<Insert Picture Here>

Demantra Diagnostic Tools

VCP PMO
Feb 2012

Agenda
This presentation summarizes some of the key diagnostic tools
that can be utilized when dealing with various Demantra errors
and problems.
Focus on the following areas:

Installation
Data model
Data loading (EP_Load and Integration Interfaces)
DB deadlocks and internal errors
Web Application troubleshooting:
Application memory errors
WS performance
Engine errors

2009 Oracle Corporation Proprietary and Confidential

Installation
The installer includes a configurable log file that can trace the
install process.

The installer needs to be configured (first panel of the installer),

otherwise no log will be written at all (in the past by default the
log was written to the $temp$ directory this is not the case any
more).
When having installation errors there are few traces that can be
analyzed:
Installer log
Screenshots of the selections in the different Installer panels.
Review db_audit_log table for object changes made by the
upgrade
DB_Exception_Log table
2009 Oracle Corporation Proprietary and Confidential

Data Model:
When having data model errors there are few traces that can be
analyzed:
An Export of the Data model
Review db_audit_log table for object changes made by the
upgrade.
LOG_DATA_MODEL table (an automatic LOG_IT log).

2009 Oracle Corporation Proprietary and Confidential

Data Loading:
Data loading includes 2 aspects Ep_Load stored procedures and
Integration Interface.
Each one of them can be monitored both for tracking errors as well
as tracking performance problems.

2009 Oracle Corporation Proprietary and Confidential

Data Loading Ep_Load:

Ep_load Errors:
Enable LOG_IT process on relevant Demantra stored
procedures as defined in LOG_IT_PARAMS table
DB_Exception_Log.

Ep_Load performance:
Count of data from the staging tables.
DB_Params table Ep_Load performance related parameters
Execute Call_DM_Build_Procedures after changing
parallelization parameters in DB_Params

DB HW definitions.
System Memory Disk IO and CPU Utilization monitoring- Either
system performance monitor or Linux top events
AWR /ADDM reports.

2009 Oracle Corporation Proprietary and Confidential

Data Loading Integration:

Integration Errors:
Collaborator & Integration log (Enable sql logging: appserver.sql,
appserver.update.sql)
Concurrent request logs (collaborator.login.user)
apscheck.sql (Note: 246150.1)
Integration Performance:
Count of data from the staging tables.
APS_Params Integration Interface related parameters
DB HW definitions.
system memory disk IO and CPU utilization monitoring- either
system performance monitor or Linux top events.
AWR /ADDM reports.

2009 Oracle Corporation Proprietary and Confidential

DB deadlock/ internal database error:

Collaborator Log
Database \udump logs (USER_DUMP_DEST specifies the pathname
for a directory where the DB server will write debugging trace files on
behalf of a user process.)

Enable LOG_IT process on selected Demantra stored

procedures as defined in LOG_IT_Params table.

2009 Oracle Corporation Proprietary and Confidential

Web Application troubleshooting:

Demantra Web application includes 2 components:
Application Server Demantra Web application deployment
inside a Web server.
Client User side browser and resources (CPU, Memory)
running SUN Java Plug-in.
They are connected by a Network (LAN, WAN).

For better troubleshooting we need to understand

where we have a problem is it on the Server side/
Client side / Both ?

2009 Oracle Corporation Proprietary and Confidential

Application out of memory errors:

Identify where the problem is by using logs:
Collaborator log server side log.
Plug-in log Client side log.

Server side issues:

Heap dump from the webserver
Thread dump from the Webserver (in case of application freeze or
slow down)
Application server memory configuration (-Xmx)
Count of Level members (need level member caching?)

Client side issues:

Heap dump from the Java plug-in.
Thread dump from the Java plug-in
Plug-in memory configuration (-Xmx)

2009 Oracle Corporation Proprietary and Confidential

10

WS performance and errors:

Identify where the problem is:
Server side AWR /ADDM reports, DB side CPU and Memory.
Application server CPU and configuration, APS_PARAMS relevant
parameters.
Client side - CPU utilization (and for how long).

Server side issues:

Demantra Index Advisor report.
Thread dump (in case of application freeze or slow down).
System Memory Disk IO and CPU Utilization monitoring- either
system performance monitor or Linux top events.
DB HW definitions.
Number of concurrent users at the time (concurrent user log).

Client side issues:

Plug-in Xmx setting
WS wizard panels screen shots.
Client side log If requested (Number of combinations in the WS).
2009 Oracle Corporation Proprietary and Confidential

11

Engine errors:
Always provide a full Engine log (manager and engine) with
relevant logging option.
DB errors:
Engine log with Sql logging depends on where the DB error is Engine
manager or engine2k.

Processor error:
Engine log with DebugPro logging groups 100/110.

Engine Out of memory errors:

No of Tasks
Count of Active Combinations (Select Count(1) from MDP_Matrix where
prediction_status = 1).

2009 Oracle Corporation Proprietary and Confidential

12

Engine errors cont:

Engine not running:
On Linux Application server log

Engine performance:
AWR report
Check whether CPU Hyper threading enabled on Engine servers Disable if
its enabled
Engine Blades/Server + DB HW definitions.
DB/Engine Blades Memory Disk IO and CPU Utilization Monitoring - Either
system performance monitor or Linux top events.

For more information check Oracle Support note 800030.1

Demantra Engine Troubleshooting Presentation (April 6th
2011).

2009 Oracle Corporation Proprietary and Confidential

13

Appedix
Configure auditing capabilities for BM configuration
changes:
Client expression Evaluator

2009 Oracle Corporation Proprietary and Confidential

14

Configure auditing capabilities for BM

configuration changes:
This is a customization enhancement introduced by DEV
application team in a white paper (Implementing Demantra Audit
log)
Currently captures Inserts/ Updates on Sessions/
Computed_fields tables by a trigger based process
The mechanism and idea can easily be enhanced for further
Metadata audit opportunities
Difference from BAL diff this mechanism captures all the
changes, not just how the latest change differs between
schemas

2009 Oracle Corporation Proprietary and Confidential

15

Configure auditing capabilities for BM

configuration changes (cont):
The logs are kept in a separate table (AUDIT_LOG ), so no
impact on the out of the box schema.
AUDIT_LOG table includes the following information (columns):

log_date
session_id
os_user
host_name
module
user_id
object_type
object_name
old_value
new_value
Message

2009 Oracle Corporation Proprietary and Confidential

16

Client expression Evaluator:

Can be used for troubleshooting client expressions that did not
pass verification in the BM
Supported from 7.3 Version and above only.

2009 Oracle Corporation Proprietary and Confidential

17

Client expression Evaluator:

New file
created

2009 Oracle Corporation Proprietary and Confidential

18

Client expression Evaluator:

SQL Parser Statement For computed_name
'perc_trade_spend' :
SELECT 0 as total_plan_spend, 0 as ttl_plan_spend_ex_slot,
0 as total_event_dollars, CAST(SYSTIMESTAMP as DATE) as
sdate From Dual

Evaluating Client Expression

Non Translated expression before evaluate:
if(isnull(( Evt Spend exS / Ttl Evt Values )),0, ( Evt Spend exS /
Ttl Evt Value ) )
Translated expression for evaluate:
if(isnull((ttl_plan_spend_ex_slot/ttlevtvalues)),0,(ttl_plan_spen
d_ex_slot/total_event_dollars))
Invalid Client Expression

2009 Oracle Corporation Proprietary and Confidential

19

Key Oracle Demantra Support Notes/Web Cast

Materials

Information Center: Oracle Demantra Demand Management (Doc ID 1381213.2)

Information Center: Oracle Value Chain Planning Demantra (Doc ID 1157065.1)
Oracle Demantra Documentation Library (Doc ID 443969.1)
Demantra Development Suggested Performance Advice plus Refernece doc (Doc ID 1157173.1)
Trouble Shooting the Demantra Worksheet Performance 7.3.0.1 and Beyond! UPDATED! (Doc ID:
470852.1)
Demantra Development Suggested Performance Advice Plus Reference Docs (Doc ID: 1157173.1)
Rebuild_Schema procedure - One of the Keys to Improved Demantra performance (Doc ID:
860576.1)
Troubleshooting EBS - DEMANTRA : User Synchronization does not work (Doc ID: 602530.1)
Step by Step Troubleshooting of the Silent Installer (Doc ID: 947322.1)
Additional Configurations needed when Running Demantra on Oracle 11g Databases (Doc ID:
1064995.1)
ORA-24247 During Shipment and Booking collections (Doc ID 1070104.1)
How to Analyze Demantra Forecast Engine Performance (Processing Time) Issues / Demantra
Engine is Slow (Doc ID 863025.1)
Improving Demantra Engine Performance of Resetting Previous Forecast For Inactive
Combinations (Doc ID 1270690.1)
How to check Demantra Performance. Troubleshooting. Database. Maintenance. Parameters.
Java. Worksheet. Engine (Doc ID 1356886.1)
Web Cast Materials (Doc ID: 800030.1)
Worksheet Design & Best Practices
Demantra Post Production Support Common Issues, Troubleshooting Tips, and Maintaining
Your Instance
Demantra Engine Troubleshooting

2009 Oracle Corporation Proprietary and Confidential

20

 2009 Oracle Corporation Proprietary and Confidential

21

Reviewing an AWR
Introduction

Introduced in Oracle 10g

Similar to a Statspack report used in prior versions
Runs as a background process
Takes snapshot of statistics and SQLs every hour.
You can manually take AWR snapshots for any operation (before and after).

2009 Oracle Corporation Proprietary and Confidential

22

Reviewing an AWR
Important report sections
Report has a DB overview section
Lists the Top 5 waits Important for troubleshooting performance bottlenecks

DB File type waits Physical IO related

Buffer type waits Logical IO related
Log Type waits Redo log related
PX Parallel query related
GC Global Cache related in a RAC environment

Longest wait times can be used for troubleshooting

Watch for DB sequential read related waits as these are single threaded reads of indexes

2009 Oracle Corporation Proprietary and Confidential

23

Reviewing an AWR
SQL related sections
SQL Time Model statistics are important for SQL tuning

Total Elapsed Time Includes all IO waits

Total CPU time Type of joins (which depend on PK, Index usage)
Total Buffer Gets Logical IO
Total Disk reads Physical IO
Total Executions Indicates SQL executions and size of worksheet
Total Parse calls SQL, Bind variable issues

For better performance, you want DB CPU time to be high compared to the total SQL processing time
Time may be spent in parsing and IO, reducing these would typically be the primary focus of the tuning
exercise, which indicate IO issues
Efficiencies listed are typically close to 100%, with decent memory settings

2009 Oracle Corporation Proprietary and Confidential

24

Reviewing an AWR
Verify Memory Settings
PGA_Aggregate_Target
SGA_Target, SGA_MAX_SIZE

Efficiencies listed are typically close to 100%, with decent memory settings
Buffer Pool statistics (default, 16k) show, how efficiently buffer cache is being used - Can
be used for sizing
Low Hits indicate relevant memory settings need to be increased

2009 Oracle Corporation Proprietary and Confidential

25

Configure Installer log:

Configure log

2009 Oracle Corporation Proprietary and Confidential

26

Exporting a Data model

Export

2009 Oracle Corporation Proprietary and Confidential

27

LOG_IT:
LOG_IT is a logging mechanism for Demantra PL/SQL database
procedure code. It is analogous to log4j in Java.
LOG_IT can be used to trace a procedure flow, show variable
values, and record performance timing without having to run a
debugger.
LOG_IT is only available for a limited number of procedures (like
CHAINING , PROPORT,SIMULATION and more) but the list
keeps growing.
The available procedures are listed in the LOG_IT_PARAMS table.

2009 Oracle Corporation Proprietary and Confidential

28

LOG_IT (cont):
How to use LOG_IT:
The customer / support can enable the logging and the contents
of the log table.
Some important points to remember:
It uses a sequential key to write logs so you know exactly the order of the log.
Each main procedure writes to one log table. ie PROPORT writes to the
LOG_PROPORT table.
Sub procedures can be set up to write to the same log table.
INSERT_UNITS and DYN_INSERT_UNITS_BRANCH_ID both write to
LOG_INSERT_UNITS.
Log tables are usually overwritting each run (truncated).
Logging is enabled by setting the LOGGING_LEVEL to be greater than 0 for
a specific procedure in the LOG_IT_PARAMS table.

2009 Oracle Corporation Proprietary and Confidential

29

LOG_IT (cont):
Logging levels:

0 = No logging
1 = Procedure and sub procedure 'start' and 'end' logs. (SP,EP)
2 = Procedure progress markers (labels) (M)
3 = Procedure data , values held in specific variables (D) (also
temp procedures and tables are not dropped)
4 = Sub Procedure progress markers (labels) (M)
5 = Sub Procedure data , values held in specific variables (D)
6 = Procedure Data, values from with in loops (L)
7 = Sub procedure progress markers inside loops (M)
8 = Sub procedure data from inside loops (L)

2009 Oracle Corporation Proprietary and Confidential

30

LOG_IT (cont):

2009 Oracle Corporation Proprietary and Confidential

31

LOG_IT (cont):
The logging is enabled by setting the LOGGING_LEVEL above 0
for a specific procedure in the LOG_IT_PARAMS table. When
you commit the change to LOG_IT_PARAMS a trigger re-writes
and compiles the procedure LOG_IT which then contains a list of
procedures that it will process.
Note: before 7.3.1.1 -Only one LOG_IT_PARAMS row should be
changed between each commit because of the way the trigger
works.
Note- LOG_IT has some performance impact on the procedures it
logs, it should not be used at a normal production mode (all log
levels should be 0 for runtime operations).

2009 Oracle Corporation Proprietary and Confidential

32

Enabling client side logs:

Traditionally, client side logs are configured via the
clientLogConf.lcf file.
Some Drawbacks:
User would have to restart the client (logout/login) in order for changes to
take effect.
All clients which would be connected to that specific Demantra server would
be exposed to the same level of verbosity.

From 7.3.0, the Client Log Categories Editor was introduced to

enable client side logging for a specific client session.
The Client Log Categories Editor loads the ClientLogConf.lcf
upon the initialization of the PartnerApplet.
The editor dialog receives the properties object as parameter to
display all the log categories listed currently in the configuration
file.
2009 Oracle Corporation Proprietary and Confidential

33

Enabling client side logs (cont):

2009 Oracle Corporation Proprietary and Confidential

34

Enabling client side logs (cont):

To activate the Editor user needs to log to the WS applet
(Partner) and press the CTRL+ALT+L button combination in the
main application screen.
If a WS is open in the Applet click on the 'Open' button to select a
worksheet and close the "Open Worksheet" dialog without choosing any
worksheet.
After that press the CTRL+ALT+L .

The editor has 4 buttons which are I18N ready:

Apply - sets the selected Log categories to the selected Log Levels and
close the dialog
Cancel - closes the dialog without doing anything
Undo Changes - resets the state of selections in the dialog to the
last saved state
Config file Settings - resets all the selections in the Dialog to
the original settings of the configuration file

2009 Oracle Corporation Proprietary and Confidential

35