Weblogic Diag Sg1

Oracle WebLogic Server 11g:
Diagnostics and Troubleshooting
Volume I • Student Guide
D61523GC20
Edition 2.0
May 2011
D72553
Author Copyright © 2011, Oracle and/or it affiliates. All rights reserved.
Bill Bell Disclaimer
This document contains proprietary y information and is protected by

y copyright
y g and
Technical Contributors other intellectual property laws. You may copy and print this document solely for your
own use in an Oracle training course. The document may not be modified or altered
and Reviewers in any way. Except where your use constitutes "fair use" under copyright law, you
Will Lyons may not use, share, download, upload, copy, print, display, perform, reproduce,
publish, license, post, transmit, or distribute this document in whole or in part without
TJ Palazzolo the express authorization of Oracle.
Serge Moiseev
The information contained in this document is subject to change without notice. If you
find any problems in the document, please report them in writing to: Oracle University,
500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
Editors warranted
t d tto be
b error-free.
f
Richard Wallis
Restricted Rights Notice
Malavika Jinka
If this documentation is delivered to the United States Government or anyone using
the documentation on behalf of the United States Government, the following notice is
Publishers applicable:
Jobi Varghese
U.S. GOVERNMENT RIGHTS
Shaik Mahaboob Basha The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or
disclose these training materials are restricted by the terms of the applicable Oracle
license agreement and/or the applicable U.S. Government contract.
Trademark Notice
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names
may be trademarks of their respective owners.
Contents
1 Course Overview
Course Objectives 1-2
Target Audience 1-3
Introductions 1-4
Course Schedule 1-5
Course Appendix 1-7
Course Practices 1-8
Classroom Guidelines 1-9
For More Information 1-10
Related Training 1-11
Oracle by Example (OBE) 1-12
2 WLST Monitoring
Objectives 2-2
WLS Domains: Review 2-3
Java Management Extension (JMX): Review 2-4
WLS MBean Hierarchies 2-5
WLS MBean Reference Documentation 2-6
Console Monitoring: Review 2-8
WebLogic Scripting Tool (WLST): Review 2-9
WLST MBean Syntax: Review 2-10
Domain Runtime 2-11
Basic Jython Syntax: Review 2-12
Basic WLST Commands 2-13
Variable Declaration 2-14
Password Management 2-15
Error Handling 2-16
File I/O 2-17
Standard Jython Libraries 2-18
WLST Example: Monitor a JMS Server 2-19
Quiz 2-20
Summary 2-23
Practice 2-1 Connecting to the Classroom Grid 2-24
Practice 2-2 Developing a Custom Monitoring Script 2-25
iii
3 Guardian
Objectives 3-2
Guardian Capabilities 3-3
Using Guardian 3-4
Guardian Architecture 3-5
Agent Installation 3-6
Collected Data 3-7
Client Installation 3-8
Guardian User Interface 3-9
Activating a Domain 3-10
Creating a Domain Inventory 3-11
Signatures and Bundles 3-12
Updating the Signature Repository 3-13
Signature Annotations 3-14
Evaluating a Domain 3-15
Evaluation Summary 3-16
Generating a Support Request 3-17
Command-Line Interface 3-18
Quiz 3-19
Summary 3-22
Practice 3-1 Using Guardian to Evaluate a Domain 3-23
4 Diagnostic Framework Essentials

Objectives 4-2
Road Map 4-3
WebLogic Diagnostic Framework (WLDF) 4-4
WLDF Architecture 4-5
WLS Logging: Review 4-6
Log Severity Thresholds 4-7
Application Logging 4-8
Server Logging Bridge 4-9
WLDF Configuration: Overview 4-10
Diagnostic Images 4-11
Capturing a Server Diagnostic Image 4-12
WLST: Downloading Diagnostic Image Files 4-13
Diagnostic Archives 4-14
Configuring Server Diagnostic Archives 4-15
Archive Retirement Policies 4-16
Archive Database Schema 4-17
Viewing Archive Contents 4-18
Creating a Diagnostic Module 4-19
iv
WLDF WLST Examples 4-20
Section Summary 4-22
Road Map 4-23
Harvester Architecture 4-24
Metric Collector Definitions 4-25
Configuring a Metric Collector 4-26
Watches and Notifications 4-28
Configuring a Watch 4-29
Watch Alarms 4-31
Configuring a JMS Notification 4-32
Configuring an Email Notification 4-33
Harvester WLST: Example 4-34
Watch WLST: Example 4-35
WLDF Sample Framework 4-36
Practice 4-1 Harvesting Diagnostic Metrics 4-38
Road Map 4-39
New Monitoring Dashboard 4-40
Viewing the Dashboard 4-41
Monitoring Dashboard Interface 4-42
Views 4-43
Built-In Views 4-44
Creating a Custom View 4-45
Metric Browser 4-46
Anatomy of a Chart 4-47
Chart and Graph Properties 4-48
Chart Styles 4-49
Current and Historical Data 4-50
Practice 4-2 Monitoring Diagnostic Metrics 4-52
Road Map 4-53
Subsystem Debugging 4-54
Console Debug Scopes 4-55
Debug Scopes: Examples 4-56
Debug Logging 4-57
WLST Debugging: Examples 4-58
Quiz 4-60
Summary 4-64
v
5 Diagnostic Instrumentation
Objectives 5-2
Road Map 5-3
Instrumentation Scenarios 5-4
Instrumentation Architecture 5-5
Monitor Actions 5-6
Application-Scoped Modules 5-8
WLS Monitor Library 5-9
Deployment Plan Review 5-11
WLDF and Deployment Plans 5-12
WLDF Deployment Plan: Example 5-13
WLDF Hot Swap 5-14
Configuring a System-Scoped Monitor 5-15
Configuring an Application-Scoped Monitor 5-17
Aspect-Oriented Programming (AOP) Concepts 5-18
Custom Monitors 5-19
Instrumentation WLST: Example 5-20
Instrumentation and Request Performance 5-21
Practice 5-1 Configuring and Monitoring Diagnostic Events 5-23
Road Map 5-24
Request Context ID 5-25
Viewing Context IDs 5-26
Request Dying 5-27
Available Dyes 5-28
Configuring a Dye Injection Monitor 5-29
Event Filtering 5-30
Configuring Dye Masks 5-31
Event Throttling 5-32
Configuring Throttle Properties 5-33
Quiz 5-35
Summary 5-38
Practice 5-2 Tracing a Client Request 5-39
6 JVM Diagnostics
Objectives 6-2
Road Map 6-3
Basic Java Concepts 6-4
Java Virtual Machine (JVM): Review 6-5
Oracle JVM Support 6-6
vi
JVM Recommendations 6-7
JVM Memory 6-8
Garbage Collection 6-9
Sun HotSpot Garbage Collection 6-10
Garbage Collection (GC) Types 6-11
Setting WLS JVM Arguments 6-12
Basic Sun JVM Arguments 6-13
JRockit Garbage Collection 6-14
Basic JRockit JVM Arguments 6-15
Out of Memory 6-16
Out-of-Memory Response 6-17
Memory Leak 6-18
JVM Crash 6-19
JVM Error Log 6-20
Road Map 6-22
JVM Tool Varieties 6-23
Java Stack Trace 6-24
Java Thread Dump: Overview 6-25
Thread Dump Signal 6-26
JVM Crash Actions 6-27
Verbose GC 6-28
Sun JVM Profiler Agent 6-29
Sun JVM Diagnostic Tools: Overview 6-30
Sun Diagnostic Tools: Examples 6-31
JVisualVM 6-33
Using JVisualVM 6-34
Practice 6-1 Troubleshooting a Running JVM 6-37
Road Map 6-38
Console JVM Monitoring 6-39
JVM WLST: Example 6-40
WLS Low Memory Detection 6-41
Configuring Low Memory Detection 6-42
Road Map 6-44
JRockit Diagnostic Tools: Overview 6-45
JRockit Diagnostic Tools: Examples 6-46
Management Communication 6-47
JRockit Mission Control (JRMC) 6-48
JRockit Discovery Protocol (JDP) 6-49
vii
JVM Browser 6-50
Management Console: Features 6-51
Management Console: General > Overview 6-52
Management Console: Runtime > Threads 6-53
Management Console: MBeans > Triggers 6-54
JRockit Flight Recorder (JFR) 6-55
Integration of JRockit Flight Recorder and WLDF 6-56
Starting the Flight Recorder from JRMC 6-57
Flight Recorder Output 6-58
General > Overview 6-59
Memory: Object Statistics 6-60
Code > Overview 6-61
Memory Leak Detector (Memleak): Features 6-62
Memleak: Trend Tab 6-63
Memleak: Type Graph 6-64
Quiz 6-66
Summary 6-70
Practice 6-2 Troubleshooting Applications on JRockit 6-71
7 Troubleshooting Java Applications

Objectives 7-2
Java Exception-Handling Concepts 7-3
Exception Chains 7-4
“Class Not Found” Errors 7-5
Class Cast Errors 7-6
Classpath: Review 7-7
WebLogic Start Script: Review 7-8
Viewing the WLS Classpath 7-9
Manifest Files and the Classpath 7-10
Domain Libraries 7-11
Java Class Loaders 7-12
Searching Class Loaders 7-13
Searching Class Loaders: Example 7-14
Default WLS Class Loader Hierarchy 7-15
Java EE Packaging: Review 7-16
Prefer Web Application Classes 7-17
Prefer Enterprise Application Classes 7-18
Client Library Errors 7-19
Null Pointer Errors 7-20
Stack Overflow Errors 7-21
viii
Too Many Open Files Errors 7-22
Quiz 7-23
Summary 7-25
Practice 7-1 Investigating Classpath Problems 7-26
8 Troubleshooting Servers
Objectives 8-2
Road Map 8-3
WLS Message Catalog: Review 8-4
Server Startup Errors 8-5
Boot Identity Errors 8-6
WLS Native Libraries 8-7
Setting the Native Library Path 8-8
Causes of Unresponsive Servers 8-9
WLS Threading Architecture 8-10
Execute Thread State 8-11
Work Managers 8-12
Work Manager Architecture 8-13
Creating a Work Manager 8-14
Creating and Using a Request Class 8-15
Assigning Work Managers to Applications 8-16
Monitoring a Server Thread Pool 8-17
Monitoring Individual Server Threads 8-18
Server Monitoring: WLST Examples 8-19
Server WLDF Image Contents 8-20
Java Deadlock Concepts 8-21
Thread Analysis 8-22
Lock Chains 8-23
Stuck Thread Detection 8-24
Overload Protection 8-25
Configuring Overload Protection 8-26
Practice 8-1 Investigating Server Problems 8-28
Road Map 8-29
WLS Deployment: Review 8-30
Deployment Errors 8-32
Application Staging 8-33
Deployment Memory Errors 8-34
Shared Library: Review 8-35
Library Errors 8-36
Deployment Debug Flags 8-37
ix
Application Error Handling 8-38
Application Monitoring: Review 8-39
Application Monitoring: WLST Examples 8-40
Quiz 8-42
Summary 8-45
9 Troubleshooting JDBC
Objectives 9-2
JDBC: Review 9-3
Data Sources: Review 9-4
JDBC Management: WLST Examples 9-5
JDBC Runtime Attributes 9-6
JDBC Monitoring: WLST Examples 9-7
JDBC WLDF Image Contents 9-8
JDBC WLDF Monitor: Review 9-9
Data Source Diagnostic Profiling 9-10
Configuring Diagnostic Profiling 9-11
JDBC Debug Flags 9-12
Other JDBC Debugging Tools 9-13
Common Configuration Errors 9-14
Configuration Error Examples 9-15
Insufficient Connection Errors 9-16
Connection Leaks 9-17
Database Cursor Considerations 9-18
Common Connection Errors 9-19
Statement Timeout 9-20
Data Sources and Database Availability 9-21
Retry Frequency and Login Timeout 9-22
Connection Testing: Review 9-23
Testing Trusted Connections 9-24
Firewall Considerations 9-25
Multi Data Source: Overview 9-26
Multi Data Source: Architecture 9-27
Java Persistence API (JPA): Overview 9-28
JPA Configuration: Overview 9-29
Troubleshooting JPA: Overview 9-30
Quiz 9-31
Summary 9-34
Practice 9-1 Investigating JDBC Problems 9-35
x
10 Troubleshooting JMS
Objectives 10-2
JMS: Review 10-3
WebLogic JMS Configuration: Review 10-5
JMS Transactions: Review 10-7
JMS Management: Overview 10-8
Console JMS Management 10-9
JMS Management: WLST Examples 10-10
JMS Runtime MBean Hierarchy 10-11
JMS Monitoring: WLST Examples 10-12
JMS Diagnostic Image Contents 10-13
JMS Message Logging 10-14
Configuring JMS Logging 10-15
JMS Debug Flags 10-16
Message Type Considerations 10-17
JMS Client Libraries 10-20
Out-of-Memory Errors and Quotas 10-21
Configuring a JMS Server Quota 10-22
Creating a Destination Quota 10-23
Message Paging 10-24
Too Many Pending Messages 10-25
Quota Blocking Policies 10-26
Thresholds and Flow Control 10-27
Configuring Thresholds 10-28
Tuning Flow Control 10-29
Lost Messages 10-30
Time to Live (TTL) 10-31
Expiration Policies 10-32
Delivery Mode 10-33
Message Redelivery 10-34
Time to Deliver (TTD) 10-35
Durable Subscriber Review 10-36
Monitoring and Managing Subscriptions 10-37
Duplicate Messages 10-38
Poison Messages 10-39
Consumer Acknowledgement Modes 10-40
Messages Out of Sequence 10-41
Unit of Order (UOO): Overview 10-42
Unit of Work (UOW): Overview 10-43
Message-Driven Beans (MDBs): Review 10-44
xi
MDB Capabilities 10-45
MDB Runtime Attributes 10-46
MDB Diagnostics and Debugging 10-47
Quiz 10-48
Summary 10-51
Practice 10-1 Investigating JMS Problems 10-52
11 Troubleshooting Security
Objectives 11-2
Road Map 11-3
Secure Sockets Layer (SSL): Review 11-4
SSL Communication: Review 11-5
WebLogic SSL Scenarios 11-6
Proxy Server SSL Scenarios 11-8
Keystore: Review 11-9
Trust Keystores 11-10
Keytool: Review 11-11
WebLogic SSL Support 11-13
SSL Configuration: Review 11-14
Restarting SSL 11-16
SSL Debug Flags 11-17
SSL Handshake Trace 11-18
Other SSL Traces 11-19
Invalid Format or Cipher Errors 11-20
Certificate Validation Errors 11-21
Host Name Verification Errors 11-22
Certificate Chains 11-23
WLS Chain Validation Utility 11-24
Missing Constraint or Policy Errors 11-25
Practice 11-1 Investigating SSL Problems 11-27
Road Map 11-28
Security Realm: Review 11-29
Security Provider Stores 11-30
Some Security Providers 11-31
Embedded LDAP: Review 11-32
Embedded LDAP Backups 11-33
Embedded LDAP Synchronization Issues 11-34
Viewing Embedded LDAP Contents 11-35
LDAP Concepts 11-36
LDAP Structure 11-37
xii
LDAP Search Operations 11-38
Resetting Admin Password in Embedded LDAP 11-39
Database Store Cache Synchronization Issues 11-40
Auditing Provider 11-41
Security Audit Events 11-42
Configuring the Auditing Provider 11-43
Realm Debug Flags 11-44
Typical Authentication Trace 11-45
Typical Role Mapping Trace 11-47
Typical Authorization Trace 11-48
LDAP Trace Log 11-49
Authentication Provider Control Flags 11-50
External LDAP Authentication Providers 11-52
LDAP Provider Configuration: Overview 11-53
Common LDAP Issues 11-56
Quiz 11-58
Summary 11-62
Practice 11-2 Investigating Security Realm Problems 11-63
12 Troubleshooting Node Manager

Objectives 12-2
Node Manager (NM): Review 12-3
Node Manager Types: Review 12-4
Node Manager Configuration: Review 12-5
Basic Java Node Manager Properties 12-6
Java Node Manager Logging 12-7
Java Node Manager Availability 12-8
Basic Script Node Manager Interface 12-9
Node Manager Server Start Parameters 12-11
Configuring Server Start Parameters 12-13
Monitoring Node Managers 12-14
Node Manager: WLST Examples 12-15
Generating Template Properties for Java NM 12-17
Node Manager Authentication 12-18
Configuring Node Manager Credentials 12-19
Node Manager Trusted Domains 12-20
Machine Enrollment 12-21
Server Boot Identity 12-22
Configuring Node Manager SSL 12-23
xiii
Quiz 12-24
Summary 12-26
Practice 12-1 Investigating Node Manager Problems 12-27
13 Troubleshooting Clusters
Objectives 13-2
Road Map 13-3
Cluster Review 13-4
Proxy Plug-in Review 13-5
Obtaining and Using Plug-Ins 13-6
Oracle HTTP Server (OHS) Review 13-7
Oracle Process Manager and Notification Server (OPMN) Review 13-9
OPMNCTL Examples 13-10
OHS Logs 13-11
Plug-in Configuration Review 13-12
Basic Plug-in Parameters 13-13
Proxy Connection Architecture 13-14
Dynamic Server List 13-16
Connection Parameters 13-17
Common Connectivity Issues 13-18
Proxy SSL Issues 13-19
Proxy Debug Page 13-20
Proxy Debug Log 13-22
Typical Proxy Trace 13-23
Practice 13-1 Investigating Proxy Problems 13-25
Road Map 13-26
Cluster Communication Review 13-27
Unicast Architecture 13-28
Session Management Review 13-29
Session Persistence Review 13-30
In-Memory Replication Review 13-31
Cluster Monitoring WLST Examples 13-32
Session Monitoring WLST Examples 13-33
Session Monitoring Attribute 13-34
Session Instrumentation 13-35
Cluster Debug Flags 13-36
Typical Cluster Heartbeat Trace 13-37
Typical Replication Trace: Primary 13-38
Typical Replication Trace: Secondary 13-39
Common Replication Issues 13-40
xiv
HttpSession API Overview 13-41
Serialization Overview 13-42
Serialization Debug Messages 13-43
Quiz 13-45
Lesson Summary 13-48
Practice 13-2 Investigating Cluster Replication Problems 13-49
A WebLogic SNMP
Simple Network Management Protocol (SNMP) A-2
SNMP Architecture A-3
Object Identifier (OID) A-4
Management Information Base (MIB) A-5
WLS MIB and OIDs A-6
Common SNMP Message Types A-7
WLS SNMP Architecture A-8
Creating an SNMP Agent A-10
Configuring an SNMP Agent A-11
SNMP Channels A-12
WLS SNMP Notifications A-13
Creating Trap Monitors A-14
Creating Trap Destinations A-15
SNMP Security A-16
Configuring Agent Security A-17
Configuring SNMP V3 Credentials A-18
Configuring Trap Destination Security A-19
WLS SNMP Utility A-20
xv
Course Overview
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Course Objectives
After completing this course, you should be able to:

• Monitor servers using the WebLogic Scripting Tool (WLST)
• Analyze a domain configuration by using Guardian
• Explain the capabilities of the WebLogic diagnostics
framework
• Configure diagnostic collectors and notifications
• Associate diagnostic monitors with custom applications
• Monitor memory usage with JRockit Mission Control
• Configure verbose logging for WebLogic subsystems
• Troubleshoot basic virtual machine, application server, and
cluster issues
• Troubleshoot JDBC and JMS problems
Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 2

Target Audience
• This course is intended for experienced WebLogic Server

administrators or those who have completed WebLogic
Server: Administration Essentials.
• Prerequisite skills include basic:
– Administration Console navigation
– WLST commands
– JDBC and JMS configuration
– Application deployment
If you are concerned about whether your experience fulfills the course prerequisites, ask the
instructor.

Introductions
Introduce yourself.
Tell us about:
• Your company and role
• Your experience with WebLogic Server
• Any previous Oracle product experience

Course Schedule
Day Lesson
1 AM WLST Monitoring
Guardian
PM Diagnostic Framework Essentials
Diagnostic Instrumentation
2 AM Diagnostic Instrumentation (continued)
JVM Diagnostics
PM JVM Diagnostics (continued)
Troubleshooting Java Applications
Troubleshooting Servers
The class schedule might vary according to the pace of the class. The instructor will provide
updates.

Course Schedule
Day Lesson
3 AM Troubleshooting JDBC
PM Troubleshooting JMS
Troubleshooting Security
4 AM Troubleshooting Security (continued)
Troubleshooting Node Manager
PM Troubleshooting Clusters

Course Appendix
Appendix A: WLS SNMP
Appendix A is intended to be additional reference material and will only be presented if time
permits.

Course Practices
• Each topic is reinforced with a hands-on exercise.

• Most exercises include a scripted solution to aid any
students who fall behind.

Classroom Guidelines
• The instructor starts each session at the scheduled time.

• Do ask questions, but please be respectful of the topic at
hand and the interests of other students.
• Ensure that cell phones and pagers are silent.
We hope that these guidelines help the class proceed smoothly and enable you to get the
maximum benefit from the course.

For More Information
Topic Website
Education and Training http://www.oracle.com/education
Product Documentation http://www.oracle.com/technetwork/indexes/documentation
Product Downloads http://www.oracle.com/technetwork/indexes/downloads
Product Articles http://www.oracle.com/technetwork/articles
Product Support http://www.oracle.com/support/
Product Forums http://forums.oracle.com
Product Tutorials http://www.oracle.com/technetwork/tutorials
Sample Code http://samplecode.oracle.com
After you complete the course, Oracle provides a variety of channels for developers and
administrators to access additional information.

Related Training
Course Title
Oracle WebLogic Server: Advanced Administration
Oracle WebLogic Server: Monitor and Tune Performance

Oracle by Example (OBE)
OBE is a series of free online tutorials that cover specific

product features with step-by-step instructions.
The Oracle by Example (OBE) series provides hands-on, step-by-step instructions on how to
implement various technology solutions to business problems. OBE solutions are built for
practical real-world situations, allowing you to gain valuable hands-on experience as well as
to use the presented solutions as the foundation for production implementation, dramatically
reducing time to deployment.

WLST Monitoring

Objectives
After completing this lesson, you should be able to:

• Describe the relationship between JMX and WLST
• Compare the WLS MBean hierarchies
• Use some basic Jython and WLST commands
• Work with files, environment variables, and exceptions in
Jython

WLS Domains: Review
Machine Machine Machine
Domain B
Domain A Domain A Domain B
Admin
Server Server server
Admin
server Server
Server
Product Product
installation Product installation installation
An Oracle WebLogic Server administration domain is a logically related group of Oracle

WebLogic Server resources. Domains include a special Oracle WebLogic Server instance
called the Administration Server, which is the central point from which you configure and
manage all resources in the domain. Usually, you configure a domain to include additional
Oracle WebLogic Server instances called Managed Servers. You deploy Web applications,
EJBs, Web Services, and other resources onto the Managed Servers and use the
Administration Server for configuration and management purposes only.
You can use a single Oracle WebLogic Server installation to create and run multiple domains,
or you can use multiple installations to run a single domain. How you organize your Oracle
WebLogic Server installations into domains depends on your business needs. You can define
multiple domains based on different system administrators’ responsibilities, application
boundaries, or geographical locations of the machines on which servers run. Conversely, you
might decide to use a single domain to centralize all Oracle WebLogic Server administration
activities.
For development or test environments, you can create a simple domain that consists of a
single server instance. This single instance acts as an Administration Server and hosts the
applications that you are developing.

Java Management Extension (JMX): Review
• All WLS configuration, management, and monitoring is

accomplished via JMX, which provides a standard Java
API for accessing management “objects” or MBeans.
• MBeans:
– Are organized hierarchically
– Can be accessed from remote systems
– On the Administration Server act as proxies to MBeans on
managed servers
Admin server Managed server
JMX client JMX client
MBeans MBeans
A managed bean (MBean) is a Java object that provides a Java Management Extensions
(JMX) interface. JMX is the Java EE solution for monitoring and managing resources on a
network. Like SNMP and other management standards, JMX is a public specification and
many vendors of commonly used monitoring products support it. WebLogic Server provides a
set of MBeans that you can use to configure, monitor, and manage WebLogic Server
resources through JMX.
Each server in the domain has its own copy of the domain’s configuration documents (which
consist of a config.xml file and subsidiary files). During a server’s startup cycle, it contacts
the Administration Server to update its configuration files with any changes that occurred
while it was shut down. Then it instantiates configuration MBeans to represent the data in the
configuration documents.

WLS MBean Hierarchies
• Every WebLogic Server hosts several different MBean

containers or “servers.”
• The Server Configuration hierarchy lets you view the
current configuration settings for resources.
• The Server Runtime hierarchy:
– Lets you monitor resource statistics
– Provides historical data since last server restart
• Administration servers also include an Edit Configuration
hierarchy to create, update, and delete configuration
MBeans for all servers in the domain.
All WebLogic Server MBeans can be organized into one of several general types. Runtime
MBeans contain information about the runtime state of a server and its resources. They
generally contain only data about the current state of a server or resource, and they do not
persist this data. When you shut down a server instance, all runtime statistics and metrics
from the runtime MBeans are destroyed. Configuration MBeans contain information about the
configuration of servers and resources. They represent the information that is stored in the
domain’s XML configuration documents.
All edits to MBean attributes occur within the context of an edit session, and only one edit
session can be active at a time within each WebLogic Server domain. Changing an MBean
attribute or creating a new MBean updates the in-memory hierarchy of pending configuration
MBeans. If you end your edit session before saving these changes, the unsaved changes will
be discarded. When you activate your changes, WebLogic Server copies the saved, pending
configuration files to all servers in the domain. Each server evaluates the changes and
indicates whether it can consume them. If it can, it updates its active configuration files and in-
memory hierarchy of configuration MBeans.

WLS MBean Reference Documentation
• Each MBean consists of attributes and operations.

• All MBeans have a name attribute.
• Attributes may reference other MBean objects (children).
1
As an alternative to using WebLogic Scripting Tool (WLST) to interactively browse a server’s

runtime MBeans, you can use the online WebLogic Server MBean Reference guide.
1. Select the Runtime MBeans category in the tree found on the left. Select an MBean.
2. Then select a link to see information about that MBean.
When connecting to a managed server, the root of the runtime hierarchy is
ServerRuntimeMBean. From this MBean, you can then obtain references to all resources
targeted to this server.
Configuration MBeans expose attributes and operations for configuring WebLogic Server
resources. This reference document organizes configuration MBeans into the following
categories:
• Domain Configuration MBeans, which configure clusters, server instances, containers
within the server (EJB and Servlet), and other services.
• System Module MBeans, which configure independently deployed modules that are
available as resources to all applications. These include JDBC, JMS, and diagnostic
modules.
• Security MBeans, which configure WebLogic security providers and provide security-
related utilities such as listing and managing lockouts on user accounts.

WLS MBean Reference Documentation
Links to child
MBean types
MBeans, like any Java objects, consist of attributes, which can be simple types such as
Strings or doubles, or they can be other MBeans. In other words, MBeans can contain other
MBeans and define a hierarchy of management objects. MBeans provide standard methods
for managing these relationships, which are typically of the form createXYZ, destroyXYZ, or
lookupXYZ, where XYZ is the type of child MBean. For example, the DomainMBean includes
an operation named lookupServer(String name), which takes as a parameter the name
that was used to create a server instance.
MBean interface can also include other arbitrary system operations that JMX clients can
invoke remotely. For example, JMS destinations can be administratively paused, resumed,
purged, and so on. Note that each attribute also has an implicit operation named get<name>,
where <name> is the attribute name. For configuration MBeans that are not read-only, there
may also be a matching set<name> operation.

Console Monitoring: Review
View console Help to determine

corresponding MBeans.
Monitoring
tabs
Every time that a resource, service, or application object can be monitored, a Monitoring tab is
available in the console for that object. Clicking it shows you the available monitoring
information for the selected object. Moreover, when the monitoring page shows information in
tabular format, you may change the way that the information is displayed. To do this, click
“Customize this table” and choose which columns to display and on what columns to sort the
table.
One way to determine which MBean types and attributes correspond to the displayed
columns is by using the context-sensitive Help link found at the top of the console. For each
column, the Help page provides a description and the corresponding MBean information.
You cannot monitor the activity of one domain through another domain. For example, you
cannot open the Administration Console for domain Y and try to monitor servers within
domain Z.

WebLogic Scripting Tool (WLST): Review
• WLST provides a command-line and scripting interface for:

– Creating new WLS domains
– Retrieving and updating WLS domain configurations
– Deploying applications
– Obtaining runtime server statistics
• With WLST, navigating MBean hierarchies is similar to
navigating folders on a file system.
connect('user','password','myhost:7001')
serverRuntime() Access this server's run-time hierarchy.
jvm = getMBean('/JVMRuntime/server1')
print 'Free Heap: ' , jvm.getHeapFreePercent()
wm = getMBean('/WorkManagerRuntimes/weblogic.kernel.Default')
print 'Pending: ' , wm.getPendingRequests()
After connecting to a running server, the serverRuntime() WLST command navigates to

the last MBean to which you navigated in the runtime MBean hierarchy or to the root of the
hierarchy, ServerRuntimeMBean. This read-only hierarchy stores the runtime MBeans that
represent the server to which WLST is currently connected.
You may then use all of the standard WLST browsing commands such as cd(), get(), and
ls() to navigate the runtime MBean hierarchy for a server and to retrieve the desired
attributes.

WLST MBean Syntax: Review
Access a child MBean from a parent by using the child’s name

attribute.
JMSRuntime
Name: ServerC.jms
JMSServers: <list> JMSServerRuntime
Name: HRJMS
JMSDestinations: <list>
JMSDestinationRuntime
Name:
BenefitsJMSModule!SyncB
enefitsQueue
getMBean('/JMSRuntime/ServerC.jms/JMSServers/HRJMS/Destination
s/BenefitsJMSModule!SyncBenefitsQueue')
In the WLST file system, MBean hierarchies correspond to drives; MBean types and
instances are directories; MBean attributes and operations are files. In WLST, you traverse
the hierarchical structure of MBeans by using commands such as cd(), ls(), and pwd() in
a way similar to how you would navigate a file system in a UNIX or Windows command shell.
For example, to navigate back to a parent configuration or runtime bean, enter cd('..'). To
get back to the root bean after navigating to a bean that is deep in the hierarchy, enter
cd('/'). After navigating to an MBean instance, you interact with the MBean using WLST
commands.

Domain Runtime
• Use the serverRuntime() command to monitor a single

server.
• Alternatively, use the domainRuntime() command to
monitor the runtime hierarchy of multiple servers via the
admin server.
WLST Server
Server
Admin
WLST Server
Server
Server
Server runtime MBeans expose monitoring, runtime control, and the current configuration
settings (read-only) of a specific WebLogic Server instance.
Some runtime MBeans are only available on the Administration Server, such as several types
of security MBeans. Access these “domain-wide” MBeans by using the domain runtime
hierarchy. This MBean server also acts as a single point of access for MBeans that reside on
other managed servers within the same domain.
If your client monitors runtime MBeans for multiple servers, or if your client runs in a separate
JVM, Oracle recommends that you connect to the domain runtime MBean hierarchy on the
Administration Server instead of connecting separately to the runtime MBean hierarchy on
each server instance in the domain. Your script then only needs to construct a single URL for
connecting to the domain runtime hierarchy on the Administration Server, instead of working
with multiple address/port combinations.
Using the domain runtime hierarchy, you can also route all administrative traffic in a domain
through the Administration Server’s secured administration port. Additionally, you could use a
firewall to prevent direct connections to managed server administration ports from outside
some network boundary.

Basic Jython Syntax: Review
list = ['ab','cd','ef']
Conditional
List variable if len(list) >= 3:
expressions
for x in list:
For loop print 'length: ', len(x)
print 'done'
else: Print a number.
print 'list too small'
def doSomething(value): Function definition

i = 1
While loop while i <= 5:
if value.endswith(str(i)):
print 'invalid:' + value
String break
concatenation i += 1
Use indentation to
...
define code blocks.
doSomething('mypassword5')
Jython programs start with the first line of code that is not commented, not indented, and is
not a function definition (def keyword). Leading white space (spaces and tabs) at the
beginning of a logical line is used to compute the indentation level of the line, which in turn is
used to determine the grouping of statements. First, tabs are replaced (from left to right) by
one to eight spaces so that the total number of characters up to and including the
replacement is a multiple of eight (this is intended to be the same rule as used by UNIX). The
total number of spaces preceding the first non-blank character then determines the line’s
indentation. Indentation cannot be split over multiple physical lines by using backslashes; the
white space up to the first backslash determines the indentation.
Jython is an interpreted language. As a result, the developer is not required to declare the
variable types at design time. The Jython interpreter assigns the types at run time. However,
each type object has an application programming interface (API), and if the API of one object
is used by your code and the type has been assigned by the interpreter to another type at run
time, the interpreter throws an exception at run time. Ensure that the correct API is used
based on the type that you expect to assign to an object at run time.

Basic WLST Commands
Command Description
connect() Connect to a server by using supplied credentials.

disconnect() Disconnect from the current server.
getMBean() Retrieve the MBean object at the given location.
Navigate to the MBean at the given location and update
cd()
the current management object (cmo) global variable.
ls() Print the attributes and values of the cmo MBean.
pwd() Print the current location in the MBean hierarchy.
startEdit() Begin a new change session.
activate() Commit all changes in the current session.
deploy() (Re-)Deploy an application to servers.
undeploy() Shut down a running application on servers.
WLST enables you to navigate a hierarchy of MBeans in a similar fashion to navigating a

hierarchy of files in a file system. For example, to navigate back to a parent MBean, enter the
cd('..') command.
In the WLST file system, MBean hierarchies correspond to drives, MBean types and
instances are directories, and MBean attributes and operations are files. In WLST, you
traverse the hierarchical structure of MBeans by using commands such as cd(), ls(), and
pwd() in a way similar to how you would navigate a file system in a UNIX or Windows
command shell.
After navigating to an MBean instance, you can interact with the MBean using other WLST
commands, such as get(), set(), and invoke(). The built-in variable cmo is set to the
current MBean each time you change directory. With this variable, you can perform
operations directly against the current management object. However, an alternative approach
is the getMBean() command, which returns the management object for a given WLST path.
Unlike cd(), getMBean() does not change the current path pointed to by cmo.

Variable Declaration
• Use variables in scripts, instead of hard-coding MBean

attribute values.
• Declare variables at the top of scripts so that they can be
easily modified.
# Variable definitions
url = 'localhost:7020'
username = 'user'
password = 'password'
dsName = 'MyDS'
targetName = 'MyServer'
connect(username, password, url)

...
jdbcSystemResource = create(dsName, 'JDBCSystemResource')
targetServer = getMBean('/Servers/' + targetName)
jdbcSystemResource.addTarget(targetServer)
Parameterize your WLST scripts. Provide variables defined and initialized in a preamble at
the top of the script. This practice promotes reuse and can aid in troubleshooting.
Alternatively, you can define variables in a separate text file and import them as variables by
using the loadProperties WLST command, as in the following example:
loadProperties('c:/temp/myLoad.properties')

Password Management
To avoid using plain text credentials in your scripts, do one of

the following:
• Require them as command-line arguments.
• Prompt the user to enter them.
• Create and use an encrypted password file.
Create encrypted password file for previously used credentials:
connect(username, password, myurl)

storeUserConfig() Stored in OS user’s home
directory by default
Connect to a server using the current password file:
connect(url=myurl)
File retrieved from the
... home directory by default
The storeUserConfig() command creates a user configuration file and an associated key
file for the identity of the current user you are connected to WLS with. The user configuration
file contains an encrypted username and password. The key file contains a secret key that is
used to encrypt and decrypt the username and password. Only the key file that originally
encrypted the username and password can be used to decrypt the values. If you lose the key
file, you must create a new user configuration and key file pair. If you do not specify file
names or locations, the command stores the files in your home directory as determined by
your JVM. The location of the home directory depends on the SDK and type of operating
system on which WLST is running. The default file names are <username>-
WebLogicConfig.properties and <username>-WebLogicKey.properties.
If you do not specify credentials as part of the connect() command, and a user
configuration and a default key file exists in your home directory, then the command uses the
credentials found in those files. This option is recommended if you use WLST in script mode
because it prevents you from storing unencrypted user credentials in your scripts.
Alternatively, you can provide the specific locations and names of these files by using the
userConfigFile and userKeyFile command arguments.

Error Handling
• Before reading, updating, or deleting resources, confirm

that they exist.
• Before creating new resources, confirm that they do not
already exist.
try:
cd('/JDBCSystemResources/' + dsName)
print 'The JDBC Data Source ' + dsName + ' already exists.'
exit()
except WLSTException:
pass
jdbcSystemResource = create(dsName, 'JDBCSystemResource')
ds = getMBean('/JDBCSystemResources/' + dsName)
if ds != None:
...
The first example shows the use of a try/except block as part of creating a new server
resource, such as a JDBC data source. The try/except block is used while attempting to
navigate to a specific configuration MBean with the cd() command. If this code block
succeeds without an exception, the MBean already exists and the script is terminated. If an
exception is thrown (the expected case), the rest of the script is executed to create the new
resource. The Python “pass” statement is a required placeholder, although it performs no
actual work.
The second example shows how similar functionality can be accomplished using the
getMBean() command. Unlike the cd() command, getMBean() does not throw an
exception if the given MBean is not found.
The use of print statements to inform the user of script progress and successful completion is
also a good practice.

File I/O
• Jython includes standard file operations.

• Consider writing script output to log files instead of to the
standard output stream.
outfile = open('script.log','w')
...
outfile.write('Server State: ' + server.getState() + '\n')
outfile.close()
infile = open('data.in','r')
line = 'temp'
while len(line) != 0:
line = infile.readline()
...
Jython includes a built-in file type and a built-in function open() for creating variables of this
type. In addition to supplying the file name and path, you can optionally indicate a mode ('r',
'w', 'a', for reading, writing and appending, respectively). In append mode, all write operations
are appended to the end of any existing file contents. The mode flag can also include a 't' or
'b' suffix to indicate text or binary mode. In text mode, newline characters are automatically
converted to the current platform's line terminator, and vice versa.
The read() command reads until the end of the file or up to a specified number of bytes, and
returns a string. If a size is not indicated, the whole file is read. This command returns an
empty string if the end of the file has been reached. The readline() command is similar,
but also stops reading if a line terminator is found. Finally, the readlines() command
returns a list of strings.
The write() command writes the supplied string to the file, and the writelines()
command writes a list of strings to the file.

Standard Jython Libraries
import sys
targetServer = sys.argv[1] Command-line

timeout = sys.argv[2] arguments
...
import os Environment
variables
if os.environ.has_key('APP_HOME'):
os.rename(filename, filename + '.bak') Create or
... rename files.
import time as jythontime

Avoid collision with the
jythontime.sleep(10) weblogic.time Java package.
...
Pause the program
(seconds).
The sys module includes some general purpose global variables. The sys.argv variable is
a list that contains the command-line arguments passed to the Jython interpreter. The
expression argv[0] evaluates to the script name.
The os module provides a portable way of using operating system–dependent functionality.
For example, the os.environ variable is a map object that provides access to system
environment variables. This module also has listdir(), makedir(), rename(), and
rmdir() functions. If you simply want to read or write a file, use the built-in open() function.
If you want to manipulate system paths, see the os.path module instead.
Jython includes several modules for working with times and dates. The time module, for
example, includes a sleep() function.

WLST Example: Monitor a JMS Server
server = 'serverA'
jmsserver = 'DefaultJMSServer'
while true:
connect('user','password','myhost:7001')
serverRuntime()
jms = getMBean('/JMSRuntime/' + server

+ '.jms/JMSServers/' + jmsserver)
print 'Messages: ' , jms.getMessagesCurrentCount()

print 'Pending: ' , jms.getMessagesPendingCount()
disconnect()
time.sleep(60)
The example in the slide connects to a server and prints the values of various JMS runtime
MBean attributes every 60 seconds. The loop is written so that it runs indefinitely, until the
process is terminated. In this example, the WLST script connects to and disconnects from the
server each time data is collected. Connection creation is a relatively expensive operation, so
this approach can have an impact on the script’s or server’s responsiveness. Alternatively,
create an initial connection, reuse it, and automatically re-create it if the connection ever fails.

Quiz
Name four valid Jython keywords.

a. def
b. deploy
c. try
d. for
e. import
Answer: a, c , d, e

Quiz
Which is not a WLST command included in a WLS installation?

a. connect()
b. cd()
c. move()
d. ls()
e. getMBean()
Answer: c

Quiz
What WLST command is necessary to monitor the health or

performance of WLS resources?
a. serverConfig()
b. redeploy()
c. startEdit()
d. serverRuntime()
e. validate()
Answer: d

Summary
In this lesson, you should have learned how to:

• Describe the relationship between JMX and WLST
• Compare the WLS MBean hierarchies
• Use some basic Jython and WLST commands
• Work with files, environment variables, and exceptions in
Jython

Practice 2-1
Connecting to the Classroom Grid
This practice covers the following topics:
• Identifying your classroom virtual machine (VM) instance
• Connecting to a classroom virtual machine by using an NX
client
• Establishing an NX session

Practice 2-2
Developing a Custom Monitoring Script
• Initializing a sample domain
• Navigating the WLS runtime MBean hierarchy
• Retrieving and displaying MBean attributes
• Using loop and condition Jython constructs

Guardian

Objectives

• Explain the purpose of a Guardian agent
• Install and update Guardian
• Generate domain inventories and evaluation summaries
• Browse problem signatures, bundles, and remedies
• Use Guardian to help generate a support request
• Automate Guardian tasks by using scripts

Guardian Capabilities
Oracle Guardian:
• Is like a virus scanner for WebLogic Server
• Checks domains for common configuration and runtime
problems and then recommends solutions
• Includes graphical and command-line interfaces
• May also scan other Oracle products running on WLS
• Is free to those who have an Oracle support account
• Can also help automate support-case creation
Oracle Guardian is a diagnostic tool for identifying potential problems in your environment
before they occur, and then providing specific instructions for resolving them. Using Guardian
is like having the entire Oracle Customer Support team scrutinize your domain and
immediately present its findings and recommendations to you at your convenience.
Each Guardian installation maintains a registry of active domains. A domain is considered
active when it is capable of being evaluated. You can activate and deactivate domains at will,
and select which to evaluate at any given time. You can also organize the domains in
Guardian into domain groups for easier management.
When you conduct an evaluation that detects a problem pattern or “signature,” you can create
a service request directly from a selected signature in an evaluation summary. Guardian
creates and saves the service request data as a service request archive for later submission
to Oracle Support.

Using Guardian
Use Guardian to evaluate domains:

• After their initial creation and setup
• As part of each development cycle, prior to production
• After any Oracle patch or service pack has been applied
• After introducing or upgrading any third-party software
• Periodically in production (each night, for example)
• While load or stress testing
As you develop an application and migrate from development to quality assurance to

production, you can run an evaluation at each stage. Guardian will help ensure that each
phase of your development process is compliant with Oracle best practices. After you update
an existing application, you can run an evaluation to assess the deployment. Guardian will
help you find any potential problems that could impact your upgrade. If Guardian earlier
detected a signature, and you subsequently applied the remedy or made other changes to
your system, you can run an evaluation to confirm that the signature is no longer detected and
no new issues were introduced.
After you install a new Oracle patch, service pack, or upgrade, or install or upgrade third-party
software, you can run an evaluation to identify any new issues that may have been
introduced.
Some signatures are designed to evaluate runtime domain settings. Running an evaluation
under heavy load can detect potential problems that would not otherwise be detected. Oracle
recommends conducting these evaluations during load and performance testing. If you are
concerned about domain settings being incorrectly changed overnight, or if your domain is
approaching certain resource limits, you can schedule evaluations to run overnight. You can
review the Evaluation Summary in the morning and decide if any detected signatures merit
further investigation.

Guardian Architecture
Domain
Admin server
JMX
GUI or script Guardian agent
Guardian
client Managed server
HTTP(S)
JMX
Guardian agent
Repository
Oracle
Support Domain
The Guardian agent is a lightweight Web application that gathers the data used for
evaluations. It collects data from the server via JMX, from the JVM, and inspects application
resources such as deployment descriptors.
Both graphical and command-line Guardian clients are available. The first time you launch the
Guardian interface it will generate a workspace folder. Guardian stores all settings,
preferences, domain inventories and domain evaluations in this workspace. To perform an
evaluation, the Guardian client communicates with the agents deployed on your servers.
The Guardian repository contains the locally persisted store of problem signatures available
for evaluation. When you download signatures from the Guardian update site, they arrive in a
Java Archive (JAR) file. The JAR file is stored in the repository/archives directory of your
Guardian installation directory.
To safeguard your domains, Guardian requires valid login credentials for all communications
between Guardian and your Guardian domains. Whenever you conduct an evaluation or
activate a domain, Guardian prompts you for the username and password of an Administrator
or Monitor account on the target domain.
SSL encryption is available for all communication with Oracle over the Internet and for all
communication with Guardian agents in your target domain(s).

Agent Installation
• A Guardian agent (.war) is bundled with WLS.

• Either deploy the agent manually to each server or use the
shortcut option available in the console.
• Update and redeploy the agent as new versions become
available from Oracle Support.
To quickly deploy the Guardian agent Web application bundled with your WebLogic Server
installation:
1. Launch the console and select the domain name in the Domain Structure panel.
2. Select the Enable Oracle Guardian Agent check box and click Save.
3. Restart all servers in your domain.
The Guardian agent included with WLS can be found at <WL_HOME>/server/lib/bea-
guardian-agent.war. If a new version of the agent becomes available, you can simply
replace this file and perform an application update, or delete the application deployment and
reinstall the new file. Do not change the default name of the Guardian agent when deploying it
manually.

Collected Data
During an evaluation, the following types of data are collected

by the agent:
• JMX (configuration and runtime)
• Application deployment descriptor settings
• JVM system properties
• JRockit JVM runtime statistics

Client Installation
• Download the client installer from the Oracle Support site

(Windows or Linux).
• An existing JVM (1.5 or later) must be in the OS path
before you run the installer.
• Launch Guardian and select a workspace location in which
to store user preferences and evaluation data.
1. Install Guardian:
a. Launch guardian_installer.exe/bin.
b. Select the location in which to install Guardian. Accept the default, or click Choose
to open a file browser from which you can select a location. If you already have
Guardian installed, be sure to install it to a different location from that of the
previous Guardian installation.
c. Review your installation details and click Install to proceed.
d. Click Done to dismiss the wizard.
2. Launch Guardian and select a Workspace location. The workspace is the directory in
which all of your Guardian data is stored, including user preferences, domain
inventories, and domain evaluation summaries. To prevent loss of work when Guardian
is updated or uninstalled, select a workspace location outside of the Guardian
installation directory.
If you select the “Use this as the default workspace and do not ask again” check box,
Guardian will use the current location from then on when starting. To resume opening the
Select Workspace dialog box at startup, select the “Prompt for workspace on startup” check
box in the “Startup and Shutdown” section of the Guardian Preferences menu.

Guardian User Interface
Main toolbar
View domain Browse all Browse Create shortcuts

inventories and problem signatures for commonly
evaluations. signatures. by bundle. performed actions.
The main Guardian toolbar is located below the main menu bar and contains action buttons
for the most common Guardian tasks. You can place your cursor over a button to get a brief
description of its functionality. The Open button opens the current selected entity in one of the
tabs, such as a domain inventory, evaluation summary, or signature. These resources are
shown in separate tabbed document panels on the right side of the interface (by default). To
print the contents of the current active document panel, use the Print button.
The navigation pane in Guardian resides on the left side of the interface and contains several
tabs to access various explorer panels. The Domain Explorer view enables you to browse,
manage, and monitor the domains you have registered with Guardian. The Signature Explorer
view allows you to browse the problem signatures in Guardian’s repository and view their
details.
Explorer and document panels contain title bar buttons and can be individually maximized,
minimized, or closed as needed. You can also drag and drop panels to reorder them. These
types of customizations are automatically saved in your workspace for subsequent Guardian
sessions.
The online help for Guardian provides a detailed overview of the tool’s capabilities. However,
for specific instructions on using the client interface, use Guardian’s integrated Help system
(available from the main menu).

Activating a Domain
Activating a domain:
• Registers the domain in your workspace
• Generates an initial domain inventory using the agent
1 Location of
admin server
1. Either click the Activate button in the main toolbar or select File > New > Domain from
the main menu.
2. On the General tab, enter the following:
- Protocol: Select “http://” or “https://.” Oracle recommends using the SSL
encryption (“https://”) option for communication between Guardian clients and
agents. Guardian utilizes open source, 128-bit encryption.
- Host Name: This is the listen address for the domain’s Administration Server.
- Port Number: This is the listen port for the domain’s Administration Server.
- Username/Password: A domain account that has administrator or monitor
privileges
- Remember Username/Password: Store the supplied credentials securely in the
Guardian workspace to avoid entering them for every subsequent task.

Creating a Domain Inventory
A domain inventory:
• Is a snapshot of a domain’s high-level configuration
(servers, applications, data sources, and so on)
• Is automatically generated prior to each evaluation
• Can also be created manually for use in a support case
Or
After activating a domain, a node is added to the Domain Explorer view, in the Target
Domains folder. An initial domain inventory entry is also added to the domain node’s
Inventory History folder. Double-click an inventory to view its details in the document pane.
To explicitly create a new inventory file for a domain:
1. Either click the Inventory button in the main toolbar or select File > New > Inventory from
the main menu.
2. Select one of the available activated domains. If prompted, also specify the
Username/Password to access this domain.
To compare two domain inventories:
1. Within the Domain Explorer, expand the Inventory History folder for a domain.
2. Select two inventory files using the Ctrl or Shift keys.
3. Right-click the selected items and select Compare. A document panel then highlights
the differences between the two files. Various box and connector shapes are used to
indicate content that has been added, removed, or updated. Right-click within this panel
to see additional display options.

Signatures and Bundles
A Guardian signature:
• Identifies a pattern in domain evaluation data that indicates
a potential problem
• Is created by Oracle Support
• Has a severity level (Info, Warning, Critical, and so on)
A signature bundle:
• Is a collection of related signatures with similar
characteristics
• Can be evaluated together as a unit
Oracle Support has identified patterns in user domains that can cause problems. These
patterns are described in XML documents called signatures. Signatures form a primary
component of Oracle Guardian because they contain the distilled knowledge of Oracle
Support for both detecting potential problems and resolving them.
Signatures describe potential problems based on information about your WebLogic Servers
and the environment in which they are deployed, including JVMs, operating systems, and
databases. Signatures contain executable logic that can identify specific versions of these
products as well as their configuration settings. Signatures also contain a remedy
recommendation and a severity level: Critical, Warning, or Info. These severities approximate
the level of attention you should give the signature when it is detected.
A signature bundle is a group of signatures that are evaluated together against one or more
specified domains. Signatures are grouped into bundles based on their characteristics. For
example, the Security Advisories bundle contains signatures that detect potential security
problems for which Oracle has issued security advisories.
The Signature Explorer is a view that allows you to browse and interact with the available
signatures. If you double-click a signature in the Signature Explorer, a Signature Details editor
opens in the document pane.

Updating the Signature Repository
To update Guardian with the latest features and signatures, do

one of the following:
• Run the Update Wizard and supply your Support account.
• Download the files from the Support site and apply them to
your installation manually.
1
2
1. Click the Update button on the main toolbar or select Help > Software Updates >
Guardian Updates from the main menu.
2. Enter your Oracle Support username and password. (Selecting the Remember
Username/Password check box is optional.)
3. On the Search Results page, select the features to install using the supplied check
boxes. Click More Info to see a more detailed description of each update.
4. After the updates are installed, you are prompted to restart Guardian. When Guardian
restarts, all the new signatures and application features you downloaded will be
available.

Signature Annotations
Annotations allow you to:

• Ignore certain signatures during evaluations
• Flag signatures of interest and include custom comments
Annotations enable you to tag a signature with one or more persistent attributes. Use the
Annotations Wizard to create and manage custom annotations. The most common uses of
signatures are (a) to indicate that a signature should be ignored or skipped during an
evaluation and (b) to record custom notes and comments about an annotation for later
reference.
1. Navigate to a signature list in either the Signature Explorer, Bundle Explorer, or an
Evaluation Summary. Right-click a signature and select Annotations > Manage
Annotations. Then, in the annotations table, click Add to create a new annotation for this
signature. Similarly, use the Edit and Delete buttons to update or remove them.
2. Enter the following values as needed:
- Type: Select “ignore” to skip this signature while evaluating the specified target
domains. Select “flag” to simply enter some comments. In either case, this
signature’s icon will now be decorated with symbols to indicate that it has
annotations.
- Name: Give the annotation some arbitrary name.
- Comment: Record custom notes about this signature.
- Domains: Specify which target domains this signature should be evaluated on.
- Evaluations: Indicate whether this annotation applies to a specific evaluation or any
evaluation.

Evaluating a Domain
1. Select one or more domains to evaluate.

2. Select which signatures to test on each domain.
1
To detect which signatures apply to your domain, you conduct an evaluation. Guardian
collects data about your domain environment and identifies which signatures apply to the
domain.
Note that the evaluation of some signatures can fail or cause the evaluation to hang. To
prevent this, use the Preferences page to set the Enable Safe Evaluation option.
1. Click the Evaluate button on the main toolbar. Alternatively, from the Domain Explorer,
right-click an available domain and select Evaluate.
2. Select one or more domains from the displayed table. In the Bundle column for the
indicated domain, select the signature bundle to evaluate. To test all problem
signatures, choose the All Signatures option. As always, enter the credentials to use to
access the domain. If you selected multiple domains, you must supply credentials for
each of them.
3. (Optional) Use the Create Shortcut check box to add a new shortcut to the Shortcut
Explorer. This is useful if you plan to perform this same evaluation routinely.

Evaluation Summary
Select a detected
signature.
View suggested
remedy and links
to related docs.
View signature
details.
When the evaluation is complete, the results are displayed in an Evaluation Summary file in
the document pane. The Evaluation Summary lists all the detected signatures, along with the
severity level, description, and recommended remedy for each.
For a given bundle of signatures, some signatures may not be used in the evaluation because
they target different products or versions than the domain is configured for. The ones that do
target what the domain has are counted as Targeted Signatures. The Targeted Signatures
are divided into Detected Signatures, which are actually found on the domain, and
Undetected Signatures, which are not found on the domain.
Optionally, you can right-click one of the detected signatures in the summary and select
Filters. Filters enable you to hide or show certain signatures according to their characteristics.
On the bottom of the summary page, the following tabs provide different representations of
the summary data:
• Overview: The default view (as described above)
• Source: Provides the actual XML source from the summary document. It includes the
raw data for each detected and undetected signature.
• Report: Provides all the information contained in the Overview tab but in a printer-
friendly format

Generating a Support Request
• Guardian can generate a file to later attach to a support

case if you are unable to resolve the issue.
• The generated archive file (.car) includes:
– Details of the selected signature
– Your domain inventory and evaluation results
– The domain’s configuration files (optional)
– The log file of a selected server (optional)
...
1 2
Evaluation
summary
A support service request is a record that is created when you submit technical questions or
issues to Oracle Support. Customers with a support contract can open a service request over
the phone or online. Suppose that despite Guardian’s assistance you are unable to resolve a
given problem signature. Before creating a new service request, Guardian can create and
save the initial service request template as a file for later submission to Oracle Support.
These service request archive files include all the information from the signature along with
the corresponding evaluation summary and domain inventory files. This enables an Oracle
support engineer to quickly begin working on your service request upon receipt of the archive.
You can also add custom attachments and notes before creating the support request file.
1. Open an evaluation summary and select a signature. Then, within the Remedy section,
click the Get More Help link. If desired, enter any Additional Service Request Notes you
want the support engineer to see.
2. Select one of the servers in your domain. By default, the selected server’s log file is
included in the support request archive. Alternatively, you can disable the inclusion of
the server log and/or domain configuration files by using the supplied check boxes.
3. Enter the destination on the local file system to which Guardian will persist the service
request archive file.

Command-Line Interface
Activate a domain:
guardianHeadless.sh –gactivateDomain
–t http://199.177.1.1:7001 -u myuser –p mypassword
–data /home/oracle/guardianWorkspace
List activated domains and their IDs:
guardianHeadless.sh –glistActiveDomains –data ...
List signature bundles in the repository and their IDs:
guardianHeadless.sh –glistBundles –data ...
Evaluate a domain against the All Signatures bundle (8):
guardianHeadless.sh –gevaluateDomainBundle –d MyDomain_myhost_7001

-b 8 -u myuser –p mypassword –data ...
The Guardian command-line interface provides a set of commands that can be issued directly
from the operating system shell, and therefore can also be scheduled to run automatically at
specific times. There is a Guardian command for each of the most common tasks you can
perform using the Guardian graphical interface. For a list of available commands, use the
–ghelp argument. To evaluate a domain, you must specify the bundle’s internal ID instead of
its display name. For example, the default bundle is 0 while the All Signatures bundle is 8.
This tool can either run a single command or a series of them placed in a separate script file
(-gscript command). Also note that on Windows all command output is directed to a file
named headless_output.txt, while on Linux it is simply written to the standard output
stream.
All commands require access to your Guardian workspace directory. If a workspace is not
explicitly specified, the default workspace is used. In addition to the commands shown in the
slide (-g<commandName>), the tool also supports the following:
• createShortcut: Create a shortcut to evaluate the specified domain and bundle.
• evaluateShortcut: Run the specified shortcut.
• listShortcuts: List the IDs of all available shortcuts in the workspace.
• inventoryDomain: Generate an inventory for an activated domain.

Quiz
Name three capabilities of Oracle Guardian.

a. Trigger email notifications.
b. Generate support request artifacts.
c. Check domains for common problems.
d. Create a domain inventory.
e. Monitor server performance.
Answer: b, c, d

Quiz
Signatures are organized into ______.

a. Domains
b. Annotations
c. Shortcuts
d. Bundles
e. Agents
Answer: d

Quiz
Name three Guardian signature severity levels.

a. Warning
b. Notice
c. Error
d. Info
e. Critical
Answer: a, d, e

Summary

• Explain the purpose of a Guardian agent
• Install and update Guardian
• Generate domain inventories and evaluation summaries
• Browse problem signatures, bundles, and remedies
• Use Guardian to help generate a support request
• Automate Guardian tasks by using scripts

Practice 3-1
Using Guardian to Evaluate a Domain
• Updating Guardian with the latest problem signatures
• Deploying the Guardian agent to a domain
• Activating and inventorying a domain
• Evaluating a domain for problem signatures
• Correcting a problem identified by Guardian

Diagnostic Framework Essentials

Objectives

• Describe the main components of the WebLogic
Diagnostic Framework
• Explain the capabilities of the diagnostic Harvester
• Configure diagnostic archives and modules
• Record MBean attributes by using WLDF
• Trigger notifications under specific MBean conditions
• Plot current and historical metrics by using the Monitoring
Dashboard
• Enable debugging messages for server subsystems

Road Map
• WLDF Administration
– Architecture
– Logging Review
– Diagnostic Images
– Diagnostic Archives
– Diagnostic Modules
• Metric Collectors
• Monitoring Dashboard
• WLS Debugging

WebLogic Diagnostic Framework (WLDF)
• WLDF provides a generic framework to gather and analyze

WLS runtime data for monitoring and/or troubleshooting.
• Use WLDF to:
– Capture a snapshot of key server metrics for distribution to
support personnel
– Capture metrics at specific code points in WLS or your
application
– Periodically record selected MBean attributes
– Send notifications when attributes meet certain conditions
WLDF consists of a number of components that work together to collect, archive, and access
diagnostic information about a WebLogic Server instance and the applications it hosts.

WLDF Architecture
WLDF
Image Capture
MBeans
Harvester Data
archive
Metric Collectors
Logs
Notifications
Watches
Code
Instrumentation
Event
Monitors archive
Data creators generate diagnostic data that is consumed by the logger and the Harvester.
Those components coordinate with the archive to persist the data, and they coordinate with
the watch and notification subsystem to provide automated monitoring. The data accessor
interacts with the logger and the Harvester to expose current diagnostic data and with the
archive to present historical data. MBeans make themselves known as data providers by
registering with the Harvester. Collected data is then exposed to both the watch and
notification system for automated monitoring and to the archive for persistence.
The instrumentation system creates monitors and inserts them at well-defined points in the
flow of code execution within the JVM. These monitors trigger events and publish data directly
to the archive. They can also take advantage of watches and notifications.
Diagnostic image capture support gathers the most common sources of the key server state
used in diagnosing problems. It packages that state into a single artifact, which can be made
available to support technicians.
The past state is often critical in diagnosing faults in a system. This requires that the state be
captured and archived for future access, creating a historical archive. In WLDF, the archive
meets this need with several persistence components. Both events and harvested metrics can
be persisted and made available for historical review.

WLS Logging: Review
• Each server has its own log file into which subsystems
record messages.
• Applications can write to the host server’s log as well.
• Servers also have access logs to record all HTTP
requests.
• Certain important messages are also broadcast to the
domain log hosted on the admin server.
Configure logging
for a server.
Each WebLogic Server instance writes all messages from its subsystems and applications to
a server log file that is located on the local host computer. By default, the server log file is
located in the logs directory below the server instance root directory (for example,
DOMAIN_NAME/servers/SERVER_NAME/logs/SERVER_NAME.log). The messages
include information about the time and date of the event as well as the ID of the user who
initiated the event.
Application developers who want to use the WebLogic Server message catalogs and logging
services as a way for their applications to produce log messages must know the Java APIs.
In addition to writing messages to the server log file, each server instance forwards a subset
of its messages to a domain-wide log file. By default, servers forward only messages of
severity level NOTICE or higher. The domain log file provides a central location from which to
view the overall status of the domain. The domain log resides in the Administration Server
logs directory. The default name and location for the domain log file is
DOMAIN_NAME/servers/ADMIN_SERVER_NAME/logs/DOMAIN_NAME.log.
Some subsystems maintain additional log files to provide an audit of the subsystem’s
interactions under normal operating conditions. The HTTP subsystem keeps a log of all HTTP
transactions in a text file. The default location and rotation policy for HTTP access logs is the
same as the server log.

Log Severity Thresholds
• A severity level is assigned to each log message.

• Severity thresholds control which log messages are:
– Generated by specific WLS subsystems
– Written to the server log file
– Sent to standard output
– Forwarded to the domain log
EJB subsystem will

generate only
Warning or higher.
Only Notice or
higher will be sent to
standard output.
The severity attribute of a WebLogic Server log message indicates the potential impact of the
event or condition that the message reports. TRACE and DEBUG messages are the lowest
severity while CRITICAL, ALERT, and EMERGENCY are the highest. Under normal
circumstances, WebLogic Server subsystems generate many messages of lower severity and
fewer messages of higher severity. In addition to writing messages to a log file, each server
instance prints a subset of its messages to standard out. Usually, standard out is the shell
(command prompt) in which you are running the server instance.
• Logger Severity Properties: Configure which log message severities are generated by
named WLS subsystems or Java packages
• Log File: Severity Level: The minimum severity of log messages going to the server
log file. By default all messages go to the log file
• Standard Out: Severity Level: The minimum severity of log messages going to the
standard out
• Domain Log: Severity Level: The minimum severity of log messages going to the
domain log from this server’s log broadcaster
• Minimum Severity to Log: The minimum severity of log messages going to all log
destinations

Application Logging
• Developers can add error, debug, and other events to the

server log or use their own independent logging system.
• Message text can be coded into the application or
maintained in external files.
• WLS supports several frameworks to help integrate server
and application logging:
– Java Logging (default)
– Apache Jakarta Commons
– Apache Log4J
• Refer to the documentation for complete setup
instructions.
You can use WebLogic logging services to keep a record of which user invokes specific
application components, to report error conditions, or to help debug your application before
releasing it to a production environment. Your application can also use them to communicate
its status and respond to specific events. A major advantage of integrating your application
logging with WebLogic logging framework is ease of management.
Many implementation options are available. First, WLS provides APIs and tools to build
custom message catalogs that support multiple languages. These catalogs are then
referenced by your application. WLS also supports alternative APIs to simply let applications
programmatically add message text to the server log, without the use of catalogs. Finally,
WLS can also integrate with other popular open-source logging frameworks including Apache
Commons and Apache Log4J.
These third-party frameworks are not distributed with WLS and must be downloaded and
added to the server classpath manually. This is most commonly done using the domain’s
/lib directory. Additional steps are required to complete the integration. For example, to
support Log4J, you must edit the server’s Logging Implementation attribute, as shown above.
To integrate Commons, you must set various Java system properties when starting the
server.

Server Logging Bridge
The Server Logging Bridge provides a mechanism for

applications that use Java or Log4J logging to redirect
messages to the WebLogic logger.
• Create a logging.properties file which is passed as
an argument to weblogic.Server on server startup.
– This file registers the Server Logging Bridge handler in the
application’s logger tree.
• For Java logging:
– Bridge is exposed as a handler object:
weblogic.logging.ServerLoggingHandler.
• For Log4J logging:
– Bridge is exposed as an appender object:
weblogic.logging.log4j.ServerLoggingAppender.
The Server Logging Bridge provides a lightweight mechanism for applications that currently
use Java Logging or Log4J Logging to have their log messages redirected to WebLogic
logging services. Applications can use the Server Logging Bridge with their existing
configuration; no code changes or programmatic use of the WebLogic Logging APIs is
required. To use the Server Logging Bridge, you only need to create a logging properties file.
Java Logging
WebLogic Server exposes the Server Logging Bridge as the handler object
weblogic.logging.ServerLoggingHandler. When the handler receives an application
log message in the form of a java.util.logging.LogRecord object, the handler
redirects the message to the WebLogic logging service destinations, such as stdout, server
log, domain log, and so on.
Log4J Logging
WebLogic Server exposes the Server Logging Bridge as the appender object
weblogic.logging.log4j.ServerLoggingAppender. When the appender receives an
application log message in the form of a org.apache.log4j.spi.LoggingEvent object,
the appender redirects the message to the WebLogic logging service destinations such as
stdout, server log, domain log, and so on.

WLDF Configuration: Overview
1. Configure and generate diagnostic images for a server.

2. Configure diagnostic archives for a server.
3. Create and target a new diagnostic system module.
4. Add metric collectors, watches, notifications, and/or
monitors to the module.
5. (Optional) Create application-scoped modules and
monitors.
Server
Instrumentation Harvester
Monitors Metric Collectors
Watches
Diagnostic Module
WLDF provides features for generating, gathering, analyzing, and persisting diagnostic data
from WebLogic Server instances and from applications deployed to them. For server-scoped
diagnostics, some WLDF features are configured as part of the configuration for a server in a
domain. Other features are configured as system resource descriptors that can be targeted to
servers (or clusters). For application-scoped diagnostics, diagnostic features are configured
as resource descriptors for the application.
The Harvester, watch, and instrumentation features are configured, packaged, and targeted
as part of a diagnostic module, similar to a JMS module resource. Only instrumentation
monitors can be defined in application-scoped modules, which are placed in the application’s
weblogic-diagnostics.xml file.
You create a diagnostic system module through the admin console or WLST. It is created as
a WLDFResourceBean, and the configuration is persisted in a resource descriptor file called
<module>.xml, where <module> is the name of the diagnostic module. The file is created
by default in the domain’s config/diagnostics directory. The file has the extension .xml.

Diagnostic Images
A diagnostic image is:

• A zip file generated by WLDF
• A detailed snapshot of a server’s configuration and runtime
metrics (JVM, JNDI, JDBC, JMS, and so on)
• Primarily intended to help Oracle Support personnel
diagnose issues
You use the diagnostic image capture component of WLDF to create a diagnostic snapshot,
or dump, of a server’s internal runtime state at the time of the capture. This information helps
support personnel analyze the cause of a server failure. You can capture an image manually
using the console or WebLogic Scripting Tool (WLST), or you can generate one automatically
as part of a watch notification.
Because the diagnostic image capture is meant primarily as a post-failure analysis tool, there
is little control over what information is captured. It includes the server’s configuration, log
cache, JVM state, work manager state, JNDI tree, and most recent harvested data. The
image capture subsystem combines the data files produced by the different server
subsystems into a single zip file.

Capturing a Server Diagnostic Image
Location of
image file (.zip)
1
3
To capture a server image by using the console:

1. In the left pane, expand Diagnostics and select Diagnostic Images.
2. Select the server for which you want to capture a diagnostic image, and click Capture
Image.
3. Enter a new directory in the Destination Directory field, or accept the default directory. If
you change the directory for this image capture, it does not change the default directory
for capturing images for this server when they are captured by other means, such as a
watch notification. Then click OK.

WLST: Downloading Diagnostic Image Files
New online WLST commands have been added for

downloading WLDF diagnostic image files:
• getAvailableCapturedImages()
– Returns a list of diagnostic images that have been created in
the image destination directory configured on the server
• saveDiagnosticImageCaptureFile()
– Downloads a specified diagnostic image capture file
• saveDiagnosticImageCaptureEntryFile()
– Downloads a specific entry within a diagnostic image capture
getAvailableCapturedImages()
Returns, as an array of strings, a list of the previously captured diagnostic images that are
stored in the image destination directory configured on the server. The default directory is
SERVER\logs\diagnostic_images. This command is useful for identifying a diagnostic
image capture that you want to download, or for identifying a diagnostic image capture from
which you want to download a specific entry.
saveDiagnosticImageCaptureFile(imageName)
Downloads the specified diagnostic image capture from the server to which WLST is currently
connected
saveDiagnosticImageCaptureEntryFile(imageName,imageEntryName)
Downloads a specific entry from the diagnostic image capture that is located on the server to
which WLST is currently connected
Example
images=getAvailableCapturedImages()
saveDiagnosticImageCaptureFile(images[0])

Diagnostic Archives
Collected WLDF metrics and events are recorded to the

server’s diagnostic archives:
• WLDF file store
(<server>/data/store/diagnostics, by default)
• WLDF JDBC store
Module1 Module2
Harvested data archive
Harvester Harvester
Instrumentation Instrumentation
Event data archive
The Archive component of WLDF captures and persists all data events, log records, and
metrics collected by WLDF from server instances and applications running on them. You can
access archived diagnostic data in online mode (that is, on a running server). You can also
access archived data in off-line mode by using WLST. You configure the diagnostic archive
on a per-server basis.
For a file-based store, WLDF creates a file to contain the archived information. The only
configuration option for a WLDF file-based archive is the directory where the file will be
created and maintained. The default directory is
<domain>/servers/<server>/data/store/diagnostics. When you save to a file-
based store, WLDF uses the WebLogic Server persistent store subsystem.
To use a JDBC store, the appropriate tables must exist in a database, and JDBC must be
configured to connect to that database. The wls_events table stores data generated from
WLDF Instrumentation events. The wls_hvst table stores data generated from the WLDF
Harvester component. Refer to the documentation for the required schema.

Configuring Server Diagnostic Archives
2
1
3
For file
store option
For JDBC
store option
Log files are archived as human-readable files. Events and harvested data are archived in
binary format, in a WebLogic persistent store or in a database.
1. In the left pane, expand Diagnostics and select Archives.
2. Click the name of the server for which you want to configure diagnostic archive settings.
3. Select one of the following archive types from the Type list:
- Select File Store to persist data to the file system. If you choose this option, enter
the directory in the Directory field.
- Select JDBC to persist data to a database. If you choose this option, select an
existing JDBC data source from the Data Source list.

Archive Retirement Policies
Recorded data can be limited by using size-based (file only)

or age-based policies.
File-based policy
(megabytes)
Age-based
policy (hours)
WLDF includes a configuration-based data retirement feature for periodically deleting old
diagnostic data from the archives. You can configure size-based data retirement at the server
level and age-based retirement at the individual archive level. Size-based data retirement can
be used only for file-based stores. These options are ignored for database-based stores.
To configure size-based policies, select a diagnostic archive for a specific server. In the
Preferred Store Size field, enter a maximum data file size, in megabytes. When this size is
exceeded, enough of the oldest records in the store are removed to reduce the size of the
store below the maximum. In the Store Size Check Period field, enter the interval, in minutes,
between the times when the store will be checked to see if it has exceeded the preferred store
size.
To configure age-based policies, select a diagnostic archive for a specific server and locate
the Data Retirement Policies table. Then click New and enter the following criteria:
• Age: Retirement age for records in hours. Older records will be eligible for deletion.
• Time: The hour of day at which the data retirement task will first run during the day
• Period: The period in hours at which the data retirement task will be periodically
performed for the archive during the day after it is first executed

Archive Database Schema
• A JDBC diagnostic archive requires two tables:

– WLS_HVST (harvested metrics)
– WLS_EVENTS (instrumentation)
• Refer to the documentation for the complete schema.
CREATE TABLE WLS_HVST (
RECORDID INTEGER,
TIMESTAMP NUMERIC default NULL, Oracle DB
DOMAIN VARCHAR2(64) default NULL, example
SERVER VARCHAR2(64) default NULL,
TYPE VARCHAR2(64) default NULL,
NAME VARCHAR2(250) default NULL,
ATTRNAME VARCHAR2(64) default NULL,
ATTRTYPE INTEGER default NULL,
ATTRVALUE VARCHAR2(4000)
);
To use a JDBC store, the appropriate tables must exist in a database, and JDBC must be
configured to connect to that database. If they do not already exist, you must create the
database tables used by WLDF to store data in a JDBC-based store. Two tables are required.
The wls_events table stores data generated from WLDF Instrumentation events, while the
wls_hvst table stores data generated from the WLDF Harvester component.
The SQL data definition language (DDL) used to create tables may differ for different
databases, depending on the SQL variation supported by the database. A sample DDL
implementation for Pointbase is provided in the online documentation:
http://download.oracle.com/docs/cd/E12839_01/web.1111/e13714/config_diag_archives.htm#
i1067779
Consult the database documentation or your database administrator when creating these
tables for your database.

Viewing Archive Contents
2
1
View archive for a

specific server.
Search for or
browse records.
WebLogic Server’s Data Accessor subsystem retrieves diagnostic information from WLDF
components. Captured information is segregated into logical data stores that are separated by
the types of diagnostic data. For example, server logs, HTTP logs, and harvested metrics are
captured in separate data stores. WLDF maintains diagnostic data on a per-server basis.
Data stores can be modeled as tabular data. Each record in the table represents one item,
and the columns describe characteristics of the item. Different data stores may have different
columns. However, most data stores have some of the same columns, such as the time when
the data was collected.
Because WLDF archives are modeled the same as log files, you can browse and search their
contents in the console using the same process you use for log files:
1. In the left pane of the console, expand Diagnostics and select Log Files.
2. In the Log Files table, select the option button next to the name of the WLDF archive
that you want to view, and click View.
3. By default, the subsequent table displays the most recent contents of the archive and
some basic columns, but this can be customized as needed. Select the option button
next to the archive record that you want to view, and click View.

Creating a Diagnostic Module
Modules group related WLDF resources and are targeted to

servers or clusters.
1
2
Target to one or Configure

more servers. resources.
To configure and use the Instrumentation, Harvester, and Watch and Notification components
at the server level, you must first create a system resource called a diagnostic system
module. System modules are globally available for targeting to servers and clusters
configured in a domain. But at most only one diagnostic system module can be targeted to
any given server or cluster.
1. In the left pane, expand Diagnostics and select Diagnostic Modules.
2. Click New. Enter a name for the module and, optionally, enter a description. Then click
OK.
3. Use the various Configuration tabs to add diagnostic components to this module.
4. To target the module to a server or cluster, click the Targets tab.

WLDF WLST Examples
Capture a server diagnostic image:
serverRuntime()
wldfCapture = getMBean('WLDFRuntime/WLDFRuntime/
WLDFImageRuntime/Image')
wldfCapture.captureImage('logs/diagnostic_images',30)
Create a diagnostic module:
cd('/')
module = cmo.createWLDFSystemResource('JMSDebugModule')
module.addTarget(getMBean('/Servers/serverA'))
Additional WLDF WLST examples can be found in the WLDF guide in the product
documentation.
Refer to the following MBeans:
• WLDFRuntimeMBean
• WLDFImageRuntimeMBean (a component of WLDFRuntimeMBean)
• WLDFServerDiagnosticMBean
• WLDFDataRetirementMBean (a component of WLDFServerDiagnosticMBean)
• WLDFDataRetirementByAgeMBean (a component of
WLDFServerDiagnosticMBean)
• WLDFResourceBean

WLDF WLST Examples
Configure a server’s diagnostic archives:
archives = getMBean('Servers/serverA/
ServerDiagnosticConfig/serverA')
archives.setStoreSizeCheckPeriod(30)
archives.setDataRetirementEnabled(true)
View the contents of a server’s diagnostic archive as XML:
startTime = System.currentTimeMillis()
endTime = 3600 * 1000
exportDiagnosticDataFromServer(
logicalName='HarvestedDataArchive',
exportFileName='wls_debug.xml',
beginTimestamp=startTime, endTimestamp=endTime,
query="ATTRNAME = 'MessageHighCount'")
The exportDiagnosticDataFromServer() WLST command executes a query on the

server side and retrieves the exported WLDF data. The results are saved to an XML file. This
is an online WLST command and therefore you must connect to a running server before
executing it. A similar offline command named exportDiagnosticData() is also available
that can extract WLDF data from a specified diagnostic archive on the local file system.
The exportDiagnosticDataFromServer() command supports the following arguments:
• logicalName: The logical name of the log file being read from the current server. Valid
values include: HarvestedDataArchive, EventsDataArchive, ServerLog,
DomainLog, HTTPAccessLog, WebAppLog, ConnectorLog, and JMSMessageLog.
This option defaults to ServerLog.
• exportFileName: The name of the file to which the data is exported. This option
defaults to export.xml.
• query: A WLDF query expression specifying the filter condition for the data records to
be included in the result set. If it is not specified, all data is retrieved. The available
parameters for a query expression vary based on the type of the data source
(HarvestedDataArchive, ServerLog, and so on). For additional examples, see the
online documentation.

Section Summary
In this section, you should have learned how to:

• Discuss the overall capabilities of WLDF
• Describe the contents of a server diagnostic image
• Capture a server diagnostic image
• Configure server diagnostic archives and retirement
policies
• Search the contents of a diagnostic archive
• Create and target a diagnostic module

Road Map
– Harvester Architecture
– Watches
– Alarms
– Notifications
• WLS Debugging

Harvester Architecture
The WLDF Harvester component periodically collects and

records target MBean attributes.
Data
archive
Harvester
Metric collectors
MBeans
Notifications
Watches
Diagnostic module
The Harvester component of WLDF gathers metrics from attributes on qualified MBeans that
are instantiated in a running server. The Harvester can collect metrics from WebLogic Server
MBeans and from custom MBeans. To be harvestable, an MBean must be registered in the
local WebLogic Server runtime MBean server. The Harvester is configured and metrics are
collected in the scope of a diagnostic module targeted to one or more server instances.
Harvesting metrics is the process of gathering data that is useful for monitoring the system
state and performance. The Harvester gathers values from selected MBean attributes at a
specified sampling rate. Therefore, you can track potentially fluctuating values over time.
The Watch and Notification component of WLDF provides the means for monitoring server
and application states and then sending notifications based on criteria set in the watches.
Watches and notifications are configured as part of a diagnostic module targeted to one or
more server instances in a domain.

Metric Collector Definitions
• A metric collector definition for the Harvester includes:

– The runtime MBean type to query
– The specific MBean instance names to query (all instances,
by default)
– The MBean attributes to collect (all attributes, by default)
– The frequency with which to sample data
• Alternatively, use the MBean expression syntax, which
also supports wildcards and complex attributes.
MBean attributes
mydomain: Type=*JMSRuntimeMBean, Name=hr*, MessagesHighCount
MBean type MBean name
Harvesting metrics is the process of gathering data that is useful for monitoring the system
state and performance. Metrics are exposed to WLDF as attributes on qualified MBeans. The
Harvester gathers values from selected MBean attributes at a specified sampling rate.
Therefore, you can track potentially fluctuating values over time. For custom MBeans, the
MBean must be currently registered with the JMX server.
You can configure the Harvester to harvest data from named MBean types, instances, and
attributes. If only a type is specified, data is collected from all attributes in all instances of the
specified type. If only a type and attributes are specified, data is collected from all instances of
the specified type. MBean type declarations must specify the full Java package name if no
wildcards are used (for example,
weblogic.management.runtime.ServerRuntimeMBean).
The sample period specifies the time between each cycle. For example, if the Harvester
begins execution at time T, and the sample period is I, the next harvest cycle begins at T + I.
If a cycle takes A seconds to complete and if A exceeds I, then the next cycle begins at T + A.
If this occurs, the Harvester tries to start the next cycle sooner to ensure that the average
interval is I.
WLDF allows for the use of wildcards (*) in type names, instance names, and attribute
specifications. WLDF also supports nested attributes using a dot delimiter, as well as complex
attributes such as arrays and maps. WLDF watch expressions also support similar
capabilities.

Configuring a Metric Collector
How often to
collect samples?
Metrics are configured and collected in the scope of a diagnostic system module targeted to
one or more server instances. Therefore, to collect metrics, you must first create a diagnostic
system module.
1. Click the name of the module for which you want to configure metric collection. Then
click Configuration > Collected Metrics.
2. To enable (or disable) all metric collection for this module, select (or deselect) the
Enabled check box. To set the period between samples, enter the period (in
milliseconds) in the Sampling Period field. To define a new collected metric, click New.
From the MBean Server Location drop-down list, select either DomainRuntime (admin
server only) or ServerRuntime. Then click Next.

Configuring a Metric Collector
Select the
MBean type.
3
Select MBean
attributes or enter a
wildcard expression.
4
Messages*,
for example
3. Select an MBean that you want to monitor from the MBean Type list. Then click Next
again.
4. In the Collected Attributes section, select one or more attributes from the Available list
and move them to the Chosen list (default is all attributes). Alternatively, WLDF supports
wildcard expressions in attribute specifications using the Attribute Expressions field.
Click Next.
5. In the Collected Instances section, select one or more instances from the Available list
and move them to the Chosen list (default is all instances). Once again, you can
alternatively enter an Instance Expression that includes wildcards. Click Finish.
6. Click Save.

Watches and Notifications
• A WLDF watch:
– Inspects data generated from metric collectors, events
generated from monitors, or server log files
– Compares data to one or more conditions or “rules”
– Triggers one or more notifications
• Available notification types include:
– JMS
– Email
– SNMP trap
– Diagnostic image capture
A watch identifies a situation that you want to trap for monitoring or diagnostic purposes. You
can configure watches to analyze log records, data events, and harvested metrics. A watch is
specified as a watch rule, which includes a rule expression, an alarm setting, and one or more
notification handlers. A notification is an action that is taken when a watch rule expression
evaluates to true. You must associate a watch with a notification for a useful diagnostic
activity to occur (for example, to notify an administrator about specified states or activities in a
running server).
Log and instrumentation watches are triggered in real time, whereas Harvester watches are
triggered only after the current harvest cycle completes.
Watches and notifications are configured separately from each other. A notification can be
associated with multiple watches, and a watch can be associated with multiple notifications.
This provides the flexibility to recombine and reuse watches and notifications, according to
current needs. Each watch and notification can be individually enabled and disabled as well.
A complete watch and notification configuration includes settings for one or more watches,
one or more notifications, and any underlying configurations required for the notification media
(for example, the SNMP configuration required for an SNMP-based notification).

Configuring a Watch
2
3
A collected metric watch can monitor any runtime MBean in the local runtime MBean server.
Log watches monitor the occurrence of specific messages and/or strings in the server log.
Watches contain a severity value that is passed through to the recipients of notifications.
Instrumentation watches are triggered as a result of the event being posted that matches
some criteria.
The example in the slide depicts a collected metric watch:
1. Click the name of the module for which you want to create a watch. Then click
Configuration > Watches and Notifications.
2. In the Watches section, click New.
3. Enter a name for the watch in the Watch Name field. Then select a Watch Type. To
enable or disable the watch, select or deselect the Enable Watch check box. Then click
Next.
4. Click the Add Expressions button to construct one or more watch expressions.
Expressions can be entered manually using the WLDF expression language or
constructed graphically. To group two or more expressions, select the check boxes
adjacent to the expressions that you want to group and click Combine. To reorder
expressions, select the check boxes adjacent to the expressions you want to move and
click Move Up or Move Down.

Configuring a Watch
Watch based on
collected MBean
attributes
7
Assign previously
configured notifications.
5. For each watch expression, select either DomainRuntime or ServerRuntime from the
MBean Server Location drop-down list. Then click Next. Select an instance from the
Instance drop-down list, and click Next again.
6. Select an attribute from the Message Attribute list and an operator from the Operator list,
and enter a value with which to compare the attribute using the Value field. The value
must be an appropriate value for the attribute chosen above. Click Next.
7. Select and move one or more existing Notifications from the Available list to the Chosen
list. Then click Finish.

Watch Alarms
• Alarms determine watch behavior after it has been

triggered.
• Watches can be disabled, or they can continue to run and
evaluate.
Trigger watch
repeatedly.
Watch must be
reactivated by
administrator.
Watch will
reactivate after
a specified time.
Each watch definition can be individually enabled and disabled using the console or WLST.
When disabled, the watch does not trigger and corresponding notifications do not fire. If
watches and notifications are disabled at the module level, all individual watches are
effectively disabled (the value of this flag on a specific watch is ignored).
Watches can be specified to trigger repeatedly, or to trigger once, when a condition is met.
For watches that trigger repeatedly, you can optionally define a minimum time between
occurrences. The default option, “Don’t use an alarm,” causes the watch to trigger whenever
possible. The “Use an automatic reset alarm” option also causes the watch to trigger
whenever possible, except that subsequent occurrences cannot occur any sooner than the
specified time interval. Finally, the “Use a manual reset alarm” option causes the watch to fire
a single time, but after it fires, you must manually reset it to fire again.

Configuring a JMS Notification
2 1
Enter JNDI names

4
of local destination
and factory.
1. Click the name of the module for which you want to create a notification. Then click
Configuration > Watches and Notifications. Scroll down and click the Notifications tab,
and then click New.
2. Select the JMS Message type and click Next.
3. Enter an arbitrary Notification Name and select whether this notification should be
enabled or not. Then click Next.
4. Enter the JMS Destination JNDI Name and Connection Factory JNDI Name that will
allow the diagnostics framework to publish a local JMS message. The published
message will be formatted as a JMS MapMessage.

Configuring an Email Notification
1 3
2
Assign previously
configured mail
session.
Override default
email subject or
body, if desired.
Simple Mail Transfer Protocol (SMTP) notifications are used to send email messages over the
SMTP protocol in response to the triggering of an associated watch. Before configuring an
SMTP notification, first create an SMTP Mail Session resource in you domain and target it to
the same server(s) that this diagnostic module is targeted to. Mail sessions define how
WebLogic connects to an enterprise mail server.
To create the notification:
1. Click the name of the module for which you want to create a notification. Then click
Configuration > “Watches and Notifications.” Scroll down and click the Notifications tab,
and then click New. Select the SNMP type and click Next.
2. Enter an arbitrary Notification Name and select whether this notification should be
enabled or not. Then click Next.
3. Select your previously configured mail session and enter one or more email recipients.
Messages sent by the diagnostics framework will contain a default subject and body, but
you can override this behavior and enter custom text.

Harvester WLST: Example
Use WLST to quickly enable or disable diagnostics while

testing or troubleshooting.
Add a metric collector to a diagnostic module:
harvester = getMBean('/WLDFSystemResources/JMSDebugModule/
WLDFResource/JMSDebugModule/Harvester/JMSDebugModule')
harvester.setSamplePeriod(300000)
harvester.setEnabled(true)
harvestType = harvester.createHarvestedType
('weblogic.management.runtime.JMSServerRuntimeMBean')
harvestType.setEnabled(true)
harvestType.setHarvestedAttributes(['MessagesHighCount'])
documentation.
• WLDFHarvesterBean (a subclass of WLDFResourceBean)
• WLDFHarvestedTypeBean (a component of WLDFHarvesterBean)

Watch WLST: Example
Add a watch to a diagnostic module:
notify = getMBean('/WLDFSystemResources/JMSDebugModule/
WLDFResource/JMSDebugModule/WatchNotification/JMSDebugModule/
JMSNotifications/ITMgmtQueue')
watches = getMBean('/WLDFSystemResources/JMSDebugModule/
WLDFResource/JMSDebugModule/WatchNotification/JMSDebugModule')
watches.setEnabled(true)
watch = watches.createWatch('JMSWatch')
watch.setRuleType('Harvester')
watch.setEnabled('true')
watch.setRuleExpression('${ServerRuntime//[weblogic.management
.runtime.JMSServerRuntimeMBean]//MessagesHighCount} > 1000')
watch.setNotifications(array([notify],
weblogic.diagnostics.descriptor.WLDFNotificationBean)
documentation.
• WLDFWatchNotificationBean (a subclass of WLDFResourceBean)
• WLDFWatchBean (a component of WLDFWatchNotificationBean)
• WLDFNotificationBean (abstract; subclasses exist for each notification type)

WLDF Sample Framework
• WLS ships with a collection of sample WLDF WLST scripts

found at
<WL_HOME>/samples/server/examples/src/examp
les/diagnostics.
• These scripts provide a framework to:
– Enable/disable generic metric collectors and watches for
WLS subsystems (JDBC, JMS, Web, EJB, and so on)
– Enable/disable notifications for sample watches
Some users find it difficult to determine what WLS metrics to collect, especially users who are
new to JMX and/or WLDF. WLS includes some WLST scripts and utilities that provide a
starting point for creating common WLDF configurations, referred to as profiles. Each profile is
represented by a Jython class that encapsulates the MBean Harvester and watch
configurations for a specific WLS subsystem. These class definitions can then be modified or
extended to suit your individual needs. In addition, you can use the framework of utility
classes separately and outside the context of profile configuration while working with WLDF
from the WLST command-line.
WLDFResource.py provides a core set of utility classes in Jython for manipulating instances
of the WLDFSystemResourceMBean and its constituent bean objects. This file also provides
various helper functions that are used by other classes and scripts in this framework.
WLDFProfiles.py contains the set of Jython classes that represent the predefined profiles,
as well as a utility class, WLDFProfileManager, for enabling or disabling groups of these
profiles. For convenience, various “enable” scripts are included for each profile, such as
enableEJBProfile.py and enableJTAProfile.py. Each also has a corresponding
“disable” script, which will remove the configured MBean instances associated with the profile.
The scripts accept command-line arguments in the form of name/value pairs to override
common settings and parameters, such as the server URL, username, password, WLDF
module name, and Harvester period.

Section Summary

• Discuss the use of the WLDF Harvester component
• Collect and record MBean metrics by using WLDF
• Configure WLDF watches and trigger notifications
• Customize a watch’s alarm settings

Practice 4-1
Harvesting Diagnostic Metrics
• Defining a JMS queue for diagnostic notifications
• Creating a system diagnostic module
• Configuring metric collectors for several MBean types
• Triggering notifications under various MBean watch
conditions
• Accessing diagnostic archives and inspecting recorded
metrics

Road Map
– New Interface to Metrics Graphing
– Views
– Charts and Graphs
• WLS Debugging

New Monitoring Dashboard
The Monitoring Dashboard:

• Is accessible from the admin console
– Under “Charts and Graphs” on the Home page, click
Monitoring Dashboard.
• Provides views and tools for graphically representing
diagnostic data
– Multiple graph types are available.
• Allows users to monitor MBean attributes
– From active runtime MBeans (polled metrics)
– From an archive collected by the Harvester (collected
metrics)
• Replaces the WLDF console extension
The Monitoring Dashboard provides views and tools for graphically presenting diagnostic
data. The underlying functionality for generating, retrieving, and persisting diagnostic data is
provided by the WebLogic Diagnostics Framework (WLDF).
You can launch the Monitoring Dashboard from the WLS admin console, or you can run it
independently. To launch it from the admin console, go to the Home page and under “Charts
and Graphs” click the Monitoring Dashboard link. The dashboard opens in its own window (or
tab). If you are not logged in to the admin console when you launch the dashboard, you are
prompted for an admin-level username and password. To access the Monitoring Dashboard
directly, use the URL http:<host>:<port>/console/dashboard (for example,
http://localhost:7020/console/dashboard).
The diagnostic data displayed by the Monitoring Dashboard consists of runtime MBean
attributes. These values are referred to in the Monitoring Dashboard as metrics. The
dashboard obtains metrics from two sources:
• Directly from active runtime MBean instances. These metrics are referred to as polled
metrics.
• From Archive data that have been collected by the Harvester. These metrics are
referred to as collected metrics.

Viewing the Dashboard
• Install the Java plug-in (1.5 or later) within your web

browser prior to using the dashboard.
• On Linux, you can simply create a symbolic link to the Java
plug-in included in the JDK that is bundled with WLS.
The Monitoring Dashboard requires the Java plug-in version 1.5 (J2SE Runtime Environment
5.0) or later. If the required Java plug-in is not already installed in your Web browser, you may
be prompted to initiate a download from the Oracle Java web site when you access the
dashboard.
Follow the instructions on the screen. The exact installation procedure varies depending on
the type of browser and platform. For example, for Firefox on Linux, you typically are required
to copy or create a symbolic link to the file libjavaplugin_oji.so into your
<browser_install>/plugins directory.

Monitoring Dashboard Interface
Explorer
Metric Browser View Display
View List
The Monitoring Dashboard has two main panels: the explorer panel and the view display
panel.
The explorer panel has two tabs:
• View List: A list of existing built-in and custom views. It also contains controls for
creating, copying, renaming, and deleting views.
• Metric Browser: A way of navigating to and selecting the specific MBean instance
attributes whose metric values you want to display in a chart in a view

Views
Views:
• Are a way to organize your charts and graphs
• Typically display metrics that are related in some way
• Are individually started and stopped
• Continue to collect data even when not being displayed
Stop
Stop all
Start active views.
(disabled
since already
running)
Copy the
Active view selected view.
Inactive view
The View List tab lists views. A view is a collection of one or more charts that display captured
monitoring and diagnostic data. You can access, create, and update views from the View List
tab. When you click the name of a view on the View List tab, that view is displayed in the View
Display on the right.
The dashboard uses icons to indicate the status of a view. A gray icon indicates that the view
is inactive and data polling is not occurring for the charts in that view. A color icon indicates
that the view is active and data polling is occurring for all charts in that view (this is true
whether or not the view is currently displayed in the View Display).
To start the data collection for a view, click the view name in the list and click the green Start
button above the tabs. To stop data collection, click the red-and-white rectangular Stop
button. To stop all active views, click the red octagonal Stop All button.

Built-In Views
• The dashboard defines built-in views for some of the more

critical runtime performance metrics.
• Built-in views cannot be modified, but they can be copied
and the copy modified.
Basic JMS metrics for all servers
Basic heap metrics for

each server’s JVM
Preconfigured
views listed
per server
The built-in views are a set of predefined views of available runtime metrics for all running
WebLogic Server instances in the domain. These views surface some of the more critical
runtime WebLogic Server performance metrics and serve as examples of the dashboard’s
chart and graph capabilities.
You cannot modify a built-in view, but you can copy it. This copy is now one of your custom
views. As a custom view, the copy can be modified, renamed, saved, and later deleted.
Custom views are available only to the user who created them and only within the current
domain.

Creating a Custom View
Click New View in the View List.

• A view is a
collection of
charts.
• Each chart
contains one
or more
graphs.
• Each graph
is associated
with an In the Metric
MBean Browser, drag
attribute. and drop
MBean
attributes.
A custom view is any view created by a user. Custom views are available only to the user who
created them. You can access a custom view again when needed.
To create a new custom view, click the View List tab. Then click the New View button. A new
view appears in the list named New View. Replace the default name with something
meaningful. Also, a new empty view appears in the View Display area.
To add charts to the custom view, use the drop-down menu above the View Display area and
click New Chart.
To add graphs to a chart, first click the Metric Browser tab. Select a server in the Servers
drop-down list and click Go. Then select an MBean type and an MBean instance. In the
Metrics list for that instance, drag an MBean attribute to a chart. A view may have as many
charts as you like and a chart may graph as many metrics as you like. Also, if a metric is
dragged to a view that contains no charts, the dashboard automatically creates a new chart to
contain the graph.
When the metrics are in place, click the green Start button to start collecting data.
To delete a custom view, select the name of the view and click the Delete (red “X”) button.

Metric Browser
• After creating a custom

view, use the dashboard’s
Metric Browser to browse Select a server.
available MBeans on a
server.
MBean
– Drag attributes to a chart. types
• Metrics are obtained from
active MBeans in running
servers or the Archive. MBean
instances
– To view only harvested
(Archive) metrics, select
Collected Metrics Only.
MBean
attributes
The Metric Browser tab displays metrics available for a single server at a time. The Server
drop-down list displays all the servers in the domain. Select a server and click the Go button.
The metrics for the selected server are displayed. If the selected server is not running (is
currently shut down), a message indicates that the dashboard cannot connect to the server.
You can track a metric based on an attribute of any registered MBean simply by dragging the
attribute to a chart on the displayed view.
As a convenience, to have only metrics that have been collected by the Harvester displayed,
select the Collected Metrics Only check box.
To determine whether a metric was collected by the harvester, select the metric. A pop-up
note provides information about the metric, including whether or not it is a collected metric.
To see metrics for all runtime MBean types regardless of whether instances of them are
active, select Include All Types.

Anatomy of a Chart
Edit Tool
Y-axis
Graph with data

points (move cursor
over for details)
Menu
Pan and zoom

controls
Legend Chart
X-axis overview
Metric Menu
A chart contains one or more graphs that show data points over a specified time span. A chart
also includes a legend that lists the data sources for each graph along with their associated
icons and colors.
When working with a view, you can do the following:
• Add charts to views.
• Add graphs to charts.
• Pan and zoom.
• Edit labels and legends by using the Edit Tool.
• Start and stop data collection for charts in a view.

Chart and Graph Properties
Chart style
Properties of
selected chart
Individual
metric
properties
When you create a chart, it is created with a default type and default properties. To change
the chart type, use the chart menu and select Chart Type. The types available are Bar Chart,
Line Plot, Scatter Plot, Vertical Linear Gauge, Horizontal Linear Gauge, and Radial Gauge.
To change the properties for a chart, use the chart menu and select Properties. You can
change the Chart Title, Y-axis Units, Color, Background Color, and Highlight Color. You can
choose Set Y-axis Range Automatically, or you can choose Y-axis Max, Y-axis Min,
Threshold Max, and Threshold Min. You can also change the Time Range.
Each graph in the chart (metric) also has properties. Use the drop-down menu next to the
metric in the legend and select Properties. Here you can change the Name of the metric as
well as choose its Marker Symbol and Color.

Chart Styles
Name Description Example

Line Plot Sequential dots connected by lines
Scatter Plot Dots only
Bar Chart Bars for each metric
Vertical Linear Shows minimum, maximum, average,

Gauge and standard deviation vertically
Horizontal Shows minimum, maximum, average,
Linear Gauge and standard deviation horizontally
Radial Gauge Shows minimum, maximum, average,
and standard deviation radially

Current and Historical Data
• To view real-time polled metrics, no Harvester is needed.

When a view is started, the runtime MBean instances are
polled.
• To view historical (collected) metrics, a Harvester must first
be configured. Metrics collected by the Harvester are
placed in the Archive. To view harvested data:
– In the View List, click the New View button.
– In the Metric Browser, select a Server.
– To see only harvested data, select Collected Metrics Only.
– Drag some attribute from the Metrics list to the new view (a
new chart will be created for you).
The Monitoring Dashboard displays two kinds of diagnostic metrics: real-time data directly
from active runtime MBeans (called polled metrics), and historical data collected by a
previously configured WLDF Harvester (called collected metrics).
Note that with polled metrics, if polling has been taking place long enough for old data to be
purged, a view will not contain all data from the time polling started.
If a Harvester was configured to harvest data for a particular metric, that historical data is
available and can be displayed. Data that matches a selected metric and within a view’s time
range will be displayed by that view.

Section Summary

• Access the Monitoring Dashboard
• Use standard views and define custom ones
• Create charts and graphs for MBean attributes
• Customize the appearance of charts and graphs

Practice 4-2
Monitoring Diagnostic Metrics
• Accessing the Monitoring Dashboard
• Monitoring standard views in the Monitoring Dashboard
• Defining a custom view in the Monitoring Dashboard
• Configuring charts within a view

Road Map
• WLS Debugging
– Debug Scopes
– Debug Logging

Subsystem Debugging
• Various WLS subsystems have the ability to produce very

detailed log messages for debugging purposes.
• You can dynamically enable debugging on specific servers
and for individual subsystems.
1
2
1. Select an existing server.

2. Click the Debug tab. Use this page to define the debug settings for this server.
3. Select one or more available debugging scopes using the supplied check boxes. Then
click Enable or Disable. For convenience, a Clear button is also provided.

Console Debug Scopes
• Debugging flags or “attributes” for WLS subsystems are

classified and organized into scopes.
• When a parent scope is enabled, all child scopes are also
enabled unless they are overridden.
Scopes
Attribute
This debugging method is dynamic and can be used to enable debugging while the server is
running. Alternatively, many debug flags can also be set as command-line arguments when
starting a server.
Examples
• -Dweblogic.debug.DebugJDBCSQL=true
• -Dweblogic.debug.DebugJMSBackEnd=true
• -Dweblogic.debug.DebugSAFSendingAgent=true
• -Dweblogic.debug.DebugJDBCJTA=true

Debug Scopes: Examples
Subsystem Scopes (weblogic.*)

JDBC jdbc.connection, jdbc.sql, jpa.jdbc, jdbc.internal
JMS jms.module, store, jms.store, jms.durablesubscribers,

messaging.kernel
Cluster core.cluster
Deployment deploy, ejb.deployment
Applications servlet, servlet.internal, servlet.internal.session, ejb.pooling,
ejb.caching, ejb.invoke, application.library
Transactions transaction.xa, transaction.twopc, transaction.recovery, jms.xa
Security security, security.ssl, security.ldap
WLDF diagnostics.harvester.core, diagnostics.watch
• weblogic.jdbc.sql.DebugJDBCSQL: Prints information about all JDBC methods

invoked, including their arguments and return values, and thrown exceptions
• weblogic.jdbc.connection.DebugJDBCConn: Traces all connection reserve and
release operations in data sources as well as all application requests to get or close
connections
• weblogic.jdbc.internal.DebugJDBCInternal: Low level debugging related to
the data source, the connection environment, and the data source manager
• weblogic.jms.backend.DebugJMSBackEnd: Prints information for debugging the
JMS Back End (including some information used for distributed destinations and JMS
SAF)
• weblogic.jms.frontend.DebugJMSFrontEnd: Prints information for debugging
the JMS Front End (including some information used for multicast)
• weblogic.jms.common.DebugJMSCommon: Prints information for debugging JMS
common methods (including some information from the client JMS producer)
• weblogic.jms.boot.DebugJMSBoot: Prints some messages at boot time regarding
what store the JMS server is using and its configured destinations
• weblogic.jms.module.DebugJMSModule: Prints information about JMS module
operations and message life cycle

Debug Logging
• By default, debug messages are written to the server log

but not to other destinations.
• While troubleshooting, modify the minimum severity of log
streams to include debug messages as necessary.
You can enable debugging by setting the appropriate Severity Level server logging
configuration attributes to true.
Under normal circumstances, WebLogic Server subsystems generate many messages of
lower severity and fewer messages of higher severity. In addition to writing messages to a log
file, each server instance prints a subset of its messages to standard out. Usually, standard
out is the shell (command prompt) in which you are running the server instance.
• Log File: Severity Level: The minimum severity of log messages going to the server
log file. By default, all messages go to the log file.
• Standard Out: Severity Level: The minimum severity of log messages going to the
standard out
• Memory Buffer: Severity Level: The minimum severity of log messages that WebLogic
accepts from subsystems and applications and then keeps in memory and eventually
distributes to destinations such as standard output or the log file

WLST Debugging: Examples
Enable specific server debug attributes:
debug = getMBean('/Servers/serverA/ServerDebug/serverA')
debug.setDebugJDBCInternal(true)
debug.setDebugJMSBackEnd(true)
debug.setDebugSSL(true)
Enable an entire debug scope:
scope =
getMBean('/Servers/serverA/ServerDebug/serverA/DebugScopes/web
logic.jdbc')
scope.setEnabled(true)
The MBean used in the slide, ServerDebugMBean, is not currently documented in the main
MBean Reference guide online, but it is documented in the MBean API Reference (Javadoc).
Search for the type weblogic.management.configuration.ServerDebugMBean. You
can also inspect this or any other MBean’s available attributes by using WLST in online mode.
For example, browse to the above location and execute the ls command.
Unlike in the admin console, the MBean used to maintain debug attributes does not organize
them using scopes. Debug scopes are simply a convenience of the console user interface.

Section Summary

• Enable debugging for specific WLS features
• Direct debug messages to log destinations

Quiz
Which of these is not a component of WLDF?

a. Watch
b. Image Capture
c. Archive
d. Guardian
e. Harvester
Answer: d

Quiz
Which of these is not an available option when you configure a

collected metric?
a. Type
b. Method
c. Attribute
d. Instance
e. Period
Answer: b

Quiz
Name three types of WLDF notifications.

a. JDBC
b. SNMP
c. SMTP
d. JMS
e. FTP
Answer: b, c, d

Quiz
In the WLDF console, a chart can include zero or more ____.

a. Graphs
b. Ports
c. Watches
d. Views
e. Flags
Answer: a

Summary

• Describe the main components of the WebLogic
Diagnostic Framework
• Explain the capabilities of the diagnostic Harvester
• Configure diagnostic archives and modules
• Record MBean attributes by using WLDF
• Trigger notifications under specific MBean conditions
• Plot current and historical metrics by using the Monitoring
Dashboard
• Enable debugging messages for server subsystems

Diagnostic Instrumentation

Objectives

• Describe some typical uses of WLS instrumentation
• Configure application-scoped instrumentation by using
WLS deployment plans
• Explain the relationship between instrumentation and AOP
• Configure WLDF monitors and actions from the WLS
library
• Define custom monitors and pointcuts
• Trace log messages for a specific client request
• Filter instrumentation events by using dyes

Road Map
• Monitors
– Architecture
– Actions
– Application-Scoped Modules
– Hot Swap
– Pointcuts
• Request Tracking

Instrumentation Scenarios
• Use the WLDF instrumentation component to collect

diagnostic data at certain points in WLS or application
code.
• Typical scenarios include:
– Determining which code is using a data source or queue
– Determining the source of HTTP session data
– Calculating how much time some code takes to execute
– Generating log messages that trace the flow of execution
– Taking a snapshot of the server at a certain point in the
application
A diagnostic monitor is a dynamically manageable unit of diagnostic code that is inserted into
server or application code at specific locations. Monitors allow you to better troubleshoot
issues that occur at the specific points in the server or application's flow of execution. They
can also be used to simply help identify the part of an application that is causing some
behavior. This approach can be especially useful to administrators who may not have direct
access to an application’s source code or documentation, or at least not have it readily
available. The concept of instrumentation assumes that the underlying application code
remains unmodified.

Instrumentation Architecture
Diagnostic monitors:
• Trigger actions at specific code locations
• Record the data from actions as events in the archive
Event
Instrumentation archive
Monitors
Actions
Code
Notifications
Watches
Diagnostic module
A diagnostic monitor is a dynamically manageable unit of diagnostic code that is inserted into
server or application code at specific locations. WLDF provides a library of predefined
diagnostic monitors and actions. You can also create application-scoped custom monitors,
where you control the locations where diagnostic code is inserted in the application.
Monitors are applied and removed from server and application Java code dynamically without
modifying the code itself. The WLDF instrumentation code is inserted or “woven” into server
and application code at precise locations. A joinpoint is a specific location in a class, for
example the entry and/or exit point of a method or a call site within a method. A pointcut is an
expression that specifies a set of joinpoints, for example all methods related to scheduling,
starting, and executing work items. For a monitor to perform any useful diagnostic work, you
must configure at least one action for the monitor. Only certain actions are available to certain
monitor types.
You use instrumentation watches to monitor the events from the WLDF instrumentation
component, similar to monitoring the MBean data collected from the Harvester component.
Watches of this type are triggered as a result of the event being posted, or when the event’s
data meets some conditions. Recall that watches respond with notifications, such as capturing
a server diagnostic image.

Monitor Actions
Action Data Collected

Display Arguments Either the values of the input arguments or the return value of
the methods at this code point
Trace Elapsed Time The time in nanoseconds that this code point took to execute
Trace Memory The number of bytes allocated by a thread while running the
Allocation code point
Stack Dump The list of method calls that led to this code point
Thread Dump The methods that all server threads are executing at this
code point
Trace Simply an indicator that this code point has executed. This
also generates a log message of TRACE severity.
Method Invocation The running minimum, maximum, average, and total time
Statistics that this code point took to execute
Method Memory The number of bytes allocated by a thread during a method
Allocation Stats call
Diagnostic actions perform some type of data collection intended to help gain insight into the
server or application. Each diagnostic action can only be used with the monitor types with
which they are compatible. In addition to the data described above, all actions also capture
general statistics such as the current time, transaction ID, and user ID, if applicable.
When attached to “before” monitors, the Display Arguments instrumentation event captures
input arguments to the joinpoint (for example, method arguments). When attached to “after”
monitors, the instrumentation event captures the return value from the joinpoint.
When executed, the Trace Elapsed Time action captures the timestamps before and after the
execution of an associated joinpoint. It then computes the elapsed time by computing the
difference. It generates an instrumentation event which is dispatched to the events archive.
The elapsed time is stored as event payload.
Trace Memory Allocation is very similar to the Trace Elapsed Time, except that the memory
allocated within a method call is traced, rather than the time to run the method.

A Method Invocation Statistics action computes statistics in memory without persisting an
event for each invocation. It makes the collected information available through the
InstrumentationRuntimeMBean, which is consumable by WLST, the Harvester, or
watches. This makes it possible to create watch rules that can combine request information
from the instrumentation system and metric information from other runtime MBeans.
The Method Memory Allocation Statistics action is very similar to the Method Invocation
Statistics action, except that the memory allocated within the code point is tracked, rather than
the time to run it.

Application-Scoped Modules
• Define monitors in WLDF system modules or application-

scoped modules.
• Configure application-scoped monitors using the descriptor
weblogic-diagnostics.xml in the META-INF
directory.
• Instrumentation must be enabled at the system level to
activate monitors in application-scoped modules.
System
module
You can provide instrumentation services at the system level (servers and clusters) and at the
application level. Many concepts, services, configuration options, and implementation
features are the same for both. However, there are differences, including the types of
monitors that are available.
Server-scoped instrumentation for a server or cluster is configured and deployed as part of a
diagnostic module, an XML configuration file located in the
<DOMAIN_NAME>/config/diagnostics directory, and linked from config.xml. Only
one WLDF system resource (and hence one system-level diagnostics descriptor file) can be
active at a time for a server (or cluster). Server-scoped instrumentation can be enabled,
disabled, and reconfigured without restarting the server.
Application-scoped instrumentation is also configured and deployed as a diagnostics module,
but as an XML configuration file named weblogic-diagnostics.xml, which is packaged
with the deployed application in the META-INF directory. As with all deployment descriptors,
simply redeploy the application for your latest instrumentation settings to take effect. For
instrumentation to be available for an application, instrumentation must be enabled on the
server to which the application is deployed.

WLS Monitor Library
System Application
Subsystem Available Code Points
Module? Module?
JDBC Before/After Connection X X
Reserve/Release Connection X X
Before/After Commit X X
Before/After Rollback X X
Before/After SQL Statement X X
JMS Before/After Message Send X
Before/After Message Receive X
JNDI Before/After Lookup X
JTA Before/After Start X
Before/After Commit X
Before/After Rollback X
Diagnostic monitors are broadly classified as server-scoped and application-scoped monitors.

The former can be used to instrument WebLogic Server classes. You use the latter to
instrument application classes. Except for the Dye Injection monitor, none of the monitors
have a built-in diagnostic action. Instead, they delegate to actions (trace, display arguments,
and so on) attached to them to perform diagnostic activity. For any delegating monitor, only
compatible actions may be attached. The compatibility is determined by the nature of the
monitor.
All monitors are preconfigured with their respective pointcuts. However, the actual locations
affected by them may vary depending on the classes they instrument. For example, the
Servlet_Before_Service monitor adds diagnostic code at the entry of servlet or Java server
page (JSP) service methods at different locations in different servlet implementations.
Although not shown in the slide, WLS also includes monitors in its library to troubleshoot Java
connectors, which are a type of Java EE application that helps other applications
communicate and integrate with some external system. Like data sources, connectors can
also participate in XA transactions using JTA.

WLS Monitor Library
System Application
Subsystem Available Joinpoints
Module? Module?
Servlet/JSP Before/After Execution X
Before/After Session Access X
Before/After Tag Execution X
EJB Before/After All Entity Methods X
Before/After Entity Business X
Methods
Before/After All Session Methods X
Before/After Session Business X
Methods
Before/After MDB Message Receive X
Before/After MDB Created X
Before/After MDB Removed X
EJBs define one or more custom “business” methods that clients can execute, but all EJBs
also include a set of standard “semantic” methods such as ejbActivate(), ejbLoad(),
and ejbRemove(). WLDF monitors are available to troubleshoot only EJB business
methods, only EJB semantic methods, or both types of methods.

Deployment Plan Review
Deployment plans:
• Add or override elements in deployment descriptors
• Are associated with an application during deployment
• Can be created with the help of the console or command-
line tools
plan.xml
DEPLOY TestDomain
MSQLDataSource
MyEJBApp
ejb-jar.xml
plan.xml
ORADataSource DEPLOY ProdDomain
A deployment plan is an XML document that is used to define an application’s deployment

configuration for a specific environment, such as development, test, or production. A
deployment plan resides outside of an application’s archive file and contains deployment
properties that add to or override those found in an application’s existing Java EE and WLS-
specific descriptors. Multiple deployment plans can be used to reconfigure a single application
for deployment to multiple systems, without requiring changes to the application archive itself.
Any external resources required by the application are subject to change when the application
is deployed to a different environment. For example, the Java Naming and Directory
Interface (JNDI) names of the data sources that are used in your development environment
can be different from those used in testing or production. Exposing those JNDI names as
variables makes it easy for administrators to use the available resources or create the
required resources when deploying the application.
Certain tuning parameters that are acceptable in a development environment are
unacceptable in a production environment. For example, it may suffice to accept default or
minimal values for EJB caching on a development machine, whereas a production cluster
would need higher levels of caching to maintain acceptable performance. To deploy the
application to a new environment, an administrator simply creates or uses a new deployment
plan as necessary.

WLDF and Deployment Plans
• Application-scoped instrumentation is an ideal candidate

for a deployment plan.
• Oracle recommends that you:
– Include an empty WLDF descriptor in each application
– Use deployment plans to add diagnostic monitors as needed
To make modifications to application-scoped diagnostic monitors, you must either edit

weblogic-diagnostics.xml directly within the application, or associate the application
with a deployment plan and edit the plan file. If you want to reconfigure an application that
was deployed without a deployment plan, you must undeploy, unarchive, reconfigure, re-
archive, and then redeploy the application. With a configuration plan, you can dynamically
change many configuration options simply by updating the plan without modifying the
application archive.
You can create deployment plans using the weblogic.PlanGenerator tool. The
weblogic.PlanGenerator tool creates a template deployment plan with placeholder
variables for selected categories of WebLogic Server deployment descriptors. After the plan is
created by the tool, modify the deployment plan either manually or using the administration
console.
To enable instrumentation for an application, instrumentation must be enabled for the server
on which the application is deployed. If server instrumentation is enabled at the time of
deployment, instrumentation is available for the application. If instrumentation is not enabled
on the server at the time of deployment, enabling instrumentation in an application has no
effect.

WLDF Deployment Plan: Example
Add an application-scoped monitor using a deployment plan:
<variable>
<name>EJBMonitor-Action</name>
<value>TraceAction</value> Set the action element
</variable> for a monitor with a
... given name.
<module-descriptor external="false">
<root-element>wldf-resource</root-element>
<uri>META-INF/weblogic-diagnostics.xml</uri>
<variable-assignment>
<name>EJBMonitor-Action</name>
<xpath>/wldf-resource/instrumentation
/wldf-instrumentation-monitor/[name="EJBMonitor"]
/action</xpath>
</variable-assignment>
...
</module-descriptor>
The weblogic-diagnostics.xml file supports the following elements, which can be

added or modified using deployment plans:
• <enabled>: Enable/disable all monitors defined for this application.
• <include>: A list of Java classes or packages that are allowed to be monitored using
instrumentation (optional)
• <exclude>: A list of Java classes or packages that are forbidden from being monitored
using instrumentation (optional)
• <wldf-instrumentation-monitor>: Define a monitor configuration.
Each monitor supports the following elements:
• <name>: The name of a built-in system monitor or a new custom monitor
• <enabled>: Enable or disable this specific monitor.
• <action>: The action to perform
Custom monitors support additional elements, which are described later.

WLDF Hot Swap
Hot swap:
• Automatically detects and applies changes to deployment
plans
• Does not require that you redeploy the application to add
or remove monitors
• Is enabled using a JVM argument
startWebLogic.sh:
...
JAVA_OPTIONS="${JAVA_OPTIONS}
-javaagent:${WL_HOME}/server/lib/diagnostics-agent.jar"
...
If you enable a feature called “hot swap” before deploying your application with a deployment
plan, you can dynamically update all instrumentation settings without redeploying the
application. If you do not enable hot swap, or if you do not use a deployment plan, changes to
some instrumentation settings require redeployment. If hot swap is not enabled, you can
“remove” a monitor, but that just disables it. The instrumentation code is still woven into the
application code. You cannot re-enable it through a modified plan without also redeploying the
application.
It is recommended that you create an empty descriptor. That provides full flexibility for
dynamically modifying the configuration. It is possible to create monitors in the original
descriptor file and then use a deployment plan to override the settings. You will, however, be
unable to completely remove monitors without redeploying. If you add monitors using a
deployment plan to an empty descriptor, all such monitors can be removed.

Configuring a System-Scoped Monitor
1. Click the name of the module to which you want to add diagnostic monitors. Then select
Configuration > Instrumentation.
2. Click the Add/Remove button.
3. To enable (or disable) the monitor, select (or deselect) Enabled. Then locate the
Diagnostic Monitors section and select one or more monitors from the Available list.
Click the right arrow button to move the monitors to the Chosen list. The name of the
monitor indicates where in the flow of execution the monitoring takes place.

Configuring a System-Scoped Monitor
4. If you are editing a delegating monitor and you want to add Actions, click the desired
actions in the Available list and move them to the Chosen list.
5. Click Finish.

Configuring an Application-Scoped Monitor
1
Create or edit
deployment plan.
2
1. Click the name of an existing application. Then select Configuration > Instrumentation.
2. Click the Add Monitor From Library button.
3. Locate the Diagnostic Monitors section and select one or more monitors from the
Available list. Click the right arrow button to move the monitors to the Chosen list. The
name of the monitor indicates where in the flow of execution the monitoring will take
place. Click OK.
4. Choose a Path for the deployment plan and click OK.
5. Click the name of a monitor to edit its settings.
6. To enable or disable the monitor, select or deselect Enabled.
7. Under Actions, click the desired actions in the Available list and move them to the
Chosen list. Click Save.

Aspect-Oriented Programming (AOP) Concepts
• AOP allows code to be inserted within an existing program

without editing the original source.
• WLS includes a Java AOP engine called AspectJ for
WLDF instrumentation.
• Joinpoint: Identifies a method in a Java class
– A call joinpoint occurs when a method is invoked.
– An execution joinpoint occurs before and after the method.
• Pointcut: Expression defining one or more joinpoints
Pointcut Description
* com.mycompany.MyClass doIt(...) A specific method in a specific class
* *.MyClass do*(...) All methods named “do” in all classes
named “MyClass”
Traditionally in object-oriented programming, if you wanted to implement a service that will cut
across your business logic modules, you would write a separate module, such as logging, and
then provide access to that module through an abstract interface. Your business logic
modules can do their logging by using the logging services, and you can change how you do
logging by changing the logging module without touching the business logic. The only
potential problem with this is that you have to embed calls to the logging module within the
business logic. You cannot flexibly or easily change where and when you want the logging to
occur. Aspect-oriented programming builds on top of object-oriented programming. You
define the points at which you want something to happen, and then you define what you want
to happen (for example, logging). You are essentially defining rules in which to weave new
code into your program. Common AOP examples include diagnostics, security, transactions,
resource pooling, and persistence.
A joinpoint is an identifiable point in the execution of a program. Currently, WLDF only
supports joinpoints that describe Java methods or constructors. WLDF supports “call” and
“execution” joinpoints, whose differences are very subtle. At a “call” joinpoint, an action will
occur when a method is invoked but before its logic is actually executed. An “execution”
joinpoint occurs around (before and after) a method’s execution. A pointcut is an expression
that specifies a set of joinpoints by matching certain characteristics. Pointcuts can also use
logical operators such as AND, OR, and NOT.

Custom Monitors
• Application-scoped modules support custom monitors to

define your own instrumentation pointcuts.
• Before and After monitors must specify call joinpoints.
• Around monitors must specify execution joinpoints.
2
call(...) or
execution(...)
A custom monitor is available only for application-scoped instrumentation and does not have
a predefined pointcut or location. A diagnostic location is the position relative to a joinpoint
where the diagnostic activity will take place. Diagnostic locations are before, after, and
around. You assign a name to a custom monitor, define the pointcut and the diagnostics
location the monitor will use, and then assign actions from the set of predefined diagnostic
actions.
To add a custom monitor:
1. Edit an application and click the Configuration > Instrumentation tab.
2. Click the Add Custom Monitor button.
When configuring a custom pointcut in WLS, the following syntax rules apply:
• Indicate either a “call” or “execution” pointcut.
• Specify a type (class or interface) and method name. Wildcards (*) can be used in class
types and method names. Use a “+” prefix to indicate all subclasses, subinterfaces, or
concrete classes implementing the specified class or interface.
• (Optional) Indicate method arguments and return type.
• Pointcut expressions can be combined with AND, OR, and NOT Boolean operators to
build complex pointcut expression trees.

Instrumentation WLST: Example
Add a built-in monitor to a WLDF system module:
instr = getMBean('/WLDFSystemResources/MyModule/
WLDFResource/MyModule/Instrumentation/MyModule')
instr.setEnabled(true)
monitor = instr.createWLDFInstrumentationMonitor(
'JDBC_Before_Connection_Internal')
monitor.setEnabled(true)
monitor.setActions(array(['StackDumpAction'],String))
documentation.
• WLDFInstrumentationBean (a subclass of WLDFResourceBean)
• WLDFInstrumentationMonitorBean (a component of
WLDFInstrumentationBean)

Instrumentation and Request Performance
To request performance data on the admin console’s

Diagnostics Request Performance page:
1. Configure WLDF instrumentation.
– The instrumentation must use the ElapsedTimeAction.
– That action must be attached to “Around” diagnostic
monitors.
2. Launch the admin console.
3. In the Domain Structure, select Diagnostics > Request
Performance.
4. Select a server and a time interval from the drop-down
lists.
If server-scoped or application-scoped instrumentation has been configured, you can display

request performance data in the WebLogic Server Administration Console. The Request
Performance page displays information about the real-time and historical views of method
performance that has been captured by means of the WebLogic Diagnostics Framework
instrumentation capabilities.
To create request performance data, the following criteria must be met:
• A WLDF system resource must be created and targeted to the server. This resource
may be created by using the WebLogic Server Administration Console or the WebLogic
Scripting Tool (WLST).
• Instrumentation in the targeted WLDF system resource must be enabled.
• Application instrumentation must be enabled with a weblogic-diagnostics.xml descriptor,
which you create in the application’s META-INF directory.
• Application instrumentation descriptors must use TraceElapsedTimeAction diagnostic
actions attached to "Around“-type diagnostic monitors.

Section Summary

• Explain the architecture of the WLDF instrumentation
component
• List some monitor and action types available in the
instrumentation library
• Use deployment plans to configure application-scoped
WLDF modules
• Configure standard and custom monitors

Practice 5-1
Configuring and Monitoring Diagnostic Events
• Using the console to generate and edit a deployment plan
• Using the console to configure application instrumentation
• Configuring built-in diagnostic monitors
• Defining pointcuts for custom monitors
• Inspecting events found in diagnostic archives

Road Map
• Monitors
• Request Tracking
– Context ID
– Request Dyes
– Event Filtering
– Event Throttling

Request Context ID
• When instrumentation is enabled on a server, a context ID

is assigned to each incoming client request.
• A context ID:
– Follows the request as it passes through various WLS
subsystems
– Is preserved for a request that spans multiple servers
– Is unique across an entire domain
Server Server
Web EJB EJB
ID: 1111 ID: 1111
ID: 1112
The WLDF Instrumentation component provides a way to uniquely identify requests (such as
HTTP or RMI requests) and track them as they flow through the system. The diagnostic
context consists of two pieces: a unique context ID and a 64-bit dye vector that represents the
characteristics of the request. The context ID associated with a given request is recorded in
the event archive and can be used to associate other server log messages with the same
request.
The diagnostic context for a request is created and initialized when the request enters the
system (for example, when a client makes an HTTP request). The diagnostic context remains
attached to the request, even as the request crosses thread boundaries and Java Virtual
Machine (JVM) boundaries. The diagnostic context lives for the duration of the life cycle of the
request. Every diagnostic context is identified by a Context ID that is unique in the domain.
Because the Context ID travels with the request, it is possible to determine the events and log
entries associated with a given request as it flows through the system.

Viewing Context IDs
All server log and WLDF archive entries include the current
context ID (if available).
By default, the Context ID column for server log files is not shown in the console. When it is
added, you can quickly find all of the log and WLDF messages associated with a single
request as it passes through different applications and WLS subsystems.
You can also filter the displayed log messages by Context ID, as in the following example:
CONTEXTID LIKE '%abcd%'

Request Dying
• Diagnostic contexts also include a series of flags or dyes.

• A special diagnostic monitor, the Dye Injector, intercepts
server requests and initializes these dyes based on
different conditions.
Dye Server
Injector
ID: 1111
ID: 1112
ID: 1113
When any request enters the system, WLDF creates and instantiates a diagnostic context for
the request. The context includes a unique context ID and a dye vector. The Dye Injection
monitor, if enabled at the server level within a WLDF diagnostic module, examines the
request to see if any of the configured dye values in the dye vector match attributes of the
request. For example, it checks to see if the request originated from the user associated with
USER1 or USER2, and it checks to see if the request came from the IP address associated
with ADDR1 or ADDR2.
For each dye value that matches a request attribute, the Dye Injection monitor sets the
associated dye bits within the diagnostic context. For example, if the Dye Injection monitor is
configured with USER1=weblogic, USER2=admin@avitek.com, ADDR1=127.0.0.1,
ADDR2=127.0.0.2, and the request originated from user weblogic at IP address
127.0.0.2, it sets the USER1 and ADDR2 dye bits within the dye vector.

Available Dyes
#
Dye Type Condition Based On
Available
ADDR 4 The incoming client IP address
USER 4 The user ID associated with this request
PROTOCOL 7 The protocol of the incoming request (HTTP,
RMI, SSL, T3, and so on)
COOKIE 4 The value of a browser cookie named
weblogic.diagnostics.dye
DYE 8 A value set programmatically using the WLDF
APIs
Use the ADDR1, ADDR2, ADDR3, and ADDR4 dyes to specify the IP addresses of clients that
originate requests. These dye flags are set in the diagnostic context for a request if the
request originated from an IP address specified by the respective property. These dyes
cannot be used to specify DNS names.
Use the USER1, USER2, USER3, and USER4 dyes to specify the user names of clients that
originate requests. These dye flags are set in the diagnostic context for a request if the
request was originated by a user specified by the respective property.
COOKIE1, COOKIE2, COOKIE3, and COOKIE4 are set in the diagnostic context for an HTTP
or HTTPS request, if the request contains the cookie named weblogic.diagnostics.dye
and its value is equal to the value of the respective property.
PROTOCOL_HTTP is set in the diagnostic context of a request if the request uses HTTP or
HTTPS protocol. PROTOCOL_SSL is set in the diagnostic context of a request if it uses the
Secure Sockets Layer (SSL) protocol. Similar flags are available named PROTOCOL_IIOP,
PROTOCOL_JRMP, PROTOCOL_T3, PROTOCOL_RMI, and PROTOCOL_SOAP.
DYE_0 to DYE_7 are available for use only by application developers.

Configuring a Dye Injection Monitor
Add the
monitor.
Use monitor properties to

configure dye conditions.
The PROTOCOL dyes

are set implicitly.
To initialize the dye vector for requests coming into the system, you must:
1. Create and enable a diagnostic module for the server (or servers) that you want to
monitor.
2. Enable instrumentation for the diagnostic module.
3. Configure and enable the Dye Injection monitor for the module. Only one Dye Injection
monitor can be used with a diagnostic module at any one time.
4. For Properties, enter the list of conditions for each dye you would like to use, such as
ADDR1, ADDR2, USER1, USER2, COOKIE1, COOKIE2, and so on.

Event Filtering
• Other diagnostic monitors can limit which types of requests

trigger their actions by using dyes.
• A dye mask indicates which dyes must be set for an event
to be recorded.
Dye MonitorA
Injector Mask: D2
Event
archive
ID: 1111
ID: 1112
ID: 1113
D2
One of the main reasons for using diagnostic context is to filter requests. Although you may
want a diagnostic action to trigger when something happens within the server or the
application, you may not want it to happen every time for every request. Dye filtering is one of
the ways you can restrict how many actions are triggered. Dye filtering is done using dye
masks. A dye mask is a is a selection of dyes from the Dye Injection monitor whose
conditions must evaluate to “true.”

Configuring Dye Masks
Edit any
monitor.
Enable
filtering.
Which dyes
must be set?
After editing an existing server-scoped or application-scoped monitor, update the Dye Mask
section. Move any of the available dyes from the Available list to the Chosen list. Also be sure
to select the EnableDyeFiltering check box. Then click Save. Remember that for application-
scoped instrumentation, these changes are saved in the application’s deployment plan.
The flags that are enabled in the diagnostic monitor must exactly match the bits set in the
request’s dye vector for an action to be triggered and an event to be written to the event
archive. For example, if the diagnostic monitor has both the USER1 and ADDR1 flags enabled,
and only the USER1 flag is set in the request’s dye vector, no action is triggered and no event
is generated.
When configuring a diagnostic monitor, do not enable multiple flags of the same type. For
example, do not enable both the USER1 and USER2 flags, because the dye vector for a given
request will never have both the USER1 and USER2 flags set.

Event Throttling
• The Dye Injector also supports properties that control how

often requests are:
– Processed by other monitors
– Simply ignored
• This diagnostic technique is known as throttling (or
sampling).
Dye Injector MonitorA
Event
ID: 1111 archive
ID: 1112
ID: 1113
ID: 1114
Throttling is used to control the number of requests that are processed by the monitors in a
diagnostic module. Throttling is configured using the THROTTLE dye, which is defined in the
Dye Injection monitor. The USERn and ADDRn dyes allow inspection of requests from specific
users or IP addresses. However, they do not provide a means to look at arbitrary user
transactions. The THROTTLE dye provides that functionality by allowing sampling of requests.
If dye filtering for a monitor is enabled and that monitor has a dye mask, filtering is performed
based on the dye mask. That mask may include the THROTTLE dye, but it does not have to. If
THROTTLE is included in a dye mask, then THROTTLE must also be included in the request’s
dye vector for the request to be passed to the monitor. However, if THROTTLE is not included
in the dye mask, all qualifying requests are passed to the monitor, whether their dye vectors
include THROTTLE or not.
On the other hand, if dye filtering is not being used, you can still use the Dye Injection monitor
to configure throttling of event data. The throttling feature is not dependent on the use of filters
in your monitors.

Configuring Throttle Properties
Edit the Dye

Injection monitor.
Sample one request

every five seconds.
Sample one request out

of every ten.
THROTTLE_INTERVAL sets an interval (in milliseconds) after which a new incoming request
is dyed with the THROTTLE dye. If the THROTTLE_INTERVAL is greater than 0, the Dye
Injection monitor sets the THROTTLE dye flag in the dye vector of an incoming request if the
last request dyed with THROTTLE arrived at least THROTTLE_INTERVAL before the new
request. For example, if THROTTLE_INTERVAL=3000, the Dye Injection monitor waits at
least 3000 milliseconds before it dyes an incoming request with THROTTLE.
THROTTLE_RATE sets the rate (in terms of the number of incoming requests) by which new
incoming requests are dyed with the THROTTLE dye. If THROTTLE_RATE is greater than 0,
the Dye Injection monitor sets the THROTTLE dye flag in the dye vector of an incoming
request when the number of requests since the last request dyed with THROTTLE equals
THROTTLE_RATE. For example, if THROTTLE_RATE=6, every sixth request is dyed with
THROTTLE.
You can use THROTTLE_INTERVAL and THROTTLE_RATE together. If either condition is
satisfied, the request is dyed with the THROTTLE dye.

Section Summary

• Use context IDs to map log messages to client requests
• Describe the capabilities of the Dye Injection monitor
• Filter WLDF events by using dye conditions and masks
• Define event sampling rates and sizes by using the Dye
Injector

Quiz
Which of the following is not an available action for a WLDF

monitor?
a. Start Server
b. Stack Dump
c. Elapsed Time
d. Display Arguments
e. Thread Dump
Answer: a

Quiz
What is the name of the deployment descriptor used to

configure application-scoped instrumentation?
a. weblogic-monitor.xml
b. weblogic-diagnostics.xml
c. weblogic-wldf.xml
d. weblogic-ext.xml
e. weblogic.xml
Answer: b

Quiz
Custom monitors include a _____ to identify one or more

locations within your application code.
a. Dye Mask
b. Harvester
c. Join Unit
d. Pointcut
e. Context
Answer: d

Summary

• Describe some typical uses of WLS instrumentation
• Explain the relationship between instrumentation and AOP
• Configure WLDF monitors and actions from the WLS
library
• Define custom monitors and pointcuts
• Trace log messages for a specific client request
• Filter instrumentation events by using dyes

Practice 5-2
Tracing a Client Request
• Enabling server debug log messages
• Using context IDs to associate log entries with a request
• Configuring a Dye Injection monitor
• Filtering recorded events by using dye masks

JVM Diagnostics

Objectives

• Explain basic JVM concepts, such as heap, garbage
collection, and memory leaks
• Update the memory and diagnostic settings for a JVM
• Investigate a JVM thread dump
• Use command-line Sun and JRockit diagnostic tools
• Monitor the JVM by using JVisualVM, WLS, and JRockit
Mission Control

Road Map
• JVM Concepts
– JVM Support
– Heap
– Garbage Collection
– Basic Settings
– Memory Leak
– Crash Files
• Basic JVM Tools
• WLS JVM Tools
• JRockit Mission Control

Basic Java Concepts
• Java code is organized into classes (generically referred to

as types).
• An instance of a class in memory is called an object.
• The class definitions themselves also must be loaded into
memory.
• Classes are organized into packages, which map to folders
on the file system.
com.mycompany.commerce.InventoryManager
An object is a software bundle of related state and behavior. Software objects are often used
to model business entities and processes. Software objects are conceptually similar to real-
world objects—they consist of state and related behavior. An object stores its state in fields
(variables in some programming languages) and exposes its behavior through methods
(functions in some programming languages). Methods operate on an object’s internal state
and serve as the primary mechanism for object-to-object communication.
A class is a blueprint or prototype from which objects are created. An object is an instance of
a specific class. An interface is a contract between a class and the outside world. When a
class implements an interface, it promises to provide the behavior published by that interface.
The concepts of classes and interfaces are often generalized using the term “type.”
A package is a namespace for organizing classes and interfaces in a logical manner.
Because software written in the Java programming language (particularly Java EE systems)
can be composed of thousands of individual classes, it makes sense to keep things organized
by placing related classes and interfaces into packages. Conceptually you can think of
packages as being similar to different folders on your computer. The Java platform provides
an enormous class library (a set of packages) suitable for use in your own applications, which
represent common, general-purpose programming requirements. For example, a
java.io.File object allows you to easily read, create, update, and delete folders and files
on the file system.

Java Virtual Machine (JVM): Review
After they are compiled, Java programs can be run on any

platform that has a supported JVM.
HelloWorld.java
Compile
HelloWorld.class
JVM JVM JVM

Windows Linux Solaris
Java programs are compiled into a form called Java bytecodes. The JVM executes Java
bytecodes, so Java bytecodes can be thought of as the machine language of the JVM. The
Java compiler reads Java language source (.java) files, translates the source into Java
bytecodes, and places the bytecodes into class (.class) files. The compiler generates one
class file per class in the source.
To the JVM, a stream of bytecodes is a sequence of instructions. Each instruction consists of
a one-byte opcode and zero or more operands. The opcode tells the JVM what action to take.
If the JVM requires more information to perform the action than just the opcode, the required
information immediately follows the opcode as operands.
The graphic in the slide shows how source code is compiled into class files that can be used
by JVMs running on multiple operating systems.

Oracle JVM Support
• WLS relies on its host JVM to manage memory and I/O.

• Refer to the “Supported Configurations” section of the
documentation to answer these questions:
– Is WLS certified on your host OS?
– Is your JVM release certified on your host OS?
– Are WLS and your JVM certified on your hardware
architecture?
– Are WLS and your JVM certified against any virtualization
technologies you employ?
x86, 32bit Oracle Enterprise Linux (UL7+) JRockit 6 Update 5 R27+
Note: The following information is subject to change. Always refer to the latest support
documentation.
If an Oracle product has been certified against and is supported on a version of Red Hat
Enterprise Linux (RHEL), it is automatically certified and supported on the corresponding
version of Oracle Enterprise Linux (OEL) (for example, RHEL4 > OEL4, RHEL5 > OEL5).
If a product is supported and certified on OEL or RHEL, it is also certified and supported in the
virtualized installation of the same version of OEL or RHEL running on Oracle VM (for
example, OEL4 > OEL4 on Oracle VM, OEL5 > OEL5 on Oracle VM, RHEL4 > RHEL4 on
Oracle VM, RHEL 5 > RHEL5 on Oracle VM). Oracle recommends using the latest update
levels and OVM versions available.
Every Oracle product that is certified on Windows is also certified and supported when
running on Windows in a virtualized environment with Oracle VM.

JVM Recommendations
Historically, WLS supports the following JVM vendors:

• Oracle
• Sun
• IBM
• HP
When they are supported, Oracle recommends that you use:
• Oracle JRockit JVM for production servers
• Sun HotSpot JVM for development servers and for running
other WLS utilities
The Java Virtual Machine (JVM) is a virtual “execution engine” instance that executes the
bytecodes in compiled Java class files on a microprocessor. To the JVM, a stream of
bytecodes is a sequence of instructions. Each instruction consists of a one-byte opcode and
zero or more operands. The opcode tells the JVM what action to take. If the JVM requires
more information to perform the action than just the opcode, the required information
immediately follows the opcode as operands.
Tuning the JVM to achieve optimal application performance is one of the most critical aspects
of WebLogic Server performance. A poorly tuned JVM can result in slow transactions, long
latencies, system freezes, and even system crashes. Ideally, tuning should occur as part of
the system startup by employing various combinations of the startup options.

JVM Memory
• The JVM itself and WLS performance packs occupy a

section of native memory that is generally static.
• A JVM’s heap:
– Is the area of memory where WLS and application objects
get created and deleted dynamically
– Has a configured maximum size
• A JVM organizes its heap into different areas, often called
generations.
Heap
Native Generation A Generation B
Native memory is the memory that the JVM uses for its own internal operations. The amount
of native memory heap that will be used by the JVM depends on the amount of code
generated, threads created, memory used during garbage collection for keeping Java object
information and temporary space used during code generation, optimization, and so on. If
there is a third-party native module, it could also use the native memory; for example, native
JDBC drivers allocate native memory.
Java heap is the memory used by the JVM to allocate Java objects. The maximum size of
Java heap can be specified using arguments on the Java command line. If the maximum
heap size is not specified, the limit is decided by the JVM considering factors such as the
amount of physical memory in the machine and the amount of free memory available at that
moment. You should specify a value for the maximum Java heap value. The Java heap
contains objects used by the Java programs, including both live and dead objects, as well as
free memory that has been allocated but not used.

Garbage Collection
• Unlike with traditional languages, Java developers do not

explicitly allocate and free memory.
• Garbage collection:
– Is the process of freeing up memory used by objects that are
no longer referenced by the program
– Is automatically performed by the JVM
– Typically defragments or “compacts” free memory as well
– Requires CPU cycles and can therefore affect program
performance
The Java heap is where the objects of a Java program live. It is a repository for live objects,
dead objects, and free memory. When an object can no longer be reached from any pointer in
the running program, it is considered “garbage” and ready for collection. The Java language
does not allow you to free allocated memory directly. Instead, the runtime environment keeps
track of the references to each object on the heap and automatically frees the memory
occupied by objects that are no longer referenced by a process called garbage collection.
The JVM heap size determines how often (and for how long) the VM collects garbage. An
acceptable rate for garbage collection is application-specific and should be adjusted after
analyzing the actual time and frequency of garbage collections. If you set a large heap size,
full garbage collection is slower but occurs less frequently. If you set your heap size in
accordance with your memory needs, full garbage collection is faster but occurs more
frequently.
In addition to freeing unreferenced objects, a garbage collector may also combat heap
fragmentation. Heap fragmentation occurs through the course of normal program execution.
A second advantage of garbage collection is that it helps ensure program integrity. Garbage
collection is an important part of Java’s security strategy. Java programmers are unable to
accidentally (or purposely) crash the JVM by incorrectly freeing memory.

Sun HotSpot Garbage Collection
• Garbage collection algorithms tend to be based on the

premise that most objects do not live long.
• The heap is divided into these generations:
– New space
– Old space
– Permanent
• New space is further divided into an Eden area and two
survivor areas.
New space
Perm Eden Old space
Survivor
Survivor
Memory in the Java HotSpot virtual machine is organized into three generations: a young
generation, an old generation, and a permanent generation. Most objects are initially
allocated in the young generation. The old generation contains objects that have survived
some number of young generation collections, as well as some large objects that may be
allocated directly in the old generation. The permanent generation holds objects that the JVM
finds convenient to have the garbage collector manage, such as objects describing classes
and methods, as well as the classes and methods themselves.
The young generation consists of an area called Eden, plus two smaller survivor spaces. Most
objects are initially allocated in Eden. (As mentioned, a few large objects may be allocated
directly in the old generation.) The survivor spaces hold objects that have survived at least
one young generation collection and have thus been given additional chances to die before
being considered “old enough” to be promoted to the old generation. At any given time, one of
the survivor spaces holds such objects, while the other is empty and remains unused until the
next collection.
When the young generation fills up, a young generation collection (sometimes referred to as a
minor collection) of only that generation is performed. When the old or permanent generation
fills up, what is known as a full collection (sometimes referred to as a major collection) is
typically done.

Garbage Collection (GC) Types
• JVMs automatically perform different types of garbage

collection under different conditions.
• A full GC typically:
– Scans the entire heap for unused objects
– Compacts live objects to minimize fragmented memory
– Requires a significant time to execute
• A partial GC typically:
– Scans a single generation for unused objects
– Promotes live objects to subsequent generations
– Is faster and occurs more frequently
There are a number of garbage collection techniques (for example, reference counts and
mark and sweep). Different versions of Java use different algorithms, but recent versions use
generational garbage collection, which often proves to be quite efficient. Garbage collection is
an important part of Java’s security strategy. Java programmers are unable to accidentally (or
purposely) crash the JVM by incorrectly freeing memory.
When a heap generation becomes full, garbage is collected by running a partial collection,
where all objects that have lived long enough in the younger generations are promoted
(moved) to an older generation, thus freeing up the younger one for more object allocation.
When the old space becomes full, a full collection is required.
The reasoning behind generations is that most objects are temporary and short lived. A partial
collection is designed to be swift at finding newly allocated objects that are still alive and
moving them away. Typically, a partial collection frees a given amount of memory much faster
than a full collection. When monitoring the heap consumption of a JVM, it is often easy to
identify the points at which different collection algorithms are used, based on the amount of
time to perform the collection and the amount of memory freed.

Setting WLS JVM Arguments
• WLS start scripts define default JVM arguments.

• Use the JAVA_HOME or JAVA_VENDOR environment
variable to override the JVM used by a specific server.
• Use the USER_MEM_ARGS and/or JAVA_OPTIONS
variables to override the JVM arguments for a specific
server.
Start a server with custom JVM settings:
export JAVA_VENDOR="Oracle"
export USER_MEM_ARGS="-Xms512m –Xmx1g"
./startWebLogic.sh
When you create a domain, if you choose to customize the configuration, the Configuration
Wizard presents a list of SDKs that WebLogic Server installed. From this list, you choose the
JVM that you want to run your domain, and the wizard configures the Oracle start scripts
based on your choice.
After you create a domain, if you want to use a different JVM, you can modify the scripts and
change either the JAVA_HOME or JAVA_VENDOR environment variables. For JAVA_HOME,
specify an absolute pathname to the top directory of the SDK that you want to use. For
JAVA_VENDOR, specify the vendor of the SDK. Valid values depend on the platform on which
you are running:
• “Oracle” indicates that you are using the JRockit SDK. It is valid only on platforms that
support JRockit.
• “Sun” indicates that you are using the Sun SDK.
• “HP” and “IBM” indicate that you are using SDKs that Hewlett Packard or IBM have
provided. These values are valid only on platforms that support HP or IBM SDKs.

Basic Sun JVM Arguments
Argument Description
-Xms Initial amount of heap (‘m’=megabytes, ‘g’=gigabytes)
-Xmx Maximum amount of heap
-XX:NewSize Initial size of the new generation in the heap
-XX:MaxNewSize Maximum size of the new generation
-XX:MaxPermSize Maximum size of the permanent generation
-XX:SurvivorRatio Ratio of the Eden area to one survivor area.

For example, 8 means that the survivor areas use 20% of
the new generation space.
• -XX:NewSize (default 2 MB): Default size of new generation (in bytes)

• -XX:MaxNewSize: Maximum size of new generation (in bytes). Since 1.4,
MaxNewSize is computed as a function of NewRatio.
• -XX:NewRatio (default = 2): Ratio of new to old generation sizes
• -XX:SurvivorRatio (default = 8): Ratio of Eden size to one survivor space size
• -XX:TargetSurvivorRatio (default = 50%): Desired percentage of survivor space
used after scavenge
• -XX:MaxPermSize: Size of the permanent generation

JRockit Garbage Collection
• JRockit is designed to optimize itself under load with

minimal configuration.
• Smaller objects are organized into nursery and old
generations.
• Larger objects are placed directly in the old space.
Nursery Old space
With JRockit, the heap may be divided into two generations called the nursery (or young
space) and the old space. The nursery is a part of the heap reserved for allocation of new
objects. When the nursery becomes full, garbage is collected by running a special young
collection, where all objects that have lived long enough in the nursery are promoted (moved)
to the old space, thus freeing up the nursery for more object allocation. When the old space
becomes full, garbage is collected there (a process called an old collection). The reasoning
behind a nursery is that most objects are temporary and short lived. A young collection is
designed to be swift at finding newly allocated objects that are still alive and moving them
away from the nursery.
During object allocation, JRockit JVM distinguishes between small and large objects. The limit
for when an object is considered large depends on the JVM version, the heap size, the
garbage collection strategy and the platform used, but is usually somewhere between 2 KB
and 128 KB. Small objects are allocated in thread local areas (TLAs). The thread local areas
are free chunks reserved from the heap and given to a Java thread for exclusive use. The
thread can then allocate objects in its TLA without synchronizing with other threads. When the
TLA becomes full, the thread simply requests a new TLA. The TLAs are reserved from the
nursery if a nursery exists; otherwise, they are reserved anywhere in the heap.

Basic JRockit JVM Arguments
Argument Description
-Xms Initial amount of heap allocated to the JVM
-Xmx Maximum amount of heap that this JVM can allocate

-Xns Size of the nursery generation in the heap
-XgcPrio A priority level that helps determine which GC algorithms the

JVM uses at run time:
• throughput: Maximize application throughput
• pausetime: Minimize how long GC runs
• deterministic: Consistent response times
-XXcompactRatio Percentage of the heap that should be compacted during

each GC cycle
The -Xms option sets the initial and minimum Java heap size. Combine -Xms with a memory
value and add a unit. If you do not add a unit, you will get the exact value you state; for
example, 64 will be interpreted as 64 bytes, not 64 megabytes or 64 kilobytes.
The –Xmx option sets the maximum Java heap size. Depending upon the kind of operating
system that you are running, the maximum value that you can set for the Java heap can vary.
Note, however, that this option does not limit the total amount of memory that the JVM can
use.
The -Xns option sets the nursery size. JRockit JVM uses a nursery when the generational
garbage collection model is used. You can also set a static nursery size when running a
dynamic garbage collector (-XgcPrio).
The -XgcPrio option sets a dynamic garbage collection mode. This garbage collector
combines all types of garbage collection heuristics and optimizes the performance
accordingly. When running this garbage collector, you need to determine only whether your
application responds best to optimal memory throughput during collection or minimized pause
times. The dynamic garbage collector will then adapt its choice of collector type, in run time,
to what best suits your application.
Advanced –XX options are subject to change without notice and are not recommended unless
you have a thorough understanding of the JVM.

Out of Memory
• JVMs trigger java.lang.OutOfMemoryError when there

is insufficient memory to perform some task.
– There is no more memory to allocate.
– Free memory is too fragmented.
• The error indicates the type of memory:
– Native
– Heap
– Specific generation (permanent, for example)
An out-of-memory condition may occur in the Java heap, when the JVM does not have
enough heap space to allocate new Java objects. A similar error can occur for native memory,
when the JVM itself or any additional native shared libraries cannot allocate space for objects
or internal operations.
Native memory errors can also occur due to applications that utilize the Java Native Interface
(JNI). JNI allows you to write native methods to handle situations when an application cannot
be written entirely in the Java programming language, such as when the standard Java class
library does not support the required platform-specific features. Many of the standard Java
library classes depend on JNI to provide functionality to the developer and the user, including
file I/O and sound capabilities. Before resorting to using JNI, developers should make sure
that the functionality is not already provided in the standard libraries. The JNI framework does
not provide any automatic garbage collection for non-JVM memory resources allocated by
code executing on the native side. Consequently, native-side code (such as C, C++, or
assembly language) must assume the responsibility for explicitly releasing any such memory
resources that it itself acquires.
An out-of-memory condition can occur when there is free memory available in the heap but it
is too fragmented and not contiguously located to store the object being allocated or moved
(as part of a garbage collection cycle).

Out-of-Memory Response
• The JVM sends errors to the standard out/error streams.

• WLS writes all Java exceptions and errors to the server log
if they are not handled by the application.
• Out-of-memory and similar system errors should not be
directly handled by applications.
• If the error occurred within an application, an error (HTTP
500 or similar) is sent to the client.
• If the error occurred with a WLS subsystem, the server
may become unstable and require restart.
The JVM throws a java.lang.OutOfMemoryError exception if it is unable to get more

memory in the Java heap to allocate more Java objects. The JVM cannot allocate more Java
objects if the Java heap is full of live objects and it is not able to expand the Java heap any
more.
In this situation or in any Java error, the JVM lets the application decide what to do after
throwing the error. For example, the application may handle this error and decide to shut itself
down in a safe way or decide to ignore this error and retry the failed task. If the application
does not handle this error, the server’s default error handling routine is executed. Typically,
this involves distributing the exception to the client or, in the case of a Web application,
displaying an Internal Server Error message. The error is also logged, unless this has been
disabled by the administrator. The thread that throws this error will also be destroyed (you will
not see this thread if you take a Java thread dump). If this error is being thrown continuously
and you have enabled server health monitoring, the server may opt to shut itself down.

Memory Leak
• Memory leaks:
– Are a common cause of out-of-memory errors
– Are less common in Java than in traditional languages
– Occur when an object is no longer needed but is not made
eligible for garbage collection
• Typical scenarios include:
– Excessive caching
– Excessive use of the HTTP session
– Failing to close DB results when finished
– Problems with dynamic class loading
At first glance, it may appear that memory leaks are not possible with Java, because the JVM
automatically destroys any objects that the application is no longer referencing. However, a
poorly designed or implemented application can still accidentally maintain a reference to an
object even after the object is no longer needed.
In Java EE applications, a common memory leak culprit is the HTTP session, which is a
specific form of caching. Recall that sessions are collections of objects that the application
wants to maintain for the lifetime of a user conversation, such as navigating a Web site. In this
case, objects are initially identified by the application, but are subsequently referenced and
maintained by the server. By default, WebLogic Server will maintain sessions for 1 hour of
inactivity before making them eligible for garbage collection. For servers hosting hundreds
and thousands of concurrent users, large sessions can quickly cause the JVM to run out of
memory. Poorly built applications will cache large amounts of data in HTTP sessions simply
for convenience or for a perceived performance benefit.
WebLogic uses classloaders to dynamically modify the classpath for individual applications at
run time. A typical example is the (re)deployment of an application. This class loading
behavior can also be customized by developers and administrators. If done improperly, this
customization can create subtle memory leaks.

JVM Crash
• Rarely, the JVM itself can fail and cause the server to
crash.
• Depending on the JVM and type of failure, a log and/or a
core file may be produced.
• A core file:
– Is a binary snapshot of the process memory at the time of
failure
– Is triggered by failing native code (JVM, native WLS libraries,
native DB drivers, and so on)
– Can be viewed using OS-specific tools
– Can be disabled or truncated based on your OS settings
On rare occasions you may encounter a JVM crash, which means that the JVM itself has
encountered a problem from which it has not managed to recover gracefully. You can identify
and troubleshoot a JVM crash by the diagnostic files that are generated by the JVM and/or
OS. A snapshot is created that captures the state of the JVM process at the time of the error.
These diagnostic files may include a core file, which typically uses the extension .core on
UNIX and .mdmp on Windows. This binary file contains information about the entire JVM
process and needs to be opened using debugging tools including with your OS. The size of
the binary crash file is usually quite large, so you need to make sure there is enough disk
space for the file to be completely written to disk. Alternatively, you can configure the JVM
and/or OS to limit the file size. For example, on UNIX, use the ulimit command or the
limits.conf file. JVMs also include options to change the file name or location of
generated core files.
The gdb debugging tool, popular on Linux, can extract useful information from core files. The
Dr. Watson tool on Windows provides similar capabilities.

JVM Error Log
The log file typically includes the following types of information:

• The OS error message
• JVM version
• Hardware and OS specifications
• Arguments and environment variables
• Heap and GC summary
• Thread summary
Error Message: Illegal memory access. [54]
Signal info : si_signo=11, si_code=2 si_addr=0x20
Version : JRockit(R) R26.4.0-63-63688-1.4.2_11-20060626-2259-
linux-ia64
GC : gencon : mmHeap->data = 0x2000000000ae0000, mmHeap->top =
0x20000000bc2e0000 ...
The JVM error log is a text file that is like an executive summary of the full memory image and
the environment in which the JVM was run at the time of the crash. This file is produced by
the JVM itself when it crashes and is useful for classifying crashes. It can also sometimes be
used to identify problems that have already been fixed. While this file by itself rarely reveals
enough information to actually find the cause of the problem, it can be useful when creating a
support case.
The exact contents of the log file will vary by JVM, but they generally include similar types of
information. On the Sun JVM, the log file is named hs_err_pid<pid>.log, where <pid> is
the process ID of the process. JRockit refers to this error log as a “dump” file with the name
jrockit.<pid>.dump.
The log shows the actual error message that the operation threw at the time of the crash.
Check your operating system vendor’s user information to find out more about the error. It
also describes how many CPUs have been used and how much memory has been consumed
by the JVM. In addition, the log includes the state of any native threads being utilized by the
JVM.
The log includes a summary of the JVM’s host environment, including the OS version and
hardware specifications. Be sure that your JVM is running on a supported configuration.

Section Summary

• Describe the relationship between WLS and the JVM
• Explain how a JVM manages memory
• Configure basic heap and garbage collection settings for a
JVM
• Describe the conditions and results of a server running out
of memory
• Describe the conditions and results of a JVM crash

Road Map
• JVM Concepts
• Basic JVM Tools
– Stack Trace
– Thread Dump
– Verbose GC
– Sun Profiler Agent
– Sun Diagnostic Tools
– JVisualVM
• WLS JVM Tools

JVM Tool Varieties
Most JVM diagnostic tools can be classified into two main

categories.
• Agents:
– Run within the JVM itself
– Are enabled and configured using JVM command-line
options
– Are triggered periodically or under certain conditions
• Client tools:
– Are external processes that communicate with the JVM and
extract data
– May use a graphical or command-line interface

Java Stack Trace
A stack trace:
• Describes a sequence of method invocations that have led
up to a specific point in a program
• Helps depict the various layers of your application
• Includes source line numbers, unless explicitly removed
during compilation
WLS code
at weblogic.jdbc.jta.DataSource.connect(DataSource.java:383)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.
connectInternal(JDBCStoreManager.java:773)
at kodo.jdbc.kernel.KodoJDBCStoreManager.
connectInternal(JDBCStoreManager.java:29)
at kodo.jdbc.sql.KodoSelectImpl.execute(KodoSelectImpl.java:28)
at com.mycompany.ejb.Customer.findByStatus(Customer.java:21)
Your application Java EE or

code framework code
If an exception or error is not explicitly handled by WLS or an application, the server will log
the error message along with a stack trace. For a typical Java EE application running on
WLS, a full stack trace for a single client request could comprise100 or more method
invocations. Developers can also programmatically print the current stack trace if desired. The
text <init> is used within a stack trace entry to indicate a constructor method of a class.
Stack traces may include calls to native functions, such as within the JVM or other native
libraries. These native calls will be indicated as such in the stack trace and will not include
detailed line numbers.
For each method call within a stack trace, you will see the containing class and package
name. Even if you are not intimately familiar with WLS or your application’s implementation
details, these package names can still help you identify which general layers of the system
are involved. For example, generally speaking, packages starting with weblogic are part of
WebLogic’s codebase, while those starting with org tend to be third-party frameworks or
libraries. In the example in the slide, we can see that this application uses two such libraries:
OpenJPA and Kodo (both of which are bundled with WLS).

Java Thread Dump: Overview
A thread dump provides a snapshot of all threads currently

running in the program:
• Current state (running, blocking/waiting, stuck, and so on)
• Current stack trace Waiting for
[ACTIVE] ExecuteThread: '12' ... a request
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:474) Running a
at weblogic.work.ExecuteThread. JSP in an
waitForRequest(ExecuteThread.java:156) ... application
[ACTIVE] ExecuteThread: '20' ...

at jsp_servlet.__MyPage._jspService(__MyPage.java:108)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
at weblogic.work.ExecuteThread.
execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
A thread dump is a snapshot of all active threads in a Java Virtual Machine in a readable
textual format. A thread dump provides information about what each of the threads is doing at
a particular moment in time.
A set of thread dumps can help analyze the change, or lack of change, in each thread’s state
from one thread dump to another. To detect deadlocks, get thread dumps several times (three
or more) on each server, spaced somewhere between 5 and 30 seconds apart.
Some threads in WebLogic are not used to process client requests but instead perform
internal operations. For example, socket reader (or muxer) threads simply poll sockets for
incoming requests and place them on a queue to be processed by other threads. The number
of these socket reader threads that are created by WebLogic can vary based on the host OS
and the type of native performance pack installed.
There are several open-source graphical tools that can help you better visualize and analyze
the data contained in a thread dump. Some may support only a specific JVM, but others
support multiple JVMs. Examples include Samurai and the TDA project. JVMs themselves
typically provide similar thread dump analysis tools as well.

Thread Dump Signal
Trigger a thread dump on a WebLogic Server instance:

• Through the Administration Console or WLST
• By sending the process a QUIT signal on UNIX
• By sending the process a BREAK signal on Windows
(Ctrl + Break)
• By using other JVM tools
Capture a thread dump on UNIX by using an OS signal:
kill -3 <WLSProcessID>
Generally speaking, thread dumps are written to the standard output stream, but there are
some exceptions. For example, you can view the state of all threads by using the WLS
Administration Console. Select a server and click the Monitoring > Threads tab. The Current
Request column indicates what the thread is currently working on. To instead trigger a thread
dump (on standard output) by using the console, click the Monitoring > Performance tab and
then click Garbage Collect.
The WLST threadDump() command displays a thread dump for the given server and, by
default, also writes this data to a file. By default, the file name is
Thread_Dump_<serverName>. If you are connected to an Administration Server, you can
display a thread dump for the Administration Server as well as for any managed server that is
running in the domain. If you are connected to a managed server, you can only display a
thread dump for that managed server.
Note that the JVM option –Xrs (or –Xnohup) can disable thread dumps on a server. This
option helps to prevent possible interference when the JVM is running as a background
process or service and receives OS signals such as CTRL_LOGOFF_EVENT or SIGHUP. If not
set, upon receiving such events, the VM tries to initiate a shutdown, but this shutdown will fail
because the operating system will not actually terminate the process.

JVM Crash Actions
Most JVMs support additional options in the event of a crash:

• Pause to permit a final thread dump or the use of OS
debugging tools.
• Run a custom script.
Pause after a fatal error on the Sun JVM:
java ... -XX:+ShowMessageBoxOnError weblogic.Server
Pause after a fatal error on the JRockit JVM:
java ... XXOnCrash:wait=true weblogic.Server
When the -XX:+ShowMessageBoxOnError option is set and a fatal error is encountered,

the HotSpot VM will display information about the fatal error and prompt the user to specify
whether the native debugger is to be launched. This technique also gives you an opportunity
to take a thread dump at the precise time of the crash. In the case of Solaris and Linux, the
output and prompt are sent to the application console (standard input and standard output). In
the case of Windows, a graphical message box displays.
When a fatal error occurs, the HotSpot VM can optionally execute a user-supplied script or
command. The script or command is specified by using the -XX:OnError=<string>
command-line option, where <string> is a single command or a list of commands separated
by a semicolon. Within <string>, all occurrences of %p are replaced with the current
process ID, and all occurrences of %% are replaced with a single %.
On Solaris OS, the pmap command displays information about the address space of a
process. In the following example, if a fatal error occurs, the pmap command is executed to
display the address space of the process:
java -XX:OnError="pmap %p" MyApplication

Verbose GC
Most JVMs can provide a trace of all garbage collection

activities for troubleshooting/tuning purposes.
• On Sun: -verbose:gc or -XX:-PrintGCDetails
• On Jrockit: -verbose:gc or -Xverbose:gcpause
java –verbose:gc ... weblogic.Server

...
[memory ] 7.160: GC 131072K->130052K (131072K)in 12.93 ms
[memory ] 30.170:FULL GC 131072K->103704K (131072K)
in 200.112 ms
java –Xverbose:gcpause ... weblogic.Server

...
[memory ] old collection phase 0-2 pause time: 100.794255 ms
[memory ] nursery collection pause time: 4.121775 ms
The -verbose and -Xverbose JRockit JVM command-line options output diagnostic
information about the system. The output is by default printed to the standard output for error
messages (stderr) but you can redirect it to a file by using the -XverboseLog command-
line option. The information displayed depends on the parameter specified with the option; for
example, specifying the parameter cpuinfo displays information about your CPU and
indicates whether or not the JVM can determine if hyper-threading is enabled. Available
debug parameters include cpuinfo, gc, gcpause, gcreport, class, load, and codegen.
The Sun HotSpot JVM supports similar command-line options. The -verbose option
supports the parameters gc, class, and jni. Additional command-line options related to
memory management include –XX:PrintGC, -XX:-PrintGCDetails, and –XX:-
PrintGCTimeStamps.
A typical garbage collection debug message includes the memory used prior to the GC, the
new amount of memory used after the GC, the current heap size, and the elapsed time. The
output format can vary, so refer to your JVM documentation. If you encounter out-of-memory
errors even when verbose GC messages indicate that there is free Java heap available, it is
likely due to memory fragmentation.
Tagtraum GCViewer is an open source that can help you visualize the data produced by the
verbose GC options. It supports both the Sun and JRockit JVMs.

Sun JVM Profiler Agent
• Most JVMs include some type of profiling feature to

determine which code requires the most memory or CPU.
• The Sun JVM Heap Profile (HPROF) agent:
– Periodically collects JVM heap or CPU statistics
– Generates a profile in text or binary format
– Is enabled using the -agentlib:hprof option
– Can be triggered similarly to how a thread dump is triggered
Heap profile in text (ASCII) format:
ID of stack trace that
TRACE 300008 ... Stack traces created these objects
SITES BEGIN (ordered by live bytes)...
percent live ... stack class
rank self accum bytes objs ... trace name
1 30% 30% 1578736 13576... 300008 char[]
2 3% 33% 918466 476 ... 300208 java.lang.String
HPROF is actually a JVM native agent library that is dynamically loaded through a command-
line option at JVM startup, and becomes part of the JVM process. By supplying HPROF
options at startup, users can request various types of heap and/or CPU profiling features from
HPROF. The data generated can be in textual or binary format, and can be used to track
down and isolate performance problems involving memory usage and inefficient code. The
following profile types are supported:
• -agentlib:hprof=heap=[all|dump|sites]
• -agentlib:hprof=cpu=[samples|times|old]
Additional options are available to control the output format (ASCII or binary), sampling
period, and stack trace depth. For example, to sample CPU information every 20 ms, with a
stack depth of 3, use:
-agentlib:hprof=cpu=samples,interval=20,depth=3
By default, heap profiling information (sites and dump) is written out to java.hprof.txt.
The output in most cases contains IDs for stack traces, threads, and objects. Each type of ID
typically starts with a different number than the other IDs. For example, traces might start with
300000.

Sun JVM Diagnostic Tools: Overview
Tool Description
jvisualvm An all-inclusive graphical interface for monitoring and profiling CPU,

JVM memory, threads, and MBeans
jconsole A more lightweight version of JVisualVM with fewer capabilities
jps Identify and print the status of local or remote JVM processes.
jinfo View or update configuration settings for a local or remote JVM.
jstat Periodically sample a JVM for memory and GC statistics.
jmap • Print current memory and GC statistics (heap).
• Print memory statistics for each class in use (histogram).
• Generate a binary or text heap dump file.
jhat Navigate the contents of a binary heap dump file as a Web
application.
jstack Print thread dump for a local or remote JVM.
You can use these tools bundled with the Sun JDK to monitor JVM performance statistics and
to troubleshoot problems. According to the Oracle Sun web site, the tools described in this
section are unsupported and experimental, and should be used with that in mind. Therefore,
although unlikely, they may not be available in future JDK versions.
JVisualVM is a graphical tool that provides detailed information about one or more local or
remote JVMs. It includes memory and CPU profiling, heap dump analysis, memory leak
detection, access to MBeans, and garbage collection analysis. JConsole is a similar tool but
does not include memory and CPU profiling features.
The jps tool lists the instrumented HotSpot Java Virtual Machines (JVMs) on the target
system. The tool is limited to reporting information on JVMs for which it has the access
permissions. If jps is run without specifying a hostid, it will look for instrumented JVMs on
the local host. If started with a hostid, it will look for JVMs on the indicated host, using the
specified protocol and port. A jstatd process is assumed to be running on the target host.
The jps command will report the local VM identifier, or vmid, for each instrumented JVM
found on the target system. The vmid is typically, but not necessarily, the operating system’s
process identifier for the JVM process. With no options, jps will list each Java application’s
vmid followed by the short form of the application’s class name or JAR file name. The short
form of the class name or JAR file name omits the class’s package information or the JAR
files path information.

Sun Diagnostic Tools: Examples
<xyz> denotes
internal JVM types.
> jmap –histo 13101 | more
num #instances #bytes class name

----------------------------------------------
1: 211110 23296528 <constMethodKlass>
2: 211110 16894408 <methodKlass>
3: 180804 4339296 java.lang.String
4: 48846 1953840 java.util.HashMap
5: 26041 416656 weblogic.security.service.
... internal.WLSIdentityImpl
Print the Java classes

that currently occupy
the most memory.
For most Sun diagnostic tools, you must specify the ID of the JVM to communicate with. In
most instances, this is the host process ID.
The jmap tool supports the following data options:
• -histo[:live]: Prints a histogram of the heap. For each Java class, the number of
objects, the memory size in bytes, and the fully qualified class names are printed. VM
internal class names are printed with the * prefix. If the live option is specified, only live
objects are counted.
• -heap: Prints a heap summary, GC algorithm used, heap configuration, and heap
usage for each generation
• -dump:[live,]format=b,file=<filename> : Dumps the Java heap in hprof
binary format to the file name. The live option is optional. If specified, only the live
objects in the heap are dumped. To browse the heap dump, you can use jhat.
• -finalizerinfo: Prints information on objects awaiting finalization
• -permstat: Prints class loader statistics for permanent generation of the Java heap.
For each class loader, its name, status, address, parent class loader, and the number
and size of classes it has loaded are printed. In addition, the number and size of
interned strings are printed.

Sun Diagnostic Tools: Examples
Print all GC stats

every 30 seconds.
> jstat –gc 13101 30s
S0C S1C S0U S1U EC EU OC ...

9216.0 192.0 0.0 160.0 37696.0 37112.4 233024.0 ...
8320.0 9344.0 240.0 0.0 36544.0 9617.2 233024.0 ...
9088.0 1536.0 0.0 1531.1 35456.0 19082.5 233024.0 ...
9088.0 1536.0 0.0 1531.1 35456.0 19318.0 233024.0 ...
...
S = survivor space E = Eden O = old space
When running the jstat tool you must specify a data option. For example, -gc prints all
statistics for the entire garbage-collected heap. The –gcold and –gcnew options print
statistics for only the old or new heap generations, respectively. The –gcutil option prints
an abbreviated set of heap statistics.
The jstat tool supports arguments to control the data collection interval, the columns that
are displayed, and the interval at which column headers are displayed. For GC statistics, the
columns include:
• S0C Current survivor space 0 capacity (KB)
• S1C Current survivor space 1 capacity (KB)
• S0U Survivor space 0 utilization (KB)
• S1U Survivor space 1 utilization (KB)
• EC Current Eden space capacity (KB)
• EU Eden space utilization (KB)
• OC Current old space capacity (KB)
• PC Current permanent space capacity (KB)

JVisualVM
JVisualVM can:
• Monitor and profile local JVMs
• Monitor JVMs on remote hosts that are also running the
jstatd process
JVisualVM is a tool that provides a visual interface for viewing detailed information about Java
applications while they are running on a JVM, and for troubleshooting and profiling these
applications. JVisualVM combines all of the functionality available in the JDK’s command-line
diagnostic tools, such as jstat, jinfo, jmap, and jstack.
JVisualVM is able to access and display data about applications running on remote hosts.
After connecting to a remote host, JVisualVM can display general data about the application’s
runtime environment and can monitor memory heap and thread activity. JVisualVM can
retrieve monitoring information on remote applications, but it cannot profile remote
applications.
The jstatd tool is an RMI server application that monitors the creation and termination of
instrumented HotSpot Java virtual machines (JVMs) and provides an interface to allow remote
monitoring tools to attach to JVMs running on the local host. The jstatd server requires the
presence of an RMI registry on the local host. The jstatd server will attempt to attach to the
RMI registry on the default port, or on the port indicated by the -p port option. If an RMI
registry is not found, one will be created within the jstatd application bound to the port
indicated by the -p port option or to the default RMI registry port if -p port is omitted. The
jstatd server can monitor only those JVMs for which it has the appropriate native access
permissions. Therefore, the jstatd process must be running with the same user credentials
as the target JVMs.

Using JVisualVM
JVisualVM allows you to:

• Inspect JVM configuration settings
• Monitor heap and thread usage in real time
• Capture and analyze a heap or thread dump
• Record and analyze a CPU or memory usage profile
• Import and view a JVM core dump (UNIX only)
When you start JVisualVM, the Applications panel is visible at the left side of the main
window. The Applications panel uses a tree structure to enable you to quickly view the
applications that are running on local and connected remote JVM software instances. It also
displays core dumps and application snapshots. For most nodes in the Applications panel,
you can view additional information and perform actions by right-clicking a node and selecting
an item from the popup menu.
The Local node displays the names and process IDs (PIDs) of Java applications that are
running on the same system as JVisualVM. When you launch JVisualVM, it automatically
displays the currently running Java applications, including JVisualVM itself. When a new local
Java application is launched, a node for that application appears under the Local node. The
application node disappears when the application terminates. If you use JVisualVM to take
thread dumps, heap dumps, or profiling snapshots of an application, they are displayed as
subnodes below the node for that application. You can right-click a dump or snapshot
subnode to save it to your local system. You can also collect and archive all the information
collected about an application and save it as an application snapshot.
The VM Coredumps node lists open core dump files. A core dump is a binary file that contains
information about the runtime status of the machine at the time the core dump was taken. You
can then extract an overview of the system properties, a heap dump, and a thread dump.

Using JVisualVM
Profile memory
Monitor usage
threads
JVisualVM displays real-time, high-level data on thread activity on the Threads tab for a
specific JVM instance. By default, the Threads tab displays a timeline of current thread
activity. You can also click a thread in the timeline to view details about that thread on the
Details tab. Use the buttons in the Timeline toolbar to zoom in and out on the current view and
to switch to the “Scale to Fit” mode. The drop-down list enables you to select which threads
are displayed. You can choose to view all threads, live threads, or finished threads.
The Profiler tab enables you to start and stop the profiling session of a local JVM. When
profiling results are available, they are automatically displayed on the Profiler tab. You can
use the toolbar to refresh the profiling results, invoke garbage collection, and save the
profiling data. Choose CPU Profiling to profile the performance of the application. Choose
Memory Profiling to analyze the memory usage of the JVM. The results display the objects
allocated by the JVM along with the classes that allocated those objects. The filter box below
the profiling results enables you to filter the displayed results according to the name of the
method. To filter the results, enter a term in the method name filter box, select a filtering
method to use, and press Return. You can see and select previous filter terms by clicking the
arrow to the right of the method name filter box.

Section Summary

• Describe what a typical Java thread dump looks like
• Use OS signals or JDK tools to trigger a stack dump
• Enable verbose garbage collection or heap profiling for a
JVM
• List some of the diagnostic tools included with the Sun
JDK
• Discuss the capabilities of Sun’s JVisualVM

Practice 6-1
Troubleshooting a Running JVM
• Starting a server by using the Sun JVM
• Performing and analyzing a server thread dump
• Configuring JVM heap settings
• Generating a heap profile by using the Sun JVM
• Analyzing a heap profile

Road Map
• JVM Concepts
• Basic JVM Tools
• WLS JVM Tools
– Console JVM Monitoring
– Low Memory Detection

Console JVM Monitoring
The WLS console lets you monitor basic JVM metrics.
1
2
3
Force a GC.
1. Select a server in your domain.

2. Click the Monitoring tab.
3. Click the Performance tab to view various settings and statistics for the underlying
server JVM. You can also use this page to force garbage collection or a thread dump.
Available statistics include Heap Free Current, Heap Size Current, Heap Free Percent, Total
Physical Memory, Free Physical Memory, GC Algorithm, Total GC Count, and Last GC Start.
The GC Pause Time Target button sets a pause time target value for pause time or
deterministic dynamic garbage collection. The target value is used as a pause time goal. This
helps to more precisely configure the dynamic garbage collector to keep pauses near the
target value. Using this option you can specify pause target values between 1 ms and 5
seconds.
Also see the –XpauseTarget command-line argument.

JVM WLST: Example
domainRuntime()
serverJVM =
getMBean('/ServerRuntimes/serverA/JVMRuntime/serverA')
print 'Heap Size: ' + serverJVM.getHeapSizeCurrent()

print 'Heap Free: ' + serverJVM.getHeapFreeCurrent()
print 'Heap Free(%): ' + serverJVM.getHeapFreePercent()
Refer to JVMRuntimeMBean in the online documentation for more details.

As always, remember that you can either connect to a specific managed server and monitor
its MBeans, or connect to the Administration Server and use it as a proxy to monitor the
managed servers. The latter approach is shown in the example in the slide.

WLS Low Memory Detection
• When WLS detects a low-memory condition, it:

– Changes its health state attribute to Warning
– Generates a log message
• (Optional) Create WLDF or SNMP notifications for this
attribute.
1
2
You can automatically log low-memory conditions observed by the server. WebLogic Server
detects low memory by sampling the available free memory a specified number of times
during a time interval. At the end of each interval, an average of the free memory is recorded
and compared to the average obtained at the next interval. If the average drops by a user-
configured amount after any sample interval, the server logs a low-memory warning message
in the log file and sets the server health state to Warning.
1. Select a server in your domain.
2. Confirm that the Configuration tab is selected.
3. Click the Tuning tab.

Configuring Low Memory Detection
• Low Memory GC Threshold: The threshold level (as a percentage) that this server
uses for logging low-memory conditions and changing the server health state to
Warning. For example, if you specify 5, the server logs a low-memory warning in the log
file and changes the server health state to Warning after the average free memory
reaches 5% of the initial free memory measured at the server’s boot time.
• Low Memory Granularity Level: The granularity level (as a percentage) that this server
uses for logging low-memory conditions and changing the server health state to
Warning. For example, if you specify 5 and the average free memory drops by 5% or
more over two measured intervals, the server logs a low-memory warning in the log file
and changes the server health state to Warning.
• Low Memory Time Interval: The amount of time (in seconds) that defines the interval
over which this server determines average free-memory values. By default, the server
obtains an average free-memory value every 3600 seconds. This interval is not used if
the JRockit VM is used, because the memory samples are collected immediately after a
VM-scheduled garbage collection. Taking memory samples after garbage collection
gives a more accurate average value of the free memory.

Section Summary

• View basic JVM information and statistics using the WLS
console or WLST
• Configure WLS low-memory detection

Road Map
• JVM Concepts
• Basic JVM Tools
• WLS JVM Tools
– JRockit Diagnostic Tools
– Management Console
– Runtime Analyzer
– Memory Leak Detector

JRockit Diagnostic Tools: Overview
• Use any of the following to monitor/troubleshoot JRockit:

– jrmc and jrcmd tools
– -Xverbose:type argument
– ctrlhandler.act file
• JRockit also includes the same jconsole, jps, and
jstat tools as the Sun JDK.
Tool Description
jrmc (JRockit Mission Control) An all-inclusive graphical interface for
monitoring and profiling CPU, JVM memory, threads, and MBeans
jrcmd Identify JRockit processes and send commands to them such as
print_threads, heap_diagnostics or print_class_summary.
The jrcmd file is a command-line tool included with the JRockit JDK that you can use to send
diagnostic commands to one or more running JVM processes. It discovers which JRockit JVM
processes are running on the machine. The syntax is as follows:
jrcmd <pid> [<command> [<arguments>]]
<pid> is the process ID of a JRockit JVM. If set to 0, commands are sent to all JRockit JVM
processes.
[<command> [<arguments>]] is any diagnostic command and its associated arguments.
If instead you want to use these same commands in the context of the JVM thread dump
signal (kill -3 on UNIX, for example), you can edit the ctrlhandler.act file. These
commands include:
• version, which prints the JRockit JVM version
• command_line, which prints the command line that is used to start the JRockit JVM
• print_threads, which prints a normal thread dump
• print_class_summary, which prints all loaded classes
• print_properties, which prints all Java system properties
• heap_diagnostics, which prints memory layout and object allocation statistics

JRockit Diagnostic Tools: Examples
> jrcmd 13101 heap_diagnostics | more
Total memory in system: 3145908224 bytes

Available physical memory in system: 1640439808 bytes
-Xmx (maximal heap size) is 536870912 bytes
Heapsize: 536870912 bytes
Free heap-memory: 245073280 bytes
Memory layout: ...
-------- Detailed Heap Statistics: --------

7.5% 2960k 126334 -1k java/lang/String
5.5% 2161k 19761 +0k java/lang/Class
0.7% 285k 12202 +0k javax/xml/namespace/QName
...
ctrlhandler.act
set_filename filename=/tmp/jvm_output.txt
print_threads
heap_diagnostics
Use diagnostic commands to communicate with a running Oracle JRockit JVM process.
These commands tell the JRockit JVM to, for example, print a heap report or a garbage
collection activity report, or to turn on or off a specific verbose module. You can enable or
disable any diagnostic command by using the -Djrockit.ctrlbreak.enable<name>=
<true|false> system property, where <name> is the name of the diagnostic command.
In addition to the jrcmd tool, you can run diagnostic commands by pressing Ctrl + Break on
Windows or kill -3 on UNIX. When you issue this signal, the JRockit JVM will search for a file
named ctrlhandler.act in your current working directory. If it does not find the file there, it
will look in the directory containing the JVM. If it does not find this file there either, it will revert
to displaying the normal thread dump. If it finds the file, it will read the file searching for
command entries, each of which invokes the corresponding diagnostic command. You can
disable this functionality by setting this JVM option:
-Djrockit.dontusectrlbreakfile=true
The set_filename command sets the file that all commands following this command will
use for printing. You can have several set_filename commands in a single file. It takes two
arguments: the file name and an optional flag to specify if you want to append to the file or
overwrite it. The defaults are stderr for the file name, and to overwrite the file.

Management Communication
• JRockit JVMs must be configured to accept management

requests from diagnostic tools.
• Perform one of the following:
– Add the -Xmanagement option when starting the JVM.
– Dynamically enable or disable the port by using jrcmd.
• Processes communicate by using JMX (default port is
7091).
> jrcmd 13101 start_management_server
> jrcmd 13101 kill_management_server
The -Xmanagement option starts the JRockit JVM concurrently with the management server
and allows you to either enable and configure or explicitly disable security features for this
connection, such as SSL encryption and authentication.
The following example overrides the default port using 1234, and disables SSL:
-Xmanagement:port=1234,ssl=false,authenticate=false
Due to the security risks and the mission-critical nature of most JRockit JVM deployments, the
new default behavior of the JRockit JVM requires that you either disable security explicitly or
configure and enable security. If you do not take these steps, the management server will not
open a port for remote access and may cause the JVM startup to halt with an error message
concerning the security configuration.
You can also dynamically enable or disable the management port by using the jrcmd
command-line tool or the Ctrl + Break handler. In either case, use the
start_management_server and kill_management_server commands.

JRockit Mission Control (JRMC)
• JRMC:
– Is a graphical monitoring and diagnostic client bundled with
JRockit
– Supports real-time monitoring, historical analysis, and
customized alerts
– Is designed to incur minimal overhead on server JVMs when
compared to similar tools
– Runs on JRockit
– Is also available as a plug-in to Eclipse
• Check Oracle.com for the most recent release.
• As a best practice, run JRMC on a separate machine (not
on the machine running the server JVM).
The Oracle JRockit Mission Control client tools suite includes tools to monitor, manage,
profile, and eliminate memory leaks in your Java application without introducing the
performance overhead normally associated with these types of tools.
To view real-time behavior of your application and of Oracle JRockit JVM, you can connect to
an instance of the JRockit JVM and view real-time information through the JRockit
Management Console. Typical data that you can view includes thread usage, CPU usage,
and memory usage. All graphs are configurable, and you can both add your own attributes
and redefine their respective labels. In the Management Console, you can also create rules
that trigger on certain events. For example, an email is sent if the CPU reaches 90% of the
size.

JRockit Discovery Protocol (JDP)
• If JDP is enabled on remote JVMs, JRMC automatically

discovers them on your network by using multicast.
• Due to network limitations, multicast is typically limited to a
single LAN.
java ... –Xmanagement:autodiscovery=true ...
Discover JVM
1 JDP Server
JRMC Client
Management
2 Agent
You can use the –Xmanagement option to make the JRockit JVM use the JRockit Discovery
Protocol (JDP) to announce its presence to management clients such as JRockit Mission
Control.
Some additional Java system properties (-D<property>) are also available to tune this
discovery behavior:
• jrockit.managementserver.discovery.period: The time to wait between
multicasting the presence in ms (default is 5000)
• jrockit.managementserver.discovery.ttl: The number of router hops the
packets being multicasted should survive (default is 1)
• jrockit.managementserver.discovery.address: The multicast group/address
to use (default is 232.192.1.212)
• jrockit.managementserver.discovery.targetport: The target port to
broadcast (default is 7091)

JVM Browser
• Use the JVM Browser view to launch tools for a specific

JVM process.
Discovered
remote JVMs
This process
Local
JVMs
Process ID
When you launch Oracle JRockit Mission Control, the JVM Browser is automatically opened.
The JVM Browser is a configurable tree view that you can use to manage your Oracle JRockit
JVM connections. It will automatically discover locally running JRockit JVMs as well as
JRockit JVMs on the network that have been started using JDP. Other plug-ins (for example,
the JRA tool) use the JVM Browser to access the connection(s) on which to operate.
Although JRMC should discover your JVMs automatically, you can also explicitly create a
connection to a JVM. Right-click within the JVM Browser under the Connectors folder and
select New Connection. Enter a host, port, and connection name.
To start other JRMC tools such as the Management Console, select a running JVM and either
use the toolbar buttons at the top of the view or right-click the JVM and select an application.
To update user preferences for the JVM Browser or other plug-ins, select Window >
Preferences from the main menu. In this case, you select JRockit Mission Control > Browser
Preferences.

Management Console: Features
JRMC tools provide a main menu, with each menu option

displaying a collection of tabs.
Menu Tab Descriptions
General View a summary of current CPU and memory usage.

MBeans • View all available JVM attributes.
• Define notifications based on JVM attribute values.
Runtime • View CPU usage, general JVM statistics, and Java system
properties.
• View more detailed memory and GC statistics.
• View thread statistics and stack traces.
Advanced • Profile JVM over time and collect statistics for selected methods and
exceptions.
• Run jrcmd commands (print_threads, ...).
JRockit instrumentation is accessible through JMX managed bean (MBean) interfaces, which
are registered in the JVM’s management server and displayed using diagnostic tools such as
the Management Console in JRockit Mission Control. The extra cost of running the
Management Console against a running JVM is small and can almost be disregarded. This
provides for low-cost monitoring and profiling of your application. Each tab in the
Management Console focuses on a specific set of JVM MBeans, although the MBean
Browser tab lets you browse and monitor the raw data for all available MBeans.
The Runtime > Memory tab allows you to monitor how efficiently your application is using the
memory that is available to it. This tab focuses on heap usage, memory usage, and garbage
collection schemes. The information provided on this tab can greatly assist you in determining
whether you have configured the JRockit JVM to provide optimal application performance.
The Runtime > System tab provides such information as the average processor load over
time and as a percentage of the overall CPU load. It also lists all system properties loaded
with the JVM.
The Advanced > Exception Count tab supports a type of profiling that counts the number of
exceptions of a certain type, providing information that is helpful when you are troubleshooting
your Java application.

Management Console: General > Overview
Memory and
CPU gauges
CPU usage
over time
Memory usage
over time
The Overview tab found under the General menu option allows you to monitor processor
behavior and memory statistics during system run time. The Overview tab is helpful in
analyzing a system’s general health because it can reveal behavior that might indicate
bottlenecks or other sources of poor system performance.
Graphs show the performance of one or more MBean attributes over time. You can add or
remove attributes to any graph in the Management Console. The values are plotted on the y
axis (vertical axis) and are usually specified in percentages or raw numbers. The time element
is plotted on the x axis (horizontal axis). Time is displayed in increments of seconds, minutes,
or hours. Each graph has a legend that uses a color patch to identify the attribute being
plotted.

Management Console: Runtime > Threads
View the status

of all threads.
View a thread’s
stack trace.
The Threads tab found under the Runtime menu option allows you to monitor thread activity
for a running application. This tab contains both a graph that plots thread usage by an
application over time and a sortable list of all live threads used by the application. It also
displayed thread stack traces.
Some Management Console tabs display data by using tables. You can add and remove
elements in a Management Console table, and you can also resize the widths of the elements
and the way they are sorted when viewing.

Management Console: MBeans > Triggers
Select JVM or
WLS MBean.
Set conditions.
Select an action.
View triggered
alert actions.
The Triggers tab found under the MBeans menu option lists all trigger rules that you have
created for a JVM and allows you to activate or deactivate any trigger using the supplied
check boxes. These rules, when violated, launch a user-defined notification that advises
users of the violating condition. Create new triggers by using the Add button. Similarly, you
can Edit or Delete existing trigger definitions.
When adding a new trigger, you must specify the MBean attribute on which it should be
based. Browse the available MBeans and select an attribute. Then set the following:
• Max Trigger Value: The maximum value at which point the rule should be triggered
• Sustained: The duration, in seconds, that the triggering condition must be met for the
action to trigger
• Limit Period: The minimum time, in seconds, between consecutive triggers
• Trigger When: Whether you want the action to be triggered when the attribute reaches
the set value and/or when it drops below the set value
The Application Alert action type simply makes the notification available in the JRMC tool
itself. The Console Output type prints the notification to standard output. The Send E-mail
action sends an email to the specified email server and address. The Thread Stack Dump
action captures a JVM stack dump and delivers it as an JRMC alert or writes it to a log file.

JRockit Flight Recorder (JFR)
• Generates a rotating buffer of diagnostic data called a

flight recording (or JFR file) that can be viewed in JRMC
• Runs within the JRockit Virtual Machine with minimal
overhead (less than 1%)
• Can be started within the JRMC client
• Can be started with JRockit VM command-line options
-XX:FlightRecorderOptions=defaultRecording=true
• Can be started with JRMC command-line options
> jrcmd 3616 start_flightrecording
defaultrecording=true
• Can be started automatically (exit/exceptions/triggers)
• Replaces the JRockit Runtime Analyzer (JRA)
The JRockit Flight Recorder (JFR) is a performance monitoring and profiling tool that makes
diagnostic information available, even following catastrophic failure, such as a system failure.
The Flight Recorder is a rotating buffer of diagnostic and profiling data that is available on-
demand. Like its aviation namesake, the Flight Recorder provides an account of events
leading up to a system (or application) failure. Whenever a Java application running on Oracle
JRockit is started with the Flight Recorder enabled, data is constantly collected in the Flight
Recorder or its buffers. In the event of a system failure, the content of this recording can be
transferred to a file and used to analyze what went wrong.
Data displayed by the Flight Recorder GUI is collected by the Flight Recorder Run Time
component, which is part of the Oracle JRockit JVM. Flight recordings are displayed in the
Flight Recorder GUI, which is part of the JRockit Mission Control Client.
To automatically create a flight recording on a failure (unhandled exception), use the following
option when you start the JRockit JVM:
-XX:FlightRecordingDumpOnUnhandledException

Integration of JRockit Flight Recorder and WLDF
WLDF provides integration points with JRockit Flight Recorder.

• WLDF can be configured to generate event data that are
then captured by the JFR file. Events can come from Web
applications, EJBs, JDBC, JTA, and JMS.
– The generation of event data is controlled by the WLDF
diagnostic volume configuration, found in the admin console:
— Environment > Server > a server > Configuration > General
— Set the Diagnostic Volume to Off, Low, Medium, or High.
• WLDF diagnostic image captures now include the JFR file.
• As processing load increases, WLDF automatically
throttles the number of WLS requests selected for event
generation and recording to the JFR file to minimize
overhead.
The volume of Flight Recorder data that is captured can be configured from the WebLogic
Server Administration Console, which allows you to specify the following settings:
• Off: No data is captured in the Flight Recorder diagnostic image. This is currently the
default.
• Low: Basic information is captured when messages with the “emergency,” “alert,” or
“critical” levels are recorded.
• Medium: Additional information is captured when messages with the “error” level and
above are recorded.
• High: In-depth information is captured when messages with the “error” level and above
are recorded.

Starting the Flight Recorder from JRMC
1. In the JVM Browser,

right-click a server and
select Start Flight
Recording.
2. Pick a template, enter
a filename and a name,
and select either “Time
fixed recording” or
“Continuous recording.”
3. Click OK.
There are standard templates that come with the Flight Recorder:
• Default: This template includes most available information. It omits some low-level JVM
data, some costly memory system data, and exception stack traces.
• Real Time: This template consumes the least overhead possible. Therefore, it collects
limited data. It includes the most important garbage collection and memory system
information; however, some tabs do not contain any information.
• Full: This template records every JVM-level event and thus provides the most data.
Conversely, it also requires the most overhead. Note that other events use default
settings from the producer.

Flight Recorder Output
To view the flight recording, select File > Open File and then
browse to the recording file.
Tab Description
General General information, such as CPU and heap usage
Memory Memory information (such as memory consumption over time and
garbage collection statistics) along with memory allocation and object
statistics
Code Code information, such as "hot" packages and methods, exceptions by
class, and code-generation statistics
CPU / CPU and thread information, such as CPU usage and thread count,
Threads lock contention and lock profiling, and hot threads and thread latency
Events Event information, such as event producers and types, event logging
and graphing, event by thread, event stack traces, and event
histograms

General > Overview
Memory in the heap

How fragmented
used by “live” How long does
is free heap?
objects. GC take?
Other items under General > Overview are:

• CPU Usage chart
• Heap Usage chart
• JVM Start Time
• JVM Version
Under the General > System tab are:
• JVM Start Time
• JVM Name
• JVM Version
• JVM Command Line Arguments
• Java Application Arguments
• Operating System Version

Memory: Object Statistics
Java classes requiring

the most heap
Other tabs under Memory:

• Overview: A graph of Memory Usage, along with GC Configuration, GC Statistics, and
Allocation
• GC General: Heap and Nursery Configuration numbers and other generational
information
• GC Graph: A graph showing Heap Size, Heap Usage, and Live Set, as well as statistics
on the longest GCs
• GC Advanced: Data on particular GCs
• Allocation: Bar charts showing Allocation by Class, Allocation by Thread, Allocation by
Size, plus a pie chart showing the Thread Local Area (TLA) by size
• Heap Contents: A graph showing the Total Heap Size, Used memory, and
Fragmentation

Code > Overview
Java packages
requiring the most
CPU time
Java classes requiring

the most CPU time
Other tabs under Code:

• Hot Methods: A list of methods requiring the most CPU time. Click a method in the list
to see predecessor and successor methods.
• Exceptions: Top exceptions by class. Click an exception class to see its stack trace.
• Code Generation: JIT Compilations and Optimizations

Memory Leak Detector (Memleak): Features
To access from JRockit Mission Control, right-click a server

and select Start Memleak.
Tab Description
Trend • Monitor Java types whose memory consumption is growing fast.

• Select a type to investigate.
Type Graph / • Visualize the relationships between types.
Type Tree • Select a specific object instance to investigate.
• In graph or tree form
Instance • Visualize the relationships between the instance in the heap.
Graph / • Inspect the attributes of an object.
Instance Tree • In graph or tree form
Allocation View stack trace showing how the selected object was created.
To perform trend analysis is to observe continuously updated object type–related information

and try to discover object types with suspicious memory growth. These object types should
then be studied in the next phase of the memory leak detection process. In the Memory Leak
Detector, this information is updated every ten seconds—or more often if there are very
frequent garbage collections. Trend analysis is a good way to find even the smallest leaks,
before they throw out-of-memory exceptions.
Studying object-type relations means following reference paths between object types. The
goal is to find interesting connections between growing object types and the types of objects
that point to them.
Instance investigation consists of finding an instance of abnormal memory size or an
abnormal amount of references being held and then inspecting that instance. When you
inspect an instance, values (for example, field names, field types, and field values) are
displayed. These values will hopefully lead you to the correct place for the error in the
application code (that is, where that particular instance of the particular object type is
allocated, modified, or removed from the collection, depending upon what the situation
implies).

Memleak: Trend Tab
Colors identify suspicious

types that have growing
memory usage.
Right-click and select

“Add to Type Graph” to
add to the Type Graph
and Type Tree.
From the Trend tab, you start the analysis of your applications. The object types with the
highest growth in bytes/sec are marked red (darkest) in the Trend Analysis table and listed at
the top of the table. For each update, the list can change and the type that was the highest
can move down the list. In addition, the table displays:
• The percentage of the Java heap occupied by the type of object
• The number of instances of the type that currently exist
• The size (in MB or KB) that the type consumes
The trend analysis should start automatically. If not, click the Start button above and to the
right of the table to start the trend analysis. Click the Pause button to freeze the updating of
trend data so that you can start to analyze the application. If you want to view more data from
the same analysis run, click the Play button again and the Memory Leak Detector resumes
displaying samples. If you instead use the Stop button and then later Start again, the current
trend data is reset.
When you find a suspected memory leak, you investigate the suspected leak further on the
Type Graph or Type Tree tab. Before anything is displayed on the tab, you need to select a
type from the Trend tab, right-click it, and then select the “Add to Type Graph” option.

Memleak: Type Graph
Number of objects Browse an interactive class

using this type relationship diagram.
Same colors used

on theTrend tab
Right-click and select List All Instances
to list them in the Instances view.
The Type Graph tab offers a view of the relationships between all the types pointing to the
type that you are investigating. For each type you also see a number, which is the number of
instances that point to that type. Once again, darker colors refer to types that appear to have
a high growth rate in terms of their memory consumption. Several buttons are available on the
toolbar to help you navigate the diagram: Zoom in, Zoom out, Reset zoom level, and so on.
Right-click a type and select List All Instances to display them in the Instances window. To
further investigate a specific instance of the selected type, right-click an instance in the
Instances view and then select Inspect Instance. The instance is displayed in the Inspector
view. When inspecting an instance, you will see all fields (variables) that the object contains.
This information will help you pinpoint where the leaking object is located in your application.
To add a specific instance to the Instance Graph or Instance Tree views, right-click an
instance and select “Add to Instance Graph.”

Section Summary

• Describe the capabilities of the jrcmd tool
• Enable the JRockit management and diagnostics agent
• List several features of JRMC
• Connect to and monitor a running JVM by using JRMC
• Record and browse diagnostic data by using JRMC
• Identify and investigate possible memory leaks by using
JRMC

Quiz
Garbage collection algorithms based on object age organize

JVM memory into _____.
a. Locks
b. Heaps
c. Generations
d. Classes
Answer: c

Quiz
Which of these is not an environment variable used by WLS

start scripts?
a. HEAP_ARGS
b. JAVA_VENDOR
c. JAVA_OPTIONS
d. USER_MEM_ARGS
Answer: a

Quiz
Name three tools included with either the JRockit SDK or Sun
JDK.
a. jstat
b. jrcmd
c. jopt
d. joracle
e. jstack
Answer: a, b, e

Quiz
Which JRMC component provides real-time monitoring of the

CPU, JVM memory, and JVM threads?
a. Memory Leak Detector
b. Java Browser
c. Management Console
d. Runtime Analyzer
e. Latency Events
Answer: c

Summary

• Define the terms heap, garbage collection, and memory
leak
• List some common JVM command-line options
• Investigate JVM memory errors
• Trigger a thread dump and analyze its contents
• Locate and use basic JVM diagnostic and profiling tools
• List the features of JVisualVM and JRockit Mission Control
• Monitor the JVM from the WLS console

Practice 6-2
Troubleshooting Applications on JRockit
• Starting a server under JRockit
• Navigating the features of JRockit Mission Control
• Monitoring CPU usage by using the Management Console
• Profiling execution time by using the Runtime Analyzer
• Profiling heap usage with the Memory Leak Detector

Troubleshooting Java Applications

Objectives

• Interpret a Java exception chain
• Describe some common Java exceptions and errors
• List several techniques by which you can make code
available to WebLogic
• Discuss the class loader search process

Java Exception-Handling Concepts
• Exceptions indicate JVM, server, or application issues.

• Checked exceptions represent recoverable issues and
should be caught and handled within your application
code.
• Unchecked exceptions or errors:
– Represent lower-level system issues
– Do not have to be handled by your application
– Are typically caught by WLS and recorded to the log as
errors
WLS HRWebAction.class HRWorkflow.class
Unchecked error not handled

by the application
When an error occurs within a method, the method creates an object and hands it off to the
runtime system. The object, called an exception object, contains information about the error,
including its type and the state of the program when the error occurred. Creating an exception
object and handing it to the runtime system is called throwing an exception.
The runtime system searches the call stack for a method that contains a block of code that can
handle the exception. This block of code is called an exception handler. The search begins with
the method in which the error occurred and proceeds through the call stack in the reverse order
in which the methods were called. The exception handler chosen is said to “catch the
exception.”
The first kind of exception is the checked exception. These are exceptional conditions that a
well-written application should anticipate and recover from (for example, a File Not Found
exception). The second kind of exception is the error. These are exceptional conditions that are
external to the application and that the application usually cannot anticipate or recover from.
For example, suppose that an application successfully opens a file for input but is unable to
read the file because of a hardware or system malfunction. An application might choose to
catch this exception, in order to notify the user of the problem, but it also might make sense for
the program to print a stack trace and exit.

Exception Chains
• When caught, an exception can simply be wrapped within

another exception and triggered again.
• This technique:
– Results in a chain of error messages and stack traces
– Is commonly done by WLS subsystems and application
frameworks
javax.faces.FacesException: #{hrWebController.update}:
javax.ejb.EJBTransactionRolledbackException: EJB Exception;
nested exception is:
java.io.IOException:
java.io.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite ...
at java.io.OutputStreamWriter.flush ...
at com.mycompany.hr.HRManagerEJB.refreshCache ...
An application often responds to an exception by throwing another exception. In effect, the first
exception causes the second exception. It can be very helpful to know when one exception
causes another. Chained exceptions in Java help the programmer do this.
When the JVM prints an exception that contains one or more nested exceptions, the stack
traces for each exception are printed to facilitate easier tracing and debugging. The full depth of
the exception chain is included. The exact exception-handling text that a class generates is not
standardized, but most WLS subsystems and third-party frameworks will include phrases such
as “nested exception is” or “caused by” to indicate an exception chain.

“Class Not Found” Errors
These type of errors typically indicate that:

• The JVM cannot locate the compiled class definition file
(.class) for a type being referenced by the program
• The compiled class definition file was available during
compilation but not at run time
java.lang.NoClassDefFoundError:
org/apache/activemq/ActiveMQConnectionFactory
java.lang.ClassNotFoundException:
oracle.jdbc.driver.OracleDriver
NoClassDefFoundError is thrown if the Java Virtual Machine or a class loader instance tries
to load in the definition of a class (as part of a normal method call or as part of creating a new
instance using the new expression) and no definition of the class can be found. The class
definition existed when the currently executing class was compiled, but the definition can no
longer be found during program execution. This error is a type of linkage error, meaning that
although a class has some dependency on another class, the latter class has incompatibly
changed after the compilation of the former class.
ClassNotFoundException is thrown when an application tries to explicitly load in a class by
name but no definition for the class with the specified name could be found. Typical examples
include:
• forName method in class Class
• findSystemClass method in class ClassLoader
• loadClass method in class ClassLoader

Class Cast Errors
• This error occurs when a program expects one type of

object but receives another.
• Typical causes include:
– A poorly designed and/or implemented application
– An application use case that was never identified or tested
– Multiple versions of the same class found by the JVM
– Custom class loader configurations
java.lang.ClassCastException: org.dom4j.DocumentFactory
at org.dom4j.DocumentFactory.getInstance ...
Every expression written in the Java programming language has a type that can be deduced
from the structure of the expression and the types of the literals, variables, and methods
mentioned in the expression. It is possible, however, to write an expression in a context where
the type of the expression is not appropriate. In some cases, for simple object types, the
language performs an implicit conversion from the type of the expression to a type acceptable
for its surrounding context. However, in most cases casting is required to direct the Java
compiler to treat a variable of a given type as another. It can be done with both primitive types
as well as user-defined types.
The Java language specification defines programming scenarios in which casting is
permissible. For example, an object can be cast “upward” to a more generic type, such as
casting from a Manager type to an Employee type. Conversely, an object can also be cast
“downward” to a more specific subclass. These casting rules are validated at compile time. If an
incompatibility is detected during this runtime validation process, the JVM raises a
ClassCastException. It indicates that the code has attempted to cast an object to a
subclass of which it is not an instance.
Java does provide an instanceof operator to allow programmers to check at run time
whether an object belongs to a specific class, or implements a specific interface. However, it is
generally considered a poor practice to use this operator in lieu of well-designed, well-tested
code.

Classpath: Review
• The Java classpath provides a static list of locations from

which the JVM searches for packages and classes:
– Folders
– Archives (JAR)
• Configure the classpath by using one of the following:
– The -classpath JVM option
– A CLASSPATH environment variable
CLASSPATH=${WL_HOME}/server/lib/weblogic_sp.jar:${WL_HOME}/ser
ver/lib/weblogic.jar:...
...
java ... weblogic.Server
java –Xms256m –Xmx512m –classpath ... weblogic.Server
In Java, the classpath is the list of directories that the virtual machine uses to search for
dependent classes in a program. You can set this classpath using an environment variable or
as a command-line parameter when you execute the virtual machine.
After installation, WebLogic Server’s classpath is already set, but you may choose to modify it
for a number of reasons, such as adding a patch to WebLogic Server, updating the version of
the database driver you are using, or adding support for Log4J logging.
The shell environment in which you run a server determines which character you use to
separate elements in the classpath. On Windows, you typically use a semicolon (;). In a BASH
shell, you typically use a colon (:).

WebLogic Start Script: Review
Product installation
Patches to add to the
classpath for all domains
setPatchEnv
Standard WLS
classpath for
all domains commEnv setWLSEnv For command-
line tools
Domain A Domain B
Domain-specific
setDomainEnv setDomainEnv classpath additions
startWebLogic startWebLogic
To start a server within a domain, you can use the generated startWebLogic script or develop
your own custom scripts. The default startWebLogic script executes your domain’s
setDomainEnv script. This script in turn calls a script named commEnv, which is included with
your product installation. The commEnv script uses the setPatchEnv script, which is
responsible for initializing variables that point to your currently installed patches. Another script
that makes use of commEnv is setWLSEnv, which is not directly used to start servers. Instead,
it provides a convenient way of initializing your environment to support WebLogic developer
and administrator tools, including Ant and WLST.
Some patches directly replace existing resources in your product installation and therefore
become effective automatically as soon as they are applied. They are not enabled through a
reference in a server start script, such as a classpath. Examples of patches that typically
contain replacement artifacts include resources for Web server plug-ins, native socket
multiplexers, and native dynamically linked libraries.
Other patches include Java class or library files that are loaded by server start scripts. These
patch files are written to a special location within your Oracle root installation folder,
<MIDDLEWARE_HOME>. These folders typically start with the name patch_.

Viewing the WLS Classpath
• For debugging purposes, startWebLogic prints the

current values of the PATH and CLASSPATH variables
before launching the server.
• Alternatively, you can use client tools such as jinfo or
jrmc to inspect the classpath for a JVM process.
> startManagedWebLogic.sh Server4 admhost:7001

...
CLASSPATH=/u01/middleware/patch_wls1031/profiles/default/sys_m
anifest_classpath/weblogic_patch.jar:/u01/middleware/jdk160_11
/lib/tools.jar: ...
...
Recall that the convenience script startManagedWebLogic simply calls the standard
startWebLogic script and initializes some basic WebLogic system properties
(-D<name>=<value>), specifically the name of the managed server and the location of the
Administration Server.
The jinfo command-line utility gets configuration information from a running Java process or
crash dump and prints the system properties or the command-line flags that were used to start
the virtual machine. If you start the target Java VM with the -classpath and
-Xbootclasspath arguments, the output from jinfo provides the settings for
java.class.path and sun.boot.class.path. This information might be needed when
investigating class loader issues.

Manifest Files and the Classpath
• JAR files can include a META-INF/MANIFEST.MF text file.

• A JAR manifest can be used to add more entries to the
classpath relative to the JAR file’s location.
• While starting up, WLS prints these manifest entries for
debugging purposes.
Adds to classpath
weblogic.jar
xmlx.jar
commEnv
MANIFEST
weblogic_patch.jar patchABC.jar
setPatchEnv
MANIFEST patchDEF.jar
The commEnv script explicitly adds the basic WLS libraries to the classpath, including
weblogic.jar. Some of these JAR files include manifest files that indirectly append
additional paths and JAR files to the system classpath.
The commEnv script calls the setPatchEnv script, which includes default definitions for the
environment variables that specify the locations of patch JAR files. By default, these patch
variables are in effect for every WebLogic Server instance that is started by using commEnv.
The setPatchEnv script variable PATCH_CLASSPATH defines a single JAR file named
weblogic_patch.jar. This file contains no Java classes but includes a manifest file. The
manifest file lists all of the specific patch JAR files that are to be dynamically included in the
server classpath. As new patches are downloaded and installed, this manifest file is updated.
With this approach, scripts such as commEnv, setDomainEnv, and startWebLogic need not
be modified.
Alternatively, you can customize a domain’s setDomainEnv script to contain its own definition
for the PATCH_CLASSPATH variable. In this scenario, the definition of PATCH_CLASSPATH in
the commEnv script is overridden for those server instances. It is important that, if you add a
patch path variable definition to a start script, you place the definition before the statement that
invokes another start script.

Domain Libraries
• WLS also supports the addition of extra JAR files to the

system classpath by using a domain’s lib folder.
• This technique affects the classpath of all servers for a
single domain.
WebLogic Server includes a lib subdirectory, located in the domain directory, that you can
use to add one or more JAR files to the WebLogic Server system classpath when servers start
up. The lib subdirectory is intended for JAR files that change infrequently and are required by
all or most applications deployed in the server, or by WebLogic Server itself. For example, you
might use the lib directory to store third-party utility classes that are required by all
deployments in a domain. You can also use it to apply patches to WebLogic Server.
The lib directory is not recommended as a general-purpose method for sharing JARs
between one or two applications deployed in a domain, or for sharing JARs that need to be
updated periodically. If you update a JAR in the lib directory, you must reboot all servers in
the domain for applications to realize the change.

Java Class Loaders
• The system classpath cannot be modified without

restarting the JVM.
• Class loaders:
– Represent locations where class definitions can be found
– Are organized as a hierarchy with parents and children
– Are dynamically created and destroyed by WLS to implement
hot deployment
Class Loader B
Children
Class Loader A app1.jar
of A
lib1.jar, lib2.jar Class Loader C
app1.jar
Class loaders are a fundamental module of the Java language. A class loader is a part of the
Java virtual machine (JVM) that loads classes into memory; a class loader is responsible for
finding and loading class files at run time.
WebLogic Server allows you to deploy newer versions of application modules such as EJBs
while the server is running. This process is known as hot-deploy or hot-redeploy and is closely
related to class loading. Java class loaders do not have any standard mechanism to undeploy
or unload a set of classes, nor can they load new versions of classes. To make updates to
classes in a running virtual machine, the class loader that loaded the changed classes must be
replaced with a new class loader. When a class loader is replaced, all classes that were loaded
from that class loader (or any class loaders that are offspring of that class loader) must be
reloaded. Any instances of these classes must be re-instantiated.
In WebLogic Server, each application has a hierarchy of class loaders that are offspring of the
system class loader. These hierarchies allow applications or parts of applications to be
individually reloaded without affecting the rest of the system.

Searching Class Loaders
The JVM uses a delegation model to search for class

definitions at run time. When a class is referenced, the class
loader goes through this process:
1. Check for the class in the cache. If the class is there, it is
returned. If the class is not found, the class loader goes to
Step 2.
2. Ask the parent class loader if the parent has (or can find)
the class. If the class is not returned from the parent, the
loader goes to Step 3.
3. Check the class loader’s own search path for the class. If
found, the class is returned and placed in its cache. If not
found, a ClassNotFound exception is thrown.
The class loader is responsible for locating libraries, reading their contents, and loading the
classes contained within the libraries. This loading is typically done “on demand.” That is, it
does not occur until the class is actually referenced by the code.
A thread of execution has a “current” or “local” class loader. When a thread need a class, it
asks its current class loader to produce or find it. The class loader does the following:
1. It checks its cache to see if it already has the class loaded. If so, the class is returned.
2. If the class is not cached, it asks its parent class loader if it has (or can find) the class.
3. If the class is not returned to the class loader by its parent, the class loader checks its
search path to see if it can find the class and return it. If the class is not found, it throws a
ClassNotFound exception.
This is the same process right up the chain of class loaders. So, if no class loader has the
class in question already cached, the bootstrap (top of the hierarchy) class loader gets the first
chance to search its path for the class and actually load it.
This delegation model is followed to prevent multiple copies of the same class from being
loaded. Multiple copies of the same class can lead to a ClassCastException.

Searching Class Loaders: Example
Class Loader C
cache
cache
6 AccessRight.class
AccessRight.class
Check cache Class found in

5 Class not found; 7 cache and
for class. ask parent.
returned
Class Loader B
cache
4
Check cache Class not found;
for class. 3
ask parent.
Current Class Loader
cache New class being

2 1 referenced in code
Check cache a = new AccessRight();
for class.
1. A class that is not already loaded is being referenced in code.

2. The current class loader checks its cache for the class being referenced. If the class is
found, it returns the class.
3. The class is not found, so the class loader asks its parent for the class.
4. The parent class loader (B) checks its cache for the class. If found, it returns the class.
5. The class is not found, so the class loader asks its parent for the class.
6. The parent class loader (C) checks its cache for the class.
7. The class is found in the cache and returned.
In the example, the class is found in a class loader’s cache. If the class is not found in any
class loader’s cache, the top class in the hierarchy (the bootstrap class loader) searches its
path for the class. If the class is found, it is placed in the bootstrap class loader’s cache and
returned. If the class is not found, a ClassNotFound exception is thrown. The next class
loader down the hierarchy catches the exception and searches its path for the class. If the
class is found, it is placed in that class loader’s cache and returned. If the class is not found,
the exception is thrown and the process continues. If the class is not found by any parent class
loader, it goes all the way back to the current class loader. When the current class loader’s
path is searched and the class is still not found, the ClassNotFound exception is thrown and
not caught.

Default WLS Class Loader Hierarchy
Bootstrap Class Standard Java libraries

Loader (java.* packages)
System Class Enterprise applications

JVM classpath
Loader are deployed as multiple
class loaders.
Web App 1 Class EJB App 1 Class

Loader Loader
EAR
Class Loader for all
Each application has EJB Modules
its own class loader.
Web Module 1 Web Module 2

Class Loader Class Loader
The bootstrap class loader is the root of the Java class loader hierarchy. The Java virtual
machine (JVM) creates the bootstrap class loader, which loads the Java development kit (JDK)
internal classes and java.* packages included in the JVM (for example, the bootstrap class
loader loads java.lang.String). The extensions class loader (not depicted) is a child of the
bootstrap class loader. The extensions class loader loads any JAR files placed in the
extensions directory of the JDK. This is a convenient means to extending the JDK without
adding entries to the classpath.
The system class loader extends the JDK bootstrap and extensions class loaders. The system
class loader loads the classes from the classpath of the JVM. Application-specific class
loaders, such as those created by WebLogic Server, are children of the system class loader.
WLS class loading is centered on the concept of an application. An application is normally
packaged in an enterprise archive (EAR) file containing application classes. Everything within
an EAR file is considered part of the same application. WLS automatically creates a hierarchy
of class loaders when an application is deployed. The root class loader in this hierarchy loads
any EJB JAR files in the application. A child class loader is created for each Web application
WAR file. Because it is common for Web applications to call EJBs, the WLS application class
loader architecture allows Web components to see the EJB interfaces in their parent class
loader. This architecture also allows Web applications to be redeployed without redeploying the
EJB tier.

Java EE Packaging: Review
EAR Parent Application

Class Loader
Or APP-
INF/lib
Web Module
Class Loader
WAR
EJB JAR
The root of a WAR hierarchy defines the document root of your Web application.
All files under this root directory can be served to the client over HTTP, except for files under
WEB-INF. All files under WEB-INF are private and are not served to a client, including XML
deployment descriptors. Individual packages and class files are placed within WEB-
INF/classes, while those that have been archived into JAR files are placed under WEB-
INF/lib.
EJB applications and modules have a very simple structure. Place packages and class files
within the root of the archive, as with any standard JAR file.
WebLogic Server provides a location within an EAR file where you can store shared utility
classes. Place utility JAR files in the APP-INF/lib directory and individual classes in the
APP-INF/classes directory. (Do not place JAR files in the /classes directory or classes in
the /lib directory.) These classes are loaded into the root class loader for the application.
This feature eliminates the need to place utility classes in the system classpath or place
classes in an EJB JAR file. Alternatively, utilize the Class-Path entry within a JAR file’s
manifest. Remember that the location specified in the manifest is relative to the path of the
containing JAR file.

Prefer Web Application Classes
• By default, class definitions in parent class loaders are

used over any definitions in the current class loader.
• A Web application or module may wish to use a different
version of some library found in the system class loader.
• Override the default behavior by using the weblogic.xml
descriptor.
...
<container-descriptor>
...
<prefer-web-inf-classes>true</prefer-web-inf-classes>
</container-descriptor>
The weblogic.xml Web application deployment descriptor contains a <prefer-web-inf-

classes> element (a subelement of the <container-descriptor> element). By default,
this element is set to “false.” Setting this element to “true” subverts the class loader delegation
model so that class definitions from the Web application are loaded in preference to class
definitions in higher-level class loaders. This allows a Web application to use its own version of
a third-party class instead of one that might also be part of WebLogic Server.
When using this feature, you must be careful not to mix instances created from the Web
application’s class definition with instances created from the server’s definition. If such
instances are mixed, a ClassCastException results.

Prefer Enterprise Application Classes
• A similar capability is available for enterprise applications.

• Use weblogic-application.xml to specify individual
Java packages to load in favor of those in parent class
loaders.
• Wildcard expressions are supported.
...
<prefer-application-packages>
<package-name>org.apache.log4j.*</package-name>
<package-name>antlr.*</package-name>
</prefer-application-packages>
In WebLogic Server, any JAR file present in the system classpath is loaded by the WebLogic
Server system class loader. All applications running within a server instance are loaded in
application class loaders that are children of the system class loader. In this implementation of
the system class loader, applications cannot use different versions of third-party JARs that are
already present in the system class loader. Every child class loader asks the parent (the
system class loader) for a particular class and cannot load classes that are seen by the parent.
The WLS Filtering Class Loader provides a mechanism for you to explicitly specify that certain
packages should always be loaded from the application, rather than being loaded by the
system class loader. Although this feature lets you bundle and use third-party JARs in your
application, it is not recommended that you filter out standard classes in the javax.*
packages or weblogic.* packages. To configure the Filtering Class Loader, add a
<prefer-application-packages> descriptor element to the enterprise application’s
weblogic-application.xml file. This element specifies the list of packages to be loaded
from the application.
The same deployment descriptor also supports a <classloader-structure> element,
which allows you to define your own custom hierarchy of class loaders for each of the modules
within the enterprise application. This feature has many consequences and limitations, and is
therefore recommended only for advanced WebLogic users.

Client Library Errors
• Remote EJB/JMS clients often require one or more WLS

client JAR files in their classpaths.
• Remote EJB clients also require access to generated EJB
client classes or “stubs,” normally packaged in JAR files.
• Missing or old versions of WLS or EJB client libraries may
result in runtime errors.
Exception in thread "main" java.lang.ClassCastException

at com.sun.corba.se.internal.javax.rmi.
PortableRemoteObject.narrow(Unknown Source)
at javax.rmi.PortableRemoteObject.narrow(Unknown Source)
at com.mycompany.SupportClient.callEJB ...
A stand-alone client is a client that has a JVM runtime environment independent of WebLogic
Server. Stand-alone clients that access WebLogic Server applications range from simple
command-line utilities that use standard I/O to highly interactive GUI applications built using the
Java Swing/AWT classes.
WLS includes both thin and full client JAR files. Although the WebLogic full client requires the
largest JAR file among the various clients, it supports the most features (clustering, JMS, store
and forward, and so on) and is the best overall performer.
When you run the appc compiler, a JAR file with the classes required to access the EJB is
generated. Make the client JAR available to the remote client’s classpath.
Although infrequent, when you generate classes with the appc compiler, you may encounter a
generated class name collision which could result in a ClassCastException and other
undesirable behavior. This is because the names of the generated classes are based on three
keys: the bean class name, the bean class package, and the configured name for the bean.
This problem occurs when you use an EAR file that contains multiple JAR files and at least two
of the JAR files contain an EJB with both the same bean class, package, or classname, and
both of those EJBs have the same name in their respective JAR files. If you experience this
problem, change the name of one of the beans to make it unique.

Null Pointer Errors
• This common error occurs when a program expects a

variable defined in the code to have a value but it does not.
• Typical causes include a:
– Poorly designed and/or implemented application
– Use case that was never identified or tested
– Missing parameter in a configuration file
– Lack of error-handling logic
• This error normally requires the use of a debugger tool to
step through and analyze the offending code.
java.lang.NullPointerException
at org.apache.struts.tiles.definition.
ComponentDefinitionsFactoryWrapper.getDefinition ...
A NullPointerException in Java is thrown when an application attempts to use a variable

that currently has no value (“null”) in a case where an object is required. Typical cases include:
• Calling the instance method of a null object
• Accessing or modifying the field of a null object
• Taking the length of null as if it were an array
• Accessing or modifying the slots of null as if it were an array
• Throwing null as if it were an exception object
A NullPointerException is an unchecked runtime exception in Java, and therefore very
few applications or third-party frameworks explicitly check for and handle it. It is assumed that
these errors should be detected during development and testing, and that they are relatively
easy to eliminate.

Stack Overflow Errors
• Recursion is a common programming technique in which a

method calls itself but uses different parameters.
• This error normally occurs when:
– Some recursive procedure runs indefinitely until the size of
the JVM call stack has reached its maximum
– The recursion is not intended or implemented correctly
• Periodically capture thread dumps to try to identify the
location of the recursive code.
java.lang.StackOverflowError
at weblogic.servlet.internal.RequestDispatcherImpl.forward
at org.apache.solr.servlet.SolrDispatchFilter.doFilter ...
A StackOverflowError in Java is usually caused by “infinite recursion,” meaning that some

method is calling itself over and over and no exit condition is ever reached. JVMs implement a
call stack to store the location and parameters of each method call. Although this call stack is
normally large enough to handle even complex, many-layered Java EE applications, infinite
recursion will cause the JVM to go past this limit and trigger an error. This is done to keep the
JVM from potentially running out of memory.
Under most circumstances the stack trace associated with this error should indicate where the
recursion is taking place. However, if this type of error occurs within the JVM, native library, or
WLS, it may cause a JVM crash. In this case, test the scenario again and capture frequent
thread dumps prior to the crash. If the problem appears to be caused by a standard Java
library, contact the JVM vendor for support.

Too Many Open Files Errors
• JVMs require OS file descriptor objects to:

– Open files
– Open TCP socket connections
• Most operating systems limit the number of descriptors
that are available to a single process.
• If this limit is reached, the JVM triggers an error.
java.net.SocketException: Too many open files
java.io.IOException: Too many open files
commEnv script:
... Edit OS configuration files or WLS
ulimit –n 8192 scripts to adjust file descriptor limit
A file descriptor is a handle represented by an unsigned integer and is used by a process to

identify an open file. It is associated with a file object that includes information such as the
mode in which the file was opened, its position type, its initial type, and so on. File descriptors
are generally unique to each process. When a child process is created with a fork subroutine,
the child gets a copy of all its parent process’s file descriptors that were open at the time of the
fork.
When a socket is closed, it first enters a TIME_WAIT state to ensure that all the data was
transmitted during the connection and so that a final acknowledgment (ACK) can end the data
transfer. This period also delays the release of the file descriptor allocated to it, and can be
tuned on your OS if necessary.
The number of file descriptors available to WLS needs to be sufficient for the following:
• Files used by the JVM to access class definitions and JARs
• Data files used by WLS and applications, such as server and subsystem logs
• Files associated with active sockets
• Files associated with closed but not yet released sockets
Your OS includes various tools and configuration files to monitor file descriptors and control
their availability to processes. For example, most UNIX systems include the lsof command-
line tool, while most Windows systems provide a similar tool named handle.

Quiz
Which of the following is not a standard Java error type?

a. ClassCastException
b. StackOverflowError
c. NoClassDefFoundError
d. JVMBuildError
e. NullPointerException
Answer: d

Quiz
Name three entities that receive their own dedicated class

loaders in WLS, by default.
a. Diagnostic modules
b. JVM classpath
c. Transaction log
d. EJB modules in an EAR
e. Web application
Answer: b, d, e

Summary

• Interpret a Java exception chain
• Describe some common Java exceptions and errors
• List several techniques by which you can make code
available to WebLogic
• Discuss the class loader search process

Practice 7-1
Investigating Classpath Problems
• Locating and analyzing a classpath error
• Browsing application and server classes
• Debugging server start scripts

Troubleshooting Servers

Objectives

• Investigate common server startup issues
• Discuss the use of WLS work managers
• Analyze server thread dumps and application deadlocks
• Investigate common deployment issues
• Monitor running applications

Road Map
• Server Diagnostics
– Startup Errors
– Native Libraries
– Thread States
– Work Managers
– Deadlocks
– Overload Protection
• Application Diagnostics

WLS Message Catalog: Review
Use the WLS online message catalog to obtain more

information about a specific log message ID.
For a detailed description of the log messages in the Oracle WebLogic Server message
catalogs, see “Oracle WebLogic Server Message Catalog” in the online documentation. There
is a link to it on the main Oracle WebLogic Server Documentation Library web page. The
index of messages describes all the messages generated by the Oracle WebLogic Server
subsystems and provides a detailed description of the error, a possible cause, and a
recommended action to avoid or fix the error.

Server Startup Errors
• Severe errors during server startup may cause:

– WLS to exit prematurely
– The JVM to crash (rare)
• Common causes include:
– Missing or invalid server start parameters
– Custom start scripts
– Missing or invalid boot credentials or SSL files
– An invalid domain configuration
– A corrupted LDAP server
<Critical> <WebLogicServer> <BEA-000362> Parsing Failure in
config.xml on line 295.>
...
<Error> <WebLogicServer> <BEA-000383> A critical service
failed. The server will shut itself down.
Possible server startup error messages include the following:

• The address <listenAddress> might be incorrect or another process is using port
<port>.
• Unable to create a server socket for port <port>. Perhaps another process is using port
<port>.
• Cannot use SSL; no certificates have been specified in the WebLogic configuration
• The initialization of SSL failed due to an exception when loading the private key, loading
the server certificates, loading the trusted CA certificates, or setting the trust validator.
• Server failed during initialization. (Exception: <exception>)
• Server shutdown due to fatal exception: <exception>
The first time you start a managed server instance, it must be able to contact the
Administration Server. Thereafter, the managed server instance can normally start even if the
Administration Server is unavailable. It can retrieve its configuration by reading its locally
cached configuration data from the config directory.
To start and stop a WebLogic Server instance, you must provide the credentials of a user who
is permitted to start and stop servers for the domain. Users must fit the criteria of the global
Admin role defined in the security realm. By default, this role includes all members of the
Administrators group.

Boot Identity Errors
• A server can retrieve credentials from an encrypted

boot.properties file during startup.
• If plain text, the server will automatically encrypt it again.
• Errors can occur if you:
– Update the corresponding user account without editing this
file
– Attempt to reuse an encrypted file on another machine
• If all else fails, delete the entire /servers/<serverName>
directory and restart.
<Critical> <Security> ... <BEA-090402> <Authentication denied:

Boot identity not valid; The user name and/or password from
the boot identity file (boot.properties) is not valid ...
If you use the Configuration Wizard to create a domain in development mode, the
Configuration Wizard creates an encrypted boot identity file in the security directory of the
Administration Server’s root directory. In production mode, you must create this file manually.
Start the Administration Server at least once and provide the user credentials on the
command line. During the Administration Server’s initial startup process, it generates security
files that must be in place before a server can use a boot identity file. If you save the file as
boot.properties and locate it in the security directory of the server’s root directory, the
server automatically uses this file during subsequent startup cycles. The first time you use this
file to start a server, the server reads the file and then overwrites it with an encrypted version
of the username and password.
If you want to specify a different file (or if you do not want to store boot identity files in a
server’s security directory), you can include the following argument in the server’s startup
command:-Dweblogic.system.BootIdentityFile.
If a managed server uses the same root directory as the Administration Server, it can use the
same boot properties file as the Administration Server, or it can use its own dedicated file. If
you use a Node Manager to start a managed server, you do not need to create a server boot
identity file. Instead, you need to specify the boot credentials in the Node Manager’s
configuration.

WLS Native Libraries
• WLS is not 100% Java. Some platform-specific libraries

are bundled with WLS to increase performance.
• These libraries are also referred to as performance packs.
• Separate libraries are also available for 32-bit and 64-bit
platforms.
• Log messages identify the use of performance packs
during startup.
<Info> <Socket> ... <BEA-000446> <Native IO Enabled.>
<Info> <Socket> ... <BEA-000438> <Unable to load performance

pack. Using Java I/O instead.>
Cannot locate libraries for
this platform/architecture
By default, WebLogic utilizes native threads to read incoming requests from sockets, if the
appropriate native libraries can be located during server startup. These special threads are
also known as muxers. The majority of all platforms provide some mechanism to poll a socket
for data. For example, UNIX systems use the poll system call and the Windows architecture
uses completion ports. Native muxers provide superior scalability because they implement a
nonblocking thread model, unlike standard Java threads. When a native muxer is used, the
server creates a fixed number of threads dedicated to reading incoming requests.
When nonnative I/O is used, I/O is performed in a blocking manner. Therefore, if the number
of socket reader threads is less than the number of active sockets, performance may degrade
significantly. It is recommended that you change your domain settings to improve the
performance. For example, make sure the native libraries for your target environment are
available to the server. Similarly, it is also possible to explicitly disable native I/O by using
server command-line parameters or server settings in the configuration repository
(Configuration > Tuning tab).
Refer to the Product Certification section of the online documentation to determine whether a
WebLogic performance pack is available and supported for your target platform, architecture,
and JVM.

Setting the Native Library Path
• Libraries are typically found at

<WL_HOME>/server/native.
• The commEnv script is responsible for:
– Setting the native library path based on the current platform
– Selecting 32-bit or 64-bit libraries, if available
...
if [ ${arch} = "ia64" ]; then
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${WL_HOME}/server/
native/linux/ia64 ...
else
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${WL_HOME}/server/
native/linux/i686 ...
Script excerpt
for Linux
The environment variable used to identify native libraries is platform dependent.

Examples
• Windows: PATH
• Linux, Tru64, Solaris: LD_LIBRARY_PATH
• AIX: LIBPATH
• HPUX: SHLIB_PATH

Causes of Unresponsive Servers
• If an unresponsive server appears to “hang”:

– New requests may be ignored or rejected
– Existing requests may time out
– CPU utilization may or may not be very high
• There are many potential causes, including:
– Insufficient resources (threads, connections, and so on)
– Out of memory
– Application or database deadlocks
– Excessive garbage collection
A server can hang for a variety of reasons. Generally, servers hang ultimately because of a
lack of some resource, which in turn prevents the server from servicing requests. For
example, due to the request volume or a problem like a deadlock, there may be no more
execute threads available to do any more work. All available resources are busy processing
previous requests.
When the server appears to hang, first try pinging the server. If the server can respond to the
ping, it may be that the application is hanging and not the server itself. Then make sure that
the server is not simply performing too much garbage collection. For example, restart the
server with -verbosegc turned on, and redirect stdout and stderr to one file.
Furthermore, if garbage collection is taking too long (more than 10 seconds), the server may
miss heartbeat messages that servers use to keep each other up to date on the topology of
the cluster.

WLS Threading Architecture
• Native socket reader threads (3) receive incoming

requests and place them on a queue.
• A pool of execute threads is created as needed to process
and respond to queued requests.
• Other internal threads are dedicated to various background
tasks.
• By default, all client requests are given equal priority.
Socket Execute Threads

Threads Application
Queue
Application
Application
WebLogic Server uses a single dynamic thread pool, in which all types of work are executed.
WebLogic Server prioritizes work based on rules that you define, and runtime metrics,
including the actual time it takes to execute a request and the rate at which requests are
entering and leaving the pool.
The common thread pool changes its size automatically to maximize throughput. The
incoming request queue monitors throughput over time and, based on history, determines
whether to adjust the thread count. For example, if historical throughput statistics indicate that
a higher thread count increased throughput, WebLogic increases the thread count. Similarly, if
statistics indicate that fewer threads did not reduce throughput, WebLogic decreases the
thread count. This strategy makes it easier for administrators to allocate processing resources
and manage performance.
The default work manager’s implicit maximum thread pool size is 400, although, due to the
context-switching overhead associated with threads, it is highly unlikely that the thread pool
will even approach this default constraint.
When a performance pack is enabled, WebLogic also uses a static pool of native socket
reader or muxer threads. These threads are responsible for receiving new requests from the
system networking layer and for handling the socket life cycle. Under healthy server
conditions, there should always be three socket reader threads.

Execute Thread State
• WLS creates and destroys threads as needed, depending

on the current server load.
• Active threads:
– Are either processing or waiting for requests
– Can be tagged with additional attributes such as Idle or
Hogging
• Standby threads are not used to process requests unless
additional throughput is needed.

at jsp_servlet.__MyPage._jspService(__MyPage.java:108)
...
[STANDBY] ExecuteThread: '13' ...
Thread dump
at java.lang.Object.wait(Native Method) excerpt
The common thread pool changes its size automatically to maximize throughput. The queue
monitors throughput over time and based on history, and determines whether to adjust the
thread count. For example, if historical throughput statistics indicate that a higher thread count
increased throughput, WebLogic increases the thread count. Similarly, if statistics indicate
that fewer threads did not reduce throughput, WebLogic decreases the thread count. This
new strategy makes it easier for administrators to allocate processing resources and manage
performance, avoiding the effort and complexity involved in configuring, monitoring, and
tuning custom executes queues.
Idle threads do not include standby threads or stuck threads. This tag indicates that a thread
is ready to pick up new work when it arrives.
Hogging threads appear to be processing a request for an unusual amount of time, basic on
historical trends. The state of these threads may eventually be changed from active to stuck,
based on your current overload protection settings.
Threads that are not needed to handle the present work load are designated as standby and
added to the standby pool. These threads are activated when more threads are needed.

Work Managers
• Service-level agreements (SLAs) describe performance

requirements for a specific application or component:
– Percentage of server resources to utilize
– Maximum response time
– Maximum number of threads to use
• An SLA is implemented on WLS using a work manager.
• If not assigned a work manager, applications use the
default one.
• To override the SLA characteristics of the default work
manager, create one named “default.”
Default Work Manager Applications

Queue
My Work Manager Applications
WebLogic Server enables you to configure how your application prioritizes the execution of its
work. Based on rules that you define and by monitoring actual runtime performance,
WebLogic Server can optimize the performance of your application and maintain service-level
agreements (SLAs). You tune the thread utilization of a server instance by defining rules and
constraints for your application by defining a work manger and applying it either globally to the
WebLogic Server domain or to a specific application component. Each distinct SLA
requirement needs a unique work manager.
To handle thread management and perform self-tuning, WebLogic Server implements a
default work manager. This work manager is used by an application when no other work
managers are specified in the application’s deployment descriptors. In many situations, the
default work manager may be sufficient for most application requirements. WebLogic Server’s
thread-handling algorithms assign to each application its own fair share by default.
Applications are given equal priority for threads and are prevented from monopolizing them.
You can override the behavior of the default work manager by creating and configuring a
global work manager called “default.” This allows you to control the default thread-handling
behavior of WebLogic Server.

Work Manager Architecture
• Request classes describe the guidelines that WLS uses to

allocate threads from the pool:
– Average percentage of execution time while under load
– Target average response time (ms)
• Constraints define additional boundary conditions that limit
when threads are allocated (can override request class).
• Error handling policies define how to handle deadlocked or
timed-out threads.
Constraint A
Request Class Work Manager
Constraint B
Applications
A request class expresses a scheduling guideline that WebLogic Server uses to allocate
threads to requests. Request classes help ensure that high-priority work is scheduled before
less important work, even if the high-priority work is submitted after the lower-priority work.
WebLogic Server takes into account how long it takes for requests to each module to
complete.
There are multiple types of request classes, each of which expresses a scheduling guideline
in different terms. A work manager may specify only one request class. A fair share request
class specifies the average thread-use time required to process requests. For example,
suppose that WebLogic Server is running two modules. The work manager for Module1
specifies a fair share of “80” and the Work Manager for Module2 specifies a fair share of “20.”
During a period of sufficient demand, with a steady stream of requests for each module so
that the number requests exceeds the number of threads, WebLogic Server allocates 80%
and 20% of the thread-usage time, respectively, to Module1 and Module2.
A constraint defines minimum and maximum numbers of threads allocated to execute
requests and the total number of requests that can be queued or executing before WebLogic
Server begins rejecting requests.
In response to stuck threads, you can define an error-handing policy that shuts down the work
manager, moves the application into admin mode, or marks the entire server instance as
failed.

Creating a Work Manager
Using the Administration Console, you can create global work managers that are used to
prioritize thread execution.
To create a global work manager:
1. In the left pane of the Console, expand Environment and select Work Managers. Click
New.
2. Select Work Manager, and click Next.
3. Enter a Name for the new work manager, and click Next.
4. In the Available Targets list, select server instances or clusters on which you will deploy
applications that reference the work manager. Then click Finish.

Creating and Using a Request Class
3
1
Edit the work manager and

assign the request class.
4
After you have created a global work manager, you typically create at least one request class
or constraint and assign it to the work manager. Each work manager can contain only one
request class, but you can share request classes among multiple work managers.
To create a global request class:
1. In the left pane of the Console, expand Environment and select Work Managers. Click
New and then select the type of global request class that you want to create. Click Next.
2. For a fair share request class, enter the numeric weight in the Fair Share field. For a
response time request class, enter a time (in milliseconds) in the Response Time Goal
field. When finished, click Next.
3. In the Available Targets list, select server instances or clusters on which you will deploy
applications that reference this request class. Then click Finish.
4. Edit your existing work manager. Select your new request class by using the Request
Class field, and then click Save.

Assigning Work Managers to Applications
Assign a WM to a web application or module by using weblogic.xml:

...
<wl-dispatch-policy>HighPriorityWM</wl-dispatch-policy>
Assign a WM to a specific EJB by using weblogic-ejb-jar.xml:

...
<weblogic-enterprise-bean>
<ejb-name>AccountManager</ejb-name>
...
<dispatch-policy>HighPriorityWM</dispatch-policy>
</weblogic-enterprise-bean>
In your deployment descriptors, you reference one of the work managers, request classes, or
constraints by its name.
An enterprise application (EAR) cannot be directly associated with a work manager, although
it can define its own application-scoped work managers. Instead, individual modules within
the enterprise application can reference global or application-scoped work managers.
For web or web-service applications, use the wl-dispatch-policy element to assign the
web application to a configured work manager by identifying the work manager name. This
web application–level parameter can be overridden at the individual servlet or JSP level by
using the per-servlet-dispatch-policy element.
For EJB applications, use the dispatch-policy element to assign individual EJB
components to specific work managers. If no dispatch-policy is specified, or if the
specified dispatch-policy refers to a nonexistent work manager, the server’s default work
manager is used instead.

Monitoring a Server Thread Pool
Print the current stack

trace for each thread.
View current thread

pool statistics.
Select a server and click Monitoring > Threads. The first table provides general information
about the status of the thread pool. The second table provides information about individual
threads. The available columns in the first table include:
• Execute Thread Total Count: The total number of threads in the pool
• Execute Thread Idle Count: The number of idle threads in the pool. This count does
not include standby threads and stuck threads. The count indicates threads that are
ready to pick up new work when it arrives.
• Pending User Request Count: The number of pending user requests in the priority
queue. The priority queue contains requests from internal subsystems and users. This is
just the count of all user requests.
• Hogging Thread Count: Returns the threads that are being hogged by a request right
now. These threads will either be declared as stuck after the configured timeout or will
return to the pool before that. The self-tuning mechanism will backfill if necessary
• Throughput: The mean number of requests completed per second
To display the current Java stack trace for active threads, click the Dump Thread Stacks
button.

Monitoring Individual Server Threads
Current status and

statistics for each thread
The second table on the server’s Monitoring > Threads tab provides the status and statistics
for individual threads, including:
• Total Requests: The number of requests that have been processed by the thread
• Current Request: A string representation of the request this thread is currently
processing
• Transaction: The XA transaction that the execute thread is currently working on behalf
of
• User: The name associated with this thread
• Idle: Returns the value “true” if the execute thread has no work assigned to it
• Stuck: Returns “true” if the execute thread is being hogged by a request for much more
than the normal execution time as observed by the scheduler automatically. If this
thread is still busy after the stuck thread max time, it is declared as stuck.

Server Monitoring: WLST Examples
Monitor the entire thread pool:
serverRuntime()
pool = getMBean('/ThreadPoolRuntime/ThreadPoolRuntime')
print 'Total Count: ' , pool.getExecuteThreadTotalCount()
print 'Idle Count: ' , pool.getExecuteThreadIdleCount()
Monitor individual work managers:
serverRuntime()
wm = getMBean('/WorkManagerRuntimes/weblogic.kernel.Default')
print 'Default WM Pending: ' , wm.getPendingRequests()
print 'Default WM Stuck Count: ' , wm.getStuckThreadCount()
wm = getMBean('/WorkManagerRuntimes/HighPriorityWM')
print 'Custom WM Pending: ' , wm.getPendingRequests()
print 'Custom WM Stuck Count: ' , wm.getStuckThreadCount()

• ThreadPoolRuntimeMBean
• ExecuteQueueRuntimeMBean
• WorkManagerRuntimeMBean
• RequestClassRuntimeMBean (a child of WorkManagerRuntimeMBean)
• MaxThreadsConstraintRuntimeMBean (a child of WorkManagerRuntimeMBean)
• MinThreadsConstraintRuntimeMBean (a child of WorkManagerRuntimeMBean)

Server WLDF Image Contents
Diagnostic images include the following files:

• JVM.img contains a thread dump.
• WORK_MANAGER.img contains basic statistics for the
thread pool and each work manager.
WORK_MANAGER.img:
Total thread count : 27

Idle thread count : 5
Standby thread count : 3
Queue depth : 5
...
--- WorkManager HighPriorityWM for app HRApp, module HRWeb
Requests accepted : 2142
Requests started : 2142
Requests Completed : 2111
A diagnostic image is a heavyweight artifact meant to serve as a server-level state dump for
the purpose of diagnosing significant failures. It enables you to capture a significant amount of
important data in a structured format and then to provide that data to support personnel for
analysis. Because it is an artifact intended primarily for internal consumption, the image
contents are not documented in detail and are subject to change.

Java Deadlock Concepts
• Sections of Java code can:

– Be marked as “synchronized,” meaning that only a single
thread can execute it at a time (Others must wait.)
– Designate an object to use as a lock
• Bottlenecks occur when multiple threads are waiting on the
same lock.
• Deadlocks occur when a thread is waiting on a lock held
by another thread, and vice versa.
Waiting on
Lock B
Lock A
Waiting on
An inadvertent deadlock in the application code can cause a server to hang. For example,
consider a situation in which thread1 is waiting for resource1 and is holding a lock on
resource2, while thread2 needs resource2 and is holding the lock on resource1. Neither
thread can progress. No new work can be introduced because the allocated number of
threads quickly becomes exhausted.
Fundamentally, this problem happens because the design and implementation of the
application have introduced the possibility of deadlocks. These types of problems may only
show up under heavy load. Therefore, these applications often pass through QA testing and
become problems in production.
A special deadlock case can occur in distributed systems. Consider the following: An
application running within a WebLogic instance invokes a service on another remote
WebLogic instance. The remote service then makes a call back to the first server. This sets
up the opportunity for a deadlock on the first server (especially under heavy load). The first
server has an execution thread that is tied up waiting for an inbound response. This inbound
response will require a thread from the same execute pool as the thread that is waiting to
receive the response. If the first server is faster than the remote server, eventually all the
threads in the execute pool will be exhausted by the server making outbound requests, with
fewer threads available for processing inbound responses.

Thread Analysis
Use thread dumps or JVM tools (such as jstack) to help

identify potential bottlenecks or deadlocks.
Waiting for a
[ACTIVE] ExecuteThread: '3' ... lock owned by
waiting for lock ExpenseItem@564290 BLOCKED another thread
com.mycompany.PayrollService.update ...
...

waiting for lock PayrollRecord@564290 BLOCKED
com.mycompany.ExpenseService.update ...
Investigate the thread dump to gain an understanding of what the threads were doing when
the server appeared to hang. If no consistent pattern emerges other than that all the threads
are busy, it is likely that your server does not have enough threads to perform the required
work. If there are not 400 threads in use, check whether the application is using a work
manager with a constraint that is artificially limiting the number of threads available to it.
Applications can sometimes appear to hang, or at least run very slowly, because there is
excessive use of Java object synchronization. The synchronization may be in place for
legitimate reasons but can happen for other less appropriate reasons. An application thread
may incorrectly be sharing a resource (for example, a JDBC connection) with another thread
due to incorrect programming. This often results in issues when the two threads both try to
enter synchronized code segments.
On occasion, developers may use synchronization to work around a multithreading issue.
Ultimately, they would like to solve the underlying issue, but using synchronization may
enable the application to pass testing and even reach production. Unfortunately, this kind of
workaround often results in a slow-running application when placed under production-level
loads.

Lock Chains
• JVMs also display lock chains at the end of a thread dump.

• Open chains identify dependencies between threads but
not necessarily a problem.
• Chains flagged as blocked or deadlocked warrant further
investigation.
...
Blocked lock chains
===================
Chain 4:
"ExecuteThread: '1' ... waiting for ExpenseItem@564290 held
by:
"ExecuteThread: '2' ... in chain 3
...
Several threads can be tied up in lock chains. Threads A and B form a lock chain if thread A
holds a lock that thread B is trying to take. The JRockit JVM analyzes its threads and
calculates the lock chains. There are three possible types of lock chains: open, deadlocked,
and blocked.
Open lock chains represent a straight dependency—thread A is waiting for B, which is waiting
for C, and so on. If you have long open lock chains, your application may have poor
performance but the stability of the system should be affected. In these cases, you may want
to reconsider how locks are used for synchronization purposes in your application.
A deadlocked (or circular) lock chain consists of a chain of threads in which the first thread in
the chain is waiting for the last thread in the chain. In the simplest case, thread A is waiting for
thread B, while thread B is waiting for thread A. Note that a deadlocked chain has no head. In
thread dumps, the JRockit JVM selects an arbitrary thread to display as the first thread in the
chain. Deadlocks can never be resolved, and the application is stuck waiting indefinitely.
A blocked lock chain is made up of a lock chain whose head thread is also part of another
lock chain, which can be either open or deadlocked. For example, if thread A is waiting for
thread B, thread B is waiting for thread A, and thread C is waiting for thread A, then thread A
and B form a deadlocked lock chain, while thread C and thread A form a blocked lock chain.

Stuck Thread Detection
• Threads running for a long time are likely deadlocked or in

an infinite loop, or they indicate an overloaded server.
• WebLogic Server:
– Periodically checks how long threads have been running
– Logs a warning if a thread becomes stuck
– Creates additional threads to handle new requests
– Sets its state to Failed after a specified number of threads
become stuck
– Can automatically shut itself down when in the Failed state
• Individual application work managers can also be
configured to detect stuck threads and reject new
requests.
WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set
period of time. You can tune a server’s thread detection behavior by changing the length of
time before a thread is diagnosed as stuck, and by changing the frequency with which the
server checks for stuck threads.
If all application threads (or a configured percentage of them) are stuck, a server instance
marks itself Failed and, if configured to do so, exits. You can configure Node Manager or a
third-party high-availability solution to restart the server instance for automatic failure
recovery.
The WorkManagerShutdownTriggerMBean configures the conditions under which an
associated work manager is automatically shut down. The trigger specifies the number of
threads that need to be stuck for a certain amount of time. A shutdown work manager refuses
new work but attempts to complete pending work. There is currently no interface in the
Administration Console for configuring this MBean type for global work managers. For
application-scoped work managers, use the <work-manager-shutdown-trigger>
element, which is a child element of <work-manager>.

Overload Protection
• An overloaded server can become unresponsive and even

unstable.
• WLS offers overload protection actions in response to:
– The number of stuck threads
– An out-of-memory JVM error
• JRockit also supports an -XXexitOnOutOfMemory
argument.
WebLogic Server has features for detecting, avoiding, and recovering from overload
conditions. WebLogic Server’s overload protection features help prevent the negative
consequences—degraded application performance and stability—that can result from
continuing to accept requests when the system capacity is reached.
You can define a maximum size of the execute queue used to accept requests. Beyond this
value, the server will refuse all web application requests and lower priority EJB requests. If the
overload condition continues to persist, higher priority requests will start getting rejected as
well, with the exception of JMS and transaction-related requests, for which overload
management is provided by the JMS and the transaction manager.
Administration network channels are an exception, however. Administration channels allow
server access only to administrative users. To ensure that overload conditions do not prevent
administrator access to the system, the limit you set on the execute queue length does not
affect administration channel requests.
A work manager can also specify the maximum requests of a particular request class that can
be queued. When both global and work manager maximum values are set, the limit that is
reached first is honored.

Configuring Overload Protection
Suspend or shut down

server if failed?
Shut down server if

out of memory?
Stuck thread
after 10 minutes
When to set server

to Failed?
Select a server and locate its Configuration > Overload tab:

• Shared Capacity For Work Managers: Total number of requests that can be present in
the server. This includes requests that are in the queue and awaiting execution as well
as those under execution. The server performs a differentiated denial of service on
reaching the shared capacity. A request with higher priority will be accepted in place of a
lower-priority request already in the queue. The lower-priority request is kept waiting in
the queue until all high-priority requests are executed. Additional lower-priority requests
are rejected immediately.
• Failure Action: Enable automatic shutdown or suspension of the server on a failed
state. The server self-health monitoring detects fatal failures and marks the server as
Failed.
• Panic Action: Enable automatic shutdown of the server when it encounters a panic
condition like an unhandled memory error. These errors can lead to an inconsistent
state; a server restart is advisable.
• Max Stuck Thread Time: The number of seconds that a thread must be continually
working before this server considers the thread stuck
• Stuck Thread Count: The number of stuck threads after which the server is transitioned
into Failed state

Section Summary

• Troubleshoot boot identity errors
• Confirm the installation and use of WLS performance
packs
• Describe the WLS threading and work manager
architecture
• Define a simple custom work manager
• Interpret thread state and lock information
• Monitor WLS threads by using the console or WLST
• Configure server overload protection

Practice 8-1
Investigating Server Problems
• Managing server overload conditions
• Using WLST and the console to monitor server threads
• Identifying potential deadlocks in thread dumps
• Limiting server resource consumption by using work
managers

Road Map
• Server Diagnostics
• Application Diagnostics
– Deployment Errors
– Shared Library Errors
– Error Pages
– Application Monitoring

WLS Deployment: Review
• Java EE applications can be deployed archived or as a

directory (“exploded”).
• Applications are given administrative names and can be in
one of several states.
State Description
INSTALLED Registered with the domain (Administration Server) as an available
application
PREPARED Latest files are distributed to target servers using one of these
approaches:
• Uploaded by the Administration Server (“staged”)
• Manually copied
• Use shared network storage
ADMIN Running on target servers, but accessible only from the admin
network channel (if configured)
ACTIVATED Running on target servers
A deployment unit refers to a Java EE application (an Enterprise Application or web

application) or a stand-alone Java EE module (such as an EJB or Resource Adapter) that has
been organized according to the Java EE specification and can be deployed to WebLogic
Server.
Installing an application refers to making its physical file or directory known to WebLogic
Server. An application can be installed as an archived EAR file or as an exploded directory.
After you have installed the Enterprise application, you can start it so that users can begin
using it.
When you first deploy an application or stand-alone module to one or more WebLogic Server
instances, you specify a deployment name to describe collectively the deployment files, target
servers, and other configuration options you selected. You can later redeploy or stop the
deployment unit on all target servers by simply using the deployment name. The deployment
name saves you the trouble of re-identifying the deployment files and target servers when you
want to work with the deployment unit across servers in a domain.

When you update an application, you can specify that WebLogic Server redeploy the original
archive file or exploded directory, or you can specify that WebLogic Server deploy a new
archive file in place of the original one. You can also change the deployment plan that is
associated with the application. Update an application if you have made changes to the
application and you want to make the changes available to WebLogic Server clients, if you
have made changes to the deployment plan, or if you want to redeploy an entirely new
archive file in a new location.

Deployment Errors
Common causes of deployment issues include the following:

• Applications files do not have correct OS permissions.
• Target servers are not in compatible states.
• Required deployment descriptors are missing or invalid.
• Deployment plan is invalid.
• Application name is invalid.
• Local or network storage is unavailable.
• WLS user is not granted deployment rights.
• Domain lock is held by another user.
• Application contents are also on the system classpath.
• Application web URLs or JNDI names are already in use.
• Custom initialization code fails.
While deploying an application, make sure that the servers have read and write permissions
of the application archive or directory. Similarly, make sure that the credentials used to
execute deployment tasks meet the criteria of the global Deployer role. Otherwise, you can
expect errors similar to:
“Access not allowed for subject: principals=[myuser, Deployers], on
ResourceType: Cluster Action: execute, Target: addDeployment”
If your applications includes an improperly configured deployment descriptor, this could lead
to deployment exceptions similar to:
“weblogic.management.DeploymentException: Error while loading
descriptors: Error parsing file “META-INF/application.xml”
If one or more target servers are not in a state that is compatible with accepting new
deployment tasks (Failed or Shutting Down states, for example), the task will fail. This is
particularly important when deploying to a cluster.

Application Staging
• When staging is enabled, deployments are uploaded to

each target server’s file system.
• Staged deployments are stored at
servers/<server>/stage by default, but this location is
configurable.
• Verify the contents of this location if deployment succeeds
but appears to be the wrong version.
In stage mode, the Administration Server copies the deployment files from their original
location on the Administration Server machine to the staging directories of each target server.
For example, if you deploy a Java EE application to three servers in a cluster by using stage
mode, the Administration Server copies the deployment files to directories on each of the
three server machines. Each server then deploys the Java EE application by using its local
copy of the archive files. When copying files to the staging directory, the Administration Server
creates a subdirectory with the same name as the deployment name.
Stage mode ensures that each server has a local copy of the deployment files on hand, even
if a network outage makes the Administration Server unreachable. However, if you are
deploying very large applications to multiple servers or to a cluster, the time required to copy
files to target servers can be considerable. Consider “nostage” mode to avoid the overhead of
copying large files to multiple servers.
The Administration Console uses stage mode as the default mode when deploying to more
than one WebLogic Server instance. weblogic.Deployer uses the target server’s staging
mode as the default, and managed servers use stage mode by default.

Deployment Memory Errors
• JVMs utilize specific heap locations for permanent data

such as class definitions.
• Deploying very large applications can result in out-of-
memory errors on the client or server JVM.
<Initiating deploy operation

for application, HRApp.ear [archive: /home/user/HRApp.ear],
to ServerC .>
java.lang.OutOfMemoryError: PermGen space
Failed to deploy the application with status failed
java ... –XX:MaxPermSize=128M weblogic.WSLT deployApp.py
Update JVM memory

for WLST client.
If you encounter an out-of-memory exception when you deploy or undeploy an application, try
increasing the permanent generation size in the JVM heap. For example, on Sun use the
command-line argument -XX:MaxPermSize=<size>. This issue tends to be less common
on the JRockit JVM.

Shared Library: Review
A Java EE shared library:

• Is a reusable portion of a web or enterprise application
• Is referenced by other deployed applications
• Avoids duplicating source files among Java EE projects
• Can contain deployment descriptors that are merged with
the application’s descriptors
WebLibraryA
page1.jsp MyWebApp
MyWebApp
lib1.jar page1.jsp
page1.jsp page2.jsp
web.xml DEPLOY
page2.jsp page3.jsp
WebLibraryB page3.jsp lib1.jar
page2.jsp web.xml Overrides
web.xml (merged)
web.xml file in library
At the web application level, a shared library is a WAR file that can include servlets, JSPs,
and tag libraries. Shared libraries can be included in an application by reference, and multiple
applications can reference a single shared library.
You can deploy as many shared libraries to Oracle WebLogic Server as you require. In turn,
libraries can reference other libraries, and so on. Because the shared library code and your
own application code are assembled at run time, rules must exist to resolve potential conflicts.
The following are the rules:
• Any file that is located in your application takes precedence over a file that is in a shared
library.
• Conflicts arising between referenced libraries are resolved based on the order in which
the libraries are specified in the META-INF/weblogic-application.xml file (for
enterprise applications) or the WEB-INF/weblogic.xml file (for web applications).
Oracle WebLogic Server supports versioning of the shared Java EE libraries, so that the
referencing applications can specify a required minimum version of the library to use or an
exact, required version. The specification version identifies the version number of the
specification (for example, the Java EE specification version) to which a shared Java EE
library or optional package conforms. The implementation version identifies the version
number of the actual code.

Library Errors
• A library typically includes a MANIFEST file to indicate its

version numbers.
• Application descriptors specify some combination of the
following:
– List of required libraries
– Required or minimum version of a library
• Deployment fails if this criterion is not met. Did not find a
required library
weblogic.management.DeploymentException:
Error while processing library references. Unresolved
application library references, defined in weblogic-
application.xml: [Extension-Name: hrcommon, Specification-
Version: 3.2, exact-match: true] ...
Required name
and version
As a best practice, you should always include version information (an implementation version,
or both an implementation and specification version) when creating shared Java EE libraries.
Creating and updating version information as you develop shared components allows you to
deploy multiple versions of those components simultaneously for testing. For example, a
production application may require a specific version of a library, because only that library has
been fully approved for production use. An internal application may be configured to always
use a minimum version of the same library. Applications that require no specific version can
be configured to use the latest version of the library. The name and version information for a
shared Java EE library are specified in the META-INF/MANIFEST.MF file, as in the following
example:
Extension-Name: myExtension
Specification-Version: 2.0
Implementation-Version: 9.0.0
After you deploy one or more shared libraries, you can then deploy applications and modules
that reference these libraries. Successfully deploying a referencing application requires that
two conditions are met:
• All referenced libraries are registered on the application’s target servers.
• Registered libraries meet the version requirements of the referencing application.

Deployment Debug Flags
• Servers produce a set of informational messages during a

deployment task.
• Use the DebugDeployment flag for additional details.
Standard deployment server log messages:
<Info> <Deployer> <BEA-149060> <Module Payroll of application

HRApp successfully transitioned from STATE_ADMIN to
STATE_ACTIVE on server ServerC.>
Server log messages with DebugDeployment enabled:
<Debug> <Deployment> <BEA-000000> <Creating target server

deploy operation 'deploy' for '[0]HRApp' initiated by
'principals=[myuser, Administrators]> ...
<Debug> <Deployment> <BEA-000000> <createDataUpdate - staging
location: /home/user/HRApp.ear>
Because deployment events are particularly important to application servers, WebLogic

Server automatically logs informational messages for any deployment task as well as for all
application state transitions. However, if more detailed information is needed regarding the
deployment process, several debug flags are available. For example, you may want to know
the user who initiated a deployment task or the staging location that was used.

Application Error Handling
• Application errors may not be logged if they are

intercepted and handled by the application:
– Programmatically
– Through deployment descriptor settings
• Web applications can display custom pages in response to
Java exceptions or HTTP errors by using web.xml.
<error-page>
<exception-type>java.io.IOException</exception-type>
<location>/pages/misc/ioerror.jsp</location>
</error-page>
<error-page>
<error-code>500</error-code>
<location>/pages/misc/generalerror.htm</location>
</error-page>
You can configure WebLogic Server to respond with your own custom web pages or other
HTTP resources when particular HTTP errors or Java exceptions occur, instead of
responding with the standard WebLogic Server error response pages.
You define custom error pages in the <error-page> element of the Java EE standard web
application deployment descriptor, web.xml. (The web.xml file is located in the WEB-INF
directory of your web application.)

Application Monitoring: Review
WLS collects various runtime application statistics, including:

• Number and size of HTTP sessions
• Servlet or web service execution times
• EJB instance pool size and utilization
• EJB/JPA cache size and utilization
• EJB transaction counts
Monitor a web
application.
In the left pane of the Administration Console, click Deployments. In the right pane, click the
application that you want to monitor, and then click the Monitoring tab. The available second
level tabs will vary depending on the type of the application. For example, a web application
or module has tabs named web Application, Servlets, Sessions, and Workload. The available
columns in these tabs include:
• Servlets: The number of Java Servlets that are deployed within this application,
including the internal WebLogic Servlets. If required, the Servlets tab displays statistics
on a per-Servlet basis.
• Sessions: A count of the current number of open sessions in this module
• Sessions High: The highest number of active sessions ever managed by this
application. The count starts at zero each time the application is activated.
• Total Sessions: A count of the total number of sessions opened since the application
was deployed

Application Monitoring: WLST Examples
Monitor a web module within an enterprise application:
serverRuntime()
web = getMBean('/ApplicationRuntimes/HRApp/ComponentRuntimes/
ServerC_/payroll')
print 'Sessions: ', web.getOpenSessionsCurrentCount()
Monitor an EJB module within an enterprise application:
serverRuntime()
ejb = getMBean('/ApplicationRuntimes/HRApp/ComponentRuntimes/
payroll.jar/EJBRuntimes/PayrollManager/PoolRuntime/
PayrollManager')
print 'Pool Size: ', ejb.getPooledBeansCurrentCount()
print 'In Use: ', ejb.getBeansInUseCount()
print 'Waiting: ', ejb.getWaiterCurrentCount()

• ApplicationRuntimeMBean
• ComponentRuntimeMBean (a child of ApplicationRuntimeMBean)
• StatelessEJBRuntimeMBean (a child of ComponentRuntimeMBean)
• StatefulEJBRuntimeMBean (a child of ComponentRuntimeMBean)
• EntityEJBRuntimeMBean (a child of ComponentRuntimeMBean)
• MessageDrivenEJBRuntimeMBean (a child of ComponentRuntimeMBean)
• EJBPoolRuntimeMBean (a child of StatelessEJBRuntimeMBean and
MessageDrivenEJBRuntimeMBean)
• EJBCacheRuntimeMBean (a child of StatefulEJBRuntimeMBean and
EntityEJBRuntimeMBean)
• EJBTransactionRuntimeMBean (a child of all EJBRuntimeMBeans)
• KodoPersistenceUnitRuntimeMBean (a child of any ComponentRuntimeMBean)
• WseeRuntimeMBean (a child of ApplicationRuntimeMBean)

Section Summary

• Describe the life cycle of a WLS application
• List some common deployment issues
• Troubleshoot library dependency issues
• Monitor running applications by using the console or WLST

Quiz
Which of these is not a type of WLS thread?

a. Service Producer
b. Socket Reader
c. Execute
Answer: a

Quiz
A work manager can be associated with a ____. (Choose two

of the following responses.)
a. Standby
b. Request Class
c. Staging Mode
d. Constraint
Answer: b, d

Quiz
Which of the following would not typically cause a deployment

failure?
a. Missing deployment descriptor
b. Performance pack disabled
c. Wrong library version
d. Lock held by another user
e. Web URI already in use
Answer: b

Summary

• Investigate common server startup issues
• Discuss the use of WLS work managers
• Analyze server thread dumps and application deadlocks
• Investigate common deployment issues
• Monitor running applications

Weblogic Diag Sg1

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Weblogic Diag Sg1

Caricato da

Copyright:

Formati disponibili

Oracle WebLogic Server 11g:

Diagnostics and Troubleshooting

Volume I • Student Guide

Bill Bell Disclaimer

This document contains proprietary y information and is protected by

4 Diagnostic Framework Essentials

7 Troubleshooting Java Applications

12 Troubleshooting Node Manager

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

After completing this course, you should be able to:

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 2

• This course is intended for experienced WebLogic Server

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 3

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 4

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 5

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 6

Appendix A: WLS SNMP

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 7

• Each topic is reinforced with a hands-on exercise.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 8

• The instructor starts each session at the scheduled time.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 9

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 10

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 11

OBE is a series of free online tutorials that cover specific

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 1 - 12

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

After completing this lesson, you should be able to:

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 2 - 2

Machine Machine Machine

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

An Oracle WebLogic Server administration domain is a logically related group of Oracle

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 2 - 3

• All WLS configuration, management, and monitoring is

Admin server Managed server

JMX client JMX client

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 2 - 4

• Every WebLogic Server hosts several different MBean

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 2 - 5

• Each MBean consists of attributes and operations.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

As an alternative to using WebLogic Scripting Tool (WLST) to interactively browse a server’s

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 2 - 6

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 2 - 7

View console Help to determine

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 2 - 8

• WLST provides a command-line and scripting interface for:

serverRuntime() Access this server's run-time hierarchy.