Tools and Tips to

Diagnose Performance
Issues
Claudio Miranda
Summa Technologies
3320

Objective

Learn some tips and tools to better diagnose

and solve the most common performance
issues of Java applications.
The main focus going on to diagnose
memory and CPU problems.

Disclaimer

The presented tools are based on the

author's experience.
These arguments can be applied to
development and production systems

Disclaimer 2

There are plenty of others free and

commercial tools available, some better
suited to specific tasks, other tools can be
easy or complex to use, it all depends of:
Time
Money

AGENDA
>
>
>
>

Where it all begins

Tools and tips to diagnose problems
Application and system performance
General tips

AGENDA
>
>
>
>

Where it all begins

Tools and tips to diagnose problems
Application and system performance
General tips

Where it all begins

> No performance requirements at project level
> Traps
The application is too slow
Speed it up at all costs
How much to tune ?
It is necessary to have performance numbers to work with
> Typical numbers
Tx/s (Average transactions per second)
Concurrent users
Number of sessions

Where it all begins

> What is a transaction ?
Database
Messaging
Connector
Filesystem
Is two phase commit required ?
> Call it a business transaction
Example: 2 DB inserts + 1 messaging
Identify the hot spots

Where it all begins

> Where is the problem ?
Memory
I/O
Networking
CPU
>

10

Where it all begins

> Personal diagnose and solve steps
Analysis => What is the matter ?
Priority => Is the application down or slow ?
Diagnose => Where is the problem ?
Solution => How to solve it ?
Sledgehammer way
Best practices
Tests => Make sure it will not happen again
Documentation => To remember what happened

11

AGENDA
>
>
>
>

Where it all begins

Tools and tips to diagnose problems
Application and system performance
General tips

12

Tools and Tips

> Memory issues
Memory Leak
OutOfMemoryError (OOME)
> CPU issues
High CPU usage
Threads
> Garbage Collector

13

Tools and Tips

Each tool and tip has different level of intrusion
on the overall system.
It ranges from simple monitoring (with ready
to use tools) to profiling and debugging.
That is one key point, to consider which
approach to use, to diagnose a problem

14

Tools and Tips

> Monitoring
Non intrusive
No pre-configuration
Monitor production systems
Easy to use
> Profiling and Debugging
Easily shows where is the problem
Nice graphics
Needs to restart system
Slowdown system runtime
Additional settings
Can distort results

15

Tools and Tips

Diagnose Memory Issues

16

Tools and Tips - Memory

> Required steps
How much physical memory is available
How many programs are using it
Who is the memory hunger ?
What are the VM parameters ?
Application did entire life cycle to prevent new classes being loaded
Initial load time

17

Tools and Tips Monitoring Memory

> How much Physical memory
> Who is the memory hunger

> Monitoring: top

RES = Physical memory

18

Tools and Tips Monitoring Memory

Java heap space
> Monitoring:
jmap
> GC Algorithm
> Heap size

19

Tools and Tips Monitoring Memory

Java heap space
> Monitoring:
jstat
> GC Activity

20

Tools and Tips Monitoring Memory

Each GC releases less memory
than before
Memory Leak indicator

21

Tools and Tips - Memory

> Track down where is the problem
Heap dump (non intrusive)
Generate a dump file as large as the heap memory
Freezes application while dump is generated
Profilers (intrusive)
Need to reconfigure VM settings
Attach a client profiler to introspect

22

Tools and Tips - Memory

> Heap dump
> jmap -dump:live,format=b,file=dump23855_appserv1 23855
> -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=heapdump.hprof

23

Tools and Tips - Memory

> Heap dump and profilers
> Dump Analyzers and Profilers
IBM Heap Dump Analyzer
Eclipse Memory Analyzer
NetBeans Profiler

24

Tools and Tips - Memory

> IBM Heap Analyzer

25

Tools and Tips - Memory

> IBM Heap Analyzer

26

Tools and Tips - Memory

> Eclipse Memory Analyzer

27

Tools and Tips - Memory

> NetBeans Profiler

28

Tools and Tips - Memory

> OutOfMemoryError
Look at log files
Search for hprof, hs_err or core files
Monitor memory usage
prstat, top, ps
Monitor GC activity
jstat, -verbose:gc, visualgc,

jconsole,

29

Tools and Tips - Memory

> OutOfMemoryError types
Java heap space
PermGen space
Out of swap space
unable to create new native thread
Requested array size exceeds VM limit
GC overhead limit exceeded
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at ConsumeHeap$BigObject.<init>(ConsumeHeap.java:22)
at ConsumeHeap.main(ConsumeHeap.java:41)

30

Tools and Tips - Memory

Java heap space
> Causes
Low RAM memory
Small heap
Memory leak
> Fix
More physical memory
Make heap larger
Fix memory leaks

31

Tools and Tips - Memory

PermGen space
> Causes
Low memory for permanent generation area
Classes on permanent generation are prevented to be collected
Classloader leak
> Fix
Make permanent generation area, larger
Disable hot deployment or class reloading for classloader leak

32

Tools and Tips - Memory

Out of swap space
> Cause
Often is caused by JNI code
No memory available to operating system
> Fix
Reserve more memory to operating system
Lower the heap size

33

Tools and Tips - Memory

Unable to create new native thread
> Cause
Big stack size (-Xss)
Small memory available
Poor thread management
> Fix
Make stack size smaller
Reserve more memory to operating system
Lower heap size
Better thread management
Java 5 thread pool

34

Tools and Tips - Memory

Requested array size exceeds VM limit
> Cause
Attempt to allocate an array that is larger than the heap size
> Fix
Lower the array size
Make heap larger

35

Tools and Tips - Memory

GC overhead limit exceeded
> Cause
Too much time spent on GC (98%) and little heap (2%) to be free
> Fix
Similar to java heap space
Tune GC behavior

36

Tools and Tips - Memory

> VM Crashes
Search for hprof, hs_err or core files
> Enable post morten diagnostics
gcore -o <file> PID
-XX:+HeapDumpOnOutOfMemoryError
-XX:OnError
> Generate thread and heap dumps to analyze it later

37

Tools and Tips - CPU

Diagnosing CPU Issues

38

Tools and Tips - CPU

> OS diagnostic tool
prstat, ps, top
> Thread behavior
Thread dump, thread analyzer
> GC Activity (again)
Monitor heap
> Profiling

39

Tools and Tips - CPU

> Thread dump and analyzer

Everybody waiting
this guy !

40

Tools and Tips - CPU

> Thread dump and analyzer

41

Tools and Tips - CPU

> Thread dump and analyzer

42

Tools and Tips - CPU

> Thread dump and analyzer

43

Tools and Tips - CPU

> Profiler
> CPU intensive task

44

Tools and Tips - CPU

> Profiler
> CPU intensive task, the caller class

45

Tools and Tips - CPU

GC Activity
Too much GC in short period of
time, degrades performance
> Fix
Adjust GC
Make heap larger

46

Tools and Tips More tools

More Tools to help

diagnose issues...
...or to mess around
your toolbox

47

Tools and Tips More tools

> Dtrace (Solaris, FreeBSD, Mac OSX)
Examine program behavior by reading points of instrumentation on java
and solaris.
> Sun Studio Tools (Solaris and Linux)
Graphical tools to help diagnose and debug memory and cpu issues
> Profilers: Jprobe, Optimize It, YourKit, (commercial)
> BEA JRockit Memory Leak Detector
> JBoss Profiler (collect first, analyze later)
> JXInsight (commercial - free dev. edition)
Monitor, diagnose and analysis of java systems
>

48

Tools and Tips More tools

> CPU and Memory statistics
dstat, prstat, vmstat, mpstat
> Networking
Wireshark (AKA ethereal), tcpdump, snoop, dig, netstat,
> System call tracing
truss, strace, dtrace
> Threading
pstack
> Process
lsof, ps,

49

AGENDA
>
>
>
>

Where it all begins

Tools and tips to diagnose problems
Application and system performance
General tips

50

Application and System Performance

Some general tips

to improve overall
performance

51

Application and System Performance

> Application Servers
Properly configure pool resources
Threads pools are used everywhere
Connection pools for Http, JDBC, JMS, JCA, JMX, EJBs

Monitor TCP connections and blocking connections to have a

general view to tune those parameters

Many threads waiting on a single monitor, adjust the pool

Http services
Enable resource cache
Configure timeout setting for http, database, session
Enable keep-alive
Put the data where it belongs

Static data on webservers

Dynamic data on appserver

Disable DNS lookup

52

Application and System Performance

> Application Servers
Clustering
When session (HttpSession and SFSB) must synchronize themselves ?
Application must be designed to work under cluster
Make sure there are no-firewall between nodes
Hardware load balancers are faster
Large session size, can bring down the cluster
Memory replication is better
Classloader and Dynamic loading
Disable redeploy, autodeploy, auto discovery and JSP compilation

53

Application and System Performance

> Java Concurrent Programming
Use thread pools
Lower VM stack size (-Xss)
Use util.concurrent classes
Collections
Atomic operations
Framework for multithread system
Synchronize the smallest possible code
Parallelize tasks on SMP systems
Be careful, context switching can be expensive

54

Application and System Performance

> Java Runtime
Disable security manager at trusted sites
Disable class verifier (-Xnoverify)
Garbage Collector
Use Parallel GC when possible
Disable verbose:gc, prefer jstat

32

32 bit systems have address space restrictions of 2

Cannot configure -Xmx4g
If possible don't enable 64 bits (-d64)

bytes

55

Application and System Performance

> Database
Have a database expert
For report systems
Configure a specific connection pool
Set a less restrictive transaction isolation level
Data cache
A cleaning policy is highly recommended
Don't use XA protocol, if possible
Use stored procedures for:
Data transformation between datasets
For large datasets
Pure JDBC for reporting or complex queries
Create indexes
Use optimistic concurrency scheme

56

Application and System Performance

> I/O System
Get rid of all System.out
Use if (log.isDebugEnabled()) before logging statements, to
prevent string concatenation
Prefer java.io.Externalizable over standard serialization
Use buffers on I/O operations
On NIO systems
Be careful of allocation outside heap, use -XX:MaxDirectMemorySize
Decide which type of allocation to use
Consider FileChannel for copy or read operations

57

Application and System Performance

> General Java Programming
Use Weak and Soft references
Bad hashCode and equals implementation can make your cache crazy
> XML
Prefer a binary formato for communication protocol
> Log management
Log levels for production and development systems
No debug messages for production systems, please
Don't log method names or package abbreviation

58

AGENDA
>
>
>
>

Where it all begins

Tools and tips to diagnose problems
Application and system performance
General tips

59

General Tips

Why is the top management

upset at stacktraces
on the server ?

60

General Tips
> Don't panic yourself into the crazy bandwagon
> Try to have a good view of the overall computing system
Infrastructure, operating system, daemons, firewalls, application servers,
configurations, what was working before
> Be careful to trust the information sources
> Don't prematurely tell the customer about the solution
> Critical understanding of the affected services and importance to the customer
> Get to know teams and respective leaders/managers
> Confirm reports told by other people
> Verify system settings

61

General Tips
>
>
>
>
>

Understand how to use the tools and its output

Don't depend on graphical consoles
Have access to the application's source code
Have a good decompiler (if the previous one fails)
When notifying the customer about the issue, bring him the fix
And have a happy customer

62

Questions ?

Claudio Miranda

Summa Technologies

http://weblogs.java.net/blog/claudio
claudio@claudius.com.br
http://www.summa-tech.com