Monitor, Diagnose, and Optimize Java Workloads on

Premises and in the Cloud

Oracle Openworld, 2015

Agenda

The company

Our platform

JVMD on EM 12

JVMD on EM13 - Expectations

Therap Services

Integrated documentation and communication services for the Developmental

Disability industry

Private and Government customers

De-facto National Leader

SaaS, 24x7 operation

1M+ lines of java code, Spring & JEE

100+ application modules, 20+ war in an ear

Heavy use of JMS, Coherence, RMI

Stats

Very efficient in resource usage

~250K active and heavy users, ~10K concurrent user (up from 8.5K
6 months ago)

Translates to 46K+ peak requests/minute, ~2.2M requests/hour

600,000 application page views/hour

Application Lifecycle

3 major releases/year, minor/bug-fix releases every few weeks

Multiple testing environments

2 identical sites each with

8 WebLogic instance (4 Hosts) - 12.1.2.0.0

1 in-memory DB (in-house)

1 Oracle DB - 12c (moved six months ago), Goldengate replication

Our EM Deployment

EM: 12.1.0.4.0

One EM instance per production data center (2), one for the dev/
staging environment.

Our system team owns it. Used by system team, operations, DBA

Tries to move very to the latest version fairly quickly. Preferably

within months but depend on product release schedule.

JVMD Use Case at Therap

Primarily for Application Performance Management (APM)

Proactively fix bottlenecks before they actually becomes problem

Diagnosing incidents

Right now primarily for production, limited high value use for big
infrastructure change (WL upgrade etc.)

Want to increase usage during dev cycles but it needs to get easier
to use (13.1 looks promising)

What it provides

Active thread by state

Thread transitions, automatic snapshots and detail informations in

snapshots (request, ECID, sql, stacktrace)

Top requests, methods, sql etc.

All sorts of weblogic, JVM metrics that we get access to

Log4j > Logback

Log4j sync logging had concurrency issue

This was limiting request/second despite low cpu

The undiscovered bottleneck was causing request piling up and

crashing the server

We moved to logback which had better concurrency and then we

moved to async logging to make it even more scalable

JVMD 13.1 - Better Data Exploration

The UI is much faster and intuitive

Allows us to slice and dice the data from many different angle

Allows us to focus on specific time (remembers it)

Allows us to create a focus workset (jvm, request, session etc.) and

look at if from different perspective

JVMD 13.1 - Better Information

Gives us a lot more information (not just top 5 like before)

Gives us information in a more statistically relevant fashion (max,

avg, count)

Allocation, DB information

Allocation Information

Allocation information at a granular level is the most difficult information to

find

Its extremely useful and essential for the cases where you need it

You can take pro-active measurement based on allocation profile because

bad allocation profile often suggest design problems

Its very hard to guess whats causing allocation and such making a very lean
application

e.g. - At one point of reducing allocation I had to change int to Integer

since the casting for map lookup was a big percentage of allocation

SQL to Application Code

You can look at the top sql and can find the stack trace

Very important for us since we have over 100 modules and 4000
queries in the system, making it sometime very hard to figure out
where a particular query comes from

Its doubly important for generic queries where we cant get the bind
variables

JFR Integration

There is nothing that is beating JFR for the absolute low level metrics

JVMD is the only tool that I know that can integrate with JFR

Thank You!
Tahseen Mohammad
Principal Software Engineer, Therap Services
tahseen ~AT~ therapservices.net