Session: D02

8 Years of Performance Solutions: Lessons Learned

Scott Hayes President & CEO, DBI IBM DB2 GOLD Consultant
Monday, May 8, 2006 1:00 p.m. 2:10 p.m. Platform: DB2 UDB for Linux, UNIX, and Windows

8 Years of UDB Performance Solutions: Life Lessons from the field

Planning, Goals, Service Level Agreements A Performance Achievement Methodology Critical Performance Metrics and Analysis Physical Design Solutions for GREAT Results Life Lessons from the field

Lessons Learned: Vision Required

BEGIN WITH THE END IN MIND

3

What are your goals? If you dont know where you want to be, you wont ever get there.

Travel is pointless without a destination

What are your performance tuning goals?

Avoid Hardware Upgrades / reduce CPU? Improve Transaction Response Time? , T A ND S R E D Add more users? O UN T O O D T T S FIR DERS Reduce Phone Rage? N U SEEK E B N TO E H T Service Level Agreement Requirements?
95% of transactions complete < 5 seconds? Application is Available 99.92%?
What does Available 4mean?

You need to establish defined goals otherwise youll never know whether or not you are or were successful. Understand your businesses needs, and then your business will be more receptive to changes you would like to make.

Lessons Learned : You Need a Game Plan

FAILURE TO PLAN IS PLANNING TO FAIL

5

Wisdom worth remembering.

A Performance Methodology
Modify DBM, DB, and BP Configurations as appropriate
Administration Performance Monitoring

Measure Overall Database Performance (Key Metrics)

Random % High CPU Read Costs GAME PLAN Table Rows Read/TX Update Engage DB, IBM DBM Index/ CFG High Sort Costs PLAY Sufficient Agents Alter Design Bufferpool Advisor BPNAME High Avg Times TO Elapsed WIN! BP & Cache Hit Ratios Inefficient Index Usage

SQL Analysis & Tuning

Implement Physical Design Changes

Administration & Space Management
6

Identify Costly SQL

Tuning is an iterative process. Change one thing at a time. Assess the health of the database. If the database health is poor, then identify the tables that are getting in trouble with high I/O rates, then find the statements that are the driving force behind those high I/O rates. Modify the physical design (add, change, drop indexes, use MDC, MQT/AST tables, etc) to reduce the table I/O burden. Re-measure, repeat. For more details, a free white paper on DB2 UDB LUW tuning is available at http://www.database-brothers.com/papers.php

Database Health Check

Random Read % (or Sync Read % (SRP)) =
100 - ( Data + Index Async Reads * 100 / Data + Index Total Reads ) db2 Get Snapshot for Database

> 90% 80 - 90 % 50 - 80 % < 50 %

7

These percentages are valid for OLTP databases. Details and guidelines for data warehouse databases are available in a free DB2 UDB Tuning White Paper available from http://www.database-brothers.com/papers.php Generally, DW RR% should be 10-30%.

 Compute Average Rows Read per Transaction (Commits + Rollbacks) for database and each table. Get Snapshot for Database + Tables

Table I/O Health Check

RR/TX < 10 RR/TX > 10 & < 50 RR/TX > 50 RR/TX > 100

Rank tables by most heavily read to least trouble makers are at the top!!!
8

Because not every transaction accesses every table is precisely why this guideline formula works so well. We expect RR/TX to be in the single digits.

Lessons Learned : Laser Focus

A WELL STATED PROBLEM IS A HALF SOLVED PROBLEM

9

You cannot solve a problem until you KNOW what problem you are trying to solve. Be careful and be wary you may find several symptoms easily but have to dig and dig to find root cause problems.

Workload Analysis > Well Stated Problem

80+% of Tuning Benefit comes from complete and accurate understanding of the SQL workload and its costs What is the most costly, most harmful, SQL during peak periods? Recent periods? Over time? Highest CPU Consumption Highest Sort Time Consumption Highest average Elapsed times Highest Read I/O (rows read)
Grouping & Cost Aggregation of similarly structured SQL statements is imperative to True Cost determination
10

A comprehensive statement workload cost analysis will often provide you with the required well stated problem.

10

SQL Equalization & Cost Aggregation

How DB2 Sees the SQLc5 Workload: Select c1, c2, c4 c4 from tbl tbl where where c5 = 0210 cpu=.1 Select c1, c2, from c8 = >0360 0680 0120 0220 cpu=.2 0500 8800 0300 0400 0450 0490 0670 Bob 0110 0190 0390 0790 2380 4560 cpu=.1 cpu=.3 100s of SQL statements per second SQL Snapshot shows 19 different statements! WRONG ANSWER! How the DBA needs to see the SQL Workload:
SQL Statement Select c1, c2, c4 from tbl where c5 = ? Select c1, c2, c4 from tbl where c5 > ? Select c1, c2, c4 from tbl where c8 = ? Totals:
11

Count TotCPU CPU% 10 11 12 13 14 15 16 2 3 4 5 6 7 8 9 1 1 2 1 19 .2 .3 .4 .5 .6 .7 .8 .9 .1 1.0 1.1 1.2 1.3 1.4 1.5 1.6 .3 .6 .2 2.4 66.6 25.0 8.33 100.00

SQL Equalization and Cost Aggregation is discussed in painful detail in US Patent # 6,772,411

Relative Costs

11

Solving Problems > Effective Solutions

Given a costly SQL statement, 3 possible solutions: 1) Physical Design Change (95%)
Add an Index Add/modify Cluster Index
Most potent weapon against poor application performance

Drop Ineffective/Costly Indexes

Low Cardinality, Skewed Distributions, Redundant Indexes

Generated Columns with new supporting Index MQT/AST tables

2) Tweak Catalog Statistics to fool optimizer (2%)

A temporary and difficult to maintain solution

3) Re-write/modify SQL (3%)

The DB2 Optimizer Re-writes SQL. Isnt re-writing re-written SQL redundant?
12

Given the odds, which horse would you bet on?

12

Lessons Learned : Small Stuff

SWEAT THE SMALL STUFF

13

Small drops of rain delivered over a period of 40 days and 40 nights had dramatic impact on earth so says the Bible.

13

 Small Tables
A simple SELECT executed with high frequency against a table with only 32 rows consumed 34% of ALL CPU time on an SMP 4-way Myth: Small tables dont need indexes Realities:
Explains dont identify costly SQL against small tables Explains dont consider frequency of execution Only Dynamic SQL Equalization finds high cost SQL Even ONE row tables can benefit from indexes
14

Doodle here.

14

Recipe for Success

When life hands you lemons, make lemonade

15

6 3

sugar

Use big lemons and bottled spring water then youll have gourmet lemonade.

15

Lessons Learned : SORT Costs

SORT
IS A FOUR LETTER WORD
16

Doodle here.

16

 SORT is a Four Letter Word

SELECT * from TB where C1 = ? Order By C3 put an internet company out of business by consuming 68% of CPU on RS6K 8-way! Myth: Small Sorts are inconsequential Realities:
Small Sorts consume CPU! Explains dont consider frequency of execution Only Dynamic SQL Equalization finds most costly Sorts Just *5* Clustering Indexes can often restore 50% of a machines capacity!!!
17

50% more capacity means twice as many users on the same hardware, or transaction rates going twice as fast, or CPU utilization cut so you can run other work on the machine.

17

Impact of SORTHEAP / Small Sorts Avoid Sort Overflows

256 - Small Sort Overflows 1MB SORTHEAP, 2 sorts, 1 overflows 512 - Small Sort completes within 2MB SORTHEAP, 2 sorts, 0 overflows Avoiding the overflow cut sort time by 20% (1/5th), reduced elapsed time by 10%, and sliced CPU burn by 15% 128 - No Order by clause (1 index rid sort)
CPU Sec Elapsed Sec 1.4 1.2 1 0.8 0.6 0.4 0.2 0 128 256 512 128/1 Sort Sec

For OLTP, < 3% sorts overflow

18

Sorts are evil, even small sorts. Use Clustering Indexes or MDC to eliminate and reduce sort costs. Get rid of order by or distinct if you dont need them.

18

Lessons Learned : Efficient Indexes

Index AND-ing Works, BUT Composite Indexes work better!

19

Its all about precision.

19

 Composite Indexes are Best

SELECT * from TB where C1 = ? And C2 = ? And C3 >= ? caused an SLA to be missed and service contracts nearly lost Myth: Use 3 single column indexes on C1, C2, and C3 Realities:
Index AND-ing can be CPU and I/O expensive A single composite index on columns C1, C2, & C3 is dramatically faster and more efficient The INDEX/Design Advisor favors composite indexes, but identifying the costly SQL is the trick >> SQL EQ!
20

Single column indexes are expensive to maintain and provide insufficient filtration cardinality on their own which leads to multiple index access which requires processing multiple indexes and that can be expensive!

20

Lessons Learned: Be Prepared

The Boy Scout Motto applies to asynchronous I/O but not DB2 Indexes!
21

I am an Eagle Scout. I know it is important to be prepared. Thats why I pack extra socks for every trip.

21

Index Design
Indexes with Cardinality = 1 are a performance death sentence. Do not create indexes just in case Indexes with Skewed distributions can be expensive to maintain on Insert, Update, Delete Redundant Indexes are expensive to maintain, consume disk, and provide no value to DB2 Drop them!

IX on C1, C2 <<- Redundant Index IX on C1, C2, C4

For multi-column indexes, place the column that is most frequently known (= predicate) first. Use Clustering Indexes to reduce Sort & CPU costs
22

For more helpful information on DB2 performance physical design, visit DBIs site at http://www.database-brothers.com/db2tips.php

22

Lessons Learned : Performance Drag

Parachutes are for Sky Divers and Drag Racing

23

Who else uses parachutes?

23

Closing Files = Putting the brakes on DB2

Default DB CFG MAXFILOP = 64
TOO SMALL FOR MOST DATABASES! Closing Files BURNS CPU and SLOWS SQL!

Ensure Database Files Closed = 0 via Database Snapshots

Increase MAXFILOP until Files Closed = 0
24

Closing Files continues to be an ongoing DB2 UDB performance problem.

24

Lessons Learned : Poisons

ONE BAD APPLE CAN SPOIL THE WHOLE BUNCH

25

I witnessed this several times as a kid growing up.

25

Wheres the rotten fruit in your database?

Which tablespace has the slowest overall disk read response time (ORMS)? ORMS Which tablespace has the slowest overall disk write response time (OWMS)? OWMS Which tablespace has the highest physical reads per transaction (PRTX)? PRTX Logical Reads (LRTX)? LRTX Is PREFETCH I/O effective and efficient (APPR)? APPR How do these tablespace metrics compare to the overall average for the database? Do any tables have (Overflows x 100/Rows Read) > 3%? Overflow Rows require double the logical I/O!
26

Get the rotten fruit out of your database before the whole thing gets spoiled!

26

Attendee Notes - Metrics

db2 get snapshot for tablespaces on DBNAME ORMS = (Total buffer pool read time (ms) / (Buffer pool data physical reads + Buffer pool index physical reads) ) OWMS = (Total buffer pool write time (ms) / (Buffer pool data writes + Buffer pool index writes) ) PRTX = (Buffer pool data physical reads + Buffer pool index physical reads) / # Commits + Rollbacks [from DB Snapshot] LRTX = (Buffer pool data logical reads + Buffer pool index logical reads) / # Commits + Rollbacks APPR = (Asynchronous pool data page reads + Asynchronous pool index page reads) / Asynchronous read requests TB RR/TX = (Rows Read / # Commits + Rollbacks)

27

Bonus notes adding notes to an attendee notes slide now thats a lot of notes! Have I mentioned yet that you can get a free white paper with even more formulas and techniques at http://www.database-brothers.com/papers.php ?

27

A Quote of Note

You cant build a reputation on what you are going to do

- Henry Ford

28

People will judge and admire you based on your actual achievements. If you can apply some of what you learn from this presentation, you should be able to rise to hero status very quickly.

28

Lessons Learned: NEVER

Use the word ALWAYS Listen to people who tell you what you CANNOT do Rely on a single snapshot, or small time slice, for making significant tuning decisions whats happening right now is often meaningless

Stop looking at the trees and try to see the forest

Use the word NEVER
29

Practice Forest Management It is important to see the big picture.

29

Passion, Commitment, and Discipline

30

Anyone care to dance?

30

Youre Done Passionately Tuning When

OLTP: Rows Read/TX/TB < 10 DB BP Sync Reads > 90% BP, Pkg Cache, & Catlg Cache hit ratios > 95% There are no bad apples No Slow TS (ORMS, OWMS) > 10ms No SQL > 10% CPU No SQL > 50% SLA time No SQL w/ Rows Read/Rows Fetched > 100 (IXEFF) No Files Closed No Lock or Token Waits Phone Rage Ends

Data Warehouse:
Prefetch is Effective (APPR > 10 for each TS) No Slow TS (ORMS, OWMS) TEMPSPACE defined where data isnt has 3-6 containers DB BP Sync Reads > 25% Catlg Cache Hit > 95% No Files Closed SQL having Frequency>1 uses MQTs / ASTs MDC tables Effective Indexes Phone Rage Ends
31

Do you have a database that meets these criteria? Congratulations! Time to celebrate! But, remember, theres always tomorrow and DB2 can change its mind when data volumes and number of users grow!

31

Session D02 8 Years of Performance Solutions Lessons Learned

Scott Hayes
Database-Brothers Inc. (DBI) Scott.Hayes@Database-Brothers.com www.Database-Brothers.com

32

The End. Please complete your session evaluation cards. Thank you!

32