Fine Tune the Performance of Your SAP Enterprise Portal

Will Carlton SAP

What Well Cover

Performance tuning your SAP Enterprise Portal: overview Performance tuning and monitoring your portal back-end Designing and tuning portal content More tuning techniques: front-end and middleware

SAP AG 2004, slide 2

Performance Tuning Is Part Science, Part Art

Performance tuning requires a rigorous scientific application of hypothesis and experimentation to prove the hypothesis correct or incorrect It is the application of engineering principles to the study of system behavior under specified conditions Tuning should begin with specific goals and should proceed using well defined practices Tuning IS NOT testing! Testing is an important aspect, but it is useless if not well-defined and properly carried out Performance tuning requires thinking about your environment and understanding the tools at your disposal
And imaginative application of available technologies and practices

SAP AG 2004, slide 3

Improving Performance Via Tuning

This presentation covers some useful tips and techniques for optimizing performance of your SAP Enterprise Portal Although much of the material describes tips for version 6.0, many of the options, settings, and techniques are also relevant for version 5.0 The tips and suggestions are based on real-world testing on real projects and reflect knowledge gained thus far with version 6.0
As with any software, new tuning and performance improvements will continue to arise as the portal platform matures

SAP AG 2004, slide 4

Factors Impacting Portal Performance

Many factors affect the overall performance of a portal OS
Kernel Memory CPU Network JVM

Portal
Load limits Auth. schemes Login / home page Logging configuration

J2EE
Memory Threads Sessions

Java iViews
Code for performance Profiling Testing/evaluation

SAP AG 2004, slide 5

Factors Impacting Portal Performance (cont.)

Another way to organize tuning tasks: Back-end
Server hardware Server software (OS, J2EE, Portal) Content configuration (iViews, etc.)

Front-end
Browser

The Middle
Proxy/Cache Hardware devices (SSL accelerators, Proxies) Network Location

SAP AG 2004, slide 6

SAP Enterprise Portal Fine-Tuning Guide

The Guide "How to Fine-Tune Performance of Enterprise Portal 6.0" is available at http://service.sap.com/ A variety of topics are covered in this excellent guide:
Tuning the Operating System Tuning the Java Virtual Machine J2EE Engine parameters Tuning the Portal Run time (Portal Platform) Tuning the Database (PCD persistence) Tuning the Browser

Please review the Fine-Tuning Guide before making any changes to your portal systems This presentation relies heavily on real-world experience gained from a number of portal projects
Much of the experience has been distilled into the Fine Tuning Guide
SAP AG 2004, slide 7

What Well Cover

Performance tuning your SAP Enterprise Portal: overview Performance tuning and monitoring your portal back-end Designing and tuning portal content More tuning techniques: front-end and middleware

SAP AG 2004, slide 8

Back-End Items You Can Performance Tune

1 - Portal server and other hardware 2 - Portal server operating system 3 - Portal server Java Virtual Machine (JVM) 4 - Portal server J2EE Software 5 - Portal server Log Analyzer 6 - Portal server monitoring settings

SAP AG 2004, slide 9

1 - Portal Server and Other Hardware

Separate the components RDBMS, J2EE, LDAP, TREX, HTTPD Utilize the system fully
Optimize memory allocation for each element (server, dispatcher) Optimize CPU allocation (processor sets, binding) Carefully review memory usage patterns on active systems Never exceed 80-85% CPU load for extended periods of time

SAP AG 2004, slide 10

1 - Portal Server and Other Hardware (cont.)

Browser / Mobile Device

Acceleration Smart Load Balance

Proxy? Cache?

Mem/Process Dedicated CPU?

Load Balancer

Web Server

Web Server

J2EE Cluster

Dispatcher Dispatcher

J2EE Cluster

Dispatcher Dispatcher

Server

Server

Server

Server

Server

Server

SAP AG 2004, slide 11

2 - Portal Server Operating System

Consult SAP Notes relevant to the Operating System of your Portal hardware
Review the O/S Dependency information in the Guide at: http://service.sap.com/instguides/
http://www.sap.com/instguides/ -> SAP Web Application Server -> Release 6.20 (or 6.30 for SP3)

Note 492222 outlines the O/S dependencies for Unix systems For Solaris, the TCP/IP stack may need some adjustments
Refer to the topic TCP/IP Tunable Parameters in Solaris Tunable Parameters Reference Manual, at:
http://docs.sun.com/db/doc/816-0607 Example: Modify the /etc/system file and configure "table size" and "wait interval" set tcp: tcp_conn_hash_size=32768 (default is 512K) set tcp_tcp_time_wait_interval=60000 (1 minute)

SAP AG 2004, slide 12

3 - Portal Server JVM

Tuning of the J2EE Engine is covered in "How to Tune SAP J2EE Engine 6.20 v3.0"
Available at http://service.sap.com/webas/ -> Tools & Demos -> HowTo Guides

Some J2EE parameters may not be optimally configured after installation Each customer's system is different and must be tuned appropriately A number of configurable parameters are explained in the Tuning Guide Performance testing will identify areas for improvement

SAP AG 2004, slide 13

3 - Portal Server JVM: Memory Settings

Java Virtual Machine tuning parameters can have a substantial impact on performance Consult SAP Note 552522 for suggestions:
-Xmx<size> specify the maximum heap size Minimum is 512MB, recommended size is 1.5GB (or more) specify the initial heap size, recommendation: [ms] = [mx]/2 ratio between young and old generation area (if you are short of main memory, increase the value up to 4 or 6; minimum: 2) initial size of young generation area

-Xms<size> -XX:NewRatio=3

-XX:NewSize=[ms] / ([NewRatio] + 1)

-XX:MaxNewSize=[mx] / ([NewRatio] + 1) maximum size of young generation area

SAP AG 2004, slide 14

3 - Portal Server JVM: Garbage Collection

Automatic Garbage Collection (GC) can adversely impact performance
Memory settings affect the garbage collector

Monitor GC events with JVM parameter verbose:gc GC events will be written to the system log: [GC 2095K->1709K(2160K), 0.0017628 secs] [Full GC 2161K->1018K(2276K), 0.0576353 secs] GC = Small (regular GC) short duration Full GC = longer duration where all objects are queried Increase heap size if you continue to see Full GC events in the log

SAP AG 2004, slide 15

3 - Portal Server JVM: Garbage Collection (cont.)

Refer to SAP Notes 634689 and 608533 for more details on memory issues and Garbage Collection Details on various VM options related to memory usage and garbage collection can be found on Sun's Java Web site:
http://java.sun.com/docs/hotspot/VMOptions.html http://java.sun.com/docs/hotspot/gc/

SAP AG 2004, slide 16

3 - Portal Server JVM: Other Settings

Previously SAP has recommended the use of compiler option server to ensure that the JIT (Just in Time) compiler is used Due to some bugs in the JVM, we no longer recommend this option for SP2
A performance penalty may be observed without this option See SAP Note 696410 for details

SAP AG 2004, slide 17

3 - Portal Server JVM: Memory Mgmt. Notes

SAP Notes related to Java Memory Management Java VM settings for EP6 on J2EE 6.20 Java HotSpot VM Memory Parameters Central Note for Memory Issues, SAP J2EE Engine 6.20 J2EE crashes with OutOfMemory on SUN JDK Obtaining Memory Profiling Information 610 134 AIX, HP, Solaris SAP J2EE Engine 6.20 on AIX: how to increase the heap size 696410 552522 634689 597187 667841 667711

SAP AG 2004, slide 18

4 - Portal J2EE Server

The J2EE Engine can be configured to suit your system capacity and usage patterns Memory allocations can be made for Dispatcher and Server The Dispatcher typically requires much less memory and CPU time
Typically only about 64MB of memory (-Xms64M)

Settings are modified with the J2EE Configtool

SAP AG 2004, slide 19

4 Portal J2EE Configtool *******

/usr/sap/[SID]/j2ee/j2ee_[INSTANCE]/configtool

SAP AG 2004, slide 20

4 - Portal J2EE: Dispatcher Tread Manager

Dispatcher Thread Manager Parameter InitialThreadCount MaxThreadCount ChangeThreadCountStep InitialRQSize ChangeRQSizeStep MaxRQSize Default 100 200 100 100 50 200 Recommended 100 200 100 100 100 500

SAP AG 2004, slide 21

4 - Portal J2EE: Server Thread Manager (cont.)

Server Thread Manager Parameter InitialThreadCount MaxThreadCount MinThreadCount InitialRQSize ChangeRQSizeStep MaxRQSize Default 40 40 40 500 100 6000 Recommended 100 200 100 1000 200 10000

SAP AG 2004, slide 22

4 - Portal J2EE: Other Tuning Opportunities

Connections manager
Manages client connections in the dispatcher

Timeout manager
A non-distributed system for scheduling tasks for future execution in a background thread

HTTP service
Receives HTTP requests from clients and responds to them by returning the HTML files that correspond to the requested URL

DBPool service
Creates and stores database connections so that they can be reused
Along with the transaction service, DBPool manages transactions and synchronizes the transactions in the server with those in the database

SAP AG 2004, slide 23

5 - Portal Log Analyzer (cont.)

Select the Logger you wish to activate

Press "Configuration Mode"

SAP AG 2004, slide 24

5 - Portal Log Analyzer

Use the Log Analyzer to pinpoint bottlenecks
Activate a log by navigating to System Administration -> Monitoring -> Portal -> Logging Console:

Activate logger (set to true)

Set log level to Info, Debug, or All

Click "Apply, then "View Mode" (at bottom of list)

SAP AG 2004, slide 25

5 - Portal Log Analyzer (cont.)

View the Logs by selecting the log detail you want (INFO, WARNING, etc.) Click "Display" to view
SAP AG 2004, slide 26

5 - Portal Log Analyzer (cont.)

Analyze the displayed logs for timing issues and events that take longer than expected

SAP AG 2004, slide 27

5 - Logging Tips For Productive Systems

Log analysis can pinpoint performance bottlenecks, but log levels set too high can dramatically impact performance in some cases > 300%!!! Make sure loggers are set to only log warnings and/or errors in productive portals Change the default logging level of "system.log" and "default.trc" by:
Edit saplogging.config in <SAPJ2EE>/cluster/server/services/log/work Change /System.severity to "NONE" Change /Application.severity to "NONE" Change .logs = log[defaultTrace] to .localLogs=log[defaultTrace]

SAP AG 2004, slide 28

6 - Portal Monitoring Settings

Disable Collecting Monitoring Data
If youre not using the SAP Computing Center Management System (CCMS) to collect and monitor portal data, disable the property for it Logon to the portal as an administrator In the top-level navigation bar, select System Administration > System Configuration From the Detailed Navigation panel on the left, select Monitoring Configuration Under JARM, Java Application Responsetime Measurement in the Monitoring Configuration iView on the right, uncheck the following:
Collect monitoring data Select Save in the iView

SAP AG 2004, slide 29

What Well Cover

Performance tuning your SAP Enterprise Portal: overview Performance tuning and monitoring your portal back-end Designing and tuning portal content More tuning techniques: front-end and middleware

SAP AG 2004, slide 30

Content Considerations
Server-side caching
Reduce server and browser response time by caching iViews and pages
In the iView or Page Editor: modify the Cache Level for each iView/page and set to "User, Shared, or Session"

With version 6.0, entire pages can be cached and sent to the browser without consulting the back-end systems Caching of pre-rendered content reduces the network traffic and load on back-end systems Most helpful with static content or content that changes infrequently Note: URL iViews cannot currently be cached on the server side with SP2
SAP AG 2004, slide 31

Content Considerations (cont.)

The Home/Welcome page should be restricted to primarily static content that can be cached Try to limit the number of iViews on the Home Page
Large numbers of iViews will require longer rendering and increased network traffic

Use Eventing to coordinate iViews that have a selection model

This will reduce traffic to only the affected iViews and not require the entire page to be redrawn

Reduce the size of graphic images, Javascript, or other assets contained in your content Limit the number of roles the user has, or use the "Merge ID" feature to collapse multiple worksets/roles in order to reduce the navigation hierarchy

SAP AG 2004, slide 32

Content Considerations - IView Types

Embedded
The Page Builder fetches the content of the iView and places it on the page in a <table> structure All of the iViews in the page are fetched in one request by the Client iViews are fetched asynchronously in multi threads by the Page Builder (server side) Each iView is included asynchronously into the Page Builder response (doContent method is called) with a special response object that stores the output of the iView in a main storage area

URL (isolated)
Isolated iViews are iViews that are fetched by an <iframe> in a separate request by the client The Page Builder creates an <iframe src="iView source" /> statement for each iView All requests go directly from the client to the (Web) content source

SAP AG 2004, slide 33

Content Considerations - IView Types (cont.)

Pumped
Pumped iViews are isolated iViews that are fetched in one request by the client (together with the page) Pumped iViews are not rendered in a table structure but rather placed inside an empty <iframe> element
Done by a Client Side JavaScript

Second request to the iView goes directly from the client to the (Web) content source
Embedded Simple HTML sources Sources with High usage of scripting and cookies Download volume Number of requests
SAP AG 2004, slide 34

Isolated No restrictions Sometimes the only option Lower (only iView) High (number of iViews)

Pumped No restrictions limited

No restrictions Very limited

Higher (whole page) Low (min. 2)

Lower (only iView) Low (min. 2)

Content Considerations - Pages

Pages should contain only one type of iViews in them isolated (or pumped) or embedded We do not block mixing isolated (or pumped) and embedded iViews, but the Page Editor warns from creating mixed pages Pages that act as sub-pages are treated as embedded, hence do not add additional <iframes> to the containing page

SAP AG 2004, slide 35

Java iView Performance

Performance of Java iViews can affect the overall performance of the Portal Use logging to determine which iViews take the greatest amount of time
The PDK covers how to add logging functionality to your components

Use the new JARM (Java Application Responsetime Measurement) API to collect monitoring information for iViews that are problematic (requires small code modifications to your Java iViews) For information regarding JARM, go to http://service.sap.com and search for "JARM" or "Java Monitoring Overview"

SAP AG 2004, slide 36

Java iView Performance Tips

Java iViews can often benefit from several performance enhancing techniques: Connection Pooling
Reduces the number of connection negotiations under heavy load and can dramatically improve throughput Provided automatically by the Connection Framework (CF), so use the CF API instead of the normal JDBC or JCO connectivity approach

Data Object Caching

A caching API is available to cache data objects returned from back-end systems Reduces load on back-end system and allows data to be shared among users or sessions

See SDN article: "Java Development Techniques for Improving Performance of EP iViews" for details
SAP AG 2004, slide 37

What Well Cover

Performance tuning your SAP Enterprise Portal: overview Performance tuning and monitoring your portal back-end Designing and tuning portal content More tuning techniques: front-end and middleware

SAP AG 2004, slide 38

Front-End Browser Cache

Improve browser response time by using the browser cache
enables your browser to cache resource files such as images, CSS, JS, txt, etc. Best option is "Automatically"

Refer to the section on Accessing the Enterprise Portal under the topic Setting Internet Browser Options in the End-User Guide at: http://help.sap.com/ep

SAP AG 2004, slide 39

Front-End Browser Compression

Compression support should be enabled to reduce network traffic for Internet Explorer:
From the client browser menu bar, select Tools, then Internet Options Choose the Advanced tab and scroll to HTTP 1.1 settings Make sure the Use HTTP 1.1 checkbox is selected If your network operates with a proxy server, verify that Use HTTP 1.1 through proxy connections is also selected. Click OK

SAP AG 2004, slide 40

The Middle Proxy/Cache

A proxy cache can offload a significant amount of work from the J2EE server, freeing resources to handle dynamic content
A 50% cache hit ratio would reduce the number of hits to the J2EE engine by

Software Proxy/Cache servers such as Squid or Apache are inexpensive, but do not offer comparable performance to commercial products
Some commercial software proxies are supported builds of Squid or Apache

Hardware Proxy/Cache servers such as NetCache or Cache Engine are more expensive but offer hit rates to 50-60% An excellent resource is:
http://www.web-caching.com

SAP AG 2004, slide 41

The Middle - Network

Network configuration can impact latency
This is not a direct performance factor (i.e., does not increase or decrease the load) Users will perceive this as a performance factor as it will impact page load/response times, however

Careful placement of components with respect to proximity to users or data sources can yield significant improvements
Impacts performance of the whole solution Also consider position of firewalls and proxies

Packet sniffers can also be useful in tracing total byte counts and network throughput

SAP AG 2004, slide 42

Pre Go-Live: Performance Benchmarking

Begin performance testing EARLY in implementation cycle Require performance metrics as part of unit testing development Build baselines for each configuration. Understand performance characteristics at each level of configuration complexity
develop cost/benefit analysis develop scalability metrics rationalize performance expectations

Keep historical performance data for analysis after go-live

SAP AG 2004, slide 43

Going Live Check

Request a Going Live Check which ensures everything is attended to. The Going Live Check consists of three sessions designed to root out any major bottlenecks or performance issues:
Analysis - checks the major system components of your SAP solution to verify system availability and consistency Optimization - checks typical business processes and the transactions involved in those processes for possible improvement in performance and through put Verification - a routine check of the system to verify the assumptions from the Analysis Session

Details at http://service.sap.com/goinglivecheck

SAP AG 2004, slide 44

Post Go-Live: Performance Benchmarking

Thoroughly test any major new functionality prior to moving it into production Periodically review logging settings and other parameters to ensure that they have not "accidentally" changed Periodically review system performance during peak load periods (such as Monday morning) using your testing tool (but don't add additional load to the system) Compare performance data for pre-go-live historical data for variances

SAP AG 2004, slide 45

SAP Developer Network sdn.sap.com

The central hub for the SAP technology community Everyone can connect, contribute and collaborate- consultants, administrators and developers Focus around SAP NetWeaver and SAP xApps High quality of technical resources Articles, how-to guides, weblogs, collaborative areas, discussion forums and downloads, toolkits and code-samples A collaboration platform, not a one-way street SAP experts from customers, partners and SAP SDN is powered by SAP NetWeaver Built on the SAP Enterprise Portal Featuring collaboration capabilities of SAP Knowledge Management
SAP AG 2004, slide 46

Resources
SAP Service Marketplace (service.sap.com)
How to Fine Tune Enterprise Portal 6.0 How to Tune SAP J2EE Engine 620

Delivered Documents (J2EE PL)

J2EE Administration (PDF/html)

Testing tips
Understand your testing tool! Utilize system monitoring capabilities Organize your data and make it meaningful

SAP AG 2004, slide 47

Resources (cont.) Test and Measurement Tools

Unix OS vendors provide many tools (built-in and purchased) that assist in analysis of performance
Use the free tools and understand them prior to implementing more complex measurement products

For example, SUN provides a comprehensive toolset for performance measurement And plenty of documentation online on how to use and understand the data.
http://developers.sun.com/solaris/articles/performance_tools.html

http://developers.sun.com/solaris/articles/prstat.html

Java tuning and performance docs are also available

http://developers.sun.com/solaris/articles/alt_thread_lib.html

SAP AG 2004, slide 48

7 Key Points to Take Home

Distribute load among different servers by separating portal, DBMS, LDAP, and other components. Choose appropriate hardware for each component. Carefully monitor CPU load during peak loads. Never exceed 85% utilization for extended periods of time. Correct tuning of JVM memory parameters is critical. Frequent Full GC events will kill your portal's performance. The Tuning Guide contains a number of useful tips and parameter settings designed to improve throughput. Reduce the complexity of your starting (home) portal page. Enable caching of iViews if possible. Web caching (hardware or software) services can offload work from the portal server for images, style sheets, etc. Ensure that the minimum amount of logging is enabled for productive systems
SAP AG 2004, slide 49

Your Turn!

Q&A
Will Carlton
SAP NetWeaver RIG (US) Development Enterprise Portal

william.carlton@sap.com

SAP AG 2004, slide 50