Performing End-to-End Root Cause Analysis Using SAP Solution Manager

Michael Klffer, Marc Thier

Contents
1 1.1 1.2 Introduction to End-to-End Root Cause Analysis .............................................. SAP Standards for End-to-End Solution Operations .................................................... Root Cause Analysis Standard at a Glance ... Challenge ................................................. Goal .......................................................... 1.3 The Basic Concept of the Root Cause Analysis Standard .......................................... Process Flow ............................................ Architecture ............................................. 2 2.1 2.2 2.3 2.4 Root Cause Analysis in SAP Solution Manager ........................................................ 13 Supportability Overview ............................... 13 Navigation Concept ...................................... 14 Help Center Concept .................................... 14 Case Study: Usage of Root Cause Analysis for an End User Performance Problem ......... 16 3 3.1 3.2 3.3 End-to-End Change Analysis ....................... 23 Procedure ...................................................... 23 Tool Overview ............................................... 23 End-to-End Change Analysis ........................ 24 Analysis procedure ................................... 24 Architecture ............................................. 26 3.4 3.5 Configuration and File Reporting ................. 27 Analysis procedure ................................... 28 Compare Multiple Instances ......................... 28 Analysis procedure ................................... 28 4 4.1 4.2 End-to-End Workload Analysis ................... 31 Procedure ...................................................... 31 Tool Overview ............................................... 31 7 7.1 7.2 7.3 End-to-End Exception Analysis ................... 61 Procedure ...................................................... 61 Tool Overview ............................................... 62 End-to-End Exception Analysis ..................... 62 Analysis Procedure ................................... 62 Architecture .............................................. 64 6.4 6 6.1 6.2 6.3 End-to-End Trace .......................................... 49 Procedure ...................................................... 49 Tool Overview ............................................... 49 End-to-End Trace Analysis ............................ 49 Analysis procedure .................................. 50 Running a Trace ........................................ 50 Analyzing a Trace ...................................... 52 Architecture .............................................. 57 Wily Transaction Trace .................................. 59
6 7 8 4 6 6 6 3

4.3

End-to-End Workload Analysis ..................... 31 Analysis procedure ................................... 32 Architecture .............................................. 34

4.4 4.5 4.6

Wily Introscope ............................................. 34 Analysis procedure ................................... 35 Java Memory Analysis .................................... 37 Analysis procedure ................................... 37 Thread Dump Analysis ................................... 38 Analysis procedure ................................... 38

5 5.1 5.2 5.3 5.4 5.5 5.6

Operation System and Database Analysis ... 43 Procedure ...................................................... 43 Tool Overview ............................................... 43 SQL Command Console ................................ 43 OS Command Console .................................. 44 File System Browser ...................................... 45 LDAP Browser ................................................ 47

www.sap-press.com 1

Contents

7.4

Log Viewer .................................................... 66 Analysis procedure ................................... 66

9.2

People Certification ....................................... 73 E2E Solution Operations Overview E-Learning E2E50 ..................... 73 Technical Core Competence Courses ...... 73 Technical Expert Competence Workshops ................................................ 74 Management Competence Courses ......... 74 End-to-End Solution Operations Starter Package ......................................... 75

8 9 9.1

End-to-End Availability Analysis ................. 69 How to Implement the Root Cause Analysis Standard ......................................... 71 Installation and Configuration of Root Cause Analysis Scenario ........................................... 71 Installation ................................................ 71 Installation Check ..................................... 72 Configuration ........................................... 72 Configuration Check ................................ 73 A 9.3

How to Measure the Success of the Implementation ............................................. 75 Glossary ......................................................... 77 Index .............................................................. 79

2 Galileo Press 2008. All rights reserved.

6 End-to-End Trace

As mentioned above, in a solution landscape with several different components based on different technologies, it is very challenging to isolate the component, which is the root cause for a single issue. End-to-End Trace has a mechanism for isolating a single user interaction through a complete landscape and providing trace information on each of the involved components for only this interaction, starting with the user interaction in the browser and ending with the data being committed to the database. To do this, at the client, a client-independent proxy and a client-dependent plug-in are installed, which provides unique correlations for scenarios, clicks, and requests to the traffic between client and server. Clientside trace data is transferred from the client to the managing system via HTTP upload. Currently, Java and .NET components are covered by Wily Introscope tracing capabilities.

To analyze a problem in Java-based components, the Wily transaction traces can be used to identify which part of a request in a Java environment caused the problem. Furthermore, the known tools for analysis of a problem with an ABAP-based component can be used.

6.2 Tool Overview

The tools available in this area are:
End-to-End Trace Analysis

This tool collects component-specific trace summaries for both functional and performance analysis in order to identify the process steps and technical components, which require detailed analysis.
ABAP SQL Trace

Also called Transaction ST05, the ABAP SQL Trace is used on ABAP systems to analyze problems when working with the persistence layer. The aggregated

6.1

Procedure

ABAP SQL Trace is displayed in the End-to-End Trace Analysis; a detailed analysis can be done on the ABAP system.
Wily Transaction Trace

The analysis of a long running user request starts with an End-to-End Trace Analysis to identify the part of the request that takes the most time. This information is useful when doing a more in-depth analysis of this component. The End-to-End Trace Analysis provides all the functionality required for such an analysis. If the outcome of an analysis is that the problem is on the client side, you can use BMC AppSight to do a client analysis. For this reason, SAP provides recording profiles for the BMC AppSight Black Box, which records all client activity in order to enable SAP to analyze the problem. In the case of a network-related problem, we recommend performing a network analysis, which is not covered in this SAP PRESS Essentials guide.

The Wily Transaction Trace is a convenient tool for tracing problems in the J2EE infrastructure. It is an application that provides traces for J2EE applications, showing the component sequence over time in relation to the component calling sequence.

6.3 End-to-End Trace Analysis

The most common use case of the End-to-End Trace Analysis is identifying long running user requests within a complex system landscape. It has capabilities for analyzing across the complete system landscape, so that the

www.sap-press.com 49

End-to-End Trace

component causing the problem can be isolated and the root cause can be identified. Therefore, the response times of each component involved in the request execution and the request path through the involved components are provided for a detailed analysis. During an End-to-End Trace Analysis, the support employee must check the distribution of the response time over client, network, and server. After that, this employee should check for manipulations in the HTTP header (e.g., done by a proxy or another active network component). SAP recommends filtering out cached requests, because theyre usually quite fast, can be handled locally on the clients browser, and dont include server actions. Another common problem can be requests that transfer large amounts of data from the server to the client. Analysis procedure The End-to-End Trace Analysis is divided into two parts. First, a trace must run and be recorded; second, the recorded trace is evaluated. Before starting a new trace, the support employee should ensure that the problem to be analyzed could be reproduced. Running a Trace To run a new trace, the browser requires a plug-in to instrument requests that are leaving the browser on the client side. In addition, the systems on the server side must be enabled to react to these instrument requests by switching on certain trace functionalities. Figure 6.1 shows the plug-ins user interface.

tively, the plug-in can be downloaded directly from the End-to-End Trace Analysis application. The links are located in the Options section of the analysis application (see Figure 6.2). SAP plans to enable end users to download the plugin themselves using a link provided through email without requiring access to SAP Solution Manager.

Figure 6.2 End-to-End Trace Analysis Options

The plug-in is available in two versions: for Windows Installer or as a self-extracting archive. The following explanation is based on the self-extracting archive version. After the download, the plug-in is available as an executable file SAPIEPlugin.exe. This file must be copied to the client, from which the trace will run. When saving the file, a new directory should be created, the naming of which is up to the user. Before starting the trace, the server side must be enabled; the related function is located in the End-to End Trace Analysis tool on the Trace
Enabling tab (see Figure 6.3).

Figure 6.1

SAP IE Plug-in User Interface

The current version of the plug-in is available for download as an attachment to SAP Note 1041556. AlternaFigure 6.3 End-to-End Trace Analysis Enable Trace

50 Galileo Press 2008. All rights reserved.

6.3

End-to-End Trace Analysis

This tab displays the systems contained in the selected solution. To enable a system to run traces, mark the corresponding row in the table and click the Enable All button. In the State column, a green icon indicates that a system has been enabled to run traces. On the Options tab, you can determine for how long the trace enabling is valid. By default, tracing is automatically disabled after 30 minutes. When all preparations are complete, the trace can run. To do this, activate the plug-in on the client side by running the executable file SAPIEPlugin.exe. This opens an empty browser window with the deactivated plug-in user interface. If you enter the browser URL, you can navigate to the point where the error occurred without having to write trace data. Then activate the plug-in before performing the action to be analyzed. To do this, enter a name for the business transaction and a name for the transaction step. Both names can be freely defined by the user and will only be used to identify the trace in the analysis tool later. In the next field, select the trace level. Depending on the settings, different traces can be activated on the server side. The different trace levels are used for different analytical purposes:
Trace Level None

vides front-end response times, server response times (HTTP logs), and derived network times.
Trace Level Medium

This trace level provides the most important information and is used for a performance trace.
Trace Level High

This trace level is used for tracing functional issues, technical correctness, and data consistency. Depending on the chosen location, this level could significantly impact the response time and therefore should not be used in a productive environment. For more information about the trace levels and what is included, see Table 6.1. To enable the automatic upload of data on the client side, enter the host name of the SAP Solution Manager installation and the HTTP port in the user interface. If there is no HTTP connection between the client and the SAP Solution Manager host, the trace data can be uploaded manually into the analysis tool at a later point in time. Once all the settings are made, you can activate the plug-in by clicking the Start button. At this point, all requests will be instrumented and measured by the plugin. During the runtime of the transaction, the plug-in displays the bytes sent and received, as well as the number of requests and responses. Furthermore, the maximum number of physical connections and the currently active connections are displayed. After the transaction has concluded and the overall response is displayed in the browser, a screen similar to Figure 6.4 should appear.

With this trace level, no trace is written on the server side. This is helpful if you want to trace only the end user experience on the client side.
Trace Level Low

With this trace level, only the HTTP logs are written. This is only useful for HTTP analysis, because it pro-

J2EE Trace level HTTP Log SQL Trace Logging IS Transaction Trace

ABAP HTTP Log SQL Trace ABAP Trace RFC Enque/ Buffer Authorization

none low medium high x x x x x x xx x x x x x x x Aggregated Non aggregated xx xx xx xx

x = Summary displayed in End-to-End Root Cause Analysis, xx = not displayed in End-to-End Root Cause Analysis

Table 6.1

Trace Levels in ABAP and Java

www.sap-press.com 51

End-to-End Trace

Figure 6.4 End-to-End Trace Start Trace

If required, you can now trace additional transaction steps. To do this, click New Step and enter a descriptive name for the next transaction step. To stop the trace, choose the Stop Transaction button. The trace data is then automatically transferred to the analysis tool in the SAP Solution Manager. If the automatic transfer is not successful, an error message appears and you can upload the data manually. The relevant file is the BusinessTransaction.xml, which is located in the plug-in directory, specifically in the folder logs and subfolder <timestamp>_ <transactionname>. To leave the browser window and the plug-in, click the Exit button. In the next step, you will analyze the trace. Analyzing a Trace If the trace data on the client side was automatically transferred, you can now see the descriptive name of the transaction as a new entry at the top of the table in the
Trace Analysis tab of the End-to-End analysis tool. You

that the correct solution is selected. Now you can mark the new trace in the table. A dialog window appears, asking whether the correct solution was selected. After the dialog window is confirmed, the progress display for the data collection becomes visible. If this step is successful, a screen similar to Figure 6.5 should appear.

may have to use the Refresh button for the entry to be visible. If you have to manually upload the file, use the entry field Upload BusinessTransaction.xml located in the lower part of this screen. First, select the correct server side trace data to upload to the analysis tool. Make sure The column State should display a green icon. The meanings of the different icons are as follows: Grey Icon: Traces have not been collected yet.
Figure 6.5 End-to-End Trace Collect Trace

52 Galileo Press 2008. All rights reserved.

6.3

End-to-End Trace Analysis

Yellow Arrow: Traces are currently collected (by another user session). Green Icon: Traces have been collected and saved successfully. As the traces are deleted by default automatically after 60 days, you may want to consider locking a trace against deletion using the checkbox in the last column. The deletion is triggered by entering the application. The lower section of the screen lists the recorded transaction steps in a table (see Figure 6.6).

The screen is divided into four sections:

Client Trace Summary

This section contains all the values measured by the SAP HTTP Plug-in.
Time (accumulated)

In this section, the network time per HTTP request between the browser and the first HTTP server component is calculated as the difference of the browser response time and the server time. The distribution of the time intervals is displayed in a pie chart.
HTTP Status Codes

This section (or table) contains all the returned status codes for all requests. The distribution of the status codes is displayed in a pie chart.
Transaction Step details

In this section, all instances and server nodes that are involved with direct HTTP communication are listed. In addition to the statistics in the upper section, it is particularly interesting to see how the accumulated times are
Figure 6.6 Transaction Steps

divided. These numbers allow you to see at a glance whether the performance problem is on the client, the network, or the server. The network time displayed in the pie chart is not measured on network layer, but rather is calculated via the difference between client response time and server response time (see Figure 6.8). Therefore, the network time must be seen as an approximation. The measurement of the client response time takes place inside the browser and starts when the first byte of an HTTP request leaves the browser and ends when the last byte of an HTTP response has arrived. Because of this type of measurement, the client response time can be impacted in several ways. In addition to network- and server-time, it includes time that was spent by the browser, as well as time spent by other processes running on the operating system (e.g., virus scan, firewall). Generally, the estimated network time reflects the actual time when the User CPU Time ratio is very low, and might be too high when the User CPU Time is very high. In the example depicted in Figure 6.7, approximately 49 seconds are displayed as the time required on the server side to execute the transaction step, which amounts to 97 % of the total time required. This leads to the assumption that the Root Cause Analysis should con-

To start the analysis, mark a transaction step in the table and click Display. The Summary tab is displayed (see Figure 6.7). Here you can find the summarized information on the collected trace and an overview of the most important metrics that have been collected during the trace.

Figure 6.7

End-to-End Trace Summary

www.sap-press.com 53

End-to-End Trace

Browser
HTTP Plug-In adds header X-CorrelationID to each request
Preprocessing and queueing time

Server

Network Delay

Figure 6.8

tinue on the server. Therefore, we will look at more details about the request processing in the Messages
table tab (see Figure 6.9).

Figure 6.9

54 Galileo Press 2008. All rights reserved.

Time

Browser Response Time

Server Response Time

Network Delay

HTTP Service write correlation ID from header to http log file HTTP Plug-In writes repsone times and browser to BusinessTransaction.xml

Rendering/ loading of sub-requests

End-to-End Trace Network Time Calculation

The Messages table shows that this single transaction step consists of 46 separate requests. Each has its own table entry with an ID, Time, Method, URL, and HTTP

End-to-End Trace Messages Table

6.3

End-to-End Trace Analysis

Figure 6.10

End-to-End Trace Messages Table Selected Request

Status. The additional columns display the following details:

ID

cRendt (Client Rendering Time)

Time required for rendering the page and loading sub-requests.

csBytes

Identification number of HTTP request.

Time

Bytes sent from client to server.

scBytes

Timestamp taken from the client.

Method

Bytes sent from server to client. Because the requests, which are processed from the cache, arent important for the analysis, you can exclude them from the display with the Hide Cached button (see Figure 6.10). In our example, there are now only five requests that are processed on the server side. The request that required the most time is highlighted in red in the column
sRT. This request has the ID 29 and was processed in

Used HTTP method.

URL

Called URL. In the table, only the last part is displayed. By selecting one line, the complete URL will be displayed above the table.
Status

HTTP status code; for example, 2XX = Successful, 3XX = Redirect, 4XX = Client Error, and 5XX = Server Error.
cPT (Client Preprocessing Time)

It contains preprocessing and queuing time in the browser.

cRespT (Client Response Time)

approximately 49 seconds on the server. To perform an even more in-depth analysis, mark this request in the table and the HTTP header view appears below the table (see Figure 6.11). The top left section displays the HTTP header of the request (Client Request Header), as it was sent from the client. Directly below this is how the server received the header (Server Request Header). In the lower right section, the header of the corresponding response (Server
Response Header) is shown as it was sent from the server

It is measured on the client, starting when a request leaves the browser until the response receives. Technically, it contains the time spent in the network and on the server side.
sRT (Server Response Time)

Time spent in the server that is taken from the HTTP log.

to the client. And above this, in the upper right, is how the client received the response (Client Response

www.sap-press.com 55

End-to-End Trace

Figure 6.11 Client Message Headers

Header). The goal here is to determine whether there are

Possible request types that can be displayed here include the following:
Incoming HTTP request

any differences between the sent and received data. The differences may originate from network components, such as proxy servers, along the transfer route, and may be the reason for the error. In the example in Figure 6.11, we can rule out any such manipulations as a cause for the error. Below the HTTP section, you can review the Server
Analysis.

These values are taken from the HTTP logs.

Incoming J2EE request

These values are taken from the Distributed Statistical Records (DSRs).
Outgoing J2EE request

These values are taken from the DSRs.

Incoming ABAP summary

These values are taken from the ABAP statistical records and correspond to an RFC/HTTP Server Destination record.
Outgoing ABAP summary

These values are taken from the ABAP statistical records and correspond to an RFC/HTTP Client DestiFigure 6.12 End-to-End Trace Involved Systems

nation record.
Incoming ABAP request

From the Summary tab (see Figure 6.12), you can see that the approximately 49 seconds required for processing on server side (from Figure 6.10 in column sRT) consist of 0.5 seconds in the J2EE-based system and 48.7 seconds in the ABAP-based system. The Requests tree tab provides more details (see Figure 6.13).

These values are taken from the ABAP statistical records and correspond to an RFC/HTTP Server record.
Outgoing ABAP request

These values are taken from the ABAP statistical records and correspond to an RFC/HTTP client record.

56 Galileo Press 2008. All rights reserved.

6.3

End-to-End Trace Analysis

Figure 6.13 End-to-End Trace Requests Tree

In Figure 6.13, we can see that the request with ID 29 was received via HTTP from the J2EE system and then processed by the Page Builder component. The processing time was 242 ms (column Net for request with ID 29). Then the request left the J2EE system and was processed in the ABAP system. Here, four function modules were executed, where the function module HRXSS_PER_INIT_ PERNR required approximately 47 seconds for processing. Obviously, this function module is the cause of the long runtime of the traced transaction. We can now review the Distributed Statistic Records (DSR) and the Single Statistic Records (SSR), see Figure 6.14.

Moreover, the table contains icons for a detailed analysis of Java and ABAP. Figure 6.15 provides a legend for the icons. By clicking an icon, the corresponding data is displayed in a new browser window. Figure 6.16 shows the ABAP Trace Summary. To display the complete ABAP trace, choose the JumpIn icon in the table in the upper left section of the screen.

Based on the gross and net times, you can now make a qualified statement on the cause of the long runtime of the function module call. The End-to-End Trace Analysis tool offers the added feature of being able to graphically display the individual requests. This allows you to see at a glance which request is responsible for the long runtimes in the transaction step. This graphical representation can be displayed in the
Messages graphics tab (see Figure 6.17).

Architecture Tracing with a high trace level activated could have a negative impact on the system resources in the analyzed system. It could increase the CPU usage as well as the storage space in a database or file system. Therefore, it is necessary that systems are protected against such situations. The system administrator can determine centrally (in SAP Solution Manager) whether tracing in a managed system is allowed (see Figure 6.18)
Figure 6.14 End-to-End Trace Statistical Records

www.sap-press.com 57

Index
.NET framework 8 ConfigStore 26 Configuration and File Reporting 23, 26, 27 procedure 28 Custom Development 4 Customer Organization 5

F
File System Browser 43, 45 Flight Recorder 66 Flight Recorder Dump 77 Full Thread Dump 77

A
ABAP 8 ABAP Application Log 64 ABAP Dump 64 ABAP Extractor 9 ABAP Parameters 24 ABAP Runtime Error 61 ABAP SQL Trace 19, 49 ABAP System Log 64 ABAP Trace 31 ABAP Update Error 64 Accumulated Response Time 77 Application Error 61 Application Log 61 Application Management Team 4 apptracing 59 Average Response Time 77

D
Dashboards 10 Data Integrity 5 Data Volume Management 5 Database Analysis 43 Database Monitor 31 DB Parameters 24 Dialog Performance Metrics 77 Distributed Statistical Records 57

G
Granularity 77

H
Health checks 11 Help Center Concept 14

I
IDoc Error 64 Incident Management 5, 7 InfoCube 77 Introscope Enterprise Manager 35 Introscope WebView 35 Introscope Workstation 35 IT Infrastructure Organization 4

E
E2E Solution Operations Curriculum 73 Employee Self-Services (ESS) 16 End Users 4 End-to-End Availability Analysis 69 End-to-End Change Analysis 13, 23, 24 architecture 26 procedure 24 End-to-End Configuration Analysis Detail Viewer 25 End-to-End Detail Viewer 25 End-to-End Exception Analysis 13, 61, 62 architecture 64 procedure 62 End-to-End Solution Operations Starter Package 75 End-to-End Trace 18, 19, 49 architecture 57 End-to-End Trace Analysis 13, 49

B
Background Performance Metrics 77 BEx Queries 77 BEx Query Designer 77 BEx Web Templates 34, 77 BMC AppSight 49 BMC AppSight for SAP Client Diagnostics 19 Business Process and Interface Monitoring 5 Business Process Champions 4 Business Process Operations 4 Business Transaction 77 Business Transaction Step 77

J
J2EE Engine 26 J2EE Errors 65 Java 8 Java Application Availability 69 Java Memory Analysis 31, 37 procedure 37 Java Parameters 24 Java System Availability 69 Job Scheduling Management 5 Jump-in 77

procedure 50 End-to-End Workload Analysis 13, 31 architecture 34 procedure 32 Enhanced Change and Transport System (CTS+) 23 eSOA Readiness 5 Exception Handling 5 Extractor Framework (EFWK) 8, 77 architecture 10

C(++) 8 Change Control Management 5 Change Group 26 ABAP 27 Change Request Management 5 Change Type 26 ABAP 27 Collect Configuration Data 27 Compare Multiple Instances 24, 28 procedure 28

K
Key Performance Indicators (KPIs) 4 Key Users 4

L
Landscape Model 78 LDAP Browser 43, 47 Log Viewer 62, 64, 66 procedure 66

78 Galileo Press 2008. All rights reserved.

Index

M
MBean 59 MDX Queries 34, 78 Minimum Documentation 5

SAP NetWeaver Exchange Infrastructure 23, 74 SAP NetWeaver Portal 6, 15, 17, 23, 24, 26, 34, 74 SAP Solution Manager Root Cause Analysis 13 scenarios 3 Service Desk 7, 17 SAP Solution Manager Scheduler 27 SAP Solution Manager Starter Pack 73 SAP Solution Support Enablement Package 19 SAP Technical Operations 4 SAPIEPlugin 50, 51 SAP-Passport 59 Scatter 32 Server Termination 61 Service Desk 17 Service Level Agreements (SLAs) 4 Service Level Reporting 11 Single Statistical Records 57 SQL Command Console 43 ST-(A)PI 71 Standards for End-to-End Solution Operations 3, 4 Support Packages 24 System Administration 5 System Error 61 System Log 61, 64 System Monitoring 6, 11

Transaction ID 59 Transactional Consistency 5 Transport requests 24 TREX 24, 34

O
OLAP Reporting 77 Operating System Monitor 31 Operation System Analysis 43 OS Command Console 43, 44 OS parameters 24

U
Unicode 71 Upgrade 5

V
View 78

P
People Certification 73 Portal Access 69 PPMS-ID 9 Program Management Office 4 Program Termination 61

W
Wily Dashboards 36 Wily Introscope 10, 31, 34, 49 for Portal 15 procedure 35 Wily Introscope Enterprise Manager 9, 71 Wily Investigator 37 Wily Metrics 36 Wily Traffic Lights 36 Wily Transaction Trace 49, 59 Workload Analysis portfolio view 18 time profile 17

R
R/3 Workload Monitor 31 Remote Supportability 5 Response Times 32 Root Cause Analysis 3, 5, 6, 7 architecture 8 basic concept 6 challenge 6 configuration 72 configuration check 73 goal 6 implementing 71 in SAP Solution Manager 13 installation 71 installation check 72 key areas 13 navigation 14

X
X-CorrelationID 59

T
Thread Dump Analysis 31, 38, 64 procedure 38 Time Profile 32 Trace analyze 52 running 50 Trace levels 51 Transaction SLG1 61 SM21 61, 64 SM50 38 SMSY 9, 23, 72 SPRO 72 ST03N 31 ST04 19, 31 ST05 31, 49 ST06 31 ST22 61

S
SAP CRM 64, 74 SAP ECC 6, 17 SAP ECC Server 24 SAP NetWeaver Business Intelligence 8, 14, 64, 74, 77 time zones 11 SAP NetWeaver Development Infrastructure 23

www.sap-press.com 79