Mobile Performance Testing Using HP LoadRunner

User Guide
V3 August 2014

Copyright

Copyright 2016, Perfecto Mobile Ltd. All rights reserved.

The information in this document is subject to change without notice. No part of this document may be
reproduced, stored or transmitted in any form or by any means, electronic or mechanical, for any
purpose without the express written permission of: Perfecto Mobile Ltd.

Perfecto Mobile Ltd. assumes no liability for any damages incurred, directly or indirectly, from any
errors, omissions or discrepancies between the software and the information contained in this
document.
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

Contents
1 Terms & Definitions .............................................................................................................................. 3
2 Introduction .......................................................................................................................................... 4
3 Setup Pre-Requisites ............................................................................................................................. 5
4 Reference Documentation .................................................................................................................... 5
5 Examine network conditions impact on mobile user experience......................................................... 6
6 Examine server load impact on mobile user experience .................................................................... 13
7 Reporting............................................................................................................................................. 17
8 Appendix A- Inside the VuGen template ............................................................................................ 18
9 Appendix B- Creating a VuGen Web Performance Script ................................................................... 22
10 Appendix C - Using the HP Password Encoder ................................................................................ 23

Table of Figures
Figure 1: Perfecto Mobile Performance Solution ......................................................................................... 4
Figure 2: Perfecto Mobile Performance solution architecture ..................................................................... 4
Figure 3: MobileCloud Automation Sample Script ....................................................................................... 8
Figure 4: Viewing TCP traffic using Wireshark ............................................................................................ 12
Figure 5: PCAPPerf network traffic analysis................................................................................................ 13
Figure 6: HP LoadRunner Analysis - Transaction Summary in Summary Report ........................................ 17
Figure 7: HP LoadRunner Controller - Interactive Schedule Graph ............................................................ 17
Figure 8: Sample Script - Execute MobileCloud Automation ...................................................................... 19
Figure 9: Sample Script - Check Execution Status of MobileCloud Automation Script............................... 20
Figure 10: Sample Script - Collect and Analyze MobileCloud Automation Script ....................................... 21
Figure 11: Create new script in VuGen ....................................................................................................... 22
Figure 12: Record web transaction in VuGen ............................................................................................. 23
Figure 13: VuGen recording toolbar ........................................................................................................... 23
Figure 14: VuGen parameter replacement dialog ...................................................................................... 23
Figure 15: HP Tool password encoder ........................................................................................................ 24

Page 2 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

1 Terms & Definitions

Term Description
HP VuGen HP Virtual User Generator
SaaS Software as a Service
KPI Key Performance Indicator, also known as transactions
SLA Service Level Agreement
CI Configuration Item
MCM Mobile Cloud Manager

* Disclaimer: Any and all references to the HP tool chain: LoadRunner, Performance Center, VuGen and
password encoder should all be considered as recommendation. Please turn to HP.com for proper
documentation and guidance regarding using the HP tool chain.

Page 3 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

2 Introduction
This document details the setup process for implementing the Perfecto mobile performance product for
measuring the mobile users experience during impacted network and server load conditions.
The Perfecto MobileCloud provides real devices, connected via Wi-Fi or cellular carriers to the internet,
and spread in different locations. The Perfecto performance product enables real mobile device testing
on simulated network and server load conditions.
This solution enables users to measure response time and availability of key transactions inside a mobile
application, under simulated real-world conditions. This new product handles this complex and
challenging process composed of unpredictable network conditions and varying server loads, alongside
mobile device performance under such conditions.

Mobile performance testing measures the impact of

Network conditions on the user experience
Application server load on the user experience

Figure 1: Perfecto Mobile Performance Solution

Figure 2: Perfecto Mobile Performance solution architecture

Page 4 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

3 Setup Pre-Requisites
Before getting started, be sure to complete the following pre-requisites list. All listed items are required,
and can be obtained from Perfecto Mobile Support.

3.1 Required
Acquired from Perfecto Mobile Support:

Perfecto Mobile Performance License

MobileCloud account, including required devices that are connected to the network
virtualization component via Wi-Fi
MobileCloud Automation License

Acquired from HP:

HP Performance Center and/or LoadRunner

HP Virtual User Generator (VuGen) installation
Download from HP BSM: BSM-Saas > Admin > EUM > Setup and maintenance >
Downloads; OR
Online from here

3.2 Optional
Packet capture analysis tools:
Wireshark, Shunra Analytics (available from HP Software), http://pcapperf.appspot.com

4 Reference Documentation
4.1 Perfecto Mobile
Help Knowledgebase
HTTP API guide
HP VuGen Sample Script and LoadRunner Template

4.2 HP
HP LoadRunner user guide
Online from here
HP VuGen user guide here

Page 5 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

5 Examine network conditions impact on mobile user experience

This section includes the following components:

Creating a MobileCloud Automation script

Network conditions commands
Device Vitals commands
Viewing results using Wireshark/PCAPPerf Analytics

5.1 Create MobileCloud Automation Script

To get started, create a MobileCloud Automation script, within the Perfecto MobileCloud.
Once your script is completed, follow the steps and figure 1 below to customize it to Mobile Monitoring.

Getting Started with MobileCloud Automation:

Refer to the Perfecto Community.

Tip:
It is recommended follow the ScriptOnce1 methodology so your script is applicable for all the
devices you are targeting.

Step 1. Start your script by doing some cleanup and proper launch.
Its very important to clean browser cache/ remove application from resident memory
and ensure a consistent script starting point to ensure script stability and timers
accuracy.
In the case of a mobile application, the recommendation is to use the following
sequence:
- Close application
- Wait(5)
- Start application
In the case of a browser, it is recommended to clean the browser cache. Its certainly
recommended to use the device generic browser (ex.: safari) as opposed to an
instrumented browser.
Step 2. Identify the KPIs to be measured.
These are the Key Performance indicators (KPIs) you are interested in monitoring.

Note:
A KPI is a measurement of response time and availability of a user facing
transaction. It can be composed of a single or multiple user action(s)
In our example, the CNN page load time is our KPI.

1
Perfecto Mobile's patented ScriptOnce technology allows the creation of device agnostic test objects. With this
technology you can write a script and re-use it on multiple platforms and networks with minimal to no
modifications to the script.

Page 6 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

Step 3. Measure each KPI

Define a timer starting point for each KPI. Use the Timer start function one line after the
desired command line. The timer name can be any name. Do this for each KPI you want
to measure.
Step 4. Place a Text checkpoint following the timer and command being measured. Make sure
to Try the likelihood of finding the object on the device UI, you want to get a value thats 98%
or higher. To get accurate timer value, select the advanced settings checkbox and set the
OCR-analysis mode to Automatic. Repeat the try and see that the high likelihood is
maintained. It is also possible to use image checkpoints, however text checkpoints will be more
reliable and accurate.
Step 5. Terminate the timer on condition success
Place a Timer stop function within the condition On Success.
If the condition is successful, the timer value will be reported, facilitating the
response time of the aspect of the KPI.
If the condition fails, the timer will not be reported, thus facilitating the
availability aspect of the KPI.
Step 6. Insert a timer checkpoint
1. Set the timer ID to be the same as the one you defined in timer start
2. Set the upper bound to a high value (ex.: 1000 seconds)
3. Set the type of timer to be UX
Step 7. Exit the script on condition failure (Optional)
You may want to close the device(s) and exit the script by placing an Exit function within
the condition On failure.
Step 8. Add network virtualization commands
Network virtualization start
Network virtualization update
Network virtualization stop
Step 9. Add device vitals commands
Device vitals start
Device vitals stop
Device get vitals

Page 7 of 24
Mobile Performance User Guide: MobileCloud Automation and HP BSM

Figure 3: MobileCloud Automation Sample Script

Page 8 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

5.2 Device vitals commands

The below MobileCloud Automation functions drive the device vitals.

Device vitals start Begin collection of device vitals for the mobile device. It is recommended to
leave the parameters field as-is (collect-all) and set the collection frequency to run at the highest
possible frequency.
Device vitals stop- this will stop the collection of the device vitals
Get Vitals- this will get an immediate reading of the vitals selected

The script execution report will have a tab labeled performance, where a CSV file will be available to
download, containing all device vitals.

At the time of writing this document not all commands are supported across all iOS/Android platforms.
Therefore it is recommended to set the error policy for the above commands to catch so that they do
not fail the script in case a device is not supported.

5.3 Network virtualization commands

The below MobileCloud Automation functions drive the network conditions.

Network virtualization start

Start network virtualization for the device with the specified conditions
Network virtualization update
Update network virtualization for the device with the specified conditions; for any condition
that is not specified, the system will continue to use the existing value
Network virtualization stop
Stop network virtualization for the device and save PCAP file to the repository

For network conditions testing, the device being used is connected to the Web via WiFi.

Use the above Network virtualization start and the Network virtualization update commands to activate
and update the network conditions. Without this activation there is no effect on the device
communication to the Web.

Page 9 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

5.3.1 Network virtualization start and update

Name Mandatory Type Default Description
Device ID Handset DUT The device for this command.
Latency Optional Number 0 Latency applied on packets in the Network. Effective values are in
the range of 0-8000 ms.
Packet loss Optional Number 0 Network packet loss.
Effective values are in the range of 0-100%.
A reasonable packet loss value should not exceed 5%
Bandwidth in Optional Number unlimited Limitation on the allowed download network bandwidth into the
device.
Effective values are in the range of 3-100,000 Kbps, or unlimited.
Bandwidth Optional Number unlimited Limitation on the allowed upload network bandwidth from the
out device.
Effective values are in the range of 3-100,000 Kbps, or unlimited.
Packet Advanced Number 0 Network packet corruption.
corruption Effective values are in the range of 0-100%.
A reasonable packet corruption value should not exceed 5%
Packet Advanced Number 0 The percentage of network packets sent immediately without any
reordering delay. Used alongside the Latency parameter. Moreover, the
packets that are sent immediately will arrive earlier than the
packets that were delayed by the defined latency value, essentially
creating a packet reordering.
Effective values are in the range of 0-100%.
A reasonable packet reordering value should not exceed 10%
Packet Advanced Number 0 Network packets duplicated.
duplication Effective values are in the range of 0-100%.
A reasonable packet duplication value should not exceed 3%
Delay jitter Advanced Number 0 Random latency variation; the actual latency between latency +-
jitter. Used alongside the Latency parameter.
Effective values are in the range of 0-8000 ms.
For example, if the latency is defined to be 100 ms and the jitter is
10 ms, this causes the added delay to be 100ms 10ms.
Correlation Advanced Number 0 Network packet value correlation, affecting the Latency,
Corruption, Reordering and Duplication.
Effective values are in the range of 0-100%.
For Latency, Corruption, Reordering and Duplication, the current
packet n value will be correlated by this % to previous packet n-1
value. In other words, the current packet value is correlated to the
previous packet value.
For example, if the correlation is defined to be 25%, the packet
loss is 5%, and the previous packet was lost, then the updated
packet loss value (for the current packet) would be 75% * 5% +
25%, which equals 28.75%. However, if the previous packet was
not lost, then the packet loss value would be 75% * 5%, which
equals 3.75%.
Blocked Advanced String Network packet block, to specific destinations, defined by domain
destinations name, IP address, and IP range destinations in IP Prefix (Slash)
notation.
Examples:
Domain name: www.google.com
IP address: 192.168.2.0

Page 10 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

IP range destination: 192.168.2.0/24

To remove a network packet block, prefix the value with a '-'. For
example, -www.google.com.
Blocked ports Advanced String Network packet block, to specific ports.
For example, to block http, define port number 80.
To remove a network packet block, prefix the value with a '-'. For
example, -80.

5.3.2 Network virtualization stop

Name Mandatory Type Default Description
Device ID Handset DUT The device for this command.

5.3.3 Suggested Profiles

Typical network emulation values

Generation Type Level Latency Bandwidth In Bandwidth Out Packet Loss

(ms) (Kbit/s) (Kbit/s) (%)
2G GPRS Good 500 50 28 1
2G GPRS Average 600 40 24 2
2G GPRS Poor 700 30 20 2
2G Edge Good 200 250 100 0
2G Edge Average 400 150 50 1
2G Edge Low 500 100 30 2
3G UMTS Good 100 400 64 0
3G UMTS Average 140 300 50 0
3G UMTS Low 210 200 30 1
3.5G HSPA Good 110 1800 400 0
3.5G HSPA Average 130 1350 300 0
3.5G HSPA Low 190 900 200 1
3.5G HSPA+ Good 100 7,300 1,500 0
3.5G HSPA+ Average 120 4,400 1,100 0
3.5G HSPA+ Low 170 2,000 600 1
4G LTE Good 80 18,000 9,000 0
4G LTE Average 100 10,000 4,400 0
4G LTE Low 120 7,000 3,000 1
4G LTE Advanced Good 70 50,000 30,000 0
4G LTE Advanced Average 80 30,000 20,000 0
4G LTE Advanced Low 100 14,000 10,000 0

Emulation of problematic networks

Problem Level Packet Loss % Corruption % Duplication % Reorder %

Steady Network 1 0 0 2
Noisy Network 2 2 0 5
Unstable Network 3 3 2 10
Very Unstable 5 5 5 20
Network

Page 11 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

5.4 Viewing results using Wireshark or PCAPPerf

View the results of your test using various tools. Follow the below installation and viewing procedures.
5.4.1 Using Wireshark
Download Wireshark from http://www.wireshark.com. Open the PCAP file (if you have a .shunra file, set
the file extension to .zip and unzip it to find the .PCAP file) with wireshark.

Figure 4: Viewing TCP traffic using Wireshark

Note that in case the packet payload is encrypted you may need to decrypt the packets payload. Heres
one link describing the process.

Page 12 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

5.4.2 Using PCAPPerf

Open http://pcapperf.appspot.com, upload the PCAP file and observe the waterfall chart/HAR file.

Figure 5: PCAPPerf network traffic analysis

6 Examine server load impact on mobile user experience

This section includes the following components:
Creating a VuGen script to simulate the server load
Creating a VuGen wrapper for the real mobile device
Best practices for configuring real device testing within LoadRunner

This use-case measures the impact of a server load on real mobile devices, leveraging the HP
Performance Center LoadRunner tool to execute the following:

Apply a significant number of virtual users on the backend

The script representing these virtual users may already exist or may be generated from a
previous recording of the device traffic as it communicates with the service.
Execute a real device test and measure KPIs, in parallel, on a sample set of devices

LoadRunner will present reporting for both approaches, as well as the script execution result for each
device from Perfecto Mobile.

Page 13 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

6.1 Creating a VuGen script from the network sniffer (PCAP) file to
facilitate the backend server load
This step is optional and only required if there is no existing backend load traffic script available; Use the
PCAP file generated in the previous section, and import it into VuGen.

Step 1. Open VuGen

Step 2. Go to File > New Script and Solution
Step 3. Select the Mobile Application - HTTP/HTML protocol
Step 4. Click Create
Step 5. Click Record to open the Recording Wizard
Step 6. Select Analyze Traffic to analyze the traffic using the PCAP file
Step 7. Browse to the PCAP file location to be used to generate a script. The PCAP file is
available by setting the extension of the .shunra file to .zip and unzipping it.
Step 8. Specify the IP address of the host, observed during the PCAP recording
WARNING:
No file is generated if the IP address specified here does not match what is in
the file.
If you've forgotten what the IP addresses are, use a PCAP file reader utility
(Wireshark) to view the file
Step 9. Click Finish to complete the process and generate the VuGen script
WARNING:
In many cases the network traffic recorded is encrypted. Please refer to VuGen
User Guide (here) for more instructions on this topic.

6.2 Create a LoadRunner Scenario

The simplest way to move forward is to adopt the LoadRunner template from Perfecto and the base
VuGen template, available here.

You will notice in the LoadRunner scenario there are two groups, each of them based on a single real
device. The base script is called Real_Device_Template and it is meant to drive real devices in the
Perfecto cloud while load is applied on the backend.

You will want to leave the number of virtual users the way it is (1 virtual user) for each of them. You can
change the group name to reflect a logical name you want to call one of the devices in your test. You
may want to extend the duration of the test by clicking on Duration at the bottom-left side of the
screen. You then want to set the runtime parameters for your script. You do that by highlighting the
group row and doing a right-click, selecting Run time Settings. Below are all the VuGen parameters:

Page 14 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

Name Mandatory Description Example

PerfectoDeviceDUT The ID of the real device to be 12345
used
PerfectoDeviceName Custom name to precede the iPhone5
transaction names in the
LoadRunner report
PerfectoScriptName Script name in Perfecto cloud Public:example\demo
PerfectoUserID Your Perfecto cloud a@b.com
username. Usually this would
be your email address
PerfectoPassword Your Perfecto Cloud Password. abcd
Note you can provide
LoadRunner encrypted
password or non-encrypted
password (see here)
PerfectoCloud Your Perfecto Cloud URL Performance.perfectomobile.com
PerfectoMoreParameters Optional Additional Parameters you &param.value1=t&param.value2=5
may want to pass to the
Perfecto script. Note that the
Perfecto script has to have
these parameters in the
variables set as runtime. To
read more on this topic please
see here

Page 15 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

Repeat the same process for the 2nd device in your script. If you do not have a 2nd device to run
your script, delete the group.
Run your scenario once to ensure it works well.
Add to your scenario more groups based on the number of real devices you want to run in
parallel to the load. Ensure the script you are pointing to is the same Real_Device_Template
script.
Add to your scenario the script you created that emulates a mobile virtual user, as described
here
Add to your scenario a web load script. See here instructions how to create one.

Eventually you should have a complete scenario as shown below. Run your scenario, observe the
response time(s) for web, virtual mobile user and the various devices. For each Perfecto script execution
you can refer back to the execution center of your cloud to get the complete Perfecto report,
.PCAP/.Shunra file and device vitals.

Once the scenario is complete, you can click on Results->analyze results to gain access to the
LoadRunner reports

6.3 Best practices configuring real device testing from inside LoadRunner
The following does not replace the LoadRunner manual. This is an explanation how to configure the real
device sampling while emulated load is running.

You may want to create several scripts that will each run on a different device while the emulated load
is in effect. Ensure that for each script:

Only one virtual user is applied at all times

For the duration of the load test, repeat the script execution on the real device

Page 16 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

7 Reporting
Refer to the HP LoadRunner user manual for instructions on viewing load session reports. If the load
session was completed successfully, you should be able to obtain KPIs from real devices that are
materially different from those extracted by the emulated load.

Refer to the reporting interface inside Perfecto portal to view specific real device script execution
report.

Figure 6: HP LoadRunner Analysis - Transaction Summary in Summary Report

Figure 7: HP LoadRunner Controller - Interactive Schedule Graph

Page 17 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

8 Appendix A- Inside the VuGen template

In the Perfecto Performance Testing solution, the role of the VuGen script is to order the execution of
the Perfecto script on the Perfecto mobile cloud (via Perfectos HTTP API). Essentially this is a wrapper
for the MobileCloud script, facilitating the communication between HP LoadRunner and the Perfecto
MobileCloud engine.

The VuGen script includes HTTP API operations that communicate with the MobileCloud.

This section describes all of the VuGen sample script components. These include:
1. Execute MobileCloud Automation Script:
Send an execution query to the MobileCloud MCM to execute the Automation script
2. Check MobileCloud Automation Script Execution Status:
Query the MobileCloud for the Automation script execution status loop querying every 30
seconds until execution is complete
3. Collect and Analyze MobileCloud Script run Report:
Translate KPIs in MobileCloud report to HP BSM - Report KPI
Return MobileCloud HTML report to HP BSM in case of error - Report Snapshot OnError

Page 18 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

8.1.1 Execute MobileCloud Automation Script

Using the target device, execute the desired script.

Figure 8: Sample Script - Execute MobileCloud Automation

How to use:
See the Start New Execution Perfecto Mobile Help KB article for more information on how to use
this HTTP API operation.

Page 19 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

8.1.2 Check MobileCloud Automation Script Execution Status

Now that the script is running, it is possible to loop through the execution status until it has successfully
completed or experienced a timeout.

Figure 9: Sample Script - Check Execution Status of MobileCloud Automation Script

How to use:
See the Get Execution Status Perfecto Mobile Help KB article for more information on how to
use this HTTP API operation.

Page 20 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

8.1.3 Collect and Analyze MobileCloud Script run Report

Once the MobileCloud Automation script has completed, it is possible to retrieve the script report in
XML format, extract the timer(s) availability and value, and report this data to HP BSM.

In this sample script TimerA is the measured KPI.

Figure 10: Sample Script - Collect and Analyze MobileCloud Automation Script

How to use:
See the Download Execution Report Perfecto Mobile Help KB article for more information on
how to use this HTTP API operation.

Page 21 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

9 Appendix B- Creating a VuGen Web Performance Script

The following information does not replace the VuGen user guide available here.

Launch VuGen, select File->new. The script type should be Web-HTTP/HTML.

Figure 11: Create new script in VuGen

You can set the script name, folder etc.

Next click the record button:

Page 22 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

Figure 12: Record web transaction in VuGen

Select the browser you want to use, the URL you want to go to and hit start recording. Your browser
will show and any transaction will record until you hit stop.

Figure 13: VuGen recording toolbar

Next, VuGen will offer a dialog to replace parameters as you prepare for multi-virtual user test.

Figure 14: VuGen parameter replacement dialog

You can now run your script, add start/stop transactions.

You can now add your web load script to your Performance Center/LoadRunner scenario.

10 Appendix C - Using the HP Password Encoder

The HP tools offer the ability to encrypt passwords.

Go to Start > HP Software > HP LoadRunner > Tools > Password Encorder.
Enter your password, click Generate, and Copy to copy the encrypted password to your clipboard.

Page 23 of 24
 Mobile Performance User Guide: MobileCloud Automation and HP BSM

Figure 15: HP Tool password encoder

Page 24 of 24