What is performance testing

Performance testing is a Testing conducted to evaluate the compliance

of a system or component with specified performance
requirements. [IEEE]
OR
A performance test is a test of a complete system carried out under
reproducible and production-like conditions, where statistically significant
response times are measured over a period of time when the system is
under a known business emulation load
Performance Tests determine the runtime behavior of the
application and its supporting infrastructure,
under certain
conditions.
Performance Testing is used to measure several system
characteristics, such as processing speed, response time, resource
consumption, throughput, and efficiency

Types of performance tests

Performance Tests (Load Tests)
Testing an application against a requested number of users. The objective is
to determine whether the site can sustain this requested number of users with
acceptable response times.
Load Testing will simulate a real time user load on the application. The
simulated user load will be the normal load that the application is subjected or
the load expected on the application when it goes live.

WHY?
Load testing the application prior to production ensures application will be
stable and any performance issues can be addressed in pre-production
phase
To ensure that the system is responsive to user interactions and handles
extreme loading without unacceptable operational degradation
To test response time and reliability by increased user traffic.
To identify which components are responsible for performance degradation
and what usage characteristics cause degradation to occur

Types of performance tests

Volume Tests:
Volume testing is a typical load testing except that a large volume of data is
populated on the database.
Volume testing is used to check storage requirements of a system to check

how system will behave with large amount of data storage and its
processing

Volume testing is subjecting the program to heavy volumes of data. E.g.

A compiler would be fed a large source program to compile
An operating system job queue would be filled to full capacity
A file system would be fed with enough data to cause the program to switch
from one volume to another

WHY?
To study the application behavior when the database is populated with
production like data and to find the impact on the application response time
and database overall health.

Types of performance tests

Stress Tests
It involves subjecting the program to heavy loads or stress.
The idea is to break the system and to see what happens when the system is
pushed beyond design limits
In Stress testing a considerable load is generated as quickly as possible in
order to stress the application and analyze the maximum limit of concurrent users
the application can support or its break point
Stress Testing should answer the following questions:

Does the system degrade gently or does the server shut down
Are appropriate messages displayed? E.g. Server not available.
Are transactions lost as capacity is exceeded
Are certain functions discontinued as capacity reaches the 80 or 90 percent level

WHY?
Stress Testing ensures the application which is tested for expected load can take
on spikes in the load condition like increase in rate of transactions and study its
impact on the system resources and helps tune and configure the system
optimally.

Types of performance tests

Endurance Testing:
Endurance test is executed with expected user load sustained over longer
period of time with normal ramp up and ramp down time.
WHY?

Endurance testing helps in uncovering the performance bottlenecks

like memory leaks, which becomes visible when the system is
subjected to normal load for a prolonged time.

Types of performance tests

Availability Testing :
Testing for availability means running an application for a planned period of
time, collecting failure events and repair time.
WHY?

To ensure the service level agreement(SLA) by comparing the

availability percentage to the original service level agreement.

Types of performance tests

Scalability Testing:
Scalability Testing is loading the system with increasing load simulating the
expected business growth (user load growth) of the application down the
years
WHY?

To determine how effectively the system can scale to accommodate

the increasing load and for system capacity planning for
procurement of more resources down the line

Manual Load Test - Why Not?

Non Scalable

Large load tests simply cannot be

implemented

Inaccurate

Response times cannot be accurately measured

Real user activities cannot be consistently repeated

Incomplete

Multiple environments cannot be configured

Multi-Client types cannot be tested

Inconclusive

Does not help isolate bottlenecks

Results data is not centralized

Automated Load Testing

PERFORMANCE MONITORS

LoadRunner
User
Controller Simulation

Internet/
WAN

Web Server App. Server

Database

Replaces real users with thousands of virtual users

Generates consistent, measurable, and repeatable load,
managed from a single point of control
Efficiently identifies performance bottlenecks across all tiers/layers with
automated reporting and analyses

Performance Testing Parameters & Objectives

Load: The number of concurrent Virtual Clients trying to access the site.
Throughput: The average number of bytes per second transmitted from the AUT to
the Virtual Clients running Scenario during the last reporting interval
OR
The number of tasks completed per unit time. Indicates how much work has been
done within an interval
Response time: The time it takes the AUT to send the object of an HTTP request
back to Virtual Client, in seconds.
OR
The time elapsed during input arrival and output delivery
Transaction Time: The time it takes to complete a successful HTTP request, in
seconds. The time of a transaction is the sum of the Connect time, Send time,
Response time and Process time.
Average Latency: The time it takes from when a request is sent until the first byte is
received.
Utilization: The percentage of time a component (CPU, Channel, Storage, File
Server) is busy

Performance Testing Typical Workflow

Performance Testing : Performance Evaluation of application includes

Load, stress and volume testing. Parameters measured are concurrent
users, transaction thru put, response times, memory and CPU usage

Load Generation through Internet

Controller

Web Server

Load Generator

App. Server

Internet

Database

In ideal situation load should be generated in the local area

network
Predictability of result for repetitive cycle is difficult, since
constant performance of carrier network is not guaranteed

Performance Testing from Offshore

License @ offshore
Vendor Network

Port

Port
Load Generator

Client Network

VPN Connectivity
Response Time

Mediator

Controller/ VUGen

Fire
a testmonitors
from Controller,
information
be
Online
capturesScenario
data from
servers
& will
network
resources
and data
Load
Generator
resides
in clients
network,

Required
nos.
of
(Hundreds/Thousand)
virtual
user
perform
Controller
is
the
central
console
Establish
VPN
connectivity
between
Vendors
network
&
sent the
back
to
controller

Controller remain
connected
to
Agentvirtual
on Loadusers
Generator
and
generates
on Application
Under
transferred
to agent
on a Load
Generator
which
resides
in
Vendor
Network
from
real
life
transaction
on
system
to
emulate
production
clients network.
Controller
& Load
Communicates
Test
to
emulate
the performance
live
conditions
mediator
throughout
theGenerator
test
&
display
online
Monitors
communicate
Controller
through
specified
port on
which Online
theenvironment
test
is managed
& with

Monitoring
over
fire wall component is installed
on specified
port
firewall.
Controller
& Mediator
machine
firewall
usingon
Monitor
over
firewall
component.
parameters
on console.
monitored
on mediator PC to monitor servers over Firewall
also communicates
on
port.
Alternatively
all
servers
can
be
monitored
directly
from controller,
Load
is
generated
in
clients
LAN,
where
application
resides, but
At the end of the test, Results
are
stored
in a database
individual
port
for environmental
each server need to bedo
opened
in firewall
sorepository
that
other
impact
response
allowing users tofactors
generate not
reports
& perform
time

Offshore Model License @ Customer Location

Client Network

Vendor Network

Offshore machine
will have remote
access
of
the
Controller

Results
are
stored
in
a
database
repository,
allowing users to
generate reports
&
perform
analysis

VPN Network

Load Generator

Thousands of Vusers
perform
real
time
transactions
on
a
system to emulate
production traffic
Response Time

Terminal in
Mediator

Vendor
network

Controller/ VUGen
Controller is the central
console from which the
load test is managed &
monitored

Real time Monitors

Capture
performance data
across
all
tiers,
servers
and
network resources
and
display
information
on
Controller console

Loadrunner licenses from Client

Terminal client access to Controller for test execution &

management

Tool Support for Testing

Performance and Monitoring tools
Dynamic analysis tools
Analyze use of system resources such as memory, hard disk,
processor, ports by the application
Detect Memory Leaks
Performance test tools
Evaluate performance under expected conditions
Evaluate system behaviour under extreme conditions
Evaluate data handling capabilities of the system
Monitoring tools
Monitoring Live system for vital performance characteristics
In-line alerts when key parameters are slipping
Tools are : HP LoadRunner (Paid). Apache Jmeter (OpenSource)

LR architecture
Automation Overcomes resource limitations
Replaces testers with Virtual Users
Runs many Vusers on few machines
Controller manages the Vusers
Controller
Monitoring &
Analysis

Vuser
host

Meaningful results with analysis tools

Repeats tests with scripted actions

Web server

Database
server

Application
server
Load Generation

System Under Test

Database
server

LR Basic flow

Phase
1
Plan Load
Test

Phase 2
Create Vuser
Scripts

Phase
3

Phase
4

Create
Scenarios

Run
Scenarios

Phase
5
Analyze
Results

LoadRunner
VUGEN
LoadRunner
CONTROLLER&
AN ALY S I S

Tune
System Based
on Analysis

Pitfall to avoid
KNOW YOUR APPLICATION
Varying the data being used during a performance test is essential.
If the data does not change, then a database may cache the

information that is being constantly requested by multiple users, and will

cease performing database queries.
What happens when the technology changes??
Performance Testing is not linear! It is extremely risky to extrapolate
Performance Testing results based on smaller tests.
Remember to anticipate unsatisfactory results. Allow time to make
changes to the system and execute the test again if needed.