PEAK PERFORMANCE

Running your first load test with JMeter

Michael Kelly
11.24.2009
Rating: -4.00- (out of 5)

Software quality news and advice

Digg This! StumbleUpon Del.icio.us

In this tip, we'll take a look at running your first load test with JMeter. JMeter is a popular open
source Java application used for load testing. It supports a variety of technologies like,
HTTP/HTTPS, SOAP, JDBC, LDAP, JMS. These are all commonly used by teams looking to
introduce load on a server, network, or even specific objects. While it's not as full featured as some
commercial tools, it's a great tool with a lot of firepower for it's price -- free.

Mike Kelly We won't be looking at getting JMeter setup and running in

Editor's note:
This article was written using
JMeter 2.3.4 running on Java
1.5.0_20. The application used
in this article is Yahoo Search
since it's a publically available
application. The example uses a
very low load to simply illustrate
how JMeter works. If you would
like to test with larger loads, do
not use Yahoo. Instead, please
use your own applications.
test plan.
this article. That's not because it's difficult - typically all I
have to do to get it setup is download and double-click to fire
it up. It's because operating systems and versions change,
so the best place to get setup advice is on the "Getting
Started" section of the JMeter user manual on the JMeter
website.
First some terminology
JMeter tests are built around the idea of a test plan. Within a
test plan you can have Thread Groups, controllers, listeners,
timers, assertions, and other elements. Each test plan is a
performance test scenario - it's the series of steps JMeter
will execute when you run the plan. The following table gives
a brief overview of the basic elements you can include in a

Element Description
As with any load test, execution is multi-threaded. The
thread group element is what controls concurrent connection
Thread Group
to your application. In end user terms, it's your group of
users.
A sampler is the basic type of controller. Very simply,
samplers tell JMeter to send requests (HTTP, SOAP, etc…)
Samplers
to a server. There are also Logic Controllers, but we won't
be using one in this tip.
Listeners are what you use to access to the information
Listeners JMeter gathers while running. They give you the pretty
charts
Timers are how you set delays in JMeter. Timers fire before
Timers
each request that a thread makes.
Assertions, like in any test tool, allow you to check for
Assertions specific behavior as your test executes. Assertions provide
the standard pass/fail results.

For full details on these and other elements, please read the Elements of a Test Plan section of the JMeter user
manual. However, this short list is enough for us to get a simple test running. If you have troubles with any particular
element when trying to get it to work with your own application, the Component Referencesection of the JMeter user
manual is awesome for understanding each element and it's settings.

Building your first test

When you first open JMeter, you should be greeted with an empty Test Plan.

To add elements to a Test Plan, you can either right-click on it, or with it highlighted select Edit -> Add from the menu.
The option of either right-clicking to add elements or using the menu to add elements will be true for all the items we
talk about in this article. To reduce confusion, I'll be referencing object by asking you to right-click on them.

The first thing we need to get a test running is a Thread Group. To add a Thread Group, right-click on the Test Plan

and select Add -> Thread Group.

For our example, we don't want to run with a lot of load. Yahoo won't like that. So we're only going to test with 10
threads - or users. In addition, we'll set the ramp-up period to 2 seconds, and we'll have each user search ten times.
The only reason I'm having them loop ten times is so the charts you see at the end will actually contain some data in
them.

Once we have our Thread Group configured, we're ready to add a timer. We'll be using a Constant Timer between
transactions to space them apart. To add a timer, right-click on the Thread Group and select Add -> Timer ->

Constant Timer.

Set the thread delay to three seconds - or 3000 milliseconds. At this point we're ready for our HTTP request. To add

a request, right-click on the Thread Group again and this time select Add -> Sampler -> HTTP Request.

For the example, I want to have you run a simple search on Yahoo. To do that, set the Server Name to
'www.search.yahoo.com'. That tells JMeter what server you're hitting. Then for the path, I want you to enter '/search?
p=testing'. That's the same as if you went to Yahoo's homepage and did a search on the word 'testing'. You can
leave everything else on the page set to the defaults.

When I'm testing, I always want to know that what I did had the desired effect. So we're going to add an assertion to
our test. To do that, right-click on the HTTP Request element and select Add -> Assertion -> Response Assertion.

There are a number of options here that might be confusing. We're not going to look at them in detail, but what we
want to do is test to make sure we're getting search results back in our response. To do that, we're going to test the
'Main sample only' and we're going to be looking at the 'Text Response.' Looking at the text response means we're
looking at the HTML coming back.

Because we want to make sure it has results, we're doing a 'Contains' pattern match on the response. At the time I
wrote this article, the top response to the search was the Wikipedia entry for Software Testing. I simply searched for
the starting text to that definition which was 'Software Testing is an empirical investigation'.

You'll notice that there's a bold HTML tag in the check. That's because that's what you'll see if you do a view source
on the page. Always check for what you see in the source code - not what you see on the screen.

Next, we're going to add a couple of listeners so we can see our results. I like the Graph Results and View Results as
Table listeners. They both offer intuitive interfaces. To add a Graph Results listener, right-click on the Thread Group

and select Add -> Listener -> Graph Results.

For this example, you might want to uncheck the Deviation and Throughput checkboxes. Next, add the View Results

in Table listener. Right-click on the Test Group and select Add -> Listener -> View Results in Table.

You now have a really simple test ready to go. Go ahead and save your Test Plan.

Running your test

Once you've saved your test, you're ready to run. From the menu, select Run -> Start. It's very possible you won't
notice anything happen. Many first time users of JMeter don't. Here are some tips.

First, when you don't have a test running, you won't see any active users. In the upper right-hand corner there's a

user activity indicator. If you're not running a test, it looks like this:

When the test is running, this indicator changes to show you how many of your users are active, like in the following

figure:

Second, you can select your listeners to see what's happening during your tests. This information is updated real-
time. This is also how you view your results after your test is complete. In the figure below, I've selected the Graph

Results view after my test has executed:

You can see that in the Graph Results, it's charted each of the individual data points during the test (that's all those
black dots scattered around) along with two lines showing the median and average response times. This is a hand
chart to notice trends and odd behaviors.

Alternatively, the View Results in Table listener shown below is a more structured view of the data:

With this listener you can see stats for each data point, along with a graphic representation of the status of the
response. If you wanted to export data, this is likely one of the listeners you'd use to save off your results to import
into another tool like Excel.
Recording and running software load tests with JMeter

Mike Kelly
11.21.2009
Rating: --- (out of 5)

Software quality news and advice

Digg This! StumbleUpon Del.icio.us

In this tip we will take a look at recording performance tests using JMeter. In a previous tip on
"Running your first load test with JMeter, we looked at manually adding a Sampler for Yahoo
Search. However, you may find that for more complicated test scenarios, you'll want the ability to
record tests. This will help you save time when creating Samplers for your test plans. One easy way
is to create multiple Samplers and record with them simultaneously.

Mike Kelly Setting up a JMeter proxy

Author's note:
This tip was written using
JMeter 2.3.4 running on Java
1.5.0_20. The version of
FireFox used in 3.0.14. The
application used in this article is
Yahoo Search since it's a
publically available application.
The example uses a very low
load to simply illustrate how
JMeter works. If you would like
to test with larger loads, do not
use Yahoo. Instead, please use
your own application.
If you're not familiar with setting up a proxy, a quick
explanation might be in order. A proxy simply acts as an
intermediary for client requests to the various servers from
which you've requested resources. By setting up a JMeter
proxy, you're allowing JMeter to record the requests made
and use the information derived to create a test plan.
If you're following along with this example, you'll want to
start with a new test plan. To do this, add a proxy by right-
clicking on WorkBench. Next click Add -> Non-Test
Elements -> HTTP Proxy Server (as shown in figure 1
below).

Once the HTTP Proxy Server is added, select it in the left navigation tree, change the port from "8080" to "9090" and
ensure the checkbox labeled "Retrieve All Embedded Resources from the HTML Files" is checked. Your settings
should look similar to those shown in figure 2 below.
Next you'll need to set up your browser to point to a proxy server. For FireFox, you can find these settings by opening
Preferences, selecting the Advanced icon, then the Network tab, and clicking the "Settings…" button. This should
open a proxy configuration screen similar to the one shown in figure 3 below.
Select the "Manual proxy configuration" radio button and enter an HTTP Proxy of localhost on port 9090. When
finished, click OK and close FireFox preferences.

((Content component not found.))

Recording information using the proxy

You should now be set up to record a test using JMeter. To start recording your tests, you'll want to start the JMeter
proxy. To do this, navigate to the HTTP Proxy Server in JMeter and click the Start button at the bottom of the screen.
This will start up the server.

At this point, JMeter will record anything you do in the browser you configured. Sometimes it's helpful to have two
browsers opened. I'll often record tests using FireFox, and then also have an Internet Explorer or Safari browser open
on the side so I can look stuff up without recording the transactions in JMeter.

For this tip's example, we'll be testing Yahoo Search. Open a FireFox browser and navigate
to http://search.yahoo.com/. Once that page loads, search for "software testing." When the search results are
returned, navigate back to JMeter and take a look at what's been recorded. As shown in figure 4 below, you should
see a number of recorded Samplers under the WorkBench.
This should be enough for our example, so let's stop the proxy server. To do this, navigate to the HTTP Proxy Server
in JMeter and click the Stop button at the bottom of the screen. You're no longer recording. If you're completely done,
you'll also want to reset the proxy settings in your browser back to what they were before you set them to the manual
configuration.

Making sense of what was recorded

Previous JMeter tips: As you look at the list of recorded HTTP requests, you'll want to figure out
Running your first load test with what you're looking at. At the end of each item name, you'll see a file
JMeter extension indicating what's being requested. For example, the first few files
JMeter is a highly from Yahoo included a couple of javascript files (.js), a cascading style sheet
(.css), and a portable network graphic (.png). These files give you the basic
recommended tool for load
search functionality for the initial page we loaded.
testing open source Java
applications, as it supports
technologies from HTTPS - After that, you'll see a number of "gossip-us-sayt" requests. The purposes of
those requests might not be immediately obvious, but if you dig in just a bit by
SOAP. Learn how run your first selecting one of them, you'll see that the server name these requests go to is
JMeter test in this tip. " sugg.us.search.yahoo.net". That's the server that provides the spelling
suggestions while you're typing (as shown below in figure 5).
Recording and running software
load tests with JMeter
Recording JMeter load tests can
simplify the creation of
Samplers needed for your test
plans. This tip explains how to
record JMeter test results and
analyze the data generated.

After that series of requests, you'll find more javascript images and actual search results coming back. Those files
represent the search results page.

At this point, you can move each of these elements from the WorkBench into your Test Plan. You can take all the
Samplers recorded or you can select individual Samplers to move up. This allows you to record a series of Samplers,
and then mix and match them as needed to create custom scenarios.

As a quick example, add a Thread Group to your test plan and move the first six recorded Samplers up under that
Thread Group. Once those have been added, add a View Results Tree Listener as well (as shown in figure 6).
You can now run this test. If you do so (just run one thread for one loop), you'll see six responses come back
successfully in the View Results Tree Listener. If navigate those results and look at the response data, you should
find the content required to load the Yahoo Search page. The easiest example to recognize is the png file in the third
result set shown in figure 7 below.

At this point, you can now customize those Samplers as needed to create the test scenarios necessary to generate
your target load.
SOFTWARE TESTING

Tips for debugging your JMeter tests

Mike Kelly
11.19.2009
Rating: --- (out of 5)

Software quality news and advice

Digg This! StumbleUpon Del.icio.us

In this tip we'll take a look at some techniques for debugging your JMeter tests. In a previous tip on "Running your
first load test with JMeter," we looked at getting JMeter setup and we ran a simple test against Yahoo Search. If
you're new to JMeter, you'll quickly discover that once you get past very simple HTTP requests you'll need to be able
to get visibility into what's happening at runtime so you can debug and tune your tests. For the examples we'll look at
in this article, we'll be using the Yahoo Example Test Plan we created in the first tip in this series.

Use listeners to see what's happening during execution

Editor's note:
This tip was written using
JMeter 2.3.4 running on Java
1.5.0_20. The application used
in this article is Yahoo Search
since it's a publically available
application. The example uses a
very low load to simply illustrate
how JMeter works. If you would
like to test with larger loads, do
not use Yahoo. Instead, please
use your own application.
it.
As we saw in the previous tip when looking at the Graph Results and View
Results listeners, listeners are what you use to access to the information
JMeter gathers while running. There are a number of listeners available, and
while in the previous article we looked at two listeners that are handy for
results reporting, in this article we'll be focused on a couple of listeners more
appropriate for debugging.
While different listeners are useful in different situations, a common listener
that I use when debugging a JMeter test is the Assertion Results listener. The
Assertion Results listener logs assertion requests and reports failures of any
Assertions within your test plan. When I'm creating a Test Plan, I try to add an
assertion to each step for just this reason. Assertions are critical to the
debugging process, and even if you take them out later, they let you know
your test is doing exactly what you expected it to do when you were creating

For example, in the following figure, you can see successful assertions added to the Yahoo Example.

Each HTTP Request log message shown above represents a successful assertion from my Test Plan. In the figure
below, you can see what an error looks like.
In this case, JMeter shows not on the HTTP Request, which indicates an assertion was attempted, but also shows
that the assertion failed and why it failed. Using the checkboxes at the top of the screen, you can filter this to display
only errors if you like.

Using the View Results Tree listener

There's one particular listener that's a bit more powerful than the others when it comes to debugging: the View
Results Tree listener. The View Results Tree shows a tree of all sample responses, allowing you to drill down into the
response for any sample. Example data provided for HTTP requests includes thread runtime information, response
headers, the request and response data, and response assertion results if there was an assertion associated with the
request.

The following two figures show examples of response data shown in text and rendered HTML format for the Yahoo
Example Test Plan:
From this listener you can not only see the details of the request sent, but you can see all the details of the response
that came back. This is important, because if your assertions are off by as much as a single space they will be wrong.
Many times I end up going to the text response data and copying the results there that I'm looking for. In addition, if
you're testing web services and need to do XPath queries, the rendered XML view of the response data is quite
helpful.

Checkout the JMeter Component Reference for a bit more detail on some of the subtle nuances of using the View
Results Tree.

Using Debug Sampler and Debug PostProcessor

JMeter has two built in debugging elements: Debug Sampler and Debug PostProcessor. Think of these two elements
as helpers for providing more information in the View Results Tree listener.

The Debug Sampler generates a sample containing the values of all the JMeter variables and properties at runtime. If
you use the Debug Sampler in conjunction with a View Results Tree listener, you'll be able to navigate those values
as shown below.

The Debug PostProcessor on the other hand, creates a "subSample" with the details of the previous sampler
properties. The Debug PostProcessor shows additional details about the request performed as shown in the figure
below:

Both of these can be handy for figuring out what's happening under load. I know that when I'm looking at the
"standard" results shown in the View Results Tree listener, I'm often focused on debugging a single transaction.
However, when I switch to looking at the data in the Debug Sampler and Debug PostProcessor, I'm often trying to
figure out what's happening under load.

Save your response data to a file

By default, listeners display information graphically in the JMeter user interface. This is a low-information density
medium, and one you'll quickly need to move past to get at the details of your test results. One of the first things you'll
need to figure out when using JMeter is how to write to files.

As you can see in the two figures above, listeners in JMeter have a "Write results to file" field. If you provide a path to
file, you can capture detailed information in either XML or CSV format. To configure what JMeter logs for a listener,
click on the Configure button next to the file name.

If you save your results in XML, the output will look similar to the following:
By contrast, if you're looking for something that you can easily manipulate in Excel, the CSV format shown below
might be more appropriate:

Each format has it's own advantages, but I find myself using CSV more than XML. Each listener saves it's own
configuration settings separately. So be careful you don't assume if you change the settings for one listener that it will
cascade to all of them. It won't. This allows you to tune different listeners to collect different information.

Be sure your load generator has enough resources

Another common issue when load testing is figuring out if your load generators have enough resources for the testing
required. If your load generation machine(s) can't adequately accommodate the demands your JMeter test is putting
on them, you'll skew your execution results.
Previous JMeter tips:
Running your first load test with
JMeter
JMeter is a highly
recommended tool for load
testing open source Java
applications, as it supports
technologies from HTTPS -
SOAP. Learn how run your first
JMeter test in this tip.
Recording and running software
load tests with JMeter
Recording JMeter load tests can
simplify the creation of
Samplers needed for your test
plans. This tip explains how to
record JMeter test results and
analyze the data generated.
the command line, add a '-l' argument before the test plan name.
One place to start is to monitor your load generators while your tests are
running. If you're using Windows, this likely means just running Perfmon. If
you're on another platform, use the Perfmon equivalent for that operating
system - they all have one. If you start to notice performance issues on the
load generator, here are some things to look at:
• Threading: The more threads your load test uses, the harder your
load generator will work. This is likely the biggest factor to
performance. If you need large-scale load testing, consider running
your test across multiple machines. You know your load generator is
over burdened when you're pinging the CPU or when you're pushing
the upper limits of your memory capacity.
• The GUI: JMeter has a "headless" mode that allows you to run
without the GUI. Rendering all those pretty charts taxes the system.
To run without the GUI, use jmeter -n -t test.jmx test.jtl from the
command line.
• The number of Listeners: Just like the GUI, it takes processing
power to populate listeners. Even if you run without the GUI, you're
still collection listener data. This is the main reason I consider the
View Results Tree and Assertion Response listeners debug tools
instead of just something you'd always use. That have overhead.
You can either remove them from your test, or if you're running from

Next steps

In future tips, we'll look a bit more at some other ways you can use JMeter in your load testing. These tips will include
recording tests and tips for making sense of all the results. In the meantime, for more on JMeter, you can always
checkout the project website at jakarta.apache.org.