AutomatedSoftwareTestingMagazine November2010

An AUTOMATED TESTING INSTITUTE Publication - www.automatedtestinginstitute.
com
AutomAted ....... S t
oftware
SPECIAL ISSUE 2010
eSting
MAGAZINE
Performance Testing Virtualized System

excerPts from the mentorA/Ati WebinAr on virtuAlizAtion
The Winner Is...
leArn more About this yeArs Ati AutomAtion honors Winners
Moving Your TesT Lab

To The
Cloud
SPECIAL 2ND ANNUAL ATI AUTOMATION HONORS COVERAGE
Are You Contributing Content Yet?

The Automated Testing Institute relies heavily on the automated testing community in order to deliver up-to-date and relevant content. Thats why weve made it even easier for you to contribute content directly to the ATI Online Reference! Register and let your voice be heard today!
As a registered user you can submit content directly to the site, providing you with content control and the ability to network with like minded individuals.
Community Comments Box
>> Community Comments Box - this comments box, available on the home page of the site, provides an opportunity for users to post micro comments in real time. >> AnnounCements & Blog Posts - if you have interesting tool announcements, or you have a concept that youd like to blog about, submit a post directly to the Ati online reference today. At Ati, you have a community of individuals that would love to hear what you have to say. Your site profile will include a list of your submitted articles. >> AutomAtion events - Do you know about a cool automated testing meetup, webinar or conference? let the rest of us know about it by posting it on the Ati site. Add the date, time and venue so people will know where to go and when to be there.
Announcements & Blog Posts
Automation Events
learn more today at http//www.about.automatedtestinginstitute.com
AutomAted S t
oftware eSting
Special Edition 2010, Volume 2, Issue 4
Contents
Cloud
9
Virtualization CoVerage
Moving Your TesT Lab To The CLoud
Most organizations that employ in-house virtualization for testing are at least studying the potential benefits of the cloud and many are making the move, either in whole or in part. Read this article to learn what those reasons are and how you might go about moving your test lab to the cloud. By Randy Hayes
exCerTs FroM: six CriTiCaL suCCess FaCTors For PerForManCe TesTing virTuaLized sYsTeMs 38
Peformance testers must understand the fundamentals of virtualization, differences among virtual server technologies, virtual workload modeling, key performance bottlenecks of virtualized systems and adapting testing for virtualization. This article provide some excerpts from the webinar that provides all of this information fto you. By Dan Downing & Dion Johnson
2nd annual ati automation HonorS CoVerage

FinaLisTs and Winners announCed
Best Open Source UNIT Automated Test Tool 13 Best Open Source FUNCTIONAL Automated Test Tool 14 Best Open Source PERFORMANCE Automated Test Tool 15 Best Commercial FUNCTIONAL Automated Test Tool 18 Best Commercial PERFORMANCE Automated Test Tool 20 Best Automated Testing Book 21 Best Automated Testing Blog 21 Best Automated Testing Forum 21
12
and The aTi auToMaTion honors Winner is...
22
You know who the winners are, but do you really KNOW who the winners are? Learn more about them.
ColumnS & departmentS

ediToriaL
The best is Yet To Come Why the best is OK.
special edition 2010
The Automated Software Testing (AST) Magazine is an Automated Testing Institute (ATI) publication. For more information regarding the magazine visit http://www.astmagazine.automatedtestinginstitute.com
www.automatedtestinginstitute.com
Automated software testing magazine
Editorial
The besT is Yet To Come

by Dion Johnson
Thats right. The title of this article has the B word in it. Im not exactly sure when it happened, but at some point the word best became profanity. It became something to fear or shun. It became the favorite target for those that sought to set themselves apart, maybe even above the masses... the masses and their other destructive devices like testing standards, wikipedia, and ESPNs top ten plays. Are these things really destructive, though? They are definitely not perfect, and may occassionally present disagreeable information, but they have true value: the value of collective wisdom. Which brings me back to the word best. Best is not a word to be feared, but rather embraced, and that is exactly what the Automated Testing Institute has done by organizing the 2nd Annual ATI Automation Honors awards. ATI invited its community to get
intelligent enough to know that the tools and resources that have been named the best may not be the absolute best in every single circumstance that you can muster. Naming them the best doesnt end evaluation, but rather helps individual users and organizations begin and more
Then, the short list may be more closely evaluated to see if the best translates into the best for you. So, dont be afraid of the word best. Often if you shine a light on those that demonize the word, youll find that they actually do promote the best concept,
...at some point the word best became profanity

quickly conclude evaluations. Take, for example, the tool categories in which awards are presented. There are hundreds - maybe thousands - of tools that can fit into any given category. ATIs tool index alone has thousands of tools listed, and the index is far from exhaustive. So when people are looking for tools to meet their needs, where do they start? Generally organizations dont have time and resources to evaluate hundreds of tools. This is the benefit that something like the ATI Automation Honors provides. By highlighting tools that have been collectively named the best by a community of users that share needs that are similar to yours, you may now more readily refine your search and create a more effective short list. but without using the word best. Then if you probe deeper, youll find that the best that they promote is their own personal doctrine. Not only is this special issue of the AST magazine not afraid of the word best, it is dedicated to it. The 2nd Annual ATI Honors awards winners and finalists are highlighted throughout this issue which provides descriptions of as well as feedback from the awardees. In addition, this issue shines a light on the topic of virtualization. Mr. Randy Hayes delivers an engaging article on cloudbased virtualization entitled, Moving Your Test Lab To The Cloud. Also, excerpts are provided from the Mentora/ATI sponsored webinar entitled, Six Critical Success Factors for Performance Testing Virtualized Systems. special edition 2010
together and collectively identify a set of automated testing tools and resources that may be considered the best, with the belief that no one will mistake the word best to mean perfect. The IT discipline is filled with people that are 4
For more information, visit http://www.networking.automatedtestinginstitute.com

special edition 2010 www.automatedtestinginstitute.com Automated software testing magazine 5
Authors and Events

Whos In This Issue?
tests for the Altair 8080 in 1979. Then in 1983 he created the first automated test tool for the PC called AutoTester for DOS. Currently Hayes is co-founder and CEO of Capacity Calibration, a software company that specializes in cloud-based Web load testing. Incorporated in 2008, Capacity Calibration develops and markets test automation solutions for medium and large companies. Its flagship product is CapCal, a Web performance testing tool intended to help reduce the effort of script creation and maintenance.
randy hayes wrote diagnostic
AutomAted S t
oftware eSting
Managing Editor Dion Johnson Contributing Editor Edward Torrie Director of Marketing and Events Christine Johnson
A PUBLICATION OF THE AUTOMATED TESTING INSTITUTE
12 years of experience and has led hundreds of performance projects on applications ranging from eCommerce to ERP for companies ranging from startups to global enterprises. He has authored numerous presentations, white papers and articles on performance testing, presented at STAR, HP Software Universe, Software Test and Performance, and Workshop on Performance and Reliability conferences. He is widely recognized among the software testing and performance community, and holds positions on the Advisory boards at STP and WOPR among others.
dan downing has over
CONTACT US AST Magazine astmagazine@automatedtestinginstitute.com ATI Online Reference contact@automatedtestinginstitute.com
aTi honors Winners
Ati and Partner events

September 2011 Verify/ATI Conference
www.verifyati.com
are featured in this issue. Open Source, Commercial and other tools and resources get the recognition they deserve!
6 Automated software testing magazine
ATI Automation Honors

Celebrating Excellence in the Discipline of Software Test Automation See the Winners Now!
www.atihonors.automatedtestinginstitute.com
2nd Annual
Moving You
To The
by Hayes Randy
Clo
Moving to the Cloud
ur
oud
I
TesT Lab
n a recent study, IBM reported that the average enterprise devotes up to 50 percent of its entire technology infrastructure to development and testing, but typically up to 90 percent of the test infrastructure remains idle. Whether or not this describes your company, the fact remains that test automation has increased both the size and number of test labs, where a test lab is a room or space dedicated to computers used exclusively for testing. With the advent of virtualization, computers in a test lab have been made far more productive along with the people using them just the ability to store different configurations for use in testing has been a great boon for testers and developers everywhere. Cloud computing, of course, is really just a specialized way of using virtualization, but it relieves you of the need to purchase, store and maintain your own hardware and provides virtually unlimited capacity for when you need it. Most organizations that use VMware or Virtual PC are at least studying the potential benefits of the cloud and many are making the move, either in whole or in part, for all the right reasons. What those reasons are and how to go about making the switch are the topics of this article.
www.automatedtestinginstitute.com Automated software testing magazine 9
Types of Cloud Computing

Its become as tricky to define the cloud as it was to define artificial intelligence in the 80s (if you can remember that far back). Microsofts Azure, for example, provides just about everything but the Windows platform itself as a cloud-based service, while Amazon and Rackspace provide both the Windows and the Linux platforms as a service. For the purpose of test automation you will most often need the Windows platform as a service, since most functional test tools run on Windows (with some exceptions, of course). So for the purposes of this
Test labs with 10 to 20 or even more machines in them are quite common, even though they are going the way of the dodo bird as of 2010. Because they consume space and power and are usually idle a great deal of the time, they are a prime candidate for migrating to the cloud. In fact there is probably no better example of why cloud computing makes sense than a test lab for these very reasons.
to load testing and even then it wasnt enough. But for functional testing you may want to avail yourself of the added capacity of the cloud for a number of reasons. You might want to run a number of tests in parallel and thus cut down on the time needed to complete a lengthy test run. For cross browser testing you may want to run tests in all the major browsers instead of just one. Practically anything you can do with virtualization in house you can do in the cloud, but without the limits inherent in a lab of any size.
Benef of the Cloud its for Testing

Aside from the space and power consumption, what makes the cloud better than using virtualization in house? Well,
Choosing the Right Cloud

Everyone knows that Amazon was the first to introduce cloud computing to
Aside from the space and power consumption, what makes the cloud better than using virtualization in house? Well, the most obvious example is load testing
article we will confine our discussion to clouds that offer the Windows platform as a service, and thus compatible with tools like HP Mercury QuickTest Pro, Micro Focus SilkTest, Smart Bear Test Complete, Worksoft Certify as so forth.
the most obvious example is load testing, in which many machines are needed to generate load. How many is a matter of the size of the load needed, but I have seen test labs with 100 computers stacked from floor to ceiling dedicated entirely
Test labs with 1O to 2O or even more machines in them are quite common, even though they are going the way of the dodo bird as of 2O1O.
the masses and remains by far the most popular and well known cloud. The Amazon Compute Cloud (EC2) supports a wide range of Windows and Linux platforms in all sizes, and takes just minutes to configure and deploy custom test machine images. Rackspace has also been an innovator in this space and has recently announced support for Windows as well as Linux platforms. Pricing ranges from 2 cents an hour to over $1 an hour based on the size of the machine in terms of memory and disk space as well as the speed and number of CPU cores. One of the benefits of the Amazon cloud is something called Virtual Private Cloud, which allows you to configure your test machine instances to run inside your firewall. Unless the application under test runs on the client machine itself you will often need to have your test machines run on the internal network, and VPC supports this. A workaround might special edition 2010
Moving to the Cloud

be to install VPN software on your test machine images, but in any case it is a matter to consider. The location of data centers is also a factor, and Amazon has a presence in the USA, Europe and Asia. While this may not be as important for functional testing it is definitely a factor in load testing, where network latency is a key metric in test results. options you require and attach them to your instances on startup. The default is all ports closed, so even to use RDP and SSH you will have to create a security group with these ports open. When you spin up a cloud instance you are given a public IP address that anyone can access. This IP address lives for the duration of that instances lifetime so if you want to maintain that same address in the future Amazon provides elastic IP addresses which are static and allow you to associate them with any instance. If you need to make sure nobody outside your organization can access your test machines you should consider using VPC so that your instances run on a sub-net inside your private network. the difference between life and death fortunately most web apps are nowhere near as sensitive).
Making the Switch

In general, the procedure for setting up a test lab on the cloud is not much different than setting one up in a virtualized environment. Start with a plain vanilla system, install and configure whatever software you need, and save it. The magic, of course, is most visible when you need to set up and tear down your instances watching a dozen or more test machines come to life in a matter of minutes is fun to watch, and even more fun when they are churning out tests and test results. One limiting factor, however, is the availability of Windows platforms neither Amazon nor Rackspace supports anything but Windows Server editions, whereas VMware supports XP, Vista and Windows 7 as well. With Microsoft promising to provide Windows as a service on the Azure cloud, maybe they will lift this limitation. But in the meantime, many companies are employing a hybrid approach that uses a combination of virtual machines and cloud machines. The days of dedicating a physical computer to run a single test are gone have you noticed?
Some Drawbacks and Limitations

OK, so youve set up an account at the cloud provider of your choice and fired up an instance of a Windows or Linux machine. Now what? Well, like any vanilla virtual machine youve got to copy whatever files you need, including programs to install. For this you need to rely on FTP commands, which is cumbersome compared to copying files from one drive to another. You will need to use RDP for Windows or SSH for Linux to log in, then attach to yet another server to download your files. With Amazons Virtual Private Cloud things are easier since you can see the machine from inside your firewall, but still you may find yourself doing a lot of manual work and wish it could be automated for you. Finally, a remote connection is never as fast as a local one so screens may not appear as snappy as you are accustomed to. And yet for the number of advantages you get these limitations are worth it for most people.
Load Testing as Killer App for the Cloud

Hardly anything else comes close to load testing as a perfect reason to use the cloud. Depending on how big of a test you need to run you can easily require dozens if not hundreds of computers to generate the load. It simply is not practical for most companies to have this many computers on hand for testing purposes, especially if they are only needed from time to time. With cloud computing you can spin up as many as you need, use them for your tests, and tear them down again. There is a running debate about the accuracy of load tests that use the cloud for load generation (or any virtual machines for that matter) because of the slowing down of the system clock that can occur if the cloud providers resources become overly taxed. Its been our experience that this effect, while certainly a possibility, occurs rarely enough and is subtle enough to have no visible effect on the results. In any case, if you need 100 computers to run a load test what are your options other than using the cloud? Its one of those trade-offs that is easy to make for most companies (unless you are testing medical or transportation equipment where milliseconds can mean
Wait! Theres More! Want more of the cloud? Check out the Volume 2 Issue 1 (V2, I1) of the Automated Software Testing Magazine (March 2010 Issue). V2 I1 has articles on how automated testing is leading the cloud charge and how open source tools are adapting to the cloud. Find back issues at:
http://www.astmagazine. automatedtestinginstitute.com
What about Security

Security has always been a major concern, and even more so nowadays with so many cyber threats that exist. While the data centers themselves are physically secure, what you expose to the outside world is entirely up to you. With Amazon, for instance, the use of security groups allows you to set the firewall special edition 2010
11
2nd Annual ATI ............ ............ Automation ............ Honors

Pioneers in
Honoring tHe best
tools and resources
in automation
The 8 award categories are: best open source, unit, functional and performance test tools; best commercial unit, functional and performance test tools; best automated testing book, best blog and best forum site. Subcategories exist in each of the main tool categories and are specific to technology, such as C++, Web and Java. Also, each tool category has a subcategory that is not technology specific, but instead honors the tool irrespective of technology for its feature list, ease of use, support and other pertinent factors. 12 Automated software testing magazine
......................
u ap so
beck
Ju n it
Kent
e st et Te pl m o c derek
i
Ole Lensmar
langone
Best Open Source Unit Test Tool

Sub-category
C++
Google C++ Testing Framework Producer: Google
cfix Producer: Johannes Passing
Check Producer: Chris Pickett
Winner
JUnit Producer: David Saff, Erich Gamma, Erik G. H. Meade, Kent Beck
Finalist
TestNG Producer: Cdric Beust
Finalist
UISpec4J Producer: Rgis Medina, Pascal Pratmarty
Sub-category
Java
Winner
NUnit Producer: Charlie Poole
Finalist
csUnit Producer: Jacob Jake W. An-
Finalist
xUnit.net Producer: Brad Wilson, Jim Newkirk
Sub-category
.NET
derson, Piers Lawson, Markus Renschler and Manfred Lange
Winner
JUnit Producer: David Saff, Erich Gamma, Erik G. H. Meade, Kent Beck
Finalist
NUnit Producer: Charlie Poole cfix Producer: Johannes Passing
Finalist
TestNG Producer: Cdric Beust
Overall
Sub-category
Winner
Runner-up
Finalist
Finalist
13
Best Open Source Fu

Sub-category
Selenium Producer: OpenQA Sahi Producer: Tyto Software WebTest Producer: Canoo Watir
Producer: Bret Pettichord
Web
Winner
T-Plan Robot Producer: T-Plan Ltd.
Runner-up
Finalist
Finalist
UISpec4J Producer: Rgis Medina, Pascal Pratmarty
Jemmy Producer: Alexandre (Shura) Iline
Sub-category
Java
Winner
SWTBot Producer: Charlie Poole
Runner-up
Finalist
White Producer: Vivek Singh
Finalist
Finalist
Sub-category
Fitnesse Producer: Micah Martin, Robert Martin
T-Plan Robot Producer: T-Plan Ltd.
.NET
Winner
Finalist
Finalist
unctional Test Tool

Flash/Flex
Sub-category
Flex Monkey Producer: Gorilla Logic Flash Selenium Producer: Paulo Caroli, Sachin Sudheendra FunFx Producer: Aslak Hellesy, Stefan Magnus Landr, Peter Nicolai Motzfeldt
Winner
Selenium Producer: OpenQA Jsystem Producer: Ignis Software
Finalist
Fitnesse Producer: Micah Martin, Robert Martin
Finalist
Overall
Sub-category
Winner
Runner-up
Finalist
Finalist
For us, this award proves that some do consider free technologies instead of just blindly paying big bucks
Shura Iline
Jemmy
15
Best Open Source Performance Test Tool

Sub-category
JMeter
Producer: Apache Software Foundation
Grinder Producer: Philip Aston
C++
fwptt Producer: Bogdan Damian
TestMaker
Producer: PushToTest
Winner
Software Foundation
Runner-up
SoapUI Producer: eviware
Finalist
Finalist
Pylot
Producer: Corey Goldberg
Sub-category
JMeter Producer: Apache
Java
Winner
JMeter
Producer: Apache Software Foundation
Runner-up
SoapUI Producer: eviware
Finalist
Finalist
TestMaker
Producer: PushToTest
Overall
Sub-category
Winner
Runner-up
Finalist
Finalist
This prize goes out to the community. Thank you. Thank you so much guys.
16 Automated software testing magazine www.automatedtestinginstitute.com
Niclas and Ole
SoapUI
Are You Contributing Content Yet?

The Automated Testing Institute relies heavily on the automated testing community in order to deliver up-to-date and relevant content. Thats why weve made it even easier for you to contribute content directly to the ATI Online Reference! Register and let your voice be heard today!
>> Community Comments Box - this comments box, available on the home page of the site, provides an opportunity for users to post micro comments in real time. >> AnnounCements & Blog Posts - if you have interesting tool announcements, or you have a concept that youd like to blog about, submit a post directly to the Ati online reference today. At Ati, you have a community of individuals that would love to hear what you have to say. Your site profile will include a list of your submitted articles. >> AutomAtion events - Do you know about a cool automated testing meetup, webinar or conference? let the rest of us know about it by posting it on the Ati site. Add the date, time and venue so people will know where to go and when to be there.
Automation Events
learn more today at http//www.about.automatedtestinginstitute.com
Best Commercial Fu
Sub-category
SilkTest Producer: Micro Focus Certify Producer: Worksoft Rational Functional Tester Producer: IBM
Web
Winner
TestComplete Producer: SmartBear Software
Finalist
QA Wizard Producer: Seapine Software Qf-Test Producer: QFS
Finalist
Squish Producer: Froglogic
Sub-category
Java
Winner
Ranorex Producer: Ranorex GmbH
Runner-up
Finalist
Finalist
SilkTest Producer: Micro Focus
Sub-category
QA Wizard Producer: Seapine Software
.NET
Winner
Finalist
eggPlant Producer: TestPlant
Finalist
RIATest Producer: Cogitek Inc
Flash/Flex
Sub-category
Winner
Finalist
Finalist
unctional Test Tool

Web Services/Soa
Sub-category
SilkPerfomer for SOA Producer: Micro Focus Lisa Test Producer: iTKO SOATest Producer: Parasoft
Winner
TestComplete Producer: SmartBear Software
Finalist
Finalist
eggPlant Producer: TestPlant
Overall
Sub-category
Winner
Runner-up
Finalist
QA Wizard Producer: Seapine Software
SilkTest Producer: Micro Focus
Finalist
Finalist
I would like to thank ATI for organizing this event, and would like to thank all of you who voted for us. Jen Herget
CEO, Ranorex GmbH
special edition 2010 www.automatedtestinginstitute.com Automated software testing magazine 19 19
Best Commercial Performance Test Tool

Web Services Web/HTTPS
Sub-category
LoadRunner Producer: Hewlett Packard AQTime Producer: SmartBear Software Producer: IBM
Rational Performance Tester
SilkPerformer
Producer: Micro Focus
Winner
LoadRunner Producer: Hewlett Packard
Runner-up
Finalist
Finalist
SOATest Producer: Parasoft
Sub-category
QA Load Producer: Micro Focus
Winner
LoadRunner Producer: Hewlett Packard
Finalist
AQTime Producer: SmartBear Software
Rational Performance Tester
Finalist
SilkPerformer Producer: Micro Focus
Overall
Sub-category
Producer: IBM
Winner
Runner-up
Finalist
Finalist
Wed like to thank the Automated Testing Institute...for this prestigious award.
President, SmartBear Software
20 Automated software testing magazine www.automatedtestinginstitute.com special edition 2010
Derek Langone
Best Automated testing Book, Blog & Forum

Best Forum
SQAForums Producer: BetaSoft Inc
Knowledge Networks
AdvancedQTP Forum Producer: SOLMAR
One Stop Testing
TekTips
Producer: Tecumseh Group
Producer: Vyom Technosoft Pvt. Ltd.
Winner
Runner-up
Autonomicon Producer: Nick Olivo
Finalist
Finalist
Corey Goldberg Blog Producer: Corey Goldberg
Automated Testing and QA Producer: Peter Kartashov
Best Blog
Winner
Finalist
Runner-up
Testlabs.com Producer: TestLabs
Finalist
I.M. Testy Producer: Testing Mentor
Finalist
Expert Shell Scripting (Experts Voice in Open Source)
Best Book
QTP Unplugged Producer: Tarun Lalwani
Software Performance and Scalability: A Quantitative Approach
Producer: Henry H. Liu
Producer: Ron Peters
Winner
Finalist
Finalist
Automated software testing magazine 21
And the AtI AUTOmATIOn HOnORS Winner is...

TI A uto A tion ma nors Ho
The AST Magazine Focuses on the ATI Automation Honors Award Winners
he ATI Automation Honors awards accept nominations from you, the automation community, and rely on your votes to determine who the best of the best really are. Awards are presented in several categories from the corporate level down to the practitioner level, resulting in a celebration of the best and most relevant new releases of tools, and the best print and online resources. Weve told you who the winners are, but do you really KNOW who the winners are? This article provides you with the opportunity to learn more about this elite group of tools and resources.
22 Automated software testing magazine www.automatedtestinginstitute.com special edition 2010
2nd AnnuAl ATI AuTomATIon Honors
We dont think there is (or will be) a single best framework - you have to pick the right tool for the particular task you are tackling.
GoogleTest FAQs
http://code.google.com/p/googletest/wiki/FAQ
Google C++ Testing Framework

oogle C++ Testing Framework, also known as Google Test, is based on the xUnit architecture, and helps write C++ tests on a variety of platforms, including Linux, Windows, and Mac. Its features include the ability to isolates tests by running each of them on different objects and grouping related tests into test cases that have the ability to share data and subroutines. Google Test is also used by the following notable projects: The Chromium projects The LLVM compiler Protocol Buffers (Googles data interchange format)
JUnit
Unit is an open source unit testing framework for the Java programming language. Created by Kent Beck and Erich Gamma, JUnit has been important in the creation of test-driven development. JUnit features include: Assertions for testing expected results, test fixtures for sharing code, common test data and test runners for running tests. JUnit has been ported to other languages including ATI Honors Winner NUnits C#.
Honors received
1st Place Winner: Best Open Source UNIT Automated Test Tool - Java 1st Place Winner: Best Open Source UNIT Automated Test Tool - Overall
Honors received
1 Place Winner: Best Open Source UNIT Automated Test Tool - C++
st
Last eLigibLe reLease

4.8.1

1.4.0
Producer/Project admin
David Saff, Erich Gamma, Erik G. H. Meade, Kent Beck
Google
Fun Fact(s):
JUnit has also won the Best Java Performance Monitoring/Testing Tool award in the 2001 and 2002 JavaWorld Editors Choice Awards (ECA)
Fun Fact(s):
The GoogleTest FAQs page begins answering one of its questions by saying, First, lets say clearly that we dont want to get into the debate of which C++ testing framework is the best.. The automation community has, however, decided to settle this debate for them by naming them the best in the C++ category. While it is true that no single tool is the best for every single purpose or person, GoogleTest has earned its way onto anyones short list of tools to consider for unit testing C++ applications.
Web Presence/Project Home:
http://code.google.com/p/googletest/ special edition 2010
honor
Kent Beck (Emailed Statement)
http://sourceforge.net/projects/junit/files/
Thank you for the
Extreme Programming Creator and JUnit developer
23
Its very exciting that the Automated Testing Institute
this way. Thanks for noticing folks!

Charlie Poole (Acceptance Speech Excerpt)
NUnit Representative
chosen to honor nunit in

has
Sahi
NUnit
Unit is a unit-testing framework for all .Net languages that is based on the xUnit architecture. The current production release, version 2.5, is the sixth major release, is written entirely in C# and has been completely redesigned to take advantage of many .NET language features, for example custom attributes and other reflection related capabilities. Charlie Poole, NUnit Developer has identified the following top ten reasons to try NUnit 2.5 (http:// nunit.net/blogs/?p=66#comments): Data-Driven Tests, Theories, Inline Expected Exception Tests, Generic Support, Lambda Support, Out-of-Process Execution and Runtime Selection, PNUnit, Source Code Display, Timeout and Delayed Constraints, Threading Attributes.
ahi is a tool for automation of web applications. Developed in Java and JavaScript, it uses simple JavaScript to execute events on the browser. Features include in-browser controls, text based scripts, Ant support for playback of suites of tests, and multithreaded playback. It supports HTTP and HTTPS. Sahi runs as a proxy server and the browser needs to use the Sahi server as its proxy. Sahi then injects JavaScript so that it can access elements in the webpage.
Honors received
runner uP: Best Open Source FUNCTIONAL Automated Test Tool - Web

v3 (2009-09-22)
Honors received
1st Place Winner: Best Open Source UNIT Automated Test Tool - .NET runner uP: Best Open Source UNIT Automated Test Tool - Overall
Tyto Software
Fun Fact(s):
According to Sahis website, products like ThoughtWorks Studios Twist and the commercial version of PushToTests TestMaker (an ATI nominee in the Open Source Functional Category) leverage Sahis technology to achieve easy automation across browsers.

2.5.3 (Version 2.5.3.9345)
Charlie Poole
Fun Fact(s):
NUnit, like many other xUnit frameworks, was originally transliterated from JUnit. The 2.0 release was a total redesign that was created as though it was done completely in C# from the start.

http://sahi.co.in/w/

http://www.nunit.org
Your acknowledgement [makes] us even more focused

Narayan Raman (Acceptance Speech Excerpt)
Sahi Representative
Had there been another testing tool that could test Javascript uI features in all browsers on all
platforms, believe me, I would have
Jason Huggins (Selenium Blog Interview Excerpt)
saved lots of time *not* writing my own tool.

Jemmy
emmy is a tool allowing you to create automated tests for Java GUI applications. Tests are written in Java, using Jemmy as a regular Java library.
Co-creator of Selenium http://googletesting.blogspot.com/2007/09/seleniums-inventor.html
Selenium
elenium is a suite of tools used to test web applications. The suite includes:
Selenium IDE Selenium RC Selenium Grid
Honors received
runner uP: Best Open Source FUNCTIONAL Automated Test Tool - Java
It provides a record/playback feature along with a test domain specific language (DSL) to write tests in a number of popular programming languages, including Java, Ruby, Groovy, Python, PHP, and Perl. Selenium deploys on Windows, Linux, and Macintosh platforms. Selenium was originally developed by Jason Huggins, who was later joined by other programmers and testers at ThoughtWorks.

Nov 20, 2009
Shura Iline
Fun Fact(s):
JSystems solution for testing Swing application is composed of an agent that uses a Jemmy package to simulate user operations on the Swing UI components. So, once again one ATI Automation Honors finalist is linked to another. Great tools stick together! (http://www.jsystemtest.org/?q=node/81).
Honors received
1st Place Winner: Best Open Source FUNCTIONAL Automated Test Tool - Web 1st Place Winner: Best Open Source FUNCTIONAL Automated Test Tool - Overall

1.0.5

https://jemmy.dev.java.net/
OpenQA
Fun Fact(s):
People often think of Selenium as one tool, but it is actually a suite of tools. There are three main tools in the suite, but in total, there are 8!
For us this proves some do consider free technologies

Shura Iline (Acceptance Speech Excerpt)
Jemmy Representative

http://seleniumhq.org/ special edition 2010
Enables customers,
and their programmers software to should
testers,
learn do, what to and
automatically
compare that to what it
Fitnesse One Minute Description Excerpt
http://www.fitnesse.org/FitNesse.UserGuide.OneMinuteDescription
actually does do.

Fitnesse
itNesse is a lightweight, open-source framework that enables customers, testers, and programmers on a software development team to collaborate on and share what the software under test should do, create acceptance tests for checking expected behavior against actual behavior, and automate these acceptance tests.
T-Plan Robot
-Plan Robot is a Java Functional automated testing tool that is developed on generic image based testing principles, and provides a human-like approach to software testing. Founded in 2005, the VNCRobot project had delivered more than 20 binary releases of free software for automated black box GUI testing over the RFB protocol until July 2009 when it was acquired by T-Plan Ltd and renamed to T-Plan Robot.
Honors received
1st Plance Winner: Best Open Source FUNCTIONAL Automated Test Tool - .NET
Honors received
1st Place Winner: Best Open Source FUNCTIONAL Automated Test Tool - Java

20100303

2.0.3
Micah Martin, Robert Martin
T-Plan Ltd.
Fun Fact(s):
Fitnesse is really a front end for developing and running FIT (Framework for Integrated Test) tests, and is a web server, wiki and automated testing tool all rolled into one.
Fun Fact(s):
The original version, VNCRobot 1.x, was released as freeware, but came into the world of open source in T-Plan Robot version 2.0 (http:// www.t-plan.com/robot/docs/faq.html#license). Although open source, the T-Plan Robot source code is not open for any public contributions. Instead, if you, Jane/Joe Automator, want to update the functionality, you have to do so via plug-ins. (http://www.t-plan.com/robot/docs/ faq.html#project)
http://sourceforge.net/projects/fitnesse/
http://www.t-plan.com/robot/
Thanks for this...

Charlie Wheeler (Emailed Statement)
T-Plan Robot Sales & Marketing Manager
on behalf of the Jsystem team and community I want to say thank you for giving us this award.
Yoram Shamir (1st annual awardsAcceptance Speech Excerpt)
CEO Ignis Software
JSystem
System is a professional open-source tool for writing and managing automated system tests. Written in Java, JSystem is based on several open source java projects, including JUnit for writing tests, Ant used to execute scenarios, and Eclipse as the development environment. Jsystem is comprised of the following components: JSystem Services (Java API), JSystem Drivers, JSystem GUI Interface (JRunner), JSystem Agent and the JSystem Eclipse plug-in
FlexMonkey
lexMonkey is an Adobe AIR application used for testing Flex and AIR based applications. Providing the functionality to record, playback and verify Flex UI interactions, FlexMonkey also generates ActionScriptbased testing scripts that you can easily include within a continuous integration environment.
Honors received
runner uP: Best Open Source FUNCTIONAL Automated Test Tool Flash/Flex
Honors received
runner uP: Best Open Source FUNCTIONAL Automated Test Tool - Overall

4.0

5.4.0
Gorilla Logic
Ignis Software
Fun Fact(s):
Based on the Adobe automation API. Stuart Stern stumbled upon it and built functionality on top of it to create Flex Monkey.
Fun Fact(s):
People often think of Selenium as one tool, but it is actually a suite of tools. There are three main tools in the suite, but in total, there are 8!
http://www.gorillalogic.com/flexmonkey

http://www.jsystemtest.org/
[are going to] over as uI development platforms of Stuart Stern (TheFlexShow podcast excerpt) choice. Gorilla Logic President/CEO, Creator of Flex Monkey
like flex ultimately take

Things
http://www.theflexshow.com
The
hands
Grinder puts control right into developers

and provides
Philip Aston (Java Performance Tuning interview excerpt)
visibility of issues.
JMeter
early
Creator of The Grinder http://www.javaperformancetuning.com/news/interview031.shtml
The Grinder
he Grinder is a Java load testing framework that makes it easy to run a distributed test using many load injector machines. Key features Generic Approach Load test anything that has a Java API. Flexible Scripting Tests are written in the powerful Jython scripting language. Distributed Framework Mature HTTP Support
Meter is designed to load test functional behavior and measure performance. JMeter may be used to test performance on static and dynamic resources (Servlets, Perl scripts, Java Objects, Queries, and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types.
Honors received
1st Place Winner: Best Open Source PERFORMANCE Automated Test Tool - Web 1st Place Winner: Best Open Source PERFORMANCE Automated Test Tool - Web
Services/SOA
1st Place Winner: Best Open Source PERFORMANCE Automated Test Tool - Overall
Honors received
runner uP: Best Open Source PERFORMANCE Automated Test Tool - Web

2.3.3
Apache Software Foundation

3.3
Fun Fact(s):
Stefano Mazzocchi of the Apache Software Foundation was the original developer of JMeter. He wrote it primarily to test the performance of Apache JServ (a project that has since been replaced by the Apache Tomcat project).
Philip Aston
Fun Fact(s):
The Grinder was originally developed for the book Professional Java 2 Enterprise Edition with BEA WebLogic Server by Paco Gmez and Peter Zadrozny. It soon found a life outside of the books pages and now finds itself being honored as a full fledge testing tool.

http://grinder.sourceforge.net/
http://jakarta.apache.org/jmeter/index.html
Jmeter can be used to simulate a heavy load on a server

JMeter Site Introduction
http://jakarta.apache.org/jmeter/index.html
Wed like to thank you guys at the ATI Automation Awards. And we also have to thank our faithful users which are the main key factor to our being here.
Ole & Niclas (Acceptance Speech Excerpt)
Co-creators of soapUI
soapUI
oapUI is an open source Web Service Testing Tool for Service Oriented Architecture (SOA). Its functionality mainly covers web service inspection, invoking, development, simulation and mocking, functional testing, load and compliance testing. Productivity enhancement features can be found in the soapUI pro version. soapUI was initially released on Sourceforge during September 2005 and is distributed under the terms of the GNU Lesser General Public License with the application and source code also provided on the basis of the Open Source GNU Lesser General Public License.
SilkTest
ilkTest is a functional and regression automated testing tool that promotes collaboration among different stakeholders such as QA professionals, business analysts and developers, allowing various groups to participate in a continuous quality process. It allows scripts to be developed visually (no script), or with VB.NET, C# or Java environments.
Honors received
1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - Web

V2009 R2
Micro Focus
Honors received
runner uP: Best Open Source PERFORMANCE Automated Test Tool - Web Services/SOA runner uP: Best Open Source PERFORMANCE Automated Test Tool - Overall
Fun Fact(s):
At one point in time SilkTest was produced by Segue Software, but Segue was acquired by Borland in 2006. Borland was then acquired by Micro Focus International in 2009. So a lot of people were confused about where to go for the tool. Adding to the confusion is the fact that as of the date of this publication, you can find SilkTest listed on both the Microfocus site (http://www.microfocus.com/ products/silk/silktest/index.aspx) as well as a separate Borland site (http://www.borland.com/us/products/silk/silktest/index.html). In their defense, however, the Borland site has a line of subtext under the Borland name reading, A Micro Focus Company.

3.0
eviware
Fun Fact(s):
With more than 2,000,000 downloads, it is the most used tool for SOA testing.

http://www.soapui.org/
Web Presence/Project Home: http://www.microfocus.com/products/silk/silktest/index.aspx
Fastest script creation of any functional test automation tool.

Micro Focus Site
http://www.microfocus.com/products/silk/silktest/index.aspx
Wed like to thank the Automated Testing Institute
for selecting testing tool

TestComplete
Derek Langone (Acceptance Speech Excerpt)
SmartBear President
our automated
TestComplete
estComplete is an automated testing tool that lets you create, manage and run tests for any Windows, Web or Rich Client software. TestCompletes script-free keyword-driven testing, test visualizer and user-friendly interface makes learning automated testing available for a wider audience of testers.
QA Wizard
A Wizard Pro automates the functional and regression testing of Web, Windows, and Java applications, and load testing of Web applications. It uses an object-based record and playback engine that works with a repository in which objects may be stored. It also has features to aid in data-driven scripting and validation of expected results.
Honors received
runner uP: Best Commercial FUNCTIONAL Automated Test Tool - Java
Honors received
1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - Java 1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - Overall

2010.1
Seapine Software

7.52
Fun Fact(s):
QAWizard is no stranger to awards. It has won several APEX awards that honor excellence in publications. They won for their users guides and product tours. In addition, the QA Wizard documentation also received a STC (Society For Technical Communication) award. The tool was also a finalist for the 15th Annual Jolt Awards.
SmartBear Software
Fun Fact(s):
TestComplete has a connection with competitor SilkTest. According to the Smartbear site, Joe Krivickas, CEO of Smartbear, was the CEO of Segue Software. Seque was the company that owned the Silk product line, including SilkTest (now a MicroFocus product), which became used by tens of thousands of development teams. Joe may not have been the biggest fan of Borlands takeover of Silk however, since according to the site, he felt the Borland takeover of Segue didnt allow him to finish what he started. As he moved on with Smartbear, he took his Segue-collaborator, Ian McLeod, the previous head of R&D at Segue, with him. Ian is now an Executive VP at Smartbear. Small world.
f u n c t i o n a l
testing with load
Rick Riccetti (Newsletter Excerpt)
Seapine Software President and CEO http://www.seapine.com/pr.php?id=182
http://www.seapine.com/qawizard.html
...now easily combine
http://www.automatedqa.com/products/testcomplete/ 30 Automated software testing magazine
ranorex is proud
to accept this award and to continue to provide a powerful testing tool to the industry
Jen Herget (Acceptance Speech Excerpt)
Ranorex CEO
Ranorex
anorex is a Windows UI test automation framework for testing many different application types including Winforms, WPF, Silverlight, Adobe Flash/ Flex, Web 2.0, Ajax, .NET and Java. The Ranorex Library is the powerful core of the Ranorex Automation Framework. Regardless how the Ranorex Library is used to implement tests, a Ranorex test can be implemented as a simple EXE file. In addition, ranorex also has key features such as capture/replay and an object repository browser.
SilkPerformer for SOA
icro Focus SilkPerformer offers a cost-effective, enterprise-class tool for software application performance and load testing, delivering automated software load, stress and performance testing in an open and sharable model.
Honors received
1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - Web/Web Services

v2009 R2
Honors received
1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - .Net 1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - Flash/Flex runner uP: Best Commercial FUNCTIONAL Automated Test Tool - Overall
Micro Focus
Fun Fact(s):
Through an interesting twist of fate, SilkPerformer and its competitor QALoad are now owned by the same company. The Silk product line was previously owned by Borland, while QALoad was a Compuware product. In 2009 when Micro Focus acquired Borland and the Quality Solutions portion of Compuware, SilkPerformer and QALoad came under the same umbrella. The only question that remains, is how long Micro Focus will continue to support two different tools that have the same purpose

2.2.2
Ranorex GmbH
Fun Fact(s):
Ranorex is so dedicated to automation that they even went as far as to automate the ATI Honors voting process for its users! The created an automated script that their users could download that automatically selected Ranorex in the pertinent categories. The user still had to manually fill out the CAPTCHA (which is the whole point of CAPTCHA), but this extra effort really shows that Ranorex has automation coursing through its very veins!
or scalability
problems early
Micro Focus Site
http://www.microfocus.com/products/silk/silkperformer.aspx
Web Presence/Project Home: http://www.microfocus.com/products/silk/silkperformer.aspx
finding performance

http://www.ranorex.com/ special edition 2010
31
At smartBear we are dedicated to providing best in breed technology at an affordable price for the QA and development teams that are focused on software Quality
Derek Langone (Acceptance Speech Excerpt)
SmartBear President
AQTime
QTime is a perfor mance profiler and memory allocation debugger for 32-bit and 64-bit Windows and .NET-connected applications. AQtime can be integrated into Microsoft Visual Studio, Borland Developer Studio or CodeGear RAD Studio or it can run as a stand-alone application. With AQtime, you can not only find a bottleneck, but also determine what caused that bottleneck.
LoadRunner
ewlett-Packard(HP) Loadrunner (LR) is a performance and load testing product for examining system behavior and performance, while generating actual system load. LoadRunner can emulate hundreds or thousands of concurrent users to put the application through the rigors of reallife user loads, while collecting information from key infrastructure components.
Honors received
1st Place Winner: Best Commercial PERFORMANCE Automated Test Tool - Web/HTTPS 1st Place Winner: Best Commercial PERFORMANCE Automated Test Tool - Web Services/SOA 1st Place Winner: Best Commercial PERFORMANCE Automated Test Tool - Overall
Honors received
runner uP: Best Commercial PERFORMANCE Automated Test Tool - Web/HTTPS runner uP: Best Commercial PERFORMANCE Automated Test Tool - Overall

6.4

9.5
Hewlett Packard
SmartBear Software
Fun Fact(s):
HP LR holds 77% load testing marketshare worldwide!
Fun Fact(s):
Borland, according to a SmartBear case study is on the record as having used AQTime to tune Delphi 2006. This is particularly interesting given that Joe Krivickas, CEO of Smartbear, used to be CEO of Seque, the company that produces a competing line of automation products (the Silk products) and was bought by Borland. See TestComplete Fun Fact for more information.
http://www.automatedqa.com/products/aqtime/
https://h10078.www1.hp.com/cda/hpms/display/main/hpms_ content.jsp?zn=bto&cp=1-11-15-17^8_4000_100__
...prevent costly
Hewlett Packard Site
application performance problems
I would like to thank all those that voted for QTP unplugged as the best book in [the] testing automation category.
Tarun Lalwani (Acceptance Speech Excerpt)
Author
QTP Unplugged
P QuickTest Professional is a functional test automation tool. It supports a Record and Playback framework out of the box, where we can record and capture our interactions with the application under test and then replay those actions later. With this book you will learn - Basic concepts of QTP - Working without Object repository using Descriptive Programming - Advanced concepts of QTP - Working with external tools Microsoft Word, Outlook, Excel Integrating QTP Scripts with Quality Center - Real life Automation problems and their solutions
Automated Testing and QA

utomated Testing and QA (At4QA) is a blog focused strictly on software testing topics, and more specifically automated software testing topics.
Honors received
1st Place Winner: Best Automated Testing Blog
Producer/Primary bLogger
Peter Kartashov
bLogger ProFiLe
* I sense this world as I am & * Quality is like health - nobody can buy it...
Honors received
1st Place Winner: Best Automated Testing Book
Fun Fact(s):
On August 31, Automated Testing and QA published a blog post that referenced 5 other blogs. 4 out of 5 of the blogs are associated with past and present ATI Automation Honors winners and finalist including: Google Testing Blog, SmartBear Blog, I.M. Testy Blog, Corey Goldberg Blog.
autHors
Tarun Lalwani
PubLisHer
KnowledgeInbox

http://at4qa.blogspot.com/
Im [an] enthusiast of test automation and this is why Im running At4QA Peter Kartashov (Acceptance Speech Excerpt) blog.
At4QA Primary Blogger
33
Its wonderful to see that some organization has awards focused on software testing categories
AJ Alhait (1st Annual Acceptance Speech Excerpt)
SQAForums
SQAForums
Autonomicon
utonomicon is a blog that features thoughts on software automation and software demonstrations.
QAForums is an internet forum site that is created by BetaSoft, Inc.
BetaSoft Inc. is a privately held corporation based in San Jose, California. Founded in 1995, its main business function is providing software testing and quality assurance services throughout the United States and Canada.
Honors received
runner uP: Best Automated Testing Blog
Honors received
1st Place Winner: Best Automated Testing Forum
Producer/Primary bLogger
Nick Olivo
Producer/admin
Betasoft Inc.
bLogger ProFiLe
Im an SE, an automated tester and a tech writer rolled into one.
http://www.sqaforums.com/ubbthreads.php
http://autonomicon.blogspot.com/
AdvancedQTP Forum
O L M A R sponsors a leading resource hub for HPs QuickTest Professional - www. AdvancedQTP.com, which helps - for free thousands of professionals with its top class team of experts and materials. SOLMAR is a global provider of professional services that range from consultancy and integration, thru development and project management, to training, job placement and outsourcing.
Established in January 2008, SOLMAR is already a leading global provider of training and coaching in the field of testing automation.
Honors received
runner uP: Best Automated Testing Forum
Producer/admin
Solmar

http://www.advancedqtp.com/
http://www.googleautomation.com
special edition 2010 www.automatedtestinginstitute.com Automated software testing magazine 35
37
r e m c o x Fr e
for Performance Testin With

Dan
Downing
Six Critical Succe

t
&
Dion
Johnson
Webinar
On October 26 the Automated Testing Institute co-sponsored a webinar with Mentora. Below are excerpts from that webinar.
Note:
For the full webinar visit:

impact, challenges and requirements surrounding virtualization and found that the adoption of server virtualization is expanding rapidly. As part of their study, IT decision makers were polled and asked to reveal the percentage of servers they had virtualized in 2009 along with the percentage of servers that they planned on having virtualized in 2011. For 2009 the survey peak was already at around 21 to 50 percent, which is pretty high when you think about how pervasive huge racks of dedicated servers was just a few short years ago. But then looking forward to 2011 the survey reveals that the peak will be closer to 70% range.
http://webinars.automatedtestinginstitute.com
For questions related to the webinar or performance testing of virtualized systems, feel free to contact Dan Downing at:
ddowning@mentora.com STATe OF AdOPTION OF SeRveR vIRTUALIzATION
Dion Johnson The reason I, as representative of the Automated Testing Institute or ATI, wanted to participate in this webinar was because server virtualization is truly a great topic to cover. The very purpose of ATI is to provide a comprehensive resource of timely information relative and pertinent to testing and test automation, and there are few topics more timely and pertinent right now than virtualization. In 2009 the research firm, Forrester Consulting evaluated the 38 Automated software testing magazine
ess Factors
[As] an IT consultant and as the managing editor of the Automated Software Testing Magazine, these findings are definitely in line with what Ive seen personally and in line with the feedback that Ive received from our readership. So there is clearly a strong tide headed towards virtualization that has already begun and promises to continue into the future. So as testers, specifically performance testers, we need to make sure were ahead of the game, because it promises to not only change how we test, but what we test. Because as virtualization moves more to production environments, we wont just be using virtualization in our development environments to test software applications, but instead well be doing more testing of the virtualized system itself. Plus, this move to virtualization also promises to introduce new or enhanced challenges that we may face
Webinar Recap
ng Virtualized Systems
ANATOMy OF A vIRTUAL SySTeM

Dan Downing So moving right on to the anatomy.
To learn about the top challenges feared most by IT decision makers, view the full webinar at:
This little diagram in the center of the slide [above], were going to walk through that here for a minute. The box labeled hardware in the middle, that black box, is known in VM speak as the host. And its basically the hardware that provides the platform for virtualization. Unlike just your typical server, it typically is a hardware that has multi all over it: multi-cpu, multi-core, multi-threaded. Typically a CPU board will present numerous sockets, multiple sockets of memory. Often times were looking at 64-bit systems because we can address expanded memory over the 4 gigabyte limit. And another thing that is multi in a good host is a multi network interface card. Typically one of the bottlenecks of virtualized systems can be bandwidth throughput, and so relying on a single or even a master/ slave NIC is not good enough for a good host. Sitting below the hardware box - that box labeled storage - is nothing more than a bunch of disk drives accessible to the host over [an] efficient network connection. - [This is a] so called storage area network, or network attached storage, one of those flavors of external storage with fast IO buses that can be hooked up to the host either through standard Ethernet cabling or sometimes through optimized fiber channel
or faster optimized IO and that also contain protocols or embed (or use) protocols that off load some of the overhead that often the CPU has to do in order to do physical IO. Again, hinting at one of the other vulnerabilities we will
bring into play here, primarily from understanding where the potential vulnerabilities of a virtualized system can be. One of the things that weve learned from experience is to look at the vulnerabilities as being the places where there
So as testers, specifically performance testers, we need to make sure were ahead of the game, because it promises to not only change how we test, but what we test.
talk about shortly that we want to be sure to understand about virtual systems. Sitting above the hardware box or the host rectangle, there, in the red box, is another box labeled hypervisor. Hypervisor is a virtual term that refers to the piece of virtualized software and there [are] actually two types that this comes in: Type 1 is the one illustrated here or attempted to illustrate, where the hypervisor is the native OS running on the host. VMware is probably the leading example of the type 1 hypervisor. Theres actually a type 2 and that is where the hypervisor or the virtualization software itself run as a guest operating system on top of whatever the hardwares native OS is. An example of this is Windows 2008 server which comes embedded with a virtualization layer that is optional. And well talk about some of these underlying technologies towards the end. And finally the boxes across the top are the slices that the host will be sliced into; and each will have allocated to it some amount of CPU, some amount of memory and will share the peripheral that is the disk IO and the network IO and whatever other peripheral that may be involved, but those are the two main ones - based on how those VMs are configured using the management utility of the hypervisor. is potential limiting resource. Number 1 area is in disk IO. Perhaps you remember our diagram where I suggested the anatomy of a good virtualized system contains fast network attached storage with efficient protocols and thats because IO is one of the bottlenecks and we want to be able to maximize the IO throughput of the host because its going to be shared by all of the various virtual partitions that that host supports. So in this first bullet were looking at a piece of the previous output from VMware capacity planner and it shows a big red X on the left hand side and that happens
To learn why someone might choose one hypervisor type over the other, view the full webinar at:
PeRFORMANCe TeSTING RULeS OF ThUMb WheN evALUATING CAPACITy PLANNING deCISIONS

Dan Downing Yes, there are some heuristics we can 40 Automated software testing magazine
to be the Oracle database server in this instance. And the number 142 thats highlighted on the right is under disk IO and that says were tracking 142 IOs per second. Well, that may be relatively large, or not, depending again on the capacity of that particular system and its IO bus, but typically thats going to be flagged by a good capacity planning software as not a good candidate and thats where the red X comes into play. So this really points to us as testers that, hey anytime were testing a virtualized system
disk IO, [potential] contention at the disk level is something to look for. Did high disk IO servers get virtualized? And if so, lets really focus to make sure that we didnt undermine scalability and performance. The number 1 vulnerability regarding a virtualized system comes sort of in two related flavors and that is redundancy and load balancing. If you think about it, most systems - most complex systems - have multiple web servers, multiple app servers, a database cluster, perhaps some file servers, mail servers, other helper systems and they all occur in more than one instance - theres three app servers, theres five web servers and so forth. And certainly I think it should be pretty obvious that the reason those dedicated systems have those capabilities is number 1 for load balancing you know the load coming into our front door is more than one web server can support or more than one app server can manage and secondly if one of those hardware boxes fails its got helpers to take up the load and hopefully not affect end user experience too dramatically.
C ontent C oday T
ontribute
To learn more performance testing implications and some specific performance testing techniques view the full webinar at:
Automation Events
learn more today at

http//www.about.automatedtestinginstitute.com
41
43

AutomatedSoftwareTestingMagazine November2010

Caricato da

Informazioni sul documento

Descrizione originale:

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

AutomatedSoftwareTestingMagazine November2010

Caricato da

Copyright:

Formati disponibili

An AUTOMATED TESTING INSTITUTE Publication - www.automatedtestinginstitute.

Performance Testing Virtualized System

The Winner Is...

leArn more About this yeArs Ati AutomAtion honors Winners

Moving Your TesT Lab

Are You Contributing Content Yet?

Community Comments Box

Announcements & Blog Posts

learn more today at http//www.about.automatedtestinginstitute.com

2nd annual ati automation HonorS CoVerage

and The aTi auToMaTion honors Winner is...

ColumnS & departmentS

Automated software testing magazine

The besT is Yet To Come

...at some point the word best became profanity

Automated software testing magazine

For more information, visit http://www.networking.automatedtestinginstitute.com

Authors and Events

randy hayes wrote diagnostic

A PUBLICATION OF THE AUTOMATED TESTING INSTITUTE

dan downing has over

CONTACT US AST Magazine astmagazine@automatedtestinginstitute.com ATI Online Reference contact@automatedtestinginstitute.com

aTi honors Winners

Ati and Partner events

special edition 2010

ATI Automation Honors

special edition 2010

Automated software testing magazine

Automated software testing magazine

Moving to the Cloud

Types of Cloud Computing

Benef of the Cloud its for Testing

Choosing the Right Cloud

Moving to the Cloud

Making the Switch

Some Drawbacks and Limitations

Load Testing as Killer App for the Cloud

What about Security

Automated software testing magazine

2nd Annual ATI ............ ............ Automation ............ Honors

Best Open Source Unit Test Tool

Google C++ Testing Framework Producer: Google

cfix Producer: Johannes Passing

Check Producer: Chris Pickett

derson, Piers Lawson, Markus Renschler and Manfred Lange

special edition 2010

Automated software testing magazine

Best Open Source Fu

Jemmy Producer: Alexandre (Shura) Iline

White Producer: Vivek Singh

Fitnesse Producer: Micah Martin, Robert Martin

T-Plan Robot Producer: T-Plan Ltd.

14 Automated software testing magazine

special edition 2010

unctional Test Tool

special edition 2010

Automated software testing magazine

Best Open Source Performance Test Tool

Grinder Producer: Philip Aston

fwptt Producer: Bogdan Damian

JMeter Producer: Apache

Niclas and Ole

saved lots of time not writing my own tool.