Topic 1

Introduction to Software Testing

LEARNING OUTCOMES
By the end of this topic, you should be able to: 1. 2. 3. 4. 5. Recognise the importance of software testing; Explain what is software testing; State how software testing fits into the software engineering lifecycle; Describe the terminologies, terms and assumptions used in reference to software testing; and Explain the roadmap as shown in section 1.4.

INTRODUCTION
Nowadays, we are increasingly dependent on software to assist as well as facilitate our daily chores. In fact, whenever possible, most hardware implementations are currently being replaced by software. From the washing machine controllers, mobile phone applications to the sophisticated airplane control systems, our growing dependency on software can be attributed to a number of factors. Unlike hardware, software does not wear out. Thus, the use of software can also help to control maintenance costs. Additionally, software is also malleable and can be easily customised as the need arises. The fact that software is malleable can be an issue as far as dependability and reliability are concerned. Here, software testing becomes immensely important especially if software is employed in harsh, critical, or life-threatening applications. Covering as much as 40 to 50 percent of the total software development costs (Beizer, 1990), testing can be considered one of the most

TOPIC 1

INTRODUCTION TO SOFTWARE TESTING

important activities for software validation and verification. Lack of testing can lead to disastrous consequences including loss of data, fortunes and even lives. For instance, consider the accident that occurred during the European Space Agencys launching of Ariane 5 in 1996. Investigation by independent researchers from Massachusetts Institute of Technology reveals that the disaster is caused by the mismatch of the hardware and software component faults (Lion, 2005). The component erroneously puts a 64 bit floating point number in to a 16 bit space, causing overflow error. This overflow error affected the rockets alignment function, and hence, causing the rocket to veer off course and eventually exploded a mere 37 seconds after lift off. This incident wasted billions of Euros as far as satellite equipments and rockets equipment on board.

Figure 1.1: Application Error Dialog Box

If the abovementioned example is beyond our comprehension, consider the following common disaster that may have happened to some of us. Let say that we are in the middle of preparing our report to be submitted to a class in 5 minutes time. Suddenly, the following screen appears (see Figure 1.1) and we have not saved our work. What happen is that we have lost our valuable data and therefore unable to submit our report on time. As noted earlier, lack of testing could also be fatal if software is being used in life threatening or safety critical applications. Consider the software that controls the airbag system of a car. Suppose that the airbag system needs to be activated within 5 milliseconds upon impact. What if, due to a software failure, the airbag system is only activated after 5 seconds instead? People could die as a result of this delay. These aforementioned examples have demonstrated some of the incidents due to software failures. The question is how can testing helps to minimise if not avoid such incidents? In order to answer this question, there is a need to understand and define as well position software testing within the software engineering lifecycle. These issues will be the subject of the next section.

TOPIC 1

INTRODUCTION TO SOFTWARE TESTING

1.1

OVERVIEW OF SOFTWARE TESTING

Software testing is the process of executing a program with the intent to find errors (Myers, 1976). The objectives for software testing include: (a) Find defects and anomalies As will be seen in later topics, software testing communities have provided a number of useful techniques that can be used to identify possible defects and anomalies. Reduce risks for software failures If most of the defects and anomalies are detected, the risks can indeed be minimised. Prove that the program is good or otherwise In some applications, there may be a need to demonstrate and prove whether the program is working well or not. Detect variation from specification Software testing can help developer detects variation from the specification, that is, to establish confidence that a program does what it is supposed to do.

(b) (c)

(d)

In order to fulfil the aforementioned objectives, software testing activities can be divided into three main stages. Termed test cycles, these stages are Test Planning stage, Test Execution stage, and Test Monitoring stage (see Figure 1.2).

Figure 1.2: Software Testing Cycles (Source: (Zamli et al., 2007a))

TOPIC 1

INTRODUCTION TO SOFTWARE TESTING

As the name suggests, the Test Planning stage involves the planning on how and what kinds of test techniques to be employed (as will be covered by later topics) as well as the resources involved (e.g. in terms of manpower, costing, timing, and tools). Test planning stage also includes consideration on failure empirical data, which is, based on known problems with similar systems. Test Execution stage involves the activities to define and execute the planned test cases. Finally, Test Monitoring stage involves analysing coverage as well as monitoring whether or not the test results conform to the specifications. Having described the activities within software testing, there is also a need to understand how software testing fits into the whole software engineering product development lifecycle. Referring to Figure 1.3, software development starts with the requirement elicitation phase. Here, the customers and stakeholders interact with the requirement engineers to produce the software specifications. Based on the specifications, software engineers and programmers collaborate to produce software design and implementations. This activity occurs in the implementation phase. Software testing falls under the validation phase which may occur in parallel with the requirement elicitation phase and implementation phase. The independent verification and validation (V&V) team needs to consult the requirement engineers for software specification. Based on the software specification, the V&V team produces the test cases to be executed against the software implementation. If the execution results satisfy the requirement specification, then the software is ready to be released. Otherwise, some additional work need to be done to the design and implementation until conformance is achieved.

Figure 1.3: Software Engineering Product Lifecycle (Source: (Zamli et al, 2007b))

TOPIC 1

INTRODUCTION TO SOFTWARE TESTING

As seen above, the purpose of testing is not to prove anything, rather to reduce the perceived risk of not working to an acceptable value. The key challenges in software testing are not only dependent on the actual execution of the test cases but also the production of quality test cases. Here, quality test cases are test cases that are likely to detect faults.

SELF-CHECK 1.1
(a) (b) (c) (d) List and discuss why software testing is important. Identify some of the common incidents due to software failures. How does software engineering product lifecycle relate to software engineering lifecycle? Why do software testing activities need to be in parallel with development?

1.2

RELATED SOFTWARE FAULT TERMINOLOGIES

In order to facilitate understanding of the material in this text, there is a need to understand the meaning of some of the important software fault terminologies. (a) Specification defines the product being created. Often, specification includes a functional requirement that expresses the features that the product will support. Sometimes, specification also outlines a nonfunctional requirement that represents the constraint on the product. Verification is an attempt to find errors by executing a program in a test or simulated environment. Validation is an attempt to find errors by executing a program in a real environment. Error/bug/defect is a mistake specification and even testing. in design, coding, requirements,

(b) (c) (d) (e) (f) (g) (h)

Fault is the representation of the error. Failure is the result of faults (i.e. faults generate failure). Incident is the visible manifestation of the failure. Test case has inputs, steps and execution conditions as well as expected results (i.e. passed and failed criterion).

TOPIC 1

INTRODUCTION TO SOFTWARE TESTING

(i) (j) (k)

Test suite is a collection of test cases. Conformance implies the outputs match with expected results. Software artifact refers to any piece of software (i.e. models/descriptions) developed and used during the software development and maintenance process. Examples are requirements specifications, architecture and design models, source and executable code (programs), configuration directives, test data, test scripts, process models, project plans, various documentations and etc.

SELF-CHECK 1.2
Can you recap and differentiate between the above mentioned fault terminologies?

1.3

SOFTWARE TESTING ASSUMPTION

To ensure objective understanding on the test results obtained, software testing activities are often undertaken with the following assumptions: (a) (b) Software testing can be used to show the presence of bugs but never their absence. Only exhaustive testing can show that a program is free from defects. However, exhaustive testing is impossible due to timing and resource constraints. As the number of detected defects in a piece of software increases, the probability of the existence of more undetected defects also increases. There is no rule of thumb on when to stop testing apart from coverage metric criteria (which will be discussed in later topics). Hence, high coverage metric does not guarantee high fault detection. It is unethical to test ones own program. A small change in ones program does not necessarily improve the program, but can potentially lead to newer bugs.

(c) (d)

(e) (f)

TOPIC 1

INTRODUCTION TO SOFTWARE TESTING

ACTIVITY 1.1
(a) Work in pairs and identify three software that are commonly used in your workplace. List out at least 5 possible consequences as a result of their failures. In terms of ethics, discuss why V&V teams must be independent from the development teams to undertake testing?

(b)

1.4

TOPIC ROAD MAP

This topic has discussed important background on software testing. In particular, this topic highlights the importance of software testing, as well as illustrates how software testing fits into the software engineering lifecycle. Additionally, this topic also highlights the common terminologies used in reference to software testing. Referring to Figure 1.4, the rest of the topics will be as follows. Topic 2 discusses on static analysis activities for software testing. Topic 3 focuses on dynamic analysis activities. Unlike static analysis activities which concentrate on error prevention (i.e. through specification review as well as code walkthrough), dynamic analysis activities focus on error detection (i.e. through program execution). As such, Topic 3 highlights black box testing methods (also termed behavioural testing) as well as white box testing methods (also termed structural testing). Also, technique for test case design based on mutation testing is also summarised. Topic 4 discusses the planning, documents, and portfolios necessary for software testing. Here, the focus of discussion is based on the IEEE Standard for Software Test Documentation (IEEE 829 1998). Topic 5 highlights the main issues concerning test case execution. In particular, this topic will highlight the pass or fail criteria, the cascading dependencies of test execution, the scaffolding technique as well as the need for test automation. Additionally, Topic 5 highlights various testing level. In doing so, Topic 5 also demonstrates the scaffolding technique with drivers and stubs enabling functional testing even when the whole code has not yet been completed. Topic 6 describes techniques to minimise the number of test cases based on coverage analysis as well as the tway combinatorial design. Topic 7 complements Topic 6 by introducing various stopping criteria for software testing. Specifically, Topic 7 explores coverage metrics analysis, fault seeding technique, as well as statistical technique on the fault detection rate over specified time. Additionally, Topic 7 will also discuss the criteria for choosing the optimum interaction strength of coverage, t, for t-way

TOPIC 1

INTRODUCTION TO SOFTWARE TESTING

test case minimisation strategy. Topic 8 explores the discussion on regression testing as well as highlights the change impact analysis techniques in an effort to reduce the need for re-testing when changes are introduced in the existing software. Finally, Topic 9 discusses some of the automated tools that can assist and facilitate the testing process.

Figure 1.4: Text Roadmap

Software testing is an integral part of software engineering. Lack of testing can lead to disastrous consequences including loss of data, fortune and even lives. Software testing activities are often conducted in parallel with development. Software testing stages encompass planning, execution and monitoring phases.

TOPIC 1

INTRODUCTION TO SOFTWARE TESTING

Error Dialogue box Test Execution Stage

Test Monitoring Stage Test Planning Stage

Beizer, B. (1990). Software testing techniques testing. Van Nostrand Reinhold. Lion, J.L (2005). Ariane 5 failure report. Retrieved January 3rd, 2008, from http://sunnyday.mit.edu/accidents Myers, G.J. (1976). The art of software testing. Boston: John Wiley and Sons. Zamli, K.Z. et al. (2007a). A tool for automated test data generation (and Execution) based on combinatorial approach. International Journal of Software Engineering and Its Applications, July 2007, pp. 19-34. Zamli, K.Z. et al. (2007b). SFITGenerator: Development of combinatorial algorithms for automatic test case generation. Proceedings of the International Robotics, Vision, Information and Signal Processing Conference 2007 (ROVISP2007), Penang, pp. 31-35.