BSIT 54 Main PG 1 65

Chapter 1
Introduction to Software Testing
1.1 LEARNING OBJECTIVES
ou will learn about:

What is Software Testing? Need for software Testing, Various approaches to Software Testing, What is the defect distribution, Software Testing Fundamentals.
1.2 INTRODUCTION
Software testing is a critical element of software quality assurance and represents the ultimate process to ensure the correctness of the product. The quality of product always enhances the customer confidence in using the product thereby increasing the business economics. In other words, a good quality product means zero defects, which is derived from a better quality process in testing. The definition of testing is not well understood. Testers use a totally incorrect definition of the word testing, and that this is the primary cause for poor program testing. Examples of these definitions are such statements asTesting is the process of demonstrating that errors are not present,
BSIT 54 Software Quality and Testing
Chapter 1 - Introduction to Software Testing
The purpose of testing is to show that a program performs its intended functions correctly, and Testing is the process of establishing confidence that a program does what it is supposed to do. Testing the product means adding value to it which means raising the quality or reliability of the program. Raising the reliability of the product means finding and removing errors. Hence one should not test a product to show that it works; rather, one should start with the assumption that the program contains errors and then test the program to find as many errors as possible. Thus a more appropriate definition could be: Testing is the process of executing a program with the intention of finding errors Here identifying the errors is more important than verifying the functioning of the programs.
What is the purpose of Testing?
To show that the software works: It is known as demonstration-oriented. Test all functions that are used to run the software which produces results. These functions are directly related to the business functions. Here, all paths of the software system are not tested. For instance, there may be an error routine which traps the errors during the execution. This may not be necessary to test as it is not relevant to the daily running of the software.
To Show that the software doesnt work: It is known as destruction-oriented. In this case, all functions including non-business functions are tested to discover the possible errors. Even non-routine functions are also tested. Exceptions are also tested. However, once a single error is detected, the testing process stops.
To minimize the risk of not working up to an acceptable level: It is known as evaluationoriented. Some times the entire software will not be tested due to many reasons like inadequate time. However, all the functions which are critical from the business point of view to the customer will be tested. The acceptable level is a mutually agreed terms between the customer and the developer.
Why do we need to Test the application?

Defects can exist in the software, as it is developed by human beings who can make mistakes during the development of software. However, it is the primary duty of a software vendor to ensure that software delivered does not have defects and the customers day-to-day operations do not get affected. This can be achieved by rigorously testing the software. The most common origin of software bugs is due to:
Poor understanding and incomplete requirements
Unrealistic schedule Fast changes in requirements Too many assumptions and complacency
Some of the major computer system failures listed below gives sample evidence that the testing is an important activity of the software quality process. 1. In April of 1999 a software bug caused the failure of a $1.2 billion military satellite launch, the costliest unmanned accident in the history of Cape Canaveral launches. The failure was the latest in a string of launch failures, triggering a complete military and industry review of U.S. space launch programs, including software integration and testing processes. Congressional oversight hearings were requested. 2. On June 4, 1996, the first flight of the European Space Agencys new Ariane 5 rocket failed shortly after launching, resulting in an estimated uninsured loss of a half billion dollars. It was reportedly due to the lack of exception handling of a floating-point error in a conversion from a 64-bit integer to a 16-bit signed integer. 3. The computer system of a major online U.S. stock trading service failed during trading hours several times over a period of days in February of 1999 according to nationwide news reports. The problem was reportedly due to bugs in a software upgrade intended to speed online trade confirmations. 4. In November of 1997 the stock of a major health industry company dropped 60% due to reports of failures in computer billing systems, problems with a large database conversion, and inadequate software testing. It was reported that more than $100,000,000 in receivables had to be written off and that multi-million dollar fines were levied on the company by government agencies. 5. Software bugs caused the bank accounts of 823 customers of a major U.S. bank to be credited with $924,844,208.32 each in May of 1996, according to newspaper reports. The American Bankers Association claimed it was the largest such error in banking history. A bank spokesman said the programming errors were corrected and all funds were recovered. All the above incidents only reiterate the importance of thorough testing of software applications and products before they are put on production. It clearly demonstrates that cost of rectifying a defect during development is much less than rectifying a defect in production.
1.3 WHAT IS TESTING?

Before testing any system, it is necessary to know what is testing and how testing is conducted. What are the different types of testing and its impact on the system? According to IEEE:
Testing is an activity in which a system or component is executed under specified conditions; the results are observed and recorded and an evaluation is made of some aspect of the system or component
Some of the points to be noted regarding testing are:

Executing a system or component is known as dynamic testing. Review, inspection and verification of documents (Requirements, design documents. Test Plans etc.), code and other work products of software is known as static testing. Static testing is found to be the most effective and efficient way of testing. Successful testing of software demands both dynamic and static testing. Measurements show that a defect discovered during design that costs $1 to rectify at that stage may cost $1,000 to repair in production. This clearly points out the advantage of early testing. Testing should start with small measurable units of code, gradually progress towards testing integrated components of the applications and finally be completed with testing at the application level. Testing verifies the system against its stated and implied requirements, i.e., is it doing what it is supposed to do? It should also check if the system is not doing what it is not supposed to do, if it takes care of boundary conditions, how the system performs in production-like environment and how fast and consistently the system responds when the data volumes are high.
The above points addresses about the testing and its importance. Once the definition of testing is known it is necessary to know the approaches to the testing.
1.4 APPROACHES TO TESTING

Many approaches have been defined in literature. The importance of any approach depends on the type of the system in which you are testing. Some of the approaches are given below:
Debugging-Oriented
This approach identifies the errors during debugging the program. There is no difference between testing and debugging. Normally, programmers while developing the programs also test it for intended objectives. When a program does not work as specified, they start verifying the program and identify the possible bottlenecks.
This process is known as debugging. Debugging reveals most of the errors specified to the program but not the system.
Demonstration-oriented
The purpose of testing is to show that the software works Here, most of the time, the software is demonstrated in a normal sequence/flow. All the branches may not be tested. This approach is mainly to satisfy the customer and no value added to the program.
Destruction-oriented
The purpose of testing is to show that the software doesnt work. The program is tested in all possible ways such that it breaks down at some point of execution. It may be static testing and dynamic testing. It may be testing on different platforms or used an unstructured approach.
Evaluation-oriented
The purpose of testing is to reduce the perceived risk of not working up to an acceptable value. A minimum acceptance criterion is fixed for any software by the customer and the system is tested. The acceptance criterion may be defined by the Customer or Developer.
Prevention-oriented
It can be viewed as testing is a mental discipline that results in low risk software. It is always better to forecast the possible errors and rectify it earlier. A set of pre-conditions are defined by the system designers at each level, i.e. system design, coding, deployment and maintenance level. These conditions must be passed before the system could be released. This ensures the minimum quality standard of the software before its release. In general, program testing is more properly viewed as the destructive process of trying to find the errors (whose presence is presumed) in a program. A successful test case is one that furthers progress in this direction by causing the program to fail. However, one wants to use program testing to establish some degree of confidence that a program does what it is supposed to do and does not do what is not supposed to do, but this purpose is best achieved by a diligent exploration for errors.
1.5 IMPORTANCE OF TESTING

Testing activity cannot be eliminated in the life cycle as the end product must be bug free and a reliable one. Imagine a situation wherein a persons balance in the account reduces without withdrawing. A wrong marks card reaches the student which may spoil the students career. Testing must be an integral part of the system development and this process cannot be eliminated.
6
Testing is important because:

Testing is a critical element of software Quality Assurance Post-release removal of defects is the most expensive Significant portion of life cycle effort expended on testing
In a typical service oriented project, about 20-40% of project effort is spent on testing. It is much more in the case of human-rated software. An example could be, at Microsoft, tester to developer ratio is 1:1 whereas at NASA shuttle development center (SEI Level 5), the ratio is 7:1. This shows that how testing is an integral part of Quality Assurance.
1.6 HURDLES IN TESTING

As in many other development projects, testing is not free from hurdles. Some of the hurdles normally encounters are:

Usually late activity in the project life cycle No concrete output and therefore difficult to measure the value addition Lack of historical data Recognition of importance is relatively less Politically damaging as you are challenging the developer Delivery commitments Too much optimistic that the software always works correctly
Based on the project and delivery schedule these hurdles have to be addressed.
1.7 DEFECT DISTRIBUTION

In a typical project life cycle, testing is the late activity. When the product is tested, the defects may be due to many reasons. It may be either due to programming error or may be due to defects in the design or defects at any stages in the life cycle itself. The overall defect distribution is shown in Figure 1.1. It is observed that most of the defects are at the requirements stage itself. This is because, the requirements may not be understood properly or requirements keep changing even after coding or frequent changes in the requirements during development which in turn changes the design. Defects at the coding level are minimum because coding process is well understood and well defined.
Design 27%
Rqmts. Design Code Other
Rqmts. 56%
Code
7%
Other
10%
Figure 1.1: Software defect Distribution
1.8 TESTING FUNDAMENTALS

Any software program that is to be quality conscious, must be rigorously tested with a predefined set of objectives. These objectives could vary from project to project. Some projects may set a few objectives whereas others could set complete objectives. It is necessary to know the objectives before testing any software system.
1.8.1 Testing Objectives

Some of the objectives of testing are

Testing is a process of executing a program with the intent of finding an error. A good test is one that has a high probability of finding an as yet undiscovered error. A successful test is one that uncovers an as yet undiscovered error.
The objective is to design tests that systematically uncover different classes of errors and do so with a minimum amount of time and effort.
Secondary benefits include
Testing demonstrates that software functions appear to be working according to specification.
Those performance requirements appear to have been met. Data collected during testing provides a good indication of software reliability and some indication of software quality.
Finally, if testing process is followed we can guarantee that testing cannot show the absence of defects, it can only show that software defects are present. Once the objectives are set, testing process needs to be understood properly.
1.8.2 Test Information Flow

Testing is a complex process and requires efforts similar to software development. A typical test information flow is shown in Figure 1.2.
Figure 1.2: Test information flow in a typical software test life cycle
In the Figure1.2,
Software Configuration includes a Software Requirements Specification, a Design Specification, and source code. A test configuration includes a Test Plan and Procedures, test cases, and testing tools. It is difficult to predict the time to debug the code, hence it is difficult to schedule.
Once the right software is available for testing, proper test plan and test cases are developed. Then the
software is subjected to test with simulated test data. After the test execution, the test results are examined. It may have defects or the software is passed with out any defect. The software with defect is subjected to debugging and again tested for its correctness. This process will continue till the testing reports zero defects or run out of time for testing.
1.8.3 Test Case Design

During testing, test data and condition under which these data must be used is to be determined. Such process is known as test case design. We need to understand the scope of the testing and design the test cases. Some of the points to be noted during the test case design are:

Can be as difficult as the initial design. Can test if a component conforms to specification It is known as Black Box Testing. Can test if a component conforms to design It is known as White box testing.
In the Testing case design we cannot prove the complete correctness of the system as not all execution paths can be tested during the test execution. Consider the flow chart example shown in Figure 1.3 which represents multiple control structures and multiple paths of program execution.
Figure 1.3: Flow chart of a typical program execution with multiple paths.
10
A program with a structure as illustrated above (with less than 100 lines of Pascal code) has about 100,000,000,000,000 possible combinations of paths. If attempted to test these at the rate of 1000 tests per second, would take 3170 years to test all paths. This shows that exhaustive testing of a software is not possible. Testing is a mandatory activity in software development cycle. This activity must begin from the requirements specification and should be addressed at each level of development cycle. Before the beginning of the test execution, one should know about the importance of testing, objectives and process of testing.
QUESTIONS
1. 2. 3. 4. 5. 6. 7. 8. What is testing? Describe briefly. What is the purpose of software testing? What are the realone for software buge? Explain. What are the approaches to testing ? Explain. Describe the hurdles in testing. Explain the origin of the defect distribution in a typical software development life cycle. What are the objectives of testing? Explain. With diagram explain test information flow in typical software test life cycle.
Chapter 2
Types of Software Testing
In God, We Trust All else, We Test - Anon

Different types of Software Testing Brief definition of each of Software Testing types
2.2 INTRODUCTION
Testing is a yardstick to ensure the appropriateness of the understanding of needs and the correctness of the corresponding deliverable (which could be a system, a product or a service). Understanding is an outcome of the learning process, whereas delivery of an output is an outcome of the process of understanding. There are multiple ways of testing to ensure appropriateness/correctness. In the following sections, different types of testing are explained briefly to gain the knowledge in the area of software testing.
11
12 2.3 TYPES OF SOFTWARE TESTING

1. Black Box Testing
Chapter 2 - Types of Software Testing
To ensure the validity of the software application without the need for having its internal structure exposed. During the testing, the program code is not available but the executable module is used. Detailed test plan and proper test environment is required. Black box testing method is used in most of the applications to ensure the business functions accurately.
2. White Box Testing

It is a method to test the software utilizing its internal structure. The complete code structure is available during the test. Normally, programmers use this method to find the errors at the program level which helps in debugging. This also helps in understanding the behavior of the program, code review, and data flow and code level test case design.
3. Grey Box Testing

It is a higher level of abstraction and less comprehensive as compared to, white box testing. It involves business case testing with parts of the internal structure exposed. Used to test the software through some understanding of its internals and is like looking under hood.
4. Alpha Testing
A person other than the developer carries out testing, in-house, and at various project milestones. Since a third person tests the software, a detailed test plan must be provided.
5. Beta Testing
This type of testing is conducted by end-users either after or in parallel with system testing. Users who are the ultimate owner of the software will test the system before deployment and the errors will be reported back to the developer. For instance, Microsoft Windows 2000 was released to a set of users world wide for a limited duration for beta testing. All these users were using earlier versions of Microsoft operating system. The goal is to test the product by the end user in all respect.
6. Unit Testing
Unit testing uses both black box and white box methods to test a module against its design specification immediately after its design. This is normally done by the developer or another programmer. Unit testing is important and mandatory for all software modules. To illustrate, if you develop a program which computes simple interest, it has to be tested for all different type of inputs. This is known as unit testing.
7. Integration Testing
Here an independent tester in association with developers tests the system after the integration. In
13
large scale business systems, it is necessary to integrate many modules developed by different people. Once it is integrated, the same has to be tested and known as integration testing.
8. System Testing
This is a pre-deployment testing to verify whether the developed system meets the requirement specifications or not by simulating the target operational environment. This is to verify whether the system is production ready or not.
9. Acceptance Testing
Here the customer carries out system testing before finally accepting the system as meeting his stated requirements. This is normally done by the customer in association with the developer. Accepting testing is must for all systems. They will test the software with the real data and conforms the software system for its correctness.
10. Configuration/ Platform Testing

Some time the system may have to work with different platforms like windows 2000, Windows NT, Unix etc,. In such cases, the system has to be tested in all platforms in which the system will be ultimately used. It is to ensure the system functionality across,

Different Hardware/ Software configurations, Multiple versions of Operating Systems, Different versions of browsers, Various plug-ins or external components Different local conditions.
This is a mandatory testing for all application before deployment.
11. Localization Testing

All systems need not be interfaced or mapped on to a English language. In some countries, English may be used. In such cases, the system has to be developed both in English and customized to local language. This type of testing is to ensure the software that localization does not affect the software functionality. It ensures the appropriateness of linguistic locale changes.
12. Internationalization Testing

To ensure that the software is able to operate with the local languages and data representation including those using multi byte character sets. Also, currency symbols used in the application will be tested.
14
13. Usability Testing
To test the effective user interface of the system by considering the human factors and done by ergonomic experts. Normally, this kind of test is performed by using checklist approach. For modern software systems, this type of testing is mandatory. 14. Performance Testing Applications are tested to ensure expected efficiency, which includes parameters such as:

Response time, Resource utilization, and Reliability
Performance testing helps in acquiring the customers confidence on the system.
15. Load Testing

The system utilization is normally depends on the number of users. More users on the system, more is the pressure or load on the system. But how to ensure that the developed system is capable of handling more loads. Load testing is to test the server and hardware system by applying load, defined by the workload pattern to yield required performance characteristics like response time, throughput and resource utilization. 16. Stress Testing It is another form of load testing. Similar to load testing in a constrained environment, it involves stressing the server beyond the expected load and verifying the behavior of the operational characteristics of the application.
17. Benchmark Testing

Benchmark is a standard mechanism to compare with other similar products. This type of testing is to compare the product with the standard benchmark available.
18. Functional Testing

Wherein the system under testing is subjected to verification against its functional behavior as laid down in the requirements document. Functional behavior is normally collected at the requirements gathering stage and the same will verified during the functional testing. However, the methods used for functional testing may be different, like, white box or black box methods.
19. Formal Inspections

It is a structural method to identify defects in various artifacts used in the development process as well
15
as in the deliverables. The artifacts include requirements document, design document, coding standards, program codes, reports and others. Code walk through is a standard method used in formal inspections. Usually, a team of people performs this activity and log defects in a formal fashion.
20. Consistency Testing

It is a type of testing method to verify the consistency of data in an application. This is important because the same data stored in two different formats may cause problems. To illustrate, a name field may use twenty characters as field length in one place and different field length at other places may cause consistency problem.
21. Regression Testing

A mechanism to conduct repeated tests on every new version of the software by using the same test scripts and test plan. This would ensure that coupling effects (wherein fixing defects in one part of the application could potentially create defects in previously correct segments) are taken care of in the development cycle. This is helpful when you are releasing multiple versions of the software.
22. Smoke Testing

Rigorous testing requires more time and people. Also, the complete software may not be ready for rigorous testing. Initially, testing is carried out on important functional behavior so that it can be taken up for further testing. Smoke testing mainly ensures that all functions work correctly on a new software before being subjected to rigorous testing. This is a pre-qualification test.
23. Sanity Testing

A cursory test to ensure all normal functions that work as expected before shipping or delivery. It is equivalent to ensuring the management of the company that their product is healthy and quality conscious.
24. Top Down Testing

To test the application, through the white box method, starting with the main program and going down unit by unit. It involves creation of stubs if a lower level unit is not ready. This method is useful if the system is large and developed many people. It is difficult to wait for all the modules to get ready before the testing actually begins. This method also ensures that the requirements are tested well in advance.
25. Bottom Up Testing

To test the application, by white box method, starting with the lower level units and going up unit by unit. It involves creation of drivers when a higher-level unit is not ready. In this method, as and when the modules are ready, testing is conducted. Here, module level tests are carried out first and system level later.
16
26. End to End Testing
Test the functions of the system by involving all possible components, which includes hardware, software, other components, processes and people.
27. Automated Testing

Testing is a complex process. For large systems, manual testing is difficult and time consuming. In order to avoid this, automated tools are used for testing. It is a method to execute series of test cases automatically during the tests using testing tools.
28. Heuristic Testing

It is a method to test the software through experiential methods, involving technical insights and participation in critical thinking.
29. User Testing

Collect all the user scenarios through real user exercises and test the software. Participation by users is very important in this type of testing.
30. Protection Testing

It is a method to verify whether information is protected or not. It is a method to find the presence of faults, which could result in corruption of data, denial of services, unauthorized access or protection against attackers.
31. Data Driven Test

Generally, test scrip is developed to test the software. Data files are separated from data files. Data driven test is the method which uses different sets of data during testing without affecting the test script.
32. Forced Error Testing

A method to drive the software intentionally to all error conditions and checks whether the handling of error routines is correct. Standard software provides the error trapping and possible recovery from the errors. This method of testing is to ensure these error traps are working correctly.
33. Database Testing

A testing method to identify database related errors. Data bases are separated from the program code. These data bases must be separated for its integrity of data, schema and other related details.
34. Destructive Testing

A method to stress the application or environment until it fails to perform the functions and then carrying out a root-cause analysis.
17
35. Preventive Testing

This type of Testing is through code review, formal inspection, design reviews and walkthroughs to prevent possible errors once the software is ready. This is a kind of static testing method and helps in documentation also.
36. Boundary Testing

To test the software with extreme input values and cause it to generate extreme output values.
37. Volume Testing

It is a kind of load testing wherein large amounts of data are processed through the system.
38. Compatibility Testing

It is similar to configuration testing to ensure whether the software works with multiple Operating System and hardware. This type of testing is important particularly for web applications. We dont know the type of OS used at the client destination.
39. Documentation Testing

Test to ensure the correctness of the documentation, which includes manuals, context helps, screen images etc. This is required when a product is sold along with manuals and other documents.
40. On-line Help Testing

It is to ensure the correctness of on-line help and proper links with the appropriate pages.
41. Installation Testing

Test the installation module for multiple versions, operating systems and different platforms. This is also known as deployment testing.
42. Graphic User Interface(GUI) Testing

Most of the modern software uses GUI method to develop user interface. These user interfaces are developed using GUI standards. This method of testing is to ensure the software is developed as per GUI standards and works correctly.
43. Security Testing

Any software is subjected to security threat. It may be due to virus or hackers may misuse the software. In order to protect the software from these threats, security testing must be conducted.
18
44. Link Testing
To verify whether the web links are pointing to the correct page and correspondingly, whether the page exists. This is very important testing in case of E-commerce applications.
45. Static Testing

This is a method of testing the program codes through walkthroughs, reviews and inspections. Here, the software module will not be executed during the testing.
46. Requirement Phase Testing

Use verification methods to test the requirements of software before the development process and ensure the requirements are testable.
47. Error-Handling Testing

Determine the ability of the software being tested to process incorrect transactions.
48. Module Testing

It is a process of testing individual subprograms, subroutines and procedures in a program. Also, test a program in small blocks as they are built.
49. Basis Path Testing

It is a white box method to test the software using the flow diagrams of all the executable paths.
50. Big Bang Testing

No integration testing; test the system at the end of development cycle by attacking all modules at once. It is fast and must be done at once.
51. Exhaustive Testing

Test the software for every possible inputs and every possible outcome. In most of the cases, it is difficult to test all possible inputs as it is time consuming.
52. Hardware Testing

Test the hardware (this is very important in embedded systems) to check if it meets the customer specifications.
53. Software Testing

Refers to a broad area of activity and is a method to test if the software conforms to requirements completely.
19
54. Production Testing

It is a part of the production process and involves testing the product that has the exact image of the deliverable. It ensures how the software works when it is deployed later.
55. Component Testing

Test individual software components and interoperability among all components.
56. E-Commerce Testing

A buzzword, which is used for end-to-end testing of web based applications.
57. Browser Testing

Test the web based applications with browsers against standards set by World Wide Web Consortium.
58. HTML Testing

Test the web application to identify deviations from Hyper Text Meta Language standards, missing tags, wrong links and lack of proper syntax.
59. Server Testing

This type of testing in web applications tests the various servers for availability, tolerability and maintainability.
60. Reliability Testing

It is a type of testing to ensure the reliability of the software for a specified period. For example, a typical web site must be available 24 x 7 days through out the year.
61. Availability Testing

To ensure the availability of the application to the user by verifying the user connections, check whether application responds to an input, 24x7 availability time and number of failed attempts to load a page. The above dictionary of testing types is designed to be a quick reference to help develop an appreciation, obtain some understanding and strengthen your testing vocabulary. Some of the testing types/definitions may overlap with each other and suggest nearly the same meaning. In such cases, it may be appropriate to refer to more elaborate compendia or specialist material (as provided below) to arrive at a better understanding. (Refer to References given at the end of the book).
QUESTION
1. Explain types of software testing in detail.
Chapter 3
Software Quality Assurance

Basic principles about the Software Quality Software Quality Assurance and SQA activities Software Reliability
3.2 INTRODUCTION
Quality is defined as a characteristic or attribute of something. As an attribute of an item, quality refers to measurable characteristics-things we are able to compare to known standards such as length, color, electrical properties, malleability, and so on. However, software, largely an intellectual entity, is more challenging to characterize than physical objects. Quality design refers to the characteristics that designers specify for an item. The grade of materials, tolerance, and performance specifications all contribute to the quality of design. Quality of conformance is the degree to which the design specification is followed during manufacturing. Again, the greater the degree of conformance, the higher the level of quality of conformance.
Software Quality Assurance Encompasses
A quality management approach for software development
20
Chapter 3 - Software Quality Assurance
21
Effective software engineering technology Formal technical reviews during the software development A multi tiered testing strategy Control of software documentation and changes made to it A procedure to assure compliance with software development standards Measurement and reporting mechanisms
Software Engineering Methods
Formal Technical Review
Measurement
Quality
Standards And Procedures
Testing
SCM & SQA
Figure 3.1: Achieving Software Quality
The term software quality is ensured by following either the process or methods of many items shown in Figure 3.1. Development of software must be carried out through software engineering methods. Ad-hoc methods do not yield proper results. While developing, one should follow standards and procedures. Standards related to screen design, documentation, file design and so on. Every software development methodology as well as maintenance must associate with Software Configuration Methods (SCM) and standard Software Quality Assessment (SQA). Normally, a SQ Analyst audits the software periodically during the development. Quality process must involve technical reviews and standard measurement must be followed. Testing is the integral process of the software quality process.
3.3
QUALITY CONCEPTS
What are the quality concepts?
22

Quality Quality control Quality assurance Cost of quality
The American heritage dictionary defines quality as a characteristic or attribute of something. As an attribute of an item quality refers to measurable characteristic-things which we are able to compare to known standards such as length, color, electrical properties, and malleability, and so on. However, software is largely an intellectual entity, more challenging to characterize than physical object. Nevertheless, measures of a programs characteristic do exist. These properties include 1. Cyclomatic complexity 2. Cohesion 3. Number of function points 4. Lines of code When we examine an item based on its measurable characteristics, two kinds of quality may be encountered:

Quality of design Quality of conformance
3.4 QUALITY OF DESIGN

Quality of design refers to the characteristics that designers specify for an item. The grade of materials, tolerance, and performance specifications all contribute to quality of design. As higher graded materials are used and tighter, tolerance and greater levels of performance are specified the design quality of a product increases if the product is manufactured according to specifications. Proper standards and documentations are used to increase the quality standards. Adequate checks must be associated during the development life cycle of the product.
3.5 QUALITY OF CONFORMANCE

Quality of conformance is the degree to which the design specifications are followed during manufacturing. Again, the greater the degree of conformance, the higher the level of quality of conformance.
23
In software development, quality of design encompasses requirements, specifications and design of the system. The requirements must be gathered formally and there should not be any ambiguity and must be complete in all respects. Specifications must be elaborated and defined formally. Design must follow standard methods and process. Quality of conformance is an issue focussed primarily on implementation. If the implementation follows the design and the resulting system meets its requirements and performance goals, conformance quality is high.
3.6 QUALITY CONTROL (QC)

QC is the series of inspections, reviews, and tests used throughout the development life cycle to ensure that each work product meets the requirements placed upon it. QC includes a feedback loop to the process that created the work product. The combination of measurement and feedback allows us to tune the process when the work products created fail to meet their specification. These approach views QC as part of the manufacturing process QC activities may be fully automated, manual or a combination of automated tools and human interaction. An essential concept of QC is that all work products have defined and measurable specification to which we may compare the outputs of each process the feedback loop is essential to minimize the defect produced. Normally, a QC team reviews the development and gives guidelines periodically to improve the software quality. They review the code, standards, process followed at every stages.
3.7 QUALITY ASSURANCE (QA)

QA consists of the editing and reporting functions of management. The goal of quality assurance is to provide management with the data necessary to be informed about product quality, thereby gaining insight and confidence that product quality is meeting its goals. Of course, if the data provided through QA identify problems, it is managements responsibility to address the problems and apply the necessary resources to resolve quality issues.
3.7.1 Cost of Quality

Cost of quality includes all codes incurred in the pursuit of quality or in performing quality related activities. Cost of quality studies are conducted to provide a base line for the current cost of quality, to identify opportunities for reducing the cost of quality, and to provide a normalized basis of comparison. The basis of normalization is usually money. Once we have normalized quality costs on a money basis, we
24
have the necessary data to evaluate where the opportunities lie to improve our process further. However, we can evaluate the effect of changes in money based terms. QC may be divided into cost associated with

Prevention Appraisal Failure
Prevention costs include:

Quality Planning Formal Technical Review on systems Testing Equipments used for running the software Training imparted for persons involved in development
Appraisal costs includes activity to gain insight into product condition the First time through each process. Examples for appraisal costs includes:

In process and inter process inspection Equipment calibration and maintenance Testing
Failure Costs are costs that would disappear if no defects appeared before shipping a product to customers failure costs may be subdivided into internal and external failure costs. Internal failure costs are costs incurred when we detect an error in our product prior to shipment. Internal failure costs includes

Rework Repair Failure Mode Analyses
External failure costs are the cost associated with defects found after the product has been shipped to the customer. Examples of external failure costs are
25
1. Complaint Resolution 2. Product return and replacement 3. Helpline support 4. Warranty work
3.8 SOFTWARE QUALITY ASSURANCE(SQA)

How do we define Quality? Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software. The above definition emphasizes three important points: 1. Software requirements are the foundation from which quality is measured. Lack of conformance to requirements is lack of quality. 2. Specified standards define a set of development criteria that guide the manner in which software is engineered. If the criteria are not followed, lack of quality will almost surely result. 3. There is a set of implicit requirements often goes unmentioned. (E.g. the desire of good maintainability). If software conforms to its explicit requirements but fails to meet implicit requirements, software quality is questionable.
3.8.1 Background Issues

QA is an essential activity for any business that produces products to be used by others. The SQA group serves as the customer in-house representative. That is the people who perform SQA must look at the software from customers point of view. The SQA group attempts to answer the questions asked below and hence ensure the quality of software. The questions are: 1. Has software development been conducted according to pre-established standards? 2. Have technical disciplines properly performed their role as part of the SQA activity?
26
SQA Activities
SQA Plan is interpreted as shown in Figure 3.2.
SQA comprises of a variety of tasks associated with two different constituencies. 1. The software engineers who do technical work like

Performing Quality assurance by applying technical methods Conduct Formal Technical Reviews Perform well-planed software testing.
2. SQA group that has responsibility for

Quality assurance planning oversight Record keeping Analysis and reporting.
QA activities performed by SE team and SQA are governed by the following plan:

Evaluation to be performed. Audits and reviews to be performed. Standards that is applicable to the project. Procedures for error reporting and tracking Documents to be produced by the SQA group
Amount of feedback provided to software project team.
Software Engineers Activities
SQA Planning Team Activities
SQA Plan
Figure 3.2: Software Quality Assurance Plan

What are the activities performed by SQA and SE team? Prepare SQA Plan for a project
27
Participate in the development of the projects software description Review software-engineering activities to verify compliance with defined software process. Audits designated software work products to verify compliance with those defined as part of the software process. Ensures that deviations in software work and work products are documented and handled according to a documented procedure. Records any noncompliance and reports to senior management.
3.8.2 Software Reviews

Software reviews are a filter for the software engineering process. That is, reviews are applied at various points during software development and serve to uncover errors that can then be removed. Software reviews serve to purify the software work products that occur as a result of analysis, design, and coding. Any review is a way of using the diversity of a group of people to: 1. Point out needed improvements in the product of a single person or a team; 2. Confirm that parts of a product in which improvement is either not desired, or not needed. 3. Achieve technical work of more uniform, or at least more predictable quality that can be achieved without reviews, in order to make technical work more manageable. There are many different types of reviews that can be conducted as part of software- engineering like: 1. An informal meeting if technical problems are discussed. 2. A formal presentation of software design to an audience of customers, management, and technical staff is a form of review. 3. A formal technical review is the most effective filter from a quality assurance standpoint. Conducted by software engineers for software engineers, the FTR is an effective means for improving software quality.
3.8.3 Cost impact of Software Defects

To illustrate the cost impact of early error detection, we consider a series of relative costs that is based on actual cost data collected for large software projects. Assume that an error uncovered during design will cost 1.0 monetary unit to correct. Relative to this cost, the same error uncovered just before testing commences will cost 6.5 units; during testing 15 units;
28
and after release, between 60 and 100 units.
3.8.4 Defect Amplification and Removal

A defect amplification model can be used to illustrate the generation and detection of errors during preliminary design, detail design, and coding steps of the software engineering process. The model is illustrated schematically in Figure 3.3. A box represents a software development step. During the step, errors may be inadvertently generated. Review may fail to uncover newly generated errors from previous steps, resulting in some number of errors that are passed through. In some cases, errors passed through from previous steps, resulting in some number of errors that are passed through. In some cases errors passed through from previous steps are amplified (amplification factor, x) by current work. The box subdivisions represent each of these characteristics and the percent efficiency for detecting errors, a function of the thoroughness of review.
Errors from previous Step DEFECTS
DETECTION
Errors passed to next step
Errors passed through Percent efficiency for Amplified errors 1:x Newly errors
Figure 3.3: Defect Amplification Model.
error detection
generated
Figure 3.4 illustrates hypothetical example of defect amplification for a software development process in which no reviews are conducted. As shown in the figure each test step is assumed to uncover and correct fifty percent of all incoming errors without introducing new errors (an optimistic assumption). Ten preliminary design errors are amplified to 94 errors before testing commences. Twelve latent defects are released to the field. Figure 3.5 considers the same conditions except that design and code reviews are conducted as part of each development step. In this case, ten initial preliminary design errors are amplified to 24 errors before testing commences. Only three latent defects exist. By recalling the relative cost associated with the discovery and correction of errors, overall costs (with and without review for our hypothetical example) can be established. To conduct reviews a developer must expend time and effort and the development organization must spend money. However, the results of the presiding example leave little doubt that we have encountered a Pay now or pay much more later syndrome.
29
Formal technical reviews (for design and other technical activities) provide a demonstrable cost benefit and they should be conducted.
Preliminary design 0 0 10 70 % 3,2 Detail Design 2 1-1.5 1 25 50 % 15 Code/Unit 5 10 -3 10 25 24 Integration Test 12 0 10 70 % Validation test 2 6 1-1.5 25 50 % System Test To 60%
24
3 0 0 60%
Latent
Figure 3.4: Defect Amplification -No Reviews
30
Preliminary design 0 0 10 0% 10, Detail Design 6 4 x 1.5 4 x = 1.5 25 0%
37
Code/Unit
10 27x3 x=3 25 94 Integration Test 47 0 10 50 % Validation test 2 1-1.5 25 50 % System Test To 20%
94
24
12 0 0 60%
Latent errors
Figure 3.5: Defect Amplification - Reviews Conducted
31
3.9 FORMAL TECHNICAL REVIEWS (FTR)

FTR is a SQA activity that is performed by software engineers. The main objectives of the FTR are:

To uncover errors in function, logic, implemented in any representation of the software To verify that software under review meets its requirements To ensure that software has been represented according to predefined standards To achieve software that is developed in an uniform manner To make projects more manageable
In addition, the FTR serves as a training ground, enabling junior engineers to observe different approaches to software analysis, design, and implementation. The FTR also serves to promote backup and continuity because numbers of people become familiar with parts of the software that they may not have other wise seen. The FTR is actually a class of reviews that include walkthrough inspection and round robin reviews, and other small group technical assessments of software. Each FTR is conducted as meeting and will be successful only if it is properly planned, controlled and attended. In the paragraph that follows, guidelines similar to those for a walk through are presented as a representative Formal technical review.
3.9.1 The Review Meeting

The Focus of the FTR is on a work product - a component of the software. At the end of review all attendees of the FTR must decide: 1. Whether to accept the work product without further modification 2. Reject the work product due to serve errors (Once corrected another review must be performed) 3. Accept the work product provisionally (minor errors have been encountered and must be corrected but no additional review will be required) The decision made, All FTR attendees complete a sign-off indication their participation in the review and their concurrence with the review team findings
3.9.2 Review Reporting and Record Keeping

The review summary report is typically is a single page form. It becomes part of the project historical
32
record and may be distributed to the project leader and other interested parties. The review issue lists serves two purposes. 1. To identify problem areas within the product 2. To serve as an action item. Checklist that guides the producer as corrections are made. An issues list is normally attached to the summary report. It is important to establish a follow up procedure to ensure that item on the issues list have been properly corrected. Unless this is done, it is possible that issued raised can fall between the cracks. One approach is to assign responsibility for follow up for the review leader. A more formal approach as signs responsibility independent to SQA group.
3.9.3 Review Guidelines

The following represents a minimum set of guidelines for formal technical reviews:

Review the product, Not the producer Set an agenda and maintain it Limit the debate Enunciate problem areas but dont attempt to solve every problem noted Take return notes Limit the number of participants and insist upon the advanced preparation Develop a check list each work product that is likely to be reviewed Allocate resources and time schedule for FTRs. Conducts meaningful training for all reviewers Review your early reviews
3.10 STATISTICAL QUALITY ASSURANCE

Statistical quality assurance reflects a growing trend throughout industry to become more quantitative about quality. For software, statistical quality assurance implies the following steps
Information about software defects is collected and categorized
33
An attempt is made to trace each defect to its underlying cause Using Pareto principle (80% of the defects can be traced to 20% of all possible causes), isolate the 20% (the vital few) Once the vital few causes have been identified, move to correct the problems that have caused the defects.
This relatively simple concept represents an important step toward the creation of an adaptive software engineering process in which changes are made to improve those elements of the process that introduce errors. To illustrate the process, assume that a software development organization collects information on defects for a period of one year. Some errors are uncovered as software is being developed. Other defects are encountered after the software has been released to its end user. Although hundreds of errors are uncovered, all can be tracked to one of the following causes.
Incomplete or erroneous specification (IES) Misinterpretation of customer communication (MCC) Intentional deviation from specification (IDS) Violation of programming standards ( VPS ) Error in data representation (EDR) Inconsistent module interface (IMI) Error in design logic (EDL) Incomplete or erroneous testing (IET) Inaccurate or incomplete documentation (IID) Error in programming language translation of design (PLT) Ambiguous or inconsistent human-computer interface (HCI) Miscellaneous (MIS)
To apply statistical SQA table 1 is built. Once the few vital causes are determined, the software development organization can begin corrective action. After analysis, design, coding, testing, and release, the following data are gathered.
34
Ei Si Mi Ti PS
= The total number of errors uncovered during the ith step in the software Engineering process = The number of serious errors = The number of moderate errors = The number of minor errors = Size of the product (LOC, design statements, pages of documentation at the ith step
Ws, Wm, Wt = weighting factors for serious, moderate and trivial errors where recommended values are Ws = 10, Wm = 3, Wt = 1. The weighting factors for each phase should become larger as development progresses. This rewards an organization that finds errors early. At each step in the software engineering process, a phase index, PIi, is computed as; PIi = Ws (Si/Ei)+Wm (Mi/Ei)+Wt (Ti/Ei) The error index EI ids computed by calculating the cumulative effect or each PIi, weighting errors encountered later in the software engineering process more heavily than those encountered earlier. EI = (i x PIi)/PS = (PIi+2PI2 +3PI3 +iPIi)/PS The error index can be used in conjunction with information collected in table to develop an overall indication of improvement in software quality.
35
DATA COLLECTION FOR STATISTICAL SQA Total Error IES MCC IDS VPS EDR IMI EDL IET IID PLT HCI MIS No. 205 156 48 25 130 58 45 95 36 60 28 56 % 22 17 5 3 14 6 5 10 4 6 3 6 100 Serious No. 34 12 1 0 26 9 14 12 2 15 3 0 128 % 27 9 1 0 20 7 11 9 2 12 2 0 100 Moderate No 68 68 24 15 68 18 12 35 20 19 17 15 379 % 18 18 6 4 18 5 3 9 5 5 4 4 100 No 103 76 23 10 36 31 19 48 14 26 8 41 435 Minor % 24 17 5 2 8 7 4 11 3 6 2 9 100
TOTA 942 LS
3.11 SOFTWARE RELIABILITY

Software reliability, unlike many other quality factors, can be measured directed and estimated using historical and developmental data. Software reliability is defined in statistical terms as Probability of
36
failure free operation of a computer program in a specified environment for a specified time to illustrate, program x is estimated to have reliability of 0.96 over 8 elapsed processing hours. In other words, if program x were to be executed 100 times and required 8 hours of elapsed processing time, it is likely to operate correctly to operate 96/100 times.
3.11.1 Measures of Reliability and Availability

In a computer-based system, a simple measure of reliability is mean time between failure (MTBF), where MTBF = MTTF+MTTR The acronym MTTF and MTTR are Mean Time To Failure and Mean Time To Repair, respectively. In addition to reliability measure, we must develop a measure of availability. Software availability is the probability that a program is operating according to requirements at a given point in time and is defined as: Availability = MTTF / (MTTF+MTTR) x100% The MTBF reliability measure is equally sensitive to MTTF and MTTR. The availability measure is somewhat more sensitive to MTTR an indirect measure of the maintainability of the software.
3.11.2 Software Safety and Hazard Analysis

Software safety and hazard analysis are SQA activities that focus on the identification and assessment of potential hazards that may impact software negatively and cause entire system to fail. If hazards can be identified early in the software engineering process software design features can be specified that will either eliminate or control potential hazards. A modeling and analysis process is conducted as part of safety. Initially hazards are identified and categorized by criticality and risk. Once hazards are identified and analyzed, safety related requirements could be specified for the software i.e., the specification can contain a list of undesirable events and desired system responses to these events. The roll of software in managing undesirable events is then indicated. Although software reliability and software safety are closely related to one another, it is important to understand the subtle difference between them. Software reliability uses statistical analysis to determine the likelihood that a software failure will occur however, the occurrence of a failure does not necessarily result in a hazard or mishap. Software safety examines the ways in which failure result in condition that
37
can be lead to mishap. That is, failures are not considered in a vacuum. But are evaluated in the context of an entire computer based system.
3.12 THE SQA PLAN

The SQA plan provides a road map for instituting software quality assurance. Developed by the SQA group and the project team, The plan serves as a template for SQA activities that are instituted for each software project. ANSI/IEEE Standards 730-1984 and 983-1986 SQA plans is defined as shown below. I. II. III Purpose of Plan References Management 1. Organization 2. Tasks 3. Responsibilities IV. Documentation 1. Purpose 2. Required software engineering documents 3. Other Documents V. Standards, Practices and conventions 1. Purpose 2. Conventions VI. Reviews and Audits 1. Purpose 2. Review requirements a. Software requirements b. Designed reviews
38
c. Software V & V reviews d. Functional Audits e. Physical Audit f. In-process Audits g. Management reviews VII. Test VIII. Problem reporting and corrective action IX. X. XI. Tools, techniques and methodologies Code Control Media Control
XII. Supplier Control XIII. Record Collection, Maintenance, and retention XIV. Training XV. Risk Management.
3.12.1 The ISO Approach to Quality Assurance System

ISO 9000 describes the elements of a quality assurance in general terms. These elements include the organizational structure, procedures, processes, and resources needed to implement quality planning, quality control, quality assurance, and quality improvement. However, ISO 9000 does not describe how an organization should implement these quality system elements. Consequently, the challenge lies in designing and implementing a quality assurance system that meets the standard and fits the companys products, services, and culture.
3.12.2 The ISO 9001 Standard

ISO 9001 is the quality assurance standard that applies to software engineering. The standard contains 20 requirements that must be present for an effective quality assurance system. Because the ISO 9001 standard is applicable in all engineering disciplines, a special set of ISO guidelines have been developed to help interpret the standard for use in the software process.
39
The 20 requirements delineated by ISO9001 address the following topic. 1. Management responsibility 2. Quality system 3. Contract review 4. Design control 5. Document and data control 6. Purchasing 7. Control of customer supplied product 8. Product identification and tractability 9. Process control 10.Inspection and testing 11. Control of inspection, measuring, and test equipment 12.Inspection and test status 13.Control of non confirming product 14.Corrective and preventive action 15.Handling, storage, packing, preservation, and delivery 16.Control of quality records 17.Internal quality audits 18.Training 19.Servicing 20.Statistical techniques In order for a software organization to become registered to ISO 9001, it must establish policies and procedure to address each of the requirement noted above and then be able to demonstrate that these policies and procedures are being followed.
40
QUESTIONS
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. With diagram explain how software quality is achieved. What are quality concepts? Explain.
Describe the terms prevention costs, appraisal cost and failure cost. What are the background issues of QA? Explain briefly. Explain the activities of SQA plan. What do you mean by software reviews? Describe briefly. What is defect amplification model? Explain in detail. What is FTR? Explain its objectives. Explain types of FTR. Explain the six well defined phases of Fagan Inspection. Briefly explain the following: i)Small Vs Large Team Reviews. ii) Single Vs Multiple Session Reviews.
iii) Non Systematic & Systematic DDT Reviews. iv) Single Site Vs Multiple Site Reviews. v) Synchronous and Arychronous Reviews.
vi) Manual Vs Computer Supported Reviews. 12. 13. 14. 15. 16. 17. 18. What are recent economic analyses? Explain. How psychological aspects of FTR are categorized? Explain in detail. What is the purpose of Review meeting? Explain. Explain the Review guidelines in detail. Describe the Statistical Quality Assurance. Mention the different causes for tracking error. Explain the measures of Reliability and Availability. Explain CMM model in detail.
Chapter 4
Program Inspections, Walkthroughs, and Reviews
What are static testing and its importance in Software Testing? Guidelines to be followed during static testing Process involved in inspection and walkthroughs Various check lists to be followed while handling errors in Software Testing Review techniques
4.2 INTRODUCTION
The majority of the programming community worked under the assumptions that programs are written solely for machine execution and are not intended to be read by people. The only way to test a program is by executing it on a machine. Weinberg built a convincing strategy that why programs should be read by people, and indicated this could be an effective error detection process. Experience has shown that human testing techniques are quite effective in finding errors, so much so that one or more of these should be employed in every programming project. The method discussed in this Chapter are intended to be applied between the time that the program is coded and the time that computer based testing begins. We discuss this based on two ways:
It is generally recognized that the earlier the errors are found, the lower are the costs of correcting
41
42
Chapter 4 - Program Inspections, Walkthroughs, and Reviews
the errors and the higher is the probability of correcting the errors correctly.
Programmers seem to experience a psychological change when computer-based testing commences.
4.3 INSPECTIONS AND WALKTHROUGHS

Code inspections and walkthroughs are the two primary human testing methods. It involve the reading or visual inspection of a program by a team of people. Both methods involve some preparatory work by the participants. Normally it is done through meeting and it is typically known as meeting of the minds, a conference held by the participants. The objective of the meeting is to find errors, but not to find solutions to the errors (i.e. to test but not to debug).
What is the process involved in inspection and walkthroughs?

The process is performed by a group of people (three or four), only one of whom is the author of the program. Hence the program is essentially being tested by people other than the author, which is in consonance with the testing principle stating that an individual is usually ineffective in testing his or her own program. Inspection and walkthroughs are far more effective compare to desk checking (the process of a programmer reading his/her own program before testing it) because people other than the programs author are involved in the process. These processes also appear to result in lower debugging (error correction) costs, since, when they find an error, the precise nature of the error is usually located. Also, they expose a batch of errors, thus allowing the errors to be corrected later enmasse. Computer based testing, on the other hand, normally exposes only a symptom of the error and errors are usually detected and corrected one by one.
Some Observations
Experience with these methods has found them to be effective in finding from 30% to 70% of the logic design and coding errors in typical programs. They are not, however, effective in detecting high-level design errors, such as errors made in the requirements analysis process. Human processes find only the easy errors (those that would be trivial to find with computerbased testing) and the difficult, obscure, or tricky errors can only be found by computer-based testing. As an example stack overflow or memory leak in a module of a program. Inspections/walkthroughs and computer-based testing are complementary; error-detection efficiency will suffer if one or the other is not present. These processes are invaluable for testing modifications to programs. Because modifying an existing program is a more error-prone process(in terms of errors per statement written) than writing a new program.
43
4.4 CODE INSPECTIONS

An inspection team usually consists of four people. One of the four people is an expert. The expert is expected to be a competent programmer, but he/she is not the programmer of the code under inspection and need not be acquainted with the details of the program. The duties of the expert include:

Distributing materials for scheduling inspections Leading the session, Recording all errors found, and Ensuring that the errors are subsequently corrected.
Hence the expert may be called as quality-control engineer. The remaining members usually consist of the programs designer and a test specialist. The general procedure is that the expert distributes the programs listing and design specification to other participants well in advance of the inspection session. The participants are expected to familiarize themselves with the material prior to the session. During inspection session, two main activities occur: 1. The programmer is requested to narrate, statement by statement, the logic of the program. During the discourse, questions are raised and pursued to determine if errors exist. Experience has shown that many of the errors discovered are actually found by the programmer, rather than the other team members, during the narration. In other words, the simple act of reading aloud ones program to an audience seems to be a remarkably effective error-detection technique. 2. The program is analyzed with respect to a checklist of historically common programming errors (such a checklist is discussed in the next section). It is Test leads responsibility to ensure the smooth conduction of the proceedings and that the participants focus their attention on finding errors, not correcting them. After session, the programmer is given a list of the errors found. The list of errors is also analyzed, categorized, ad used to refine the error checklist to improve the effectiveness of future inspections.
The main benefits of this method are:

Identifying early, errors in the program, The programmers usually receive feedback concerning his or her programming style and choice of algorithms and programming techniques. Other participants are also gain in similar way by being exposed to another programmers errors and programming style.
44
The inspection process is a way of identifying early the most error-prone sections of the program, thus allowing one to focus more attention on these sections during the computer based testing processes.
4.5 AN ERROR CHECKLIST FOR INSPECTIONS

An important part of the inspection process is the use of a checklist to examine the program for common errors. The checklist is largely language independent as most of the errors can occur with any programming language.
Checklist for Data-Reference Errors

1. Is a variable referenced whose value is unset or uninitialized? This is probably the most frequent programming error; it occurs in a wide variety of circumstances. 2. For all array references, is each subscript value within the defined bounds of the corresponding dimension? 3. For all array references, does each subscript have an integer value? This is not necessarily an error in all languages, but it is a dangerous practice. 4. Check for dangling reference problem? Note: The Dangling reference problem occurs in situations where the lifetime of a pointer is greater than the lifetime of the referenced storage. 5. Are there any explicit or implicit addressing problems if on the machine being used, the units of storage allocation are smaller than the units of storage addressability? 6. If a data structure is referenced in multiple procedures or subroutines, is the structure defined identically in each procedure? 7. When indexing into a string, are the limits of the string exceeded?
Checklist for Data-Declaration Error

1. Have all variables been explicitly declared? A failure to do so is not necessarily an error, but it is a common source of trouble. 2. If all attributes of a variable are not explicitly stated in the declaration, are the defaults well understood? 3. Where a variable is initialized in a declarative statement, is it properly initialized?
45
4. Is each variable assigned the correct length, type, and storage class? 5. Is the initialization of a variable consistent with its storage type?
Checklist for Computation Errors

1. Are there any computations using variables having inconsistent (e.g. No arithmetic) data types? 2. Are there any mixed mode computations? 3. Are there any computations using variables having the same data type but different lengths? (Example: short int X + long int Y = short int Z) 4. Is the target variable of an assignment smaller than the right-hand expression? (Example int Product = float variable 1 * double of variable 2) 5. Is an overflow or underflow exception possible during the computation of an expression? That is, the end result may appear to have a valid value, but an intermediate result might be too big or too small for the machines data representations. 6. Is it possible for the divisor in a division operation to be zero? 7. Where applicable, can the value of a variable go outside its meaningful range? 8. Are there any invalid uses of integer arithmetic, particularly division? For example, if I is an integer variable, whether the expression 2*I/2 is equal to I depends on whether I has an odd or an even value and whether the multiplication or division is performed first.
Checklist Comparison Errors

1. Are there any comparisons between variables having inconsistent data types (e.g. comparing a character string to an address)? 2. Are there any mixed-mode comparisons or comparisons between variables of different lengths? If so, ensure that the conversion rules are well understood. 3. Does each Boolean expression state what it is supposed to state? Programmers often make mistakes when writing logical expressions involving and, or, and not. 4. Are the operands of a Boolean operator Boolean? Have comparison and Boolean operators been erroneously mixed together?
Checklist for Control-Flow Errors

1. If the program contains a multi way branch (e.g. a computed GO TO in FORTRAN), can the index variable ever exceed the number of branch possibilities? For example, in the Fortran statement,
46
GOTO (200,300,400), I
Will I always have the value 1, 2, or 3? Here, the index value must not go beyond 3 as there are only 3 branches. 2. Will every loop, function or program module eventually terminate? Devise an informal proof or argument showing that each loop will terminate 3. Is it possible that, because of the conditions upon entry, a loop will never execute? If so, does this represent an oversight? For instance, for loops headed by the following statements: DO WHILE (NOTFOUND) DO I=X TO Z What happens if NOTFOUND is initially false or if X is greater than Z? 5. Are there any non-exhaustive decisions? For instance, if an input parameters expected values are 1,2, or 3, does the logic assume that it must be 3 if it is not 1 or 2? If so, is the assumption valid?
Checklist for Interface Errors

1. Does the number of parameters received by this module equal the number of arguments sent by each of the calling modules? Also, is the order correct? 2. Do the attributes (e.g. type and size) of each parameter match the attributes of each corresponding argument? 3. In the procedural language check whether Does the number of arguments transmitted by this module to another module equal the number of parameters expected by that module? 4. Do the attributes of each argument transmitted to another module match the attributes of the corresponding parameter in that module? 5. If built-in functions are invoked, are the number, attributes, and order of the arguments correct? 6. Does the subroutine alter a parameter that is intended to be only an input value?
Input/Output Errors with respect to file handling

1. If files are explicitly declared, are their attributes correct? 2. Are the attributes on the file OPEN statement correct? 3. Is the size of the I/O area in storage equal to the record size?
47
4. Have all files been opened before use? 5. Are end-of-file conditions detected and handled correctly? 6. Are there spelling or grammatical errors in any text that is printed or displayed by the program? The above error checklist helps for all code reviewers to assess the quality of code written by the programmer. This also helps in reducing the post deployment work and increases the documentation.
4.6 WALKTHROUGHS
The code walkthrough, like the inspection, is a set of procedures and error-detection techniques for group code reading. It shares much in common with the inspection process, but the procedures are slightly different, and a different error-detection technique is employed. The walkthrough is an uninterrupted meeting of one to two hours in duration. The walkthrough team consists of three to five people to play the role of moderator, secretary (a person who records all errors found), tester and programmer. It is suggested to have other participants like:

A highly experienced programmer, A programming-language expert, A new programmer (to give a fresh, unbiased outlook) The person who will eventually maintain the program, Someone from different project and Someone from the same programming team as the programmer.
The initial procedure is identical to that of the inspection process: the participants are given the materials several days in advance to allow them to study the program. However, the procedure in the meeting is different. Rather than simply reading the program or using error checklists, the participants play computer. The person designated as the tester comes to the meeting armed with a small set of paper test casesrepresentative sets of inputs (and expected outputs) for the program or module. During the meeting, each test case is mentally executed. That is, the test data are walked through the logic of the program. The state of the program (i.e. the values of the variables) is monitored on paper or a blackboard. The test case must be simple and few in number, because people execute programs at a rate that is very slow compared to machines. In most walkthroughs, more errors are found during the process of questioning the programmer than are found directly by the test cases themselves.
48
QUESTIONS
1. 2. 3. 4. 5. 6.
Is code reviews are relevant to the software testing? Explain the process involved in a typical code review. Explain the need for inspection and list the different types of code reviews. Consider a program and perform detailed reviews and list the review findings in detail. Explain the difference between code walk through and Inspection. What are the duties of moderator during code inspection. Explain different types of errors for Inspection.
Chapter 5
Test Case Design
Dynamic testing of Software Applications White box and black box testing Various techniques used in white box testing Various techniques used in black box testing Static program analysis Automation of testing process
5.2 INTRODUCTION
Software can be tested either by running the programs and verifying each step of its execution against expected results or by statically examining the code or the document against its stated requirement or objective. In general, software testing can be divided into two categories, viz. static and dynamic testing. Static testing is a non-execution-based testing and carried through mostly by human effort. In static testing, we test, design, code or any document through inspection, walkthroughs and reviews as discussed in Chapter 2. Many studies show that the single most cost-effective defect reduction process is the classic structural test; the code inspection or walk-through. Code inspection is like proof reading and developers will be benefited in identifying the typographical errors, logic errors and deviations in styles and standards normally followed. BSIT 54 Software Quality and Testing
49
50
Chapter 5 - Test Case Design
Dynamic testing is an execution based testing technique. Program must be executed to find the possible errors. Here, the program, module or the entire system is executed (run) and the output is verified against the expected result. Dynamic execution of tests is based on specifications of the program, code and methodology.
5.3 WHITE BOX TESTING

This testing technique takes into account the internal structure of the system or component. The entire source code of the system must be available. This technique is known as white box testing because the complete internal structure and working of the code is available. Testers are transparent to the entire code and design documentation during the testing. White box testing helps to derive test cases to ensure: 1. All independent paths are exercised at least once. 2. All logical decisions are exercised for both true and false paths. 3. All loops are executed at their boundaries and within operational bounds. 4. All internal data structures are exercised to ensure validity. White box testing helps to:

Traverse complicated loop structures Cover common data areas Cover control structures and sub-routines Evaluate different execution paths Test the module and integration of many modules Discover logical errors, if any Helps to understand the code Uncover execution problems during testing
Why the white box testing is used to test conformance to requirements?
Logic errors and incorrect assumptions most likely to be made when coding for special cases. Need to ensure these execution paths are tested. This is also called as test of logical coverage.
51
May find assumptions about execution paths incorrect, and so make design errors. White box testing can find these errors. Typographical errors are random. Just as likely to be on an obscure logical path as on a mainstream path.
5.4 BASIS PATH TESTING

All software programs will have multiple paths of execution. The main aim is to derive the logical complexity of the procedural design of software and use this as a guide for defining a basic set of execution paths. Test cases, which exercise basic set, will execute every statement at least once.
5.4.1 Flow Graph Notation

Flow graph notation helps to represent various control structures of any programming language. Commonly used notations for representing control flow are shown in Figure 5.1.
Figure 5.1: Notations used for control structures
On a flow graph (Figure 5.1):

Arrows called edges represent flow of control Circles called nodes represent one or more actions Areas bounded by edges and nodes called regions
52
A predicate node is a node containing a condition
Any procedural design program can be translated into a flow graph by using these notations. Later the flow graph can be analyzed for various paths within it. Note that compound Boolean expressions at tests generate at least two predicate node and additional arcs. Figure 5.2 represents a sample control flow diagram and the corresponding flow diagram. Example:
Figure 5.2: Control flow of a program and the corresponding flow diagram
5.4.2 Cyclomatic Complexity

The cyclomatic complexity gives a quantitative measure of the logical complexity. This value gives the number of independent paths in the basis set, and an upper bound for the number of tests to ensure that each statement is executed at least once. An independent path is any path through a program that introduces at least one new set of processing statements or a new condition (i.e., a new edge)
53
Figure 5.3: Sample program and corresponding flow diagram
In Figure 5.3, the statements are numbered and the corresponding nodes also numbered with the same number. The sample program contains one DO and three nested IF statements. From the example we can observe that:
Cyclomatic Complexity of 4 can be calculated as: 1. Number of regions of flow graph, which is 4. 2. #Edges - #Nodes + 2, which is 11-9+2=4. 3. #Predicate Nodes + 1, which is 3+1=4.
The above complexity provides the upper bound on the number of tests cases to be generated or independent execution paths in the program. The independent paths (4 paths) for the program shown in Figure 5.3 are given below:
Independent Paths:
54
1. 1, 8 2. 1, 2, 3, 7b, 1, 8 3. 1, 2, 4, 5, 7a, 7b, 1, 8 4. 1, 2, 4, 6, 7a, 7b, 1, 8
Cyclomatic complexity provides upper bound for number of tests required to guarantee the coverage of all program statements.
5.4.3. Deriving Test Cases

Test cases are designed in many ways. The steps involved for test case design are: 1. Using the design or code, draw the corresponding flow graph. 2. Determine the cyclomatic complexity of the flow graph. 3. Determine a basis set of independent paths. 4. Prepare test cases that will force execution of each path in the basis set. Note: some paths may only be able to be executed as part of another test.
5.4.4. Graph Matrices

Graph matrices can automate derivation of flow graph and determination of a set of basis paths. Software tools to do this can use a graph matrix. A sample graph matrix is shown in Figure 5.4. The graph matrix:

Is a square matrix with number of sides equal to number of nodes Rows and columns of the matrix correspond to the number of nodes in the flow graph Entries correspond to the directing edges.
The matrix can associate a number with each entry of the edge. Use a value of 1 to calculate the cyclomatic complexity. The cyclomatic complexity is calculated as follows:
For each row, sum the values of each column and subtract 1 with the sum
55
Sum these final column totals and add 1.
In the example given in Figure 5,4, final value for each row is (1, 1,0,1,0,0,0,0). After adding these values with addition of 1 becomes 4. Some other interesting link weight can be measured by the graph as:

Probability that a link (edge) will be executed Processing time for traversal of a link Memory required during traversal of a link Resources required during traversal of a link
Figure 5.4: Example of a graph matrix
56 5.5 CONTROL STRUCTURE TESTING
In programs, conditions are very important and testing such conditions is more complex than other statements like assignment and declarative statements. Basic path testing is one example of control structure testing. There are many ways in which control structure can be tested.
5.5.1. Conditions Testing

Condition testing aims to exercise all logical conditions in a program module. Logical conditions may be complex or simple. Logical conditions may be nested with many relational operations. It can defined as:
Relational expression: (E1 op E2), where E1 and E2 are arithmetic expressions. For example, (x+y) (s/t), where x, y, s and t are variables. Simple condition: Boolean variable or relational expression, possibly proceeded by a NOT operator. Compound condition: composed of two or more simple conditions, Boolean operators and parentheses along with relational operators. Boolean expression: Condition without relational expressions.
Normally errors in expressions can be due to due to one or all or the following:

Boolean operator error Boolean variable error Boolean parenthesis error Relational operator error Arithmetic expression error Mismatch of types
Condition testing methods focus on testing each condition in the program and it could be of any type of conditions. There are many strategies to identify errors. Some of the strategies proposed include:

Branch testing: Every branch is executed at least once. Domain Testing: Uses three or four tests for every relational operator depending on the complexity of the statement.
57
Branch and relational operator testing: Uses condition constraints. Based on the complexity of the relational operators, many branches will be executed.
Example: C1 = B1 & B2

Where B1, B2 are Boolean conditions.. Condition constraint of form (D1,D2) where D1 and D2 can be true (t) or false(f). The branch and relational operator test requires the constraint set {(t,t),(f,t),(t,f)} to be covered by the execution of C1.
Coverage of the constraint set guarantees detection of relational operator errors.
5.5.2. Data Flow Testing

First, a proper data flow diagram like control flow (see basis path flow) is drawn. Then selects test paths according to the location of definitions and use of variables. Any variables that have been defined in any program behave in the following way: D: define the variable, normally defined in declarative section, U: use the variables which are defined earlier, in the program. K: kill the variable, which is another state of the variable at any time of the execution of the program. Any variable that is part of the program will undergo any of the above state. However, the sequence of states is important. We can avoid following anomalies during the program execution:

DU: Normal, UK, UU: Normal, DD: Suspicious DK: Probable bug KD: Normal KK: Probable bug KU: bug UD: Normal
An example could be:
58
DU: Normal means a variable is defined first and then used in the program which is normal behavior of the data flow in the program. DK: Probable bug means a variable is defined and then killed before using in the program. This may be bug as why the variable is defined and killed with out using in the program. This type of testing is useful and can be automated.
5.5.3. Loop Testing

Loops are fundamental to many algorithms. Loops can be categorized as, define loops as simple, concatenated, nested, and unstructured. Loops can be defined in many ways. Consider the example given in Figure 5.5 which represents control structures and loops for a typical program.
Simple
Nested
Concatenate
Unstructured
Figure 5.5: Different types of Loops
59
Along with testing the control constructs, we have to test loops also. To test the loops, following guidelines may be followed:
Simple Loops of size n: o Skip loop entirely o Only one pass through the loop o Two passes through the loop o m passes through loop where m<n. o (n-1), n, and (n+1) passes through the loop. This helps in testing the boundary of the loops.
Nested Loops o Start with inner loop. Set all other loops to minimum values. o Conduct simple loop testing on inner loop. o Work outwards and take the next nested loop. o Continue until all loops are tested.
Concatenated Loops o If independent loops, use simple loop testing. o If dependent, treat as nested loops.
Unstructured loops o Dont test - redesign. This is known as poor design.
Care must be taken to test the loops as loops are complex compare to control structures.
5.6 BLACK BOX TESTING

Functional tests examine the observable behavior of software as evidenced by its outputs, without any reference to internal functions. This kind of tests is from the user point of view, which means as if the user is testing as in the normal business functions.
Black box tests normally determine the quality of the software. It is an advantage to create the quality criteria from this point of view from the beginning. In black box testing, software is subjected to a full range of inputs and the outputs are verified for its correctness. Here, the structure of the program is immaterial. Black box testing technique can be applied once unit and integration testing is completed. It focuses on functional requirements.
60
It is compliment to the white box testing.
The main objective of the black box testing is to find: 1. Incorrect or missing functions 2. Interface errors 3. Errors in data structures or external database access 4. Performance errors 5. Initialization and termination errors After knowing the basic objectives of black box testing, how do you conduct black box testing? Various techniques are used for black box testing. Some of the techniques used for black box testing are discussed below:
5.6.1 Equivalence Partitioning

The main objective of this method is to partitioning the input so that an optimal input data is selected. Steps to be followed are: 1. Divide the input domain into classes of data for which test cases can be generated. 2. Attempting to uncover classes of errors, if any. 3. Identify the both right and wrong input data while partitioning the data. 4. Test the program for all types of data based on equivalence classes for input conditions. An equivalence class represents a set of valid or invalid states. An input condition is a specific numeric value, range of values, a set of related values, or a Boolean condition. Equivalence classes can be defined by:
If an input condition specifies a range or a specific value, one valid and two invalid equivalence classes defined. If an input condition specifies a Boolean or a member of a set, one valid and one invalid equivalence classes defined.
Test cases for each input domain data item developed and executed. This method uses less number of input data compare to exhaustive testing. However, the data for boundary values are not considered. This method though reduces significantly the number of input data to be tested; it does not test the combinations of the input data. Example: If an input condition specifies that the range of values of the input variable items must be from 1 to 999, one valid equivalence class would be 1<=items<=999 and 2 invalid equivalence classes would be items<1 and items>999.
61
5.6.2. Boundary Value Analysis

It is observed that boundary points for any inputs are not tested properly. This leads to many errors. Large numbers of errors tend to occur at boundaries of the input domain. Boundary Value Analysis (BVA) leads to selection of test cases that exercise boundary values. BVA complements equivalence partitioning i.e. select any element in an equivalence class, select those at the edge of the class. Examples: 1. For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1). Consider If a = 1 and b = 999 then the boundary values would be 0,1,2,999,1000 etc. 2. If input conditions specify a number of values n, test with (n-1), n and (n+1) input values. 3. Apply 1 and 2 to output conditions (e.g., generate table of minimum and maximum size). 4. If internal program data structures have boundaries (e.g., buffer size, table limits), use input data to exercise structures on boundaries. BVA and equivalence partitioning both helps in testing the programs and covers most of the conditions. This method does not test the combinations of input conditions.
5.6.3. Cause Effect Graphing Techniques

Translation of natural language descriptions of procedures to software based algorithms is error prone. Example: From US Army Corps of Engineers: Executive Order 10358 provides in the case of an employee whose work week varies from the normal Monday through Friday work week, that Labor Day and Thanksgiving Day each were to be observed on the next succeeding workday when the holiday fell on a day outside the employees regular basic work week. Now, when Labor Day, Thanksgiving Day or any of the new Monday holidays are outside an employees basic workbook, the immediately preceding workday will be his holiday when the non-workday on which the holiday falls is the second non-workday or the non-workday designated as the employees day off in lieu of Saturday. When the non-workday on which the holiday falls is the first non-workday or the non-workday designated as the employees day off in lieu of Sunday, the holiday observance is moved to the next succeeding workday.
How do you test code, which attempts to implement this?

Cause-effect graphing attempts to provide a concise representation of logical combinations and corresponding actions.
62
1. Causes (input conditions) and effects (actions) are listed for a module and an identifier is assigned to each. 2. A cause-effect graph developed. 3. Graph converted to a decision table. 4. Decision table rules are converted to test cases. Simplified Symbols used are shown in Figure 5.6.
Fig 5.6: Representation of cause-effect nodes
63
5.6.4. Comparison Testing

In some applications the reliability is critical. Hence redundant hardware and software may be used. For redundant software, use separate teams to develop independent versions of the software. Test each version with same test data to ensure all provisional identical output. Run all versions in parallel with a real-time comparison of results. Even if it run only one version in the final system, for some critical applications can develop independent versions and use comparison testing or back-to-back testing. When outputs of versions differ, each is investigated to determine if there is a defect. Some method does not catch errors in the specification.
5.7 STATIC PROGRAM ANALYSIS

This strategy helps in identifying errors without executing the program. Peer reviewers and programmers will use this strategy to uncover probable static errors.
5.7.1. Program Inspections

This is part of the static testing where in the code is tested for its design standards, possible logical errors, type errors through code walk through. The details have covered in the previous Chapter.
5.7.2. Mathematical Program Verification

If the programming language semantics are formally defined, one can consider program to be a set of mathematical statements. We can attempt to develop a mathematical proof that the program is correct with respect to the specification. If the proof can be established, the program is verified and testing to check verification is not required. There are a number of approaches to proving program correctness. We will only consider axiomatic approach. Suppose that at points P(1), .. , P(n) assertions concerning the program variables and their relationships can be made. The assertions are a(1), ..., a(n). The assertion a(1) is about inputs to the program, and a(n) about outputs. We can now attempt, for k between 1 and (n-1), to prove that the statements between
64
P(k) and P(k+1) transform the assertion a(k) to a(k+1).
Given that a(1) and a(n) are true, this sequence of proofs shows partial program correctness. If it can be shown that the program will terminate, the proof is complete.
5.7.3. Static Program Analyzers

Static analysis tools scan the source code to try to detect errors. The code does not need to be executed. It uses most useful for languages which do not have strong typing. It can check: 1. Syntax. 2. Unreachable code 3. Unconditional branches into loops 4. Undeclared variables 5. Uninitialised variables 6. Parameter type mismatches 7. Uncalled functions and procedures 8. Variables used before initialization 9. Non-usage of function results 10.Possible array bound errors 11. Misuse of pointers
5.8 AUTOMATED TESTING TOOLS

Automation of testing is the state of the art technique where in number of tools will help in testing program automatically. Programmers can use any tool to test his/her program and ensure the quality. There are number of tools are available in the market. Some of the tools which help the programmer are: 1. Static analyzer 2. Code Auditors
65
3. Assertion processors 4. Test file generators 5. Test Data Generators 6. Test Verifiers 7. Output comparators. Programmer can select any tool depending on the complexity of the program. All tools only assist in testing but the initial effort must be more to design and develop test cases.
QUESTIONS
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. What is black box testing? Explain. What are the different techniques are available to conduct black box testing? Explain different methods available in white box testing with examples. What is basis path testing? Explain. Explain the different notations used for control structures. What is cyclomatic complexity? Explain in detail. Write a short note on Graph matrices. Explain the condition testing in detail. Which anomalies are avoided during the program execution? Explain. Explain different ways of loops. Explain the guidelines for testing the loop. Explain the main objectives of equivalence partitioning. What do you mean by boundary value analysis? Explain. Explain the cause effect graphing techniques. Explain the following: 1) 2) Static program analysers. Automated testing tools.

BSIT 54 Main PG 1 65

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

BSIT 54 Main PG 1 65

Caricato da

Copyright:

Formati disponibili

Chapter 1

Introduction to Software Testing

1.1 LEARNING OBJECTIVES

ou will learn about:

BSIT 54 Software Quality and Testing

Chapter 1 - Introduction to Software Testing

What is the purpose of Testing?

Why do we need to Test the application?

Poor understanding and incomplete requirements

BSIT 54 Software Quality and Testing

1.3 WHAT IS TESTING?

Chapter 1 - Introduction to Software Testing

Some of the points to be noted regarding testing are:

1.4 APPROACHES TO TESTING

BSIT 54 Software Quality and Testing

1.5 IMPORTANCE OF TESTING

Chapter 1 - Introduction to Software Testing

1.6 HURDLES IN TESTING

1.7 DEFECT DISTRIBUTION

BSIT 54 Software Quality and Testing

1.8 TESTING FUNDAMENTALS

1.8.1 Testing Objectives

Secondary benefits include

Testing demonstrates that software functions appear to be working according to specification.

Chapter 1 - Introduction to Software Testing

1.8.2 Test Information Flow

BSIT 54 Software Quality and Testing

1.8.3 Test Case Design

Chapter 1 - Introduction to Software Testing

Types of Software Testing

In God, We Trust All else, We Test - Anon

2.1 LEARNING OBJECTIVES

ou will learn about:

BSIT 54 Software Quality and Testing

12 2.3 TYPES OF SOFTWARE TESTING

Chapter 2 - Types of Software Testing

2. White Box Testing

3. Grey Box Testing

BSIT 54 Software Quality and Testing

10. Configuration/ Platform Testing

This is a mandatory testing for all application before deployment.

11. Localization Testing

12. Internationalization Testing

Chapter 2 - Types of Software Testing

Response time, Resource utilization, and Reliability

Performance testing helps in acquiring the customers confidence on the system.

15. Load Testing

17. Benchmark Testing

18. Functional Testing

19. Formal Inspections

BSIT 54 Software Quality and Testing

20. Consistency Testing

21. Regression Testing

22. Smoke Testing

23. Sanity Testing

24. Top Down Testing

25. Bottom Up Testing

Chapter 2 - Types of Software Testing

27. Automated Testing

28. Heuristic Testing

29. User Testing

30. Protection Testing

31. Data Driven Test