Testing is a process used to help identify the correctness, completeness and quality of

developed computer software. With that in mind, testing can never completely establish
the correctness of computer software.

There are many approaches to software testing, but effective testing of complex
products is essentially a process of investigation, not merely a matter of creating and
following rote procedure. One definition of testing is "the process of questioning a
product in order to evaluate it", where the "questions" are things the tester tries to do with
the product, and the product answers with its behavior in reaction to the probing of the
tester. Although most of the intellectual processes of testing are nearly identical to that of
review or inspection, the word testing is connoted to mean the dynamic analysis of the
product—putting the product through its paces.

The quality of the application can and normally does vary widely from system to system
but some of the common quality attributes include reliability, stability, portability,
maintainability and usability. Refer to the ISO standard ISO 9126 for a more complete
list of attributes and criteria.

Testing helps is Verifying and Validating if the Software is working as it is intended to be

working. Thins involves using Static and Dynamic methodologies to Test the application.

Because of the fallibility of its human designers and its own abstract, complex nature,
software development must be accompanied by quality assurance activities. It is not
unusual for developers to spend 40% of the total project time on testing. For life-critical
software (e.g. flight control, reactor monitoring), testing can cost 3 to 5 times as much as
all other activities combined. The destructive nature of testing requires that the developer
discard preconceived notions of the correctness of his/her developed software.

Software Testing Fundamentals

Testing objectives include

1. Testing is a process of executing a program with the intent of finding an error.

2. A good test case is one that has a high probability of finding an as yet undiscovered
error.
3. A successful test is one that uncovers an as yet undiscovered error.

Testing should systematically uncover different classes of errors in a minimum amount of

time and with a minimum amount of effort. A secondary benefit of testing is that it
demonstrates that the software appears to be working as stated in the specifications. The
data collected through testing can also provide an indication of the software's reliability
and quality. But, testing cannot show the absence of defect -- it can only show that
software defects are present.
 Testing Start Process

Testing is sometimes incorrectly thought as an after-the-fact activity; performed after

programming is done for a product. Instead, testing should be performed at every
development stage of the product. Test data sets must be derived and their correctness and
consistency should be monitored throughout the development process.

If we divide the lifecycle of software development into “Requirements Analysis”,

“Design”, “Programming/Construction” and “Operation and Maintenance”, then testing
should accompany each of the above phases. If testing is isolated as a single phase late in
the cycle, errors in the problem statement or design may incur exorbitant costs. Not only
must the original error be corrected, but the entire structure built upon it must also be
changed. Therefore, testing should not be isolated as an inspection activity. Rather testing
should be involved throughout the SDLC in order to bring out a quality product.

The following testing activities should be performed during the phases:

1. Requirements Analysis
- Determine correctness
- Generate functional test data.
2. Design
- Determine correctness and consistency
- Generate structural and functional test data.

1. Programming/Construction
- Determine correctness and consistency
- Generate structural and functional test data
- Apply test data
- Refine test data
2. Operation and Maintenance
- Retest

The following test activities should be performed during this stage:

1.1 Invest in analysis at the beginning of the project - Having a clear, concise and formal
statement of the requirements facilitates programming, communication, error analysis
and test data generation.

The requirements statement should record the following information and

decisions:

a. Program function - What the program must do?

b. The form, format, data types and units for input.

c. The form, format, data types and units for output.

d. How exceptions, errors and deviations are to be handled.

e. For scientific computations, the numerical method or at least the required accuracy of
the solution.

f. The hardware/software environment required or assumed (e.g. the machine, the

operating system, and the implementation language).

Deciding the above issues is one of the activities related to testing that should be
performed during this stage.
1.2 Start developing the test set at the requirements analysis phase - Data should be
generated that can be used to determine whether the requirements have been met. To do
this, the input domain should be partitioned into classes of values that the program will
treat in a similar manner and for each class a representative element should be included in
the test data.

In addition, following should also be included in the data set:

(1) boundary values

(2) any non-extreme input values that would require special handling.

The output domain should be treated similarly.

Invalid input requires the same analysis as valid input.

1.3 The correctness, consistency and completeness of the requirements should also be
analyzed - Consider whether the correct problem is being solved, check for conflicts and
inconsistencies among the requirements and consider the possibility of missing cases.

Design

The design document aids in programming, communication, and error analysis and test
data generation. The requirements statement and the design document should together
give the problem and the organization of the solution i.e. what the program will do and
how it will be done.

The design document should contain:

• Principal data structures.

• Functions, algorithms, heuristics or special techniques used for processing.
• The program organization, how it will be modularized and categorized into
external and internal interfaces.
• Any additional information.
Here the testing activities should consist of:

- Analysis of design to check its completeness and consistency - the total process should
be analyzed to determine that no steps or special cases have been overlooked. Internal
interfaces, I/O handling and data structures should specially be checked for
inconsistencies.

- Analysis of design to check whether it satisfies the requirements - check whether both
requirements and design document contain the same form, format, units used for input
and output and also that all functions listed in the requirement document have been
included in the design document. Selected test data which is generated during the
requirements analysis phase should be manually simulated to determine whether the
design will yield the expected values.

- Generation of test data based on the design - The tests generated should cover the
structure as well as the internal functions of the design like the data structures, algorithm,
functions, heuristics and general program structure etc. Standard extreme and special
values should be included and expected output should be recorded in the test data.

- Re-examination and refinement of the test data set generated at the requirements
analysis phase.

The first two steps should also be performed by some colleague and not only the
designer/developer.

Here the main testing points are:

- Check the code for consistency with design - the areas to check include modular
structure, module interfaces, data structures, functions, algorithms and I/O handling.

- Perform the Testing process in an organized and systematic manner with test runs dated,
annotated and saved. A plan or schedule can be used as a checklist to help the
programmer organize testing efforts. If errors are found and changes made to the
program, all tests involving the erroneous segment (including those which resulted in
success previously) must be rerun and recorded.

- Asks some colleague for assistance - Some independent party, other than the
programmer of the specific part of the code, should analyze the development product at
each phase. The programmer should explain the product to the party who will then
question the logic and search for errors with a checklist to guide the search. This is
needed to locate errors the programmer has overlooked.

- Use available tools - the programmer should be familiar with various compilers and
interpreters available on the system for the implementation language being used because
they differ in their error analysis and code generation capabilities.
- Apply Stress to the Program - Testing should exercise and stress the program structure,
the data structures, the internal functions and the externally visible functions or
functionality. Both valid and invalid data should be included in the test set.
- Test one at a time - Pieces of code, individual modules and small collections of modules
should be exercised separately before they are integrated into the total program, one by
one. Errors are easier to isolate when the no. of potential interactions should be kept
small. Instrumentation-insertion of some code into the program solely to measure various
program characteristics – can be useful here. A tester should perform array bound checks,
check loop control variables, determine whether key data values are within permissible
ranges, trace program execution, and count the no. of times a group of statements is
executed.

- Measure testing coverage/When should testing stop? - If errors are still found every
time the program is executed, testing should continue. Because errors tend to cluster,
modules appearing particularly error-prone require special scrutiny.

The metrics used to measure testing thoroughness include statement testing (whether
each statement in the program has been executed at least once), branch testing (whether
each exit from each branch has been executed at least once) and path testing (whether all
logical paths, which may involve repeated execution of various segments, have been
executed at least once). Statement testing is the coverage metric most frequently used as
it is relatively simple to implement.

The amount of testing depends on the cost of an error. Critical programs or functions
require more thorough testing than the less significant functions.

Operations and Maintenance

Corrections, modifications and extensions are bound to occur even for small programs
and testing is required every time there is a change. Testing during maintenance is termed
regression testing. The test set, the test plan, and the test results for the original program
should exist.

Modifications must be made to accommodate the program changes, and then all portions
of the program affected by the modifications must be re-tested. After regression testing is
complete, the program and test documentation must be updated to reflect the changes.
 Testing Stop Process

This can be difficult to determine. Many modern software applications are so complex,
and run in such as interdependent environment, that complete testing can never be done.
"When to stop testing" is one of the most difficult questions to a test engineer. Common
factors in deciding when to stop are:

• Deadlines ( release deadlines,testing deadlines.)

• Test cases completed with certain percentages passed
• Test budget depleted
• Coverage of code/functionality/requirements reaches a specified point
• The rate at which Bugs can be found is too small
• Beta or Alpha Testing period ends
• The risk in the project is under acceptable limit.

Practically, we feel that the decision of stopping testing is based on the level of the risk
acceptable to the management. As testing is a never ending process we can never assume
that 100 % testing has been done, we can only minimize the risk of shipping the product
to client with X testing done. The risk can be measured by Risk analysis but for small
duration / low budget / low resources project, risk can be deduced by simply: -

• Measuring Test Coverage.

• Number of test cycles.
• Number of high priority bugs.

The Software Assurance Technology Center (SATC) in the Systems Reliability and
Safety Office at Goddard Space Flight Center (GSFC) is investigating the use of software
error data as an indicator of testing status. Items of interest for determining the status of
testing include projections of the number of errors remaining in the software and the
expected amount of time to find some percentage of the remaining errors.

To project the number of errors remaining in software, one needs an estimate of the
total number of errors in the software at the start of testing and a count of the errors found
and corrected throughout testing. There are a number of models that reasonably fit the
rate at which errors are found in software, the most commonly used is referred to in this
paper as the Musa model. This model is not easily applicable at GSFC, however, due to
the availability and the quality of the error data.

At GSFC, useful error data is not easy to obtain for projects not in the Software
Engineering Laboratory. Of the projects studied by the SATC, only a few had an
organized accounting scheme for tracking errors, but they often did not have a consistent
format for recording errors. Some projects record errors that were found but did not
record any information about resources applied to testing. The error data frequently
contained the date of entry of the error data rather than the actual date of error discovery.
In order to use traditional models such as the Musa model for estimating the cumulative
number of errors, one needs fairly precise data on the time of discovery of errors and the
level of resources applied to testing. Real world software projects are generally not very
accommodating when it comes to either accuracy or completeness of error data. The
models developed by the SATC to perform trending and prediction on error data attempt
to compensate for these shortcomings in the quantity and availability of project data.

In order to compensate for the quality of the error data, the SATC developed a software
error trending models using two techniques, each based on the basic Musa model, but
with the constant in the exponential term replaced by a function of time that describes the
'intensity' of the testing effort. The shape and the parameters for this function can be
estimated using measures such as CPU time or staff hours devoted to testing. The first
technique involves fitting cumulative error data to the modified Musa model using a least
squares fit that is based on gradient methods. This technique requires data on errors found
and the number of staff hours devoted to testing each week of the testing activity. The
second technique uses a Kalman filter to estimate both the total number of errors in the
software and the level of testing being performed. This technique requires error data and
initial estimates of the total number of errors and the initial amount of effort applied to
testing.

The SATC has currently examined and modeled error data from a limited number of
projects. Generally, only the date on which an error was entered into the error tracking
system was available, not the date of discovery of the error. No useful data was available
on human or computer resources expended for testing.

What is needed for the most accurate model is the total time expended for testing,
even if the times are approximate. Using the sum of reported times to find/fix individual
errors did not produce any reasonable correlation with the resource function required.
Some indirect attempts to estimate resource usage, however, led to some very good fits.
On one project errors were reported along with the name of the person that found the
error. Resource usage for testing was estimated as follows: A person was estimated to be
working on the testing effort over a period beginning with the first error that they
reported and ending with the last error that they reported. The percentage of time that
each person worked during that period was assumed to be an unknown constant that did
not differ from person to person. Using this technique led to a resource curve that closely
resembled the Rayleigh curve (Figure 1).
Figure 1: Test Resource Levels for Project A

On most of the projects, there was good conformity between the trend model and the
reported error data. More importantly, estimates of the total number of errors and the
error discovery parameter, made fairly early in the testing activity, seemed to provide
reliable indicators of the total number of errors actually found and the time it took to find
future errors.

Figures 2 shows the relationship between reported errors and the SATC trend model
for one project. The graph represents data available at the conclusion of the project. This
close fit was also found on other projects when sufficient data was available.

Figure 2: Cumulative Software Errors for Project A

On another project, different estimates of the total number of errors were obtained when
estimates were made over different testing time intervals. That is, there was inconsistent
agreement between the trend model and the error data over different time intervals.
Through subsequent discussion with the project manager it was learned that the rate of
error reporting by the project went from approximately 100% during integration testing to
40% during acceptance testing. Furthermore, there was a significant amount of code
rework, and testing of the software involved a sequential strategy of completely testing a
single functional area before proceeding to test the next functional area of the code.

Thus, the instability of the estimates of the total errors was a useful indicator of the
fact that there was a significant change in either the project's testing and reporting
process. Figure 3 shows the results for this project. Note the change in slope of the
reported number of errors occurring around 150 days. The data curve flattens at the right
end of the curve due to a pause in testing, rather than a lack of error detection. This
project is still undergoing testing.

Figure 3: Cumulative S/W Errors for Project B - Flight S/W

If error data is broken into the distinct testing phases of the life cycle (e.g., unit, system,
integration), the projected error curve using the SATC model closely fits the rate at which
errors are found in each phase.

Some points need to be clarified about the SATC error trend model. The formulation
of the SATC equation is the direct result of assuming that at any instant of time, the rate
of discovery of errors is proportional to the number of errors remaining in the software
and to the resources applied to finding errors. Additional conditions needed in order for
the SATC trending model to be a valid are:

1. The code being tested is not being substantially altered during the testing process,
especially through the addition or rework of large amounts of code.
2. All errors found are reported.
3. All of the software is tested, and testing of the software is uniform throughout the
time of the testing activity.

Condition 1 is present to ensure that the total number of errors is a relatively stable
number throughout the testing activity. Conditions 2 and 3 are present to ensure that the
estimate of the total number of errors is in fact an estimate of the total errors present in
the software at the start of testing - no new errors are introduced during testing. If testing
is not "uniform" then the rate of error discovery will not necessarily be proportional to
the number of errors remaining in the software and so the equation will not be an
appropriate model for errors found. No attempt will be made here to make precise the
meaning of the word "uniform".

The SATC developed this model rather than using the standard Musa model because it
seems less sensitive to data inaccuracy and provides for non-constant testing resource
levels. An additional benefit from this work is the application of the Rayleigh curve for
estimating resource usage. Future work by the SATC will continue to seek a practical
balance between available trend analysis theory and the constraints imposed by the
limited accuracy and availability of data from real-world projects.

Testing Strategy
Test Strategy

How we plan to cover the product so as to develop an adequate assessment of quality.

A good test strategy is:

• Specific
• Practical
• Justified

The purpose of a test strategy is to clarify the major tasks and challenges of the test
project.

Test Approach and Test Architecture are other terms commonly used to describe what I’m
calling test strategy.

Example of a poorly stated (and probably poorly conceived) test strategy:

"We will use black box testing, cause-effect graphing, boundary testing, and white box
testing to test this product against its specification."

Contents:

 Creating Test Strategy

 Defining Test Strategy
 Requirements in Test Strategy
 Key points to remember
 Creating a Test Strategy

The test strategy is a formal description of how a software product will be tested. A test
strategy is developed for all levels of testing, as required. The test team analyzes the
requirements, writes the test strategy and reviews the plan with the project team. The test
plan may include test cases, conditions, the test environment, a list of related tasks,
pass/fail criteria and risk assessment.

Inputs for this process:

• A description of the required hardware and software components, including test

tools. This information comes from the test environment, including test tool data.
• A description of roles and responsibilities of the resources required for the test
and schedule constraints. This information comes from man-hours and schedules.

• Testing methodology. This is based on known standards.

• Functional and technical requirements of the application. This information comes
from requirements, change request, technical and functional design documents.
• Requirements that the system can not provide, e.g. system limitations.

Outputs for this process:

• An approved and signed off test strategy document, test plan, including test cases.
• Testing issues requiring resolution. Usually this requires additional negotiation at
the project management level.

Defining a Test Strategy

A solid testing strategy provides the framework necessary to implement your testing
methodology. A separate strategy should be developed for each system being developed
taking into account the development methodology being used and the specific application
architecture.

The heart of any testing strategy is the master testing strategy document. It aggregates all
the information from the requirements, system design and acceptance criteria into a
detailed plan for testing. A detailed master strategy should cover the following:

Project Scope

Restate the business objective of the application and define the scope of the testing.
The statement should be a list of activities that will be in scope or out of scope. A
sample list would include:

* List of software to be tested

* Software configurations to be tested
* Documentation to be validated
* Hardware to be tested

Test Objectives

The system under test should be measured by its compliance to the requirements and
the user acceptance criteria. Each requirement and acceptance criteria must be
mapped to specific test plans that validate and measure the expected results for each
test being performed. The objectives should be listed in order of importance and
weighted by Risk. Features and Functions to be Tested

Every feature and function must be listed for test inclusion or exclusion, along with a
description of the exceptions. Some features may not be testable due to a lack of
hardware or lack of control etc. The list should be grouped by functional area to add
clarity. The following is a basic list of functional areas:

* Backup and recovery

* Workflow
* Interface design
* Installation
* Procedures (users, operational, installation)
* Requirements and design
* Messaging
* Notifications
* Error handling
* System exceptions and third-party application faults

Testing Approach

The approach provides the detail necessary to describe the levels and types of testing.
The basic V-Model shows what types of testing are needed to validate the system.
More specific test types include functionality, performance testing, backup and
recovery, security testing, environmental testing, conversion testing, usability testing,
installation and regression testing. The specific testing methodology should be
described and the entry/exit criteria for each phase noted in a matrix by phase. A
project plan that list the resources and schedule for each testing cycle should also be
created that maps the specific testing task to the overall development project plan.

Testing Process and Procedures

The order of test execution and the steps necessary to perform each type of test
should be described in sufficient detail to provide clear input into the creation of test
plans and test cases. Procedures should include how test data is created, managed and
loaded. Test cycles should be planned and scheduled based on system availability and
deliverable dates from development. All application and environmental dependencies
should be identified along with the procedures necessary to gain access to all the
dependent systems.

Test Compliance

Every level of testing must have a defined set of entry/exit criteria which is used to
validate that all prerequisites for a valid test have been met. All mainstream software
testing methodologies provide an extensive list of entry/exit criteria and checklist. In
addition to the standard list, additional items should be added based on specific
testing needs. Some common additions are, environmental availability, data
availability, and validated code which is ready to be tested.

Each level of testing should define specific pass/fail acceptance criteria, to ensure to
ensure that all quality gates have been validated and that the test plan focuses on
developing test that validate the specific criteria defined by the user acceptance plan.

Testing Tools

All testing tools should be identified and their use, ownership and dependencies
defined. The tools category includes manual tools, such as templates in spreadsheets
and documents as well as automated tools for test management, defect tracking,
regression testing and performance/load testing. Any specific skill sets should be
identified and compared against the existing skills identified for the project to
highlight any training needs.

Defect Resolution

A plan to address the resolution of failed tests needs to be created that lists the
escalation procedures to seek correction and retest of the failed tests along with a risk
mitigation plan for high-risk test. Defect tracking should include basic metrics for
compliance based on number and type of defect found.

Roles and Responsibilities

A matrix listing the roles and responsibilities of everyone involved in the testing
activities, along with the anticipated amount of their time allocated to the project,
must be prepared.

Process Improvement

The entire testing process should be focused on process improvement. The strategy
should list ways to monitor progress and provide constant feedback. This feedback
can serve to enhance the process, deliverables and metrics used in the testing. Root
cause analysis should be performed on all reported defects to help isolate the true
nature of the problem and prevent unnecessary repeat offenses.

Deliverables

All deliverables should be defined and their location specified. Common deliverables
are test plans, test cases, test scripts, test matrix and a defect log.

Schedule

All testing activities should be combined into one master testing schedule. The
schedule should include an estimate of time for each task and the dependences for
each. Testing resources should be assigned to each task and quality gates should be
listed to insure oversight of the entire process.

Environmental Needs

All the requirements of the testing environment need to be listed. Common ones
include a description of the environment's use, management, hardware and software,
specific tools needed, data loading and security requirements.

Resource Management

The skills of all personnel involved in the testing effort need to be assessed and the
gaps noted so that a comprehensive training program can be designed. Specialty skills
that will not be filled with in-house staff will require job descriptions and budgeting.

Risk and Contingencies

Planning for risk in advance and ways to mitigate it are essential for a robust strategy.
A risk assessment that is prioritized by severity of risk and covers technology,
resource, schedule and environmental issues should feed a detailed plan to mitigate
each red flag.

Approvals and Workflow

All items on the critical path must go through an approval cycle. The procedures for
approval and escalation must be well defined and assigned to resources prior to the
start of the testing.

The above covers the main sections of a well-drafted and documented testing
strategy. The more detail that you include in the strategy document, the less ambiguity
and chance for deviation there will be throughout the project

The completion of the strategy signals the beginning of the test planning phase. For
 each type of testing identified in the master test strategy there should be a test plan
identifying the components to be tested, the location of the test data, the test
environment needs, the test procedures, resources required, and the tests schedule.
For each plan a series of test conditions should be identified so that test cases with
expected results can be generated for later execution.

What to include in the Testing Strategy

During the analysis phase, you gather and validate the business requirements for the
solution. It makes sense that the Testing Strategy is completed during this phase as well.
In a sense, you are defining the overall testing requirements.

The purpose of the Testing Strategy is to define the overall context for the entire testing
process. The process is different depending on the specific characteristics of your
solution. In many respects, this is the most important part of the testing process, since all
future testing decisions will be made within the context of the strategy. Here are the basic
parts of the testing strategy:

• Project Overview: You can copy this from the Project Definition.

• Business Risks: These are high-level risks of the project that will affect the
overall testing strategy. For instance, the risk of doing business on the Internet
may drive the need for rigorous system tests of firewalls, technical architecture,
and security. The risks can be classified as high, medium, and low, depending on
the nature and impact of the problem. For each high and medium risk, identify
what elements in the overall testing approach will help ensure that the potential
problem does not occur.
o Testing Milestones: This section gives the reader a preliminary overview
of the testing timelines. Obviously, since this document is created in the
analysis phase, these dates are subject to later revision.

o Testing Approach: This describes the testing process at a high level,

including how you will conduct unit testing, integration testing, system
testing, and acceptance testing. (If your project is large enough, each of
these might be its own section.) This is where you make fundamental
decisions regarding the type of testing that makes sense for your project.
For instance, if you are implementing a packaged solution, the approach
 may start in system testing, with the vendor providing close support. If
you are doing iterative development cycles, the testing approach will
reflect this overall development life cycle. For system testing, define the
major testing events, such as stress testing, security testing, disaster
recovery testing, usability testing, and response time testing.

o Testing Environment: Think through the technologies and facilities

needed for the testing process. If the overall testing environment needs are
understood up front, it will be easier to break out the specific activities
required to put the environment in place. In addition, you may need to
plan for and acquire some parts of the environment well in advance.

Depending on your project, there may be other high-level sections to include,

such as testing objectives, testing assumptions, testing organization, and testing
tools, along with effort and cost estimates.

Key points to remember

The developer’s role here is not trivial. As you contribute to this strategy document, keep
these three points in mind:

• If you are working on a large project, you need to formulate an overall Testing
Strategy during the analysis phase. The Testing Strategy defines the overall
approach to testing and describes how the testing process will ensure that the
solution has the appropriate level of quality and reliability.

• The Testing Strategy provides the overall guidelines from which all future
testing decisions are made. A well-crafted Testing Strategy allows the rest of the
testing process to be defined more effectively.

• The Testing Strategy needs to be understood and approved by the sponsor. If the
strategy is accepted, there is a much better likelihood that the final solution will
meet the customer’s expectations.
 Risk Analysis:

A risk is a potential for loss or damage to an Organization from materialized threats. Risk
Analysis attempts to identify all the risks and then quantify the severity of the risks.A
threat as we have seen is a possible damaging event. If it occurs, it exploits vulnerability
in the security of a computer based system.

Contents:

 Risk Identification
 Schedule Risk
 Risk Assessment
 Risk Assessment in Public Health

• Risk Assessment in Auditing

• Criticisms of quantitative risk assessment
• Risk Management
• Create a risk management plan
• Implementation
• Review and evaluation of the plan
• Limitations
• Areas of Risk Management
• Risk management and business continuity

Risk Identification

1. Software Risks:

Knowledge of the most common risks associated with Software development, and the
platform you are working on.

2. Business Risks: Most common risks associated with the business using the
Software

3. Testing Risks: Knowledge of the most common risks associated with Software
Testing for the platform you are working on, tools being used, and test methods being
applied.

4. Premature Release Risk: Ability to determine the risk associated with releasing
unsatisfactory or untested Software Prodicts.

5. Risk Methods: Strategies and approaches for identifying risks or problems

associated with implementing and operating information technology, products and
process; assessing their likelihood, and initiating strategies to test those risks.
 What is Schedule Risk?

In your project, you have to estimate how long it takes to complete a certain task.

You estimate that it usually takes 15 days to complete. If things go well it may take 12
days but if things go badly it may take 20 days.

In your project plan you enter 15 days against the task.

The other information, the best case estimate of 12 days and the worst case estimate of 20
days, is not entered into the project at all.

If this seems familiar, then you already go through the process of identifying uncertainty
or risk. By entering only the most likely duration a great deal of additional information is
lost. But with Schedule Risk this extra information is used to help produce a much more
realistic project.

And you are not just limited to durations. Uncertainty in resources and costs can also be
modeled in your project to produce an even greater depth and accuracy to the information
available to you.

Who should use Schedule Risk Analysis?

The simple answer is - anyone who manages a project! If you are running projects that
are time and/or cost critical, risk analysis will help you manage your projects more
effectively and help reduce the chances of your project being late and over budget.

Pertmaster is used by project planners of all levels, from those just entering into the
Schedule Risk arena to the world's leading risk experts.

How easy is it to use?

Very easy. You do not need to be an expert in risk and statistics to be able to use schedule
risk. With normal project planning, the level of detail and complexity that you build into
the project is up to you and your requirements. This is the same with Schedule Risk. Very
little extra information is required as a minimum but you have the ability to provide a
great deal of very specific additional information if you require it.

Pertmaster is acclaimed as being very easy to use. By simply following the tutorials and
examples you will be able to incorporate risk into your project with ease. Pertmaster
includes a Quick Risk (link) facility that lets you add risk to your project in seconds.

Risk Assessment
Risk assessment may be the most important step in the risk management process, and
may also be the most difficult and prone to error. Once risks have been identified and
assessed, the steps to properly deal with them are much more programmatical.

Part of the difficulty of risk management is that measurement of both of the quantities
in which risk assessment is concerned can be very difficult itself. Uncertainty in the
measurement is often large in both cases. Also, risk management would be simpler if a
single metric could embody all of the information in the measurement. However, since
two quantities are being measured, this is not possible. A risk with a large potential loss
and a low probability of occurring must be treated differently than one with a low
potential loss but a high likelihood of occurring. In theory both are of nearly equal
priority in dealing with first, but in practice it can be very difficult to manage when faced
with the scarcity of resources, especially time, in which to conduct the risk management
process. Expressed mathematically,

Financial decisions, such as insurance, often express loss terms in dollars. When risk
assessment is used for public health or environmental decisions, there are differences of
opinions as to whether the loss can be quantified in a common metric such as dollar
values or some numerical measure of quality of life. Often for public health and
environmental decisions, the loss term is simply a verbal description of the outcome, such
as increased cancer incidence or incidence of birth defects. In that case, the "risk" is
expressed as:

If the risk estimate takes into account information on the number of individuals exposed,
it is termed a "population risk" and is in units of expected increased cases per a time
period. If the risk estimate does not take into account the number of individuals exposed,
it is termed an "individual risk" and is in units of incidence rate per a time period.
Population risks are of more use for cost/benefit analysis; individual risks are of more use
for evaluating whether risks to individuals are "acceptable".

Risk assessment in public health

In the context of public health, risk assessment is the process of quantifying the
probability of a harmful effect to individuals or populations from certain human
activities.

In most countries, the use of specific chemicals, or the operations of specific

facilities (e.g. power plants, manufacturing plants) is not allowed unless it can be
shown that they do not increase the risk of death or illness above a specific threshold.
For example, the American Food and Drug Administration (FDA) regulates food
safety through risk assessment. The FDA required in 1973 that cancer-causing
compounds must not be present in meat at concentrations that would cause a cancer
risk greater than 1 in a million lifetimes.

How the risk is determined

In the estimation of the risks, three or more steps are involved, requiring the inputs of
different disciplines.

The first step, Hazard Identification, aims to determine the qualitative nature of the
potential adverse consequences of the contaminant (chemical, radiation, noise, etc.) and
the strength of the evidence it can have that effect. This is done, for chemical hazards, by
drawing from the results of the sciences of toxicology and epidemiology. For other kinds
of hazard, engineering or other disciplines are involved.

The second step for chemical risk assessment, Dose-Response Analysis, is

determining the relationship between dose and the probability or the incidence of effect
(dose-response assessment). The complexity of this step in many contexts derives mainly
from the need to extrapolate results from experimental animals (e.g. mouse, rat) to
humans, and/or from high to lower doses. In addition, the differences between individuals
due to genetics or other factors mean that the hazard may be higher for particular groups,
called susceptible populations. An alternative to dose-response estimation is to determine
an effect unlikely to yield observable effects. In developing such a dose, to account for
the largely unknown effects of animal to human extrapolations, increased variability in
humans, or missing data, a prudent approach is often adopted by including safety factors
in the estimate of the "safe" dose, typically a factor of 10 for each unknown step.

The third step, Exposure Quantification, aims to determine the amount of a contaminant
(dose) that individuals and populations will receive. This is done by examining the results
of the discipline of exposure assessment. As different location, lifestyles and other factors
likely influence the amount of contaminant that is received, a range or distribution of
possible values is generated in this step. Particular care is taken to determine the exposure
of the susceptible population(s).

Finally, the results of the three steps above are then combined to produce an estimate of
risk. Because of the different susceptibilities and exposures, this risk will vary within a
population. The decisions based on the application of risk assessment are sometimes
based on a standard of protecting those most at risk. This problem raises the question of
how small a segment of a population must be protected. What if a risk is very low for
everyone but 0.1% of the population? A difference exists whether this 0.1% is represented
by

• all infants younger than X days or

• recreational users of a particular product.

If the risk is higher for a particular sub-population because of abnormal exposure rather
than susceptibility, there is a potential to consider strategies to further reduce the
exposure of that subgroup. If an identifiable sub-population is more susceptible due to
inherent genetic or other factors, there is a policy choice whether to set policies for
protecting the general population that are protective of such groups (as is currently done
for children when data exists, or is done under the Clean Air Act for populations such as
asthmatics) or whether if the group is too small, or the costs to high. Sometimes, a
suitable position is to at least limit the risk of the more susceptible to some risk level
above which it seems too inequitable to leave them out of the risk

Acceptable Risk Increase

The idea of not increasing lifetime risk by more than one in a million has become
common place in public health discourse and policy. How consensus settled on this
particular figure is unclear. In some respects, this figure has the characteristics of a
mythical number. In another sense, the figure provides a numerical basis for what to
consider a negligible increase in risk.

In part, the one in a million benchmark arose early in public health risk assessment
history when risk assessment was a tempering analysis to existing statutory language
such as the Delaney Clause prohibition on any use of introduced carcinogens or where
environmental statutes were using a "best technology" decision rule. Some current
environmental decision making allows some discretion to deem individual risks
potentially "acceptable" if below one in ten thousand increased lifetime risk. Low risk
criteria such as these do provide some protection for the case that individuals may be
exposed to multiple chemicals (whether pollutants or food additives, or other chemicals).
But both of these benchmarks are clearly small relative to the typical one in four lifetime
risk of death by cancer (due to all causes combined) in developed countries.

Individuals may be tempted to advocate the adoption of a zero-risk policy. After all the 1
in a million policy would still cause the death of hundreds or thousands, of people in a
large enough population. In practice however, a true zero-risk is possible only with the
suppression of the risk-causing activity. More stringent requirements, or even the 1 in a
million one, may not be technologically feasible at a given time, or so expensive as to
render the risk-causing activity unsustainable.

In the interest of public health, the risks vs. benefits of the possible alternatives must be
carefully considered. For example, it might well be that the emissions from hospital
incinerators result in a certain number of deaths per year. However, this risk must be
balanced against the available alternatives. In some unusual cases, there are significant
public health risks, as well as economic costs, associated with all options. For example,
there are risks associated with no incineration (with the potential risk for spread of
infectious diseases) or even no hospitals. But, often further investigation identifies further
options, such as separating noninfectious from infectious wastes, or air pollution controls
on a medical incinerator, that provide a broad range of options of acceptable risk - though
with varying practical implications and varying economic costs. Intelligent thought about
a reasonably full set of options is essential. Thus, it is not unusual for there to be an
iterative process between analysis, consideration of options, and then further analysis.

Risk assessment in auditing

In auditing, risk assessment is a very crucial stage before accepting an audit engagement.
According to ISA315 Understanding the Entity and its Environment and Assessing the
Risks of Material Misstatement, "the auditor should perform risk assessment procedures
to obtain an understanding of the entity and its environment, including its internal
control."

The main purpose of risk assessment procedures is to help the auditor understand the
audit client. Aspects like client's business nature, management structure and internal
control system are good examples. The procedures will provide audit evidence relating to
the auditor’s risk assessment of a material misstatement in the client’s financial
statements. Then, auditor obtains initial evidence regarding the classes of transactions at
the client and the operating effectiveness of the client’s internal controls.

In auditing, audit risk includes inherent risk, control risk and detection risk.

Criticisms of quantitative risk assessment

Barry Commoner and other critics have expressed concerns that risk assessment tends to
be overly quantitative and reductive. For example, they argue that risk assessments ignore
qualitative differences among risks.

Some charge that assessments may drop out important non-quantifiable or inaccessible
information, such as variations among the classes of people exposed to hazards. O'Brien
further claims that quantitative approaches divert attention from precautionary or
preventative measures.

Risk Management
Risk management is a structured approach to managing uncertainty through, risk
assessment, developing strategies to manage it, and mitigation of risk using managerial
resources.

The strategies include transferring the risk to another party, avoiding the risk, reducing
the negative effect of the risk, and accepting some or all of the consequences of a
particular risk.

Some traditional risk managements are focused on risks stemming from physical or legal
causes (e.g. natural disasters or fires, accidents, death and lawsuits). Financial risk
management, on the other hand, focuses on risks that can be managed using traded
financial instruments.

Objective of risk management is to reduce different risks related to a preselected domain

to the level accepted by society. It may refer to numerous types of threats caused by
environment, technology, humans, organizations and politics. On the other hand it
involves all means available for humans, or in particular, for a risk management entity
(person, staff, organization).

Establishing the context involves

1. Identification of risk in a selected domain of interest

2. Planning the remainder of the process.
3. Mapping out
4. Defining a framework for the activity and an agenda for identification.
5. Developing an analysis of risks involved in the process.
6. Mitigation of risks using available technological, human and organizational
resources.

Identification

After establishing the context, the next step in the process of managing risk is to identify
potential risks. Risks are about events that, when triggered, cause problems. Hence, risk
identification can start with the source of problems, or with the problem itself.

1. Source analysis:Risk sources may be internal or external to the system that is the
target of risk management. Examples of risk sources are: stakeholders of a
project, employees of a company or the weather over an airport.
2. Problem analysis:Risks are related to identified threats. For example: the threat
of losing money, the threat of abuse of privacy information or the threat of
accidents and casualties. The threats may exist with various entities, most
important with shareholders, customers and legislative bodies such as the
government.
When either source or problem is known, the events that a source may trigger or the
events that can lead to a problem can be investigated. For example: stakeholders
withdrawing during a project may endanger funding of the project; privacy information
may be stolen by employees even within a closed network; lightning striking a Boeing
747 during takeoff may make all people onboard immediate casualties.

The chosen method of identifying risks may depend on culture, industry practice and
compliance. The identification methods are formed by templates or the development of
templates for identifying source, problem or event. Common risk identification methods
are:

1. Objectives-based risk identification :Organizations and project teams have

objectives. Any event that may endanger achieving an objective partly or
completely is identified as risk. Objective-based risk identification is at the basis
of COSO's Enterprise Risk Management - Integrated Framework
2. Scenario-based risk identification:In scenario analysis different scenarios are
created. The scenarios may be the alternative ways to achieve an objective, or an
analysis of the interaction of forces in, for example, a market or battle. Any event
that triggers an undesired scenario alternative is identified as risk - see Futures
Studies for methodology used by Futurists.
3. Taxonomy-based risk identification:The taxonomy in taxonomy-based risk
identification is a breakdown of possible risk sources. Based on the taxonomy and
knowledge of best practices, a questionnaire is compiled. The answers to the
questions reveal risks. Taxonomy-based risk identification in software industry
can be found in CMU/SEI-93-TR-6.
4. Common-risk Checking:In several industries lists with known risks are
available. Each risk in the list can be checked for application to a particular
situation. An example of known risks in the software industry is the Common
Vulnerability and Exposures.
5. Risk Charting:This method combines the above approaches by listing Resources
at risk, Threats to those resources Modifying Factors which may increase or
reduce the risk and Consequences it is wished to avoid. Creating a matrix under
these headings enables a variety of approaches. One can begin with resources and
consider the threats they are exposed to and the consequences of each.
Alternatively one can start with the threats and examine which resources they
would affect, or one can begin with the consequences and determine which
combination of threats and resources would be involved to bring them about.

Assessment

Once risks have been identified, they must then be assessed as to their potential severity
of loss and to the probability of occurrence. These quantities can be either simple to
measure, in the case of the value of a lost building, or impossible to know for sure in the
case of the probability of an unlikely event occurring. Therefore, in the assessment
process it is critical to make the best educated guesses possible in order to properly
prioritize the implementation of the risk management plan.

The fundamental difficulty in risk assessment is determining the rate of occurrence

since statistical information is not available on all kinds of past incidents. Furthermore,
evaluating the severity of the consequences (impact) is often quite difficult for immaterial
assets. Asset valuation is another question that needs to be addressed. Thus, best educated
opinions and available statistics are the primary sources of information. Nevertheless,
risk assessment should produce such information for the management of the organization
that the primary risks are easy to understand and that the risk management decisions may
be prioritized. Thus, there have been several theories and attempts to quantify risks.
Numerous different risk formulae exist, but perhaps the most widely accepted formula for
risk quantification is:

Rate of occurrence multiplied by the impact of the event equals risk

Later research has shown that the financial benefits of risk management are less
dependent on the formula used but are more dependent on the frequency and how risk
assessment is performed.

In business it is imperative to be able to present the findings of risk assessments in

financial terms. Robert Courtney Jr. (IBM, 1970) proposed a formula for presenting risks
in financial terms. The Courtney formula was accepted as the official risk analysis
method for the US governmental agencies. The formula proposes calculation of ALE
(annualised loss expectancy) and compares the expected loss value to the security control
implementation costs (cost-benefit analysis).
Potential risk treatments

Once risks have been identified and assessed, all techniques to manage the risk fall into
one or more of these four major categories: (Dorfman, 1997)

• Avoidance (aka elimination)

• Reduction (aka mitigation)
• Retention
• Transfer (aka buying insurance)

Ideal use of these strategies may not be possible. Some of them may involve trade-offs
that are not acceptable to the organization or person making the risk management
decisions. Another source, from the US Department of Defense, Defense Acquisition
University, calls these ACAT, for Avoid, Control, Accept, or Transfer. The ACAT
acronym is reminiscent of the term ACAT (for Acquisition Category) used in US Defense
industry procurements.
 Risk Avoidance

Once risks have been identified and assessed, all techniques to manage the risk fall into
one or more of these four major categories: (Dorfman, 1997)

• Avoidance (aka elimination)

• Reduction (aka mitigation)
• Retention
• Transfer (aka buying insurance)

Includes not performing an activity that could carry risk. An example would be not
buying a property or business in order to not take on the liability that comes with it.
Another would be not flying in order to not take the risk that the airplane were to be
hijacked. Avoidance may seem the answer to all risks, but avoiding risks also means
losing out on the potential gain that accepting (retaining) the risk may have allowed. Not
entering a business to avoid the risk of loss also avoids the possibility of earning profits.

Risk Reduction

Involves methods that reduce the severity of the loss or the risk of the loss from
occurring. Examples include sprinklers designed to put out a fire to reduce the risk of loss
by fire. This method may cause a greater loss by water damage and therefore may not be
suitable. Halon fire suppression systems may mitigate that risk, but the cost may be
prohibitive as a strategy.

Modern software development methodologies reduce risk by developing and delivering

software incrementally. Early methodologies suffered from the fact that they only
delivered software in the final phase of development; any problems encountered in earlier
phases meant costly rework and often jeopardized the whole project. By developing in
iterations, software projects can limit effort wasted to a single iteration.

Outsourcing could be an example of risk reduction if the outsourcer can demonstrate

higher capability at managing or reducing risks. In this case companies outsource only
some of their departmental needs. For example, a company may outsource only its
software development, the manufacturing of hard goods, or customer support needs to
another company, while handling the business management itself. This way, the company
can concentrate more on business development without having to worry as much about
the manufacturing process, managing the development team, or finding a physical
location for a call center.
 Risk Retention

Involves accepting the loss when it occurs. True self insurance falls in this category. Risk
retention is a viable strategy for small risks where the cost of insuring against the risk
would be greater over time than the total losses sustained. All risks that are not avoided or
transferred are retained by default.

This includes risks that are so large or catastrophic that they either cannot be insured
against or the premiums would be infeasible. War is an example since most property and
risks are not insured against war, so the loss attributed by war is retained by the insured.
Also any amounts of potential loss (risk) over the amount insured is retained risk. This
may also be acceptable if the chance of a very large loss is small or if the cost to insure
for greater coverage amounts is so great it would hinder the goals of the organization too
much.

Risk Transfer

Means causing another party to accept the risk, typically by contract or by hedging.
Insurance is one type of risk transfer that uses contracts. Other times it may involve
contract language that transfers a risk to another party without the payment of an
insurance premium. Liability among construction or other contractors is very often
transferred this way. On the other hand, taking offsetting positions in derivatives is
typically how firms use hedging to financially manage risk.

Some ways of managing risk fall into multiple categories. Risk retention pools are
technically retaining the risk for the group, but spreading it over the whole group
involves transfer among individual members of the group. This is different from
traditional insurance, in that no premium is exchanged between members of the group up
front, but instead losses are assessed to all members of the group.

Create a risk management plan

Select appropriate controls or countermeasures to measure each risk. Risk mitigation

needs to be approved by the appropriate level of management. For example, a risk
concerning the image of the organization should have top management decision behind it
whereas IT management would have the authority to decide on computer virus risks.

The risk management plan should propose applicable and effective security controls
for managing the risks. For example, an observed high risk of computer viruses could be
mitigated by acquiring and implementing antivirus software. A good risk management
plan should contain a schedule for control implementation and responsible persons for
those actions.

According to ISO/IEC 27001, the stage immediately after completion of the Risk
Assessment phase consists of preparing a Risk Treatment Plan, which should document
the decisions about how each of the identified risks should be handled. Mitigation of risks
often means selection of Security Controls, which should be documented in a Statement
of Applicability, which identifies which particular control objectives and controls from
the standard have been selected, and why.

Implementation

Follow all of the planned methods for mitigating the effect of the risks.

Purchase insurance policies for the risks that have been decided to be transferred to an
insurer,avoid all risks that can be avoided without sacrificing the entity's goals, reduce
others, and retain the rest.

Review and evaluation of the plan

Initial risk management plans will never be perfect. Practice, experience, and actual loss
results will necessitate changes in the plan and contribute information to allow possible
different decisions to be made in dealing with the risks being faced.

Risk analysis results and management plans should be updated periodically. There are
two primary reasons for this:

• to evaluate whether the previously selected security controls are still applicable
and effective, and
• to evaluate the possible risk level changes in the business environment. For
example, information risks are a good example of rapidly changing business
environment.

Limitations

If risks are improperly assessed and prioritized, time can be wasted in dealing with risk of
losses that are not likely to occur. Spending too much time assessing and managing
unlikely risks can divert resources that could be used more profitably. Unlikely events do
occur but if the risk is unlikely enough to occur it may be better to simply retain the risk
and deal with the result if the loss does in fact occur.

Prioritizing too highly the risk management processes could keep an organization
from ever completing a project or even getting started. This is especially true if other
work is suspended until the risk management process is considered complete.

It is also important to keep in mind the distinction between risk and uncertainty. Risk can
be measured by impacts x probability.
Areas of risk management

As applied to corporate finance, risk management is the technique for measuring,

monitoring and controlling the financial or operational risk on a firm's balance sheet.

The Basel II framework breaks risks into market risk (price risk), credit risk and
operational risk and also specifies methods for calculating capital requirements for each
of these components.

Enterprise risk management

In enterprise risk management, a risk is defined as a possible event or circumstance

that can have negative influences on the Enterprise in question. Its impact can be on the
very existence, the resources (human and capital), the products and services, or the
customers of the enterprise, as well as external impacts on society, markets, or the
environment. In a financial institution, enterprise risk management is normally thought of
as the combination of credit risk, interest rate risk or asset liability management, market
risk, and operational risk.

In the more general case, every probable risk can have a preformulated plan to deal with
its possible consequences (to ensure contingency if the risk becomes a liability).

From the information above and the average cost per employee over time, or cost accrual
ratio, a project manager can estimate

• the cost associated with the risk if it arises, estimated by multiplying employee
costs per unit time by the estimated time lost (cost impact, C where C = cost
accrual ratio * S).
• the probable increase in time associated with a risk (schedule variance due to risk,
Rs where Rs = P * S):
o Sorting on this value puts the highest risks to the schedule first. This is
intended to cause the greatest risks to the project to be attempted first so
that risk is minimized as quickly as possible.
o This is slightly misleading as schedule variances with a large P and small
S and vice versa are not equivalent. (The risk of the RMS Titanic sinking
vs. the passengers' meals being served at slightly the wrong time).
• the probable increase in cost associated with a risk (cost variance due to risk, Rc
where Rc = P*C = P*CAR*S = P*S*CAR)
o sorting on this value puts the highest risks to the budget first.
 o see concerns about schedule variance as this is a function of it, as
illustrated in the equation above.

Risk in a project or process can be due either to Special Cause Variation or Common
Cause Variation and requires appropriate treatment. That is to re-iterate the concern about
extremal cases not being equivalent in the list immediately above.

Risk management activities as applied to project management

In project management, risk management includes the following activities:

• Planning how risk management will be held in the particular project. Plan should
include risk management tasks, responsibilities, activities and budget.
• Assigning a risk officer - a team member other than a project manager who is
responsible for foreseeing potential project problems. Typical characteristic of
risk officer is a healthy skepticism.
• Maintaining live project risk database. Each risk should have the following
attributes: opening date, title, short description, probability and importance.
Optionally a risk may have an assigned person responsible for its resolution and a
date by which the risk must be resolved.
• Creating anonymous risk reporting channel. Each team member should have
possibility to report risk that he foresees in the project.
• Preparing mitigation plans for risks that are chosen to be mitigated. The purpose
of the mitigation plan is to describe how this particular risk will be handled –
what, when, by who and how will it be done to avoid it or minimize consequences
if it becomes a liability.
• Summarizing planned and faced risks, effectiveness of mitigation activities, and
effort spent for the risk management.

Risk management and business continuity

Risk management is simply a practice of systematically selecting cost effective

approaches for minimising the effect of threat realization to the organization. All risks
can never be fully avoided or mitigated simply because of financial and practical
limitations. Therefore all organizations have to accept some level of residual risks.

Whereas risk management tends to be preemptive, business continuity planning (BCP)

was invented to deal with the consequences of realised residual risks. The necessity to
have BCP in place arises because even very unlikely events will occur if given enough
time. Risk management and BCP are often mistakenly seen as rivals or overlapping
practices. In fact these processes are so tightly tied together that such separation seems
artificial. For example, the risk management process creates important inputs for the BCP
(assets, impact assessments, cost estimates etc). Risk management also proposes
applicable controls for the observed risks. Therefore, risk management covers several
areas that are vital for the BCP process. However, the BCP process goes beyond risk
management's preemptive approach and moves on from the assumption that the disaster
will realize at some point.

Release Life Cycle

Introduction

A software release is the distribution, whether public or private, of an initial or new and
upgraded version of a computer software product. Each time a software program or
system is changed, the software engineers and company doing the work decide on how to
distribute the program or system, or changes to that program or system.

Software patches are one method of distributing the changes, as are downloads and
compact discs.

 Software release stages

 End of life

Release Life Cycle Stages

The software release life cycle is composed of different stages that describe the stability
of a piece of software and the amount of development it requires before final release.
Each major version of a product usually goes through a stage when new features are
added, or the alpha stage; a stage when it is being actively debugged, or the beta stage;
and finally a stage when all important bugs have been removed, or the stable stage.

Intermediate stages may also be recognized. The stages may be formally announced
and regulated by the project's developers, but sometimes the terms are used informally to
describe the state of a product. Conventionally, code names are often used by many
companies for versions prior to the release of the product, though the actual product and
features are rarely secret.
 Pre-alpha

Sometimes a build known as pre-alpha is issued, before the release of an alpha or beta.
In contrast to alpha and beta versions, the pre-alpha is not "feature complete". When it is
used, it refers to all activities performed during the software project prior to software
testing. These activities can include requirements analysis, software design, software
development and unit testing.

In Open Source world, there are several types of pre-alpha versions. Milestone
versions include specific sets of functionality and are released as soon as the functionality
is complete. Nightly builds are versions that are usually automatically checked out from
the revision control system and built, typically over night; these versions allow the testers
to test the recently implemented functionality immediately, and find the new bugs.

Alpha
The alpha build of the software is the build delivered to the software testers, that is
persons different from the software engineers, but usually internal to the organization or
community that develops the software. In a rush to market, more and more companies are
engaging external customers or value-chain partners in their alpha testing phase. This
allows more extensive usability testing during the alpha phase.

In the first phase of testing, developers generally test the software using white box
techniques. Additional validation is then performed using black box or grey box
techniques, by another dedicated testing team, sometimes concurrently. Moving to black
box testing inside the organization is known as alpha release.

Beta

A beta version is the first version released outside the organization or community that
develops the software, for the purpose of evaluation or real-world black/grey-box testing.
The process of delivering a beta version to the users is called beta release. Beta level
software generally includes all features, but may also include known issues and bugs of a
less serious variety.

The users of a beta version are called beta testers. They are usually customers or
prospective customers of the organization that develops the software. They receive the
software for free or for a reduced price, but act as free testers.

Beta versions test the supportability of the product, the go-to-market messaging (while
recruiting Beta customers), the manufacturability of the product, and the overall channel
flow or channel reach.

Beta version software is likely to be useful for internal demonstrations and previews to
select customers, but unstable and not yet ready for release. Some developers refer to this
stage as a preview, a prototype, a technical preview (TP) or as an early access. As the
second major stage in the release lifecycle, following the alpha stage, it is named after the
Greek letter beta, the second letter in the Greek alphabet.

Often this stage begins when the developers announce a feature freeze on the product,
indicating that no more feature requirements will be accepted for this version of the
product. Only software issues, or bugs and unimplemented features will be addressed.

Developers release either a closed beta or an open beta; closed beta versions are released
to a select group of individuals for a user test, while open betas are to a larger community
group, usually the general public. The testers report any bugs that they found and
sometimes minor features they would like to see in the final version.
An example of a major public beta test was when Microsoft started releasing regular
Windows Vista Community Technology Previews (CTP) to beta testers starting in
January 2005. The first of these was build 5219. Subsequent CTPs introduced most of the
planned features, as well as a number of changes to the user interface, based in large part
on feedback from beta testers. Windows Vista was deemed feature complete with the
release of build 5308 CTP, released on February 22, 2006, and much of the remainder of
work between that build and the final release of the product focused on stability,
performance, application and driver compatibility, and documentation.

When a beta becomes available to the general public it is often widely used by the
technologically savvy and those familiar with previous versions as though it were the
finished product. Usually developers of freeware or open-source betas release them to the
general public while proprietary betas go to a relatively small group of testers. Recipients
of highly proprietary betas may have to sign a non-disclosure agreement. A release is
called feature complete when the product team agrees that functional requirements of the
system are met and no new features will be put into the release, but significant software
bugs may still exist. Companies with a formal software process will tend to enter the beta
period with a list of known bugs that must be fixed to exit the beta period, and some
companies make this list available to customers and testers.

As the Internet has allowed for rapid and inexpensive distribution of software, companies
have begun to take a more flexible approach to use of the word "beta". Netscape
Communications was infamous for releasing alpha level versions of its Netscape web
browser to the public and calling them “beta” releases. In February 2005, ZDNet
published an article about the recent phenomenon of a beta version often staying for years
and being used as if it were in production-level. It noted that Gmail and Google News, for
example, had been in beta for a long period of time and were not expected to drop the
beta status despite the fact that they were widely used; however, Google News did leave
beta in January 2006. This technique may also allow a developer to delay offering full
support and/or responsibility for remaining issues. In the context of Web 2.0, people even
talk of perpetual betas to signify that some software is meant to stay in beta state. Also,
"beta" is sometimes used to indicate something more like a release candidate such as the
Halo 3 public beta.

Origin of 'alpha' and 'beta'

The term beta test applied to software comes from an early IBM hardware product test
convention dating back to punched card tabulating and sorting machines. Hardware first
went through an alpha test for preliminary functionality and small scale manufacturing
feasibility. Then came a beta test to verify that it actually correctly performed the
functions it was supposed to and could be manufactured at scales necessary for the
market, and then a c test to verify safety.
 With the advent of programmable computers and the first shareable software
programs, IBM used the same terminology for testing software. Beta tests were
conducted by people or groups other than the developers. As other companies began
developing software for their own use, and for distribution to others, the terminology
stuck and now is part of our common vocabulary.

Seigo Stage

The Seigo Stage is the stage in software development when the libraries for the product
are ready (or very close to being ready) but the rest of the product is still in need of
polishing up. The stage comes between the Beta and Release Candidate stages of
software development because the software product is still not ready to be a potential
final product but libraries of the software are release quality.

Origin of the 'Seigo Stage'

The term Seigo Stage was invented after a heated discussion on The Linux Action Show!
podcast regarding the naming of the KDE 4 second release candidate not being up to the
standard of what a release candidate should be. The discussion took place between the
two presenters of the show and Aaron Seigo via Skype. It was finally agreed that the
release candidate had been termed as such because it was beyond beta quality as the
libraries were of release candidate quality. At this point one of the presenters said that
they should rename the stage KDE 4 had reached, the name they choose was the Seigo
Stage.

Release candidate

The term release candidate refers to a version with potential to be a final product, ready
to release unless fatal bugs emerge. In this stage, the product features all designed
functionalities and no known showstopper-class bugs. At this phase the product is usually
code complete.

Microsoft Corporation often uses the term release candidate. During the 1990s, Apple
Inc. used the term "golden master" for its release candidates, and the final golden master
was the general availability release. Other terms include gamma (and occasionally also
delta, and perhaps even more Greek letters) for versions that are substantially complete,
but still under test, and omega for final testing of versions that are believed to be bug-
free, and may go into production at any time. (Gamma, delta, and omega are,
respectively, the third, fourth, and last letters of the Greek alphabet.) Some users
disparagingly refer to release candidates and even final "point oh" releases as "gamma
test" software, suggesting that the developer has chosen to use its customers to test
software that is not truly ready for general release. Often, beta testers, if privately
selected, will be billed for using the release candidate as though it were a finished
product.

A release is called code complete when the development team agrees that no entirely
new source code will be added to this release. There may still be source code changes to
fix defects. There may still be changes to documentation and data files, and to the code
for test cases or utilities. New code may be added in a future release.

Gold or general availability release

The gold or general availability release version is the final version of a particular
product. It is typically almost identical to the final release candidate, with only last-
minute bugs fixed. A gold release is considered to be very stable and relatively bug-free
with a quality suitable for wide distribution and use by end users. In commercial software
releases, this version may also be signed (used to allow end-users to verify that code has
not been modified since the release). The expression that a software product "has gone
gold" means that the code has been completed and "is being mass-produced and will be
for sale soon." Other terms for the version include gold master, golden master, “GM”,
gold release, or gold build.

The term gold anecdotally refers to the use of "gold master disc" which was commonly
used to send the final version to manufacturers who use it to create the mass-produced
retail copies. It may in this context be a hold-over from music production. In some cases,
however, the master disc is still actually made of gold, for both aesthetic appeal and
resistance to corrosion.

RTM or RTW

Microsoft and others use the term "release to manufacturing" (RTM) to refer to this
version (for commercial products, like Windows XP, as in, "Build 2600 is the Windows
XP RTM release"), and "release to Web" (RTW) for freely downloadable products.
Typically, RTM is weeks or months before General Availability (GA) because the RTM
version must be stamped to disc and boxed etc.

Box copy

A box copy is the final product, printed on a disc that is included in the actual release,
complete with disc graphic art. This term is used mostly by reviewers to differentiate
from gold master discs. A box copy does not necessarily come enclosed in the actual
boxed product - it refers to the disc itself.
 Stable or Unstable

In open source programming, version numbers or the terms stable and unstable
commonly distinguish the stage of development. The term stable refers to a version of
software that is substantially identical to a version that has been through enough real-
world testing to reasonably assume there are no showstopper problems, or at least that
any problems are known and documented.

On the other hand, the term unstable does not necessarily mean that there are problems -
rather, that enhancements or changes have been made to the software that have not
undergone rigorous testing and that more changes are expected to be imminent. Users of
such software are advised to use the stable version if it meets their needs, and to only use
the unstable version if the new functionality is of interest that exceeds the risk that
something might simply not work right.

In the Linux kernel, version numbers are composed of three numbers, separated by a
period. Between versions 1.0.0 and 2.6.x, stable releases had an even second number and
unstable release an odd one. As of Linux 2.6.x, the even or odd status of the second
number no longer holds any significance. The practice of using even and odd numbers to
indicate the stability of a release has been used by other open and closed source projects.

End of life

Sometimes software companies stop selling or supporting their software products (e.g.,
not releasing new patches).

At this point the product will be said to be in the status of "legacy", "vintage" or "end
of life". For instance, on August 15, 2007, Apple announced that AppleWorks reached
"end-of-life status".