Software and Systems Verification (VIMIMA01)

Introduction
Overview of V&V techniques

Istvan Majzik, Zoltan Micskei

Budapest University of Technology and Economics

Fault Tolerant Systems Research Group

Budapest University of Technology and Economics

Department of Measurement and Information Systems
 Main topics of the course
 Overview (1)
o V&V techniques, Critical systems
 Static techniques (2)
o Verifying specifications
o Verifying source code
 Dynamic techniques: Testing (7)
o Developer testing, Test design techniques
o Testing process and levels, Test generation, Automation
 System-level verification (3)
o Verifying architecture, Dependability analysis
o Runtime verification
2
 Who is this course for?
System Engineer • Requirements, verifying specification

Architect, Designer • Modeling and verifying designs

Developer, Coder • Verifying source code, unit testing

Test Designer • Test processes and techniques

Test Engineer • Test automation, integration and system tests

Safety Engineer • Certification, development standards

3
 Stereotypes

“Testing is destructive.”
“Testing is just pushing buttons
and supplying values randomly.”

“If your are not good for a

developer, you can be a tester.”

“Testing is boring.”

“I tested in the debugger…”

4
 V&V (and testing) in reality
V&V (and testing) is creative!
How is this working? How can I prove it works?
How should it work? How can it fail?

V&V (and testing) is constructive!

Testers are not breaking the SW (it was broken)
Passion for quality
Testers help make the system better

V&V (and testing) requires a different mindset

Intuition Attention to details …
Systems level thinking Specific knowledge
5
 V&V is context dependent!
Telco
• E2E, conformance…
• Protocol testing
• ITU, ETSI…
Critical systems Enterprise, web
• Safety • Agile, Lean
• Process, standards • ISTQB
• Documentation

V&V
6
 Useful resources (download now!)
 IEEE standards
o 24765-2010 Systems and SW engineering – Vocabulary
o 29148-2011 Requirements engineering
o 29119 Software testing
• Part 1 Concepts and definitions
• Part 2 Test processes
• Part 3 Test documentation
 International Software Testing Qualifications Board (ISTQB)
o Foundation Level Syllabus (2011)
o Glossary of Testing Terms
 Hungarian Testing Board (HTB)
o Glossary / Kifejezésgyűjtemény (magyar fordítás)

7
MOTIVATION

8
 Different kinds of faults

Development phase Operational phase

• Specification faults • Hardware faults

• Design faults • Configuration faults
• Implementation faults • Operator faults

V&V during Fault tolerance

design (e.g. redundancy)

9
 Software is the cause of problems
„Defibtech issues a worldwide recall of two of its defibrillator
products due to faulty self-test software that may clear a
previously detected low battery condition.” (February 2007)

„Cricket Communications recalls about 285,000 of its cell

phones due to a software glitch that causes audio problems
when a caller connects to an emergency 911 call. (May 2008)”

10
 How many bugs do we have to expect?

Source: K-R. Hase: „Open Proof in Railway Safety Software”, FORMS/FORMAT Conference, December 2-3, 2010, Braunschweig, Germany

11
Distribution and cost of bugs

Early V&V reduces cost!

12
 V&V: Verification and Validation
Verification
„Am I building the system right?”
Check consistency of
development phases
Conformance of designs/models
and their specification
Objective; can be automated
Fault model: Design and
implementation faults
Not needed if implementation is
automatically generated from
specification
13
Validation
„Am I building the right system?”
Check the result of the
development
Conformance of the finished
system and the user requirements
Subjective; checking acceptance
Fault model: problems in the
requirements
Not needed if the specification is
correct (very simple)
OVERVIEW OF V&V TECHNIQUES

14
 Learning outcomes

 List typical V&V activities (K1)

 Classify the different verification techniques

according to their place in the lifecycle (K2)

15
 Typical steps in development lifecycle
Requirement
analysis System Schedule, sequencing
System engineer depends on lifecycle model!
specification

Architecture
design
Architect
Module
design

Module
Developer,
implementation coder
System
integration
Test
System
delivery
engineer
Operation,
maintenance

16
 Requirement analysis
Requirement Task V&V criteria V&V technique
analysis
- Checklists
System Defining functions, - Risks
specification actors, use cases - Failure mode and
- Criticality
effects analysis
Architecture
design

Module
design

Module
implementation

System
integration

System
delivery

Operation,
maintenance

17
 System specification
Requirement
Task V&V criteria V&V technique
analysis - Completeness
Defining functional - Reviews
System - Unambiguity
specification and non-functional - Static analysis
requirements - Verifiability
- Simulation
Architecture - Feasibility
design

Module
design
Analysis
Module Reality
implementation
Design Implementation
System space
integration Modeling 18 Implementation
- structuring
System Designing space
delivery - abstraction
- decomposition
Operation,
maintenance
 Architecture design
abstraction
Requirement Analysis
analysis Design space
Mapping
System Structuring
(automated)
specification design space
and mapping
Architecture
design
Implementation space formality
Module
design

Module
implementation Task V&V criteria V&V technique
System - Decomposing - Function coverage - Static analysis
integration modules - Conformance of - Simulation
System - HW-SW co-design interfaces - Performance,
delivery - Designing - Non-functional dependability,
Operation,
communication properties security analysis
maintenance
 Module design (detailed design)
Requirement
analysis
System modellje
Rendszer model Requirement spec.

System Formal y n
specification Automatikus
Automated model
verification modellellen
checking
Architecture
design
OK Counter-
Module example
design

Module
implementation
Task V&V criteria V&V technique
System
integration - Designing detailed - Correctness of - Static analysis
behavior critical internal - Simulation
System
delivery (data structures, algorithms and - Formal verification
algorithms) protocols - Rapid prototyping
Operation,
maintenance

20
 Module implementation
Requirement
analysis

System
specification

Architecture Task V&V criteria V&V technique

design
Code is
- Coding conventions
Module - Software - Safe
design - Code reviews
implementation - Verifiable
Module
- Static code analysis
- Maintainable
implementation

System - Verifying module - Conformance to - Unit testing

integration implementation module designs - Regression testing
21
System
delivery

Operation,
maintenance
 System integration
Requirement
Task V&V criteria V&V technique
analysis - Conformance of
System
- Integrating modules integrated
- Integration testing
specification - Integrating SW with behavior (incremental)
Architecture
HW - Verifying
design communication
Module
design

Module
implementation

System
integration

System
delivery

Operation,
maintenance

22
 System delivery and deployment
Requirement
analysis

System
specification

Architecture
design Task V&V criteria V&V technique

Module - System testing

design - Assembling - Conformance to
complete system system specification - Measurements,
Module monitoring
implementation

- Conformance to - Validation testing

System - Fulfilling user
integration requirements and - Acceptance testing
expectations
expectations - Alfa/beta testing
System
delivery

Operation,
maintenance

23
 Operation and maintenance
Requirement
analysis

System Tasks during operation and maintenance:

specification - Failure logging and analysis (for failure prediction)
Architecture
- V&V of modifications
design

Module
design

Module
implementation

System
integration
Mini-lifecycle
System for each
delivery
modification
Operation,
maintenance

24
V&V TECHNIQUES IN
CRITICAL SYSTEMS

25
 Learning outcomes

 Recall the safety concepts of critical systems (K1)

 List typical activities required by standards (K1)

26
 Safety-critical systems

Safety: “The expectation that a system does not,

under defined conditions, lead to a state in which
human life, health, property, or the environment is
endangered.” [IEEE]

27
 Certification

 Certification by safety authorities

 Basis of certification: Standards

o IEC 61508: Generic standard (for electrical, electronic
or programmable electronic systems)
o DO178B/C: Software in airborne systems
o EN50128: Railway (software)
o ISO26262: Automotive

28
 Safety concepts
 Safety function
o Intended to achieve or maintain a safe state
 Safety integrity
o Probability of a safety-related system satisfactorily
performing the required safety functions under all
stated conditions and within a stated period of time
 Safety Integrity Level (SIL)
o Based on risk analysis
o Tolerable Hazard Rate (THR)

29
 Basics of determining SIL
Risk analysis -> THR -> SIL
Frequency of
hazardous event System Software
safety safety
integrity integrity
level level
4 4
Risk THR SIL 3 3
2 2
Consequence of
1 1
hazardous event
0 0

SIL Probability of dangerous failure per

15 years lifetime:
1 failure in case
of 750 equipment
hour per safety function
1 10-6  THR < 10-5
2 10-7  THR < 10-6
3 10-8  THR < 10-7
4 10-9  THR < 10-8
30
 Demonstrating SIL requirements
Different approaches for types of failures

 Random failures (e.g. HW)

o Qualitative analysis (statistics, experiments…)

 Systematic failures (e.g. SW)

o Rigor in the engineering
o Recommendations for each SIL
o Process, techniques, documentation, responsibilities

31
 Example: Process (V model)
Operation,
maintenance

Requirement System val. System

analysis design validation

System System test System

specification design verification

Well-defined
Architecture Integration test System
design design integration phases

Module Module test Module Verification of

design design verification
each step

Module
implementation

32
 Example: Techniques (EN 50128)

o M: Mandatory
o HR: Highly recommended (rationale behind not using
it should be detailed and agreed with the assessor)
o R: Recommended
o ---: No recommendation for or against being used
o NR: Not recommended

33
 Example: Document structure (EN50128)
System Development Phase Software Maintenance Phase
System Requirements Specification Software Maintenance Records
System Safety Requirements Specification Software Change Records
Software Planning Phase
System Architecture Description
Software Development Plan
System Safety Plan Software Assessment Phase
Software Quality Assurance Plan
Software Configuration Management Plan Software Assessment Report
Software Verification Plan
Software Integration Test Plan Software Requirements Spec. Phase
Software/hardware Integration Test Plan Software Requirements Specification Software Validation Phase
Software Validation Plan Software Requirements Test Specification Software Validation Report
Software Maintenance Plan Software Requirements Verification Report

Software/hardware Integration Phase

Software/hardware Integration Test Report

Software Architecture & Design Phase

Software Integration Phase
Software Architecture Specification
Software Integration Test Report
Software Design Specification
Software Architecture and Design Verification Report
30 documents in a
systematic structure
 Specification Software Module Design Phase
Software Module Design Specification
Software Module Testing Phase
Software Module Test Report
 Design Software Module Test Specification
Software Module Verification Report
 Verification
Coding Phase
Software Source Code & Supporting Documentation
Software Source Code Verification Report

34
 Example: Responsibilities (EN 50128)
Organization Person
SIL 0:
ASS
DES, VER, VAL

SIL 1 or 2:
ASS
DES VER, VAL

SIL 3 or 4: MGR
ASS DES: Designer (analyst,
DES VER, VAL architect, coder, unit
tester)
or: VER: Verifier
MGR
VAL: Validator
ASS ASS: Assessor
DES VER VAL MAN: Project manager

35
 BACKGROUND MATERIAL

(For reference only, recommended to come back at the end of the course
to see how many techniques are familiar)

36
IEC 61508 V&V methods

37
IEC 61508 V&V methods – Testing

38
IEC 61508 V&V methods – Static analysis

39