Synergies between INCOSE SE Handbook, CMMI

and DO-178B
Dr. Antonio Monzon, CSEP
Airbus Military
John Lennon Av., 28906 Getafe (Madrid)
Spain
+34 605 71 22 32
antonio.monzon@military.airbus.com
http://www.linkedin.com/in/antoniomonzon/en
Copyright 2012 by Dr. Antonio Monzon. Published and used by INCOSE with permission.

Abstract. One of the biggest challenges in multi-normative organizations, in the field of

industrial areas like aerospace and defense, is to make all technical regulations compatible. A
huge amount of effort is spent by these organizations to maintain their accreditations
periodically. Although each standard has its own purpose and appraisal procedure, it seems to
be useful to know about the commonalities among norms, as well as extracting best practices
from the application of its requirements to the accomplishment of the others.
Particularly the INCOSE Systems Engineering Handbook can be considered as general
lifecycle framework for systems, the Capability Maturity Model for Integration (CMMI) is a
reference model to assess the process efficiency and RTCA DO-178B can be used as basis for
software certification. In literature very few contributions can be found to explain how these
standards interrelate and how they can work together to achieve more efficient organizations.
This experience paper aims to shed some light on the common aspects of these three standards
applicable for the production of on-board software. The combination proposal included in this
paper is the result of the experience of the author in the application of these standards (and
others) in the military avionics domain. Conclusions could be useful for their application to
other domains or even for future potential combined accreditations.

Introduction

Aerospace industry has been traditionally interested in standardization because of the inherent
complexity of the systems to be developed and the pressure of the airworthiness authorities to
guarantee safety.
Several standards have been put in place since late 60s from the Mil-Std-499 until last version
of ISO/IEC15288:2008 [6] to cover the system development life cycle.
In parallel to this international standardization mainstream, the International Council on
Systems Engineering (INCOSE) has produced a handbook [4] conceived as a practical
guidance reference for Systems Engineers.
In addition to these initiatives, the Software Engineering Institute (SEI) has promoted over the
last decade the de facto standard CMMI [3] (Capability Maturity Model for Integration) as a
reference framework to be used by organizations developing systems (embedding or not
software) to assess their maturity as productive entities. Although CMMI comes from the
software world, the latest version "for integration" considers general aspects of systems
production, regardless their applicability to software. This movement towards system's world
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

makes CMMI an interesting complement of SE standards to cover those process improvement

aspects other standards are lacking. The alternative standard ISO15504 (aka as SPICE) was
discarded from the comparison because it is considered less common than CMMI.
The airworthiness authorities require in addition the fulfillment of certification standards for
the development of complex systems (SAE ARP4754 [9]) and complex software (RTCA
DO-178B [8]). These standards pay a special attention to safety aspects.
Finally in the context of the development of systems for military application, additional norms
must be fulfilled, i.e. NATO AQAP 160 [1] or AQAP 2210 [2].
In such a complex normative context it is always difficult to know which are the borders among
norms and standards. In this context, many mappings can be found in the literature among
CMMI and classical ISO-series, but no or very few mappings are available for INCOSE SE
Handbook, DO-178B or AQAPs. For this reason it was considered relevant to publish the
experience on the comparison among these standards grounded on the experience gained in
their application.

Table 1: Multiple standards applicable to airborne systems

Product Lifecycle Standards

Airworthiness
Military Specific Certification
General Purpose
(NATO)
Corporate &
Engineering ISO 9001 / AS9100 FAR (FAA)
AQAP 2110
Department EN 9100 EN (EASA)
Level
System INCOSE SE Handbook
Development ISO/IEC 15288 SAE ARP 4754
Level CMMI-DEV
SW CMMI-DEV v1.2 ML 3
AQAP 2210 RTCA
Development ISO/IEC 12207
AQAP 160 / 169 DO-178B
Level EN 9115
HW
Development RTCA DO-254
Level

INCOSE SE Handbook and CMMI Summary

For conciseness reasons this section will not describe in detail the standards CMMI and
INCOSE SE Handbook. Their main characteristics are just highlighted and a summary of their
mutual mapping is presented.

Capability Maturity Model for Integration (CMMI) is a process improvement model that
provides organizations with the essential elements of effective processes, which will improve
their performance. CMMI permits two deployment approaches: continuous and staged. In the
first, it is possible to assess the capability of the processes and its purpose is just improving the
weakest parts of the organization. In the second representation the objective is to measure the
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

Table 2: CMMI to INCOSE SE HB Mapping.

maturity of the overall organization. Five maturity levels are considered in the model: Level 1 -
Initial (Chaotic), Level 2 - Managed, Level 3 - Defined, Level 4 - Quantitatively Managed and
Level 5 Optimizing. In addition, the model permits three constellations: CMMI for
development, CMMI for services and CMMI for acquisition. These three share a set of
practices which are considered as the CMMI common core.

For the purposes of this paper, the CMMI for development constellation and the specific and
generic practices for the Maturity Level 3 of the staged representation are considered. The
process areas of the levels 2 and 3 are shown in Table 2, organized by categories.

The INCOSE SE Handbook defines the discipline and practice of systems engineering (SE).
This handbook provides an authoritative reference to understand the discipline in terms of
content and practice. This reference contains a set of 25 processes organized in three categories:
Technical, Project, and Organizational Project-Enabling. Each process in the handbook has the
following structure: Purpose, Description, Inputs, Outputs, Process Activities, Common
approaches and tips, Process Elaboration and References. The list of processes is shown in the
Table 3 (numbering of processes from previous version of the HB).

From the reference [7] the following mapping results between CMMI and INCOSE SE HB can
be highlighted (the mapping strategy is not repeated in this paper for conciseness reasons):

An organization holding a management system aligned with the INCOSE SE HB will

cover about 60% of the CMMI Level 3 specific practices (see Table 2). With respect to
the CMMI Level 2 practices, the coverage is slightly better: 66%.
An organization with a CMMI accreditation of maturity level 3 covers 76% of the
activities included in the INCOSE SE Handbook (see Table 3).
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

Table 3: INCOSE SE HB to CMMI Mapping.

ID INCOSE SE Process Coverage by CMMI CMMI PAs
4.2 Stakeholder Requirements Definition 100% RD, REQM, IPM
4.3 Requirements Analysis 100% RD
4.4 Architectural Design 100% TS
4.5 Implementation 100% TS, VER, PI
Technical 4.6 Integration 100% PI
Processes 4.7 Verification 100% VER
(64% ) 4.8 Transition 100% TS, PI, VAL
4.9 Validation 100% VAL, PMC
4.10 Operation 0% N/A
4.11 Maintenance 0% N/A
4.12 Disposal 0% N/A
5.2 Project Planning 100% PP
5.3 Project Assessment 86% PMC
Project 5.4 Control 100% PMC
Processes 5.5 Decision-making 100% DAR, IPM
(97% ) 5.6 Risk Management 100% RSKM, PMC
5.7 Configuration Management 100% CM
5.8 Information Management 100% PP, PMC
6.2 Enterprise Environment Management 83% OPD, OPF
6.3 Investment Management 83% IPM, PMC
Enterprise and
6.4 SLC Processes Management 100% OPD, OPF
Agreement
6.5 Resource Management 88% IPM, PP, OT
Processes
6.6 Quality Management 100% PPQA, OPD, OPF
(77% )
6.7 Acquisition 88% IPM, SAM, VAL
6.8 Supply 0% N/A
Average 76%

RTCA DO-178B Essentials

The main aspects of RTCA DO-178B are described in this section. The information contained
does not pretend to be exhaustive but representative of the standard, according to the
experience of the author.

The RTCA DO-178B is considered as a mean of compliance, acceptable by the certification

authorities, to fulfill airworthiness requirements for airborne equipments embedding complex
software. This norm is related to others of the same kind, particularly to the system level
equivalent SAE ARP4754.

The RTCA DO-178B is structured in the following sections:

System Aspects Relating to Software Development:

This section describes the way data is flowing between system and software levels. In
addition it describes the software levels depending on the criticality of the failure
conditions (at system and aircraft levels). Finally it contains some additional
considerations on safety such as dissimilarity, partitioning or safety monitoring.
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

Software Life Cycle:

This section describes the software development phases, processes and activities. It
pays special attention to transitions between consecutive phases.
Software Planning Process:
This section describes the plans and standards required to produce software in
accordance to the standard. Plans are expected to have some specific sections defined in
this section (e.g. development environment, language and compiler or software test
environment).
Software Development Process:
This section describes the main processes needed to develop software in accordance to
the standard, i.e. requirements, design, coding and integration.
Software Verification Process:
This section describes the way verification has to be performed in order to be compliant
with the norm. The main verification activities are: software reviews and software
testing.
Software Configuration Management Process:
This section describes the way configuration has to be managed. It includes the
configuration identification, the creation of baselines or the management of problems
and changes.
Software Quality Assurance Process:
This section describes the way QA activities have to be performed in order to be
compliant with the norm. QA is expected to
Certification Liaison Process:
This section describes the way the applicant and the certification authority
communicate in order to properly achieve the expectations. Means of compliance,
interim audits and final acceptance are part of this process.
Software Life Cycle Data:
This section describes the content of all the artifacts to be produced and maintained
during the whole life cycle of the software product in accordance to the norm. The main
artifacts are: plans, source code, design, accomplishment summary, verification results.
Additional Considerations:
This section describes other aspects relevant to the norm. In particular, it takes into
account topics such as the usage of pre-existing software, tool qualification or
alternative methods.

The most relevant topics of the RTCA DO-178B from its application point of view are:

Plans:
The plans are an essential aspect for the norm. The plans have to be created at the
beginning of the project and must be followed strictly. Any deviation from the plans has
to be documented formally and its application tracked explicitly. There is a specific
audit aimed to ensure that all plans exist and are being followed (SOI#1).
Traceability:
Traceability is the generic mechanism that enables control of the changes, ensures the
overall completeness and that the problems identified by the V&V activities will
continue through implementation. It is expected that traceability exists between
requirements (high and low), between requirements and V&V activities and between
requirements and data elements recording problems and changes. Traceability is
required not only from a formal point of view (i.e. there are links or references among
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

elements) but essentially from a functional perspective. The functional content of the
requirements must be followed in a logical way across the levels of abstraction to the
source code. There is a specific audit to demonstrate that traceability exists at all levels
and is properly used (SOI#2).
Verification:
The verification activities applied to various artifacts (requirements, design, source
code, etc.) represent the most important aspect of this standard. As it is impossible to
verify that the system works correctly in all circumstances, the standard gives crucial
importance to all activities that serve to demonstrate that the products of the processes
at all levels have been reviewed with the aim of detecting errors. As part of this
verification process the standard also expects that all detected errors have been
addressed through the corresponding change proposals and that these change proposals
are implemented properly. One of the differentiating aspects of this norm is the concept
of verification of the verification, by which it has to ensure that the verification
activities (particularly their descriptions) can in fact verify that the product complies
with the requirements at their level. It is required to generate evidence that verification
activities are consistent with the requirements which fulfillment are intended to
demonstrate. There is a specific audit dedicated to ensuring that the verification
activities have been properly managed (SOI#3).
Safety:
The most characteristic aspect of this norm is that explicitly addresses the topic of
safety. The standard specifies 5 criticality levels of the software (DAL level) which
involve different constraints on the process, particularly on V&V.
Quality Assurance:
The norm gives great importance to Quality Assurance activities because it expects
someone independent from the implementation, to ensure that processes and plans are
being followed properly. The role of QA gets much more involvement in the product
details than other standards such as ISO or CMMI. QA role is expected to have
software specialist skills (e.g. capable of reading and understanding source code from a
functional viewpoint). The QA role must have the prerogative to stop the transition
between phases and is expected to generate evidence of their activities and to monitor
closely the closure of the findings identified during the formal audits and monitoring of
a specific plan for QA.
Configuration Management:
Another very relevant topic to the standard is the configuration management. It is
considered that no formal development can be started until a requirements baseline
exists (frozen requirements prior to implementation). Any artifact (product) must be
uniquely identified and should be under configuration control. All plans must have a
version assigned and there must be a master document that identifies the applicable
versions of all documents to a comprehensive product baseline. All proposed changes
must be formally approved by an established mechanism (typically a CCB) and be
followed to implementation, leaving evidence that the change has occurred in
accordance to the provisions in the proposal. A specific plan for CM is expected.
Tool Qualification:
The tools involved in the software construction process must be controlled. This
standard gives special attention to the qualification of tools (design, code generation,
management and execution of test cases, etc.) to make it clear that they do exactly what
is expected of them. It is necessary to provide specific evidence (qualification kit) as
part of the formal acceptance of the product.
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

Comparative analysis of CMMI, INCOSE SE HB and DO-178B

In this section the commonalities and differences among CMMI, DO-178B and INCOSE SE
HB are highlighted. The assessment methods are compared as well.

Figure 1. INCOSE SE HB, CMMI and DO-178B structural mapping

The Figure 1 represents the processes/topics which can be considered as common to all three
standards as well as those specific to each. This only represents the comparison of the
requirements contained in each standard (structural mapping). In addition to this structural
mapping, the evaluation methods will also be compared in this section as well as the SW
specificities and the level of detail of the evidences required by each. Although CMMI does not
include proper processes, the associated process areas (PAs) have been assumed as processes
for comparison purposes.

The common processes to the three standards are basically those related to the classic V-Model:
requirements engineering, design, implementation, integration and verification. In addition, the
management processes dedicated to project planning and monitoring are also shared. Finally
two transversal processes are also common: configuration management and quality assurance.

Measurement, risk management, decision making and validation processes are shared only
between CMMI and INCOSE SE HB.

The CMMI distinctive processes are basically those related to process improvement (OPF in
CMMI notation). MA (Measurement and Analysis) is closely related to this process area and
can be considered also as CMMI specific with respect to INCOSE SE HB (even sharing a
similar name). The process areas applicable at high maturity (levels 4 and 5) are also specific to
CMMI, but will not be considered for the comparison included in this paper.
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

The INCOSE SE HB specific processes are supply/acquisition, operation, maintenance and

disposal. They are not considered explicitly by CMMI or DO-178B. These are also shared with
other life cycle standards such as ISO 12207 and 15288.

Regardless the safety process (which is in fact not described within the standard), CMMI
practices cover 70% of the rest of the normative content of the DO-178B. This rate is due to the
specificities of the DO-178B. A similar analysis performed against CMMI shows that
DO-178B fulfillment represents 54% of CMMI practices. This basically means that only half
of the CMMI level 3 specific practices are covered by DO-178B requirements. This
comparison does not take into account the compliance of CMMI generic practices (which
would represent a lower coverage rate). The main difference comes from the level of detail
expected for certain products and the software specificities of the DO-178B with respect to
CMMI. In particular, the evidences related to traceability and verification are much more strict
for the DO-178B than for CMMI. Although CMMI does not impose any product per se, the
model suggests some typical work products. These are the elements from which the
conclusion is extracted.

Table 4: Comparison of model elements and assessment methods.

INCOSE SE HB CMMI DO-178B
Process Area (PA)
Process Process
Specific Goal (SG)
Model Objectives Activities (detailed)
Specific Practice (SP)
Elements Inputs (generic) Deliverables
Generic Goal (GG)
Outputs (generic) (detailed)
Generic Practice (GP)
SCAMPI
Audits (SOI)
Assessment Appraisals (C, B and A)
None Typically 4 are
Method Typically 2-3 are
necessary
necessary for level 3

The concerns related to standard comparison (see Table 4) can be summarized as follows:

INCOSE SE Handbook is a process reference which main focus is promoting

excellence in system development through the application of Systems Engineering
principles (holistic view, multidisciplinary approach, full perspective of the system as a
whole, etc.). Distinctive aspects of SE HB are: the interaction of Systems Engineering
disciplines with the system lifecycle and the consideration of non-technical processes
as relevant for the success of the system.
CMMI is a reference model to evaluate the maturity of organizations in producing
systems (with or without software). It is focused on the way organizations are
structured to capitalize the experience gained in the projects. Measurement is a central
aspect of the model which enables the continuous improvement based on quantitative
analysis of the process performance. Distinctive aspects of CMMI are: measurement,
process improvement and appraisal method.
DO-178B is a standard which enables the certification of projects (not organizations). It
is focused on the overall quality of the product and requires detailed evidences that
processes are working as expected. Key aspects for DO-178B are: plans, verification,
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

traceability, Quality Assurance and Configuration Management. Distinctive aspect of

DO-178B is safety.

Figure 2. SW specificity versus evidence demanding

One of the main differences between CMMI and DO-178B is that the first is oriented to the
certification of the organization as a whole and the second is only concerned to the certification
of a concrete project. DO-178B requires very detailed evidences (see Figure 2) just from the
project under evaluation because it is concerned on the quality of the software product. CMMI
instead focuses in the capability of the organization (organizational unit) in producing software.
CMMI appraisals require providing evidences from several projects covering the majority of
the organization business. In some aspects CMMI appraisals are stricter than DO-178B audits,
because the number and kind of evidences (PIIs in CMMI notation) is very important.

As a reference, during a CMMI appraisal to get maturity level 3 around 1.000 evidences are
reviewed in order to demonstrate compliance with 365 practices (149 SPs and 216 GPs). In
addition to the number of evidences and the number of practices appraised, another indicator of
the level of demand of CMMI is that the appraisal method requires gathering verbal
affirmations from the team members involved in the projects. Typically around 20% of the
staff is interviewed to get these affirmations. This part of the appraisal is the most important
because it provides confirmation that the practices are in fact known and used by the teams.
The objective of the interview is to assess the level of institutionalization of the practices
described in the model. Affirmations can contradict the evidences and the application of a
practice, and therefore they are crucial for the success of a CMMI appraisal. An SCAMPI class
A (appraisal to get maturity level) typically takes around five days. A previous work is required
to gather all the evidences that will be requested during the formal appraisal. Confidence is a
key factor for CMMI. It is gained in successive appraisals with increasing visibility of the
organization performance to the lead appraiser. Typically three appraisals are required to get a
maturity level (especially for ML 3): a gap-analysis of the process manuals (SCAMPI class C),
a preliminary analysis of the institutionalization level (SCAMPI class B) and a full analysis of
the organization (SCAMPI class A). Prior to the final appraisal for maturity, an intermediate
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

readiness review is performed in order to ensure that all the evidences are present and fit for
purpose. If the result of this review is negative the appraisal process may be delayed or even
cancelled. The lead appraiser gains confidence on the maturity of the organization through
these consecutive evaluation actions and may suggest at any point corrective actions or even a
partial or total cancellation of the process. Another distinctive aspect of the evaluation method
(SCAMPI) is that the final decision on the appropriateness of the evidences and affirmations
does not rely on just one individual but on the consensus of the appraisal team. The lead
appraiser participates in the evaluation process providing guidance to interpret the model
objectives and practices, but decisions are agreed within the team. The size of the appraisal
team varies depending on the scope of the assessment, but typically ranges from 4 up to 10
members. The composition of the team is also a distinctive aspect of the method. It is expected
that at least one of the members is internal to the appraised organization. The reason for that is
that evidences content and affirmations must be understood in the context of the organization.
External point of view is required to ensure objectivity, but it is also relevant to have the
internal perspective to properly place evidences in context.

The DO-178B audits (called SOIs) are the equivalent mechanism to the CMMI appraisals.
These audits are performed in front of the certification authority and typically take two days per
session. The way audits are performed is slightly different than CMMI interviews. They are not
in fact interviews but presentations and demos on certain topics agreed in advance with the
certification authority. These presentations and the associated evidences (typically documents)
must be shown to the authority and the questions on the topics answered during the session. If a
question cannot be answered for any reason, an action is taken to be solved after the audit.
Another difference with CMMI appraisals is that SOI audits rely more on the evidence content
than on the affirmations (answers to the questions). The total number of evidences reviewed
during an audit depends very much on the stage of the project execution and the size of the
product to be evaluated. During an SOI#1 typically only the plans are reviewed and perhaps
some additional supporting documents (10-15 documents). During an SOI#2 the total number
of presentations used is around 12; the number of evidences reviewed is 15 general documents
(plans, records, reports, etc.) and around 10 more per CSCI (including demos of supporting
tools). The strategy followed in the DO-178B audits is incremental. Each SOI has clear
pass/fail criteria and once an audit is considered as passed the evidences shown are not
revisited again unless pending actions are taken to be solved in further SOIs. The audits are
conducted typically by one individual (certification authority) and supported by one or several
DCSs. The questions are indistinctly made by the auditor and the DCSs, but the final decision
on the approval of the audit relies on the auditor. This is one of the most important differences
with CMMI appraisal method. Confidence is also a relevant factor for DO-178B and it is
gained through the incremental and successive presentation of evidences to the certification
authority. Audits are the main mechanism to handle this confidence and their results are
binding to all parties. The way confidence is handled in this case is slightly different than in the
case of CMMI. An audit may serve to gain or lose confidence and the way to regain confidence
is to provide evidence that the actions raised during the audits are tracked to closure.

In contrast to CMMI and DO-178B, no explicit method exists to evaluate adherence to

INCOSE SE HB. For this reason this standard has to be considered just as a reference. The way
to accomplish adherence evaluation at the organizational level is to rely on other life cycle
standards such as ISO 12207 or their military versions (AQAP 160 and AQAP 2110/2210). As
long as current version of INCOSE SE HB shows an ISO 15288 shape, and last versions of ISO
12207 and 15288 share exactly the same structure, the assessment is possible due to the
one-to-one mapping at process and activity levels. The ISO software life cycle standards are
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

evaluated the same way than ISO 9XXX series: an authorized audit team reviews the processes
of the organization and during a single session (typically two days), and all the requirements of
the standard are assessed through a question-answer mechanism.

Proposal for the combination of standards

In this section a combination of the standards is proposed in order to better take advantage of
the strengths of each. This combination is possible because they have in fact different purposes
and apply to different abstraction levels. This fact allows an excellent complementarity of these
three standards. Although many commonalities can be observed from a structural point of view,
the main differences are in the level to which the processes apply.

The strategy for the combination proposed in this paper consists in organizing the processes in
layers: organization, system and software. These layers are not independent and the processes
may affect different layers (particularly those related to the engineering processes).
Nevertheless it is possible to assign characteristic processes to each layer. The proposal is to
use in each layer the most appropriate process according to the criteria described below (see
Figure 3).

Figure 3. Conceptual combination of INCOSE SE HB, CMMI and DO-178B

The first layer is the organizational which involves product life cycle processes such as Supply,
Operation, Maintenance and Disposal. These processes have to do respectively with the
delivery, utilization, support and retirement of the system, and represent the highest level in the
layered structure. The INCOSE SE Handbook is the only standard addressing these processes.
This reference is therefore considered the appropriate at this level.
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

The Supply Process is invoked [4] to provide the means necessary for conducting the project,
with the result of a product (airborne system in our concrete domain). Acquisition Process is
alternative to the development, in order to optimize the final product cost. The Supply Process
involves the execution of the development cycle. The Operation Process intends to provide the
appropriate resources to operate the system. In parallel, the Maintenance Process aims to
sustain the system capabilities during its operation. Finally the Disposal Process manages the
deactivation and/or disassembly of the system once its operational life has ended.
All these processes are specific of the INCOSE SE Handbook and their consideration is
necessary to sustain the system within its intended operational scenario.

The second layer represents the proper system development. At this level, it seems reasonable
to apply the project management and engineering processes involved in a classical
development life cycle. In addition to these project management processes (not represented in
the Figure 3), the relevant processes which are common to the SE HB and to CMMI are:
Validation, Measurement, Risk Management and Decision Making. The Validation Process is
understood as the compliance mean to ensure that the system is achieving its intended use in its
intended operational environment. The Measurement Process aims to gather data to assess the
performance of the processes and the quality of the products, in order to make decisions based
on objective criteria. Decision Making and Risk Management processes ensure that decisions
are properly justified and taken, and threats are identified and mitigated.
The organizational processes specific to CMMI which apply at this level are Training and
Process Improvement. The first has to do with the sustainment of the skills required to perform
the management and development tasks across the development life cycle. The second intends
to gather all the information available in the projects and the organizational assets (mainly from
Measurement) in order to detect opportunities for improvement. The main goal of this process
is to make an organization more profitable by reducing the production costs and promoting
quality as a key success factor.

The third layer is the proper software development. The idea here is to apply DO-178B
processes and constraints to produce reliable and maintainable software. Although many
alternative software development life cycles are available (agile, incremental, iterative,
cascade, spiral, etc.), the one considered appropriate for the development of certified software
for aviation (under strict safety constraints) is the one proposed by the DO-178B.
Regardless the specificities of this standard on safety assessment, QA and CM, the most
relevant difference with other standards is the way verification is conceived. The DO-178B
explicitly mentions three layers for the specification of software: High Level Requirements
(SW HLR), Low Level Requirements (SW LLR) and Source Code.

The verification activities affecting the SW product can be summarized as follows:

Review of the SW HLR and SW LLR (e.g. checklists against SW Requirements

Standard).
Review of the SW Source Code (e.g. checklists against SW Coding Standard).
Review of the SW HL and LL Test Cases (e.g. checklists against SW Testing
Standard).
Formal testing at HL and LL.

The number and depth of evidences on verification is especially relevant for DO-178B.
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

Success story

The approach described in the previous section was applied to the establishment and
maintenance of the SW Management System of the On-Board Software organizations within
the Engineering and Technology directorate of Airbus Military.

The success of the approach is supported by the achievement of two successive accreditations
of CMMI maturity level 3 (in 2008 and 2011), the AQAP 160 certification (in 2005), the
AQAP 2210 certification (in 2011) and the certification (according to DO-178B) of the
fly-by-wire control computer of the air-to-air refueling system of the Airbus derivative MRTT
(DAL A) and the application software of the Military Mission Computer of the A400M (DAL
C).

In addition to the successful certification of the organization and their projects, the following
benefits of this combined approach can be highlighted:

1. Better integration of system and software levels: the usage of Integrated Product Teams
(IPTs) has improved the communication and the effectiveness of the organization
(particularly successful in the A400M program).
2. Clearer reporting to program level: a robust set of metrics is generated to assess the
software production effectiveness and efficiency. The indicators selected to feed the
engineering dashboard are the SW Defect Density and the Requirements Volatility.
3. More efficient process improvement: the improvements to processes are proposed
directly by the projects and are in line with the business objectives.
4. Easier fulfillment of multiple standards: the combination of standards enables a
simplified fulfillment of the requirements of the standards, particularly in those projects
under DO-178B constraints.

Note: The details of the tailoring and the concrete benefits from the combination of the
standards are not included in this paper for industrial property and security restrictions.

Conclusions

After the analysis of the commonalities and differences among INCOSE SE HB, CMMI and
DO-178B the following observations can be extracted:

INCOSE SE HB compliance does not guarantee CMMI nor DO-178B compliance: the
main reasons are that the HB applies at organizational level, does not impose a concrete
life cycle and has no associated evaluation method; therefore the standard itself does
not guaranty any further compliance.
DO-178B compliance does not guarantee CMMI compliance: the reasons are clear,
first DO-178B focuses only in a limited set of the practices required by CMMI, and
second the levels of applicability of both standards are quite different.
CMMI compliance does not guarantee DO-178B compliance: although CMMI
involves almost all the processes considered in DO-178B, the level of detail required
for certain deliverables imposed by the second does not permit a direct compliance of
DO-178B. In addition, safety concerns are not taken into account in CMMI.
CMMI or DO-178B does not guarantee INCOSE SE HB compliance: the basic reason
is that the HB covers several aspects not treated by the other two standards.
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

From these observations it can be concluded that direct compliance among standards cannot be
achieved. Nevertheless, there is a partial structural coverage of standards (summarized in Table
5). According to the experience of the author, the better way to combine the standards is to
follow a layered strategy, applying the parts of the standard which better fit the purposes at
each level, taking into account the normative constraints. In this line, to make the standards
compatible it is necessary to enlarge the set of processes and to increase the detail of some
deliverables in order to make them actually compatible. The result of this combination is a
departmental Software Management System which collects the requirements from all three
standards in a consistent way. In the case of the INCOSE SE HB, the usage of additional life
cycle standards is necessary in order to supplement the lack of evaluation method, in particular
the AQAP series (military version of the ISO 12207).

Table 5: Structural coverage of standards.

INCOSE
CMMI DO-178B
SE HB

INCOSE
60% 42%
SE HB

CMMI 76% 70%

DO-178B 41% 54%

Note: This table must be read by rows. As an example, in the second row, CMMI specific
practices cover 76% of INCOSE SE HB process activities and 70% of DO-178B process
requirements. The mapping is bi-directional for the three standards under comparison.

The main problem found in the application of this combined normative is the variability in the
evaluation methods. Even considering that the departmental process definition fully covers the
three standards from a structural point of view, the assessment processes are quite different and
require specific audits and appraisals.

A future improvement to ease the accreditation of multi-standard organizations could be to

propose crediting or compensation mechanisms which could allow the convalidation of certain
parts of the standards fulfillment. In particular it would be especially useful that ISO-series
would recognize CMMI accreditations in order to avoid unnecessary repetition of process
audits. In addition, some DO-178B audits could be partially covered by the evidences extracted
from a CMMI accreditation process. Finally a more detailed mapping of INCOSE SE HB with
CMMI is suggested in order to better highlight the synergies among them in future editions of
the handbook.
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

References

[1] AQAP 160: NATO Integrated Quality Requirements for Software throughout the Life
Cycle, NATO Standardization Agreements, September 2001
[2] AQAP 2210: NATO Supplementary Software Quality Assurance Requirement, NATO
Standardization Agreements, November 2006
[3] CMMI for Development, Version 1.2. SEI Technical Report, CMU/SEI-2006-TR-008,
August 2006.
[4] INCOSE 2010 Systems Engineering Handbook - A guide for system life cycle processes
and activities, Version 3.2. ed. Cecilia Haskins, January 2010.
[5] ISO/IEC 12207:2008: Systems and software engineering Software life cycle processes,
Geneva: International Organization for Standardization, issued 1 February 2008.
[6] ISO/IEC 15288:2008: Systems and software engineering System life cycle processes,
Geneva: International Organization for Standardization, issued 1 February 2008.
[7] Monzon, A.: Bi-directional Mapping between CMMI and INCOSE SE Handbook.
Embedded Real-Time Software International Conference, Toulouse, May 2010.
[8] RTCA DO-178B: Software Considerations in Airborne Systems and Equipment
Certification, RTCA Inc., 1992.
[9] SAE ARP4754: Certification Considerations for Highly Integrated or Complex Aircraft
Systems, Society of Automotive Engineers Inc., 1996.

Biography

Antonio Monzon is Process Manager in the Systems/SW Processes, Methods and Tools
Department of Airbus Military. He holds an MsC in Aerospace Engineering and a PhD in
Telecommunications Engineering both by the Technical University of Madrid (UPM). He has
17 years of experience mainly in on-board software embedded in military avionics for tactical
mission applications, but also in other domains such as telecommunications or banking.
He is Certified Systems Engineering Professional (INCOSE CSEP) and EADS Black Belt and
participates in trans-national EADS groups as expert in Model Based Systems Engineering and
process improvement.
He has been project manager, site coordinator and appraisal team member in three CMMI
accreditations, audit coordinator in the SOIs for the DO-178B certification of the Military
Mission Computer of the A400M and technical responsible of the AQAP certifications of
Airbus Military.
In addition he is also interested in research and academia. He periodically participates as
speaker in different Systems/SW Engineering international events, is member of international
engineering organizations (e.g. INCOSE) and contributes to standardization initiatives (ISO).
 Synergies among INCOSE SE Handbook, CMMI and DO-178B

Acronyms

AQAP: Allied Quality Assurance Procedure

CCB: Configuration/Change Control Board
CM: Configuration Management
CMMI: Capability Maturity Model for Integration
CSCI: Computer Software Configuration Item
DAR: Decision Analysis & Resolution
DCS: Delegated Certification Specialist
HB: Handbook
INCOSE: International Council of Systems Engineering
ISO: International Standardization Organization
IPM: Integrated Project Management
MA: Measurement and Analysis
MRTT: Multi-Role Tanker Transport
OPD: Organizational Process Definition
OPF: Organizational Process Focus
OT: Organizational Training
PA: Process Area
PI: Product Integration
PMC: Project Monitoring & Control
PP: Project Planning
PPQA: Process and Product Quality Assurance
QA: Quality Assurance
RD: Requirements Development
RM: Requirements Management
RSKM: Risk Management
RTCA: Radio Technical Commission for Aeronautics
SAM: Supplier Agreement Management
SCAMPI: Standard CMMI Appraisal Method for Process Improvement
SE: Systems Engineering
SEI: Software Engineering Institute
SOI: Stage of Involvement
SP: Specific Practice
SW: Software
TS: Technical Solution
VAL: Validation
VER: Verification