DO Qualification Kit

Simulink Report Generator

Test Cases and Procedures
R2015b, September 2015

How to Contact MathWorks

Latest news:

www.mathworks.com

Sales and services:

www.mathworks.com/sales_and_services

User community:

www.mathworks.com/matlabcentral

Technical support:

www.mathworks.com/support/contact_us

Phone:

508-647-7000

The MathWorks, Inc.

3 Apple Hill Drive
Natick, MA 01760-2098
DO Qualification Kit: Simulink Report Generator Test Cases and Procedures

COPYRIGHT 20102015 by The MathWorks, Inc.

The software described in this document is furnished under a license agreement. The software may be used or copied only under
the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written
consent from The MathWorks, Inc.
FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the
federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees
that this software or documentation qualifies as commercial computer software or commercial computer software documentation
as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and
conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification,
reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or
other entity acquiring for or through the federal government)and shall supersede any conflicting contractual terms or conditions.
If this License fails to meet the governments needs or is inconsistent in any respect with federal procurement law, the
government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.
Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a
list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective
holders.
Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for more
information.

Revision History
September 2010
April 2011
September 2011
March 2012
September 2012
March 2013
September 2013
March 2014
October 2014
March 2015
September 2015

New for Version 1.3 (Applies to Release 2010b)

Revised for Version 1.4 (Applies to Release 2011a)
Revised for Version 1.5 (Applies to Release 2011b)
Revised for Version 1.6 (Applies to Release 2012a)
Revised for Version 2.0 (Applies to Release 2012b)
Revised for Version 2.1 (Applies to Release 2013a)
Revised for Version 2.2 (Applies to Release 2013b)
Revised for Version 2.3 (Applies to Release 2014a)
Revised for Version 2.4 (Applies to Release 2014b)
Revised for Version 2.5 (Applies to Release 2015a)
Revised for DO Qualification Kit Version 3.0 (Applies to Release 2015b); renamed to Simulink
Report Generator Test Cases and Procedures

Contents
1 Introduction ...................................................................................................................................... 1-1
2 Test Cases and Procedures ............................................................................................................... 2-1
2.1 System Design Description Report ......................................................................................... 2-1
2.2 Model XML Comparison Report ............................................................................................ 2-1
3 Traceability Between Operational Requirements and Test Cases and Procedures........................... 3-1
3.1 System Design Description Report Traceability Matrix....................................................... 3-2
3.2 Model XML Comparison Report Traceability Matrix ......................................................... 3-6
4 Executing Test Cases and Procedures .............................................................................................. 4-1
4.1 Executing Test Cases and Procedures for System Design Description Report ....................... 4-1
4.2 Executing Test Cases and Procedures for Model XML Comparison Report .......................... 4-3
5 Test Results ...................................................................................................................................... 5-1
5.1 System Design Description Report ......................................................................................... 5-2
5.2 Model XML Comparison Report ............................................................................................ 5-4

vi

1 Introduction
This document comprises test cases and procedures (Reference DO-330 Sections 10.3.3 and
10.3.4) for the following capability of the Simulink Report Generator verification tool:

System Design Description report

Model XML Comparison report

This document is intended for use in the DO-330 tool qualification process for TQL-5 tools.
See also the DO Qualification Kit User's Guide, R2015b.

1-2

2 Test Cases and Procedures

Requirements-based test cases and procedures have been developed from the System Design
Description Report Operational Requirements and Model XML Comparison Report
Operational Requirements sections in the following document:
Simulink Report Generator Tool Operational Requirements
To access the tool operational requirements document, on the MATLAB command line, type
qualkitdo to open the Artifacts Explorer. The document is in Simulink Report Generator.

2.1 System Design Description Report

The test cases and procedures exercise the System Design Description (SDD) report capability,
and reside in the Simulink test models identified in System Design Description Report
Traceability Matrix. The test procedure applies each SDD report capability to a subset of the
preceding test models, and resides in the report generator file identified in System Design
Description Report Traceability Matrix.

2.2 Model XML Comparison Report

The test cases and procedures exercising the Model XML Comparison report capability include
Simulink Report Generator setup file qualkitdo_xmlcomp_rs.rpt and associated mfiles, mat-files, and Simulink test models. Test files and their allocation to the tool operational
requirements are identified in Model XML Comparison Report Traceability Matrix.
The report setup file invokes MATLAB code that sequentially modifies values of each
qualifying parameter of the test model using pre-generated test values and runs XML
comparison between the original and modified model. Generated comparison reports are
evaluated to confirm that the reports contain the correct information for the corresponding
parameters. Qualification status is assigned to PASS if all generated reports are correct. The
summary qualification test report summarizes the outcome of the tests run and include lists of

the qualified parameters in the appendixes. Figure 1 shows the Model XML Comparison testing
workflow.

Figure 1: Test Workflow

2-2

3 Traceability Between Operational

Requirements and Test Cases
and Procedures

3.1 System Design Description Report Traceability

Matrix
The test cases and procedures cover the Tool Operational Requirements for the System Design
Description (SDD) report, as shown in the following table. The test models are used by the test
procedure qualkitdo_rptgenext_tcp.rpt.
System Design Description Report Traceability Matrix
Test Model

Requirement ID

Expected
Result

slvnvdemo_fuelsys_docreq.slx
slvnvdemo_fuelsys_htmreq.slx

SDD_mdlver
SDD_checksum
SDD_userdef_title
SDD_userdef_subtitle
SDD_userdef_authors
SDD_blkdiag
SDD_execorder
SDD_name
SDD_path
SDD_SF_chart
SDD_data
SDD_SFtarget
SDD_reqs
SDD_configset
SDD_rptformat_PDF
SDD_rptformat_HTML
SDD_rptformat_DOC
SDD_rptformat_RTF
SDD_mdlver
SDD_checksum
SDD_userdef_title
SDD_userdef_subtitle
SDD_userdef_authors
SDD_blkdiag
SDD_blkparas
SDD_execorder
SDD_mdlref
SDD_configset
SDD_rptformat_PDF
SDD_rptformat_HTML
SDD_rptformat_DOC
SDD_rptformat_RTF

Included in
pre-generated
reports

sldemo_mdlref_basic.slx

Included in
pre-generated
reports

3-2

Test Model

Requirement ID

Expected
Result

SDD_cont.slx
SDD_discont.slx
SDD_discrete.slx
SDD_logic.slx
SDD_lookup.slx
SDD_math.slx
SDD_other.slx
SDD_signals.slx
SDD_sources.slx

SDD_mdlver
SDD_checksum
SDD_userdef_title
SDD_userdef_subtitle
SDD_userdef_authors
SDD_blkdiag
SDD_blkparas
SDD_mdlver
SDD_checksum
SDD_userdef_title
SDD_userdef_subtitle
SDD_userdef_authors
SDD_blkdiag
SDD_data
SDD_name
SDD_path
SDD_chksum
SDD_SF_chart
SDD_SFtt
SDD_eml
SDD_SFtarget
SDD_configset
SDD_rptformat_PDF
SDD_rptformat_HTML
SDD_rptformat_DOC
SDD_rptformat_RTF

Included in
pre-generated
reports

3-3

Test Model

Requirement ID

Expected
Result

SDD_eml_call_gf.slx

SDD_mdlver
SDD_checksum
SDD_userdef_title
SDD_userdef_subtitle
SDD_userdef_authors
SDD_blkdiag
SDD_outputs
SDD_execorder
SDD_name
SDD_path
SDD_SF_chart
SDD_SFgraphfunc
SDD_SFdata
SDD_eml
SDD_SFtarget
SDD_configset
SDD_rptformat_PDF
SDD_rptformat_HTML
SDD_rptformat_DOC
SDD_rptformat_RTF
SDD_mdlver
SDD_checksum
SDD_userdef_title
SDD_userdef_subtitle
SDD_userdef_authors
SDD_blkdiag
SDD_execorder
SDD_name
SDD_path
SDD_chksum
SDD_SF_chart
SDD_SFstates
SDD_SFtrans
SDD_SFevents
SDD_SFgraphfunc
SDD_SFdata
SDD_SFtarget
SDD_configset
SDD_rptformat_PDF
SDD_rptformat_HTML
SDD_rptformat_DOC
SDD_rptformat_RTF

Included in
pre-generated
reports

SDD_SF_ArithLogic.slx
SDD_SF_data_check.slx
SDD_SF_event.slx
SDD_SF_exit_substate.slx
SDD_SF_model.slx

Included in
pre-generated
reports

3-4

Test Model

Requirement ID

Expected
Result

SDD_subsys.slx

SDD_mdlver
SDD_checksum
SDD_userdef_title
SDD_userdef_subtitle
SDD_userdef_authors
SDD_blkdiag
SDD_inputs
SDD_outputs
SDD_blkparas
SDD_execorder
SDD_name
SDD_path
SDD_chksum
SDD_configset
SDD_rptformat_PDF
SDD_rptformat_HTML
SDD_rptformat_DOC
SDD_rptformat_RTF
SDD_mdlver
SDD_checksum
SDD_userdef_title
SDD_userdef_subtitle
SDD_userdef_authors
SDD_blkdiag
SDD_execorder
SDD_name
SDD_path
SDD_SF_chart
SDD_SFtt
SDD_SFdata
SDD_SFtarget
SDD_configset
SDD_rptformat_PDF
SDD_rptformat_HTML
SDD_rptformat_DOC
SDD_rptformat_RTF

Included in
pre-generated
reports

SDD_truth_table_model.slx
SDD_TT_Block.slx

Included in
pre-generated
reports

3-5

Test Model

Requirement ID

Expected
Result

SDD_Version_Description_Test.slx

SDD_mdlver
SDD_checksum
SDD_userdef_title
SDD_userdef_subtitle
SDD_userdef_authors
SDD_blkdiag
SDD_descr
SDD_inputs
SDD_outputs
SDD_configset
SDD_rptformat_PDF
SDD_rptformat_HTML
SDD_rptformat_DOC
SDD_rptformat_RTF

Included in
pre-generated
reports

3.2 Model XML Comparison Report Traceability

Matrix
Bi-directional traceability between the Tool Operation Requirements for the Model XML
Comparison Report capability and test cases, procedures and associated test files is provided in
the traceability matrix:
qualkitdo_xmlcomp_trace.xlsx
The traceability matrix is located in:
matlabroot/toolbox/qualkits/do/rptgenext/

3-6

4 Executing Test Cases and

Procedures
4.1 Executing Test Cases and Procedures for System
Design Description Report
The applicant will execute the test cases and procedures for the System Design Description
(SDD) report capability in the installed environment.
Note:

To run the test cases and procedures, copy the

matlabroot/toolbox/qualkits/do/rptgenext folder and subfolders to a
location where you have write access, and then run the test cases and procedures from
there.

Before you execute the RPT file, start a new MATLAB session.

To execute the RPT file, you must have MATLAB Report Generator and Simulink
Report Generator licenses.

To execute some test models, you must have Stateflow and Simulink Verification and
Validation licenses.

Execute the System Design Description Report test procedure:

1.

Configure the report generation:

a.

Specify the model covered in the report using the

qualkitdoCtrl.rptgenext.model variable in the base workspace:
At the MATLAB command line, enter the following command, where
model_file is the name of the test procedure:

qualkitdoCtrl.rptgenext.model = {'model_file'};
b.

Specify the report format you want using the

qualkitdoCtrl.rptgenext.reportFormat variable in the base
workspace:
At the MATLAB command line, enter
qualkitdoCtrl.rptgenext.reportFormat = {'PDF'}; to
generate a report in PDF format
qualkitdoCtrl.rptgenext.reportFormat = {'HTML'}; to
generate a report in HTML format
qualkitdoCtrl.rptgenext.reportFormat = {'DOC'}; to
generate a report in DOC format
qualkitdoCtrl.rptgenext.reportFormat = {'RTF'}; to
generate a report in RTF format

2.

Create the report:

Using the GUI:
a.

Double-click the qualkitdo_rptgenext_tcp.rpt file to open the

Report Explorer.

b.

Select File > Report. Simulink Report Generator creates the summary
report and the model level reports.

At the MATLAB command line, enter

report ('qualkitdo_rptgenext_tcp')
3.

After the report generation is complete, review the test report results. For more
information, see Test Results.

4-2

4.2 Executing Test Cases and Procedures for Model

XML Comparison Report
The applicant will execute the test cases and procedures for the Model XML Comparison report
capabilities in the installed environment.
Note:

To run the test cases and procedures, copy the

matlabroot/toolbox/qualkits/do/rptgenext folder and subfolders to a
location where you have write access, and then run the test cases and procedures from
there.

Before you execute the RPT file, start a new MATLAB session.

Set the Java heap size for your MATLAB session to at least 3072 MB before you
execute model coverage RPT files. To check the Java heap size, open the MATLAB
Preferences dialog box and select General > Java Heap Memory. If the Java Heap Size
value is less than 3072 MB, change it to 3072 MB, click OK, and restart MATLAB. (If the
maximum available heap size value is less than 3072 MB, select the maximum value.) This
may help you avoid java.lang.OutOfMemoryError messages.

To execute the RPT file, you must have MATLAB Report Generator and Simulink
Report Generator licenses.

To execute some test models, you must have Stateflow and Simulink Verification and
Validation licenses.

Execute the Model XML Comparison Report test procedure:

1.

If you need to qualify the operation of the comparison tool on models saved in previous
releases, you may specify two Simulink releases to test.

At the MATLAB command line, enter

qualkitdoCtrl.xmlcomp.mdlFormat = {'FORMAT1', 'FORMAT2'}
where FORMAT1 and FORMAT2 specify one of the following Simulink releases:
'R14', 'R14SP1', 'R14SP2', 'R14SP3', 'R2006A', 'R2006B',
'R2007A', 'R2007B', 'R2008A', 'R2008B', 'R2009A','R2009B',
'R2010A', 'R2010B', 'R2011A', 'R2011B', 'R2012A',
'R2012A_MDL', 'R2012A_SLX', 'R2012B', 'R2012B_MDL',
'R2012B_SLX', 'R2013A', 'R2013A_MDL', 'R2013A_SLX',

4-3

'R2013B', 'R2013B_MDL', 'R2013B_SLX', 'R2014A',

'R2014A_MDL', 'R2014A_SLX', 'R2014B', 'R2014B_MDL',
'R2014B_SLX', 'R2015B'
Examples of use:
- To qualify the comparison of models saved in R2012b and R2015b, enter:
qualkitdoCtrl.xmlcomp.mdlFormat = {'R2012B',
'R2015B'}
- To qualify the comparison of models from release R2012b to R2014a, enter:
qualkitdoCtrl.xmlcomp.mdlFormat = {'R2012B',
'R2014A'}
- If Models are saved in the current release,
qualkitdoCtrl.xmlcomp.mdlFormat does not need to be set. It will
default to the current release.
2.

Run the RPT file:

Using the GUI:

Double-click the qualkitdo_xmlcomp_rs.rpt file to open the Report
Explorer
Select File > Report. Simulink Report Generator creates the summary report
and the model level reports

At the MATLAB command line, enter

report ('qualkitdo_xmlcomp_rs.rpt')

Simulink Report Generator runs the Model XML Comparison Report tests and stores the
comparison reports and summary qualification report in this generated folder:
rptgenext/outputs/
The summary qualification report
qualkitdo_xmlcomp_qualificationreport.html contains information about
test environment, executed test files, and qualification results.

4-4

3.

After Simulink Report Generator has finished running the test procedure, verify that there
are no warning or error messages in the Simulink Report Generator message list window.

4.

Review the summary qualification report. For more information, see Test Results.

4-5

5 Test Results

5.1 System Design Description Report

Executing the following test procedure generates the following test result files. The test result
files provide the Tool Verification Results in a summary report and multiple model level
reports.
System Design Description Report Test Summary
Test Procedure

Test Results

qualkitdo_rptgenex outputs\qualkitdo_rptgenext_qualificationreport_for_<OS>.pdf2
t_tcp.rpt1
outputs\qualkitdo_rptgenext_qualificationreport_model.pdf3
outputs\qualkitdo_rptgenext_qualificationreport_model.doc3
outputs\qualkitdo_rptgenext_qualificationreport_model.rtf3
outputs\qualkitdo_rptgenext_qualificationreport_model.html3

Notes:
1 Requires MATLAB Report Generator, Simulink Report Generator, Simulink, Stateflow, and
Simulink Verification and Validation licenses.
2 Overview report.
3 Model level report.
In the above listed test results, model stands for
slvnvdemo_fuelsys_docreq, slvnvdemo_fuelsys_htmreq, sldemo_mdlref_basic,
SDD_cont, SDD_discont, SDD_discrete, SDD_eml_call_gf, SDD_logic, SDD_lookup,
SDD_math, SDD_other, SDD_SF_ArithLogic, SDD_SF_data_check, SDD_SF_event,
SDD_SF_exit_substate, SDD_SF_model, SDD_signals, SDD_sources, SDD_subsys,
SDD_truth_table_model, SDD_TT_Block, SDD_Version_Description_Test.

The applicant will:

Review the Tool Verification test results.

Confirm that test reports are generated without errors or warnings.
Configure the test results in a configuration management system.

For each test result file, the DO Qualification Kit for Simulink Report Generator contains a
pre-generated version of the reports with reviewed content. For each test result file, compare the
expected results (pre-generated version) to the actual results (locally generated version).
Perform this review only on the file formats that will be used for the specific project for which
the Simulink Report Generator is to be qualified.
When reviewing the actual results (locally generated version) in the qualification reports, the
Stateflow IDs for unlabeled transitions can differ from those provided in the expected results
(pre-generated version). However, the reports -generate the traceability information, even for
models using unlabeled Stateflow transitions. The SDD report capability of Simulink Report
Generator generates traceability information for a labeled Stateflow transition using the
transition label for the name of the transition. For example, [fails>=5]. In the case of

5-2

unlabeled transitions, however, SDD uses a string with the word Transition followed by an
ID. The ID is the value of the ID property of the Stateflow transition object,
Stateflow.Transition. Since the value of the ID property for a transition object changes
every time the model is opened, regenerating the report for the same model might lead to
different transition names. The transition names might not match those provided in pregenerated version of the report.
The actual model checksums (locally generated version) in the qualification reports can differ
from the checksums provided in the reference reports (pre-generated version). The locally
generated version of the model checksums depend on the execution environment and can be
platform-specific. If the execution platform of the locally generated version is not the same as
the execution platform used to generated the reference reports (pre-generated version), the
model checksums can differ.

5-3

5.2 Model XML Comparison Report

Executing the aggregate test procedure file qualkitdo_xmlcomp_rs.rpt generates and
stores the following data in /outputs/:

Generated summary qualification report

qualkitdo_xmlcomp_qualificationreport.html

Generated Model XML Comparison reports

Note: Summary qualification report will include lists of the parameters that were tested
in the appendixes.

The applicant will:

1.

Review the test results:

Review the summary qualification report

qualkitdo_xmlcomp_qualificationreport.html, generated when
qualkitdo_xmlcomp_rs.rpt executes.
-

Check that the information regarding test environment is correct.

In the Qualification Test Results section of the report, check that the
qualification test outcome is evaluated as PASS.
Note 1: If the qualification status is evaluated as FAIL, manually review the
content of the comparison reports for parameters evaluated as FAILED to assess
the impact of the failed parameter with respect to the software life cycle activities
performed using the tool. The status for each parameter is provided in the
appendices of the summary qualification report.
Note 2: A list of corresponding test cases is provided in the CMP page of the
traceability matrix qualkitdo_xmlcomp_trace.xlsx.

2.

Configure the content of the folder comprising the test cases, procedures and results in a
configuration management system.

5-4