A Software

Engineering

Experiment

in Software

Component

Generation*

Jeffrey

Richard

B. Kieburtz

Bell

James Hook

M.

Dino

P. Oliva

Tim

Laura

Abstract
This

paper

neering
constructing

gram

that

common

mand,

and

software

of

domain,

with

reusable
were

applied

constructing

modules

communications

for

and

to

ter

message

military

information

The

employed

of use of the

of test

examples.

personnel

Test

taken

The

results

and

jects

used

Ada

templates

to

that

were

The

levels

when

when

sofiware

they

modules

differences

in

are statistically

exceeding

domain

the

the

component

that

SDRR

subused

structed
matic

average

program

the

potential

duce

to improve

the incidence

velopment
results

and
by

human

of errors

comparing

productivity
This

conducted
the

to test

performance

and
reports
these

re-

$5.0001996
of ICSE-18

IEEE

in a selected
developed

to conreasonably

target
with

domain-specific
a software

language

SDRR

com-

language

component

from

in which
a selected

domain,

a translator

de-

provides

the

tion,

for
early

the

specification

prototyping

of

language
a generated

that
solu-

hy-

of subjects

*The research reported here was supported by a contract

with the USAF Materiel Command.
Research
Center,
Oregon
Graduate Instit pacific Software
OR
tute of Science & Technology, P,O. Box 91000, Portland,
http://www.cse.
ogi.edu/PacSoft/.
Authors
97291-1000
USA,
,mckinney]
@cse. ogi.
edu
e-mail addresses: [click

Proceedings

generator

a declarative,

a reusable
tion

0270-5257/96

translation
into

con-

of auto-

have

in software
paper

seman-

The

prises

reliabil-

generators

introduced

maintenance.

of an experiment

potheses

software

the

implementation.

that

code

informal

techniques

definition

of

provided

from

and

for

contribution

technology

advanced

semantic

compilable

use for

language.

transformation

to specify

believed

uses

a high-level

for

the

a soft-

to formalize

unique

generator

a specification

generator

efficient,

usability.

is widely

is in

of

problem
It

method

specify

abstractions

experts
The

is

language

It is intended

designs.

CenSDRR

is designed

high-level

domain

of their

tech-

designer

can

language

domain.

definition

Introduction

in the

tic

vert

application

the

Reuse)

the

specification

the

was

from

an

In

specialist

to capture

and

Institute.

for

a program

szgnijicant

productivity,

Research

construct

99 percent.

generation,

synthesis

A specification

A software

flexibility,

software
Software

applications

vernacular

using

by the Pacific

to

Keywords
Software

each

of the

productivity

than

solution.

tasks.

constructed
Reliability

Graduate

in

reusable

for

interface

the

ware

performance

programming
was

a domain-specific

description

specifica-

introduced

generator

of the subjects

at confidence

with

independent

of message

greater

implement

of specifications.

performance

ity,

the

importance

C31 systems.

error

the program

set

conducted

by an

consisted

are

fewer

was

supervised

Force

to conduct

on a common

experiment

and
Am

subjects

technologies

cases

from

main

achieved

sets

The

supplied

contractor.
tions

two

four

same

Design

Oregon

upon

in which

experiment

trials

of the

their

technology,

[4, 5], an experimental

based

sys-

with

generator

developed

method,

com-

to the

software
(Software

nique

Walton

state-of-the-art

templates,

method

pro-

Lewis

generator

another

SDRR

a reuse

Ada

Lisa

a software

The

domam-specific

Jeffrey

Smith

code

for

(C31).

tems

Ira

applying

engi-

technology

technologies

validation

control,

new
from

sets

Both

problem

translation

has been compared

employs

templates.

of

which

generators

languages

technology

Kotov

using

results

in

program

specification

presents

experiment

McKinney

Alex

Sheard

of the

interface

that

prototype

a back-end
selected

542

program

pipeline

transformation
reshapes

and
improves

translathe code

translator,

code generator
target

and

language,

specification.

that

produces

conforming

code

in a

to a software

The
over

user

the

of a program

algorithms,

standards

that

mentation

of a software

cost and reliability

tion.

The

to

SDRR

the

level.

applying

it in a single,

comparing

it

an

notions

more

flexible

monly

held

ers would
reusable
that

generation

plement

effort

many

design

Ten
reuse

assemble

systems

An

capture

hardware

help

to assure

such

in code

interface
costly

types

and

and

of a resulting
Ideas

software

trying

domain

sign

that

Often,
cations

which

within

kit

There
common

aspects

A soflware
of

entities

will

make

artifacts,

makes

it

on

an

and

or more

on the
concepts

while

which

(components,

[7] is

a software

product

and

set

These

of

how

may

be

of

components,
a large

or

a very

or

use

sophisticompo-

[11].
MTV

Templates
reuse

the

a language-based

from

MTV

kits.

and

are

view

of the

Templates,

to the

code

applica-

the

emphasis

templates,

interface

in-

for

and

these

applications

en-

technologies
technology

compared

Engineering

types

of fields

ing numeric

fields
There

which

fields

in

an external

to

be

These

The

to

application,

to

of a set
to the
includ-

records

and enu-

specify

patterns

as strings

templates

of bits

contain

spe-

are to be replaced

templates

produce

(MTV)

consists

text,

that

code

selected

correspond

encoded

identifiers

the

in a messagee,

width,

message.

bracketed

It

is a template

are

for

that

expected

of any

the

Communications

systems.

templates

the

by

validation

Control,

(C31)

program

in

[9]
and

Command,

which

developed

Institute
translation

Force

with

was

can

Ada

by

be edited

functions

specialized

to a particular

Templates

solution,

for

by
the

message

format.
the

templates

that

functions

they

stream

that

543

translate

represent

test

somatically.

scale,

MTV

is synthesized

which

simple

templates.
small

the

of prototype

merations.

and
prototype

in-

seen by

adapt

templates

hidden

visible

was

a message.
A

are

Information

MTV

The

are related.

archi-

existing,

is on synthesis

prototype

reuse

parameters.

the kinds

interfaces)

and

provides

templates-based

Air

cially

of domain-

are central.

types,

may
and

of domain-specific

With

the

generic

applications

describes

and

emphasis

the

and

With

architecture,

the

The

in an application

framework

of 12 Ada

domain.

idea

by
from

the interface

to select

Generator
the

Two

and

are:

up

MTV-G,

engineer.

for

a process

application

framework

Software

requirements
kit

MTV

domainmessage

de-

one

the

search-and-replace

variations
certain

inclusion

the

The

domain.

but

im-

[3].

components.

provides

tools

into

1.2

ap-

of appli-

of new

that

a synthetic

gineer.

products.

and

guided

an appl;ation

engineer

for

generator

domain

tools

construction

are many
reuse,

as

flexible

reuse

forms

of components

or customized

nents

constitute

performance

describing

Genera-

providing

framework,

is on customization

workproducts

domain-specific

centered

a specified

specific

for

[10].

by prototype

bu-ilds

inference

tion

to new

of software

of

templates.

framework

generation

such

pre-implementation

an architecture

enables

the

a more

is a particular

languages

them,

This

or customization

prototype

creation

application

terface,

of compo-

reusable

a class

of reusable

using

A reuse

and

class

specifications.

For

components

components

to

in

to

share

domain-specific
a collection

the

that

integration

fami-

decisions

affect

evolved

engineering

reuse

envisioned.

details.

capture

is common

the

for

cated

With

product.

have

to

way

concept

technology

software

generation

of features,

Both
fit

has not developed

design

adversely

of

to the

software

algorithm

can

on reuse

proach,

late

existing

library

compatibility

standards,

too many

to adapt

applications

the

industry

govern

reuse

a comdevelop-

This

had

the software

which

ago,

designed.

that

Lacking

of

was that

em-

be directed

integration

the

collections

a standard

as some

reuse
be-

years

process

from

classes

tecture,

moving

in analogy

were

as successful

reasons,

from

components,

systems

to
been

towards

artifacts.

standards

nents.

libraries

of software

soft ware

lies and

can

facility

was

state-of-the-art

have

the

or synthesis

is, by
domain

approach

vision

has not been

tion

application

technology

whether

support

components.

tegration

of reuse

hardware

for

Tools that

synthesized

in reuse

yond

upon
assembly

implefor the

here

this

available,

phasizes

automa-

described

successful

depending

coding

technology

of software

A domain-specific
Concepts

or

supporting

non-trivial

with

and

control

technology.

1.1

the

design

in exchange

experiment
how

and

ages of design

and

automation

to evaluate

and

the

y advant

The

designed

reuse

to

component

method

to take

a new

sacrifices

representations

contribute

time,
attempt

generator

data

ations

perform
This
harness

into
and

validity
the

and
package

package

between

internal
checks

a set of edited

Ada

messages

synthesis
for

an

data
on the

the
ar;

their

of six
data-

structures,
contents

generation
performed

of
of

au-

For

the

MTV

the SDRR
ified

in

for

MTV-G

tions

with

interfaces
In

executable
tested

before
automation

tous.

The

interpreted
validity

may

typical

MTV

several

sensors.

Each

reuse
modules.

this

experiment

was

already

MTV

because

the

available,

can

Final

be fur-

on fields

of the

from

Hypotheses

domain

was selected

for

of

IBM

Federal

productivity
by

for

versus

Division

custom

coding

tralize,

compari-

reported

in

a more

reliability

and

templates

code

reuse

The
test

that

The

may

developer

of applications

[6].

This

level

differences

with

that

designed

experiment

least

[2] that

benefits
We

believe

as effective

The

evalu-

of code

other

experiment

we report

hypotheses

tor

that

was

the

MTV

this

experiment

developed

Templates.

with

MTV

SDRR

Specific

was designed

the

MTV

hypotheses

to

of analysis.

obtain

with

technologies.

instances

of message

as can

Templates.

in
and
with

creating

or

validation
the

MTV

modifying
module

will

be more

a message
when

using

ranthat

matching

(the
by

Vott

of the

factor

control

same

of the

such

experiment

subjects,

is

of the faccan be de-

measured
to

data.

control

as the

for

order

technologies,

the

a common

of progress

in
these

of

order
work

in learning

new

factor

remaining

factors

have been

of analysis.
systematic
there

remains

tasks.
task

replication,
a and

if the

systematic

individual

of variable

in a software

to

only

is differences
However,

is essential

two

rate

extraneous,

of

Porter,

544

the

controlled,

translation
than

in the

The

effects

MTV-G

(C)

factors

of independence

analysis

of tasks
and

productive

Templates.

degree

factors,

after

and

extra-

variation

independently

systematic

difficulty

gies)
2. Productivitydevelopers

for

vary

it

successfully

to at least

specification

The

factor

subjects.

independence,

be the

Even

for

and

to those

residual

them

of subjects

should

neu-

of extraneous

of trials,

be accounted

of presentation

in

of analy-

one must

to control

respect

human

by statistical

environment,

tested

can be configured

can

Genera-

technology

of the

potentially

are:

1. FlexibilityMTV-G
as many

on here

comparing

an experi-

to eliminate

replication

(vi)
(vii)

familiarity

factor

effects

available

important

to make

training

(viii)

used,

the

work

motivation,

of a single

matching

with

(iv)

systematically.

most

termined
To

(A)

(B)

abilities

tor

reuse

reusable

as any

are

tech-

the

experience,

and

technologies

of trials

vary

the

inconsistent

times

the

be in-

(i)

In designing

as possible,

trial,

templates,

modules.
at

each

and

of tasks,

effect

techniques

domization

by

code

is not

csse

factors

reusable

recent

are

1980s

techniques.

several

the

showed

productivity

wit h object-oriented
code

late

the

of human

technologies,

procedures.

insofar

factors.

at

the

of difficulty

in

Phase

in the technologies,

education
ability

design

one may

including

the

prior

to measure

the

[5].

as this

of training

in individual
level

four

increase

as

be found

Reuse:

Report

of factors

of learning
(v)

can

and

such

order

task-specific

approximately

domain-specific,

of productivity

ated

improved

in

(ii)

of

details

of the performance

used,

sis, in this

Experiment

Systems
was

using

the

gathered

attributes

difficulty

overview

Technical

nologies

ment

solution

a basis

and

by a variety

with

is difficult

templates-based

data

MTV-G

Templates.

Reliability

fluenced

boredom,

Thus

for

in an experiment

differences

similar

of details,

components

its

a brief

subjects

neous

Unpublished

in
de-

perceive

MTV

Complete

Design

environment,

son.
2

will
the

gives

Scientific

(iii)

several

performs

section

Measurable
or

is trans-

messages

providing

introduced

in a solution

Design

experiment.

Software

ubiqui-

incorporates

module

This
of the

of bits

that

a multitude

The

be

Templates.

to use than

Experiment

be

is to translate

As a message

of software

can

for soft-

as streams

structure

MTV

than

an
with

are

module

be performed

in

easier

compiled.

modules

to translate

differs

conventional

and

candidate

C31 system

modules

but

MTV

in MSL

arrive

data

checks

functions

written

by a controller.

different

for

that

an internal

message.

translator,

of an MTV

MTV

with

will

4. Usabilitydevelopers

func-

provides

message

MTV

veloped

defects
MTV-G

templates-based

is generated

because

with

is

MTV-

of six

MTV-G

is an excellent

messages

into

lated,

code

function

incoming

of the

specification

domain

ware

ther

of the

Ada

MTV

those

for

development

de-

MSL

language

however,

prototype

a message

bytes

like

(MSL)

package

3. Reliabilityfewer

using
is spec-

application.

an Ada

addition,

which

language

MTV

specification

synthesizes

solution.

The

the

created

of a message

specification

specifically

domain-specific

G.

(MTV-G)

the format

a message

signed
the

Generator

technology,

factors
Basili

of factors
engineering

variability

This

is assigned
through
are
affecting

in both
which

diluted

[1] give

in

a detailed
human

experiment.

yin

the

is controlled

by

technolo-

the

random

import

ante.

discussion
performance

3.1

Trial

task

Message
trials

format

were

message

taken

varied

record

fields

fields

an

fields

were

fields,

all
were

value

Inter-field

value

constraints

An

Force

message

to

the

232

riod

prior

alternative

duct

field.

use.

terface

Control

specifies

the

scribes

the

acter

Document
layout

(ICD),

of each

length

field

by an In-

a tabular

of a message

length

if the

is specified

with

form

field-by-field.

that

and

gives

selected
tems

supplied

by personnel

Center

viewed

to

its

who

were

ysis

of

data

for

obtained

not

the

were
otherwise

to

OGI

would

were

being

the

were

nents.

The

both

subjects

technologies,

sample

Intermetrics,
These
tions

Inc.

subjects

persons

first

or anal-

3.5

test

contracted

were

selected
Each

gramming

They

domain.

had

There
subjects

for the particular

in conducting

the experiment,

experience

was

a disadvantage
experiment
facility
sion

required
in

trials
subjects

in

using
were

in Cambridge,
of the

were
are
but
order
MTV

conducted
Massachusetts.

was provided

and

years

To

to

and
were

then

in

in the

the

reverse

applica-

put

that

Templates.

subject

in

each

mented

at
The

Intermetrics

to use MTV

were

assigned

545

the

per-

formats

and

format
through

consisted

Eight

of the

chosen
tasks

there

were

to

subjects.
six of the

to individual

with
The

subjects
that
in

the

initial

to

ICDS,

then

imple-

selected

ex-

and

sequences

were

a subject

were

tasks

MTV-G

task

imple-

there

initial

subject
randomly

done

Templates.

technologies

was
thus

four

be

se-

56 modifications

112 maintenance

tasks,

eight

modifications

specifications

Each

mes-

simulated

formed

and

the

of ICDs

initial

for

cumulative

message

to the constraint
two

not

translation

was simulated

were

28 maintenance

were

would

message

specifications

tasks

14 were

in-

in advance

as a message

implemented

that

of the

by Intermetrics.

new

activity

technology.

cept

subject

supervi-

for

thus

among

or

the

as individuals.

in each of the two technologies,

Each
three

told

developing

software

of seven

of the

design

and

to measure

in

maintenance

ICD,

Each

feedback

experiment

designed

both

formats.

The

initial

user

sequences

specifications

be assigned

Ada

them

an

data
infor-

of changes.

each consisting

24 initial

of them

was

their

what

privacy

performance

trial

message

format

mented

somewhat

required

their

packages

initial

told

were

the

UNIX

that

recorded

any

subjects

MTV

not

environment

in

controlled

informed

subject

the

series

and

on identical

were

without

Maintenance

in all.

two

but

protect

12 independent

ries,

pro-

of the

Direct

and

trained

were

The

of subjects

maintenance.

in

and

at the

conwere

subjects

in a restricted

collected

validation

sage

males

to

to

development

environment

data,

maintaining

for

super-

Ada

it was felt
not

peThe

experiment
were

two

MTV-G
were

the

technology

in its

of

two

and

trials.

they

in each

subjects

experiment

The

with

qualifica-

probably

tasks

the

use of

experiment.

similar

two

knowledge

over-qualified

the

a BS or MS degree

one to three

use

recorded.

a cumulative

for the experiment.

to have

no prior

These

in

the

over

a ten-day

of the

was varied:

and

Experiment

evolves.

compo-

experience

to provide
trials

held

and had

experience.

females.

trained

to beginning

to conduct

science

development
MTV

hands-on

and experience.

computer

tion

were

prior

in parallel
of specified

including

problems

vise four

engaged

the

to evaluate

The

maintenance

trainers

be monitored,

data

be used

and

subjects

use

which

experience

The

of the

that

specification,

simulated

in
in

automatically,

tegrity

personnel

acceptance

work

in
Four

trained

worked

intervention.

Subjects

and

experiment

monitoring

formance
3.2

Generator

of the

order

subjects

mation

review,

in the design

generate

reOGI

on points
After

to other

involved

Sys-

the

clarification

given

of the

collected

Environment

workstations.

were
were

by

inconsistency.

experiment,

each

They

trials

MTV

start

The

in

Work
The

intended

Electronic

AFB.
of

the

during

Templates,

3.4

char-

experiment

Force

beginning

or apparent

specifications

who

Air

at Hanscom

the

manager,

of ambiguity

use in the

at the

(ESC)

prior

project

the

for

of

prior

experiment

ICDS

were

order.

interpretation.
The

a par-

It pre-

or a terminating

is variable,

that

provided

training
first

signed
him/her

was
also

work.

The

MTV

data

environment

persons

scaling.

use

were

their

subject

informed

he experiment.

to the

subjects

each

which

of the

in

monitoring

encountered.

format

form,

Templates

trained
Air

of training,

oft

Training

vari-

variant

course

MTV

character-

and numeric

were

C31

and

start
consent

uses to be made

Variable-

in a single

ranges

Force

message

Both

the

ticipant

in detail.

of fields

Up

specified

specified

rich

included.

lists

occurred.

Air

individual
40.

Training
At

experiment

of different

are quite
in

approximately

optional

patterns

Some

a number

types

bit-based

fields,

string

from

the

unclassified

specifications

7 to

and

for

actual,

of field

from

length
ant

from

These

number

based

specifications

specifications

systems.
The

3.3

specifications

14
that

randomized,

alternated

use

six

No

tasks.

subject

was

message
task

assigned

to

specification

assigned

technology

3.5.1

to

implement

in both
a subject

in which

the

Out-of-order

the

same

technologies.

required

subject

task

use

experiment

a test

first

of the

was trained

The

initial

The

test

same

first.

a module

Although

the maintenance

in random

also

constrained

maintenance
decessor
blocked

task

not

it often

on his/her

of a predecessor

whose

task

a predecessor

next

assigned

were

many

3.6

before

subject
task

start

with

to obtain

tasks

did

not

active

allow

Subjects

to

checked

pertinent

until

envi-

monitoring

logged

than

one

were

outside

to

exit

from

the

ifying

not

related

to the

not

to work

on exper-

environment.

was confirmed

ex-

and

were

never
on

which

and

views,

subjects

the

versations
concerned
and

bug,

work-

3.7

that

passed

so.

In

problems,

problems

with

the ICD

currently

working.

general

technical

their
and

acceptance

such

as a suspected

or the environment,

Ada

an

acceptance
messages

and

of the
test
and

whose
25 that

message

of a task

data

comprised
the

accep-

of scope

the

task

able

the

task

with

MTV

the

design
were

task.

be tested

only

package.

The

to

validate

the

technology.

of one

they

the
support

or suggest

unless

they

an extended
en-

was that

by the subject

to allow

no attempt

were

would

to be done

remedy

person,

and

instruc-

that

either

such

out-of-

the intended

tasks

intended

be

MTV

contact

with

features

be written,

However,

report
further

For

it was

other

MTV-G,

to be completed.

technical

to

subject

provided

would

or the

received

assem-

would

support

with

the

were
ICDS,

specifications

instructed

to be done
that

trials
Force

technical

until

Templates,

to provide

used

Air

message

to supply

of MSL

Success-

the

in either

of existing

task

was

data.

Ada

were

off-

result-

to confirmation

could
an

subject

actions
The

test

solution

to their

For tasks

the

these

terminated

data

amount

package,

subjected

tests

ac-

to actu-

tasks

Subjects
problems

prior

translation,

acceptance

some

for

full-function
complete

the

task

was made

extensions

specifically

ei-

to the basic
requested

by

a subject.

ticipated

was

specification

violated

Com-

its

program

for the experiment

that

to require

completion

recorded.
passed

overnight.

compiling

a variety

As it turned
for

then

test

the data

technology

criteria

criterion

was
same

Out-of-scope

ther

data

into
acces-

a substantial

the

or

of a common

from

As

breaks

to be completed.

predominantly

accep-

directly

could

an Ada

Templates

same

or by the

con-

code.

of these

new templates

inter-

technical

test-

For

provided

of initiating

the

MTV

version

spec-

post-experiment
that

duration,

implementation

acceptable

were

the

reported

completion

normal

the

one another

testing.

for unit

module

to run

lunch

building

tion.

collaboratively

were
it

to
were

pre-loaded

not

when

option

package

suspend

arounds.

Task
The

to work

they

of short

general

compiler

not

been

compile

the

after

scope

in post-experiment

to ask one another

did

were

told

to show

were allowed

questions,

and

solution.

Conformance

data

were

of an MTV-G

completion

out

tasks

had

that

a subject

Ada

against

remedy

a task

They

Ada

bled

environ-

also asked
the

rule

subjects

on tasks

during

Since

interviews.
The

line,

3.8

task

data

Generator

required

anticipated
personal

work

build
allowed

environment

more

was

completion

to the task.

the subject

runs

recorded.

outcomes

MTV

testing

The

the

test

not

capability,

was

in

monitoring

have

requested

They

this

to files
The

was preWhen

the

then

ful

time.

tasks

with

assignments

he/she

accessible

a subject

to perform

iment

There

Test

or accept ante

own

occurred

generating

ing

experiment.

experiment

access

were

periment.

the

task,

the

completion.

at any

ment

of order.

Test

a subject

test.

ally

his/her

test

provided

by

created.

were

in files

of a task

of time

occurred.

of task
of

a new

files remained

the

it out

this

schedule

identifier

ronment

perform

to

the

to subjects.

tests

the

obtained

These

In or-

completion

progress

testing,

ceptance

a subject

awaiting
to

outcomes

prototyping

comple-

subject.

the

Because

was

rules

A randomized
pared

pending

another

allowed

and

a subject

testing

environment

tance

its preworked

in the experiment,

cases in which

Work

that

was blocked

was
task

by

flow

until

subjects

assignment

task

work

Since

been

his/her

pletion

had
unit

sible

was

that

provided

the

to sub-

tasks

of modifications.

happened

next

assigned

of these

be performed

completed.

rates,

assigned

were

ordering

sequence

could

been

der to maintain

of

tasks
the

by the

had

at different
tion

order,

be invoked

as either

and

environment

could

A subject
tance

jects

monitoring

that

identified
ing

performance

harness

25

specifi-

alyzed

the

to find

work-arounds

which

546

fewer

by the OGI

requesting

cation.

out,

out-of-scope
task

research

tasks

The

to complete

requested

than

team

specifications.

modifications.

a subject

MTV-G

extensions

were
had

when

they

subjects
most

There

was

and

received

tasks
a single

deemed
been
first

were

ananable

without
case in

an extension

to the

MSL

specification

language

to

allow

a task

4.1

to

be completed.
Several

tasks

required

templates

in order

Templates.

The

making

the

plates.

to

necessary

to modify

the

were

tasks

subjects

that

could

or modifying

MTV

successful

to the
did

not

or add

with

generally

modifications

the

problems

creating

complete

subjects

However,

scope

the subject

Ada

encounter

be handled

in
tem-

Data

over

a series

analysis

of variance.

pothesis

that

sources

The

experiment

were

Data

extracted

experiment

from

a large

that

were

computer

environment

amount

directly

files

consist

maintained

effort

subjects

allocation

multiple

by the

is being

of

on

tribute

reports,

a weekly

effort

hours

submitted

basis.

These

by

and

at-

was

assessment
filled

out

forms,
by

a subject

each

task,

or at the

still

being

attempted.

tions

that

scale
the

can

In

design

as order

runs,

the

the

the

outcome,

if relevant,

activity.

The
account

Edit

summaries

edit

Subject
each
to

subject.

The

scripts

This

of the

The

was
were

The

taped,
were

in the

outcomes
design
all

design

that
the

con-

relative

tasks.

differences

between

with

re-

a confidence

experiment

to

flexibility

was

meet

hypothesis

varied

specifica-

was that

to at least

specification

as can

demonstrated

Scope:

the

the

MTV

as many

the

MTV

by examining

spectrum

could

Gen-

instances
Templates.

several

charac-

of problem

instances

that

handle;

Extensibility:
handle

detailed,

held

ex-

the

ability

out-of-scope

prepared

Performance:
Ada

In the

prepared
and

System
pilable

5.1.1

with

to respond

been

eliminated

familiarity

to

modify

a system

to

problems;

conclusion

of the

asked

had

answers

activity.

were

order

prior

used

teristics:

to save

a finely

completion

that

interviews

subjects

logs.

at the

gives

of the

solution

RCS

interviews
subject

This

its

a de-

technologies,

confirmed

can be configured

of message

flow.

of editing

After

The

builds,
by the

Other

tions

activities

provide

subjects.

was

been

Flexibility

erator

accessed

technologies

or greater,

Results

sub-

and

have

of 99%

two

the

noted,

differ-

among

factor

individual

5.1

session

activity

files

archive

of questions

in advance.

which

from

summary

individual

a series

the

the

was programmed

perceptions.

periment,

the

work

RCS

session.

time-stamped

at

summaries

extracted

to the

record

to

the

observed

and

were

or

observable

unaccountable

of the

level

experiment

which

of each

of on-line

file

of each

of

of the

compilations,

and

session

environment

images

at

times

type

experiment

the

These
times

transactions,

etc.)

tailed

by

task

otherwise

means

the

Analfactor

experiment,

with

of use of the

An

variability

ported

of this

technologies

Unless

on a

usability

the

complexity

logged

ended,

(edit

was

perception

the

in

particular

tributed

of ques-

a number

subjects
and

environment.

and

if a task

were

be as-

a single
to the

be correlated
differences

other

the unaccount-

distributed.

account

tested.

such

the

into

relationship

the

an analysis

can reasonably

normally

of

by ran-

all conceivable

Then

whose

of the experiment.

completed

a series

with

with

form

environment.

identifier,

occurred

asks

the

task

summaries

began
test

assess

and

monitoring
jects

of a week,
It

of the

technology

Session

end

assessment

as he/she

be answered

of 1-5 to
difficulty

A task

can take

value

for

In

an

of the hy-

of the observable

tested.

and

individual

factors,

to tasks.

values.

are observed

in performance
the

be accounted

variation

are being

could

test

to eliminate

of

using

in the mean

factors

of training
Task

observed

that

that

ences

the

reports

in the

ysis of variance

or

factors
Task

cannot

to be random,

performance

compared

is a st atistical

metric

that

the

were

differences

of systematic

those

sumed

of data

reported

This

it is important

able variations

produced

forms.

compared
of trials

observed

variation

of variance

templates.

analysis

subjects

dom

by

collection

in several

of

directly

than
4

hod

that

a performance

out-ofsimply

Met
Data

tered

tran-

erator

for

Scope
course

scope

547

the
for

Extensibility

of the

experiment,

and nine
The

problem

out-of-scope
MTV

of specifications

ability

problem

and

one out-of-scope

Templates.

analysis.

code

the

subjects

in using
problems

Generator
than

to produce

did

com-

instances;

using

handled
the

encoun-

the MTV

MTV

the

GenMTV

a broader
Templates.

To

handle

were

the

developed

MTV

out-of-scope
for

Templates.

extension

to

compass

the

tension

it

team

For

the

not

MTV

MTV

language

was

problem.

the

designed

expertise

and

to

en-

this

ex-

subjects

of the

implemented

300

them-

OGI

effort
two

reported

by

technologies

each

subject

on equal

in

numbers

lan-

g 200

suc-

:150

MSL

total
of the

250

research

the

the

each

of tasksl.

a minor

sufficient

the

using

and

However,

by

1 shows

extensions

Generator

Generator,

be provided

required

who

the

MSL

problems,

the

out-of-scope

could

selves;

both

guage.
Extension
cessful

of the MTV

in

handling

the

Extensibility

was

gained

ming

ability

duce

new code

instances,

new

accepting
scope

nine

not

:100

program-

them

handled

of

not

of the

within

by

workarounds

One-third
resolved

50

to pro-

in the majority
were

by finding

solutions.
were

Ada

allowed

However,

but

encountered.

the

problems

templates,

partial

problems

that

templates.

problems

was partially

through

subjects

extensibility

writing

the

of the

Templates

or

out-of-

the course

CD
Subject

of

experiment.
Figure

5.1.2

Beginning-to-end

system

of the

12 initial

maintenance
with

specifications

modifications

MTV

were

Templates.

and
the

unusual

typing

strategies

within

resulted

in Ada

code

manifested

error

that

in the Sun Ada

abandonment
Of the
completed

with

G failed

to

within

the
to

the

fully

for

piler

V1. 1 (j)

still

components,
limits

on line

ing for

the

prototyping

exceed

length

and

did

file

these

Ada
the
some

size.

All

solutions

so that

the

impede

lack

The

30

be more

hypothesis,

productive

with

the

for each

technology.

product

required

of each

same,

so the

inversely

effort

proportional

required

The

was modified

development

second

test-

was done

at the

mode

Generator

at the

the

tasks

to simulate

pro-

corresponds

to

tasks.

ex-

In.

*co
:mod

cuwcoco

Effort

will

this

Figure

(log)

the

work

2: Distribution

of effort

hours

per

task.

1The time logged by subjects in support tasks, such as developing or modifying templates to support out-of-scope specifications, investigating
problems with acceptance data, or diagnosing compiler errors, was not counted as effort expended in
performing the assigned tasks.

work

ostensibly

productivity.

Hours

technol-

experiment,

to perform

mode

with

Ada
the

developers

was

axis
scale.

code

acceptance

through

that

In this

to his/her

modes.

is associated

The

tails

a logarithmic

MTV

subject

two

the

horizontal

capacity

ogy than the MTV

Templates,
was tested by measuring the amount
of work completed
per reported
hour
of effort

shows

using

the

com-

generated

improvement

productivity

by

a specification

:0

Productivity

Because

extended,

in

are dis-

success-

Ada

of a generated

progress

2.

hours

problems

code
Sun

Figure

completed

effort

task

periment.
5.2

tasks

problems

compiler-defined

Generator
not

of the

of

of hours

maintenance.

of tasks

reported

are

compressed

number

original

improvement

solved

handle

which

level,

component

three
capacity

However,

cannot

MTV

for

gram

MTV-

chart

distribution

smaller

all were

the

been

in which

However,

produces

instances.

to accomplish

of number
versus

distributions

Each

in the

tasks,

Post-experiment

now

has

sequence.

space

subsequently

generator
all

code

and

generator.

generator

and

Ada

in

of the

Templates

resulting

distributions

technology

played

but

an unrecoverable

Generator.

time

each

expressed

MTV

56 maintenance

MTV

generate
due

the

of the

and

the

task

56

use of legal

VI. 1 (j),

of a complete
12 initial

sequences

compiler,

47 of the

successfully

However,

expended

performance
The

Eleven

1: Effort

was

Figure

548

The
tion

ratio

of the

of the

MTV-G
data

relative
were

timate

the

tors.

probability

collected
tal

rather

elapsed

below

spent

in
two

and

gain

effort

The

three

of

times

MTV

to-

identification

reported

on

MTV

Time

docu-

software
soft-

by the MTV

shown

Templates

by

the

domain,

experiment

software

exhibits

technology
of

In

is

reuse

reuse

not

limitation

depends

opportunities.

was

the

a typical

that

on

human

The

always

obvious

post-experiment

on template

Some
Reported

reuse.

of

or straight-

debriefing,

subjects

reuse:

Time

Time

templates

periment.
Not

On-Line*

3,20

for

task:

Elapsed
in

Editor*

nor

generated

of frequently-repeated

and

Templates

Templates/MTV-G

Elapsed

Ratio

per

is represented

reuse-based

commented
of average

family

Productivity

forward.

MTV

the
that

in productivity

editing,

sheets.

Ratios

is reused),
of the

real.

5.2.1

of the

time

For

modules

(which

integration

con-

reported
in

analysis
and

fac-

time

results

spent

on-line,

the

is 99.570.

the

the

time

ware

to unaccounted

from
shows

artifact,

to es-

difference

working

domain

mentation

These

to use of the
level

subjects

measures:

time

time

than

obtained

table

effort

subject

of the

results

was 2.92.

observed

inal

with

of variance

is attributable
significance

measures
The

the

indica-

observed

an analysis
that

calculated

the

hours.

is a direct

Templates,

to

technologies,

Other
firm

to MTV

on 57 tasks

The

which

improvement

subjected

the means
different

means,

productivity

2.67

2.92

were

I do not

all templates

ing,

and

had

a new

never

know
were

we had

used

what
covered

to discover

application

in the

they

ex-

do.

in the train-

them

when

requiring

the

we
tem-

plate.
* collected

automatically
For reuse

The

small

ment
ity

number

posed

a risk

differences

other

their

measured

to 3:1.

These

vation

differences

but

similar

it

can

be expected
and

include.
to

of software
typically

be

thesis

It

gained

accounts

by

the

to hold

for

sorts

obser-

for only

activity

they

wisdom

do

that

the

alone,

and

experiment

accounts

for

far

It encompasses

conversion

(the

ICD)

a specification,

unit

testing

into

of a solution.

there

productivity

as that

activity

at

that

language.
by

occur

their

inclusion
through

different

for

templates

sites

the same

be-

Distribu-

description
with

non-standard

the
tech-

reimplement

extension.

templates

That

ac-

and inhibits

more

than

of application

MTV

module

sured

the

just

in

this

they

coding.

coding

account

number
given
as much

believed

was correct.

and

the

first

additional

549

of their

own

until

If a module
that

its

reliability
we

tests

to completion

to the subjects

submission,

the

a system,

of acceptance

testing

time

of

in

was necessary,

for acceptance

for orig-

indicator

operating

to pass prior

instructions

requirements

design,

It does not

for

a predictive

has failed

of the total

accounted

plus

engineers

Reliability
As

perform
and

in MTV-

extensions

is no opportunity

that

new

the specifi-

is managed

to accomplish

in

understood.
using

of a language

leaks

file

Enhancements

of the

There

to reExten-

result

and

language

users

process.

proliferates

5.3

productivity

what

7 to 15 per cent

monitored

to all

when

tivity

in a project.

The

as MSL

needs
template

management.

is accomplished

versions

templates

user

solution

as a vehicle.

of reuse

nology

to these

Templates

of enhancements

an accepted

and

file

to be documented

in MTV-G

in updated

for

the
individual

reuse.

improving

in coding

MTV

available

tion

in

all subjects

of these

consider

a scheme

language

come

did

observed

particular

of each

G are expressed

experience.

is conventional

engineers

that

Templates,

purpose

have
the

Reuse

subject,

This

of MTV

files

cation

of up

MTV-G

the same.

the implications
to

that

observed

using

are not

is important

is little

the

training

In evaluating

cost

was nearly
with

their

every

to

template

in

performance

the

must

sions

with

of having

by ratios

For

and

1 shows

differences

between

in productivity

subjects,

do not

the

experi-

matched
Figure

technologies.

Templates

differences

well
differed

with

two

in spite

in individual

strongly

is consistent

data,

were

of productivity

the MTV

with

who

the

be reproducible
that

and experience,

use of the
ratio

would

however,

productivities

correlate

the

the productiv-

individuals

of education

not

in

that

Note,

levels

their

used

we observed

subjects.

selected

of subjects

to the conclusion

member

that

they

a module

that

testing

they

not

confident

its accept ante

failure

score
test

was

as

submit

passed

an acceptance

The
could

of a module

should
were

an

mea-

of the task.

were

and

of

have

it

that

it

test

on

zero.

Each

was run

on the

40

5.4.1

35

On

Perceptions

the

rate

task

from

assessment

1 (easy)

encountered

30

MTV-G

%
20
5

for MTV

and

the

was

2.8.

for

This

problem

tions

test

When
failures.

the

ment?,

added
not

test

run.

the

one to its failure

completed

The

module
Figure

same

for

graph

for

point,

a task

MTV

accept ante

0.8;

for

MTV

firms

the

were

number

defects

with

MTV-G.

5.4

to

tem

test

this

ratio

runs

size.

Sometimes

they

were

tory

workarounds

field

across

appear

MTV-G

tasks

is

caused

These

data

tions.

which

con-

as variable-length

difference

with

a 97%

2.3 in the

the hypothesis

in modules

that

implemented

Usability

programmer

ogy.

For

were

required

tem

problems

MTV-G
Users

perceptions

assessment
cluding

work

forms

were

captured

completed

on each

task

both

immediately
and

in

the

after

task
con-

sues.

in the post-experiment

dle system

debriefings.

and

2In [8] it is suggested that a better estimate of significance

of a test for difference of means of Poisson-distributed
factors is
obtained if the data are preconditioned
by taking a square root.
This removes the functional relationship
between the mean and
the variance, if the distribution
is indeed Poisson. When this
was done, the difference in the means remained significant, at a
!3Syo
confidence ievel.

flexibility

5.4.2
The
ing
sults.

550

subjects
two
Their

numeric
issues

also

implementa-

representations

variant

record

such

types

with

were

subjects

feedback

limits

spec-

about

productivity
of their

confirmed
some

is-

must

han-

the portability

produced.

gives

systypes.

domain

programmer

program

perceptions

data

non-domain-related

it necessarily

technologies

programmers

to express

into

the dotechnol-

lower-level

of necessary

an application

y of the

from

domain

concerned

Perceptions

the

satisfac-

a large

are isolated

delving

details,

sometimes

the use of the MTV-G

creation

the

Whenever

and
partially

Templates

Templates,
be

and

without

hard

anticipated

templates.

MTV

allowed

ifications

MTV

with
to

adjust

an

boundary

of limitations

main

the

Bit/byte

sysrun-

encountered
to

as splitting

for new data

new

kinds

only

MTV

and

subjects
needed

seek

and

of the
involved

difficulties,

successful,

fields

using

These

ation

were

within

experi-

the MTVno difference

subjects

exceeded

to

Requirements

handled

mean

the

that

fields.

problems

identified

The

such

two

is 1.8.

of nearly

they

required

the

technologies.

they

data

accep-

accommo-

in

of flexibility

by

allowing

commenced.

perceived

two

where

handle

of variance,

confirms

are introduced

or

fourth

the

discovery.

before

the

the

ESC,

used

subjects

ex-

specifica-

better

ICDS

represent

problems

one outlying

for
it

problem

limits

of

the

the

generate

experiment

limitations

data

by
to

the

biased

problem

technology

range

the

show

graphs

of

frame-

generated

had

time

the

aa identified

limits,

scaling

(The

before

while

15 failures
These

tasks
of

The

test

technology.

part

volunteered

provided

of the four

In general,

distributions.

to an analysis

of failed

not

were

passed. )

Templates

levelz.

fewer

does

of failures

significance

confidence

applied

of acceptance

in each

there

number

subjected

was

the

post-experiment

had

The

enough

between

Templates

time

Templates

Poisson

mean

on a mod-

an acceptance

data

distribution

for which
tests

Work

passed

of test

performed

to approximate

were

the

tasks

it

run.

3 shows

failures

The

suite

on each

score.

until

on

in the

as a result

Which

G technology,

was

wsa 1.9

implementations

subjects

personnel

been

entire

three

that

of MTV-G.

data

in flexibility
module

to

average

in the MTV-G

four

and

only

asked,

dated

ule

The

a perception
flexibility

OGI

actually

personnel

tance

of failures

that

in favor

had

OGI

8910

test

Templates

of the

instances

periment

of acceptance

asked

of difficulty

implementations

to note

two

misconception

3: Distribution

MTV

indicates

It is interesting

Figure

were
level

a specification.

of greater

debriefings,

Number

the

work.

S15
z
10

01234567

subjects

Generator

average

the subjects

H MTV Templates

forms,

to 5 (difficult)

in modifying

difficulty

2
~ 25

of flexibility

productivity
the
clues

in us-

measured

re-

as to the

rea-

sons

for

Their

their

better

average

all tasks,
formed

from

to

subjects

while
rate

ing on this

the

task,

it

to

average

was

statistically

3.5.

agree)

was

All

4.2

significant

asked

While

to determine

three

with

of these

parts

Temwere

level

not

two

of the

During

the post-experiment

confirmed

more

that

productive

using

maintenance

how

nologies,

of at least

the

plates

stated,

a GUI

are

Ada

the

code;

tion
When

only

like

which
me

code.

the

during

field

widths

that

sub-

isting

template

from

instantiation.

say

5.5

the

using

areas

have

been

Confidence

When

the

forded

them

a given

subjects

of the

the code

directly

misgivings

widely

to

and

in assembly
then

domain-specific

accepted,

the

need

Average

correct

to see the

hours

in the

Ada

desire

Std.

code

both

problems

Also,

one subject
or edit

cues

produced

were

an ex-

gone

definitive

advantage

with

SDRR

after

realized

the

technology

MTV
in the

a lower

rate

developers.

in a direct

to be the best
software
the

results

for

and

applications

past,

to compiled

of

These

comparison

currently

available

components.

quantitative

The

results

of

generated

MTV

Ratio

test

from

8.17

4.2

13.5

O.8

1.8

y:

number
runs

2.92

2.80

task

deviation

Avg.

similar

of

2.25

failed

lanMTV-G

to see the compiler

become

effort
per

Reliability

to check

transition

languages

errors

for

Terndates
Productivity:

Confidence

wanted

of

Templates,

MTV-G

lack

languages

types

problems

results

the
dur-

In the

the

textual

summarizes

con-

about
is the

the

indi-

greater

times

personal

work

analysis:

several

code,

during

the

of

of their

in the
technolo-

difference.

positions.

developing

table

sessions

produce

the

common

Programmers
As

Ada

was compelling.

were

programming
output.

the

generator

moderate,

surfaced

for

following

af-

equally

complaint

debriefing

to examine

only

guages.

that

felt

is believed

technology

correctness

expressed

other
A

solutions

technology

the

four

the

their

which
in

of the
and

post-experiment

of ability
ability

asked

technologies.

technology

ing the

was

three

both

in

confidence

in MTV-G

in

MTV-G

were

greater

solution,

confidence
fident

of subjects

what

Ada

Interestingly,
the two

improvement
from

impeded
Templates,

a difference

MTV

has

defects

easMSL

use of the

to reconstruct

the

of productivity

which

the

editing

a pronounced

advantages
with

5.4.3

bit

constructed

delivered

generation.

and

experiment

Generator

to improve

the

was
the

MTV

between

it was difficult

demonstrating

and

perceive

for

of

it

of problems.

noted

since

Subjectively,

that

the

characterize

Summary
This

descripapproach.

code

and

the
loca-

poor,

measurements

to

was

however,

data.

was a significant

subjects

noted

at

productivity

Ada

with

good

not

to
easy

MTV-G

In particular,
For

insertion

was

ease of error

stated

were

were

the

asked
the

this

errors.

did

It
for

about

with

forms

was 3.0.

subjects

tracing

there

Tem-

would
look

was

that

technologies,

This

offered

although

When

tech-

One

to

disagree)

debriefing,

messages

also

assessment

rating

in MTV-G.

of error

both

MTV-G

improve

suggestion

performance

end

errors

allowed

made,

benefit

make

Ada

how to further

the

generator

the

and

MTV

would

dont

initial

the use of

the

be expected

statement,

Templates

four

messages

specifically

gains.

front

and

remarkably

asked

MTV-G,

for

Solution

go generate

sounds

more

productivity

a text-based
types,

cated

identified

interface

(... Template

or even

these

for

gies,

were

with

subjects

user

as important

for

asked

be improved

of the four

sophisticated

sub-

they

the

consistent

subjects

task

to questions

error

error

all four

that

both

When

could

three

a more

believed

MTV-G,

tasks.

productivity

debriefing,

they

can

1 (strongly

average

MTV

of syntactic

likelihood

jects

ject

were

only

debugger

99%.

for

tion

the

the

post-experiment

location

MTV

with

answers

compilers

MTV-G,

in

(from

The

the

ier to locate

differences

at a confidence

for

During

to

asked

errors.

and

subjects

work-

which
For

while

com-

A greater

were

be completed.

3.5,

was 4.2,

was 3.7.

correctness

agreement
agree))

to locate

was

were

were

their

5 (strongly

with
it

MTV-G

subjects

to rate

agreement

a task

for

Subjects

with

task,

the statement,

it was easy

remained

of

when

with

their
this

response

observed

response

plates

parts
response

per-

performed

affirm

diminish.

over

tasks

5 (strongly
on

to subjectively

MTV-G.

15 for

to rate

to

which

agreement

of a task

those

Templates

was

their

37 for

working

average

the MTV

difference

was

asked

While

determine

plete,

using

difficulty

1-100,

disagree)

statement,

easy

and
When

1 (strongly

the

from

MTV-G

Templates.

in

of implementation

on a scale
with

MTV

performance

rating

dling

more

inal

output

551

also

a greater
design

than

exhibited
range
the

superior

of specifications
MTV

Templates.

flexibility

in

han-

within

the

orig-

This

result

was

achieved

despite

already

been

the fact

used

Furthermore,
its

first

MTV-GS
tions,

serious

The
of its

that

with

of the

favored

message

in

1977.

Charles

Plinta}

was

[9]

specifica-

substantially

it or any

experiment

software

MTV-G

although

man.

alleviated

from

similar

about

tion

had

had

the

solution

the

most

advantages

[10] Dennis

important

ware

of generating

promising,

new

tional

that

55-60.

technology

Volpano

A.

A.

Porter

detection

quirements

inspections:

IEEE

it y and

productivity

of

software

replicated

C. Briand,
impact

y in

Communications

Basili.

Formal

re-

In

Conference
IEEE

Engineer-

and Walc61io
of reuse

on

object-oriented

the

ACM,

39

qual-

systems.
(to

appear),

1996.

[3] Don

Batory,

Jeff

Thomas.

ceedings

Vivek

Singhai,

Scalable

of

ACM

Foundations

Marty

software

SIGSOFT

of Sofiware

Sirkin,

libraries.
93f

and

In Pro-

Symposium

Engineering,

on

December

1993.

[4] Jeffrey
and

Bell

et al.

reuse:

November

[5] Pacific

Proceedings,

final

February

for

reliability

demonstration.
pages

In

396-404.

ACM,

Research

Center.

scientific

and

SDRR

project

technical

report,

1995.

[6] Walter

Ellis.

[7] Martin

private

Griss

kits

In

of

Proceedings
Conference,

communication,

and

domain-specific
tions

design

1994.

Software

Phase

Software

A proof-of-concept

94

TRI-Ada

CMU/SEIEngineering

University,

Richard

B.

December

Kieburtz.

Proceedings

on Software

Computer

Waldinger
Towards

Etghth
Engineering,

Society,

Kevin

Wentzel.

for a flexible
1994

ACM

SA C94,

June,

software
Software

pages

47-52,

1993.
Hybrid

factory.
ApplicaMarch

1994.

552

and
kinder,

August

Michael
gentler

of the 1994

Methods.

September

experi-

1995.

Lionel
the

R.

on Soflware

June

Measuring

V.

for

Transactions

R. Basili,

Melo.

and

methods

21(6):563575,

[2] Victor

Risstransla-

SoftInternapages
1985.

components.

Comparing

ing,

Wiley

Michael

Report
Software

Mellon

and

templates.

In Proceedings

ment.

and

John

C31 message

Technical

Carnegie

II.

the Physt-

1989.

References
Votta

Lee,
for

and Ex-

and

volume

12 ESD-89-TR-20,

PHION:

G.

Edition,

and validation.

[11] Richard

[1]L.

Statistics

in Engineering

Kenneth

A model

Institute,

expe-

It has demonstrated

a highly
software

perceptions

no prior

2nd

F. C. Leone.

technology.

specifications.

is indeed

developing

in their

they

has confirmed

hypotheses

SDRR

has been

cal Sciences,
& Sons,

no errors

encountered

and

Design

in

89-TR-

subjects

The

robustness

encountered
large

N. L. Johnson
perimental

version.

usability,

rience

[8]

had

system.

problem

to handle

a problem

in a new

for

subjects

most

inability

Templates

demonstrated

the

The

the MTV

in a deployed

MTV-G

trial

MTV-G.

that

1994.

U.S.

Naval

Monterey

Lowry.
formal

AMmethods.

Workshop

Postgraduate

on

School,