A Design Research Methodology

(overview)
Research
Clarification

DESCRIPTIVE STUDY I

PRESCRIPTIVE STUDY

DESCRIPTIVE STUDY II

Lucienne Blessing
3TU PhD Event
June 2010

Status of design research

It is no simple matter to define the contents, the research
approach and the community behind research in
engineering design. (Cantamessa)

It is not easy to see the trends of evolution, to identify
landmarks of development, to judge the scientific
significance of the various approaches, and to decide on the
target fields for investments. (Horvath)

3TU PhD Event June 2010

Lucienne Blessing

Status of design research

Rapidly growing but:

Many loosely coupled, ill-defined strands of research;

No overview;

Referencing islands;

No agreed terminology;

Many different topics and approaches;

Lack of consistency in approach

Still pre-theoretical

Transfer into industry poor

No established methodologie(s)

3TU PhD Event June 2010

Lucienne Blessing

Research clarification: status

Research is often not clarified:

Aims and objectives often fuzzy or too high level

Stated aims are hardly used as performance metrics

Industrial success criteria often not appropriate

No established metrics exist

3TU PhD Event June 2010

Lucienne Blessing

Developing understanding: status

Most rapidly growing research area, which has become
an integral part, but

Variety of aims and approaches

Publications often insufficiently specified

Flavours of the year in research methods

Correlations between pairs of factors

Few links with success

Assumptions, paradigm, worldview unclear
Of theand
111methodology
papers describing empirical studies:

Little reflection on methods
10% did not state the research approach,
22% did not give the unit of analysis,
41% did not declare the sample size
25% did not present the implications of the
findings (Cantamessa on ICED proceedings)
3TU PhD Event June 2010

Lucienne Blessing

Developing support: status

Developing support has a long tradition and is still a
dominant theme, but

Little evidence of extensive use of empirical data or link with
industry, although this is improving substantially

Importance of empirical data for the development of support
often underestimated

Motivation, underlying assumptions and view on design not
articulated

Alternative solutions rare

Implementation issues usually absent

No approach to develop demonstrators

Little reflection on methodology

3TU PhD Event June 2010

Lucienne Blessing

Evaluating support: status

Evaluation of the proposed support is often poor

Usually not properly evaluated

Focus on functionality, not on implementation, training,
organisational or other issues
Focus on complete support rather than core concept
Conclusions go beyond what evaluation method and findings allow
Little evidence that aims have been realised

3TU PhD Event June 2010

Lucienne Blessing

Issues that need addressing

The lack of overview of research results

The lack of use of results in practice

The lack of scientific rigour

Systematic development and validation of knowledge

Established research methodology

3TU PhD Event June 2010

Lucienne Blessing

You must learn from the mistakes of others.

You can't possibly live long enough to make
them all yourself.
Sam Levenson, teacher and comedian

3TU PhD Event June 2010

Lucienne Blessing

What is design research?

product
tools & methods

Understanding

organisation

micro-economy

process
Support

macro-economy

people

Improving design (product and process)

3TU PhD Event June 2010

Lucienne Blessing

10

Aims of design research

Aims:

the formulation and validation of models and theories about
the phenomenon of design, as well as

the development and validation of knowledge, methods and
tools - founded on these models and theories

to improve the design process (i.e. support industry producing
successful products).
It is the methodology that makes a topic of investigation scientific.
Design research must be scientific in order for the results to have
validity in some generic, practical sense. For this, design research has
to develop and validate knowledge systematically. This requires a
research methodology.

3TU PhD Event June 2010

Lucienne Blessing

11

Research process in social sciences

(Frankfort-Nachmias)

Problem
Generalization

Hypotheses

Theory
Research design

Data analysis

Data collection
3TU PhD Event June 2010

Measurement

Lucienne Blessing

12

The real way of doing things?

3TU PhD Event June 2010

Lucienne Blessing

13

Characteristics of design research

Not only aims at increasing our understanding about design

(design-as-is), but also at the use of this understanding to
change the actual situation (design-as-should-be).

This requires two theories/models: about the existing situation

and about the desired situation.

Design research includes theory/model development and

support development (to get from the existing to the desired
situation)

Challenge:

Designing is a complex activity: many different influencing

variables that are strongly interrelated, and both the influencing
factors and links between these are dynamic.

3TU PhD Event June 2010

Lucienne Blessing

14

DRM: A Design Research Methodology

Basic method

Stages

Literature &
Analysis

Research
Clarification

Empirical data &

Analysis

DESCRIPTIVE STUDY I

Assumption &
Experience &
Synthesis

Empirical data &

Analysis

3TU PhD Event June 2010

PRESCRIPTIVE STUDY

DESCRIPTIVE STUDY II

Lucienne Blessing

Results

Goals

Understanding

Support

Evaluation

15

Example project

Goal:

Improve the reliability of products to improve product
performance (increase sales).

Assumptions:

Improved reliability improves product performance

Improved product performance increases sales

Timely assessment of reliability increases the changes of
improving product reliability

Available support is ineffective

Research questions:

Are the assumptions correct?

How to improve?

3TU PhD Event June 2010

Lucienne Blessing

16

Research Clarification stage (RC)

The role of the Research Clarification Stage is:

to identify the goals the research is expected to realise; the focus; the
main research problems, questions and hypotheses; the relevant
disciplines and areas to be reviewed and those to contribute:
to develop an initial representation of the existing and of the desired
situation, the Initial Reference and Impact Models

Make explicit current understanding and beliefs

Determine

the initial set of influencing factors thought to be relevant

those factors that may be suitable success criteria;

the believed links between the factors in the existing as well as
the desired situation, in particular those linked to the success
criteria;

3TU PhD Event June 2010

Lucienne Blessing

17

Example: initial reference and

impact models
Amount of
profit

Market share

_
+ [A]

Maintenance
cost

[A]
_

+
_ [A]
_

Company
image
Amount of
profit

Market share

_ + [A]

[A]
+

Product
reliability

[A]

3TU PhD Event June 2010

+ + [A]+
Support

Company
image

Maintenance
cost

[A]

Product
reliability
18

Research Clarification stage (RC)

The role of the Research Clarification Stage is:

to identify the goals the research is expected to realise; the focus; the
main research problems, questions and hypotheses; the relevant
disciplines and areas to be reviewed and those to contribute:
to develop Initial Reference and Impact Models, i.e., an initial picture of
the existing and of the desired situation;
to provide a focus for the DS-I stage in finding the factors that
contribute to, hinder or prohibit success;

Overall research plan + preliminary understanding

(Initial Reference and Impact Models)

3TU PhD Event June 2010

Lucienne Blessing

19

DRM: A Design Research Methodology

Basic method

Stages

Literature &
Analysis

Research
Clarification

Empirical data &

Analysis

DESCRIPTIVE STUDY I

Assumption &
Experience &
Synthesis

Empirical data &

Analysis

3TU PhD Event June 2010

PRESCRIPTIVE STUDY

DESCRIPTIVE STUDY II

Lucienne Blessing

Results

Goals

Understanding

Support

Evaluation

20

Descriptive Study I Stage (DS-I)

The role of the Descriptive Study I stage is:

to obtain a better understanding of the existing situation by

identifying and clarifying in more detail the factors that influence the
preliminary Criteria and the way in which these factors influence the
Criteria;

Methods:

Literature review of empirical studies

Empirical Studies

3TU PhD Event June 2010

Lucienne Blessing

21

Reading literature

3TU PhD Event June 2010

Lucienne Blessing

22

Empirical studies
Some essentials:

Plan, prepare, prepare, prepare

Formulate

Know your paradigm and your methods

Ensure that

Research questions: clear, unspecific, answerable, value-free

Hypotheses: clear, specific, testable, value-free

the aim,
the research questions and/or hypotheses,
the type of data collected,
the way it is collected, processed and analysed,
the interpretations and conclusions

all match.
Get advice and try out
Focus, focus, focus

3TU PhD Event June 2010

Lucienne Blessing

23

Descriptive Study I Stage (DS-I)

The role of the Descriptive Study I stage is:

to obtain a better understanding of the existing situation by identifying

and clarifying in more detail the factors that influence the preliminary
Criteria and the way in which these factors influence the Criteria;
to complete the Reference Model including the Success Criteria and
Measurable Success Criteria;
to suggest the factors (possible Key Factors) that might be suitable
to address in the PS stage, as these are likely to lead to an
improvement of the existing situation;

3TU PhD Event June 2010

Lucienne Blessing

24

Reference Model
Success Criterion

_
Market share

Other factors

Measurable
Criterion
Quality of
production

[1]

Price

+
0
[2]+ [A]_
[1]
+
Customer
satisfaction
_
_ [1]

+
[4]
+
Warranty
cost
[3,4]

[1]
+

Product
reliability

+
Operating
cost

+
+ [3]

+
Maintenance
cost

Quality of
product
+

[5]

[3]
_

Key Factor

Other factors
_
Productspecificity of
DfR- methods
3TU PhD Event June 2010

[A]

0[E]

+ +
Use of
DfR-methods

Lucienne Blessing

% of
project time left
to improve

25

Descriptive Study I Stage (DS-I)

The role of the Descriptive Study I stage is:

to obtain a better understanding of the existing situation by identifying

and clarifying in more detail the factors that influence the preliminary
Criteria and the way in which these factors influence the Criteria;
to complete the Reference Model including the Success Criteria and
Measurable Success Criteria;
to suggest the factors (possible Key Factors) that might be suitable to
address in the PS stage, as these are likely to lead to an
improvement of the existing situation;
to provide a basis for the PS stage to systematically develop
support that addresses the factors with the strongest influence on
success;
to provide a reference for the evaluation of the developed support in
the DS-II stage.

Reference Model, Success and Measurable Criteria, Key Factors,

Initial Impact model, implications for support development
3TU PhD Event June 2010

Lucienne Blessing

26

DRM: A Design Research Methodology

Basic method

Stages

Literature &
Analysis

Research
Clarification

Empirical data &

Analysis

DESCRIPTIVE STUDY I

Assumption &
Experience &
Synthesis

Empirical data &

Analysis

3TU PhD Event June 2010

PRESCRIPTIVE STUDY

DESCRIPTIVE STUDY II

Lucienne Blessing

Results

Goals

Understanding

Support

Evaluation

27

Prescriptive Study
The role of the Prescriptive Study stage is:

to use the understanding from DS-I or DS-II to determine the factors to

be addressed (the Key Factors) to improve the existing situation;
to develop an Impact Model describing the desired, improved
situation expected as a consequence of addressing the selected Key
Factors;
to select the part of the Impact Model to address and to determine the
related Success and Measurable Success Criteria;

3TU PhD Event June 2010

Lucienne Blessing

28

Impact Model
preliminary
Success
Factor

market share

price

+[2]
+

other factors

customer
satisfaction

warranty
cost

[1]

++

product
reliability

other factors

use of
DfR methods
3TU PhD Event June 2010

_
_

maintenance
cost

quality of
product
quality of
production

operating
cost

_
_

+
+

preliminary Measurable
Succsss Factor

_
_

preliminary
Key Factor

+
% of
project time left
to improve

Lucienne Blessing

assess
reliability
in early design
stages
29

Prescriptive Study
The role of the Prescriptive Study stage is:

to use the understanding from DS-I or DS-II to determine the factors to

be addressed in PS (the Key Factors) to improve the existing situation;
to develop an Impact Model describing the desired, improved situation
expected as a consequence of addressing the selected Key Factors;
to select the part of the Impact Model to address and to determine the
related Success and Measurable Success Criteria;
to develop the Intended Support and to realise this in the Actual
Support to such a level of detail that an evaluation of its effects can take
place against the Measurable Success Criteria;

3TU PhD Event June 2010

Lucienne Blessing

30

Actual support vs Intended support

functionality of
Actual Support
core
contribution
functionality of
Intended Support

3TU PhD Event June 2010

Lucienne Blessing

31

Prescriptive Study
The role of the Prescriptive Study stage is:

to use the understanding from DS-I or DS-II to determine the factors to

be addressed (the Key Factors) to improve the existing situation;
to develop an Impact Model describing the desired, improved situation
expected as a consequence of addressing the selected Key Factors;
to select the part of the Impact Model to address and to determine the
related Success and Measurable Success Criteria;
to develop the Intended Support and to realise this to such a level of
detail that an evaluation of its effects can take place against the
Measurable Success Criteria;
to evaluate the Actual Support with respect to its in-built functionality,
consistency, etc., the Support Evaluation;
to develop an Outline Evaluation Plan DS-II.

Impact model, Actual Support, Intended Support

Support evaluation
3TU PhD Event June 2010

Lucienne Blessing

32

DRM: A Design Research Methodology

Basic method

Stages

Literature &
Analysis

Research
Clarification

Empirical data &

Analysis

DESCRIPTIVE STUDY I

Assumption &
Experience &
Synthesis

Empirical data &

Analysis

3TU PhD Event June 2010

PRESCRIPTIVE STUDY

DESCRIPTIVE STUDY II

Lucienne Blessing

Results

Goals

Understanding

Support

Evaluation

33

Descriptive Study II
The role of Descriptive Study II is to identify:

whether the support can be used in the situation for which it is

intended and whether it does address the factors it is supposed
to addressed (Application Evaluation = usability)
To identify whether the support contributes to success,
addressing usefulness, implications and side effects (Success
Evaluation = usefulness)

Application Evaluation
Success Evaluation
(Model evaluations)

3TU PhD Event June 2010

Lucienne Blessing

34

Evaluation
preliminary
Success
Factor

market share

price

+[2]
+

other factors

customer
satisfaction

warranty
cost

product
reliability

other factors

use of
DfR methods
3TU PhD Event June 2010

Application evaluation

[1]

++

Success evaluation

_
_

maintenance
cost

quality of
product
quality of
production

operating
cost

_
_

+
+

preliminary Measurable
Succsss Factor

_
_

preliminary
Key Factor
Support evaluation

+
% of
project time left
to improve

Lucienne Blessing

assess
reliability
in early design
stages
35

Keypoints
DRM emphasises

the need for formulating success as well as measurable criteria;

the importance of descriptive studies to increase our
understanding of design products and processes to inform the
development of design support;
the systematic development of design support;
the different types of evaluation necessary to assess the
developed support
the different possible types of research projects

3TU PhD Event June 2010

Lucienne Blessing

36

Variations, iterations and entry points

Basic method

Stages

Literature &
Analysis

Research
Clarification

Empirical data &

Analysis

DESCRIPTIVE STUDY I

Assumption &
Experience &
Synthesis

Empirical data &

Analysis

3TU PhD Event June 2010

PRESCRIPTIVE STUDY

DESCRIPTIVE STUDY II

Lucienne Blessing

Results
Results

Goals

Understanding

Support

Evaluation

37

As to DRM:

Those are my principles,

and if you dont like them,
..... well, I have others.
Groucho Marx

3TU PhD Event June 2010

Lucienne Blessing

38