Educational Design Research:

an Introduction

Tjeerd Plomp
Professor Emeritus
University of Twente
Enschede, The Netherlands
1
Design-Based Research Collective (2003):
educational research is often divorced from the
problems and issues of everyday practice – a split
that resulted in a credibility gap and creates a
need for new research approaches that speak
directly to problems of practice and that lead to
the development of ‘usable knowledge’
(authors from domain of Learning Sciences)

2
Van den Akker (1999; director of Netherlands
Institute for Curriculum Development)
about research in domain of curriculum
development and implementation:
many ‘traditional’ research approaches such as
experiments, surveys, correlational analyses,
with their emphasis on description hardly
provide prescriptions that are useful for design
and development problems in education

3
Reeves (2006) for the domain of educational
technology:
there is “a legacy of ill-conceived and poorly
conducted research that results in no significant
differences or, at best, in modest effect sizes”

4
Quotes illustrate that
many researchers in various ‘corners’
of our field belief that there is a need
for an alternative research approach
that is directly relevant for problems in
educational practice.

5
 In this presentation:
• Focus of research, research functions and
approaches
• What is design research
• Research question and other outputs
• Generalizibility in design research
• How to conduct DR – formative evaluation
• Design research dilemmas
• Concluding remarks
6
 Focus of research project

Dependent on research question/goal

research may address a need
– to describe
– to explain, understand, predict
– to compare
– to evaluate
– to design/develop
Called: research functions
7
 NOTE:
One resq => more than one res function
Example:
if aim of research is
to design and develop a teaching-learning strategy
for acquiring the competency of mathematical
modelling (in grade 11 & 12),
then researchers may first want to understand and
carefully describe what barriers students
experience with mathematical modelling.
But: there is primary research function
8
 Research designs
to realize research functions e.g.:
• Survey: to describe, compare, evaluate
• Case studies: to describe, to explain
• Experiments: to explain, to compare
• Ethnography: to describe, to understand
• Correlational research: e.g. to describe, to
explain
• Evaluation research: e.g. to determine the
effectiveness of a program
9
 Research approaches, e.g.:
Design research:
to design/develop an intervention (e.g. program, T/L
strategy, process, product, system, etc) with the purpose
– to improve practice and contribute to body of
knowledge
OR
– to develop a theory
(and possibly: to inform decision making and policy
development)

Note: intervention is container term

10
 Research approaches, e.g.:

So, when conducting research:

Research problem, question or goal
=> Decide on primary and other research
functions
=> Choose research approach(es)

11
 What is Educational Design Research
(EDR)?
Distinction between:
• Development studies aimed at research-based
solutions for complex problems in educational
practice
• Validation studies aimed at development or
validation of a theory
Research on interventions versus research
through interventions
(McKenney & Reeves, 2012)
12
 What is Educational Design Research
(EDR)?
Educational design research – type development
studies - is
the systematic study of designing, developing
and evaluating educational interventions - as
solutions for problems for which no ‘how-to-
do’ guidelines exist - aimed at
(i) optimal interventions AND
(ii) theories (design principles) on ‘how’ and ‘why’
these interventions work
13
 What is Educational Design Research
(EDR)?
Educational design research – type validation
studies - is
the systematic study of designing, developing
and evaluating educational interventions –
e.g. learning environments – with the purpose
to develop and validate
theories about learning, learning
environments (and the like) OR to validate
design principles.
14
 When is Design Research
Appropriate?
Eamonn Kelly (2009):
• Initial state(s) unknown or unclear
• Goal state(s) unknown or unclear
• Operators to move from initial states to goal
states are unknown or how to apply the
operators is unclear

15
 Why embark on Design Research?
Various reasons – e.g.:
1. Need to solve a complex problem in
education practice
2. Need to elaborate and/or validate a
theory
(e.g. whether the theory of Realistic
Mathematics Education is valid in a
certain context)
16
 Problems may exist at various levels - for
example:
System level:
Need for system for e-learning to serve specific
group of students

Institutional level:
What are effective methods for collaborative
learning

17
Educational Design Research encompasses
systematic educational design processes.

The reverse is not true:

Not all systematic educational design can
be called research

18
Iterations of systematic design
cycles

design & develop

analysis prototype

evaluation
Revision: yes?
No? STOP
19
 Design research methodology can be
expressed as follows:

ID of problems Development of
with practitioners prototype solutions

Revision

Reflection to produce Testing solutions

design principles in context

(adapted from Reeves, UoGeorgia, Athens, USA)

20
Susan McKenney, Univ of Twente:

21
 Phases in Design Research
• Preliminary phase:
needs and content analysis, review of state-of-art of
literature => conceptualisation & design specifications
• Development or Prototyping phase
(iterative design phase + formative evaluation)
<=> micro-cycles of research
• Assessment phase (semi-summative evaluation)
ALL Phases:
systematic reflection and documentation (resulting in
design principles)
22
 Generic Design Research Model

Practitioner and User Participation

Refinement of Problem,
Solution, and Method

Tentative
Consult Formative
Product Practical
Evaluation
Researchers Experts & Products/
Approaches
Practitioners Results
Other Analyze Focused
Practitioners Redesign & Refinement
Sources Practical Literature
Context Review of Products & Theories

Collaborative Analyze Tentative Contribution

Promising Design to
Reflection
Examples Principles Theoretical

Problem in Context
Preliminary Investigation
of Problem, Context,
& Approaches Refinement of
Design Theory

Expert and Researcher Participation

Phases
Identification of Tentative Tentative Products Prototyping & Assessment of Problem Resolution
Problem Identification & Theories Preliminary Products & Theories & Advancing Theory
Products & Design Principles

23
Revision 5, Dated: 05-20-05 based on Tom Reeves second response on 05/15/05 regarding the “initial identification of the problem” and Wademan additional iterative processing enhancements.
 On terminology:
(Educational) Design Research
stands for a ‘family’ of related research approaches
• Design studies, design experiments
• Design-based (implementation) research
• Development research
• Participative action research
• Formative research
• Engineering research
(e.g. Van den Akker et al., 2006):
But a number of characteristics in common!
24
 Common characteristics

• Interventionist
• Iterative
• Process oriented
• Utility oriented: merit of design
measured
• Theory oriented: based upon &
contributing to
• Involvement of practitioners

25
 A closer look:

If research is aimed at understanding or

knowing
• what would this mean for design research?
• what do we aim to understand, to know?
KEY: what are the research goals?

26
Context Z:

Intervention X Outcomes
Input => Process Y1, Y2, …, Yn

design principles or intervention theory

or local instruction theory or design theory
27
Context Z:

Intervention X Outcomes
Input => Process Y1, Y2, …, Yn

Research goal – Development:

(i) developing a research based intervention
as solution to complex problem
and
(ii) constructing (re-usable) design principles
Development research
28
Context Z:

Intervention X Outcomes
Input => Process Y1, Y2, …, Yn

Research goal – Theory:

(i) designing learning environments
with the purpose
(ii) to develop and validate theories
about learning, learning environments, etc
Validation research
29
Context Z:

Intervention X Outcomes
Input => Process Y1, Y2, …, Yn

Research goal – Implementation or upscaling:

(i) Implementing a particular program
and
(ii) a strategy and conditions under which
this can happen (design principles)
‘Implementation’ research
30
Context Z:

Intervention X Outcomes
Input => Process Y1, Y2, …, Yn

In general:
Research goal may vary, but always:
(i) design/development of intervention
and
(ii) design principles or theory validation
31
Context Z:

Intervention X Outcomes
Input => Process Y1, Y2, …, Yn

Intervention ‘theory’
Local instruction theory
Intervention ‘hypothesis’
Design principle(s)

32
 Research question in design research
If research goal is development of intervention:
What are the characteristics of an <intervention X>
to realize outcomes Y in context Z
OR in case of developing/validating a theory
(tentatively):
What is an adequate learning and teaching
strategy for <topic B> in context C in order to cope
with the main difficulties in the learning of this
topic
33
 Examples of research questions

#1 what are the characteristics of an effective

in-service programme for mathematics
teachers through which they develop the
ability to apply student-centred pedagogical
methods (Tecle, 2003)

34
 Examples of research questions
#2 What is an adequate learning and teaching
strategy for genetics in upper secondary
biology education in order to cope with the
main difficulties in learning and teaching
genetics, and
to promote the acquisition of a meaningful
and coherent understanding of hereditary
phenomena? ( Knippels, 2002)
35
 Outputs of Design Research

• Design principles or validated theory

• Interventions (programs, products,
processes)
 Initiated to meet a need felt in a complex,
practical situation
 Design Research by definition practically
relevant
• Professional development 36
 Outputs of Design Research (DR)

Professional development:
Characteristic is collaboration of researchers
and practitioners
 DR as form of professional development
 practitioners develop awareness of how
research can contribute to improving
professional context
37
 Design principles
(in development studies)

Heuristic statements like:

If you want to design intervention X,
for the purpose of Y in context Z,
then you are best advised to give that intervention
the characteristics A, B, and C (substantive
emphasis),
and to do that via procedures K, L, and M (procedural
emphasis )
because of arguments P, Q, and R.”
(Van den Akker, 1999)
38
 Design principles
• substantive: knowledge incorporated in
interventions
• procedural: what activities/steps to be
taken
• theoretical arguments providing directions
• empirical evidence about impact of
principles
BUT: they are heuristics
39
 Design principles
• Heuristic: refers to experience-based
techniques for problem solving, learning,
and discovery.
• however: no guarantee for success in
other contexts, because developed in
certain context!
• But: will be increasingly powerful when
validated in various contexts.

40
 Design principles
• Heuristics – no guarantee for success in
other contexts, because developed in
certain context!
• But: will be increasingly powerful when
validated in various contexts.
QUESTION: can design principles or local
(instruction) theories be generalized from
one context to another
• This question of generalization similar as in
experimental research or case studies
41
 Generalizability in Design Research:
Yin (2003) about case study research:
striving to generalize findings to some broader theory:

Context Z:
Intervention X Outcomes
Y1, Y2, …, Yn
Input => Process

design principles as intervention theory

local instruction theory
42
 Generalizability in Design Research:
A theory (design principles or local theories) must be
tested through replications of findings in a 2nd, 3rd or
more cases:
if same results in various cases, then ….
the theory (= design principles or local theories)
might be accepted for a much larger number of
contexts.
This replication logic is the same that underlies design
research!
Analytical generalization.
(compare with ‘transferability’ in qualitative research)
43
 Generalizability in Design Research:
So: design principles must be seen
as ‘heuristic’ statements providing guidance and
direction,
but do not give ‘certainties’.

When we give proper weight to local

conditions, any generalization is a working
hypothesis, not a conclusion!
(Lee Cronbach, 1975, cited by Tom Reeves)

44
 Part 2:
How to conduct Design Research

- Role of formative evalution

- Formulating research questions per
iterations/phases
- Quality criteria and other attributes
- Aspects of the research design per
iteration/phase
- Design research dilemmas
- Concluding remarks
45
 Guiding principles
for scientific research
• Pose significant questions that can be investigated
• Link research to relevant theory
• Use methods that permit direct investigation of the
question
• Provide a coherent and explicit chain of reasoning
• Replicate and generalize across studies
• Disclose research to encourage professional scrutiny
and critique
(Shavelson & Towne, 2002)

46
 How to conduct Design Reserach
Tenets:
• Rigor of research (see next slide)
• Relevance for practice
– Research team has good working knowledge of
target setting
• Collaboration with professionals from target
setting
• Iterations - emphasis on formative evaluation
to refine intervention and design theory.
47
 Formative evaluation
= Evaluation to improve
Aimed at:
(i) Improving prototype towards intended
outcome
(ii) Sharpening underlying tentative design
principles (towards elaborated set)

48
 Quality criteria for interventions:
1. Relevance or Content Validity:
• need for the intervention
• based on contemporary scientific insights
2. Consistency or Construct Validity
2. Practicality:
- can be used in settings for which developed
3. Effectiveness:
- yields desired results
- cost-benefit ratio
49
 Shift in emphasis on criteria
Stage Criteria
Preliminary Emphasis mainly on relevance
and consistency (i.e. validity),
research
less on practicality
Development initially: consistency and
or practicality;
Prototyping increasingly practicality and
gradually effectiveness
phase
Assessment practicality and effectiveness
phase 50
 Development stages of a prototype:
- Design specifications: first and general
description of intervention
- Global intervention: some or all components are
given some detail – horizontal prototype
- Part of intervention in detail- for use by target
group – vertical prototype
- Complete intervention: whole intervention
sufficiently detailed for use in intended target
setting
51
 Formulating research questions
for various iterations/phases of project:
The syntax of these research questions is:
'What is the [quality criterium a, b, c and/or d] of
[element of] the intervention that is in
[development stage w, x, y, z].‘
Example:
What is the relevance [quality criterion] of the
content [conceptual element] of a quick reference
manual for using Chinese characters that is at a
global stage of devt [development stage] ?
52
 Formative evaluation is part of
process of doing research:

Each iteration is a ‘micro-cycle of research’

=> therefore each iteration/version/prototype:
- its evaluation/research question(s)
- its evaluation/research design
=> address all components of a research design:
‘what’, ‘why’, ‘who’, ‘where’, ‘how’ and ‘when’
53
 Many possible methods of
formative evaluation:
• screening or self-evaluation
• expert apppraisal / focus group
(nb: ‘experts in what?’)
• walk through or 1-to-1 evaluation (with
representative of target group)
• micro evaluation in small group of target
users
• try-out and/or field test
Example from Nieveen (1999, 2007) 54
 And evaluation activities
Evaluation method Possible activities for the gathering of information

Screening  Using a checklist

Focus group  Interviewing
Walkthrough  Using a checklist
 Interviewing
 Observing
Micro-evaluation  Observing
 Interviewing
 Administering a questionnaire
 Testing or requesting a learning report
Try-out  Observing
 Interviewing
 Administering a questionnaire
 Testing or requesting a learning report
 Requesting logbooks 55
56
 Layers of formative evaluation
Early stages informal evaluation:
self-evaluation or screening, walk through or
1-to-1 evaluation, expert review
then small group evaluation
aimed at practicality and effectiveness
to full field test – can become separate effect
study
BUT: each cycle is a piece of research!
57
 Preliminary paper comp based final version
comp based based versions
version version
users experts users experts users users users
(n=5) (n=3) (n=5) (n=6) (n=4) (n=4) (n=17)

Vali-
dity √ ea √ ea
Practi-
cality √ wt √ wt √ ea √ me √ to √ ft
Effective
ness √ to √ ft
ea: expert appraisal to: tryout
wt: walk through ft: field trial
me: micro- evaluation
58
 Formative evaluation e.g.:
Design Global Partly Completely
specs design developed developed
Relevance • Screening • Screening • Screening • Focus group
• Focus group • Focus group • Focus group • Micro-
evaluatie
Consistenc • Screening • Screening • Screening • Focus group
y • Focus group • Focus group • Focus group • Micro-eva

Practica- • Screening • Screening • Walkthrough • Walkthrough

lity • Focus group • Focus group • Focus group • Focus group
• Micro-eva • Micro-eva
• Try-out
Effective- • Screening • Screening • Focus group • Focus group
ness • Focus group • Focus group • Micro-eva • Micro-eva
• Try-out
59
 Formative evaluation e.g.:
Design Global Partly Completely
specs design developed developed
Relevance • Screening • Screening • Screening • Focus group
• Focus grp • Focus group • Focus group • Micro-eva

Consistency • Screening • Screening • Screening • Focus group

• Focus grp • Focus grp • Focus group • Micro-eva

Practica- • Screening • Screening • Walkthrgh • Walkthrgh

lity • Focus group • Focus grp • Focus grp • Focus grp
• Micro-eva • Micro-eva
• Try-out

Effective- • Screening • Screening • Focus grp • Focus grp

ness • Focus group • Focus group • Micro-eva • Micro-eva
• Try-out60
 Example: T/L strategy for science education

Tryout in
3 classrooms Appraisel by
Appraisel by 3 experts
3 experts

Version I Version II Version III Version IV

Tryout with
Design guidelines & tchr educ studs
specifications
61
 Formative evaluation e.g.:

Design Global Partly Completely

specs design developed developed
Practicality:
expected •Screening •Screening •Walkthrough •Walkthrough
•Focus •Focus group •Focus group •Focus group
group
actual •Micro-eva •Micro-eva
•Try-out
Effective- •Screening •Screening •Focus group •Focus group
ness: •Focus •Focus group
expected group

actual Micro-eva •Micro-eva

•Try-out
62
63
Example: Cycles as micro-cycles of research

64
65
66
 Design research dilemmas
(see e.g. van den Akker et al., 2006)

1. Researcher in multiple roles:

also designer, evaluator, implementer
• Be open for professional scrutiny
• From creative designer towards critical
researcher
• Quality research design with strong ‘chain of
reasoning’ (Krathwohl, 1998)

67
 Design research dilemmas
1. Researcher in multiple roles (contd):
• Quality research design:
– Triangulation: of data source, instruments
– Attention for validity and reliability of
instruments
– Empirical testing
– Systematic documentation and reflection
• Many tactics:
- member checks
- multiple observers
etc 68
 Design research dilemmas

2. real-world settings bring real-world

complications
• Researcher as a ‘cultural stranger’ in target
setting
• Pay attention to gaining trust
• Invest in understanding target setting
• Involvement of professionals in multiple
roles
69
 Concluding remarks:

Design research can be relevant for

educational practice and policy because it
may result in:
- Design principles / validated theory
- Interventions as a solution to a need that is
experienced
- Involvement of professionals

70
 Concluding remarks:

For design research to really have impact in

practice and to develop design principles or
theories that are generalizable we need:
• Research programs (go beyond single
projects)
• ‘communities of researchers AND
professionals’

71
 Why ‘design research’ and
not ‘action research’?
Both
• Real-world problems
• Aim at improving practice
• Cyclical in nature
• Participative
BUT:
design research aims to contribute to theory -
has focus on producing generalizable design
principles
72
Important sources:

Van den Akker, J.,

Gravemeijer, K, McKenney,
S. & Nieveen, N. (Eds).
(2006). Educational design
research.
London: Routledge. ISBN10:
0-415-39635-2 (pbk)

73
 An Introduction to
Educational Design
Research
Tjeerd Plomp & Nienke
Nieveen (Eds)
Published by: SLO –
Netherlands Institute for
Curriculum Development.
Enschede, 2009 (126 pages)
Pdf file freely available via:
http://www.slo.nl/organisatie/int
ernational/publications/

October 2013 revised edition: http://international.slo.nl/edr 74

Published by: SLO –
Netherlands Institute for
Curriculum Development.
Enschede

Part B: already available

http://international.slo.nl/
edr

Part A: will be released

mid-October 2013

75
 End of part 1

Thank you!

(t.plomp@utwente.nl)

Part 2:
How to conduct Design Research
76