2/5/2014

RESEARCH METHODS IN CS
Atul Gupta

Software Development
 A multi-disciplinary field
 We need to investigate
 Which tools, techniques, and processes
they use?,
 What social and cognitive processes
surrounding them?
 How individual software engineer
develop software?
 How teams and organizations
coordinate their efforts?

8/23/2017

1
 2/5/2014

Evaluations in Software
 We need to know two things:
1. What assessment techniques are available to us?
2. Which should be used in a given situation?

8/23/2017

Evaluations in Software
 The main objective of the ESE course
 General Objectives
 What is Research?
 How do we do it?

 Specific Objectives
 What is Experimental Research?
 W5H2 about Experimental Research

 How to argue and convince others?

 Statistical Methods

2
 2/5/2014

Evaluations in CS
 Theoretical/Analytical
 Proof oriented; proving properties of abstract artifacts
 Solving mathematical problems like those based on
graphical models, probability, coding theory,
 Empirical
 Evidences
oriented, extrapolating from observed
phenomenon

Empirical Research Methods

 Controlled Experiments (including Quasi-
Experiments)
 Case Studies (both exploratory and confirmatory)
 Survey Research
 Ethnographies
 Action Research

3
 2/5/2014

Empirical Evaluations
Quantitative Qualitative
Objective Subjective
Research questions: How many? Strength of Research questions: What? Why?
association? Is better?
"Hard" science "Soft" science
Test theory Develops theory
Measurable (in numeric terms) Interpretive
Report statistical analysis. Report rich narrative, individual; interpretation.
Subjects Participants
Establishes relationships, causation Describes meaning, discovery
Generalizations leading to prediction, Patterns and theories developed for understanding
explanation, and understanding
Highly controlled setting: experimental setting Flexible approach: natural setting (process oriented)
(outcome oriented)
Sample size: n Sample size is not a concern; seeks "informal rich" sample

"Counts the bugs" Provides information as to "which bugs are worth

counting"

Empirical Investigations
 Common tasks
 Identify goals, questions and measures (metrics)
 Choosing research method(s)

 Planning, designing, and carry out investigations

 Data collection,

 Data Analysis

 Results

 Validity considerations

 Conclusions

4
 2/5/2014

Two Motivating Examples

 To assess the
effectiveness of a novel
fisheye-view file
navigator
 To assess how developers
in industry use (or not use)
UML diagrams during
software design

5
 2/5/2014

Research Questions
Kind of research questions

 RQ. “Is a fisheye-view file  RQ. “how widely are

navigator more efficient UML diagrams used as
than the traditional view
for file navigation?” collaborative shared
 Ambiguous !
artefacts during
 Who?
design?”
 File navigation?  Ambiguous !
 Circumstances?
 Efficiency?

Defining the precise meaning of terms is Crucial !

Formulating Exploratory Questions

 Existence Questions
 Ex: “Does X exists?”
 Is file navigation something that users actually do?”
 “Do collaborative shared artefacts actually exist?”
 Classification Questions
 Ex: “What is X like?”
 How can we measure efficiency for file navigation?”
 What are all the types of collaborative shared artefacts?”
 Descriptive-Comparative Questions
 Ex: “How does X differ from Y?”
 How do fisheye views differ from conventional views?”
 “How do UML diagrams differ from other representations of design information?”

6
 2/5/2014

Formulating Base-rate Questions

 Frequency and distribution questions
 Ex:
“How often does X occur?”, “What is an average
amount of X?”
 Descriptive-Process questions
 Ex: “How does X normally work?”
 Relationship questions
 Ex:“Are X and Y related?”
 Correlation and Causality

Formulating Base-rate Questions

 Causality Relationship questions
 Causality questions -“Does X cause Y?”
 Causality-Comparative questions - “Does X cause more
Y than does Z?”
 Causality-Comparative Interaction “Does X or Z cause
more Y under one condition but not others?”

7
 2/5/2014

Slide adapted from 2014 Steve Easterbrook CSC2130: Empirical Research Methods for Computer Scientists

Slide adapted from 2014 Steve Easterbrook CSC2130: Empirical Research Methods for Computer Scientists

8
 2/5/2014

Slide adapted from 2014 Steve Easterbrook CSC2130: Empirical Research Methods for Computer Scientists

Slide adapted from 2014 Steve Easterbrook CSC2130: Empirical Research Methods for Computer Scientists

9
 2/5/2014

Choosing a Research Method

 Depends
 available resources,
 access to subjects,

 opportunity to control the variables of interest,

 the skills of the researcher

 how closely the method aligns with the question(s) that

have been posed

10
 2/5/2014

Controlled Experiments
 Investigation of a testable hypothesis suggesting
causal relationships
 Identification of study variables and instruments
 Independent , dependent, block variables
 Subjects, objects

 Design of experiments
 More of a quantitative than qualitative method
 Theory driven
 Validity consideration

Controlled Experiment
 Strengths
 Theory driven
 Exercise maximum degree of control over study
variables
 Repeatable

 Better generalization is possible

 A tool for SE education

11
 2/5/2014

Controlled Experiments
 Weaknesses or Limitations
 Small Scope (duration, size)
 Underlying assumptions may be wrong

 Tends to be artificial

Case Studies
 “An empirical inquiry that investigates a
contemporary phenomenon within its real-life
context, especially when the boundaries between
phenomenon and context are not clearly evident.”
[Yin 2002]
 Exploratory vs. confirmatory
 Selection of a case is a crucial step
 Purposive sampling vs. random sampling
 Quantitative as well as qualitative

12
 2/5/2014

Case Studies
 Strengths
 Offer in-depth understanding of a phenomenon in real
situations
 Situations where effects are expected to be wide
ranging, or take a long time (e.g. weeks, months, years)
to appear (large scope)
 situations where the context is expected to play a role
in the phenomena

Case Studies
 Weaknesses
 the data collection and analysis is more open to
interpretation and researcher bias
 Limited generalization

13
 2/5/2014

Surveys
 Characteristics of a broad population of individuals
 Use of interviews/questionnaires for data collection

 Selection of a representative sample from a well-

defined population
 More of a qualitative than quantitative

Surveys
 Strengths
 Smallto large scope
 Generalizable

14
 2/5/2014

Surveys
 Weaknesses
 control for sampling bias
 Low response rate can be problematic

 Inappropriate survey questions

 Difficult to interpret or less insightful

Ethenographics
 Field studies
 Study a community of people
 focus on a broad technical community (e.g. Java
programmers in general), or a small, closely knit
community (e.g. a single development team)
 Researcher as an observer or a participant
observer
 The biggest challenge is to perform detailed
observation, data collection, and analysis

15
 2/5/2014

Action Research
 Solving a real-world problem while simultaneously
studying it
 A long drawn method
 An iterative approach to problem solving
 Immaturity as an empirical method (tends to be ad
hoc)
 An appealing framework for mixing research with
professional activities

Mixed-Mode Approaches
 All methods have limitations
 Weaknesses of one method can be compensated for by
the strengths of other methods
 Key decisions involve
 the strategy for data collection
 the sequence in which different methods are employed
 quantitative and qualitative data
 Three general approaches
 Sequential explanatory strategy
 Sequential exploratory strategy
 Concurrent triangulation strategy

16
 2/5/2014

Data Collection Techniques

 Separate lecture

Empirical Validity
 Construct validity:
 the “right” (relevant and minimal) metrics, design
 “Are we actually measuring what we intend to measure?”
 Internal validity:
 the “right” data values.
 “Does the data really follows from the experimental concepts?”
 Conclusion validity:
 the right (appropriate) data analysis?
 External validity:
 the “right” (representative) context
 “Can the results of the experiment be generalized?”

17
 2/5/2014

Threats to validity
Examples:
 Violated assumptions of statistical tests
 Researchers may influence the results by looking for a
specific outcome.
 If the group is very heterogeneous there is a risk that the
variation due to individual differences is larger than due to
the treatment.
 Incorrect design of the Experiment
 Effect of confounding factors
 Research bias
 …

Other Dangers
 Manipulating several explanatory variables
simultaneously so the effect on the response variable
become difficult to be interpreted
 Over-complex design
 Carryover effects or learning
 Insufficient subjects (lack of power)
 Unrealistic task/environment
 Unrepresentative subjects - usually students

18
 2/5/2014

Confounding Effects in SE Experiment

 Technique Learning Effect
 Object Learning Effect
 Boredom Effect
 Enthusiasm Effect
 Experience Effect
 Procedure Effect

Practical Considerations
 Relate to time, budget and personnel resources, and
access to data
 Methods that are primarily qualitative
 Designing good research question
 Researching skills – Observation, recording social
behavior, patience,
 Methods that are primarily quantitative
 Require more significant time in the planning
 Pilot studies
 Researching Skills – critique, management, innovation

19
 2/5/2014

Conclusions
 An overview of research methods
 Each has its own strengths and weakness
 Use of mixing-methods
 Validity considerations

References
 Selecting Empirical Methods for Software
Engineering Research, Steve Easterbrook,
Janice Singer, Margaret-Anne Storey,
Daniela Damian, Book Chapter in Guide to
Advanced Empirical Software Engineering, Forest
Shull, Janice Singer, and Dag I.K. Sjøberg, Springer
2008

20