Data collection instruments

Dr Azleena Mohamad
Signpost
Questionnaires
Field notes
Educational records
Quantitative research methods
Attitude scales
Digital research tools
Triangulation
What is a
Questionnaire?
A set of questions
designed to generate the
statistical information
from a specific
demographic needed to
accomplish the research
objectives
 How questions are specified and put
together will influence the respondents
willingness to participate & the
responses they provide
 Purposes of the Questionnaire
Ensures standardisation and comparability of
the data across interviews everyone is asked
the same questions
Increases speed and accuracy of recording
Facilitates data processing
Allows the researcher to collect the relevant
information necessary to address the
management decision problem
Improper design can
lead to
Incomplete information
Inaccurate data
Higher costs
 Criteria to consider
Does it provide the necessary
information?
Does it consider the respondent?
Does it meet editing, coding and
data processing requirements?
 Designing the
Questionnaire
Determine objectives: Plan what to
measure.
Decide on format. E.g. personal interview,
telephone, self.
Formulate questions to obtain the needed
information
Decide on the wording of questions
Decide on the question sequence and
layout of the questionnaire
Using a sample, test the questionnaire for
omissions and ambiguity
Correct the problems (pretest again, if
necessary)
 The Major Decisions in
Questionnaire Design
1. Content - What should be asked?
2. Wording - How should each
question be phrased?
3. Sequence - In what order should
the questions be presented?
4. Layout - What layout will best serve
the research objectives?

The most difficult step is specifying

exactly what information is to be
Content: Questions must meet 5 requirements
1. Are the questions relevant? Do
they pertain to the research
problem?
2. Are the questions accurate? Do
they accurately depict the
attitudes, behaviors, etc. intended
to investigate?
3. Do respondents have the
necessary information?
Qualify respondents
4. Do respondents understand and
interpret the question correctly?
5. Will respondents give the
How should the questions be asked?
Format: How much freedom do we give
respondents in answering questions

Open-ended questions
What do you look for most in a job?
Is there anything else you would like to add about
the product?
 Open Ended Questions:
Advantages Disadvantages
Wide range of responses and Ability and/or willingness of
information can be obtained respondent to answer

Answers based on respondents not Interviewers ability to record

researchers frame of reference answers quickly or summarize
consumers terms accurately & probe effectively

Lack of influence. Don't channel Interviewers attitude influences

respondents thinking response

Can help interpret closed-ended Time consuming (interview sessions,

questions - why tabulation, classification,
assignment, validation)

Particularly useful as introduction to Difficulty in coding

survey or topic

When its important to measure the Require respondents to be articulate

salience of an issue

Respondents may miss important

points

Non-response
Closed-ended questions
(Fixed-alternative responses)
What do you look for most in a job?
____ Work that pays well
____ Work that gives a sense of
accomplishment
____ Work where you make most decisions
by yourself
____ Work that is steady with little chance
of being laid
off.
Closed ended questions
Advantages
Ease of understanding
Requires less effort on part of
interviewer and respondent
Ease of tabulation & analysis
Less error prone
Less interviewer bias
Less time consuming
Answers directly comparable
Disadvantages
Middle/Neutral categories
often selected inappropriately
(ignorance, safety)
Less opportunity for self-
expression or subtle
qualifications
Less involving for respondents
Order of response categories
can have major impact on
results
Key tradeoff

Want to get respondent to address

issues our research is concerned
with (Forced response) and at
same time give respondent
opportunity to honestly opt out of
question
(i.e., Dont Know, No Answer,
Neither Agree nor Disagree) so as
not to dilute data collected
 Dichotomous Questions
Should the government give
consumers an energy rebate?
1.Agree
2.Disagree

Advantages Disadvantages
Easy to administer and Prone to large amounts
tabulate of error since polarized
responses prevent
gaining information on
the range of variation

Fail to communicate any

intensity of feeling
 Multiple Choice Questions
Are all possible alternatives included?
Too many alternatives
Position Bias

Scaled Response Questions

Closed ended questions where the response
choices are designed to capture an intensity of
feeling (Likert, Staple, Semantic differential)
Easy to code and more powerful statistical tools
Main problem: Respondent misunderstanding
Question Wording
Can have major impact on how
respondent interprets question
All respondents should interpret
in the same way
Things to avoid
Complexity: use simple, direct, conversational
language
leading questions -- that suggest or imply certain
answers
loaded questions -- suggest social desirability, or
are emotionally charged. Have you purchased a
high quality Sony TV this year
Ambiguity and vagueness: Words such as often,
occasionally, usually, regularly,
frequently, many, should be used with
caution. If these words have to be used, their
meaning should be explained properly.
Which district is bigger? Seremban or Nilai? Would
your answer be based on population or area?
 More things to avoid
long-worded questions

double-barreled questions.
Questions that refer to two or more
issues within the same question.
Where respondent may agree with
only 1 part of multipart statement.
Do you think Nike offers better
pricing and variety than other
brands?
Making implicit assumptions
 More things to avoid
burdensome questions - that may
tax the respondents memory
How many tubes of toothpaste
have you purchased in the last 3
months?
Have you purchased toothpaste
in the past week?
Embarrassing, sensitive, or
threatening questions:
Have you charged more on your
credit card than you should?
Question Sequence & Layout Decisions
Initial stages
Screening or qualifying questions:
Have you been to the movie theatre this
month?
Need to gain & maintain respondents
cooperation
Make questionnaire simple for
interviewer to administer
Opening questions should be interesting,
simple, and easy to answer.
Which theatre did you go to last?
Transition Questions
What aspects of the theatre did you like
best?
Questions directly related to research
objectives which require more effort and
get respondent thinking about topic
Difficult and Complicated Question
The following 10 questions relate to the
characteristics of theatres
Respondent now committed and can see
an end in sight
Classifying and Demographic Questions
What is your average annual household
income?
May not be answered (sensitive or
threatening) but most questions have
been answered
 Other Considerations
Questions should flow logically from one to
the next
General questions should be asked before
more specific ones
Earlier questions should not influence
response to later ones
Questions should flow from factual and
behavioural questions to attitudinal and
opinion questions
Questions should flow from the least
sensitive to the most sensitive.
Questions should flow from unaided (which
brand do you prefer) to aided questions (which
brand do you prefer Tide, Cheer, Gain)
 Demographic questions should come at
the end.
1. initial questions - screening and rapport
questions
2. Middle - product specific questions
3. End demographic questions.
 Other Considerations
Use multiple questions instead of one
Similar questions together
consistent mindset for
respondents
Develop a logical flow
Use transitions between sections (E.g.
In this section we ask questions about
X)
Distinguish between instructions,
questions and responses
Distinguishing question and responses:
CAPS or BOLD or Underline versus lower case or unformatted

INSTRUCTIONS LAYOUT A:
Do you agree, disagree or have no opinion that this
company has:
A good vacation policy - disagree/not sure/agree.
Good management feedback - disagree/not sure/agree.
Good medical insurance - disagree/not sure/agree.

INSTRUCTIONS LAYOUT B:
Does this company have:
___ Disagree Not Sure Agree
A good vacation policy 1 2 3
Good management feedback 1 2 3
Good medical insurance 1 2 3
Filter and pivot questions should
be used as necessary.

A FILTER question is one that

screens out respondents who are
not qualified to answer a second
question.
A PIVOT question is a type of
filter question that is used to
determine what version of a
second question to ask.
 Developing a logical flow
If the questionnaire deals with several topics,
complete questions on a single topic before
moving on to a new topic
If topics are related, ask questions on related
topics before asking questions about unrelated
topics
If you ask questions about behaviors over some
time period, follow chronological order
backward in time
When changing topics, use some transitional
phrase
 Questionnaire Layout
The layout and physical attractiveness
of a questionnaire are important
aspects
Questionnaires should be designed
to appear as short as possible
Questionnaires should not appear
overcrowded
Leave lots of space for open ended
questions
Questionnaires in booklet form are
often recommended
Opening
Provide first name (at least)
Provide name of organisation doing
research
Provide reason for survey and topic
(objective)
State that no selling will be involved and
no personal data other than for
statistical purposes
Tell respondent approximate time to
complete
Reinforce that respondents time is
Closing
Thank for time
Ask if they had a positive
experience and remind them that
their opinions count
 Pretesting and Correcting Problems
Purpose of pretest: To ensure that
the questionnaire meets the
expectations in terms of the
information that will be obtained
Is question necessary does it serve
a purpose, will info be used?
Missing important variables does it
provide the info needed?
Match questions to objectives
 Pretest Specific Questions For
Variation
Meaning
Task difficulty
Respondent interest and attention
Ambiguous, ill-defined, loaded, double-barreled
questions
Pretest the Questionnaire
Flow of the questionnaire
Skip patterns
Length
Respondent Interest and Attention
Questionnaire Design Flow Chart
Step 1: Specify what information will be sought
Step 2: Determine type of questionnaire and method of
administration
Step 3: Determine the content of individual questions
Step 4: Determine form of response for each question
Step 5: Determine wording for each question
Step 6: Determine sequence of questions
Step 7: Determine physical characteristics of
questionnaire
Step 8: Revise steps 1-7, revise if necessary
Step 9: Pretest questionnaire, revise if necessary
Step 10: implement
A QUESTIONNAIRE
IS ONLY AS GOOD
AS THE QUESTIONS
IT ASKS
Field notes
Records of what the observer has seen or heard
Contain literal descriptions as well as personal
reactions and comments on what the observer has
experienced and thought during an observation
session
May be guided by a protocol developed before the
observation session
May take the form of anecdotal records compiled as
part of a more systematic authentic assessment or
portfolio effort
What you write depends on what you are looking for.
Positioning
Components of Effective Observation
Degrees of Participation
Participant Becomes a part of, and a participant
Observer in the situation under observation
Engage in activities
Observe activities, people &
physical aspects
Privileged A teachers aide during specialists
Observer time
Passive Observer Watches but does not participate in
the situation while observing it
Present only to observe whats
going on
How to write field notes
(Wolcott, 1994)
Components of Effective Observation
Field notes
Observe and Attune to what you actually
record everything record through verbatim
conversations, maps and
illustrations, photos, and video
and audio recordings
Observe and look Try to see beyond the routine
for nothing and look with a fresh
perspective
Look for What are the unintended
paradoxes consequences of action?
The dont
Do not remove yourself from the site when
observation is supposed to take place
Educational records
(Mills, 2014)
Archival documents
Journals
Maps
Video and audio recordings
Artifacts
Archival documents
Sources that teacher researcher can access:
Student records
Minutes of meetings (faculty, PTA, school board)
Newspaper clippings about significant events in
the community
Students work
Journals/ Diaries/ Logs
Students journals provide researcher with a
valuable window into the students world ( as how
homework assignments provide parents with
insights of their childrens daily experiences)
Can also use a daily journal to keep a narrative
account of their perspectives of what is happening
in their classrooms
Maps
Extremely helpful for teacher researchers in trying
to monitor movements in a classroom data that
are not easily recorded in a narrative form
Video and audio recordings
Provide teacher researchers with another data
source when the teacher is fully engaged in
teaching but still wants to capture:
Classroom events
Interactions
Artifacts
Classrooms rich sources
Written or visual sources of data that contribute to
our understanding of what is happening in our
classrooms and schools
Online focus group
(Stewart & William, 2005)
Synchronous (real-time) and asynchronous (non
real-time)
The interaction among participants will likely to
yield the best information when
They are similar and cooperative with each other
Time to collect information is limited
Participant may be hesitant to provide information
on one-to-one basis
(Kruger, 1994; Morgan, 1988, Stewart & Shamdasani, 1990)
 Advantages & Disadvantages

Advantages
Participants can be questioned over long periods of
time
Larger number can be managed
More heated and open exchanges occur
Disadvantages
Obtaining complete informed consent
Recruiting individuals to participate
Choosing time to convene given different
international time zones
Photo elicitation
(Pink, 2001; Clandinin, 2006)
Participants are shown pictures (their own/ those
taken by researcher)
Participants are asked the content of the pictures
(Denzin & Lincoln, 1994)
Quantitative Data
Collection Tools
Teacher-made tests
Standardised tests
School-generated report cards
Attitude scales
Likert scales
Semantic differentials
Teacher-made tests

Used to aid teachers in their ability to monitor and

adjust instruction (Creswell, 2014)
Should not rely solely on the unit tests provided
by textbook companies to determine whether
students have achieved mastery of specific goals
and objectives
Teachers are the experts, thus must make their
own test from scratch
Data from this instrument provides accessible
information about how well their students are
responding to a particular teaching or curriculum
innovation
Standardised tests
Intend to provide teachers, principals, parents and
other stakeholders with individual student
achievement data
Provide teachers with a snapshot of how their
students are performing on a given subject test
relative to all other students taking the test
Can be a source that contributes to our
understanding of how teaching practices affect
our students
School-Generated Report Cards
Valuable data source for
teachers (who can quantify student achievement)
Students and parents (who can interpret the data
and set goals accordingly)
Attitude Scales
Many teacher researchers are curious about the
impact of their work on students attitudes
Scales used to measure attitudes, such as Likert
scales and semantic differentials
Allows teacher researchers to determine what an
individual believes, perceives, or feels (Gay, Mills
& Airasian, 2012, p.150)
Likert scale
Ask students to respond to a series of statements
indicating whether they:
Strongly agree (SA)
Agree (A)
Undecided (U)
Disagree (D)
Strongly disagree (SD)

Each point corresponds to a point value and a score is

determined by adding the point values for each
statement
A high point value on a positively stated item would
indicate a positive attitude and a high total score on the
test would be indicative of a positive attitude (Gay,
Mills & Airasian, 2012, p. 130)
Semantic differential

Asks a student (or parent) to give a quantitative

rating to the subject of the rating scale on a
number of bipolar adjectives
Boring __ __ __ __ __ __ __ Exciting
-3 -2 -1 0 1 2 3
Each location on the continuum between the
bipolar words has an associated score
Totaling the score teacher researcher can
determine whether a childs attitude is positive or
negative
Usually have five to seven intervals, with a neutral
attitude being assigned a value of zero
Digital research tools
Web-Based survey tools
SurveyMonkey.com
Zoomerang
LimeSurvey
eSurveyspro
Triangulation
The desire to use multiple sources of data
Researchers should not rely on any single source of
data, interview, observation or instrument
Action researchers complete a triangulation matrix
a simple grid that shows the various data sources that
will be used to answer each research question (Sagor,
2000, p. 19-20)
Pelto and Pelto (1978, p. 122) called this as the multi-
instrument approach
Wolcott (1988, p. 192) suggests that the teacher is the
research instrument who, in collecting data, utilises a
variety of techniques over an extended period of time
Can involve both qualitative and quantitative
approaches
Triangulation Matrix
Exercise
Research Data Source
Questions 1 2 3

1.
2.
3.
4.
5.
Example
Research Data Source
Questions 1 2 3

1. Student Survey Computer

Preexisting Knowledge
knowledge? Pretest
2. Pretest Posttest Teacher help
Keyboarding
speed?
3. Pretest Posttest Timed typing
Appropriate software software teacher
use? constructed
4. Time on School lab Student survey Parent survey
computers? records
Any questions?