Scaling, Reliability & Validity

CHAPTER 7
SCALING,
RELIABILITY &
VALIDITY
April 2017 Halilah Haron 1
Learning Objectives
Understand the levels of measurement.

Understand rating scales.
Understand ranking scales.
Goodness of measure.

RELIABILITY &
VALIDITY
SCALING
LEVEL OF
MESUREMENT

Levels of Measurement
• Categorical in nature
Nominal
• Categorical & Sequential in nature

Ordinal
• No absolute zero
Interval
• With absolute zero

Ratio

Data- Example
Gender
• Female
• Male

Data- Example
Ethnic Group
• Bumiputera
 Malay
 Other Bumiputera
• Chinese
• Indians
• Others

Data- Example
Marital status
• Divorced/Permanently separated
• Widowed
• Married
• Never married

Data - Example

Data- Example
Type of
houses
• Bungalow
• Condominium
• Double-terrace house Bungalow
Condominium
• Flat
• Kampong house
Double-terrace house Kampong house
Flat

Nominal Data
Characteristics
 Non-number / Qualitative
 Categorical

Exercise
Give some examples of nominal data.
Program (Statistics,
Job (professional, computer science, actuary,
mathematics)
Currency (pound, Us Dollar,

Subjects (research Euro, Ringgit Malaysia,
methodology, Baht, Rupiah, Riyal, Peso,
regression, etc) Dirham, Won. Yen, etc.)

Data - Example
Educational qualification
 Secondary
 Certificate
 Diploma
 Degree
 Masters
 Ph.D
 Professional qualification
Data - Example

Data - Example
Rate the importance of these residential environment
features when buying a house at retirement age.
Level of importance
Features 1 2 3 4 5
CONVENIENCE
AMENITY
HEALTH
SAFETY
COMMUNITY
OTHERS
1 - Not important at all

2 - Not important
3 - Uncertain
4 - Important
5 - Very important

Data - Example
The following are seven cell phone features. Please rank any
five according to your preference. A ‘1’ represents ‘Most
preferred’ while a ‘5’ represents ‘Least preferred’
Voice ________
SMS ________
Graphics ________
Downloadables ________
Browser ________
Camera ________
Geopositioning ________

Ordinal Data /Ranked Data
Characteristics
 Categorical
 Has sequence
 Statistical measures: median and
other nonparametric methods

Exercise
Give some examples of ordinal data.
Stage (Stage 1, Stage 2, Job qualification

Stage 3, Stage 4) ( managers, executive,
supervisors, clerks)
Cast (Brahmin, Satria, … ) BMI Level (underweight,

normal, overweight, obese)

Interval Data - Example
Age Group
 0 – 14
 15 – 24
 25 – 44
 45 – 64
 65 and over
Source: Department of Statistics , Malaysia

When responses to various items that measure a
variable on any form of Likert scale are summed across
the items, an interval scale is obtained as illustrated
below.
PERCEIVED LOYALTY
VALUE

Interval Data

WEALTH = Money ?
WEALTH = Money +
Material Possession +
Education + Children
+ Family + Friends +
Health + Relationship
+ Inspiration +
Spiritual
Halilah Haron
April 2017 21
Interval Data-Example

To summarize …
Interval Data
 In terms of numbers
 No absolute zero (subjective)
 Statistical measures: mean,
variance, std deviation,
nonparametric methods
Exercise
Give some examples of interval data
Intention to leave
Customer satisfaction
Tahap kefahaman Tahap kesedaran

Ratio Data - Example
Absolute
zero

China
Makkah
al
Mukarramah
World’s Tallest Building

Absolute zero

WEALTH = Amount
of Money
Absolute
zero
Halilah Haron
April 2017 27
Ratio -Example

Ratio - Example

Ratio Data- Example

Example
Some variables can be presented in several
formats to achieve good response in specific
situations, such as age. Age can be represented
using different formats such as the following

Format A : (As a ratio scale)
Age (as of 01.01.14) : _______ years
Format B : (As an interval scale)

Age
Less than 20 years
Between 20 – 30 years
Between 30 – 40 years
More than 40 years

Format C: (As an ordinal scale)
Age
Young adults (Less than 40 years)
Older adults (More than 40 years)

To summarize …
Ratio Data
 In terms of numbers
 Have absolute zero (objective)
 Statistical measures: mean,
variance, std deviation,
nonparametric methods
Nominal Scale
Dichotomous
Category Scale
Scale
(Multiple
(Two
responses)
responses)

• Died/Survive
Dichotomou • Defective/Good
s Scale • Response/No response
• Pass/Fail
• Urban/ Suburban/ Rural

• Malay/ Chinese/ Indian/
Category Others
Scale • Government/ Semi-
government/ Private

Rating Scale
Ordinal Interval
Scale Scale

Ordinal Scale
Graphic
Constant
Rating
Sum Scale
Scale

Graphic Rating Scale
A scale of between 7 through 10 points
Normally used for evaluating performance

Word placement on the word-graphic rating scale by pediatric patients
Pediatric Nursing, Jan-Feb, 1997 by Lisa Sinkin-Feldman, Mary Tesler, Marilyn
Savedra
Nurses who care for children in pain agree that accurate assessment of the pain is essential to the
development of an effective treatment plan. Accuracy of pain assessment is increased by the use
of a valid, reliable, and developmentally appropriate pain assessment tool that the child has been
taught to use.
One tool used to assess pain intensity in children and adolescents is the Word-Graphic Rating
Scale (WGRS), a component of the Adolescent Pediatric Pain Tool (APPT) (Tesler et al., 1991;
Savedra, Holzemer, Tesler, & Wilkie, 1993). The Word-Graphic Rating Scale was adapted for
children and adolescents from an adult scale. The tool consists of five descriptive words that
denote degrees of pain intensity. The words are printed approximately equidistant under a 100
mm line. No research had examined the placement of these words on the line. This descriptive
study sought to establish where a sample of hospitalized children and adolescents would place
the descriptive words on a 100 mm line anchored by "no pain" and "worst possible pain" to
describe degrees of pain intensity.
…… Scores can range from 0 for no pain to 100 for worst possible pain. The children are asked
to bisect the line at the point that best represents their pain. The distance to the mark is measured
with a 100 mm ruler from the 0 point. The number is recorded as the child's pain intensity.

Constant Sum Scale
Example: The following question asks you to divide 100 points between a set
of options to show the value or importance you place on each option.
Distribute the 100 points giving the more important reasons a greater number
of points. The computer will prompt you if your total does not equal exactly 100
points.
Example: A group of clinicians were asked to allocate 100 points across 10

commonly prescribed drugs with the points reflecting the frequency that they
prescribed the drugs to the last 100 patients with a particular disease.
This type of question is used when you are relatively sure of the reasons
for purchase, or you want input on a limited number of reasons you feel are
important. Questions must sum to 100 points and point totals are checked by
a java script.

An Example – Constant Scale

Interval Scale
Itemized
Likert Numeric
rating
scale al scale
scale

… Interval Scale
Semantic
differential Stapel scale
scale

Likert Scale
• Rensis Likert
• Initially a 5-point scale to measure level of agreement
• The responses start from positive to negative or vice
versa.
• Developed to a 7 or 9-point scale
Whether a 7 or 9-point scale fares better than a 5-point scale is

debatable. Some researchers say no difference and others say the
more choice of responses the better. (Please see page 151 of
textbook)

Strongly Agree Neither Disagr Strongly
Agree ee Disagree
If the price of raw materials fell firms would 1 2 3 4 5

reduce the price of their food products.
Without government regulation the firms would 1 2 3 4 5
exploit the consumer.
Most food companies are so concerned about 1 2 3 4 5
making profits they do not care about quality.
The food industry spends a great deal of 1 2 3 4 5
money making sure that its manufacturing is
hygienic.
Food companies should charge the same price 1 2 3 4 5
for their products throughout the country

Itemized Rating Scale
• Measuring other than level of agreement (satisfaction,
importance, likeliness, interest, etc)
• Usually a 5 or 7- point scale with anchors.
• With neutral point (balanced rating scale).
• Without neutral point (unbalanced rating scale). (See
example on page 220 of textbook).
• The anchors start from positive to negative or vice versa.
• Since similar to Likert scale, many researchers claimed
itemized rating scale as Likert-scaled.

An Example – Itemized Rating Scale
Please circle your answer to the given questions based on the following
scales:
1 2 3 4 5
Very Unlikely Neither Likely Very Likely
Unlikely Unlikely
Nor
Likely
anchors anchors

An Example – Itemized Rating Scale

Staple Scale
• Simultaneously measures both the direction
and intensity of the attitude towards the item
under study.

An Example - Staple Scale

Semantic Differential Scale
• A rating scale that has two bi-polar adjectives
at each end.
• The adjectives should be a mixture of positive
to negative and vice versa. (Good-Bad, Weak-
Strong, Hot-Cold, etc)
• Does not have a neutral or middle selection.

An Example - Semantic Differential Scale

Numerical Scale
Bipolar adjectives at both ends like semantic
differential scale.

An
Example –
Numerical
scale
Paired Comparisons
• A powerful voting, prioritization and concensus technique.

• More commonly used by a team
• Subjective opinion-based technique

Forced Choice
• Rank objects relative to one another among
the alternatives provided
• Formats using 4 favorable items, from which
the rater chooses the items most
characteristic of the person rated, prove
superior to other formats.

An Example – Forced Choice

An Example – Forced Choice

Comparative Scale
Provides a benchmark or a point of reference to
assessing attitudes toward current situation,
object or event.

An Example - Comparative Pain Scale

An Example - Comparative Pain Scale
Comparative Pain Scale
0 No pain. Feeling perfectly normal.
Very light barely noticable pain, like a
1
Very Mild mosquito bite or a poison ivy itch. Most of
the time you never think about the pain.
Minor pain, like lightly pinching the fold of

2 skin between the thumb and first finger with
Minor
Discomforting the other hand, using the fingernails. Note
Does not interfere with most
that people react differently to this self-test.
activities. Able to adapt to
pain psychologically and with
medication or devices such as
cushions. Very noticable pain, like an accidental cut, a
blow to the nose causing a bloody nose, or a
3 doctor giving you an injection. The pain is not
Tolerable so strong that you cannot get used to it.
Eventually, most of the time you don't notice
the pain. You have adapted to it.

Strong, deep pain, like an average toothache, the
initial pain from a bee sting, or minor trauma to part
of the body, such as stubbing your toe real hard. So
strong you notice the pain all the time and cannot
4
completely adapt. This pain level can be simulated
Distressing
by pinching the fold of skin between the thumb and
first finger with the other hand, using the fingernails,
and squeezing real hard. Note how the similated pain
Moderate
is initially piercing but becomes dull after that.
Interferes with many
Strong, deep, piercing pain, such as a sprained ankle
activities. Requires
when you stand on it wrong, or mild back pain. Not
lifestyle changes but 5
only do you notice the pain all the time, you are now
patient remains Very
so preoccupied with managing it that you normal
independent. Unable to Distressing
lifestyle is curtailed. Temporary personality
adapt to pain.
disorders are frequent.
Strong, deep, piercing pain so strong it seems to
partially dominate your senses, causing you to think
somewhat unclearly. At this point you begin to have
6
trouble holding a job or maintaining normal social
Intense
relationships. Comparable to a bad non-migriane
headache combined with several bee stings, or a bad
back pain.

Same as 6 except the pain completely dominates your
7 senses, causing you to think unclearly about half the time. At
Very this point you are effectively disabled and frequently cannot
Intense live alone. Comparable to an average migraine headache.
Pain so intense you can no longer think clearly at all, and

have often undergone severe personality change if the pain
8
has been present for a long time. Suicide is frequently
Utterly
contemplated and sometimes tried. Comparable to childbirth
Severe Horrible
or a real bad migraine headache.
Unable to engage in normal
activities. Patient is disabled
and unable to function
independently. Pain so intense you cannot tolerate it and demand pain
9 killers or surgery, no matter what the side effects or risk. If
Excruciating this doesn't work, suicide is frequent since there is no more
Unbearable joy in life whatsoever. Comparable to throat cancer.
Pain so intense you will go unconscious shortly. Most people

have never experienced this level of pain. Those who have
10
suffered a severe accident, such as a crushed hand, and lost
Unimaginable
consciousness as a result of the pain and not blood loss, have
Unspeakable
experienced level 10.

Goodness of Measures
• What instrumentation(s) are necessary to
measure the variables?
• How sure are we that the instrumentations
can actually measure the variables (eg.
perception, satisfaction, intention,
acceptance, etc.) we are investigating?

Item Analysis
• The goal of item analysis is to select those
items that are most related to the
construct.
• Methodology – two independent t-test is
carried out to compare if there are significant
differences between high-scorers and low
scorers

Reliability
• Reliable refers to the ability of the items to
measure the variable over time and space
despite uncontrollable environmental
conditions and the state of the respondents.

Test-retest
• Survey conducted on the same individual using the same
instrument on 2 different occasions (a few days, 3 months, etc)
• Scores on the 1st administration are compared to scores on the
2nd administration using correlation (r).
• One concern of test-retest reliability is termed carryover effects.
• Carryover effects benefit individuals from already having taken a
test. This enables them to solve problems more quickly or
correctly the second time they take the same test
• To eliminate carryover effects researchers often give 2 highly
similar forms of the test to the same people at different times.

Split-half
• Divide or split the instrument into halves (each half is an
alternative form) randomly or by some other
predetermined method (i.e., split on similar content; odd-
even spilt)
• Administer a test to a group of individuals
• Correlate the scores on one half with those on the other
half

Cronbach Alpha
•One of the most popular reliability statistics in use (Cronbach, 1951).
•A measure of internal consistency (how well each individual item in a
scale correlates with the sum of the remaining items).
•Ranges from 0 to 1.
•The widely-accepted social science cut-off ≥ .70 . Also suggested by
Nunnally and Bernstein (1994)
•some use .75 or .80
•A value as low as 0.6 might be okay for an exploratory study. (G. David
Garson)

Reliability Statistics
Cronbach's Alpha
Based on N of
Cronbach's Alpha Standardized Items Items
.700 .719 6

Item-Total Statistics
Scale Corrected Squared Cronbach's

Scale Mean if Variance if Item-Total Multiple Alpha if Item
Item Deleted Item Deleted Correlation Correlation Deleted
most days i am
enthusiastic about my 24.36 13.227 .492 .349 .645
work
i feel that i am happier in
my work than most other 24.80 12.660 .566 .396 .622
people
my job is pretty interesting 24.31 12.619 .595 .394 .615
i enjoy my work more than
25.58 14.458 .185 .062 .743
my leisure time
Recode 24.36 11.993 .514 .311 .632
Recode 24.79 13.048 .338 .228 .695

How to report reliability
Component Number of items Cronbach Alpha
Work 6 0.700
Spiritual 5 0.806
Health condition and Physical 6 0.583

activities
Emotional and mental outlook 6 0.720
Leisure and social activities 6 0.817
Attitude 7 0.628

Content Validity
• A function of how well the dimensions and
elements of a concept have been described.
• Does the measure adequately measure the
concept?
• Evaluated by subject experts.

Concept
Individual related factor
 Women’s attitude towards
their roles Dimensions
 Work related factors

 Role conflict
 Role ambiguity
 Role overload
 Work commitment Work  Family Satisfaction
family  Job Satisfaction
conflict
 Family related factors
 Family conflict
 Marital commitment Subdimensions
 Parental commitment

Construct Validity
Does the instrument tap the concept as

theorized?
Measured exploratory factor analysis

(EFA)

Factor Analysis Output
Factor
1 2 3 4
Indicate company's MOTIVATIONAL
FACTORS for environmental
disclosures. (Fulfilling stakeholders' .791
needs and rights to the information)

disclosures. (Develops better .705
relationships with stakeholders)

disclosures. (It represents corporate .653
environmental responsibility)

disclosures. (It represents corporate .584 .360
environmental responsibility)

Factor 1 Factor 2 Factor 3 Factor 4
Indicate company's Indicate company's Indicate company's Indicate company's

MOTIVATIONAL FACTORS for MOTIVATIONAL FACTORS MOTIVATIONAL FACTORS MOTIVATIONAL FACTORS
environmental disclosures. for environmental for environmental for environmental
(Fulfiling stakeholders' needs disclosures. (Environmental disclosures. (A response disclosures. (It may prevent
and rights to the information) disclosures attract more towards global a legally imposed
investment) environmental Issues) requirement)

environmental disclosures. for environmental for environmental for environmental
(Develops better relationships disclosures. (Enhances disclosures. (Improves disclosures. (Company
with stakeholders) company's opportunities to environmental management practices a transparent
access wider market) system) reporting system)

environmental disclosures. (It for environmental for environmental for environmental
represents corporate disclosures. (It is an disclosures. (Helps company disclosures. (Fulfiling
environmental responsibility) evidence of the true and fair identify opportunities for suppliers' requirement)
view of operations) costs saving)
Indicate company's Indicate company's

MOTIVATIONAL FACTORS for MOTIVATIONAL
environmental disclosures. FACTORS for
(Enhances corporate image)
environmental disclosures.
(Compliance with
corporate environmental
policies)

Factor No of items Cronbach Alpha
1 4 0.796
2 3 0.838
3 3 0.786
4 4 0.586

Summary
The results of a study can only be as good as the
measures that tap the concepts in the theoretical
framework.
Use instruments that are reputable (more frequently

used).
ADAPT or ADOPT an established measure to suit the

setting.
Avoid developing your own instrument unless necessary.

April 2017
THANK YOU!!
Halilah Haron 88

Scaling, Reliability &amp; Validity

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Scaling, Reliability &amp; Validity

Caricato da

Copyright:

Formati disponibili

CHAPTER 7

Understand the levels of measurement.

April 2017 Halilah Haron 2

April 2017 Halilah Haron 3

• Categorical & Sequential in nature

• With absolute zero

April 2017 Halilah Haron 4

April 2017 Halilah Haron 5

April 2017 Halilah Haron 6

April 2017 Halilah Haron 7

April 2017 Halilah Haron 8

Double-terrace house Kampong house

April 2017 Halilah Haron 9

April 2017 Halilah Haron 10

Currency (pound, Us Dollar,

April 2017 Halilah Haron 11

April 2017 Halilah Haron 13

1 - Not important at all

April 2017 Halilah Haron 14

April 2017 Halilah Haron 15

April 2017 Halilah Haron 16

Stage (Stage 1, Stage 2, Job qualification

Cast (Brahmin, Satria, … ) BMI Level (underweight,

April 2017 Halilah Haron 17

April 2017 Halilah Haron 18

April 2017 Halilah Haron 19

April 2017 Halilah Haron 20

April 2017 Halilah Haron 22

Tahap kefahaman Tahap kesedaran

April 2017 Halilah Haron 24

April 2017 Halilah Haron 25

World’s Tallest Building

April 2017 Halilah Haron 26

April 2017 Halilah Haron 28

April 2017 Halilah Haron 29

April 2017 Halilah Haron 30

April 2017 Halilah Haron 31

Format B : (As an interval scale)

April 2017 Halilah Haron 32

Older adults (More than 40 years)

April 2017 Halilah Haron 33

April 2017 Halilah Haron 35

• Urban/ Suburban/ Rural

April 2017 Halilah Haron 36

April 2017 Halilah Haron 37

April 2017 Halilah Haron 38

April 2017 Halilah Haron 39

April 2017 Halilah Haron 40

Example: A group of clinicians were asked to allocate 100 points across 10

April 2017 Halilah Haron 41

April 2017 Halilah Haron 42

April 2017 Halilah Haron 43

April 2017 Halilah Haron 44

Whether a 7 or 9-point scale fares better than a 5-point scale is

April 2017 Halilah Haron 45

If the price of raw materials fell firms would 1 2 3 4 5

April 2017 Halilah Haron 46

April 2017 Halilah Haron 47

April 2017 Halilah Haron 48

April 2017 Halilah Haron 49

April 2017 Halilah Haron 50

April 2017 Halilah Haron 51

April 2017 Halilah Haron 53

April 2017 Halilah Haron 54

Scaling, Reliability & Validity

Scaling, Reliability & Validity