Non-Response Bias

One of the difficulties in surveying people (whether by mail, telephone, or direct approach) is that
some choose not to respond. Assume that you have decided to conduct a study which requires a
sample size of 100. If you only expect 10% of those surveyed to respond to your questionnaire, what
should you do? A nave answer is, "Simply send out 1000 questionnaires!"

Unfortunately, the demographics of respondents and nonrespondents may differ substantially. To

base estimates for the entire population merely on the data collected from respondents therefore
might leave you exposed to substantial sampling bias.

A form of stratified sampling is typically used to overcome non-response bias. An initial mass
mailing of questionnaires takes place, with identifying codes placed on each questionnaire (or its
return envelope). When the submission deadline for responses is reached, estimates can be made
for the stratum of "people who respond to the initial mailing." Crossing these people (by
cross-referencing the codes on their responses) off the initial mailing list leaves a list of people all of
whom are now known to be in the other "people who don't respond" stratum. The initial response rate
is used to estimate the relative sizes of the two strata.

A sample of those who didn't respond is now recontacted, using a more expensive approach
designed to obtain responses from everyone.(The expense is typically related to an incentive of
some kind.) Their data provides estimates for the second stratum, and the study can then be
completed.

In order to see how this might all be carried out, assume that you are asked to conduct a
mail-based marketing survey. You would put together your questionnaire, and then conduct a pilot
study. Assume that you send out 100 questionnaires, get 20 back, and then recontact 30 of the
nonrespondents in order to learn how much of an incentive must be provided to them in order to
obtain their responses (and to collect data from them).

Focusing on the estimation of mean annual income, assume the results of the pilot study are as
displayed below:

pilot study respondents (A) non-respondents (B)

cost/contact $1 $10
mean $30,000 $40,000
stdev $5,000 $8,000
fraction of sample 20% 80%

(It costs about $1 to handle each original mailing, and the recontact incentives will cost you about
$10/recontact.)

The full study can now be roughed out. The yellow cells will eventually contain the size of the initial
mailing and the number of nonrespondents who will be recontacted. The cells with red entries will,
when the study is completed, contain actual results, and the actual estimate and its margin of error
(at the 95%-confidence level) will appear below. For planning purposes, we temporarily fill in results
from the pilot study.

full study respondents (A) non-respondents (B)

mailings 100
 mean $30,000 $40,000
stdev $5,000 $8,000
responses 20 80 non-responses
fraction of sample 20% 80%
30 recontacts

estimated mean $38,000

margin of error $2,460
target margin of error $1,000
cost of study $400

I've set up Excel's "Solver" to find the least-expensive way to conduct the full study while achieving
the target margin of error in the final estimate. Select "Tools / Solver / Solve" to find the solution.

s2A 2
2 sB f ( 1f A )
1. 96 f 2
A +( 1f A ) +( x A x B )2 A
( f An A) nB nA
Mathematical note: The formula for the margin of error in the final estimate looks just like the standard
formula for stratified sampling, but with an extra term. The last term takes into account the fact that
we're using the study to estimate the relative sizes of the strata, rather than taking those sizes as
"known" from some previous study.