Q U A L I T Y

A W A R D S

Measuring Performance
After Meeting Award Criteria
Study compares perceived success
to financial data of award winners and applicants
by
Timothy M. Bergquist and Kenneth D. Ramsing

QUESTION OFTEN ASKED AMONG QUALITY

award criteria have on U.S. companies.2 The

practitioners is whether companies

study, An Assessment of the Operational and

that receive quality awards actually

Financial Impact on Companies of Quality Awards in

perform better than others. This

the United States, is the first to reexamine the

issue has been of major concern

GAOs original 20 measures and investigate

since the Malcolm Baldrige National Quality

MBNQA and state-level quality award winners

Award (MBNQA) was first presented in 1988.

and applicants. It is also the first to extend such

After the General Accounting Office (GAO) pub-

measures to organizations that never applied for

lished the results of its 1991 study on quality

a quality award. The results of this study can be

management practices of companies that had

used to better understand the links between per-

received MBNQA site visits, several other studies

formance and quality award criteria.

tried to assess issues related to quality awards.

A 1993 report by the Conference Board sum-

Understanding the GAO study

The GAO study was conducted to determine

marized 20 studies that had been conducted

regarding quality management practices, but

the importance of total quality management

only three of those studiesone of which was

(TQM) practices on the performance of U.S. com-

the GAO studyfocused on quality awards.

panies. The GAO developed a general TQM

No other study used the same GAO measures

model for companies exhibiting the direction and

to assess performance until 1996. This article pre-

interrelationships of the total quality processes to

sents the results of that 1996 research study,

improve performance and the direction of

which used the same 20 performance measures

improvement for each measure.3 Six common fea-

as the GAO to investigate the effects quality

tures that contributed to improved company

66

QU A L I T Y P R O G R E S S

S E P T E M B E R

1 9 9 9

performance were defined: customer focus, management and leadership, employee involvement,

TABLE 1

Status of State Quality Awards

as of February 1999

open corporate culture, fact-based decision-making and partnership with suppliers.

Investigators from the GAO interviewed 20 of

MBNQA-like Award

U.S. Senate
Productivity Award

Arizona93
Arkansas95

Alabama86

the 22 companies that experienced a site visit

California94

California94

from the MBNQA in either 1988 or 1989. The

Connecticut(2)88, 93

Louisiana84

impact of TQM practices on company perfor-

Florida93

Maryland83

Georgia99

mance was examined in four areasemployee

Nevada89

Hawaii95

Virginia83

relations, operating procedures, customer satis-

Idaho96

faction and financial performanceusing 20

Illinois95
Kansas96

Other Quality Award

performance measures.4 An expected direction

Kentucky97

California94

of improvement (up or down) was assigned to

Louisiana95

Delaware92

each measure.

Maine91

Louisiana88

Massachusetts92

Maryland86

Michigan94

Nebraska93

companies in the study, the GAO determined

Minnesota91

Rhode Island93

the average annual percentage change for every

Mississippi95

South Dakota85

Missouri93

Wyoming86 (ended92,

Based on interviews and data provided by the

measure and the direction of each measures

change: positive, negative or unchanged.
These 20 measures demonstrated overwhelmingly positive improvement in a companys

New Hampshire95
New Mexico93 (SPA84)

No Award

New York92
North Carolina91

performance when employing TQM practices.

Ohio99

However, the study indicated that implement-

Oklahoma94

ing TQM endeavors took time to yield

restarted97)

New Jersey93

Alaska
Colorado
Montana

Oregon94

North Dakota

Pennsylvania94

significant results. On average, the companies

Rhode Island95 (Other94)

studied took 2.5 years to yield results.5, 6, 7, 8

South Carolina95
Texas93
Utah95

One of the largest problems with the GAO

West Virginia
Wisconsin

Tennessee93

MBNQA and state-level quality awards

Vermont

Award in development

Washington94
Indiana

study was that it did not consider any quality

Iowa

award other than the MBNQA. This may be

due to the fact that such programs were not as
typical at the time of the GAO study as they are

Improvement Act was signed on August 20, 1987.

today.

The MBNQA recognizes outstanding quality

Established by an Act of Congress and designated as Public Law 100-107, the MBNQA

management and customer satisfaction of U.S.

companies in three categories: manufacturing,
QU A L I T Y P R O G R E S S

S E P T E M B E R

1 9 9 9

67

M E A S U R I N G T H E P E R F O R M A N C E O F Q U A L I T Y AWA R D W I N N E R S

service and small business. In 1999, health care and

education categories were added. The examination
items and areas to address have continued to evolve
and are updated every two years to ensure relevance.
A major outcome of the MBNQA was the development of quality award programs at the state level.
These state-level quality awards generally take one of
three forms: (1) patterned almost identically after the
MBNQA, (2) developed under the U.S. Senate
Productivity Award (SPA) program or (3) a local
award, independent of existing criteria. The categories and rigor of applying for them vary from state
to state.9,10
The number of award programs changes yearly as
more states develop awards or modify existing ones.
Currently, 33 states have developed MBNQA-like
awards, six states utilize the SPA program and eight
states have other types of quality awards. Table 1
describes the status of all state quality award programs as of February 1999, including the first year
each award was presented.
The increase in these awards and the weight of
importance they have grown to hold made it necessary to include their existence in the 1996 study.

Performance measures of the study

Due to the increased number of quality award programs, the 1996 study needed additional measures of
performance to include the value of using local award
criteria. A careful review of past studies on quality
awards, literature on world-class companies and case
studies dealing with strong quality management practices, led to the development of such measures.
An Assessment of the Operational and Financial Impact
on Companies of Quality Awards in the United States
expanded the GAOs 20 measures to a total of 40 measures. These measures were grouped into three
general areas:
Operational impacts pertain to the ability of a company to meet customer needs. These 22 measures
can be categorized into three subareas: employee
related, operations related and customer related.
The employee related impacts concern changes in
such areas as overall employee satisfaction, attendance and turnover, safety and health, and the use
of work teams. The operations related impacts
include changes in such factors as reliability, ontime delivery, errors or defects, costs of quality and
production costs. The customer related impacts concern changes in issues involving customer
satisfaction, complaints and retention. The GAO
study had 16 operating measures.
Financial impacts relate to the companys bottomline monetary and competitive issues. These 13
68

QU A L I T Y P R O G R E S S

S E P T E M B E R

1 9 9 9

TABLE 2

Performance Measures

Twenty Operational Measures Taken From the GAO Study

Employee Related
1. Employee satisfaction.
2. Attendance.
3. Turnover.
4. Safety/health.
5. Suggestions received.
Operations Related
9. Reliability.
10. Timeliness of delivery.
11. Order-processing time.
12. Errors or defects.
13. Product lead time.
14. Inventory turnover. *
15. Costs of quality.
16. Cost savings.
Customer Related
20. Overall customer satisfaction.
21. Customer complaints.
22. Customer retention.
Financial Measures
23. Market share.
24. Sales per employee.*
25. Return on assets.*
26. Return on sales.*

NOTES:
Measures 6-8, 17-19, and 27-40 were another part of the project.
* indicates available in Compustat.

measures include changes in items such as market

share, sales per employee, return on assets and
sales, share price per earnings, net profit, operating
expenses and inventory. The GAO study only examined four financial related measures. Many have
expressed concern that quality awards should indicate profitable companies, yet the awards do not
directly measure financial performance. By implementing a quality improvement process, the results
should, in the long run, show up on the bottom line.
The award related impacts addressed five specific
issues regarding quality awards: the internal use of
the award criteria, the use of award feedback infor-

mation, the extent of benchmarking, the use of quality management practices, and the time and
expense involved in applying for the award. The
GAO study did not address any award related measures.

The research study

While the GAO study evaluated companies that
received MBNQA site visits between 1988 and 1989,
this research study looked at winners and applicants
of both the MBNQA and state-level award programs
between the years 1990 and 1995. Nonapplicants were
included to provide a basis for comparison. The categories of companiescommon to almost all the
awardswere manufacturing, service and small business. Organizations in health care, education and
government were not included.
Two data sources were used in this study: a largescale, nationwide mail survey and Compustat, a
financial database of public companies. Compustat
contains financial measures on more than 8,000 stockissuing companies, primarily taken from financial

TABLE 3

statements and annual reports that are in the public

realm, as determined by the Securities and Exchange
Commission.
Even though the mail survey examined 40 measures
and the Compustat data covered 13 measures, this
article is only concerned with the results of the 20
measures taken from the GAO study. Due to this
focus, only four Compustat measures can be considered, as they are the only measures to overlap the
GAOs original 20 (see Table 2).

The respondents
The mail survey was sent to 1,122 companies, which
consisted of 343 quality award winners, 393 quality
award applicants and 386 non-applicants. A total of
193 companies (17%) responded to the mail survey.
Public companies made up 564 or about half of the
total number of companies that received the survey.
Of these 564 public companies, 418 had the data for all
13 Compustat measures; therefore, only these public
companies could be analyzed where Compustat data
was concerned. Thirty-eight of these 418 companies

Mail Survey ResultsChange in Measures

Winners

Measure
#

Measure
Name

GAO Direction of
Improvement

Mean

Maximum sample size

Applicants

Sample
Size

Mean

Sample
Size

103

Nonapplicants
Mean

Sample
Size

64

26

Employee Related Measures

1
2
3
4
5

Employee satisfaction
Attendance
Turnover
Safety/health
Suggestions received

up
up
down
up
up

5.431
4.919
3.580
3.280
5.368

102
86
100
93
87

5.333
4.825
4.164
3.696
5.091

63
57
61
56
55

5.238
4.905
3.800
3.316
5.412

21
21
20
19
17

up
up
down
down
down
up
down
up

5.940
5.598
4.178
3.000
3.093
5.260
2.914
5.522

100
102
90
101
86
77
93
92

5.590
5.548
4.586
3.217
3.421
4.813
3.429
5.078

61
62
58
60
57
48
56
51

5.696
5.696
4.500
2.714
3.526
5.250
2.444
5.350

23
23
22
21
19
16
18
20

up
down
up

5.961
2.653
5.477

103
101
86

5.381
2.967
4.833

63
60
60

5.680
2.958
5.273

25
24
22

up
up
up
up

5.643
5.805
5.726
5.671

84
82
84
85

4.944
5.389
5.259
5.039

54
54
54
51

5.478
5.760
5.542
5.524

23
25
24
21

Operations Related Measures

9
10
11
12
13
14
15
16

Reliability
Timeliness of delivery
Order-processing time
Errors or defects
Product lead time
Inventory turnover
Costs of quality
Cost savings

Customer Related Measures

20
21
22

Overall customer satisfaction

Customer complaints
Customer retention

Financial Measures
23
24
25
26

Market share
Sales per employee
Return on assets
Return on sales

QU A L I T Y P R O G R E S S

S E P T E M B E R

1 9 9 9

69

M E A S U R I N G T H E P E R F O R M A N C E O F Q U A L I T Y AWA R D W I N N E R S

Survey ResultsPrimary Reason (Mode) for Change

TABLE 4

Winners
Measure
#

Measure
Name

Direction of
Improvement

Mode

Maximum Sample Size

Sample
Size

Applicants
Percent

Sample
Mode

103

Size

Nonapplicants
Percent

Sample
Mode

64

Size

Percent

26

Employee Related Measures

1
2
3
4
5

Employee satisfaction
Attendance
Turnover
Safety/health
Suggestions received

up
up
down
up
up

7
7
7
7
7

92
63
73
74
78

54.35%
53.97%
49.32%
55.41%
70.51%

7
7
7
7
7

58
42
43
34
44

37.93%
52.38%
30.23%
50.00%
52.27%

7
7
7
7
7

18
13
14
11
13

83.33%
61.54%
42.86%
72.73%
76.92%

up
up
down
down
down
up
down
up

7
7
7
7
7
7
7
7

91
94
80
91
72
64
80
76

49.45%
54.26%
53.75%
69.23%
51.39%
54.69%
72.50%
68.42%

7
7
7
7
7
7
7
7

49
50
48
47
43
38
44
42

55.10%
58.00%
47.92%
59.57%
53.49%
52.63%
61.36%
64.29%

7
7
7
7
7
7,8
7
7

18
16
17
17
14
13
14
15

55.56%
68.75%
52.94%
64.71%
42.86%
23.08%
71.43%
93.33%

up

92

63.04%

52

59.62%

20

90.00%

down
up

7
7

88
72

67.05%
62.50%

7
7

45
45

66.67%
62.22%

7
7

17
17

82.35%
88.24%

up
up
up
up

7
7
7
7

72
75
75
74

37.50%
41.33%
42.67%
40.54%

7
7
7
7

44
46
44
45

38.64%
34.78%
40.91%
40.00%

7
7
7
7

19
20
19
16

42.11%
40.00%
36.84%
43.75%

Operations Related Measures

9
10
11
12
13
14
15
16

Reliability
Timeliness of delivery
Order-processing time
Errors or defects
Product lead time
Inventory turnover
Costs of quality
Cost savings

Customer Related Measures

20
21
22

Overall customer
satisfaction
Customer complaints
Customer retention

Financial Measures
23
24
25
26

Market share
Sales per employee
Return on assets
Return on sales

also completed the mail surveyincluding five winners, 13 applicants and 20 nonapplicants.
Demographic data was compiled from both the survey and Compustat database. Of the companies, 62%
were manufacturing, and 38% were service. About
98% of the survey respondents indicated they were
implementing quality management practices. Half of
the survey respondents were quality managers or
directors, one-quarter were presidents or vice presidents, and the remaining quarter were project or
factory managers.
Survey respondents also indicated that the average
time between implementing TQM practices and winning a quality award was 5.5 yearsover twice as
long as indicated by the GAO study.

The mail survey

The mail survey was organized into five sections:
(I) company background information, (II) award
information, (III) change in performance of measure,
(IV) reasons for change in performance and (V)
70

QU A L I T Y P R O G R E S S

S E P T E M B E R

1 9 9 9

additional information.
Section III asked respondents to estimate the average annual percentage change for each measure
between 1990 and 1995. The following seven point
Likert scale was used, including a percentage for each
category:
1Decreased greatly, 11%+
2Decreased moderately, 6 to 10%
3Decreased slightly, 1 to 5%
4No change, 0%
5Increased slightly, 1 to 5%
6Increased moderately, 6 to 10%
7Increased greatly, 11%+
In addition, an N/A option was provided and coded
as a 0 for analysis purposes.
Section IV of the survey requested the reason for the
change in the measure. Twelve options were provided, including an other option.
The results from the mail survey are shown in
Table 3. The number, name and expected direction of
improvement are given for each measure. The mean

TABLE 5

Compustat Results
Winners

Measure
#

Measure
Name

Direction of
Improvement

Mean

Maximum sample size

Applicants

Sample
Size

Mean

Nonapplicants

Sample
Size

17

Mean

20

Sample
Size
379

Operational Measure
14

Inventory turnover

up

4.84%

15

3.00%

18

3.06%

304

up
up
up

-5.11%
-0.28%
-11.81%

16
13
17

1.77%
35.51%
-4.52%

20
17
20

0.61%
-4.27%
-5.32%

339
360
363

Financial Measures
24
25
26

TABLE 6

Sales per employee

Return on assets
Return on sales

Common Company Analysis

Survey Data

Measure
#

Measure
Name

Direction of
Improvement

Correlation
Coefficient

Mean

Maximum sample size

COMPUSTAT Data

Sample
Size

Mean

38

Sample
Size
3

Operational Measure
14

Inventory turnover

up

0.432

5.069

29

4.727

33

up
up
up

-0.017
0.097
-0.311

5.789
5.703
5.576

38
37
33

4.086
4.684
2.842

35
38
38

Financial Measures
24
25
26

Sales per employee

Return on assets
Return on sales

and sample size are also reported for the three groups
of companies who responded to the mail survey
award winners, award applicants and nonapplicants.
Mean values less than four indicate the percent
change decreased, while values greater than four indicate the percent change increased. All N/A responses
were excluded from the survey analysis.
Generally, the mail survey results for all three types
of organizations followed in the same expected direction of improvement as found in the GAO study.
However, for safety/health (measure 4) and order
processing time (measure 11), the survey results indicated an opposite direction than might be expected for
all three groups.
The mail survey requested respondents to identify
the reason for the positive change in the measure for
their company. Table 4 presents the results of their
responses. The most common reason given by all three
groups was implemented quality management practices.
More than half of the respondents gave this reason for
several measures. Clearly, respondents believed that
the changes in these measures were directly related to

the implementation of quality management.

The last section of the survey asked if respondents
thought the award criteria had a positive impact on
their companys performance. About 89% of winners,
77% of applicants and 42% of nonapplicants said it had.
A linkage appears to exist between award criteria and
perceived company performance.

Financial results
Table 5 shows the Compustat results for the same
three groups of companies. Note the sample size difference. This is primarily due to the fact that most
quality award winning companies are not public companies. The percentage change was actually calculated
from the data and is reported as a percentage.
The three financial Compustat measures offered disturbing results. The mean values were negative
(opposite from the expected direction of improvement) for the winners in all three measures. The same
can be said for one measure regarding applicants and
two measures where nonapplicants are concerned.
There does not appear to be any consistency.
QU A L I T Y P R O G R E S S

S E P T E M B E R

1 9 9 9

71

M E A S U R I N G T H E P E R F O R M A N C E O F Q U A L I T Y AWA R D W I N N E R S

Perceived performance vs. actual financials

A comparison was made between the mail survey
and Compustat results for the four measures that both
sources had in common. Table 6 compares the results
of the four measures among the 38 public companies
that both responded to the survey and had data in
Compustat. To accomplish this comparison, the
Compustat results were converted to the 1 to 7 Likert
scale used in the survey. The analysis was performed
by combining the data from the survey and
Compustat for the 38 companies.
In all four measures, the mail survey respondents
thought they were doing better (by expected direction
of improvement) than was indicated by the
Compustat data. The mean values for the survey data
are greater than for the Compustat datasometimes
by a large margin.
This is corroboration that, in general, respondents to
surveys think they are doing better than they actually
are. A review of these 38 company surveys showed
that the respondents were primarily quality managers, not financial managers. A correlation analysis
for all 38 companies identified two correlationsmeasures 24 and 26as negative (see Table 6). In other
words, these respondents thought almost the opposite
of what their company was actually doing.

Analyzing the results

Based on the results from this research study, we
cannot conclusively determine whether quality award
winning companies perform better than others.
However, since 89% of the winners and 77% of the
applicants who responded to the mail survey believed
that using award criteria did have a positive impact
on company performance, a link appears to exist
between award criteria and perceived company performance. Even though there is no clear proof that
award criteria yields positive results, implementing
quality management practices does seem to have an
impact on how positively employees judge the organizations operations and overall performance.

Harvard Business Review, 1991, pp. 80-93.

6. Hiam, Does Quality Work? A Review of Relevant
Studies (see reference 1).
7. Y.K. Shetty, The Quest for Quality Excellence: Lessons
from the Malcolm Baldrige Quality Award, SAM Advanced
Management Journal, 1993, pp. 34-40.
8. B. Stratton, The Value of Implementing Quality, Quality
Progress, July 1991, pp. 70-71.
9. T.M. Bergquist, An Assessment of the Operational and
Financial Impact on Companies of Quality Awards in the
United States (See reference 2).
10. T.M. Bergquist, A Comparative Study of State Quality
Awards, Proceedings of the 26th Annual Meeting of the
Decision Sciences Institute, 1995, pp. 1707-1709.

BIBLIOGRAPHY
Bemowski, K., The State of the States, Quality Progress, May
1993.
Bergquist, T. M., State Quality Award Directory:
Introduction, Quality Digest, Vol. 14, No. 11, 1994.
Bergquist, T. M., and T. Dedlow, The Backbone of State
Quality Awards: The Examiners, The Journal for Quality and
Participation, Vol. 19, No. 4, 1996.
Clarke, M. K., and E. N. Dobson, Promoting Quality Businesses:
A State Action Agenda (Washington, DC: National Governors
Association, 1992).
Dobson, E. N., Designing and Implementing a State Quality Award
(Washington, DC: National Governors Association, 1993).
Dusharme, D., 1997 State Quality Award Directory, Quality
Digest, Vol. 17, No. 2, 1997.
Dusharme, D., State Quality Award Directory, Quality Digest,
Vol. 14, No. 11, 1994.
Politi, J., State Quality Award Network, unpublished working
paper.
U.S. Department of Commerce, Malcolm Baldrige National
Quality Award: 1999 Award Criteria (Gaithersburg, MD:
National Institute of Standards and Technology, 1998).
U.S. State Quality Award Mini-Directory, The Journal for
Quality and Participation, Vol. 19, No. 4, 1996.

TIMOTHY M. BERGQUIST is an associate professor of manage-

REFERENCES
1. A. Hiam, Does Quality Work? A Review of Relevant
Studies, Report No. 1043 for The Conference Board, 1993.
2. T.M. Bergquist, An Assessment of the Operational and
Financial Impact on Companies of Quality Awards in the
United States, unpublished doctoral dissertation, 1996.
3. GAO/NSIAD-91-190, Management Practices: U.S.
Companies Improve Performance Through Quality Efforts
(Washington, DC: U.S. Government Printing Office, 1991).
4. Ibid.
5. D.A. Garvin, How the Baldrige Award Really Works,
72

QU A L I T Y P R O G R E S S

S E P T E M B E R

1 9 9 9

ment at Northwest Christian College in Eugene, OR. He received

his doctorate in business from the University of Oregon-Eugene.
Bergquist also has masters degrees in statistics, business administration and operations research from various universities. He is
an ASQ member.
KENNETH D. RAMSING is a professor of management at the

University of Oregon-Eugene. He has a doctorate in business

from the University of Oregon-Eugene.