Journalq/'Mant([dcturingvol.
24/No.Systems~l
A Review of Lean Assessment in Organizations:
An Exploratory Study of Lean Practices by
Electronics Manufacturers
Toni L. D o o l e n and Maria E. Hacker, Industrial and Manufacturing Engineering, Oregon State University,
Corvallis, Oregon, U S A
Abstract
This paper describes the development of a survey instru-
ment to assess the implementation of lean practices within
an organization. Results of a literature review, which was used
to identify lean manufacturing practices and existing lean as-
sessment tools, are presented.The findings of this review were
synthesized to develop an instrument to assess both the num-
ber and the level of implementation of a broad range of lean
practices in an organization. As part of a larger research
project, an exploratory study was completed using the sur-
vey. A cross section of electronic manufacturers in the Pacific
Northwest was used for the exploratory study. Analysis of the
survey results from the exploratory study are summarized in
this paper to illustrate how the survey can be used to under-
stand what factors might contribute to the implementation of
lean practices. In the exploratory study completed, for exam-
ple, it was found that while electronic manufacturers have
implemented a broad range of lean practices, the level of im-
plementation does vary and may be related to economic, op-
erational, or organizational factors.
Keywords: Lean, Manufacturing, Electronics, Lean Produc-
tion, Operations Management
Introduction
With the publication of the book The Machine That
Changed the World, lean manufacturing practices
have found acceptance in many manufacturing op-
erations over more traditional mass production tech-
niques. Womack, Roos, and Jones (1990) studied the
implementation of lean manufacturing practices in
the automotive industry on a global scale. Other re-
search efforts, such as the Lean Aerospace Initiative
(2002), have looked at the unique challenges of imple-
menting lean manufacturing practices in a specific
type of industry. The larger research project for which
this survey development effort was initiated had a
similar focus--understanding the implementation of
lean manufacturing practices within the electronics
manufacturing industrial sector. To study the use and
ultimately the impact of lean manufacturing practices
on electronics manufacturers, a literature review was
55
2005
completed to determine if existing assessment meth-
ods could be applied to this study.
Both the research and practitioner literature were
reviewed to identify a complete set of lean practices
and existing research instruments that might be le-
veraged for the study. From this review, a number of
lean practices were identified, but no single research
instrument was found that captured this broad array
of practices. As a result, it was necessary to develop
a new tool for the study. The primary contribution of
this paper is to describe the development of this as-
sessment tool and to illustrate, through an exploratory
study, how the tool can be used to assess both the
range of lean practices implemented, as well as the
level of adoption of these practices within an organi-
zation. The exploratory study, conducted using a
sample of electronic manufacturers, demonstrates
how the instrument can be used to understand factors
that may limit or constrain the implementation of lean
practices. A selective review of lean implementation
literature is summarized in the next section to pro-
vide a foundation for interpreting the results of the
exploratory study.
Overview of Factors Impacting
Lean Implementation
Manufacturers in many different business sectors
and in countries across the world have invested in
the implementation of lean manufacturing practices.
Many of these implementations are grounded in the
principles and practices incorporated in the Toyota
Production System and often referred to as lean pro-
duction. Womack and Jones (2003) defined five lean
principles to eliminate waste in organizations. The
principles are: specifying value, identifying the value
stream, flow, pull, and perfection. While the principles
of lean are relatively constant, there are many differ-
ent practices that can and have been implemented in
Journal o/" Manu/2lcturing Systems
Vol. 24/No. 1
2005
the quest for value creation in the manufacture of
products. Just-in-time (JIT) production systems, to-
tal productive maintenance (TPM), cellular manufac-
turing, single-minute exchange of die (SMED), mixed
model production (MMP), and mistake-proofing are
just a few of the many examples of practices that are
associated with a lean production system. The suc-
cessful application of various lean practices has been
documented in a variety of industries, such as aero-
space, computer manufacturing, and automotive as-
sembly (MacDuffie, Sethuraman, and Fisher 1996;
Laughlin 1995; Houlahan 1994). Some manufactur-
ing organizations, however, have experienced only
limited success in using lean manufacturing practices
to increase competitiveness. Researchers have ex-
plored the portability of lean practices both within
and between different manufacturing sectors. A re-
view of previous research exploring the limitations
of lean manufacturing was completed. Based on this
review, there is evidence that factors such as chang-
ing economic conditions, high levels of demand un-
certainty, high-mix, low-volume product portfolios,
and rigid organizational structures may limit the ap-
plicability of lean manufacturing practices or may
prevent manufacturers from realizing the full ben-
efits of these practices.
Some authors contend that other manufacturing
strategies may be superior to lean under certain mar-
ket conditions (Cooney 2002; Katayama and Bennett
1996). In particular, Cooney asserts that market char-
acteristics of an industrial sector should influence the
type of production strategy chosen. A push system
utilizing batch production was found to be effective
for automotive component manufacturers given un-
stable customer demand and short-term customer re-
lationships. Katayama and Bennett found that some
Japanese manufacturers who adopted a lean manu-
facturing strategy were faced with problems due to
variations or reduction in product demand; success
associated with lean in the 1990s was attributed to
favorable market conditions in Japan and the rest of
the world.
Several researchers (Cooney 2002; Fliedner and
Vokurka 1997; James-Moore and Gibbons 1997; Ma-
son-Jones, Naylor, and Towill 2000; Yusef and
Adeleye 2002) contend that lean practices do not pro-
vide a compelling competitive edge in all operational
environments and have identified alternative opera-
tional practices. Yusef and Adeleye found that orga-
56
nizations that focus on volume flexibility, technol-
ogy leadership, speed to market, and new product
development outperform organizations that focus on
low cost and quality. Strategies to develop and main-
tain supplemental capacity within the overall supply
chain (in direct contrast to the lean philosophy of
waste reduction) have also been shown to provide a
significant competitive advantage (Mason-Jones,
Naylor, and Towill 2000). Batch production, along
with craft production methods, were found to pro-
vide automotive suppliers with the flexibility needed
to be successful when rapid product and process in-
novation were the norm (Cooney 2002). Finally,
James-Moore and Gibbons (1997) determined that
the level of lean manufacturing adoption in the aero-
space industry was lower than expected as a result of
high product mix and low volumes.
Previous research has also identified the manage-
ment of human resources as a significant issue in the
implementation of lean (Agrawal and Graves 1999;
Bamber and Dale 2000; Nicholas 1998; Yauch and
Steudel 2002). Bamber and Dale found that "rigid
hierarchical organizational design" was the single
biggest obstacle to the adoption of lean production
practices. Yauch and Steudel showed that cultural
factors play a direct role in impeding the formation,
design, implementation, and operation of manufac-
turing cells. Agrawal and Graves found that hierar-
chical organizational practices exacerbated the
separation between design, board fabrication, assem-
bly, and test operations and inhibited the implemen-
tation of lean practices in electronic manufacturing
organizations. Similarly, when product designers,
process engineers, and workers are isolated function-
ally, geographically, or across organizations, decen-
tralization of authority b e c o m e s much more
challenging, and the ability to effect change through
continuous improvement activities is more difficult.
This can also hinder kaizen (continuous improve-
ment) activities, which are foundational to lean (Detty
and Yingling 2000).
Today, the electronics industry faces many of the
challenges identified in these previous studies. Rap-
idly rising customer expectations, globalization of
both markets and competition, accelerating pace of
technological change, and rapid expansion of tech-
nology access are the norm for electronic manufac-
turers (Haughey 2002; NGM Project 1997). High
levels of competition and shifting market demand

have led to a distributed supply chain. With the ex-
tensive use of outsourcing, many electronic manu-
facturers are functionally specialized. In the assembly
of a printed circuit boards, for example, design, board
fabrication, assembly, and test are potentially dis-
persed both organizationally and geographically. To
be able to study the impact of market, operational,
and structural factors on electronic manufacturers, it
is necessary to assess the implementation and use of
lean manufacturing practices within an organization.
The remainder of this paper focuses on the develop-
ment of an assessment tool to enable this study and
the exploratory study completed to illustrate how the
assessment tool may be used.
Lean Assessment Tool Development
Both research papers and practitioner works were
examined to identify a comprehensive set of manu-
facturing practices considered to be essential in a lean
manufacturing enterprise and to identify existing sur-
vey instruments that could be used to complete a
broad-based assessment of the lean practices used
within an organization. The decision to engage in lean
practices is often part of an organization's manufac-
turing strategy. As such, an assessment of the level of
a lean implementation should include components of
both a tactical and strategic nature. Hayes and Wheel-
wright (1984) d e f i n e d key structural and
infrastructural decision categories to evaluate in de-
veloping a manufacturing strategy. The four struc-
tural c a t e g o r i e s i n c l u d e d capacity, facilities,
technology, and vertical integration. The four
infrastructural categories included workforce, qual-
ity, production planning/materials, and organization.
In an extension to this framework, Hayes, Wheel-
wright, and Clark (1988) discuss the importance of
external perspectives on manufacturing strategy.
Similarly, Miltenburg (1995) describes another frame-
work for manufacturing strategy linking "manufac-
turing levers" to m a n u f a c t u r i n g outputs. In
Miltenburg's framework, six manufacturing levers are
defined--human resources, organization structure
and controls, production planning and control, sourc-
ing, process technology, and facilities.
Consistent with this concept that manufacturing
strategies must be multidimensional, a central tenant
of the lean enterprise is that lean is multifaceted. This
conceptualization of lean as multidimensional is sup-
57
Journal ~f'Mam(/ctcturing S3'stems
Vol. 24/No. 1
2005
ported by a wide range of researchers (Womack,
Roos, and Jones 1990; Bentley, Nightingale, and
Taneja 2000; Cochran et al. 2000; Detty andYingling
2000). Previous researchers have taken an approach
of defining specific lean manufacturing practices and
relating them to different functions of the manufac-
turing enterprise. For instance, James-Moore and
Gibbons (1997) identified five key constructs--waste
elimination, flexibility, people, process control, and
optimization--in developing a "lean automotive
model." Other researchers have used similar models
to define lean practices associated with the manufac-
turing function, as well as to define the "lean enter-
prise" (Karlsson and Ahlstrom 1996a; Bentley,
Nightingale, and Taneja 2000). A lean enterprise is
defined as an organization that uses lean concepts
and practices not just in the manufacturing function
but in everything it does (Womack, Roos, and Jones
1990). Panizzolo (1998) defined six areas for inter-
vention within a manufacturing enterprise to study
the implementation of lean manufacturing practices
among Italian manufacturing plants--manufacturing
equipment and processes, shop-floor management,
new product development, supplier management,
customer relations, and workforce management. In
the next step of this study, copies of existing lean
survey instruments were obtained and reviewed in
an effort to identify an instrument or combination of
instruments that might be used for a broad-based as-
sessment of lean practices consistent with the
conceptualization of lean as multidimensional.
Review of Lean Surveys and Assessment Tools
A number of tools have been developed to help
organizational leaders assess their progress toward
becoming a lean enterprise. For this study, copies of
seven different assessment instruments were re-
viewed. Surveys have also been used by researchers
in previous research studies focused on linking lean
practices with organizational performance. Five dif-
ferent research surveys were identified and reviewed.
In some cases, the complete research instrument was
not included in the paper reviewed. Where possible,
the author was contacted, and a copy of the actual
survey instrument was obtained. Tables 1 and 2 sum-
marize the research surveys and the industrial assess-
ment tools reviewed. The research surveys tended to
focus on assessing a group of lean practices within a
higher-level construct, such as JIT or TQM. The seven
Journal o/'Manu[acturing Systems
Vol. 24/No. l
2005

S u m m a r y of Research Surveys Reviewed
Survey Identification
Fullerton, McWatters, and Fawson (2003)
Nightingale and Mize (2002),
Lean Enterprise Self-AssessmentTool
Perez and Sanchez (2000)
Panizzolo (1998)
Shah and Ward (2003)
Table 1
Description and Lean Aspects Included
This research was based on a surveydevelopedto measure the level of JIT imple-
mentation within an organization.Ten JIT elements were definedfor the research,
and 11 correspondingsurveyitems were developedto assess the level of JIT
implementation.
This research study describes the structure of an assessment tool created by the
Lean Aerospace Initiative.The tool includes three sections: lean transformation
leadership, life cycle processes, and enabling infrastructure.Fifty-fourlean practices
are included in the tool. These practices were chosen as being indicators of the
behaviors that lean organizationsshould engage in rather than being comprehensive.
This research was based on a field survey of automotivesuppliers in Aragon. Data
collection includedorganizationaldemographics (number of employees,age of
organization, sales, and ownership), source of technologyinnovation,use of flexible
production technologies(JIT), and workforceand workplaceflexibilitymeasures
(teams, job rotation, and training).
This research was based on field surveysof Italian manufacturersfrom a wide range
of industrial sectors. The surveyitems were developedto probe the implementation
of lean practices in six differentareas of intervention(process and equipment,
manufacturingplanning and control, human resources, product design, supplier
relationships, customerrelationships). Some of the lean practices includedwithin
the interventionareas includesetup reduction, cellularmanufacturing,error
proofing, leveledproduction,teams, design for manufacturability,JIT, total cost
supplier evaluation,on-timedeliveries, and customerinvolvementin product design.
This research study was based on an annual surveyof manufacturingmanagers in
1999 by publishers of bldustryWeek. The surveyincluded questions on the level of
implementationof 22 differentlean practices, includingpractices related to JIT,
TPM, TQM, and human resource management.

industrial assessment tools reviewed included many
different lean practices, but were inconsistent in the
practices included as well in the mechanism for evalu-
ating the level of implementation. As a result of this
review, a list of practices and a modified approach
for assessing the level of implementation of each prac-
tice were developed.
An overall structure integrating the multiple di-
mensions of a lean enterprise was synthesized from
these previously developed instruments. Within this
structure, six impact areas were defined and then used
as the basis for the development of survey questions
regarding the adoption of specific lean manufactur-
ing practices. The six areas incorporated into the sur-
vey were: manufacturing equipment and processes,
shop floor management, new product development,
supplier m a n a g e m e n t , customer relations, and
workforce management. Each impact area was fur-
ther specified by defining a family of supporting ac-
tivities or practices. A synthesized set of impact areas
and lean practices is important in assuring that the
survey assesses the adoption of an overall lean pro-
duction strategy with practices impacting the full ar-
ray of organizational activities. The specific practices
and item development structure are described further
in the next section.
Survey Item Development
A set of defining practices was developed for each
impact area. The set of practices and the correspond-
ing impact areas are summarized in Table 3. Although
a universal set of lean practices is not fully specified
in the research literature, several studies along with
the reviewed surveys and assessment tools were used
to develop a set of representative practices for each
of the impact areas. Womack, Roos, and Jones (1990)
documented the importance of setup time reduction,
error-proofing equipment, shop-floor organization,
total quality management (TQM), and value addition
in the context of manufacturing processes. Similarly,
other researchers have shown the importance of total
productive maintenance (TPM), cellular manufactur-
ing, cycle time reduction (James-Moore and Gibbons
1997; Panizzolo 1998; Bentley, Nightingale, and
Taneja 2000; Karlsson and Ahlstrom 1996a). In the
case of product development, the significance o f con-
current engineering techniques has been studied and
found to play an important role in a "lean" product

58
 Journal of Mamt~dcttH'ing Systems
Vol. 24/No. l
2005

Table 2
Summary of Seven Industrial Assessment Tools Reviewed

Survey Identification Description and Lean Aspects Included

Jordan and Michel (2001),
Survey of Perceptions of a
Company's Leanness
Lean Enterprise Implementation Group (1999),
The 360o Lean Audit
Lean Learning Center (2003),
The Lean Company Survey
Northwest High Performance Enterprise
Consortium (2002), HPEC Assessment
Robert Abair Associates, Inc. (2002),
Lean Checklist Self-Assessment
Wisconsin Manufacturing Extension Partnership
(2002), Lean Business Assessment
Wisconsin Manufacturing Extension Partnership
(2001), How Lean is Your Culture?
This tool is a 36-question survey used to assess a company's leanness.
Different versions of the survey are available for various stakeholders (executives,
employees, investors, suppliers, and customers). The combined set of surveys
addresses topics such as waste identification and reduction, continuous
improvement, value stream management, flow, and human resource topics such as
employee development and leadership.
This assessment tool is used to evaluate the level of implementation of policies,
process management, lean tools and techniques, and supply chain integration
activities. Lean tools and techniques included in the assessment are workplace
organization, waste, flow, pull, quality, standards, PDCA, equipment effectiveness
and reliability, and leveled production.
This benchmarking survey requests information on performance changes
attributable to lean or continuous improvement efforts, infrastructure details (who is
responsible for lean efforts), functional involvement in lean, and implementation
details (types of lean tools implemented).
This benchmarking tool is structured to identify a broad range of measurable
outcomes resulting from a lean implementation, including changes in management
practices, quality achievements, employee involvement, flexible manufacturing
practices, maintenance practices, inventory management processes, and new product
development processes.
This self-assessment tool allows organizations to establish a baseline
measurement of their lean progress. The tool includes a range of management and
lean practices, such as lean education, training, statistical process control, JIT,
kaizen, heijunka, 5S, SMED, poka-yoke, waste, workforce flexibility, performance
measures, and QFD.
This self-assessment tool is used to help organizations identify and prioritize
improvement activities. The tool addresses 10 lean principles and a range of lean
practices, including flow production, leveled mixed-model production, quick
changeover, automation with a human touch, pull systems, autonomous
maintenance, and kaizen reporting.
This short self-assessment tool is designed to help managers identify cultural
factors that can support or inhibit the sustainability of lean manufacturing
initiatives.

d e v e l o p m e n t process (Karlsson and Ahlstrom 1996b). p e r v a s i v e n e s s o f the 29 lean m a n u f a c t u r i n g p r a c t i c e s

T h e i m p o r t a n c e o f using c o n c u r r e n t e n g i n e e r i n g in
designing a new p r o d u c t has also been well estab-
lished ( W o m a c k , Roos, and Jones 1990; Takeuchi and
N o n a k a 1986; H e n k e , Krachenberg, and Lyons 1993).
W o m a c k , Roos, and Jones d e m o n s t r a t e d the i m p o r -
tance o f supplier relationships in a lean m a n u f a c t u r -
i n g c o n t e x t . F o l l o w - o n r e s e a r c h has f o u n d t h a t
investing in l o n g - t e r m relationships, evaluating sup-
pliers, and e x c h a n g i n g i n f o r m a t i o n are key aspects
o f a healthy relationship b e t w e e n a m a n u f a c t u r e r and
supplier ( C o n t r e r a s and M a s t 2001). T h e i m p o r t a n c e
o f evaluating the total costs in selecting a supplier
has also b e e n d o c u m e n t e d (Ellram 1994).
O n c e the task o f d e v e l o p i n g the i m p a c t areas and
defining the associated practices was c o m p l e t e d , in-
dividual s u r v e y items w e r e d e v e l o p e d to assess the
s u m m a r i z e d in Table 3. Six additional s u r v e y items
w e r e d e v e l o p e d and i n c l u d e d at the end o f the s u r v e y
to provide d e m o g r a p h i c i n f o r m a t i o n on both the c o m -
p a n y and the r e s p o n d e n t . T h e set o f d e m o g r a p h i c
items r e q u e s t e d i n c l u d e d total n u m b e r o f e m p l o y e e s ,
annual sales, products or services p r o d u c e d b y the
organization, and the position title, f u n c t i o n a l area,
and years o f service o f the r e s p o n d e n t . T h e final i t e m
o n the s u r v e y was a space for r e s p o n d e n t s to p r o v i d e
any c o m m e n t s related to either their o r g a n i z a t i o n ' s
i m p l e m e n t a t i o n o f lean o r any c o m m e n t s related to
the survey i n s t r u m e n t itself.
T h r e e - t o - s e v e n f o l l o w - o n items w e r e d e s i g n e d f o r
26 o f the 29 lean m a n u f a c t u r i n g practices listed in
Table 3. T h e s u r v e y was d e s i g n e d to direct r e s p o n -
dents to f o l l o w - o n items, o n l y if the overall p r a c t i c e

59
Journal ~/" Manq/~wturing Systems
Vol. 24/No. 1
2005

Table 3 After the initial survey development, an electronic

Mapping of Lean Manufacturing Practices to Six Impact Areas
copy of the survey was distributed and reviewed by
Lean Manufacturing
ImpactAreas four other researchers with expertise in the imple-
Principles and Practices
ManufacturingEquipment Setup time reduction mentation of lean practices and two researchers with
and Processes Work standardization expertise in survey development. Four of the review-
Cellularmanufacturing ers were associated with universities in the United
Mistake or errorproofing States, one with a European university, and the sixth
Valueidentification
with a university in South America. As a result of
Total productivemaintenance
Shop-floororganization this set of reviews, the number and wording of many
Total qualitymanagement of the follow-on items were modified. In general,
Cycletimereduction the review of the survey by this set of experts iden-
Shop-Floor Management Productionscheduling tified inconsistencies in terminology and in item
Lot size reduction
Pull flowcontrol structure. These inconsistencies were addressed. The
New ProductDevelopment Partsstandardization final instrument was 16 pages in length.
Concurrentengineering Exploratory Study Sample
Design for manufacturability
Supplier Relationships Supplier evaluation Company names and contacts for the exploratory
Total cost evaluation study were obtained using a variety of methods. Pub-
Informationexchange lished listings of manufacturers including Web site
Long-termrelationships listings, direct contact with the researchers, and con-
CustomerRelationships Deliveryperformance tacts established as a result of participation in pro-
improvement
Demandstabilization fessional societies/organizations were the three
Servicesto enhancevalue primary means of obtaining company contact in-
Customerrequirements formation. Pre-letters introducing the research and
analysis surveys were mailed to 27 companies engaged in
Product customization
electronics manufacturing located in the states of
WorkforceManagement Multifunctionalworkforce
Work delegation Oregon and Washington. Senior manufacturing per-
Employeeevaluation sonnel with broad visibility to organizational prac-
Pay for performance tices were the target recipients of the mailed survey.
Formal rewardsystem When specific contacts were not identified for a
company, the survey was addressed to the manu-
was used within the organization. The structure cho- facturing or process engineering manager. Four
sen for the survey items not only minimized the amount weeks after the initial mailing, a second survey and
of time to complete the survey, but also provided the cover letter were mailed to all companies in the
respondent with additional details on a particular prac- original sample who had not responded. Of the com-
tice through the follow-on items. Because the follow- panies receiving surveys, 40% are involved in ei-
on items referenced more specific aspects of the lean ther the m a n u f a c t u r e o f silicon wafers or in
practice in question, this reduced the likelihood that semiconductor manufacturing, 30% are involved in
the respondent would misunderstand the overall prac- the manufacture or assembly of printed circuit
tice. The structure also provided a means for deter- boards, and 30% are involved in the manufacture of
mining the practices being implemented, as well as subassemblies or equipment used by electronic
the details of how a particular organization was choos- manufacturers. Thirteen completed surveys were re-
ing to implement the practice. turned and used for all subsequent analyses. This
For each item, respondents were asked to evaluate corresponds to a response rate of approximately 48%.
how often a specific practice was used within their Electronic products are the result of a series of
organization. A five-point Likert scale (Always, Most complex manufacturing processes, including wafer
of the time, Some of the time, Rarely, or Never) was manufacture, semiconductor manufacture, printed
used. The content and responses for two of the overall circuit board manufacture, and board assembly. Each
lean practices, along with their more specific follow- of these segments of the electronics industry is
on items, are shown in Table 4. unique in the type of process used, but as a whole,

60

Table 4
Example Survey Items and Responses
Item
Does your organization make an effort to reduce setup times?
(If YES, then go to follow-on items)
In your effort to reduce setup times, how often do you engage in
each of the following practices?
• Encourage employees to find ways to reduce setup times
• Educate employees on the importance of reducing setup times
• Use checklists during the setup of a machine
• Use devices to reduce operator error during setup
• Complete setup operations offline
Does your organizationengagein work standardizationpractices?
(If YES, then go to follow-onitems)
How often do you engagein each of the followingpracticesto
facilitate standardizationof work?
• Use documentedstandardoperatingprocedures
• Train employeesto use standardoperatingprocedures
• Updatestandardoperatingprocedures
these manufacturers share two important character-
istics-utilizing high-technology processes and ma-
terials and being subject to rapid product development
cycle times and short product life cycles.
Exploratory Study Results and
Discussion
To test the usefulness of the research instrument
developed, an analysis of the survey results from the
exploratory study was completed. The primary inter-
est in the study was to develop an instrument that could
be used to assess both the type and extent of practices
being implemented within organizations as part of their
overall manufacturing strategy. As such, the extent of
implementation was determined for the entire sample
by calculating an average score and standard devia-
tion for each of the six impact areas. For each com-
pany, the implementation score was obtained by
averaging the level of implementation for all practices
associated with an impact area. Higher scores are in-
dicative of higher levels of implementation. The re-
suits are summarized in Table 5. The manufacturing
processes and equipment impact area and the customer
relations impact area received the highest average
scores (3.70) for the entire set of completed surveys.
New product development received the lowest aver-
age score (3.13). The amount of variability observed
was similar for each of the six impact areas.
61
Journa! ~["Mamgt~zcTuring Systellzs
Vol. 24/No. 1
2005
Response
NO or YES
RELY
ME
VEROF THE TIME
~OwAY
ST OF THE TIME
S
NO or YES
RELY
M E OF THE TIME ALWAYS
OST OF THE TIME
S
Based on demographic information provided by
the respondents, the number of employees was used
to stratify the data to allow comparison between
smaller and larger organizations. Larger companies
have more resources and a broader range of exper-
tise within their organizations. Both of these fac-
tors support a supposition that larger companies will
implement a larger variety and/or a greater number
of lean practices. The distinction between large and
small companies was based solely on the number of
employees working in the organization, as reported
by the survey respondent. Because company sizes
ranged from very small (less than 10 employees) to
very large (thousands of employees), large compa-
nies were defined as having 100 or more employ-
ees. Of the 13 companies included in this analysis,
five companies were identified as having fewer than
100 employees, and the remaining eight companies
had more than 100 employees. A two-sample, non-
parametric test (Mann Whitney) was used to com-
pare the two samples for each impact area. The Mann
W h i t n e y statistic (analogous to an i n d e p e n d e n t
sample t-test for normally distributed data) tests
whether or not two sampled populations are equiva-
lent in location. The Mann Whitney statistic is a
conservative test that is appropriate for samples with
any underlying distribution. The Mann Whitney test
was used to determine whether or not any differences
in the level of implementation for larger and small
Journal ~/'Mam(fitcturing Systems
Vol. 24/No. 1
2005

Table 5
Average and Standard Deviation for Each Impact 5.0
Area for Sample (n = 13)
Average
4.0
Implementation Standard
Impact Area Level Deviation
Manufacturing equipment
3.0
and processes 3.70 0.651 _=
Shop-floor management 3.49 0.606
New product development 3.13 0.627
_=
2.0
Supplier management 3.41 0.707 Size

Customer relationships 3.70 0.555 [ " ] Small

Workforce management 3.50 0.470 1.0 Dta~ge
Mfg Process New Prod Dev Cust Rel
Table 6
Shop Floor Supplier Mgmt Wkf Mgmt
Average Implementation Score for Small (n = 5) and
Large (n = 8) Companies
Small Large Figure 1
Impact Area Companies Companies p-value Comparison of Average Implementation Scores for Small
and Large Companies. (Lines on/in box represent upper
Manufacturing equipment quartile, median, and lower quartile.)
and processes 3.13 4.06 0.013
Shop-floor management 3.41 3.54 0.570
New product development 3.11 3.14 0.932 The implementation of lean practices has often
Supplier management 3.21 3.53 0.558
Customer relationships 3.42 3.87 0.242
been associated with discrete part manufacturers
Workforce management 3.34 3.60 0.462 (Shah and Ward 2003). The sample used for this study
included a mix of organizations with some being dis-
companies were statistically significant. The results crete part manufacturers and others being process-
are summarized in Table 6. based manufacturers. To study whether or not the type
There is evidence that larger companies included of manufacturing organization was related to the level
in this sample have implemented lean practices to a of lean implementation, each organization included
greater extent than smaller companies responding to in the study was classified into one of three catego-
this survey. In particular, a significant difference in ries based on survey responses--printed circuit board
the average implementation of lean practices associ- manufacturer or assembly, equipment manufacturers,
ated with manufacturing equipment and processes wafer or semiconductor manufacturers. This group-
was found (p-value = 0.013). The level of implemen- ing of companies was chosen because the technical
tation is summarized graphically in Figure 1. One of systems and business models for these types of manu-
the large companies responding to the survey, for facturers differ. For example, the level of capital in-
example, indicated that their organization had spent vestment necessary for wafer and semiconductor
the last year "introducing lean methods to the floor manufacturing is significantly higher than the capi-
and management staff," and resources were being tal investment necessary for surface mount assembly
assigned to begin a "three to four year full imple- or equipment assembly.
mentation of lean practices" at the beginning of the Organizations involved in circuit board manufac-
organization's new fiscal year. These findings are turing and assembly had the lowest average imple-
consistent with previous research. Shah and Ward m e n t a t i o n score in all six i m p a c t areas. A
(2003) found that larger plants across a variety of Kruskal-Wallis test (the non-parametric analog of
industrial sectors are more likely than smaller plants one-way analysis of variance) was performed to de-
to implement lean practices. Similarly, Shah and Ward termine whether or not observed differences were
did not find a significant difference in the implemen- significant. The average scores and the p-values for
tation likelihood of crossfunctional workforce prac- the Kruskal-Wallis statistic are summarized in Table
tices based on organizational size. No difference was 7. Implementation scores by the type of manufactur-
found between small and large organizations in this ing organization are also displayed in Figure 2. A
study related to workforce management practices. The marginally significant difference was found for the
similarity in findings provides initial validation of manufacturing equipment and processes impact area
the new survey. (p-value = 0.089).

62
 Jottrltal of Mam(fitcturillg Systems
Vol. 24/No. 1
2005

Table 7
Average Implementation Score for Printed Circuit and Assembly (u = 4), Equipment (n = 4),
and Wafer or Semiconductor (u = 5) Manufacturers
Printed
Circuit Wafer and
and Equipment Semiconductor
Impact Area Assembly Manufacturers Manufacturers p-value
Manufacturing equipment and processes 3.41 3.44 4.15 0.089
Shop-floor management 3.14 3.74 3.61 0.493
New product development 2.53 3.34 3.31 0.137
Supplier management 2.86 3.80 3.53 0.205
Customer relationships 3.25 3.85 3.94 0.180
Workforce management 3.30 3.43 3.72 0.291

Variations in operational strategies resulting from

5.0 ¸
differences in product volume and product variety
may explain this finding. Narrative comments from aa

one survey respondent (a large organization) involved o 4.0

in the assembly of complex printed circuit boards

supported this interpretation. "We are currently mak- =
3.0

ing our second attempt at lean. Our business, like all E

high-tech manufacturing, rides the economical roller Organization

2.0
coaster...Many of the advantages an OEM has avail- [ ' - ~ P C B & Assembly

able are not available to us, as many of our processes
are dictated by our customers." Perez and Sanchez
(2000) found that the level of adoption of flexible
production technologies differed between compo-
nent-level and subsystem assembly automotive sup-
pliers, but they did not find significant differences
for the other practices.
At the item level, average scores and standard de-
viations for 26 of the original 29 items are summa-
rized in Table 8. Question 27 (Does your organization
have a formal employee evaluation system or not?),
question 28 (Does your organization use a perfor-
mance related pay system, or not?), and question 29
(Does your organization have a formal reward sys-
tem for personnel who perform well, or not?) did not
use Likert scales. The average implementation scores
for work standardization, shop-floor organization
(5S), and total productive maintenance (TPM) indi-
cate that the respondents were implementing these
practices most of the time; whereas design for
manufacturability (DFM) and mistake or error-proof-
ing practices were implemented to a lesser extent by
the responding organizations.
At the item level, all respondents indicated that
supplier evaluation, delivery performance improve-
ment activities, and multifunctional workforce de-
velopment were being implemented. On the other
hand, fewer than 40% of the respondents indicated
that they practiced total supplier cost evaluation, mis-
~Equip M fg
1.0 ~Wafer & Semi
Mf$ Process Ne~ Prod Dev Cust R¢I
Shop Floor Supplier Mgmt Wkf Mgmt
Figure 2
Comparison of Average Implementation Scores for Companies
Based on the Type of Manufacturing Organization. (Lines on/in
box represent upper quartile, median, and lower quartile.)
take or error proofing, or design for manufacturability.
The percentages of respondents implementing at least
some of the activities associated with a particular
practice are summarized for all 29 items in Table 8.
Non-parametric testing was completed at the item
level to ascertain which specific practices demon-
strated significant differences based on company size
and the type of manufacturing organization. The level
of implementation was found to be significantly dif-
ferent between smaller and larger companies for three
practices (value identification, cycle time reduction,
and production scheduling improvement). Significant
differences were observed for two practices when the
type of manufacturing organization was used as the
grouping variable (cycle time reduction and customer
requirements analysis). These results are summarized
in Tables 9 and 10, respectively.
In addition to completing the structured survey
questions, a number of respondents also added nar-
rative comments, which provide additional insight
into the implementation levels and the percentages

63
Journal of Manz([~lcturing Systems
Vol. 24/No. 1
2005

Table 8
Summary Statistics and Percentage of Organizations Implementing Specific Practices by Item
Lean Practice Percentage Average Standard Deviation
Supplier evaluation 100% 3.21 1.02
Delivery performance improvement 100% 3.96 0.70
Multifunctional workforce 100% 3.69 0.53
Work standardization 92% 4.70 0.58
Value identification 92% 3.39 0.41
Cycle time reduction 92% 3.68 0.97
Product customization 92% 3.91 0.69
Employee evaluation 92% N/A N/A
Setup time reduction 85% 3.76 0.53
Total quality management 85% 3.44 0.99
Production scheduling improvement 85% 3.54 0.81
Information exchange with suppliers 85% 3.53 0.71
Long-term relationships with suppliers 85% 3.76 0.80
Customer requirements analysis 85% 3.61 1.05
Pay for performance 83% N/A N/A
Lot size reduction 77% 3.34 0.83
Pull flow control 77% 3.70 1.08
Work delegation 77% 3.37 0.53
Total productive maintenance 69% 4.04 0.89
Parts standardization 69% 3.18 0.88
Formal reward system 69% N/A N/A
Concurrent engineering 62% 3.40 0.65
Demand stabilization 62% 3.54 0.85
Services to enhance product value 62% 3.58 0.76
Shop-floor organization (5S) 58% 4.24 0.66
Cellular manufacturing 54% 3.39 1.02
Design for manufacturability 38% 2.67 0.85
Total cost evaluation 31% 3.12 0.64
Mistake or error proofing 27% 2.67 0.29

Table 9
of organizations implementing specific practices. For Average Lean Practice Implementation Score for Small
(n = 5) and Large (n = 8) Companies
example, one respondent involved in the assembly
of printed circuit boards indicated that design-for- Lean Practice and Small Large
Impact Area Companies Companies p-value
manufacturability practices were not implemented in
Value identification
their organization. "Because we are a contract manu- (Manufacturing equipment
facturer, we don't do any design, but we do give feed- and processes) 2.97 3.68 0.004
back to our customers on their designs, but we don't Cycle time reduction
have a formal system." Similarly, another contract (Manufacturing equipment
manufacturer involved in the assembly of printed cir- and processes) 2.80 4.13 0.026
cuit boards indicated that some shop-floor manage- Production scheduling
improvement (Shop-floor
ment practices (pull production) associated with lean management) 2.87 3.92 0.047
were not easily applied within their business envi-
ronment. "JIT (just-in-time or pull production) meth- The differences in the level of implementation,
ods d o n ' t work well in contract m a n u f a c t u r i n g while not statistically significant for all impact ar-
because we don't have a reliable schedule for more eas, are quite interesting when put in the context of
than a week and often less..." Similarly, for smaller the unique challenges faced by the different types of
organizations involved in contract manufacturing, manufacturing organizations included in the explor-
some lean practices, such as cellular manufacturing, atory study. For example, wafer and semiconductor
become much more challenging. "As a small com- manufacturing organizations had the highest average
pany with a lot of different customers and a schedule scores for the manufacturing equipment and processes
that changes all the time, we don't have enough con- impact area, the customer relations impact area, and
sistency to set up cells" the workforce management impact area. Wafer and

64

Table 10
Average Lean Practice Implementation Score for Printed Circuit and Assembly (n = 4), Equipment (n = 4),
and Wafer/Semiconductor (n= 5) Manufacturers
Lean Practice and Impact Area
Cycle time reduction (Manufacturing equipment and processes)
Customer requirements analysis (Customer relations)
semiconductor manufacturing are process-intensive
industries. Thus, the higher level of implementation
of lean practices focused on manufacturing equip-
ment and processes is not unexpected. The organiza-
tions in this category were also all large organizations,
for example, more than 100 employees. The higher
level of implementation of lean practices focused on
workforce management may be related to having or-
ganizational resources available, such as human re-
s o u r c e specialists. E l e c t r o n i c e q u i p m e n t
manufacturers' average scores were highest for shop-
floor management, new product development, and
supplier management. Equipment manufacturers uti-
lize a variety of assembly processes and often rely
on outsourcing for subassemblies and components.
These factors may explain, in part, the higher imple-
mentation levels for practices related to shop-floor
management and supplier management. Customers
of electronic manufacturing equipment are faced with
rapidly changing requirements, which may force
equipment manufacturers to design equipment that
will meet new product and process requirements. The
higher level of implementation of practices associ-
ated with new product development may be a result
of this cycle.
Conclusions
Based on both empirical and anecdotal evidence
contained in the lean literature, it could be argued
that electronic manufacturers are subject to a variety
of challenging conditions that limit the applicability
of lean production practices. This exploratory study
established that even though these conditions may
limit the applicability of lean practices, all of the or-
ganizations in this study had implemented lean prac-
tices to some extent. The results of this study suggest
that organizational size and the type of manufactur-
ing may be significant factors.
As a result of the survey development and subse-
quent exploratory study, three opportunities for fu-
ture research were identified. First, significant
65
Journal ~/'Manz(fitcturing Systems
Vol. 24/No. 1
2005
Wafer and
Printed Circuit Equipment Semiconductor
and Assembly Manufacturers Manufacturers p-value
3.27 2.95 4.52 0.016
2.63 4.25 4.15 0.048
differences were not observed for all impact areas or
between all subcategories of organizations. A larger
study with multiple organizations for each of the or-
ganizational types being compared would provide
insight into whether or not the number and extent of
lean practice implementation differs when organiza-
tional characteristics are taken into account.
The second opportunity is related to the adminis-
tration of the survey. Specific definitions were not
provided for the items included in the survey. While
the items being rated were validated by a small num-
ber of researchers familiar with lean and by survey
development experts, it is possible that respondents
were unfamiliar with the terms used. It is also pos-
sible that respondents might be unable to accurately
assess the level of implementation of a specific prac-
tice because it was identified within the target orga-
nization with a different name. One indication of
terminology ambiguity is incomplete surveys or sur-
veys with missing responses. These problems were
not observed in the surveys returned for this explor-
atory study. Respondents did not indicate any con-
cern with t e r m i n o l o g y or d e f i n i t i o n s in the
open-ended request for feedback at the end of the
survey. The issue of respondent knowledge and varia-
tion in item interpretation are potential threats to the
validity of the findings. One way to mitigate this threat
would be to include a glossary of terms used with the
survey instrument.
The third opportunity is related to the selection of
a simple manufacturing sector to validate the instru-
ment. While some of the relationships found are con-
sistent with previous research involving other
industrial segments, a larger study is needed before
applying the instrument to a broader spectrum of
manufacturing organizations. The findings from the
exploratory study warrant further investigation of the
relationships between organizational characteristics
(product vs. process focus, volume, variety, market
drivers, and so on) and the implementation of lean
practices. Such research would help determine if the
 Journal ~/' Mant([~tcturing Systems
Vol. 24/No. 1
2005
differences observed hold true for a larger sample of
electronic manufacturers or for a broader cross sec-
tion of manufacturing organizations.
The primary contribution of this paper was to de-
scribe the development and validation of a survey to
measure both the number and level of adoption of
lean practices by an organization. A review of 12 dif-
ferent surveys and assessment tools was completed
along with a review of related literature on the po-
tential limitations of a lean manufacturing strategy.
A synthesis of this information was used to define
six lean impact areas and to identify 29 unique lean
practices. Results from an exploratory study were also
analyzed and discussed. The exploratory study
sample, while small, did encompass a variety of
manufacturing organizations from the electronics
industry. The survey did provide a mechanism for
comparing the lean implementation strategies of dif-
ferent organizations within this industrial sector. By
comparing both the variety and extent of lean prac-
tices being implemented, this study has demonstrated
that it is possible to empirically evaluate the applica-
bility of lean practices to different manufacturing
organizations. The findings from this research have
been used to identify opportunities for further devel-
opment of the assessment tool and to justify the ad-
ministration of the survey to a larger number of
organizations.
Acknowledgments
This study was supported by the National Science
Foundation under grant No. SES-0217866.
References
Agrawal, A. and Graves, R.J. (1999). "A distributed systems model for
estimation of printed circuit board fabrication costs." Production Plan-
ning and Control (vl0, n7), pp650-658.
Bamber, L. and Dale, B.G. (1999). "Lean production: A study of applica-
tion in a traditional manufacturing environment." Production Plan-
ning and Control (vl 1, n3), pp291-298.
Bentley, G.; Nightingale, D.; and Taneja, A. (2000). "Lean Enterprise
Model." http ://lean.mit.edu/events/workshops/I/iles ~oubl ic/Le lem.pdf
Cochran, D.S.; Eversheim, W.; Kubin, G.; and Sesterhenn, M.L. (2000).
"The application of axiomatic design and lean management principles
in the scope of production system segmentation." lnt'l Journal of Pro-
duction Research (v38, n6), pp1377-1396.
Contreras, L. and Mast, M. (2001). Lean Strategies Group LLC,
www. leanstrategies, com.
Cooney, R. (2002). "Is "lean" a universal production system?" hlt'l Join=
hal ¢~f"Operations & Production Mgmt. (v22, n 10), pp 1130-1147.
Detty, R.B. and Yingfing, J.C. (2000). "Quantifying benefits of conversion to
lean manufacturing with discrete event simulation: A case study." lnt'l
JoulTud of PirMuction Research (v38, n2), pp429-445.
Ellram, L. (1994). "Total Cost Modeling in Purchasing." Center for Ad-
vanced Purchasing Studies, www.capsresearch.org/ReportHTMs/
tem.htm).
66
Fliedner, G. and Vokurka, R. (1997). "Agility: Competitive Weapon of
the 1990s and Beyond?" Production and Inventory Mgmt. Journal
(Third Quarter), pp19-24.
Fullerton, R.R.; McWatters, C.S.; and Fawson, C. (2003). "An examina-
tion of the relationship between JIT and financial performance." Jour-
nal o f Operations Mgmt. (v21), pp383-404.
Haughey, J. (2002). "Market pulse: electronics industry crawls in second
quarter." EBN (Sept. 24, 2002).
Hayes, R.H. and Wheelwright, S.C. (1984). Restoring our Competitive
Edge, New York: John Wiley and Sons.
Hayes, R.H.; Wheelwright, S.C.; and Clark, K.B. (1988). Dynamic Manu-
facturing: Creating the Learning Organization. New York: The Free
Press.
Henke, J.; Krachenberg, R.; and Lyons, T. (1993). "Cross-functional teams:
Good concept, poor implementation?" Journal of Product hmovation
Mgmt. (vl0, n3), pp216-229.
Houlahan, C.J. (1994). "Reduction of front-end loading of inventory: Mak-
ing the airframe industry lean through better inventory management"
Cambridge, MA: Massachusetts Institute of Technology.
James-Moore, S.M. and Gibbons, A. (1997). "Is lean manufacture uni-
versally relevant? An investigative methodology." hlt'l Journal of Op-
erations and Production Mgmt. (v17, n9), pp899-911.
Jordan, J.A., Jr. and Michel, EJ. (2001). The Lean Company: Making the
Right Choices. Dearborn, MI: Society of Manufacturing Engineers.
Karlsson, C. and Ahlstrom, P. (1996a). "Assessing changes toward lean
production." hzt'l Journal of Operations and Production Mgmt. (vl6,
n2), pp24 -21.
Karlsson, C. and Ahlstrom, P. (1996b). "The difficult path to lean product
development." Jounud t~/ PnMuct hmow~tion Mgmt. (n 13), pp283-295.
Katayama, H. and Bennett, D. (1996). "Lean production in a changing
competitive world: A japanese perspective." htt'l Journal of Opera-
tions and Production Mgmt. (vl6, n2), pp8-23.
Laughlin, K. (1995). "Increasing competitiveness with a cellular process."
hzdustrial Engg. (April), pp30-33.
Lean Aerospace Initiative ¢2002). http://leun.mit.edu/.
Lean Enterprise Research Center (1999). www.cfac.uk/carbs/Iondlerc/
index.htnd.
Lean Learning Center (2003). wwn:leanlearningcentet:com/.
MacDuffie, J.P.; Sethuraman, K." and Fisher, M.L. (1996). "Product variety
and manufacturing performance: Evidence from the international auto-
motive assembly plant study." Mgmt. Science (v42), pp350-369.
Mason-Jones, R.; Naylor, B.; and Towill, D.R. (2000). "Lean, agile, or
leagile? Matching your supply chain to the marketplace." htt'l Jour-
nal of Production Research (v 16, n2), pp4061-4070.
Miltenburg, J. (1995). Manu]~wturing Strategy: How to Formulate and
bnplement a Winning Plan. Portland, OR: Productivity Press.
Nicholas, J. (1998). Competitive Manufacturing Management: Continu-
ous hnprovement, Lean Production, and Customer-Focused Quali~.
Boston: Irwin McGraw-Hill.
Nightingale, D.J. and Mize, J.H. (2002). "Development of a lean enter-
prise transformation maturity model." ln/ommtion, Knowledge, Sys-
tems Mgmt. (v3), pp15-30.
Northwest High Performance Enterprise Consortium (2002).
www.nwhpec.org/.
NGM Project (1997). Next Generation Manttfiwturing: Framework Jbr
Action. Bethlehem, PA: Ability Forum.
Panizzolo R. (1998). "Applying the lessons learned from 27 lean manu-
facturers. The relevance of relationship management." lnt'l Journal
of Production Economics (v55), pp223-240.
Perez, M.P. and Sanchez, A.M. (2000). "Lean production and supplier
relations: A survey of practices in the Aragonese automotive indus-
try." Technovation (v20), pp665-676.
Robert Abair Associates, Inc. (2002). www.robet'tabairassociates.com.
Shah, R. and Ward, P.T. (2003). "Lean manufacturing: Context, prac-
tice bundles, and performance." Journal t~/'Operations Mgmt. (v21),
pp129-149.
Takeuchi, H. and Nonaka, I. (1986). "The new new product development
game." Harvard Business Review (Jan.-Feb.), pp137-146.

Wisconsin Manufacturing Extension Partnership (2001). ww~:wmep.orgZ
Womack, J. and Jones, D. (1993). Lean Thinking: Banishing Waste and
Create Wealth in Your Corporation. New York: Free Press.
Womack, J. and Jones, D. (2003). Lean Thinking: Banishing Waste and
Create Weulth in Your Colporation. New York: Free Press.
Womack, J.; Roos, D.; and Jones, D. (1990), The Machine That Changed
the World. New York, NY: Rawson and Associates.
Yauch, C.A. and Steudel, H.J. (2002). "Cellular manufacturing for small
business: Key cultural factors that impact the conversion process."
Journal of Operations Mgmt. (v20), pp593-617.
Yusuf, Y.Y. and Adeleye, E.O. (2002). "A comparative study of lean and
agile manufacturing with a related survey of current practices in the
UK." hzl '1 Journal of Plvduction Research (v40), pp4545-4562.
Authors' Biographies
Dr. Toni L. Doolen is an assistant professor in the Industrial and Man-
ufacturing Engineering Dept. at Oregon State University. She received a
BS in material science and engineering and a BS in electrical engineering
from Cornell University in 1987. In 1991, she received an MS in manu-
facturing systems engineering from Stanford University. She received her
PhD in industrial engineering from Oregon State University in 2001. Her
67
Journal q(Mam([acturing ~,stems
Vol. 24/No. 1
2005
research is focused on manufacturing systems design, lean manufactur-
ing, work group effectiveness, and mobile technology in education. Prior
to seeking her PhD, she gained 12 years of manufacturing experience at
Hewlett-Packard Co. as an engineer, senior member of technical staff,
and manager.
Dr. Maria E. Hacker is an associate professor at Oregon State Univer-
sity-Cascades Campus. She teaches both industrial engineering and busi-
ness management courses. Prior to returning to the university to earn her
doctorate at Virginia Tech, she was a plant manager at one of Procter and
Gamble's largest manufacturing facilities. She now teaches, creates, ad-
vises, and applies methods for achieving organizational performance im-
p r o v e m e n t t h r o u g h performance m e a s u r e m e n t s y s t e m s , p r o j e c t
management tools, customer service interventions, total quality manage-
ment, and high-performance work team initiatives. Her work has been
published in many journals, including: 1EEE Transactions on Engineer-
ing Management; Operations & Production Munagement; Quali~ Mun-
agement Journal; Public Sector Management; National Productivity
Review; Munaging Service Quality; Measuring Business Excellence; and
Team PerJbrmance Management. Her experience includes leading and
facilitating improvement activities in public, not-for-profit, government,
and international organizations.