An Historical Guide to the

WIA Performance Measures

Georgia Department of Labor

February 2007
February 2007

Please note: This guide was created after the initial enactment of WIA in 2000. With the
implementation of Common Measures in 2006, certain elements contained within this document
have become obsolete. For the most current information regarding performance that all local
areas within Georgia should be following, please refer to the GDOL Common Measures
Resources Guide, which was distributed on March 2006, and the USDOL Training and
Employment Guidance Letter (TEGL) 17-05. This TEGL may be accessed on the DOL-ETA
web site at http://wdr.doleta.gov/directives/attach/TEGL17-05.pdf

1
February 2007

Introduction

This guide has been prepared to help local partners involved in planning for WIA to understand
the WIA performance measures. Basic information is presented in “Q&A” format in the main
body of the guide. More detailed or technical information is contained in appendices that we
hope will be useful to those most directly involved in performance measurement.

While some important questions regarding common performance measurement remain

unanswered at this time, this guide includes what we know now and lays out what you can
expect to happen next. We will update and expand the guide in the months to come.

Concern has been expressed about placing emphasis on performance measurement at this point
in the development of a program that should be customer-driven. We hope that the process of
designing local programs will be highly collaborative, focusing on customer needs and the
resources available for meeting those needs. Within that process, the performance measures
should be seen as parameters within which programs must operate − much like allocation size
and applicable laws and regulations.

Although we expect that the performance measures are meaningful indicators of how well our
workforce investment system is serving its customers, performance on these measures should
never become an end in itself. Rather, it should be a product of well-designed and well-
implemented systems, responsive to local needs and both building on and expanding local
opportunities.

2
February 2007

Understanding the WIA Performance Measures

1. Why are the WIA performance measures important?

The WIA performance measures are one component in the comprehensive WIA performance
accountability system. Other components include the system for identifying eligible training
providers, the related “consumer report card,” and shared outcomes and comparable performance
measures among partner programs. Together, these components focus attention on:

The importance of performance accountability at all levels

The interdependence of partner programs
The right of the public to valid consumer information and quality service

Because outcomes are shared across all programs with a common customer, collaborating
programs have a strong interest in one another’s success. At the same time, incentives and
sanctions at each level create consequences for performance. Providers may be removed from
the eligible provider list for poor performance or attract customers with strong performance.
Local areas may be subject to corrective actions for failure to meet their negotiated performance
levels, but states may give incentive awards for high levels of performance. States may have
their allocations reduced for failure to meet negotiated performance levels, or earn incentives
generated from funds withheld from other states. Each level’s success depends on the level
below, so each level has an interest in doing what it can to support and promote that success.

But most of all, the performance measures are important because they represent the aspirations
of our customers − for skill development and acquisition of meaningful credentials, for stable
employment over time, for improved earnings, and for positive experiences when they seek our
help.

2. What are the performance measures and to whom do they apply?

The WIA performance measures are listed in Table 1, along with the groups of customers to
which each measure applies. Here are the rules to remember:

• Except for the two customer satisfaction measures, the performance measures apply
separately to customers served under three funding streams: adults, youth, and dislocated
workers.

• Within youth services, measures apply separately to older (19 to 21) and younger (14 to
18) youth, based on the age of the customer at registration.

• Youth age 18 through 21 may be served as youth, adults, or both.

• Outcomes of customers served under more than one funding stream are counted under
each appropriate set of performance measures.

3
February 2007

• Customer satisfaction measures apply across all funding streams to two groups of
customers: employers and participants.

Table 1
WIA Performance Measures and Applicability by Customer Group
Youth
Older Younger
Dislocated Youth Youth
Performance Measures Adults* Workers (19 – 21) (14 – 18) Employers
Entered Employment Rate X X X
Employment Retention Rate X X X
Earnings Change in Six Months X X
Earnings Replacement Rate X
Employment & Credential Rate X X
Credential Rate X
Skill Attainment Rate X
Diploma or Equivalent Rate X
Retention Rate X
Customer Satisfaction Rates:
Participants X X X X
Employers X
* Includes youth age 18 to 21 served as adults.

3. Within each customer group, are all customers counted in the performance
measures?

No, only those customers who are registered under WIA are counted in the performance
measures.

4. Which customers must be registered under WIA?

• Local areas may draw on a variety of resources in providing workforce services, but only
customers who receive services funded under the WIA must be WIA-registered.

• All youth receiving services of any level under the Title I youth funding stream must be
registered under WIA.

• Among adults and youth age 18 to 21 served under the adult or dislocated worker funding
streams, WIA registration depends on the level of service received. Only those
customers receiving core services with significant staff assistance, intensive services, or
training funded by WIA must be WIA-registered.

4
February 2007

5. What types of services are included in “core services with significant staff
assistance, intensive services, and training”?

Table 2, below, lists types of WIA services and provides examples of each. Note that Georgia
has not identified any core services that involve “significant staff assistance,” so registration is
effectively limited to recipients of WIA-funded intensive services and training.

WIA Service Levels and Registration Requirements
WIA Service Level
Core Information
Core Self-Service
Core Staff-Assisted
(Minimal Intervention)
Table 2
WIA Registration
Service Examples Required?
ƒ General information such as pamphlets or No
directions
ƒ Resources such as books or videos
ƒ Labor market information
ƒ Consumer Report information (hard copy)
ƒ Training provider and local Workforce
Investment Board program performance
reports (hard copy)
ƒ Information on filing Unemployment
Insurance claims
ƒ Job fair information
ƒ Orientation to the one-stop
ƒ Internet browsing
ƒ Resources on-line, including: No
ƒ Information on job availability, including
Job Information System
ƒ Information on partner agency and local
community services
ƒ Information on support service availability
ƒ Information on financial aid
ƒ Self-directed assessment, Georgia Career
Information System
ƒ Labor market information
ƒ Consumer report information
ƒ Training provider and local Workforce
Investment Board program performance
reports
ƒ Information on filing Unemployment
Insurance claims
ƒ Internet registration for America’s Job No
Bank, etc.
ƒ Outreach
ƒ Group workshops in interviewing, job
search, resume writing, financial
management, etc.
ƒ Job fairs

5
February 2007
WIA Service Levels and Registration Requirements
WIA Service Level
Core Staff-Assisted
(Moderate Assistance)
Intensive Services
(Significant Assistance)
Training Services
Table 2
WIA Registration
Service Examples Required?
ƒ Initial assessment No
ƒ Service needs evaluation
ƒ Information and referral to services
ƒ Job matching, job referrals, job search
assistance, or job development
ƒ Job clubs
ƒ Career guidance, Georgia Career
Information System
ƒ Follow-up activities and reassessment for
services
ƒ Filing Unemployment Insurance claims
and Worker Profiling
ƒ Unemployment Insurance Eligibility
Review Program
ƒ Comprehensive, in-depth and diagnostic Yes, if WIA funded.
assessment
ƒ In-depth interviewing and evaluation
ƒ Development of individualized
employment plan
ƒ Individual or group career counseling
ƒ Service coordination or case management
ƒ Basic workforce readiness, pre-vocational
skills
ƒ Adult basic education, GED preparation
ƒ Out-of-area job search, relocation
assistance
ƒ Internships, work experience
ƒ Referrals to training
ƒ Intensive job development
ƒ Occupational skills training Yes, if WIA funded.
ƒ On-the-job training
ƒ Programs that combine workplace training
with related instruction
ƒ Training programs operated by the private
sector
ƒ Skill upgrading and retraining
ƒ Entrepreneurial training
ƒ Job readiness training
ƒ Adult education and literacy activities in
combination with other training services
ƒ Customized training
6
February 2007

6. Are WIA registrants who are receiving services from a partner program, such as
Technical Education, Adult Literacy, or Vocational Rehabilitation, still considered
active WIA participants?

Yes, WIA registrants continue to be considered Tracking Participation in

active WIA participants as long as they are receiving
any partner services. However, care must be taken to
maintain current records on service participation
through case management or tracking systems.
(Mandatory and optional partner programs are listed
in Appendix A.)
7. When are the outcomes of WIA customers
counted?
• Outcomes are counted and reported quarterly.
• For all measures except the skill attainment
rate for younger youth (age 14 to 18) and the
employer customer satisfaction measure,
measures are based on customers exiting in
the quarter being reported.
• Customers who exit, return, and exit again
are counted once in each quarter in which
they exit.
• USDOL has distinguished between “hard”
and “soft” exits. Hard exits occur upon case
closure or completion. Soft exits occur when
a registrant has not received any WIA or
WIA partner funded service for 90 days and
is not scheduled to receive any future
services. Because soft exits occur
automatically, it will be important for local
areas to maintain current data on customer services, including partner services.
Partner Programs
Suppose a customer enrolls in a technical
institute following WIA-funded intensive
assessment and career counseling at his local
one-stop. The one-stop assigns a case
manager to maintain contact with the
participant, offering additional services as
needed and recording the customer’s activities
and progress. When the customer finishes
training and goes to work, the case manager
records placement data and information on the
credential earned and closes the case (hard
exit). Subsequent employment and earnings
are tracked automatically through UI wage
record match.
But what if the one-stop doesn’t follow-up
with the customer while he was in training?
After 90 days with no record of receipt of
services, the customer’s case would be closed
automatically (soft exit) and, again, subsequent
employment and earnings would be tracked.
Even if the customer worked part time while
he was in school, the effect on the local area’s
performance would be negative since post-
participation earnings would be low and the
area would not receive credit for the credential
when it was earned. The bottom line is that
partners need to communicate about their
shared customers.

• A customer’s exit date is the last date of WIA funded or partner service. It determines his
or her exit quarter. (Notice that soft exits will consistently generate exits in the previous
quarter.)

• Planned gaps in service of more than 90 days due to a delay in the beginning of training
or a medical condition that prevents an individual from participating will not trigger a
soft exit.

7
February 2007

• The skill attainment rate is not an exit-based measure, so skill attainments are counted as
they occur.

8. WIA mandates the use of Unemployment Insurance wage data. How will that
work?

The following WIA measures use wage record data:

• Entered employment rate

• Employment retention rate
• Earnings change
• Earnings replacement rate
• Employment and credential rate

Nationally, over 90 percent of all workers are covered under Unemployment Insurance (UI).
Each quarter, their employers must report the total wages paid to each employee, by Social
Security number. The Georgia Department of Labor administers the UI program for our state and
will routinely match these wage reports to WIA customer records at the state level.

For the WIA performance measures, a customer is considered “employed” in a given quarter if
he or she has any reported earnings for that quarter, regardless of the amount earned. However,
because extremely high values can be indicators of problems such as duplicate data entry,
typographical errors, or use of the same Social Security by more than one person, records with
unusually high earnings levels will need to be examined and may be deleted.

The WIA measures use five quarters of wage record data: the 2nd and 3rd quarters prior to the
quarter of registration (or, in the case of dislocated workers, quarter of dislocation), and the 1st,
2nd, and 3rd quarters following the quarter of exit.

There can be up to a three quarter delay between the quarter in which wages are earned and the
quarter in which data on those earnings are available. For example, earnings for the 1st post exit
quarter for WIA registrants who exit during July through September 2000 may not be available
until the July to September 2001 quarter. Complete performance data will not be available for
this group until the January through March 2002 quarter. Clearly, UI wage record based
measures are not useful for day-to-day program management, and local areas will need to look at
other indicators of performance.

9. What about employment that is not covered by Georgia’s Unemployment Insurance

program, such as federal employment, self-employment, or employment in another
state?

Because, in some states, a significant percentage of the workforce is not covered by Unemploy-
ment Insurance, USDOL will permit − but not require − the use of supplemental data sources for
reporting employment. USDOL has issued guidelines regarding use of supplemental data (see
Appendix B, USDOL Instructions on Reporting Supplemental Data). These include the
following:

8
February 2007

• Supplemental data may be obtained either by contact with participants or their employers
or by record sharing or automated record matches with other employment-related data
bases.

• Such data must be recorded within 30 days after it becomes clear that there is no wage
record for an individual.

• Supplemental data may be used in calculating entered employment, employment reten-

tion, and employment and credential rates, but not in calculating earnings change or
earnings replacement rate.

• All data and methods used to supplement wage data must be documented and are subject
to audit.

Before deciding whether to collect supplemental data, local areas may wish to consider the
following:

• Georgia’s UI coverage rate is close to the national rate.

• A federal clearinghouse designed to facilitate interstate wage matching is being

developed, and Georgia is participating as a pilot state.

• Local areas are not required to take non-covered employment into consideration in setting
proposed levels of performance.

10. How are the WIA performance measures calculated?

In general, calculation of the WIA performance measures is complex, with subtle variations in
the same or similar measures across the five customer groups − adults, older youth, younger
youth, dislocated workers, and employers. Operational definitions for the performance measures
are contained in Table 3. In looking at this table, pay particular attention to the “Included”
column. It’s tricky.

The State is developing an automated system that will track performance and calculate the
performance measures.

9
February 2007

Table 3
Calculating the WIA Performance Measures
Performance
Measure Group Included Numerator Denominator
Entered Adults Those who were not employed at Number employed in the 1st Number who exit during the
Employment (18 & Older) registration1 quarter after exit quarter
Rate
Dislocated All Number employed in the 1st Number who exit during the
Workers quarter after exit quarter
Older Youth Those who were not employed at Number employed in the 1st Number who exit during the
(19 – 21) registration1 and who are not quarter after exit quarter
enrolled in post-secondary
education or advanced training in
the 1st quarter after exit
Employment Adults Those employed in the 1st quarter Number employed in the 3rd Number who exit during the
Retention Rate (18 & Older), after exit quarter after exit quarter
Dislocated
Workers
Older Youth Those employed in the 1st quarter Number employed in the 3rd Number who exit during the
(19 – 21) after exit and who are not enrolled quarter after exit quarter
in post-secondary education or
advanced training in the 3rd quarter
after exit
Earnings Adults Those employed in the 1st quarter Total UI-reported earnings in the Number who exit during the
Change (18 & Older) after exit 2nd and 3rd quarters after exit quarter
minus total UI-reported earnings
in the 2nd and 3rd quarters before
registration
Older Youth Those employed in the 1st quarter Total UI-reported earnings in the Number who exit during the
(19 – 21) after exit and who are not enrolled 2nd and 3rd quarters after exit quarter
in post-secondary education or minus total UI-reported earnings
advanced training in the 3rd quarter in the 2nd and 3rd quarters before
after exit registration

10
February 2007

Table 3
Calculating the WIA Performance Measures
Performance
Measure Group Included Numerator Denominator
Earnings Dislocated Those employed in the 1st quarter Total UI-reported earnings in the Total UI-reported earnings in the
Replacement Workers after exit 2nd and 3rd quarters after exit 2nd and 3rd quarters before
Rate dislocation2

Employment Adults Those who received training Number who were employed in Number who exit during the
and Credential (18 & Older), services the 1st quarter after exit and quarter
Rate Dislocated received a credential by the end of
Workers the 3rd quarter after exit3
Credential Rate Older Youth All Number who were in employment, Number who exit during the
(19 – 21) post-secondary education, or quarter
advanced training in the 1st quarter
after exit and who received a
credential by the end of the 3rd
quarter after exit3
Skill Younger Youth All in-school youth and all out-of- Total number of attained basic Total number of basic skill goals
Attainment (14 – 18) school youth assessed to be in skills plus number of attained plus number of work readiness
Rate 4 need of basic skills, work work readiness skills plus attained goals plus number of occupational
readiness skills, or occupational occupational skills during the skills goals set targeted for
skills reporting period completion in the reporting period
Diploma or Younger Youth Those who register without a Number who attained a secondary Number who exit during the
Equivalent (Age 14 – 18) secondary school diploma or school diploma or equivalent by quarter
Attainment equivalent and who are not in the end of the 1st quarter after exit
Rate secondary school at exit

Retention Rate Younger Youth Those not in secondary school at Number of youth found in one of Number who exit during quarter
(Age 14 – 18) exit the following categories in the 3rd
quarter after exit:
ƒ Post secondary education
ƒ Advanced training
ƒ Employment
ƒ Military service
ƒ Qualified apprenticeships

11
February 2007

Table 3
Calculating the WIA Performance Measures
Performance
Measure Group Included Numerator Denominator
Participant WIA-registered Participants who have exited from Index calculated based on responses to three questions addressing
Customer customers WIA and who are selected for satisfaction, expectations, and service ideal.
Satisfaction inclusion in the survey sample.
Employer One-stop Employers who have received a Index calculated based on responses to three questions addressing
Customer employer completed service involving satisfaction, expectations, and service ideal.
Satisfaction customers personal contact with one-stop
staff and who are selected for
inclusion in the survey sample.
Notes:
1 Employment status at registration is based on information provided by the registrant.
nd rd
2 If there is no date of dislocation, or if the date of dislocation occurs after registration, the 2 and 3 quarters prior to registration are used.
3 Credentials may be obtained while the participant is enrolled in WIA.
4 For additional information on skill goals and skill attainment, see page 13.

12
February 2007

11. What constitutes a “credential,” for purposes of the credential-based measures?

USDOL has defined “credential” broadly, to include both nationally and state or locally
recognized credentials. (See Appendix C, Definitions.) In addition to secondary diplomas,
GEDs, and post-secondary diplomas, the definition includes all credentials recognized by state
education agencies. Licensure and industry-recognized certificates are also included in the
definition. Please note:

With the exception of the diploma or equivalent rate for younger youth (age 14 to 18), none of
the credential-based measures give credit solely for obtaining a credential. Adults and dislocated
workers must also be employed in the 1st quarter after exit, and older youth (age 19 to 21) served
under the youth program must also be employed, in post-secondary school, or in advanced
training in the 1st quarter after exit.

All of the credential-based measures incorporate some lag time between exit and when the
credential must be obtained. In the case of younger youth (age 14 to 18) and the diploma or
equivalent rate, the credential must be obtained by the end of the 1st quarter after exit. In all of
the remaining credential-based measures, the credential must be obtained by the 3rd quarter after
exit. In fact, however, credentials will often be earned prior to exit.

Data on receipt of credentials will need to be obtained and documented at the local level.

12. The skills attainment measure is very different from the other measures, since it
compares goals attained to goals set, rather than specific outcomes to number of
individuals exiting from WIA. How will this measure work?

USDOL has established the following rules for measuring skill attainment rate:

• If a customer is deficient in basic literacy skills, he or she must set at least one basic skills
goal. In addition, he or she may also set goals for work readiness or occupational skills.

• Goals must be set for all younger youth (age 14 – 18) who are either in school or out-of-
school and assessed to be in need of basic skills, work readiness skills, or occupational
skills.

• All youth included in this measure must have at least one, and not more than three, skill
goals in a year. These goals may involve any combination of the three types of goals.
For example, a customer could have two basic skills goals plus one occupational skill
goal.

• Pre-assessment and post-assessment of skill goals are required.

• Use of standardized test procedures, such as standardized tests or performance-based

assessments with standardized scoring methods are encouraged. Where such tests or
assessments are not available, assessment techniques must be objective, unbiased and
conform to widely accepted, clearly defined criteria, be field tested for utility,
consistency, and accuracy, and provide instructions for valid test administration.

13
February 2007

• All data and methods used to determine

achievement of skill attainment goals must
be documented and are subject to audit.
• Target dates for accomplishing goals may
not be more than a year from the date on
which they are set, but they may be
extended to accommodate periods of time in
an inactive or “hold” status.
• Goal attainment should be recorded in the
quarter in which it occurs. Failure to meet a
goal should not be recorded until the goal
attainment period has expired.
Definitions of basic literacy skills deficient, basic
skills goal, occupational skills goal, and work
readiness skills goal have been established. (See
Appendix C, Definitions.)
13. What about the two customer satisfaction
measures? How will customer
satisfaction be measured?
Tests Recommended by USDOL
ƒ Test of Adult Basic Education (TABE)
ƒ DOL Workplace Literacy Test
(DOL-WLT)
ƒ Adult Measure of Educational Skills
(AMES)
ƒ Adult Basic Learning Exam (ABLE)
ƒ Adult Literacy Test (ALT), Simon &
Schuster
ƒ Armed Forces Qualifying Test (AFQT)
ƒ Basic Occupational Literacy Test
(BOLT)
ƒ California Achievement Test (CAT)
ƒ Career Ability Placement Survey (CAPS)
ƒ Comprehensive Adult Student
Assessment System (CASAS)
ƒ General Aptitude Battery (GATB)
ƒ Iowa Test of Basic Skills (ITBS)
ƒ Metropolitan Achievement Test (MAT)
ƒ Reading Job Corps Screening Test
(FJCST)
ƒ Wide Range Achievement Test (WRAT)

USDOL requires that we use the copyrighted American Customer Satisfaction Index (ACSI) and
scoring methodology to measure the satisfaction of participant and employer customers. This
index, which is based on telephone survey data, is widely used in both industry and government.
Detailed information on the ACSI and the required data collection process is contained in
Appendix D, Customer Satisfaction Measures.

14. Where can I learn more about the WIA performance measures?

The most current and comprehensive federal information on the WIA performance measures is
contained in two issuances: USDOL Training and Employment Guidance Letters No. 7-99 and
8-99. These TEGLs and other timely information can be downloaded from
www.usworkforce.org and www.icesa.org.

14
February 2007

Negotiated Levels of Performance

The levels of performance to which the State and each local area will be accountable will be
reached through negotiation − between USDOL and the State and between the State and each
local area. Under Section 136(c) of WIA, local levels of performance on each of the perform-
ance measures are to be "based on the State adjusted levels of performance,” taking into account
“the specific economic, demographic, and other characteristics of the populations to be served in
the local area.”

For those of us who are accustomed to the JTPA performance standards, in which required levels
of performance are generated using regression equations that take into consideration character-
istics of the population served and labor market factors, negotiated performance levels will take
some getting used to. While the JTPA system meant that local areas did not know until the end
of the program year what their final standards were, the standards were “self-correcting.” That
is, if the population served turned out to be substantially different from the expected population,
required levels of performance changed automatically. WIA’s negotiated performance level
approach does not offer that kind of flexibility. However, as we gain experience under WIA, we
will have better information on which to base performance levels.

1. Have State adjusted levels of performance been set?

Georgia submitted baseline data and proposed levels of performance to the U. S. Department of
Labor in its State Plan. Negotiations have taken place, but our plan has not been approved yet.
In addition, because we will not have any information on local program design until initial local
plans are submitted, we have advised USDOL that our proposed levels of performance may need
to be revised. However, the proposed State levels are listed in Table 4 below.

2. When will negotiation of local performance levels take place?

Although local areas are asked to include planned levels of performance for Year 1 in their initial
plans, final negotiation of performance levels will occur when the comprehensive plans are
submitted in the fall.

Year 1 performance will not be very meaningful in terms of service under the WIA system, since
it will be based on the outcomes of JTPA terminees from October 1999 through June 2000 plus
WIA registrants meeting the criteria for inclusion in the performance measures during July
through September 2000. However, performance in terms of the WIA measures during this
transitional period will help all of us become familiar with those measures.

15
February 2007

Table 4
Statewide Performance Levels (Pending USDOL Approval)*
Performance Measure Year 1 Year 2 Year 3

Adults
Entered Employment Rate 67% 67.5% 68%
Employment Retention Rate 80% 80% 80%
Earnings Change $3,425 $3,435 $3,445
Employment & Credential Rate 60% 60% 60%

Dislocated Workers
Entered Employment Rate 74% 74% 74%
Employment Retention Rate 89% 89% 89%
Earnings Replacement Rate 94% 94% 94%
Employment & Credential Rate 60% 60% 60%

Older Youth, 19 - 21
Entered Employment Rate 68% 70% 72%
Employment Retention Rate 81% 81% 81%
Earnings Change $2,750 $2,750 $2,800
Credential Rate 50% 50% 50%

Younger Youth, 14 – 18
Skill Attainment Rate 72% 72% 72%
Diploma or Equivalent Attainment Rate 55% 55% 55%
Retention Rate 58% 60% 62%

Customer Satisfaction
Participant 66% 67% 68%
Employer 70% 71.5% 73%
* Because of the negotiation process, the levels shown are different from the levels that
appear in the State Plan and the Local Planning Instructions.

3. Will local areas be expected to meet the statewide levels of performance?

Ultimately local performance will need to “roll up” into the statewide levels. However, we know
that conditions vary substantially across our state. Therefore, local performance levels should
reflect local circumstances.

4. What will the State take into consideration in negotiating local performance levels?

Our baseline data have made it clear that the outcomes of those who receive training are likely to
be substantially different from the outcomes of customers receiving intensive services only, so
service mix for the local area will need to be considered. (See Appendix E, Baseline
Performance Data.) In addition, local labor market conditions and characteristics of the
population served have historically been related to program outcomes. In the coming months,
we will add labor market data to our existing baseline data so that we can study the relationships
between the WIA performance measures, customer characteristics, and labor market conditions.

16
February 2007

Appendix A
Mandatory and Optional WIA Partner Programs

Mandatory Partners

• Employment Service
• Adult Education
• Postsecondary Vocational Education
• Vocational Rehabilitation
• Welfare-to-Work
• Title V of the Older Americans Act
• Trade Adjustment Assistance
• NAFTA Transitional Adjustment Assistance
• Veterans Employment and Training Programs
• Community Services Block Grants
• HUD Employment and Training Activities
• Unemployment Insurance
• Job Corps
• U. S. Department of Labor: Indian and Native American Programs; Migrant and
Seasonal Farmworker Programs; Youth Opportunity Grants; demonstration, pilot,
multiservice, research, and multistate projects; and national emergency grants

Optional Partners

• School-to-Work
• Bureau of Apprenticeship and Training
• Department of Transportation Employment Support Programs
• TANF
• USDA Food Stamp Employment and Training Programs
• Other partners designated by local areas

i
February 2007

Appendix B
USDOL Instructions on Reporting Supplemental Data

[Proposed reporting instructions issued in the Federal Register on April 3, 2000, included the
following language regarding reporting of supplemental data on employment:]

While the majority of employment in a state’s workforce will be “covered” in the UI wage
records, certain types of employers and employees are excluded by Federal standards or are not
covered under a state’s UI law. “Uncovered” employment typically includes Federal
employment, postal service, military, railroad, self employment, some agricultural employment,
and employment where earnings are primarily based on commission. States have flexibility in
methods used to obtain information on participants in “uncovered” employment. Examples
include:

1) Case management, follow-up services, and surveys of participants to determine that the
participant[s] are employed and written documentation of that employment; or

2) Record sharing and/or automated record matching with other employment and
administrative data bases to determine and document employment. These databases
include, but are not limited to:

• Office of Personnel Management (Federal Career Service);

• United States Postal Service;
• Railroad Retirement System;
• State Department of Revenue or Tax (State income tax for self-reported
occupations);
• U. S. Department of Defense; and
• Government Employment Records (State government, local government, judicial
employment, public school employment, etc.).

All data and methods to supplement wage record data must be documented and are subject to
audit. Computer records from automated record matching are considered to be a valid written
record. A telephone response from the participant must be accompanied by a written document
such as a W2 form, pay stub, 1099 form, or other written documentation. Telephone verification
of employment with employers is acceptable, but must also be documented. For self-employed
individuals, telephone verification with major clients/contracting entities is also acceptable, but
must be documented. States must wait two full quarters after the quarter of wage activity before
making the decision to use supplemental sources of data. Supplemental data must be recorded
within 30 days after the individual was found missing in the wage record.

ii
February 2007

Appendix C
Definitions

The definitions that follow were issued by USDOL in Training and Employment Guidance Letter
No. 7-99.

Advanced training – an occupational skills employment/training program, not funded under Title I
of WIA, which does not duplicate training received under Title I. Includes only training outside of
the One-Stop, WIA and partner, system (i.e. training following exit).

Basic literacy skills deficient - the individual computes or solves problems, reads, writes, or speaks
English at or below the 8th grade level or is unable to compute or solve problems, read, write, or
speak English at a level necessary to function on the job, in the individual’s family, or in society. 1

Basic skills goal - measurable increase in basic educational skills including reading
comprehension, math computation, writing, speaking, listening, problem solving, reasoning, and
the capacity to use these skills.

Credential – nationally recognized degree or certificate or State/locally recognized credential.

Credentials include, but are not limited to, a high school diploma, GED or other recognized
equivalents, post-secondary degrees/certificates, recognized skill standards, and licensure or
industry-recognized certificates. States should include all State Education Agency recognized
credentials. 2

Date of dislocation – The last day of employment at the dislocation job. If there is no date of
dislocation, date of registration will be used instead.

Employed at registration – An individual employed at registration is one who is currently working

as a paid employee or who works on his or her own business, profession or farm, or works 15
hours or more as an unpaid worker in an enterprise operated by a member of the family, or is one
who was not working, but has a job or business from which he or she was temporarily absent
because of illness, bad weather, vacation, labor-management dispute, or personal reasons, whether
or not paid by the employer for time-off, and whether or not seeking another job. (definition was
adopted from TEGL 14-00, change 1, Attachment E)

Employed in quarter after exit quarter – The individual is considered employed if UI wage
records for the quarter after exit show earnings greater than zero. UI Wage records will be the
primary data source for tracking employment in the quarter after exit. When supplemental data
sources are used, individuals should be counted as employed if, in the calendar quarter after exit,

1
States and localities may establish their own definitions of “basic literacy skills deficient.” However, their
definitions must include the above language.
2
The USDOL definition includes the following guidance: “In addition, States should work with local
Workforce Investment Boards to encourage certificates to recognize successful completion of the training
services listed above that are designed to equip individuals to enter or re-enter employment, retain
employment, or advance into better employment.” [See definition of “training services.”]

8/4/2000 xix
February 2007

they did any work at all as paid employees (i.e. received at least some earnings), worked in their
own business, profession or worked on their own farm.

Exit date – the last date of WIA funded or partner service received (except follow-up services).
There are two ways to determine exit date:
• a participant who has a date of case closure, completion or known exit from WIA-funded or
non-WIA funded partner service within the quarter (hard exit); or
• a participant who does not receive any WIA-funded or non-WIA funded partner service for 90
days and is not scheduled for future services except follow-up services (soft exit). 3
Participants who have a planned gap in service of greater than 90 days should not be considered as
exited if the gap in service is due to a delay before the beginning of training or a health/medical
condition that prevents an individual from participating in services.

Exit quarter – the quarter in which the last date of service (except follow-up services) takes place.

High school diploma equivalent – a GED or high school equivalency diploma recognized by the
State.

Military service – reporting for active duty.

Not employed at registration – An individual who does not meet the definition of employed at
registration.

Occupational skills goal - primary occupational skills encompass the proficiency to perform actual
tasks and technical functions required by certain occupational fields at entry, intermediate, or
advanced levels. Secondary occupational skills entail familiarity with and use of set-up
procedures, safety measures, work-related terminology, record keeping and paperwork formats,
tools, equipment and materials, and breakdown and clean-up routines.

Post-secondary education – a program at an accredited degree-granting institution that leads to an

academic degree (e.g., AA, AS, BA, BS). Does not include programs offered by degree-granting
institutions that do not lead to an academic degree.

Qualified apprenticeship – a program appoved and recorded by the ETA/Bureau of

Apprenticeship and Training (BAT) or by a recognized State Apprenticeship Agency (State
Apprenticeship Council). Approval is by certified registration or other appropriate credential.

Training services – include WIA-funded and non-WIA funded partner training services. These
services include:
• occupational skills training, including training for nontraditional employment
• on-the-job training
• programs that combine workplace training with related instruction, which may include
cooperative education programs
• training programs operated by the private sector
• skill upgrading and retraining

3
A soft exit cannot be determined until 90 days has elapsed from the last date of service, but the exit date
recorded is the last date of service.

8/4/2000 xx
February 2007

• entrepreneurial training
• job readiness training
• adult education and literacy activities in combination with other training
• customized training conducted with a commitment by an employer or group of employers to
employ an individual upon successful completion of the training

Work readiness skill goal – work readiness skills include world of work awareness, labor market
knowledge, occupational information, values clarification and personal understanding, career
planning and decision making, and job search techniques (resumes, interviews, applications, and
follow-up letters). They also encompass survival/daily living skills such as using the phone,
telling time, shopping, renting an apartment, opening a bank account, and using public
transportation. They also include positive work habits, attitudes, and behaviors such as
punctuality, regular attendance, presenting a neat appearance, getting along and working well with
others, exhibiting good conduct, following instructions and completing tasks, accepting
constructive criticism from supervisors and co-workers, showing initiative and reliability, and
assuming the responsibilities involved in maintaining a job. This category also entails developing
motivation and adaptability, obtaining effective coping and problem-solving skills, and acquiring
an improved self image.

8/4/2000 xxi
February 2007

Appendix D
Customer Satisfaction Measures

Customer satisfaction measures have been established for two groups of customers: WIA
participants and employers receiving services from one-stop centers. USDOL has chosen to use
the copyrighted American Customer Satisfaction Index (ACSI) and scoring methodology. It has
purchased a license to use the ACSI for 500 completed participant and 500 completed employer
surveys per state, per year. The State will be responsible for conducting these surveys.

Because this number of surveys is not sufficient for valid measurement of customer satisfaction at
the local level, additional surveys will need to be conducted. We will work with local areas to
address issues such as design of survey instruments, data collection methodology, funding, and
State and local responsibilities.

1. Who will be surveyed?

Random samples will be drawn from two customer groups:

• Title I-B participants who have exited from WIA

• Employers who have received a substantial service involving personal contact with One-
Stop staff

The “substantial service” criterion excludes employers who request a brochure or standard
mailing, ask a question that is answered with little expenditure of staff time, or use electronic self-
services. Examples of substantial services include staff-facilitated job orders, customized labor
market information requests, and on-the-job or customized training activities.

2. How many surveys must be completed?

Each state must complete 500 participant surveys and 500 employer surveys a year. Completed
surveys must be generated from data collection processes achieving response rates of 50 percent or
higher. A completed survey is defined as a survey in which all three questions regarding customer
satisfaction have been answered.

USDOL has been silent on the issue of the number of surveys that must be completed within each
local area, but, clearly, 500 surveys statewide will not provide data that can be used to calculate
valid measures of local customer satisfaction. We will consider the issue of sample size after local
plans have been received and we have some sense of the number of people local areas expect to
serve.

3. How will customer satisfaction data be collected?

The responses are to be obtained using a uniform telephone methodology. According to USDOL,
estimates of the cost of telephone surveys nationwide run an average of $15 per completed survey.
Although USDOL has not yet issued guidance on cost allocation, it is our understanding that the

8/4/2000 xxii
February 2007

cost of collecting customer satisfaction data will be a program cost, rather than an administrative
cost.

4. When will surveys be conducted?

The surveys will be conducted on a rolling basis within the following time frames:

Participants will be contacted within 60 days of the exit date or the date that an exit date has been
determined. This means either 60 days after the date of an exit interview or 60 days after the 90
days have elapsed since the last service date.

Employers will be contacted within 60 days of the completion of the service or 30 to 60 days after
a job order has been listed where no referrals have been made.

5. What types of questions will be asked?

Following an introduction used to determine whether the interviewer is speaking to the right
person and establish the appropriate frame of reference, respondents will be asked to rate the
service they received on three dimensions:

• Their overall satisfaction with the service received

• The extent to which the service met his or her expectations
• How close the service was to the respondent’s ideal for such services

Ratings will be made on a scale from 1 to 10.

6. How will customer satisfaction scores be calculated?

A customer’s ACSI score is the weighted average of his or her responses to the three questions.
This score is then transformed to an index reported on a 0-100 scale. An overall State or local
customer satisfaction score is simply the average of the individual index scores, rounded to the
nearest whole number.

8/4/2000 xxiii
February 2007

Appendix E
Baseline Performance Data

1. Where did the baseline data come from?

Although USDOL mandated the use of JTPA data as a baseline for WIA performance, Georgia
also submitted data for Employment Service customers who received the equivalent of WIA
intensive services from a field service office (FSO) during a six-month period. In general, four
types of customers received such services:

UI Claimants receiving services under the Claimant Assistance Program

Veterans served by veterans’ programs
TANF recipients and applicants served under the TANF contract
Other individuals who were disadvantaged in the workplace in some way, including people with
disabilities, the economically disadvantaged, and those lacking a high school diploma

UI claimants were excluded from these baseline data because they are not typical of the population
likely to be served under WIA and because their need for assistance will continue to be met
through the Claimant Assistance Program. The outcomes of the remaining recipients of intensive
services are seen in the entries for Adults and Older Youth listed under “Intensive Labor
Exchange” in the table below.

All of the dislocated workers included in the baseline data were JTPA Title III customers.
However, performance for “Education and Training” is for those served by SDAs, while
performance for “Intensive Labor Exchange” is for those receiving basic readjustment services
from FSOs.

Baseline data from both JTPA and the Employment Service were matched against UI data so that
performance could be calculated in terms of the WIA performance measures.

In calculating the combined baseline figures, we assumed a mix between intensive services only
and training of 50/50 for adults and older youth (age 19 – 21). This mix was chosen simply
because we had no basis on which to make any assumptions about what the actual mix would be.
The mix for dislocated workers was set at 80% intensive services to only 20% education and
training, based on the historic ratio of participation between basic readjustment services and
training. These assumptions will be reconsidered after we receive local plans.

8/4/2000 xxiv
February 2007

U. S. Department of Labor Projected Performance and Georgia Baseline Data

Georgia Baseline Data

USDOL Education Intensive
Projected And Labor
Performance Indicator Average Training1 Exchange2 Combined

Adults
Entered Employment Rate 71% 71% 56% 64%
Employment Retention Rate 78% 81% 80% 80%
Earnings Change in Six Months $3,700 $3,506 $2,965 $3,235
Employment and Credential Rate 60% *** *** 60%

Dislocated Workers3
Entered Employment Rate 77% 76% 71% 72%
Employment Retention Rate 85% 91% 89% 89%
Earnings Replacement Rate 92% 120% 88% 94%
Employment and Credential Rate 60% *** *** 60%

Older Youth (Age 19 – 21)

Entered Employment Rate 63% 75% 64% 70%
Employment Retention Rate 77% 81% 81% 81%
Earnings Change in Six Months $3,150 $3,037 $1,917 $2,477
Credential Rate 50% *** *** 50%

Younger Youth (Age 14 – 18)

Skill Attainment Rate 72% *** N/A 72%
Diploma or Equivalent Attainment 55% *** N/A 55%
Rate
Retention Rate 54% 60% *** 60%

1 Based on customers exiting from JTPA during the period from October 1997 through
September 1998.
2 Based on job seekers receiving services similar to WIA intensive services and exiting
from Georgia’s field service offices during the period from September 1997 through
March 1998.
3 Based entirely on JTPA participants served by SDAs (Education and Training) and FSOs
(Intensive Labor Exchange)
*** Cannot be calculated using available data.

8/4/2000 xxv
February 2007

2. The USDOL projected average levels for earnings gains seem very high. Can we
really expect to see average increased earnings of over $3000?

Performance on the earnings measures depends very substantially on prior earnings. When we
looked at the outcomes of all FSO customers during the baseline period, the average change was
very low – less than $300. However, among those receiving intensive services, there was a clear
split between UI claimants and all other customers. Earnings remained essentially the same for
the claimants, but increased substantially (the $2,965 seen in the table, above) for other recipients
of intensive service. That is, the UI claimants had solid earnings before losing their jobs, and
when they returned to work, they earned about what they had earned before. The other
Employment Service customers provided with intensive services had very low earnings in the
months prior to receiving services. With help, they improved their employment situation and their
earnings increased substantially.

We see the same thing in the JTPA baseline data. JTPA serves individuals who are economically
disadvantaged. That is, they are people with very low earnings in the period prior to enrollment.
Following participation, their employment situation improves, and their earnings also increase
substantially.

The USDOL projected average levels for the two earnings gain measures have the effect of
targeting WIA services on individuals who have been unemployed, or underemployed, for an
extended period prior to their registration for WIA.

3. It appears that customers who received education and training fared better than
those receiving intensive services.

Yes. Entered employment rates were consistently higher for those who received training than for
those receiving intensive services only, although employment retention rates were essentially the
same for the two groups. Similarly, performance on the earnings change measure for adults and
older youth (age 19 – 21) and the earnings replacement rate for dislocated workers was higher
among those who received education and training than among those receiving intensive services
only.

It is not surprising that outcomes are generally better for recipients of training. After all, more
time and resources were invested in these customers, and the skills they learned enhanced their
competitiveness in the job market. However, not everyone registered under WIA will be either
willing or able to enter training. In many cases, intensive services will be the services most
responsive to our customers’ needs.

8/4/2000 xxvi
February 2007

Appendix F
WIA Performance Measure Baseline Data by State Service Delivery Regions

Background. Although the WIA performance measures are different from the measures used in
assessing the performance of programs funded under JTPA and Wagner-Peyser, the WIA
measures that are based on Unemployment Insurance (UI) wage data can be calculated for people
served under those programs. The U. S. Department of Labor instructed states to do just that,
using JTPA terminees for the period from October 1997 through September 1998, in order to set
baseline performance levels for WIA. Georgia chose to add individuals who received services
similar to WIA intensive services from the state’s Career Centers to its baseline data so that it
would be more representative of the scope of services to be provided under WIA. The Career
Center customers were active during the period from October 1997 through March 1998.

Findings from the baseline data are presented in the tables and charts that follow. They are
organized by the twelve state service delivery regions. To a large extent, these regions coincide
with local WIA areas. Where they do not, small WIA areas have generally been carved out of
larger regions for administrative purposes. In most small areas, we do not have enough data to
provide a valid basis for planning and negotiation. However, the regions reflect the labor markets
within which these local areas will operate. In addition, labor market data that may be important
in understanding variation between areas − and useful in negotiating local levels of performance −
are readily available by region. Local areas may also provide additional, more localized data to
inform the negotiation process.

In the tables and charts, comparisons are made between “training” and “intensive services only.”
Data on training for Adults and Older Youth are based on JTPA Title II terminees. Data on
“intensive services only” for the same groups are based on Career Center customers, excluding UI
claimants, who received services similar to the intensive services authorized under WIA.
Claimants were excluded because they will continue to receive such services through the Claimant
Assistance Program. The majority of the Career Center customers in the baseline data were
economically, educationally, or physically disadvantaged and therefore were similar to those
served under Title II of JTPA. Performance for Adults and Older Youth are shown both
separately and combined because outcomes for the two groups were very similar and because the
number of cases available for Older Youth is very small.

Data on Dislocated Workers are based entirely on terminees from Title III of JTPA. Those who
received “training” were served by SDAs, while those shown under “intensive services only”
received basic readjustment services from a Career Center.

Factors Influencing Performance. In general, the factors that are likely to have an effect on
performance include labor market conditions, the background and characteristics of the people
served, the types of services provided, and the quality of those services. Ideally, negotiated
performance measures should take the first three types of factors into consideration and let quality
shine through. However, unlike the JTPA performance standard adjustment process, which fine-
tuned performance expectations to actual labor market conditions and participants served,
negotiation of the WIA performance measures takes place during the planning process, and the

8/4/2000 xxvii
February 2007

resulting levels remain fixed. Therefore, emphasis should be placed on factors that reflect
enduring conditions or fundamental decisions regarding program focus.

We are currently doing analyses of the relationships between Georgia’s baseline performance and
factors that we believe may influence performance. Examples include regional factors such as
unemployment rates, average annual wages, or indicators of economic growth or decline, and
registrant characteristics such as educational attainment or economic disadvantage. We will share
what we learn in a future package on negotiating local performance measures. However, the
baseline data may help you become familiar with the wage-based measures mandated under WIA.
They may also help you think about the implications of program design choices for performance.

Baseline Performance

The wage-based performance measures, which are calculated separately for each of the groups
listed, include the following:

• Entered employment rate (Adults, Older Youth, and Dislocated Workers)

• Employment retention rate (Adults, Older Youth, and Dislocated Workers)
• Earnings change (Adults and Older Youth)
• Earnings replacement rate (Dislocated Workers)

The operational definitions of these measures are technically complex, especially with regard to
who is included in each calculation. However, all of them are based on either the presence or
amount of wages reported to the Unemployment Insurance program during a particular quarter or
set of quarters. (See Table 3, Calculating the WIA Performance Measures, on page 9 of this
Guide.)

In looking at the baseline data, it is useful to think about variation, both across regions and
between services, and whether it appears to be “systematic” or “random.” Systematic variation is
that part of the differences between regions that we can account for. For example, one might
expect regions with higher unemployment rates to have lower entered employment rates. Random
variation is that part of the differences between regions that we cannot explain. It is there, but we
don’t know why, and it should not be the basis for differences in performance expectations.

In general, we do not see strong evidence of large, systematic geographic differences. However,
we will explore this issue further by examining relationships between labor market conditions and
performance, and we expect that an understanding of geographic differences will be useful in fine-
tuning performance expectations.

On the other hand, outcomes were generally better for individuals who received training than for
those who received intensive services only. Therefore, we would expect to see higher levels of
performance from areas that provide training to more of their WIA registrants. However, for
many customers, intensive services may be the best, and most cost effective, form of help a local
area can offer. In order to avoid creating a disincentive to providing intensive services to those
who need them, service mix will need to be taken into consideration in the negotiation process.

8/4/2000 xxviii
February 2007

Among the measures, employment retention rates showed the least variation, both across regions
and between services − especially among Dislocated Workers. (One reason why there is little
variation is that only those where were employed in the first quarter following exit are included in
the measure, so it is unaffected by differences in entered employment rates.) As we learn more
about the retention measures, we may discover that performance expectations for these measures
should vary within a narrow range, if at all.

We have the most to learn about the earnings measures. The earnings replacement rate for
dislocated workers shows that training made a strong contribution to earnings. But while earnings
replacement rates were very consistent across regions for those who received intensive services
only, there was a great deal of variation across regions for those who received training. We know
that the character of dislocated worker programs can change dramatically, depending on the types
of workers affected by dislocation. Therefore, while the very high earnings replacement rates seen
in some areas are cause for celebration, they are probably not very helpful when it comes to setting
realistic performance expectations.

The baseline earnings changes for Adults and Older Youth vary widely across regions, and,
although earnings changes were generally higher for those who received training, the relationship
between the type of service received and earnings change is not consistent across areas. We do
know, however, that earnings changes tend to be greatest for those customers with the lowest pre-
participation earnings, and that, therefore, this measure has the effect of targeting services on those
most in need of assistance. While it will probably be important to take characteristics of the
population being served into consideration in negotiating this measure, the negotiation process
should not be used to shift services away from those who need them most.

Finally, in thinking about the implications of wage-based WIA measures for local programs, it is
important to remember that the state will also be held accountable for its performance. That
performance can only come from the performance of local areas. Across all areas, negotiated
performance levels will need to “roll up” to a reasonable expectation that the state will meet its
negotiated levels of performance.

8/4/2000 xxix
February 2007

Appendix G

Data for Use in Negotiating WIA UI Wage-Record Based Performance Measures

Background

Under WIA, proposed state and local performance levels are to be negotiated between federal and
state partners and between state and local partners. The challenge is how to make the negotiation
process valid and equitable, resulting in performance levels that reflect a realistic understanding of
differences among states and between local areas within states. What follows does not provide a
standard "formula" for coming up with planned performance. However, it does give you some
background information and concepts to think about as you develop reasonable performance
expectations.

The national JTPA performance standard system addressed this issue by using a statistical
modeling process that adjusted local standards based on a set of factors shown to have an effect on
local performance. For example, if an area served a high percentage of participants who lacked a
high school diploma, its performance standards were lowered because this population was harder
to place and less likely to retain their jobs. Although this approach is not being used under WIA,
it can help us understand the factors that effect performance so that we can consider them in the
negotiation process.

The information that follows is based on the outcomes of customers who either terminated from
JTPA or exited from FSOs during the baseline period. The outcomes are expressed in terms of the
WIA UI wage-record based measures. Training (SDAs) and “intensive services only” (FSOs)
were examined separately. Statistical models could not be developed for outcomes for older youth
because there were so few members of this group in the baseline data. However, findings
regarding the adult measures suggest factors that could be considered in negotiating performance
levels for older youth.

The factors tend to be related to one another. Therefore, some factors that you might expect to be
important in explaining outcomes were not because other, related, and more powerful factors did
the job. For example, one of the customer characteristics tested was whether or not the customer
was receiving TANF. Because factors for gender and prior earnings level accounted for the
differences between TANF recipients and other customers, the TANF factor did not add anything
to our understanding of outcomes. The same thing happened with unemployment rate, which was
related to all of the other labor market factors tested.

The outcomes associated with various levels of important factors are shown in the tables that
follow. We hope that this information will help you think about the ways in which your area and
your customers differ from those of other areas, and the implications of those differences for your
expected performance. We expect that findings from the baseline data will inform the negotiation
process but will not, by themselves, determine the resulting performance levels.

8/4/2000 xxx
February 2007

Factors

Type of Services Provided. In Appendix F of this Guide, we saw that, for most measures, differences
between training and intensive services within service delivery regions tended to be greater than differences
between service delivery regions.

Under JTPA, type of services provided was excluded from the performance standard adjustment process.
However, JTPA was primarily a training program, serving individuals who were willing and able to
participate in training. The exception to this focus on training was JTPA's ability to provide basic
readjustment services under Title III. Georgia found that a substantial majority of the dislocated workers
served under JTPA chose basic readjustment services only.

When we compare the outcomes of the baseline dislocated workers who received training with those who
received basic readjustment services using the WIA measures, we see that those who received training did
somewhat better. This difference was especially true in the case of earnings replacement rate. That is, an
investment of participant time and JTPA funds in training helped the trainees get better wages. That does not
mean, however, that basic readjustment services (which are similar to WIA intensive services) were not
helpful in facilitating the transition from dislocation to reemployment. It simply means inexpensive, short-
term services cannot be expected to generate the same results as longer-term, more costly services.

We believe that both intensive services and training are needed, and the outcomes for both should
be considered in setting state and local standards. Therefore, planned mix of services should be
taken into consideration in negotiating performance levels. (See "Statewide Baseline Outcomes
by Type of Service," below.)

Statewide Baseline Outcomes by Type of Service

Intensive
Training Services Only
Performance Measure
Entered Employment Rate
ƒ Adults 73% 56%
ƒ Older Youth 77% 58%
ƒ Dislocated Workers 76% 71%
Employment Retention Rate
ƒ Adults 81% 80%
ƒ Older Youth 81% 79%
ƒ Dislocated Workers 91% 89%
Earnings Change
ƒ Adults $3506 $2985
ƒ Older Youth $3037 $2279
Earnings Replacement Rate
ƒ Dislocated Workers 120% 89%

8/4/2000 xxxi
February 2007

While there is no prescribed methodology for predicting WIA performance, a good place to start
in thinking about performance is the statewide baseline performance levels. Planned mix of
services between training and intensive services can be considered by calculating a weighted
average of the two. For example, suppose you expect that 40 percent of your adult WIA
registrants will enroll in training, while the remaining 60 percent will receive intensive services
only. Applying these percentages to the statewide baselines (multiplying the weights times the
baseline rates and adding the results) would give you a projected adult entered employment rate of
approximately 63%.

Service Weight Baseline Product

Training 40% 73% 29.2%
Intensive Services 60% 56% 33.6%
Predicted Performance (Sum) 62.8%

Customer Characteristics. In general, the baseline data include two types of individuals:
dislocated workers and people who had some disadvantage in the labor force. Those served under
Title II of JTPA were economically disadvantaged or had other barriers to employment. When
claimants and dislocated workers were excluded from those provided intensive services by FSOs,
the data showed that the majority of these customers were also disadvantaged. The table below
summarizes customer characteristics in the baseline period.

Statewide Baseline Customer Characteristics by

Program Area and Type of Service

Adults Dislocated Workers

Characteristic Intensive Intensive
Training Services Training Services
Gender
ƒ Male 30% 40% 37% 36%
ƒ Female 70% 60% 63% 64%
High School Graduate
ƒ Yes 75% 76% 91% 87%
ƒ No 25% 24% 9% 13%
Person with Disability
ƒ Yes 5% 3% 2% 1%
ƒ No 95% 97% 98% 99%
Earnings in the 3rd Quarter
Before Entry:
ƒ Less than $1000 70% 64%
ƒ $1000 to $2499 15% 12%
ƒ $2500 and Higher 15% 24%
Earnings in the 3rd Quarter
Before Entry:
ƒ Less than $2500 37% 23%
ƒ $2500 to $4999 33% 38%
ƒ $5000 and Higher 30% 39%

8/4/2000 xxxii
February 2007

Among the customer characteristics examined for their relationship to WIA outcomes were
gender, high school diploma or GED, disability, and receipt of TANF. In addition, level of UI-
reported wages in the third quarter before receipt of services was examined. The 3rd quarter prior
to participation was chosen because earnings typically decline just before individuals seek
employment assistance or training. Going back in time provides a better picture of the
individual’s “normal” employment and earnings and helps distinguish between those with
temporary versus enduring employment problems. In meaning, this factor is similar to the more
familiar characteristics of economic disadvantage or poor work history. As you will see, lower
prior earnings are associated with lower entered employment and retention rates but much higher
earning gains and earnings replacement rates.

In considering whether characteristics of your customers should be considered in setting your

performance levels, the first question to ask is whether your customers are significantly different
from those of other areas. (See table, "Statewide Baseline Customer Characteristics by Program
Area and Type of Service, above.) For example, if you expect that about 25 percent of your adult
customers will lack a high school diploma or GED, you are typical. There is no need to adjust
your expected performance levels for that fact. However, if you expect that, based on
coordination relationships and service goals in your area, 15% of your adult trainees will be
persons with disabilities, you will probably want to take that fact into consideration.

Again, calculating a weighted average will help you see the likely effect of this commitment on
your performance.

Persons with
Disabilities Weight Baseline Product
Yes 15% 61% 9.2%
No 85% 72% 61.2%
Predicted Performance (Sum) 70.4%

Labor Market Factors. Because a variety of labor market factors had been tested over the years for use in
the JTPA performance standard adjustment process, we attempted to look at the relationships between those
factors and WIA outcomes. Some of those factors could not be calculated from available data, and
substitutions were made. Factor values were linked to individual customer data based on county of residence.
The following factors were tested:

• Annual unemployment rate (1997)

• Average weekly wage (1997)
• Average annual earnings in wholesale and retail trade (1997)
• Percentage employed in agriculture, manufacturing, or mining (1997)
• Percentage of population below poverty level (1995)
• Employee/resident worker ratio (1997)
• Percentage change in population (1990 to 1997)
• 1993 Beale Code Definitions of Metro/Nonmetro Status

8/4/2000 xxxiii
February 2007

To take the effect of a labor market factor into consideration, look at the difference between the
baseline value and the value that applies to your area and apply that difference to your baseline, or
modified baseline, figures. (The most current available data on the factors that proved to be
important in explaining outcomes will be provided separately.)

Levels of Performance Associated with Significant Factors

The tables that follow show the levels of performance associated with the customer characteristics and
labor market factors that had a significant effect on performance. Levels of the labor market factors
have been selected based on their distribution in the baseline population. Different levels of earnings
in the third quarter prior to receipt of services have been chosen for adults and dislocated workers
because the range of earnings was substantially higher for the dislocated workers than for the
customers included in the baseline data for adults.

In some cases, effects of the factors on performance are modest, and may not be worth considering. In
other cases, they are dramatic. In particular, earnings in the 3rd quarter prior to participation helps
explain the very substantial differences in earnings change and earnings replacement rate that we saw
between service delivery regions in Appendix F. Regions that served customers with lower prior
earnings had higher earnings gains and replacement rates. Decisions that you make regarding
targeting of services based on work and earnings history will have a substantial effect on your
performance on these measures.

8/4/2000 xxxiv
 February 2007

WIA Outcomes by Customer Characteristics: Training Recipients

Earnings
Entered Employment Rate Employment Retention Rate Earnings Replacement
Characteristic Dislocated Dislocated Change: Rate:
Adults Workers Adults Workers Adults Dislocated
Workers
Gender
ƒ Male 72% 73% 87% $4,028 121%
ƒ Female 78% 84% 93% $3,287 123%
High School Graduate
ƒ Yes 84% $3,698 123%
ƒ No 71% $2,896 107%
Disability
ƒ Yes 61% * 75%
ƒ No 72% 91%
Earnings in 3rd Quarter
Before Entry:
ƒ Less than $1000 68% 76% $4,638
ƒ $1000 - $2499 77% 88% $2,300
ƒ $2500 and Higher 84% 90% $454
Earnings in 3rd Quarter
Before Entry
ƒ Less than $2500 86% 464%
ƒ $2500 to $4999 92% 110%
ƒ $5000 and Higher 95% 79%
* Based on fewer than 50 individuals.

8/4/2000 xxxv
 February 2007

WIA Outcomes by Labor Market Factors: Training Recipients

Earnings
Entered Employment Rate Employment Retention Rate Earnings Replacement
Labor Market Factor Dislocated Dislocated Change: Rate:
Adults Workers Adults Workers Adults Dislocated
Workers
Average Weekly Wage
ƒ Less than $400 74%
ƒ $400 to $499 74%
ƒ $500 and Higher 79%
% of Population in Poverty
ƒ Less than 13% $4,027
ƒ 13% to 22% $3,050
ƒ 23% and Higher $3,515
Beale Code
ƒ Metropolitan County 75%
ƒ Nonmetropolitan Cnty 78%
* Based on fewer than 50 individuals.

8/4/2000 xxxvi
 February 2007

WIA Outcomes by Customer Characteristics: Intensive Services Only

Earnings
Entered Employment Rate Employment Retention Rate Earnings Replacement
Characteristic Change: Rate:
Dislocated Dislocated Adults Dislocated
Adults Workers Adults Workers Workers

Gender
69% $3,943
ƒ Male 72% $2,363
ƒ Female
High School Graduate
ƒ Yes 59% 72% 82% 89% $3,345
ƒ No 47% 70% 71% 85% $1,629
Disability
ƒ Yes 41%
ƒ No 56%
Earnings in 3rd Quarter
Before Entry:
ƒ Less than $1000 52% 75% $5,053
ƒ $1000 - $2499 63% 84% $1,612
ƒ $2500 and Higher 64% 87% -$603
Earnings in 3rd Quarter
Before Entry
ƒ Less than $2500 86% 157%
ƒ $2500 to $4999 89% 90%
ƒ $5000 and Higher 90% 77%

8/4/2000 xxxvii
 February 2007

WIA Outcomes by Labor Market Factors: Recipients of Intensive Services Only

Earnings
Entered Employment Rate Employment Retention Rate Earnings Replacement
Labor Market Factor Dislocated Dislocated Change: Rate:
Adults Workers Adults Workers Adults Dislocated
Workers
% Employed in
Manufacturing, Agriculture,
& Mining
ƒ Less than 20% 74%
ƒ 20% to 39% 69%
ƒ 40% and Higher 67%
% of Population in Poverty
ƒ Less than 13% $3,750
ƒ 13% to 22% $2,760
ƒ 23% and Higher $2,700
Beale Code
ƒ Metropolitan County 58%
ƒ Nonmetropolitan Cnty 52%

8/4/2000 xxxviii