Sei sulla pagina 1di 24
Using Evaluation Tools to Drive Quality Improvement 
 Andrew Sarkin, PhD TEQI Summit October 15,

Using Evaluation Tools to Drive Quality Improvement

Using Evaluation Tools to Drive Quality Improvement 
 Andrew Sarkin, PhD TEQI Summit October 15, 2015

Andrew Sarkin, PhD TEQI Summit October 15, 2015

Using Evaluation Tools to Drive Quality Improvement 
 Andrew Sarkin, PhD TEQI Summit October 15, 2015
Using Evaluation Tools to Drive Quality Improvement 
 Andrew Sarkin, PhD TEQI Summit October 15, 2015
Using Evaluation Tools to Drive Quality Improvement 
 Andrew Sarkin, PhD TEQI Summit October 15, 2015
Using Evaluation Tools to Drive Quality Improvement 
 Andrew Sarkin, PhD TEQI Summit October 15, 2015
Using Evaluation Tools to Drive Quality Improvement 
 Andrew Sarkin, PhD TEQI Summit October 15, 2015
Agenda
Agenda

! Introduction to quality improvement

! Examples of quality improvement

! A basic strategy for positive change

! The PDSA cycle for quality improvement

! Setting SMARTER goals for TAY programs

! Measuring quality improvement

! Quality Improvement Planning Worksheet

! Comments, sharing ideas, and questions

! Measuring quality improvement ! Quality Improvement Planning Worksheet ! Comments, sharing ideas, and questions
! Measuring quality improvement ! Quality Improvement Planning Worksheet ! Comments, sharing ideas, and questions
What is Quality Improvement?
What is Quality Improvement?

! Quality improvement involves systematic actions that lead to measureable improvements in quality of services, access to services, client outcomes, and/or client satisfaction. ! In order to improve your program or county’s services, client outcomes, and client satisfaction, we implement manageable changes in our current system.

or county’s services, client outcomes, and client satisfaction, we implement manageable changes in our current system.
or county’s services, client outcomes, and client satisfaction, we implement manageable changes in our current system.
What is Quality Improvement?
What is Quality Improvement?

! A program’s current system consists of:

Inputs: Resources (e.g., people, information, materials, regulations)

Processes:

Activities that are carried out by the program How these activities are done (i.e., when they occur, where they occur, who provides these activities)

Results: Outcomes (e.g., services provided, changes in client health, changes in client satisfaction)

activities) ◦ Results : Outcomes (e.g., services provided, changes in client health, changes in client satisfaction)
activities) ◦ Results : Outcomes (e.g., services provided, changes in client health, changes in client satisfaction)
Why Engage in Quality Improvement?
Why Engage in Quality Improvement?

Improved outcomes:

Improved services provided

Improved client health

Improved client satisfaction with services

Improved efficiency:

Reduction or elimination of unnecessary or less effective services or processes

Avoided costs:

Improved ability to anticipate and solve problems before they occur

Improved ability to recognize problems early on and address them more promptly

Improved program reputation:

Programs that focus on quality improvement are appealing to current and potential funders and other community partners

Programs that focus on quality improvement are appealing to current and potential funders and other community
Programs that focus on quality improvement are appealing to current and potential funders and other community
Continuous Quality Improvement Takes Many Forms - Examples ! Adding a substance abuse counselor to

Continuous Quality Improvement Takes Many Forms - Examples

Continuous Quality Improvement Takes Many Forms - Examples ! Adding a substance abuse counselor to a

! Adding a substance abuse counselor to a Transitional Age Youth (TAY) program that has a high prevalence of addiction problems

! Reducing wait times to improve satisfaction of people getting services at a TAY program

! Increasing outreach to targeted groups who are not using a program as much as expected

! Providing technical training to enhance staff ability to utilize outcomes information for individual treatment

! Providing technical training to enhance staff ability to utilize outcomes information for individual treatment
! Providing technical training to enhance staff ability to utilize outcomes information for individual treatment
Your Examples
Your Examples

! What are some brief examples of quality improvement projects that you have done, or wanted to implement for Transitional Age Youth, in your counties or programs?

projects that you have done, or wanted to implement for Transitional Age Youth, in your counties
projects that you have done, or wanted to implement for Transitional Age Youth, in your counties
Implement Positive Change
Implement Positive Change

! Identify an area for creating a positive change

! Set SMARTER goals for improvement

! Create and implement a strategy for change

! Monitor the process of implementing change

! Measure the impact on your goal (outcomes)

! Disseminate the results for learning

! Organize using the Plan-Do-Study-Act cycle

! NOTE – It is beneficial to include all of your stakeholders at each stage of the process.

Plan-Do-Study-Act cycle ! NOTE – It is beneficial to include all of your stakeholders at each
Plan-Do-Study-Act cycle ! NOTE – It is beneficial to include all of your stakeholders at each
The Plan-Do-Study-Act (PDSA) Cycle
The Plan-Do-Study-Act (PDSA) Cycle
The Plan-Do-Study-Act (PDSA) Cycle • Determine what changes should be made • Prepare for the next

Determine what changes should be made

Prepare for the next cycle

• Determine what changes should be made • Prepare for the next cycle

Study

Analyze data

Compare results to predictions

Summarize and reflect on what was learned

• Analyze data • Compare results to predictions • Summarize and reflect on what was learned

Do

Act

Act • Objective (SMART goals) • Predictions • Plan to carry out the cycle (Who? What?

Objective (SMART goals)

Predictions

Plan to carry out the cycle (Who? What? When? Where?)

Data collection plan

Plan

• Carry out the plan • Document observations • Record data

Carry out the plan

Document observations

Record data

When? Where?) • Data collection plan Plan • Carry out the plan • Document observations •
When? Where?) • Data collection plan Plan • Carry out the plan • Document observations •
When? Where?) • Data collection plan Plan • Carry out the plan • Document observations •
SMART Goals
SMART Goals

! Specific

! Measurable

! Attainable

! Relevant

! Time-bound

SMARTER Goals are also:

! Evaluated

! Reviewed

! S pecific ! M easurable ! A ttainable ! R elevant ! T ime-bound SMARTER
! S pecific ! M easurable ! A ttainable ! R elevant ! T ime-bound SMARTER
SMART Goals Exercise
SMART Goals Exercise

! There will be zero suicides per year in our county among Transitional Age Youth 16-25.

SMART Goals Exercise ! There will be zero suicides per year in our county among Transitional
SMART Goals Exercise ! There will be zero suicides per year in our county among Transitional
SMART Goals Exercise
SMART Goals Exercise

! We will reduce wait times for Transitional Age Youth mental health clients by 2017.

SMART Goals Exercise ! We will reduce wait times for Transitional Age Youth mental health clients
SMART Goals Exercise ! We will reduce wait times for Transitional Age Youth mental health clients
SMART Goals Exercise
SMART Goals Exercise

! There will be a 10% increase from 2015 to 2017 in the overall satisfaction with services as measured by the MHSIP Consumer Perception Survey, among TAY clients.

to 2017 in the overall satisfaction with services as measured by the MHSIP Consumer Perception Survey,
to 2017 in the overall satisfaction with services as measured by the MHSIP Consumer Perception Survey,
Measure Program Improvement
Measure Program Improvement

! Process Measures

Outreach and engagement with targeted population

Discharge reasons and referral patterns

Involvement of family and friends

Medication and treatment adherence

Satisfaction with services

! Outcome Measures

Mental health as rated by a clinician (MORS, IMR)

Self-reported client-centered outcomes (RMQ, MHSIP)

Changes in the magnitude of improvement

Percentage of people who improve significantly

outcomes (RMQ, MHSIP) ◦ Changes in the magnitude of improvement ◦ Percentage of people who improve
outcomes (RMQ, MHSIP) ◦ Changes in the magnitude of improvement ◦ Percentage of people who improve

My beliefs were considered as

Process Measures Examples part of the services that I
Process Measures Examples
part of the services that I

received.

I felt comfortable asking questions

about treatment and medication.

4.8

4.6

4.4

4.2

4.0

3.8

4.8 4.6 4.38 4.4 4.2 4.00 4 3.8
4.8
4.6
4.38
4.4
4.2
4.00
4
3.8
4.64 4.23
4.64
4.23

cal year 2011-2012 (N=26)

4.4 4.2 4.0 3.8 4.8 4.6 4.38 4.4 4.2 4.00 4 3.8 4.64 4.23 cal year

Fiscal year 2011-2012 (N=22)

4.4 4.2 4.0 3.8 4.8 4.6 4.38 4.4 4.2 4.00 4 3.8 4.64 4.23 cal year
Outcome Measures Examples I deal more effectively with daily
Outcome Measures Examples
I deal more effectively with daily

My mental health has improved

problems

4.0

3.8

3.6

3.4

3.2

3.0

3.72 3.15
3.72
3.15

4.0

3.8

3.6

3.4

3.2

3.0

3.88 3.40
3.88
3.40

scal year 2011-2012 (N=33)

3.8 3.6 3.4 3.2 3.0 3.72 3.15 4.0 3.8 3.6 3.4 3.2 3.0 3.88 3.40 scal

Fiscal year 2011-2012 (N=35)

3.8 3.6 3.4 3.2 3.0 3.72 3.15 4.0 3.8 3.6 3.4 3.2 3.0 3.88 3.40 scal
Reduced Inpatient Admissions Percentage of Clients with Inpatient Admissions
Reduced Inpatient Admissions
Percentage of Clients with Inpatient
Admissions

44.0%

33.0%

22.0%

11.0%

0.0%

43.8% 43.5% 41.1% 26.2%
43.8%
43.5%
41.1%
26.2%

One Year Pre

One Year Post

Before P After PIP
Before P After PIP After PIP

Admissions 44.0% 33.0% 22.0% 11.0% 0.0% 43.8% 43.5% 41.1% 26.2% One Year Pre One Year Post
Admissions 44.0% 33.0% 22.0% 11.0% 0.0% 43.8% 43.5% 41.1% 26.2% One Year Pre One Year Post
Admissions 44.0% 33.0% 22.0% 11.0% 0.0% 43.8% 43.5% 41.1% 26.2% One Year Pre One Year Post
Improving Job Opportunities Percentage of Transitional Age Youth with Jobs
Improving Job Opportunities
Percentage of Transitional Age Youth with
Jobs

90

67.5

45

22.5

0

84.4 69.2 45.7 43.4
84.4
69.2
45.7
43.4

Baseline

6 Months

Before P After PIP
Before P After PIP After PIP

90 67.5 45 22.5 0 84.4 69.2 45.7 43.4 Baseline 6 Months Before P After PIP

PIP = Performance Improvement Project

90 67.5 45 22.5 0 84.4 69.2 45.7 43.4 Baseline 6 Months Before P After PIP
Quality Improvement Worksheet
Quality Improvement Worksheet

INSERT PICTURE OF FORM HERE

Quality Improvement Worksheet INSERT PICTURE OF FORM HERE
Quality Improvement Worksheet INSERT PICTURE OF FORM HERE
Discussion and Questions
Discussion and Questions

! How can we use these tools to help empower evaluation and quality improvement?

! How can we create better tools for continuous program improvement?

! What other tools or strategies are useful to accomplish quality improvement for TAY?

! How can we improve these trainings and what topics should we address in the future?

! Anything you want to share that is relevant!

improve these trainings and what topics should we address in the future? ! Anything you want
improve these trainings and what topics should we address in the future? ! Anything you want
Resources
Resources

! U.S. Department of Health and Human Services (http://www.hrsa.gov/quality/)

o HRSA has a Quality Improvement toolkit available on their website, which also offers an introductory training on QI. It is available as a pre-recorded audio and video modules in Real Player software format.

! SMART Goals Worksheet

! SAMHSA PDSA Worksheet

SMARTerGoals.pdf ! SAMHSA PDSA Worksheet http://www.integration.samhsa.gov/pbhci-learning- community/pdsa.pdf ◦
SMARTerGoals.pdf ! SAMHSA PDSA Worksheet http://www.integration.samhsa.gov/pbhci-learning- community/pdsa.pdf ◦
Thank you!
Thank you!

Andrew Sarkin, PhD asarkin@ucsd.edu hsrc.ucsd.edu mhse.ucsd.edu

Thank you! Andrew Sarkin, PhD asarkin@ucsd.edu hsrc.ucsd.edu mhse.ucsd.edu
Thank you! Andrew Sarkin, PhD asarkin@ucsd.edu hsrc.ucsd.edu mhse.ucsd.edu
Change in the Magnitude of Improvement for an Outcome 5 4 3 2 1 2012

Change in the Magnitude of Improvement for an Outcome

Change in the Magnitude of Improvement for an Outcome 5 4 3 2 1 2012 2013
5 4 3 2 1 2012 2013
5
4
3
2
1
2012
2013

BaselineChange in the Magnitude of Improvement for an Outcome 5 4 3 2 1 2012 2013

6 MonthChange in the Magnitude of Improvement for an Outcome 5 4 3 2 1 2012 2013