Using Evaluation Tools to Drive

Quality Improvement
Andrew Sarkin, PhD
TEQI Summit
October 15, 2015

Agenda
!
!
!
!
!
!
!
!

Introduction to quality improvement

Examples of quality improvement
A basic strategy for positive change
The PDSA cycle for quality improvement
Setting SMARTER goals for TAY programs
Measuring quality improvement
Quality Improvement Planning Worksheet
Comments, sharing ideas, and questions

What is Quality Improvement?

!

Quality improvement involves systematic

actions that lead to measureable
improvements in quality of services, access to
services, client outcomes, and/or client
satisfaction.
In order to improve your program or countys
services, client outcomes, and client
satisfaction, we implement manageable
changes in our current system.

What is Quality Improvement?

!

A programs current system consists of:

Inputs: Resources (e.g., people, information, materials,
regulations)
Processes:
Activities that are carried out by the program
How these activities are done (i.e., when they
occur, where they occur, who provides these
activities)
Results: Outcomes (e.g., services provided, changes
in client health, changes in client satisfaction)

Why Engage in Quality Improvement?

Improved outcomes:

Improved services provided

Improved client health
Improved client satisfaction with services

Improved efficiency:

Reduction or elimination of unnecessary or less effective

services or processes

Avoided costs:

Improved ability to anticipate and solve problems before they

occur
Improved ability to recognize problems early on and address
them more promptly

Improved program reputation:

Programs that focus on quality improvement are appealing to

current and potential funders and other community partners

Continuous Quality Improvement

Takes Many Forms - Examples
!

Adding a substance abuse counselor to a

Transitional Age Youth (TAY) program that has a
high prevalence of addiction problems
Reducing wait times to improve satisfaction of
people getting services at a TAY program
Increasing outreach to targeted groups who are not
using a program as much as expected
Providing technical training to enhance staff ability
to utilize outcomes information for individual
treatment

Your Examples
!

What are some brief examples of quality

improvement projects that you have done, or
wanted to implement for Transitional Age
Youth, in your counties or programs?

Implement Positive Change

!
!
!
!
!
!
!

Identify an area for creating a positive change

Set SMARTER goals for improvement
Create and implement a strategy for change
Monitor the process of implementing change
Measure the impact on your goal (outcomes)
Disseminate the results for learning
Organize using the Plan-Do-Study-Act cycle
NOTE It is beneficial to include all of your
stakeholders at each stage of the process.

The Plan-Do-Study-Act (PDSA) Cycle

Act

Determine what
changes should be
made
Prepare for the next
cycle

Analyze data
Compare results to
predictions
Summarize and reflect
on what was learned

Study

Plan

Do

Objective (SMART
goals)
Predictions
Plan to carry out
the cycle (Who?
What? When?
Where?)
Data collection plan
Carry out the plan
Document
observations
Record data

SMART Goals

Specific
! Measurable
! Attainable
! Relevant
! Time-bound
!

SMARTER Goals are also:

! Evaluated
! Reviewed

SMART Goals Exercise

! There

will be zero suicides per

year in our county among
Transitional Age Youth 16-25.

SMART Goals Exercise

! We

will reduce wait times for

Transitional Age Youth mental
health clients by 2017.

SMART Goals Exercise

! There

will be a 10% increase

from 2015 to 2017 in the
overall satisfaction with
services as measured by the
MHSIP Consumer Perception
Survey, among TAY clients.

Measure Program Improvement

!

Process Measures

Outreach and engagement with targeted population

Discharge reasons and referral patterns
Involvement of family and friends
Medication and treatment adherence
Satisfaction with services

Outcome Measures

Mental health as rated by a clinician (MORS, IMR)

Self-reported client-centered outcomes (RMQ, MHSIP)
Changes in the magnitude of improvement
Percentage of people who improve significantly

My beliefs were considered as

part of the services that I
received.

I felt comfortable
asking questions
Process Measures
Examples

4.8

4.8

4.64

4.6

4.6

4.38

4.4

4.4

4.23

4.2

4.0

about treatment and medication.

4.2

4.00

3.8

scal year 2011-2012 (N=26)

3.8

Fiscal year 2011-2012 (N=22)

Outcome Measures Examples

I deal more effectively with daily
problems

My mental health has improved

4.0

4.0

3.72

3.8

3.8

3.6

3.6

3.4

3.4

3.2

3.15

3.0

scal year 2011-2012 (N=33)

3.88

3.40

3.2

3.0

Fiscal year 2011-2012 (N=35)

Reduced Inpatient Admissions

Percentage of Clients with Inpatient
Admissions

44.0%

43.8%

41.1%

43.5%

33.0%

26.2%

Before PI
After PIP

22.0%

11.0%

0.0%

One Year Pre

One Year Post

Percentage of Transitional Age Youth with

Improving
Job
Opportunities
Jobs
90

84.4
69.2

67.5

45

43.4

45.7

Before PI
After PIP

22.5

Baseline

6 Months
PIP = Performance Improvement Project

Quality Improvement Worksheet

INSERT PICTURE OF FORM HERE

Discussion and Questions

!

How can we use these tools to help empower

evaluation and quality improvement?
How can we create better tools for continuous
program improvement?
What other tools or strategies are useful to
accomplish quality improvement for TAY?
How can we improve these trainings and what
topics should we address in the future?
Anything you want to share that is relevant!

Resources
!

U.S. Department of Health and Human Services

(http://www.hrsa.gov/quality/)
o HRSA has a Quality Improvement toolkit available on
their website, which also offers an introductory
training on QI. It is available as a pre-recorded audio
and video modules in Real Player software format.
SMART Goals Worksheet
http://www.ucdmc.ucdavis.edu/facultydev/pdfs/
SMARTerGoals.pdf
SAMHSA PDSA Worksheet
http://www.integration.samhsa.gov/pbhci-learningcommunity/pdsa.pdf

Thank you!

Andrew Sarkin, PhD

asarkin@ucsd.edu
hsrc.ucsd.edu
mhse.ucsd.edu

Change in the Magnitude of

Improvement for an Outcome
5

Baseline
6 Month

2012

2013