Johns Hopkins Bloomberg School of Public Health HPM- 300.600 - Introduction to Health Policy Unit on Medical Care PolicyDr.

Jonathan Weiner Copyright 2005J. Weiner and The Johns Hopkins University Version8/8/05

Health Policy Analysis Checklist

For the Development, Selection, and Assessment of Program Policies Within Health Care Organizations This outline is intended as a guide to suggest issues/approaches you should consider when undertaking an applied policy analysis project. Not all aspects below are appropriate for all projects, and other issues beyond those noted here should sometimes be considered as well. Frequently the sequencing of real-world policy analysis is different than described here, and often the process is iterative; going back-and-forth from phase to phase. This document embraces most of the Bardach framework, but it expands upon it to also consider implementation, evaluation, and management issues that have direct relevance to public health policy analysis. This checklist was developed with special reference to health care delivery/financing organizations either within the private or public sectors. This document should also be relevant to most other health policy contexts. 1) Understand and define the problem: a) QuantifyBe specific about problems scope and magnitude i) Who is affected ii) Specific nature of the problem b) Past history c) Causes i) Key factors (separate into actionable vs. not) ii) Factors associated with action vs. inaction (errors of omission, missed opportunities vs. errors of commission)

2) Sources of information/evidence: a) Background literature on scope of problem b) Existing data on this population: i) Internal to organization ii) Other sources c) New data collection/analysis i) Re-analysis of existing data ii) New surveys/special studies iii) Commission problem assessment research (for major initiative) 3) Develop alternative solutions: a) Best practices/benchmarks (literature and consultants) b) Input from stakeholders/decision makers (also start of consensus process) c) Approaches to consider within complex health care system i) Financial incentives (provider, consumer, other) ii) Education/information (provider, consumer, other) iii) Organizational infrastructure New program Modification to existing program iv) Collaborate with others (community, other organizations) v) Policy considerations Internal policies Payers (e.g., employer or Medicaid) policies State (or federal) law or regulation 4) Identify criteria to be used in decision-making framework: (These are potential criteria used to assess alternative policies.) a) Population benefit i) Quality of life/outcomes of care ii) Concordance with evidence-based process standards iii) Improve access for those not getting ideal treatment iv) Reduce disparities in access, quality of care, outcomes b) Cost/economics
2

i) Cost/benefit (maximize positive effects given input costs) ii) Efficiency per unit of service (potential measure: QALYs, DALYs) iii) Contain costs (to stay within available resource budget and to insure margin between input and output costs) c) Ethics/equity i) Maximize total good (utilitarianism) ii) Consider fairness of resource allocation decisions, with attention to need, equity, etc. iii) Fulfill individuals rights to certain goods or outcomes (liberalism) iv) Consider trade-offs between individual autonomy and the common good d) Administrative feasibility i) Degree of organizations control ii) Timeframe of implementation/success iii) Resources required for developing and implementing policy (distinct from cost benefit of program itself) iv) Simplicity/robustness of policy/intervention (the fewer moving parts, the better) v) Legal/regulatory issues e) The perspectives of various constituencies (beyond patient/consumer above) i) Providers ii) Sponsors (employers, Medicaid agency) iii) Community groups and politicians iv) Employees of your organization v) Board of directors (shareholders, if investor owned) Consider costs and benefits to various constituencies, and the politics involved 5. Estimating impact of your Policy alternative: (Note: steps 5 and 6 are to some extent done iteratively/concurrently.) Use a phased assessment. All potential alternatives should be assessed (scored) on preliminary basis. Move on to in-depth assessment for top alternatives. Various approaches (often termed evidence-based policy tools): a) Review the existing evidence
3

i) The published literature. If possible, rely on evidence-based medicine (EBM) meta-analysis. ii) The experience to date of other providers/programs (possible site visits) b) Estimate impact on health outcomes using quantitative techniques (often drawn from the decision sciences). These may include operations research/forecasting/statistical analysis. c) Estimate economic impact using cost-effectiveness and cost-benefit analysis. (Note that these techniques can be applied during both this policy development phase and the program evaluation phase.) i) Array alternatives based on dollar inputs and outputs (CBA) or ii) Dollars as inputs and a variety of outputs (CEA) e) As needed, perform policy formulation evaluation research i) Collect information from your delivery system to help populate the decision matrix (see #6 below) ii) Initiate a small-scale pilot program and evaluate to aid in selecting one policy alternative. (Obviously, only for major initiatives or extremely innovative ones for which there is no track record) f) Get some help on all of the above i) Get expert opinions (such as group process/Delphi panels) ii) Hire paid consultants iii) Commission policy development project by outside experts 6. Decision-making process and development of decision matrix: a) Form decision-making group. (Need to decide which constituencies will have advise and/or consent roles.) Develop final decision matrix with groups input b) Develop decision matrix (see model in Appendix) by operationalizing a subset of the criteria you identified above. Determine relative importance (relative weights) of each criterion. As needed, do sensitivity analyses of alternative weights (see Appendix). c) For each alternative, develop a quantitative score for its characteristics along each decision criteria to help array trade-offs. As needed, undertake
4

sensitivity analysis assessing implications of different scores if outcome is unclear (see Appendix). d) After completing the matrix, consider making changes i) Think about the possibility of combining some of the strengths of multiple policy alternatives into a new combined policy approach. ii) Modify aspects of the matrix as indicated by decision making body. iii) As appropriate, involve other key executives and board members, in selecting final policy alternative. e) As appropriate to the context, develop final decision memo to carefully document both quantitative and qualitative rationale for selection of policy alternative and to summarize above decision-making process. This is mainly an internal document but will also be used in external processes (see #7). 7. Policy advocacy: getting the policy adopted a) Develop series of public documents: you will need one-pager, 35-pager, and detailed document (if a major initiative) i) Include rationale for policy decision (based on decision memo) ii) Explain why other alternatives were rejected iii) Offer some details of implementation b) Develop strategy/PR campaign (tell some real stories) c) Meet with key constituencies one-on-one and in groups to get feedback and buy-in d) Advisory group should address, as appropriate, final program design, implementation, and evaluation. (The development of a neat package to hand-off can be considered part of the advocacy phase.) 8. Implementation, improvement, and evaluation: (Beyond Bardach) Program implementation and management issues are largely out of the scope of this class. But the policy analysis process does not stop with the decision memo. The following actions can be considered as part of the health care policy process. Moreover, in the real world, often the same analyst will be involved in the policy analysis phase of a project, and these subsequent steps.
5

a) Policy team should liaise with implementation/program management group. b) Develop evaluation plan. This is part of the policy selection and implementation continuum: policy/planning/implementation/management/ evaluation. i) Formative evaluations emphasize attainment of program operational objectives and are essential during the program formation process. This can be considered a type of quality improvement assessment. ii) Summative evaluations are necessary to determine if the program or policy has had its desired overall effect. (These are also called impact or end-result evaluations.) c) Program (policy) improvement: revise your program/policy based on experience and formative evaluation. Consider expanding, eliminating or otherwise modifying the program/policy. d) Consider disseminating successes (or failures?) to a wider audience i) Within your organization ii) Your community iii) Nationally/internationally (via publications)

Appendix: Decision making matrix for evaluating policy alternatives Using the matrix below, for each policy alternative, describe what you know in terms of each decision criterion (or sub-criterion) you developed based on factors outlined in #4 of the Health Policy Analysis Checklist. These criteria should include both positive and negative characteristics. Attempt to assign a relative importance weight to each criterion (or subcriterion). For example, a criterion might contribute 40% to the final decision, while others would be weighted at 10%, 20% etc, summing up to total of 100%. Based on some of the approaches outlined in section #5 of the checklist, develop a score for each policy alternative along each decision criteria dimension (or sub-dimension). You may use a 1-10 score or --, -, =, +, ++ .
Policy Alternatives (Use More Sheets For Other Alternatives) I II

Criterion for scoring (Note subcriteria as appropriate) - Population Benefit

Weight _____ - Cost/Economics

Score_____

Score_____

Weight _____

Score_____

Score_____

- Ethics/Equity

Weight _____ - Administrative Feasibility

Score_____

Score_____

Weight _____ -Constituency Perspectives

Score_____

Score_____

Weight _____

Score_____

Score_____

Bottom Line Summary: Option #1 Overall Score:____

Option #2 Overall Score:___

To develop a bottom line score to support your decision process, multiply each criterions weighting factor by the score for each alternative policy. Then sum scores across each criterion to calculate a bottom line overall score for each alternative. (You may use a 1100 final score, or some other scale.) You will likely want to repeat this summation process several times with slightly different scores or weights, to gauge the sensitivity of your final decision to your scores on each criterion.
8