Innovative Approaches to Mental Health Evaluation

20201.

PREFACE

The idea for this book evolved over a series of coffee break discussions between the editors concerning the nature and prospects for the field of mental health program evaluation. In the process of conducting program evaluations, we have frequently had the experience, as have many professionals in the field, of being regarded as somewhat suspicious outsiders who have been commissioned to check up on program administrators and staff to see that they have been performing their duties adequately. Part of the reason for this distrust is that many evaluators have been insensitive to the special problems, attributes, characteristics, and limitations of the particular program area that they are evaluating. To carry out adequately an evaluation of a program, it is necessary to emphasize both halves of the term program evaluation. Evaluation designs and methods must be custom tailored to the unique qualities of the program and program areas that are to be assessed. Equally important, evaluators must be thoroughly familiar with the assessment tools of their trade.

In other words, evaluators bring with them an armamentarium of techniques and methods that must be selectively adapted and molded to the unique attributes and characteristics of the programs to be assessed. However, the evaluator’s effectiveness in designing and conducting a relevant evaluation study, in providing managers with useful results, and in having recommendations implemented is enhanced to the degree that the evaluator is sensitive to the specific issues and concerns inherent in the policy area or program under investigation. Evaluation should not be viewed as an end in itself, but as one integral part of a program development and management system.

The conceptual framework of this book reflects our belief that the foremost goal of evaluation is to produce relevant information geared to the decision-making needs of managers, service providers, and the public. Our intention in editing this book has been to place a more explicit focus on integrating evaluation strategies with the specific program areas than has traditionally been done. However, although we believe that the mental health policy issues of the 1980s will shape program evaluation methods and techniques, we are also aware that, as a budding subdiscipline, it is important to examine promising evaluation methodologies and approaches in their own rights. As a result, we have organized this volume into two sections. In Part I, Major Mental Health Issues: The Evaluative Response, eight chapters are presented describing what we believe to be the most salient mental health issues for the 1980s, and the evaluation strategies and approaches necessary to meet the challenge of these issues. On the basis of federal agency forward plans, the recently enacted Mental Health Systems Act, and other legislation, and the recent mental health research and policy literature, it appears that particular initiatives of the 1980s will be focused on prevention programs, linking health and mental health delivery systems, accountability in assuring quality of services, deinstitutionalizing the chronically mentally ill, and providing for greater local participation in mental health program management. These issue areas are those from which we have selected authors to write about what they believe to be the cutting edge of evaluation on these topics:

Wolfe and Schulberg begin this volume with an overview to what they foresee to be the mental health system of this decade, and the general trends in program evaluation to meet the needs of this system. The six chapters following Wolfe and Schulberg address these policy issues from an evaluation viewpoint.

In Chapter 2, Lorion and Lounsbury highlight a number of important conceptual distinctions and methodological issues that are essential to evaluating and improving mental health prevention programs. Wertlieb and Budman, in Chapter 3, review the current status and future directions for assessing health-mental health service delivery linkage programs, and offer their views of the most promising evaluative strategies for the coming decade. Samuels presents, in Chapter 4, a case study of an evaluation of the most important federal health-mental health linkage program attempted thus far. Then, in Chapter 5, Tash, Stahler, and Rappaport address the unique problems of how best to conduct an evaluation of a quality assurance system; and, in Chapter 6, Bachrach and Lamb discuss their contention that evaluations of deinstitutionalization programs have been based on faulty assumptions utilizing inappropriate measurements. Finally Dinkel, Windle, and Zinober, in Chapter 7, argue that program evaluation has suffered limited utility because of a lack of public participation. They offer suggestions for increasing the role of citizen evaluation.

Part II focuses on promising evaluation methodologies, approaches, and issues that are relevant to the emerging organizational and political environment of the mental health system of the 1980s:

Chapter 8, by Schmidt, Beyna, and Haar describes the process of evaluability assessment, a method that examines the gaps and problems in management’s program description and devises alternative descriptions or strategies for ameliorating these problems. In Chapter 9, Cook and Shadish construct and apply a metaevaluative model for assessing the congressionally mandated requirement in community mental health centers whereby 2% of each center’s total budget must be spent for evaluation. Wholey, in Chapter 10, discusses the necessity for evaluators to work in collaboration with managers to create the results-oriented management environment. In Chapter 11, Levine reviews the basic concepts in cost-effectiveness evaluation, and discusses how to apply these to evaluating mental health services delivery programs. Rocheleau, in Chapter 12, presents a plea for greater utilization of qualitative methods in program evaluation, and contends that these methodologies provide essential complements to the more frequently employed quantitative techniques. Chapter 13, by Majchrzak and Tash, reviews the essential elements of a performance measurement system and argues for one that combines the use of both global and specific measures. Bigelow, Stewart, and Olson, and Brodsky, in Chapter 14, describe an outcome measurement system in use in the state of Oregon that assesses client satisfaction, productivity, and quality of life. Such a performance measurement system will receive increasing attention as the mandates of the Mental Health Systems Act become implemented. Stevenson and Ciarlo, in Chapter 15, offer strategies to overcome one of the evaluator’s greatest banes—the lack of utilization of evaluation results and recommendations. Finally, in the last chapter, we summarize what our authors believe to be the necessary tools, methods, strategies, and issues to be addressed in the field of mental health program evaluation for the coming decade.

We hope that this book will be of interest to mental health administrators, managers, researchers, students, and evaluators of varying orientations and backgrounds. Our choice of authors is intended to provide the reader with a broad spectrum of evaluation perspectives from the academic–researcher to the government administrator-policymaker to the local center-level program evaluator. We believe that innovations in evaluation are occurring at all levels in the administrative hierarchy, each stratum having an impact on the others. Techniques developed by university researchers often are eventually disseminated to the practitioner. Similarly, new approaches to integrating evaluation and management at the government administrator level frequently filter down to the center-level manager (or sometimes reach him or her through blunt mandates).

We hope that the innovations and approaches presented in this volume will facilitate the dissemination of new ideas both within and across the various levels of program administration and evaluation. In a time of fiscal constraint, program evaluation will be increasingly called upon a provide feedback to the decision-making processes of program management and program development in the mental health arena. This improvement to mental health services and programs is the ultimate objective of program evaluation, and provides the basis for all that follows in this volume.

I

MAJOR MENTAL HEALTH ISSUES: THE EVALUATIVE RESPONSE

1

THE DESIGN AND EVALUATION OF FUTURE MENTAL HEALTH SYSTEMS

JOHN C. WOLFE and HERBERT C. SCHULBERG

Publisher Summary

The energy problems, world events, economy, and many other contemporary factors will shape the future of the mental health system and developments within the field itself. The ebb and flow of external and internal events reflect the Hegelian philosophy and insure a mental health system that constantly experiences change. This chapter provides a snapshot view of future systems and subsystems that relate to the mental health field. The greatest challenge of the present decade resides in the ability and energy of mental health system to bring healing to its member components, and to develop and maintain a vision of the future that rises above the morass of each day’s events. The accountability system has a significant role in determining whether or not programmatic priorities are being met with regard to the patient populations needing care, and in terms of the ability of the service delivery system to improve the functioning of its several target populations.

EDITOR’S INTRODUCTION

Wolfe and Schulberg set the stage for the following chapters by describing what they believe to be the mental health system of the coming decade. They see the emerging system as one increasingly oriented toward closer collaboration between general health and mental health. The mental health field’s multiple interest groups will be forced to work together to survive in an era of fiscal conservatism. The chronically mentally ill, the elderly, children, and minority groups will be highlighted for attention and resources. There will also be a broadening of the traditional domain of mental health concern to include such areas as weight control and life management consultation. Primary prevention will become a more visible part of the mental health system, and pressures from the major mental health disciplines for reimbursement of their individual services will continue to be intense.

The authors then discuss the role that evaluation will play in the future. They anticipate decreased participation of citizens and consumers in the evaluative process, with a larger focus on accountability studies.¹ Wolfe and Schulberg see evaluative attention being focused on such issues as providing clarification as to which populations are most appropriately served by community mental health centers (CMHCs), developing more detailed indices of performance measures for CMHCs, and ascertaining which benefits are received by which client groups. The authors still see major difficulties confronting the use and design of outcome studies. They suggest that research into the effectiveness of primary prevention, as well as cost–benefit analyses, however, will become a significant venture during the next several years. Overall, they view the accountability system as having a particularly significant role in determining programmatic priorities.

The mental health system of the future will be no more singular than that of today. In fact, the overall system will probably contain even more component subsystems than exist at present. This chapter attempts to provide a snapshot view of future systems and subsystems that relate to the mental health field; most are already familiar, but their functions and significance are expected to change. The reader could organize and identify these components by different groupings; however, the following are evident to us in the often chaotic world of mental health service delivery:

The Political System

The Patient System

The Delivery System

The Reimbursement System

The Manpower System

The Accountability System

Each system will be described in terms of its major present and future characteristics and the implications for change of the delivery of mental health services during the 1980s.

THE POLITICAL SYSTEM

History operates, according to Hegel, in a dialectical process that starts with a thesis, develops an antithesis, and later a synthesis. The synthesis becomes the thesis of the future and the entire process begins anew.

At the risk of oversimplifying past events, the political system in the United States for the past 20 years reflects this Hegelian dialectic as it relates to mental health. The Mental Retardation Facilities and Community Mental Health Centers Construction Act of 1963 was signed by President John Kennedy. Alongside other liberal legislation of the 1960s, the Community Mental Health Centers Act offered monies and new resources for the care of the mentally ill. During the 1970s, the federal administration attempted to eliminate this effort; funds were impounded, and threats were made to terminate all federal support of the program. In the late 1970s and in 1980, the mental health field experienced new hope with the creation of the President’s Commission on Mental Health (PCMH), which seated Mrs. Rosalynn Carter, the first lady, as its honorary chairperson. The 1978 report of the PCMH resulted in the administration’s proposal of the Mental Health Systems Act (MHSA), which was enacted in 1980. The MHSA represents the synthesis of the liberal policies of the 1960s (the thesis) and the reactionary policies of the 1970s (the antithesis). It purports to offer new directions with limited resources.

The 1980s offer a most fascinating canvas on which to paint the future relationship of the political and mental health systems. The Congress finds itself reacting to the taxpayer’s demand for relief and a new phrase—the Proposition 13 mentality—is used to describe many congressional actions. The legislative branch is now demanding that programs be accountable and cost effective. Some policymakers speak profoundly of mainstreaming mental health into health, while others talk about multiple funding sources. The political rhetoric becomes increasingly oriented toward the mental health system as though it is an entity whose parts are discernible and are synchronized. The political system itself is taking more powers away from the presidency; more decisions affecting mental health care are being made in the courts, and the capacity of Congress to deal knowledgeably with the mass of legislation and issues that it confronts is diminishing.

How will these developments affect the political system’s future relationship to mental health? It is safe to assume that, over the next 5 years, there will be an increased fiscal conservatism unlike anything experienced thus far. The juxtaposition of inflation and recession is a paradox even for the economist, and it encourages fiscal controls and conservatism. The Congress will find itself caught between the taxpayer and the new right (antiabortion, return to fundamental values of yesteryear, etc.) on the one hand, and those with a vested interest in mental health care on the other. Consequently, federal categorical dollars for mental health will continue to shrink in purchasing power, if not in actual volume, even though Titles XVIII and XIX will expand and provide a little more coverage. The only way to make the political system responsive to this country’s mental health needs is through massive lobbying efforts by all members of the field. This strategy is problematic, however, because of differences between the various disciplines, community care givers, and institutional providers, and the profit and nonprofit sectors.

The complexities of affecting public policy become even greater considering the relationship of federally mandated health planning agencies and the mental health system. The next few years will produce serious power struggles between the various health planning agencies within each state and the state departments of mental health. However, mental health and health planning agencies share enough compatible goals to work cooperatively by 1985.

The last half of this decade portends a brighter future for the mental health system. The antithesis of the early 1980s will occur as a result of economic, political, and international developments. The multiple interest groups in the mental health field will have been forced to work together to survive; coalitions will have developed, and individual agencies and associations will be significant members of a stronger and broader power base.

THE PATIENT SYSTEM

During the past 2 decades, a change of emphasis from institutional care to community care has occurred. Concomitant with this has been a change in the diagnostic categories of patient populations. Information from the National Institute of Mental Health indicates that 15.9% of additions to federally funded CMHCs in 1971 were diagnosed as schizophrenic; in 1977, the proportion had dropped to 14% (NIMH, unpublished data). Although CMHCs have been accused of treating those who are less severely mentally ill, upon further analysis, that may not be the case. It is possible that people are now being identified and treated in the early stages of emotional disturbance (or pathology); consequently, fewer cases are ultimately diagnosed as schizophrenic. Recent NIMH data indicate that the substantial increase in admissions of patients with other diagnoses has diluted the proportion of admissions of patients with schizophrenia, and the rapid expansion of the number of CMHCs has diluted the average number of admissions of patients with schizophrenia per CMHC [Goldman, Regier, Taube, Redick, & Bass, 1980 p. 86]. In fact, these authors emphasize "that the average absolute number of admissions per year of patients with schizophrenia … has remained relatively stable [Goldman et al., 1980 p. 85]." Nevertheless, some professionals and laymen continue to perceive that community mental health providers are neglecting schizophrenics and other seriously ill persons.

American society seems to have a penchant for emphasizing certain behavioral problems during a particular time period. For example, in the 1960s and 1970s public and political efforts were directed toward resolving both drug and alcohol problems. Substance abuse was, is, and will be troublesome; there had been a long history of neglect in this area that helped to focus resources toward combating its resulting problems. The fact remains, however, that the demand characteristics of some societal segment, other than professionals, created the resources that led to the development of needed treatment, research, and training facilities.

During the early 1980s, several patient populations will receive attention from the public and policymakers as well as from the media. The chronically mentally ill, the elderly, children, and minority groups are consistently presented as receiving short shrift from the mental health establishment. We do not dispute the veracity of such claims but would emphasize that a variety of external events cause particular population groups to be highlighted for attention and resources. Programs subsequently follow the demand.

A broadening of the spectrum perceived to be the domain of mental health concern will occur in the next decade. Behaviors such as smoking and overeating, and life crises such as separation, death, divorce, job stress, and job change will become more predominant in the diagnostic array of problems treated by mental health facilities. The more traditional kinds of pathology will continue in part of the patient population, but these new patient populations will affect the design and operation of service delivery systems.

THE DELIVERY SYSTEM

The form and philosophy of the mental health service delivery system has undergone dramatic change over the past several centuries. Deistic and demonological theories explained behavior in early cultures. People of the seventeenth and eighteenth centuries witnessed the development of institutions to deal with problems of the indigent, the dependent, and the vagrant (Rothman, 1971). The late eighteenth and the early nineteenth centuries ushered in the rise of moral treatment. However, the climate of the second half of the nineteenth century changed markedly from therapeutic optimism to pessimism, from a moral humanistic philosophy to reliance on scientific classification, from awareness of social factors to somatism. These changes, combined with other factors, resulted in the transformation of the small asylums into large custodial institutions [Golann & Eisdorfer, 1972, pp. 4, 5]. At the end of the nineteenth century, psychoanalysis introduced concepts regarding the importance of early childhood experiences as a basis of behavior, and addressed the relationship between normal and abnormal behaviors. Following World War II, dissatisfaction with the country’s mental health services increased, and new program directions centered around the concept of short-term treatment (Levenson, 1972, p. 687). Creation of the Joint Commission on Mental Illness and Mental Health in 1955 led to the development of CMHC programs in 1963 and the bold new approach to solving the problems of mental illness (Kennedy, 1963, p. 2).

It is not our purpose here to expound on all of these changes nor to describe comprehensively the elements that helped to create them. Rather, it suffices to acknowledge that change has occurred and to emphasize that change is occurring. The discovery and use of psychoactive drugs, and the revolution of the CMHC movement were both instrumental in shifting the service delivery system’s focus from inpatient to outpatient care. Thus, since 1955, the population of long-term residential mental institutions has declined from 600,000 to approximately 160,000 persons today (Sharfstein, 1980). This shift has created a very different system for delivering mental health treatments and provides a clue to future events.

Although the 1963 Community Mental Health Centers Act called for greater efforts in preventing mental illness, there was little activity by professionals as well as a dearth of resources for those wishing to become involved in this area. The public and policymakers wanted to see (and count) the results of tax dollars being spent to treat the mentally ill. Primary prevention efforts were supposedly less visible; it would take years to observe and measure the results, as research and evaluation methods were primitive at best. As a result, primary prevention was neglected.

In the late 1970s and 1980, a movement to make primary prevention a legitimate and visible part of the mental health system began. A new discipline formed out of this movement, and professionals who had previously identified with traditional mental health disciplines now assumed new roles as primary prevention professionals. The federal government has identified prevention as a priority within the Department of Health and Human Services (formerly the Department of Health, Education, and Welfare) and, during the next decade, this fledgling effort will be expanded. Community mental health centers will develop primary prevention programs aimed at populations at risk, such as those experiencing life crises (e.g., death, divorce, separation, birth, marriage, job change). We expect that other parts of the mental health system (e.g., independent entrepreneurs) will also participate in this developing delivery system. An increased emphasis on self-help groups will, in turn, provide prevention programs around specific life problems for many people who might otherwise enter the treatment system as patients. Conflicts and problems already are evident between this new self-help system and its work force and professionals who feel threatened by its emergence. It will be fascinating to observe whether the existing traditional mental health system can accommodate and absorb the new primary prevention movement, or whether the latter will spin off and develop its own power base.

Finally, there will be continuing movements toward increased ambulatory care for all patients within the mental health system. Short-term hospitalization will be utilized extensively. At the same time, there will be a reconceptualization (a synthesis) with regard to the chronically and most severely mentally ill. At least some of these patients will be returned to institutional settings providing better quality care. This will occur as both the public and professionals emphasize that this approach is cost effective and, given our present knowledge base, the best form of treatment for certain patients.

THE REIMBURSEMENT SYSTEM

The mental health system finds itself in the unenviable position (or enviable, depending on one’s perception) of being part of several larger systems. It is part of the health care system, the social or human service system, the vocational system, and the housing system. Funding is received from all of these major systems, but only with much difficulty. A CMHC must obtain funds from a variety of sources to carry out its mandate and meet the multiple needs of the population seeking help. The training and background of the CMHC staff, therefore, necessarily encompasses these several larger systems within which they operate. Institutional and long-term mental health care providers have had to confront less funding diversification in the past but, even here, the scene is changing.

Given the present complexities, which are likely to be compounded by the oscillating moods of Congress (possibly reflecting the public value system that appropriates generously at some times but, at others, works to decimate health and social programs), and international events ever more directly affect America’s fiscal and energy resources, it is difficult to predict future reimbursement patterns.

Nevertheless, indicators suggesting possible changes in reimbursement opportunities for the mental health system exist. For example, CMHC directors and boards are now considering diversification, that is, how can revenues legitimately be generated from sources other than categorical federal, state, and local dollars? (This even may include owning a commercial restaurant!) Public sector programs are beginning to realize the need for management systems that will maximize their abilities to capture fees for services, and state associations of CMHCs are lobbying for legislation to require private insurance carriers to reimburse for mental health treatment.

The 1980s will not be a decade of plenty in funding for the mental health system; it will, however, be a decade of experimentation in new efforts to capture scarce dollars. Some of the existing reimbursement mechanisms, such as Medicare, Medicaid, CHAP, and Champus will be gradually modified, but the change in these mechanisms will be slow and painful. Categorical mental health dollars from governmental bodies (federal, state, and local) will be difficult to obtain in the first half of the 1980s but will become more readily available after 1985. Pressures from the major mental health disciplines (psychiatry, psychology, social work, and nursing) for reimbursement of their individual services will continue to be intense.

THE MANPOWER SYSTEM

Previously described changes in the nature of the delivery of mental health service and fiscal reimbursement systems could profoundly affect its manpower system. Interesting events are already occurring that will shape the balance of disciplines providing care. Psychiatry is experiencing a decline in the number of medical graduates pursuing psychiatric residencies (from approximately 10% to 4%), and the profession is "concerned about the problem of adequate psychiatric involvement in public mental health services, including community mental health centers [Beigel, 1980]." Psychology is aggressively pursuing its identity as a discipline that is legitimately eligible for third-party reimbursement, and social workers and nurses are close behind in this quest. Mental health counselors have now established themselves as legitimate providers eligible for Blue Cross/Blue Shield reimbursement in the state of Maryland (Lesser, 1980).

There are many reasons for the decrease of psychiatric involvement in CMHCs, including, but not limited to, the issues of who should direct a CMHC, the proper tasks to be performed by a psychiatrist, the medical model orientation of a psychiatrist versus the human service or social model orientation of other disciplines, the lack of appropriate public mental health role models within psychiatric educational institutions, and the psychiatrist’s ability to earn more income with fewer administrative problems and interdisciplinary disputes in the private than in the public sector. This last issue warrants particular attention here.

Assuming that one of the basic motives for employment is to earn the maximum income possible, and assuming that professionals will avoid overly stressful work situations as much as possible, it is reasonable to predict that the other mental health disciplines (psychology, social work, nursing, and counseling) will follow psychiatry’s movement away from public service and CMHCs. As psychologists and members of other disciplines become eligible for third-party reimbursement for their services, they too will desert the public nonprofit system of organized care for the private profit-making sector. In this latter realm, they will not have to contend with and/or manage large organizations staffed by multidisciplinary professionals and nonprofessionals, boards of directors, muptiple funding sources, government regulations and guidelines, and interagency problems. Instead, they can achieve higher income levels with fewer and less obviously stressful situations. Thus, it is conceivable that, by the end of this decade, we may find the mental health service delivery system short of professionally trained personnel. Organized systems of mental health care (CMHCs and other mental health providers) may be predominantly staffed by the paraprofessional. In essence, evolving reimbursement policy intentionally may create a two-track system of mental health care in this country: one for persons having money and/or private and third-party coverage to reimburse for services, and the second for the poor and the indigent. The former system will be staffed by the major mental health disciplines, and the latter by nonprofessionals.

A critical question is whether anything can intervene to forestall this destructive and probable development. We suggest that if the public sector can design incentive mechanisms for those working in the mental health system (and there are the beginnings of such a concept), if the professional disciplines can reconcile their turf fights, and if the profit and nonprofit sectors can meaningfully collaborate in the delivery of care, it is possible that separate and unequal systems can be avoided. For this more beneficial outcome to occur, and there will be those trying to make these various events occur, new coalitions must be developed with the superordinate goal focused on quality care for all patients. In the absence of such a collective professional mission, the public will react adversely to the mental health system’s inability to get its act together and everyone, particularly patients, will be the losers. In essence, the mental health system will be held accountable for its narrow, self-centered conception of its societal role.

THE ACCOUNTABILITY SYSTEM

After presenting our vision of how future mental health system components will be designed and operated, we turn to foreseeable accountability mechanisms. Taxpayers will be inquiring more than ever about the effectiveness with which huge sums of public money are spent, and increasingly sophisticated accountability systems will be needed to provide meaningful responses. As these public pressures unfold during the 1980s, ironically, we anticipate a decreased role for citizens and consumers in the evaluative process. The 1975 Community Mental Health Centers Amendmerits encouraged these groups to become involved, but they have not sought nor been offered adequate training to competently participate in program evaluation’s technical tasks. Thus, the conduct and interpretation of accountability studies will primarily reside in the technocratic realm, while citizens and consumers will retain a voice in shaping the broad directions of evaluation.

Of the several system components previously described, we expect those pertaining to patients and service delivery to be most closely scrutinized by policymakers and administrators. The remainder of this chapter, therefore, focuses on conceptual and methodological issues intrinsic to the assessment of these components during the coming decade.

The Patient System

The question of whether or not appropriate clientele are being served by community mental health centers, and the degree to which client needs are being met has provoked considerable debate. Numerous studies to determine whether economically disadvantaged and minority groups are overtly or covertly dissuaded from seeking psychiatric care, and whether their treatment attrition rates differ from those of other clinical groups have been carried out (Rosen et al., 1980; Tischler et al., 1975). We expect that these populations, as well as the severely ill and others at high risk, will receive growing attention and early intervention to forestall more severe pathology. Evaluative procedures are needed, therefore, to clarify which groups are being treated and whether the volume of resources expended by CMHCs for their care is consistent with public priorities.

This form of program accountability is classified as the evaluation of effort, and such studies are performed on the premise that a patient’s participation in care implies a positive outcome. Client count data, and measures of how staff spend their time have long been collected through the NIMH annual Inventory of Comprehensive Community Mental Health Centers. In fact, the series of Mental Health Statistical Notes produced over the years by the NIMH Division of Biometry and Epidemiology on the basis of this and similar inventories has provided a meaningful picture of service utilization trends and shifts in the patient population being treated within psychiatric facilities. Nevertheless, it became apparent several years ago that evaluations of effort would recede in significance relative to outcome studies (Schulberg, 1977). Surprisingly, however, the former have emerged with renewed vigor in the 1980 Mental Health Systems Act (MHSA) under the label of performance measurement. That indicators of effort rather than criteria of effectiveness are the major evaluative strategy of MHSA is somewhat chagrining. This denouement reflects both the continuing conservatism of program directors and lingering difficulties in the design of outcome studies.

Analyses by Connolly and Deutsch (1980), and Keppler-Seid, Windle, and Woy (1980) of the rationale for and implications of utilizing performance measures emphasize their complexities and limitations. Nevertheless, 4 of the 13 performance indicators selected by NIMH pertain to CMHC accessibility, or admission patterns. They are the rates of significant minority members, persons under age 18, and persons over age 65 per 100,000 community residents admitted as patients, and the number of severely mentally disabled persons as a proportion of the total caseload of the CMHC. It is conceivable that experience with these initial performance measures will lead, in the coming years, to additional and more detailed indices of whether or not appropriate clientele are being served in CMHCs. We urge that this approach to accountability in the patient system proceed cautiously, bearing in mind that evidence of increased program effort may or may not signify that enrolled groups are benefitting from CMHC interventions. Given the predilection of administrators and others to reify body counts as if they denote far more than they actually do, prudent interpretation of these initial performance measurements seems particularly wise.

Another likely influence upon accountability studies of the patient system are the newly designed epidemiologic analyses of case prevalence. Community studies of how many and what types of persons need psychiatric care (the denominator in need assessment ratios) have been conducted for many years with the hope that improved diagnosis and treatment of mental disorders will follow from a better understanding of their patterns. However, even the most rigorous of these earlier investigations suffer from the ambiguity of whether cases uncovered through household interviews, for example, those classified in the Midtown Manhattan and Stirling County studies as manifesting severe impairment, resemble patients seen in care giving facilities. In recent years, however, epidemiologists have begun to assess caseness in community surveys with the same structured interviews as are used by clinicians to achieve research diagnoses. Thus, Weissman and Myers (1978) employed the Schedule For Affective Disorders (Endicott & Spitzer, 1978) in determining the prevalence of depression among New Haven residents, a strategy that permitted them to interpret more validly their findings about psychiatric need in the community. Through the fiscal support now provided by the NIMH Epidemiologic Catchment Area grant program, such community studies will become increasingly common and pertinent for determining true population needs. Their findings have significant potential for influencing the performance standards to be used in assessing the relevance and adequacy of CMHC services for particular populations at risk.

The Delivery System

The 1963 Community Mental Health Center Act was instrumental in shifting psychiatric care away from inpatient settings in distant, large institutions to ambulatory facilities in local, smaller settings. Given the major restructuring entailed in this transfer, much pressure has been exerted to determine whether patients indeed have benefitted from this altered pattern of service delivery. If, as we anticipate, primary prevention programs grow during the coming years, similar pressures will be exerted to assess their efficacy. Furthermore, as it has become apparent that the major portion of America’s care for emotionally disturbed persons occurs within the general health sector rather than within the specialty psychiatric system, the need to ascertain the benefits and liabilities of this reality are crucial as well.

What accountability mechanisms are available to study service outcome, and how adequate are they for administrative and policy purposes in the decade of the 1980s? The theoretical usefulness of outcome evaluations is generally acknowledged by administrators, clinicians, and researchers, but they also express considerable caution about our ability to conduct such assessments. Nevertheless, the priority assigned this accountability strategy is growing, and it is expected to have the salutary benefit of ultimately resolving the methodological issues still plaguing outcome studies. As this occurs, the cost–benefit ratio of evaluating the effectiveness of the service delivery system will improve markedly (i.e., the value of the information yielded by these studies for decision-making purposes will surpass their fiscal costs and scientific compromises).

Evaluations of the benefits generated by a service delivery system must consider the several target populations at whom care is directed and the specific outcome come indices pertinent to each (Schulberg, in press). Whereas clients are the primary concern of most administrators and clinicians, during the coming years, greater attention will be focused upon the mental health system’s direct, or indirect, effect upon other target populations as well. The families of clients, the communities within which clients reside, and the staff of psychiatric facilities will be studied to determine whether outcome patterns vary among these several target populations even though we have assumed until now that all groups benefit from a properly functioning system. Despite our faith that the services of a community mental health center are compatible, even synergistic, for all possible beneficiaries, goal conflicts may well exist among the target groups. Future outcome studies will consider not only whether an individual target group benefits relative to its own needs, but also whether positive or negative interactive patterns develop among the several populations affected by the service delivery system.

The issue of what benefits are derived by which client groups is critical with regard to recently developed community support programs for the chronically mentally ill (Schulberg & Bromet, in press). These demonstration programs seek to affect a wide variety of target groups whose goal compatibility is still unclear. The need to carefully evaluate synergistic as well as incongruous outcomes of comprehensive pilot efforts was stressed by Bachrach (1980). She noted that successful model programs for chronic mental patients share common structural elements, but their activities cannot be readily reproduced nor their effectiveness necessarily be generalized. Failure to explicate a program’s evident as well as subtle features and consequences, therefore, risks policy recommendations of dubious