The International Journal of Psychosocial Rehabilitation

A Clinical Impact Evaluation of Integrated and Disease Specific Substance Abuse Program Models in Honduras

Rosa Aguilera, Lic.
Projecto Victoria

Arthur Anderson, Ph.D.
Saybrook Institute & Antioch University

Edwardo Gabrie, M.D.
Santa Rosita

Mohemy Merlo, M.D.
Santa Rosita

Teresa Paredes, Lic.
Projecto Victoria

Rosario Pastrana, M.D.
Santa Rosita
 

Citation:Aguilera, R., Anderson, A.J., Gabrie, E., Merlo, M., ParedesT., & Pastrana, R. (1999) A Clinical Impact Evaluation of Integrated and Disease Specific Substance Abuse Program Models in Honduras.  International Journal of Psychosocial Rehabilitation, 3, 97-167

---

Abstract
This study investigated and compared the clinical impact and program effectiveness of 2 treatment programs for mentally ill chemical abusers (MICA patients) in Honduras. A review of the central issues in MICA patient treatment, treatment strategies/models, and program development is provided. Patient treatment needs, historical context for divisions of service or systems, treatment philosophies, and program model components are also reviewed. The potential benefits of treating MICA patients in both integrated and disease-specific treatment programs is presented in addition to an in-depth discussion of methodological approaches used to determine mental health program effectiveness. This provides the rationale for the comparative program evaluation methodology used in this study to determine the effectiveness of integrated and disease-specific treatment models to meet the treatment needs of their target MICA populations.

This study utilized an innovative methodology by employing both standard indicators, developed by the principal investigator, and local indicators that were developed by the programs. This method is actually a modified 4th generation responsive evaluation model, in which the standard indicators were negotiated by all the stakeholders of each treatment program. The local program stakeholders focused their indicators to gather information they deemed critical for quality improvement in their programs. Each of these stakeholders had an active stake in the outcome of this investigation and were given an equal opportunity to decide what level of participation and involvement they would have in indicator determination, measurement, and data analysis.

This evaluation investigated the clinical impact of program services in Santa Rosita (a disease-specific program) and Proyecto Victoria (an integrated program). Both programs are located within 40 kilometers of Tegucigalpa, Honduras, Central America. Results of all standard indicators were compared and contrasted to determine the clinical and programmatic benefits of each program model in treating a similar MICA population.

Though diluted by socioeconomic factors, the results generally support the findings of the original pilot study. Higher functioning MICA patients tended to have more positive outcomes in structured disease-specific programs, while lower functioning patients respond better to treatment in integrated program settings. The more sensitive measures of clinical improvement, determined through the use of the LOFA, displayed a 40% reduction in psychotic symptoms and mental illness, and a 36% improvement in interpersonal relations in the disease-specific program. The integrated program measures show a 22% reduction in psychiatric symptoms and 22% improvement in community life and interpersonal relations. The need for greater cultural sensitivity in investigating mental health and substance abuse programs in developing nations is also discussed.

---

This work was supported by a international development grant from the Social Science Research Council. New York, NY. Grant # 95-0168.  We thank the clinical staff of Santa Rosita and Projecto Victoria for their assistance with this project and input into the process evaluation aspects of the study.  Special thanks to IHADFA for their assistance with the project and publication of the results in the Honduran Press.
Correspondence may be addressed to Dr. A.J. Anderson, Suite 3, 34 Frognal Lane, London NW3 7DT  UK

---

Short Cuts
   1. Introduction
   2. Historical Context and Literature Review
   3. Epistemological Issues
   4. Hypothesis
   5. Method
       Program Descriptions (Santa Rosita) (Projecto Victoria)
   6. Limitations and Delimitations
   7. Results
   8. Discussion
   9. References

---

1. INTRODUCTION

The chronic mentally ill patient who also suffers from substance abuse problems (drugs, alcohol or both) poses a unique set of difficulties for treatment programming. Such patients present a variety of individual, social, fiscal, and political challenges to effective program planning, design, implementation, and evaluation. Dually diagnosed patients not only require intensive psychiatric treatment for mental illness, but concomitant treatment for substance abuse symptomatology as well. As a consequence, these patients tend to stretch the ability of traditional community-based treatment programs to deliver adequate services to effectively meet their multiple treatment needs. This is particularly true in developing countries, such as Honduras, where resources for mental health care systems are severely limited. The relative lack of funds for mental health and substance abuse programs in developing countries further complicates treatment for this population and has a significant impact on efficacy of treatment for both single and dual diagnosis patients at the community level (Des Jarlais, Eisenberg, Good, & Kleinman, 1995). Such problems in all mental health delivery and prevention systems have led to the development of a variety of treatment models designed to treat mentally ill chemical abuse (MICA) patients (Bachrach, 1984; Drake, Antosca et al.,1991; Minkoff 1987). This review of such program models, their underlying theoretical and philosophical assumptions, and historical development demonstrates the utility of each model to adequately meet the multiple needs of MICA patients.

In industrialized nations, deinstitutionalization and the corresponding increase in the number of homeless mentally ill has been associated with the emergence of a growing population of patients with concomitant mental illness and chemical abuse (MICA patients) (Drake, Osher & Wallach, 1989). Numerous studies have demonstrated a rate of substance abuse and or dependency among the mentally ill at between 32 and 85 % (Safer, 1987; Schwartz & Goldfinger, 1981). MICA patients are the most frequently cited population of dually diagnosed patients in the professional literature (PsycINFO, 1993). They have been reported to utilize higher rates of acute hospitalization, have histories of more housing instability, homelessness, criminality, and homicidal or suicidal behavior than either the mentally ill or chemical abusers alone (Caron, 1981; Drake et al. 1989; Osher & Kofoed, 1989; Safer, 1987). Poor medication compliance and response to treatment have also been linked to this dual disorder (LaPorte, 1989; McClelland, 1986).

Though remarkable progress has been made in improving general health for developing nations, unfortunately this has been accompanied by a deterioration in mental health for the dually diagnosed and other populations. In many areas outside Europe and North America, reported cases of schizophrenia, depression, dementia, and concomitant substance abuse have risen dramatically. In low-income societies, 24.4 million people will be affected by some form of mental illness by the year 2000. This is an increase of 45% since 1985 (Kleinman & Cohen, 1997). Rapid urbanization, chaotic modernization, and economic restructuring of many societies have fractured social supports and extended family structures, increasing violence, substance abuse and suicide (World Health Organization [WHO], 1995).

MICA patients in these low income societies are particularly affected by the lack of clinical resources and options because their multiple disabilities require more clinical resources than are generally available. When combined with an increase in overall use of both medical and psychiatric care facilities, MICA patients generally are not treated in programs designed to meet their multiple needs (WHO, 1995). In developing nations the clinical needs of such patients must be evaluated in the context of their social structures to effectively treat this difficult and growing population in a culturally sensitive manner (Kleinman & Cohen, 1997).

In Honduras, the Ministry of Health Institute for Alcoholism and Addiction (IHADFA) estimated that alcoholism and addiction to substances is present in at least 45% of all patients seen in the national psychiatric hospital, Santa Rosita (IHADFA, 1994). Though no longitudinal data was presented to suggest a trend toward increased numbers of dually diagnosed patients, it was noted that clinical observation data from acute care emergency rooms point toward an increase in the MICA subpopulation. This mirrors the results of epidemiological studies conducted in the United States during this same period (Miller, 1994).

MICA patients have not only created significant treatment challenges for traditional treatment programs, but for the entire mental health and addiction treatment care systems (Minkoff, 1991). Bachrach (1986-87) has referred to MICA patients as "system misfits" who do not measure up to the typical 'patient profile' within either the mental health or addiction systems of care. Traditional mental health programs are often poorly equipped to address dependency and ongoing intensive recovery needs of MICA patients, while addiction programs generally have difficulty treating MICA patients with psychotic symptoms or who require medication and psychotherapy to resolve a variety of various mental health issues.

Historically, treatment modalities for dual diagnosis populations have been developed to deal specifically with symptom reduction and long-term rehabilitation for each particular population. However, these programs have met with limited degrees of success in treating the dually diagnosed (McLellan, 1986; Schucket, 1985). MICA patients have multiple treatment needs and interactive symptoms, requiring a more integrated approach than is generally employed (Breakey, 1987; Miller, 1994). Depression, delusions, and hallucinations, for example, are often related to, caused by, or intensified by substance abuse and addiction (Minkoff, 1987).

Breakey (1987) notes that there are few, if any, efficacy or evaluation studies among the MICA treatment program reports published in the professional literature. He also notes that most of the published reports have been descriptive and anecdotal in nature, generally describing the treatment strategy that was developed for MICA patients, but failing to report their clinical findings. Minkoff (1987) contends that MICA patients can receive effective treatment that will directly address their addiction and mental illness, but only in programs that are designed to specifically address both constellations of symptoms.

Treatment Program Models

A variety of hybrid program models have been proposed and developed to meet the multiple clinical needs of MICA patients (Evans and Sullivan, 1990; Minkoff 1989; Osher and Kofoed, 1989). These models generally fall into one of two categories.

1. Disease-specific models with modifications - These traditional substance abuse or mental health programs attempt to treat the multiple symptoms of MICA patients by incorporating additional mental health treatment or addiction counseling into their spectrum of services. Despite these enhanced techniques, the primary clinical focus in such programs generally remains on the principal diagnosis of mental illness or substance abuse. Disease-specific programs often link their patients to other treatment programs to address those symptoms that cannot be resolved in the original program due to staffing, modeling, or other programmatic constraints. In mental health programs that link patients with substance abuse programs, a traditional approach to treating either mental illness or substance abuse can be utilized by the respective programs. Because of this, the linkage of multiple treatment programs is more of a treatment strategy than an independent model. Thus, programs using linkage models can be considered hybrids of existing disease-specific program models.

2. Integrated programs: These programs incorporate the clinical resources and systems necessary to not only meet the multiple clinical needs of MICA patients within a single program, but do so in an individualized manner, customizing treatment planning and services to meet the needs of individual MICA patients. Integrated programs provide a mix of services, such as group and individual rehabilitation therapies, psychoeducation, case management services for long- term follow-up, and other expressive therapies to treat mental illness. Most integrated programs for MICA patients also provide substance abuse treatment, pharmacotherapy, and group therapy that specifically address the independent living needs of patients recovering from both mental illness and addiction. With such a wide array of services, integrated programs can tailor services to meet the specific needs of individual patients. Patients who appear to abuse substances in an effort to self medicate can be provided with treatment plans that emphasize recovery from mental illness. Other patients who present with severe addiction symptomatology and secondary symptoms of mental illness can have their treatment focused more on the recovery from addiction. In this way the special needs of each patient can be effectively addressed.

Most disease-specific treatment models for MICA patients emphasize sequential program modeling in which patients attend collateral treatment after they have met their current treatment goals in substance abuse or mental health (Minkoff, 1991). In cases where patients are linked to other disease-specific programs to resolve those issues that cannot be treated in the original programs, the hybrid linkage strategy is often used.

Such hybrid strategies emphasize a parallel treatment process that requires patients to attend collateral treatment in another program for the mental health or substance abuse treatment they cannot receive in their current program. Such parallel service systems attempt to deal with both addiction and mental illness simultaneously, while independent disease-specific, sequential models first treat the mental illness or substance abuse, then send the patient to another program to work on the remaining symptoms.

However, in both disease-specific and hybrid linkage programs, generally only one treatment philosophy is stressed for MICA patients and it is typically substance abuse treatment (Minkoff, 1991). In such programs, mental illness and underlying pathology are often treated as secondary to the substance abuse and the primary treatment phases and components generally mirror that of traditional substance abuse treatment programs (Osher & Kofoed, 1989). This may be due to the fact that withdrawal from substances is often the most emergent problem for MICA patients who seek assistance. Consequently, the addiction symptomatology becomes the primary focus of diagnosis and initial treatment, and guides the treatment planning process. This results in a concentration on treatment for addiction and minimizes the focus on mental health and recovery from mental illness.

Effective treatment for either the addiction or mental illness symptomatology first requires clinician understanding of the interaction between all presenting symptoms. Thus, the first step in meeting the treatment needs for MICA patients is a complete assessment of all presenting symptoms. However, in many traditional disease-specific program models, initial assessment and instrumentation are often selected to measure only the aspects of the patients' symptom constellation that can be treated at that facility. As a consequence, other deficits, such as medical illness, history of trauma, skill deficits or inadequate/dysfunctional support systems, perceptual disturbances, and deficits in cognition are neglected (Koegel & Burnam, 1988; Wright & Weber, 1987). On the other hand, integrated programs are generally designed to take the full range of patient symptoms and distress into account, and customize treatment to meet these patient needs.

The development of these models has not been based so much on the clinical efficacy of the models, but more on availability of funding and political interest in treating specific patient populations (Humphreys & Rappaport, 1993). This fragmentation of program models has been perpetuated through the development of artificial and arbitrary administrative divisions at the federal, state, and local levels, without regard to clinical measures of success for the various program models. Consequently, it is possible that many public sector and grant-funded programs continue to be financed through a variety of funding streams with little or no demonstrable clinical success. This siphons critical funds from those programs that use more clinically viable models. Demonstrating clinical viability requires comprehensive impact evaluations of programs with each funding stream and patient population. However, in the past 10 years no comprehensive evaluation studies of mental health programs have been published in the professional literature. Though other social sectors (i.e. health and education) have utilized evaluation study methodologies, mental health and substance abuse subsectors researchers evaluate only selected aspects of programs, such as cost-effectiveness, therapeutic alliance, and various psychotherapy and pharmacotherapy intervention strategies (Newman & Tejeda, 1996). This approach concentrates on the molecular effects of therapeutic interventions without regard to the overall or molar impact that programs have on MICA patients. Molecular effects are specific changes in patient recovery that are directly related to the specific interventions. For example, noting frequency of substance abuse during a specific period of time after a specific cognitive behavioral intervention is a molecular effect. Molar effects are global comprehensive changes in patient recovery that are due to the total impact of the treatment environment, patient contact by clinical and administrative staff, and effects of all treatment interventions. Molar effects in the previous example would be to note all biopsychosocial changes in addition to the frequency of substance abuse. These molar effects describe patient improvements that are not only to the cognitive-behavioral intervention, but the impact of the entire program on the patient.

Many patients may, in fact, benefit from a variety of program models and contexts. However, without careful examination of the full range of treatment effects that entire programs produce and without comparisons made between the clinical effects of programs that treat similar patient populations, no valuation of clinical worth or viability can be determined on a program-by-program basis.

This study of treatment outcomes for mental health programs that use the two major program models (integrated and disease-specific) investigates the relative effectiveness of each model to treat the dually diagnosed MICA patient. The results also enable program planners to modify their programs to more effectively treat MICA patients.

Rationale and Objectives

Dually diagnosed patients in general and specifically MICA patients have complex treatment needs and interactive symptomatology that require a more integrated approach than is generally employed in programs that use disease-specific models (Breakey, 1987). Programs that combine the full range of mental health treatment methodologies and technologies with those of substance abuse programs would provide patients with more treatment resources at their disposal. It therefore seems more likely that such integrated treatment models would be clinically more effective in treating the dually diagnosed MICA patient than disease-specific models. However, given the severe therapeutic challenges that substance abuse and dependency present, a more restrictive, traditional substance abuse model may actually provide increased efficacy for MICA patients as well.

This evaluation of the various treatment outcomes of an integrated program and a disease-specific program in Honduras, treating a similar patient population, demonstrates the relative clinical effectiveness and cost-efficiency of each program model to treat MICA patients. It was hypothesized that the results would show significant clinical and fiscal advantages of using an integrated approach over a disease-specific approach for MICA patients.

Specifically, this study compared and evaluated the clinical impact of a disease-specific program with that of an integrated program for MICA patients. The clinical outcome indicators and other study variables were selected to demonstrate each model's effectiveness in meeting the programs' stated therapeutic goals for MICA patients. They were selected to rule out outcomes that may be due to differences between the programs, such as population / sub-population differences, number of services delivered, and level of patient participation (See Chapter 3). The selected indicators compare relative degrees of outcome effectiveness between the two programs and are not process oriented. As such, they directly relate not only to the goals of these two programs, but to the treatment goals of all MICA programs. Since the location, staffing pattern, outplacement resources, and goals of the programs are similar, and the study population approximately the same in terms of diagnostic and demographic composition (MICA male patients from urban centers in Honduras), the results illustrate the relative impact of disease-specific and integrated program models for successful MICA treatment. The differential rates of the effectiveness of each program produced information as to the relative utility of each model in an applied context.

In general, program models that integrate many services within a single program cost more than those that focus on only one symptom constellation. However, this may not hold true when evaluating the relative costs between the programs in terms of the costs that are only associated with successful treatment outcomes. Therefore, in addition to clinical measures, a comparison of relative costs per successful treatment outcome (cost-efficiency analysis) was also performed. These costs differ from the budgeted program costs and added additional comparative data to the investigation. It was hypothesized that, though integrated programs have higher initial costs, the cost-efficiency rates would be lower for programs with integrated models than those with disease-specific models. This is based on the first hypothesis that predicted a greater number of successful treatment outcomes for integrated programs than disease-specific models.

To effectively perform a comprehensive impact evaluation of mental health programs, cost-effectiveness and cost-efficiency data must also be measured and evaluated (Hammer, 1996). Cost-efficiency analysis only requires a comparison of direct program costs with immediate short term patient benefits. Because the goal of cost-efficiency is to measure the cost of attaining program goals, it was utilized within the context of this study as a comparative measure of program effectiveness without the more global assessment that a full cost-benefit analysis would entail.

2. HISTORICAL CONTEXT AND LITERATURE REVIEW
The use of an integrated model appears to have distinct advantages over disease-specific models of care for MICA patients. A detailed review of the historical development, theoretical/philosophical assumptions, model components, and efficacy highlights these strengths as a model for effective treatment.

Over time, established research and treatment programs for population-specific diagnostic categories have produced barriers to patient care. This is due to overspecialization of treatment programming and tends to limit access or reduce services for the dually diagnosed. Clinician, program, institutional, and funding biases have contributed to the development of programs that are focused on treatment within disease-specific categories, such as mental illness or substance abuse. These biases are generally in the direction of treatment of primarily single diagnosis symptomatology. They have resulted in the development of treatment programs and associated techniques that concentrate on one aspect of patient pathology while excluding others, such as psychotic spectrum and mood disorder symptomatology. Self-help programs such as Alcoholics Anonymous (AA) and Narcotics Anonymous (NA) Programs are examples of programs that use such treatment strategies (Cummings, 1993).

Many AA and NA programs discourage the use of all substances, including psychotropic medication used to treat mental illness (Ridgely, Goldman & Willenbring, 1990). This ignores the biologically based causes of mental illness and limits biologically focused treatment for psychosis, depression, and many other symptom constellations. In many of these programs, all aspects of care that appear to be in conflict with the goal of moral and social improvement that underlies the 12-step method used in AA and NA are discarded as potentially harmful to the substance abuse treatment. Thus, severely disabled MICA patients may only receive part of the treatment mix they need for full recovery from both mental illness and addiction.

In general, this bias within systems of care, or paradigmatic bias, is due to the evolution of separate administrative divisions and funding pools that foster effective political and administrative organization at the expense of creative and innovative clinical care. Artificial and arbitrary divisions at the federal, state and local government levels continue to promote this process and consequently prevent programs from developing joint projects or crossing service boundaries to more effectively treat and manage patients with multiple diagnoses (Drake, Osher, Wallach, 1991; Ridgely et al., 1990). Often otherwise eligible patients who seek treatment at disease-specific program facilities and who happen to have co-existing disorders are refused admission to programs, or are prematurely discharged from such treatment programs solely on the basis of their category of pathology (Galanter et al, 1988). This situation has caused many population specific treatment programs to be overutilized and restrict entry due to space limitations, while other, less restrictive community mental health programs remain underutilized (Cummings, 1993).

Prior to deinstitutionalization, almost all types of dually diagnosed patients received care from an integrated state hospital system. However, with the reduction of long-term, state and federal institutional beds came a corresponding rise (albeit slow) in various streams of funding for community mental health centers and more recently for substance abuse programs. In addition, separate funding streams were also developed for the long-term community-based treatment of mental retardation and child/adolescent disorders. Each of these funding streams produced a corresponding division in both clinical research and service delivery.

The philosophies of treatment tended to vary as new funding streams and divisions of services developed. Mental health center models tended to adopt a medical or biochemical deficit philosophy, while substance abuse programs developed treatment programs that were based on an internal character deficit philosophy (Valliant, 1983). Other funding streams for mentally retarded developmentally disabled and adolescent disorders produced programs based on combined medical and social environmental/ecological deficit philosophies (Humphreys & Rappaport, 1993).

Brower, Blow and Bereford (1989) identified five distinct treatment philosophies that have emerged in disease-specific treatment program models. He writes that many programs typically employ moral deficit, learning/behavioral, disease, self-medication, or social deficit philosophies of treatment. Though each of these treatment philosophies have advantages when applied to a target population, each are compromised by their rigid adherence to that particular philosophy and are therefore limited in their effectiveness.

The moral deficit philosophy is historically the oldest model for both substance abuse and mental health treatment. In this model, mental illness results from a moral weakness and lack of willpower. The goal of rehabilitation is to increase the patients' willpower to resist their 'evil' cravings for substances or resist the irrational urges of mental illness and become ‘good’. Though the moral deficit philosophy has the advantages of holding patients accountable and responsible for the consequences of their actions, the major disadvantage of this treatment philosophy is that it places the treating clinician in an antagonistic relationship with the patient. In such programs, clinicians must adopt a judgmental stance that is blaming and punitive. The moral deficit philosophy is often embraced by patients themselves who feel guilty for their past actions and who readily assess themselves as bad and weak willed. Though this treatment philosophy may help some chemical abusers, it could be disastrous for the MICA patient, who has no control over the biochemical imbalances that caused the mental illness or the substance abuser who may be hypersensitive to blame.

Disease-specific programs utilizing a learning/behavioral philosophy assume that substance abuse and other deficit behaviors are caused by the learning of maladaptive habits (Marlett, 1985). In this case, the patient is viewed as someone who has learned 'bad' habits through no particular fault of their own. The goal of treatment is to teach new behaviors and cognitions that are more adaptive. The main advantages of utilizing this model are that clinicians are neither punitive or judgmental in their service delivery, and the learning of new, more adaptive habits is the primary focus of treatment. Unfortunately, such models shift the focus of control to the patient, thus fueling the patient's denial of either mental illness or substance abuse. Since they may deny that they are out of control, they may deny that any problem exists. For MICA patients, who may resolve their chemical abuse or mental illness problem, this could have serious consequences because the remaining clinical deficits will not be resolved.

The disease/deficit philosophy is perhaps the dominant model used among disease-specific program providers today (Brower et al., 1989). In programs that adopt this philosophy, substance abusers are seen as individuals who are ill and unhealthy, not because of an underlying mental illness, but due to the disease of chemical dependency itself. Because there is no known cure for this 'disease', the patient is considered always and forever ill. The treatment in this case is complete abstinence. Chemical abusers are expected to "change from using to not using, from ill to healthy, and from unrecovered to recovering" (Brower et al., 1983, p.150). Although guilt is relieved because patients are not held responsible for developing chemical dependency, and treatment is neither punitive nor judgmental, this treatment philosophy may not account for patients who return to normal asymptomatic drinking. When applied to mental health, this model cannot account for spontaneous remission either. Since these 'diseases' are considered incurable and only manageable from a disease-deficit perspective, no spontaneous recoveries or remissions should be possible.

Programs that adopt a self medication philosophy assume that chemical dependency occurs either as a symptom of mental illness or as a coping mechanism for underlying psychopathology. The patient is viewed as someone who uses chemicals to alleviate the symptoms of a mental disorder such as depression. The goals of treatment for these programs emphasize improvement in mental functioning. Chemical abusers and the mentally ill are expected to change from mentally ill to psychologically healthy, using medication that more appropriately addresses their individual symptom constellations. The major advantage of these programs is that psychiatric problems are diagnosed and treated along with the substance abuse symptoms. However, this is also the model's main disadvantage as well. Assuming mental illness as the etiology for chemical abuse negates the possibility that chemical abuse causes or exacerbates the psychopathology. Because the focus of treatment is on the resolution of underlying mental illness, the chemical abuse problems that may be the true clinical etiology may not be resolved for MICA patients.

Social deficit philosophies of treatment tend to view chemical dependency and mental illness as the result of environmental, cultural, social, peer, or family influences (Beigel & Ghertner, 1977). Substance abusers and the mentally ill are viewed as products of external forces such as poverty, drug availability, peer pressure, and family dysfunction (Brower et al, 1989). The goal of treatment in these programs is to improve social functioning by altering their environment or their coping responses to perceived stressors. This may involve group therapy, attending self-help groups, residential treatment, and interpersonal therapy; all with the goal of improving social skills. An advantage in assuming a social deficit philosophy is that the role of the social environment is brought into clinical focus and treatment is geared toward reintegrating patients into their social milieu. The main disadvantage in adopting this treatment philosophy for the treatment of MICA patients lies in its exclusive treatment of social factors for problems that are often multifactored.

By accepting any of these underlying assumptions alone, and relying solely on one philosophic stance, researchers and practitioners perpetuate the status quo by remaining uncritical about the problems inherent in their models. This process has, as a consequence, produced service barriers that have discouraged or excluded large numbers of dually diagnosed patients from seeking, being admitted to, or successfully completing appropriate professional treatment programs (Bachrach, 1986; Humphreys & Rappaport, 1993.

Program Components of Integrated Program Models

An integrated system of care for MICA patients incorporates more comprehensive treatment philosophies and strategies than traditional disease-specific models. Integrated approaches allow for the use of the most appropriate level and type of treatment technologies available to rehabilitate patients at their individual level of need. Thus, integrative treatment plans can be customized to meet both the mental health and addiction needs of the patient.

Traditional disease-specific and linkage programs tend to be more generic in nature, requiring patients to conform to the expectations of the program, as opposed to the program conforming to the needs of the patient. Many substance abuse models emphasize group and individual counseling in a highly structured, substance-free, restrictive environment. These programs generally promote abstinence from all substances, including psychotropic medication with addictive potential, such as benzodiazepines. Long-term aftercare treatment focuses solely on sobriety issues. On the other hand, disease-specific models in mental health concentrate on functional adaptation and rehabilitation in a less restrictive milieu, but minimize the problems of addiction. It is assumed in each of these program models that patients will be motivated to participate in treatment to alleviate their distress. Those who do not conform to the mandates of these programs are considered treatment resistant or treatment refractory and are encouraged to seek help elsewhere or are discharged from the program.

Developing a comprehensive and more effective system of care requires the use of a wide array of services delivered under a conceptual framework that merges both addiction recovery and psychiatric rehabilitation. Minkoff (1989) has identified an integrated conceptual framework for treatment of MICA patients and the key concepts for developing such programs. The critical elements for developing such a system are as follows:

"1. Chronic psychotic disorders and substance dependency are both viewed as examples of chronic mental illness, with many common characteristics (biological etiology, hereditability, chronicity, incurability, treatability, potential for relapse and deterioration, denial, and guilt), despite distinctive differences in symptomatology.

2. Each illness can fit into a disease and recovery model for assessment and treatment, where the goal of treatment is to stabilize acute symptoms and then engage the person who has the disease to participate in a long-term program of maintenance, rehabilitation, and recovery.

3. Regardless of the order of onset, each illness in considered primary. Further, although each illness can exacerbate the symptoms of and interfere with the treatment of the other, the severity and level of disability associated with each illness is regarded as essentially independent of the severity and level of disability associated with the other.

4. Both illnesses can be regarded as having parallel phases of treatment and recovery. Those phases include acute stabilization, engagement in treatment, prolonged stabilization/maintenance and rehabilitation/recovery. Osher and Kofoed (1989) have further subdivided the engagement phase into engagement, persuasion, and active treatment; prolonged stabilization is the intended outcome of active treatment.

5. Although, in dual diagnosis patients, progress in recovery for each diagnosis is affected by progress in recovery for the other, the recovery processes commonly proceed independently. In particular, progress in recovery may depend on patient motivation, and patient motivation for treatment of each illness may vary. Thus, patients may be engaged in active treatment to maintain stabilization of psychosis, while still refusing treatment for stabilization of substance abuse." (Minkoff, 1991, p.18)

Such a conceptual framework has a number of implications for program model design. Each system of care within the integrated model must include program elements that meet the needs of the patient in every phase of recovery and rehabilitation. In addition, the program must address levels of severity and disability within each phase of rehabilitation. For example, programs must provide for acute detoxification services for both psychotic andor non-psychotic patients; deliver services for the stabilization of psychosis, whether the patient is in active substance withdrawal or not; and provide individual and group therapy services that are designed for various degrees of dysfunction in both substance abuse or mental illness. Operating under this combined conceptual framework of type and severity of dysfunction requires integrated models to be staffed with sufficient numbers and types of clinicians who can provide the customized, comprehensive treatment to relieve all types of symptoms at all levels of severity.

In addition to the comprehensive provision of the mix of services, an integrated program should provide for acute stabilization, continuity of care, and ongoing stabilization and rehabilitation for both addiction and mental illness symptomatology. Relapse occurs often in both mental illness and substance abuse. Programs must possess or link with adequate facilities to stabilize patients during acute episodes and relapses. In addition, maintaining a vast array of services under one program umbrella, provides for continuity of care by short circuiting the "ping-pong treatment" of bouncing back and forth between various programs (Ridgely et al, 1990) . This usually occurs in linkage programs and creates a discontinuity of services for the patient and confusion in treatment planning for clinicians. Finally, ongoing stabilization and long-term rehabilitation must be designed into the phases of treatment to enable patients to build on the gains made within the integrated program. This may take the form of case management or ongoing day treatment. These program components reduce the incidence of relapse for both mental illness and addiction and promotes patient re-integration into the community (Harris & Bergman, 1987).

The characteristics and program elements listed above generally describe common characteristics of integrated programs in residential and hospital settings. A review of the literature on integrated MICA programs also identifies five common characteristics for outpatient programs as well.

1. Abstinence is a goal, not a requirement.
2. Patients with substance abuse and substance dependence are treated together.
3. Group models, with either staff of peer leaders, are fundamental.
4. Patients progress from (a) low-level education or "persuasion" groups, in which patients have high denial and low motivation, to (b) "active treatment" groups, in which they are more motivated to consider abstinence and are willing to accept more confrontation, to (c) abstinence and support groups, in which they have mostly committed to abstinence and help each other to learn new skills to attain or maintain sobriety.
5. Involvement of available family members is recommended. (Minkoff, 1991, p.23)

By incorporating this vast array of services under an integrated conceptual framework, MICA patients, who typically fail in traditional treatment due to low levels of motivation or programmatic bias against either substance abuse or mental health issues, can be treated at their individual level and scope of dysfunction. The development of an integrated program model builds on the most effective treatment technologies available in addiction and mental health, while overcoming the differences that separate the systems and treatment programs.

Nuttbrock, Rahav, Rivera, Ng-Mak and Struening (1997) attempted to verify the positive impact of integrated programs in their investigation of clinical outcomes for patients in two residentially based treatment programs. Using the Brief Psychiatric Rating Scale (BPRS)(Overall & Gorham, 1962) and Schneider and Struenings' Specific Levels of Functioning Scale (SLOF) (1983), they compared clinical outcome data for patients in an integrated community residence program and an integrated therapeutic community program. Using a pretest posttest design, MICA patients in both programs were rated by counselors at the beginning and end of their treatment program to determine what specific impact the integrated model had on the patients' functional improvement and reduction in symptoms associated with psychopathology.

All patients in these programs were assessed on the dimensions of psychopathology and level of functioning. The assessment of psychopathology identified patient levels of psychotic ideation, generalized anxiety, agoraphobia, cognitive disorientation, and hostility. Functional level examined patient's personal care, instrumental activities (observed capacity to perform everyday activities of daily living), interpersonal relationships, social acceptance, and work skills. Patients were assessed within 1 month of entering the program and on their 12^th month to determine the effects of treatment on these dimensions.

The results of this investigation demonstrated positive treatment effects on the dimensions of personal care and interpersonal relationships. In addition, there were significant reductions in levels of hostility. These results suggested that all patients can benefit from an integrated program, regardless of their severity of psychopathology and distress. However, this study was not focused on a comparative evaluation of the programs’ clinical models used in the community residence and therapeutic community. Thus, no conclusions can be made as to the relative benefit each model has for various patient subpopulations within the programs. The main contribution of the study for the purposes of this investigation is its methodological contribution. Assessing patient improvement in terms of psychological and functional progress produced outcome data that could be used to compare clinical impact.

Pilot Study:
A Comparison of Integrated and Disease-specific Models

Anderson (1996a) evaluated the treatment outcomes of two transitional living community MICA treatment programs. This study illustrates the differential efficacy for a traditional, disease-specific treatment model and an integrated program approach. Using a traditional evaluation model of choosing program performance indicators and then monitoring patient outcomes across those indicators, he demonstrated higher levels of program effectiveness for the integrated program model in the treatment of low functioning MICA patients.

This investigation evaluated patient outcomes for two treatment programs at Bellevue Hospital Center, New York City, New York, that treat homeless, male, mentally ill chemical abusers. Clinical outcomes in the MICA Transitional Living Community Program (MICA-TLC), a disease-specific substance abuse treatment program, were compared with outcome data from the Transitional Living Community (TLC), an integrated, MICA residential mental health program. Both residential programs have program goals of rehabilitating MICA patients over a 6 month period and placing graduating patients in community-based housing. A comparison of patient outcomes for the same efficacy indicators was performed to determine the relative value of integrated and disease-specific treatment models in treating MICA patients.

Outcome measures on 76 male patients from the TLC and 149 patients from the MICA-TLC were included in this investigation. The patients for both programs under study were selected on the basis of their meeting the diagnostic and homeless admission criteria for the programs. Patients who were homeless for at least 3 months, who were ambulatory and no longer in need of acute care, and who were diagnosed with a major Axis I disorder of schizophrenia, major depression, or bipolar disorder and an additional substance abuse diagnosis were admitted to both treatment programs directly from the same inpatient acute care units of the hospital and New York City Shelter Programs, on a space available basis. No other conditions for admissions were imposed.

Evaluation Criterion

The two programs were evaluated by the following outcome criteria:

Indicator 1: Successful Treatment Outcomes - The relative percentages of patients who met and maintained a functional level of GAF = 80 were included as a measures of the models' success.

Indicator 2: AMA Discharge: the percentage of patients who did not complete treatment and who left the programs against medical advice (AMA) were compared across programs as a measure of the treatment program's inability to meet the needs of the target population.

Indicator 3: Correlation of Service Hours Delivered: Bivariate correlations between actual number of service hours delivered and successful treatment outcomes by program were performed to evaluate whether variable amounts of treatment affected overall program outcomes and success. Total service hours delivered to each patient in a program were correlated with the relative percentage of successful outcomes and AMA discharges in each program.

Indicator 4: Subpopulation Outcomes: Differential success rates between diagnostic subpopulations (i.e., schizophrenic, substance abusers versus mood disordered, substance abusers) were also compared to determine whether the models were best suited to one subpopulation or another. This measure was also applied to the criteria indicators listed above.

Indicator 5: Relative Rates of Recidivism: Percentage rates of patient recidivism for program graduates were also determined as a measure of the program's relative inability to effectively rehabilitate their target MICA population.

Indicator 6: Patient Satisfaction: Patients in both groups completed a standard New York State Office of Mental Health patient satisfaction survey (NYSOMH, 1990) within a month of their graduation from their respective treatment programs. This provided a measure of the patients' qualitative level of satisfaction within each program model.

Indicator 7: Cost-efficiency Rate: Relative cost per service unit was determined for patients who successfully met the goals of the program (See Indicator 1). This measure was included to illustrate differential program costs for those patients who met program goals and was considered a gross measure of program efficiency within the context of this study.

(Anderson, 1996, pp. 9-13)

Pilot Study Results Large effect size differences between the two programs and their respective therapeutic models were found in this comparative impact evaluation. TLC and MICA-TLC patient dispositions for a 30-month period are demonstrated in Figures1-4 and Tables 1-4. Within the context of this investigation, graduates were defined as those patients who had completed the therapeutic program, had reached and maintained a Global Level of Function of at least 80, and had remained in community-based placement for at least three months.

With no significant differences found in patient age, SES, race, severity of substance abuse, length of substance abuse, or hospitalization history, and including only the 76 MICA patients of the TLC unit, the TLC Program more than doubled the rate of the positive therapeutic outcomes of the MICA-TLC Program. This occurred even though the MICA-TLC had delivered 35% more service hours per patient than the TLC program during the same period. The TLC program had delivered an average of 22.3 hours of group and individual treatment to patients weekly, while the MICA-TLC program had delivered an average of 30.1 hours per week. In addition, the cost-efficiency rate of $40.69 per unit of service for the MICA-TLC was found to be double that of the TLC, which was calculated at $19.04 per unit of service. This was primarily due to the TLC's higher number of successful treatment outcomes.

From the therapeutic success and failure rates, the differences in rates of recidivism, patient satisfaction survey results, and level of cost-efficiency, the integrated TLC program model appeared to have distinct advantages over the MICA-TLC program model in the treatment of severe MICA patients with low levels of education and high numbers of past hospitalization. However, from the results of the subpopulation and characteristics correlations, the disease-specific, therapeutic community model of the MICA-TLC program seemed to work well with patients who are more highly educated and have fewer previous hospitalizations.

The high degree of MICA-TLC clinical failures and recidivism and lower levels of patient satisfaction appear to be due to the inability of a traditional substance abuse model to treat low functioning MICA patients. Since the MICA-TLC admitted primarily low functioning patients, the clinical failure rate that was demonstrated in the results is understandable. The Integrated model appeared to be more suited to treating such patients than the MICA-TLC.
 

 

Table 1 Patient Characteristics
	TLC		MICA		All Refs
	N=76		N=149		N=360
	%		%		%
Diagnosis - Primary Axis I
Psychotic Spectrum	76.3		68.5		68.4
Mood Spectrum	23.7		30.1		25.1
other	0.0		1.4		6.5
Diagnosis - Secondary Axis 1
None	1.3		3.4		2.5
Polysubstance Abuse	57.9		53.0		50.0
Crack/Cocaine Dependency	11.9	*	24.8		22.9
Alcohol Dependency	26.6		11.4	*	20.9
Other	2.3		7.4		3.7
Diagnosis - Axis II
Personality Disorders	3.9		3.3		8.0
Medications
Neuroleptic	76.3		68.4		66.0
Antidepressive	15.8		17.5		18.1
Anxioletic	1.3		0.7		2.0
Lithium	6.6		6.0		5.0
Anticonvulsive	0.0		2.0		0.0
None	0.0		5.4		8.9
Ethnicity
AfroAmerican	48.7	*	71.1	*	62.3
Caucasian	34.2	*	15.5		19.3
Hispanic	13.2		13.4		15.0
Other	3.9		0.0		3.4
Marital Status
Single	94.8		85.9		81.6
Married	2.6		2.7		7.3
Sep./Divorced	2.6		10.7		8.0
Widowed	0.0		0.7		3.1
Prison History	72.4		75.2		63.0
Military History	9.2		16.8		12.8

Note: The All Refs (all referrals) Column represents the patient characteristics of all  MICA patients referred to the two programs in  the hospital and shelter systems.  This not only includes those patients who entered the programs, but those who  refused admissions to the programs as well. * = p<.05 two tailed, for significant differences in between group comparisons of percentages.

 

Table 2 
Comparison of Patient Characteristics 
TLC         MICA TLC 
N=76       N=149 

	mean	s.d.	mean	s.d.	t-value	F
Age	40.80	10.259	34.91	7.706	4.84	8.357	a
Education	11.01	1.963	10.87	2.410	0.46	0.550
Previous Hospitalizations	2.54	1.879	3.75	2.224	-4.07	0.454
Suicide Attempts	0.49	1.390	1.17	2.132	-2.52	3.915	b
Length of Stay in Program	137.89	108.289	111.37	79.395	2.09	15.869	a
Months of Homelessness	30.62	34.691	43.00	42.667	-2.19	3.014
Treatment Hours (Daily)	3.26	0.854	5.43	0.940	-16.91	1.802	b
Welfare Benefit Amount	433.79	153.425	290.31	258.504	4.45	59.190	a

 

Table 3 
Patient Outcome Summary 

			AMA	%		Placed	%		Hosp.	%		Total	%
TLC PROGRAM
At Discharge from Program			18	23.68	**	51	67.11	*	7	9.21		76	100
Three Months Post-Discharge			9	11.84	*	34	44.74	**	8	10.53		51	51
(in community-based housing)
TLC TOTAL FINAL OUTCOMES			27	35.53	**	34	44.70	**	15	19.74		76	100
MICA-TLC Program
At Discharge from Program			93	62.42	**	43	28.86	*	13	8.72		149	100
Three Months Post-Discharge			19	12.75	*	14	9.39	**	10	6.71		43	28.8
(in community-based housing)
MICA-TLC FINAL OUTCOMES			112	75.17	**	14	9.39	**	23	15.44		149	100
*p < .05. **p < .01

 

Table 4 
Correlations of Significant Subpopulation 
Characteristics At the Time of Discharge from Program 

Services hours (daily)	Discharge Type Placed		Previous Hosp.     Episodes		Education      Years
TLC	.38	**
MICA TLC	.60	**
No. Prev. Hospitalizations
TLC	.31	*
MICA TLC	-.14	*
Months Homeless (Lifetime)
TLC	.34	**	.49	**
MICA TLC			.13	*	-.24	**
Length of Stay in Program
TLC	.38	**
MICA TLC	.61	**	-.14	*	.20	*

 
This pilot study examined two treatment programs for MICA patients that use very different treatment models and appears to demonstrate advantages to using integrated models to treat lower functioning MICA patients. The efficacy rates of the two transitional living communities suggest that the use of an integrated approach, which emphasizes the individualized mix of treatment options, produces greater patient satisfaction and yields higher levels of efficacy than traditional, disease-specific programs currently provide. In addition, integrating services within a single program reduced the costs of successful therapeutic treatment and duplication of effort because patients were treated within the same facility. This leads to the proposition that the adoption of a integrated program model might allow for the customization of program services to meet the needs of most MICA patients and produce greater overall treatment gains for this population.

The main problems with this investigation, however, lie not in the indicators chosen or in the results obtained, but in the methodological approach used to determine net program worth and value. From the top down approach used to evaluate the treatment outcomes of these two programs, we cannot assume that the obtained results were not in fact due other factors outside the seven chosen indicators. Because all seven indicators were chosen by the investigator in a traditional evaluation research model, the study failed to incorporate the programmatic priorities of all the other stakeholders in the study, for example, the funding source, hospital administration, and the programs’ clinical staff. The Community Support Services (CSS) Program Director, the investigator, was the only stakeholder viewpoint represented in this study. This has been termed, "A failure to accommodate value pluralism" and represents an evaluator bias that is often present in evaluation studies of this type (Guba & Lincoln, 1989, p.34). Other indicators, chosen by the treatment team, the funding source, or hospital administration, may have changed the outcome of this study; by either diluting the effect sizes, attributing the treatment gains to different factors, or strengthening the claim of net value and worth for the TLC program. As a consequence, the results of these seven indicators cannot fully attribute worth and value to one program over another. An additional problem with this approach lies in the weakness of the measures of clinical improvement (outcomes) that may be related to the program models. No specific tests of patient functional or psychological improvement were performed and only gross measures of program completion were used as measures of positive outcomes. Use of measures to identify specific changes in patient functioning and distress levels would produce more significant results.

Overall mental health service and substance abuse program effectiveness is difficult to assess (Barkman et al., 1996; Newman & Tejeda, 1996). Due to a variety of methodological and paradigmatic differences, program evaluation studies are often biased, rendering their results inconclusive, at best. The following section reviews the methodological issues in program research and points toward ways in which such bias can be reduced from program evaluation studies.

3. EPISTEMOLOGICAL ISSUES : REVIEW OF RESEARCH METHODOLOGIES IN MENTAL HEALTH SERVICES

Mental health programs, along with other health and social welfare services, are coming under increased scrutiny and evaluation. With consistently smaller budget allocations for mental health, many federally funded programs are being radically changed, some severely curtailed, and the relationship between government and private sector providers is being realigned (Inouye, 1983; Klerman, 1974). In addition, state and local government agencies that were expected to reimburse mental health programs for federal shortfalls in funding, have not been universally successful in meeting this challenge (National Institutite for Mental Health [NIMH], 1991a). Thus, new research into the efficacy of mental health services has been called for to meet these growing challenges. However, because traditional research methods have not been readily applicable to the study of applied health programs and many researchers are reluctant to investigate programs in naturally occurring settings, a search for more appropriate methodological approaches and researchers trained in such approaches is now underway (Newman, Howard, Windle, & Hohmann, 1994; Newman & Tejeda, 1996).

These developments are especially significant for mental health services, which have been more regulated and financially supported by government than any of the other services within the health sector in the past 200 years (Rothman, 1971). Throughout this period there have been numerous cycles of mental health reform and innovation, followed by phases of criticism, dissension and retrenchment (Bockman, 1963; Caplan & Caplan, 1969; Deutsch, 1948, Grob, 1973). While the "reforms in mental health have coincided with periods of progressive social change in the larger American Society, phases of reactions, criticism and retrenchments have occurred with the aftermath of war and economic decline" (Klerman, 1974, p. 783). The economic slump of the early ‘90s following the prosperity and massive federal spending of the ‘80s, continues the cyclic pattern of change in mental health services.

After a decade and a half of growth in mental health and substance abuse services, a number of criticisms have been leveled at the mental health sector. Chief among these criticisms is that of program ineffectiveness (NIMH, 1989). To date, there are very few applied or experimental research studies that address program effectiveness (PsycINFO Index, 1995). Without knowing what programs or treatment models effectively work for a variety of patient populations in a variety of settings, legislators and grant funding sources have no way of planning where and how their limited resources should be spent.

Inadequate research strategies and methodologies have been cited as the primary reason why such program efficacy research has not been studied (Newman et al., 1994). Mental health services research cuts across the disciplines of economics, sociology, political science and psychology. One of the prime purposes of mental health service research is to provide empirical evidence and support to guide policy decisions at all levels of government and nongovernmental organizations (NGOs). However, until recently, very few evaluation studies of mental health programs or their associated models have been reported in the professional psychological literature (PsycScan, 1994).

Newman et al. (1994) writes that, until recently, clinical psychologists have tended to ignore and not conduct mental health services research. This has been a direct result of the limitations in methodological training that psychologists receive and a bias against such research in the publication standards of the professional and academic literature. According to Newman et al., studies that assess success rates and program effectiveness in mental health service programs have not been generally deemed worthy of publication. In addition, most clinical psychologists and clinical researchers are trained in experimental and quasi-experimental techniques that make it very difficult to adequately evaluate the global, multifaceted, molar effects found in applied treatment programs (Guba & Lincoln, 1989). The methodological deficits and bias toward the scientific method in professional publications have made it difficult for psychologists to develop and utilize research

methodologies to fully assess the efficacy of mental health service programs and program models in both the public sector and NGOs (Clarke, 1995; Newman & Howard, 1991).

Researchers who investigate programs and clinical factors related to improving the quality and impact of mental health services are often handicapped by the perceived legitimacy of their applied research and the methodological approaches they utilize (Newman et al., 1994). Historically, traditional research methods and journal/grant review criteria have not taken into account the global questions and systemic points of view necessary to fully understand the therapeutic delivery systems under evaluation. Thus, there has not been a coordinated, sustained effort to determine program efficacy for the majority of human service project initiatives (Fetterman et al., 1996; NIMH, 1989).

Many program and project research evaluations attempt to present the factors and/or 'facts' uncovered in their program evaluations. Such evaluations generally utilize traditional experimental, quasi-experimental, causal comparative, correlational, and other approaches. Such evaluations attempt to identify the most salient factors for good program performance on a molecular level. In the process, many of the characteristics and practices that define a successful program may be ignored or dismissed as inappropriate or unimportant to the objectives of the research. For example, in an investigation of a psychotherapy program, investigators may choose to only examine the mean or median number of therapeutic hours received in a voluntary outpatient program. Though this may or may not relate to overall patient satisfaction, motivation, and progress, such process measures do not determine the overall performance or level of program effectiveness. Thus, almost no programmatic conclusions can be made on the basis of this information.

Despite significant clinical and basic research progress made in the diagnosis and treatment of mental disorders over the past two decades, many questions about how to provide high quality, effective treatment services still have not been answered. For people with severe, persistent, disabling mental disorders, this situation means that individual diagnoses may be inaccurate, treatment plans inadequate or ineffective, and essential services unavailable (Lalley et al., 1992; NIMH, 1991a). As a consequence, such individuals are often forced to not only endure a lonely struggle against the despair and distress caused by their mental illness, but must also negotiate a confusing, fragmented maze of human services, created by a wide range of often well-meaning public and private sector service providers.

Instead of concentrating on determining the individual facts and salient factors associated with successful treatment outcomes, human service researchers should be more concerned with one global question that allows for a more holistic examination of program worth: "What works, for whom, under what circumstances?" (NIMH, 1991a, p.vii). The net effect of any treatment program or human service project is determined by the integrated use of multiple, interactive program components, delivered at a site conducive to recovery or rehabilitation to a population that will be receptive to such treatment (Breakey, 1987; Minkoff, 1991). Since successful treatment outcomes depend on the global interaction of all these factors, research methodologies used to investigate such programs must also mirror this global, molar intervention to accurately determine whether successful outcomes have indeed occurred.

Critical Methodological Factors in Assessing Program Efficacy

Over the years there have been a number of review articles that have called attention to the need for an increased research effort into efficacy studies of mental health services (Inouye, 1983; Klerman, 1974; Newman & Tejeda, 1996). However, until recently, these articles only addressed the need to increase the research effort without recognizing the methodological developments that would be necessary to adequately assess program strengths.

Klerman (1974) gives the first comprehensive account of the state of mental health service research. In his descriptive article he not only identifies the major stakeholders that should be included in evaluation research, but outlines the major concerns voiced by each constituency. He notes that while the public at large, the courts, mental health professionals, and government agencies all have an active stake in the results of such research, each has a different focus and agenda for the outcomes of evaluation research studies, and requires varied types of data with which to formulate their concerns as to how, where, and in what manner mental health programs should operate. The identification and recognition of the needs of all major stakeholders in any mental health program is a critical step that is often overlooked in most evaluation studies (Guba and Lincoln, 1989; Newman et al., 1994).

The public at large has an active stake in new mental health services evaluations (Farkas & Anthony, 1979). In some cases, an adversarial climate has developed among mental health program critics in the general public and mental health professionals and administrators. Because many critics feel that mental health has expanded too much into areas that had previously been regarded as social deviance or legal misdemeanor, such as substance abuse and treatment for the homeless, the public at large mirrors professional uncertainty about treatment adequacy, clinical training for paraprofessionals, and about what are or are not effective treatment strategies for various patient populations. This reflects a lessening of public trust and confidence in mental health services that parallels the erosion of funding and governmental support (Farkas & Anthony, 1979). In addition, community groups are now seeking a more active voice in the operation of mental health service programs within their neighborhoods and catchment areas. In general, these groups want to ensure that treatment programs maintain standards that will protect their community, and not place the public or patients at risk (Fetterman et al.,1996; NIMH, 1991).

Federal and state courts have been become increasingly involved in mental health service programs over the past 20 years. Prison-based substance abuse and mental health programs have markedly increased over the past decade and a half (NIMH, 1991). However, most of these programs have evolved due to court mandated levels of care and have mainly documented their measures of criminal recidivism as the sole measure of program efficacy (Robitscher, 1972a). Since the courts have mandated this treatment and view success in treatment as key a factor in rehabilitating both the involuntary hospital patient and the mentally ill legal offender, they have become more interested in mental health services evaluation as well. Thus, the court systems are active stakeholders in any efficacy evaluation of mental health services and seek information as to the type and level of services that will be delivered to disordered, disabled, and incarcerated individuals.

The court system has also been a major contributor to the development of program models and standards of practice for both hospital and community-based mental health service programs. Due to general concern for patients' civil rights and for the possible infringement of their personal liberties in cases of involuntary hospitalizations, a number of landmark court decisions mandated not only effective treatment but treatment at the least restrictive level (Klerman, 1972; NIMH, 1989; Robitisher 1972a). Moreover, there has been increased concern about the depersonalization and institutional dependence fostered by large public hospitals. Within the mental health professions, there is a general awareness that large hospital based programs become professionally and therapeutically bankrupt, and thus ineffective. This sentiment fostered the creation of community mental health centers, which are felt to provide alternatives to the low levels of institutional care previously provided to the poor and disabled in large public hospitals (Westin, 1972).

Such mental health program interest by the court and legal systems has been the most significant reason for reforms in mental health services including the community mental health center programs (Klerman, 1972). By concentrating on the difficulties and dissatisfaction encountered with the large public mental hospitals, particularly county, state, and Veterans Administration hospitals, the courts have mandated improvements in treatment and service programming that have led to significant reforms in mental health services. These reforms have resulted in decreases in program size, emphasis on community-based treatment, and increased intensity of treatment in both community and hospital-based programs. They have also been shown to increase the probability of more rapid discharges and reduced recidivism (Farkas & Anthony, 1991; Lamb, 1972; Ullman, 1964).

"At all levels of government-federal, state and local evaluation efforts are frequently initiated by fiscal and budgetary agencies "(Klerman, 1972, p. 784). Increasingly, political, fiscal and administrative decisions regarding mental health programs and their associated treatment models are being made on the basis of fiscal goals to deliver the most effective programs possible for the least amount of funding. In addition, state and local agencies charged with monitoring and promoting mental health service research and delivery have been increasing their efforts to determine what constitutes effective mental health programming for a variety of patient populations. Initiatives to reform mental health care have precipitated legislation to develop and expand state and local commissions to further investigate program utility and effectiveness (Frank et al., 1994; Scott & Ginsburg, 1994).

At the federal level, mental health program research has become a high priority for agencies responsible for mental health funding and monitoring. In fact, research funding at this level has been increased from $90 million in 1992 to $369 million budgeted for Fiscal Year 1997 (NIMH, 1991a). With the reorganization of ADAMHA (Alcohol Drug Abuse and Mental Health Agency) in 1992, the (NIMH), National Institute on Drug Abuse (NIDA), and the National Institute on Alcohol Abuse and Alcoholism (NIAAA) must devote no less than 15% of their total budget for health care services research, all coordinated by NIMH [ADAMHA Reorganization Act of 1992, 464R (f)(2)]. This increase in funding clearly demonstrates an increased level of commitment by the federal government to improving service research and overall program effectiveness for the mental health program consumer.

Mental health consumers and their families have a very active stake in the outcome of program research but, until recently, have had little influence on the research process. Families of the mentally ill themselves have to shoulder tremendous financial and emotional burdens. Each year, 65% of discharged psychiatric patients (approximately 1.5 million), return home and live with their families (NIMH, 1991a). Due to the high cost of hospitalization, many of these patients return home earlier than they would have in the past, still disabled by psychiatric symptoms (NIMH, 1991a). State and local governments are also beginning to recognize the stake that patients and their families have in the mental health treatment by enacting legislation to give the consumer and his/her family a voice in the therapeutic process (NYSOMH, 1990).

The NIMH (1991a) credits the National Alliance for the Mentally Ill (NAMI) with a successful lobbying effort for the inclusion of the mentally ill and their families into the evaluative research process. NAMI together with Family Alliance for the Mentally Ill (FAMI) were developed as grass-roots organizations to serve as advocacy groups on behalf of mentally ill persons and their families. These organizations bring the problems and issues of the mentally ill and their families to the attention of local, state, and federal governments. In evaluative research studies, NAMI has championed the cause of "research designed to identify ways to help patients readjust to the community within the least restrictive environment possible and to prevent relapse through early intervention" (NIMH, 1991a, p.13). In "Clinical Services Research: Enhancing the Real-World Applications of Clinical Science" (1991b), NIMH has outlined the critical points that NAMI and FAMI advocate for service evaluation research studies.

- Researchers must carefully assess the ramifications of family involvement in the care of a mentally ill member -including the characteristics and conditions of caregiving families; the degree and varieties of family stress; the effectiveness of various coping and adaptation patterns; timing and extent of caregiver burnout; and the impact of various kinds of respite care for family members.

- Studies should be performed that are focused on family issues and produce results to assist families with mentally ill members in functioning more effectively and with less turmoil. Such studies would provide effective education in technique for dealing with mentally ill persons without succumbing to the overwhelming anxiety; motivating the patient to become more self-sufficient; and understanding and communicating appropriate expectations. An essential need is for research on the long-term effectiveness of such family education programs-both in helping the patient and in reducing the family's burden.

- For generations, families with mentally ill relatives have dealt with the problems of violence toward family members, exacerbated now as a result of a substance abuse by the seriously mentally ill. Investigators must focus on the predictable, frightening and violent behavior that patients may exhibit toward family members, with the goal of developing more efficient criteria for predicting such behavior and more objective ways to manage and prevent it. Such studies must place a high priority on meeting the needs of families for early education, prevention, and intervention. In this connection, attention should be given to identifying techniques of family adaptation and have proved effective.

-Dealing with mental illness is expensive. Families become frustrated and angry as their savings dwindle, often with meager results. A guide on how to obtain the most effective services even with limited personal resources would be a welcome aid. Such a guide-based on evaluative research, not an opinion-could help families make informed decisions about more selective use of mental health services.

(NIMH, 1991a, p.12-13)

Together, these points highlight the very active stake that the mentally ill and their families have in evaluative research. Program evaluations that do not take these points into consideration cannot be considered complete.

Mental health professional groups also have an active stake in the success of programs and on the outcomes of evaluation studies of program services (Isaac, 1971; Weiss, 1972). Concern for effective program planning and therapeutic results can be seen in the various calls for evaluation research by a number of professional groups. The American Psychiatric Association, The National Association for Mental Health, and American Psychological Association (APA) have, at various times, all called for increased research efforts in program effectiveness (Klerman, 1972; Lalley, 1992; Newman 1996, 1994, 1991). Though these calls for more research demonstrate an interest and active stake on the part of professional groups, little if any comprehensive evaluation research has been performed to date.

In response to the NIMH (1991a) call for increased service research, the APA published a special section on mental health services research in the Journal of Consulting and Clinical Psychology (Newman et al., 1994). In this special section, the authors not only reviewed the U.S. "National Plan of Research to Improve Services", but presented a series of research articles that exemplified methodological developments in this area of psychological research. These studies were presented to demonstrate innovative methodological approaches in assessing program and service efficacy. However, each of the studies appear to be focused on a different aspect of overall program efficacy, and are consequently limited in their ability to demonstrate overall program effectiveness or meet the research goals set forth in the NIMH national agenda. All of these studies failed to demonstrate program or service efficacy due to the positivist -reductionist approach utilized by the researchers (Anderson, 1996b).

Though the methodological approaches used in each of these studies are generally accepted as innovative scientific investigations suitable for publication in the psychological literature, each study has difficulty accounting for the full range of interaction between not only the program variables but from the point of view of each of the main stakeholders in mental health service programs. Thus, these studies have difficulty providing a comprehensive account of program effectiveness. Though these methodological approaches may be instrumental and valuable as part of an overall evaluation effort, they cannot be used as the sole basis for determining success in mental health service programs. A comparison-contrast of the main methodological approaches, presented in the following section demonstrates the advantages of departing from the reductionism inherent in traditional approaches in favor of a more comprehensive evaluation methodology.

Comparison-Contrast of Research Designs in Services Research

Correlational, causal-comparative, and evaluative research methodologies are the most common research approaches used to evaluate service programs (Borg & Gall, 1989). Though they have much in common, they differ in their utility, comprehensiveness and ability to establish cause-and-effect relationships among study variables with a strong degree of certainty. As a consequence, they also differ in their ability to predict future effects and causal patterns that can be attributed to the study variables. This difference is primarily due to the limitations of the methodologies to attribute the full range of possible causes to effects observed in natural or artificial/experimental settings. Though each method has situational and experimental advantages over the others in program research, each varies in its situational utility as well.

Correlational Designs

While the correlational method is well suited to establishing relationships between the variables, it cannot demonstrate cause-and-effect relationships by itself. The correlational method is restricted to quantifiable data in the data set and therefore limited in its utility. Though readily applicable to quasi-experimental study situations, it is often difficult to apply in natural settings, where identification and measurement of the most important variables often becomes difficult. This problem is illustrated by the results obtain in the Yeaton (1994) study which investigated the relationship of patient attendance in self-help group meetings to successful completion of an alcohol treatment program.

This study examined the relationship between rates of patient attendance and successful completion of programs. The relative rates of attendance in the service milieu of self-help groups are compared to the rates of successful treatment outcomes as a measure of programmatic effectiveness in treating substance abuse. However, collateral treatments for substance abuse and/or other deficits were not discussed. In addition, the actual service components of the self-help program were not discussed. As a consequence, any relationship between the study variables of attendance and outcome becomes inconclusive. The actual effects that were noted may be due to variables outside the scope of this investigation that were related to the study outcome variables. Thus, from the relationship found by this single correlational indicator, programmatic effectiveness cannot be inferred.

Correlational method research studies are best suited to discovering relationships solely among study variables. As illustrated by the Yeaton (1994)

investigation, it is very difficult to apply this approach to mental health service studies. Identifying and then measuring all the variables that define a program’s level of effectiveness becomes almost impossible using only this approach. When compounded by the various interests and focus of each stakeholder associated with a treatment program, the correlational method becomes almost useless in defining what works, for whom, under what circumstances.

Causal-Comparative Designs

The causal-comparative method is des