The International Journal of Psychosocial Rehabilitation

A Clinical Impact Evaluation of Integrated and Disease Specific Substance Abuse Program Models in Honduras

Rosa Aguilera, Lic.
Projecto Victoria

Arthur Anderson, Ph.D.
Saybrook Institute & Antioch University

Edwardo Gabrie, M.D.
Santa Rosita

Mohemy Merlo, M.D.
Santa Rosita

Teresa Paredes, Lic.
Projecto Victoria

Rosario Pastrana, M.D.
Santa Rosita
 

Citation:Aguilera, R., Anderson, A.J., Gabrie, E., Merlo, M., ParedesT., & Pastrana, R. (1999) A Clinical Impact Evaluation of Integrated and Disease Specific Substance Abuse Program Models in Honduras.  International Journal of Psychosocial Rehabilitation, 3, 97-167


Abstract
This study investigated and compared the clinical impact and program effectiveness of 2 treatment programs for mentally ill chemical abusers (MICA patients) in Honduras. A review of the central issues in MICA patient treatment, treatment strategies/models, and program development is provided. Patient treatment needs, historical context for divisions of service or systems, treatment philosophies, and program model components are also reviewed. The potential benefits of treating MICA patients in both integrated and disease-specific treatment programs is presented in addition to an in-depth discussion of methodological approaches used to determine mental health program effectiveness. This provides the rationale for the comparative program evaluation methodology used in this study to determine the effectiveness of integrated and disease-specific treatment models to meet the treatment needs of their target MICA populations.

This study utilized an innovative methodology by employing both standard indicators, developed by the principal investigator, and local indicators that were developed by the programs. This method is actually a modified 4th generation responsive evaluation model, in which the standard indicators were negotiated by all the stakeholders of each treatment program. The local program stakeholders focused their indicators to gather information they deemed critical for quality improvement in their programs. Each of these stakeholders had an active stake in the outcome of this investigation and were given an equal opportunity to decide what level of participation and involvement they would have in indicator determination, measurement, and data analysis.

This evaluation investigated the clinical impact of program services in Santa Rosita (a disease-specific program) and Proyecto Victoria (an integrated program). Both programs are located within 40 kilometers of Tegucigalpa, Honduras, Central America. Results of all standard indicators were compared and contrasted to determine the clinical and programmatic benefits of each program model in treating a similar MICA population.

Though diluted by socioeconomic factors, the results generally support the findings of the original pilot study. Higher functioning MICA patients tended to have more positive outcomes in structured disease-specific programs, while lower functioning patients respond better to treatment in integrated program settings. The more sensitive measures of clinical improvement, determined through the use of the LOFA, displayed a 40% reduction in psychotic symptoms and mental illness, and a 36% improvement in interpersonal relations in the disease-specific program. The integrated program measures show a 22% reduction in psychiatric symptoms and 22% improvement in community life and interpersonal relations. The need for greater cultural sensitivity in investigating mental health and substance abuse programs in developing nations is also discussed.



This work was supported by a international development grant from the Social Science Research Council. New York, NY. Grant # 95-0168.  We thank the clinical staff of Santa Rosita and Projecto Victoria for their assistance with this project and input into the process evaluation aspects of the study.  Special thanks to IHADFA for their assistance with the project and publication of the results in the Honduran Press.
Correspondence may be addressed to Dr. A.J. Anderson, Suite 3, 34 Frognal Lane, London NW3 7DT  UK

Short Cuts
   1. Introduction
   2. Historical Context and Literature Review
   3. Epistemological Issues
   4. Hypothesis
   5. Method
       Program Descriptions (Santa Rosita) (Projecto Victoria)
   6. Limitations and Delimitations
   7. Results
   8. Discussion
   9. References



1. INTRODUCTION

The chronic mentally ill patient who also suffers from substance abuse problems (drugs, alcohol or both) poses a unique set of difficulties for treatment programming. Such patients present a variety of individual, social, fiscal, and political challenges to effective program planning, design, implementation, and evaluation. Dually diagnosed patients not only require intensive psychiatric treatment for mental illness, but concomitant treatment for substance abuse symptomatology as well. As a consequence, these patients tend to stretch the ability of traditional community-based treatment programs to deliver adequate services to effectively meet their multiple treatment needs. This is particularly true in developing countries, such as Honduras, where resources for mental health care systems are severely limited. The relative lack of funds for mental health and substance abuse programs in developing countries further complicates treatment for this population and has a significant impact on efficacy of treatment for both single and dual diagnosis patients at the community level (Des Jarlais, Eisenberg, Good, & Kleinman, 1995). Such problems in all mental health delivery and prevention systems have led to the development of a variety of treatment models designed to treat mentally ill chemical abuse (MICA) patients (Bachrach, 1984; Drake, Antosca et al.,1991; Minkoff 1987). This review of such program models, their underlying theoretical and philosophical assumptions, and historical development demonstrates the utility of each model to adequately meet the multiple needs of MICA patients.

In industrialized nations, deinstitutionalization and the corresponding increase in the number of homeless mentally ill has been associated with the emergence of a growing population of patients with concomitant mental illness and chemical abuse (MICA patients) (Drake, Osher & Wallach, 1989). Numerous studies have demonstrated a rate of substance abuse and or dependency among the mentally ill at between 32 and 85 % (Safer, 1987; Schwartz & Goldfinger, 1981). MICA patients are the most frequently cited population of dually diagnosed patients in the professional literature (PsycINFO, 1993). They have been reported to utilize higher rates of acute hospitalization, have histories of more housing instability, homelessness, criminality, and homicidal or suicidal behavior than either the mentally ill or chemical abusers alone (Caron, 1981; Drake et al. 1989; Osher & Kofoed, 1989; Safer, 1987). Poor medication compliance and response to treatment have also been linked to this dual disorder (LaPorte, 1989; McClelland, 1986).

Though remarkable progress has been made in improving general health for developing nations, unfortunately this has been accompanied by a deterioration in mental health for the dually diagnosed and other populations. In many areas outside Europe and North America, reported cases of schizophrenia, depression, dementia, and concomitant substance abuse have risen dramatically. In low-income societies, 24.4 million people will be affected by some form of mental illness by the year 2000. This is an increase of 45% since 1985 (Kleinman & Cohen, 1997). Rapid urbanization, chaotic modernization, and economic restructuring of many societies have fractured social supports and extended family structures, increasing violence, substance abuse and suicide (World Health Organization [WHO], 1995).

MICA patients in these low income societies are particularly affected by the lack of clinical resources and options because their multiple disabilities require more clinical resources than are generally available. When combined with an increase in overall use of both medical and psychiatric care facilities, MICA patients generally are not treated in programs designed to meet their multiple needs (WHO, 1995). In developing nations the clinical needs of such patients must be evaluated in the context of their social structures to effectively treat this difficult and growing population in a culturally sensitive manner (Kleinman & Cohen, 1997).

In Honduras, the Ministry of Health Institute for Alcoholism and Addiction (IHADFA) estimated that alcoholism and addiction to substances is present in at least 45% of all patients seen in the national psychiatric hospital, Santa Rosita (IHADFA, 1994). Though no longitudinal data was presented to suggest a trend toward increased numbers of dually diagnosed patients, it was noted that clinical observation data from acute care emergency rooms point toward an increase in the MICA subpopulation. This mirrors the results of epidemiological studies conducted in the United States during this same period (Miller, 1994).

MICA patients have not only created significant treatment challenges for traditional treatment programs, but for the entire mental health and addiction treatment care systems (Minkoff, 1991). Bachrach (1986-87) has referred to MICA patients as "system misfits" who do not measure up to the typical 'patient profile' within either the mental health or addiction systems of care. Traditional mental health programs are often poorly equipped to address dependency and ongoing intensive recovery needs of MICA patients, while addiction programs generally have difficulty treating MICA patients with psychotic symptoms or who require medication and psychotherapy to resolve a variety of various mental health issues.

Historically, treatment modalities for dual diagnosis populations have been developed to deal specifically with symptom reduction and long-term rehabilitation for each particular population. However, these programs have met with limited degrees of success in treating the dually diagnosed (McLellan, 1986; Schucket, 1985). MICA patients have multiple treatment needs and interactive symptoms, requiring a more integrated approach than is generally employed (Breakey, 1987; Miller, 1994). Depression, delusions, and hallucinations, for example, are often related to, caused by, or intensified by substance abuse and addiction (Minkoff, 1987).

Breakey (1987) notes that there are few, if any, efficacy or evaluation studies among the MICA treatment program reports published in the professional literature. He also notes that most of the published reports have been descriptive and anecdotal in nature, generally describing the treatment strategy that was developed for MICA patients, but failing to report their clinical findings. Minkoff (1987) contends that MICA patients can receive effective treatment that will directly address their addiction and mental illness, but only in programs that are designed to specifically address both constellations of symptoms.

Treatment Program Models

A variety of hybrid program models have been proposed and developed to meet the multiple clinical needs of MICA patients (Evans and Sullivan, 1990; Minkoff 1989; Osher and Kofoed, 1989). These models generally fall into one of two categories.
1. Disease-specific models with modifications - These traditional substance abuse or mental health programs attempt to treat the multiple symptoms of MICA patients by incorporating additional mental health treatment or addiction counseling into their spectrum of services. Despite these enhanced techniques, the primary clinical focus in such programs generally remains on the principal diagnosis of mental illness or substance abuse. Disease-specific programs often link their patients to other treatment programs to address those symptoms that cannot be resolved in the original program due to staffing, modeling, or other programmatic constraints. In mental health programs that link patients with substance abuse programs, a traditional approach to treating either mental illness or substance abuse can be utilized by the respective programs. Because of this, the linkage of multiple treatment programs is more of a treatment strategy than an independent model. Thus, programs using linkage models can be considered hybrids of existing disease-specific program models.

2. Integrated programs: These programs incorporate the clinical resources and systems necessary to not only meet the multiple clinical needs of MICA patients within a single program, but do so in an individualized manner, customizing treatment planning and services to meet the needs of individual MICA patients. Integrated programs provide a mix of services, such as group and individual rehabilitation therapies, psychoeducation, case management services for long- term follow-up, and other expressive therapies to treat mental illness. Most integrated programs for MICA patients also provide substance abuse treatment, pharmacotherapy, and group therapy that specifically address the independent living needs of patients recovering from both mental illness and addiction. With such a wide array of services, integrated programs can tailor services to meet the specific needs of individual patients. Patients who appear to abuse substances in an effort to self medicate can be provided with treatment plans that emphasize recovery from mental illness. Other patients who present with severe addiction symptomatology and secondary symptoms of mental illness can have their treatment focused more on the recovery from addiction. In this way the special needs of each patient can be effectively addressed.

Most disease-specific treatment models for MICA patients emphasize sequential program modeling in which patients attend collateral treatment after they have met their current treatment goals in substance abuse or mental health (Minkoff, 1991). In cases where patients are linked to other disease-specific programs to resolve those issues that cannot be treated in the original programs, the hybrid linkage strategy is often used.

Such hybrid strategies emphasize a parallel treatment process that requires patients to attend collateral treatment in another program for the mental health or substance abuse treatment they cannot receive in their current program. Such parallel service systems attempt to deal with both addiction and mental illness simultaneously, while independent disease-specific, sequential models first treat the mental illness or substance abuse, then send the patient to another program to work on the remaining symptoms.

However, in both disease-specific and hybrid linkage programs, generally only one treatment philosophy is stressed for MICA patients and it is typically substance abuse treatment (Minkoff, 1991). In such programs, mental illness and underlying pathology are often treated as secondary to the substance abuse and the primary treatment phases and components generally mirror that of traditional substance abuse treatment programs (Osher & Kofoed, 1989). This may be due to the fact that withdrawal from substances is often the most emergent problem for MICA patients who seek assistance. Consequently, the addiction symptomatology becomes the primary focus of diagnosis and initial treatment, and guides the treatment planning process. This results in a concentration on treatment for addiction and minimizes the focus on mental health and recovery from mental illness.

Effective treatment for either the addiction or mental illness symptomatology first requires clinician understanding of the interaction between all presenting symptoms. Thus, the first step in meeting the treatment needs for MICA patients is a complete assessment of all presenting symptoms. However, in many traditional disease-specific program models, initial assessment and instrumentation are often selected to measure only the aspects of the patients' symptom constellation that can be treated at that facility. As a consequence, other deficits, such as medical illness, history of trauma, skill deficits or inadequate/dysfunctional support systems, perceptual disturbances, and deficits in cognition are neglected (Koegel & Burnam, 1988; Wright & Weber, 1987). On the other hand, integrated programs are generally designed to take the full range of patient symptoms and distress into account, and customize treatment to meet these patient needs.

The development of these models has not been based so much on the clinical efficacy of the models, but more on availability of funding and political interest in treating specific patient populations (Humphreys & Rappaport, 1993). This fragmentation of program models has been perpetuated through the development of artificial and arbitrary administrative divisions at the federal, state, and local levels, without regard to clinical measures of success for the various program models. Consequently, it is possible that many public sector and grant-funded programs continue to be financed through a variety of funding streams with little or no demonstrable clinical success. This siphons critical funds from those programs that use more clinically viable models. Demonstrating clinical viability requires comprehensive impact evaluations of programs with each funding stream and patient population. However, in the past 10 years no comprehensive evaluation studies of mental health programs have been published in the professional literature. Though other social sectors (i.e. health and education) have utilized evaluation study methodologies, mental health and substance abuse subsectors researchers evaluate only selected aspects of programs, such as cost-effectiveness, therapeutic alliance, and various psychotherapy and pharmacotherapy intervention strategies (Newman & Tejeda, 1996). This approach concentrates on the molecular effects of therapeutic interventions without regard to the overall or molar impact that programs have on MICA patients. Molecular effects are specific changes in patient recovery that are directly related to the specific interventions. For example, noting frequency of substance abuse during a specific period of time after a specific cognitive behavioral intervention is a molecular effect. Molar effects are global comprehensive changes in patient recovery that are due to the total impact of the treatment environment, patient contact by clinical and administrative staff, and effects of all treatment interventions. Molar effects in the previous example would be to note all biopsychosocial changes in addition to the frequency of substance abuse. These molar effects describe patient improvements that are not only to the cognitive-behavioral intervention, but the impact of the entire program on the patient.

Many patients may, in fact, benefit from a variety of program models and contexts. However, without careful examination of the full range of treatment effects that entire programs produce and without comparisons made between the clinical effects of programs that treat similar patient populations, no valuation of clinical worth or viability can be determined on a program-by-program basis.

This study of treatment outcomes for mental health programs that use the two major program models (integrated and disease-specific) investigates the relative effectiveness of each model to treat the dually diagnosed MICA patient. The results also enable program planners to modify their programs to more effectively treat MICA patients.

Rationale and Objectives

Dually diagnosed patients in general and specifically MICA patients have complex treatment needs and interactive symptomatology that require a more integrated approach than is generally employed in programs that use disease-specific models (Breakey, 1987). Programs that combine the full range of mental health treatment methodologies and technologies with those of substance abuse programs would provide patients with more treatment resources at their disposal. It therefore seems more likely that such integrated treatment models would be clinically more effective in treating the dually diagnosed MICA patient than disease-specific models. However, given the severe therapeutic challenges that substance abuse and dependency present, a more restrictive, traditional substance abuse model may actually provide increased efficacy for MICA patients as well.

This evaluation of the various treatment outcomes of an integrated program and a disease-specific program in Honduras, treating a similar patient population, demonstrates the relative clinical effectiveness and cost-efficiency of each program model to treat MICA patients. It was hypothesized that the results would show significant clinical and fiscal advantages of using an integrated approach over a disease-specific approach for MICA patients.

Specifically, this study compared and evaluated the clinical impact of a disease-specific program with that of an integrated program for MICA patients. The clinical outcome indicators and other study variables were selected to demonstrate each model's effectiveness in meeting the programs' stated therapeutic goals for MICA patients. They were selected to rule out outcomes that may be due to differences between the programs, such as population / sub-population differences, number of services delivered, and level of patient participation (See Chapter 3). The selected indicators compare relative degrees of outcome effectiveness between the two programs and are not process oriented. As such, they directly relate not only to the goals of these two programs, but to the treatment goals of all MICA programs. Since the location, staffing pattern, outplacement resources, and goals of the programs are similar, and the study population approximately the same in terms of diagnostic and demographic composition (MICA male patients from urban centers in Honduras), the results illustrate the relative impact of disease-specific and integrated program models for successful MICA treatment. The differential rates of the effectiveness of each program produced information as to the relative utility of each model in an applied context.

In general, program models that integrate many services within a single program cost more than those that focus on only one symptom constellation. However, this may not hold true when evaluating the relative costs between the programs in terms of the costs that are only associated with successful treatment outcomes. Therefore, in addition to clinical measures, a comparison of relative costs per successful treatment outcome (cost-efficiency analysis) was also performed. These costs differ from the budgeted program costs and added additional comparative data to the investigation. It was hypothesized that, though integrated programs have higher initial costs, the cost-efficiency rates would be lower for programs with integrated models than those with disease-specific models. This is based on the first hypothesis that predicted a greater number of successful treatment outcomes for integrated programs than disease-specific models.

To effectively perform a comprehensive impact evaluation of mental health programs, cost-effectiveness and cost-efficiency data must also be measured and evaluated (Hammer, 1996). Cost-efficiency analysis only requires a comparison of direct program costs with immediate short term patient benefits. Because the goal of cost-efficiency is to measure the cost of attaining program goals, it was utilized within the context of this study as a comparative measure of program effectiveness without the more global assessment that a full cost-benefit analysis would entail.

2. HISTORICAL CONTEXT AND LITERATURE REVIEW
The use of an integrated model appears to have distinct advantages over disease-specific models of care for MICA patients. A detailed review of the historical development, theoretical/philosophical assumptions, model components, and efficacy highlights these strengths as a model for effective treatment.

Over time, established research and treatment programs for population-specific diagnostic categories have produced barriers to patient care. This is due to overspecialization of treatment programming and tends to limit access or reduce services for the dually diagnosed. Clinician, program, institutional, and funding biases have contributed to the development of programs that are focused on treatment within disease-specific categories, such as mental illness or substance abuse. These biases are generally in the direction of treatment of primarily single diagnosis symptomatology. They have resulted in the development of treatment programs and associated techniques that concentrate on one aspect of patient pathology while excluding others, such as psychotic spectrum and mood disorder symptomatology. Self-help programs such as Alcoholics Anonymous (AA) and Narcotics Anonymous (NA) Programs are examples of programs that use such treatment strategies (Cummings, 1993).

Many AA and NA programs discourage the use of all substances, including psychotropic medication used to treat mental illness (Ridgely, Goldman & Willenbring, 1990). This ignores the biologically based causes of mental illness and limits biologically focused treatment for psychosis, depression, and many other symptom constellations. In many of these programs, all aspects of care that appear to be in conflict with the goal of moral and social improvement that underlies the 12-step method used in AA and NA are discarded as potentially harmful to the substance abuse treatment. Thus, severely disabled MICA patients may only receive part of the treatment mix they need for full recovery from both mental illness and addiction.

In general, this bias within systems of care, or paradigmatic bias, is due to the evolution of separate administrative divisions and funding pools that foster effective political and administrative organization at the expense of creative and innovative clinical care. Artificial and arbitrary divisions at the federal, state and local government levels continue to promote this process and consequently prevent programs from developing joint projects or crossing service boundaries to more effectively treat and manage patients with multiple diagnoses (Drake, Osher, Wallach, 1991; Ridgely et al., 1990). Often otherwise eligible patients who seek treatment at disease-specific program facilities and who happen to have co-existing disorders are refused admission to programs, or are prematurely discharged from such treatment programs solely on the basis of their category of pathology (Galanter et al, 1988). This situation has caused many population specific treatment programs to be overutilized and restrict entry due to space limitations, while other, less restrictive community mental health programs remain underutilized (Cummings, 1993).

Prior to deinstitutionalization, almost all types of dually diagnosed patients received care from an integrated state hospital system. However, with the reduction of long-term, state and federal institutional beds came a corresponding rise (albeit slow) in various streams of funding for community mental health centers and more recently for substance abuse programs. In addition, separate funding streams were also developed for the long-term community-based treatment of mental retardation and child/adolescent disorders. Each of these funding streams produced a corresponding division in both clinical research and service delivery.

The philosophies of treatment tended to vary as new funding streams and divisions of services developed. Mental health center models tended to adopt a medical or biochemical deficit philosophy, while substance abuse programs developed treatment programs that were based on an internal character deficit philosophy (Valliant, 1983). Other funding streams for mentally retarded developmentally disabled and adolescent disorders produced programs based on combined medical and social environmental/ecological deficit philosophies (Humphreys & Rappaport, 1993).

Brower, Blow and Bereford (1989) identified five distinct treatment philosophies that have emerged in disease-specific treatment program models. He writes that many programs typically employ moral deficit, learning/behavioral, disease, self-medication, or social deficit philosophies of treatment. Though each of these treatment philosophies have advantages when applied to a target population, each are compromised by their rigid adherence to that particular philosophy and are therefore limited in their effectiveness.

The moral deficit philosophy is historically the oldest model for both substance abuse and mental health treatment. In this model, mental illness results from a moral weakness and lack of willpower. The goal of rehabilitation is to increase the patients' willpower to resist their 'evil' cravings for substances or resist the irrational urges of mental illness and become ‘good’. Though the moral deficit philosophy has the advantages of holding patients accountable and responsible for the consequences of their actions, the major disadvantage of this treatment philosophy is that it places the treating clinician in an antagonistic relationship with the patient. In such programs, clinicians must adopt a judgmental stance that is blaming and punitive. The moral deficit philosophy is often embraced by patients themselves who feel guilty for their past actions and who readily assess themselves as bad and weak willed. Though this treatment philosophy may help some chemical abusers, it could be disastrous for the MICA patient, who has no control over the biochemical imbalances that caused the mental illness or the substance abuser who may be hypersensitive to blame.

Disease-specific programs utilizing a learning/behavioral philosophy assume that substance abuse and other deficit behaviors are caused by the learning of maladaptive habits (Marlett, 1985). In this case, the patient is viewed as someone who has learned 'bad' habits through no particular fault of their own. The goal of treatment is to teach new behaviors and cognitions that are more adaptive. The main advantages of utilizing this model are that clinicians are neither punitive or judgmental in their service delivery, and the learning of new, more adaptive habits is the primary focus of treatment. Unfortunately, such models shift the focus of control to the patient, thus fueling the patient's denial of either mental illness or substance abuse. Since they may deny that they are out of control, they may deny that any problem exists. For MICA patients, who may resolve their chemical abuse or mental illness problem, this could have serious consequences because the remaining clinical deficits will not be resolved.

The disease/deficit philosophy is perhaps the dominant model used among disease-specific program providers today (Brower et al., 1989). In programs that adopt this philosophy, substance abusers are seen as individuals who are ill and unhealthy, not because of an underlying mental illness, but due to the disease of chemical dependency itself. Because there is no known cure for this 'disease', the patient is considered always and forever ill. The treatment in this case is complete abstinence. Chemical abusers are expected to "change from using to not using, from ill to healthy, and from unrecovered to recovering" (Brower et al., 1983, p.150). Although guilt is relieved because patients are not held responsible for developing chemical dependency, and treatment is neither punitive nor judgmental, this treatment philosophy may not account for patients who return to normal asymptomatic drinking. When applied to mental health, this model cannot account for spontaneous remission either. Since these 'diseases' are considered incurable and only manageable from a disease-deficit perspective, no spontaneous recoveries or remissions should be possible.

Programs that adopt a self medication philosophy assume that chemical dependency occurs either as a symptom of mental illness or as a coping mechanism for underlying psychopathology. The patient is viewed as someone who uses chemicals to alleviate the symptoms of a mental disorder such as depression. The goals of treatment for these programs emphasize improvement in mental functioning. Chemical abusers and the mentally ill are expected to change from mentally ill to psychologically healthy, using medication that more appropriately addresses their individual symptom constellations. The major advantage of these programs is that psychiatric problems are diagnosed and treated along with the substance abuse symptoms. However, this is also the model's main disadvantage as well. Assuming mental illness as the etiology for chemical abuse negates the possibility that chemical abuse causes or exacerbates the psychopathology. Because the focus of treatment is on the resolution of underlying mental illness, the chemical abuse problems that may be the true clinical etiology may not be resolved for MICA patients.

Social deficit philosophies of treatment tend to view chemical dependency and mental illness as the result of environmental, cultural, social, peer, or family influences (Beigel & Ghertner, 1977). Substance abusers and the mentally ill are viewed as products of external forces such as poverty, drug availability, peer pressure, and family dysfunction (Brower et al, 1989). The goal of treatment in these programs is to improve social functioning by altering their environment or their coping responses to perceived stressors. This may involve group therapy, attending self-help groups, residential treatment, and interpersonal therapy; all with the goal of improving social skills. An advantage in assuming a social deficit philosophy is that the role of the social environment is brought into clinical focus and treatment is geared toward reintegrating patients into their social milieu. The main disadvantage in adopting this treatment philosophy for the treatment of MICA patients lies in its exclusive treatment of social factors for problems that are often multifactored.

By accepting any of these underlying assumptions alone, and relying solely on one philosophic stance, researchers and practitioners perpetuate the status quo by remaining uncritical about the problems inherent in their models. This process has, as a consequence, produced service barriers that have discouraged or excluded large numbers of dually diagnosed patients from seeking, being admitted to, or successfully completing appropriate professional treatment programs (Bachrach, 1986; Humphreys & Rappaport, 1993.

Program Components of Integrated Program Models

An integrated system of care for MICA patients incorporates more comprehensive treatment philosophies and strategies than traditional disease-specific models. Integrated approaches allow for the use of the most appropriate level and type of treatment technologies available to rehabilitate patients at their individual level of need. Thus, integrative treatment plans can be customized to meet both the mental health and addiction needs of the patient.

Traditional disease-specific and linkage programs tend to be more generic in nature, requiring patients to conform to the expectations of the program, as opposed to the program conforming to the needs of the patient. Many substance abuse models emphasize group and individual counseling in a highly structured, substance-free, restrictive environment. These programs generally promote abstinence from all substances, including psychotropic medication with addictive potential, such as benzodiazepines. Long-term aftercare treatment focuses solely on sobriety issues. On the other hand, disease-specific models in mental health concentrate on functional adaptation and rehabilitation in a less restrictive milieu, but minimize the problems of addiction. It is assumed in each of these program models that patients will be motivated to participate in treatment to alleviate their distress. Those who do not conform to the mandates of these programs are considered treatment resistant or treatment refractory and are encouraged to seek help elsewhere or are discharged from the program.

Developing a comprehensive and more effective system of care requires the use of a wide array of services delivered under a conceptual framework that merges both addiction recovery and psychiatric rehabilitation. Minkoff (1989) has identified an integrated conceptual framework for treatment of MICA patients and the key concepts for developing such programs. The critical elements for developing such a system are as follows:

"1. Chronic psychotic disorders and substance dependency are both viewed as examples of chronic mental illness, with many common characteristics (biological etiology, hereditability, chronicity, incurability, treatability, potential for relapse and deterioration, denial, and guilt), despite distinctive differences in symptomatology.

2. Each illness can fit into a disease and recovery model for assessment and treatment, where the goal of treatment is to stabilize acute symptoms and then engage the person who has the disease to participate in a long-term program of maintenance, rehabilitation, and recovery.

3. Regardless of the order of onset, each illness in considered primary. Further, although each illness can exacerbate the symptoms of and interfere with the treatment of the other, the severity and level of disability associated with each illness is regarded as essentially independent of the severity and level of disability associated with the other.

4. Both illnesses can be regarded as having parallel phases of treatment and recovery. Those phases include acute stabilization, engagement in treatment, prolonged stabilization/maintenance and rehabilitation/recovery. Osher and Kofoed (1989) have further subdivided the engagement phase into engagement, persuasion, and active treatment; prolonged stabilization is the intended outcome of active treatment.

5. Although, in dual diagnosis patients, progress in recovery for each diagnosis is affected by progress in recovery for the other, the recovery processes commonly proceed independently. In particular, progress in recovery may depend on patient motivation, and patient motivation for treatment of each illness may vary. Thus, patients may be engaged in active treatment to maintain stabilization of psychosis, while still refusing treatment for stabilization of substance abuse." (Minkoff, 1991, p.18)

Such a conceptual framework has a number of implications for program model design. Each system of care within the integrated model must include program elements that meet the needs of the patient in every phase of recovery and rehabilitation. In addition, the program must address levels of severity and disability within each phase of rehabilitation. For example, programs must provide for acute detoxification services for both psychotic andor non-psychotic patients; deliver services for the stabilization of psychosis, whether the patient is in active substance withdrawal or not; and provide individual and group therapy services that are designed for various degrees of dysfunction in both substance abuse or mental illness. Operating under this combined conceptual framework of type and severity of dysfunction requires integrated models to be staffed with sufficient numbers and types of clinicians who can provide the customized, comprehensive treatment to relieve all types of symptoms at all levels of severity.

In addition to the comprehensive provision of the mix of services, an integrated program should provide for acute stabilization, continuity of care, and ongoing stabilization and rehabilitation for both addiction and mental illness symptomatology. Relapse occurs often in both mental illness and substance abuse. Programs must possess or link with adequate facilities to stabilize patients during acute episodes and relapses. In addition, maintaining a vast array of services under one program umbrella, provides for continuity of care by short circuiting the "ping-pong treatment" of bouncing back and forth between various programs (Ridgely et al, 1990) . This usually occurs in linkage programs and creates a discontinuity of services for the patient and confusion in treatment planning for clinicians. Finally, ongoing stabilization and long-term rehabilitation must be designed into the phases of treatment to enable patients to build on the gains made within the integrated program. This may take the form of case management or ongoing day treatment. These program components reduce the incidence of relapse for both mental illness and addiction and promotes patient re-integration into the community (Harris & Bergman, 1987).

The characteristics and program elements listed above generally describe common characteristics of integrated programs in residential and hospital settings. A review of the literature on integrated MICA programs also identifies five common characteristics for outpatient programs as well.

1. Abstinence is a goal, not a requirement.
2. Patients with substance abuse and substance dependence are treated together.
3. Group models, with either staff of peer leaders, are fundamental.
4. Patients progress from (a) low-level education or "persuasion" groups, in which patients have high denial and low motivation, to (b) "active treatment" groups, in which they are more motivated to consider abstinence and are willing to accept more confrontation, to (c) abstinence and support groups, in which they have mostly committed to abstinence and help each other to learn new skills to attain or maintain sobriety.
5. Involvement of available family members is recommended. (Minkoff, 1991, p.23)
By incorporating this vast array of services under an integrated conceptual framework, MICA patients, who typically fail in traditional treatment due to low levels of motivation or programmatic bias against either substance abuse or mental health issues, can be treated at their individual level and scope of dysfunction. The development of an integrated program model builds on the most effective treatment technologies available in addiction and mental health, while overcoming the differences that separate the systems and treatment programs.

Nuttbrock, Rahav, Rivera, Ng-Mak and Struening (1997) attempted to verify the positive impact of integrated programs in their investigation of clinical outcomes for patients in two residentially based treatment programs. Using the Brief Psychiatric Rating Scale (BPRS)(Overall & Gorham, 1962) and Schneider and Struenings' Specific Levels of Functioning Scale (SLOF) (1983), they compared clinical outcome data for patients in an integrated community residence program and an integrated therapeutic community program. Using a pretest posttest design, MICA patients in both programs were rated by counselors at the beginning and end of their treatment program to determine what specific impact the integrated model had on the patients' functional improvement and reduction in symptoms associated with psychopathology.

All patients in these programs were assessed on the dimensions of psychopathology and level of functioning. The assessment of psychopathology identified patient levels of psychotic ideation, generalized anxiety, agoraphobia, cognitive disorientation, and hostility. Functional level examined patient's personal care, instrumental activities (observed capacity to perform everyday activities of daily living), interpersonal relationships, social acceptance, and work skills. Patients were assessed within 1 month of entering the program and on their 12th month to determine the effects of treatment on these dimensions.

The results of this investigation demonstrated positive treatment effects on the dimensions of personal care and interpersonal relationships. In addition, there were significant reductions in levels of hostility. These results suggested that all patients can benefit from an integrated program, regardless of their severity of psychopathology and distress. However, this study was not focused on a comparative evaluation of the programs’ clinical models used in the community residence and therapeutic community. Thus, no conclusions can be made as to the relative benefit each model has for various patient subpopulations within the programs. The main contribution of the study for the purposes of this investigation is its methodological contribution. Assessing patient improvement in terms of psychological and functional progress produced outcome data that could be used to compare clinical impact.

Pilot Study:
A Comparison of Integrated and Disease-specific Models

Anderson (1996a) evaluated the treatment outcomes of two transitional living community MICA treatment programs. This study illustrates the differential efficacy for a traditional, disease-specific treatment model and an integrated program approach. Using a traditional evaluation model of choosing program performance indicators and then monitoring patient outcomes across those indicators, he demonstrated higher levels of program effectiveness for the integrated program model in the treatment of low functioning MICA patients.

This investigation evaluated patient outcomes for two treatment programs at Bellevue Hospital Center, New York City, New York, that treat homeless, male, mentally ill chemical abusers. Clinical outcomes in the MICA Transitional Living Community Program (MICA-TLC), a disease-specific substance abuse treatment program, were compared with outcome data from the Transitional Living Community (TLC), an integrated, MICA residential mental health program. Both residential programs have program goals of rehabilitating MICA patients over a 6 month period and placing graduating patients in community-based housing. A comparison of patient outcomes for the same efficacy indicators was performed to determine the relative value of integrated and disease-specific treatment models in treating MICA patients.

Outcome measures on 76 male patients from the TLC and 149 patients from the MICA-TLC were included in this investigation. The patients for both programs under study were selected on the basis of their meeting the diagnostic and homeless admission criteria for the programs. Patients who were homeless for at least 3 months, who were ambulatory and no longer in need of acute care, and who were diagnosed with a major Axis I disorder of schizophrenia, major depression, or bipolar disorder and an additional substance abuse diagnosis were admitted to both treatment programs directly from the same inpatient acute care units of the hospital and New York City Shelter Programs, on a space available basis. No other conditions for admissions were imposed.

Evaluation Criterion

The two programs were evaluated by the following outcome criteria:

Indicator 1: Successful Treatment Outcomes - The relative percentages of patients who met and maintained a functional level of GAF = 80 were included as a measures of the models' success.

Indicator 2: AMA Discharge: the percentage of patients who did not complete treatment and who left the programs against medical advice (AMA) were compared across programs as a measure of the treatment program's inability to meet the needs of the target population.

Indicator 3: Correlation of Service Hours Delivered: Bivariate correlations between actual number of service hours delivered and successful treatment outcomes by program were performed to evaluate whether variable amounts of treatment affected overall program outcomes and success. Total service hours delivered to each patient in a program were correlated with the relative percentage of successful outcomes and AMA discharges in each program.

Indicator 4: Subpopulation Outcomes: Differential success rates between diagnostic subpopulations (i.e., schizophrenic, substance abusers versus mood disordered, substance abusers) were also compared to determine whether the models were best suited to one subpopulation or another. This measure was also applied to the criteria indicators listed above.

Indicator 5: Relative Rates of Recidivism: Percentage rates of patient recidivism for program graduates were also determined as a measure of the program's relative inability to effectively rehabilitate their target MICA population.

Indicator 6: Patient Satisfaction: Patients in both groups completed a standard New York State Office of Mental Health patient satisfaction survey (NYSOMH, 1990) within a month of their graduation from their respective treatment programs. This provided a measure of the patients' qualitative level of satisfaction within each program model.

Indicator 7: Cost-efficiency Rate: Relative cost per service unit was determined for patients who successfully met the goals of the program (See Indicator 1). This measure was included to illustrate differential program costs for those patients who met program goals and was considered a gross measure of program efficiency within the context of this study.

(Anderson, 1996, pp. 9-13)
Pilot Study Results
Large effect size differences between the two programs and their respective therapeutic models were found in this comparative impact evaluation. TLC and MICA-TLC patient dispositions for a 30-month period are demonstrated in Figures1-4 and Tables 1-4. Within the context of this investigation, graduates were defined as those patients who had completed the therapeutic program, had reached and maintained a Global Level of Function of at least 80, and had remained in community-based placement for at least three months.

With no significant differences found in patient age, SES, race, severity of substance abuse, length of substance abuse, or hospitalization history, and including only the 76 MICA patients of the TLC unit, the TLC Program more than doubled the rate of the positive therapeutic outcomes of the MICA-TLC Program. This occurred even though the MICA-TLC had delivered 35% more service hours per patient than the TLC program during the same period. The TLC program had delivered an average of 22.3 hours of group and individual treatment to patients weekly, while the MICA-TLC program had delivered an average of 30.1 hours per week. In addition, the cost-efficiency rate of $40.69 per unit of service for the MICA-TLC was found to be double that of the TLC, which was calculated at $19.04 per unit of service. This was primarily due to the TLC's higher number of successful treatment outcomes.

From the therapeutic success and failure rates, the differences in rates of recidivism, patient satisfaction survey results, and level of cost-efficiency, the integrated TLC program model appeared to have distinct advantages over the MICA-TLC program model in the treatment of severe MICA patients with low levels of education and high numbers of past hospitalization. However, from the results of the subpopulation and characteristics correlations, the disease-specific, therapeutic community model of the MICA-TLC program seemed to work well with patients who are more highly educated and have fewer previous hospitalizations.

The high degree of MICA-TLC clinical failures and recidivism and lower levels of patient satisfaction appear to be due to the inability of a traditional substance abuse model to treat low functioning MICA patients. Since the MICA-TLC admitted primarily low functioning patients, the clinical failure rate that was demonstrated in the results is understandable. The Integrated model appeared to be more suited to treating such patients than the MICA-TLC.
 

 
Table 1 Patient Characteristics
TLC
MICA
All Refs
N=76
N=149
N=360
Diagnosis - Primary Axis I
Psychotic Spectrum
76.3
68.5
68.4
Mood Spectrum
23.7
30.1
25.1
other
0.0
1.4
6.5
Diagnosis - Secondary Axis 1
None
1.3
3.4
2.5
Polysubstance Abuse
57.9
53.0
50.0
Crack/Cocaine Dependency
11.9
*
24.8
22.9
Alcohol Dependency
26.6
11.4
*
20.9
Other
2.3
7.4
3.7
Diagnosis - Axis II
Personality Disorders
3.9
3.3
8.0
Medications
Neuroleptic
76.3
68.4
66.0
Antidepressive
15.8
17.5
18.1
Anxioletic
1.3
0.7
2.0
Lithium
6.6
6.0
5.0
Anticonvulsive
0.0
2.0
0.0
None
0.0
5.4
8.9
Ethnicity
AfroAmerican
48.7
*
71.1
*
62.3
Caucasian
34.2
*
15.5
19.3
Hispanic
13.2
13.4
15.0
Other
3.9
0.0
3.4
Marital Status
Single
94.8
85.9
81.6
Married
2.6
2.7
7.3
Sep./Divorced
2.6
10.7
8.0
Widowed
0.0
0.7
3.1
Prison History
72.4
75.2
63.0
Military History
9.2
16.8
12.8

Note: The All Refs (all referrals) Column represents the patient characteristics of all  MICA patients referred to the two programs in  the hospital and shelter systems.  This not only includes those patients who entered the programs, but those who  refused admissions to the programs as well. * = p<.05 two tailed, for significant differences in between group comparisons of percentages.

 
Table 2 
Comparison of Patient Characteristics 
TLC         MICA TLC 
N=76       N=149 
mean
s.d.
mean
s.d.
t-value
F
Age
40.80
10.259
34.91
7.706
4.84
8.357
a
Education
11.01
1.963
10.87
2.410
0.46
0.550
Previous Hospitalizations
2.54
1.879
3.75
2.224
-4.07
0.454
Suicide Attempts
0.49
1.390
1.17
2.132
-2.52
3.915
b
Length of Stay in Program
137.89
108.289
111.37
79.395
2.09
15.869
a
Months of Homelessness
30.62
34.691
43.00
42.667
-2.19
3.014
Treatment Hours (Daily)
3.26
0.854
5.43
0.940
-16.91
1.802
b
Welfare Benefit Amount
433.79
153.425
290.31
258.504
4.45
59.190
a
Note: F= Levene’s Test for Equality of Variances. Within each row comparison, significant differences (p<.05, two tailed) are indicated by an ‘a’ at the end of the row. A ‘b’ at the end of a row indicates only a marginal effect (p<.10, two tailed)
 
 
 
Table 3 
Patient Outcome Summary 
AMA
%
Placed
%
Hosp.
%
Total
%
TLC PROGRAM
At Discharge from Program
18
23.68
**
51
67.11
*
7
9.21
76
100
Three Months Post-Discharge
9
11.84
*
34
44.74
**
8
10.53
51
51
(in community-based housing)
TLC TOTAL FINAL OUTCOMES
27
35.53
**
34
44.70
**
15
19.74
76
100
MICA-TLC Program
At Discharge from Program
93
62.42
**
43
28.86
*
13
8.72
149
100
Three Months Post-Discharge
19
12.75
*
14
9.39
**
10
6.71
43
28.8
(in community-based housing)
MICA-TLC FINAL OUTCOMES
112
75.17
**
14
9.39
**
23
15.44
149
100
*p < .05. **p < .01
 
Table 4 
Correlations of Significant Subpopulation 
Characteristics At the Time of Discharge from Program 

 

Services hours (daily)

Discharge Type

Placed 

Previous Hosp.
 
 

Episodes

Education 
 
 

Years

TLC
.38
**
MICA TLC
.60
**
No. Prev. Hospitalizations
TLC
.31
*
MICA TLC
-.14
*
Months Homeless (Lifetime)
TLC
.34
**
.49
**
MICA TLC
.13
*
-.24
**
Length of Stay in Program
TLC
.38
**
MICA TLC
.61
**
-.14
*
.20
*
                Note: ** p < .001; * p < .005, one tailed, bivariate correlations of patient characteristics
 
This pilot study examined two treatment programs for MICA patients that use very different treatment models and appears to demonstrate advantages to using integrated models to treat lower functioning MICA patients. The efficacy rates of the two transitional living communities suggest that the use of an integrated approach, which emphasizes the individualized mix of treatment options, produces greater patient satisfaction and yields higher levels of efficacy than traditional, disease-specific programs currently provide. In addition, integrating services within a single program reduced the costs of successful therapeutic treatment and duplication of effort because patients were treated within the same facility. This leads to the proposition that the adoption of a integrated program model might allow for the customization of program services to meet the needs of most MICA patients and produce greater overall treatment gains for this population.

The main problems with this investigation, however, lie not in the indicators chosen or in the results obtained, but in the methodological approach used to determine net program worth and value. From the top down approach used to evaluate the treatment outcomes of these two programs, we cannot assume that the obtained results were not in fact due other factors outside the seven chosen indicators. Because all seven indicators were chosen by the investigator in a traditional evaluation research model, the study failed to incorporate the programmatic priorities of all the other stakeholders in the study, for example, the funding source, hospital administration, and the programs’ clinical staff. The Community Support Services (CSS) Program Director, the investigator, was the only stakeholder viewpoint represented in this study. This has been termed, "A failure to accommodate value pluralism" and represents an evaluator bias that is often present in evaluation studies of this type (Guba & Lincoln, 1989, p.34). Other indicators, chosen by the treatment team, the funding source, or hospital administration, may have changed the outcome of this study; by either diluting the effect sizes, attributing the treatment gains to different factors, or strengthening the claim of net value and worth for the TLC program. As a consequence, the results of these seven indicators cannot fully attribute worth and value to one program over another. An additional problem with this approach lies in the weakness of the measures of clinical improvement (outcomes) that may be related to the program models. No specific tests of patient functional or psychological improvement were performed and only gross measures of program completion were used as measures of positive outcomes. Use of measures to identify specific changes in patient functioning and distress levels would produce more significant results.

Overall mental health service and substance abuse program effectiveness is difficult to assess (Barkman et al., 1996; Newman & Tejeda, 1996). Due to a variety of methodological and paradigmatic differences, program evaluation studies are often biased, rendering their results inconclusive, at best. The following section reviews the methodological issues in program research and points toward ways in which such bias can be reduced from program evaluation studies.


3. EPISTEMOLOGICAL ISSUES : REVIEW OF RESEARCH METHODOLOGIES IN MENTAL HEALTH SERVICES

Mental health programs, along with other health and social welfare services, are coming under increased scrutiny and evaluation. With consistently smaller budget allocations for mental health, many federally funded programs are being radically changed, some severely curtailed, and the relationship between government and private sector providers is being realigned (Inouye, 1983; Klerman, 1974). In addition, state and local government agencies that were expected to reimburse mental health programs for federal shortfalls in funding, have not been universally successful in meeting this challenge (National Institutite for Mental Health [NIMH], 1991a). Thus, new research into the efficacy of mental health services has been called for to meet these growing challenges. However, because traditional research methods have not been readily applicable to the study of applied health programs and many researchers are reluctant to investigate programs in naturally occurring settings, a search for more appropriate methodological approaches and researchers trained in such approaches is now underway (Newman, Howard, Windle, & Hohmann, 1994; Newman & Tejeda, 1996).

These developments are especially significant for mental health services, which have been more regulated and financially supported by government than any of the other services within the health sector in the past 200 years (Rothman, 1971). Throughout this period there have been numerous cycles of mental health reform and innovation, followed by phases of criticism, dissension and retrenchment (Bockman, 1963; Caplan & Caplan, 1969; Deutsch, 1948, Grob, 1973). While the "reforms in mental health have coincided with periods of progressive social change in the larger American Society, phases of reactions, criticism and retrenchments have occurred with the aftermath of war and economic decline" (Klerman, 1974, p. 783). The economic slump of the early ‘90s following the prosperity and massive federal spending of the ‘80s, continues the cyclic pattern of change in mental health services.

After a decade and a half of growth in mental health and substance abuse services, a number of criticisms have been leveled at the mental health sector. Chief among these criticisms is that of program ineffectiveness (NIMH, 1989). To date, there are very few applied or experimental research studies that address program effectiveness (PsycINFO Index, 1995). Without knowing what programs or treatment models effectively work for a variety of patient populations in a variety of settings, legislators and grant funding sources have no way of planning where and how their limited resources should be spent.

Inadequate research strategies and methodologies have been cited as the primary reason why such program efficacy research has not been studied (Newman et al., 1994). Mental health services research cuts across the disciplines of economics, sociology, political science and psychology. One of the prime purposes of mental health service research is to provide empirical evidence and support to guide policy decisions at all levels of government and nongovernmental organizations (NGOs). However, until recently, very few evaluation studies of mental health programs or their associated models have been reported in the professional psychological literature (PsycScan, 1994).

Newman et al. (1994) writes that, until recently, clinical psychologists have tended to ignore and not conduct mental health services research. This has been a direct result of the limitations in methodological training that psychologists receive and a bias against such research in the publication standards of the professional and academic literature. According to Newman et al., studies that assess success rates and program effectiveness in mental health service programs have not been generally deemed worthy of publication. In addition, most clinical psychologists and clinical researchers are trained in experimental and quasi-experimental techniques that make it very difficult to adequately evaluate the global, multifaceted, molar effects found in applied treatment programs (Guba & Lincoln, 1989). The methodological deficits and bias toward the scientific method in professional publications have made it difficult for psychologists to develop and utilize research

methodologies to fully assess the efficacy of mental health service programs and program models in both the public sector and NGOs (Clarke, 1995; Newman & Howard, 1991).

Researchers who investigate programs and clinical factors related to improving the quality and impact of mental health services are often handicapped by the perceived legitimacy of their applied research and the methodological approaches they utilize (Newman et al., 1994). Historically, traditional research methods and journal/grant review criteria have not taken into account the global questions and systemic points of view necessary to fully understand the therapeutic delivery systems under evaluation. Thus, there has not been a coordinated, sustained effort to determine program efficacy for the majority of human service project initiatives (Fetterman et al., 1996; NIMH, 1989).

Many program and project research evaluations attempt to present the factors and/or 'facts' uncovered in their program evaluations. Such evaluations generally utilize traditional experimental, quasi-experimental, causal comparative, correlational, and other approaches. Such evaluations attempt to identify the most salient factors for good program performance on a molecular level. In the process, many of the characteristics and practices that define a successful program may be ignored or dismissed as inappropriate or unimportant to the objectives of the research. For example, in an investigation of a psychotherapy program, investigators may choose to only examine the mean or median number of therapeutic hours received in a voluntary outpatient program. Though this may or may not relate to overall patient satisfaction, motivation, and progress, such process measures do not determine the overall performance or level of program effectiveness. Thus, almost no programmatic conclusions can be made on the basis of this information.

Despite significant clinical and basic research progress made in the diagnosis and treatment of mental disorders over the past two decades, many questions about how to provide high quality, effective treatment services still have not been answered. For people with severe, persistent, disabling mental disorders, this situation means that individual diagnoses may be inaccurate, treatment plans inadequate or ineffective, and essential services unavailable (Lalley et al., 1992; NIMH, 1991a). As a consequence, such individuals are often forced to not only endure a lonely struggle against the despair and distress caused by their mental illness, but must also negotiate a confusing, fragmented maze of human services, created by a wide range of often well-meaning public and private sector service providers.

Instead of concentrating on determining the individual facts and salient factors associated with successful treatment outcomes, human service researchers should be more concerned with one global question that allows for a more holistic examination of program worth: "What works, for whom, under what circumstances?" (NIMH, 1991a, p.vii). The net effect of any treatment program or human service project is determined by the integrated use of multiple, interactive program components, delivered at a site conducive to recovery or rehabilitation to a population that will be receptive to such treatment (Breakey, 1987; Minkoff, 1991). Since successful treatment outcomes depend on the global interaction of all these factors, research methodologies used to investigate such programs must also mirror this global, molar intervention to accurately determine whether successful outcomes have indeed occurred.

Critical Methodological Factors in Assessing Program Efficacy

Over the years there have been a number of review articles that have called attention to the need for an increased research effort into efficacy studies of mental health services (Inouye, 1983; Klerman, 1974; Newman & Tejeda, 1996). However, until recently, these articles only addressed the need to increase the research effort without recognizing the methodological developments that would be necessary to adequately assess program strengths.

Klerman (1974) gives the first comprehensive account of the state of mental health service research. In his descriptive article he not only identifies the major stakeholders that should be included in evaluation research, but outlines the major concerns voiced by each constituency. He notes that while the public at large, the courts, mental health professionals, and government agencies all have an active stake in the results of such research, each has a different focus and agenda for the outcomes of evaluation research studies, and requires varied types of data with which to formulate their concerns as to how, where, and in what manner mental health programs should operate. The identification and recognition of the needs of all major stakeholders in any mental health program is a critical step that is often overlooked in most evaluation studies (Guba and Lincoln, 1989; Newman et al., 1994).

The public at large has an active stake in new mental health services evaluations (Farkas & Anthony, 1979). In some cases, an adversarial climate has developed among mental health program critics in the general public and mental health professionals and administrators. Because many critics feel that mental health has expanded too much into areas that had previously been regarded as social deviance or legal misdemeanor, such as substance abuse and treatment for the homeless, the public at large mirrors professional uncertainty about treatment adequacy, clinical training for paraprofessionals, and about what are or are not effective treatment strategies for various patient populations. This reflects a lessening of public trust and confidence in mental health services that parallels the erosion of funding and governmental support (Farkas & Anthony, 1979). In addition, community groups are now seeking a more active voice in the operation of mental health service programs within their neighborhoods and catchment areas. In general, these groups want to ensure that treatment programs maintain standards that will protect their community, and not place the public or patients at risk (Fetterman et al.,1996; NIMH, 1991).

Federal and state courts have been become increasingly involved in mental health service programs over the past 20 years. Prison-based substance abuse and mental health programs have markedly increased over the past decade and a half (NIMH, 1991). However, most of these programs have evolved due to court mandated levels of care and have mainly documented their measures of criminal recidivism as the sole measure of program efficacy (Robitscher, 1972a). Since the courts have mandated this treatment and view success in treatment as key a factor in rehabilitating both the involuntary hospital patient and the mentally ill legal offender, they have become more interested in mental health services evaluation as well. Thus, the court systems are active stakeholders in any efficacy evaluation of mental health services and seek information as to the type and level of services that will be delivered to disordered, disabled, and incarcerated individuals.

The court system has also been a major contributor to the development of program models and standards of practice for both hospital and community-based mental health service programs. Due to general concern for patients' civil rights and for the possible infringement of their personal liberties in cases of involuntary hospitalizations, a number of landmark court decisions mandated not only effective treatment but treatment at the least restrictive level (Klerman, 1972; NIMH, 1989; Robitisher 1972a). Moreover, there has been increased concern about the depersonalization and institutional dependence fostered by large public hospitals. Within the mental health professions, there is a general awareness that large hospital based programs become professionally and therapeutically bankrupt, and thus ineffective. This sentiment fostered the creation of community mental health centers, which are felt to provide alternatives to the low levels of institutional care previously provided to the poor and disabled in large public hospitals (Westin, 1972).

Such mental health program interest by the court and legal systems has been the most significant reason for reforms in mental health services including the community mental health center programs (Klerman, 1972). By concentrating on the difficulties and dissatisfaction encountered with the large public mental hospitals, particularly county, state, and Veterans Administration hospitals, the courts have mandated improvements in treatment and service programming that have led to significant reforms in mental health services. These reforms have resulted in decreases in program size, emphasis on community-based treatment, and increased intensity of treatment in both community and hospital-based programs. They have also been shown to increase the probability of more rapid discharges and reduced recidivism (Farkas & Anthony, 1991; Lamb, 1972; Ullman, 1964).

"At all levels of government-federal, state and local evaluation efforts are frequently initiated by fiscal and budgetary agencies "(Klerman, 1972, p. 784). Increasingly, political, fiscal and administrative decisions regarding mental health programs and their associated treatment models are being made on the basis of fiscal goals to deliver the most effective programs possible for the least amount of funding. In addition, state and local agencies charged with monitoring and promoting mental health service research and delivery have been increasing their efforts to determine what constitutes effective mental health programming for a variety of patient populations. Initiatives to reform mental health care have precipitated legislation to develop and expand state and local commissions to further investigate program utility and effectiveness (Frank et al., 1994; Scott & Ginsburg, 1994).

At the federal level, mental health program research has become a high priority for agencies responsible for mental health funding and monitoring. In fact, research funding at this level has been increased from $90 million in 1992 to $369 million budgeted for Fiscal Year 1997 (NIMH, 1991a). With the reorganization of ADAMHA (Alcohol Drug Abuse and Mental Health Agency) in 1992, the (NIMH), National Institute on Drug Abuse (NIDA), and the National Institute on Alcohol Abuse and Alcoholism (NIAAA) must devote no less than 15% of their total budget for health care services research, all coordinated by NIMH [ADAMHA Reorganization Act of 1992, 464R (f)(2)]. This increase in funding clearly demonstrates an increased level of commitment by the federal government to improving service research and overall program effectiveness for the mental health program consumer.

Mental health consumers and their families have a very active stake in the outcome of program research but, until recently, have had little influence on the research process. Families of the mentally ill themselves have to shoulder tremendous financial and emotional burdens. Each year, 65% of discharged psychiatric patients (approximately 1.5 million), return home and live with their families (NIMH, 1991a). Due to the high cost of hospitalization, many of these patients return home earlier than they would have in the past, still disabled by psychiatric symptoms (NIMH, 1991a). State and local governments are also beginning to recognize the stake that patients and their families have in the mental health treatment by enacting legislation to give the consumer and his/her family a voice in the therapeutic process (NYSOMH, 1990).

The NIMH (1991a) credits the National Alliance for the Mentally Ill (NAMI) with a successful lobbying effort for the inclusion of the mentally ill and their families into the evaluative research process. NAMI together with Family Alliance for the Mentally Ill (FAMI) were developed as grass-roots organizations to serve as advocacy groups on behalf of mentally ill persons and their families. These organizations bring the problems and issues of the mentally ill and their families to the attention of local, state, and federal governments. In evaluative research studies, NAMI has championed the cause of "research designed to identify ways to help patients readjust to the community within the least restrictive environment possible and to prevent relapse through early intervention" (NIMH, 1991a, p.13). In "Clinical Services Research: Enhancing the Real-World Applications of Clinical Science" (1991b), NIMH has outlined the critical points that NAMI and FAMI advocate for service evaluation research studies.

- Researchers must carefully assess the ramifications of family involvement in the care of a mentally ill member -including the characteristics and conditions of caregiving families; the degree and varieties of family stress; the effectiveness of various coping and adaptation patterns; timing and extent of caregiver burnout; and the impact of various kinds of respite care for family members.

- Studies should be performed that are focused on family issues and produce results to assist families with mentally ill members in functioning more effectively and with less turmoil. Such studies would provide effective education in technique for dealing with mentally ill persons without succumbing to the overwhelming anxiety; motivating the patient to become more self-sufficient; and understanding and communicating appropriate expectations. An essential need is for research on the long-term effectiveness of such family education programs-both in helping the patient and in reducing the family's burden.

- For generations, families with mentally ill relatives have dealt with the problems of violence toward family members, exacerbated now as a result of a substance abuse by the seriously mentally ill. Investigators must focus on the predictable, frightening and violent behavior that patients may exhibit toward family members, with the goal of developing more efficient criteria for predicting such behavior and more objective ways to manage and prevent it. Such studies must place a high priority on meeting the needs of families for early education, prevention, and intervention. In this connection, attention should be given to identifying techniques of family adaptation and have proved effective.

-Dealing with mental illness is expensive. Families become frustrated and angry as their savings dwindle, often with meager results. A guide on how to obtain the most effective services even with limited personal resources would be a welcome aid. Such a guide-based on evaluative research, not an opinion-could help families make informed decisions about more selective use of mental health services.

(NIMH, 1991a, p.12-13)
Together, these points highlight the very active stake that the mentally ill and their families have in evaluative research. Program evaluations that do not take these points into consideration cannot be considered complete.

Mental health professional groups also have an active stake in the success of programs and on the outcomes of evaluation studies of program services (Isaac, 1971; Weiss, 1972). Concern for effective program planning and therapeutic results can be seen in the various calls for evaluation research by a number of professional groups. The American Psychiatric Association, The National Association for Mental Health, and American Psychological Association (APA) have, at various times, all called for increased research efforts in program effectiveness (Klerman, 1972; Lalley, 1992; Newman 1996, 1994, 1991). Though these calls for more research demonstrate an interest and active stake on the part of professional groups, little if any comprehensive evaluation research has been performed to date.

In response to the NIMH (1991a) call for increased service research, the APA published a special section on mental health services research in the Journal of Consulting and Clinical Psychology (Newman et al., 1994). In this special section, the authors not only reviewed the U.S. "National Plan of Research to Improve Services", but presented a series of research articles that exemplified methodological developments in this area of psychological research. These studies were presented to demonstrate innovative methodological approaches in assessing program and service efficacy. However, each of the studies appear to be focused on a different aspect of overall program efficacy, and are consequently limited in their ability to demonstrate overall program effectiveness or meet the research goals set forth in the NIMH national agenda. All of these studies failed to demonstrate program or service efficacy due to the positivist -reductionist approach utilized by the researchers (Anderson, 1996b).

Though the methodological approaches used in each of these studies are generally accepted as innovative scientific investigations suitable for publication in the psychological literature, each study has difficulty accounting for the full range of interaction between not only the program variables but from the point of view of each of the main stakeholders in mental health service programs. Thus, these studies have difficulty providing a comprehensive account of program effectiveness. Though these methodological approaches may be instrumental and valuable as part of an overall evaluation effort, they cannot be used as the sole basis for determining success in mental health service programs. A comparison-contrast of the main methodological approaches, presented in the following section demonstrates the advantages of departing from the reductionism inherent in traditional approaches in favor of a more comprehensive evaluation methodology.

Comparison-Contrast of Research Designs in Services Research

Correlational, causal-comparative, and evaluative research methodologies are the most common research approaches used to evaluate service programs (Borg & Gall, 1989). Though they have much in common, they differ in their utility, comprehensiveness and ability to establish cause-and-effect relationships among study variables with a strong degree of certainty. As a consequence, they also differ in their ability to predict future effects and causal patterns that can be attributed to the study variables. This difference is primarily due to the limitations of the methodologies to attribute the full range of possible causes to effects observed in natural or artificial/experimental settings. Though each method has situational and experimental advantages over the others in program research, each varies in its situational utility as well.

Correlational Designs

While the correlational method is well suited to establishing relationships between the variables, it cannot demonstrate cause-and-effect relationships by itself. The correlational method is restricted to quantifiable data in the data set and therefore limited in its utility. Though readily applicable to quasi-experimental study situations, it is often difficult to apply in natural settings, where identification and measurement of the most important variables often becomes difficult. This problem is illustrated by the results obtain in the Yeaton (1994) study which investigated the relationship of patient attendance in self-help group meetings to successful completion of an alcohol treatment program.

This study examined the relationship between rates of patient attendance and successful completion of programs. The relative rates of attendance in the service milieu of self-help groups are compared to the rates of successful treatment outcomes as a measure of programmatic effectiveness in treating substance abuse. However, collateral treatments for substance abuse and/or other deficits were not discussed. In addition, the actual service components of the self-help program were not discussed. As a consequence, any relationship between the study variables of attendance and outcome becomes inconclusive. The actual effects that were noted may be due to variables outside the scope of this investigation that were related to the study outcome variables. Thus, from the relationship found by this single correlational indicator, programmatic effectiveness cannot be inferred.

Correlational method research studies are best suited to discovering relationships solely among study variables. As illustrated by the Yeaton (1994)

investigation, it is very difficult to apply this approach to mental health service studies. Identifying and then measuring all the variables that define a program’s level of effectiveness becomes almost impossible using only this approach. When compounded by the various interests and focus of each stakeholder associated with a treatment program, the correlational method becomes almost useless in defining what works, for whom, under what circumstances.

Causal-Comparative Designs

The causal-comparative method is described as well suited to demonstrating significant relationships, group norms, and traits in natural settings (Borg & Gall, 1989). This method can also be used in study situations where experimental manipulation is difficult or impossible, such as in mental health service studies. However, the causal-comparative method can only demonstrate causality from the data presented within the narrow scope of the study variables and is, therefore, also limited in its ability to suggest causality in either experimental or natural settings. Alternative interpretations are often possible when this method is employed (Borg & Gall, 1989; Wood, 1974). Thus, this method is similarly limited in it utility and comprehensiveness to demonstrate a program's level of efficacy or effectiveness.

This problem was demonstrated in a study described by Uehara, Smukler, and Newman (1994). The investigation attempted to answer the question, "Who needs what services and what degree of care?" and was actually a field test of a procedure to identify and link the psychological, social, and physical functioning needs of individuals with severe and persistent mental illness to specific levels and types of treatment programs and rehabilitative services. Using the Level of Need Care Assessment Scale (LONCA scale) to assess patients' level of functioning, the researchers attempted to match the level of patient need to the level of care in a clustering method. The LONCA scale is an instrument designed to measure patients’ functional level across a wide range of psychosocial dimensions. It was speculated that the use of such a scale to place patients in programs that would specifically meet their needs would improve the treatment outcomes for this population. The degree of dysfunction would determine the level of appropriate care to be provided.

Though 65 case managers carefully rated each patient's level of need and the resulting data set was factor analyzed to cluster patients into groups that might benefit from different levels and types of care in community-based settings (Uehara et al., no actual outcome data were provided to substantiate the underlying assumption that there is a causal connection between level of dysfunction and the various outcomes that patients experience in a variety of treatment programs. In addition, critical variables in patient recovery and program operations were not taken into account. The effects of patient's level of motivation for treatment, demographic profiles, diagnostic groupings, level of patient satisfaction, program modeling, program environment, and other factors may also play an important role in determining whether a patient will respond to treatment within a given treatment program or not. Using their method, these variables could not be taken into account. Thus, the results of this study remain inconclusive.

The causal-comparative method, like the correlational method and relational methods in general, is limited in its ability to establish cause-and-effect relationships between study variables. As noted in the Yeaton (1994) and Uehara et al. (1994) studies, both methods can be criticized because they attempt to break down complex behavior into very simple components. Understanding the causal variables or set of variables that are related to the complex activities or traits of a mental health service program is beyond the scope of the research study when these methodologies are utilized. Generally, the phenomena and behaviors associated with an operational service program are so poorly understood that incomplete sets of study variables are chosen for study and analysis. This appears to have been the case with all the studies profiled in the Journal of Consulting and Clinical Psychology special section on mental health service program efficacy studies. In addition, analysis of multiple variables from one setting will not expose the complex interaction of variables across multiple settings or subject groups in other programs. These and other problems of measurement and analysis contaminate or weight inferences and result in erroneous or misleading conclusions. This limits the comprehensiveness and utility of both methods in studies of natural events and phenomena, such as those that occur in mental health service programs.

Evaluative Designs

Of the principal research methodologies used to determine the effectiveness of mental health service programs, the evaluative method is the most suited to identifying and measuring qualitative indicators and demonstrating program worth and relative value to its consumers. Thus, it is the most applicable approach to the study of mental health program efficacy.

Evaluation studies usually examine cause-and-effect relationships in natural settings. Without the constraints of the experimental study controls or sole use of quantifiable data sets, evaluative studies can identify the most salient relationships among all quantitative and qualitative variables in service programs. Because of this, the evaluative method is more comprehensive and has a higher degree of utility in natural settings than the other two methods.

The evaluative method is an applied research method that focuses on determining the merits of educational, job training, health care, and other institutional programs in health, education and welfare. This approach differs from correlation and the causal-comparative methods in that it not only looks at the relationship of a few, obvious variables to determine a cause, but it alsoexamines all observed variables that may impact the goals of the program under study. With evaluative methodology, the causes of the positive or negative program outcomes become the main focus of study. Using program goals and actual performance measures in meeting those goals, evaluation researchers attempt to locate factors related to the actual program outcomes. Traditionally, an evaluator will work directly with program leadership, staff, and consumers to determine the most salient factors that define program performance with regard to the goals of a program. In ideal evaluation study situations, mental health service program staff, directors, funding sources, patients, families, community groups, and all others who have a stake in the investigation are invited into the variable identification phase. In this phase they identify the critical program variables to be used in the study, the methods of data collection, and subsequent data analysis techniques that will be used to determine program efficacy levels. Once determined and agreed upon by all the stakeholders, these factors and study procedures produce results that can be returned to the stakeholders of the program to implement program modifications and improvements. Thus, the results of an evaluation research study can be used to modify program operations to increase performance toward meeting those program goals more effectively.

Unlike other research methodologies, evaluation research is usually initiated by someone's need for a decision to be made about policy, program management, or strategic planning. By contrast, experimental methodologies initiate studies based on a hypothesis and the research is conducted to reach a conclusion about the relationship between the variables and whether to reject or accept the hypothesis. In evaluation research, where the focus is on making practical decisions that will impact the effectiveness of a program, the emphasis is on testing variables against program goals in a decision-making process, rather than hypothesis testing.

This decision process examines the impact of a mental health treatment program's components and modes of service delivery in meeting the stated treatment and outcome goals of the program, then uses the evaluation data to redefine and modify the service program to more adequately meet the needs of the patients. In addition, the goals and objectives of the program and treatment components are reexamined to improve the relative worth of the program for all the stakeholders associated with it.

DonGiovanni (1988) performed an evaluation research study of a program for mentally ill chemical abusers. As in the Yeaton (1994) study, patient participation rates in group therapy and other program components were compared with patient outcomes as measures of mental health program effectiveness. However, this study also included measures to determine the overall level of patient satisfaction in the program, attempted to measure recidivism, polled referring hospital staff as to their opinion of the program, and also surveyed community-based mental health providers to obtain data on perceived program effectiveness. These measures incorporated the opinions and views of all the major stakeholders associated with this program. Thus, the results of this study demonstrated not only the overall level of effectiveness and relative worth of the program, but punctuated the need for additional program modifications to coordinate all the services of the program in a more cohesive, comprehensive fashion.

The DonGiovanni study (1988) illustrates the distinctive characteristic of evaluation research studies. This type of study examines the relative worth and merit of a program or program components. Thus, judgments of programmatic merit and worth that are not emphasized in other research methods are not only appropriate, but necessary in the evaluation of a program's effectiveness in its natural setting. Causal factors or variables that impact program effectiveness are also judged as to their worth, merit, and value in meeting program goals.

Evaluation research draws heavily from other methodologies. Qualitative as well as quantitative data collection and analytical techniques are often used. In the DonGiovanni evaluation (1988), correlational data was used along with the qualitative data from surveys to determine the program's relative worth. Because of this, cause-and-effect determinations arise from a richer, more comprehensive data set than with sole use of quantitative data and advanced statistics. Thus, the use of evaluation research methods in this mental health service program allowed for a more comprehensive determination of what worked for whom, under what conditions than other methodologic approaches.

The only major limitation of the evaluation research method in mental health service studies lies in their generalizability. Due to the applied nature of this method, programmatic and situational variables tend to be specific to the program under study. It is therefore often difficult to generalize the results to even similar program types. Since each program study is situated in different physical environments, with different staff, and other characteristics, the evaluation study becomes customized to that program's variables. Thus, generalizing the conclusions of one program evaluation to other programs may be difficult because many of the salient variables change from one program to another.

However, many of the lessons learned from one program can be applied and tested in other, similar treatment settings and may serve as models to enhance program or program component effectiveness in those programs as well. When performed on a program-by-program, case-by-case basis, such evaluation research data may serve as valuable tools for program modification and improvements.

Finally, the evaluative method is not constrained to hypothesis testing, but seeks to functionally establish the most salient variables operating in the natural settings of mental health service programs. Evaluation studies attempt to determine the impact of complex variable interaction on the goals of the program. The primary advantage of using this method is to provide data to policy and decision makers that can be used to improve program performance to more successfully meet program goals. Thus, it is both comprehensive and readily applicable to studies in naturally occurring mental health program settings that require data for both research and program improvement.

Fourth Generation Evaluation Research

Guba and Lincoln (1989) have traced the development and expansion of evaluation research and have refined the methodological approach to not only reflect state of the art enhancements in health and mental health program assessment, but provide a potent vehicle for program improvement as well. They note that the first generation of evaluation concentrated on the systematic collection of data and measurement of phenomena, while the second generation dealt with description of patterns of strengths and weaknesses with respect to certain stated objectives. The third generation of evaluation research focused on judgments of relative worth between programs and program components.

Though the various generations of evaluation built on the gains of preceding phases, with each successive generation providing a foundation for more detailed and sophisticated assessments of programs and organizations, there remained significant limitations in evaluation methodologies. The main problems with the first three generations were a tendency toward managerialism, failure to accommodate value-pluralism, and an overcommitment to an experimental paradigm of inquiry. A tendency toward managerialism means that the evaluator and the clinician/manager/

administrator responsible for the program under study become either too close to remain objective and impartial or become adversarial. This may contaminate or shade the results of the study. This also occurs in traditional correlational and causal comparative studies. Failure to accommodate value-pluralism refers to the inability of the investigator to incorporate the values and viewpoints of all those who have an active stake in the outcome of the study. This was clearly the case in the pilot study (Anderson, 1996a).

Finally, most first, second, and third generation evaluation research studies, as with other social scientific methodologies, tend to make sole use of the scientific method to determine the 'truth' or 'truths' underlying a phenomenon instead of focusing on the overall worth of the programs and services to the patients and the communities they serve. The recognition of these deficits in evaluation methodologies led to the development of what has been referred to as "fourth generation evaluation research" (Guba & Lincoln, 1989).

The fourth generation of evaluation research is responsive evaluation. It has been termed responsive because it seeks out different stakeholder views in determining the variables and instruments that will be used in the investigation and then responds to the needs of all those who have an active stake in the evaluation process and results.

Fourth generation, responsive evaluations have four main phases that may be reiterated or overlap. In the first phase stakeholders are identified and solicited for those claims and issues they want to bring into the research study. Guba and Lincoln have identified three main classes of stakeholders who would have an active interest in a program investigation and its outcomes: "The agents, those persons involved in producing, using or implementing the [study results]; the beneficiaries, those persons who profit in some way from the use of the [study] outcomes; and the victims; those persons who are negatively affected by the [study]"(1989, p. 40). In the second phase, all stakeholders are introduced to the others to begin the negotiating process through comments, agreements or disputes to determine what issues and topics will be assessed by what instrumentation. The third phase involves further information collection as non-resolved disputes are investigated and further negotiated. Finally, in the fourth phase, negotiation among stakeholding groups, under the guidance of the evaluator, takes place to reach a consensus and the information is collected, analyzed, and disseminated to all the stakeholders for comment and publication.

Using the process oriented, fourth generation responsive summative worth evaluation methodology would improve the current state of mental health services research and fulfill many of the goals set forth in the U.S. national plan of research to improve services (NIMH 1991a). Instead of researchers and program directors choosing critical program variables and using current correlational, causal comparative or quasi-experimental methods to establish a 'scientific truth', program staff, patients, funding sources, governmental agencies, and other interested stakeholders could collaboratively agree on the critical study variables and study methodologies that would be used to determine the relative value of the mental health service. The results could then be used not only to determine what works in one program or another for given patient populations, but could also be used as a tool to improve services in the study programs as well. As each new set of data within a program is analyzed, remedial action plans could be collaboratively agreed upon and new evaluation data obtained and analyzed to ensure a process of continuous quality improvement. Thus, in using this responsive evaluation methodology, not only would overall levels of mental health program effectiveness be obtained, but a mechanism could be established to allow for continuous quality improvement in the program over time.

The DonGiovanni (1988) evaluation study identified all major stakeholders and included their participation in the study and in the program modification phase after the results were analyzed. Though these stakeholder did not have a role in designing the evaluation criteria or study methodology at the onset of the investigation, they participated in the results and program modification phases after the initial levels of effectiveness were obtained. In addition, it was noted that ongoing program evaluation involving all the major stakeholders would continue into subsequent evaluations of that program. This goes far beyond the simple collection of data that traditional evaluation and experimentally based studies reported on and includes a program monitoring and improvement mechanism for future program improvements. Thus, in this and other responsive evaluation studies, the participation of all associated vested interest groups becomes not only a research tool but a programmatic problem solving mechanism as well (Fetterman et al., 1996).

This evaluation study, described in the following section, modified and refined the fourth generation responsive evaluation approach to include all stakeholders in every step of the evaluation process and provided for an expert facilitator to ensure that program characteristics and patient measures, which are common to all programs of the study type, were included as study indicators. The main problem with participative research of program models often lies in the expertise of the stakeholders. If the stakeholders have training and expertise in research designs and methodologies, they may be able to determine and measure those programmatic indicators that are germane to many programs of their particular type. However, this is generally not the case. Clinical staff, administrators, funding sources, and the mental health consumers often have little or no training in program evaluation. Thus, the results may demonstrate program effectiveness for their local program, but these results cannot be generalized to other, even similar, programs. Having an expert facilitator, in addition to all the stakeholders of a program, reduced the chances of managerialism bias or failure to accommodate value pluralism and increased the probability that the study results and conclusions would be generalizable to programs of a similar type. This innovation directly addressed the main criticism of managerialism in traditional program services research and generalizability of results in evaluation research designs.

By introducing both standard and local types of indicators, this study not only produced valuable program efficacy data, but introduced an innovative continuous quality improvement mechanism to the study programs to demonstrate areas of measurable changes in the programs that can be used for future program improvements. The standard indicators of successful treatment outcomes (AMA discharges, correlations of service hours delivered, subpopulation outcomes, recidivism rates, patient satisfaction, and rates of cost efficiency) were chosen to address those program and patient factors that are common to all MICA programs. The local indicators were determined by the programs’ staff and other stakeholders to monitor what they believe to be important variables that demonstrate their program's effectiveness. As they continue to monitor and measure these indicators beyond the study period, they will be developing a pool of data that can be used for continuous programmatic change and improvement. Thus, the use of two types of indicators provided not only program effectiveness data but a mechanism for long-term continuous quality improvement as well.

The continuous quality improvement mechanism was incorporated into the programs’ operations as a mechanism for future programmatic changes and innovations to improve the quality of patient care. After the initial agreement was reached on the nature and measurement of local indicators by program staff and other stakeholders, the first data collection phase of the study provided baseline data for decision making at the local level. All stakeholders were given an opportunity to participate in developing action plans to improve the quality of care at the program level and time-tables were established to measure the impact of those action plans. The results of the second data measurement phase of the local indicators evaluated the impact of the action plans on program effectiveness. This research design improved on that of DonGiovianni (1988) by incorporating greater stakeholder participation in the decision making process at the local level, while providing for generalizability of results.

Rationale for Choosing Responsive Evaluation Methodology

Of the most commonly used research methods, the evaluative approach is the most comprehensive and applicable research method for the investigation of complex variable interactions and is well suited to determining cause-and-effect in natural settings, as well as the relative value of mental health programs. Without the constraints of experimental study controls and manipulations, evaluative research can identify the overall, global relationships among all important factors that operate in a mental health program's input, process, content, and products. This method offers a more comprehensive approach to applied research problems than either causal-comparative or correlational methodologies and is an extremely useful tool for many applied research projects that cannot be experimentally manipulated in a scientific paradigm. Thus, evaluation research is the most useful approach of the principal investigative approaches in determining the global programmatic determination of 'what works, for which patients, under what circumstances' in mental health treatment programs.

Evaluation research methods can incorporate the views of allied professionals from a variety of disciplines, program administrators, directors, patients, families, and community action boards more readily than other methodological approaches. In addition, it can be more readily applied to the evaluation of a wider variety of mental service programs than traditional research approaches (Fetterman et al., 1996; Guba & Lincoln, 1981; Windle & Lalley, 1992). Since the intent of this research is to determine the value or worth of services in a particular program or program model, evaluation research studies can provide efficacy data that fit within the worldview of all interested stakeholders. This leads to greater public acceptance and better direction in mental health program planning, funding, and clinician training. Most importantly, evaluation research promotes more effective treatment programs to enrich the lives of those patients and families who must endure the financial, social, and personal costs of mental illness on a daily basis.


4. HYPOTHESIS
1. Similar program evaluation studies of this kind have found differential rates of program effectiveness between integrated and disease-specific program models for MICA patients (Anderson, 1996a). This was not only reflected in the clinical indicators, but in the cost efficiency and patient satisfaction results as well. Based on this, it was hypothesized that MICA patients with relatively lower initial GAF and educational levels would respond better to treatment in the integrated program environment than in the disease-specific model. Conversely, it was also hypothesized that patients with higher GAF and educational levels would make greater treatment gains in a program that uses a more structured disease-specific model. A comparative analysis of the patient’s demographic history (i.e., educational level, treatment history, and work experience) and clinical functioning assessments with the data obtained in the indicators for successful graduates, patients who leave the programs against medical advice, subpopulation outcomes for graduates and AMA discharges, and recidivism rates should demonstrate differential rates of program model effectiveness for patients who enter the programs with relatively high or low functioning levels as determined on the LOFA (See Chapter 3). High and low functional levels could not be defined in advance of the study as they were only relative to the patients within the two study programs.

2. Both study program models have a relative economic value in the delivery of program services to patients. It was hypothesized that though an integrated program has higher overall staffing and operating costs, it also has higher levels of cost efficiency. Indicator 7 compares the cost efficiency rates of both programs in this investigation. This hypothesis predicts that the results of the efficiency rate comparisons will demonstrate differential economic value for both models.

3. Indicator 6 measured patient satisfaction within each program. A comparison of the results of patient satisfaction surveys within and between each program will demonstrate differential rates of patient satisfaction. Due to the less restrictive nature of the integrated program model, it was hypothesized that overall, patients would report higher rates of satisfaction with an integrated treatment model.

Additional Research Questions

1. Within this study there was an underlying assumption that each program model possesses relative value and worth for the effective treatment of MICA patients that can be determined through an analysis of the indicator data within and between the program models. Such statistics and qualitative data should not only point the way toward improving the quality of care in the study programs but could also be generalizable to the two therapeutic models as well.

2. The innovative use of both standard and mutually agreed upon local indicators can provide sufficient data to determine the relative worth and value of not only individual programs but their treatment models as well. An analysis of all the indicators used in this study should at least point toward the advantages of using both top-down (standard indicators) and bottom-up (local indicators) to determine relative value and clinical worth of mental health programs.


5. METHOD
The primary goal of this investigation was to examine the relative treatment outcomes for these two main program models and determine the clinical and programmatic impact of each model on dually diagnosed MICA patients. Such results enable program planners to create or modify existing programs to more effectively treat their target dual diagnosis populations. This investigation utilized a responsive evaluation methodology. Indicators were therefore chosen to not only produce quantified data on the program model's effectiveness, but to introduce a quality improvement mechanism to the two study programs in Honduras, Central America.

Study Design

This investigation evaluated patient outcomes for two treatment programs that treat homeless or marginally housed, mentally ill, chemical abusing late adolescents and adult males (16-62 yrs.), who entered the programs between January 10, 1997 and June 13, 1997. Clinical and programmatic outcomes in Santa Rosita, a traditional disease-specific, drug and alcohol treatment program, were compared with outcome data from Proyecto Victoria, a comprehensive, community-based rehabilitation program that utilizes an integrated program model. Since both programs had goals of rehabilitating MICA patients and promoting long-term, drug-free community tenure have almost identical staffing patterns, with clinical staff who are comparably trained and are located in the same geographic area, a comparison of program outcomes, based on standard indicators, could be performed to determine the relative value of integrated and disease-specific treatment models to treat these patients. Use of local indicators provided for additional measures of program effectiveness that were either specific to that program or applied as an additional comparative measure between the two programs. Because all local indicators were determined by the stakeholder committee (patients, administrators, families, community members, and funding sources) within their cultural context, they also provided measures that were culturally specific to the program, community, region, and country that the program and patients reside in. The local indicators were therefore free of the cultural bias that would occur if only standard, outside measures of effectiveness were applied.

The basic design of this investigation can be classified as a quasi-experimental two- way non-randomized control group design (Campbell & Stanley, 1963). The advantage of using this particular design in this investigation is that it allows for the comparison of the pretest and posttest measures of change (LOFA score changes), due to the treatment that was delivered. No control group was used in this investigation because the two treatment groups were roughly equivalent, with only the type of treatment varying between the two groups.

Description of Treatment Programs

Santa Rosita
The Santa Rosita alcoholism treatment program is a disease-specific, 3 month treatment program, located in a rural setting 27 kilometers north of Tegucigalpa, Honduras on the road leading to San Pedro Sula. Within Santa Rosita, the alcoholism treatment program serves all patients who present with a substance abuse diagnosis or who are referred by families, clinics, and other tertiary care hospitals in Honduras for acute and chronic substance abuse. This includes patients diagnosed with concomitant psychiatric disorders and those who only diagnosed with substance abuse/addiction and associated symptomatology. This program has been in operation for 15 years and is operated by the Honduran Ministry of Health as a governmental program .

The Santa Rosita alcoholism program is located in a segregated section of the Santa Rosita Hospital campus that occupies an area of 2 square acres. The entire hospital is fenced in by chain-link fence with razor wire to prevent patients from eloping from the facility. Guards are posted at all gateways. Patients admitted to the program are isolated from the general population of chronic patients for their entire stay at the program. This ensures that no substances will be brought into the program and that patients can concentrate on their treatment plans, without the influence of nonprogram patients that could interfere with their treatment. Patients within this program are housed in a large dormitory style ward broken into three main areas. Access to and from the program area and dormitory is monitored by the nursing station, located at the main entrance to the program area. Adjoining the program is a large garden (maintained by the patients) with tables, benches, and an area for group activities. A large treatment room for group therapy also adjoins the dormitory area.

This program uses a five-phase graduated model of treatment that is functionally collapsed into three main phases. Patients who enter in an intoxicated state are admitted as Level 1 patients. These patients are restricted to an isolated area until they are free of all detoxification symptoms and are graduated to Level 2. Level 2 and Level 3 patients are housed in the next section of the dormitory and are considered to be actively working on not only residual detoxification issues but beginning their long-term rehabilitative process. Finally, Levels 4 and 5 patients sleep in the furthest section away from the nursing station. These patients have been in the program the longest and are preparing for their rehabilitation and reintegration back into the community. The final phase, Level 5 patients, concentrates on the gains made in treatment and prepares patients for continuing their treatment in an outpatient setting. All graduating patients are not only referred to outpatient facilities to continue their psychiatric rehabilitation, but Alcoholics Anonymous as well.

Treatment plans are generally generic in nature. Almost all patients received the same level and type of treatment within the Santa Rosita program. This consists of at least 1 therapy hour daily with a psychiatrist, a weekly session with a psychologist, weekly sessions with a social worker and 2-3 hours of group therapy for substance abuse, occupational therapy, and a daily Alcoholics Anonymous meeting.

The Santa Rosita program is staffed with round the clock nursing teams of an RN and a nursing assistant. 2 full time Psychiatrists, a part time (6 hour weekly) night Psychiatrist, a 1/2 time Occupationall therapist, an AA facilitator (6 hours weekly), a volunteer spiritual counselor (1 hour on Saturdays) and 6 hours of individual psychotherapy sessions from a Psychologist weekly. Though team meetings for treatment planning do occur, they are not regularly scheduled as the individual disciplines and departments direct the activities of the Psychologist, Social worker, and Occupationall therapist, in a very traditional medical model. The focus of this treatment program appears to be primarily on substance abuse and psychosocial rehabilitation. Appendix A provides additional information about the nature and scope of the Santa Rosita treatment program.

Proyecto Victoria
Proyecto Victoria is an integrated treatment program, located in a rural area 20 kilometers from Tegucigalpa Honduras and 5 kilometers on the road to Olancho. In Proyecto Victoria, the treatment program serves all patients who present with a substance abuse diagnosis or who are referred by families, clinics, and other tertiary care hospitals in Honduras for acute and chronic substance abuse. As with Santa Rosita, this also includes patients diagnosed with concomitant psychiatric disorders and associated symptomatology. The treatment program has been operational since 1981 and is owned and operated independently as a nongovernmentalt organization.

Proyecto Victoria is situated in a rural setting that occupies 75 acres on the side of a mountain. The campus is large and fenced in by a chain link fence. Access is only monitored at the front gates by a single guard. Within the program, patients and staff live and work in housing that was either constructed on site by patients and volunteers using local funds or constructed by contractors using funds donated as part of an international grant to the program. In each small house of the patient residential area, two to three patients are assigned to live together in a group living environment. Housing and compliance with housing rules and regulations are monitored by mental health worker staff. Individual and group treatment occurs in the program operations area of the program campus and is individually tailored to meet the treatment needs of each patient. Since the program receives no support from governmental sources, it makes the most of its resources by growing its own food, raising livestock, and selling excess beans and other products in the local markets.

In their rural setting, with planted fields and concentration on self-sufficiency within the program, Proyecto Victoria has a milieu of farm life and pastoral calm that is very different from the hospital and clinical environment associated with more traditional programs.

This program also uses a five- phase treatment model that is also functionally collapsed into three phases. As patients progress through this 3 month program, they gain progressive responsibility and liberties at each phase. Upon graduation, all patients are sent to a sister program in San Pedro Sula, where they continue their recovery from mental illness and substance abuse in an outpatient treatment program. Connection to AA is a requirement of all graduating patients. In the past, some of these graduates have returned to work as mental health workers within the Proyecto Victoria program.

With this program model, treatment is customized to meet the needs of each patient. Patients with more severe emotional or psychotic difficulties are provided more psychotherapy than group therapy, while those who primarily suffer from substance abuse receive more group therapy, re-socialization and occupational therapy. All patients are provided at least some individual psychotherapy, socialization skills training, spiritual therapy, occupational therapy, and substance abuse counseling. However, this treatment mix tends to vary according to the needs of the patients. All patients are expected to work on the farm as part of their occupational therapy and to offset the cost of their stay within the programs.

The Proyecto Victoria program is staffed with a round-the-clock substance abuse/mental health worker teams who live at the facility. These are trained staff equivalent to nursing assistants in the United States and provide the monitoring and facilitation of activities for patients, 7 days a week. The clinical program is run as a team, with weekly team meetings to discuss the progress and level of all patients within the program. Treatment plans are discussed as a team and actions and recommendations for patient treatment are based on the patient's individual neEd. Clinical staff consists of one psychiatric social worker, a (half time) psychologist, a physician (half time), and seven mental health workers. Other activities, such as occupational therapy, recreational therapy, and spiritual guidance and therapy are provided within the team environment by program staff who received their training on the job and not in a formal setting. There were four such individuals at the time of the field visit to Proyecto Victoria. The emphasis of this program is full rehabilitation and not just abstinence from all substances. The treatment program includes a full range of services including the following: psychiatric, social, spiritual, and occupational rehabilitation. Appendix B provides additional information about the treatment program of Proyecto Victoria.

Subject Selection

The patients in both programs under study were selected on the basis of their meeting the diagnostic admission criteria for the programs. Patients who were male, ambulatory, and not in need of acute care, diagnosed with a major Axis I disorder and/or an Axis II diagnosis, and had a significant history of substance abuse were admitted directly from street and hospital referral sources of Honduras on a space available basis. Though the two study programs also admitted patients without the additional mental illness diagnosis, only the MICA patients were included in this investigation. All participating patients were required to read and sign a consent form to participate in this investigation (Appendix C). Those who refused to give such consent were not subjected to the ratings or satisfaction instruments and were not be counted in the measures for either successful or unsuccessful outcomes. Such patients were, however, reported as an aggregate number who refused to participate in the study.

The degree of subject similarity was confirmed through the use of categorical logistic regression analysis. This procedure was performed on all subject characteristics within the categories of the two study programs. Logistic regression determined whether variables, such as patient age, sex, race, prison history, work history, treatment history, diagnosis, functional level at the onset of treatment, and severity, duration, and type of substance abuse significantly differed between the programs at the onset of treatment. Finally, with a similar MICA population and identical program goals of rehabilitation to the functional level necessary for successful community placement, the relative success of each treatment model could be compared and a comparative success rate determined.

Sample Size

The integrated program, Proyecto Victoria contributed the data on 44 patients for this investigation. This included 40 patients who participated in the study and 4 who refused to participate. The disease-specific program, Santa Rosita enrolled 49 patients during the study period, with 46 participating and 3 refusing to participate. This brought the total sample size of participating patients to 86 patients. Since this study used a wide variety of statistical tests on this sample size, the possibility existed for both Type I and Type II errors. The false positives that contribute to Type I errors of finding a difference that is not there, and small effect size results that may lead to Type II errors of not finding a positive result that is there were addressed by advanced statistical analysis and testing (see Data Analysis).

Patient Characteristics

The programs' goals of patient rehabilitation and functional community assimilation of mentally ill chemical abusers within a 3 month time frame were identical for each program. In addition, both groups required patients to meet the target diagnostic and functional criteria for treatment in the programs and from the programs' point of view were expected to functionally improve over a 3 month period to the point of community readiness. Since this study compared patient outcomes against the program goals, we can consider them similar only in this respect. Both groups were comprised of male MICA patients with similar characteristics from the same geographic location.

Standard Indicators

The two programs will be evaluated by the following standard criteria:

Indicator 1: Successful Treatment Outcomes.

This was determined by results of the Level of Functional Assessment scale (LOFA), a 53 item scale that quantifies functioning across independent areas of social, cognitive and physical skills (Uehara et al., 1994) (Appendix D; see Instruments). The independent, paid raters were two Honduran psychiatrists, one clinical psychologist with Master’s of Arts degrees in psychology, and one social worker with an MSW. All raters had at least 5 years of clinical experience with substance abuse and chronic mental illness in Honduras. These raters evaluated each patient’s functional level and addiction severity within 2 days of the patient’s arrival to the programs and again as patients left the programs to be placed in their community-based settings. Level and type of addiction, as determined by the LOFA, were noted along with other diagnostic and demographic data on the Patient Information Sheet (Appendix E). These raters were also responsible for the collection of all demographic and clinical data on the Patient Information Summary Sheet. In addition, the raters participated in the same training for functional assessments. This training consisted of one 8-hour instructional seminar in the use of the LOFA and Clinical Data Sheets and 8 practice assessments performed on the same patient samples. Practice assessments continued until raters achieved interrater reliability of at least r=.80. Interrater reliability to this level was reconfirmed at the midpoint of the study to ensure that interrater agreement was being maintained and there was no rater 'drift'. Patient evaluations at the time of program entry and upon discharge detailed the specific changes in their functional level due to their treatment. Successful completion and discharge from both programs required patients to have reached a functional level of at least GAF=70, as determined by the LOFA. This level corresponded to the rehabilitative discharge criterion for both treatment programs and implied that patients would be discharged with low levels of residual symptoms but at least functionally able to live independently in the community. Only those patients who met this rehabilitation goal were counted as meeting their program's objectives for successful treatment outcomes. Patients who failed to reach this level of rehabilitation remained in the programs until they had reached this functional goal. These patients were not counted in the Indicator 1 statistics for successful program completion because the additional treatment was not consistent with the time frame goals of the programs.
Indicator 2: AMA Discharge.
Within the context of this evaluation, the percentage of patients who did not complete treatment and who left the programs against medical advice were compared across programs as a measure of the treatment program's inability to meet the needs of the target population. Since both program models had clinical failures of this type, the percentage of patients who left each program against medical advice was included as AMA Discharge measures of negative clinical outcomes. Patients discharged for cause due to substance abuse relapse or violence behavior could also have been included in this measure. However, during this investigation there were no discharges for cause in either program. Occurrences of significant relapse that could lead to discharge is rare in these programs due to the following reasons: -The traditional drug/alcohol program is a voluntary lock down program with one scheduled urine analysis (UA) weekly and one random UA. Though all patients are free to leave the programs at any time, their departure is automatically considered an AMA discharge. Their participation is voluntary in that they agree to enter and remain in the highly supervised program during their course of treatment.

-In the integrated program, drug/alcohol use is strongly discouraged, UAs are done on suspect abusers and progressive levels of loss of privileges and other disincentives are used to discourage drug and alcohol use. In the integrated program, every effort is made to rehabilitate patients across all psychosocial areas, including substance abuse. When this effort fails, the patients generally leave on their own because they cannot tolerate the loss of privileges or other negative sanctions that curtailed their activities.

Indicator 3: Service Hours Delivered.

A correlation between actual number of service hours delivered and the percentage of successful treatment outcomes per program was performed to evaluate whether variable amounts of treatment affect overall program outcomes and success. Total service hours delivered to each patient in a program was correlated with the relative percentage of successful outcomes and LOFA subscale improvement scores. This demonstrates the relationship between amount of services delivered and successful outcomes.

Data collection methods were as follows:

Clinical contact hours were noted weekly from the clinical contact notes in each patient’s chart. This information was recorded on patient contact summary forms (Appendix F) weekly. Type and duration of services that were provided during that week were summarized and tallied on this form for each patient. Each patient contact hour is considered a service unit. These service units were summed for each patient at the completion of the study and then compared with patient functional level or overall outcome in the program in correlations.
Indicator 4: Subpopulation Outcomes.
Differential success rates between diagnostic subpopulations (i.e., schizophrenic, substance abusers versus mood disordered, substance abusers) were also analyzed to determine whether the integrated and disease-specific models were best suited to one subpopulation or another. This measure was also applied to the criteria indicators listed above.
Indicator 5: Relative Rates of Substance Abuse Recidivism
Percentage rates of patient recidivism for program graduates were also determined as a measure of the programs' relative inability to effectively rehabilitate their target MICA population. Recidivism within the context of this study was defined as any patient who left the program and resumed substance abuse activities that resulted in at least two episodes of intoxication during a single week. Though this data is also included in the total AMA discharge data of Indicator 2, substance abuse recidivism data provides a refined comparative measure of the programs’ ability to effectively address the substance abuse issues. Substance abuse recidivism data were collected by the independent LOFA raters in case follow-up investigations of all patients who left the programs against medical advice. Raters interviewed both the patient and family members to determine the degree and pattern of substance use or abuse after the patient left the program. Substance abuse recidivism, in the context of this study, was defined as a return to substance use that resulted in patient intoxication. Determination of intoxication was based on the clinical observation of the paid raters and the behavioral reports of patients or family members. The paid raters assessed recidivism by contacting the patient and his family within 1 month of discharge.
Indicator 6 : Patient Satisfaction.
Patients in both groups completed a Spanish version of the New York State Office of Mental Health (NYSOMH) standard patient satisfaction survey (see Instruments) at the 3 month point in their program or upon successful completion and discharge from the program (Appendix G). This provided a measure of the patients' qualitative level of satisfaction within each program model. Repetitive complaints in each program were also noted as a qualitative component of this indicator.
Indicator 7: Cost Efficiency Rate.
Relative cost per service unit was determined for patients who successfully meet the goals of the program (see Indicator 1). This measure was included to illustrate differential program costs for those patients who met program goals and was considered a gross measure of program efficiency within the context of this study. The cost figures were determined by dividing the total annual operating costs (personnel, medication, and supply costs) by the sum of the service hours for patients who met the 'successful outcome' criteria. Program development costs, capital improvements, and other non-operating costs were excluded from this analysis. Cost-benefit and cost-effectiveness analyses were not be performed, since the necessary variables for such analysis are beyond the scope of this investigation.
Local Indicators

The two programs were also asked to form a committee, which included all the stakeholders that may have an interest in each program. This included the governmental monitoring agencies, funding sources, program directors, clinical staff, administrators, the primary investigator/evaluator and patient representatives. The stakeholder committee was asked to select at least three indicators that reflect or have an impact on either the effectiveness or quality of treatment. Both programs met on several occasions to identify local indicators and each program chose indicators that either impacted the quality of care in their facilities or indicators that would help them refine their patient recruitment efforts.

Once determined, these indicators were monitored, analyzed by the committees, and data reported to the primary investigator in addition to the standard indicators. After reporting these results, these two committees developed action plans to address deficit areas determined in either the standard or local indicators for their particular programs. These committees met monthly and developed action plans to either address problematic areas determined through local indicator data measures and/or develop new action plans for the next monthly interval. The local indicators selected by the stakeholder committees are listed as follows:

Santa Rosita-(Disease-Specific Model) Local Indicators

Indicator 8: Number of patients who entered outpatient treatment-Santa Rosita
The actual number of patients who not only completed treatment, but entered AA or other outpatient treatments settings were summed and presented as a percentage of all patients who graduated from Santa Rosita.
Indicator 9: Improvement in access road conditions at Santa Rosita:
The 6 kilometer access road to the hospital from the main highway is a dirt road that provides the only access for staff, patients and families to the hospital. As such, it exists in various states of repair during the wet and dry seasons, depending on government resources to maintain the road. Since access by staff and families to the hospital is critical to effective patient care, this infrastructure indicator was chosen to monitor and improve this critical access way. For this indicator, road conditions were noted by the committee monthly and reported to hospital administration for further action.
Indicator 10: Transportation-Santa Rosita:
Since Santa Rosita is located in a rural area far from the capital, adequate ambulance and bus service is essential to effective patient care. Access for staff, families, and patients depends on the three bus stops that occur daily. Patients, family, and staff who miss these buses must walk the 6-kilometer dirt road to the main highway in order to secure transportation back to the city. The ambulance provided for this hospital stopped working prior to this fiscal year. Thus, there was no method of transporting patients with questionable stability. This infrastructure indicator monitored the transportation situation of Santa Rosita and reported the needs of the program to hospital administration and the government on a monthly basis.
Proyecto Victoria-(Integrated Model) Local Indicators
Indicator 11: Family history of substance abuse-Proyecto Victoria.
All patients who entered the program were surveyed on whether there were others in their immediate family with a history of substance abuse to the point of addiction or dependency. This number was then correlated with patient success in the programs (Indicators 1 and 2) to determine the degree to which family history of abuse predicts success or failure in this drug treatment program. This indicator was chosen to provide information on patient characteristics to refine their future staffing patterns and patient recruitment efforts. The goal of this was to ultimately provide enhanced quality of care to their patients and their families.
Indicator 12: Duration of Substance Abuse-Proyecto Victoria.
Upon admission to the program, all patients were asked how many years they have been abusing substances. This information was then correlated with success indicators within the program, summed, and presented as percentage figures. The goal of this indicator was to help determine which patient characteristics represent the model patient for this program, so that patient recruitment efforts can target that type of patient in the future.
Indicator 13: Type of substance abused.
The admissions history also gathered data on the primary and secondary types of substances abused by patients. This provided the clinical team with information that could be used to modify treatment plans to more effectively address the issues associated with recovery from each type of substance. As with Indicator 12, this indicator was also used to help determine a realistic patient model for future program planning and recruitment efforts.
Instruments

1. Level of Functional Assessment. (LOFA). This scale contains 48 items associated with specific level of functioning, covering nine specific domains. Each item is evaluated on a five-point Likert-like scale. Domains are grouped into the three broad categories of social, psychological, and physical functioning. Due to the specific nature of each of the 48 functional items, the LOFA assumes a high degree of assessor familiarity with the patient.

This scale was selected for this study because of its ability to quantify patient functional levels more systematically than the Global Assessment of Functioning (GAF) and it provides for the systematic rating of functional deficits in the areas of community living, interpersonal relations, mood, psychotic symptoms, substance abuse, and physical functioning that could not otherwise be assessed. In addition, LOFA scores can be readily converted to GAF scores due to its high correlation to the GAF in the original trials (Drake et al., 1990b). Patient scores on each LOFA component are significantly correlated with GAF scores (r=.67 for the social component, .60 for the psychological, and .50 for the physical component). Thus, patients’ specific functional levels can be determined through the use of the LOFA. Moderate associations have been found between the LOFA substance abuse scale and the Drake et al. (1990) substance abuse scale (r=.73) (Uehara et al. 1994). The combined functional assessment of both substance abuse and general functioning make the LOFA an ideal instrument to assess MICA patients. The paid raters in this investigation maintained an interrater reliability of at least r=.80 throughout the study. This was determined after the practice/training sessions at the beginning of the study and reassessed at the mid-point of the investigation. Initially, the paid raters were asked to rate patients in 10 practice sessions. Interrater reliability was then analyzed on the basis of 3 additional assessment sessions in which all raters assessed the same 3 patients. Mid-point reliability was determined on the basis of another set of assessments by all raters on three additional patients.

2. New York State Office of Mental Health (NYSOMH) Satisfaction Survey: This consumer satisfaction survey was developed by the State of New York to help programs comply with the parts of the NYSOMH 1990 certification regulations that mandate determinations of consumer satisfaction. This survey consists of 19 'fill in the blank' qualitative questions about the program, staff, and treatment. There are also three objective questions that use a 10 point Likert scale to rating to assess satisfaction with the program, staff, and facilities (NYSOMH, 1990). These instrument were scored by taking the percentage of positive and negative responses for the 19 subjective questions and the sum of the quantified answers from the three objective questions. For the 19 subjective questions, those responses that included positive adjectives (e.g., effective, beneficial, good, etc.) were scored as positive responses, and responses that included negative adjectives (e.g., worthless, poor, etc.) were scored as negative responses. Blank responses and neutral adjectives(e.g., OK, fair, etc.) were not counted as either positive or negative. This data was aggregated by program and served as the assessment of overall patient satisfaction level with the programs.

3. In addition to using a set of standard indicators that can be applied to almost any health care program, both programs were also asked to determine at least three local indicators of successful treatment that are particular to their programs. This information was of greatest value to the quality improvement component of this study. As information became available on both sets of indicators (standard and local), the major stakeholders of each program met and agreed on the remedial actions necessary to improve the quality of care and improve future performance. Continuing this process beyond the 6 month scope of this study resulted in a trend toward continuous improvement of care on a local programmatic level. Data collection for these indicators was noted on the patient information sheets.

Translation of Instruments and Consent Forms

The patient survey, consent, LOFA, and all patient data forms were subjected to a two-way translation method to ensure cultural sensitivity and meaning integrity through the translation from English to Spanish (Appendixes A-E). Translations of all forms from English to Spanish were independently translated back to English. This was done to ensure that the meaning was not skewed in the translation process and that cultural norms were taken into account for the translated documents. All translations were performed by the Peace Corps Language Department in the Peace Corps Honduras country office, under the direction of the principal investigator. The principal investigator was rated at a technical language fluency level in Spanish by the U.S. State Department-Honduras at one step below native speaker level.

Data Analysis

This investigation used a wide variety of statistical tests on a sample size of 86 patients. Thus, there was the possibility of both Type I errors and Type II errors. To address this issue, all results were subjected to statistical tests to ensure significance power analysis using Cohen's definitions for small, medium and large effects (Cohen, 1988). Only medium (d>.49) and large effects (d>.79) were reported as significant findings in this study. All data was analyzed using SPSS-PC, version 6.0 and SAS, version 6.12 for PC. A sample of the raw data and analysis for these patient outcomes is summarized in Appendix H.

Subject characteristics, such as age, educational level, years of work, diagnosis, treatment history, and GAF/substance abuse severity levels were analyzed within and between programs. This analysis was performed by an examination of the descriptive statistics, chi-squared, independent samples t-tests, and Levene’s Test for Equality of Variances. Correlational analysis and logistic regression of patient characteristics by program group were also used to determine the degree to which the program groups covaried. All correlations were subjected to Fisher’s Transformation of r to Z for comparative analysis. Determining the degree of covariance was necessary to establish the degree of similarity between the two groups. Regression analysis for all significant differences found among the two program groups was performed to determine whether the differential indicator results and study effects remained after the covariate had been partialled out of the analysis. Power analysis was performed on results of all indicators.

Indicators were analyzed as follows:

1. Indicators 1, 2, and 5: Successful Patient Outcomes, degree of change in LOFA/GAF scores, AMA Discharge and Substance Abuse Recidivism. Determination of successful outcomes by program were analyzed by chi-squared, independent samples t-tests and categorical logistic regression. Bivariate correlational analysis was also used on the patient characteristics and outcome results of these indicators to determine what patient characteristics best predict clinical success or failure by program type. Though some of the patient characteristics are presented as percentages, degree and significance of covariance were determined through the use of the above mentioned analytic procedures on the raw data set.

2. Indicators 3 and 4: Correlation of Service Hours Delivered and Subpopulation Outcomes. Multiple bivariate correlations of service hours to outcomes and subpopulation differences to outcomes with tests for significance were performed to determine whether the amount of clinical contact or subpopulation correlated with outcomes in the programs. Chi-squared and Wilcoxon Matched-Pairs Signed-Ranks tests were also applied to establish covariance and significance. Fisher’s r-to-Z transformation was performed to permit the analysis of significant differences between the correlations.

3. Indicator 6: Patient Satisfaction. Percentages of positive and negative responses were tallied by group. The raw data was also tested by chi-squared for variance and significance. For the subjective questions on the patient satisfaction survey, use of positive adjectives were counted as positive responses for that item, negative adjectives counted as a negative response, and neutral or blank responses were not counted. The total number of positive and negative responses were summed for each patient and then summed as a group by program. The three objective (Likert-scaled) items in this questionnaire were included in the summation process for each patient and program.

4. Indicator 7: Cost Efficiency. Total annual operating cost was divided by the sum of the service hours delivered to patients who had successful program outcomes in both programs.


6. LIMITATIONS AND DELIMITATIONS

Limitations
Limitations are limiting factors inherent in the study that should be clarified and made explicit. Within this investigation, there were three main limitations that are relevant to this and other evaluation studies: (a) evaluator and stakeholder bias; (b) generalizability of results; (c) equality of groups.

a. Evaluator and stakeholder bias. In program evaluation studies, a sole evaluator method contributes to managerialism and a failure to accommodate value pluralism. As such, the points of view of all program participants and stakeholders are not taken into account and can account for a skewing of the results in the direction of the evaluator’s own beliefs and assumptions (Fetterman et al., 1996, Guba & Lincoln, 1989). Formation of a committee, consisting of all stakeholders, who select and measure the local indicators, reduces such bias and leads to study results that more accurately illustrate program effectiveness. However, since all stakeholders have an active interest in the positive study outcomes for their programs, they may overtly or covertly choose and measure indicators that shed a more positive light on their program. For this reason objective scales for GAF and objective measures of patient progress in Indicators 1 through 6 were chosen by the principal investigator. These standardized objective measures reduced the amount of subjective bias on the part of the stakeholders while still giving them a voice in the investigation.

b. Generalizability of Results. The degree to which the findings of a responsive evaluation study can be generalized to other programs and models has been disputed (Borg & Gall, 1989; Weiss, 1972). Since all treatment programs exist within a unique environment, have different staffing patterns, and interpret and implement therapeutic models idiosyncratically, it is difficult , if not impossible, to generalize the results of one evaluation study to other, even similar programs with 100% accuracy. However, by structuring the study to focus on two programs that have similar staffing patterns, with similarly trained professionals, who treat a similar population, the therapeutic models could be compared and contrasted for these two programs. The principal investigator made no assumptions about whether the findings and results could be generalized to other programs and models. In order to generalize the findings of evaluation studies to other similar programs, the number of study programs treating a similar MICA population would have to be increased to a much larger number to reach valid and significant conclusions about their treatment models. However, the high level of financial and human resources required to perform a study that could reach this level of generalizablity far exceeds that of this study and was therefore beyond the scope of this investigation.

c. Equality of Groups. One of the purposes of this study was to compare two programs with roughly equivalent patient populations that differ only in their approach to treatment. However, finding full congruency between groups in natural settings may not be possible within the context of this type of investigation. Thus, significant diagnostic and demographic differences found among the two program groups were subjected to an ANCOVA to determine whether the effect sizes and significance of the outcomes remained after the significant variables had been partialled out of the analysis. In addition, logistic regression was performed to examine the cojoint effects of the large number of variables, both with and without the differential variable(s). This provided additional information as to the effects and significance of the differential variables between the groups on the outcome indicators.

Delimitations

This study was designed to compare the therapeutic models of two treatment programs for MICA patients. As such, two major restrictions were made to facilitate a more accurate impact assessment of the two treatment models under study: (a) selection of a cost efficiency indicator, (b) selection of patient performance instead of effort indicators as measures program effectiveness.

a. Selection of cost efficiency indicator. Cost efficiency was chosen as an indicator because of its ability to demonstrate relative economic success within each program. Unlike cost-effectiveness, the cost-efficiency indicator gives an objective rate of cost for patients who successfully complete the programs and can be compared across programs. In economic terms, cost-effectiveness analysis would have required longitudinal measures of costs and economic benefits to the participating patients, their families, their communities, and the society as a whole (Seitz, 1979). Such analysis was beyond the scope of this study.

b. Selection of performance over effort measures. Suchman (1967) advocated the measurement of patient effort in substance abuse programs. He argued that performance measures on tests and behavioral assessments cannot adequately measure patient gains in therapeutic programs. Indicators of patient effort and program participation were advocated as a valid approach to assessing patient outcomes in programs. Within this study, however, only patient participation hours (service hours) were collected and correlated to performance. In this sense, the effort measures (service hours) are actually used as a comparative measure to validate the performance measures in Indicators 1,2, 4, and 5 (see Indicator 3). Within the context of this study, measures of effort alone were insufficient to compare relative rates of patient and programmatic progress/effectiveness.


7. RESULTS

Comparisons of Patient Characteristics:
Tables 5 and 6 detail the results of analysis of equality and covariance. Significant between group differences were only found in 5 of the 33 characteristics selected for comparison. On average, Proyecto Victoria patients tended to be 7 years younger, had 19.5% more convictions, were 80% less alcohol dependent, stayed 15 more days in the program, had greater numbers of suicide attempts, and were homeless at more than double the rate of those in Santa Rosita. However, these differences appear to be more directly related to programmatic factors, than sample differences in the population. Patients who were being considered for admission to the programs had the option of rejecting admission, even when accepted to the programs. Thus, the entire referral pool of patients more accurately represents this population of MICA patients because those who deselect themselves from the programs or who choose one program over another were included and can be compared between the programs. When the entire pool of patient referrals to each program was compared to the pool of admitted patients, the significant differences disappeared. In addition, all patient characteristics and instrument scores were subjected to logistic categorical regression and Cohen's power analysis. All significant differences that remained are reported in these results and can be attributed to the program recruitment and selection process at the time of admissions.

Though these characteristic variances are statistically significant, the variation between the study groups is insufficiently large to account for the various patient and differential clinical outcomes. For example, the outcome differences between the two programs on the LOFA scale and subscales were statistically unaffected by differences in population or characteristics. The differences in clinical impact that were found between the programs remained significant after being subjected to logistic regression analysis and were found to not be related to the differences in characteristics between the groups. Hence, all outcome differences are related to the clinical models that were used on the patients who agreed to remain in the programs and actually graduated from the programs.

Standard Indicator Results

Indicators 1 and 2:
Results for these indicators are demonstrated in Figure 5, Figure 6 and Table 7. There were no significant differences in Graduate and AMA rates found between the two programs. The percentage of graduates for Santa Rosita was 93.7% and 82.7% for Santa Rosita, while AMA discharge percentages were 6.3% for Santa Rosita and 17.3% for Proyecto Victoria. The differential rates between the programs was not found to be significant in independent samples t-test results. Both programs retained almost all of their patients to the point of graduation and far exceeded the performance of the programs evaluated in the pilot study. This may be a due to cultural differences as other indicators (patient satisfaction) appear to show this same trend (See Discussion). There were, however, significant differences found between the programs on the LOFA subscales.

 
Table 5 
Patient Characteristics by Honduran Program
SR
PV
All Subjects
N=46
N=40
N=86
Diagnosis - Primary Axis I
Psychotic Spectrum
38.5
25.0
31.8
Mood Spectrum
57.4
73.1
65.3
Other
4.1
1.9
3.0
Diagnosis - Secondary Axis 1
Intoxication (all substances)
72.3
65.1
68.7
Polysubstance Abuse
64.1
60.2
62.2
Cannabis Dependency
15.0
25.0
20.0
Alcohol Dependency
100.0
*
21.0
*
60.5
Cocaine Dependency
1.2
*
47.5
*
24.3
Diagnosis - Axis II
Personality Disorders
3.9
3.3
3.6
Medications
Neuroleptic
66.2
*
42.1
54.2
Antidepressive
43.0
38.0
40.5
Anxioletic
24.0
10.2
17.1
Lithium
0.0
1.2
0.6
Anticonvulsive
1.2
1.2
1.2
None
0.0
12.9
6.5
Ethnicity
Mestizo
97.9
80.0
89.0
Indian
0.0
5.0
2.5
Hispanic
0.0
10.0
*
5.0
Black
2.1
5.0
3.6
Marital Status
Single
50.0
71.8
60.9
Married
21.7
20.5
21.1
Separated/Divorced
6.5
5.2
5.9
Common Law Marriage
11.7
2.1
6.9
Work
Full Time
56.5
*
12.8
*
34.7
Part Time
28.3
17.7
18.0
Unemployed
15.2
*
79.5
*
47.4
Housing
Streets
4.3
7.0
5.7
Unstable Housing
2.2
1.2
1.7
Permanent Res
32.6
2.3
*
17.5
With Family
56.5
88.3
72.3
With Friends
4.4
1.2
2.8
Prison History
13.0
32.5
*
22.8
Note: The All Referrals Column represents the patient characteristics of all patients referred to the two programs from all sources. * = p< .05, two tailed,for significant differences in between group comparisons of percentages.
 
Table 6 
Equality of Patient Characteristics Between Programs 
Santa Rosita Proyecto Victoria
N = 46 N = 40
mean
s.d.
mean
s.d.
mean
difference
F
Age
39.50 
9.98 
32.23 
14.02 
7.285 
6.334 
a
Education
7.37 
4.12 
7.33 
3.32 
0.045 
1.552 
Months Employed
21.22 
49.62 
12.45 
16.41 
8.767 
3.766 
b
Previous Hospitalizations
2.65 
2.33 
0.80 
1.11 
1.852 
7.879 
a
Suicide Attempts
0.22 
0.70 
0.68 
1.10 
-0.458 
11.262 
a
Number of Convictions
0.09 
0.35 
0.75 
1.81 
-0.663 
14.278 
a
Months of Homelessness
0.43 
1.39 
1.51 
4.82 
-1.080 
7.545 
b
Treatment Hours Daily
5.10 
0.65 
3.98 
1.47 
0.887 
2.852 
 Note: F= Levene's Test for Equality of Variances. Within each row comparison, significant differences (p < .05, two tailed)
 are indicated by an 'a' at the end of the row. A 'b' at the end of the row indicates only a marginal effect (p < .10, two tailed).
 Confirmation of variances and significance was performed by logistic categorical regression for all characteristics and
 clinical outcomes.
Santa Rosita's disease-specific program demonstrated greater gains in the reduction of psychotic symptoms, reduction of symptoms associated with mental illness, improved community life scores,and a greater overall reduction in dangerous behavior than Proyecto Victoria.

This may be related to Santa Rosita's greater reliance on the use psychotropic medication to control and modify patient behavior (See Table 5). However, this relationship was not specifically tested in this study and is purely speculative. Santa Rosita also significantly exceeded Proyecto Victoria in the overall improvement in GAF scores (Figure 6). However, this may be due to the fact that their patients entered at a much lower level of functioning (12 GAF points lower than those of Proyecto Victoria), were not yet stabilized psychiatrically, and many were in active withdrawal from their addictive substances.

Indicators 3, 4, and 5

Results of multiple bivariate correlations of service hours to outcome, sub-populations differential outcome analysis and relative rates of recidivism are

summarized in Table 8, Figure 7, and Figure 8. Significant correlations were found between improvement in GAF Scores and number of clinical contact hours that patients received in the integrated program at Proyecto Victoria.

 Table 7
 Change in LOFA Subscale Scores

                       Santa Rosita              Proyecto Victoria                SR-PV
LOFA Subscales LOFA

T1

LOFA 

T2

Change LOFA 

T1

LOFA

T2

Change Difference
Community Life
2.5 
4.3 
1.8 
3.1 
4.0 
0.9 
0.9 
*
Interpersonal Relations
2.8 
4.6 
1.8 
3.1 
4.4 
1.3 
0.5 
Mental Illness
2.5 
4.4 
1.9 
3.1 
4.2 
1.1 
0.8 
*
Psychotic Symptoms
2.6 
4.6 
2.0 
4.5 
4.4 
-0.1 
2.1 
*
Substance Abuse
2.8 
3.4 
0.6 
3.6 
4.3 
0.7 
-0.1 
Dangerous Behavior
3.0 
4.6 
1.6 
3.8 
4.6 
0.8 
0.8 
*
Personal Care
4.8 
4.8 
0.0 
4.9 
4.5 
-0.4 
0.4 
Physical Functioning
4.9 
5.0 
0.1 
4.9 
5.0 
0.1 
0.0 
Note: The range of the LOFA scale is 1 to 5. * indicates p < .05, two tailed, determined through Wilcoxon Matched-Pairs Signed-Ranks and Chi-squared tests.
 
This correlation was particularly high, exceeding that of the disease-specific, Santa Rosita program by r=.35. These results suggest that positive treatment outcomes are directly related to the amount of treatment these patients received in the integrated program. However, the positive correlations found in between service hours and Change in GAF for both programs also suggests that both treatment models can effect positive rehabilitative change, provided the patient is motivated to remain in treatment and actively participate in the programs.

No significant differences were found for diagnostic subpopulations for either positive or negative treatment outcomes, nor were significant differences found in the recidivism rates between the two programs. Though recidivism rates appear to vary between the programs, these results were not found to be significant and met the criteria for low statistical power when subjected to Cohen's power analysis.

On the other hand, significant correlations were found between educational level and change in GAF level, education and previous hospitalizations, homelessness and previous hospitalizations, and education and homelessness. These results suggest that patients with less education, more previous hospitalizations, and more time spent in a homeless conditions benefit more from the integrated program of Proyecto Victoria than from Santa Rosita. However, in this investigation, there was no significant correlation between change in GAF score and level of functioning as measured upon admissions to the program. From these results and those of the pilot study, it appears that success in traditional disease-specific programs is more likely for more educated, older patients who have greater hospitalizations and experience with treatment systems,

 
Table 8 
Subpopulation Characteristics Correlations 
Change in GAF Education
GAF Improvement Z Scores  Z-Score Difference Years Z Scores  Z-Score
Difference
Age
Santa Rosita
.39
*
.4118
.2308
Proyecto Victoria
-.18
*
.1820
Services hours (daily)
Santa Rosita
.08
.0802
Proyecto Victoria
.43
*
.4599
.3797
Prev. Hosp. (Sub. Abuse)
Santa Rosita
.33
*
.3428
.2727
Proyecto Victoria
-.07
.0701
Education
Santa Rosita
.40
*
.4236
.3132
Proyecto Victoria
-.11
.1104
Months Homeless (Lifetime)
Santa Rosita
-.41
.4356
.1802
-.29
*
.2986
.2285
Proyecto Victoria
.25
.2554
-.07
.0701
Note: p<.05, one tailed, bivariate correlations of patient characteristics converted to Z scores for analysis and
determination of significant differences. Fisher's r to Z score transformations and corresponding analysis were
used in this analysis.



 
 
 
 
 

while younger patients, with less education and experience with treatment programs tend to respond better and gravitate toward more integrated programs. As in the pilot study, this study suggests that patients need higher levels of cognitive skills to successfully negotiate the disease-specific program and remain until graduation than in an integrated program, where treatment plans are individually geared to the level of the patient.

Indicator 6

Patient Satisfaction Survey results are displayed in Figure 9 and Table 9. This survey was administered to a sample of the subjects in this study due to time constraints, data collection problems and patient refusal to participate. Of the 46 patients in the Santa Rosita program, 40 (88%) completed the patient satisfaction survey. At Proyecto Victoria, 36 (90%) of the entire pool 40 patients completed the survey. This standard NYSOMH (1990) survey consists of 19 fill in the blank questions about various aspects of the program, and three objective response questions that used a 10-point Likert scale, with 10 being the most satisfied. Qualitative responses of a negative nature were listed and aggregated. These results are listed in Table 9.

Of the 88% who responded in the Santa Rosita program and 91% who responded in the Proyecto Victoria program, the results are almost identical between the two program. Overall, there was no difference in the satisfaction levels between the two groups. This may however, be more related to cultural factors (See Chapter 8). On the other hand, the qualitative responses between the two programs demonstrate differences in patient dissatisfaction areas between the two programs. In Proyecto Victoria the patients main concern appears to have been focused on the treatment process issues of mandatory physical exercise and discharge planning, while patients in Santa Rosita were more concerned about infrastructure and physical environment

 
TABLE 9
Qualitative List of Patient's Negative Opinions
No. of 
Main Complaints
Complaints
Proyecto Victoria
Too much exercise/physical therapy
Insufficient discharge planning
Bad food
Inadequate lavatory facilities
No mail allowed, mail monitored
Not enough psychologists for treatment
Santa Rosita
Bad food
29 
Theft of patients' personal property
Dormitory too small
Not enough family treatment
Too cold at night
Too far from the city
 
issues of bad food and theft of patient property. This provided the programs with clear areas for programmatic improvement and concentration for their quality improvement committees.

Indicator 7

Based on an annual operating budget of $143,376 for the Santa Rosita program and $59,300 for the Proyecto Victoria program, the rate of cost-efficiency for successful outcomes at Santa Rosita was computed at $6.66 per service unit, while that of the Proyecto Victoria was found to be $4.86. This difference of $1.80 per service unit is significant and corresponds with that of the pilot study, where the integrated program model costs significantly less. However, in this study the difference appears to be due to the programs cost-efficiency and not a differential number of successful graduates.

On the other hand, these results also show that both programs operate at extremely low funding levels and produce positive clinical impact with very limited resources.

Local Indicator Results

Indicator 8:

Number of patients who enter outpatient treatment - All patients who graduated from Santa Rosita (93% of all patients) enrolled in and attended at least one session of ambulatory care in either AA or mental health outpatient treatment. These results are demonstrated in Figure 5.

Indicator 9:

Improvement in access road conditions at Santa Rosita - After repeated notifications of the road conditions (three notifications to Santa Rosita Hospital administration with copies to the ministry of health) during the study period, no improvement in road conditions actually occurred. In fact, over the last 2 study months, the road had actually deteriorated with the coming of the rainy season.

Indicator 10:

Transportation - Santa Rosita. Despite three notifications to hospital administration and the Ministry of Health, the transportation situation remained the same at Santa Rosita; 3 bus trips daily to and from the hospital and no working ambulance.

Indicator 11:

Family history of substance abuse in the program at Proyecto Victoria is displayed in Figure 10. Though there were no significant correlations between family history of substance abuse and either patient characteristics or outcomes, these results helped this program gain a better understanding of its target population and enabled them to adjust their mix of clinical services to more effectively meet their patients' needs.


 

Indicator 12:

Duration of substance abuse for patients enrolled in Proyecto Victoria is illustrated in Figure 11. Though again there was no statistically significant correlation between length of substance abuse and patient characteristics or clinical outcomes, the finding that the vast majority of patients have abused substances between 6 and 8 years assisted this program in modifying its clinical services to meet these patients’ needs in the future.

Indicator 13:

Primary and secondary types of substances are listed in Table 10 below. Though statistical analysis found no significant correlational relationship between the type of substance abused and degree of patient change in GAF or other indicators of success in the program, it provided the program with valuable information with which to plan future substance abuse treatment and educational program modules .

 
TABLE 10
Primary and Secondary Substances 
Proyecto Victoria
Primary Substances
Frequency
% of patients
Alcohol
12 
30.0 
Cocaine (Crack)
7.5 
Cocaine (Nasal)
20.0 
Cocaine (Freebase)
5.0 
Marijuana
10 
25.0 
Inhalants
7.5 
Secondary Substances
Alcohol
22.5 
Sedatives
2.5 
Cocaine (Crack)
5.0 
Cocaine (Nasal)
5.0 
Cocaine (Injectable)
2.5 
Cocaine (Freebase)
1
2.5 
Marijuana
15 
37.5 
Inhalants
7.5 
Santa Rosita
Primary Substances
Frequency
% of patients
Alcohol
46 
100.0 
Secondary Substances
Barbiturates
4.3 
Sedatives
2.2 
Cocaine (Crack)
2.2 
Marijuana
15 
32.6 

8. DISCUSSION
This study investigated the relative impact of a traditional, disease-specific program model and an integrated program model on the treatment outcomes for MICA patients. Based on the NIMH (1991) guidelines for mental health service research, this study began with the question of what works, for which MICA patients, in two programs that use different therapeutic approaches to address both mental illness and substance abuse. Building on the methodology of the pilot study, the results of this investigation have clinical and programmatic implications for all the stakeholders who were associated with the study programs. These findings have important implications for policy makers, administrators, and clinical program staff.

Though many of the findings of this investigation were significantly diluted by the differences in patient populations that could not have been foreseen in the planning stages of the study, generally the results illustrated in Table 7 point toward specific areas of patient improvement for both programs. These results are consistent with those of the pilot study that was more evenly matched for patient characteristics and therapeutic environment. At Santa Rosita there was a 40% reduction in psychotic symptoms/mental illness and 36% improvement in measures of interpersonal relations and community life. Though less dramatic, the reduction in the measures of mental illness (22%) and improvement in interpersonal relations/community life (22%) were significant. In both programs there was a significant reduction in dangerous behavior. These findings demonstrate specific areas of positive change that occurred while the patients were in these treatment programs. In addition, the correlational chi-squared and regression analysis did not suggest that severely dysfunctional patients could not benefit from MICA treatment. This is also consistent with the findings of the Nuttbrock et al. (1997) study that used similar measures to demonstrate specific areas of patient improvement in treatment programs.

Though the importance of the study results may have been compromised by sample differences, the therapeutic success rates, defined by the change in LOFA subscale and full scale scores, the results of patient satisfaction survey results, high levels of patient improvement, and clinical changes on the LOFA subscales point toward distinct advantages of both programs for the treatment of different types of patient subpopulations. The integrated Proyecto Victoria program model appears to have distinct advantages in the treatment of severe MICA patients, who are younger, more poorly educated, and have many past hospitalizations. In aggregate, these patients tended to respond to the increased service hours and more comprehensive rehabilitative approach provided in an integrated setting. On the other hand, older MICA patients, with more education, job experience, family connections, and more experience with hospitalization for mental illness and substance abuse tended to make greater gains in the structured, disease-specific program of Santa Rosita. These results confirm the findings of the pilot study and validate those results in a cross-cultural setting.

Within this study, the use of local indicators and the more refined measure of functioning provided a more comprehensive assessment of patient outcomes than the results found in the pilot study. The results illustrated in Table 6 and Table 8 demonstrate program specific subpopulation differences that relate to positive change in patients’ level of functioning. Viewed together with the LOFA clinical impact statistics, which demonstrate the degree of impact across a wide range of psychosocial functional areas, it is clear that GAF assessments of functional level is not the only critical variable to predicting successful treatment. Overall functioning including cognitive ability, experience with medical and psychiatric sub systems, age, and degree of social connectivity in work, family and social settings appear to be at least as salient in predicting success in both the disease-specific and integrated program. This is consistent with the results of the pilot study as well.

Socioeconomic Factors Influencing Patient Outcomes

Cross-cultural and socioeconomic factors influenced the results of this investigation on Indicators 1, 2, and 6. Within Honduras, there is no national safety net for MICA patients to rely on when they are intoxicated or psychotic. Unlike many cities in the United States, where multiple municipal shelters, hospitals and nongovernmentalt programs serve the needs of such patients, in Honduras, those patients who do not remain in the few programs that do exist, or do not have families to shelter and feed them, have no recourse. This may in part account for the wide difference between the results of study programs and the pilot study programs. In New York City, patients will be housed and fed and, in general, can negotiate systems to maintain their lifestyles on the streets if necessary. However, in developing countries, without such a social service network, many patients would actually starve to death. The high rate of graduates, the low rates of AMA discharges, relatively low rates of recidivism, and high level of patient satisfaction in both Santa Rosita and Proyecto Victoria may reflect this cultural difference in patient response to their programs.

These differences appear to have significantly influenced the results of the study and lead to only a partial validation of Hypothesis 1. Though no significant differences were found between the graduation or AMA rates, significant differences were found in the LOFA subscales between the program. This added partial validity to the first hypothesis. The more sensitive measure of the LOFA Subscales found differences in the degree and type of effect that each program had on specific levels of functioning.

The second hypothesis was, however, unaffected by these factors, since a significant difference was found in the service unit costs for successfully treated cases. The integrated program model at Proyecto Victoria produced an almost equal number of program graduates for 27% less cost.

On the other hand, differential patient satisfaction, appears to have been significantly diluted in this study, due to socio-economic factors, and no conclusions can be reached in this area. Patients answered the surveys in the presence of the raters. Thus, their need for approval from the paid raters (a possible cultural trait) may have influenced their assessment of the overall program. Qualitative data on the satisfaction indicator added additional data that provides at least some conclusions about the differential patient satisfaction in the program. However their overall opinion appears to be the same for both program models.

Negotiation Within the Responsive Evaluation Model

From the onset of this investigation, negotiation strategies played a major role in the planning and implementation of this study. After first obtaining authorization from the Ministry of Health to undertake the project and identify the stakeholders, 15 separate meetings were held to not only outline the project, but determine the needs of the government and local stakeholders. In general, the negotiation process tended to delegate the decisions of the stakeholders to lower levels. In the final stakeholder negotiations, the Ministry of Health, IHADFA, program administrators, and families delegated all decisions about local indicators to the clinical staffs of the two programs. This produced indicators that were important to the programs, but were not necessarily outcomes of treatment. The transportation and environmental indicators that were chosen by the programs affect treatment, but were not outcomes in themselves. However, they remained in the study because they were considered essential indicators by the program staff and all stakeholders agreed on the content.

On the surface, this delegation of authority and somewhat inappropriate selection of indicators suggests a weakness in using a responsive evaluation model. Inappropriate selection of indicators in an outcome study investigation did not enrich the findings of differential effectiveness between the programs. However, the use of the environmental and other non-outcome indicators in this investigation served to punctuate program needs, with the intent of improving funding and treatment. Thus, it became an important tool for the stakeholders of the programs. In addition, by delegating authority to the clinical staff, the major stakeholders finally reached agreement as to the appropriateness of indicators in a culturally approved manner.

Implications of Culturally Sensitive Patient Outcome Methodologies

Cultural and professional bias exists in most intercultural endeavors. This is particularly true in international mental health, where the views and practices of international consultants from industrialized nations often conflict with the resources, needs, and treatment philosophies of developing nations (Kleinman & Cohen, 1997). The underlying beliefs of mental health professionals from developed nations cannot readily explain or accept the differences in symptomatic expression or prevalence of mental illness due to cultural, biological, and psychosocial differences of life in developing countries. Thus, the advice and technical assistance given by international consultants of the World Health Organization (WHO), other international organizations, and NGOs generally mirrors practices in their native countries. This often sidetracks and/or defeats mental health initiatives, and ignores basic tenants of public health practice (Des Jarlis et al.,1995). In this investigation, the responsive impact evaluation model and instrumentation were selected on the basis of their ability to obtain data that was culturally sensitive for use in Latin America. Though many of the results confirm the findings of North American studies, they also suggest culturally specific areas for program change and modification.

Study Limitations and Future Research Opportunities

One of the biggest problems with this investigation is the degree of dissimilarity between the treatment programs and the sample populations. In applied settings, program service models often operate in vastly different environments, treating vastly different patient populations. Though this study attempted to isolate the critical variables that demonstrate patient outcomes and program impact related to the treatment models, the differences in sites and populations significantly diluted the results of many of the indicators. Thus, the obtained results may or may not be related to the study programs or models, but other programmatic or population factors. Future evaluation efforts could avoid this problem by more carefully matching programs, clinicians, and patient populations so that the only differences lie in the therapeutic models of integrated and disease-specific treatments.

Another major difficulty with this study lies in the monitoring of the services that were actually delivered. Though the principal researcher was on site to monitor many of the interviews and functions of the study, the actual services to patients were not monitored. This Type III error could be addressed by the addition of a qualitative evaluation of the services that actually take place in service program. This qualitative information directly reflects on the quality and duration of services delivered and would have enriched the outcome evaluation of the two treatment programs. Had this been performed in this study, perhaps the positive correlation between contact hours and GAF at Proyecto Victoria could have been more adequately understood. As such, it is difficult to draw definitive conclusions from much of the data in this study due the Type III error.

Conclusions

Though the actual results of this investigation of two programs using two different program models in Honduras cannot be generalized beyond the study programs, the use of culturally sensitive program and patient outcome measures that produced significant results can be generalized throughout the developing world. The modified responsive impact evaluation model used in this investigation produced results that identify areas of programmatic strength and relative weakness in the treatment of Honduran MICA patients. This methodological strategy can be readily applied to any single or multi-program investigation and serve the needs of all stakeholder groups.

Evaluations of this type serve all the stakeholders that need information as to the overall effectiveness of program services. Thus, from the point of view of the clinicians, patients, families, administrators, and Ministry of Health officials who participated in this study, all the results were of great value in not only establishing the level of program effectiveness, but for the identification of problematic areas that can be changed to improve patient care in the future.
Finally, this investigation examined program indicators for only two programs. Expanding the scope of this study to compare outcomes of many programs that are matched for clinical population, size, staffing patterns, geographic location, and service goals, using the modified responsive evaluation model of this investigation, may demonstrate other advantages to using the respective models. Understanding the impact of a variety of culturally appropriate program models that treat MICA patients in developing countries will contribute knowledge and expertise to the struggle against the growing incidence of substance abuse and mental illness in such countries. It would help to refine our answers to the question of "what works, for whom under what circumstances" in a culturally sensitive manner (NIMH, 1991, p. vii; WHO, 1995).


Recommendations based on this clinical impact evaluation

Santa Rosita

1. Continued monitoring and reporting of infrastructure deficits, such as the bus transportation and ambulance situation. If one level of hospital administration and health ministry receives and fails to act on the deficits in these areas, additional reports should be forwarded to higher levels of the Ministry of Health until action occurs in these areas. Infrastructure deficits, including patient security and security of personal property, are the main problems facing the alcoholism program at Santa Rosita and should remain the focus of future quality improvement efforts.

2. A team environment should be encouraged in which psychiatrists, psychologists, social workers, nursing staff and activity therapists are all permanently assigned to the program and not serve as in-house consultants to the program. The traditional hospital/medical model of direct assignment of case load and clinicians serving multiple programs and departments in the hospital fragments services within specific programs. Assignment of clinical staff to specific programs will enhance a team approach to treatment. Such changes would refine and enhance treatment planning and service delivery within the program and lead to even greater clinical impact for the patients and families.

3. General team meetings on a biweekly basis should be encouraged. This should include not only the clinical team, but family representatives, administration, and patient representatives, so that all viewpoints can be taken into account when discussing problematic areas of the program and planning actions to improve services.

4. Food service policy, procedures and resources should be reviewed and improved to more adequately meet the needs of the patients. This could be readily included as a future local indicator for the continuous quality improvement process, already in operation.

Proyecto Victoria

1. More liberal use of psychotropic medication would increase the clinical impact in the mental health and psychotic behavior areas of functioning. Using psychopharmacology in addition to the already vast array of services would decrease the length of stay in the program for patients and provide more rapid rehabilitation and higher levels of gains in the program.

2. Greater levels of clinical staffing would provide patients with increased resources for individual therapy that currently cannot be provided. This would also decrease the current burden on the clinical staff, who are currently stretched to the limit of effective service in this program.

3. General team meetings on a biweekly basis should be encouraged. This should include not only the clinical team, but family representatives, administration, and patient representatives, so that all viewpoints can be taken into account when discussing program operations and planning actions to improve services.

4. Patient concerns over their clinical care could be given additional attention. Individual patient dissatisfaction over discharge planning, inadequate attention by professional staff, and certain group activities should be explored with patients on an individual basis and treatment plans modified to address the patients’ concerns whenever possible.



REFERENCES
ADAMHA Reorganization Act of 1992, Pub. L. No. 102-321, 464R (f)(2).

American Psychiatric Association (APA) (1994). Diagnostic and statistical manual of mental disorders. 4th Ed. Washington, DC: American Psychiatric Press.

Anderson, A. J. (1996). A comparative impact evaluation of two therapeutic programs for mentally ill chemical abusers. Manuscript submitted for publication.

Anderson, A. J. (1996a). Methodologic approaches in mental health service research. Unpublished doctoral candidacy essay, Saybrook Institute.

Bachrach, L. L. (1984). The homeless mentally ill and mental health services: An analytical review of the literature. In H.R. Lamb. (Ed.) The homeless mentally ill (p. 11-33). Washington DC: American Psychiatric Press.

Bachrach, L. L. (1986-1987). The context of care for the chronic mental patient with substance abuse. Psychiatric Quarterly, 58, 3-14

Barkham, M., Rees, A., Stiles, W. B., Shapiro, D., Hardy, G. E., & Reynolds, S. (1996) Dose-effect relations in time-limited psychotherapy for depression. Journal of Consulting and Clinical Psychology, 64(5), 927-935.

Beigel, A., & Ghertner, S. (1977). Toward a social model: An assessment of social factors which influence problem drinking and its treatment. In B. Kissin & H.

Begleiter (Ed.), The biology of alcoholism. Treatment and rehabilitation of the chronic alcoholic (Vol. 5, p. 197-233). New York: Plenum.

Bockman, J. S. (1963). Moral treatment in American psychiatry. New York: Springer.

Borg, W. R., & Gall, M. D. (1989). Educational research. New York: Longman, 587-540.

Breakey, W. R. (1987). Treating the homeless. Alcohol and Research World, 11, 42-47.

Brower, K. J., Blow, F. C., Beresford, T. P. (1989). Treatment implications of chemical dependency models: An integrative approach. Journal of Substance Abuse Treatment, 6(3), 147-157.

Caron, C. (1981). The new chronic patient and the system of community care. Hospital and Community Psychiatry, 32, 475-478.

Clarke, G. N. (1995). Improving the transition from basic efficacy research to effectiveness studies: methodological issues and procedures. Journal of Consulting and Clinical Psychology, 63 (5), 718-725.

Cohen, J. (1988). Statistical power analysis for behavioral sciences (2nd ed.). New York: Academic Press.

Cummings, N. A. (1993). Psychotherapy with substance abusers. In G. Striker & J. R. Gold (Ed.) Comprehensive handbook of psychotherapy integration. New York: Plenum Press.

Des Jarlais, R., Eisenberg, L., Good. B., & Kleinman, A. (1995). World mental health: Problems and priorities in low-income countries. Cambridge, Oxford University Press.

DonGiovanni, V. J. (1988). An evaluation of an ongoing treatment program for the psychiatrically impaired substance abuser. Investigation Archives, Indiana University of Pennsylvania.

Drake, R. E., Antosca, L., Noordsy, D. L., Bartles, S. J., & Osher, F. C. (1991). Specialized services for the dually diagnosed. In K. Minkoff and R. E. Drake (Ed.), Dual diagnosis of major mental illness and substance disorder (New directions in mental health (p. 67-67). San Francisco, Josse-Bass.

Drake, R. E., Osher F. C., & Wallach, M. (1989). Alcohol use and abuse in schizophrenia a prospective community study. Journal of Nervous and Mental Disease, 177, 408-414.

Drake, R. E., Osher, F. C., Noordsy, D. L., Hurlbut, S. M., Teague, G. B., Beaudett, M. S. (1990). Diagnosis of alcohol use disorders in schizophrenia. Schizophrenia Bulletin, 16, 57-67.

Drake, R. E., Osher, F., & Wallach M. (1991) Homelessness and dual diagnosis. American Psychologist, 46(11), 1149-1158.

Evans, K. & Sullivan, J. M. (1990) Dual diagnosis: counseling the mentally ill substance abuser. New York: Guilford Press.

Farkas, M. D., and Anthony, M. A. (1991) Psychiatric rehabilitation programs. Baltimore, MD: John Hopkins University Press.

Fetterman, D. M., Kaftarian, S. J., & Wandersman, A. (1996). Empowerment Evaluation. London: Sage.

Galanter, M., Castaneda, R. & Ferman, J. (1988). Substance abuse among general psychiatric patients: Place of presentation, diagnosis and treatment. American Journal of Drug and Alcohol Abuse, 14, 211-235.

Glaser, F. B. (1980). Anybody got a match? Treatment research and match hypothesis. In G. Edward's & M. Grant (Ed.), Alcoholism treatment in transition (p. 178-196). London: Croom.

Guba, E. G. & Lincoln, Y. (1981) Effective evaluation. San Francisco: Josey-Bass.

Guba, E. G. & Lincoln, Y. (1989) Forth generation evaluation. Newbury Park, CA: Sage.

Hammer, J. S. (1996 May) Economic analysis for health projects. Policy Research Working Paper: The World Bank Policy Research Department. May.

Harris, M., and Bergman, H. (1987). Case management with the chronically mentally ill: A clinical perspective. American Journal of Orthopsychiatry. 57 (2), 296-302.

Humphreys, K., & Rappaport, J. (1993). From the community mental health movement to the war on drugs. American Psychologist, 48(8), 892-901.

IHADFA (1994). Reporte Annual de Ministerio de Salud. Ministry of Health, Honduras, (132-141), August.

Inouye, D. (1983). Access, stigma, and effectiveness. American Psychologist. August, 912-917.

Isaac, S. (1971). Handbook of research and evaluation. San Diego: Edits.

Klerman, G. L. (1972). Public trust and professional confidence. Smith College Studies in Social Work, 17, 115-124.

Klerman, G. L. (1974). Current evaluation research on mental health services. American Journal of Psychiatry, 131 (7), 783-787.

Klerman, G. L., Kellam, S., & Leiderman, H. (1971). Research aspects of community mental health centers: report of the APA task force. American Journal of Psychiatry, 127, 993-998.

Kleinman, A. & Cohen, A. (1997). Psychiatry's global challenge. Scientific American. March, 86-89.

Koegel, P., & Burnam, M. A. (1988). Alcoholism among homeless adults in the inner city of Los Angeles. Archives of General Psychiatry. 45: 1011-1018.

Laporte, D. J., McLellan A. T., Lamb, R. & O'Brien, C. P. (1989). Treatment response in psychiatrically impaired drug abusers. Comprehensive Psychiatry, 22, 411-419.

Lalley, T. L., Hohmann, A. A., Windel, C. D., Norquist, G. S., Keith, S. J., & Burke, J. D. (1992). Caring for people with severe mental disorders: A national plan to improve services. Schizophrenia Bulletin, 18, 559-700.

Marlatt, G. A. (1985). Relapse prevention: Theoretical rationale and overview of the model.

In G. A. Marlatt & J. R. Gordon (Ed.) Relapse prevention (p. 3-70). New York: Guilford Press.

Marlatt, G. A. (1988). Matching clients to treatment: Treatment models and stages of change. In D.M. Donovan & G. A. Marlatt (Ed.), Assessment of addictive behaviors (p. 474-483). New York: Guilford Press.

McLellan, A. T. (1986). Psychiatric severity as a predictor of outcome from substance abuse treatment, in R. E. Mever (Ed.) Psychopathology of Addictive Disorders. New York, Guilford Press.

Miller, N.S. (1994). Treating coexisting psychiatric and addictive disorders. (P. 7-23). Center City, Minnesoda: Hazelden Educational Materials.

Minkoff, K. (1987). Beyond deinstitutionalization: A new ideology for the postinstitutional era. Hospital and Community Psychiatry, 38, 945-950.

Minkoff, K. (1989). An integrated program model for dual diagnosis of psychosis and addiction. Hospital and Community Psychiatry. 40 (10), 1031-1036.

Minkoff, K. (1991). Program components of a comprehensive integrated care system for seriously mentally ill patients with substance disorders. New Directions for Mental Health Services, 50, 95-106.

National Institute of Mental Health. (1989). The future of mental health services research (DHHS Publication No. DM 89-1600). Washington, DC: U.S. Government Printing Office.

National Institute of Mental Health. (1991a. Caring for people with severe mental disorders: A national plan of research to improve services. In C. A. Traube, D.

Mechanic, & A. A. Hohmann, (Ed.) (DHHS Publication No. ADM 91-1762). Washington, DC: U.S. Government Printing Office.

National Institute of Mental Health. (1991b). Clinical services research: enhancing the real-world applications of clinical science. In Caring for people with severe mental disorders: A national plan of research to improve services. (DHHS Publication No. ADM 91-1762). Washington, DC: U.S. Government Printing Office.

Newman, F. L., & Howard, K. I. (1991) Introduction to the special section on seeking new clinical research methods. Journal of Consulting and Clinical Psychology, 59, 8-11.

Newman, F. L., Howard, K. I., Windle, C. D., and Hohmann, A. A. (1994). Introduction to the special section on seeking new methods in mental health services research. Journal of Consulting and Clinical Psychology. 62, 667-669.

Newman, F. L., & Tejeda, M. J. (1996) The need for research that is designed to support decisions in the delivery of mental health services. American Psychologist, 51(10), 1040-1050.

Nuttbrock, L., Rahav, M., Rivera J., Ng-Mak, D., Struening, E. (1997). Mentally ill chemical abusers in residential treatment programs: effects of psychopathology on levels of functioning. Journal of Substance Abuse Treatment. 14(3), 269-274.

NYSOMH-(1990). Part 585, mental health regulations for outpatient services. New York State Office of Mental Health. Albany, NY: NYSOMH.

Osher, F. C., Kofoed, L. L. (1989). Treatment of patients with psychiatric and proactive substance abuse disorders. Hospital and Community Psychiatry, 40(10), 1025-1030.

Overall, J. E., & Gorham, D.R. (1962) The brief psychiatric rating scale. Psychological Reports, 10, 199-812.

PsycINFO (1993). Dual Diagnosis Index. Alexandria VA: American Psychological Association.

PsycScan (1994). Program effectiveness index. Alexandria VA: American Psychological Association.

PsycINFO Index (1995). Program effectiveness/efficacy index. Alexandria VA: American Psychological Association.

Ridgely, S. M., Osher, F. C., Talbott, J. A. (1987). Chronic mentally ill young adults with substance abuse problems. Treatment and Training Issues. Rockville, MD: Alcohol, Drug Abuse and Mental Health Administration.

Ridgely, S. M., Goldman, H., Willenbring, M. (1990). Barriers to the care of persons with dual diagnoses: Organizational and financing issues. Schizophrenia Bulletin, 16, 123-132.

Robitscher J. (1972). The right to psychiatric treatment: a social-legal approach to the plight of the state hospital patient. Villanova Law Review, 18:11-36.

Robitscher J. (1972). Courts, state hospitals, and the right to treatment. American Journal of Psychiatry, 127: 993-998.

Rothman D. (1971). The Discovery of the Asylum: Social Order and Disorder in the New Republic. Boston: Little, Brown and Co.

Safer, D. (1987). Substance abuse by young adult chronic patients. Hospital and Community Psychiatry, 38, 511-514.

Schnieder, L. C. , & Struening, E. I. (1983). SLOF: A behavioral rating scale for assessing the mentally ill. Social Work Research and Abstracts, 41, 9-21.

Schucket, M. A., (1985). The clinical implications of primary diagnostic groups among alcoholics. Archives of General Psychiatry, 42, 1043-1049.

Schwartz, S. R. and Goldfinger, S. M. (1981). The New Chronic Patient: Clinical Characteristics of an Emerging Subgroup. Hospital and Community Psychiatry. Vol. 32, 470-474.

Scott, J. P., & Ginsburg, B.E. (1994) The seville statement on violence revisited. American Psychologist, 49, No. 10, 849-850.

Seitz, N. (1979). Financial Analysis. Reston, Virginia: Reston Publishing Company.

Suchman, E. A. (1967). Evaluative research, principles and practice in public service and social action programs. New York: Russel Sage Foundation.

Uehara, E. S., Smukler, M., & Newman, F. L. (1994). Linking resource use to consumer level of need: Field test of the level of need-care assessment (LONCA) method. Journal of Consulting and Clinical Psychology, 62, 695-709.

Valliant, G. E. (1983). The Natural History of Alcoholism. Cambridge, Mass.: Harvard University Press.

Westin, D. (1972). cited in Drug Research Reports15 (49), Dec. 2, p.RN3.WHO (World

Health Organization) (1993). Mental health care in the western pacific region: present

status, needs, and future directions. International Journal of Mental Health. Vol. 22 (1), 101-116.

WHO (World Health Organization) (1995). World health report: bridging the gap. Geneva, World Health Organization.

Windle, C., & Lalley, T. L. (1992). Recent findings from NIMH's services research program. Administration and Policy in Mental Health, 19 (5).

Wood, S. (1974). Fundamentals of psychological research. Englewood Cliffs, NJ: Prentice-Hall.

Yeaton, W. H. (1994) The development and assessment of valid measures of service delivery to enhance inference in outcome-based research: Measuring attendance at self-help group meetings. Journal of Consulting and Clinical Psychology, 62, 686-694.


        Appendices available upon request
 



Copyright © 2000, Southern Development Group, S.A.  All Rights Reserved.
A Private Non-Profit Agency for the good of all, published in the UK & Honduras