Anderson, A. J. (1999) A Comparative Impact Evaluation
of Two Therapeutic Programs for
Mentally Ill Chemical Abusers. International Journal of Psychosocial Rehabilitation. 4, 11-26
Abstract: This study investigated the relative impact of a traditional, disease specific program model and an integrated program model on the basis of treatment outcomes for homeless, mentally ill chemical abusers (MICA) patients. Patient outcomes in these two hospital based, residential programs that treat MICA patients and that varied only in their treatment models were compared across seven indicators: successful community placement, treatment failures, service hours, subpopulation outcomes, recidivism, patient satisfaction, and cost efficiency. It was found that the integrated program model tended to produce greater gains with a low functioning, homeless MICA population, while the disease specific, therapeutic community model may be more suited to higher functioning MICA patients. A recommendation is made to expand the scope of efficacy and outcome research with different program models.
A Comparative Impact Evaluation of TwoTherapeutic Programs for Mentally Ill Chemical Abusers
Introduction: Patients diagnosed with severe mental illness who also suffer from other psychiatric and medical disorders present a variety of individual, social, fiscal and political challenges not only for program funding, but for planning and implementing effective rehabilitative treatment programs as well. Severe mental illness coupled with severe personality, behavioral, addictive, cognitive or physical disease, stretches the ability of traditional community based treatment programs to effectively treat these patients. This has led to the development of a wide variety of treatment models designed to serve the multiple needs of these patients (Bachrach, 1984; Drake, 1991; Minkoff 1987). An evaluation of the clinical effectiveness of these models would not only lead to more cost effective use of limited treatment resources, but more importantly, improve clinical treatment options for the dually diagnosed patient.
Focusing a program impact evaluation on current treatment models for the most commonly reported dually diagnosed population, mentally ill chemical abusers (MICA), may demonstrate the differential rates of effectiveness for the most commonly used program models (PsycInfo, 1993). Such results would be useful to policy planners, administrators, legislators and program developers who must design and implement the most effective treatment programs possible with limited resources. More importantly, this information can be used to not only determine what treatments and programmatic approaches are effective, but to improve the quality of care for severely dysfunctional MICA patients.
Treatment programs for the dually diagnosed primarily fall into two main categories: disease specific and integrated program models (Minkoff,1991). Disease specific program models tend to focus treatment on what they consider to be the primary area of distress and minimize the importance or urgency of other areas of dysfunction. Many substance abuse treatment programs and traditional hospital based mental health programs typically model their treatment programs in this manner. Integrated program models, found in both hospital and community based settings, are designed to provide individualized treatment planning and services that focus attention on all areas of patient dysfunction within a single program.
The development of these models has not been based so much on the clinical efficacy, but more on availability of funding and political interest in treating specific patient populations (Humphreys & Rappaport, 1993). This fragmentation of program models has been perpetuated through the development of artificial and arbitrary administrative divisions at the federal, state and local levels without regard to clinical measures of success for the various program models. Consequently, it is possible that many public sector and grant funded programs continue to be financed through a variety of funding streams with little or no demonstrable clinical success. This siphons critical funds from those programs that use more clinically viable models.
Dually diagnosed patients in general and specifically MICA patients have complex treatment needs and interactive symptomatology that require a more integrated approach than is generally employed (Breakey, 1987). It therefore seems more likely that integrated treatment models would be clinically more effective in treating the dually diagnosed MICA patient than disease specific models. However, given the severe therapeutic challenges that substance abuse and dependency present, a more restrictive, traditional substance abuse model may actually provide increased efficacy for the MICA patient as well. Evaluating the various treatment outcomes produced by each program model, treating a similar patient population, should demonstrate the relative clinical effectiveness and cost efficiency of each program model to treat MICA patients.
This exploratory study investigates the clinical impact of each program model on MICA patients. Specific outcome indicators and study variables were selected to demonstrate each model's effectiveness in meeting the programs' stated therapeutic goals for MICA patients, and to rule out outcomes that may be due to differences between the programs, such as population differences, number of services delivered, and level of patient participation. These indicators compare relative degrees of outcome efficacy between the two programs and are not process oriented. As such, they directly relate to the goals of the two programs. Since the location, staffing pattern, outplacement resources, and goals of the programs are identical, and the study population similar in terms of diagnostic and demographic composition, the results should illustrate the relative impact of disease specific and integrated program models in the treatment of homeless, MICA patients.
Method Design and Program Characteristics
This investigation evaluated patient outcomes for two treatment programs at Bellevue Hospital Center, NYC, NY, that treat homeless, male mentally ill chemical abusers. Clinical outcomes in the MICA Transitional Living Community Program (MICA TLC), a disease specific substance abuse treatment program, are compared with outcome data from the Transitional Living Community (TLC), an integrated, MICA residential mental health program. Both residential programs have program goals of rehabilitating MICA patients over a six month period and placing graduating patients in community based housing. A comparison of patient outcomes for the same efficacy indicators was performed to determine the relative value of integrated and disease specific treatment models in treating MICA patients.
TLC Integrated Program Model: This 30 bed voluntary residential rehabilitation program is located at the 30th Street Facility of the Bellevue Hospital Campus on the 6th floor. This program for homeless males with severe and persistent mental illness is administered by the Director of Community Support Services and is staffed by one psychiatrist, three social workers, one psychologists, two nurses, five nurse aides, three rehabilitation/activity therapists, and two clinical case managers. Individual therapy, group therapy, rehabilitative group sessions, case management support, substance abuse counseling, double trouble (addiction and mental illness) groups, and pharmacotherapy were customized within individualized treatment plans to meet the needs and desires of the patient.
In this program, a voluntary treatment contract is negotiated with each patient and he is free to leave the program during the day, provided he has fulfilled his contractual treatment schedule for that day. Patients are typically scheduled for 3 to 5 hours of treatment daily. This program focuses on the whole spectrum of mental illness and substance abuse issues in an open, non threatening environment, normalizing substance abuse issues as just one more treatment challenge that the patient has the responsibility for overcoming.
MICA TLC Disease Specific Program Model: This is also a 30 bed residential rehabilitation program, located at the 30th Street Facility of the Bellevue Hospital Campus on the 6th floor. This program, also designed to treat homeless males with severe and persistent mental illness and substance abuse, is also administered by the Director of Community Support Services and is staffed by one psychiatrist, four social workers or psychologists, two nurses, five nurse aides, three rehabilitation/activity therapists, and two clinical case managers. This program adopted a traditional generic self help, therapeutic community treatment approach where traditional substance abuse groups and individual therapy are combined with pharmacotherapy to reduce dependency needs and facilitate social re integration. Confrontational approaches and a highly structured 'house' hierarchy maintain an almost military atmosphere that structures the lives of all the patients in the program.
Like other disease specific MICA programs, patients agree to remain on the unit and are supervised for all off program activities. Unlike the TLC program, treatment plans are generally identical for all patients and stress abstinence and social responsibility. Services include Double Trouble Groups, Substance Abuse Rap Groups, AA, Resocialization Groups, Individual Psychotherapy and Community reentry groups focused on maintenance of sobriety in the community. This program maintains a highly structured, substance free, restrictive environment focused on delivering a similar level and type of treatment to all admitted patients.
Data Collection and Subject Inclusion:
All male, mentally ill chemical abusing patients who entered the programs after 4/25/91 and who were discharged by 11/25/93 were included in the investigation. The admission criteria of homelessness and a major Axis I diagnosis in addition to a substance abuse diagnosis are the same for both programs. In addition, all patient referrals to the programs come from the same Bellevue inpatient psychiatric units and New York City homeless shelter programs. Thus, the patient 'pool' for these two programs are assumed to be identical for the purposes of this study (See Subject Selection). This was confirmed by an analysis of variance and comparison of demographic characteristics between the groups on the dimensions of age, diagnosis, substance abuse severity (in years and type of abuse), prison history, suicide history, medication, and number of previous hospitalizations. Since this subject pool was thus determined to be grossly equivalent, the relative success of each treatment model could be compared and a comparative rate determined across indicators (Tables 1 & 2).
Within the context of this study, successful rehabilitation of patients and program graduation required a Global Assessment of Functioning (GAF) level of at least 80. The functional level was recorded on New York City Department of Mental Health functional assessment forms. The addiction severity index is a survey instrument to record the type of substance abused, duration of substance abuse and housing, prison and work history. The determination of functional level at the time of program graduation and the degree of addiction severity were made by clinical case managers within the programs. These levels and scores were subsequently confirmed by clinical case managers from an affiliated case management program. The administration of the addiction severity index required no additional training since it is a self report questionnaire that could be independently completed by patients. The case manager raters from the affiliated case management program were responsible for long term patient follow up of those patients who graduated and were placed in community based housing. Three month post graduation outcome information on patients who were placed in community based, residential programs and apartments was performed by these same Clinical Case Managers who recorded their findings on New York State Office of Mental Health Form 143a, Parts 1 & 2. This data provides the basis for post graduation placement and recidivism data.
Outcome measures on 76 male patient from the TLC and 149 patients from the MICA TLC were included in this investigation. The patients for both programs under study were selected on the basis of their meeting the diagnostic and homeless admission criteria for the programs. Patients who were homeless for at least 3 months, who were ambulatory and no longer in need of acute care, and who were diagnosed with a major Axis I disorder of Schizophrenia, Major Depression, or Bipolar Disorder and an additional substance abuse diagnosis were admitted to both treatment programs directly from the same inpatient acute care units of the hospital and NYC Shelter Programs, on a space available basis. No other conditions for admissions were imposed.
Though the TLC program also admitted patients without the additional substance diagnosis, only the MICA patients were included in this study. In the MICA TLC all admitted patients were included in this study. No MICA patients were excluded from this study in either program.
Equivalency of Subject Characteristics: In terms of the programs' goals of rehabilitation and placement of mentally ill chemical abusers within a specified time frame, the subjects selected from both programs are considered are identical; they are both comprised of only MICA patients. In addition, both groups meet the target diagnostic and functional criteria for treatment in the programs and from the programs' point of view were expected to functionally improve over a six month period to the point of community readiness. Since this study compares patient outcomes against the program goals, they are considered identical only in this respect. Both groups are comprised of male, homeless, MICA patients with similar backgrounds from the same geographic location, receiving treatment in the same hospital facility.
Measures and Analysis
The two programs were evaluated by the following outcome criteria:
Indicator 1: Successful Treatment Outcomes The primary program goal for each program is to rehabilitate patients over a three to six month period to the point where they are functionally able to live independently in the community and maintain psychiatric stability and abstinence from psychoactive substances for a period of six months after placement in their community setting. Patients who were rated as functionally ready for independent living, GAF level (APA, 1994) of least 80, at the time of program graduation and three months after community placement were considered functionally capable of community based living and considered successful in fulfilling the programs' rehabilitation goals. The relative percentages of patients who met and maintained this functional level were included as a measures of the models' success.
Indicator 2: AMA Discharge Within the context of this evaluation, the percentage of patients who did not complete treatment and who left the programs against medical advice (AMA) were compared across programs as a measure of the treatment program's inability to meet the needs of the target population. Since both program models had clinical failures of this type, the percentage of patients who left each program against medical advice or for violent threats or actions (cause) were included as AMA Discharge measures of negative clinical outcomes.
Patients who were discharged before program graduation for substance abuse relapse were also included in this measure. However, there were few of these outcomes due to the following reasons:
The MICA TLC is a voluntary lock down program with 2 scheduled urine analyses (UA) weekly and one random UA.
In the TLC program, drug/ETOH was strongly discouraged, Uas done on suspect abusers, ETOH breath analysis done on a regular basis, and progressive levels of loss of privileges and other disincentives used to discourage drug and alcohol use. In the TLC program every effort was made to rehabilitate patients across all psychosocial areas, including substance abuse. When this failed, the patients generally left AMA on their own when they could not tolerate the loss of privileges or other negative sanctions that curtailed their activities.
If patients decompensated to an acute stage they were rehospitalized and this is counted in the recidivism data.
Once in community placements, they were generally given UAs in most cases but this data was unavailable because it occurred in Private Non Profit programs not connected to the hospital. In almost all cases the patients were placed in community residences that discouraged use and monitored patients for abuse. Though few were actually 'kicked out' of these residences, many left not due to psychiatric reasons but due to a return to substance abuse. In either case their functional level can be assumed to have dropped because they were no longer able to live independently.
Indicator 3: Correlation of Service Hours delivered. Bivariate correlations between actual number of service hours delivered and successful treatment outcomes by program was performed to evaluate whether variable amounts of treatment affected overall program outcomes and success. Total service hours delivered to each patient in a program were correlated with the relative percentage of successful outcomes and AMA discharges in each program. This evaluated the relationship between amount of services delivered and successful treatment outcomes.
Data collection methods: These hours were collected in the accounting data used for medicaid reimbursement. This listed the type of service given, to what patients for how long, by each clinician. Each week, all clinicians would fill out their patient summary sheets that include all patient contact hours for that week. Each of these hours is considered a service unit. These service units were summed for all patients and then compared with patient functional level or overall outcome in the program in bivariate correlations.
Indicator 4: Subpopulation Outcomes Differential success rates between diagnostic subpopulations (i.e., schizophrenic, substance abusers versus mood disordered, substance abusers), was also compared to determine whether the models are best suited to one sub population or another. This measure was also applied to the criteria indicators listed above.
Indicator 5: Relative Rates of Recidivism Percentage rates of patient recidivism for program graduates was also determined as a measure of the program's relative inability to effectively rehabilitate their target MICA population. Recidivism within the context of this study is defined as any patient who returns to the hospital or another treatment program during or after his placement in community based housing. This may be due to either decompensation to an acute psychiatric phase, or a return to substance abuse to a degree that requires rehospitalization or additional rehabilitative treatment in a residential treatment facility. Indicator 6: Patient Satisfaction Patients in both groups completed a standard New York State Office of Mental Health patient satisfaction survey (NYSOMH, 1990) within a month of their graduation from their respective treatment program. This provided a measure of the patients' qualitative level of satisfaction within each program model.
Indicator 7: Cost Efficiency Rate Relative cost per service unit was determined for patients who successfully met the goals of the program (See Indicator 1). This measure was included to illustrate differential program costs for those patients who met program goals and is considered a gross measure of program efficiency within the context of this study. The cost figures were determined by dividing the total annual operating costs (personnel, medication and supply costs) by the sum of the service hours for patients who met the 'successful outcome' criteria. Program development costs, capital improvements, and other non operating costs were excluded from this analysis. Cost benefit and cost effectiveness analysis were not performed, since the necessary variables for such analysis were beyond the scope of this investigation.
Comparisons of Patient Characteristics: Tables 1 and 2 detail the results of MANOVA and comparison of means by independent samples t tests. Significant between group differences were only found in five of the 33 characteristics selected for comparison. On average, MICA TLC patients tended to be 6 years younger, comprised of 23% more Afro Americans, 15.2% less alcohol dependent, stay 26 less days in the program, and receive $143.48 less welfare than those in the TLC. However, these differences appear to be more directly related to programmatic factors, than sample differences in the population. Patients who were being considered for admission to the programs had the option of rejecting admission, even when accepted to the programs. Thus, the entire referral pool of patients more accurately represents this population of MICA patients because those who deselect themselves from the programs are included and can be compared between the programs. When the entire pool of patient referrals to each program was compared to the pool of admitted patients, the significant differences in age and ethnicity disappeared. It is likely that older, alcohol abusing patients (2X more likely to be Caucasian) would find the environment of the MICA TLC too restrictive and seek entry to the TLC Program or refuse admission to either program. One hundred thirty five patients who were referred to the programs refused admission; 74% of these were referrals to the MICA TLC. Length of stay differences were due to the high rate of MICA TLC 'dropouts' (Figure 1) and welfare benefit differences were due to a reluctance on the part of the MICA TLC staff to supply substance abusers with cash benefits that were felt to be potential sources for 'drug money'.
Though these characteristic variances are statistically significant, the variation between the study groups is insufficiently large to account for the various patient and program outcomes displayed in Figure 1 and 3. For example, the effect size differences between the patient outcomes of the two programs cannot be accounted for by the significant between group differences in diagnosis. The 12.9% difference in crack/cocaine dependency in the MICA TLC Program represents a total of 19 patients, while alcohol dependency differences of 15.2% in the TLC program only represents 12 patients. Using a worst case scenario, excluding these differential patients from each program's differences in successful community placement, the TLC program still produced more than twice the number of placed graduates. Thus, the effects of the diagnostic and other differences probably contributed to the differences in outcomes, but cannot account for the full effect size differences displayed in Figure 1.
Indicators 1 and 2 - Results for these indicators are demonstrated in Figure 1 and Table 3. Though rehospitalization rates were found to be almost equal for both programs, the TLC program had almost 5 times more successful placements than the MICA TLC (by percentage) with only half the AMA rate of the MICA TLC. Total Placements and AMA rates varied significantly between the two groups (p < .01).
Indicators 3 and 4 - Results of multiple bivariate correlations of service hours to outcome and sub populations differential outcome analysis are summarized in Table 4 and Figure 2. Significant correlations were found between successful placements and number of clinical contact hours that patients received in the program. This correlation for the MICA TLC was particularly high; exceeding that of the TLC program by r=.2219. In addition length of program stay was also correlated to positive treatment outcomes at almost the same level. These results suggest that positive treatment outcomes are directly related to the amount of treatment these patients received in both programs. It also suggests that both treatment models can effect positive rehabilitative change, provided the patient is motivated to remain in treatment and actively participate in the programs.
No significant differences were found for diagnostic subpopulations for either positive or negative treatment outcomes. However, significant correlations were found between those patients who were placed and the number of previous hospitalizations. In addition, months of homelessness negatively correlated with positive treatment outcomes. These results suggest that patients with more previous hospitalizations may benefit from the TLC more than the MICA TLC. The negative correlation between previous hospitalizations and length of stay in the MICA TLC lends additional support for this conclusion.
Months of homelessness and length of stay in the MICA TLC Program also significantly correlated with years of education. MICA TLC patients with more education tended to have spent less time in a homeless situation. However the most interesting statistic here is the relationship between length of stay and education. This result suggests that those who stay longer in the program are also those with the highest level of education. This is not the case in the TLC program, where no significant correlations were obtained. In view of the demanding nature of the highly structured MICA TLC program this result is understandable. For patients to successfully negotiate the program and remain until graduation, they would need higher levels of cognitive skills than in the TLC, where treatment plans are individually geared to the level of the patient.
Indicator 5- A comparison of recidivism rates between the programs is presented in Figure 3. While rehospitalization for psychiatric relapse remained grossly equivalent between the programs, recidivism due to substance abuse relapse was significantly higher in the MICA TLC (36.9% to 11.8%, p < .05, two tailed). The MICA TLC had almost three times more substance abuse relapses that resulted in a return to homelessness or a residential drug treatment than the TLC program.
Indicator 6- Patient Satisfaction Survey results are displayed in Figure 4. This survey was only administered to a sample of the subjects in this study due to time constraints, data collection problems and patient refusal to participate. This standard NYSOMH survey consists of 20 fill in the blank questions about various aspects of the program. Patients who responded positively on six questions or less were considered not satisfied with the program; those who reported six to 12 positive responses were rated as somewhat satisfied; and those who responded with 12 or more positive responses were considered very satisfied with the program.
Of the 55% who responded in the TLC program and 62% who responded in the MICA TLC, almost twice as many patients reported that there were very satisfied with the program in the TLC program than the MICA TLC. In addition, while the percentage who were very satisfied was roughly equal to those who reported to be somewhat satisfied in the TLC, this was not the case in the MICA TLC. In the MICA TLC almost three times as many patients reported to being only somewhat satisfied, compared to those who reported being very satisfied.
Indicator 7-Based on an operating budget of $606,000 for the TLC and $732,000 for the MICA TLC, the rate of cost efficiency for successful outcomes in the TLC was computed at $19.04 per service unit, while that of the MICA TLC was found to be $40.69. This difference is due to the low number of successful outcomes in the MICA TLC. Based solely on this, it could be concluded that the TLC Program is twice as cost efficient in successfully treating MICA patient than the MICA TLC.
This study investigated the relative impact of a traditional, disease specific program model and an integrated program model on the treatment outcomes for homeless, MICA patients. Based on the National Institute of Mental Health (1991) guidelines for mental health service research, this study began with the question of what works, for which MICA patients, in two programs that use different therapeutic approaches to address both mental illness and substance abuse. The patient outcome results demonstrate clear differences between the programs and their therapeutic models.
From the therapeutic success and failure rates, the differences in rates of recidivism, patient satisfaction survey results, and level of cost efficiency, the integrated TLC Program model appears to have distinct advantages over the MICA TLC in the treatment of severe MICA patients with low levels of education and high numbers of past hospitalization. However, from the results of the subpopulation and characteristics correlations, the disease specific, therapeutic community model of the MICA TLC seems to work well with patients who are more highly educated and have fewer previous hospitalizations. The high degree of MICA TLC clinical failures and recidivism and lower levels of patient satisfaction appear to be due to the inability of a traditional substance abuse model to treat low functioning MICA patients. Since the MICA TLC admitted primarily low functioning patients, the clinical failure rate is understandable. The TLC Integrated model appears to be more suited to treating such patients. Higher functioning MICA patients with less distress may fare better in the highly structured traditional environment where the locus of control is external, than in a treatment milieu that customizes services to meet the needs of patient, requiring an internal locus of control.
Finally, as an exploratory study, this investigation examined program indicators for only two programs. Expanding the scope of this study to compare outcomes of many programs that are matched for clinical population, size, staffing patterns, geographic location and service goals may demonstrate other advantages to using the respective models and refine our answers to the question of "what works, for whom under what circumstances" (NIMH, 1991, p.vii).
American Psychiatric Association (APA) (1994). Diagnostic and Statistical Manual of Mental Disorders. 4th Ed. Washington, DC: American Psychiatric Press.
Bachrach, L. L. (1984). The homeless mentally ill and mental health services: An analytical review of the literature. In H. R. Lamb. (Eds.) The homeless mentally ill (pp. 11 33). Washington DC: American Psychiatric Press.
Breakey, W. R. (1987). Treating the homeless. Alcohol and Research World, 11, 42 47.
Drake, R. E., Antosca, L., Noordsy, D. L., Bartles, S. J., Osher, F. C. (1991). Specialized services for the dually diagnosed. In K. Minkoff and R. E. Drake (Eds.), Dual diagnosis of major mental illness and substance disorder (New directions in Mental Health (pp. 67 67). San Francisco: Josse Bass.
Drake, R. E., Osher F. C., Wallach, M. (1989). Alcohol use and abuse in schizophrenia a prospective community study. Journal of Nervous and Mental Disease, 177, 408 414.
Drake, R., Osher, F., Wallach M. (1991) Homelessness and Dual Diagnosis. American Psychologist, 46(11), 1149 1158.
Humphreys, K., & Rappaport, J. (1993). From the community mental health movement to the war on drugs. American Psychologist, 48(8), 892 901.
Minkoff, K. (1987). Beyond deinstitutionalization: A new ideology for the postinstitutional era. Hospital and Community Psychiatry, 38, 945 950.
Minkoff, K. (1991). Program components of a comprehensive integrated care system for serious mentally ill patients with substance disorders. New Directions for Mental Health Services, 50, 95 106.
National Institute of Mental Health. (1991). Caring for People With Severe Mental Disorders: A National Plan of Research to Improve Services. In C. A. Traube, D. Mechanic, & A. A. Hohmann, (Eds.) (DHHS Publication No. ADM 91 1762). Washington, DC: U.S. Government Printing Office.
NYSOMH (1990). New York State Office of Mental Health. Part 585, mental health regulations for outpatient services. Albany, NY: NYSOMH.
Psycinfo (1993). Dual Diagnosis Index. Alexandria VA: American Psychological Association.
Copyright © 2000, Southern Development Group,
S.A. All Rights Reserved.
A Private Non-Profit Agency for the good of all, published in the UK & Honduras