Treatment of Aptitude × Treatment Interactions

The main effects in psychotherapy research have been smaller than expected. Rather than concluding that psychotherapy has weak effects, clinical researchers have argued that average effect sizes are reduced because of mismatches between clients and treatment. Hence, Aptitude × Treatment interaction (ATI) research has been viewed as a promising new frontier in psychotherapy research. If ATI research is to become a productive and progressive program, then researchers will need to focus their attention on interesting and meaningful ATIs. This will require greater theoretical precision and a stronger emphasis on construct validity. Specific issues addressed in this article include Type II and Type III errors, manipulation checks from both the patient and practitioner perspective, considerations of the strength of treatment, the need to test rival hypotheses, and the desirability of collaborative research.

Psychotherapy outcome research is a frustrating business. Experiments are not easy to arrange and control, outcomes are difficult to measure–often even to define–and results are often disappointing. Psychotherapy ought to work better than it appears to. One possible explanation for why it does not is that the effects of psychotherapy depend on specific characteristics of patients and the therapies to which they are exposed (i.e., effects of psychotherapy may depend on Aptitude × Treatment Interactions (ATIs; Cronbach & Snow, 1977 ). More specifically, the ATI hypothesis states that appropriate matching of patients with treatment will result in better outcomes. This hypothesis has been optimistically interpreted by many clinical researchers to mean that ATI research can uncover psychotherapy effects that, compared with main effects, are stronger and more reliable. Unfortunately for the optimists, as we will assert and try to account for in this article, compared with main effects, ATIs in psychotherapy research may be infrequent, undependable, and difficult to detect. The purpose of this article is to take a sober look at the realities and probable impact of ATI research in terms of psychotherapy theory and practice. The ATI approach is not a quick fix to the problem of disappointing results in psychotherapy research. This article outlines a variety of stringent conditions necessary for adequate ATI research. Ironically, if our recommendations are heeded, it is likely that subsequent research will uncover previously "hidden" main effects more frequently than interactions.

The existence of an interaction does not necessarily mean that circumstances exist under which the two treatments are equal, or that if one is better for one thing, the other must be better for something else. Interactions can be disordinal or ordinal ( Cronbach & Snow, 1977 ). In the case of disordinal interactions (panel 1 of Figure 1 ), the lines connecting the like treatment conditions cross. This, for some reason, is often taken as firmer evidence for an interaction than a "mere" ordinal interaction in which the lines do not cross (at least not within the range of variables studied; see panel 2 of Figure 1 ). If statistical significance means anything, it surely means that a significant ordinal interaction is just as dependable as a similarly significant distortional interaction. Some of the mythical superiority of disordinal interactions may be an artifact from the use of statistical models that give priority to main effects. Partialing out main effect variance has differential effects on disordinal and ordinal interactions, and, as a consequence, ordinal interactions may simply not be statistically significant after main effect variance is removed.

Moreover, under some circumstances, apparent ordinality may simply reflect limitations on the representations of independent or dependent variables ( Cronbach and Snow, 1977 ). For example, consider the apparent ordinal interaction depicted in panel 3 of Figure 1 . For therapists with 2 years of experience, therapy modalities Y and Z do not differ in effectiveness, but for therapists with 4 years, Z is superior. But consider the dotted lines extending to the level of 0 experience, a level not tested in the experiment. ¹ The depiction suggests that the ordinal interaction actually observed would have been disordinal had different levels of the experience variable been included. For inexperienced therapists, Therapy Y would be a better choice. In fact, had only 0 and 2 years of experience been studied, one would have observed quite a different ordinal interaction with Y superior at the lowest level and Y and Z equal at the highest (2-year) level.

The interaction "results" displayed in Figure 1 also permit another observation: the pernicious potential of the inclination to connect two data points by a straight line. The line drawn in panel 4 almost cries out for the interpretation, "With increasing therapist experience, Z is increasingly the intervention to be recommended." That cry should be resisted for reasons portrayed in panel 4, where it is evident that lines of almost any sort may connect two data points. Nevertheless, a straight line is the best guess when only two data points are available. Therefore, anyone who intends to model growth and change should begin with a minimum of three data points. It may be that some ATIs can be measured only by nonlinear models.

A second way in which an Aptitude × Treatment interaction might be manifested, however, is in the relative efficiencies of two treatments. Whiskey is a "stronger" drink than beer, but that simply reflects the concentration of alcohol per fluid ounce. Beer is just as intoxicating as whiskey if one consumes enough of it. Similarly, two therapies could conceivably differ not in terms of the terminal effect but in the "dose" required to get there. One needs to be cautious in interpreting findings comparing 8 sessions of Therapy Y with 8 of Therapy Z. If Y takes 16 sessions to achieve the same effect of 8 Z sessions, that might make Z a preferable therapy (e.g., in terms of cost-effectiveness; Scott & Sechrest, in press ), but that is not an absolute difference between the outcomes of the two therapies. Moreover, psychotherapeutic processes are not necessarily affected by the rate of change. Therefore, some ATIs may reflect differences in the rate of change rather than a qualitative difference in process and outcome.

A third possibility is that Aptitude × Treatment interactions may be specific to the outcome measure(s) chosen. Therapy Z may produce more improvement in self-esteem than Y for female than male patients, but no difference may be observable on any other outcome measure. An obvious implication is that researchers should use multiple outcome measures, and interpreters and users of research should be cautious in inferring the existence of generalized Aptitude × Treatment interactions without considering the possibility of an Aptitude × Treatment × Outcome interaction. It should be noted here, however, that a recent attempt to identify different outcomes for different treatments of depression failed to find any Treatment × Outcome interactions ( Imber et al., 1990 ). This failure may have occurred because, although Imber et al. did enter client aptitudes into their analyses (level of depression), the interaction may not have been significant for a variety of other reasons, such as problems with statistical power, lack of measurement precision, or the simple possibility that there was no interaction effect.

What are the boundaries of what constitutes "treatment" for the purposes of discussing Aptitude × Treatment interactions? If we consider the entire panoply of interventions relevant to the domain of "psychotherapy," then it is manifestly absurd to think that there are no ATIs (e.g., see Lazarus, 1990 ). It is now widely conceded that behavioral intervention involving exposure to the stimulus is the treatment of choice for phobias; however, such interventions would be irrelevant at best for a wide range of other conditions. For example, it is doubtful that simply exposing depressed patients to depressing stimuli will make them any better. Matching different problems with differentially effective treatments automatically makes for an interaction. Behavioral interventions tend by their very nature to be tailored somewhat to the requirements of different problems so that their use assumes an ATI. The recommendation of breathing retraining for hyperventilation is so obvious an instance of an assumed ATI that it is not even interesting; no one would suppose that breathing retraining would be useful in treating bruxism or brutomania.

On the other hand, if psychotherapy has no specific effects, if it is all just a matter of the quality of the relationship, then the search for ATIs is similarly diminished in interest. The search would be reduced to the search for therapist—patient matching variables that would foster the development of a high quality therapeutic relationship. Requirements for matching therapists and patients are not of great practical interest because they are difficult to meet under most conditions of practice. Therapists in agencies are usually under pressure to deliver services and do not often have the luxury of declining to treat a person because of less than optimal matching. Therapists in independent practice are under similar pressures, albeit for somewhat different reasons.

We assume that the interest in ATIs reflects assumptions about strategic and tactical options potentially open to therapists (i.e., that if a therapist knew of ATIs, the therapist could capitalize on them by optimal behavior). Interest in ATIs assumes that therapists are capable of planful flexibility in their decisions about how to approach cases. If the ATI reflects differences in the effectiveness of two or more modalities of therapy (e.g., as appears to be the case for treatment of phobias), the ethical therapist has either to master multiple modalities or decline to treat a patient nonoptimally and refer the patient to a therapist competent in the better modality. In actuality, most therapists probably consider themselves generalists and try to treat almost every patient who enters their office. Thus, ATIs may prove to be more interesting in theory development than in actual practice.

A medical analogy can further elucidate the complexity of treatment decisions in the context of ATIs. A cardiologist would not attempt to treat a patient presenting with a sore knee but would refer the patient to an orthopedist (another modality). The orthopedist might conclude that the first challenge would be to reduce swelling and inflammation and might prescribe an anti-inflammatory drug, rather than a painkiller. That tactical decision would involve an ATI. The physician's overall initial strategy might be to rely on natural healing processes on the assumption that the problem was caused by a severe sprain with no critical tissue damage. That strategic approach would involve an assumed ATI. If the problem were caused by tearing of the anterior cruciate ligament, then natural healing processes would produce an unsatisfactory outcome, and surgery would be recommended. The ATI would be a Diagnosis or Problem × Treatment interaction.

The same orthopedist might be faced with another patient who could be considered for a total knee replacement. The doctor might decide that the patient's age and lifestyle would not justify such a radical procedure. With a young but not highly active patient, watchful waiting could be a better treatment choice if a few years of moderate disability were rewarded by improvements in prosthetic technology during the waiting period. That would be a Patient Characteristic × Treatment interaction. If alternative treatments for a damaged knee involved a trade-off between discomfort and reduced mobility, that would be a Treatment × Outcome interaction.

In order to exploit ATIs when they are discovered, it is essential to know exactly how the treatment works and what the mechanisms are. Any ATI is observed in a particular context of therapists, patients, problems, circumstances, and so on. If we are to know how the observed ATI is to be applied to some new set of conditions, then we must know exactly the nature of the interaction in the first place. Our current understanding of most psychotherapies and behavioral interventions is scarcely more sophisticated than would be represented by a description of a medical "treatment" as "some red pills." A pharmacotherapy is not regarded as completely satisfactory until its specific mode of action is understood (i.e., which chemical operating on what structure to produce what response). Acetyl salicylate (aspirin), for example, was a useful therapeutic for centuries, but the medical community was never completely comfortable with it until they began, recently, to understand its various modes of action.

When one considers that a century after the invention of psychotherapy major disagreements still exist about such a fundamental issue as whether there are any specific treatment effects (e.g., Lazarus, 1990 ; Strupp, 1989 ), one can believe that we are still at the early aspirin stages in our understanding. Even if ATIs were found, we would not be in a position to interpret them correctly and exploit them in other than the most direct empirical way. In general, as we will elaborate on later, we expect psychotherapy theorists to benefit more from ATI findings than will practicing therapists.

Common sense suggests that there should be at least some Aptitude × Treatment interactions (ATI) in psychotherapy. The concept of the ATI has received serious attention in the educational psychology literature (see Cronbach & Snow, 1977 ) and has been proposed as an important, if not essential, strategy for research in all areas of psychology (e.g., Cronbach, 1975 ). Nonetheless, very little systematic work on ATIs in psychotherapy has been done, and very few replicable ATIs have been reported.

An important reason ATI research has not become a reality is the fact that measuring and interpreting interaction effects is much more difficult than dealing with main effects. ATI research requires greater precision than general effects research. Rather than comparing general packages of treatment delivered to a broad class of patients, in ATI studies one needs to know precisely what it is about the patient that interacts with a precisely defined component of treatment. Interaction effects need to be shown to occur above and beyond the additive influence of main effects, and this requires studies with large sample sizes and at least four treatment cells. Thus, compared with the search for main effects, research on interactions requires better measurement, more subjects, a wider variety of conditions, and specific a priori hypotheses.

To complicate further the problems of ATI research, traditional statistics are relatively insensitive to interaction effects ( Wahlsten, 1990 ). Some of the statistical disadvantage in interaction research arises from the tendency of scientists to consider main effects before interaction effects. This preference is arbitrary, but it supposedly promotes parsimony and has become the accepted rule (actually it may be more parsimonious to propose one interaction instead of two main effects). Some consequences of the "parsimonious approach" are these: (a) Interactions have to be gleaned from the variance left over after main effects are removed, which reduces the likelihood that interactions will be significant; (b) the magnitudes of interaction effects may often be underestimated; and (c) the magnitudes of main effects are often overestimated. We are in general agreement with Dawes's (1988) point that interactions may not be very impressive after main effects are allowed for. But researchers who think interactions are important should understand the handicap they place themselves under in using analytical models that look for interaction effects only in the residuals from main effects predictions. Analyses of variance (ANOVAs) have that unhappy property, and the search for interaction effects might better be carried out by General Linear Models involving regression methods and individual growth curve analyses.

The search for ATIs is also impeded by the inclinations of many investigators to convert continuous measures to categorical measures, thereby sacrificing critical information. That inclination has probably been contributed to by a preference for ANOVA statistics and the custom of graphing interactions by the mean values of individual cells. What Cronbach and Snow (1977) refer to as "ATI regressions" are simply plots of regressions. These have the advantage of showing the shape of the entire function (e.g., revealing nonlinearity if it exists). Interactions will ordinarily be seen as differences in slopes of regression lines, although Cronbach and Snow note that interactions may occasionally affect variances. Not only are regression analyses likely to have greater power for detection of interactions ( Cohen & Cohen, 1983 ) but they are certain to be more sensitive to other features of the phenomena under study.

Few psychotherapy studies are planned to reveal interactions, and most of them have inadequate statistical power to detect interactions, especially given the preference researchers have shown for ANOVAs. Cronbach and Snow (1977) suggest, for example, that an ATI study with subjects assigned randomly to groups should have about 100 subjects per treatment, a sample size much larger than almost any psychotherapy studies. Blanchard, Appelbaum, Radnitz, Morrill, et al. (1990) , in what is a fairly representative example of therapy research, had only 116 cases to allocate to four treatment conditions. It is not surprising that they found no differences among the three active treatments, although two actually had an odds ratio of better than 1.7 for producing improvement when compared with the third. Another study of treatment of a small sample of elderly patients found no differences between treatments ( Thompson, Gallagher, & Breckenridge, 1987 ), although at the end of treatment "major depression still present" was twice as frequent in the cognitive as compared with the behavior therapy group. The possibility in such studies for detecting an interaction indicating that one of the treatments would be better than the others with a particular type of patient is virtually nil.

Unfortunately, power analyses are rarely reported for psychotherapy studies of any kind, an omission that is going to have to be corrected, but that is going to prove painful. Journal editors must begin to insist on properly done power analyses. Power analyses must be done before therapy studies are undertaken if they are to be useful. After the fact analyses permit capitalization on chance to a considerable, although unknown, degree because all the estimates must necessarily be considered biased.

With the exception of the above-mentioned statistical considerations and the minimum requirement that two treatments and two aptitudes must be compared, the difficulties of ATI research are basically similar to those faced in all types of psychotherapy research. Many of these issues are discussed else-where in great detail; however, it strengthens the purpose of this article to remind ATI researchers not to perpetuate problems that continue to plague psychotherapy research. As mentioned earlier, simply looking for ATIs will not automatically result in psychotherapy research successes. On the contrary, the ATI approach may exacerbate past problems that, if left uncorrected, could lead to dismal failure. Some of the more troublesome of these research liabilities are discussed later in this article.

Before discussing possible methodological problems that have made ATIs elusive, it is important to consider the possibility that ATIs do not exist, or at least that they are rare. Virtually every comprehensive analysis of psychotherapy outcomes (e.g., Smith, Glass, & Miller, 1980 ; Landman & Dawes, 1982 ; Luborsky, Singer, & Luborsky, 1975 ) ultimately concluded that type of therapy, experience of the therapist, credentials of the therapist, and so on are unrelated to outcome. If any client variables are consistently related to outcome of therapy, they generally support no more of a conclusion than that clients who are bright, verbal, motivated, and not so bad off in the first place tend to do better in therapy. Those are main effects, rather than interactions, and not very interesting ones. These conclusions have not, however, been much of a deterrent to speculation about ATIs. For example, an ATI reported by Jacobson, Follette, and Pagel (1986) for marital therapy was not only not particularly large in size but it was found only at immediate posttherapy measurement and not at follow-up. Talley, Strupp, and Morey (1990) found an interaction between therapists' tendencies toward affiliative or hostile response and patients' similar tendencies but only for therapist-rated improvement on a single item scale. Other interactions were reported, but they were similarly inconsistent across independent and dependent variables.

Despite the fairly consistently negative outcomes of the search for ATIs, the search is unabated. Why? One answer may lie in what Dawes (1979) calls "arguing from a vacuum." People have a strong tendency to believe and argue that if one desired solution does not work, it must be true that something else will. Psychotherapists would like to believe that therapist experience has a good bit to do with outcome, and when that cannot be shown, the response is to believe that the answer must lie in interactions. If type of therapy does not appear to be related to outcome of therapy, then the effect of type of therapy must lie in an interaction with other variables. All that remains is to ferret out and display the interaction.

In some important ways, the persistence of psychotherapy researchers in searching for ATIs resembles the error in thinking that Dawes (1988) identifies with the commitment to "sunk costs." So much effort has been expended in the attempt to find the "silver bullet" of psychotherapy that it is simply too painful to abandon all that investment, cut losses, and try something else. As the punch line of an old joke has it, "There must be a pony in there somewhere!"

To a metascientist the movement toward ATI research might be viewed as a symptom of a degenerating program of research. Programs can be said to be degenerating if they (a) fail to yield new predictions or empirical successes and/or (b) deal with empirical anomalies through ad hoc maneuvers that overcomplicate rather than clarify the problem of interest ( Gholson & Barker, 1985 ). Perhaps psychotherapy researchers should be seriously and dispassionately reconsidering the core assumptions of their theories rather than building an elaborate ATI structure on a crumbling theoretical foundation.

The disappointment over empirical outcomes notwithstanding, the case for interactions is, unfortunately, a priori discouraging. Here is why, as Dawes (1991) makes clear. If there is an interaction but no main effect for the treatment variable, then either the interaction must be very small in magnitude in relation to unexplained (error) variance or the interaction must be disordinal (as in panel 1, Figure 1 ). Now if an interaction involving, for example, therapist experience were disordinal, that would mean that experienced therapists were less helpful, and perhaps harmful, to some clients. Not only would such a conclusion seem unlikely on the face of it, but it would immediately plunge the field into serious ethical difficulties. It would clearly be unethical to assign a client to any form of intervention known to be suboptimal. Thus, many ATIs, if they exist at all, may be more confusing and troublesome than past failures to find strong main effects.

Historically, the possibility of making Type I errors has been more carefully guarded against than the chance of making Type II errors. Meanwhile, the possibility of making Type III errors has been virtually ignored. This oversight is serious because Type III errors override both Type I and Type II errors. Who cares if a hypothesis is erroneously accepted or rejected if the hypothesis is misspecified to begin with? Therefore, to minimize and control for Type I and Type II errors at the expense of Type III errors is a fallacy of misplaced precision ( Mitroff & Featheringham, 1974 ). Most of the issues relevant to controlling and minimizing Type III errors fall under the province of what Mahoney (1978) calls "theoretical validity," which he defines as the extent to which an experiment has some logical bearing on a specific hypothesis or theory. Unfortunately, owing to the proliferation of diverse paradigms for explaining and studying human psychopathology, it is difficult to reach agreement on the minimum standards for theoretical validity. Nevertheless, some aspects of theoretical validity may be less debatable than others. For instance, determining whether a hypothesis is clearly stated a priori is much less problematic than trying to decide whether the hypothesis is theoretically relevant to a certain paradigm. Thus, even though many issues regarding theoretical validity are entrenched in specific schools of psychotherapy, it should be possible to list several methodological issues related to theoretical validity that are ubiquitous across paradigms of psychotherapy.

Concerns over Type III errors pertain to all four of the most widely recognized types of experimental validity: internal, external, construct, and statistical conclusion validity ( Cook & Campbell, 1979 ). Of these four, statistical conclusion validity is probably the least affected by Type III errors. Nevertheless, calculations of statistical power may be dependent on theorybased estimates of treatment effect size ( Scott & Sechrest, 1989 ). When this is the case and the wrong theory is applied or the right theory is misapplied, then a Type III error can result in a problem with estimating statistical power. Thus, statistical conclusion validity might be affected by Type III errors, although threats to statistical conclusion validity do not appear to contribute to Type III errors. Nonetheless, it should be emphasized that statistical conclusion validity issues are, on other grounds, critical to study and interpretation of ATIs.

Proper concern with Type III errors increases as attention shifts to internal validity. The confidence with which one can assert that the outcome of an experiment is attributable to the intervention and to no other variables (i.e., internal validity) is not a purely objective deduction. Formulations of experimental problems, and as a result internal validity, depend to a large extent on the researcher's ability to conceive of and control for rival explanations of treatment effects. Determining whether an effect occurred is a relatively simple endeavor compared with the process of attributing the effect to the independent variable. It is easier to show that a treatment works than to explain how it works, and misattributing a cause to an effect is a Type III error. For example, Jacobson, Follette, and Pagel (1986) may have shown that behavioral marital therapy is more beneficial for egalitarian couples than for others (although only immediately posttherapy), but that finding is not necessarily easily explained; and simply stating that the outcome was some ATI related to the "egalitarian" qualities of the couple would be a mistake. In order to reduce problems with Type III errors, we need to know more about specific and unique qualities of concepts such as "egalitarian" and "behavioral marital therapy."

Problems associated with misunderstanding how a treatment works are especially distressing when one attempts to disseminate findings from an effective study, a shortcoming recognized by Jacobson et al. (1986) . The legitimacy of various generalizations of research findings across persons, places, and times (and other dimensions; see Cook, 1990 ) is the essence of external validity. Even though conclusive evaluation of external validity is based on empirical demonstrations of the replicability of treatment across different settings and subpopulations, most inferences about generalizability are based on theoretical interpretations of treatment. As a result, poor understanding of treatment variables can be expected to result in flawed generalization of treatment.

Out of Cook and Campbell's (1979) list of validities, construct validity is the most closely linked with Type III errors. Construct validity is completely subsumed by the concept of theoretical validity, although a few considerations pertinent to theoretical validity are not traditionally associated with construct validity (e.g., the issue of clinically vs. statistically significant change). We will not attempt to make sharp distinctions between theoretical and construct validity, and these terms may be used interchangeably, as they are in this article. However, we favor the expression "theoretical validity" because it emphasizes the importance of theory in psychotherapy research and promotes the notion that researchers should occasionally look beyond the four types of validity listed by Cook and Campbell (1979) .

Theoretical validity refers to the adequacy of our understanding of the experimental variable(s) being studied. If the variables are not operationally defined in a manner that clearly and completely represents the theoretical construct of interest, then the experiment can result in a Type III error. Likewise, if the variables are not accurately described (i.e., poorly reported), there is a risk that subsequent readers and researchers will commit Type III errors. In summary, any severe threat to construct or theoretical validity will almost certainly result in a Type III error, and these errors imply that the worth of the entire experiment is diminished, at least in terms of theoretical or practical meaning. Unfortunately, the psychotherapy literature is replete with instances of poor, and probably inaccurate, description of interventions. Type III errors (i.e., failing to understand what treatment really is) could be the primary reason why main effects research has been disappointing. If these errors are left uncorrected, Type III errors could be an even more formidable barrier to ATI research.

A final note on the theoretical validity of treatment concerns the importance of the timing of formulations of theoretical explanations. Hypotheses that are formulated before the experiment (i.e., a priori hypotheses) are presumably tested by the experiment. Hypotheses generated after inspection of the data (i.e., a posteriori hypotheses) are actually untested and may represent little more than speculation. Thus, accepting a posteriori hypotheses as fact can result in Type III errors. An example is provided by the National Institute of Mental Health (NIMH) Depression Collaborative Research Program, for which a recent analysis ( Elkin et al., 1989 ) appeared to show a reasonable, but unanticipated, ATI. More severely depressed patients got greater benefit from drug treatment than from other treatments, although no greater benefit from drugs existed for less severely depressed patients. This ordinal ATI would have been considerably more persuasive had it been predicted in advance, because compared with hindsight, a priori prediction suggests a greater understanding of the topic of interest.

One possibility is that more studies can be carried out collaboratively. Successful completion of collaborative studies is not easy, but it is wasteful of money and effort to do studies that will not accomplish their aims. It should be recognized, however, that power to detect effects is not solely a matter of sample size (see also Higgenbotham, West, & Forsyth, 1988 ). It may be that for some effects the reduction of alpha from .05 to .10 would be justified; too much emphasis is placed on statistical significance in any case ( Cohen, 1990 ). A second determinant of statistical power is the effect size anticipated, which may depend heavily on the strength of the intervention. Strength of intervention is often under the control of the investigative team, and treatment should be planned to be strong to begin with, and lapses in integrity of treatment should be protected against ( Sechrest & Redner, 1979 ). Investigators should also, if they value and predict interactions, think of giving them priority over main effects in their statistical models. Finally, statistical power is a function in part of the size of the error term. The magnitude of experimental error is also very often under control of the investigative team. Experimental error can be reduced by decreasing heterogeneity in the sample, by better measurement procedures, by greater precision in conducting the experiment, and other maneuvers ( Sechrest & Yeaton, 1981b ).

We believe strongly that if psychotherapeutic interventions are going to be improved substantially, and particularly if that improvement is to be derived from ATIs, better theory is going to be required, and that theory will have to have its basis in fundamental psychological research. Twenty-five years ago, Goldstein, Heller, and Sechrest (1966) proposed that psychotherapy should be, first and foremost, psychological, and that meant grounding theory and practice in the basic concepts and findings of the field. They did not believe that much progress could be made by trying to develop what would be a separate, isolated discipline of psychotherapy. Moreover, the three authors demonstrated by systematic reviews of research literature that hypotheses important to the psychotherapeutic enterprise could be derived from more basic research in the field. Despite a good bit of assent from others at the time, even some acclaim, the main thesis of Goldstein, Heller, and Sechrest has been, we think, largely ignored. Some of their ideas have been realized to some extent (e.g., the importance of generalization as an aspect of psychotherapy), but for the most part current literature on psychotherapy appears to pay scant attention to research on basic behavioral and conceptual processes. One cannot determine that, for example, cognitive therapy owes any more than the most general debts to cognitive psychology. If cognitive therapy is to develop and improve, one would think that it ought to take account of what cognitive psychologists are learning about cognition. A recent volume, Psychotherapy and Behavior Change ( Higgenbotham et al., 1988 ), which is a descendent of Goldstein, Heller, and Sechrest, provides contemporary instances of the need to apply more basic research in developing thinking and research about psychotherapy. We would observe, however, that the material reviewed and the hypotheses developed tend strongly to suggest that any major improvements in psychotherapy are more likely to be in the form of main effects than ATIs.

When testing psychological theories, researchers need to determine if their operationalization of the theory is accurate and representative of the problem of interest. Theoretical validity problems need to be subjected to the rigorous methodology of the multitrait—multimethod approach ( Campbell & Fiske, 1959 ), which has been largely ignored in psychotherapy research. This approach attempts to get at the true meaning of a measure (the construct "true score") and, at the same time, to identify and attempt to neutralize the effect of any systematic bias introduced by measurement and analytic strategies. Thanks to modern statistical procedures such as structural modeling researchers are now in the position to undertake theory-driven determinations of convergent and divergent validity of experimental measures.

Once the underlying traits of experimental measures are determined, dependent variables (DVs) can be chosen that should reveal the operation of the theorized mechanisms of change. For example, if a treatment for drug users focuses on self-concept because improved self-esteem is thought to result in better recovery, then treatment evaluation should use measures of self-esteem. Assuming that the DVs are perfectly psychometrically sound (which is rarely the case), if no reliable differences are recorded for these DVs, then the construct validity of the treatment should be questioned. In the drug abuser self-esteem example, failure to observe changes in self-esteem threatens the construct validity of treatment, at least to the extent that self-esteem is critical to the overall treatment theory. The methodology of recording changes in hypothesized mediating variables has been called "manipulation checks" ( Aronson & Carlsmith, 1968 ). In our opinion, if the psychotherapy research paper does not report manipulation checks, then the reader should be skeptical about the construct validity of treatment.

Manipulation checks have also been shown to be useful in studying the believability of alternative treatments, especially those intended as placebos ( Dush, 1987 ). If, as is usually assumed, placebos have their effect through arousal of expectancies for change, if they do not arouse those expectancies, they are not good placebos. Manipulation checks can be used to determine whether alternative treatments arouse similar expectancies, are equally believable, and so on. Alternatively, manipulation checks may reveal that alternative, nonspecific treatments were more "active" than they were intended to be. An example is provided by Blanchard, Appelbaum, Radnitz, Michultka, et al. (1990) , who found that the placebo condition probably functioned much like a relaxation treatment. Elkin et al. (1989) also noted that the placebo condition in the National Depression Collaborative Research Program was not inactive.

We also think that manipulation checks should be more generally used to assess the nature of therapy as perceived by patients. For example, the National Depression Collaborative Research Program intended to compare the effects of interpersonal and cognitive behavioral therapies, for which extensive manuals were prepared in order to achieve uniformity in implementation of treatment. We are not, however, aware of any attempts to determine whether patients experienced the two therapies in any different ways (e.g., whether patients in cognitive behavior therapy viewed their therapists or therapy in any way different from the views of interpersonal therapy patients). This type of information is crucial to ATI research because patient aptitudes may have a major effect on perception of treatment and subsequent outcome. Unfortunately, with the exception of detailed process analyses on the role of patient expectancy in psychotherapy (e.g., Elliott, 1986 ), there appears to be very little information about the phenomenology of patienthood.

Dawes (1991) also makes the point that the fallback position of asserting that it is the importance of "the relationship" or the "therapeutic alliance" is weak unless we have measures of the relationship or alliance that predict outcome. Even then, the causal connection of the quality of the therapeutic alliance to outcome may be uncertain. The therapeutic alliance is likely to grow stronger when things are going well, and the perception of improvement (e.g., in symptoms) might well lag behind the perception of the quality of the alliance without any necessary inference of a causal relationship. In recovering from medical illnesses, people often report feeling much better or even "well" long before their relevant biomedical parameters have shown much change.

A critical parameter in the understanding of any intervention, including psychotherapy, is the strength of the treatment. Strength is viewed here as a close analog to the strength of a drug treatment. A strong treatment of aspirin would be 10 mg; a weak treatment would be 2 mg. What would "strong" psychotherapy look like? We do not know. We can, however, conjecture as follows, allowing our description to reflect rather more common sense and consensus than actual empirical evidence. Strong psychotherapy might include