DOCUMENT RESUME

ED 128 377 TM 005 499

AUTHOR Garvick, GeoffreyTITLE Program Evaluation Project Report, 1969-1973. Chapter

Five: A Construct Validity Overview of GoalAttainment Scaling.

INSTITUTION Program Evaluation Resource Center, Minneapolis,Minn.

SPONS AGTACr National Inst. of Mental Health (DHEW), Rockville,Md. Div. of Mental Health Services Program.

PUB DATE Jun 74GRANT NIMH-5-R01-1678904NOTE 21p.; For related documents, see TM 005 495-501AVAILABLE FROM Program Evaluation Project, 501 Park Ave. South,

Minneapolis, Minnesota 55415 ($1.00)

EDRS PRICE MF-$0.83 HC-$1.67 Plus Postage.DESCRIPTORS Evaluation Methods; *Goal Orientation; *Measurement

Techniques; *Mental Health Programs; *ProgramEvaluation; Scores; *Validity

IDENTIFIERS *Goal Attainment Scaling

ABSTRACTThe P.E.P. Report 1969-1973 focuses on the various

findings and activities of the Program Evaluation Project. Theestablishment of validity is one of the major tasks of the developersof a measurement methodology. In this chapter, it is argued that theconstruct validity approach is essential to an understanding of thevalidity of Goal Attainment Scaling, since there are no clearcriteria available for concurrent validation. The chapter discussesdata from a variety of studies on Goal Attainment Scaling in aneffort to illustrate various facets of the construct validityapproach when applied to the methodology. The findings underscore theidea that Goal Attainment Scaling can be applied in a variety ofsettings. (Author/RC)

************************************************************************ Documents acquired by ERIC include many informal unpublished ** materials not available from other sources. ERIC makes every effort ** to obtain the best copy available. Nevertheless, items of marginal *

reproducibility are often encountered and this affects the quality *af the microfiche and hardcopy reproductions ERIC makes available *via the ERIC Document Reproduction Service (EDRS) . EDRS is not *

* responsible for the quality of the original document. Reproductions ** supplied by EDRS are the best that can be made from the original. ************************************************************************

A CONSTRUCT VALI DI TY OVERV I EW

N.\ OF GOAL ATTA INVENT SCALI NG

CO(NJr-1

CHAPTER FI VE:

uJ

A REPORT ON FOUR YEARS OF

STAFF EFFORT AT Ti-iE PROGRAM

EVALUAT I ON PROJECT

CHAPTER FIVE

Program Evaluation Project Report, 1969-1973

A CONSTRUCT VALIDITY OVERVIEW OF GOAL ATTAINMENT SCALING

Geoffrey Garwick

June, 1974

Thomas J. Kiresuk, Ph.D., DirectorProgram Evaluation Project

501 Park Avenue SouthMinneapolis, Minnesota 55415

Developed Ldder Grant #5 RO1 1678904, National Institute ofMental Health, Department of Health, Education, and Welfare

Acknowlegements are given to all of the Program Evaluation Project Staff,especially Donna Audette, James Baxter, Joan Dreyer, Susan Jones andLaurence Kivens for his valuable editorial comments. Dr. Robert Shermangave helpful advice and Dr. David Weiss contributed subtantial improve-ments to the chapter.

TABLE OF CONTENTS

General Introdu tion to the P.E.P. Report 1959-1973

Synposis

Introduction

I. Variables Relevant to Goal Attainment ScalingConstruct Validity

II. Generating and Representing Hypotheses aboutGoal Attainment Scaling

Validity Measures for Goal Attainment Scaling

Conclusion

Program Evaluation Project Staff Listing

Page Number

1

2

3

5

7

8

14

For further information, please contact Ms. Joan Brintnall, Program EvaluationProject, 501 Park Avenue South, Minneapolis, Minnesota 55414.

4

As of January, 1974, the Program Evaluation Project is funded by a three year collaborative grant

tith the Mental Health Services Division of the National Institute of Mental Health. The purpose of the

rant is to emphasize the coordination and dissemination of informtion on a variety of program evaluation

ethodologies. Currently, it is expected that the title of the organization will be changed to the Program

valuation Resource Center during 1974.

Further information on the Goal Attainment Scaling methodology and program evaluation is available in

ther written and recorded materials from the Program Evaluation Project office. Chapter One, "Basic Goal

ttainment Scaling Procedures", Chapter Three, "An Introduction to Reliability and the Goal Attainment

caling Methodology", and Chapter Nine, "Evaluation of the Adult Outpatient Program, Hennepin County Mental

ealth Service" of the P.E.P. Report 1969-1973 are now available. Additional chapters will be released

lis year as they are completed.

1

5

SYNOPSIS FOR CHAPTER FIVE

A CONSTRUCT VALIDITY OVERVIEW OF GOAL ATTAINMENT SCALING

PURPOSE: The establishment of validity is one of the major tasks of the developers of a measurementmethodology. In this chapter, it is argued that t! construct validity approach s e sential to an under-standing of the validity of Goal Attainment Scaling, since there are no clear criteria available for con-current validation.

The chapter discusses data from a variety of studies on Goal Attainment Scaling in an effort to il-lustrate various facets of the construct validity approach when applied to the methi.dology. The follow-ing chapter presents the results of one particular study of the validity of Goal Attainment Scaling.

MAJOR FINDINGS: It is emphasized here that there have been many approaches to validity, but the Cronbach-Feehl concept of construct validity seems to be the most inclusive. If there is a basic construct under-lying Goal Attainment Scaling it is the "attainment of expectations", but this major construct is ac-companied by many other variables related to the many possible ways in which Goal Attainment Scaling canbe applied.

list of these accompanying variables and a system for illustrating hypotheses, about Goal AttainmentScaling are presented. This system is utilized to present several findings, the clearest being that aspredicted, the Goal Attainment score is not significantly related to client characteristics such as age,sex, education, marital status or intelligence. In one study of adult outpatients and day treatment cases,Goal Attainment scores based o- therapist scoring were correlated from a .58 to .84 with two questions ofglobal ratings of treatment outcomes answered by the therapists. The correlations of the Goal Attainmentscore with the consumer satisfaction Index was .23 for one group with correlations for individual consumersatisfaction items ranging from -.12 to .46. The Goal Attainment score was shown to be correlated .31with predictive accuracy for one group of adult outpatients. In general, correlations with other measures,as expected, are positive, but low to moderate, with early results from the drug effectiveness study, forexample, where all concurrent validity coefficients were .52 or less.

In general, the original Kiresuk-Sherman hypotheses about Goal Attvinment Scaling are supported. Thefindings underscore the idea that Goal Attainment Scaling can be applied in a variety of settings.

6

2

The two central concepts in the analysis ofmeasurement systems have historically been valid-ity and reliability. The reliability of Goal At-tainment Scaling is discussed at some length inChapters Three and Four of the P.E.P. Report 1969-1973.

This chapter presents suggestions for approach-ing the study of the "validity" of Goal AttainmentScaling. The Introduction is an attempt to linktraditional psychometric comments on validity,especially construct validity, to the specialcharacteristics of Goal Attainment Scaling.Section I is an outline of the variables avail-able for validity studies of Goal Atta'imentScaling by the Program Evaluation Project staff.Section II discusses briefly the generation ofhypotheses related to Goal Attainment Scaling.Section III shows Program Evaluation Projectdata applied to issues in Goal AttainmentScaling validity.

Introduction

As Ebel (1961) comments, validity is an im-precise term due to "logical and operationallimitations of the concepts of validity itself".He argues that "...faster progress will be madetoward more educational and psychological testsif validity is given a much more specific and re-stricted definition than is usually the case,and if it is no longer regarded as the supremelyimportant quality of every mental test".The classic statement on validity is Lindquist's(1942) observation that "the validity of a testmay be defined as the accuracy with which itmeasures that which it 's intended to measureor as the degree to which it approaches infalli-bility in measuring what it purports to measure".Loevinger (1957) argues that "this definition istoo vague, too remote from factual measuringoperations, to be useful ..." Ebel also arguesthat this definition is too general and notparticularly useful. He proposes that thesepoints of test "quality" be examined inst2ad:

1. The importance of the inferences thatcan be made from the test scores.

2. The meaningfulness of the test scores,based on:

a. An operational definition of themeasurement procedure.

b. A knowledge of the relevance ofthe scores to other measures, from,

i. Validity coefficients, pre-dictive and concurrent.

Other correlation coefficientsor measures of relationship.

3

c. A good estimate of the reliability ofthe scores.

d. Appropriate norms of examinee perform-ance.

3. The convenience of the test in use.

Ebel concludes that these concepts answer theprimary questions about the utility of a test, andthat points 2a, 2b-i, and 2b-ii need not be estab-lished for all tests.

A more stringent approach is presented byRaymond Cattell (1964) who argues that some verymathematically inclined "psychometricians have some-times seemed lost in their labyrinthine fastnessesfrom logic, from common sense, and certainly frompsychological perspective". He suggests that valid-ity is "...the capacity of a test to predict somespecified behavioral measure (or set of measures)other than itself". This use c "predict" could betaken, for the purposes of outcome evaluative mea-sures, in the very broadest sense, such as "a highscore for treatment X on measure Y for group Q pre-dicts a high outcome for other groups similar togroup Q who receive treatment X". In any case, Cat-tell specifies three parameters of validity:

1. "Degree of Abstraction of the ReferentCriterion." This parameter might also becalled, the generalizability of results,with one extreme being a very concrete test,such as "ability to clean armadillos quick-ly", and the other extreme being a very con-ceptual test, such as "creativity".

2. "Degree of Naturalness of the Criterion."This parameter varies from a critr inbased on a "Natural" situation su_ uscoping with a real disaster, to correlationwith an "Artificial" criterion situation,such as another test.

3. "Degree of Directness of Validation." Thisparameter varies from "direct" or completecorrelation with the criterion to "iAirnctor circumstantial" correlation with thecriterion. The directress refers to thepatterns by which variables are correlated.For example, two measures could both becorrelated .3 wtth criterion X, and yet mea-sure #1 could correlate with only one aspectof criterion X, whereas measure #2 could re-late to an entirely different aspect of cri-terion X.

After introducing these three parameters, Cat-tell discusses some of the concepts frequently called"validity". He maintains that there are a group of"utility coefficients" which are not validity co-efficients at all and includes "face validity", "con-tent validity" and "semantic validity" in this groupof non-validity concepts. Mosier (1947), writingmuch earlier, also recommends that "face validity" bedropped from the vocabulary because it has too many

confusing and vague definitions. Cattell (1964)appears to see the "...integration of psychomet-rics with personality theory and general psycho-logical theory..." as an ultimate goal for im-proving validity conceptualizations, in much thesame way that Loevinger (1957) stresses it. In

fact, Loevinger maintains that "content validityis established by the judgment of the investigatorthat the items are valid; it is thus also contin-gent upon a special, non-generalizable circum-stance, to wit, the particular investigator....Since ad hoc arguments are scientifically of minormportance, if not actually inadmissable, what isleft, construct validity, is the whole of the sub-ject of a systematic scientific point of view".

The viewpoints presented above reveal therange in theoretical thinking about validity.In this discussion, construct validity isutilized as a basis for the presentation, sinceit allows for flexibility and subsumes mall!earlier "forms" of validity.

This approach to the validity issue is basedon Cronbach and Meehl's (1955) version of the con-cept of "construct validity", which they added toearlier "types" of validity: such as concurrent,content, and predictive. It is very germane toGoal Attainment Scaling theory that Cronbach andMeehl observe (evidently citing Gaylord), "Whenan investigator believes that no criterionavailable to him is fully valid, he perforce be-comes interested in construct validity becausethis is the only way to avoid the infinitefrustration of relating every criterion to somemore ultimate standard... Construct validitymust be investigated whenever no criterion oruniverse of content is accepted as entirelyadequate to define the quality to be measured."

This lack of a clear-cut criterion for com-parison is a central issue in the discussion ofGoal Attainment Scaling validation. There is noclear-cut criterion for either mental health ortherapy effectiveness in mental health. With theuse of Goal Attainment Scaling, in effect, a newcriterion for mental health treatment is selected,that is, "the degree to which expectations fortreatment are achieVed." The expectations may bedeveloped by the clinicians, the clients, otherpersons, or some combination of involved peo'ons.Since this form of criterion is so new, no im-mediately applicable standard for comparison withGoal Attainment Scaling has been located. Al-though some instruments may be suitable forexamining certain aspects of Goal AttainmentScaling, none have been identified which are ascomprehensive or "expectation-oriented" asGoal Attainment Scaling.

If a clinician developed the expectations asappearing on a particular follow-up guide, theresult could be taken theoretically as a one per-son sample of the expectations for treatment whichwould be imposed by the hypothetical average clin-ician. In any case, as noted above, the use ofindividualized expectations as a criterion is sounusual that there are no -asily applicable con-

4

cuiTent measures and a construct validity approachappears essential.

The construct, for Meehl and Cronbach, is"...some postulated attribute of people assumedto be reflected in test performance.... A con-struct has certain associated meanings carriedin statements of this general character: Per-sons who possess this attribute will, in situ-ation X, act in manner Y (with a stated prob-ability)." It is noteworthy that, even in thisvery pertinent article, the language of psycho-metrics is not well-adapted to outcome evaluationin general, or Goal Attainment Scaling in particular.For pv-poses of the discussion of Goal AttainmentScaling, the above phrase would have to be changed toread, perhaps, "Agencies, treatment modes, or personswho possess this attribute will, in situation X, actin manner Y or will have acted in manner Y (with astated probabilifir.'

Despite their relatively straightforward def-inition of a "construct" the authors emphasize thatthere is no single, simple coefficient of constructvalidity, but that construct validity must be thoughtof in terms of a "nomological net" of constructslinked by testable hypotheses. Meehl and Cronbach(1955) delineate construct validity by a series ofaxioms:

1. Constructs of varying degrees of defini-tiveness are defined by a network ofpropositions.

2. There must be predicted relationshipsamong variables.

3. The network must be explicit.

4. Many types of evidence are relevant toconstruct validity and both high and lowcorrelations may be useful evidence forthe proposed nomological net.

5. When a predicted relationship fails to beobserved, the network of constructs mustbe redefined.

6. There is no simple coefficient of constructvalidity.

7. General scientific )rocedures are used.

Cronbach and Meehl's discussion, like that ofmost of the authors mentioned before, is based large-ly on the concept of the trait or characteristic. Anevaluation technique, however, is not usually aimedat measuring a trait of a person, but rather at mea-suring a pattern of -hanges Jr effects in an agency(which may be measured either for the agency as awhole or through the sum of effects on a number ofpersons). Wiggins (1973) describes a "trait" as"...a hypothetical construct which provides an organ-izing principle for relating a variety of superfi-cially dissimilar behaviors under a single dispo-sitional unit". If there is a trait underlying theGoal Attainment score, it would be "the tendency toattain expectations" as noted above. The Goal

Attainment expectations could be set by one moreof a number of sources, such as the client, thetherapist, and so on. Clearly, even if Goal At-tainment Scaling can be linked to trait concepts,the methodology demands a loose application ofmany validity ideas.

These experts on psychometrics have been citedabove to suggest the flexibility of the "validity"concept. Possibly the most basic characteristicof validity is that, as Nunnally (1967) says,"Validity is a matter of degree rather than anall-or-none property, and validation is an unendingprocess.... Strictly speaking, one lidates nota measuring instrument, but rather some use towhich the instrument is put." Nunnally concludesthat, "... sufficient evidence for construct valid-ity is that the supposed measures on the instrument... behave as expected".

This general commentary on validity has beenused to introduce a few construct-oriented guide-lines. Possible variables related to the "attain-ment of expectations" construct underlying GoalAttainment Scaling are outlined in the first twosections. Then in the third section, findings onvalidity will be summarized for Goal AttainmentScaling.

I. Variables Relevant to Goal Attainment ScalingConstruct Validity

As observed above, Goal Attainment Scaling isnot a typical "trait-type" problem in validity.Instead, the methodology involves a collection ofcharacteristics and activities which can be inter-connected by predictions and hypotheses. It isthe utility of Goal Attainment Scaling in evalu-ation methodology which must be examined in thecontest of a network of constructs.

Ebel's three points related to an instrument's"quality" or validity wem presented in the firstpage of this discussion. His divisions of"quality" seem appropriate to the considerationsof the practical issues of validating an outcome-oriented methodology like Goal Attainment Scaling.Thus, in the third part of this chapter whereconstruct validity issues are empirically examined,Ebel's outline will be followed as a matter ofconvenience.

This presentation is not intended to be acontribution to the reliability versus validitycontroversy. Ebel (1961), as mentioned previously,includes reliability as a basic factor in test"quality". Nunnally (1967) states that " ... con-sistency is a necessary but not suffiCient con-dition for construct validity". Therefore, re-liability is subsumed in part of the followingdiscussion, although a more thorough descriptionof these reliability results is presented inChapters Three and Four of the P.E.P. Report 1969-1973.

In summary, the validity issues for this dis-cussion of Goal Attainment Scaling orientedto a construct validity perspective in whichbasic questions are asked about the methodology'sutility and measurement properties in the pro-gram evaluation context. (The effectiveness ofutilizing Goal Attainment Scaling as part of thetherapeutic process is not within the scope ofthis chapter, except in reference to specificmeasurement studies.) This perspective may in-clude correlations of the Goal Attainment scoreswith criteria, a procedure often called "concurrentvalidity", or content analyses which could be re-lated to the so-called "content validity", but thcbasic thrust of the discussion is to examine thenetwork of relationships between constructs, theGoal Attainment scores and other variables.Anastasi (1970) comments bluntly, however, that"...content, criterion-oriented, and constructvalidity do not correspond to distinct or logical-ly coordinate categories. On the contrary, con-struct validity is a comprehensive concept, whichincludes the other types."

The primary construct being examined is "out-come" or "attainment of expectations". "Outcome"could actually be considered a collection of dif-ferent constructs. For example, "outcome after onemonth of treatment" is certainly a different setof expectations, and a different construct, withdifferent postulated attributes, different predic-tions, and a different meaning than outcome at sixmonths."

The basic components of Goal Attainment Scalingprocedures may be considered in terms of character-istics and activities. Characteristics are attributesof the persons involved in the Goal Attainment Scalingsituations and activities are considered to be be-haviors or procedures of the Goal Attainment Scalingprocess. Both aspects are represented here by thedata available from Program Evaluation Project data,some of which is incomplete.

Characteristics

Al. Of the persons (or agency) being repre-sented on the Goal Attainment Follow-upGuide.

A2. Of the person(s) constructing the GoalAttainment Follow-up Guide.

A3. Of the person(s) scoring the Goal At-tainment Follow-up Guide.

Activities

Bl. The rules and procedures used to constructthe Goal Attainment Follow-up Guide.

9

32. The treatment being used in the agency.

B3. The way in which the Goal Attainmentresults are expressed.

B4. The way the Goal Attainment Scaling re-sults are used.

Here are some more specific variables includedunder the seven major components mentioned above:

Al. The person represented on the follow-upguide. (If organizations were being rep-resented on the follow-up guide, theircharacteristics would be listed here, suchas size, income, available evaluative re-sources and so on.)

a. Truthfulness, completeness and abil-ity to communicate and/or predict.

b. Age, sex, intelligence, and othervariables.

c. History of treatment.

d. Problems presented and diagnosis.

A2. The person constructing the Goal Attain-ment Follow-up Guide.

a. Who is it? the client, a spouse ofthe client, a relative, a clini-cian(s), some combination of theabove persons, or others?

b. How experienced is the constructorat Goal Attainment Scaling and hasthe corstructor been trained inGoal Attainment Scaling?

c. How accurate is the constructor'sability to predict outcome?

d. Personality and demographic character-istics of the constructor and/or ed-ucational or discipline background.

A3. The person scoring the Goal AttainmentFollow-up Guide.

a. Who is it? the client, the personconstructing the follow-up guide, theperson giving the treatment or a sep-arate person?

b. How experienced and skilled is theperson at scoring the Goal AttainmentFollow-up Guide?

c. Personality, educational, demographicand discipline characteristics of thescorer. (If other than the client?)

d. How skilled at interviewing is theperson scoring?

Bl. Rules and procedures used to constructthe Goal Attainment Follow-up Guide.

a. What form of the Goal AttainmentFollow-up Guide is used?

b. What rules of construction are usedand are the follow-up guides re-checked to see if the rules are met?

6

c. Is the Goal Attainment Follow-up Guideaimed at 'a specific date in the futureand hdw far off is that date?

d. Is the level at intake used?

e. Can scales be differently weighted?

f. Is the Goal Attainmeat Scaling semi-standardized, standardized or com-pletely idiosyncratic?

. g. Is the treatment to bethe persnn who will beto the follow-up guide

82. The treatment being used.

received and/ortreating knownconstructor?

a. What form of treatment is being usedand what is available?

b. What rules of treatment choice (e.g.random) are used and s their usemonitored?

c. Is the treatment limited in time andhow long does it last?

d. Is the Goal Attainment Follow-up Guideavailable to the treater or used as partpart of treatment?

e. Can treatments be changed during theclient/treater interaction and howoften are they changed in practice?

f. Is there any limit on the type or num-ber of problems to be treated?

83. The type of measure used to express theGoal Attainment results.

a. Goal Attainment sco,-e mean (depends onfollow-up results).

i. Kiresuk-Sherman Goal Attainmentscore (varies from 20 to 80).

Scale-by-scale Goal Attainmentscore, (varies from -2 to +2)for either an individual scaleor the mean for an entire follow-up guide.

b. Goal Attainment score variance (dependson follow-up results).

c. Change score (depends on follow-upresults and whether the initial statusof the client is noted on the follow-up guide).

d. Predictive accuracy also called Mean In-accuracy Score (depends on follow-upresults).

e. Contents or types of problems includedon the Goal Attainment Follow-up Guide.

f. Reaching arbitrarily established levels,regardless of expectations or change.

g. Are rules and procedu-es established forthe follow-up scorer?

h. How often are scales unscoreable andhow ccnfident is the follow-up scorerin the amiracy of his score?

B4. The way the Goal Attainment score resultsare used.

a. Who receives the results, supervisors,no one, clients, treaters, legislators?

b. What are the consequences of the re-sults, i.e., information, salary,employment, advancement, publicity,peer pressure, etc.?

How long will the results be receivedand how many times or how often?

d. Will the results be used in conjunc-tion with Jifferent measures, i.e.,costs?

This array of variables illustrates the immen-sity of the validation task for Goal AttainmentScaling, and this list is not necessarily complete.There are thirty-seven components listed here, someof which contain multiple variables.

In addition to the three characteristics andfour activities listed previously, a related char-acteristic which should be discussed here is theeffect of the experimental design in which GoalAttainment Scaling is being applied. It should beunderstood that tioal Attainment Scaling is a mea-surement tool which can be utilized within a rangeof desired degrees of formal scientific or experi-mental procedures, such as control groups, randomassignment, impartial interviewers, etc. Such ex-perimental procedures are part of the total matrixof the utilization of Goal'Attainment Scaling butare not necessarily included in the list of GoalAttainment Scaling activities and characteristics.

Other variables are available for some pur-poses as partial comparative criteria. These in-clude a variety of commonly utilized outcome cor-relates and are other methods for measuring whatis usually assumed to be some segments of "out-come". Goal Attainment Scaling is designed to bemore comprehensive and more sensitive to clientoutcome than such measures (Kiresuk and Sherman,1968). Available criteria include:

I. Minnesota Multiphasic Personality Inven-tory (MMPI) and other personality measures.

2. Intelligence Quotient (IQ) results.

3. Brief Psychiatric Rating Scale.

4. Self-Rating Symptom Scale.

5. Consumer satisfaction scores.

7

6. Therapist ratings of global improvement.

7. Differences among groups receiving dif-ferent treatments.

8. Taylor Manifest Anxiety Scale.

These eight potential criteria, plus the thirty-seven components of Goal Attainment Scaling present-ed above, total at least forty-five components forthe study of Goal Attainment Scaling constructvalidity. Obviously, many other criteria could beutilized, further expanding the list of componentswhich could be examined.

II. Generating and Representing Hypotheses aboutGoal Atta:nment Scaling

Clearly, not all the possible relationshipsamong these activity and characteristic compo-nents of Goal Attainment Scaling can be repre-sented, but a cmss section of hypothetical linkscan be suggested for some of the more importantcomponents. Predicted relationships among thecomponents will be represented by arrows (44.40)with a superscript of "high" for a predicted cor-relation of over (.70), "moderate" for a predictedcorrelation of (.40) to (.69), "low" for a pre-dicted correlation of (.39) to (.15) and "no"for a correlation of (-.14) to (.14). Positivecorrelations are "+" and negative correlationsare "-", "Crit" refers to one of the eight cri-terion variables, but other codes refer to theGoal Attainment Scaling components listed pre-viously.

Construct and variable relationships can behypothesized and represented for Goal AttainmentScaling theory with this system of activities,characteristics, and criteria. One of the dif-ficulties with the construct validity approach,however, is that standards for the number andnature of the links required in the nomologicalnetwork have not been clearly established. Inaddition, since it could be assumed that changeis expected over time for "outcome" measures, thereare actually different constructs with differentsets of variable relationships when there are dif-ferent lengths of time between follow-up guide con-struction and follow-up scoring. For example, al-though age would not be expected to be associatedwith the outcome construct of Goal Attainment ifthe follow-up is completed inside a year or so, agecould have some effect if the follow-ups were com-pleted only after five years and either childrenor elderly persons were included as Figure I illus-trates.

Blctime ofover twoears

FIGURE I

These and other theoretical considerations implythat any summary comments about the constructvalidity and Goal Attainment Scaling should becautious.

The hypotheses associated with Goal Attain-ment Scaling are still fairly rudimentary andonly a few have been tested extensively. Themost firmly established hypothesis is that GoalAttainment scores are relatively uiresponsiveto the demographic characteris:ics of either in-dividuals or groups of clients. This lack of re-lationship was predicted because the very pro-cess of setting the Goal Attainment Scalinglevels is based on developing expectations whichshould allow for the unique features, demographicor otherwise, of the client. The evidence inrelation to other hypotheses, as discussed below,is somewhat sparser. The collection and analysisof data relevant to the hypotheses contivues.

The original Kiresuk and Sherman article of1968 predicted that the Goal Attainment scoreshould have a low to moderate correlation witnalready existing outcome measures. Since theGoal Attainment score is 1) based on an indivudal-ized measurement system, it sheuld not have ahigh correlation with non-individualized measures,and 2) since it is specific and goal-oriented, itshould differ from global or change-orientedmeasures even if they are individualized.

Sections of the Hypthetical Nomological Net forGoal Attainment Scaling.

FIGURE Ila:

A2d(Characteristics of fol-

134c

)(feedback

low-up guide constructor)

requencylow to moderate /

No Nol\s\\\*low

e3.BID

(GA score (GA scoremean

outcome10.

Novariable)

low}..er

-SO _moderate

lo

availabil)tyof follow-upguide during

(contents

`skp guide)

e follow-tn

Figure Ila illustrates several hypotheses thatcotod be tested. For example, the part of the dia-gram in the upper left quadrant of Figure IIa ad-vances the hypothesis that the Goal Attainment scoremean, as a representative oF the "outcome" construct,(83a), shou1d have a correlation with criterion 6.

Figure IIb and lIc are also representations ofinter ,ariable relationships. Figure lIc suggests

that the mean Goal Attainment score should be slightlyrelated to the accuracy of predictions by the follow-up guide constructor and to the relationship of thefollow-up guide constructor to treatment.

8

FIGURE Ilb:

FIGURE Ilc:

low

low

(Accuracy of thefollow-up guideccnstructionpredictions)

B3a

(GA scoremean)

(Relationship oftne follow-up guideconstructor and client)

III. Validity Measures fur Goal Attainment Scaling

In the following section, this representationalsystem is used to discuss some hypotheses about GoalAttainment Scaling with pertinent data.

The three major areas of test "quality" estab-lished by Ebel will be used to separate the presen-tation of these components' interrelationships intosmaller units for ease of discussion. Ebel's pointswere presented previously on page 3.

Examples will be presented in relation to eachof these points.

A. "The Importance of the Inferences that Can BeMade"

Clearly if a criterion of effectiveness inmental health treatment could be established, itwould be a very significant contribution to eval-uation and psychopathology knowledge.

Specific examples of investigating treatmenteffectiveness include:

1. Differences Among Four Outpatient Treat-ment Modes

In a group o:" clients randomly distributedamong treatment modes, the mean Goal Attainmentscore should very among modes which hiwe differentdegrees of effectiveness.

FIGURE III

B2a

Type ofTreatment

B3aDifferences in Mean

Goal Attainment Scores

The results in the earliest study using GoalAttainment Scaling suggested that differencesamong modes were not great when all clients wereconsidered as a group regardless of other vari-ables. On June 5, 1972 for 186 nonrandomly aassigned subjects, the four modes of therapy inthe original Program Evaluation Project studyvaried only from 48.7 for Day Treatment to 50.2for Individual Therapy, 50.3 for Drug Clinic to51.1 for Group Therapy. (The procedures for thisstudy are discussed in other P.E.P. Report 1969-1973 chapters.)

Total results for all 249 randomly assignedclients as of 1973, reveals even less variationamong treatment modes in terms of Goal Attainmentscore.

Treatment Z 50.0Treatment Y 50.2Treatment X 50.3Treatment W 50.8

These means are not statistically different evenat the p < .10 level and certainly there is littleclinical significance to such miniscule differ-ences. (The above data are based on assignedtreatment modes, not actual treatment patterns.)

More recent data for the randomly assignedcases in this four-mode study of the ProgramEvaluation Project, whose procedures are de-scribed elsewhere in the P.E.P. Report 1969-1973,are slightly more encouraging, as Table I suggests,when separated by randomization pattern.

There are, however, no Goal Attainment scoredifferences which reach the p < .10 level. It isnoteworthy that the Consumer Satisfaction Indexdifferences in means also do not reach this levelof statistical significance. It is not clear,however, that these treatment would be expectedto be significantly different in outcome, espe-cially since the meaning of these various modeswere not rigorously defined.

TABLE I: Outcome Scores for Clients Who Stayed inAssigned Treatment Mode at Least Half ofTheir Treatment Sessions for the Four ModeStedy

TREATMENT W . TREATMENT X TREATKENT V TREATMENT 2

PATTERN

I

MeanGoal

AttainmentScore

47.34

(4.3)

52.29

('1.3)

48.82

(4.3)

47.89

(4.4)

MeanCnnsumerSatisfactionIndex

57.14

(4.3)

67.77

('1.3)

55.55

(4.3)

81.54

(104)

PATTERN

11

MeanGoal

AttainmentScore

47.27

(4.6)

41.41

(4.7)

48.97

0-9)

44.Mean

ConsumerSatisfactionIndex

63.09

(4.6)

68.97

(4.7)

67.45

(4.9)

PATTERN

III

ma.GoalAttainmentScore

52.65

MO49.04

(4-9)

56.32

(N.140

Mean

Consum,,SatisfactionIndex

72.81

(rpm

76.19

(4.9)

76.36

(4.14)

PATTERM

IV

M4anGoalAttainment

Score

52.49

0,419)

52.89

(N.62)

it 111111

KeanConsumerSatisfactionIndex

80.29

(M.118)

72.96

(.61)

9

13

2. Two Treatment Modes at a Day TreatmentCenter

Another study involving randomization of 14clients between two therapy modes, however, hasshown statistically significant differences. In

the Day Treatment Center of the Hennepin CountyMental Health Service, half of the clients wererandomly assigned to a situation in which they pre-pared their own Goal Attainment Follow-up Guides,while the other half of the clients did not directlyset goals for themselves. A clinician constructedstandard Goal Attainment Follow-up Guides for bothgroups. Based on follow-up interviewer ratings ofthese clinician-prepared follow-up guides, theclients who were involved in their own goal-settinghad a mean Goal Attainment score of 71, while theclients who did not set goals had a mean Goal Attain-ment score of 59, a difference significant at thep < .015 level (two-tailed). In addition, clients pre-paring their own follow-up guides reported greaterconsumer satisfaction with the significance of thedifference in means ranging, in terms of specificquestions, from p < .05 to p < .10.

3. Client Characteristics and the Goal Attain-ment Score

In constrast to the treatment comparisons madein points number 1 and number 2 above, most non-

treatment variables should not be related to the meanGoal Attainment score. The individualized develop-ment of the Goal Attainment Follow-up Guide shouldtake into account such client-specific differenceswhen the expectations on the follow-up guide are set.

FIGURE IV

//Client's Demographic /4 MEan GoalVariables Attainment

Alb, Alc, Ald ScoreB3a

This no-difference example is significant sinceit (a) suggests, by contrast, the importance of GoalAttainment score differences due to different degreesof treatment effectiveness and (b) indicates that theGoal Attainment Follow-up Guide can be speciallyadapted to each client, which is a key assertion ofGoal Attainment theory. This portion of Goal Attain-ment Scaling validation is strongly supported. In

January of 1972, a series of linear regressions werecalculated for the relationship between the Goal At-tainment score of 50.55. (Baxter, Tripp, 1972) The

correlations did not reach the p<.10 level of sta-tistical significance. The highest correlation wasfor income measures, which were correlated .20 and.18 with the 3oal Attainment score. See Table II.

No

TABLE II: Correlations with Goal Attainment ScoreMean (Pearson Product Moment) or Equivalent

AGE .034

SEX -.049

MARITAL STATUS .01

NUMBER OF CHILDREN .075

"IS INCOME ADEQUATE" .111

INCOME SOURCE .184

INCOME AMOUNT .204

HIGHEST GRADE COMr..ETED .054

Presenting problems mentioned by the clientsalso had low statistically non-significant correla-tions with Goal Attainment Scaling according to thedata in that study. Some of these presenting prob-lem variables are dichotomous or polychotomous andwere correlated accordingly. A few examples arepresented here in Table III.

10

4.4

TABLE III: Judgments on a Standardized InformationForm Correlated

Severity of Problem (N = 193) .102

Thoughts of Suicide (N = 178) .153

"Are Meds being Taken Currently?"(N = 158)

.142

A June, 1972 report by Baxter and Trippmentioned before suggests that even cross-analysisby age and sex does not lead to statistically dif-ferent (at the p < .10 lvel) mean Goal AttainmentScaling scores, as shown in Table IV.

TABLE IV: Goal Attainment Scores by Sex and Age-Groupfor 186 Clients

Sex Young Old

Male 50.11 53.00

Female 50.17 50.57

Thus, in these samples Goal Attainment scoresseem to be largely independent of client character-

istic variables.

A similar find Ag is suggested by data re-leased on September 15, 1971 resulting in a lowcorrelation between Shipley-Hartford Intelligencescores and the Goal Attainment scores. (Meade, 1971)

B. "The Meaningfulness of the Test Scores"

This is Ebel's second major aspect of validity.He includes four subpoints, each of which is dis-cussed below.

1. Operational Definition of the Measure-

ment Procedures

A large amount of operational flexibility is

basic to Goal Attainment Scaling. This method-

ology is not a single, rigidly determined set ofprocedures, but a collection of guidelines fromwhich procedures may be, within some limits, es-tablished to fit the needs of each agency. The

various Project publications are intended to out-line a range of approaches and to describe whathas been done in the Project's research. The pub-

lications are not designed to establish definitive,permanently fixed proceduresi but to allow a rangeof implementations of Goal Attainment Scaling.(See Programmed Instruction in Goal Attainment

Scaling, Garwick, 1973 and chapter one of theP.E.P. Report, 1969-1973.)

2. Knowledge of the Relationship of theScores to Other Measures through Co-efficients

This topic is the second subpoint within Ebel'ssecond general area, "The meaningfulness of thetest scores". This aspect of a measuremeot systemwould commonly be called "concurrent validity",i.e., validation through other measurement devices.In this instance, the "other measures" used forcomparison should be other program evaluation ortreatment outcome measures, as opposed to theclient characteristic measures discussed earlier.

As suggested earlier, Goal Attainment scoresare not intended to have a particularly high cor-relation with other treatment outcome measurementdevices, since Goal Attainment Scaling is such aradically different evaluation system (Kiresuk andSherman, 1968).

a. Concurrent Validity in the Drug Effective-ness Study of the Campbell-Fiske Matrix

The predicted relationships within some of themajor outcome measures of the Program EvaluationProject "Drug Effectiveness Study", appear below ina part of the nomological net.

FIGURE V

low .

CCrit 8 )Taylor MAS

(CritBPRS)

+ high

(CritSRSS)

+high

The actual results, bused on the first blockof 20 cases completed for the Drug Study appear inTable V in the following modified Campbell-Fiske(1959) matrix. This matrix is based on data sup-plied by Baxter and Jones (1973).

This application of the matrix is modifiedfrom the Campbell-Fiske concept by the assumptionthat the clients' outcomes at a given time are theequivalent in the matrix of a "trait". For example,initial status is one trait, outcome at three weeksafter intake is another trait, outcome after twomonths is the fourth trait. There are four vari-ables to measure these traits, except that theTaylor Manifest Anxiety Scale was not administered atat the three week's follow-up. (The places on thematrix which are left blank (because the ManifestAnxiety Scale was not administered at the threeweek follow-up are marked by a capital "X".) Thus,

each of the four measurement points in the DrugEffectiveness Study are utilized in the matrixin the same relationship as a personality traitwould be utilized in the original conception ofthe matrix. The matrix should be interpreted,of course, in terms of the construct relationshipsexpected for Goal Attainment Scaling.

The concurrent or "convergent" validities arerepresented on the matrix by the numbers not in-cluded in the triangles. As predicted, correlationsbetween Goal Attainment Scaling and the other mea-sures are generany low. Except for the correlationbetween Goal Attainment Scaling and the ManifestAnxiety SCI:t' at the two month follow-up which was.52, the co(relations are all below .30, with sevenbelow .20. Only the .52 correlation reaches thep < .05 level of significance (two-tailed). By con-trast, for the ten concurrent validity correlationsnot involving Goal Attainment Scaling, (for example,the correlations between the Taylor Manifest AnxietyScale and the Self-Rating Symptom Scales, which is.73 at the initial measurement, .74 at the two month'sfollow-up, and .80 at the three month's follow-up)eight are over .30, with five over .50 and threeover .70.

Th2 first type of "discriminant validity",which requires that convergent validities be largerthan correlations between different measures, inWiggins' discussion of the Campbell and Fiske Matrix.(Wiggins, 1973) This validity criterion is not metconsistently in this study by any of the outcome mea-surement devices, and particularly not by the GoalAttainment Scaling data, suggesting that correlationsbetween two different outcome measures taken at asingle follow-up time are not consistently higherthan correlations involving different measures ordifferent follow-up times. Similarly, the secondand third types of discriminant validity are notmet, suggesting that correlation in outcome dueto "method" variance (which in this case is notreally "method" variance but rather should becalled follow-up variance) is as great or greaterthan correlation due to different follow-up times.

The solid triangles illustrate the "reliabil-ity" correlations. Since in reality, all theseoutcome measures are based on three differenttime :,ome change might be expected in the rela-tive ,erformance of the clients. Out of the sixGoal Attainment Scaling correlations, four arebelow (.20), while for the 15 correlations fromthe other measures, only two are less than (.30),11 are greater than (.50) and six are greaterthan (.60). This pattern of outcome reliabilitysuggests that the Goal Attainment score is lessstable than the other outcome measues. Whetheror not this smaller degree of stability repre-sents a greater sensitivity to real shifts inoutcome will have to be determined by futurestudies. The Drug Effectiveness Study is con-tinuing and later data may help illuminate thesituation.

b. Concurrent Validity and the ConsumerSatisfaction Index

Consumer satisfaction results have also been

11

,15

TABLE V: An Outcome Evaluation Multitrait-Multimethod Matrix for the Drug Effectiveness (Valium versus Psycho-therapy) Study

(N=20) Method 1(Goal Attainment Scat ng)

Method 2(Brief Psychiatric Rating Scale)

Retnod 3(Self Rating Symptom Scale)

Method 4(Manifest Anxiety Sea e)

Traits

_Ini tial 3

Weeks

2

Months6

MdnthsInitial 3

Weeks2

Months6

MonthsInitial 3

Weeks2

Months6

Months

Initial 3

Weeks2

MOnths6

Mcnths

Method 1

iGoal Attain-ment Scal ing)

Int tial

3 Weeks2 Months6 Months

-.09-.12-.22

7

.20 .47

Method 2(Brief Psy-chiatricRatingScale)

Ini tial

3 Weeks2 Months6 Months

,..,-.04.-"-

, .15--i -.22

L.05

-:.3-0

-,J6.07 -.11

-.1-7

-.20-.26--.23- :

-.2-8 -1

.08 I- -.12 ;

c._06-

.11

.52

-.06

-.

.75 .50

Method 3(Self RatingSymotomSeal e)

Int tial

3 Weeks2 Months6 Months

r.,.,21 -; .24- -1 .12

. .19

- ...11--.11

.28 -

.16

-.13

- -.27

--.23- --.39- -

.24 1,

.10 i-.09z.111 -4

r-.47 7---"----.i0

I .25-

.43

L 04

- .55".60-.51

.-37

-- .27

....Sr"

.2a -

-..201.38 ,

- -.33 I

- 42.-

.58

.66

.47

7-8-6-----_,.52 .64 -----__L

Method 4(Manifest

AnxietyScale)

In 1 t 1 al

3 Weeks*2 Months6 Months

--.1c--: x-i -.30

L .33

:...j5

...x

-.16-.05

-:27"_ X

...5.-2 --.34- -

:23 1X I

-.31 I

.17-

-. .zz '1 X ---.._

! .27

L .11

-.44Xr

728.51

.52-....X

- .38-

.21

.17 1

. X:- ...05 I

- .23'

- .73 .-

I is.......

; .84

i .61

- k7C....7i.

./C1- --....74

.69

.52

.7b. -

.18-,X :

-- .42 i-_,eo

x.84

.49 x .66

The Manifest Anxiety Scale was nOt stoned at the two week follow-up.

compared with the Goal Attainment score. In a

February 1974 report, Dreyer noted a Pearson-Product Moment Correlation between Baxter's sevenitem Consumer Satisfaction Index and Goal Attain-ment score of .21 for 686 followed-up cases. In

a 1973 report, Baxter presented correlations be-tween the Goal Attainment score and 12 of thevarious individual items on the Consumer Satis-faction inventory for 202 randomly assignedclients. These 12 correlations ranged from -.12to .46, with four over .20. This report alsoshowed the Goal Attainment score and the Con-sumer Satisfaction Index which is based on sevenitems to be correlated .23 (for 199 clients).

c. Concurrent Validity and Therapist Ratings

A useful addition to the Goal Attainment Scal-ing validation data is the combination Validity/Reliability Study by Baxter. (Baxter, 1973) In

this study, therapists were asked to completethree procedures for clients followed up afterApril 15, 1972: 1) answer two global rating ques-tions about a client's progress in therapy, 2)score the client's Goal Attainment Follow-upGuide before seeing the scores from the follow-up

interviewer, and 3) rate the "relevance, "optimism"versus "pessimism", and need for additional scales

for each client's scales. The global ratings andthe relevance, optimism/pessimism, and need for ad-

ditional scales are valuable concurrent validationprocedures.

It would be expected that the correlations be-tween Goal Attainment scores and the individualglobal ratings would be somewhat higher than betweenGoal Attainment scores and more standardized measuressuch as the BPRS or SRSS as illustrated in Table V.

Table VI shows the recorded Pearson Correlations for

the Validity/Reliability Study. When therapist glo-

bal ratings and Goal Attainment scores compare, the

correlations range from .582 to .849.

12

16

TABLE VI

"Indlcate how well, in your opinion.each patient did In relation to thetypical patient in your caseload..."

"Indicate how successful, 'n yOurooinion. was your Interaction withthis Patient in relatioq to tletypical patient in your caseload..."

OutpatientN 53

i Day TreatmentN 8

OutpatientN 53

Day TreatmentN 8

Correlationwi th Thera-pist Follow-Up SCOre

.582 .689 .684 .849

Correlationwi th Fel I Ow-

Up Interview-er Follow-upScore

.325 .192 .319 .507

d. Concurrent Validity and the MMPI

Mauger has found a Pearson Correlation of .285between the Goal Attainment score and the MMPI mean-change Index, and a correlation of .306 between theGoal Attainment change score and the MMPI mean-changescore. Neither coefficient reached the p<.05 levelof statistical significance.

e. Concurrent Validity and PredictiveAccuracy

In an analysis of data for the forty-four casesin the original reliability study (see the P.E.P. Re-port, 1969-1973 chapters on follow-up and reliability)a number of measures were investigated (Twedt, 1974).In this study, for each client, one follow-up guidewas constructed by the intake interviewer and a sec-ond follow-up guide was constructed by the therapist.Then, these two follow-up guides were combined andthis combined follow-up guide was scored independ-ently at two different interviews by two differentinterviewers.

For the therapist-constructed follow-up guides,the Goal Attainment score was correlated only .18with the Consumer Satisfaction Index, but was cor-related .31 (significant at the p < .05 level) withpredictive accuracy of the follow-up guide (i.e.,mean inaccuracy score, absolute deviation for the de-viation for the expected mean of 50). There was nocorrelation which reached the p < .05 level cf sig-nificance between either the Goal Attainment scorefor the therapist or for the intake interviewerfollow-up guides and variables of age, sex, or num-ber of treatment sessions with the clients, al-though intake interviewer Goal Attainment score wascorrelated -.23 with age of the client. For the in-take interviewer constructed follow-up guides, theGoal Attainment score was correlated only .24 with'..he Consumer Satisfaction Index and .16 with predic-tive accuracy of the follow-up guide. Thus, the pre-dictive accuracy was correlated significantly withthe Goal Attainment scores for the therapist but notfor the intake interviewer, suggesting that the pre-dictions could have influenced the course of ther-apy in this case, which is very compatible with thefact that intake interviewers were not involved inthe therapy directly.

All these results suggest that Goal AttainmentScaling is not highly correlated with other mea-surement systems. Nonetheless, there is a posi-tive correlation with the other systems, and thecorrelations are in the anticipated low to mod-erate range.

The "Change score" is another possible concur-rent measure. Present data suggest moderate cor-relations between the Goal Attainment score andthe Goal Attainment Change score in the .10 to .30range for various groups. (See the chapter on theChange score in the P.E.P. Report, 1969-1973.)

3. Reliability Measures

Reliability, Ebel's third subpoint in hissecond general area, "The Meaningfulness of theTest Scored", has been investigated repeatedlyfor Goal Attainment Scaling. It is not clearthat his logic is correct when he included re-liability under the "meaningfulness" rubic, butit is included here in order to follow his out-line. Extensive discussion of the reliabilityresults are available in the chapter on Re-liability and the Goal Attainment ScalingMethodology in P.E.P. Report 1969-1973.

a. One such study, the Interdisciplinary Re-liability Follow-up Study, was completed late in1972. (See the Follow-up Chapter in the P.E.P.Report, 1969-1973.) The study examined difTeT-7ences between interviewer discipline, telephoneversus in-person interviewing, and the firstversus the second interview. This arrangementof repeated trails will tend to minimize the es-timate of reliability, since follow-up interview-ers and other variables can change in the inter-val between the two interviews (a mean of 27 days).(See the chapter on Reliability of Goal Attain-ment Scaling in the P.E.P. Report, 1969-1973.)

Even in these demanding circumstances, theoverall Pearson Correlation for Goal Attainmrnt

13

17

scores in the first and second interviews was.595 (N=99). When the discipline of the inter-viewer waL held constant, the correlation was.621 'ir VSW interviewers (N=13) and .750 forRN oier,iewers (N=10). When the discipline ofthe interviewer in follow-up one was differentthan that of the interviewer in follow-up two, thecorrelations were slightly lower.

b. The Combination Validity/Reliability Studymentioned previously gives another viewpoint onReliability (Baxter, 1973). Here, therapist andthe follow-up interviewers scored the Goal Attain-ment Follow-up Guides independently. Again, thisis an extremely severe test of reliability, sincethere was a time span between the two interviews,and the sources of information were clearly muchdifferent for therapist and follow-up interviewers.Nevertheless, as Table VII illustrates, the cor-relations ranged from .46 to .85.

TABLE VII

Pearson Product-Moment Correlat on betwen r, )al Attainmen t Scores forcllents based on independent (1) therapist-scorings and (2) follow-upInterviewer scorings.

Adul t OPD H 53 .507

Daj Treatment N 8 .048

Al I Cl lents 11 61 .458

These various results suggest a fair degrra ofreliability with internal consistency coefficientsin the .60 range, and inter-rater agreement re-liability coefficients in the .45 to .75 range.There seems to be sufficient Goal Attainment scoreconsistency to make the methodology useful in anumber of situations.

4. Appropriate Norms of Examinee Performance

Ebel's fourth subpoint under his general areaof "The Meaningfulness of the Test Scores", is some-what difficult to translate directly into Goal At-tainment Scaling terms. Perhaps the most usefuldata on "norms" would be the mean Goal Attainmentscore of 50 and standard deviation of about 10,where each agency is seen as an "examinee". Asmentioned earlier, most agencies using Goal Attain-ment Scaling have achieved such nor-s.

AnoLher possible approach to establishing normsis some type of content analysis. A beginning onsuch norms is available from "Expectations and Goalsfor Clients at a Community Mental Health Service".(Garwick and Lampman, 1972) This study shows somenorms for quantifiable variables for the MentalHealth Service, but does not examine norms from otheragencies, although Goal Attainment Follow-up Guidesfrom several other agencies have been collected, andwill be further analyzed in the future.

C. Technical Refinements, or the "Convenience ofthe Test"

Ebel's thirgol major area of "test quality" re-

fers to the ease of the test's implementation andinterpretation. Some improvements have been madein the convenience of Goal Attainment Scaling, mostrecently, Baxter's Conversion Key for CalculatingGoal Attainment Scores from Unweighted Scores (Bax-ter (1973) and Garwick and Brintnall's Goal Attain-ment Scaling Calculation Tables (1973). In general,however, the convenience of the methodology couldbe improved further. The Goal Attainment Scalingmethodology seems to be inherently attractive to manyclinicians and administrators, but both (1) construc-tion of the Goal Attainment Follow-up Guide and (2)interpretation of the Goal Attainment score may ap-pear inconvenient to some.

1. Convenience of Construch of the GoalAttainment Follow-Up Guide

Some persons have praised the Goal AttainmentScaling concept because it enables evaluation to be-come part of the therapy and can be incorporated in-to the interaction with the client. Others, how-ever, have complained of the difficulty of trainingpersonnel in Goal Attainment Scaling and in findingenough time to produce the Goal Attainment Follow-upGuides. One possible amelioration of this difficul-ty is the client construction of the follow-upguides, as illustrated by the Guide to Goals, FormatOne approach (Garwick, 1972).

The "review" or "monitoring" of the Goal Attain-ment Follow-up Guide is a closely related and dif-ficult problem. As the Program Evaluation Projectstaff began accumulating Goal Attainment Follow-upGuides, it seemed that some follow-up guides pro-duced by the clinicians included clerical or logicalor descriptive shortcomings which rendered follow-upscoring very difficult or questionable ("Manual onFollow-up Assessment", Garwick, et. al. 1972). Var-ious monitoring and follow-up guide revision tech-niques have been utilized, but none has been com-pletely satisfactory, due to clinician dislikeof the monitoring and a limited empirical basisof the monitoring criteria. The monitoring isexpensive for the evaluation staff, and evident-ly unattractive to the clinical staff, yet theinconvenience and cost of follow-up guides withsevere errors can,be troublesome. Thus, reviewand revision of the Goal Attainment Follow-upGuide and the amount of time required far train-ing and complr ng the follow-up guide are twoof the more frequent complaints.

The client-specific nature of the Goal At-tainment Scaling methodology, however, is ap-parently popular. One possibility for increas-ing clients involvement is the "programmed"Guide to Goals, Format One which was mentionedpreviously. If this device makes it possibleto allow clients t.r.. construct their own follow-

up guides with inimal clinical supervision, theconvenience r:t' Cual Attainment Scaling may begreatly increa5ed. One possibility for minimizingthe need for review is a "Semi-standardized" sys-tem where idiosyncratic (i.e., client-specific)construction is retained but a catalogue of os-sible Goal Attainment variables is used to decrease thethe time required for follow-up guide construction.

14

18

2. Convenience of the Interpretation ofthe Goal Attainment Score

The Goal Attainment score is based on theformula from Kiresuk and Sherman's 1968 article.This formula seems fr-midable to some. One ofthe earliest attempts to ease Goal Attainmentscore calculation was the release of a short,simplified description of the calculation insimple steps. As mentioned above, Baxter hasproduced a conversion key and recently Shermanhas suggested the possibility of simpler, scale-by-scale calculation of follow-up level scoremeans for each follow-up guide. (See ChapterOne of the P.E.P. Report, 1969-1973.) A seriesof tables which have been developed, make itpossible to obtain the Goal Attainment scorewithout calculation for Goal Attainment Follow-up Guides with from one to five scales (Garwickand Brintnall, 1973).

Because of the newness of Goal AttainmentScaling, comments on interpreting the Goal At-tainment score have been purposely restricted.In general, the Goal Attainment score is saidto be "the degree to which expectations foroutcome at some certain time are reached".

Conclusion

Too often, it is forgotten that the Goal At-tainment score is merely a tool, like the MMPI,the Strong Vocational Interest Blank, or theWechsler Adult Intelligence Scale. Like any mea-surement tool, the Goal Attainment score is asuseful as the desici or system with which it isutilized. The degree of scientific irreproach-ability or clinical completeness depends on theway the methodology is integrated into a practicalor experimental procedure.

References.

Anastasi., A. Psychological testing. (5th ed,)USA: Macmillan Company, 1970, P. 121.

Baxter, J. Goal attainment scaling conversion keyfor equally weighted scales. UnpublishedProgram Evaluation Project Report, February,1973.

Unpublished data report from theProgram Evaluation Project, 1973.

& Jones, S. Unpublished data report fromthe Program Evaluation Project, 1973.

& Tripp, R. Unpublished data report fromthe Program Evaluation Project, 1972.

Campbell, D.T. & Fiske, D.W. Convergent and discrim-inant validation by the multitrait-multimethodmatrix. Psychological Bulletin, 1959, 56, 81-105.

Cattel, R.B. Validity and reliability: a proposedmore basic set of concepts. Journal ofEducational Psychology, 1964, 55, 1-22.

Dryer, J. Unpublished data report from the ProgramEvaluation Project, 1974.

Ebel, R.L. Must all tests be valid. AmericanPsychologist, 1961, 16, 640-674.

Garwfck, G. Guide to goals, format one. UnpublishedProgram Evaluation Project Report, October, 1972.

Programmed instruction in goal attain-ment scaling. Unpublished Program EvaluationProject Report, October, 1973.

& Brintnall, J. Tables for calculatingthe goal attainment score. Unpublished ProgramEvaluation Project Report, August, 1973.

, Grygelko, G., Makela, W. & Jones, S.Manual for the standardized assessment of thegoal attainment follow-up guide. UnpublishedProgram Evaluation Project Report, July, 1972.

& Lampman, S. Expectations and goals forclients at a community mental health service.Unpublished Program Evaluation Project Report,July, 1972.

Kiresuk, T.J. & Sherman, R.E. Goal attai lent scaling:a general method for evaluating comprehensivecommunity mental health programs. CommunityMental Health Journal, 1968, 4(6), 443-453.

Lindquist, E.F. A first course in statistics (Rev.) Boston: Houghton Mifflin, 1942.

Loevin,er, j. Objective tests as instruments ofpsychological theory. Ps cholo ical Re orts, 1957Vol. 3, Monograph Supplement 9, 635-694.

Mauger, P. A study of the construct validity of GoalAttainment Scaling. Chapter Six, P.E.P. Report1969-1973, June, 1974.

Mosier, C.I. A critical examination of the conceptsof face validity. Educational & PsychologicalMeasurement, 1947, 7, 191-205.

Nunnally, J.C. Psychometric theory. New York: McGraw-Hill, 1967. Pp. 78-79, 92.

Twedt, R. Unpublished data report from the ProgramEvaluation Project, 1974.

Wiggins, J.S. Personality & prediction, princirlescf personality assessment. USA: Addisnn-WesleyCompany, T973. Pp. 407-409.

15

19

PROGRAM EVALUATION PROJECT STAFF LISTING

CURRENT STAFF MEMBERS

Thomas J. Kiresuk, Ph.D.Principal Investigator 1969-1974

Donna M. AudetteResearch Assistant 1970Follow-up Assistant 1971Follow-up c'upervisor 1972-1973Utilization Consultant 1974

James BaxterResearch Assistant 1970Operations Manager 1971

Assistant Coordinator 1972-1973Re-design Coordinator 1974

Diane BergEditorial Secretary 1973-1974

Janis Bibee, M.A.Editorial Secretary 1972Editorial Assistant 1973Assistant to the Editor 1974

Joan BrintnallStudent Assistant 1971

Secretary-Receptionist 1972Administrative Assistant 1973Dissemination, Consultation,and Utilization Supervisor 1974

Joan DreyerResearch Clerk 1972Research Assistant 1973Research/Administrative Clerk 1974

Geoffrey Garwick, M.A.Research Applications Consultant 1970Program Evaluation Coordinator 1971

Deputy Assistant Director 1972

Assistant Director 1973Dissemination, Consultation,and Utilization Consultant 1974

Carolyn JaspersonStudent Assistant 1973-1974

Susan JonesResearch Assistant 1971

Research Associate 1972-1974

Laurence Kivens, M.A.Applications Analyst 1971

Editorial Supervisor 1971

Editor 1972-1974

Mary KnepperAppointment Interviewer 1970-1973Administrative Assistant 1974

Judy LongAdministrative Secretary 1974

Sander LundManagement Applications Supervisor 1971

Assistant Coordinator 1971-1972Coordinator for Administration 1973Assistant Director 1974

Nancy PetersenSecretary 1973Research Assistant 1974

Michael SaundersProgremmer Analyst 1971-1972Programmer Supervisor 1973-1974

Robert Sherman, Ph.D.Associate Investigator 1969-1974

Roger TwedtStudent Assistant 1974

Mary Ellen WhalenStudent Assistant 197?-1974

PREVIOUS STAFF VOIMRS

Anita BjornsunRescarcn Ansistant 1370

Barbara BlazickStudent Assistant 17O

Mary OurocheEditorial Secretary 1972

David FeigalResearch Assistant 1970-1971

Thomas GriffinStudent Assistant 1970

Marilee GrygelkoStudent Assistant 1971-1972Research Assistant 1973

Edward GutmanStudent Assistant 1971

Colleen HalleyStudent Assistant 1971-1973

Robert KearneyEditorial Assistant 1972

Karen Kohout, M.A.Research Analyst 1969Research Supervisor 1970-1971

Sherry LampmanAdministrative Assistant 1970Linguistic Analysis Consultant 1971

Content Analysis Supervisor 1972

William Makela, M.A.Follow-up Supervisor 1970-1972

2 0 Charles Meade41

Research Assistant 1970-1972

16

Deirdre MeadeSecretary-Receptionist 1971

Administrative Assistant 1972

Sylvia MuilenbergAdministrative Assistant 1969

Administrative Supervisor 1970

Mils OlssonStudent Assistant 1971

Research Assistant 1972-1973

Carol PollockResearch Clerk 1972

William ProckCommunity Applications Supervisor 1971-1972

Peter ReeStudent Assistant 1970

Martha RosenSecretary-Receptionist 1970

Editorial Assistant 1971-1972

Susan SalasinCoordinator 1969-1970Assistant Director 1971-1972Research Applications Consultant 1973

Richard TrippProgramming Supervisor 1970Design and Analysis Supervisor 1971-1972

Mary TroneEditorial Secretary 1973

Carol VanderpoolSecretary-Receptionist 1971-1972

Administrative Assistant 1973

Cynthia WetterlandSecretary-Receptionist 1970

Allen WichelmanMedical Records Clerk 1970Management Applications Supervisor 1971

Sue WrightClinical Applications 1971

Carole ZimbroltResearch Analyst 1969-1971

FOLLOW-jP INTERVIEWERS - CURRENT

Kathleen Bergum, M.S.W.

Charles Besnett, M.S.W.

Carol Dethmers, B.A.

Marcia Frankenberg, B.A.

George Meirick, M.S.W.

FOLLOW-UP INTERVIEWERS - PREVIOUS

Mary Ann Anzelc, R.N.

21

17

James Bergum, M.S.W.

Roanne Borkon, R.N.

Larry Bultena, M.S.W.

Scott Craven

Jeanine Emmons, R.N.

Barbara Gusek, R.N.

Mary Keturakat, R.N.

Steve Lapinsky

Betty Metz, B.A.

Mladiline Sachs, R.N.

James Snope, M.S.W.

Milt Somerfleck, M.S.W.

CONSULTANTS

Dean Beaulieu, Ph.D.Clinical Coordinator 1971-1974

James Boen, Ph.D.Statistical Consultant 1969-1973

Byron Brown, Ph.D.Statistical Consultant 1969-1973

Arthur Funke, Ph.D.Dissemination Consultant 1969-1974

Stephen Greenwald, M.D.Medieations Consultant 1959-1973

Ann RussellClinical Consultant 1970

Robert Spano, A.C.S.W.Patient Follow-up Coordinator 1969-1973

Wyman SpanoEditorial Consultant 1972-1973

Robert Walker, M.A.Data Applications Coordinator 1973