.

S.

47

DOCUMENT RESUME

ID 220 504 TM 820 518-

-AUTHOR Crane, Laura R. And OthersTITLE Measurement.of Evaluation Utilization: Preliminary

Results..PUB DATE Mar 82NOTE 40p.; Palier presented at the Annual Meeting of the

-American Educationil Research Association,(66th, NewYork, NY, March 19-23, 1982).

EDRS PRICEDESCRIPTORS

IDENTIFIERS.

ABSTRACT

4F01/PCO2 Plus Postage.Compensatory Education; DecisiOn Making;^ElementarySqcondary Education; Factor Structure; *InformationUtilization;-*Models; *Program Evaluation; *TestConstructioh*Elementary Secondary Educatiom Act Titld I;*Evaluation Utilization

Based on Leviton and Hughes' Conceptualization ofvariable Clusters that affect evaluation utilization, a procedure formeasuring utilizatiop potential was deve14ed. Five clusters ofvariablesoate consistently related to utilizatiOn: relevance,credibility, communication, information processing, and userinvolvement and advocacy. The communication and informationprocessing factors were combined for this study. To relate the fourremaining factors to decision-making within the context of.Title Iprograms, Title I decision areas were identified. They include fundallocation, program adoptien or change, Staffing, student selection,and test selectioh. The generation of items for the pilot instrumbntwas a multistage process involving Title I Technical AssistanceCenters across the country-. From the several hundred items generated,a 65-iteT pilot instrument was developed and, pilot tested. Maximumlikelihood factor.analysis using a subset of items revealed a set ofcorrelated factors related to Levitçn and Hughes' conceptualization.The exploratory factor analysis resu ts were not confirmed on asecond sample, but this may have bee due to small sample sizes. Thisresearch indicates that the possibility of developing a scale toassess utilization potential is very real. (Author/BW)

*******R***************************************************************: t .

* Reproductions supPlied by EDRS are the.best that can be made *.

* from the'origfnal document. *;**********************************************.*************************

1 .

I

a

1

MEASUREMENT OF EVALUATION UTILIZATIONPreliminary Results

Laura R. CraneRichard W. Naccarato

Educational Testing ServiceMidwestern Regional Office

Evanston, Illinois

SubramanianBell &

Chicago,

KandaswamyHowellIllinois

a

DEPARTMENT OF EDUCAT/ONNATIONAL INSTITUTE OF EDUCATION

EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)

X Thri Oxionern has Dien reproduced es(vowel heal the pee. o eapaciaian

Mnor changes have been made to wort,,reprooucuon clueirry

Pants of vew oPrrons suftscl m tha Cepa

frfe'M neeenarlY evestert Otfcw NIEPOV 'X PCAKZY

'PERMISSION TO REPRODUCE THISMATERIAL HAS BEEN GRANTED BY

S. K 41.41dig tt

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)-

Paper presented at the annual meeting of the American EducationalResearch Association, New York, March 1982.

t;

1

Introductionc

During the 1970s there was ebnsiderable focus on evaluation utili-

zation and many articles began to appear on this topic. More.recently,

several analytic reviews have appeared (e.g., Haenn, 1980; Hansen, Martin

and Oxford, 1979; Leviton and Hughes, 1981). The reviews all attempted

to identify clusters or "utilization factors" and involved at least two

and usually three or more levels of factors with each major,factor

divided,into subfactors.

The three reviews mentioned here had a fair amount of overlap in

terms of references, but approached the task through somewhat different

perspectives. The review by Haenn was considered in the context of

school district utilization; the Hansen et. al. review was considered in

the context of Title I evaluation. The Leviton and Hughes review con-

sidered utilization in a much broader context, not restricted to school

related environments. Additionally, the Leviton and Hughes paper devoted

considerable attention to the definitional and methodological ptoblems

inherent in research on evaluation utilization.

Much of the research on utilization has, of necessity, relied on the

use of interviews (e.g., Caplan, 1977; David, 1978) and intensive case

studies (e.g., Alkin, et. al:, 1979; Patton, 1978) and questionnaires.

Leviton and Hughes cite four major problems these techniques have with

respect to utilization research: 1) "...it is difficult to document that

utilization occurs, because evaluations are frequently used informally...,"

2) "demonstrating that change, at any level, was caused at least in part

by evaluationi...," 3)"...the question of base rates .for compariion.

Until. recently, we believed the base rate for utilization wils very low.

We are learning that fhe fault may lie with our measures...," and 4)

1

2

...the unit of analysis: What is an instance of utilization? It is

necessary to quantify utilization if ve are to show that it can be

enhancgd." (Leviton and Hughes, p. 533). Although not a methodological

problem per se, one might also add cost considerations as a problem with

interview and case study approaches to research on utilization.

Given the methodological problems inherent in much of the Utiliza

tion research, One might ask the question: Is there Any way of develop

ing a scale that would measure utilization directly? The answer is

probably in the negative, but it might be possible to develop a scale

that would measure "potential" for utilization where the higher the

potential the more likely it is that utilization occurs. Working within

the utilization conceptual framewotk developed by Leviton and Hres,

this paper reports preliminary results of a pilot study designed to

develop such an instrument. Briefly described in the next section are

the five clusters (hypothetical factors) conceptualized h(dLeviton and

Hughes.

Hypothetical Factors

ton and Hughes identified five clusters of variables that seemed

consistentl ted to utilization. The five clusters are: 1) Relevance,

2) Credibility, Communication, 4) Information Processing and 5) User

dvocacy. - Each of these is briefly discussed below.Involvement

)4Relevance. The primary concern of this category is whetherthe eva .

/uatiori meets ehe user's needs. This ia viewed as essential to utilization.

Relevance assumes that the evaluation.answers the necessary questions or

measures goals that are viewed as important by users. One variable within

this cluster is timeliness: the need to have evaluation results in time

4

1r

3

for oonsideratiOn before key decisions are made.

Credibility. Credibility of the information and the information

1 producer also affects utilization. CredibilitY*of evaluation as it is

compared to alternative available sources of itjemation, preconceptions

or biases of users toward research, trust in the research, and perceived1

quality of the informatioft are some aspects of credibility that affect

utilization.

Communication. Another category of variablei is broadly referred

4

to as communication. The pertinent variables included are associated

with communication as it occurs within the bureaucratic hierarchies where

the evaluation is focused. Important to utilization is the frequency of

contact between the producers of the evaluation apd the potential users.

Also important is the existence of networks in the bureaucratic hierarchy

which assure that, during dissemination, valuable information is not left

out or dilstorted.

Information Processing. Information processing refers to the impor-

tance of-translating evaluation findings into specific implications. Eval-

uations must be structured to answer specific questions and they should

tt

be clear in terms of their goals and objectives. The information process-

ing style of the administratot (user), generally different from the

evaluator's style, has implications for utilization. Evaluations should

be presented in a style familiar to the user..

' User Involvement and AdvOcacy. User involvement and advocacy are

Variables of a political nature. Utilization is affected by be level.

1

of interest and commitment of decision makers to ihe process of evalua-

tion as well as.to the program being evaluated. .The direction of the

evaluation results, i.e., w#ether or not they support theiadvocate'S

position, will impinge upon utilization:

Four Factors. For our purposes, it was felt that the "Information

Processing" cluster or factor had considerable overlap with the "Communi-

cation" factor and we elected to combine the two factors. Our interest

was in the development of a scale that would assess utilization potential

within the context of Title I decision-making. Using the conceptual

.framework developed by Levitom and Hughes, re.Oed to four factors: 1)

Rtlevance, 2) Credibility, 3) Communication, and 0 User Involvement and

Advocacy we believed such a scale, could be developed. To relate the in-

strument to decidion-making, Title I decision areas were identified as

discussed in the next section.

IDecis ion Areas r

Within Title I, at least five areas were identified where school

district personnel made decisions. These areas were: 1) fund allocation,

2) program adoption or change, 3) staffing, 4) student selection, and 5)

test selection. Evaluation has the potential for informing administrators

making decisions in each of these areas as described below.

Fund Allocation. School administrators are always making decisions

concerned with funding. In a Title I context, such decisions 'Might be

concerned with how funds should be allocated to different projects,(e.t.,

reading and math projects).

Program Adoptton or Change. Decision-making in tilts area is concerned

'with selection of.a program best meeting the needs of students served or

5

with how an operating program might be imprred. Which program will

best serve the needs of students? Would increasing time( pent in drill

improve the program? These are example questions

required in the area of program adoption or change.

that ,imply decisions

Staffing. Staffing decisions might include deciding what staffing'

pattern is required, deciding which staff shoufd participate pr what

staff training is necessary to successfully implement or operate a\., .

program.

Student Selection. Deciding which students should participate in a

program is an obvious decision that must be made. Peihafs-more cruc-ial

in this area is deciding what selection process should be implemented.

Test Selection. Deciding on a testing program that serves multiple

purposes can involve many decisions. For example, "Does a test acculrately-

reflect program goals?" or "Should outof,level eesting be-used?" Are

questioni which need to be considered when test stteGtion decisions are

being made.

.Summary of Decision-Areas. The,decision areas briefly discussed in

this section are only a few of the areas where school district administra

tors make decisions. The five Areas--fund allocation, program adoption

or change, staffing, studene selection, and test selection were discussed .

in this section because these areas are general across school districts

and Title I.

Aop,

Methodp"

Development of the Instrument

The generation of items for the pilot instrument was a multistige

process involving Title I Technical Assistance Centers (TACs) across the

country. Each TAC receiv concept paper describing the "factors" and

our approach to developing item, a brief description of the "factors"

and a matrix showing the five decision areas wiihin which Title-I decision-

making was likely tdoccur ief description and matrix are in Appen-

dix A). Each TAd was asked to develop items for each decision area

within two "factors."

From the several hundred items generated, after categorizing and

_ editing, a sixty-five item pilot instrument was developed. This instru-.

'ment was reviewed by state and local school district staff and by TAC

staff. Based on reviewer comments, additional editing was done before a

final pi ot instrument was distributed.

instrument requested respondeints to rate each of the sixty-

,five items o dimensions--the degree to which the described condition

Akisted in the district and the importance of the condition. Both ratings

were done on five point acales with a "1" indicating little existence (or

importance) and a "5" indicating considerable existence (or importance).

Although the instrument was very lengthy, all items were retained for

piliot testing With the tntent of reducing the instrument to about thirty

items in its final form.

Sampling 4

.Three states agreed to allow pilot testing of tha instrument. From

lists ofachool districts havin& a Title I pipil enrollment of at least

6

7

100, randamsamples of 100 districts per st'ite were drawn. Each state

sample was then randomly split into two samples (I and II) of size 50:

Sample I instruments requested respondent identification (name business

address and telephone number) while Sample II instruments did not requedt

* respondent identification. Both inst ruments had,a page requesting back-

ground information (posiiion, number of years in Tiqe I, highest degree

earned, grade levels served by Titie I programs, and appriximate number

of children served:in Title I progiams). ,A camparison of Sample and

Sample II ressonses on the background.information is given in Table 1. A

total of 223 instruments were returned, 114 fram Sample I districts and

lp9 fram Sample II districts.indicating a slightly higher return rate from

the ft respondent identification" Sampie I districts. Across both samples

district median number of pupils served in Title I reading programs

was 140 and the median number served .in Title I math programs was 75.

Phase I.Analyses

Each item was Classified accarding to Elle hypothetical factor for

which it had been developed--Relevance (R), Credibility (Cr), Communica-

ion (Co), and User,Involvement Reliabilities were computed for

each hypothetical factor separately for the existence (ES) and Importance

(IS) scales. Results of these analyses are shown in Table h As shown

, in Table 2, the hypothetical fdctor reliabilities were quite respectable

(minimum of .81).

Factor analytic techniques were then.,4plied separately to each,:

scile, a "very simple structure" (VSS) analysis was done (kevelle and

Rocklin, 1979).1 Results of these arikyses were used to specify para-

meters for tnaximum likelihood confirmatory factor analyses using LISREL

cif

4

;

r

Table 1

Background Information from Sample I (Respondents Identified)And Sample II (Respondents Not Identified) Questionnaires

"Question". Sample I

(n=114)

Sample II

(n=109)

Total

(n=223)

PositionaAdministrator 76 79 155Evaluator 49 31 80Teacher .28 22 50Other , 25 6

.

31

Years' Experience

in Title I5 yrs or less 49 49 986 yrs 10 yrs 32 29 61

more than'10 yrs 33 31 63

Highest DegreeDoctorate 6 7 13

Speciilist 20 22 42Masters ,,61 56 117

Bachelor , 27 24 51

aRespondents check6d all that applied.

Table 2

PhAse I Nypothetical Factor Scale Reliabilities

'Hypothetical Number ReliabilityFactor of Items ES IS

Relevance (R) 15 .88 .89

Credibility (Cr) 11 .81 .82

Communication (Co) 20 .88 .90User Involvement (UI) 19 .91 .91

11

10,

_Odreskog and Slirbom, 1978). Both LISREL analyses resulted in significant

rs indicating that the four fact% model, as specified, was ndi con-

firmed. These results wet-6 somewhat puzzlifig, VSS analyses indicated one

large general factor dominated by User Involvement items, .but also

including Rtlevance and Communication,items. It seemed quite possible

that the,LISREL models were misspecified.

Since the concept of a scale to measure evalu\a.tion utilization poten-

tial is relatively new, we felt additional, more exploratory, analyses

should be conducted. These analyses are described in the next section

on Phase II analyses.

Phase II analyses

The main concern of the Phase II analyses was to conduct exploratory

factor analyses that would lead to a set of factors that could be con-

firmed in subsequent analyses. To accomplish this, we decided to randomly

split the samples into two subsamples (Sample A and Sample B). SaMple A

would be used for exploratory analyses and Sample B would be uied for

confirmatory analyses. This decision created another problem, subsample

n's would most likely be muchltoo small to factor analyze the full

65-item set. It was decided to work with a 30-item subset.

Item,SelectiOn. To Select the 30-Aem subset, We first had the 65-

items independently classified into the four hypothetical factor cate--,

gories.(R) Cr, Co and UI) by three raters. A total of 60 items were

assigned to the kame hypothetical factor category by at least ewo of 'the

raters while 27 of the items were assigned to the same category by all

three iaters. Twenty-seven items, of the 30-icem subset were those items

assigned to 1 categOry by all three raters. Since'Only four User Involve-

.;

12

4.-

'11

ment (UI) items were within the 27 items, three additional Ul items were

selected at random from' the item& that two raters placed in the DI

Category. Same items were placed in categories Aifferent from their

original category designatiOn. Most of die 27 (23), *ere cateogrized,by

all three raters into their orilginal tafegory. The 30 items are.given

in Appendix B. Table 3 'shows reliabilities for the Existence Scale

hypothetical factors based on the 30item subset.

Table 3

Phass II Hypothetical Factor Existence Scale

Reliabilities for Subsample A, Subsample Band Total Sample .

Hypothetical Number ,

Factor of ItemsSubsampleaA .8

TotalaSample

(109) (114) (223)

..../

Relevance (R). 7 .80 .82 .81

Credibilitf (co fi .76 .77 .77

Communication (Co) 10 .82 .83 .82

Utter Involvement (Ul) 7 .76 .75 .75

aSamPle n's are given in parentheses.

a

Subsample A exploratory analyses. Exploratory common factor analyses

using maximum likelihood procedures developed and described by J8re8kog

/altd Van Thillo '(1971) were,done with data from Subsample A. Multiple R2

coefficients.were used for initial communality estimates. Since it_*es

assumed that the hypothetical factors were correlated, oblique rotational

peocedures were employed. The SPSS Subprogram JFACTOR (Burns, 1977) was4

used for the,exploratory analyses.

The JFACTOR program also.'prints results of three statistical tests

to- determine the suitability of.a correlation matrix ior factor analysis

^

12

(Dziuban and Shirkey, 1974). Bartletts's test of sphericity was rejecied

(411239.05, p<.001) indicating the Stibsample A correlation matrix was

suitable for aceoring. Inspection of the off-diagonal elements of the

anti-image matrix showed 13.33 percent of the elemepts were greater than

zero (>.09),-small enough to indic'ete the matrix suitable for factoring.

Finally, Kaiser's measure of sampling adequacy of .78 (almost the "merito-

rious" range in the .80s) indicating the matrix was suitable for factoring.

(Dziuban and Shirkey provide a brief discussion of each test.)

,Nine factors'were extracted from the Suhsample A correlation matrix

before a nonsignificant Chi Square value was reached. ( significant Chi

Square value indicates that a significant amount of Air ance remains in

the residual correlation matrix.) The pattern matrix was rotated oidiquely

uiing the Kaiser.Normalization procedure. Application of the Scree test

(Gorsuch, pp. 152-156) and visual inspection of the rotated structure

matrix indicated that at, least five of the factors were interpretable.

As a check'that the .rotated pattern matrix attained simple structure,

the five criteria by Thurstone (Gorsuch, pp. 164-165) were applied. The

.results of the tests for simide structure.are given,in Table 4.

14

3

4

'tAr:

- 4.

13

Table 4

Thurstones Five .SiMple Structure CriteriaResults.orl Subsample A Rotated Pattern Matrix

Criteria1. Each varkable has at leist

one zero loading.a

2. Each factbr has at least one '

set of linearly iddependenevariables with zero factorloadings.1

3. For every .p.sir of factors

there are several variableswhoseilloadings are zero on -*

one factor but not the other.. .

4. For every pair of factors alarge proportion of varia-bles have zero loadings on

,ihoth whenever More thanr about four factors are

present._

5. For every pair of factorsonly a small number ofvariables shouldSheenonzero loachngs on bothfactors.

Met

Yes

Yes

58% of the factorpairs had 50% ormore of the varia-bles meet this cri-teria.

97%.of the factor.'pairs had 20% ormore of the vari-a-

bles with zero load-ings on both lactors.

92% of the factor .

pairs had 20% or.fewer of the varia-bles with nonzpro.--.

, loadings on bothfactors.

aA zero loading was defined as any loading strictly less than .1 inabsolute value.

The rotated pattern matrix appeared to meet the criteria for simple struc-

ture. The Subsample A torFelation matrix, rotated factor pattern and

,structure matrices, and the factor correlation matrix are given in

. Appendix C.

The interpretation of factors found Subsample A is based for the

most part onthe structure matrix,(see Appendix C). This matrix gives

'the correlations heiyeen a variable and a factor.!

14

The items in each of the four pre-determined categories (R, Cr, Co and

UI) tended to correlate highly with one, and in some cases two, of the

factors resulting from the analysis. Six of the seven User Involvement

items correlated highly with the sixth factor of the nine factor configu-

ration.

The Credibility items correlated highlY for the most part with fac-

tors 1 and 3, having somewhat higher correlations on'the average with

factor 3.

The ten Communication' items correlated most highly with factor 4,

with a substantial number also Forrelating (though not as laghly) with

factor 6.

The fourth category of items, Relevance, seemed to be the most defin-

itive in term; of tile factor structurf. These itemi clearly and nearly

exclusively correlated on factor 5.

Hence, the oblique rotational procedures resulted' in items within

the categories correlating primarily,with a total of five factors, with

the remaining four factors being of rlat.ively small significance.. ,

The next step in the analysis, and often the most difficult one in

factor analysis, was an attempt to draw an .interpretation of the major

resulting five factors in light of the four-item categories. In an

effort to further interpretation of the factiors, the content of each

item was examined in relation to factors with which the item was highly

correlated.

'The first area for investigation of item content was for those items

. .

which correlated significantly ;rith more than one factor. For rample,

the five items within the Commusication-category which correlated signifi-.

11, .2LU

.

,

'

s

15

1

cantly with both factors 4 and 6 were examined on item content. Four of

thest five items hail correlated highly.with factor 4 which, upon closer

investigation of 'item content, could be interpreted as a "Cammunication"

factor. Of the ten total Communication items, five items correlated

significantly with factor 6, which appeared to be a "User Involvement"

factor. 'The wording of the five items which overlapped factors 4 And 6

(Communication and User Involvement) was examined,. In four of the;i five

pre-categorized Communication items the words "ulied" or "useful" terms

were found, indicating reason why same sampled respondents.might lave

interpreted such items in a "user invalvement" 'sense as well as in a

ft communication" sense.

The other item category which contained a number of items which cor-

related on more than one factor were the Credibility items--four of six

correlating highly with both factors 1 and 3. An examination of the

terminology for these four items which had cOrrelated with factoi 1

showed the use of "decison-making" terms within ffie content of each item.

04

Since factor 3 had been temporarily termed the "Credibility" factor, it

seemed worthy to examine the content of items which correlated only with

factor 1 and no other factors, and those Credibility items which did notir

correlate with 'factor 1. Othtr items which correlated with factor 1, and

--;

not with other factors, had similar "decision-making" terminology within_

them. Factor A, therefore, seemed to- be interpretable as A .Decision-

Making factor, different from the four *A-detprmined categories. Those

Credibility items which did not correlaIwith.tfie Decision-Making factor

I did not contain terminology Mentioning "decisions," but rather could

be termed ttpurett Credibility items.,

4

te .41

16

In summary, ,the exploratory analysis on*.Subsample A produced a nine--

factor structure. Four of these factors were relatively minor in terms

of the number of items within any of the four pre-determined categories

which correlated highly on any one of the factors, i.e. item correlations

with these four factors were generally smaller and dispersed across all

four pre-determined categories of items (R, Cr, Co and UI). Theinterpre-

tation of the exploratory factor analysis is that the four pre-determined

categories of items correlated differentially primarily on five factors,

which upon examination of the iteds with respect to the, factors resulted

in the factors being interpreted as Relevance, Credibility,iCammunication,.

User Involvement and Decision-Making.

Subsample B confirmatory analyses. The program LISREL (J6reskog and

Sbrbom, 1978) was used for cotifirmatory analyses. Basically, by using

the Subsample A pattern matrix and factor correlation matrix (see Appendix

C) with the Subsample B correlation matrix a maximum likelihood test on

the-residual matrix is available. The relationship between the matrices

is shown below.

O.*

y 13-11/A.

Where 4E. for our case is the COI-relation matrix determined by the factor

pattern matrix (4y), the irterf'actor eorrelation matrix (Y) and the er-

ror matrix ( ). I is an identity matrix. The maximum likelihood.test

performed is on Eq. where 2. is the Subsample B correlation matrix and .

is estimated from the above equation. 'A nonsignificant rindicates the. .

results from Subsample A-were confirmed on Subsample B.

When estimates1for both AyAnd 17 were fixed and taken from Subsam-

ple A results, A moat stringeAtItest,-Ithe fal.69-X-p4itern and correlations

17

"V '-

were not confirmed ( 0(.40, 717.41, p<.0000). This test required confirma-

tion of both the factor pattern and the intercorrelation among factors.

When Only estimates for .11ywere fixed, the test, if nonsignificant, would

confirr Subsample A factor pattern, but not the intercorrelation

among factors. This test also did not confirm the Subsample A pattern

matrix (Appr58 .21, p<.0000). Other, less restricted confirmatory analy-.

ses (e.g., J.estimates only partially fixed) are currently being con-

sidered.

Discussion, Summary and Implications

Based op the Leviton and Hughes' conceptualization of variable clus-

ters that affect utili4tion, a procedure for Measuring utilization

11

potential was described. 'Preliminary results of the pilot effort, within

the context of Title / ev luation utilization, were presented,

Maximum likelihood factor analyses using a subset of items with ai

random half of the data base Alearly indicated that a set of correlated

factors related to the Leviton and Hughes' factor conceptualization were

I

found. Additionally, a,decision-making factor also appeared that may

have been due to the wording of same items. ---

The exploratory factor analysis results were not confiTmed on a second

sample. However, this should not be taken as an indication that either

the Leviton and Hughes conceptualization was not confirmed or that an

instrument for measuring utilization potential is not possible. By split-

ting the sample, our effective n's became quite small for the application

of factor analytic techniques. When sample size is considered, we feel

the results are highly suggestive. The possibility of developing a scale

18

to assess utilization potential is very real and its 4evelopment uld

provide a powerful methodological tool for utilization research and

for ideintifying areas where training might increase utilization.

References

Alkin, Marvin C., Daillak, Richard, and White, Peter. Using Evaluations:,

Does Evaluation Make a Difference? Beverly 4115, CA: Sage Publications,Inc., 1979.

r

Burns, Phillip. SPSS Subprogram JFACTOR-1J6reskog Factor Analysis. Evanston,

IL: Northwestern University, 1977.

Caplan, Nathan. Social research and national policy: What gets used, bywham, for what purpose, and with what effects? In Stuart S. Nagel (Ed.),Policy Studies Review Annual, (Vol. 1). Beverly Hills, CA: Sage Publi-cations, Inc., 1977. (Reprinted from Inteinational_ Social Science Journal,1976, 28.)

David, Jane L. Local Uses of Title I Evaluations. (Research Report EPRC 21),Menlo Park, CA: SRI International, July, 1978.

Dziuban, Charles D. and Shirkey, Edwin C. When is a correlation matrix appro-priate for factor analYsis? Psychological Bulletin, 1974, 81, (6), 358-361.

Haenn, Joseph F. Reasons why evaluations and testing don't inform. Paperpresented at the Annual Meeting of the American Educational ResearchAssociation, Boston, MA: April, 1980.

Hansen, Joe B., Martin, Joy M. and Oxford, Rebecca L. The Use of Evaluation:An Analytical Review. Region VIII TAC, Portland, OR: Northwest RegionalEducational Laboratory, 1979.

Jiireskog,.Karl G. and SOrbom, Dag. LISREL IV User's Guide. Chicago: Inter-

national Educational Services, 1980.7fr

JOreskog, Karl G. and Van Thillo, Marielle. New rapid algorithms for factoranalysis by unweighted least squares', generalized least squares and maxi-mum likelihood. Educational Testing Service, Research Bulletin 71-5.

Leviton, Laura C. and Hughes, Edward F. X. A review and synthesis of researchon the utilization of evaluations, Evaluation Review, 1981, 5, (4), 525-548.

Patton, Michael Q. Utilization-Focused Evaluation. Beverly Hills, CA: Sage

Publications, Inc., 1978.

Revelle, William and Rocklin, Tom. VSIMPL. Unpublished manuscript, 1979.

Appendix A

Summary of Utilization FactorsInstructions for Developing /tem Concepts.U414ftion of Evaldation Information Matrix

4.

tilization Factors

'Leviton and Hughes (1980) have proposed a five factor structure for the construct

of utilization.Although it is possible to question any or all of the proposed,

factors, ve helieve that out goal of measuring utilization will be more readily

ichieved.by hsing a relatively concise conceptualization of the construct thtaby attempting to employ one of the more ekaborate models described the,litera-

ture.

As described by Leviton and Hughes, eadh of the five factors 1ssociated with

.eviluation utilization is comprised of a Cluster of discrete variables. We have

chosen to paraphrase, and call these variables "facilitating conditions";

Factor Facilitating.Conditions

1. ReleInce

2., Commun4catimx

3. Credibilit:y.

4., User InvOlvement andAdvocacy

1'

eir a. Evaluation addresses client needs; iA.ini Title I program, evaluation dataare perceived as relevant to informationneeds,of teachers, program people, and

administrators.

b. Evaluations provide timely infoimation,i.e. data are available when needed fordecision making.

a. There is formal and informal interactionbetween evaluator and users ofevaluation information (teachers, programpeople, administrators).

b. As information filters through the bureau=-cratic hierarchy, no information is distort

or omitted.

The evaluation information communicated to

each user is specific to his/her needs.

d. The information cammunicated to each user

is in a form he/she can readily'assimilate.

a. Information from evaluation is seen asan important part of the total knowledge

base on which decisions are based.

b. Information users have confidence in the

evaluation findings.

a. Key personnel have a sense of ownershipof the evaluation results, and believethat results will validly inform decision?'making.

b. Information users are willing to seriouslY'sconsider evaluation results in'the context

of their decision-making.

A fifth factor, "Information Processing", is described to include such conditionsas clarity of reports, and unique characteristics'of information required by dif-ferent decision..makers. We feel there is much overlap with "Communication" in thisfactor, and so have elected to omit Information Processing from our schema ofUtilization structure.

,

Th f r factor structure we have chosen to repredent the notion of evaluation

ut ation can be used, therefore, to develop a measurable definition of utilize-. We propose the following definitioh:

Evaluati6 utilization occurs when decisions are made in the context of:

1. Belief in the relevance of evaluation information2. Awareness of the importahce,of communicating evaluation results

to all potential users3. Faith in the credibility of the data, and4. User involvement and advocacy of evaluation as decision-making tool.

,

Instructions for Developing Item Concepts

-

The Region V TAC used a modified Nominal Group Technique (NGT) as described byDelbecq et al (1975) to generate items for the four columns and five rows of thematrix. Based upon our experience with this method of item concept generation,we offer the following suggestions for a procedure to be used by all TACs.

1. Appoint one person to assume responsibility for coordinating the item concept writing effort. This person,should carefully read the position paper,identify persons to participate in item concept writing, distribute copiesof paper to identified writers, chair two meetings of writers, and send inthe iesulting item concepts to Region V.

We suggest that each TAC focus on writing item concepts for two columns (tobe decided at this TAC Directors' Meeting) of the matrix. Each individualitem writer.should write items for only one column.

2. The item writing coordinator should 'onvene a meeting to explain the modeland the nature of the proposed instrument. This will involve:

a. describing the four factor model derived from the Leviton and Hughespaper;

b. describing the five Title I decision areas;c. displaying the matrix, and sample item cOncepts for one column (provided);d. describing the two scales on which LEA respondents will be asked to rate

each item. Namely, an LEA Tit e I Evaluator and/or Title I ProgramAdministrator will be asked to rate the extent to which,the conditiondescribed in the item exists n his/her district, and also to.rate theextent to which Technidal Ass stance would be welcomed to facilitatethe described condition;'

e.. assigning one column to each individual item writer.

3. Item concept writing may be done in the meeting, or individually. Ourexperience suggests that individual item concept writing might be mostefficient, once all participants fully understand the task.

4.. All items should be returned to the CoOrdinator who will have them typedand copies made for each item concept writer;

5. At a second meeting, items are discussed and ranked, by matrix cell. Duplicate items and/or rela concepts may be eliminated or combined and thelist refined prior to ranking.

6. All item concepts, ranked in the second meeting, should be returned to. Laura Crane, Region V, by November 3, 1980.

TAC V ETS/MR0

Matrix Column 3: EVALUATION' CREDIBILITY

DECISION AREA /TEMS (write two for eaChdecision area)

1. Funds Allocation

/.1. Title.I Evaluation Data is an important info sourcesin grade allocation ofTitle I funds.

1.

2. -Program Adoption. or change

1. Title I EvalUation Data is usually a very accurate source in pinpointingprogram problems.

2. Iltle I Evaluation Data is normally considered in the context of progrma'planning.

3. _Staffing

1. Title I Evai.uatipn Data provides an accurate assessment of staffing problems.

!. Student Selection

1: Student test data is a major basis (but not only basis) fOr project selection.

5. Test Selection

1. Title I tests are routinely examined for appropriateness to program goals.,

2. Title I test resulti are routinely eXaMined for validity (Proper levels'adminittered,

UTILIZATION OF EVALUATION INFORMATION'

ITEM MATRIX

DECISION AREA

Relevance

FACTOR

Communication Credibility 1 User Involvement

Funds Allocation

Program Adopt&or Change

Staffing

Student,,Selectinn

- Pr.

Test Selection

75-8vo

4

Appendix B

30-Item SubsetaelevanceCredibilityCommunicatIonUser Involvement

.

Aur93

er

S.

. 1 'i 1 ., '0, $

't ,,

I.

'Ff.ELE-VA:,NICE .t

'5.4:NALUATIMDATA ARE USED TO DECIDE WHICH TEST BEST poatHEs THE oa4EcTrvEs

1

OF 7HE PROJECT.

;7. STAFF WMADkINISTER AND USE TESTS CHECK WHETHER CR NOT THE TEST CONTENT

MATCHES INSTRUCTION BEFORE SELECTING TESTS. .

1

I 13. STUDENT PERFORMANCE DATA ARE AVAILABLE IN TIME TO ASSIST IN STUDENT SELECTION.

a

24. TITLE I EVALOATION REPORTS ARE RECEIVED IN TIME TO ASSIST ADMLNISTRATORS IN

rWING DECISIONS ABOUT CHANGING THE PROJECT(S).

O'

30. EVALIATION DATA ARE AVAILABLE IN TIME 70 BE USEFUL IN DECISIONS CONCERNING

Ttsr*SELECTICII

.33. EVALUAtION INFORMATION FROM THE PREVIOUS YEAR IS AVAILABLE IN TIME TO INFORM

THE-FOLPHING YiAR'S FUNDS ALLOCATION DEOISICNS,

401 EVALUATION DATA ARE AVAILABLE WHEN STAFpING EECISIONS ARE BEING MADE.

C R E L I TY

/..

3.-TITLE I EVALUATION PROVIDES A SUFFICIENTLY CREDIBLE SOURCE OF INFDRMATJON

10 BE CONSIDERED WHEN FUNDINGDECISIONS ARE BEING MADE.

ID...EVALUATION USERS HAVE CONFIDENCE IN. THE QUALITY OF THE EVALUATION.DATA USED*WM

IN MAKING FUNDING DECISIONS.

27, EVALUATION DATA ARE CONSIDERED SUFFICIENTLY VALID FOR MAKING DECISIONS' ABOUT THE

HIRING OF AIDES.

32. PROGRAM PLANNERS BELIEVE THAT EVALUA3 DATA ACCURATELY REFLECT THE STATUS

' Of A PROJECT. P

58, TITLE I STUDENTS ARE SELECTED WITH CONFIDENCE ON THE BASIS OF.EVALUATION DATA,

63, EVALUATION DATA ARE CONSItERED A SUFFICIENTLY CREDIBLE SOURCE Of INFORMATION

FOR MAKING DECISIONS AbOUT CNE METHOD OF INSTRUCTION VS. ANOTHER.

4

4

COMMUNICA.TION

21, TEACHERS ROUTINELY RECEIVE EVALUATION INFORMATION ABOUT THE OVERALL.IMPACT

OF THEIR TITLE I PROjECT.

28. A DESCRIPTION OF DISTRICT TEST SELECTION PROCEDURES IS-AVAILABLE TO ANYONE,

I NTERESTED

2. PARENTS UNDERSTAND HOW THE STUDENT SELECTION PROCESS.WOOKS.

31. PARENTS ARE AWARE OF HOW EVALUATION RESULTS ARE USED IN PROGRAMMATIC DECISIONS.'

34, TEACHERS UNDERSTAND THE IMPORTANCE OF ADHERING TO_THE TITLE I EVALUATION AND

REPORTIliG SYSTEM REQUIREMENTS FOR STUDENT SELECTION.

35, 5FLUATION DATA ARE PRESENTED IN A FORM WHICH IS USE5i. FOR PROJECT PLANNING.

39. MEETINGS ARE CONDUCTED TO MAKE SURE THAT STUDENT SELECTION CRITERIA ARE CLEARLY

DEFINED AND LNDERSTOOD BY ALL APPROPRIATE PERSONNEL,

49, TITLE I ADMINISTRATORS iINWHOW EVALUATION RESULTS CAN BE USED TO DETERMINE.

. JF A Ned TEST.IS NEEDED.

53. EVALUATION INFIDRMATION USEFUL-FOR FUNDING,DE9SIONS IS PRESENTED IN AN UNDER-

STANDABLE FASHION.

.11I ' 'V. 0 3 ' 104

-

4

USER IN'VOL-VEMENT

2. USERS OF TEST INFORMATION ARE.PERIODICALLY ASKED FOR _THEIR REACTIONS TO THE

TESTS IN USE,

8. PROJECT ADMINISTRATORS ARE ROUTINELY INVOLVED IN EVALUATION PLANNING SO,THE

, EVALUATION WILL ADDRESS THEIR OEtIFI-O NEEDS,4.

15. TITLE I TEACHERS HAVi' ACCESS TO DECISION-MAKER§ TO VOICE\THEIR SUGGESTIONS

FCR PROJECT CHANGES.

37. TITLE I STAFF REVIEW EVALUATICX.1 RES6LTS FOR PURPOSES OF,4LLOCATING PROJECT

FUNDS TO DIFFERENT PROjECT COMPONENTS. '

43. PERSONS.RESPONSIBLE FOR FUNDING DECISIONS ARE ROUTINELY INVOLVED IN EVALUATION.

.PLANNING ACTIVITIES.

57, PROJECT LECISION-MAKERS ARE WILLING TO SPEND TI-iE TIME REQUIRED TO INSURE THE

USEFULNESS OF EVALUATIONS FOR THEIR NEEDS,lb

61. PERSONS RESPONSIBLE.FOR PRIM PROJECT CHANGE DECISIONS ARE FORMALLY INVOLVED

IN EVALUATION 1PLANNING ACTIVITIES,.

4

_

, Appendix C

Subsample A Correlation MatrixSubsample A Pattern MatrixSubsmnple A Structure MatrixSubsample A_Factor Correlations

4

Subsample A Correlation Matrix (04S)

712-.F.

.2831912E18E

IZ7E .25944I43E .15940157E.

T--I61E.16188.15917

I?1E, .09009128E .12373129E. .02050

.v7213-6--

.06154-7-71-3117. I34E

°

135E .18592.23586'

, 149E .06704, .153E .12017

. I56E .1419613E .05931I1OE .0811-50-127E132E

I 13E130E33E41)

I2EI8412.5t.137E143E157E161E121E

q 128EI-29E

134E135E139E.145E153E156E

4- Ilk;58E

bSt.

:17E,

140E

.037-24.12682

871-1-26162

. 3343G

. 25482

.12E84-

. 12713

. 03431

18E

283191:00000.361-011.26891.48381.35059.40694.22747.2-.7642.04248

..09629. 36343

.

.4245041657.33035. 39976.41759--

1-37E. .

4

. .09362 .25944.38100 .26891

1."0-00-04----- .25.8.5"9p.25859 1.00000.47300 .3139250321 .32056 A_

.32430 .18022

.27829 .25E46

.09500 .31700-.02153 .03688.D6492 72:98:.32610 22156.29494 ,501a13--.13798 __.1637/_. 352-52 .26562.35837 .43E73. 42611 .39989. 33943 .21256

---73-6522 425-013.21696 51334.31597 41976

__4.269.5A___ _.4.1647

. 2'024 .39990

. 1 277 .30258

.35684 . 437211-:1.-07f4-- ---.43-02-4-.21767 41211.14702 .39698

Yk§A1.11

. 3Q03535286

. 03572-.1:5573- 4-. 28122.120454

w3ltiO

. 16188

. 35859

. 32056

. 502051.0000031190. 3s75222525.28714. 31811-e:28573.51033. 29495.43911--.52885.50200

1591.7.40694-TM 47-18 0 2 2

.6087 0

. 31190-1.04000.17675..2426A19820

. 39076-.31163.23143. 26246Z4253

. 30830

.43979

15112.2370: .30510.27243 43357.35gS3 --41-21.1-13-2--

.25950 :.1447518065 23659

14881..2-719-9----- .-Z3538-.38,777 .2-0260

1941-5-47958 .249b6'

I43Z'

.15940 .

. 483811

.313921.00000

.60870

. 21183.626087. 09884324-8-7.24568437686

_ 014_6_42_.24818

,.46538.51117.36411

.Z420030057

.02690

.09488-.19888

.1825023696

-1-21r7-

.09009

.22747

-7.28E-

.12373

.27E42 .

;Min.31700.26087.22525.72'426U-

T219Z

.02050 .

.04248

,.0688.C9884.28714

.26263C9329

.25646

.21183

.39752sit:75

1400000.24285

-.26263 409329 1. 000044-27949 76172.11418 .14713

.33404 2340Q 18624

.18763 1419: 34260.3796T .WEEFT-

.37175 .37719 .19951

.35288 .28701 .24933'0026

.18407 .09544

.28083 .43410 .C7845

.25325 .25078 .193,94."

.20084-- a-CSBUB-

.22635 .14246 .09616

.16511 13883 22118

.19344O- .24507.3411SZ

.12623=.01129--

. 3 5 2 7 6 .38391 .G49.495 .'.02172

.22940 .42542 .14644

2t5

_ _/ ZE__LaE_-I1.5EI37E

fV4121I 28EI29E

31E134E135E.1-39EI49E

13E110 E

158EI 63E15E . 25067 ... 01496

- I ZE__ . 38232 _ s Z4Et2_1. I13E . 251 51 .1.5197

I24E .2 365S '.16992'I 30E . 221 98 .14524133E 10_3_336 .1 3820

I34E

- .17296 00615'433595 OS629

.1 8992 .32610. 25985 22156. 32487 .2450.33811

39076 .31163'.3 4253 34434

2794c. .11418.61728 .147/3

1.0000C 26155 ,

:388,2.261g51:H8gg

23708-37328 .37700,32087 36220404'4 _ _ 5 .--. 25914. .16742.2 70 350.21228

----.-2-8T13.21453

.33490

. 25388.19527'

.35605

140= 29354 - 10572----- -.49-3-00 .244-55

135E

.4,18592.36343.2 4 4.5 1t.ro-33--

. 2314333404

..23400.1-8624

' .38852*."02ri

29-8135-.40053.:41g4;.41475-.45872.18gi.4 2

22-25-02934+827416

_L2.0_200. 35743-32523.36162

32406

.2358604871

.19 798

.1637714E42

------. 25 li.5"".26246.18763

.1_1-41953426031748

.237082989 &

0697632472370220111 4

.10187

.19561

.1 2814

.T6-25 2

.23440-07234_._09.723

.25357-

. 03440

. 18591

153,7 156E.12017 .14196

1 8Z -.--41817/ 330-35 .. 3 837

. . 4 67.. .1i8i9'I43E . 4+65 38 ..51117

. 528 -17. 502-011--1 61 E .30830 .43979fill :MI; :1R581IZ9E 41 9551 Z4-913

!!,* TE . . :iEM :'igt4.,......LI 59337 l+95L7ti

. 32472- .-370-224, 47665 29809

1..00000 53128149E,I 53E ,156E . 531 28 1.00000

5 '18313 .28681,156I1OE127E- I 32E1581163E15E

, 17E113E124

--TAU

. .501 31.50849

_4_39669 _r- . 3607926337

.11673.18068 ,

31481.1 9318..32575. 30843. 3031 5.1 4196.17607'.178:0-75i 20123r.32305. 33258

of6421----.1 7200.32127 4

__. :23403.44678

I4SE

4, 06704.42460. 3.5252'. 2656224818:43911--

.2425353866

.37967G6052

ie 373283770.40053:16-976- ---7"

1. t0000.47665-.298P9.31 71

r 9.A145+8

27614

4,292024, 27038

21128.36735.38858. 284013495-3-

I3E. 05931 0-8098.399 /6-. 33943 .36522. 31256 .25033:36411 . 39097.14659; .38-056

' .2951 7 . 36208- . 25194 32070

.11+623 .21 306

1 2 7

t3724.-341-17.21696. 51334.24200.318f

15112407181;9544

. 0-0-261:1 . -D-31:11TE ..711786. :iP11+ :Bigg

.414+75 .45872 41825

. 0111-4---- 7 -.10-1-8.7 -4,-19361

.31710 .50159 .3554859819 .50.131 .50849

.28681 e 31481 19318-1. U00-0-0--- . b,5461. 74--a-s91a--

.55461 1.00000 .40504.40390 *405'04 1.. C0000.26998 28703 .346774 2049 514-38 294-99.22350 .33729 39249.36452 .30764 4,23352.20288 .14547 . 08627-

... 27-54-3---------72-991-2 7u1.6-n---27499 .30974 . 31077

.31 314 .3690_0:- 392i0?2_379.______. ..265.71 .4Z4Z834670 . 4147-4 21716'

_

..r

,

I8E

I 37EI 43E

i 1 57E161EI21E

0- I 29E

i i4EI35E

3-I 49 EI 53 E

1 32E158E163E15EI7E

177-rfsr--! I24E

I30-E133E

- 240t

31--WEI61EIZ1E128E1-Z9t131E

4 iTa':=39E

rit9E

Ii0E127e

if -igiE

1 firI7E

130

s.

132E

.12682 .09252

. 30935 .11846* 31517 .26854.41976o :,. 16447. 300574 .27689:2-3745-----727 2-43-. 3 0 5 1 0 .43357. 28083 -.25325.4 34 10 . 25078. 0 7845 .1 9394

.,

I63E18731 .

15E*

.26162,. 22803 . 30035 . 35286.21617 6 24024 .1727737448 .39900 .30258.10599 02E90 jp 09488

25-9 5-0. 21182 .14475 .23059.20084 .22 E35 .16511.13482 .14246 . 13884*.05806 .09E1 E .22118

:MP) :iitR4.40261 .22250 .29348 .27416 .20200.12-1114----T-;16252-----TZTWO------a-723-4----- .T9723--

.26847

.18068

.176U7

.21288-'

.1454Z

.C86Z7

.10730-,7-truir0.10314.58530

1.0000U974-2

.129134

.k32331

.17476-;-113-665---------

.27E14

.39669

.32575

.26998

:i0M1.00000-.-31835----1ortrow.31786.1.31-88.10730

-M5407.35728.39942

421E9i0"0--- ---.-34151F0------.Z507-9----.-

.26215 .29202

.36079 .26337#30843 .30315.32049 72350-----51438.29499 -

.33729

.39249.31855 ' .31786

.271198..11E73.1419636452.3 0.764.23052.13186

.31155-9.30589 1 . 0000004944 .15169

......01006 .10314.18581-- .19591.27950 .35377.18422 .30475.21588 .05917

.. U4944

.151691. 00000..56530.3nia.--28468.48829.23190

-T2-031-13-

113E :

.25482

.11357235684.

. 3721113.864. 3604V.14881.1 9344.2 45077-1-2623:2 51 51.1 51 97357432-535T--211 28

:irctqZ7543---2991214460

. 3059D019.19591. 33941.09742

1. 00-na.33916.35931#2123961.25

. 0-

\124E 130E

.1 2684 .12713-.18573 -72n-2Z-. 10714 .21767.4 3024 . .41211

1___._1_066.7 __.: _Al-8.2"V- -e . 2739g-- .3877

.:33E I4OE

.03431

_Iiiial

-724323--.q893

.3686T

.16975

.46500

25533 -.47958-.19415 24966. 23055 .22940.30882

40-2172 :1iii4.03336 .29354. 13820 .10572

--:'-gilit --:-Mgt-7--28401 ;34953

..fini -6.1W2a

.zeS7g . 341,10

.26573 .$1474

.42428 . 29716

.39421

0.05917

....,.,25.908.-----

. 25179

.34911.

-:23190 .- . 20319.1747.6 . 186654-2122'S-'. 6 7 21 5 .45956. 5442 .35952

13atli-- -1:4811--

. 23538

.3495034863

.23659

.20260

.35276

.38391

.22198.. 16992 14524. 32523 3 6 16

0 344 0- 1. 85 9-12

3 6 73 5 3 8 8 5 8172 0 0e0123'. Z7499 .51..5i.4.30974 .3690031077, .39240

.3172-6 .39942 '

72790 ,.1842-2.35377 .3047528468 .48820

. 2913* .32331.35931-

q.000.00 .653084'65308 - 1.00000

---#..0.7215 .54382.4595.6; -695. 2

-

.37

ILE*- SAvEUS (CREATEC - 01/28/82)

AFTER ROTATION wrTH KAISER NORMALIZATION--TOBLIQUE FACTOR PATTERN MATRIX

'DELTA = 0

Subaample A Pattern Matrix

FACIOR

.0019513670.05896.26338.09355.17036

-.08666

FACTOR 2

.160-617

.2Q3-84

.06424

.15498.15526..00817.06178

/2E

iliE137EI43E157E41E128 i

IREI35E139EI4SE

.-04644

.15284

.00571',1Q270

-.35983.19300.01158.948-03

#113(04..04420.0561601242.02296

-e.05705.04465

-017377

132E158E

FACTOR 3

-.14166TM5 1.19323. 09586

-.13114

.0094105721405154.033.37

--.25458-.04641.13218.04043O 02-9-85

FACTOR\je

.21 07

.16 69.05781.01535

.2004044438

-.43853

FACTOR 5_ _ _ _

. 04186-9E2585-..1364v. 30949. C3256. 00966

IR'?94.21448 .31429.01026 .07398

.22300 .04332-.26218 ..00368.13096 .1502306-13 -.C7462

1.0k 98 (1129

e 4 2U.175

.00738 . 30806. 01862 .12698

.17261 .06740

. 09677 .01649.16110.164.66.

44

1 tI7E113E124E-130E133 E140E

. 1

.026800.

.54796use -.10208 .11844

_7!..15657 02226. 08074- .25663-. 04839 .00247.17238 -.04877

-T

t

12E18E '

115ElarE143E157

FA"TOR 6

.15301

.48301

.39469

.87653

.21534

FACTOR 7

0564000939512170-.03231.12212

0 s, ... 02112 -.4-8494 .13846

-.02188 -.04237 .1948§-.18183 ....n...0.4788 91.,P.

;091-02,--- 1...07724. 15306 -.01509 .76842

-.02243 -.08134 .35905

FACTOR 8

.34743-T.12484.01586

.23f5T-

.05449

.03579

FACTOR 9

.03437.19421.322390.(t55.15134.33855

12 E129EI 31E134E135 E139 E149E153E

01. 06274

-.04088. 20026. 1-0-414--.06120-.02555-.04851. 01808

.

.1 1

.10276"

.831

.665

.12622

. 13173

. 42054-.09001.13275

.05838.07339. 08509

--03-2-9139.09328.08470' .02073. 00960

.01832.01836

-.05753-.113-76-9.2270e.19765.00040

-.0900

ii'tI32E158E

I7EI13E

4 .

. 19829 -.10671-.134740- ---ii-.113011ET-

. 14326 . .00)969 ,.

.16108 .i99.0::87183

.".0.119Z22-966.07283

. 0038V.05495-40116V94

6g811746

.243

.40327 ,

-0772-4..0039i.61244tO?Ci5.9. 00664201-98-

.10188

.T3261----

. 06411..03914

-.nnis.20 2//--

i.24-79.10191 .02493 5640

-.00562 .05743 -.05428-allil -.02247 f1132

_

-"AAR

:3 8

4

3

5

6

a I 2EI 8E15E

TO" I 37E

Subeample- A Structure Metrix

FACTOR 1 FACTOR 2

.; 1 28EI 29EI 31_E

$0.

.7, I 35 E

fl4E

igtEtin132E158E

I7EI1 3EI24E_

14 130E:9 33E

140E

:.

3r) I 2Ei 18E

'8! 115Es7/ 137 E

I 143E4-al I57E

: -ftE. 128E°i 129E'2 I 31 E

1 .54t

"4 filE

. 13525

. 39134

. 37237

. 49103

. 45021452101

.345644-09517.2046E. 30548. 59642

4ao.

-797483

2665035752.19682.35477. 035?4.187o9

FACTOR 3

220722912727001

...42883

::gig)13071

FACTOR 4 FACTOR 5

-.4,07769-.43E02.r.36311.-.25278

.07286 ,009874

.25538 -.27461

. 21043 ..36051-.27532-.31835

,:

. 45 48 4-6

.40611

.33156

.27953

.191123

.15725

.2048018666 .27801'.35954 0+539327301 .21230.44948 14759

.22062 .15299

.61007 08946. .52039 .06381

.304-51T ./4357- .87947 .17404

149051 32069

:aN An

_1 28193 . ... 381 32:1-9583--.10811 -.36135.02658 .'.33522.12692 ...99155.(645 -.i-1344.54105 .14766.23134 -.22833

6 7___ E A

-.36551-.10071.0502(

.... 22133*4)0890*00004

:Piga.26755 .20675 . .01747.12952 .80464 00555.41811 7519t .02900alrE197,7-.38D33.16(48 .416081

29843 .06083-.09942

.6rt

-. 09366- 30550

=.42?42

=:gt5-.39515-.13203-.43627-.39704'-.42263.11337

-.99280..r.51l111

-710 .3331337-83944211

. 28501

.24201

. S1/14112995437468

14524. 21837. 14621.51474

.250Q2

.42820

. C1390

.23529

. 17003--

.41375

. 17g836o3529448

--.2827 8-.27278-.26E72-.Z3536- 27488-.21557

34342

26E68 .76562- 36560 :52423

QR 9111834

-.0093.041Z68 .

*79776-3096251776

.eg5ii

.10865

.15345

.11-Pg

*31196.44534

-1-4-tz-4+---- ---:-Ega-------gligi-- 1119-:.41 7108

ElenI.32E58E

1 457622.372

93441/

BE ' 2426317954-

4. I 24E

7-40E

3033E 20152

....TO 987

34435 ,

.29803 27423

.21100 .018710977362031

9 8 .172241-013-8 02 07505452 07982

31 WE.24266

.249314 04478 .40 860

.05006 43024 28873-1-1-17-62

.14146 . 0442c 7264-48-8135

25168 é 9,048 7

4

70,

Subsample A Factor Correlations

1FA OR 2FA OR 3

FACTOR 1

1.00000.14801

-.36633-.34874.3200C.42161.16271.05199

, FACTOR 40' FACTOR 5

FACTOR 6'? FACTOR 7p--"FACTOR. 8

FACTOR 9

z

.32374

sFA,CTOR 6

li-FACTOR .421.61FACTOR 2 .19154FACTOR, 3 -$28952

a, FACTOR 4 -.35843tAUTOR 2 .22706-

:I FACTOR 6 1.000TO

REM ;rzy-oir

.22393

.18878

FACTOR 2

.148 i1.0°0 0-.16J3

--=.21247--.28732.19154.07606

-.13004--.09617

FACTOR 3

..36633-.16173

_1.00000

-.28196-.28952-.20330

-.23869

FACTOR

-.39874-.21247.30174

-.35216-.35843-.20996

-.15E74

4' FACTOR 5

.3200028732

-..c8190

1.00000.22706 P'.10,789.C4017.25616

FACTOR 7 FACTOR 8 FACTOR 9

16271 -.05199- .32374.07606 -.13004 .09E17

-.20330 -.02048 -.23869-.20996 -.094-20 -.15E74..111789 .714-01( b.22393 .04104 .18 87 8

1.00000 .14733-.065491.00000 01550

133___

0155R 1.00000

2;a

'.4 t

66

.0

4

-