8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

1/24

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

2/24

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

3/24

CONTRUCT VALIDITY

3 ASPECTS

SUBSTANTIVE VALIDITYSTRUCTURAL VALIDITYEXTERNAL VALIDITY

ACCDG TO LOEVINGER THESZE THREE ASPECTS AREMUTUALLY EXCLUSIVE ,EXHAUSTIVE OF POSSIBLE LINES OFEVIDENCE FOR CONTRUCT VALIDITY AND MANATATORY

AND ARE CLOSELY RELATED TO THREE STAGES IN THE TESTCONTRUCTION PROCESS, CONSTITUTION OF THE POOL OF ITEMAND CONSEQUENT SELECTION OF ITEMS TO FORM THE SCORINGKEY AND CORRELATION OF TEST SCORES WITH CRITERIA ANDOTHER VARIABLES

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

4/24

SUBST TIVE V LIDITY PH SE

IS CENTERED ON THE TASKS OF CONTRUCTCONCEPTUALIZATION AND

DEVELOPMENT OF THE INITIAL ITEM POOL

review of related literature - important as if the reviewreveals that there is already a good psychometricallysound measures of the contruct, the scale developer mustask himself whether the new measure is, in fact necessaryand if so why

however, the existence of psychometrically sound measuresof the construct does not necessarily preclude the development ofa new instrument

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

5/24

construct conceptualization

- 2nd important functrion of the revie is to develop aclear conceptualization of the target construct

the literature review will reveal alternativeconceptualization of the construct, related contructs

that are potentially importantand potential pitfalls to consider in thescale development process

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

6/24

development of the initial item pool

very critical step in the scale

construction

no existing data analytic technique canremedy seriousdeficiencies in an item poor

primary consideration during this stepis to generate items sampling ALLcontent that is potentially relevant tothe the target construct

overinclusiveness - should characterizethe initial item pool

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

7/24

relevance - refers to the appropriateness of ameasure's items for the target construct.when applied to the scale cosntructionprocess this principle suggest that allitems in the finished measure should fall

within the boundaries target construct

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

8/24

representativeness- refers to the degree to which the item pooladequately samples content from all important aaspects of the target construct

-item pool should contain itemsreflecting all contentareas relevant to the target construct

-item p[ool should include items reflecting alllevels of the trait that need to beassessed

classical trait theory - ussually favors selection of items withmoderate selection of items with MODERATEendorsement probabilities

item response theory - offers valuable tools for quantifyingthe "trait level" of the items in the pool

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

9/24

item writing

item clarityresponse format

use of simple and straightforward language

avoid trendy) or colloquial expressionavoid complex or convoluted items

writing items with stemsi worry about ....

writing mixture of posivitively and negativelykeyed items

context neutral

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

10/24

item writing

response format

dichotomouspolytomous

use of anchoring schemes based onagreement - strong disagree to

strongly agreeperceived similarity - uncharacteristic of me

to characteristic of me

frequency - never to always

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

11/24

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

12/24

STRUCTURAL VALIDITY PHASEPSYCHOMETRIC EVALUATION OF ITEMSAND PROVISIONALSCALE DEVELOPMENT

LOEVINGER defined structural component ofconstruct validity as the "extent to whichstructural relations between testitems parallel the structural manifestationsof the trait being measured

structural relations between test aned

nontest manifestation of the targetconstruct should be parallet to the extentpossible - structural fidelity

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

13/24

item selection strategies

rational theoretical approach- scale developer simply writes items that APPEARconsistent with his or her particular theoretical

` construct

pitfalls - discriminant validity suffers

replicated rational selection (MMPI2) - involves askingmany trained raters (who are given detaileddefinition of the target construct - to select itemsfrom a pool that most clearly tap the construct,

given their interpretations of the definition andthe items. Only items that achieve a high degreeof consensus make the final cut

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

14/24

criterion keyed item selection - items are selected for ascale based solely on their ability to discriminatebetween individuals from a "normal group and those froma prescribed criterion group. Item content is thereforirrelevant

pitfalls- atheoretical and fail to help advancepsychological theory in a meaningful way

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

15/24

internal consistency approach to item selection-the goal is to identify relativelyhomogenous scales thatdemontrate good discriminant validity

accomplished with some variant of factor

or component analysis, oftencombined with classical and modernpsychometric approaches tohone factor based scales

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

16/24

data collection

psychometric evaluation of items

factor analysis- extrem,ely useful to the scaledeveloper who wishes to create homogenous scales t

that exhibit good discriminant validity

ultimately, however, the most important criterion forchoosing factor structure is the psychological andmeaningfulmess of the resultant factors

good candidate items are those that load at leastmoderately on the primary factor and minimally on theother factors

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

17/24

internal consistency and homogeneity

once a reduced pool of candidate items has been

identified throughfactor analysis, additional item level analysis should beconducted to hone the scales

the goals at this stage is to identify a set of items whoseintercorrelations match the internal organization of the

target construct

estimator -- coefficient alpha which are are functions of 2parameters

average interitem correlation

number of items on the scale

alternativesexamination of interitem correlation (.15 to .50)conducting confrimatory factor analysis

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

18/24

item response theory

parameter for item difficultyitem discrimination

item difficulty - also known as threshold or locationrefers to the point along the trait contimuumat which a given item has a 50 percentprobability of being endorsed in the keyeddirection

discrimination - reflects the degree of psychometricprecision or information, that an itemprovides at its difficulty level

standard error of measurement of a scale isequal to the inverse squareroot of informationat every point along the trait continuum

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

19/24

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

20/24

EXTERNAL VALIDITY PHASEVALIDATION AGAINST TEST AND NONTEST CRITERIA

concerned with two basic aspect of construct validityconvergent and discriminant validitycriterion related validity

whereas the structural phase primarily involvesanalyses of the items WITHIN the new measure, thegoal of the external phase is to examine whetherrelations between the new measure and the importanttest and nontest criteria are congruent withone's theoretical understanding of the target construct

and its place in the nomologicalnet

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

21/24

convergent validity - extent to which a measure correlateswith other measures of the same construct

discriminant validity - extent that a measure does notcorrelate with measures of other constructs that aretheoretically or emperically distinct

can be assessed by MULTITRAIT-MULTIMETHOD

(MTMM) - in such a matrixmultiple measures of at least twoconstructs are correlated andarranged to highlight several importantaspect of convergentand discriminant validity

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

22/24

absolute convergent correlations will depend on specificaspects of the measures being correlated

concept of method variance - suggest thatself rating of the same construct generallywill correlated more strongly than will selfratings and peer rating

heterotrait-heteromethod triangels -correlationabove and below the convergentcorrelations

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

23/24

convergent validity correlations

should be higher thanthe correlations obtained between that

variable and any other variable and any

other variable having neither trait

nor method in commonconvergent correlations generally should behigher than the correlations in the hetero-

monomethod triangesthat appear above and to the right of the

heterosexual block

the same pattern and trait interrelationshipshould be shown in all of the heterotrait

triangles

8/14/2019 itemformulationpsychologicaltestingfinalxxxx1-110821092921-phpapp01.ppt

24/24