the concept 5.2 opq: inefficient measurement or broad

The Concept 5.2 OPQ:Inefficient Measurement or broad occupational assessment? (Subtitled: the Reality behind the Myths)

And…

The comparability of the 16PF Form A and the 16PF5: Some observations on the 16PF5 test. (Also subtitled: the Reality behind the Myths).

Barrett and Paltiel, January 1995: BPS Occupational Conference, Warwick University, UK

Click to add titleThe OPQ Model of Personality

3 attribute domains describe an individual’s personalityDeductive, a priori model construction An ENERGIES domain somewhere in the middle...

FEELING

THINKING

RELATING


Model constructs should be completely specified Relations between constructs should be specified a priori Causal paths should be defined with testable specifications The measurement model underlying the psychological model

should be completely specified (linear, non-linear, chaotic, probabilistic vs true-score) Testable components of the model should be clearly differentiated

from speculative opinion


2 Concept manual pages describe the features of the OPQ model of personality.No testable features of the model are specifiedA series of generalities serve as the basis for model

definition. E.g. “People vary in their behaviour” ... “Personality is a changing thing but still shows certain stabilities”


Various conceptual models proposed -tested statisticallyScales are claimed to be empirically-identified

“psychologically meaningful and measurable scales” “The Concept Model stands alone as a

psychologically meaningful and statistically justifiable model of personality...” p.4 OPQ Factor manual


Inductive methods of analysis confirm the deductive model 32 Concept scales identified by Factor Analysis

from an initial pool of 40 scales, followed by further item and factor analysis on 1500 items. CONTRARY to the received wisdom concerning

the OPQ, 30 Concept scales were initially identified by Item and Factor Analysis


Therefore, one testable feature of the concept model is that 30 scales should be able to be identified by Factor Analytic techniques. In the Concept Manual introduction to the OPQ

tests, the deductive concept model development is contrasted with the inductive factor model development. It is now apparent that this distinction is an illusion

OPQ Scale Development - Stated AimsHigh internal consistency Items correlate higher with their own scale than

with other scales High face validity of items Enhanced customer acceptability Occupational Relevance - work based personality

OPQ Scale Development - The Reality

Problem #1: Too long?The OPQ CM5.2 test seems too long - it appears to be a rather inefficient measure of some personality characteristics


By reducing the length of some scales with high alphas and high item-total correlations, from 8 items to a minimum of 4 items (67 items removed)

re-scoring the test and correlating the original scale scores with the shortened scale scores

re-computing alphas for the shortened scales

doing all this on two separate samples of respondents (N=420 and N=621)


Scale Name LongAlpha

ShortAlpha

No. ofItems

Long vsShort R

Persuasive 0.76 0.72 5 0.94

Modest 0.87 0.89 4 0.93

Artistic 0.86 0.85 4 0.94

Practical 0.91 0.92 4 0.97

Active 0.81 0.85 5 0.96

Averaged coefficients from both samples (N1 + N2 = 1041)


Problem #2: Evidence?Currently, there is no public-domain empirical

evidence available to demonstrate that all of the 30 concepts can be isolated as “measurable scales”.

Also, there is no evidence to support the empiricalexistence of the 8, 14, or 19 factor models


No factor patterns for either the Concept or Factor Models published by SHL

Matthews, Stanton, Graham, and Brimelow (1990)-94 respondents - 30 scales factored - only a 5 factor model partially replicated

Matthews and Stanton (1994) - 2000 respondents -item and scale factor analysis - 21 concepts identified -only a 5/6 factor model partially replicated


Barrett, Kline, Paltiel, and Eysenck (accepted JOOP, pending revision) - 621 respondents - item and scale factor analysis - 19 concepts identified, once again, only the 5 factor model given partial support.

No support for the empirical existence of the 3 super-constructs of Relating, Thinking, or Feeling.


Problem #3: Item Complexity?

Up to 21% of the Concept Model items correlate higher than 0.5 with non-keyedscale scores, even though they correlate higher than this with their own keyed scale score.

(from Barrett, Kline, Paltiel, and Eysenck (submitted) - N=621 respondents)


Problem #4: Occupational Validity?

Is this test really occupationally valid - any more so than a more conventional, general personality questionnaire?


When making multiple hypothesis tests, in order to control for the likelihood of obtaining statistically significant results by chance, under conditions of non-independent hypothesis testing, it is usual to compute a Bonferroni significance level (α) -which is given by:

α’ =(α ÷ n)

where α’= adjusted alpha (actual level to use)α = desired alphan = the number of significance tests to be made


So, if we carry out say 10 significance tests at a desired significance level of 0.05, we would in fact look for rejection of the Null Hypothesis at a p value of 0.005, not 0.05if we carry out say 30 significance tests at a

desired significance level of 0.01, we would in fact look for rejection of the Null Hypothesis at a p value of 0.0003, not 0.01


A concrete example of the effect of this correction applied to some SHL validation data taken from the OPQ Factor Manual, chapter: The Validity of the OPQ ... pages 4-5

234 trainee cashiers - OPQ scores vs performance ratings

Before correction - 61 significant correlations reported

After correction (16 scales correlated with 50 performance criteria)= 800 tests = an actual required p value of 0.0000625

on this basis, correlations above 0.26 will be significant at 0.05

Thus, 8 out of the 61 reported are conservatively significant.


If we ignore the statistical significance of the correlations, and instead ask what might the “true” value be for a correlation, we can compute a 95% confidence interval for any coefficient. That is, we can say that the true value would fall somewhere between an upper and lower value in 95 out of 100 studies. Thus, for a “significant” correlation between say Relaxed

and adopts correct security practices with cash (–0.213)the true value lies somewhere between –0.09 and –0.33


SHL have responded to the criticisms of “fishing” or “data dredging” by reporting a series of studies that correlate peer ratings of subordinates with OPQ Concept scale scores, within a predictive hypothesis testing framework:


Dulewicz(1992) - predictive -100 existing and potential managers - 40 job competencies - boss ratings -hypothesised relations with a subselection 0f the 30 concept scales - 57 predictions, 35 significant, average significant correlation of 0.23. Interestingly, Dulewicz refers to the 30 factors of the OPQ Concept 5 questionnaire (p.2). Also, he declined to use his own 12 “supra-competencies” in favour of the 40 because “...their definitions were more similar to the OPQ factors” (p.2).


Robertson and Kinder (1993) - predictive - 20 SHL studies in a meta-analysis - personnel practitioner and psychologists’ ratings of job competencies, correlated with OPQ test scores -average validity of 0.20.


Saville, Nyfield, Sik, and Hackston (1991) -predictive - cross validation samples of data - 440 (1984) and 270 (1988) managers’ ratings of subordinate personality and performance criteria. 5 Job Performance Criteria - cross validation over 10 out of 30 scales - average 1988 personality vscriterion correlation = 0.28


Gibbons, Baron, Nyfield, and Robertson (1995) predictive study - 468 UK managers - correlating new OPQ “big 5” factors with 5 scale scores from the SHL Inventory of Management Competencies. Results indicated average correlation of 0.26. What was so wrong with the OPQ FM5 questionnaire, as advertised in the 1993 Factor Manual?


Problem #5: Item Redundancy?

Are some OPQ Concept Scales really “scales” in the accepted sense of the word,

or do they consist of paraphrases of the same item?


As Peter Saville has indicated in a conference paper presented in 1989, the item composition of a personality scale should not consist of a single item asked several different ways (referring specifically to the 16PF scale of Harria vs Premsia(Tough Minded).


Factor I - 16PF Form A - Tough Minded

Item No. Paraphrased Text11 I would rather be a playright than a construction engineer112 Being a guidance counsellor appeals to me more than an

engineering manager.138 The beauty of a poem appeals to me more than that of a

well-made gun


Factor I - 16PF Form B - Tough Minded

Item No. Paraphrased Text87 I do not tend to be interested in mechanical matters112 I would rather be a philosopher than a mechanical engineer163 I would rather watch a concert artist than a programme on

new inventions on the television

ITC = corrected item-total correlation

Relationships (& SD) Thinking Style Feelings

OPQ Concept Alpha & Mean ITC’s

R1 .79 .50 T1 .89 .68 F1 .85 .59R2 .77 .48 T2 .90 .71 F2 .75 .46R3 .60 .31 T3 .87 .63 F3 .83 .57R4 .89 .67 T4 .72 .43 F4 .85 .60R5 .78 .51 T5 .74 .44 F5 .77 .50R6 .85 .62 T6 .62 .32 F6 .59 .30R7 .87 .63 T7 .77 .49 F7 .82 .56R8 .66 .36 T8 .85 .59 F8 .79 .52R9 .68 .39 T9 .59 .30 F9 .68 .37

T10 .80 .52 F10 .75 .45D1 .70 .39 T11 .76 .48

Scale Alpha ITC Scale Alpha ITC Scale Alpha ITC

N=621 Barrett, Kline, Paltiel, and Eysenck

Examples of OPQ Concept 5.2 scale items(paraphrased for copyright purposes)

T8-Innovative alpha=.85 Mean ITC=.59

8 I do not find it easy to generate creative ideas 39 People approach me for creative ideas70 I find it hard to be inventive101 New ideas come easily to me132 My ideas are rarely innovative163 I enjoy coming up with lots of valuable ideas194 I rarely have many original ideas225 I generally have an original approach to problems


Examples of OPQ Concept 5.2 scale items(paraphrased for copyright purposes)

R5-Affiliative alpha=.78 Mean ITC=.51

29 I prefer my own company to that of others60 I get much pleasure from other people's company91 Companionship is not a major concern to me122 I develop close attachments to people153 I rarely long for the company of others184 I have a large number of friends215 I do not like making new friendships246 I get enjoyment from the companionship of others


The POP questionnaireR1 Persuasiveness I like selling, whether ideas or productsR2 Controlling I like organising and taking charge of people.R3 Independent I speak my mind even if its unpopularR4 Outgoing I am an outgoing and sociable personR5 Affliliative I enjoy being in the company of others.R6 Soc Confident I am at ease in social settings.R7 Modest I am modest about my achievements.R8 Democratic I like the group to participate in decision-making.R9 Caring I am sensitive to other people's problems.T1 Practical I enjoy repairing objects or devices. T2 Data Rational I enjoy working with numbers and statistics.T3 Artistic I appreciate the performing and literary arts.T4 Behavioural I like analysing other people's behaviour.T5 Traditional I am described as something of a traditionalist.T6 Change Orient. I am usually critical of people's ideas.

The POP questionnaire - ResultsUncorrected Corrected

R1 Persuasive .64 .98 I like selling, whether ideas or productsR2 Controlling .73 >1.0 I like organising and taking charge of people.R3 Independent .48 .79 I speak my mind even if its unpopularR4 Outgoing .69 .98 I am an outgoing and sociable personR5 Affliliative .58 .88 I enjoy being in the company of others.R6 Soc Confident .64 .91 I am at ease in social settings.R7 Modest .68 .99 I am modest about my achievements.R8 Democratic .57 .93 I like the group to participate in decision-making.R9 Caring .51 .74 I am sensitive to other people's problems.T1 Practical .88 >1.0 I enjoy repairing objects or devices. T2 Data Rational .84 >1.0 I enjoy working with numbers and statistics.T3 Artistic .79 >1.0 I appreciate the performing and literary arts.T4 Behavioural .64 .98 I like analysing other people's behaviour.T5 Traditional .56 .85 I am described as something of a traditionalist.T6 Change Orient. .48 .80 I am usually critical of people's ideas.T7 Conceptual .68 >1.0 I enjoy the discussion of hypothetical ideas.T8 Innovative .74 >1.0 I generate creative and innovative ideas.

Median r .64 .98

The OPQ The OPQ -- Summary ConclusionsSummary ConclusionsThe OPQ Model of personality is poorly specified and untestableThe Concept Model is itself a factor modelThe OPQ Concept 5.2 might be shortened by almost one halfOnly about 20 out of 30 the Concept scales are measurableNo empirical evidence to support 8,14, or 19 factor modelsThere is little data to support the contention that the OPQ is more occupationally relevant than any other personality testSome OPQ scales appear entirely redundant - being single item rewordsOverall, test development appears poor, with little regard paid to the psychometrics of test design and the substantive content of some of the scales generated.

The Evolution of a Revolution?The Evolution of a Revolution?

The 16PF5The 16PF5

In 1949 Raymond Cattell published the first edition of 16PF...it was a revolutionary concept; measuring the whole of human personality using structure discovered through factor analysis.

A good thing gets better...New Fifth edition represents a controlled natural evolution of 16PF...continuing to measure the same 16 factors first identified by Dr Cattell over 40 years ago.With the 5th edition, we have made a number of significant enhancements without changing the basic structure of the test.

The 16PF5The 16PF5

“The new edition of 16PF represents an advance on earlier editions, and it is expected that the appeal of this alreadywidely used personality instrument will broaden considerablyin selection and development contexts. The new British Standardisation data also serves to enhance its relevance.”Wendy Lord` p.65, last para, Personnel Management, Feb. 1994.

“The remarkable thing about the 16PF was, and still is, the depth of the analysis it provides...” Wendy Lord` p.65, 2nd. para, Personnel Management, Feb. 1994.

The 16PF5The 16PF5

“The advantage of measuring source traits, as the 16PF does, is that you end up with a richer understanding of the personbecause you are not just describing what can be seen butalso the characteristics underlying what can be seen..” Wendy Lord` p.65, 8th. para, Personnel Management, Feb. 1994.

The 16PF5 The 16PF5 -- Why?Why?

• Simplified language of items - clearer, shorter

• Decrease gender, cultural, racial bias

• Questions more acceptable to candidates

• Reduce testing time

• Increase cohesiveness of each scale

• Easier to score

• Factor measurement fine-tuned

• Reduce scale overlap

The 16PF5The 16PF5

α stability2 A / BA .37 .78 .64C .53 .60 .54E .60 .62 .59F .68 .54 .70G .52 .27 .56H .77 .75 .80I .56 .68 .60L .44 .67 .36M .21 .55 .23N .27 .47 .26O .57 .73 .65

Q 1 .39 .61 .47Q 2 .43 .58 .48Q 3 .48 .46 .58Q 4 .69 .61 .70Av. .5 0 .5 8 .4 8

16PF-A Reliability

The 16PF5 The 16PF5 -- internal consistency internal consistency

“The aim was to move the sixteen scales slightly further apart from each other, to make them more independent of each other. In other words, the aim was to make each scale morecohesive [=within-scale item intercorrelations] within itself..”Wendy Lord, Presentation to the 16PF User’s group, June 1994.

“We’ve raised the Internal Consistency of the factors without sacrificing the diversity of item content which makes 16PF data so rich...” ASE Publicity blurb


α Stability2A .69 .77C .78 .67E .66 .69F .72 .69G .75 .76H .85 .79I .77 .76L .74 .56M .74 .67N .75 .70O .78 .64

Q 1 .64 .70Q 2 .78 .69Q 3 .71 .77Q 4 .76 .68Av. . 7 4 . 7 0

16PF-5 Reliability


R.B. Cattell’s views (J.Beh.Sci., 1972, 1, 169-187):

“The charge that within-scale item intercorrelations(=cohesive ..Wendy Lord) are low within the 16PF (from 0.01 to 0.14 in this paper) are correct, ...but one would have thought that informed psychometrists today would have recognized that this might actually be a virtue”

“The fact is that although the obsession of early psychometristswith internal consistency, under the impression that it was reliability, has long passed out of well-informed discussion, it dies hard as a superstition...”


“The ideal internal consistency is NOT the greatest attainablebut an optimal LOW value based upon various combined

psychometric considerations.”

and finally, in response to others’ criticisms of the extremeitem heterogeneity of the original 16PF (lack of “Cohesiveness” in Wendy Lord’s terminology), Cattellstates:“..(these criticisms)... seem to indicate a need for appraisal of their grasp of psychometry”

The 16PF5 The 16PF5 -- internal consistency internal consistency Example of 16PF-A

16PF A Factor G alpha= .52 Mean ITC = .28

-If Isaw two neighbours children fighting I would reason with them

-When I see 'sloppy' untidy people I get disgusted and annoyed

-I think that freedom is more important than good manners & respect for the law

-People sometimes call me careless even though I'm a likeable person

-I close my mind to well meant suggestions of others, even though I know I shouldn't

-In thinking of difficulties at work I try to plan ahead before meeting them

I find the sight of an untidy room very annoying

(16PF5 Q3 item: I don't usually mind if my room is messy)

-I always make it a point in deciding anything to refer to basic rules of right and wrong

-I am a fairly strict person, insisting on always doing things as correctly as possible


“The diversity of the (16PF5) item content in each scale means that the scales retain breadth. In other words, the increasedprecision has not been at the expense of the depth of theanalysis. The nature of what is being measured stays the same.”Wendy Lord, SDR, December 1994

The 16PF5 The 16PF5 -- internal consistency internal consistency Example of 16PF5

16PF -5 Factor G Alpha: 0.75 Mean ITC = .41

I'd rather see a home that doesn't have too many rules

I value respect for rules and good manner more than easy living.

Most rules are made to be broken when there are good reasons for it.

I get annoyed if people insist that I follow every single minor safety rule

People should insist more than they now do that moral standards should be strictly followed

If a person is clever enough to get around the rules without seeming to break them, he/she should do so only if there is a special reason

I think that being free to do what I want is more important than good manners & respect for rules.

The 16PF5 The 16PF5 -- UK UK anglicizationanglicization

• “All items were written and selected based upon specified criteria, including updating and simplifying language, avoiding content that might lead to gender, race, or disability bias, making material cross-culturally translatable, and avoiding language that is unacceptable in employee selection settings” H.Cattell, Executive Summary, The Revised 16PF - Psychometric Issues Symposium, 1993, Ontario

• “..This had a further effect of making the items more transalantic and less American, reducing the need for major revisions for English language users outside of the US...” UK Technical Supplement.


UK Anglicization??

“ASE prepared the UK standardization version directly from an ASCII file of the US version. A team including chartered

psychologists, trainers in the 16PF & publishing specialists

worked on the anglicization process”

—Movie ---> film

—Math ---> Maths

—Business office ---> office

—program ---> programme


Not Mentioned??

I would rather exercise by:

— 97 US a. fencing or dancing ? c. wrestling or baseball— 42 UK a. fencing or dancing ? c. wrestling or cricket — 39 US I enjoy racy & slapstick humour of some TV shows

— 94 UK I enjoy saucy & slapstick humour of some TV shows


Not Done??

43 UK I prefer the beauty of a poem to an expert football strategy 35 UK I frequently have periods when it’s hard to stop a mood of self-pity


16PF-5 (US) 16PF-5 (UK) 16PF-5 (US) in UKA .69 .69 .62C 78 .73 .77E .66 .68 .71F .72 .74 .72G .75 .70 .71H .85 .87 .87I .77 .76 .76L .74 .60 .73M .74 .71 .71N .75 .72 .78O .78 .77 .75Q1 .64 .65 .69Q2 .78 .75 .79Q3 .71 .74 .77Q4 .76 .73 .78

Average .74 .72 .74

The alpha Reliabilities of the 16PF5 - UK vs US

The 16PF5 The 16PF5 -- EquivalenceEquivalence

• New Fifth edition represents a controlled natural evolution of 16PF...continuing to measure the same 16 factors first identified by Dr Cattell over 40 years ago. US Publicity blurb

• “The nature of what is being measured remains the same.The latest edition of 16PF stays true to the original factor structure” Wendy Lord, SDR, Dec. 1994

• “..85% of the items were drawn from the existing forms ... of the 16PF” UK Technical Supplement

The 16PF5 The 16PF5 -- EquivalenceEquivalence• “The 16PF fifth edition, although updated and

revised,continues to measure the same 16 primary personality factor scales identified by Cattell over 45 years ago” p.3, para 1, US Administration Manual, IPAT

• “Users would be unwise to assume that scores from the 16PF form A and the 16PF5 are interchangeable.” p.13, para 3, UK Standardisation of the 16PF5: A supplement of Norms and Technical Data,ASE Ltd.

• “Clearly some of the scales show a level of correspondence that would suggest that they have changed in their scale characteristics...” Wendy Lord, Presentation to the UK 16PF User Group, June 1994

Scale UncorrectedR

2-weekRetest

Corrected RAlpha

Corrected RA 0.59 0.72 1.17C 0.57 0.75 0.92E 0.55 0.70 0.86F 0.80 0.99 1.13G 0.46 0.55 0.76H 0.85 0.98 1.04I 0.71 0.85 1.09L 0.15 0.20 0.29M 0.21 0.26 0.54N 0.19 0.25 0.43O 0.60 0.80 0.91Q1 0.15 0.19 0.30Q2 0.51 0.58 0.90Q3 0.52 0.60 0.87Q4 0.60 0.72 0.85

A comparison between scores on

the 16PF form A and the 16PF5 test

(N=100 Sixth Form School Pupils). UK Technical

Supplement, ASE

A comparison between scores on

the 16PF form A and the 16PF5 test

(N=100 Sixth Form School Pupils). UK Technical

Supplement, ASE

The 16PF5 The 16PF5 -- Scale EquivalenceScale Equivalence

The 16PF5 The 16PF5 -- Scale EquivalenceScale EquivalencePF5(us) PF5(uk) 15FQ

A .52 .59 .47C .54 .57 .56E .55 .55 .63F .71 .80 .69G .41 .46 .63H .80 .85 .70I .55 .71 .71L .38 .15 .43M .17 .21 .34N .26 .19 .26O .67 .60 .68Q1 .31 .15 .55Q2 .66 .51 .53Q3 .42 .52 .41Q4 .48 .60 .66

Average .49 .50 .55

A comparison between the US administration manual data, UK Technical supplement data, and Psytech15FQ UK scale data with the 16PF-A scores

A comparison between the US administration manual data, UK Technical supplement data, and Psytech15FQ UK scale data with the 16PF-A scores

The 16PF5 The 16PF5 -- Factor EquivalenceFactor Equivalence

The 16PF5 The 16PF5 -- Factor EquivalenceFactor Equivalence

Scale 16PF5 vs 16PFA

Extraversion 0.81

Anxiety 0.79

Independence 0.70

Self Control 0.65

Tough Mindedness 0.38

using 2nd order sten scores, N=462, US Administration manual, p.76

The 16PF5 The 16PF5 -- conclusionsconclusions• There is an unresolved disparity between Cattell’s

position on test construction and that adopted by the developers of the new edition of the 16PF

• 16PF5’s new cultural transferability appears questionable

• The UK anglicization of the test seems to have more commercial than substantive psychological value

• Half the 16PF5 scales do not appear to be equivalent to their earlier counterparts

• The global factor comparability also appear to lack equivalence

the concept 5.2 opq: inefficient measurement or broad

Documents