Measure for Measure(ment)

James ShanteauJames Shanteau

University Distinguished Professor of University Distinguished Professor of Psychology Emeritus Psychology Emeritus

Kansas State UniversityKansas State University

Bayes Conference – 2/14

Goals and Caveats

Three goals:(1) Summarize why measurement is essential(2) Review some prior studies using measurement(3) Consider what more needs to be done

Goals and Caveats

Three goals:(1) Summarize why measurement is essential(2) Review some prior studies using measurement(3) Consider what more needs to be doneCaveats:(1) Concern with psychological measurement of

stim-ulus values, sometimes called psychological scaling

(2) Focus on graphical analyses, which are based on underlying numerical estimates of psychological measures

Quotes About Measurement“Thou shall not have in thine house diverse measures.

But a perfect & just measure shalt thou have” (Deuteronomy)

“If you cannot measure it then it is not science” (Lord Kelvin)

“The first steps in the path of discovery … the first approximate measures, are those which add most to the existing knowledge of mankind” (Charles Babbage)

“To understand God’s thoughts one must study statistics … the measure of his purpose” (Florence Nightingale)

“Whenever you can, count” (Sir Francis Galton)

“Research is 4 things: brains with which to think, eyes with which to see, machines with which to measure, & 4th money” m (Albert Szent-Gyorgyi)

Quote by Psychologists on Measurement

“In order to formulate quantitative laws (in psychology),

the relevant properties must be expressible by numbers.

The process by which scientists (do this) is called

measure-ment.” (Coombs, Dawes, & Tversky, 1970)

Where Is Discussion of Measurement? To find additional comments on Measurement in

Psychology, I looked at recent books with “Measurement” in title, eg:

Edwards, W., (Ed.) (1992). “Utility theories: Measurements and applications.” Boston: Kluwer Academic Press.

Weiss, D. J. (2006) “Analysis of variance and functional measurement.” Oxford: Oxford University Press.

Lockhart, P. (2012). “Measurement.” Cambridge, MA: Belknap Press of Harvard University Press.

Only 1 of 3 included any content on psych measurementGuess which one?

Where Is Discussion of Measurement? To find additional comments on Measurement in

Psychology, I looked at recent books with “Measurement” in title, eg:

Edwards, W., (Ed.) (1992). “Utility theories: Measurements and applications.” Boston: Kluwer Academic Press.

Weiss, D. J. (2006) “Analysis of variance and functional measurement.” Oxford: Oxford University Press.

Lockhart, P. (2012). “Measurement.” Cambridge, MA: Belknap Press of Harvard University Press.

Only 1 of 3 included any content on psych measurementGuess which one?Only Weiss book includes discussion of actual

measurementsThe Dilemma: Why is “Measurement” widely talked

about, but so rarely done?

Some Preliminary Comments

(1)(1) Measures = parameters of models, stated or Measures = parameters of models, stated or

unstatedunstated

=> Validity of measures depends on validity of models=> Validity of measures depends on validity of models

(2) Estimating Scaling values ≠ Weighting values(2) Estimating Scaling values ≠ Weighting values

=> Different estimation methods required=> Different estimation methods required

(3) Typically try to get interval-scale estimates(3) Typically try to get interval-scale estimates

=> Need to establish 2 points, eg, slope & intercept=> Need to establish 2 points, eg, slope & intercept

(4) Computer programs simplify getting actual (4) Computer programs simplify getting actual

measuresmeasures

=> => “Its too hard to do measurement” “Its too hard to do measurement” is no longer an is no longer an

excuseexcuse

Example of Weight MeasurementMy 1My 1stst study as grad student involved estimation study as grad student involved estimation of serial position weights in dynamic decision of serial position weights in dynamic decision making tasks, eg, probability revisionmaking tasks, eg, probability revision

These curves show weights of serial positions for These curves show weights of serial positions for different lengths of stimuli, Resp 4 to Resp 15different lengths of stimuli, Resp 4 to Resp 15

See generalized recencySee generalized recency

Note: weight estimatesNote: weight estimates

are not normalizedare not normalized

Example of Weight Measurement

How to Compare Weights?

When Are Weight Measures Important?

In study of expertise by Ettenson, Krogstad, & In study of expertise by Ettenson, Krogstad, & Shanteau, 3 groups of auditors judged common Shanteau, 3 groups of auditors judged common set of cases described by 8 cuesset of cases described by 8 cues

Expert (Partners) & intermediate (Seniors) Expert (Partners) & intermediate (Seniors) auditors relied primarily on 1 cue, whereas auditors relied primarily on 1 cue, whereas graduate (Stu-dents) made use of many more graduate (Stu-dents) made use of many more cuescues

Has implications for teachingHas implications for teaching

students to focus on whatstudents to focus on what

is vital in a given contextis vital in a given context

Example of Scale Value Measurement

Why Care About Measurement of Scale Values?

In an analysis of willingness to donate various In an analysis of willingness to donate various organs while living, Skowronski & Shanteau had organs while living, Skowronski & Shanteau had 4 groups evaluate various donations while living4 groups evaluate various donations while living

Found all groups (even Anti-Donors) were willing Found all groups (even Anti-Donors) were willing to donate to relativesto donate to relatives

This finding has important implications for This finding has important implications for increasing rates of organ donationincreasing rates of organ donation

Everyone willing to donateEveryone willing to donate… … under some situationsunder some situations

No one is No one is Anti-DonationAnti-Donation

But most are But most are Anti-Stranger Anti-Stranger

Example of Joint Weight & Scale Value Measurements

My contribution to measurements involved using My contribution to measurements involved using bilinear models to estimate both weight & scale bilinear models to estimate both weight & scale values; bilinearity => diverging fan of straight values; bilinearity => diverging fan of straight lineslines

On left, objective values => irregular pattern On left, objective values => irregular pattern unclearunclear

On right, subjective values => bilinear pattern On right, subjective values => bilinear pattern evidentevident

Shows linkage between model Shows linkage between model testing & measurementtesting & measurement

See support for SEU model:See support for SEU model: SEU = Subj Prob x UtilitySEU = Subj Prob x Utility

What About Non-Numeric, Verbal Stimuli?

Traditionally measurement approaches Traditionally measurement approaches transform numeric stimuli into psych values, eg, transform numeric stimuli into psych values, eg, psychophysicspsychophysics

But measurement not constrained to numeric But measurement not constrained to numeric stimulistimuli

Challenge is getting interval scale valuesChallenge is getting interval scale values

This can be solved by using This can be solved by using

anchor stimuli, eg,anchor stimuli, eg,

““No Chance” No Chance” & & “Sure Thing”“Sure Thing”

““$.50” $.50” & & “$60”“$60”

Other Examples of Measurement

More Quotes About Measurement

““If measurement matters at all, it is because it If measurement matters at all, it is because it

must have some conceivable effect on must have some conceivable effect on

decisions and behaviour” decisions and behaviour” (Douglas W. Hubbard)(Douglas W. Hubbard)

“ ’“ ’By measurement to knowledge’ I should write By measurement to knowledge’ I should write

as a motto above the entrance to every labora-as a motto above the entrance to every labora-

tory” tory” (Heike Kamerlingh Onnes)(Heike Kamerlingh Onnes)

““An experiment is a question which science An experiment is a question which science

poses to nature and a measurement is the poses to nature and a measurement is the

recording of nature’s answer” recording of nature’s answer” (Max Planck)(Max Planck)

What’s Holding Back Measurement?Given benefits of measurement, why isn’t

everyone estimating subjective values?Three possible reasons:(1) Confusions over producing interval scales

=> Need more examples to illustrate “how to do”

(2) Seemingly arbitrary assumption required=> That is why measurement & model testing linked

(3) “Forrest Young Problem” => He said MDS too sophisticated for most users=> Need to make measurement easy

Some Caveats About Measurement

““Those who think ‘Science is Measurement’ Those who think ‘Science is Measurement’

should search Darwin’s works for numbers and should search Darwin’s works for numbers and

equations” equations” (David Hubbel)(David Hubbel)

““Our clocks do not measure time … time is Our clocks do not measure time … time is

defined to be what our clocks measure” defined to be what our clocks measure”

(Anonymous, NIST)(Anonymous, NIST)

““Not everything that counts can be counted and Not everything that counts can be counted and

not everything that can be counted counts” not everything that can be counted counts”

(Albert Einstein) (Albert Einstein)

Comments and Questions? Your turn …

Norman Anderson

Measure for Measure(ment)

James ShanteauJames Shanteau

University Distinguished Professor of University Distinguished Professor of Psychology Emeritus Psychology Emeritus

Kansas State UniversityKansas State University

Bayes Conference – 2/14

Issue 3: What About Individual Differences?

Group analyses often conceal sizeable individual Group analyses often conceal sizeable individual differencesdifferences

One advantage of FM is ability to analyze both One advantage of FM is ability to analyze both levelslevels

See differences in both model shape & valuesSee differences in both model shape & values

Norman Anderson on Measurement “The logic of (FM) consists in using the postulated

behavior-al laws to induce a scaling on the dependent variable” (1962)

“A guiding principle of FM is that measurement scales are derivative from substantive theory” (1970)

“Measurement is fundamental” (1996)“FM reverses the traditional approach & makes

measurement an organic component of substantive investigation” (1981)

“All that is necessary is to have a valid integration function. That function provides base & frame for measurement” (1981)

“Measurement… is the link between the world of behavior & the world of science … (it) is a vital feature of science … meas-urement is thus an integral part of substantive theory” (2001)

“Much of progress consists in improvements in measurement, both in its empirical base & its conceptual base” (2001)

Norman Anderson (NHA) Story 50 years ago, NHA was debating with advocates of

Conjoint Measurement (CM). One of his major arguments was that CM wasn’t being used to actually measure anything, but FM was.

However, there were almost no examples of actual measures by FM at that time.

NHA quickly put together a paper with actual measures of adjectives => it was published in Psych Reports (?) => I have been unable to find either the paper or the citation

However, my memory of this story was the motivation for the present talk, ie, we talk-the-talk about measurement, but how often do we walk-the-walk?

If measurement is such a vital part of FM, why don’t we see measured values reported in most every study

But, such reports of numerical measures is rare. Why?

Purpose & OrganizationPurposePurpose:: To revisit role that measurement plays To revisit role that measurement plays

in Functional Measurement (FM)in Functional Measurement (FM)

Organization of TalkOrganization of Talk::

(1)(1) Describe role of measurement in FM logicDescribe role of measurement in FM logic

(2)(2) Present examples of measurement in FM Present examples of measurement in FM studies studies

(3)(3) Consider some possible barriers to use of FMConsider some possible barriers to use of FM

(4)(4) Take comments & answer questions (if I can)Take comments & answer questions (if I can)

Example of Joint Weight & Scale Value Measurements

Can see application of bilinear analysis in Can see application of bilinear analysis in reanalysis of Tversky (left panel)reanalysis of Tversky (left panel)

See similar patterns in Slovic & Lichenstein dataSee similar patterns in Slovic & Lichenstein dataNote: both original analyses involved additivity Note: both original analyses involved additivity (par-allelism) assumptions based on objective (par-allelism) assumptions based on objective valuesvalues

Issue 3: What About Individual Differences?

Group analyses often conceal sizeable individual Group analyses often conceal sizeable individual differencesdifferences

One advantage of FM is ability to analyze both One advantage of FM is ability to analyze both levelslevels

Example of FM Using RTsNearly all studies using FM have used rating dataNearly all studies using FM have used rating dataOne exception: Weiss’ work using ordinal dataOne exception: Weiss’ work using ordinal dataAnother exception: FM analysis of RTs in Another exception: FM analysis of RTs in question-answering task introduced by question-answering task introduced by AndersonAnderson

Results provided new insights into priming Results provided new insights into priming effectseffects

Key is bilinear analysis of cumulative Response Key is bilinear analysis of cumulative Response TimeTime

Allows bilinear scalingAllows bilinear scaling∴ ∴ Can “measure” timesCan “measure” times=> just like rating data=> just like rating data

Priming found in both mem-Priming found in both mem-ory & decision timesory & decision times

Decision priming over 2xDecision priming over 2xeffect of memory priming effect of memory priming

More Quotes on Measurement “Measurement is the 1st step to control & eventually

to improvement. If you can’t measure something, you can’t understand it” (H. James Herrington)

If measurement matters at all, it is because it must have some conceivable effect on decisions & behaviour” (Douglas W. Hubbard)

“’By measurement to knowledge’ I should write as a motto above entrance to every laboratory” (Heike Kamerlingh Onnes)

“An experiment is a question which science poses to nature, & a measurement is the recording of nature’s answer” (Max Planck)

“If you cannot measure it you cannot control it” (J Grebe)

“Man is the measure of all things” (Protagoras)

Quote by Psychologists on Measurement “Although measurement & scaling have sometimes

been used interchangeably in the literature, we have

chosen to distin-guish between them. Measurement is

concerned with the con-ditions under which various types

of scales can be construct-ed. The actual process of

assigning numbers … is called scal-ing” (Coombs, 1964)

“In order to formulate quantitative laws (in psychology),

the relevant properties must be expressible by numbers.

The process by which scientists (do this) is called

measure-ment.” (Coombs, Dawes, & Tversky, 1970)

Some Preliminary Comments

(1)(1) Measures = parameters of models, stated or Measures = parameters of models, stated or

unstatedunstated

=> Validity of measures depends on validity of models=> Validity of measures depends on validity of models

=> There is direct link between model fit & => There is direct link between model fit &

measurementmeasurement

(2) Estimating Scaling values ≠ Weighting values(2) Estimating Scaling values ≠ Weighting values

=> Different estimation methods required=> Different estimation methods required

=> Lead to different interpretation and usage=> Lead to different interpretation and usage

(3) Typically try to get interval-scale estimates(3) Typically try to get interval-scale estimates

=> Need to establish 2 points, eg, slope & intercept=> Need to establish 2 points, eg, slope & intercept

(4) Computer programs simplify getting actual (4) Computer programs simplify getting actual

measuresmeasures

=> “Its hard to do measurement” is no longer an => “Its hard to do measurement” is no longer an

excuseexcuse

Some Caveats on Measurement“Not everything that is observable & measurable is

predictable, no matter how complete our past observations may have been” (Sir William McCrea)

“Those who think ‘Science is Measurement’ should search Darwin’s works for numbers & equations” (David Hubel)

“Investigators seem to have settled for what is measur-able instead of measuring what they would really like to know” (E D Pellegrino) => Note: Drunk & Lamppost problem

“Our clocks do not measure time … Time is defined to be what our clocks measure” (Anonymous, NIST)

“Not everything that counts can be counted & not every-thing that can be counted counts” (Albert Einstein)

Are Measured Values Constant Across Tasks?

Within many tasks, eg, impression formation, Within many tasks, eg, impression formation, mea-sured values appear to be more-or-less mea-sured values appear to be more-or-less constantconstant

But across tasks, see sizable differences, eg, in But across tasks, see sizable differences, eg, in sub-jective probability values for gambles vs sub-jective probability values for gambles vs datingdating