Measurement

17.871Spring 2004

Topics in Measurement

• From abstraction to measure• Sources of error• What to do about error• Practical ways to improve measurement

The Mapping

Theory YX

yxObservation

eyex

Mapping from the Abstract to the Measurement

• Some abstract things we try to measure– Alienation– Moral decay– Democracy– Party identification– Fear of defeat– Terrorism– Ideology

Sources of Error in Measurement(Mosteller)

• Conceptual or design error• Bad breaks in random sampling• Survey question wording• Non-random out-selection• Transcription errors• Calculation & mechanization errors

What to Do About Error

• Practice safe data– Know where your data come from– Watch for anomalies– Use multiple measurement techniques– Collect as much data as possible and

disaggregate

Practical things to do about measurement

• Distinction between a measure and an indicator– Measure: straightforward quantification of a

variable of interest– Indicator: quantification of a variable that is

believed (or known) to be highly correlated with the “real” variable of interest

Examples of Measures

• Income in $$• Age in years• Votes• Number of wars (hmmmmmm…..)• Campaign contributions

• Measurement issues tend to focus on the quality of the data-gathering method, especially sampling

Examples of Indicators• Public opinion

– Party identification– Trust in government– Ideology

• Economics– Gross domestic (national) product

• Characterizations of political systems– Freedom– Transparency– Democracy

How do we generate indicators?• Trust others and hope for the best

– GDP, etc.– Presidential approval

• Use a single measure and hope for the best (or convince ourselves it’s OK)– 7-point ideology scale– 7-point party identification scale– Codings of “wars” or “rally events”– Bulletin of the Atomic Scientist “nuclear clock”

• Use multiple measures creatively

Multiple-measure indicators• “Trust in government” battery

– How much of the time do you think you can trust the government in Washington to do what is right?

– Would you say the government is pretty much run by a few big interests looking out for themselves or that it is run for the benefit of all the people?

– Do you think that people in the government waste a lot of money we pay in taxes, waste some of it, or don’t waste very much of it?

– Do you think that quite a few of the people running the government crooked, not very many are, or do you think hardly any of them are crooked?

• External political efficacy battery– Sometimes politics and government seem so complicated that a

person like me can’t really understand what’s going on– People like me don’t have any say about what the government

does– Public officials don’t care much what people like me think.

Trust and Efficacy over Time

Other Multiple-Variable Indicators

• Americans for Democratic Action Support Scores

• Freedom House freedom assessment• MIT Teaching Quality• Transparency International’s “Corruption

Perceptions Index”• DNominate Scores

ScalingOr, How to Create Multiple Indicators More Generally

Imagine an unobserved factor that gives rise to observable factors

• Democracy• Healthiness• Racism• Conservatism• Religiosity• Teaching quality

Measure 1 e1b1

b2

Measure 2 e2Unobservedfactor . . .

Measure nbn e3

Fairness ofelections

e1b1

b2

Meaningfulnessof elections

e2

Polyarchyb3

Freedom of expression e3

b4 Unbiased reporting ofofficial pronouncements

e4

SEG (Fall ’02)well-preparedpresentations

1

7

1

7

1

7

1 71

7

1 7

explains clearly

good visual aids

1 7

1 7

good info tech

speaks well

2 7

1 7

encouragesquestions

stimulatesinterest

1 7

1

7

1 7

1

7

2

7

1

7

availableoutside class

Two-factor (pure) polyarchyexample

Free

dom

of e

xpre

ssio

n

Fairn

ess

of e

lect

ions

A B DC FE G H I J

Polyarchy Polyarchy

What We Observe in the Pure Case

Fairn

ess

of e

lect

ions

Freedom of expression

Two-factor (impure) polyarchyexample

Free

dom

of e

xpre

ssio

n

Fairn

ess

of e

lect

ions

A B DC FE G H I J

Polyarchy Polyarchy

What We Observe in the Impure Case

Fairn

ess

of e

lect

ions

Polyarchy

faire

lt

frexp.5 3.5

.5

3.5

USSR,Libya, Iran,China

Congo, LebanonHungary

Bhutan

Egypt, Sri Lanka,Mexico

Jamaica,Greece

Peru, India,IsraelArgentina,

U.S., Finland

Source: http://www.nd.edu/~mcoppedg/crd/polydat.htm_______________________________________________________________

Simple way to create multiple-indicator scale with just 1 factor

SX X

sii

Ni t t

tt

T

=−

= =∑ ∑

1 1

( ),