Survey Measurement of Preference Parameters

Presentation by Miles Kimball

Osaka University, 2004

Outline

• What We Want to Know

• A New Approach: Survey Measurement of Preference Parameters

• Challenges in the Survey Measurement of Preference Parameters

• Where to Go from Here

What We Want to Know: A Partial Wish-List We All Have in Common

• Time Preference• Elasticity of

Intertemporal Substitution

• Altruism/Envy• Labor Supply

Elasticities • Risk Aversion• Prudence• Temperance

• Flow Utility • Overall Present

Discounted Value of Utility

• Marginal Propensity to Consume

• Shadow Interest Rate• Shadow Wage• Risk Aversion of

Value Function

The Need for a New Approach

• Despite decades of research using data on actual economic choices, there is still no consensus on the values of many basic preference parameters.

• In the absence of persuasive identification of the values of preference parameters, convenient calibrations continue to reassert themselves. – log utility– additive separability – zero or infinite labor supply elasticity

Advantages of Hypothetical Experiments

• Exogenous

• Large enough to overcome frictions

• Simplified situations highlight different aspects of preferences

• Close connection to economic theory

• Close connection to economic intuition

The Disadvantage of Hypothetical Experiments

• Many economists have doubts about whether people can really answer these questions meaningfully.

• These doubts deserve to be taken seriously, without letting them paralyze the research program.

• This requires dealing seriously with the cognitive limitation of respondents and measurement error.

Challenges in the Survey Measurement of Preference

Parameters

• Cognitive limitations of respondents

• Measurement error

• Linking to economic theory

• Survey Mechanics

Evidence that Survey Respondents

Have Important Cognitive Limitations

• Question order and the precise wording of questions often matter.– First response on paper and pencil questions, last

response in phone interviews– Yes-man effect– Status quo bias– Anchoring (effects of the response categories)– Overweighting of recent events– Guessing what the interviewer would think was

reasonable

Strategies for Dealing with Respondents’ Cognitive Limitations• Get many people’s reactions to a question, particularly

the reactions of professional survey methodologists.• Make a vignette (story) for the hypothetical situation that

is as concrete and believable as possible given the objective.

• Break the question down into pieces that are each individually easy to understand—even at the cost of complicating the later analysis of the question.

• When the question still seems too hard, go back to the drawing board and design a new question to get at the desired concept.

A Slogan

• To maximize the reliability of survey answers to difficult questions, substitute researcher effort for respondent effort wherever possible.

How to Gauge How Seriously Cognitive Limitations Might Be

Affecting Responses

• Pretest.• Look at nonresponse rates.• Debrief respondents. (“Why did you answer the

way you did on that question?”)• Try different variants of the same basic question.

– Vary the wording– Vary the order– Vary cutpoints and the unfolding sequence

Measurement Error

• Even after doing everything possible to make the questions as easy as possible for respondents to understand and answer, there will still be measurement error.

• Fortunately, econometric techniques can deal with this measurement error.

• The appropriate econometric techniques are different from those we are used to because survey measurement of preference parameters often yields more information about the characteristics of the measurement error than other econometric applications.

Linking to Economic Theory

• Economic theory gives precise cardinal definitions of elasticities and other parameters that make questions about preference parameters quantitative.

• This distinguishes economics from psychology, which typically has only ordinal concepts.

• Enjoying this strength of economic theory requires designing hypothetical situations to match the economic theory as closely as possible.

Survey Mechanics

• Getting a representative sample.• Getting a high response rate in order to avoid

serious biases.• Taking into account the advantages and

disadvantages of different modes.– In person (highest quality, highest cost) – Mail (low response rates a problem, at least in the

U.S., and no interviewer feedback on how respondents are handling questions)

– Phone (often the interior optimum)– Internet (emerging method with great potential)

Case Study #1: Risk Aversion

• Question Design: – Risk over permanent income to get at the

underlying utility function.– Discrete choices to reduce the cognitive

burden.– Rewritten to avoid status quo bias.– Vignette to motivate the hypothetical situation.

Case Study #1: Risk Aversion(continued)

• Implementation– Asked in multiple waves to get a measure of

measurement error. Imputation of a cardinal value to each category taking this into account.

– Econometric adjustment to OLS that takes into account the measurement error and its.

– Starting point and order of unfolding varied.– Small risk questions. – Variant questions for retired respondents.

Case Study #2: Labor Supply

• Question Design: – We use the theoretical relationship relationship

between income and substitution effects.– Income effects are the easiest for people to think

about.– Winning the lottery is an obvious vignette, since

people really think about that possibility.– The lottery is large and an easily understood size.– The decision is broken down into quit or not, then if

not quit, reduce hours or not and by how much. – The question sequence is designed to match the

family structure.

Case Study #2: Labor Supply(continued)

• Implementation– Asked in multiple waves of the HRS to allow

us to get at measurement error. (This aspect is still unanalyzed.)

– We make an analytical and econometric adjustment for the special status of quitting.

– We have put some variant questions on the Survey of Consumers, but due to lack of funding for survey time, only a fraction of everything that should be looked at.

Case Study #3: Time Preference and the EIS

• Question Design: – Choice over consumption profiles to get at

preferences. (In contrast to choices over money that get at the individual’s shadow interest rate.)

– Limitation to two time periods and discrete choices to reduce the cognitive burden. Elicitation of second choices to gain more total information.

– Bar graphs to make the numbers more vivid. – The preamble is a compromise, trying to control for

inflation and health care costs but not try the respondent’s patience.

Case Study #3: Time Preference and the EIS

(continued)

• Initial Implementation– Initial implementation on the Wave I of the

HRS had an inconsistency between the consumption growth rates and the ABCDE labels. Also there was not enough resolution. In person interviews allowed use of graphs.

Case Study #3: Time Preference and the EIS

(continued)• Revised Implementation

– The HRS has a mailout that has a reasonable response rate because of the preexisting relationship. We implemented a revised version of the question with more resolution and a systematic relationship between ABCDE labels and consumption growth rates.

– In the ongoing analysis, we have statistically modeled status quo bias (in this case the tendency to continue answering the same letter, regardless of true underlying preferences) and measurement error. The procedure for dealing with the measurement error also easily handles the discrete choice aspect.

Case Study #3: Time Preference and the EIS

(continued)

• Internet redesign and implementation – Bob Willis has headed a project

experimenting with internet interviewing. – This will allow us to test variants of the time

preference/intertemporal substitution question, including a variant with continuous choice over consumption bars using a mouse.

– This is still in the design stage. Technical hurdles remain.

Where to Go From Here

• Issues in the Choice of a Strategy

• Overall Strategy Recommendation

• High Priority Topic Areas

Issues in the Choice of a Strategy

• Key Question #1: How much time and effort should be spent nailing down each preference parameter? (The tradeoff between thoroughness and getting many things done all at once.)

• Key Question #2: What is the best mode?• Key Question #3: What is the best way to

tap into existing wisdom and resources for survey measurement?

Overall Strategy Recommendation #1: Put Thoroughness First

• Our discipline of Economics is skeptical enough of survey measures of preference parameters that it is important to be thorough and do the best job possible in measuring each preference parameter to have an impact. Getting many things done all at once is of no avail if the answers are not seen as credible.

• Even a booster of survey measures of preference parameters should only be fully persuaded of the results after doing the most careful design, implementation and analysis possible.

• In the future these measures might guide the policy for nations.

• Since surveys in many nations can use the measures, it is worth paying a high fixed cost at the design stage.

Overall Strategy Recommendation #2: Phone Survey Initially +

Eventually an Internet Survey.

• Getting a representative sample is quite valuable given the interest in the population average values of preference parameters. This can be achieved well be either phone or in-person surveys, but phone surveys are cheaper and more practical.

• Internet surveys can do everything mail surveys can do and more and are more flexible for experimental work. This is an emerging mode. Efforts are being made to figure out how to get more representative samples in this mode.

Overall Strategy Recommendation #3: Piggyback on Existing Surveys

• The Survey of Consumers provides important services– The Survey of Consumers handles the practical side

of pretesting, getting a representative sample and doing the interviewing.

– It provides expertise in question design based on long experience.

– Its staff is easy to work with. They are used to riders.

Overall Strategy Recommendation #3: Piggyback on Existing Surveys

(continued)

• The Survey of Consumers has several structural advantages – It provides basic demographic data for free, as well as

some expectational data. – It has a panel structure. – It is in the field every month, so it is possible to have a

useful feedback loop.– It is part of an academic center of survey research,

with access to all the intellectual resources of the Survey Research Center.

Overall Strategy Recommendation #3: Piggyback on Existing Surveys

(continued)

• The Survey of Consumers is a good launching pad for other efforts.– An existing question could generate a representative

sample of internet users under 50. – Questions that are successful on the Survey of

Consumers and generate interesting analyses are natural candidates for modules or even core inclusion on the HRS and PSID.

– Once questions are fully developed, the machinery of the Survey of Consumers can be used to run an entirely new phone surveys if desired.

High Priority Topic Areas (In Order of Priority, as I See It)

• Happiness

• Aspects of Declining Marginal Utility

• Labor Supply Elasticities

• The Marginal Propensity to Consume

• Aspects of the Value Function

Happiness

• Recent research suggests that studying the time-series properties of happiness is crucial

• Because it is in the field continuously, and has information on the exact day people answer questions adding a few questions to measure happiness on the Monthly Survey of Consumers would be of great value in seeing the effects of news events on happiness.

• Most other questions can be asked once or twice---this really should be on every month, for as far in the future as can be arranged.

• The needed measures exist.

Aspects of Declining Marginal Utility

• In the usual theory, all of the following are said to reflect primarily the rate at which the marginal utility of consumption declines– risk aversion– resistance to intertemporal substitution– the income elasticity of the statistical value of a life– the long-run wage elasticity of labor supply– the elasticity of altruism with respect to subsidies or to

leaky buckets

Aspects of Declining Marginal Utility (continued)

• Each of these suggests a different survey measure.

• If the usual theory is correct, data using all these different survey measures would allow one to pin down the rate at which the marginal utility of consumption declines much more accurately.

• If the usual theory is incorrect, these separate measures will indicate not only the failure, but the nature and direction of the failure.

• The state of the art is well advanced for these kind of survey measures. This set of measurements could be done right away.

Beyond Declining Marginal Utility

• Some of the techniques for measuring the rate at which marginal utility declines (u’’) can be adapted to measure higher derivatives of the utility function in a direct way.

• The value of such measurement is great because it has been even harder to identify these higher order parameters by existing data.

Labor Supply Elasticities

• The income elasticity of labor supply, the Frisch elasticity of labor supply and the long-run elasticity of labor supply are all of great practical importance.

• Separate measures of these different concepts are possible, allowing a test of the theoretical relationship between them.

• Also of interest: – the relationship between consumption and labor in the utility

function – the effect of job characteristics on labor supply– endogenous age of retirement

• The state of the art is also getting to be well-advanced here, though these are unavoidably more difficult than the typical question in the previous topic area.

The Marginal Propensity to Consume

• The MPC turns out to be quite difficult to measure. Our initial efforts had serious problems. – The reported MPC’s for permanent and transitory

shocks were quite close to each other.– When asked to distribute permanent income between

consumption, saving for early retirement, ordinary saving, giving to relatives and giving to others, the average MPC was about 24%.

– We interpret this as the 1/n phenomenon.

The Marginal Propensity to Consume (continued)

• There is a variety of evidence in the literature for mental accounting.

• Shapiro and Slemrod have used the Survey of Consumers to study the impact of tax rebates, tax cuts and withholding changes. The results are intriguing but raise as many questions as they answer.

• Basic research is needed before we can accurately measure the MPC.

• The intellectual resources exist to pursue that basic research given funding for the needed survey data.

Aspects of the Value Function

• In measuring preference parameters, the idea is to construct hypothetical situations that abstract from aspects of an individual’s situation other than preferences.

• It is also interesting to obtain measures that mix together aspects of preferences and aspects of the situation---especially when the relevant aspects of preferences are separately measured for comparison.

• When we know how to measure these well, time series could be especially interesting.

Aspects of the Value Function(continued)

• Examples:– The absolute risk aversion of the value

function (risk version over dollar or yen gambles)

– The marginal propensity to consume – The shadow interest rate– The shadow wage.

Nonstandard Arguments of the Value Function

• This is quite speculative, but one could imagine trying to measure nonstandard arguments of the value function. For example:– intensity of work effort– habits– cumulated fatigue

Summary

• Survey measurement of preference parameters has great promise.

• More generally, many things can be measured that have never been measured before.

• If we build a strong foundation for the future, survey methods can transform the conduct of economics in the 21st century.

• There are fascinating places to start.