Management Accounting Research: Experimental

Approaches

Joan Luft

Michigan State UniversityManagement Accounting Section

Mid-Year Meeting

January 2005

2

What does experimental research (not) do well?

1. Approaches that often succeed Examples of influential experimental research in the

social sciences

2. Approaches that often fail Examples of experimental studies in management

accounting that you do not see published

3. Contributions to management accounting What does experimental research do to help us

understand the problems of accounting practice?

3

1. Approaches that succeed: classic examples from experimental economics and psychology

Different traditions of “good experimentation” in economics and psychology, but common characteristics of successful research Generating original and influential ideas from mundane

observations Learning from failure (unsatisfactory experiences):

throwaway explanations versus theory-building explanations Laboratory testing as a process of clarifying & refining theory

in order to make it more usable for a broad array of types of research

4

Example 1: an asset-market experiment

Buyers & sellers (students, classroom setting) seek each other out and negotiate trades.

Sellers have an initial endowment of (artificial) assets with assigned values.

Buyers also have assigned values for assets: surplus from trade arises when (for example) a seller who owns an asset worth $10 to her finds a buyer to whom the asset is worth $15, and they trade at a price of (say) $13.

5

An unsatisfactory experience

Prices did not reach equilibrium (over 46 markets), and considerable potential surplus from trade was not captured.

Researcher tentatively concluded that (a) intersection of supply & demand curves did not predict actual market results, but (b) participants and readers were unconvinced in many cases.

What was the problem with the experiment?

6

Throwaway explanations

“Experiments with student subjects don’t tell us much.”

“There wasn’t enough money at stake.” “The task in the experiment didn’t look

the way it does in the real world.”

What could be wrong with these explanations?

7

What’s wrong with throwaways?

Not true It is not the case in general that student subjects,

modest stakes, or simplified tasks yield results that are uninformative about the “real world.”

Not fruitful As generalizations, these statements do not provide

the basis for generating further interesting ideas. Relying on throwaway explanations, instead of

looking further, can cause researchers to miss opportunities for discovery.

8

Failed asset market: Theory-building explanation

Observations by an irritated student participant in the market: Exchange proceeded through a series of private

bilateral negotiations rather than a central mechanism (e.g., auction)

Bids, asks, and prices were not public. “Bad experiment, unlike real world”? No: some real-world “markets” work like this others do

not. How much (what kind of) difference does this make?

Important new idea at the time!

9

Clarifying & refining theory: the effects of variation in market institutions

Is the potential surplus from trade captured and distributed in the same way in different kinds of competitive markets?

auctions (e-Bay) posted-price markets (department stores) multiple bilateral negotiations (selling major software to

business customers)?

What are effects of market size, duration, detailed rules of exchange, etc.? Auction design (first-price, second-price)? Posted-price design (dept stores vs. airlines)?

10

Contributions of market-design experiments: 1 Lab work allows researchers to:

“Turn one dial at a time,” find out “what happens if I change only this element or that element of the market design?”

Clarify, refine ideas about exactly what matters to market design and why.

Suggests what to model in analytic research and what to look for in archival data.

Research is sometimes limited not by lack of data but by lack of ideas with which to make sense of, or find interesting patterns in, the data.

11

Contributions of market-design experiments: 2 Practice: Dialogue between theory & experiment on

market design has influenced public policy (treasury-bond auctions and bandwidth auctions) and design of private-sector Web-based markets.

Management accounting (potential): Effect of product-market characteristics on product

costing? Internal (designed) markets as substitutes for

institutions like traditional budgeting? (HBR, April 2004)

12

Example 2: unsatisfactory experiences with teaching

Two faculty members commiserate with each other about the difficulty of getting students to grasp basic statistical reasoning & decision models.

13

Throwaway explanations

Students don’t work hard enough. They’re immature; they don’t have

enough experience of the real world. The textbooks are not very good. Etc.

14

Theory-building explanation

It’s not that students are stupid and lazy, but they’re thinking in systematically different ways, which they brought to the classroom with them and will probably take out into life with them as well. But what are these ways?

Can we describe them unambiguously, and generalize usefully about them?

15

Some contributions of experimentation in cognitive psychology

Behavior in non-experimental settings (financial markets, medicine, sports, etc.), consistent with models of heuristic judgment has been found

… once researchers knew what they were looking for and how to make sense of it!

“What to look for and how to make sense of it” was defined by clarifying & refining theory through lab testing (initial descriptions of heuristics sometimes incomplete, unclear, hard to use in research outside the psych lab) (See Gilovich, Griffin & Kahneman, Heuristics & Biases, 2002)

16

Example: Availability heuristic

Does sensational media coverage of a business (accounting) failure unduly influence managers’ judgments of the probability that similar failure threatens their firm (or a supplier or customer)?

Vivid or repeated coverage could make the reported failure more available in memory, and this availability has been shown to influence probability estimates.

17

Refining “availability” in the lab to aid efficient search for effects outside the lab

(1) What is the “availability” that influences probability judgments?

Availability = many rather than few instances come to mind (same number of actual instances have been encountered)

Availability = instances come to mind easily rather than with difficulty (number may be the same)

(2) Individuals & organizations may guard against this availability bias. How? Evidence of availability effects may not be observed bias but costly

steps taken to prevent it. What can reduce the bias (effort, judgment strategies)? What do

people think reduces it?

18

2. Approaches to experimentation in management accounting that often fail Testing (apparently) obvious assertions

… without making it clear why it is interesting or valuable to test them

Testing poorly-specified assertions from practitioner literature … without improving the specification (clarifying &

refining!) to create interesting testable statements

19

Testing the obvious

Examples: is testing these hypotheses a top priority?

If people do not have the information that is clearly necessary to make decision X, then they will not do a very good job of making decision X.

When people choose what information to use, they are more likely to use information type Z if they believe it is useful than if they believe it is not useful (all else equal).

20

Testing poorly specified assertions

Examples: attempts to test the value of ABC, TQM, EVA, BSC, and other TLA’s. Researchers sometimes hope to use lab

settings to eliminate the natural confounds, self-selection, endogeneity problems, and some (not all) proxy measurement problems that plague archival research.

21

Difficulties with this approach

The lab is useful for testing theories. TLA’s are often not theories.

They are loose bundles of economic insights, heuristics for applying these insights, and persuasive rhetoric. If we “test” a TLA in the lab, what exactly

do we test?

22

What do we test …?

Example: need to choose a decision task and provide (create) ABC or BSC information for the experiment

If it is clearly “better information” for the experimental task (e.g., less biased product costs in a decision setting where bias clearly matters), we are likely to be back to the problem of testing the obvious.

23

Is “more reality” the answer? Probably not

Consider a firm that has adopted a TLA and is not sure how well it is working. (Non-obvious.)

Suppose we took information from the firm’s actual TLA system and asked appropriately chosen participants to make decisions with it that are actually made by users of this system in the firm. Randomly assign non-TLA information to other similar

individuals & see how decisions differ.

What do we conclude if TLA-based decision is “better”? Unique instance? What generalizable assertion can we make?

Do we have any idea what properties of the information, task, setting or people produced the result?

24

What properties matter? Example

A well-known BSC property is the four-category system of classifying performance measures. Does this property matter?

Does it make a difference whether (for example) the financial measures are grouped together & labeled “financial”?

Yes (Lipe & Salterio, AOS) What kind of difference? Why?

We need ideas, to help make sense of observed patterns!

25

3. Clarifying & refining ideas about management accounting in the lab

Example: do the conventions of accounting reports (conservatism, aggregation, periodicity, etc.) aid or mislead managers in significant business decisions?

An accounting report (like an economic model or a laboratory task) is a selective abstraction from the “real world.” What are the consequences (benefits & costs) of

the way accountants select and abstract?

26

Concerns from practice

It is sometimes claimed that ABC, BSC, etc., support better decision-making by managers, and that “traditional” (oversimplified, delayed, wrongly aggregated, biased, incomplete) accounting can lead managers to make poor decisions about pricing, product mix, production, customer relations, etc.

Note that this is a claim about decision-facilitating not decision-influencing (contracting) uses of accounting information, & as such the claim has been challenged.

27

Can these managers be misled?

Consider a manager who is involved in particular business processes every day, and who has detailed data (often non-accounting) on relevant activities and expenditures.

Will the incompleteness or bias in a summary accounting report cause her to misunderstand what drives success in her business? Doesn’t she know her business, apart from the accounting reports?

Perhaps the only real importance of the formal management-accounting report is the fact that it is contractible information that can be used in the reward system.

28

How could accounting sometimes mislead well-informed, well-motivated decision-makers?

Example: we sometimes observe: Decision-makers fail to use information they

certainly do know and consciously intend to use. Their decisions are influenced by factors that they

are not aware of or would not choose to use in decisions.

Can accounting influence these unconscious elements of decision-making? What does psychology theory tell us about this?

29

Example: Two-system (dual-process) theories of reasoning System 2: Deliberate, effortful, relatively slow,

consciously controlled processes, often dependent on abstract rules (e.g., adding columns of numbers)

System 1: Automatic, rapid, uncontrolled processes, often context-dependent (e.g., “gut-feel” judgments, recognizing familiar faces) Efficiency/accuracy/adaptability tradeoffs Limited ability to choose between systems (e.g., can decide

to review gut-feel judgments more carefully; can’t decide not to recognize a familiar face)

(Sloman, Psych. Bull. 1996; Stanovich & West in Gilovich et al. Heuristics & Biases)

30

Fundamental properties of accounting reports

Distinctive structure (e.g., income statement, balance sheet)

Regular periodicity (monthly, quarterly …) Classification & and labeling (assets, expenses,

profits …)

How might these properties influence unconscious (System 1) processing, & sometimes mislead well-informed managers (unless costly steps are taken to counteract the effects)?

31

Example 1: accounting report structure influences problem structuring

Experiment: Participants understand the concept of opportunity costs and apply it correctly in a personal-finance decision.

But the more financial-accounting training they have, the more they ignore opportunity costs in a very similar business decision – presumably because they (automatically) structure the business decision by thinking in income-statement (not opportunity-cost) terms. The accounting model has become their economic model (Vera-Munoz TAR 1998).

Doesn’t show that “accountants in the real world always ignore opportunity costs.”

Does show that accounting-report structure can influence people’s problem-structuring in ways they are not fully aware of. This is a pattern we could look for in a variety of settings.

32

Example 1, cont. What might prevent the omission of

opportunity costs in decision-making in firms? Firm procedures? (For important decisions, firms

have rules that force people away from this income-statement structuring.)

Work experience? (We train people in accounting but they get over it.)

Vera-Munoz, Kinney & Bonner (TAR 2001): it’s more complicated than that … (more theory development probably needed)

33

Example 2: Periodic repetition in accounting reports makes items salient & thus influential

Individuals make investment decisions, are evaluated, and later evaluate others who make similar decisions. Expected return on investments is known; actual return provides no additional information about quality of investment decisions.

Individuals begin with a belief in the principle that investment decisions should be evaluated based on expected (not actual) return.

Individuals’ belief in and use of this principle is undermined by making decisions under a system where they themselves are evaluated based on actual return—which is not advantageous for them!

Bigger effect when their own work was evaluated after each of 12 decisions rather than once, cumulatively, after all 12. (No difference in payoff, just in timing/ repetition of feedback.)

Frederickson et al. JAR 1999

34

Conclusion

The lab is a place to clarify initial (perhaps incomplete or imperfectly specified) versions of theories, to unpack their implications, to grow and prune and refine them.

The “real world” is not a good place for doing this particular kind of testing, but our understanding of what we see in the real world is enhanced by what we learn in the lab.