interpreting and communicating checkoff program evaluation results dr. gary w. williams

INTERPRETING AND COMMUNICATING CHECKOFF INTERPRETING AND COMMUNICATING CHECKOFF PROGRAM EVALUATION RESULTSPROGRAM EVALUATION RESULTS

Dr. Gary W. WilliamsDr. Gary W. WilliamsProfessor, Agricultural EconomicsProfessor, Agricultural Economics

Director, Texas Agribusiness MarketDirector, Texas Agribusiness MarketResearch Center (TAMRC)Research Center (TAMRC)

Dr. John P. NicholsDr. John P. NicholsProfessor, Agricultural EconomicsProfessor, Agricultural Economics

Interim Head, Department of Agricultural EconomicsInterim Head, Department of Agricultural Economics

Texas A&M UniversityTexas A&M UniversityCollege Station, TexasCollege Station, Texas

““Quantitative Evaluation of Advertising and Promotion Programs”Quantitative Evaluation of Advertising and Promotion Programs”Commodity Promotion Evaluation WorkshopCommodity Promotion Evaluation Workshop

for Researchers and Managers for Researchers and Managers

2006 Spring NEC-63 Meeting2006 Spring NEC-63 MeetingMonterey, CaliforniaMonterey, California

March 9, 2006March 9, 2006

The ResearchThe ResearchIsn’t Done …Isn’t Done …

… … Until the Results Are Until the Results Are Communicated!Communicated!

Why Do Checkoff Groups WantWhy Do Checkoff Groups WantTheir Programs Evaluated?Their Programs Evaluated?

To demonstrate their effectiveness in To demonstrate their effectiveness in expanding markets and enhancing profit.expanding markets and enhancing profit.

To assist in the design and adjustment of To assist in the design and adjustment of the program’s long-run strategic planthe program’s long-run strategic plan

To provide the necessary economic To provide the necessary economic ammunition to respond to legal challengesammunition to respond to legal challenges

Simply to satisfy the legislative requirementsSimply to satisfy the legislative requirements

To serve the information needs of contributors,To serve the information needs of contributors,industry, and other stakeholdersindustry, and other stakeholders

What Do Evaluation StudiesWhat Do Evaluation StudiesTypically Provide?Typically Provide?

Measures of Program “Effectiveness”Measures of Program “Effectiveness”

Retail Sales and Price ImpactsRetail Sales and Price Impacts

Benefits to Producers and Other Program Benefits to Producers and Other Program ContributorsContributors

Other Measure of Effectiveness:Other Measure of Effectiveness:

- Market Share Impacts- Market Share Impacts- Export Competitiveness Impacts- Export Competitiveness Impacts

- Production and Industry Growth Impacts- Production and Industry Growth Impacts

What Do Evaluation StudiesWhat Do Evaluation StudiesTypically Provide?Typically Provide?

Aggregate Measures of EffectivenessAggregate Measures of Effectiveness

Benefit-Cost Ratio (BCR):Benefit-Cost Ratio (BCR):

Marginal Rate of Return (MRR):Marginal Rate of Return (MRR):

Internal Rate of Return (IRR):Internal Rate of Return (IRR):

The total benefitsThe total benefits of the program divided by the total costs of the program divided by the total costs of promotion.of promotion. The percentThe percentchange in program benefits from a 1%change in program benefits from a 1%change in advertising expenditures.change in advertising expenditures. The returnThe returnthat a firm earns if it expands or investsthat a firm earns if it expands or investsin itself rather than investing that moneyin itself rather than investing that moneyelsewhere. elsewhere.

Communication DilemmaCommunication Dilemma

Results of Checkoff ProgramResults of Checkoff ProgramEvaluations Are Often:Evaluations Are Often:

Not Well UnderstoodNot Well Understood

Considered “Unbelievable”Considered “Unbelievable”

… … instead of instead of the additional sales from each additional dollar the additional sales from each additional dollar in advertisingin advertising

… … instead of howinstead of how responsive demand is to advertisingresponsive demand is to advertising

Economic Jargon Often Gets in the WayEconomic Jargon Often Gets in the Way““Advertising Elasticity”Advertising Elasticity”

RESULTS NOT UNDERSTOODRESULTS NOT UNDERSTOOD

““Marginal Effects of Advertising”Marginal Effects of Advertising”

… … instead of instead of how your advertising affects your competitors’ how your advertising affects your competitors’ markets and how their advertising affects your markets and how their advertising affects your marketmarket

““Spillover Effects of Advertising”Spillover Effects of Advertising”

… … instead of more easily understood instead of more easily understood concepts like “profit” and “revenue”concepts like “profit” and “revenue”

““Producer Surplus” “Economic Rent”Producer Surplus” “Economic Rent”

Results Not Straight-ForwardResults Not Straight-Forwardin Their Interpretationin Their Interpretation

Estimated AMEs reported in different forms Estimated AMEs reported in different forms and calculated in different waysand calculated in different ways


-- RetailRetail BCR: increase in retail sales revenue per BCR: increase in retail sales revenue per dollar invested in advertisingdollar invested in advertising

-- Industry Profit or Producer Surplus Industry Profit or Producer Surplus BCR: increase in industry profit or BCR: increase in industry profit or producer surplus per dollar invested in advertisingproducer surplus per dollar invested in advertising

-- Simultaneous Simultaneous BCR: increase in returns per $ of advertising assuming BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables demand as well as supply, price and other relevant market variables can change as a result of advertisingcan change as a result of advertising

-- Static Static BCR: increase in returns per $ of advertising assuming nothing BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising(including price) changes except demand as a result of advertising

-- Discounted Discounted BCR: any of previous BCRs in which the returns are BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of moneydiscounted to present value to account for the time value of money




-- RetailRetail BCR: increase in retail sales revenue per dollar invested in BCR: increase in retail sales revenue per dollar invested in advertisingadvertising

-- Industry Profit or Producer Surplus Industry Profit or Producer Surplus BCR: increase BCR: increase in industry profit or producer surplus per dollar in industry profit or producer surplus per dollar invested in advertisinginvested in advertising







-- Static Static BCR: increase in returns per $ of advertising BCR: increase in returns per $ of advertising assuming nothing (including price) changes except assuming nothing (including price) changes except demand as a result of advertisingdemand as a result of advertising








-- Simultaneous Simultaneous BCR: increase in returns per $BCR: increase in returns per $ of advertising assuming demand as well as supply, of advertising assuming demand as well as supply,

price and other relevant market variables can price and other relevant market variables can change as a result of advertisingchange as a result of advertising











-- Discounted Discounted BCR:BCR: any of previous BCRs in which the any of previous BCRs in which the returns are discounted to present value to account returns are discounted to present value to account for the time value of moneyfor the time value of money



No Clear Criteria for Interpreting AMEsNo Clear Criteria for Interpreting AMEs


-- BCR > 1 typically interpreted to mean that the BCR > 1 typically interpreted to mean that the returns to advertising > the costsreturns to advertising > the costs

-- But how high must the BCR be in order to justify But how high must the BCR be in order to justify a conclusion that benefits of the program a conclusion that benefits of the program sufficiently exceed the costs to warrant sufficiently exceed the costs to warrant continuation of the program?continuation of the program?

-- Typical BCRs range from 2:1 to 10:1. So how do Typical BCRs range from 2:1 to 10:1. So how do we interpret a BCR of 50:1 or 100:1?we interpret a BCR of 50:1 or 100:1?

-- And how do we interpret a BCR at the bottom of And how do we interpret a BCR at the bottom of or below the typical range? Negative?or below the typical range? Negative?

Results Do Not Answer FundamentalResults Do Not Answer FundamentalQuestions of Checkoff ContributorsQuestions of Checkoff Contributors

Is the Advertising a “Good” Investment for Is the Advertising a “Good” Investment for Those Who Pay for the Program?Those Who Pay for the Program?


-- Could they do better by keeping checkoff money and Could they do better by keeping checkoff money and investing in other common investment alternatives?investing in other common investment alternatives?

-- Economists:Economists: A fundamental concern for program A fundamental concern for program evaluations is the opportunity cost of the checkoff evaluations is the opportunity cost of the checkoff funds.funds.

-- Non-Economists:Non-Economists: If contributors think they could do If contributors think they could do better by investing checkoff funds in financial better by investing checkoff funds in financial instruments and earn a greater return, then they will instruments and earn a greater return, then they will be unconvinced by an estimated BCR that is positive be unconvinced by an estimated BCR that is positive and “high.”and “high.”

Results Do Not Answer FundamentalResults Do Not Answer FundamentalQuestions of Checkoff ContributorsQuestions of Checkoff Contributors

Whether or Not and How Contributors Personally Whether or Not and How Contributors Personally Benefit from Their Contributions?Benefit from Their Contributions?


-- Contributors expect that the money they Contributors expect that the money they contribute to a Checkoff Program will be spent in contribute to a Checkoff Program will be spent in such a way that they will individually be better off such a way that they will individually be better off than they would have been without contributing.than they would have been without contributing.

-- The AMEs we typically provide do little to The AMEs we typically provide do little to communicate how different groups of contributors communicate how different groups of contributors with different economic characteristics benefit with different economic characteristics benefit from checkoff program activities.from checkoff program activities.

Costs Are Observable But Benefits Are NotCosts Are Observable But Benefits Are NotContributors See the Effects of Checkoff Assessments Contributors See the Effects of Checkoff Assessments on their Bottom Lines But Have No Way of Knowing on their Bottom Lines But Have No Way of Knowing What Portion of Their Revenues Are Directly What Portion of Their Revenues Are Directly Attributable to Those AssessmentsAttributable to Those Assessments

RESULTS NOT “BELIEVABLE”RESULTS NOT “BELIEVABLE”

-- Evaluations specifically intended to measure the Evaluations specifically intended to measure the portion of industry revenues that can be attributed to portion of industry revenues that can be attributed to checkoff promotion programs. checkoff promotion programs.

-- Compare actual sales or revenues to nebulous Compare actual sales or revenues to nebulous concepts like “what might have been earned in the concepts like “what might have been earned in the absence of the program.”absence of the program.”

-- We tell producers that the $5 or $6 of revenue earned We tell producers that the $5 or $6 of revenue earned for every $ spent on advertising is actually in their for every $ spent on advertising is actually in their pockets but they just don’t know it.pockets but they just don’t know it.

Costs Are Observable But Benefits Are NotCosts Are Observable But Benefits Are NotContributors See the Effects of Checkoff Assessments Contributors See the Effects of Checkoff Assessments on their Bottom Lines But Have No Way of Knowing on their Bottom Lines But Have No Way of Knowing What Portion of Their Revenues Are Directly What Portion of Their Revenues Are Directly Attributable to Those AssessmentsAttributable to Those Assessments


-- Evaluations specifically intended to measure the Evaluations specifically intended to measure the portion of industry revenues that can be attributed to portion of industry revenues that can be attributed to checkoff promotion programs. checkoff promotion programs.

-- Compare actual sales or revenues to nebulous Compare actual sales or revenues to nebulous concepts like “what might have been earned in the concepts like “what might have been earned in the absence of the program.”absence of the program.”

-- We tell producers that the $5 or $6 of revenue earned We tell producers that the $5 or $6 of revenue earned for every $ spent on advertising is actually in their for every $ spent on advertising is actually in their pockets but they just don’t know it.pockets but they just don’t know it.

This Concept Has ProvedThis Concept Has ProvedDifficult to Communicate.Difficult to Communicate.

Large BCRs Often Taken AsLarge BCRs Often Taken AsImplying Large Market ImpactsImplying Large Market Impacts

Checkoff Programs Are Only One of Many Factors Checkoff Programs Are Only One of Many Factors That Impact MarketsThat Impact Markets


-- Total checkoff assessment collections compared to Total checkoff assessment collections compared to farm sales are usually quite small (less than 0.5% in farm sales are usually quite small (less than 0.5% in most cases).most cases).

-- Large BCRs can be calculated by dividing a small Large BCRs can be calculated by dividing a small return by an even smaller cost.return by an even smaller cost.

-- So even if the BCR is large, the actual market impacts So even if the BCR is large, the actual market impacts may actually be quite small and easily overwhelmed may actually be quite small and easily overwhelmed by the effects of the many other market forces.by the effects of the many other market forces.

-- Nevertheless, “large” positive BCRs are often taken to Nevertheless, “large” positive BCRs are often taken to mean that the program has a large impact on market mean that the program has a large impact on market demand, prices, profitability.demand, prices, profitability.

Checkoff Boards Tend to OversellCheckoff Boards Tend to OversellActual and Potential Program ImpactsActual and Potential Program ImpactsContributors Come to Expect Large Impacts When Contributors Come to Expect Large Impacts When the Impacts Are Usually Quite Modestthe Impacts Are Usually Quite Modest


-- When they do not see the large effects they have been When they do not see the large effects they have been told to expect, they disbelieve the research results. told to expect, they disbelieve the research results.

-- Contributors must be educated about the true Contributors must be educated about the true potential of their checkoff programs.potential of their checkoff programs.

-- Sell checkoff programs as tools to contribute to higher Sell checkoff programs as tools to contribute to higher demand, prices, and profits in good years and to demand, prices, and profits in good years and to reduce downside pressure in bad years rather than as reduce downside pressure in bad years rather than as a panacea for the financial problems faced by a panacea for the financial problems faced by contributors.contributors.

Focus on InappropriateFocus on InappropriateMeasures of “Effectiveness”Measures of “Effectiveness”

Market Price a Key Market Indicator But Market Price a Key Market Indicator But a Poor Indicator of Program Effectivenessa Poor Indicator of Program Effectiveness


-- Price effects of advertising depend on how responsive Price effects of advertising depend on how responsive supply is to price changes.supply is to price changes.

-- In industry with no supply controls and highly elastic In industry with no supply controls and highly elastic (price responsive) supply, most benefits(price responsive) supply, most benefits

of advertising come from a sales increase rather than of advertising come from a sales increase rather than from a price increase.from a price increase.

-- In industry with inelastic (price unresponsive) supply, In industry with inelastic (price unresponsive) supply, most advertising benefits come from price increase most advertising benefits come from price increase rather than from sales increase.rather than from sales increase.

-- BUT long-run supply response can dampen price BUT long-run supply response can dampen price effects of advertising over many years.effects of advertising over many years.

Focus on InappropriateFocus on InappropriateMeasures of “Effectiveness”Measures of “Effectiveness”

The Appropriate Measure of “Effectiveness” Depends The Appropriate Measure of “Effectiveness” Depends on Objectives of Promotion Programon Objectives of Promotion Program


-- We tend to report BCRs and other AMEs because we We tend to report BCRs and other AMEs because we assume that the driving objective of all promotion assume that the driving objective of all promotion programs is to enhance profitability.programs is to enhance profitability.

-- Strategic Plans often reveal other objectives or sub-Strategic Plans often reveal other objectives or sub-objectives of more immediate interest and relevance.objectives of more immediate interest and relevance.

ALB VISIONALB VISION

ALB STRATEGIC PLANALB STRATEGIC PLAN

““A unified, thriving American Lamb Industry that concentrates A unified, thriving American Lamb Industry that concentrates its resources around a prioritized and measurable plan that its resources around a prioritized and measurable plan that fosters the opportunity for profitability of all Contributors.”fosters the opportunity for profitability of all Contributors.”

1.1. Increase American Lamb demand by 19% by 2010Increase American Lamb demand by 19% by 20102.2. Protect & enhance the business climate for American Protect & enhance the business climate for American

lamblamb3.3. Leverage the ALB budgetLeverage the ALB budget4.4. Expand markets and market share for American lambExpand markets and market share for American lamb

5.5. Establish open and interactive communications with all Establish open and interactive communications with all segments of the industrysegments of the industry

6.6. Gather research data and evaluate programsGather research data and evaluate programs

““A unified, thriving American Lamb Industry that concentrates A unified, thriving American Lamb Industry that concentrates its resources around a prioritized and measurable plan that its resources around a prioritized and measurable plan that fosters the opportunity for profitability of all Contributors.”fosters the opportunity for profitability of all Contributors.”

4.4. Expand markets and market share for American lambExpand markets and market share for American lamb

1.1. Increase American Lamb demand by 10% by 2010Increase American Lamb demand by 10% by 20102.2. Protect & enhance the business climate for American Protect & enhance the business climate for American

lamblamb3.3. Leverage the ALB budgetLeverage the ALB budget

5.5. Establish open and interactive communications with all Establish open and interactive communications with all segments of the industrysegments of the industry

6.6. Gather research data and evaluate programsGather research data and evaluate programs

ALB VISIONALB VISION

ALB STRATEGIC PLANALB STRATEGIC PLAN

Recognize that CommunicationRecognize that Communicationis a Two-Way Streetis a Two-Way Street

For Researchers to Communicate Better, Then For Researchers to Communicate Better, Then Checkoff Groups Need to Better Communicate Checkoff Groups Need to Better Communicate What What They WantThey Want from an Evaluation from an Evaluation

IMPROVING COMMUNICATIONIMPROVING COMMUNICATION

-- What do you want to know about your programs?What do you want to know about your programs?

-- What metrics do you focus on?What metrics do you focus on?

-- What implies program success to you?What implies program success to you?

Increased Sales?Increased Sales? Higher Price?Higher Price?

Industry Growth?Industry Growth?

Increased Exports?Increased Exports? Reduced Imports?Reduced Imports?

Recognize that CommunicationRecognize that Communicationis a Two-Way Streetis a Two-Way Street

Also, Researchers Need to Know Also, Researchers Need to Know Why a Program Why a Program Evaluation is WantedEvaluation is Wanted


-- Legislative reporting requirements? Legislative reporting requirements?

-- Assist in program management? Assist in program management?

-- Strategic Plans provide insights into appropriate Strategic Plans provide insights into appropriate evaluation objectives and measures.evaluation objectives and measures.

Let the researchers design the study objectivesLet the researchers design the study objectivesand measures of effectiveness.and measures of effectiveness.

Researchers need to know what that means to youResearchers need to know what that means to youso they can provide measures to meet your needs.so they can provide measures to meet your needs.

Inform Boards About WhatInform Boards About WhatCan and Cannot Be DoneCan and Cannot Be Done

They Usually Do Not Understand ExtensiveThey Usually Do Not Understand ExtensiveData RequirementsData Requirements


-- Consumer attitude and awareness studies typically done by Consumer attitude and awareness studies typically done by advertising firms, market research firms, and business advertising firms, market research firms, and business school researchers.school researchers.

They Need to Understand the Types of Evaluations They Need to Understand the Types of Evaluations Typically Not Done ByTypically Not Done ByAgricultural EconomistsAgricultural Economists

-- The typical checkoff program evaluation attempts to The typical checkoff program evaluation attempts to measure directly the effects of advertising on demand measure directly the effects of advertising on demand without concern for attitudes and awareness.without concern for attitudes and awareness.

PROMOTIONPROMOTIONRETAIL RETAIL

SALES/PRICESALES/PRICE

Provide reasonable Provide reasonable estimates of the returns to estimates of the returns to specificspecific checkoff program checkoff program activities.activities.

Provide reasonable estimates Provide reasonable estimates of the returns to checkoff program activities in the of the returns to checkoff program activities in the aggregateaggregate..



They Also Need to Understand What Our Tools Will They Also Need to Understand What Our Tools Will Allow US to Do and NOT Do EffectivelyAllow US to Do and NOT Do Effectively-- CAN DO EFFECTIVELY:CAN DO EFFECTIVELY:

-- CANNOT DO EFFECTIVELY:CANNOT DO EFFECTIVELY:

?? FirmFirmPerformancePerformance



They Also Need to Understand What Our Tools Will They Also Need to Understand What Our Tools Will Allow US to Do and NOT Do EffectivelyAllow US to Do and NOT Do Effectively-- CAN DO EFFECTIVELY:CAN DO EFFECTIVELY: Provide reasonable estimates Provide reasonable estimates

of the returns to checkoff program activities in the of the returns to checkoff program activities in the aggregateaggregate..

-- CANNOT DO EFFECTIVELY:CANNOT DO EFFECTIVELY: Provide reasonable Provide reasonable estimates of the returns to estimates of the returns to specificspecific checkoff program checkoff program activities.activities.

All All FirmFirm

ActivitiesActivitiesFirmFirm

PerformancePerformance

Other Ways to Improve CommunicationOther Ways to Improve CommunicationIMPROVING COMMUNICATIONIMPROVING COMMUNICATION

Reduce economic jargon in reportsReduce economic jargon in reportsStandardize and categorize measures ofStandardize and categorize measures ofprogram effectivenessprogram effectivenessDevelop measures that more readily conveyDevelop measures that more readily conveythe success or failure of checkoff programsthe success or failure of checkoff programsin meeting their objectivesin meeting their objectivesBe careful not to imply that large estimated returns Be careful not to imply that large estimated returns mean large program effects on demand, prices, mean large program effects on demand, prices, profits, etc.profits, etc.Inform and educate on what checkoff programs can Inform and educate on what checkoff programs can reasonably be expected to achieve in terms of reasonably be expected to achieve in terms of magnitudes of effectmagnitudes of effect


Be careful not to oversell what advertising programs Be careful not to oversell what advertising programs can accomplishcan accomplish

$ Sales$ Sales

$ Advertising Expenditures$ Advertising Expenditures

If you take all the credit for If you take all the credit for periods when sales and periods when sales and prices are growing …prices are growing …

… … then you’ll have to takethen you’ll have to takethe blame if sales and price the blame if sales and price ever drop.ever drop.

And contributors won’t And contributors won’t believe the estimates of believe the estimates of positive returns to the positive returns to the advertising paid for withadvertising paid for withtheir checkoff assessments. their checkoff assessments.

Other Ways to Improve CommunicationOther Ways to Improve Communication

interpreting and communicating checkoff program evaluation results dr. gary w. williams

Documents