estimating the effectiveness of advertising: some pitfalls in econometric methods

Estimating the Effectiveness of Advertising: Some Pitfalls in Econometric MethodsAuthor(s): Richard E. QuandtSource: Journal of Marketing Research, Vol. 1, No. 2 (May, 1964), pp. 51-60Published by: American Marketing AssociationStable URL: http://www.jstor.org/stable/3149922 .

Accessed: 02/09/2013 04:00

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact [email protected].

.

American Marketing Association is collaborating with JSTOR to digitize, preserve and extend access toJournal of Marketing Research.

http://www.jstor.org

This content downloaded from 194.214.27.178 on Mon, 2 Sep 2013 04:00:25 AMAll use subject to JSTOR Terms and Conditions

http://www.jstor.org/action/showPublisher?publisherCode=ama

http://www.jstor.org/stable/3149922?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp


51

Estimating the Effectiveness of Advertising: Some Pitfalls in Econometric Methods

RICHARD E. QUANDT*

) The attempt to estimate the effectiveness of advertising by means of standard econometric models often fails because (i) data are contaminated, (ii) the preconditions for the applicability of the statistical model are not met, (iii) the underlying economic model is defective. These sources of difficulties are discussed in detail, some suggested ways of avoiding the difficulties are examined, and the future of such approaches is assessed.

In recent years more and more business firms have turned to economists and statisticians for analytic advice concerning the effectiveness of their advertising effort, as well as for specific sales forecasts for short, inter- mediate and long runs. This phenomenon seems to be a natural accompaniment of the noted successes achieved by operations researchers in other business problems, such as production scheduling, inventory control, etc.

The forecasting of sales, the prediction of demand, and the study of the effectiveness of advertising are all substantially similar pursuits and may be referred to as demand studies. In the following, it is assumed that we are dealing with a commodity produced by a single firm; i.e., the commodity is sufficiently differentiated by appearance, composition, condition, wrapping, label, etc., as to be identifiable as the product of a given firm. We are thereby specifically excluding from considera- tion certain staple goods and raw materials which are identifiable essentially only by grade but not by producer. Thus we are concerned with the determinants of the demand for a monopolistic competitor's product --one which is unique, yet for which more or less close substitutes exist.'

Some approaches used in demand studies for products of this kind are as follows:

1) Methods based in some sense on the autocorrela- tion properties of the time series to be forecast; most often this is retail or wholesale sales or factory ship- ments. One can distinguish at least two methods falling

in this category, namely (a) exponential smoothing2 according to which the forecast for a variable depends upon the previous value of the variable and the error of this previous forecast, and (b) time-series decomposi- tion8 by which usually trend, cyclical and seasonal com- ponents are identified and are extrapolated for forecasting purposes.

2) Methods which conceive demand to be a random variable obeying a Markov process.4 Statistics on brand switching can be made to yield a matrix of transition probabilities which can then be used to calculate the limiting distribution of market shares among the various competing brands.

3) Structural econometric methods which attempt to provide causal explanations for the phenomenon to be studied (i.e., explanations for the quantity demanded) by (a) setting up economic hypotheses consisting of statements about relationships among key variables and (b) subjecting these hypotheses to empirical tests by various statistical and econometric methods.

The objective of this paper is to describe and analyze the applicability of certain statistical and econometric methods to estimating the demand function for a differentiated, brand-named consumer good which is produced not to order but in anticipation of demand. We shall thus be concerned exclusively with the third of the above-mentioned approaches to the study of demand. In discussing the pitfalls of various concrete procedures we shall undoubtedly name some difficulties which have bearing on the other two approaches as well.

We shall refer here to a hypothetical firm, named simply The Firm, which shall be endowed with as many realistic problems as seems possible.

It must be emphasized that, although many of the difficulties discussed here have been encountered in practice, it is not implied that actual cases will be beset by all of them. Endowing The Firm with a plethora of

* Richard E. Quandt is professor of economics at Princeton University.

Most of the concrete cases which provided illustrations of problems discussed in this paper arose from projects which were the joint responsibility of Professor William J. Baumol and myself. My foremost debt is therefore to him. I am also grateful for comments to Tibor Fabian and Harold W. Kuhn. However, the views expressed in this paper are not necessarily shared by them. The responsibility for errors remains mine in any case. This paper was completed under support from the National Science Foundation.

'Demand studies for staple goods are numerous. See, for example, [6] and [10].

SSee [9]. Standard computer programs are available for this.

'See [4] and [7].



52 JOURNAL OF MARKETING RESEARCH, MAY 1964

woes conveniently enables one to summarize efficiently the nature of econometric problems and their possible solutions. With this intent of the paper in mind, one should interpret any relatively pessimistic conclusions not as signals to abandon the econometric approach but rather as incentives to work toward the elimination of the difficulties.

Section 2 is devoted to a description of The Firm in broad terms and to a discussion of data requirements and problems. Section 3 deals with single equation demand models, various difficulties arising from them and some attempts to avoid the difficulties. Finally, Sec- tion 4 discusses simultaneous equation models. Again, any similarity between The Firm and any real, existing business organization is purely coincidental.

THE FIRM AND PROBLEMS CONCERNING DATA

The Firm produces 68 different products which are sold in all fifty States. We assume that the objective is to devise and test a model of the demand for just one of these 68 commodities. This commodity will be referred to as the Product. For historic and no longer well remembered reasons The Firm divided the United States into 13 sales districts. Each sales district was further divided into from 2 to 5 sales territories. The total number of sales territories is 49. It is to be especially noted that no sales district happens to coincide with any of the census regions and that no sales territory coincides with any of the states. Miraculously, however, the boundaries of sales territories coincide with county lines; each territory thus contains an integral number of counties. Not unexpectedly, of course, the sales districts do not coincide with the Nielsen districts either.

In deciding what kinds of data are necessary for estimating demand functions it would be desirable to have recourse to a body of economic theory capable of providing guidance in this matter. To some extent such a body of economic theory exists and it is easy to derive theoretical demand functions from appropriate, plausible utility functions which will make the quantity demanded depend upon consumers' incomes, all prices and advertising expenditures. But economic theory dis- appoints us in at least two respects: (a) it gives no, or at best very weak, indications as to the factors which cause differences between the utility functions of different individuals. Thus, theory does not provide us with an explicit list of additional variables to be included in the analysis;5 (b) it gives no indication as to the probable algebraic form of the demand function.6

For the sake of completeness we note that some choice has to be made with respect to the dependent variable in any demand study. The dependent variable is usually either sales (in physical or value terms) or market share. Much of the subsequent discussion would remain essentially the same, regardless of which one we used in the various models. To deal with market share would be easier but also less informative, in that using market share figures would not allow us to estimate the growth of the entire market. For con- venience it was decided, perhaps somewhat arbitrarily, to conduct the discussion with reference to sales as the dependent variable.

In any case, the demand for the Product is thought to depend on three basic types of variables: (a) economic, (b) geographic-demographic and (c) marketing.

Economic Variables The economic variables are exogenous to The Firm

and essentially describe economic conditions and buying power at the time and place at which demand is to be estimated. Household disposable income (X1) is the most relevant statistic, and to obtain such a figure for The Firm's territories requires considerable mental gyrations.7 Other statistics could be used, such as the percent of the labor force that is unemployed but this variable does not seem to have more accuracy or explanatory value than the income figure.

Geographic-Demographic Variables The climate is thought to have some effect on the

sales of the Product; degree-days (X2) is a convenient indicator of the climate.8

The educational level of the population is also thought to influence sales of the Product; thus it becomes necessary to obtain figures on, say, the fraction of the adult population which has completed high school (X3). The age distribution may also be relevant and therefore we shall include as the next variable the fraction of the population aged more than 21 years (X4). Next, the urban-rural character of territories is perhaps of relevance and we shall consider the fraction of the population living in urban areas as the appropriate urban-rural index (X5). We need finally the size of the population (X6) in each of the territories. The last four variables or the raw materials for them are obtainable on a county basis from the Census of Population; the

I Of course, in any demand study a decision is ultimately made concerning the inclusion or exclusion of variables from an equation. On the whole these decisions rest on a mixture of theory, intuition, previous experience, introspection and casual empiricism. It does not seem possible to explain precisely the genesis of these decisions.

SExcept, of course, for well known theorems such as the one requiring demand functions to be homogeneous of degree zero in prices and income.

7Specifically, we can obtain annual figures on personal income by States from Personal Income by States (Supplement to the Survey of Current Business). These figures have to be ad- justed to correspond to the territories of The Firm by assuming that each state is homogeneous and by calculating for each territory a population-weighted average income. Alternatively we can obtain the income (distribution) of families decenni- ally from the Census of Population, Vol. II.

8 Degree-days can be obtained monthly from Climatological Data, National Summary (U.S. Dept. of Commerce, Weather Bureau); however, not on a county basis but by weather sta- tions only.



ESTIMATING ADVERTISING EFFECTIVENESS: SOME PITFALLS IN ECONOMETRIC METHODS 53

fact that these figures are not as recent as might be desirable is to be regretted but perhaps change in these variables is sufficiently slow that it really does not matter that one does not possess the latest figures.9

Marketing Variables The difficulties begin to multiply in earnest when we

come to marketing variables. These are essentially of two kinds: (a) those controlled by The Firm and (b) those controlled by other firms producing competing products. Marketing variables controlled by The Firm are:

(1) The price of the Product (P). There are fortu- nately very accurate price statistics in the files of The Firm by month and territory for five consecutive years.

(2) Newspaper and magazine advertising (A1). The dollar amount spent on newspaper and magazine advertising is available, but there is difficulty in breaking it down by territories. Most of this advertising is in national media and the suggestion that the total amount be allocated to territories in proportion to the readership of the magazine is rejected on the grounds that: (a) readership cannot be estimated accurately enough and (b) there is no known relationship between readership and exposure to advertisements. At least, however, a fairly reliable breakdown of the total by sales districts is obtained.

(3) Radio and television (A2). Again, there is no problem in breaking down the national total by sales districts, but it is impossible to get a breakdown by territories.

(4) Point of purchase advertising, consisting of dis- plays in stores and the like (A3).

(5) Direct selling effort (A4), consisting of the expenditure on salesmen in the field. Records on the territorial breakdown of expenditures on salesmen are unreliable since salesmen occasionally cross territorial boundaries. Even if they did not, such figures would be of little value since each salesman handles a number of products and it is impossible to ascertain how any given salesman allocates his time among various products. The suggestion to allocate salesmen's time in proportion to sales must be rejected as totally spurious.

(6) The dollar value of promotions, special deals and similar devices (A5), important as they may appear a priori, is also difficult to estimate on a territorial basis. Here much autonomy rests in the hands of local managers and neither the duration of promotions nor the location at which they take place can be ascertained exactly. At best, reliable district totals can be obtained.

Since The Firm's competitors engage in the same types of advertising effort, it would be desirable to obtain data on competitors' advertising effort according to

the same or substantially similar breakdown. This, of course, is altogether impossible. First, the sales districts and territories of competitors do not coincide with those of The Firm. Second, even if they did, the requisite information could not be obtained in any case. Third, it is not even clear who the competitors are. In some parts of the country it is thought that a local manufacturer repre- sents the only serious competition; in another part of the country the major competition is from a national brand; in a third region a national brand as well as several local brands compete with the Product of The Firm. The problem is ultimately "solved" by taking figures on total competitive advertising from Nielsen reports and "ad- justing" them to correspond to the sales districts of The Firm. This variable is then denoted by A,.

Finally, the competitive products' prices have to be taken into account. For this purpose an average price is calculated for the competition (P,) in which each competitor's price is weighted by his market share. Of course, prices have not remained constant during the year and so it is necessary to get an average price for each competitor over the unit period of observation; nor is it possible to associate the resulting price averages unambiguously with The Firm's sales districts without some questionable adjustments. As a final difficulty, it is recognized that some of The Firm's other products compete with the Product; whether and how they should be treated in calculating the average competitors' price requires some difficult decisions.

Given the kind of information thought to be necessary for estimating demand, what can be said about the general accuracy of the data collected at great effort? Some variables have accurate values for each territory. For some others a territorial breakdown is possible only with some questionable assumptions. For still others a territorial breakdown is impossible but a precise district breakdown is available. For the remain- ing variables not even an accurate district breakdown seems possible. In addition virtually no required information is available for more than the last three years (except, of course, economic-demographic information derived from government sources) since three years earlier The Firm changed advertising agencies and the former agency did not keep appropriate records needed for demand estimation.

SINGLE EQ UATION MODELS The objective is to predict sales and for this purpose

a number of models could be set up. As already indicated, economic theory gives little help in finding specific formulations. The actual procedure employed is rule-of-thumb, according to which a number of mathe- matically simple hypotheses are tested first. Our attitude is frankly experimental and we attempt to combine intuitive plausibility of models with statistically significant findings. For statistical purposes we adopt as a fundamental classification the categories of time-series and cross-sectional models.

' Assumptions of this kind are often made. They are often

testable hypotheses which are not, in fact, subjected to the test of data because testing would be too expensive or time consum- ing. Accordingly they remain in the category of hopes.




Cross-sectional Models Models of the cross-sectional variety are characterized

by the fact that they take a snapshot, over a given short period of time, of a cross-section of the economy's relevant sectors. For the purposes of various models we establish the following definitions:

Si = Sales per capita in region i, Xli = disposable income per capita in region i, X21 =-- degree days in region i, Xi = fraction of population which has completed

high school in region i, X4i = fraction of population aged more than 21

years in region i, Xs5 = fraction of urban population in region i, Pi = the price of the Product in region i,

An = the dollar value per capita of newspaper and magazine advertising in region i,

A2i =- the dollar value per capita of radio and television advertising in region i,

A3i =- the dollar value per capita of point of purchase advertising in region i,

A4q =per capita direct selling expenditures in region i,

As- =per capita expenditures of promotions in region i,

Ai = per capita advertising expenditures by competitors in region i,

Pi= average price of competitors' product in region i.

Cross-sectional Models: Model 1 We have not yet indicated whether region i refers

to the ith sales district (of which there are only 13) or the ith sales territory (of which there are 49 but for which some of the independent variables are known only with considerable error). Difficulties arise in either case. The first and simplest model one might attempt to test is

Si = ao + alXi-

+ a2X21i a3Xsi + a4X4i + asX5i + boPi + blAli + b2A21t- b3A3i + b4A4i + b5A5i- + b6Aei + b7P1i. (1)

Clearly i cannot refer to sales districts because there are no degrees of freedom for fitting equation (1). Thus one either omits some variables or aggregates some others or both. One could, for example, omit X3 as well as X6, since both the measure of educational levels and the measure of urbanization are (or might be) highly correlated with X1 (mean disposable income). Several of the advertising effort variables could be aggregated and one could define, perhaps, a new variable As by As -= As A4 + A As. This, of course, displeases The Firm's chief economist who wishes to use the demand model for a rational allocation of advertising expenditures among competing media. But even if his objections were disregarded or overcome, these procedures would not help materially: there are still ten parameters to be estimated and only thirteen observations.

It is therefore decided to disregard the relative in- adequacy of the territorial breakdown of the data and to let "region i" refer to "territory i." There are now sufficiently numerous degrees of freedom and least squares estimates are obtained for the coefficients of equation (1).

It is a reasonable question to ask how the goodness of the results shall be judged. The statistic upon which most reliance is placed will depend on the purpose for which the analysis is conducted. If the primary goal is to judge the sign of slopes or perhaps elasticities, one would tend to rely heavily on significance tests based on one or more regression coefficients. In other circumstances the error of forecast may be the most relevant test criterion. An F-test may be the single best test of the general linear hypothesis. For the sake of completeness all of these should be examined. Since we shall refer frequently to the goodness of an estimated equation we shall generally abridge the examination of the various goodness measures. Whatever results are cited in this regard are to be construed as being indicative of the general outcome of several tests. Accordingly, we find that only a,, b3 and b4 are significantly different from 0 on the .05 level of significance using the t-test. The coefficient b4 is negative, indicating that increased direct selling effort reduces sales contrary to reasonable expectations. The correlation coefficient is .55. The result is generally unsatisfactory.

Just to be certain, the calculation is repeated with another set of data, namely data from the previous year. The correlation coefficient is of the same order of magnitude but the signs of at least half the regression coefficients differ from their signs in the previous run. This result is consistent with the fact that the equation from period t is, on the whole, useless for forecasting for period t + 1. The result continues to be unsatisfactory.

Unfavorable and surprising results require interpretation and rationalization. It would be convenient to ascribe the blame to inadequate data. However, such a move would be premature. Model 1 is admittedly naive and the attempt to interpret the unsatisfactory results leads to the formulation of a new model.

Cross-sectional Models: Model 2 It is no longer hypothesized that given levels of

independent variables cause a certain level for the dependent variable, sales per capita. The rationale for Model 1 becomes questionable when one considers that it implies that, ceteris paribus, a one dollar increase in newspaper and magazine advertisement per capita will have the same effect on sales per capita whether the increase occurs in the City of New York or in the State of Montana. This no longer seems reasonable, not even if one makes allowance for such obvious factors as climate and income differentials. There may be too many other economic as well as noneconomic variables that affect behavior.




The new model which temporarily replaces Model 1 states that it is the change in sales from one year to the next by regions that is affected by certain variables such as the changes in the various advertising expenditures from one year to the next by regions. Explicitly Model 2 is written as

ASi = ao + alAXni

+ a2X21 + a3X3i + a4X4i + asX5i + b0APi + blAAn1 + b2AA21 + b3AA31 + b4AA4i b5AA5si b6AA i + b7APci (2)

It may be noted that all variables appear as changes except for X2, X3, X4 and Xs. In any given region no material change can be expected in these four variables from one year to the next--and perhaps X1, average disposable income, should be included in this category as well. They are permitted, however, to remain in Model 2, since their levels may influence the change in sales.

The existence of a static relation between temporal changes in variables on a cross-sectional basis does not contradict the earlier finding that if one regresses the absolute level of sales on the absolute levels of the independent variables, the signs of corresponding regression coefficients were not the same year after year. In other words, it is not inconsistent with Model 2 that Model 1 yields some regression coefficients which have opposite signs in two subsequent years. A simple numerical illus- tration suffices to demonstrate this.

Let there be two variables x and y on which observations are available in three regions and for two succes- sive years. The numerical values are displayed in Table 1.

Table I HYPOTHETICAL FIGURES ILLUSTRATING SOME

CONSEQUENCES OF CROSS SECTIONAL ANALYSIS

Independent variable x Dependent variable y

Region Year t: xt Yearf t- 1: xt-, Year t: yt Year t-- : yt-

I 7.5 5.0 7.5 5.0 2 8.0 10.0 6.0 10.0 3 10.0 13.0 3.5 9.0

It is obvious by inspection that if one regresses y or x in year t, the slope of the regression line will be negative and if one calculates the same regression for year t-1, the slope of the line is positive. If we calculate regional changes from one year to the next we obtain a good (positive) relationship between Ay and Ax. Hence, a firm relationship between annual changes in cross-sectional variables is indeed consistent with unstable relationships (from one year to the next) between the corresponding absolute levels of the variables.

Being reassured on this point, Model 2 is estimated by least squares. The results are better than those obtained with Model 1: The correlation coefficient is now .68 and five regression coefficients, namely bo, bl, b3, b' and b7, are significantly different from zero. Unfortu-

nately b3 is negative, indicating that a positive change per capita in point of purchase advertising induces a negative change in sales per capita. In addition b7 is negative, indicating that an increase in competitors' prices tends to reduce (!) the sales of The Firm. Un- expected results require interpretation. Although we cannot be certain, we may suspect that the value of b3 may actually measure the effect of changing sales upon changing point of purchase advertising levels rather than the reverse. In other words, since the allocation of point of purchase advertising is largely in the hands of local managers, they may be able to react quickly to declining sales by increasing A3. Unfortunately, the ability to provide a reasonable interpretation does not undo the unfavorable result.

Table 2 HYPOTHETICAL FIGURES ILLUSTRATING SOME

CONSEQUENCES OF CROSS SECTIONAL ANALYSIS

Region Axt Ayt I 2.5 2.5 2 -2.0 -4.0 3 -3.0 -5.5

Critique of the Cross-Sectional Approach At this point the entire cross-sectional approach be-

gins to appear questionable. In order to clarify the reasons for this, we must discuss the basic rationale for this approach. Assume that 2n paired observations are obtained on an independent variable x and a dependent variable y where the pair (xi, yj) belongs to or is drawn from the itu region.10 If a least squares equation is calculated, the underlying model must be given by

yi = a + bxi + ui (3)

where the u1 are error terms. Equation (3) expresses the hypothesis that the same

a and b "applies" to every region; that is to say, except for the behavior of the u1 which are usually assumed to be independently and normally distributed with zero mean and constant variance, a given change A in the value of the independent variable in the ith region produces a change in the value of the dependent variable in the ith region which is numerically the same as the difference between a yi and yj (i = j) that exists when the corresponding xi and xj exhibit the same numerical difference A.

To put this in a slightly different manner, consider the (point) prediction problem that arises after a and b have been estimated. The estimated equation is

y- = a + bx1. (4)

10It is not necessary to specify here what x and y refer to. They may be absolute levels as in Model 1 or annual changes as in Model 2. However, the subsequent argument probably weakens still further the rationale for using Model 1.




Consider now the following two questions: (a) By how much would the predicted value of yi (sales per capita in region i) increase if the value xi (advertising expenditures per capita in region i) were increased by Axi? (b) By how much does yi exceed

.j (i.e., by how

much do sales per capita in region i exceed those in region j) if xi - xj has the value AxO?

The answers to the questions are bAxi and b Ax0 respectively, which are the same if Axi = AxO.

It is not necessarily reasonable to assume--as this cross-sectional approach implicitly does-that this is the case. The cross-sectional approach attempts to take cognizance of the fact that the fundamental relationship between the key variables y and x is not necessarily the same for every unit in a cross-section. It does this by adding other (demographic, geographic, economic) variables zj, z2, . . . etc., in the regression equation, hoping that they might account for the differences. But the use of such additional variables implies that we conceive the behavior of each unit in the cross section to follow one of the m relations

yl = al + bxl + ul

Y2 = a2 + bx2 + u2 (5)

y, = am + bxm + us where subscripts refer to the various units in the cross- section"" and where the ui are error terms. It is then reasonable to attempt to find a variable z which essentially "explains" the differences in the y-intercepts of equations (5), the equations which explain the behavior of individual cross sections. The amended (single) cross-sectional relationship aggregated over the m units then is

yi = a + bxi + czi + ui. (6) Such an approach cannot account for possible differences in the slopes b of the relations given by (5) which

P3 E E Qg

Q3

R p P2

R2 R3

E Q

Figure I

explain the behavior of the various units in the cross section. In reality the situation may be as indicated in Figure 1.

We assume that three points P1, P2 and P3 (referring to three regions) have been observed. On the basis of these points the estimated line is (approximately) EE; yet the points were generated (without error) from underlying regional relations R1, R2 and R3. Clearly EE might be a very bad predictor since observations in the three regions will move along lines R1, R2 and R3 but never along EE.

If these arguments are taken seriously-and indeed they should be--one might be tempted to abandon Model 2 as well. But at this point, in extremis, one hope remains. Model 2 can be reestimated for a different pair of years. If, as one hopes, relations R1, R2, R3, etc., governing the behavior of individual units in the cross- section, are parallel to EE, the regression coefficients of the estimated equation EE will be substantially unchanged when EE is reestimated for a different pair of years. The reason for this is that irrespective of the variation in x by regions for a different pair of years the scatter of observed points will continue to cluster close to the first estimated line EE.

If R1, R2, etc. are not parallel to EE, it would happen only by the sheerest coincidence, requiring possibly very peculiar compensating shifts in the values of the x-variables, that the new EE was substantially the same as the old one. Thus, referring to Figure 1 again, in order to have a substantially unchanged EE line with comparable standard error of estimate, the new set of points would have to be approximately Q1, Q02, and Q3.

On the basis of these arguments Model 2 is refitted for another pair of years. The results are not encourag- ing. Although the correlation coefficient does not change by much, only two of the previously significant regression coefficients continue to be significantly different from zero, namely b0 and bl. Unfortunately the sign of bl is now negative. It is small consolation that two other coefficients, previously not significant, now turn out to be significantly different from zero.

The Firm's staff economist and statistician exhibit the gravest doubts at this point concerning the usefulness of continued application of the cross-sectional approach. There are various, as yet untried formulations, some of which involve nonlinearities of one form or another. Particularly promising are hypothesized interactions between pairs of variables. It is readily agreed that it may not be unreasonable to suppose that X1, X2, P, A1, A2, As, A4, As, Pe, Ae may all interact with each other. This gives (102 = 45 pairwise interactions alone; far

too many to be included in the regression equation. Six of the 45 possible pairwise interactions are selected as being particularly plausible. Both Model 1 and Model 2 are then reestimated, this time including cross product terms between pairs of variables. Nevertheless, the results are not materially better than before. The re-

1' Regions in the particular case of The Firm.




gressions explain hardly more than 50 percent of the variation in the dependent variable and the signs of given regression coefficients tend to be unstable when the regressions are reestimated for a subsequent year.

It seems best that one abandon, at least temporarily, the cross-sectional approach in estimating the demand for the Product. Regrettably, one cannot know with re- liability all the causes of the failure. It is possible that the inaccuracy and generally contaminated nature of the data precluded the finding of any systematic cross- sectional relation on the sales territories' level. In addition, perhaps the conditions underlying a valid application of a cross-sectional approach were not met. Possibly the "correct" interaction terms were not included in the regressions. It is finally possible--though not probablel2--that the true regression coefficients are unstable from year to year. These and other reasons probably all contribute to the failure of the cross- sectional approach. Whatever the reasons, the approach is temporarily abandoned.

Time Series Models The second major attempt to explain sales consists

of taking each sales district by itself and attempting to explain current sales on the basis of current and past values of certain key variables. The immediate advantage of the time series approach by sales districts is that the geographic-demographic-economic variables can probably be safely neglected. One reason for this is that time series data are available only for a 3 year period and none of these variables is likely to have changed substantially within any given sales district. This also means that we no longer have to measure variables on a per capita basis over so short a period of time-which is the convention we shall adopt. An immediate disadvantage of the approach is that only 18 observations are available for each data series--one observation for each bi-month. Thus the number of coefficients that can be estimated is not very large.

Time Series Models: Models 3 and 4 Variables A3, A4 and A5 are not available bimonthly

and therefore have to be omitted from the analysis. We are thus left with the variables S, A,, A2, P, A, PC. Being concerned with degrees of freedom we further reduce the number of variables by considering only the price ratio R = P/PC. The first time series model, called Model 3, is

St -= ao + alAlt + a2A2t - a3Act + a4Rt + ut (7)

where the subscript t refers to the tth bimonth and where ut is the usual error term. Such an equation is to be estimated for each of the 13 sales districts.

Even before Model 3 is estimated it appears unlikely to yield good results. The data series are highly auto-

correlated and it is probable that the error term ut, representing the effect of omitted variables, will also be highly autocorrelated. Such a condition is likely to result in a seriously bad estimate of the variances of the regression coefficients.'l Indeed, the application of the Durbin-Watson test to the residuals from least squares regressions indicates that the residuals are highly autocorrelated. Model 4 is therefore immediately proposed and is

ASt = ao 0 - aAAt + a2AA2t -+ 3 aAAct + a4ARt - ut. (8)

Now, we observe that taking first differences reduces the number of observations by one. We could reduce the number of parameters to be estimated by requiring ao to equal zero; it seems preferable, however, to leave ao unrestricted and to test the hypothesis that its value is zero.'4 But Model 4 immediately suggests a number of other formulations. Since the number of plausible alternatives to (8) is very large, only a few will be stated here.

Time Series Models: Models 5 and 6 Model 5 is given by

ASt = ao + aAAt - a2AltAAt + a3AA2t + a4A2tAA2t + a5AAct +- a6ARt + ut. (9)

Model 5 has the advantage of permitting the existence of diminishing returns to advertising. Assume, for example, that AA2t= AAct = ARt = 0. Equation (9) then reduces to

ASt = ao0 + aAAt +- a2AltAAlt and if ao 0

ASt At a- + a2At. (10) AAlt

If there are diminishing returns to newspaper advertising but newspaper advertising on the whole still has a positive effect on sales, we would expect to find a, > 0 and a2 < 0. Note that the possibility of diminishing returns is not allowed in (9) with respect to the variables A, and R. The reason for this is purely practical since we do not wish to increase the number of parameters to be estimated more than is necessary. One may also note that the particular arrangement in (9) of the time subscript t is somewhat arbitrary and other models can be obtained by changing the hypothesis about the nature of lags. For example, one might formulate Model

St = ao + alAAlt_1 +- a2Alt_1nAlt_1 + a3AA2t- 1+ a4A2t- 1A2t- 1 + a5Act_ 1 + a6ARt + ut. (11)

Other models, which it does not seem worthwhile to write out in detail, may involve logarithmic formulations

" Obviously we mean "subjectively probable."

13 For a detailed discussion see [3]. 1 Another approach is to estimate (8) under the linear re-

striction ao = 0 and then to test the validity of the restriction. See [8].




or changes not over a single bimonth but over two bimonths. For the sake of simplicity we assume that Models 3-6 are estimated for each of the 13 sales districts.

Time Series Models: Results Model 3 is eliminated on the basis of high autocorre-

lation of residuals. Model 4 gives almost uniformly bad results: judging by the correlation coefficient alone we find that in only one sales district does Model 4 provide any sort of "explanation" by yielding a correlation coefficient of .74; in all others the correlation coefficient is less than .55. Models 5 and 6 give results which are comparable, Model 5 holding the edge. In one sense these results are astonishingly good: 10 of the 13 correlation coefficients are in excess of .8. Difficulties of interpretation arise, however, with the signs of some of the regression coefficients.

The following distinct cases predominate: (1) (Four sales districts) a, > 01, a2 < 0, a3 > 0,

a4 < 0. This case has the clear interpretation that there are diminishing returns to advertising via both media. With the signs of the coefficients as stated it is possible that with a sufficiently high level of advertising the level of sales will actually decline; it is verified, however, that in practice this does not take place by substituting into the regression equations the highest observed values of

A1 and A2 and noting that at no point does a, + a2AI and a3 + a4A2 become negative.

(2) (Three sales districts) Either al > 0 and a2 > 0 or a3 > 0 and a4 > 0. The interpretation must be that increasing returns to advertising expenditures prevail for one or both media. Although this in itself is not implausible, it effectively prevents the use of the calculated regression equation for marketing purposes. The reason is that these results do not allow us to go beyond the recommendation that advertising expenditures should be increased in the affected districts. We cannot say by how much expenditures should be increased since we must reject the notion that the regression equations can be extrapolated indefinitely in cases when increasing returns seem to prevail.

(3) (Three sales districts) Either al < 0 and a2 > 0 or a3 < 0 and a4 > 0. This is a somewhat paradoxical case in which, at low levels of advertising, sales diminish as advertising expenditures increase, while at high levels of advertising, sales increase (at an increasing rate) as advertising expenditures increase. Clearly in the high ranges of advertising expenditures, this is an increasing returns case. Unfortunately, the observed scatter of points is such that for some values of A1 (A2) the value of alt + azA1 (as + a4A2) is positive and for others

it is negative. These negative slopes A at low values AA

of A are difficult to explain even if one uses fairly complex explanations resting on notions of critical levels of advertising below which and above which behavior is significantly different in kind.

(4) (Three sales districts) a1 < 0 and a2 < 0 or as < 0 and a4 < 0 or as > 0. These are substantially ob- surd cases and the only consolation is that they are the cases with the least significant regression and correlation coefficients of all 13 districts.

On the whole, the results are not quite as disappoint- ing as those of the cross-sectional models. Nor are the results as promising as they would have to be before they could be used for a joint determination of optimal advertising levels for each medium in each district. If the results were uniformly good from the statistical point of view and if no increasing returns to advertising existed in any region, one could use the following simple approach to the optimal allocation of advertising expenditures.

Define Aijt as the advertising expenditures on the ith medium in the jth district in the tth time period. Assume for simplicity that we wish to maximize profit with respect to the Aijt but not with respect to the price ratio R. Assume that one can obtain coefficients kj (j = 1, ... 13) which denote profit per unit of sales in the jth district. Finally, let At denote total advertising in the tth

period. The maximization problem of The Firm for the tth period can be stated as follows:

13 Maximize EkjSjt -At (12)

J=1

2 13

subject to 1: 13 Ajt =At (13) =1 j=l

Sjt = Sjt--1 +- ao +• ajl(Aijt - Aijt-1) -- aj2Ajt(Aijt - Aijt _ 1) + aja3(A2jt- A2Jt- 1) + aj4A2jt(A2jt

- A2jt- 1) Vjt for j = 1, . .., 13

(14)

where (14) is the same as (9) except that (a) we have written out the differences explicitly, (b) we have re- placed the terms an5AAct + a6ARt by the catch-all term Vjt, and (c) we have introduced the subscript j to refer to the jth sales district. Maximizing (12) subject to (13) and (14) can be accomplished by forming the usual Lagrangian expression and setting the partial derivatives equal to zero. This results in a system of simultaneous linear equations the solution to which provides the optimal allocation of advertising expenditures. Since the coefficients of this system are known only statistically, this is not a trivial matter, since the probability distribution of the coefficients induces a probability distribution on the solution vector; hence it becomes relevant to speak of confidence intervals for the solution.'5 But since some of the sales districts exhibit increasing returns and some yield nonsense results, such a generalized procedure is

I See [1] and [5].




meaningless in any case. The best that can be accomplished under the circumstances is suboptimization for the four sales districts in which statistical results are good and diminishing returns exist. Probably the safest procedure is to allocate a total dollar amount for advertising to the aggregate of the four districts and to use a straightforward analogue to equations (12), (13) and (14) to determine the breakdown of this total among districts and media. Even so, it is not evident that useful results have been obtained, since after the confidence region is constructed it is found that some portions of the confidence region are not admissible for reasons which only the staff of The Firm understands.

It is difficult to give a completely impartial evaluation of the results up to this point. Undoubtedly, the time series approach has been somewhat more fruitful than the cross-sectional approach. Its results, however, still fall short of what one could ideally expect. The results make sense in at least some respects: The Firm's staff -but not the statistical analyst--can suggest reasons why increasing returns should have shown up in some districts. Unfortunately, the rationale appears only after the statistical finding. The field of statistics appears slightly redeemed, but one feels that the results must be considered scant in comparison with the amount of effort expended upon obtaining them.

SIMULTANEOUS EQUATION MODELS A next attempt to improve results consists of recog-

nizing that advertising expenditures are not exogenous but are jointly determined with sales. In other words, it is possible that causality here is a two-way street: not only do advertising expenditures affect sales but sales may influence the amount of dollars allocated to advertising.

If the various models are reformulated in order to take into account this eventuality, we obtain systems of simultaneous equations in which each jointly determined variable is explained by one equation. One well- known consequence is that statistical methodology changes and we can no longer estimate coefficients by the classical least squares procedures since classical least squares does not even yield consistent estimators in such cases.16

The economists at The Firm immediately and per- suasively argue that there is no point in attempting to modify the various time series models in accordance with this theoretical refinement since advertising budget decisions are made annually and the time series are based on a unit of bimonths; hence any "reverse causality" from sales (bimonthly) to advertising dollars (bimonthly) is extremely unlikely. The counter-argu-

ment that district managers--having been given an annual budget--can still carry out intertemporal allocations and reallocations of the budget in response to sales figures is either ignored or further countered by the re- mark that this does not happen. This contention is re- inforced by the fact that the unit of time is the bimonth; a period probably too short to allow At to be materially influenced by St.

It is therefore decided to reexamine the cross-sectional models from this point of view, but with a great deal of trepidation because of the already known data deficiencies of these models. For the sake of argument, we pretend that Model 1 is selected as the basic equation.17 It is agreed that the variables most likely to be jointly dependent with sales are AlI (newspaper and magazine advertising), A21 (radio and television advertising), Asi (point of purchase advertising) and Ae (competitive advertising). The latter is immediately discarded on the grounds that we cannot write down an equation "explaining" competitors' advertising strategy short of building a full scale model of all of competition: a task too great to be attempted.

In order to ensure that the basic equation (1), in which we are interested, is identified, we must find some variables which contribute to explaining All, A21, A31 but which do not directly affect S1, that is, do not occur in equation (1). The new variables introduced are:

C11 = The unit cost in territory i of radio and television spot advertisements.

C21 = The unit cost in territory i of newspaper and magazine advertisements.

Cai = A measure of The Firm's franchise in territory i, as measured by the fraction of stores in each territory which carry the Product.

Three new equations are set up to complete the model:

Al -= eo + elS1 + e2X11 + e3Cl (15)

A21 = cO + clSi + c2Xli + c3C21 (16)

A3i = do + d1Si + d2X11 + d3C31. (17) It is then attempted to estimate the coefficients of equations (1), (15), (16) and (17) simultaneously by the various methods available for this purpose. A standard computer code is employed for this purpose which estimates the coefficients by four methods: (a) by direct least squares which is equivalent to pretending that each equation exists in isolation, (b) by two-stage least squares, the method of limited information maximum likelihood and the method of full information maximum likelihood, each of which recognizes the jointly dependent nature of the variables Si,

Alt, A21 and A3i. Dif-

ficulties arise immediately because Si and C3i are very highly correlated and the computation procedure employed to obtain full information maximum likelihood

"l It is not the objective here to expound an elementary theory of simultaneous equation estimation. It is therefore assumed that the reader is familiar with the problems of identifi- cation and estimation. See [2].

" The discussion does not hinge on which model is selected as the basic equation.




estimates does not converge. Equation (17) is therefore discarded and the model estimated consists of equations (1), (15) and (16).

The comparison of the results with earlier single equation estimates is discouraging. (1) Whether we consider two-stage least squares estimates, limited information estimates or full information estimates, the standard deviations of the estimated coefficients are very large relative to the coefficients themselves. (2) Repetition of the calculation for a different year shows radically changed sign patterns for the coefficients. (3) The sign pattern of the coefficients as estimated by the various simultaneous equation techniques does not make as good intuitive sense as it does in the single equation case. (4) The multiple correlation coefficients for equations (15) and (16), obtained in the standard manner by applying direct least squares to each, are .01 and .04 respectively. It is therefore regretfully con- cluded that no simultaneous determination of Si, A11, A21 is at work; no improvement can be obtained by the relatively more sophisticated formulation. It is also agreed on the principle of "why throw good money after bad"-that it is not worthwhile to reformulate Model 2 in terms of such simultaneous interdependence. A final attempt is made by replacing variables by their logarithms and reestimating the model; when the results turn out to be even worse, the project is abandoned.

CONCLUSION In principle numerous other things could be tried.

Cross-sectional and time-series models could be pooled. Other statistical techniques, such as canonical correlation, could be employed. More variants of more models could be tested. Yet none of this seems very en- couraging.

Complete pessimism is not justified. One may note in particular some relatively quite successful studies--of a different variety, however [7]. Some useful results have even emerged from the econometric studies discussed above. From the time series study actual numerical allocations of advertising budgets have become feasible for four regions. For some other regions it has been determined with some confidence that increasing returns to advertising prevail, thus suggesting to The Firm a policy of more detailed study of the revelant regions. The findings of diminishing returns in some and increasing returns in other districts makes good sense to staff members of The Firm familiar with the histories of the districts.

But it is fairly clear that the determination of sales, by district, territory or time period, is a complex affair which cannot be successful without much more detailed and accurate data and without more sophisticated models. Not only are the data contaminated in one way or another, but they are so lacking in detail that some potentially highly important parts of a general model have to be completely discarded-such as competitive

interactions. It is not even clear that the superposition of all the difficulties discussed above has not resulted in rendering an econometric approach completely inap- plicable.

There are perhaps some ways out of the difficulties, but none of them is inexpensive. Several things must be tried, preferably simultaneously. Among these are (1) a complete overhaul of the data-gathering methods and facilities of The Firm-yielding accurate, if not necessarily exhaustive, data. (2) A sampling approach in- volving direct investigation of households on a cross- sectional as well as time series basis, possibly connected with a laboratory situation in which consumers are ex- posed to a controlled variety of circumstances. (3) A systematic reexamination of the problems of model formulation, leading to a greater variety of more sophisticated economic models. More than anything, we need to turn back, perhaps, to more classical methods of statistics and experimental design. It is possible that the conceptual and practical contamination of data and confounding of models can be avoided by subjecting approximately randomized sets of retail outlets to vary- ing treatments and applying analysis of variance techniques to the results which could then more properly be thought to come from carefully designed experimental situations. These proposals and others like them involve and raise perhaps as many problems as they solve. It is clear, however, that without a great deal of experimen- tation and improvement of the type suggested above, the standard statistical and econometric methods cannot yield truly satisfactory answers.

REFERENCES 1. G. E. P. Box and J. S. Hunter, "A Confidence Region for

the Solution of a Set of Simultaneous Equations with an Application to Experimental Design," Biometrika, 41 (1954), 190-199.

2. W. Flood and T. C. Koopmans, Studies in Econometric Method, Cowles Foundation Monograph No. 14, New York: John Wiley & Sons, 1953.

3. J. Johnson, Econometric Methods, New York: McGraw- Hill, 1963.

4. R. B. Maffei, "Brand Preferences and Simple Markov Processes," Operations Research, VIII (1960), 210-218.

5. R. E. Quandt, "Probabilistic Errors in the Leontief Sys- tem," Naval Research Logistics Quarterly, 5 (1958), 155- 170.

6. R. Stone, The Measurement of Consumers' Expenditure and Behavior in the United Kingdom 1920-1938, Cam- bridge: Cambridge University Press, 1954.

7. L. G. Telser, "The Demand for Branded Goods as Esti- mated from Consumer Panel Data," Review of Economics and Statistics, XLIV (1962), 300-324.

8. G. Tintner, Econometrics, New York: John Wiley & Sons, 1952.

9. Peter R. Winter, "Forecasting Sales by Exponentially Weighted Moving Averages," Management Science, VI (1960), 324-42.

10. H. Wold and L. Jureen, Demand Analysis, New York: John Wiley & Sons, 1953.



estimating the effectiveness of advertising: some pitfalls in econometric methods

Documents