chapter€¦ · use of lagged variables, taking the log of the dependent variable, or intro -...

CHAPTER 13

TIME-SERIES ANALYSIS,PROGRAM ASSESSMENT,AND FORECASTING

It is common for public managers and nonprofit executives to collectobservations over a period of time, be it in years, months, weeks, ordays. Unemployment rates, welfare caseloads, ticket sales, tax revenues,

and expenditures are all examples of data that can be recorded and stored asobservations across time. Such data files require what is called time-seriesanalysis.

Although there are several techniques for conducting time-seriesanalysis, we will concentrate on regression-based techniques in this chapter.Such techniques can be used to forecast future values of an outcomevariable, analyze the relationships among variables over time, and evaluateprogram interventions.

As with all regression analysis, however, the analytic power of regression-based time-series analysis comes at a price: careful attention to theassumptions on which these techniques are based. We ask the same kindsof questions we face with regression analysis of cross-sectional surveydata. For example, Is the dependent variable continuous? Is/are theindependent variable(s) continuous or dichotomous dummy variables? Aretroublesome outliers present? Are the variables normally distributed? Is thespread of the residuals uniform across the range of predicted values on thedependent variable? Are the independent variables highly correlated witheach other?

305

� TIMES-SERIES ANALYSIS PRESENTS UNIQUEANALYTIC CHALLENGES

Time-series analysis, however, also presents additional concerns for the dataanalyst because of one or more of the following characteristics that typifytime series:

• Observations in time often display strong seasonal or cyclical pat-terns that can dominate or mask any relationship that you want to esti-mate, any long-term trend you would like to understand, or any short- andlong-term program effects that you would like to evaluate. Examples ofsuch cyclical patterns include absences from work, which tend to be high-est on Fridays of each week; farm incomes that are highest during the peri-ods in which crops are harvested; and violent crimes that spike duringsummer months. Fortunately, there are techniques for removing short-term fluctuations or cyclical effects from time-series data through the cre-ation of moving averages or through seasonal adjustments of the data.

• One observation in a time series is likely to be highly related to theobservation before and after it. This phenomenon in a statistical sense iscalled autocorrelation or serial correlation. Its presence causes estimates ofstatistical significance to be exaggerated and makes the unsuspecting ana-lyst conclude that the variables are statistically related to one another whenthey are in fact only spuriously related as a consequence of their joint rela-tionship to the underlying, time-dependent processes.1 Fortunately, thereare steps that you as an analyst can take to test for autocorrelation and toremove it from your data.

• Programs or events that take place at a particular time may produceresults only after a lapse of time. These effects may also be short-lived or,conversely, build momentum or reach a tipping point that generates mul-tiplicative effects in the long run. In other words, the effects may notappear immediately after a program intervention or may not be monoto-nic or linear. Here again, techniques exist for detecting these conditionsand for applying the appropriate adjustments through steps such as theuse of lagged variables, taking the log of the dependent variable, or intro-ducing new variables to assess program effects that are short, cumulative,or temporary.

Let’s illustrate each of these special problems in time-series analysis,starting with the challenges presented by cyclical patterns.

306–�–STATISTICAL PERSUASION

DETECTING AND REMOVING �

SEASONAL/CYCLICAL EFFECTS

Meier, Brudney, and Bohte (2009, pp. 365–368) provide a case of seasonaleffects for us to examine (which I have modified here). A city’s director ofpublic works is troubled by his sanitation workers’ absences from work. Heknows that shorthanded crews do not work as efficiently as fully staffedcrews and that it is difficult to forecast the need for and availability of substi-tute workers. He would prefer to minimize the use of substitute workers andshort-staffed crews by reducing the number of absences among his full-timesanitation workers. He, therefore, creates an incentive program wherebyunused sick leave at the end of each fiscal year can be “cashed in” as a bonus.

Obviously, the director of public works would like to know if hisincentive program is achieving its objective. So he collects data on employeeabsences for 30 workdays after he announces the new incentive program.That is to say, he creates a time series with which to analyze the results of hisprogram announcement.

These fictional data are included in the file “Sanitation Absenteeism,”which you should open in SPSS to follow the steps below. Let’s first displayworker absences graphically.

Step 1: Click Graphs/Scatter/Dot.

Step 2: Select Simple Scatter (if not already selected), and click Define.

Step 3: Move the variable ABSENCES into the box for Y-axis and thevariable DAY_NUMBER into the X-axis box. Click Continue and OK.

Step 4: Edit the chart in Chart Editor to draw a line connecting theobservations, and show the data labels (to identify which days of theweek are the spikes and troughs in the series) by clicking on one ofthe circles and selecting Elements and clicking on the Interpolation line(to connect the circles).

The resulting graph should look something like Figure 13.1. The num-bers in the boxes next to each circle on the graph represent the day of theweek (1 = Monday, 2 = Tuesday, etc.). We can see from this chart that mostabsences during this period occur on Fridays and Mondays of each week,while Wednesdays and Thursdays have the fewest absences. It is also thecase that each cycle is defined by a week or five successive days. Absenceson Monday are high, decline on each successive day to reach their lowest

Time-Series Analysis, Program Assessment, and Forecasting–�–307

point each Thursday, and then spike to the highest level each week onFriday, repeating a similar pattern each following week. It is somewhat diffi-cult, however, to detect from the graph above whether the programannouncement produced fewer employee absences. In fact, the absences atfirst appear to have grown during the first 2 weeks after the director’sannouncement.

You have at least two tactics available to you to make the effect of thenew bonus program more visible.

Moving Averages/Smoothing

The first tactic is to remove the short-term cyclical effects in this patternby calculating a moving average of observations that is equal in length to thenumber of observations that characterize each cycle—in this case, 5 days. In


302520151050

Day_number

35

30

25

20

15

10

5

Ab

sen

ces

5

43

21

5

43

21

5

4

3

21

5

4

3

21

5

4

3

2

1

5

4

3

2

1

Figure 13.1 Cyclical Patterns of Worker Absenteeism After Launch ofNew Incentive Program to Reduce Absences From Work

effect, you create a new variable from the average values of five adjacentobservations, and you accomplish what is called “smoothing” the time series,thereby removing the short-term cyclical pattern.

Step 1: Click Transform/Create Time Series.

Step 2: Move the variable for which you would like to calculate a movingaverage (e.g., Absences) into the variable box, and select Centered MovingAverage from the Difference drop-down list. In the case at hand, change theSpan to 5. Rename the variable ABSENCES_MA. Click on the Changebutton. Click OK.

You can see this new variable in the Data View screen of the SPSS.sav file,which should look something like the following (without the bracket, whichI’ve inserted and commented on below):


Note, as the bracket indicates, that the mean of the first 5 observationson ABSENCES becomes the first value of the third case (18.2). The mean ofthe next 5 observations appears as the moving average value for the fourthcase (18.4).

Step 3: A scattergram of this new variable across time is shown inFigure 13.2.


302520151050

Day_number

25

20

15

10

5

0

MA

(Ab

sen

ces,

5,5)

Figure 13.2 Five-Day Moving Average of Absenteeism

The visual display of a five-term moving average (Figure 13.2) stronglysuggests that the new bonus incentive program cut the incidence of absen-teeism after an initial period of about 2 weeks, when the program appearedto have no effect.

There is a problem, however, with this interpretation that is notuncommon in evaluating program effects through the use of time series. Towit, the data were collected from the point at which the new program wasinitiated. We know nothing about the pattern of absences before theprogram began. What, for example, would you conclude if the data displayedthe pattern shown in Figure 13.3?

You might say that the new program actually slowed a longer-termdecline in absences for nearly 2 weeks, after which the decline inabsences continued at its preprogram pace. The short and sweet lessonof this illustration is that you can be much more confident about yourconclusions if you collect data prior to a program or policy intervention.We will return to this type of “interrupted time-series analysis” later in thechapter.

Regression Controls for the Short-Term Cyclical Effects

A second tactic for assessing the effects of a program produces moreprecise estimates of how much of an effect the bonus program had, whileproviding us with the diagnostics tools useful in knowing whether theseregression estimates are biased or inflated. This tactic does not smooth thedata to eliminate short-term cycles. Instead, it uses all the data available bycontrolling for the short-term cyclical effects reflected, in this case, by thedays of the week.


302520151050−5Day_number

30

25

20

15

10

5

0

Ab

sen

ces_

5ma

New program begins

Figure 13.3 Time Series That Includes Observations Prior to theIntroduction of Program

In this latter regard, you create four dummy variables from theindicator for the day of the week (Monday, Tuesday, etc.) and enterthese dummy variables into the regression equation along with anindependent variable for the passage of time (i.e., DAY_NUMBER) andthe dependent variable, ABSENCES. (See Chapter 11 for a discussion ofhow to create dummy variables for a categorical variable with morethan two nonmissing values.) In effect, the equation we estimate willtell you, after controlling for the day of the week, the extent to whichdays since the passage of time affect absenteeism among sanitationworkers.

After creating these new dummy variables, request the followingregression analysis, using Analyze/Regression/Linear. Ask for all the statistics,plots, and options from the more complete regression analysis in theprevious chapter.

Some of the results of this analysis are presented below.


By inspecting the correlation table (not shown here), we can feelconfident that our model is not troubled by multicollinearity. Norelationship among the independent variables exceeds an absolute valueof .25. (Remember that our rule-of-thumb standard here is |.70|.) The

collinearity statistics in the Coefficients table (tolerance and VIF[variance inflation factor]) are also well within their recommendedlimits, further supporting the conclusion that our model has not violatedthis assumption.

Not surprisingly, the model appears not to be troubled by univariateoutliers. While the histogram of the dependent variable is not perfectly bell-shaped, the mean and median are reasonably close together, and thekurtosis and skewness are relatively close to their respective standard errors,all suggesting the absence of troubling outliers and a fairly normaldistribution. The residual statistics table (also not shown here) revealsMahalanobis and Cook’s distances and centered average values that arewithin suggested limits for outliers and normality (see Chapter 12 for adiscussion of these guidelines).


In the partially reproduced Coefficients table above, we see that theunstandardized regression coefficient (B) for the passage of time(Day_number) is −.46 (Sig. = .000). In other words, absences declined byabout one person every 2 days over the entire 30-day period. Thedummy variables for days of the week also provide coefficients that indi-cate how many fewer (because of the negative coefficients) absencesoccurred each day of the week in comparison with the omitted day(Friday). This can be interpreted as the persistence of these “seasonal”effects even after the announcement. When controlling for the effects ofthe new program, we find six fewer absences on Mondays in comparisonwith Fridays, nine fewer absences on Tuesdays in comparison withFridays, and so on.

A plot of time against the equation’s standardized residuals (Figure13.4), which we saved as part of this regression analysis, shows a suspicioussnakelike pattern, which is also indicative of autocorrelation. A conserva-tive or prudent analyst would adjust the data to eliminate the autocorrela-tion, the steps for which we will turn to in the next example.

The Model Summary table (above) shows that the combination of ourindependent variables explains about 84% of the variance in absenteeism(using the adjusted R2 statistic).

Beware, however, that the Durbin-Watson statistic of 1.254 departs fromour recommended value of 2.0. As you may recall, substantial departures(either more or less) from 2.0 may indicate the presence of auto- or serialcorrelation, which will exaggerate the explanatory prowess of the model’sindependent variables. Unfortunately, SPSS does not provide a test of theDurbin-Watson’s statistical significance. To assess this requires you to consult atable of critical values, below or above which the presence of autocorrelationwill be indicated. These critical values are determined by the number ofobservations (30 in this example) and the number of independent variables(5 in this example). Consulting a table for the Durbin-Watson distribution(Berman, 2007, p. 292) shows a lower critical value of 1.07 and an upper criticalvalue of 1.83 for a p value of .05.

Plugging these values into the chart below from Berman (2007, p. 247)suggests that our value of 1.254 falls into that inconclusive lower region of1.07 to 1.83.


Certain

0 dl du (4 −du) (4 −dl)2 4

CertainMaybe MaybeNone

Negativeautocorrelation

Positiveautocorrelation

DETECTING AND CORRECTING FOR AUTOCORRELATION �

The substantive question we’re interested in answering in the followingexample is the extent to which the gross domestic product (GDP) in theUnited States between 1960 and 2006 affects the percentage of resi-dents above the age of 16 who are defined as “out of the labor force”(i.e., people who are unemployed and not looking for work, sometimesreferred to as “discouraged” workers). This data set is labeled “Welfareand Economics.”

First, run some descriptive statistics on these two variables:

• GDP_in_trillions• Percent_out_of_labor_force_plus16


302520151050

Day_number

2.0

1.0

0.0

−1.0

−2.0

Sta

nd

ard

ized

Res

idu

al

Residuals Plotted Against Time for Employee Absenteeism Analysis

Figure 13.4 Plot of ZRESID by Time for Absenteeism Example

The mean percentage of U.S. residents (above the age of 16) who arenot working and not looking for work is 36.5% during the period 1960through 2006. The median is a nearby 36%. The lowest percentage out ofthe labor force is 32.9; the highest, 41.3. Skewness and kurtosis are fairlyclose to their respective standard errors, suggesting that our formal test ofwhether this variable is normally distributed will confirm our suspicionshere that it is normally distributed.

The mean GDP (standardized to 2000 dollars) is $6.13 trillion over thecourse of these 47 years. GDP ranges from a low of $2.5 trillion to a highof $11.3 trillion. Kurtosis and skewness statistics relative to their standarderrors appear to suggest a normally distributed variable. Ditto here for thelikelihood of the K-S statistic confirming our suspicions of a normaldistribution.

The one-sample K-S statistics indicates that neither of these distributionssignificantly departs from a theoretical normal distribution, as reflected inthe table below:


Request a one-sample Kolmogorov-Smirnoff (K-S) statistic to formallytest whether their distributions are normal. And plot each of these two vari-ables over time, using the Scatter/Dot function.

Your summary descriptive statistics might look something like thefollowing:

It is, however, useful to see a picture of the levels of both of these variablesover this period of time, which we can generate through the Graphs/Scatter/Dot command in which GDP and the percentage of U.S. residents outof the labor force are entered as dependent (Y-axis) variables and the variablefor year is entered as the X-axis variable. Requesting each of these graphs (andediting them for presentation here) should produce something that looks likeFigures 13.5 and 13.6.


2005200019951990198519801975197019651960Year

45

40

35

30

25

20

Per

cen

tag

eO

ut

of

Lab

or

Fo

rce

Figure 13.5 Percentage of People Out of the U.S. Labor Force,1960–2006


Figure 13.6 GDP (in 2000 Dollars) in Trillions, 1960–2006

2005200019951990198519801975197019651960Year

12

10

8

6

4

2

GD

P

Okay, let’s examine the relationship between GDP and percentage ofU.S. residents out of the labor force during this 47-year period by usingregression analysis.

Step 1: Click on Analyze/Regression/Linear.

Step 2: Move the variable for percentage of residents out of the laborforce into the dependent variable box and GDP (in constant 2000 trilliondollars) into the independent variable box. Request the same sets ofstatistics, plots, and saved variables as we did in our previous regressionanalysis, and click on OK.

Here is some of that output with my interpretative remarks:

Ignore the constant. It has no meaningful interpretation in thisexample. The unstandardized coefficient for the effect of GDP on laborforce participation is −1.06. That is to say, for each increase of $1 trillionin GDP, we see a corresponding decrease of a little more than 1 per-centage point in U.S. residents out of the labor force.


Levels of GDP explain about 80% of the variation in the percentageof U.S. residents who are out of the labor force. The Durbin-Watson sta-tistic, however, is troublesome. If we were to look up the lower andupper critical values for this statistic at p = .05 for one independent vari-able and 47 observations, we’d find 1.49 and 1.58, respectively. Our 0.051is well below the lower critical value, strongly suggesting the presence ofautocorrelation.

The plot of standardized residuals over time in Figure 13.7 has a trouble-some “snakelike” pattern that also suggests autocorrelation. (Note that we plotresiduals against a measure of time. It’s not a plot of standardized predictedvalues against standardized residuals as we’ve seen previously.)

But how do you correct for the autocorrelation that may be exaggeratingthe relationship between the GDP and labor force participation?

There are two widely used solutions to the problem of autocorrelation:

1. Introduce time as a variable in the time-series regression equation.

or

2. Use first-order differences between successive years on your indepen-dent and dependent variables.

This first tactic is predicated on the reasonable argument that you’reremoving the dependency of each year’s observation on the previous year’sobservation by explicitly introducing time into the equation. (There is inci-dentally a two-step version of this, in which you regress your dependent

variable on a measure of time [e.g., year] and save the residuals for furtheranalysis. Your residuals will have had the effects of Father Time removedfrom the series for further analysis.)

The second tactic employs first differences between successiveobservations in the time series (i.e., the observations I have for Time 3 is thedifference between that variable at Time 3 and Time 2). This tactic is basedon another reasonable presumption: If two variables are truly related to eachother over time, then their increases or decreases from one time period tothe next (e.g., the differences from one year to the next) should also berelated.

Let’s demonstrate how to take first differences and see if our conclusionsdiffer from the prior analysis and if we can eliminate the pesky problem ofautocorrelation.


Figure 13.7 Plot of Standardized Residuals Over Time From GDP andOut-of-Labor-Force Regression

1960

−2.0

−1.0

0.0

1.0

2.0

3.0

1970 1980 1990Year

Sta

nd

ard

ized

Res

idu

al

2000 2010


Step 2: Move both variables into the Variables box:, and make sure thatthe Function is “Difference” and the Order is “1.” Click OK.

Step 3: Repeat the same regression analysis as above, using the firstdifference variables.

Part of your output should look something like the following:


On average, the percentage of residents out of the labor force declinedby .15 each year during this 47-year period, while GDP increased each yearby $0.19 trillion. Changes in GDP from one year to the next are only weaklyrelated, however, to yearly changes in the percentage of residents out of thelabor force, as the table below demonstrates.

Substantively, an increase of GDP of $1 trillion in a single year (about fivetimes the average yearly increase in GDP across this period of time) wouldbe associated with a drop of about 1/3 of a percentage point in people out ofthe labor force. Another way to phrase this would be to say that an averageyearly increase in GDP ($0.19 trillion) would result in a decline of 0.06 per-centage point of people out of the labor force (−0.309 × 0.19 = 0.06). Small,but better than nothing, you might conclude.

Our model summary is no more impressive in terms of effect sizes.Year-to-year differences in GDP explain less than 2% of the variation inyear-to-year changes in the percentage of people out of the labor force.A table of critical values for the Durbin-Watson statistic with one inde-pendent variable and 46 observations shows lower and upper values of1.48 and 1.57, respectively, which suggests that our first-order differenc-ing failed to eliminate the problem of autocorrelation.2 An alternativestrategy would be to transform the variables by taking their log andrerunning the regression analysis. That task lies ahead for the adventur-ous and ambitious reader.

� EVALUATING PROGRAM INTERVENTIONS

As Berman (2007, pp. 249–252) writes, program interventions or policychanges can result in a number of different effects, all of which can be mod-eled in a regression analysis.

• Pulse: An effect that occurs only during a brief period of time, afterwhich the outcome returns to levels prior to the intervention

• Period: An effect that lasts somewhat longer than a pulse effect but,like it, sees outcomes returning to pre-intervention levels

• Step: A lasting effect that changes the outcome variable to a new level,at which it remains for a considerable length of time

• Increasing or cumulative: A lasting effect that grows in impact withthe passage of time

• Lagged: An effect that may take any of the forms above but that onlyappears after some period of time after the program or policy interventionbegins


These effects can be illustrated as follows:


0,………....0, 1, 0………….0

Policy

Postpolicy

Pulse impact

0,………....0, 1, 1……….....1

Policy

Step impact

0,………....0, 1, 1,………….1, 0.................0

Postpolicy

Period impact

0,………....0, 1, 2, 3, 4, 5, 6…

Increasing or cumulative impact

0,………....0, 1, ……………1

Lagged, step impact

Prepolicy

Prepolicy

PolicyPrepolicy

PolicyPrepolicy

PolicyPrepolicy

Let’s illustrate how we might model these effects in another examplemodified from Meier et al. (2009, pp. 427–428).

In this fictional story, Goose Pimple, Vermont, officials have noticedincreasing petty lawlessness in town, as reflected in more illegal parking.They decide to hire a parking enforcement officer (PEO; previously known

as a “meter maid”) and expect that he will have an immediate, but lasting,effect (i.e., step effect) and a longer-term cumulative impact after he figuresout where and when likely offenses are to occur (i.e., increasing effect).Unlike the director of public works in the first example, Springfield officialsbegan monitoring the number of tickets issued 8 weeks prior to hiring andtraining the PEO. They then collected data on the number of tickets issuedfor the 8 weeks after he hit the streets and ask you, their analyst, todetermine whether their policy is “working.”

Your data can be found on the course Web site as the Excel file “ParkingTickets.” First step: Import this Excel data file into SPSS, and save as anSPSS.SAV file.

Now, generate a graph of parking tickets across the 16 weeks for whichyou have data. (Consult previous examples if you don’t recall how to requestsuch a graph from SPSS.) After some editing, your graph should looksomething like Figure 13.8.


Figure 13.8 Parking Tickets Issued Before and After ParkingEnforcement Officer Goes to Work

Week Number

151050

Par

kin

gT

icke

ts

100

80

60

40

No PEO PEO at Work

Notice in the Excel file (or SPSS Data View) how the two program effectsbeing evaluated here (step and cumulative) are constructed below:

With Parking_tickets as the dependent variable, request SPSS to calculatethe appropriate regression statistics, graphs, and diagnostics to assess theeffects of hiring a PEO in contrast to the longer-term effects of the passage oftime.

We can see from the Correlations table (not shown here) that we mayhave a problem with multicollinearity insofar as all independent variables arerelated to each other above the general guideline of [.70]. Fortunately, theformal tests for collinearity (tolerance and VIF) are within suggested limits(see far right of the Coefficients table). (Tolerances are not less than .10, andVIFs are not greater than 10.)


� PULLING IT ALL TOGETHER: WELFARECASELOAD DECLINE AND THE ECONOMY

The welfare reforms of the Clinton administration sparked considerable contro-versy. Although these reforms were accompanied by corresponding increases inthe Earned Income Tax Credit, many “experts” (especially concentrated amongthose who might be considered “liberal”) predicted dire consequences for wel-fare recipients and their children. So strong were such sentiments that twohighly placed and well-respected members of the Department of Health andHuman Services resigned in protest, returning to their tenured university posi-tions. Even those who believed that the reforms were necessary were soon sur-prised at how dramatic their apparent effects were in reducing welfare caseloads.As the Bush administration assumed the mantle of power in January 2001, wel-fare rolls were about half of what they had been prior to the reforms of 1996.Many people at that time, however, were unclear about the extent to which thewelfare reforms produced these results or whether the booming economy of thelate 1990s was the principal cause of this decline.

Time-series data exist through 2006 to provide an opportunity to answerthat question. Such analysis, however, requires special attention to the pitfalls

The model summary table (below) indicates that our three independentvariables explain about 94% of the variation in parking tickets issued duringthis 16-week period. The Durbin-Watson statistic of 1.853 falls within theacceptable range of 1.73 and 2.27 for 16 observations and three independentvariables.


of time-series analysis, especially where serial correlation is likely to bepresent. Insofar as variables tend to increase (or decline) with time—often asa result of processes of inflation and population growth—time-series analysiswill often appear to reveal strong relationships among these variables when infact they are the spurious result of shared temporal processes.

Begin such an analysis with descriptive statistics and graphical displays ofthe principal dependent and prospective independent variables. Thedescriptive statistics suggest that our primary variables are normally distributed,but their histograms (not shown here) suggest that percentage of residents onwelfare and percentage out of the labor force are bimodally distributed.


A one-sample K-S test provides a more formal assessment of their nor-mality. (Recall that the K-S statistic’s null hypothesis is that the distributionsare normally distributed, and we would hope not to reject that hypothesis.)The Sig. values for K-S indicate that our dependent variable is not normallydistributed and comes close to declaring the same thing for percentage ofresidents out of the labor force.

Let’s pause here and see if any transformations of our dependent vari-able will help change its distribution. Although not shown here, taking thenatural log, square root, square, and inverse of this variable does not pro-duce a normally distributed variable. Mumble something about the centrallimit theorem, and plunge right ahead with the original dependent variable.Let’s turn to something more aesthetically appealing: a graph (Figure 13.9).

Figure 13.9 is surely worth a thousand words (or at least the next 191).President Kennedy entered office with fewer than 2% of U.S. residents onwelfare. When President Nixon assumed office 8 years later, the proportionhad nearly doubled, thanks largely to President Johnson’s Great Societyprograms. The rate of increase continued its steady climb until the first yearof Nixon’s second term in 1973. The percentages moved up and downsomewhat thereafter, but within a relatively narrow band, until the beginningof a second wave of increases in 1990, which reached their historical peak in1993, Bill Clinton’s first year in office.

The vertical line in Figure 13.9 marks 1996, the year in which new welfarereform was enacted. This program replaced an emphasis on job training andeducation with an emphasis on “work first” and the clear message that welfarewas to be limited to no more than 5 years of eligibility (although states had


Figure 13.9 Percentage of U.S. Residents on Welfare, 1960–2006

2005200019951990198519801975197019651960Year

6.0

5.0

4.0

3.0

2.0

1.0

0.0

Per

cen

tag

eo

fR

esid

ents

on

Wel

fare

1.401.51

1.621.76

1.88

2.46

3.03

3.81

4.57

5.475.37

5.11

4.684.454.46

4.554.60

4.56

4.60

4.744.59

4.765.03

5.205.27

5.09

5.23

4.95

4.15

3.33

2.862.54

2.312.24

2.151.98

1.67

5.38

5.04

some flexibility here). Interestingly, welfare rolls had already begun to declineprior to the enactment of the new law. But this is not an uncommon pattern assome states—permitted to depart from federal guidelines—adopt changes thatare subsequently picked up by federal legislation that mandates other states tofollow. And what was happening with the economy during this period of time?

Figure 13.10 displays several available, and commonly reported,economic indicators. It’s hard to tell from eyeballing these two charts,however, just what the effects of economics on welfare use might be. Let’sturn to regression analysis for that answer.

We are interested here in estimating both the step and the cumulativeeffects of welfare reform (when controlling for economic conditions). You willfirst have to create two new variables for these two types of program effects.Also, create a variable to represent time trends—that is, a variable with values 1,2, 3, 4, 5, 6, and so on for each year in the file.

After creating these three new variables, I entered them into a regressionanalysis with percentage of residents on welfare as my dependent variable


Figure 13.10 Selected Economic Indicators, 1960–2006

45

40

35

30

25

20

15

10

5

0

2005200019951990198519801975197019651960

Percentage out of the labor force

Unemployment rate

GDP in trillions (2000 $$)

Year

Sel

ecte

dE

con

om

icIn

dic

ato

rs

and the three new variables, as well as the unemployment rate, GDP inconstant 2000 trillions of dollars, and the percentage of residents over 16who were considered out of the labor force (i.e., those people out of workand not looking for a job). It would be easy to jump into an interpretation ofthe correlation and regression coefficients, but that would be unwise giventhe statistical challenges that time-series data present for regression.

Indeed, the Durbin-Watson statistic for this regression equation is 0.420,which is far below the acceptable lower critical value of 1.26 for n = 47 andk = 6. Our somewhat snakelike plot (Figure 13.11) of standardized residualsagainst time is an equally troublesome indication of autocorrelation. Our VIFstatistics are beyond the acceptable limits for all but the step-impact andunemployment variables. We appear not to have any multivariate outliers.And our probability-probability (P-P) plot (not shown here) isn’t too bad forour normality assumptions, but that’s the least of our troubles here.

We’ve got to conduct some major surgery on these variables if we are toproduce a believable story about the effects of the welfare reforms whencontrolling for economic conditions. (By the way, I should have run the firstregression by “setting aside” the observations for years 2005 and 2006 sothat I can later compare these levels with those predicted from myregression equation. This comparison provides an informal test of how wellmy regression model is doing in capturing the dynamics of change. I’ll do


Figure 13.11 Standardized Residuals Over Time Are Somewhat“Snakelike”

50403020100Time_trend

2.00

1.00

0.00

−1.00

−2.00

Sta

nd

ard

ized

Res

idu

al

that in the analysis from this point forward and demonstrate later whatthose comparisons are.)

One tactic to correct for autocorrelation is to take first differences ofyour measures. A second tactic is to use weighted least squares techniques,but let’s see if we can eliminate the pesky problem of autocorrelationthrough first differencing our substantive measures (we’ll leave the policyimpact variables and time trend alone).


Step 2: Move the dependent variable and the three economic variablesinto the Variables box:, and make sure that the Function is “Difference”and the Order is “1.”

Step 3: Repeat the same regression analysis as above.

SPSS doesn’t enable you to request a plot of standardized residualsagainst time, so you have to save these residuals from the regression analy-sis and request a separate scatterdot, an edited version of which is presentedin Figure 13.12.


Figure 13.12 Standardized Residuals Over Time for First-DifferenceRegression Model for Welfare Caseloads

2005200019951990198519801975197019651960

Year

3.0

2.0

1.0

0.0

−1.0

−2.0

−3.0

Sta

nd

ard

ized

Res

idu

al

The graph fails to reveal the snakelike pattern that would be indicativeof heteroscedasticity and nonlinearity. No residual exceeds ±3.0, somultivariate outliers are also not present. Tolerance statistics are well withinstandards to rule out multicollinearity. The P-P plot is similarly reassuring inthis regard.


1.00.80.60.40.20.0Observed Cum Prob

1.0

0.8

0.6

0.4

0.2

0.0

Exp

ecte

dC

um

Pro

bNormal P-P Plot of Regression Standardized Residual

Dependent Variable: DIFF(percent_residents_on_welfare,1)

The Durbin-Watson statistic improves but continues to indicate serialcorrelation. I can’t think of any other reasonable transformations to elimi-nate autocorrelation. The time trend variable is largely unrelated to welfareuse and is incorporated in the model anyway. There are no seasonal pat-terns to the variables that I can detect and remove by taking their movingaverages. I’m going to note these difficulties in the technical appendix butwill proceed at this stage to formulate the regression equation and see howwell it does in predicting the welfare percentages for 2005 and 2006, whichwe have excluded from the model estimation below.

The step-function variable appears to make a substantial contribution toyear-to-year differences in the percentage of U.S. residents on welfare. Iwould predict a decline of 0.84% in welfare use from one year to the nextafter passage of the legislation in 1996 when controlling for all other vari-ables. The cumulative impact is surprisingly positive, with year-to-yearchange in welfare use when controlling for other factors. This is a rareinstance in which a zero-order relationship changes signs when controllingfor other variables in the equation, as you can see in the correlation columncomparison with partial correlations. This suggests a longer-term, largelysuppressed, tendency for welfare use to increase a small, but increasing,amount each year after passage. It’s an odd finding to be sure and may be theresult of the very high correlation between the step- and cumulative-functionvariables (r = .86). This level of multicollinearity can cause signs to flip and isvery likely an artifact of this statistical problem rather than some quirky, inex-plicable substantive result.


This finding may also suggest that we exclude the cumulative functionfrom the analysis and stick with an estimate of just the step impact. Largechanges in sign or counterintuitive findings often call for such an adjustmentin our statistics. The following analysis includes the cumulative-impactvariable. (I could have compared these results with an analysis in whichthe cumulative effect variable is excluded to see, for example, whether the

predictions improve. If they do, it would be justifiable to exclude the cumu-lative effect from the regression equation and recalculate our predicted val-ues for the dependent variable.)

Year-to-year changes in unemployment rates and the percentage ofdiscouraged workers has an understandable impact on year-to-year changesin the percentage of residents on welfare.

Year-to-year changes in GDP also has a counterintuitive interpretation,although it is far from statistically significant and moves from an intuitivenegative correlation as a zero-order correlation to a negligible but positivepartial correlation.

Beware that the first differences will require some careful applicationwhen using these variables to predict future values. They are predictedvalues in light of the prior year’s value, so we’ll have to take that into account,as we do below.

The predictive regression equation is as follows:

Ydiff 2005 = 0.25 − 0.84(Step impact) + 0.11(Cum. impact) − 0.01(Timetrend) + 0.11(Unemploymentdiff) + 0.23(Out of lab forcediff) + 0.38(GDPdiff).

Plugging in the values for each of these variables (remember to use thefirst difference values in 2005 for the unemployment rates and percentageout of the labor force variables), we arrive at the following values:

Ydiff 2005 = 0.25 − 0.84(1) + 0.11(8) − 0.01(45) + 0.11(−0.4) +0.23(−0.10) + 0.38(0.33) = −0.49

Also remember that our dependent variable is a first difference. To findour predicted level of residents on welfare requires us to subtract 0.49 fromthe percentage of residents on welfare in 2004 (1.62). Our predicted valuefor the 2005 percentage of residents on welfare is, therefore, 1.13.

Our equation for the predicted value of the percentage of residents onwelfare in 2006 can be expressed by the following equation:

Ydiff 2006 = 0.25 − 0.84(1) + 0.11(9) − 0.01(46) + 0.11(−0.5) +0.23(−0.10) + 0.38(0.32) = −0.51.

Our predicted value for the 2006 percentage of residents on welfare is1.51 − 0.51 = 1.0%. These predictions compare with the actual percentagesof 1.51 and 1.40. Our model “overshoots” its mark in predicting more rapiddeclines in welfare use than were actually realized. We may be observingwhat are called “floor effects,” where it becomes increasingly difficult tomove toward 0% (similarly, there are “ceiling effects,” which make most out-comes difficult to achieve 100%).


To complete the next stage of the analysis, we will include the 2005 and2006 data and reestimate a regression equation so that we can predict levelsof welfare use in 2007, 2008, and 2009, assuming a yearly decline of 3% in thethree economic indicators that are included in the regression analysis. Wemight want to use an Excel spreadsheet for this purpose. But first, the newregression estimates (using first differences for the dependent and economicvariables). The coefficients table is reproduced below:


Here’s a comparison of the new regression equation (with 2 more obser-vations) with the prior equation:

New:

Ydiff = 0.25 − 0.79(Step impact) + 0.09(Cum. impact) − 0.01(Timetrend) + 0.11(Unemploymentdiff) + 0.25(Out of lab forcediff) + 0.37(GDPdiff).

Old:

Ydiff2005 = 0.25 − 0.84(Step impact) + 0.11(Cum. impact) − 0.01(Timetrend) + 0.11(Unemploymentdiff) + 0.23(Out of lab forcediff) + 0.38(GDPdiff).

The differences between the two equations are rather small. Usingthe newest model and predicting yearly “declines” in our economic mea-sures of 3% (i.e., the country would be “worse off ” on each of the threemeasures) would produce the following predictions for the percentage ofresidents on welfare:

Graphically, this continuation of welfare caseload declines would looksomething like Figure 13.13.

Finally, what would have happened in a “It’s a Wonderful Life” world inwhich the welfare legislation was not passed in 1996 and we continued as wehad before? This requires us to fit a regression model to the series from 1960through 1996 and then predict the percentage of welfare recipients based onthat model for 1997 to 2006.

Obviously (hopefully), this new regression model will not include step- andcumulative-impact variables (“You were never born George Bailey.”). We will stickwith our first difference variables. But it’s not clear whether a model with thethree economic indicators provides a better or worse fit to the data than does amodel with a more limited number of variables. Although not shown here,regression equations with a time trend variable and different combinations of thethree economic indicators discovered that the model with only first differences inunemployment explained more of the variance in the first-differencetransformation of percentage on welfare. Explaining more of the variance in thedependent variable will also mean that that model does the best job in predictingfuture values of welfare use. So that’s how I backed into that choice. Parts of thiswould probably be useful to include in the policy memo but more so in thetechnical appendix. It is also the case that the overall model does not explain asmuch of the variation in welfare use prior to the implementation of TANF


Figure 13.13 Historical and Predicted Welfare Rates, 1960–2009

20102005200019951990198519801975197019651960Year

6.0

5.0

4.0

3.0

2.0

1.0

0.0

Per

cen

tag

eo

fR

esid

ents

on

Wel

fare

0.79

1.081.29

1.401.62

1.761.88

2.05

2.46

3.03

3.81

4.57

5.04

5.385.47

5.375.11

4.68

4.454.46

4.554.60

4.56

4.60

4.474.59

4.765.03

5.20

5.09

5.184.95

4.15

3.33

2.86

2.542.31

2.151.98

1.67

(Temporary Assistance for Needy Families) as did the regression equation for thefull data series (adjusted R2s of .53 vs. .20 in the Model Summary table below).


Figure 13.14 Standardized Residuals Over Time

200019951990198519801975197019651960Year

4.0

2.0

0.0

−2.0

−4.0

Sta

nd

ard

ized

Res

idu

al

Our Durbin-Watson statistic still indicates serial correlation, which weshould caution our readers about in the technical appendix, but ourcollinearity statistics are well within the suggested limits. A plot of standard-ized residuals against time is not too snakelike, suggesting no problems withnonlinearity or with heteroscedasticity. Nor do there appear to be multivari-ate outliers exceeding ±3.0 in standardized residuals (Figure 13.14).

The regression results provide us with the following regression equation:

Ydiff = 0.26 − 0.01(Time trend) + 0.09(Unemploymentdiff).

I find it somewhat easier to run these simple formulas in Excel andimport the results (via cut-and-paste steps) into the Data View file in SPSS.Here are the results of those calculations in Excel:


I prefer SPSS’s graphical facilities, however, so I’ll plot these predicted valuesof welfare use (if welfare legislation had not been passed) as shown in Figure 13.15.

Figure 13.15 Historical Versus Predicted Levels of Welfare Use Had Welfare Reform NotTaken Place, 1960–2006

6.00

5.00

4.00

3.00

2.00

1.00

201020001990198019701960

Actual percentage on welfare

Predicted percentage onwelfare without reforms

Year

Per

cen

tag

eo

fR

esid

ents

on

Wel

fare

If you are a fan of reduced welfare rolls, you can’t help but love the 1996reforms and make Bill Clinton into a hero. And you might conclude, “It’s thelaw, stupid, not the economy.”

NOTES �

1. In budgeting, this phenomenon has its own special status as “incrementalism,”where next year’s budget is created with a firm eye on this year’s budget, with smallincremental adjustments made to each line item.

2. The alternative tactic of regressing time (e.g., year) on both variables, savingthe residuals, and then regressing these residuals on each other was no more suc-cessful in eliminating autocorrelation in this case.


To practice and reinforce the lessons of this chapter of the book, turnto Exercise 9 in the Student Workbook (available at http://www.sagepub.com/pearsonsp/).

chapter€¦ · use of lagged variables, taking the log of the dependent variable, or intro -...

Documents