sensitivity analysis for cost-benefit studies of justice policies: a primer

Sensitivity Analysis for Cost-Benefit Studies of Justice Policies: A Primer

September 13, 2012

Ben Bryant, Staff Economist, Millennium Challenge Corpora7on Carl Ma5hies, Senior Policy Analyst, Cost-‐Benefit Analysis Unit, Vera Ins7tute of Jus7ce

• September 13, 2012

Sensitivity Analysis for Cost-Benefit Studies of Justice Policies: A Primer Ben Bryant

Economist Department of Policy and Evaluation Millennium Challenge Corporation

Carl Matthies Senior policy analyst

Cost-Benefit Analysis Unit Vera Institute of Justice


Agenda

Introductions and housekeeping 5 minutes

Sensitivity analysis as part of cost-benefit analysis (CBA) Perspectives included in a CBA Deterministic sensitivity analysis Probabilistic sensitivity analysis Q&A and conclusion

10 minutes 5 minutes 15 minutes 15 minutes 10 minutes


Housekeeping items

•  Use the chat feature to send your questions at any time during the webinar.

•  If you need webinar support or for troubleshooting: § Type questions into the chat box. § Call 800-843-9166. § Send e-mail to [email protected].

•  This webinar is being recorded.

•  The recording and PowerPoint slides will be posted on cbkb.org.


Webinar preview

You will learn:

§ Why sensitivity analysis is a critical part of cost-benefit analysis

§ Different techniques for conducting sensitivity analysis

§ How to interpret the sensitivity analysis results


Sensitivity Analysis as Part of Cost-Benefit Analysis (CBA)


Review: Basic steps of CBA

§  Specify the set of alternative projects.

§  Decide whose benefits and costs count.

§  Catalog the impacts and select measurement indicators.

§  Predict the impacts quantitatively over the life of the project.

§ Monetize all impacts.

§  Discount benefits and costs to obtain present values.

§  Compute the net present value (NPV) of each alternative.

§  Perform sensitivity analysis.

§  Recommend/choose a project.


Review: Basic steps of CBA Specify the set of alternative

projects.

Decide whose benefits and costs count.

Catalog the impacts and select measurement indicators.

Predict the impacts quantitatively over the life of the project.

Monetize all impacts.

Discount benefits and costs to obtain present values.

Compute the net present value (NPV) of each alternative.

Perform sensitivity analysis.

Recommend/choose a project.

Can be political

Requires significant topical knowledge

Requires significant quantitative skills and effort

Involves only calculations, in theory

Varies from simple to an entire study


Strengths and weakness of CBA

Some strengths:

§ CBA informs and fosters transparency in policy process by making benefits and costs explicit.

§ It also uses a common unit of measurement for potentially more efficient allocation of resources.

Some weaknesses:

§ Even with common measurement ($), people may disagree on how to value outcomes.

§ Results may vary, subject to choices about who counts and what counts.


State initiatives to legalize marijuana

Decisions on tax rate and regulatory

regime

Remove penalties

for sale and possession

Net impact on state and

local budgets

Marijuana consumption (quantity and

patterns of use)

Source: Kilmer et al. (2010)


A web of uncertain variables

Decisions on tax rate and regulatory

regime

Penalties removed

for sale and possession

Net impact on state and

local budgets

Marijuana consumption (quantity and

patterns of use)

Tax revenues from legal marijuana

sales

Other factors that

influence budgets

(e.g., criminal justice and treatment

costs; federal spending; tourism)

Changes in production and

distribution costs

Tax evasion and tax-induced

shift in mix of marijuana types

Marijuana prices consumers face

Consumer price

sensitivity

Non-price effects on

consumption

Source: Kilmer et al. (2010)


Characterizing uncertainty in a CBA

The way we characterize uncertainty is related to the methods we use:

• Best guess (percent change)

• Plausible bounds (deterministic, break-even)

• Probability distribution (Monte Carlo)

• Distinct stakeholder preferences (scenarios)


Questions?

Use the chat feature to send us your questions.


Perspectives Included in a CBA


Perspectives

• A program or policy’s net benefits can be highly sensitive to the costs and benefits included (who and what “counts”).

• Think of changing the perspective as a form of sensitivity analysis.


Which crime costs are included?

Source: McCollister, French, and Fang (2004)

Offense

Crime Victim Cost

Criminal Justice

System Cost

Crime

Career Cost

Total Tangible

Cost

Murder $744,239 $392,352 $148,555 $1,285,146

Rape/Sexual Assault $5,561 $26,479 $9,212 $41,252

Aggravated Assault $8,635 $8,641 $2,126 $19,472

Robbery $3,274 $13,827 $4,272 $21,373

Arson $11,453 $4,392 $584 $16,429

Motor Vehicle Theft $6,114 $3,867 $553 $10,534

Residential Burglary $1,654 $4,127 $681 $6,169

Larceny $479 $2,879 $163 $523


Which crime costs are included?

Source: McCollister, French, and Fang (2004)

Offense Total Tangible Cost Total Intangible Cost

Murder $1,285,146 $8,442,000

Rape/Sexual Assault $41,252 $199,642

Aggravated Assault $19,472 $95,023

Robbery $21,373 $22,575

Arson $16,429 $5133

Motor Vehicle Theft $10,534 $262

Residential Burglary $6,169 $321

Larceny $3,523 $10


Perspectives may affect project choice

Program

Estimated Impact

Annual Program Cost

Annual Net Benefit

CJS

Annual

Net Benefit

CJS + TV

Annual Net Benefit

CJS + TV + IV

A Prevents 100 motor vehicle thefts per year

$300K

+$87K

+$700K

+$730K

B Prevents 10 sexual assaults per year

$300K

-$35K

+$20K

+$2.0M

CJS: Criminal justice system TV: Tangible victim costs IV: Intangible victim costs


Questions?



Deterministic Sensitivity Analysis


Deterministic sensitivity analysis

This type of analysis examines relationships in specific, carefully chosen ways:

•  Partial sensitivity analysis varies model inputs separately to see how sensitive the CBA result is to each input’s value, all else remaining equal.

•  Extreme scenario analysis defines worst and best cases.

•  Break-even analysis solves for model inputs when the NPV equals zero.


The base case as starting point

•  A preliminary result derived from plausible or best-estimate assumptions

•  A reference point or benchmark for sensitivity analysis

•  A starting point—not necessarily the most likely outcome!


Example: Jail work-release program

•  Inmates get three days off sentence for every two days worked.

•  The daily marginal cost is slightly higher than for jail overall, but the program potentially reduces sentences and recidivism.

•  Assumptions: •  A six-year time horizon starts at Year Zero. •  No recidivism costs will occur until Year One. •  Only first-time offenders participate in the program. •  Non-compliers must serve out their entire sentence. •  The program has no effect on non-compliers.


The work-release CBA model

Benefits Formula

Cost savings from reduced jail time

npq(SR)(JMC)(0.6)

Cost savings from program effect on recidivism

npq(SR)(BRR)(E)

Costs Formula

Additional staff

s

Program cost for compliers

npq(SR)(PMC)(0.4)

Program cost for non-compliers

np(1-q)(SR)(f) (PMC)+ np(1-q)(JMC)(SR)

Model Inputs Hiring two new corrections staff (s) Number of eligible participants (n) Participation rate (p) Program compliance rate (q) Program marginal cost/day (PMC) Jail marginal cost/day (JMC) Initial sentence range for eligible individuals (ISR) Average program failure point for non-compliers (f) Baseline 5-year recidivism rate (BRR) Change in recidivism rate among participants (E) Sentence range for recidivists (RSR) Discount rate (d)



Model Inputs Base-Case Estimate Hiring two new corrections staff (s) $80,000/year

Number of eligible participants (n) 1,000

Participation rate (p) 0.9

Program compliance rate (q) 0.8

Program marginal cost (PMC) $75/day

Jail marginal cost (JMC) $55/day

Initial sentence range for eligible individuals (ISR) 120 Average program failure point for non-compliers (f) 0.1

Baseline 5-year recidivism rate (BRR) 0.5

Change in recidivism rate among participants (E) 0.2

Sentence range for recidivists (RSR) 180

Discount rate (d) 0.03

NPV for work-release program $6,520,950

The base case looks favorable, but what if there are departures from the mean values?


Partial sensitivity analysis



Model Inputs Base-Case Estimate Lower Bound Upper Bound Hiring two new corrections staff (s) $80,000/year $ 70,000/year $90,000/year

Number of eligible participants (n) 1,000 700 1300

Participation rate (p) 0.9 0.8 1.0

Program compliance rate (q) 0.8 0.7 1.0

Program marginal cost (PMC) $75/day $65/day $80/day

Jail marginal cost (JMC) $55/day $50/day $60/day

Initial sentence range for eligible individuals (ISR) 120 15 220 Average program failure point for non-compliers (f) 0.1 0.05 0.4

Baseline 5-year recidivism rate (BRR) 0.5 0.3 1.0

Change in recidivism rate among participants (E) 0.2 -0.2 0.6

Sentence range for recidivists (RSR) 180 120 365

Discount rate (d) 0.03 0.03 0.07

NPV for work-release program $6,520,950


Partial sensitivity analysis

Number of eligible participants (n)

Program marginal cost (PMC)

Jail marginal cost (JMC)

Sentence range for recidivists (RSR)

Baseline 5-year recidivism rate (BRR)

Program compliance rate (q)

Participation rate (p)

Change in recidivism rate among participants (E)

Average program failure point in non-compliers (p)

Initial sentence range for eligible individuals (ISR)

Hiring two new corrections staff (s) 90,000 70,000

1,300


Worst-case and best-case scenarios

Model Inputs Base-Case Estimate Worst Case Best Case

Hiring two new corrections staff (s) $80,000/year $90,000/year $70,000/year

Number of eligible participants (n) 1,000 1300 1,300

Participation rate (p) 0.9 1.0 1.0

Program compliance rate (q) 0.8 0.7 1.0

Program marginal cost (PMC) $75/day $80/day $65/day

Jail marginal cost/day (JMC) $55/day $60/day $60/day Initial sentence range for eligible individuals (ISR) 120 15 220 Average program failure point for non-compliers (f) 0.1 0.4 0.05

Baseline 5-year recidivism rate (BRR) 0.5 1.0 1.0 Change in recidivism rate among participants (E) 0.2 -0.2 0.6

Sentence range for recidivists (RSR) 180 120 365

Discount rate (d) 0.03 0.07 0.03

NPV for work-release program $6.5M -$43.3M $104.0M


Break-even analysis

The program would still break even if its marginal cost were $91/day, all else remaining equal.

Model Input Estimate

Hiring two new corrections staff (s) $80,000





Jail marginal cost/day (JMC) $55/day

Initial sentence range for eligible individuals (SR) 120

Average program failure point for non-compliers (f) 0.1



Sentence range for recidivists (SR) 180


NPV for work-release program $0


Break-even analysis

The program would break even if it had no effect on recidivism, all else remaining equal.

Model Input Estimate

Hiring two new corrections staff (s) $80,000





Jail marginal cost/day (JMC) $55/day

Initial sentence range for eligible individuals (SR) 120

Average program failure point for non-compliers (f) 0.1



Sentence range for recidivists (SR) 180


NPV for work-release program $0


Questions?



Probabilistic Sensitivity Analysis: Monte Carlo Simulations


Probabilistic sensitivity analysis

This type of analysis changes all model inputs at once.

•  The other methods tell us little about the likelihood of various program outcomes.

•  Monte Carlo simulation repeatedly draws random values for each model input to create a probability distribution of outcomes.


MS Excel add-ons for Monte Carlo simulation


Step 1: Define assumptions.


Define assumptions for all uncertain inputs.


Step 2: Set the forecast cell.


MC simulations provide the distribution of outcomes.


Distribution of outcomes


Questions?



Wrap-Up


Review

We covered:

§ Why sensitivity analysis is a critical part of CBA

§ Different techniques for conducting sensitivity analysis

§ How to interpret the sensitivity analysis results


Takeaway points

§ Base-case CBA is just a starting point. § Sensitivity analysis is a way of managing

uncertainty, not eliminating it. § Computation and simulation are

relatively easy. § Defining input boundaries and probability

distributions is harder.


Reminders

§ Please complete the evaluation form before you leave this webinar.

§ To receive information and notifications about upcoming webinars and other events:

•  Visit the Cost-Benefit Knowledge Bank for Criminal Justice at cbkb.org.

•  Follow us on Twitter at twitter.com/CBKBank.


Contact information

Ben Bryant [email protected]

Carl Matthies [email protected]

213-223-2445

E-mail: [email protected]

Website: cbkb.org

The Cost-Benefit Knowledge Bank for Criminal Justice (CBKB) is a project of the Vera Institute of Justice funded by the U.S. Department of Justice’s Bureau of Justice Assistance.

•  Website (cbkb.org) •  CBA toolkit •  Snapshots of CBA literature •  Podcasts, videocasts, and webinars •  Roundtable discussions •  Community of practice

This project is supported by Grant No. 2009-MU-BX K029 awarded by the Bureau of Justice Assistance. The Bureau of Justice Assistance is a component of the Office of Justice Programs, which also includes the Bureau of Justice Statistics, the National Institute of Justice, the Office of Juvenile Justice and Delinquency Prevention, and the Office of Sex Offender Sentencing, Monitoring, Apprehending, Registering, and Tracking. Points of view or opinions in this presentation are those of the authors and do not represent the official position or policies of the Millennium Challenge Corporation or the United States Department of Justice.


Thank you.

sensitivity analysis for cost-benefit studies of justice policies: a primer

Technology