1 toward a standard benefit-cost methodology for publicly-funded s&t programs crossing borders,...

1

Toward a standard benefit-cost Toward a standard benefit-cost methodology for publicly-funded S&T methodology for publicly-funded S&T

programsprograms

Crossing Borders, Crossing Crossing Borders, Crossing BoundariesBoundaries

2005 Joint CES/AEA Conference2005 Joint CES/AEA ConferenceToronto, CanadaToronto, Canada

October 26-29, 2005October 26-29, 2005

Jeanne PowellJeanne PowellEconomic Assessment Office

Advanced Technology ProgramNational Institute of Standards and Technology

2National Institute of Standards and Technology • Technology Administration • U.S. Department of Commerce

OutlineOutline

BackgroundAEA 2005: Analysis of Differences and

Considering an Approach to Comparability

Moving Forward


BackgroundBackground

ATP Mission and Project Selection Criteria

Evaluation FrameworkRole and Types of Benefit-Cost AnalysisPerformance MetricsBenefit-Cost Studies To DateStudy Similarities


ATP Mission …ATP Mission …To accelerate the development of

innovative technologies for broad

national benefit through

partnerships with the private sector.

ChemistryChemistry

BiotechnologyBiotechnology

ElectronicsElectronics

ManufacturingManufacturing

AdvancedAdvancedMaterialsMaterials

PhotonicsPhotonics

InformationInformationTechnologyTechnology


Scientific and Technological Merit (50%)– Technical innovation– High technical risk with evidence of

feasibility– Detailed technical plan

Potential for Broad-Based Economic Benefits (50%)– National economic benefits– Need for ATP funding– Pathway to economic benefits

ATP Selection CriteriaATP Selection Criteria


What We Measure WhenWhat We Measure WhenE

CO

NO

MIC

IM

PA

CT

SE

CO

NO

MIC

IM

PA

CT

S

Short-TermShort-Term Mid-TermMid-Term Longer-TermLonger-Term

1010or Moreor MoreYearsYears

-1 0 1 2 3 4 5 6 7 8 -1 0 1 2 3 4 5 6 7 8 99AnnounceAnnounce

Compe-Compe-titiontition

AnnounceAnnounceAwardAward

Complete Project

Post-Project Period

TotalEconomic Benefits

Benefits toAwardees

(Inputs/Outputs) Award/participant

characteristics Project funding R&D partnering Acceleration of R&D Innovative technology

development– Patents– Publications– Competitive advantage– Prototype products &

processes

(Outputs/Outcomes) Commercial activity

– New products– New processes– Licensing

Attraction of capital Strategic alliances Company growth

(Impacts) Broad national economic

benefits– Return on investment

– Public– Private– Social

– Inter-industry diffusion– Increased GDP/ tax base– Societal impacts


Types of B-C Case Studies in ATPTypes of B-C Case Studies in ATP

Traditional cash-flow based, return on investment measures– Benefit-to-cost ratios– Internal rates of return– Net present value

Hedonic price index models Macroeconomic models


Benefit-cost Analysis:Benefit-cost Analysis:Return on Investment MeasuresReturn on Investment Measures

Private return– return to ATP project participants on their

own investment

Public return– return to nation on ATP investment

Social return– return to ATP project participants and

nation on total ATP and private

investment


Performance MetricsPerformance Metrics

Return on Investment MetricsBenefit-to-Cost ratio is computed by

dividing the present value of benefits by the present value of investments

Net present value (NPV) is the present value of benefits minus the present value of investments

Internal rate of return is calculated by iterative solution for a rate at which the discounted value of investments equals the discounted value of benefits


ATP’s B-C Studies To DateATP’s B-C Studies To Date

12 studies published, including quantitative case studies of 28 projects

2 studies underway, including quantitative case studies of 4 projects

All studies performed by independent contractors


ATP’s B-C Studies To DateATP’s B-C Studies To DateShort Title (# of Projects)

Number Contractor Pub Date

Study Cost-$

Photonics Cluster (2) GCR 05-879

Delta Research

2005 228,000

Composites Cluster (2) GCR 04-863

Delta Research

2004 200,000

2mm Project—Retrospective (1)

GCR 03-856

MIT 2004 357,000

HDTV Joint Venture (1) GCR 03-859

RTI International

2004 75,000

A-Si Detectors for Digital Mammography (1)

GCR 03-844

Delta Research

2003 136,000

Component-Based Software (8)

GCR 02-834

RTI International

2002 154,000

Closed-Cycle Refrigeration (1)

GCR 01-819

Delta Research

2001 77,000

Digital Data Storage (2) GCR 00-790

RFF 2000 138,000

Flow-Control Machining in Auto Industry (1)

NISTIR 6373

NIST-Building & Fire Research

1999 90,000

Tissue Engineering (7) GCR 97-737

RTI International

1998 122,000

2mm—Early Assessment (1)

GCR 97-709

CONSAD 1997 25,000

Printed Wiring Board (1) GCR 97-722

Albert N. Link

1997 22,500


Study SimilaritiesStudy Similarities

Address ATP Mission, emphasizing public benefits to industry customers and end users and need for ATP funding to achieve these benefits

Cash-flow based approaches use:– Good practices consistent with public

finance literature; NPV, B:C, IRR– OMB-mandated 7% real discount rate (all

but 1 study) and constant $ estimates– Year of ATP project start for base year– Relatively short study periods

Results are not presented as “representative”, but rather as portfolio minimums


AEA 2005AEA 2005

Analyzing Differences and Considering an Approach to Study Comparability


AEA 2005: Analyzing Differences and AEA 2005: Analyzing Differences and Considering an Approach to Study Considering an Approach to Study

Comparability Comparability

Analyzing four major differences :

1) Study timing relative to ATP funding and uncertainties about future outcomes

2) Identifying the counterfactual and attribution of benefits

3) Which metrics? Public versus social return on investment

4) Adjusting for timing differences across studies and projects


AEA 2005: Analyzing Differences and AEA 2005: Analyzing Differences and Considering an Approach to StudyConsidering an Approach to Study

Comparability Comparability

Considering a standard approach based on three criteria:1) Meeting evaluation objective

2) Quality and accuracy

3) Comparability


1. Study Timing and Uncertainties 1. Study Timing and Uncertainties About Future OutcomesAbout Future OutcomesIssues:

Uncertainties concerning technology benefits are reduced over course of innovation process– At time of ATP funding, risks of technical failure (or

of meeting only limited technical objectives) are very high.

– Market/financial/business risks remain high during and after ATP funding phases of technology development and into commercialization phases

– Innovation life-cycle/timing varies by technology area Earliest studies conducted early in innovation

process – No projects had matured– Direct experience with range of possible outcomes

(and empirical data about innovation process) was limited


1. Study Timing and Uncertainties About 1. Study Timing and Uncertainties About Future Outcomes: Contrasting Studies Future Outcomes: Contrasting Studies

Study

Timing Relative to

ATP Funding

Technology Status

Treatment of Uncertainty

Tissue Engineering(7 cases)

Early in ATP for 3 cases; soon after ATP for 4 cases

Technologies not developed and/or not ready for commercialization(Biotech projects have very long timeline)

Barriers to meeting technical and economic goals not assessed. Sensitivity analysis performed but showed results not very sensitive to input variables tested. Considered only one application per case in order to be conservative

Component-Based Software(8 cases)

Soon after ATP

Technologies complete; in commercialization(IT projects have short timeline)

Selection emphasized projects with revenues to date and near-term prospects.Benefit analysis quantified only products actually on the market and assumed short product lives.


1. Study Timing and Uncertainties About 1. Study Timing and Uncertainties About Future Outcomes: Contrasting Studies Future Outcomes: Contrasting Studies

Notes: Tissue Engineering: 3 technologies made

considerable technical progress and commercialization continues to evolve but much more slowly than anticipated; 3 technologies failed to develop and companies dissolved; 1 is in transition to new company, future as yet is uncertain

Component-Based Software: focus on existing products and short-term product life reduced the amount of research required per project and enabled study of more projects


1. Study Timing and Uncertainties 1. Study Timing and Uncertainties About Future Outcomes: ConclusionsAbout Future Outcomes: Conclusions

Under conditions of considerable technical/business uncertainty about future outcomes:

Rigorous, complex economic models may or may not generate useful performance metrics.

Probability-based outcome measures that consider the broad technical, market, and business risks, as well as technology benefits, are likely needed

Metrics are more reliable/capable of precision as projects overcome technical, market, and financial barriers


2. Identifying the Counterfactual and 2. Identifying the Counterfactual and Attribution of BenefitsAttribution of Benefits

Issue:Where does project being analyzed start and

end? And what is the counterfactual, or baseline?– In theory, investment costs and benefits

included in analysis need to measure increment over the counterfactual (what would have happened without the project being measured).

– In practice, benefits (e.g., profit from sale of a product embodying new technology) derive from multiple public and private investments and from one or two of many product generations.


2. Identifying the Counterfactual and2. Identifying the Counterfactual and Attribution of Benefits—Contrasting Studies Attribution of Benefits—Contrasting Studies

Study Approach

Tissue Engineering

ATP project benefits are defined in terms of acceleration of benefits and increased probability of technical success relative to a hypothetical project without ATP. Public plus private benefits are compared to public plus private investments to compute social return on social investment.

Component-Based Software

Identified “project” that resulted in identifiable product sales and compared public/private benefits with all costs associated with that project to compute social returns

A-Si Detectors for Digital Mammography and Composites

Company interviews indicated “no project without ATP”; public benefits from product sales are compared to ATP investment to compute public return on ATP investment

Photonics Technologies

50% attribution to ATP where two different federal programs providing similar levels of funding were deemed jointly responsible for realization of any benefits.


2. Identifying the Counterfactual and 2. Identifying the Counterfactual and Attribution of Benefits—Conclusions Attribution of Benefits—Conclusions

Researchers use a variety of mechanisms to model benefits relative to a counterfactual (e.g., increased probability of achieving benefits, acceleration, allocation in accordance with amount or importance of funding sources)

ATP-funded technologies are often embodied in multiple products, each of which resulted from a number of different public/private funding sources and “projects”. Researchers measure benefits of products with strongest ATP linkages and share attribution where needed

Social rate of return has weaker assumption about “causation” than public return on ATP $ does.


3. Which metrics? 3. Which metrics?

Social return metrics– Economics literature compares social rate of return to

private rate of return (to see economic spillover) and then further compares private rate of return to private hurdle rate to identify market failure that justifies public funding

– Cost-shared projects imply comparison of benefits to both public and private recipients with investments from all sources

AND/ORReturn on ATP investment

– Directly addresses: What is the ATP program impact?– Involves substantially less historical/company proprietary

information than social return metrics


3. Which metrics? 3. Which metrics? Contrasting StudiesContrasting Studies

Issue:Some studies/contractors emphasize

social return metrics; others emphasize public return



Social return metrics Public return metrics(NC = Not computed)

NPV ($M)

B:C IRR (%)NPV ($M)

B:C IRR(%)

Component-based Software 1 -1.2 .37 N/M NC NC NC

2 789 39 363 NC NC NC

3 -1.22 .63 N/M NC NC NC

4 29.6 9.6 103 NC NC NC

5 2.06 1.8 31 NC NC NC

6 21 7.6 51 NC NC NC

7 52 18 136 NC NC NC

8 1.2 1.2 13 NC NC NC



Social return metrics (NC = Not Computed)

Public return metrics(NC = Not Computed)

NPV ($M)

B:C IRR (%) NPV ($M)

B:C IRR

Digital Video (Midpoint)

165.9 4.24 28.6 NC NC NC

A-Si Detector(base case)

NC NC NC 219 125 69

Composites(base cases)1

NC NC NC

552 221 57

2 NC NC NC 510 187 58

Photonics(base cases)1

NC NC 43 184 75 49

2 NC NC 51 201 244 350


3. Which metrics? Social versus 3. Which metrics? Social versus Public Returns Public Returns——ConclusionsConclusions

Both have merit for different purposes, and both have advantages and disadvantages– Social return metrics reflect cost-sharing

aspect of ATP and other public-private partnership projects; however, they require company-proprietary data and historical data about multiple funding sources (generally only rough estimate is feasible) .

– Public return on ATP investment provides measure of direct program impact; however, it requires judgments about of “causation”


4. Adjusting for Timing Differences4. Adjusting for Timing Differences Across Studies and Projects: Across Studies and Projects:

Constant $ and Base Year Constant $ and Base Year Issue:Different case studies based cash flow

estimates on constant dollars of different years

Although nearly all studies used the same discount rate (7%-real), they discounted cash flow estimates to different points in time (start of each project). (This practice evolved in financial world’s prospective analysis of potential investments and is used in Excel.)

Thus, reported NPV metrics are not comparable across projects.


4. Adjusting for Timing Differences 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ Across Studies and Projects: Constant $

and Base Year and Base Year EExample 1: Ranking does not changexample 1: Ranking does not change

Component-Based

Software 8 Cases

Reported NPV ($M)

Constant $ Year

Base Year for

Discounting

Adj. NPV--Consta

nt 2005$ ($M)

Adj. NPV--

2005 as Base Year ($M)

Aesthetic Solutions

-1.2 (7)

2000 1995 -1.3 -2.6 (7)

Commerce One JV

789 (1)

2000 1997 880 1,513 (1)

Extempo -1.22 (8)

2000 1995 1.36 -2.68 (8)

Intermetrics 29.6 (3)

2000 1998 33.0 53.0 (3)

Real-Time Innovations

2.06 (5)

2000 1996 2.30 4.23 (5)

SciComp 21 (4) 2000 1995 23 46 (4)

Tom Sawyer 52 (2) 2000 1996 58 107 (2)

Xerox PARC 1.2 (6) 2000 1995 1.3 2.6 (6)



and Base Year and Base Year Example 2 : Ranking ChangesExample 2 : Ranking Changes

Other Cash-Flow –Based

Cases

Reported NPV ($M)

Constant $ Year

Base Year for

Discounting

Adj. NPV--

Constant 2005$

($M)

Adj. NPV--

2005 as Base Year ($M)

Digital Video 165.9 (6)

2002 1995 177.7 349.5 (5)

A-Si Detector 219 (3) 2002 1998 236 374 (4)

Composites:Applied SciencesLincoln Composites

552 (1)510 (2)

20032003

19971998

579535

996 (1)

860 (2)

Photonics:X-Ray OpticsIon Optics

184 (5)201 (4)

20042004

19942000

188206

396 (3)

288 (6)


Adjusting to Constant 2005$Adjusting to Constant 2005$

NPV in Constant 2005$ =

Where: IPDGDP is Implicit Price Deflator for GDPand IPDGDPstudy is IPDGDP for Constant $ year used in study

1 + Q2 2005 IPDGDP – IPDGDPstudy

IDPGDPstudyx Reported NPV


Adjusting Base Year to 2005Adjusting Base Year to 2005

Adjusted NPV for Base Year 2005 =NPV in Constant 2005$ x 1.07n

Where: n = 2005 minus base year used in study


4. Adjusting for Timing Differences 4. Adjusting for Timing Differences Across Studies and Projects: Across Studies and Projects:

Constant $ and Base Year ExampleConstant $ and Base Year Example

Note:Rankings “before and after”

adjustment are for illustrative purposes only.– NPVs indicate net benefit magnitudes– NPVs do NOT indicate investment

efficiency



and Base Yearand Base YearConclusions: Both computed value of NPV and rank order can

change by adjustment to constant $ and common base year

NPV adjustments to constant $ and common base year are easy to make—directly from reported NPV

Comparability and aggregation of NPV results indicate need for such adjustments although practitioners may differ on best approach

IRR and B:C do not require adjustment; however, they cannot be aggregated


Moving ForwardMoving Forward

Work with contractors/researchers toward greater understanding of need for comparability of B-C results while seeking improved methodologies

Recognize and build on strengths of individual contractors/researchers ; help them overcome inconsistencies with comparability goals

Improve understanding of B-C methodologies among S&T evaluation community

Work with NIST experts in both B-C analysis and standards development


Call Jeanne Powell: 301-975-4196 Send e-mail to: [email protected]

Visit our website: www.atp.nist.gov

View our publications: www.atp.nist.gov/eao/eao_pubs.htm

For further information:For further information:

1 toward a standard benefit-cost methodology for publicly-funded s&t programs crossing borders,...

Documents

national economic benefitsneed

department of commercetypes

department of commercewhat

project benefits

broadbased economic

economic benefitscompanies

broad national benefit

atp fundingpathway