A Practical InvestmentOptimization Tool by Means of Certainty-Uncertainty Searches

A Presentation for the Transportation Data Palooza: A Showcase ofInnovative Technology Solutions at the United States Department of

Transportation Headquarters, Washington DC

Brian G Chow

May 9, 2013

Chow -May 2013 #2

DOT Has Long Used Benefit-Cost Methodology for its Investment Analysis and Optimization

• For project selection and optimization, DOT performs standardized investment/performance analysis, e.g.

− Highway Economic Requirement System (HERS)

− National Bridge Investment Analysis System (NBIAS)

− Transit Economics Requirements Model (TERM)

• The foundation of these analytical tools is a benefit-cost methodology

Chow -May 2013 #3

Benefit-Cost Methodology Can be Enhanced by Casting in a Portfolio Framework

• Methods currently used by DOT can be improved by adding features to better deal with the post-9/11 environment and a stressful budget

− Need to express results in confidence level in order to address uncertainties in input data, future outcomes, and risk

• Current DOT methods based on expected values and supplemented by sensitivity analysis can be inadequate

− Need to allow the selected projects to meet multiple requirements individually

• Current DOT methods tend to score and sum a potential project’s contributions to various requirements in “oranges,” “applies,” etc., risking that selected projects are all great in providing “oranges” but together still providing too few “apples”

Chow -May 2013 #4

The Essential Difficulty of Portfolio Optimization Under Uncertainty

• Difficult to find the optimal portfolio among so many possible portfolios

− Number of possible portfolios grows exponentially as the number of projects grows merely linearly

− Computers in the foreseeable future cannot search for the optimal portfolio by brute force

Number of Projects

(binary choices)

Number of Possible

Portfolios

Search Time (10,000 runs for

0.1 sec/run)4 16 4.4 hours

10 1.0x103 12 days

33 8.6x109 2.7x105 years

75 3.8 x1022 1.2x1018 years

200 1.6x1060 5.1x1055 years

Chow -May 2013 #5

Approximations Used in Traditional Methods Are Problematic for Real-World Investment Optimization

• While traditional methods work well in a certainty world, they are ill-equipped to perform investment selection and optimization in a real-world, which is rife with uncertainties

− A popular traditional method: Expected Value with Sensitivity Analysis

• Pre-select about 10 possible portfolios; go through uncertain scenarios to find the most robust portfolio; and finally check it with sensitivity analysis and what-if analysis

• Since there are typically billions or trillions of possible portfolios, the chance that the optimal portfolio is among the chosen ten is practically nil

− Other traditional methods place unrealistic limitation on the numbers of decision variables, uncertain parameters and uncertain scenarios

Chow -May 2013 #6

RAND’s PortMan Has a New Approach to Perform Investment Optimization under Uncertainties

1. Simulate 10,000 future states of the world (FSW) by making random combination of the uncertain parametersa. Each

generated FSW is a certainty state

2. Use a mixed-integer linear programming model to automatically identify the local optimal solution or portfolio among trillions of possible solution for a given certainty state

3. Use multiple straightforward search rules, one at a time, to iteratively going through Steps 1 and 2 to arrive at the global optimal portfolio

Chow -May 2013 #7

One of the Search Rules Used1. Select the project that is most frequently chosen among the

10,000 local optimal solutions as the first project (P1) in the global optimal solutiona. This is like forming a team for a job that is full of uncertainties

as to what actual tasks will have to be performedb. The idea behind forming an optimal team is to start by picking

the most valuable player first

2. Generate another 10,000 FSWs but insisting that P1 is in the global optimal solution and select the most frequently chosen project among these 10,000 FSWs (each with P1) as the second project (P2) for the global optimal solutiona. The idea is to select the second team member to best

complement the first team member

3. Repeating Step 2 for additional projects until the budget for developing and/or implementing selected projects is fully committed

Chow -May 2013 #8

• Total remaining lifecycle budget ($35B) is the sum of the total remaining S&T budget ($2B) and the total implementation budget ($33B)

• If a given portfolio meets all requirements and constraints within budgets in 9100 out of 10,000 FSWs, the feasible percentage is 91%

Note: RLCC is the remaining lifecycle cost; S&T is science and technology; Feasible percentage is a proxy for confidence level

Source: Chow et al., Toward Affordable Systems II, MG-979-A, 2011

Optimal Investment Portfolio with PortMan Running in Uncertainty Mode

(based on 10,000 future-state-of-the-world runs for each data point)

Chow -May 2013 #9

Comparison of PortMan with Two Traditional Methods-I

• Benefit/Cost Ratio− For each project, divide the total contributions

to all requirements by the remaining S&T budget

− Optimal portfolio is composed of projects with the highest ratios until the total remaining S&T budget is fully committed

− This optimal portfolio is run to determine its feasible percentage under uncertainty (i.e. among 10,000 FSWs)

Chow -May 2013 #10

Comparison of PortMan with Two Traditional Methods-II

• Certainty Model− A mixed-integer programming model is used

with expected parameters as inputs, subject to a given total remaining R&D budget and a total remaining lifecycle budget (R&D plus implementation)

− The optimal portfolio for this certainty case is obtained with an objective function for maximizing the total project values (i.e. contributions)

− This optimal portfolio is run to determine its feasible percentage under uncertainty (i.e. among 10,000 future-state-of-the-world runs)

Chow -May 2013 #11

Note: ATO=Army Technology Objective, Army’s highest priority S&T projects

The selection cannot be derived from an analysis of benefit-cost ratios—a PortMan-type model is essential

The Best ATOs to Select Are Not Necessarily the Ones with High Benefit-Cost Ratios

Chow -May 2013 #12

At Equal Cost, PortMan Does No Worse and Often Does Better Than Traditional Methods under Uncertainty

At equal confidence level, PortMan does not cost more and often costs less than traditional methods

to meet all requirements under uncertainty

0.7 25 100% 100% 28%1.0 20 68% 56% 0%1.2 20 40% 0% 0%

1.4 7 100% 74% 0%1.3 6 95% 25% 0%1.4 6 93% 25% 0%1.6 6 86% 77% 0%1.8 6 68% 60% 61%1.9 6 63% 56% 63%

R&D=Research & DevelopmentTRRD=Total Remaining R&DTRLC=Total Remaining Lifecycle, which includes TRRDRTBF=Ready-To-Be-Fielded

With Approximate Consideration of RTBF Systems

With Full Consideration of RTBF Systems

Budget Feasible PercentageThis

ModelCertainty

ModelTRRD

$BillionTRLC

$BillionBenefit/Cost Ratio Model

Chow -May 2013 #13

Advantages of This New Approach1. This new approach allows DOT to incorporate real-world situations

(uncertainties) into its investment analysis, selection and optimizationa. It tells under what situations,

i. current benefit-cost based methods can continue to be used as good approximations

ii. current methods needed to be revised and how

2. Relative to highly mathematical search algorithms, the straightforward search rules used in this approacha. Are much easier to implementb. Are much easier to understand why such a rule can find the (global)

optimal portfolioc. Are much easier to see the flaw of a rule so as enabling the design of

complementary rule(s) to mitigate the flawPortMan has been used on past RAND projects and currently is being used for three ongoing projects sponsored by the Army, the National

Institute of Justice, and the Centers for Disease Control and Prevention. Perhaps, it would be of interest to DOT as well

Chow -May 2013 #14

Free Reports: Downloaded at Links below or Requested from chow@rand.org

• Chow, Brian, Portfolio Optimization by Means of Multiple Tandem Certainty-Uncertainty Searches: A Technical Description, RR270-A/OSD, 2013 (http://www.rand.org/pubs/research_reports/RR270.html)

• Chow, Brian, Richard Silberglitt, Caroline Reilly, Scott Hiromoto, and Christina Panis, Choosing Defense Project Portfolios: A New Tool for Making Optimal Choices in a World of Constraint and Uncertainty, RB-9678-A, 2012 (http://www.rand.org/pubs/research_briefs/RB9678.html)

• Chow, Brian G., Richard Silberglitt, Caroline Reilly, Scott Hiromoto, and Christina Panis, Toward Affordable Systems III: Portfolio Management for Army Engineering and Manufacturing Development Programs, MG-1187-A, 2012 (http://www.rand.org/pubs/monographs/MG1187.html)

• Chow, Brian, Richard Silberglitt, Scott Hiromoto, Caroline Reilly, and Christina Panis, Toward Affordable Systems II: Portfolio Management for Army Science and Technology Programs Under Uncertainties, MG-979-A, 2011 (http://www.rand.org/pubs/monographs/MG979.html)

• Chow, Brian, Richard Silberglitt, and Scott Hiromoto, Toward Affordable Systems: Portfolio Analysis and Management for Army Science and Technology Programs, MG-761-A, 2009 (http://www.rand.org/pubs/monographs/MG761.html)