merrick decision risk analysis white paper

March 2012

Decision & Risk AnalysisBy: Justin Mao-Jones

ConsultantThe Merrick Consultancy

Merrick & Company2450 S. Peoria Street Aurora, CO 80014-5475

Tel: 303-751-0741 Fax: 303-751-2581 www.merrick.com

TECHNICAL PAPER

Make Better R&D Decisions with Decision and Risk Analysis

Justin Mao-Jones The Merrick Consultancy i

TABLE OF CONTENTS

Section Page

introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 uncertainty, deciSion criteria, and Weighted averageS . . . . . . . . . . . . . . . . . . . . . . 2What doeS a deciSion Problem look like in d&ra? . . . . . . . . . . . . . . . . . . . . . . . . 3riSk analySiS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6comPuter SimulationS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6other uSeful toolS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Influence DIagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Value of InformatIon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 utIlIty functIons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

incorPorating d&ra into buSineSS and Strategic Planning . . . . . . . . . . . . . . . . . . . . 8Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8referenceS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9


Justin Mao-Jones The Merrick Consultancy 1

IntroductionDecision and risk analysis (D&RA) is a field of study incorporating concepts and philosophies that are designed to help organizations make better decisions. According to the Stanford Research Institute, D&RA is a discipline that seeks to apply logical, mathematical, and scientific procedures to the decision problems of top management1. At Merrick, D&RA has five basic steps:1. Frame the decision problem2. Model the frame3. Quantify the risks and uncertainties4. Analyze the model5. Make decisions

Figure 1 - D&RA is a cost-effective way to help decision-makers identify which pieces of information are most important to project success, thus facilitating decision efficiency.

In large part, D&RA is about understanding the uncertainties inherent in the decision problem by applying probabilities to a range of possibilities for each potentially significant uncertainty. With the help of mathematical and graphical tools, the practitioner can bring simplicity to complicated scenarios. As a project progresses, valuable information comes to light and uncertainty is reduced. D&RA identifies those uncertainties that are most important to project success, allowing decision-makers to optimize information gathering activities. This increases the rate at which valuable information is accrued. Having more valuable information earlier on in the project facilitates better project definition and avoids costly design changes down the road.

Figure 2 Value of D&RA over the project timeline. With better information earlier on, decision-makers can make good decisions sooner. This promotes stronger project definition and avoids costly design changes down the road. As the project progresses, it becomes more

and more difficult to influence positive change to project value, and thus D&RA can be an important tool in front-end project development.



D&RA has been highly successful in practical applications across industries and government. It has become an industry standard in oil and gas exploration and pharmaceutical R&D. A Society of Petroleum Engineers benchmarking effort found that petroleum companies introducing D&RA greatly improved their industry ranking in just a few years2. According to the Society of Decision Professionals, a review of hundreds of projects across industries suggests that decision consulting often yields value from 100 up to 1000 to 13. In fact, a 1993 decision consulting effort at Smith-Kline Beecham returned an estimated 1300 to 1 value to cost (in-house analyst costs and consulting fees) ratio3,4. Over the years, Merrick has helped several clients improve project value by up to 20%, shift R&D resources to higher potential initiatives, identify major risks (often hidden beneath the surface), cut project development time by several months, and save millions in conceptual engineering costs. Needless to say, D&RA has high value potential.

Uncertainty, Decision Criteria, and Weighted AveragesEvery D&RA should follow a few simple rules. They are as follows:

Rule #1: Uncertainty should be modeled with probability distributions (a range of possibilities combined with probabilities assigned to each of those possibilities). In general, the simpler the probability distribution the better, but sometimes more complicated distributions are appropriate, such as a curve fitted to an empirical data set. The goal of defining an uncertain variable is to capture a reasonable representation of its possible values. It is not to portray a perfect depiction of the uncertain variable. Generally speaking, such thinking is inherently flawed, because in the future uncertain variables will ultimately have a single point value. For example, we may put an uncertainty range on the capital cost of a project, but the actual money spent will be a fixed dollar value.A common method for defining uncertain variables is to use the P10/50/90 framework. It is conveniently simple when eliciting information from subject matter experts. Mathematically speaking, the P10, P50, and P90 represent the 10th, 50th, and 90th percentiles of the ranges, respectively. Figure 3 shows an example of how the P10/50/90 framework is used to model uncertainty. Conceptually, the P10 is lowest value that the expert thinks that the uncertain variable could be. The P50 is the most likely value. The P90 is the highest value that the expert thinks the variable could be. Room is left below the P10 and above the P90 for highly unlikely scenarios, such as Acts of God.

Figure 3 An example of how a P10/50/90 range is used to model uncertainty.



Rule #2: To decide among viable options, the decision-maker must have a single, meaningful metric (i.e. a decision criterion). A meaningful metric is not always obvious. Many business decision-makers commonly use financial metrics such as net present value and internal rate of return. In complicated decision problems, return on investment metrics may be insufficient to handle the nuances of the problem frame. Important criteria may be difficult to quantify and incorporate into an overall decision criterion, such as weighting safety and functionality considerations against each other. Methods such as multi-attribute utility theory are used in these instances to derive a utility function that serves as the decision criterion. Deriving the right decision criterion in the framing session is one of the most important exercises in the D&RA process.Rule #3: The decision-maker should choose the option with the best expected decision criterion (i.e. the high-est or lowest weighted average decision criterion). For example, a frame using net present value (NPV) as the decision criterion would choose an alternative with the highest expected NPV. Conversely, a frame may choose the alternative that maximizes the probability of an acceptable NPV. As an extension of this rule, when model-ing future decisions the practitioner should assume that the decision-makers will be making good decisions, unless special circumstances of the frame logically dictate otherwise.

It is important to note that a good decision can lead to a bad outcome, an unlucky scenario. Conversely, a bad decision can result in a good outcome, a lucky scenario. Good decisions serve to maximize the likelihood of good outcomes. Bad outcomes do not imply bad decision-making, unless they reveal poor framing, faulty logic, or inappropriate use of resources1. With the right frame, appropriate use of available information, and a model that adequately incorporates the nuances of the decision problem, a decision-maker should expect that he or she is making the best decision possible.

What Does a Decision Problem Look Like in D&RA?At D&RA conferences, one will often hear practitioners extol the virtues of getting the frame right. Thats because after years of experience, practitioners have discovered that the most important and often overlooked part of the process is the frame. With the right frame, you will analyze the right issues, and come to the right decisions. So it goes, with the wrong frame you will get the wrong answer.

The frame should encompass the following: Objectives: what decision(s) do we need to make? Major issues: what do we need to worry about, such as major uncertainties? Scenarios: what could our long-term environment look like? Alternatives: what are the distinctly different options at each decisional node? Decision criterion: how will we measure the effectiveness of decisions?

In R&D projects, decision-makers must decide which ideas are worth pursuing and how to pursue them. Merrick recently worked with a biofuels client that was deciding between three reactor concepts, four ways of utilizing each reactor, four energy integration schemes, and three distinctly different product profiles, for a total of 144 alternatives. The objective was to choose the alternatives that had the highest potential for long-term success and to provide clarity on project risks and opportunities. Major issues included lack of development (high uncertainty) in particular areas. Capital costs were a major concern and development of the reactors was in a state of high flux. The decision criterion was the maximum affordable capital cost that would produce a specified internal rate of return.

The example above focused on technology selection. A different kind of frame could focus on decisions that lead up to technology selection. R&D focuses on preliminary studies, research campaigns, bench and pilot scale testing, etc., to find good ideas, flush out bad ones, and develop commercially viable technology. In other words, a frame could optimize resource allocation.

Many R&D frames have the following general issues:1. Need for long-term growth and success 2. Numerous alternatives3. Limited funding4. Limited time frames to produce results5. High uncertainty



To illustrate the application of D&RA to an R&D decision problem, consider the following example of an R&D decision at a hypothetical biofuels company. Suppose that the company has devised a methodology for improving yield values of algae strains for the production of algae oil, which can be converted to biodiesel. The company has spent the last two years developing a particular strain, Strain A, with moderate success. It has also identified a new, though relatively unknown, Strain B that has the potential for even higher yields. The team feels confident that, because of the strain development knowledge and skills acquired over the past two years, it could develop Strain B at a greatly accelerated pace.

The manager, however, is not convinced that a mid-project audible is a good idea. If the company cannot show results within the next two years, future fundraising will become very difficult. Work on Strain B would mean partially or completely diverting resources from work on Strain A. The team is already on track to produce moderate, but acceptable results with Strain A. And there is always the possibility that Strain B may not meet the high expectations or may even turn out to be a dud.

The manager could decide to table Strain B for now and continue to devote all resources towards Strain A. He is confident that his team can prove moderate yields within one to two years. Alternatively, he could decide to divert his entire team towards Strain B. Strain B could prove to have moderate to high yields, but may not be able to be grown beyond low concentrations. This would necessitate much higher capital and operating costs in a commercial facility to produce the same amount of algae as Strain A and thus be a major disadvantage. It would take about 6 months to determine whether or not Strain B can be grown at high enough concentrations. If it turns out that it can, he estimates that it will take about one to two years to prove the yields for Strain B. If it turns out otherwise, he can divert his team back towards Strain A development.

Figure 4 Decision tree of R&D direction based on time to complete research.

Within these two scenarios are three main variables: time to complete research, Strain B productivity, and Strain B concentration constraint. One way to frame this problem is to look at it from the standpoint of whether or not sufficient yields will be proven in time to show the investors. In this frame, the manager would choose the alternative that maximizes the chance of research completion within two years. Figure 4 illustrates this frame with a decision tree. Focusing on Strain A yields 100% certainty that the research will be completed on time. Focusing on Strain B, however, yields an 84% probability that research will be completed on time.

If the team determines that Strain B can grow in high concentrations, there is 100% probability that the yields can be proven within the two year time frame, assuming that Strain B development is concurrent with the six month study. If not, the team will have lost six months to work on Strain A, leaving only 1.5 years to finish the work. The team estimates that there is an equal chance that the work will take 1, 1.5, or 2 years to complete. This means that there is only a 67% chance that they will be done on time. If the team assumes ahead of time that there is a 50% chance that Strain B can grow in high concentrations, then the probability of completing the work on time is 84%.



This analysis clearly shows that continuing work on Strain A is the best path forward, because it ensures that research on Strain A will be complete within two years. However, with a longer-term frame, this answer could easily change.

Consider that the manager must also take into account the long-term future of the company. He decides to measure this by looking at the investment potential, i.e. projected NPV, of a commercial facility utilizing either strain. In this frame, Strain B gains an advantage in that higher yields would mean a higher NPV.

Figure 5 Decision tree on research direction based on NPV of future commercial facility.

Figure 5 illustrates this new frame with a decision tree. Based on financial projection analysis, the team concludes that a commercial facility using Strain A would be worth $100 million in NPV. Using Strain B, however, NPV could range from $120 up to $150 million. The team estimates a 50% chance that Strain B yields will be high versus moderate, though always better than Strain As. If growth in high concentrations is possible, then the expected NPV of a Strain B commercial facility is $135 million. If not, then the company must look at the investment potential of Strain A, given that it only has 1.5 years left, having already wasted 6 months. Under the scenario that the team does not finish the research in time, the yields of Strain A will be lower than target yields. The team estimates that with these lower yields, the commercial facility will only be worth $80 million in NPV. Thus, the expected NPV of the commercial facility will then be $93 million.

Assuming again a 50% chance that Strain B can grow in high concentrations, the expected NPV of the commercial facility by following Strain B research path is $114 million. Therefore, the manager should decide to divert resources to Strain B to achieve an expected increase in NPV of $14 million.

Keep in mind that this frame chooses the alternative that will yield the highest expected NPV after the two years are up. The question it answers is: Given the amount of time I have available, what research pathway will yield the highest potential for long-term success? In essence, it is a resource allocation problem.

This example is, of course, oversimplified. In many cases, deadlines take a backseat to funding constraints. The frame could have read Given the amount of funding I have available, what research pathway will yield the highest potential for long-term success? In real-life problems, there will likely be other decisional nodes and tens of uncertainties. Furthermore, there may be other alternatives. For example, the manager could have evaluated the option of diverting only 50% of his team to Strain B.

As the number of alternatives, decisional nodes, and uncertainties increase, the complexity of the problem grows exponentially. For example, assume that the manager was also deciding between three algae oil recovery technologies. Also assume that there are 10 uncertainties, each with three outcomes. With two research pathways for algae strain development, three recovery technology options, and thirty points of uncertainty, there are 180 scenarios to consider. Toss in a few more research studies and uncertainties and that number rises above a thousand!



Risk AnalysisRisk evaluation is a natural part of the decision problem. Uncertainty is essentially a lack of information and risk is the potential impact of making decisions without that information. Risk analysis investigates this relationship and is often illustrated using a tornado diagram, as illustrated in Figure 6. The tornado diagram depicts uncertainties ranked by potential impact to the decision criteria. The larger the range of impact to decision criteria, the larger the combined risk and potential upside. The uncertainties at the top of the tornado have the highest potential impact. Each bar of the tornado is generated by ranging the corresponding input variable across its uncertainty range while keeping all other variables at their 50th percentile values (the P50s). Typically, inputs are ranged from their 10th percentile value (the P10) to their 90th percentile value (the P90).

Figure 6 Tornado diagram of example biofuels project. Each bar is generated by swinging its corresponding uncertainty to its P10 and P90 values and calculating NPV while holding all other uncertainties at their P50 values. Uncertainties are ranked from top to bottom in

order of highest impact to NPV (i.e. largest dark green bar).

The point of the diagram is to very clearly show how important each uncertainty is relative to the others. It generates a discussion on which variables warrant special attention such as risk management, an exercise in reducing uncertainty.

Sometimes, reducing uncertainty can affect decisions. In Figure 6, for example, feedstock composition has a very high impact to NPV. This suggests that the decision-makers should give feedstock selection high importance and possibly spend money to investigate potential feedstock options. Indeed, the tornado diagram can lead practitioners and decision-makers to explore new directions when uncertainties unexpectedly rise to the top.

Just because an item does not appear on the tornado diagram or is ranked low does not mean that it is not important. Instead, it means that the uncertainty is not large enough to make a difference. For example, efficient operation of plant steam boilers may be important, but such a system is well understood across industries. The efficiency will likely fall within a small range of values with minimal relative impact to overall project economics.

Computer SimulationsComputer simulation helps facilitates D&RA. These simulations are designed to process decision alternatives and uncertainties, which are modeled with probability distributions. One way to do this is with decision trees, as shown previously in the algae development example.



Decision trees can be an essential tool in analysis and make presentation of analysis to the decision-makers much easier for the practitioner. The tree is a hierarchy of decisions and uncertainties. Each node represents a particular decision (typically represented by a square) or uncertainty (typically represented by a circle). Each branch represents a possible decision, when stemming from a decision node, or a possible outcome, when stemming from an uncertainty node. Figure 4 and Figure 5 are simple examples of a decision tree.

Monte Carlo analysis is another method for determining expected values of decision pathways and analyzing risks. It is especially useful when dealing with highly complex problems. The analysis is performed through repeated random sampling of all independent uncertain variables (inputs) in order to obtain a distribution of values for each dependent variable of significance (outputs). In each Monte Carlo iteration, each input is assigned a deterministic value randomly chosen from its probability distribution and the deterministic value of each output is calculated. Multiple iterations yield a distribution of values for each output. As more and more iterations are performed, each outputs distribution represents a closer and closer approximation of the outputs actual distribution. With the Monte Carlo distributions, one can derive expected decision criteria values for alternatives as well as uncertainty metrics such as P10/50/90 values. The uncertainty metrics are helpful for understanding the overall uncertainty in each alternative. Commercially available software such as @Risk and Crystal Ball are commonly used Monte Carlo simulators.

Other Useful Tools

Influence DiagramsInfluence diagrams are graphical depictions of how components in the decision problem affect each other. The diagram is also useful as a conceptual basis for creating a mathematical model of the frame. In Figure 7, which is a simplified version of an influence diagram, feedstock throughput is a function of feedstock composition and feedstock availability. All arrows eventually lead to the decision criteria. It is important that the computer model captures these relationships so that the D&RA results are based on a reasonable representation of reality.

Figure 7 Simplified influence diagram

Value of InformationUncertainty exists when we lack information about the future. Consequently, information has value, because it can change the future that we choose. The value of information (VOI) is calculated as the change in expected value when new information is presented. This exercise is typically performed before investing in new



information. Sometimes, information has no value, because it does not change the decision. Knowing this, however, can be valuable by preventing unwarranted uses of resources.

Utility FunctionsD&RA can be used to quantify anything, including soft or emotional issues. For example, a manufacturing company might be choosing among a number of different sites for a new facility, but must wrestle with proximity to highway and rail, workforce availability, traffic safety, noise ordinances, site buffering, land use impacts, etc. In this case, the practitioner can derive scales to quantify and rank each siting attribute and weight each attribute under an overall scale accordingly.

There are established methods for deriving utility scales such as SMART and Multi-Attribute Utility Theory5. Different methods apply under different circumstances. Most are based on behavioral research and scientific data.

Incorporating D&RA into Business and Strategic PlanningMany industry best practices for project development such as Front End Planning advocate that D&RA be applied during conceptual phase planning, because it is a successful tool for evaluating various decision scenarios. The conceptual phase succeeds the feasibility stage, in which rough numbers are estimated and then applied to generate options and alternatives, and precedes the detailed scope phase, in which key design documents and detailed project plans are developed. Important decisions are made during the conceptual phase and can have significant and lasting impact to the success of a business or project, and thus this phase is ideal for D&RA.

Clearly, D&RA should not be used to make every decision. According to the Stanford Research Institute1, D&RA is most appropriately applied to problems that have several of the following characteristics:

Uniqueness: the problem is one of a kind. Importance: a significant portion of the organizations resources is involved. Uncertainty: many key factors are known only imperfectly. Long run implications: the organization will be affected by the results of the decision for many years. Complex preferences: the desires of the decision-maker are not clearly formulated.

Strategic planning activities, which come before business planning, also have valuable uses for D&RA. A typical strategy management process addresses five main questions: 1) where are we today?; 2) what does the future look like?; 3) where do we want to be down the road?; 4) how do we get there?; 5) how do we measure our progress? D&RA comes into play in identifying how to close the gap between the desired future state and the current state of things. This exercise uses simpler models and is much less-involved in strategic planning than in business planning activities, because strategic planning is a higher level exercise.

SummaryD&RA is a decision-making tool. It is a construct for defining and understanding decision problems and is ideally suited to problems of high complexity, high importance, and high uncertainty. At Merrick, we often find that the D&RA journey is as important as the result. Every part of the process provides the D&RA team with a much better understanding of the problems facing decision-makers. With D&RA, decision-makers can expect to make better decisions and manage those uncertainties that are most impactful to project success.



References1. Matheson, J. E., and Howard, R. A. An Introduction to Decision Analysis, Stanford Research Institute, Palo

Alto, CA, 1968. Reprinted in Howard and Matheson (Eds.), The Principles and Applications of Decisions Analysis, SDG, Inc., Menlo Park, CA (1983), 17-55.

2. Jonkman, R. M., Bos, C. F. M., Breunese, J. N., Morgan, D. T. K., Spencer, J. A., and Snden, E. Best Practices and Methods in Hydrocarbon Resource Estimation, Production and Emissions Forecasting, Uncertainty Evaluation and Decision Making, Paper SPE 65144 presented at the 2000 SPE European Petroleum Conference, Paris, France, October 24-25.

3. Menke, M. M., Spetzler, C., Keelin, T. The Value of DA/DQ: Building a Compelling Case for Decision Makers, Presentation through Society of Decision Professionals Learning Exchange, February 2, 2011.

4. Sharpe, P., Keelin, T. How SmithKline Beecham Makes Better Resource-Allocation Decisions, Harvard Business Review March-April 1998. Reprint 98210.

5. Goodwin, P., Wright, G. Decision Analysis for Management Judgment, 4th Edition, West Sussex, United Kingdom: John Wiley & Sons Ltd., 2009.

merrick decision risk analysis white paper

Documents

decision criteria

decision efficiency

decision problems

decision consulting

decision problem2

value of dra

decision risk analysisby

project value