Balancing Risk in the R&D Portfolio ReportOctober 2019

iriweb.org

AcknowledgementsIRI MembersJennifer Cahalan, Entegris, Inc.Sofia Elias, MetalsaNorm Golm, Regal Beloit CorporationChuck Hardy, MichelmanPerla Hernandez, MetalsaTom Kavassalis, Xerox CorporationSteve Moskowitz, Entegris, Inc.Natalia Navarrete, MetalsaJoel Schall, Henkel CorporationKent Young, Sherwin-WilliamsMarcie Zaharee, The MITRE Corporation

IRI Staff LiaisonLee Green

2

iriweb.org

ContentsIntroduction

Part 1: Survey Resultsi. Survey Demographicsii. Section 1: How does culture, training, and experience impact risk

assessment?iii. Section 2: How do organizations show risk in their R&D portfolios?iv. Section 3: How are risk assessments and other factors shaping

organizations’ portfolios?

Part 2: Curated Resourcesi. Section 1: Portfolio Management Approachesii. Section 2: Resource Managementiii. Section 3: Portfolio-Level Risk

3

iriweb.org

Introduction

4

This report summarizes results from an IRI survey exploring how organizations document risk and use it in assessing their R&D portfolios.

The topic of risk in the context of portfolio management is an evergreen one for practitioners of innovation and R&D. In assessments of key challenges facing its members, IRI consistently receives requests for further information on the topic. To add to the large body of knowledge already existing in this space, IRI formed a team of members to create a survey to better understand the current practices related to risk assessment in R&D portfolios. The survey of our members and other organizations was conducted in April – May 2019. This report presents the results of that survey and a further literature review conducted by the research team.

Part 1 of this report presents the results of the survey, noting key findings in three main sections:

• How does culture, training, and experience impact risk assessment?

• How do organizations show risk in their R&D portfolios?• How are risk assessments and other factors shaping

organizations’ portfolios?

Part 2 summarizes additional resources that the research team recommends for further study of the topic, organized by subtopics within risk and portfolio management.

• Portfolio management approaches• Resource management• Portfolio-level risk

iriweb.org

PART 1: SURVEY RESULTS

5

iriweb.org

Survey Demographics

Most of the survey respondents were from large organizations from the industrial and consumer products sectors.

6

The survey had 51 respondents, most of whom (84%) were from companies with revenue of over $1 billion in 2018. Smaller companies (2018 revenue of < $1billion) made up only 8% of respondents, with a small number of nonprofits and government agencies also responding.

The respondents represented organizations from 15 different sectors. The most well represented were: Chemicals, Gases, & Advanced Materials (25%), Consumer Products and Food, Tobacco, and Related Products (17%), and Industry, Machinery, Equipment & Products (15%).

Aerospace & Defense

Building Products

Chemicals, Gasses & Advanced Materials

Computer, Software &

Related Products

Consumer Products

Energy, Power Supply

Federal Lab, Government

Food, Tobacco & Related Products

Healthcare, Medical Products &

PharmaceuticalsIndustry Machinery,

Equipment & Products

Paper & Allied Products

Petroleum & Related Products

R&D Services

Textile, Apparel & Advanced Materials

University/Academia

Other

Respondents by Sector

< 1002%

101 - 1,0006%

1,001 - 5,00031%

5,001 - 10,00023%

10,001 - 50,00017%

> 50,00013%

N/A8%

Revenue in 2018 (in millions)

iriweb.org

Section 1: How does culture, training, and experience impact risk assessment?

7

iriweb.org

The basics: Are organizations assessing risk and how long have they been doing it?

92% of respondent organizations are assessing risk and the majority (57%) have been doing so for 1 –5 years.

8

While almost all organizations are assessing risk, most (68%) have only been doing so for five years or less. However, the other conclusions from the survey are similar between organizations that had been assessing risk for six or more years and those assessing risk for five years or less.

5

27

14

4

0 5 10 15 20 25 30

< 1 year

1 – 5 years

6+ years

N/A

Tenure assessing risk

Number of respondents

Yes92%

No8%

Organizations Assessing Risk in their Portfolios

iriweb.org

How does risk aversity affect risk assessment?

Risk averse companies are slightly less likely to document portfolio level risk than non risk averse companies (19% vs. 10%)

9

Of the respondents, 20% rated their organizations as risk averse (1 – 2 on a 5-point scale, 5 being least risk averse) and 51% rated their organization as not risk averse (4 – 5 on a 5-point scale).

Notable differences:• Individuals performing risk assessments in risk averse

companies are less likely to have been trained for the task (30% trained in risk averse vs. 46% trained in non risk averse).

• Risk averse companies are slightly less likely to document portfolio level risk than non risk averse companies (19% vs. 10%)

• Risk averse companies are significantly more likely than non risk averse companies to use a portfolio management software tool (30% vs. 4%).

• Risk averse companies are more likely to include emerging technology trends as a factor in balancing their portfolios from year to year.

4%

16%

29%

43%

8%

Risk Aversity on a scale of 1 to 5 (1 being very willing to take risks; 5 being very risk

averse) 1

2

3

4

5

iriweb.org

Are the individuals performing the risk assessment trained for the task?

Training (or lack of) did not have a significant impact on respondents’ answers to other survey questions.

10

Just over half of firms are using trained staff to perform risk assessment, but this had no significant impact on respondents’ answers to other survey questions.

No, 55%Yes, 45%

Are the Individuals Performing the Risk Assessment Trained for the Task?

iriweb.org

Section 2: How do organizations show risk in their R&D portfolios?

11

iriweb.org

How are organizations documenting portfolio level risk?

Most organizations (57%) use a scorecard to document risk in their portfolios.

12

Scorecards were by far the most used means (57%) of documenting risk at the portfolio level. Small percentages use software tools (10%) and databases (12%), but a surprising number (16%) don’t document risk at the portfolio level at all. Respondents who use scorecards have typically been assessing risk for longer (6+ years) and are using trained staff to do the risk assessments. With small variations, all of the different documentation methods still consider the same elements when assessing their portfolios.

2

3

3

6

8

29

0 5 10 15 20 25 30 35

Software Tool (Purchased)

Other

Software Tool-In House

Database

We don't

Scorecard

Number of Respondents

How Do You Document Portfolio Level Risk?

iriweb.org

How are organizations minimizing bias in their risk assessments?

Over 1/3 of firms do not take measures to minimize bias in their risk assessments.

13

Over one third of organizations (35%) have no formal methods in place for minimizing bias in their risk assessments. For those that do, assessments done by different groups/functions (27%) and a normalization process (16%) were the top two methods. Two respondents also added that their organizations use group discussions and collaborative assessments to minimize bias.

Anonymous Assessment

2%

Other5%

Weighting15%

Normalization Process

16%

Assessment Done By Different

Groups/Functions27%

We don't35%

How Do You Minimize Bias?

iriweb.org

Section 3: How are risk assessments and other factors shaping organizations’ portfolios?

14

iriweb.org

What sources of risk are organizations evaluating?

Technology readiness and schedule/timeline were the two most evaluated sources of risk.

15

Most respondents were evaluating:• Customers;• Competitors;• Schedule/timeline;• Manufacturing;• Regulatory risk; and• Technology readiness.

Fewer than half looked at Proposed business model (go to market strategy) (44%) and supply chain (48%). One respondent also added that their organization factors in safety to their staff and damage to test labs.

8

22

24

26

29

29

31

37

43

0 5 10 15 20 25 30 35 40 45 50

Other

Proposed Business Model

Supply Chain

Manufacturing

Competitors

Regulatory Risk

Customer

Schedule/Timeline

Technology Readiness

Number of Respondents

Sources of risk evaluated

iriweb.org

How are organizations using risk in assessing their portfolios?

Portfolio balance and uncovering strengths and weaknesses in the portfolio were the main reasons organizations use risk in their assessments.

16

Respondents were asked to rank the importance of a variety of uses for risk in portfolio assessment. Portfolio balance was clearly the primary reason respondents’ organizations were using risk in assessing their portfolios, with 38% ranking it as their most important factor. Uncovering strengths and weaknesses in the portfolio was the second highest factor with 21% of respondents ranking it as their first or second priority. Respondents were split over Identifying new threats and opportunities as a factor with 21% ranking it as their highest factor and the same number ranking it as their lowest factor. Informing capital allocations had low rankings across the board.

2.9

3.1

3.33

3.34

3.92

4.4

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Informing Capital Allocations

Informing Staffing Allocations

Comparing and Exploring Alternatives

Identifying New Threats and Opportunities

Uncovering Strengths & Weaknesses in the Portfolio

Portfolio Balance

RANKING - AGGREGATE SCORE

How Do You Use Risk in Assessing Your Portfolio?

iriweb.org

In addition to risk, what value drivers are organizations considering in their portfolio assessments? Strategic fit and profit

potential are the most common value drivers that organizations use in their portfolio assessments.

17

Most organizations consider the following value drivers in their portfolio assessments:• Market attractiveness;• Growth;• Profit potential;• Strategic fit;• Differentiation/IP protection;• Sales (new and defending existing).

Reputation (18%) and new customer capture (20%) were the least considered drivers.

34 3338

45

34

912

32

10

21

4

Num

ber o

f res

pons

es

iriweb.org

What factors, in addition to risk and value drivers, are organizations using to balance/re-balance their innovation/R&D portfolios from year to year?

Strategic alignment is the primary factor that organizations use when balancing/re-balancing their portfolio.

18

Respondents were asked to rank their top three factors and strategic alignment emerged as the clear winner with 92%. Market forecasts (52%) and direct customer interest (54%) were other key factors. Brand fit was of least importance.

0

5

10

15

20

25

30

35

40

45

50

Num

ber o

f res

pons

es

iriweb.org

PART 2: CURATED RESOURCES

19

iriweb.org

Section 1: Portfolio Management Approaches

20

iriweb.org

Tools for Managing Early-Stage Business Model Innovation (Article)

21

Abstract: Evans, J., and Johnson, R. offer a two-stage approach that combines a modified risk return framework with a new concept, Innovation Readiness Levels (IRLs). IRLs represent a broadening of the Technology Readiness Level concept, offering a way to assess an organization’s capabilities with regard to a particular business model. An IRL could be defined for each capability (such as implementing a training program for a specific job code), but at Lockheed Martin, they have chosen to group capabilities by functional area and then define an IRL for each functional area. For a given business model, there is a Human Resources Readiness Level (HRRL), a Finance Readiness Level (FRL), etc. He offers a comparison of TRL against IRL where the IRLs are intended to measure the degree to which the company or function has demonstrated the capabilities needed to move from its existing business model to the envisioned new business model.

Link to Article:https://www.iriweb.org/articles/rtm-tools-managing-early-stage-business-model-innovation

John D. Evans, Ray O. Johnson; Research Technology Management, Volume 56, Issue 5, September-October 2013

iriweb.org

Success Factors & Failure Modes in Portfolio Management (Brown Bag Webinar)

Richard Sonnenblick; IRI Brown Bag, October 5, 2012

22

Abstract: In this webinar, Richard Sonnenblick, CEO Enrich Consulting discusses what works, what doesn’t and why in the realm of project forecasting, portfolio management, and project selection for investment. He identifies three challenges to portfolio management -- projects that should be cancelled but don't; no linkage between project selection and portfolio strategy; and/or no balance between short term and long-term goals. To mitigate uncertainty, he suggests asking the right questions such as “What to do with limited resources?”, “How do we broaden the portfolio and maintain growth?”, and “Are NPV’s right, do they provide balanced growth?” He also provides tenants for selecting the proper mix of projects such as projects that provide value, projects that have strategic alignment, and a portfolio of projects with varying levels of risk.

Link to Webinar:https://www.iriweb.org/articles/brown-bag-success-factors-failure-modes-portfolio-management

iriweb.org

Strategic Portfolio Management (Brown Bag Webinar)Michel Delifer, Robert Cooper; IRI Brown Bag, April 21, 2017

23

Abstract: In this webinar, Dr Robert Cooper, Founder Cooper & Associates discusses the innovation diamond and the four vectors that drive successful innovation with emphasis on the Product Innovation & Technology Strategy vector. He identifies attributes of best performers, discusses portfolio management challenges, and describes a five-step process to define product innovation strategy.

Link to Webinar:https://www.iriweb.org/articles/brown-bag-strategic-portfolio-management

iriweb.org

70/20/10 and How to Get There (Brown Bag Webinar)

24

John Bacon, Bill Weber; IRI Brown Bag, June 6, 2016

Abstract: In this webinar John Bacon, Founder iP2Biz and Bill Weber GM iP2Biz discuss the importance of balancing innovation portfolios with 70% Core (optimizing existing products), 20% Adjacent (expending from existing business to new company business) and 10% Transformational (developing breakthrough products for markets that don’t exist) type projects, and offers methods to help innovation initiatives start producing returns.

Link to Webinar:https://www.iriweb.org/articles/brown-bag-702010-and-how-get-there

iriweb.org

Valuing and Comparing Small Portfolios (Article)Rick Mitchell, Francis Hunt and David Probert; Research Technology Management, Volume 53, Issue 2, March-April 2010

25

Abstract: Messrs Rick Mitchell, Francis Hunt, and David Probert discuss a portfolio scenario where the project uncertainties are mainly internal and thus can't be hedged by investing in unrelated projects. When a company has a large portfolio of projects, risk can be balanced by use of average and variance but with small portfolios this is of little value because there are too few projects to balance things out. An effective process for valuing and comparing small portfolios is to create a decision tree of possible outcomes at each gate and an associated cost and value for each option. Once complete, a Monte Carlo simulation can be used to create a confidence vs. value curve. Projects can then be ranked by the ratio between their “upside” value and “lowest” value. Once ranked, resources can be assigned to projects until there are no resources available.

Link to article: https://www.iriweb.org/articles/rtm-valuing-and-comparing-small-portfolios

iriweb.org

Benchmarking Best NPD Practices–I (Article – First in a 3-Part Series)

Robert G. Cooper, Scott J. Edgett and Elko J. Kleinschmidt; Research Technology Management, Volume 47, Issue 1, January-February 2004

26

Overview: This first of three articles reports the results of the most recent American Productivity and Quality Center study on performance and best practices in new product development. Performance results achieved on a number of NPD metrics are reported first, followed by a set of Best Performing businesses, what these businesses have in common and—most important what differentiates them from the rest. This first article focuses on the “people side” of NPD—on how project teams are organized and on other winning characteristics of teams. The climate and culture for innovation within the business proves to be one of the strongest drivers of NPD performance, and the specifics of what the Best Performers have done to achieve this positive climate are outlined. Finally the role of senior management in setting the stage for the business’s NPD effort is examined.

Link to article:https://www.iriweb.org/articles/rtm-benchmarking-best-npd-practices-i

iriweb.org

Benchmarking Best NPD Practices–II (Article –Second in a 3-Part Series)Robert G. Cooper, Scott J. Edgett and Elko J. Kleinschmidt; Research Technology Management, Volume 47, Issue 3, May-June 2004

27

Overview: Translation of strategy into new product initiatives is the focus of this second of three articles reporting the results of the most recent American Productivity and Quality Center study on performance and best practices in new product development. Having a new product strategy for your business is clearly linked to positive performance, and the article outlines what the elements or components of a best-in-class innovation strategy are, and their relative impacts. Strategy ultimately must be reflected in spending decisions; best performing businesses undertake a higher proportion of more innovative NPD projects, while the worst performers have a timid NPD project portfolio. There is also strong evidence that a formal portfolio management approach improves NPD performance overall. Finally, the issue of resources at the NPD team level is probed. Best performers were found to have far more resources available for NPD project teams, especially in nontechnicalareas, and to provide much sharper focus in theallocation of these resources.

Link to article: https://www.iriweb.org/articles/rtm-benchmarking-best-npd-practices-ii

iriweb.org

Benchmarking Best NPD Practices–III (Article – Third in a 3-Part Series)Robert G. Cooper, Scott J. Edgett and Elko J. Kleinschmidt; Research Technology Management, Vol 47, Issue 6, November-December, 2004

28

Overview: The new product process by which firms drive new product projects from inception through to commercialization, and the methods, practices and tactics embedded within that process, are the focus of this final of three articles reporting the results of the recent American Productivity and Quality Center study on performance and best practices in new product development. Many of the decisive activities that were identified turn out to be poorly executed, while a handful of tasks emerge as pivotal to NPD performance. Most firms now employ a systematic, formal, new product process, but the nature of the process and the way it is implemented are the true keys to success. For instance, delivering a differentiated, superior product is one practice that strongly separates the Best and Worst Performers. Market information, up-front homework, stable product definition, and voice-of-customer research are found to be relatively weak practices in businesses’ new product efforts, but all strongly discriminate between the Best and Worst Performers.

Link to article:https://www.iriweb.org/articles/rtm-benchmarking-best-npd-practices%E2%80%94iii

iriweb.org

Section 2: Resource Management

29

iriweb.org

Making R&D Portfolio Management More Effective (Article)

Michael M. Menke; Research-Technology Management, Volume 56, Issue 5, September-October 2013

30

Abstract: Michael Menke summarizes a benchmarking study of portfolio management practices. This summary lists the 50 best practices related to PPM and companies self-assessments of how they perform in each. Strong capability in resource management was identified as a common weakness across respondents. Additionally, the summary identifies a list of 11 practices which are seen as high importance, but which companies generally do not perform well. These could be areas of opportunity for many companies as they perform internal assessments of their own processes.

Link to article:https://www.iriweb.org/articles/rtm-making-rd-portfolio-management-more-effective

iriweb.org

Optimizing Project Portfolios (Article)Ross Seider; Research Technology Management, Volume 49, Issue 5, September-October 2006

31

Abstract: In this article, Ross Seider describes how the theory of constraints applies to managing a portfolio of projects. Not surprisingly, the pace of a project is affected by its most constrained resource. However, as that resource becomes more constrained, the other programs they are constrained across become increasingly impacted. In fact, research shows that when resource capacity is not considered in gate reviews, the phase & gate development process can actually add to the resource overload problem in an organization. The author describes an approach for managing a portfolio of projects via a regular cross-functional review process. The approach is simple, spreadsheet-based, and forces project prioritization decisions to take place.

Link to article:https://www.iriweb.org/articles/rtm-optimizing-project-portfolios

iriweb.org

Section 3: Portfolio-Level Risk

32

iriweb.org

Managing Uncertainty During R&D Projects (Article)Marc Wouters, Berend Roorda, Ruud Gal; Research Technology Management, Volume 54, Issue 2, March-April 2011

33

Abstract: Firms make significant investments in R&D projects, yet the economic return is often difficult to predict because of significant technological and commercial uncertainty. We present an innovative and practical method for managing R&D projects, and we discuss its application to a large R&D investment by Philips Lighting. The method, which we call the project portfolio option-value (PPO) method, provides an innovative way to represent, discuss, and value uncertainty in R&D projects. The PPO method is not about “perfect” or “complete” valuation models, but rather about providing a comprehensive but not-too-detailed view of major challenges and key criteria for success. The method is designed for a complex setting in which many uncertainties exist about technology and the market; in which the order in which uncertainties are resolved and decisions will be made cannot be specified in advance; in which interdependencies exist among R&D projects; and where transparency is vital.

Link to article:https://www.iriweb.org/articles/rtm-managing-uncertainty-during-rd-projects

iriweb.org

Reducing Risk Management’s Organizational Drag (Whitepaper)

CEB, 2014

34

Abstract: In the early days, risk management was traditionally dominated by financial and hazard risks—predicting or managing against a lack of liquidity or a catastrophic typhoon. In the present day, when those types of risks can be transferred through hedging and insurance, they have taken a backseat to strategic, operational, and reputational risks that assurance functions and business leaders must identify and manage themselves. These business risks, if not managed correctly, can dramatically affect an enterprise’s financial results, brand, and even ability to operate—having asevere negative impact on shareholders, customers, and employees.

Link to whitepaper:https://www.cebglobal.com/content/dam/cebglobal/us/EN/top-insights/executive-guidance/pdfs/eg2014q3-reducing-risk-managements-organizational-drag.pdf

To learn more, contact Lee Green.green@iriweb.org703-647-2597