evaluating radical innovation portfolios · in current businesses. for example, airbus (eads)...

EVALUATING RADICALINNOVATION PORTFOLIOS

Herersquos a process for removing some of the uncertainty surroundingbreakthrough innovation projects in large companies

Albert S Paulson Gina Colarelli OrsquoConnor and Daniel Robeson

OVERVIEW Radical innovation in large firms is arisky endeavor exacerbated by high levels of uncertaintyand long cycle times Moreover management lacksevaluation tools that are appropriate for the degree ofambiguity of the information available from these typesof projects Add to this the difficulties of managing in thecontext of the large firm where many processes are tunedtoward repetition and continuous improvement and itbecomes clear that commercialization of radical innova-tion (RI) is a difficult proposition at best Portfoliomanagers will almost invariably build a team learnappropriate processes clarify their mandate within thecompany generate initial projects and begin to enrichtheir portfolios only to have their efforts cancelled outdue to an unsatisfactory financial return The ldquowhathave you done for me latelyrdquo question plagues thesegroups and ends up costing the company investment ofresources in opportunities that are never leveragedHaving an evaluation tool to assess the relative values ofprojects within the context of the entire radical innova-tion portfolio and to track changes in the relative value

of the portfolio over time can help RI portfolio managersarticulate their contribution to the companyrsquos growthstrategy and can aid in innovation strategy development

KEY CONCEPTS radical innovation portfolio evalu-ation real options

It is truly amazing to observe the growth in managementsophistication regarding breakthrough or radical innova-tion (RI) over the past ten years Firms that once experi-enced RI only by happenstance through theperseverance of strong champions protected from orga-nizational rules by their patrons in senior managementhave now developed management systems for encourag-ing RI These groups oversee portfolios of potentialradical innovations and are increasingly faced withportfolio-level issues of diversification pacing balanceand valuation What was once considered the exceptionhas become the new frontier of management attentionIndeed empirical research has established that largefirms that successfully engage in radical innovation com-

Albert Paulson is the Frank and Lillian Gilbreth ChairedProfessor in the Technologies of Management at Rens-selaer Polytechnic Institute Troy New York Histeaching and research interests center on innovationrisk management financial engineering large-scalemodeling investment and statistics He has publishedover 100 articles monographs and books and hasconsulted nationally and internationally for corpora-tions and governments He received his PhD and MSin statistics from Virginia Polytechnic Institute and BAfrom Kings College paulsonascapitalnet

Gina Colarelli OrsquoConnor is associate professor andacademic director of the Radical Innovation ResearchProgram and associate director of the Severino Centerfor Technological Entrepreneurship at Rensselaer Poly-technic Institutersquos Lally School of Management andTechnology Previously she worked for McDonnellDouglas and Monsanto Chemical Her teaching and

research focus on how large established firms linkadvanced technology development to market opportuni-ties and how they create new markets She has publishedmore than 25 articles in refereed journals and isco-author of Radical Innovation How Mature FirmsCan Outsmart Upstarts (Harvard Business School Press2000) She received her PhD in marketing andcorporate strategy from New York Universityoconngrpiedu

Daniel Robeson is a doctoral candidate in managementat Rensselaer Polytechnic Institutersquos Lally School Hehas had extensive business experience at MorganStanley Dean Witter American National Can andEastman Kodak His research interests focus on radicalinnovation at the large established firm He haspublished in the Journal of Engineering and TechnologyManagement He holds an MBA in management andtechnology from RPI dcrobesonaolcom

SeptembermdashOctober 2007 170895-630807$500 copy 2007 Industrial Research Institute Inc

mercialization efforts are more successful financiallythan those that do not (1ndash3)

This article presents the principal results of IndustrialResearch Institute-sponsored empirical research con-cerning evaluation of a portfolio of potential break-through innovations The research covers 12 years inthree phases at more than 30 large American companiesthat have proclaimed their intention to achieve repeat-able breakthrough innovation as a corporate capabilityFurther details concerning the study and participatingcompanies are provided in a later section See also (4) forresults concerning organizational structures

Radical innovation hubs or groups responsible formanaging these portfolios continue to be subject tointense scrutiny especially in times of limited capacityor senior leadership changeover A companyrsquos capacityto invest in potential breakthrough innovations can belimited in many ways Potentially ruinous litigation pro-ceedings extraordinary cost overruns on another innova-tion collapse of a principal market due to massiveovercapacity resulting in potentially fatal financial diffi-culties rapid acceptance of a new technological productwhich results in the dramatic decline of an incumbentproduct again resulting in severe financial difficulties orthe strategic decision to sell a profitable business unit ayear or so into the future are just a small number of eventsthat can limit or even eliminate capacity

Historically the average life expectancy of a radicalinnovation hub has been about four years (5) althoughour research leads us to believe that average life spans arelengthening especially for those companies goingthrough their second or third version of a hub Part of theproblem is the difficulty that RI hub managers face injustifying their RI portfolio because its economic value isso hard to quantify This occurs because of high uncer-tainty on multiple dimensions associated with RIprojects high failure rates at the project level that lead tohigh churn rates in the early stages at the portfolio levellengthy project maturation times and the fact that valueto the firm from any single project may accrue in indirectways (exposure to new market partners new learningnew technology frontiers) not traceable to any specificrevenue path

Our research on the management structures andprocesses associated with radical innovation has beenongoing in the Research-on-Research (ROR) committeeof the IRI (Research-Technology Managementrsquospublisher) since 1995 This research has highlighted thatRI hub managers need a mechanism to communicate thevalue of their work to the organization on a regular basisThey need a tool that helps articulate the value of the RIportfolio at any point in time as well as the change in itsvalue over time Traditional financial approaches cannothandle the extreme uncertainties inherent in the radicalinnovation management process and while real option

approaches are inherently better they must be adjusted inways that square with managersrsquo experiences See forexample the approach espoused by van Putten andMacMillan (6) MacMillan et al (7) and the commentsand criticisms of Paulson and Zhong (8)

Consequently an ROR sub-committee has taken as oneof its objectives the development of an RI portfolioevaluation tool whose fundamental assumptions devel-opment and results are described here The tool ispresented on pp 25ndash28 and an online version is availableat httpwwwiriincorg

Evaluating RI Portfolios

Evaluation of economic value of breakthrough innova-tion portfolios through future time horizons is at bestfiendishly difficult and at worst impossible We took athree-pronged approach to developing an instrument thatwould capture the majority of the information that mustbe obtained in order to sensibly evaluate a set of potentialbreakthrough innovations The end result was a series ofquestions constituting an evaluation instrument or toolThe answers to the questions may require much researchor much work on many fronts but it is worth it thequestions and answers address the valuation issues fromembryonic efforts through full commercialization andinclude firm-specific issues as well as issues beyond thefirmrsquos control

First we reviewed the literature to determine whatresearch must be conducted by the firm making thepotential innovation investment to come to a sensible anddefendable evaluation Next we sifted through all of thedata generated by our 12-year longitudinal researchstudy to determine what our research companies foundwould impute value how why and when valueschanged and what they found to determine risk andhow why and when risk changed

Finally we needed to find a couple of paradigms fromquantitative finance that might provide theoreticallymotivated guidance on valuing innovation investmentsthrough future time horizons We settled on the portfolio

Large firms thatsuccessfully

commercializeradical innovations

do better financially

Research Technology Management18

selection (PS) and the real options (RO) paradigms Eventhough we always knew these paradigms could onlyrarely produce sensible valuations in the highly uncertainenvironment of breakthrough innovation they nonethe-less could be dissected to suggest insights into key infor-mation that managers and executives need in order toassess the value of a portfolio of potential innovationsThese insights were used to partially guide and comple-ment our empirical research concerning our studycompaniesrsquo programs to build continuing and enduringbreakthrough innovation capabilities The makeup of ourevaluation instrument is governed by the combinedinputs of these three prongs

Portfolio selection is concerned with choosing fromamong multiple potential innovation investments Thereal options methodology is concerned with thevaluation of a single such investment Portfolio evalua-tion is closely related to portfolio selection Recent dis-cussion of the pros and cons of portfolio selectionparadigms from the financial markets point of view arereadily available for example in (9) Applications of realoptions in an innovation framework are discussed in (7)

Tool Makeup Implied by PS and RO Paradigms

The traditional portfolio selection paradigm of quantita-tive finance involves selecting from among a group ofprospective investments the amount to be invested ineach in such a way that the overall expected return oninvestment within some future time horizon ismaximized subject to the constraint of not exceeding apre-specified level of expected risk The total amount tobe invested is budgeted in advance The word expected inthis setting refers to forecast anticipated or estimatedquantities Return on investment is measured in one ofthe usual ways (profits revenues etc) Risk is measuredin terms of standard deviation or volatility of theexpected returns on the portfolio (See ref 10 for a quan-titative strategic accounting point of view concerningvaluation)

Without providing detailed rationalization we nowsummarize the key management information a usefulportfolio evaluation tool must incorporate in order toconceptually satisfy the requirements of the paradigmsof quantitative finance

1 The tool must address interrelationships in risk amongpotential innovation investments and among currentbusinesses Breakthrough innovation investments maybe inherently risky on their own account but may alsodrastically reduce risk in other potential investments orin current businesses For example Airbus (EADS)assumed a giant risk in deciding to build the giantAndash380 passenger aircraft Boeing declined to competeHowever the slippage in deliveries of the passengerAndash380 resulted in the demise of the freight version and

the loss of profits may have fatally impacted the smallerAndash350 (11)

2 The tool must address interrelationships in return ThePS and RO paradigms do not address this facet properlyA considerable number of the companies in ourempirical research indicated that the very participation inradical innovation projects imparts value to other inno-vation investments to existing products and servicesand to current and future customers in amounts thatexceed the outlay of the particular RI thrust This ldquoupliftrdquois often very difficult to quantify on an accounting basis(10) but when it is quantified it alone often more thanjustifies the investment

3 The tool must address all the major foreseeablesources of risk Both PS and RO call for a one-dimensional measure of risk This means that a risk indexmust be constructed which is not always sensiblemdashhowdoes one capture in a single number the operationalpolitical currency technological financial and macrorisks all of which our research companies face in mostpotential RI investments (11)

4 The tool must ensure that increased risk and uncer-tainty penalize the investment decision as in PS but not inRO The managers in our research study were unanimouson this point see also (7)

5 The PS and RO paradigms are not sufficiently flexibleto address the issues our breakthrough innovationportfolio managers face The tool must accommodatelearning and discovery in a flexible way and mustdevelop within-company and outside-of-company infor-mation But the tool must also limit the number of issuesto be addressed and focus on what we have determinedfrom our companies to be the most important issuesbecause the number of valuation forces can be very largeindeed

Flexibility is required because the breakthrough innova-tion environment is highly dynamic In uncertain andrisky environments the objectives might possibly be tolearn and discover as opposed to explicit evaluation(Many of the companies in our research program takethis tack because their long-term strategic objective is tobe in a particular business for the long term)

A useful evaluationtool must address the

interrelationshipsin risk

SeptembermdashOctober 2007 19

6 The PS and RO paradigms reduce valuation to a singledimension The executives in our research study continu-ally emphasized that many dimensions of risk value andconstraints drove their evaluations and managementactions The tool must allow management to obtain infor-mation on most major dimensions so they can beaddressed appropriately and simultaneously Becausethe breakthrough innovation environment is so fluid thetool must accommodate periodic updates The updatescan also be determined by learning and discovery efforts

7 The tool must allow for relative valuation of break-through portfolios While valuation is always importantfor portfolios of potential investments honest discussionregarding the contribution of the RI portfolio to overallstrategic growth and renewal of the firm will often bemore important PS and RO both imply the ability toobtain precisemdashand to some degree absolutemdashvaluations RO more so than PS (12)

Prospective estimated value can be assessed in one oftwo ways absolute or relative The ability to obtain anabsolute valuation is rare It can only be obtained wherethere is little uncertainty and low risk In the financialmarkets this essentially describes the valuation of a USTreasury bill a very short-term instrument In this case adiscounted cash flow analysis provides the Treasurybillrsquos value Long-term bonds including Treasury bondsare subject to high levels of risk and uncertainty

Relative valuations are the norm in a world of risk anduncertainty Even in the highly liquid and closely studiedequity markets the value of a company is often estimatedfrom its interrelationships with other companies that aresimilar to it in a number of respects and from thesecompaniesrsquo interrelationships with other companies thatare not so similar etc (The same valuation methodologyis used for the bond markets) The fluctuations (volatili-ties) in equity valuations can be breathtakingmdash25 50100 percent of the current equity price in the market andmuch more depending on issues of risk and uncertaintyencompassing the equity And option valuations havetheir own measure of fluctuationsmdashthat is volatilityimplied by investorsrsquo expectations of future prices for theequitymdashwhich can be well in excess of 100 percent

Relative to the evaluation of a portfolio of potentialbreakthrough innovations over future time horizons thesituation is much worse than the corresponding valuationsituation for equities in a market Accordingly the tooloffered here provides a list of issues which we havefound to be essential in divining valuation The toolrequires inputs of judgments which can then beprocessed to obtain an index relative to a potential valueThis index can be compared to 1) 100 the potentialvalue 2) previous values obtained for the portfolio and3) the companyrsquos objectives Our company participantshave found that in the pursuit of breakthrough innova-

tion dollar values can be so misleading that they shouldgenerally be disregarded

Portfolio Evaluation

Our principal objective is to provide help in articulatingthe value of a portfolio of highly risky highly uncertainlong-term investments at a point in time under thecondition that the evaluation will continue to take placeperiodically As time passes uncertainties relative tospecific projects are resolved others emerge and internaland external events constantly modify the viability ofinvestments in prospective innovations In dynamicenvironments decisions need to be re-examined in lightof new information or new events Some investmentsshould be carried forward as previously anticipatedothers increased others killed and still others put intohibernation The information required to make suchdecisions may range from the emergence of a single factor event to a long and complex sequence of learning anddiscovery involving for example technological costinfrastructure software competitor and strategic issues

Three separate evaluations are necessary to evaluate anoverall portfolio of radical innovation investments Firstit is necessary to estimate for each project within theportfolio a stand-on-its-own evaluation in accordancewith the projectrsquos progress relative to technical marketorganizational and resource-related challenges In somesituations an adjusted real options approach such as thatproposed in (7) is an appealing and realistic valuationapproach since it explicitly accounts for flexibilitymajor assumptions discovery and learning and the com-bination of uncertainties In highly uncertain situations afocus on rapid and systematic learning and discovery islikely to be more useful than an explicit evaluation byany means We found in our research that managementsthat are simultaneously sensitive to the multiple sourcesof risk uncertainty and returnmdashin contrast to focusingon isolated aspects of the innovation chainmdashenjoy sub-stantial success in bringing innovations to full commer-cialization

Three separateevaluations arenecessary to

evaluate an overallRI portfolio


Second it is necessary to evaluate the interrelationshipsamong each breakthrough innovation project theremaining projects and the other activities of the firmThe value of the portfolio is greater than the sum of thevaluations of each individual innovation project consid-ered on its own isolated merits Accounting for the valuesof the interrelationships between the individual projectsand other firm activities is generally an accountingnightmare but should be explicitly handled in some way

For example some participants in our long-term longi-tudinal study claimed that the value of the contributionsof the radical innovation projects to interrelated activi-ties products and services of the firm and to participat-ing customers was well in excess of the investment in theproject itself and better yet that the value increased overtime In addition learning from one project is often trans-ferred to other projects in the RI portfolio as well as toexisting businesses in the mainstream organization Mostof our participating companies indicate that muchtransfer of learning and value occurs prior to anyfinancial returns accruing to the breakthrough innovationinvestments

Third a sound portfolio evaluation requires assessmentof the individual innovation projects relative to businessstrategy business constraints business model capacityfor innovation competition and competenciesmdashboththose available and those to be learned Also macro-factors may impute opportunities or impediments thatcan heavily influence innovation activities For instancethere can be no doubt that the current environment forcrude oil provides an opportunity for a major thrust intoall manner of innovations related to extracting energymaterials from oil sands

We and many other researchers and practitioners havefound that evaluations are often wildly wrong forecast-ing either much too optimistic (high) or much too pessi-mistic (low) a return Nevertheless it is worthwhile todevelop a set of criteria questions and learning require-ments based on the experiences of companies activelyinvolved in radical innovation which will providevaluable inputs into any valuation methodology

Valuation is a multiple dimension entity which isusefully regarded as a network of interrelationships andinterfaces (13 Chapter 7 and 14 Part 1) The evaluationof a portfolio of potential breakthrough innovationsinvolves taking account of the forces of valuation suchas among many others revenue flows over time profit-ability over time market share over time cost structureof the businesses associated with the innovation overtime positive and negative impacts to existing andemerging businesses acquisition of existing customersfrom competitors competitive structure of businessenvironment barriers to entry rate of technologicalobsolescence creation of new markets and customerssize of markets interaction with customers a host of

macro-economic factors including currencies environ-mental thrusts taxation and so on Accordinglyvaluation in high-uncertainty environments can rarely besummarized in a single dimension such as cash flow indollars Therefore management of breakthrough innova-tion paths and strategies in these situations should not begoverned predominantly by dollar value summarizationsof a potential innovation over all the driving forces ofvalue

While our research has overwhelmingly shown that theuse of a single index of value is not necessarily sensiblein managing a portfolio of potential innovations it hasalso determined that it is not sensible to use a largenumber of value measures Our research companiesindicate that a suitably selected and relatively smallnumber of groupings of categories of information(extracted for the decision-making processes) can bevery useful in understanding how to manage throughtime and events Consequently we have settled onarranging the most useful information for managementinto eight groups There is nothing sacred about thenumber 8mdashit seems to have been effective in our limitedvalidation studies

As explained shortly we incorporated in the instrumentthe ability to develop eight indexes of managerial infor-mation or a grand index over the eight sub-indexesWhether indexes should be computed or not or up-datedor not on a periodic or event-driven or discovery-drivenbasis is up to the individual user Nevertheless the infor-mation we require to be developed for evaluating aportfolio is that which our more than 30 companies havefound to be most significant in coming to decisionsregarding investment in the portfolio of businesses andinnovations

We have not provided a means to develop a monetaryvalue even though the information that our tool requireswould cover the great majority of what is required todevelop a monetary index But in a world of great risk

Much transfer oflearning and value

occurs prior to anyfinancial returns

accruing tothe project


and uncertainty along multiple dimensions such a dollarvaluation would likely yield the value X plusmn kX where k isa very large number (3 or more) (Our engineeringfriends would find this amusing since it is tantamount todeclaring for example that the weight-bearing capacityof a bridge is 10 plusmn 30 tons but this type of uncertainty isquite common in the financial world)

While this is very different from other approaches itdrives the organization to develop information to answerquestions that are most important for senior manage-ment How much are we stretching into new domains Isthe portfolio of projects in a position to contribute to themarket in unique ways Do the projects leverage oneanother Are the project teams staffed appropriately Isour overall portfolio aligned with our strategic intent

Challenges Associated with RI Portfolios

We found that almost all of the managers and executivesresponsible for breakthrough innovation at theircompanies encountered more or less the same problemsWhat is the current value of our portfolio of potentialinnovations How does this value change over time Afew of the firms had set corporate strategies that dictatedthe businesses that they intended to compete in if notdominate over the very long term These firms found itcompelling to focus on experimentation learning anddiscovery across a broad range of technologies andmarkets rather than on explicit numerical evaluations ofpotential breakthrough innovations

Most firms had their own unique ways of estimatingvalue For these firms decision-making at the projectand portfolio level ranged from informal and opaqueconsensus among upper management to formal and moretransparent decision-making by committees utilizingexisting financial tools However the managers respon-sible for RI portfolios generally expressed their need foran instrument that would assist them in providing anapproach to sensibly valuing their companyrsquos portfolios

RI portfolio managers are regularly challenged toexpress the value of their portfolios to peer and senior-level managers in the organization as part of the ongoingbattle for resource allocation between immediate needsand future investment opportunities None of thesemanagers was completely comfortable with theapproaches they were using or had available to themAlmost all agreed that a checklist of the drivers ofvaluation that management needs in order to make suchan evaluation could be very helpful

Based on our observations of the firms in our study weidentified three difficulties in valuing an RI project orportfolio

1 The uncertainties and risks associated with theportfolio are so large that the information required tomake a financial decision with traditional financial toolsis not sufficiently reliable

2 The dimensionality that must be considered fordecision-making related to RI projects and portfoliosis very high Some of the dimensions that must beaddressed in assessing value are choice of technologiesmarkets to be developed size of potential markets costsof pursuing the potential innovation operations politicalenvironment taxes profitability organizational struc-tures economic environment and competition

3 The decision-making related to RI projects and the RIportfolio spans several levels in the firm

With these three identified challenges we began theprocess of developing a tool that would ostensibly helpmanagers make wise decisions on their projects and port-folios within the RI context

Developing the Tool

Interviews with the 12 companies that participated inPhase II coupled with inputs from the additional partici-pants in Phase III helped us to identify the critical driversof value that are actually used (some implicitly ratherthan explicitly) to assess RI projects (see ldquoThe RadicalInnovation Research Programrdquo next page) While all ofthe participating companies used formal checklists toevaluate projects sometimes we observed that decisionswere driven by other latent criteria such as portfoliobalance across earlier and later-stage projects and acrosstechnology domains as well as portfolio size Inaddition we noted that in the early stages of the opera-tions of the breakthrough innovation teams theiremphasis was almost always all about ideas anddiscovery and less so about markets competitionmacro-factors portfolio balance portfolio diversitycross-influence among projects and businesses etc Weidentified such criteria as important from our empiricalresearch and included them in the tool

From this input we generated a list of 55 items that webelieved captured important drivers of portfolio evalua-tion at the project portfolio and firm levels (capacitysolvency strategy business model macro-factors com-petition etc) These were presented to the ROR subcom-mittee in February 2005 and once again their input wasused to improve the tool We reduced the number ofitems and grouped them into the eight categories

I Projectrsquos Impact on Company Renewal

II Projectrsquos Impact on Company Growth

III Projectrsquos Impact on the Market

IV Projectrsquos Impact on the Portfolio

V Team Capabilities and Pace of Project

VI Firm Capabilities for this Project

VII External Environmentrsquos Impact on the Project

VIII Portfolio Health


Each category contains between 3 and 13 questions AnExcel-based tool was developed to allow for automaticscoring In addition at the direction of the subcommitteewe modified the tool to allow items not consideredrelevant to a specific firm or project to be disabled so asnot to affect the scoring This feature was added becausedrivers of valuation differ from firm to firm and fromproject to project even within the same firm

Validating the Tool

In May 2005 the ROR subcommittee tested the tool in aworkshop setting and provided feedback regarding thescoring system and any additional items they believedwere needed Feedback was generally positive and wesolicited other IRI member companies to try the tool ontheir entire RI portfolios Two such companies volun-teered The first was one of the worldrsquos largest integratedoil and chemical companies The company annuallyplows back an enormous amount of its profits intopotential innovation and breakthrough innovationinvestments The second was one of the worldrsquos largestintegrated imaging companies For many years it also hasmade major investments in potential breakthrough inno-vations

We visited each company and presented the tool in detailto the people responsible for the RI portfolio projectsThe 20 individuals who provided feedback were respon-sible for almost all of the radical innovation efforts at thetwo companies

Their feedback indicated they considered the tooleffective to highly effective on an absolute scale on

almost all dimensions (Complete tabulated feedback isavailable from the authors) Those who found our toolslightly ineffective focused on two issues First waswhether using the tool caused new insights to emergethat managers previously hadnrsquot considered (items 718) The second was the degree to which respondents feltthat the tool accurately portrayed the portfoliorsquos value orthe value of individual projects to the portfolio (item 13)Given that the output of this exercise was an index ratherthan a dollar value this second result is not surprisingThe change in the index over time will be the determin-ing factor in valuing the portfoliorsquos usefulness

Our feedback outcomes also reflect responses to theportfolio evaluation tool relative to other methodsrespondents are currently using Mean responses to thesequestions for the majority of the items are higher thanthose given for absolute effectiveness This can be inter-preted to mean that the PE tool offered an improvementin terms of portfolio evaluation over what the twocompanies were currently experiencing In some casesthis improvement was dramatic including

bull Item 9 confidence in the accuracy of the value of theportfolio

bull Item 10 value of projects relative to one another

bull Item 12 synergistic effects across the projects

bull Item 13 contribution of each project to the portfolio

bull Item 16 attention this will receive from senior man-agement

bull Item 18 number of surprises that arose

The Radical Innovation Research Program

This Radical Innovation portfolio evaluation study wasperformed as part of a larger research program on RI thathas been ongoing since 1995 sponsored by the IndustrialResearch Institutersquos Research-on-Research committee inconjunction with a group of academic researchers Thefirst phase of the research from 1995ndash2000 examined 12RI projects in ten large companies Air Products andChemicals Analog Devices DuPont GE GM IBMNortel Networks Polaroid Texas Instruments andUnited Technologies This phase determined that RIefforts are very unsystematically managed but that aseries of challenges confronting RI project teams could bebetter managed if companies eschewed the every-blind-squirrel-will-occasionally-find-a-nut (more-or-less adhoc) approach and took a more systematic and systems-oriented approach (15)

Phase II of this same research program from 2001ndash2005was a cross-case comparison of 12 companies that had adeclared strategic intent to develop or evolve their RIcapability In this phase we began investigating andanalyzing the top-management-driven systems-level

approaches We observed that a number of ldquograss rootsrdquoapproaches to developing continuing capability in break-through innovation for corporate growth and renewal hadtaken root but had not thrived The results of our empiricalresearch were boiled down into our evaluation tool

The companies involved in this second longitudinal studywere 3M Air Products and Chemicals Albany Interna-tional Corning Dupont GE IBM JampJ ConsumerProducts Kodak MeadWestvaco Sealed Air and ShellChemicals An additional nine companies (Bose DowCorning Guidant HP Intel PampG PPG Rohm amp HaasXerox) served as our validation set and could be charac-terized as Phase III of the program (We still continue ourinteractions with many of the companies of all threephases)

The executives and managers of the radical innovationthrusts of these companies met four times over the courseof the data collection period listened to our interpretationof the data and reflected back their own experiences Weobserved the challenges firms faced as they attempted toinstitutionalize an RI capability and the mechanisms theyuse to confront those challengesmdashThe Authors


All other dimensions were rated at least as effective ascurrent solutions with the exception of

bull Item 5 extent to which we became aware of new issuesregarding specific projects

bull Item 7 extent to which we became aware of new issuesregarding pacing of projects in the portfolio

bull Item 15 degree to which I could give answers I feelconfidence in

The first two of these can be easily explained since thetest occurred in a 3ndash4 hour workshop setting (due to timeconstraints) in which participants were asked to select asubset of the projects to evaluate Because the entire setof projects was not evaluated pacing could not be deter-mined and those projects for which management had themost to learn were likely not selected for the exercise Interms of item 15 it became apparent from both the RORsubcommittee test and the company workshops that col-laboration is necessary and likely one of the mostvaluable benefits of using the tool

Additional comments provided greater richness to thefeedback Participants wanted a thorough review andtraining session associated with the tool Some of theitems require interpretation within their own industrycontexts Items they believed should be dismissed as notrelevant at first glance turned out to be highly valuedwhen interpreted for their own innovation systems Thisrequired a fair amount of discussion Suggested additionsregarding networks have been added

Our final observation in terms of validation of the toolrsquosusefulness is that both companies expressed theirintention to incorporate the tool into their portfolioevaluation systems and processesmdasha sure sign that itoffers something of value

Using the Tool

The tool is relatively straightforward The portfoliomanager should work with each project leader to fill outSections IndashVII for each project in the portfolio Eachproject requires scoring in two columns The first columnallows that specific item to be disabled and thus notincorporated in the scoring by entering a 0 in thatspecific cell and entering a 1 or any number in the cellnext to it

As each section is completed an index score is computedat the bottom of that section for that project The indexcompares the actual score given to the total scorepossible for that section After all sections are completedfor each project the portfolio manager fills out sectionVIII which focuses on overall portfolio health Theembedded macro in the toolrsquos Excel spread sheet auto-matically generates three metrics 1) an index for eachproject across all sections labeled ldquoTotal Valuerdquo 2) anindex that aggregates the value for all projects labeledldquoMaximum Potential Value and 3) a final index thatincorporates the overall portfolio health over and abovethe aggregate value of each project to arrive at the totalportfolio value (ldquoPortfolio Totalsrdquo)

Text continues p 29

Portfolio performance as percent of maximum potential for dimensions I through VII






Much research and effort may be required to answer thequestions posed in the tool but the information sogathered is perhaps the most important result of exercis-ing the tool The development of a summary of what thetool is telling management is often very difficult toaccomplish We along with the research participantshave found that graphical representations will oftenprove effective in communicating the information Thereare many ways to do this and each company will want todevelop its own

We offer one simple way in the bar chart on p 24 Eachof the seven project-related dimensions (categories Ithrough VII) is aggregated across all projects in theportfolio This analysis provides clues to the RI portfoliomanager regarding specific areas that may require focusor improvement In the example provided the marketimpact dimension (ldquoExternal Environmentrsquos Impact onthe Projectrdquo) has the lowest performance as a percent ofmaximum potential of any of the seven dimensions

Consequently the portfolio manager may wish to scruti-nize initial project screening criteria to ensure thatquestions about the value that the project will bring to themarket are seriously considered On the other hand theprojects in the portfolio appear to be addressingCompany Renewal almost to their full potential Ifrenewal is the firmrsquos overarching objective for the RIportfolio a portfolio manager would be pleased withsuch a result Similar analyses can easily be built into thespreadsheet in accordance with the firmrsquos needs

Systematic Learning and Discovery

The instrument presented here has proven to beextremely valuable in isolating drivers of value and insuggesting the managerial paths necessary to take apotential innovation all the way through to full commer-cialization We have not attempted to put dollar values onpotential radical innovations because our empiricalresearch has indicated that when the participants recog-nized the uncertainties around the returns were muchlarger than the expected returns dollar valuation wouldnot be the sensible way to decide whether to pursue thepotential innovation or not However we have designedour instrument in such a way that if a company wishes toperform a dollar valuation the information gathered withour instrument will provide a superior approach todivining an economic evaluation of the portfolio ofpotential innovation investments not just a portfolio ofpotential breakthrough innovations

Further we have provided a sensible approach toproviding executives and managers with the questionsthat lead to the answers concerning how they invest overtime to grow and renew their company By trackingchanges in the value of their investments over timemdashnotin monetary terms but in terms that are important to thecompanyrsquos strategic intent renewal objectives riskprofile and capabilitiesmdashwe believe RI portfolio

managers will be better prepared We have set theframework for expansion of our tool into a systematiclearning and discovery approach to innovation

Acknowledgment

We gratefully acknowledge the following people whohave contributed to the development of the toolco-chairs of the IRIrsquos Research-on-Research sub-committee Radical Innovation III Ian Elsum AlanAyers Dave McKeough and Steve May all themembers of the sub-committee who spent significanttime reviewing and commenting on the tool and thosemembers at the two companies willing to invest in testingthe tool Finally we are deeply indebted to RTMrsquos editorfor his care and attention to improving this article

References1 Sorescu A B 2002 Sources and Consequences of Radical Inno-vation Unpublished doctoral thesis University of Houston Texas2 Sorescu A B Chandy A K and Prabhu J C 2003 Sources andFinancial Consequences of Radical Innovation Insights from Phar-maceuticals Journal of Marketing 67 pp 82ndash1023 Wuyts S Dutta S and Stremersch S 2004 Portfolios ofInterfirm Agreements in Technology-Intensive Markets Conse-quences for Innovation and Profitability Journal of Marketing 68pp 88ndash1004 OrsquoConnor G C Paulson A S and DeMartino R 2007 Orga-nizational Approaches to Building a Radical Innovation DynamicCapability International Journal of Technology Management forth-coming5 Fast N D 1978 New Venture Departments Organizing for Inno-vation Industrial Marketing Management 7 pp 77ndash906 van Putten A and MacMillan I 2004 Making Real OptionsWork Harvard Business Review 83 pp 131ndash1417 MacMillan I C van Putten A B McGrath R G andThompson J D 2006 Using Real Options Discipline for UncertainTechnology Investments Research-Technology Management 49pp 29ndash378 Paulson A S and Zhong R 2006 Extensions of Structures ofReal Options Working paper Rensselaer Polytechnic Institute9 Radcliffe R C 1997 Investment 5th Edition Reading Addison-Wesley10 Merrifield D B 2006 Strategic Accounting for RampDResearch-Technology Management 49 pp 9ndash1311 Schwarz N D 2007 Big Plane Big Problems Fortune Marchpp 94ndash10112 Copeland T and Antikarov V 2001 Real Options New YorkTexere13 Christensen C M and Raynor M E 2003 The InnovatorrsquosSolution Boston Harvard Business School Press14 Christensen C M 1997 The Innovatorrsquos Dilemma BostonHarvard Business School Press15 Leifer R et al 2000 Radical Innovation Boston HarvardBusiness School Press

Our instrument hasproven extremely

valuable in isolatingdrivers of value


mercialization efforts are more successful financiallythan those that do not (1ndash3)

This article presents the principal results of IndustrialResearch Institute-sponsored empirical research con-cerning evaluation of a portfolio of potential break-through innovations The research covers 12 years inthree phases at more than 30 large American companiesthat have proclaimed their intention to achieve repeat-able breakthrough innovation as a corporate capabilityFurther details concerning the study and participatingcompanies are provided in a later section See also (4) forresults concerning organizational structures

Radical innovation hubs or groups responsible formanaging these portfolios continue to be subject tointense scrutiny especially in times of limited capacityor senior leadership changeover A companyrsquos capacityto invest in potential breakthrough innovations can belimited in many ways Potentially ruinous litigation pro-ceedings extraordinary cost overruns on another innova-tion collapse of a principal market due to massiveovercapacity resulting in potentially fatal financial diffi-culties rapid acceptance of a new technological productwhich results in the dramatic decline of an incumbentproduct again resulting in severe financial difficulties orthe strategic decision to sell a profitable business unit ayear or so into the future are just a small number of eventsthat can limit or even eliminate capacity

Historically the average life expectancy of a radicalinnovation hub has been about four years (5) althoughour research leads us to believe that average life spans arelengthening especially for those companies goingthrough their second or third version of a hub Part of theproblem is the difficulty that RI hub managers face injustifying their RI portfolio because its economic value isso hard to quantify This occurs because of high uncer-tainty on multiple dimensions associated with RIprojects high failure rates at the project level that lead tohigh churn rates in the early stages at the portfolio levellengthy project maturation times and the fact that valueto the firm from any single project may accrue in indirectways (exposure to new market partners new learningnew technology frontiers) not traceable to any specificrevenue path

Our research on the management structures andprocesses associated with radical innovation has beenongoing in the Research-on-Research (ROR) committeeof the IRI (Research-Technology Managementrsquospublisher) since 1995 This research has highlighted thatRI hub managers need a mechanism to communicate thevalue of their work to the organization on a regular basisThey need a tool that helps articulate the value of the RIportfolio at any point in time as well as the change in itsvalue over time Traditional financial approaches cannothandle the extreme uncertainties inherent in the radicalinnovation management process and while real option

approaches are inherently better they must be adjusted inways that square with managersrsquo experiences See forexample the approach espoused by van Putten andMacMillan (6) MacMillan et al (7) and the commentsand criticisms of Paulson and Zhong (8)

Consequently an ROR sub-committee has taken as oneof its objectives the development of an RI portfolioevaluation tool whose fundamental assumptions devel-opment and results are described here The tool ispresented on pp 25ndash28 and an online version is availableat httpwwwiriincorg

Evaluating RI Portfolios

Evaluation of economic value of breakthrough innova-tion portfolios through future time horizons is at bestfiendishly difficult and at worst impossible We took athree-pronged approach to developing an instrument thatwould capture the majority of the information that mustbe obtained in order to sensibly evaluate a set of potentialbreakthrough innovations The end result was a series ofquestions constituting an evaluation instrument or toolThe answers to the questions may require much researchor much work on many fronts but it is worth it thequestions and answers address the valuation issues fromembryonic efforts through full commercialization andinclude firm-specific issues as well as issues beyond thefirmrsquos control

First we reviewed the literature to determine whatresearch must be conducted by the firm making thepotential innovation investment to come to a sensible anddefendable evaluation Next we sifted through all of thedata generated by our 12-year longitudinal researchstudy to determine what our research companies foundwould impute value how why and when valueschanged and what they found to determine risk andhow why and when risk changed

Finally we needed to find a couple of paradigms fromquantitative finance that might provide theoreticallymotivated guidance on valuing innovation investmentsthrough future time horizons We settled on the portfolio

Large firms thatsuccessfully

commercializeradical innovations

do better financially





















































Developing the Tool













Validating the Tool


























Using the Tool



Text continues p 29














Acknowledgment

























































Developing the Tool













Validating the Tool


























Using the Tool



Text continues p 29














Acknowledgment







Validating the Tool


























Using the Tool



Text continues p 29














Acknowledgment













Using the Tool



Text continues p 29














Acknowledgment

















Acknowledgment






evaluating radical innovation portfolios · in current businesses. for example, airbus (eads)...

Documents