Omega 36 (2008) 895–912www.elsevier.com/locate/omega

R&D project evaluation:An integrated DEA andbalanced scorecard approach�

Harel Eilat, Boaz Golany∗, Avraham ShtubFaculty of Industrial Engineering and Management, Technion, Israel Institute of Technology, Haifa 32000, Israel

Received 14 August 2005; accepted 5 May 2006Available online 7 February 2007

Abstract

We present and demonstrate a multi-criteria approach for evaluating R&D projects in different stages of their life cycle. Ourapproach integrates the balanced scorecard (BSC) and data envelopment analysis (DEA) and develops an extended DEA model.The input and output measures for the integrated DEA–BSC model are grouped in “cards” which are associated with a “BSCfor R&D projects”. The BSC is embedded in the DEA model through a hierarchical structure of constraints that reflect the BSCbalance considerations. We illustrate the proposed approach with a case study involving an industrial research laboratory thatselects and executes dozens of R&D projects every year.� 2006 Elsevier Ltd. All rights reserved.

Keywords: Data envelopment analysis; Balanced scorecard; Project evaluation

1. Introduction

This paper develops an analytical model aimed atevaluating research and development (R&D) projects indifferent stages of their life cycle. It may be applied toproject proposals—as part of a project selection pro-cedure, or to ongoing projects—during their initiation,planning, execution or closing stages. Based on the eval-uation, management has to decide which project propos-als should be selected, which ongoing projects shouldbe continued, or which resource level should be associ-ated with each selected or continued project. The eval-uation of projects at their closing stages should allow

� This manuscript was processed by Associate Editor John Semple.∗ Corresponding author. Tel.: +972 4 829 4512;

fax: +972 4 829 5688.E-mail address: golany@ie.technion.ac.il (B. Golany).

0305-0483/$ - see front matter � 2006 Elsevier Ltd. All rights reserved.doi:10.1016/j.omega.2006.05.002

the creation of a knowledge base of “best practices” and“lessons learned” that would be communicated through-out the organization for continuous learning.

The R&D project evaluation problem is a challengingdecision-making problem faced by decision makers thatdeal with R&D management. The evaluation involvesmultiple criteria measuring rewards, relevance to theorganization’s mission and objectives, strategic leveragepotential, probability of technical and commercial suc-cess, etc. Once the criteria are determined, they shouldalso be weighted to reflect the preferred emphasis ofthe organization. The focus on future events and oppor-tunities in a dynamic environment cause much of theinformation required to be at best uncertain and at worstunavailable. Opinions and judgments often have to sub-stitute for data, and measures could be estimated onlyqualitatively. While quantitative measures like return-on-investment (ROI) are sometimes hard to estimate,qualitative metrics like market familiarity and customer

896 H. Eilat et al. / Omega 36 (2008) 895–912

satisfaction are potentially important. The lack of re-liable quantitative measures is especially apparent innot-for-profit organizations, where qualitative mea-sures usually take a larger share in the overall eva-luation.

Despite these difficulties, projects should be evalu-ated and prioritized, as they compete for resources. Themodel we propose in this article tries to respond to thesechallenges by integrating two well-established manage-rial methodologies: balanced scorecard (BSC) [1] anddata envelopment analysis (DEA) [2].

The BSC is a management tool composed of a col-lection of measures, arranged in groups, and denotedas cards. The measures are related to four major man-agerial perspectives, and are aimed at providing topmanagers with a comprehensive view of their busi-ness. The cards offer a balanced evaluation of the or-ganizational performance along financial, marketing,operational and strategic dimensions. BSC combinesfinancial and operational measures, and focuses bothon the short- and long-term objectives of the organi-zation. It was motivated by the realization that tradi-tional financial measures by themselves are inadequatein providing a complete and useful overview of organi-zational performance. In [1], a number of different BSCstructures are presented for different industries. Indeed,many organizations have adopted the BSC approach toaccomplish critical management processes, clarify andtranslate their vision and strategy, communicate andlink strategic objectives and measures, plan and alignstrategic initiatives, and enhance strategic feedback andlearning. A specific BSC model for projects was firstsuggested by Stewart [3].

DEA [2,4] is a mathematical programming tech-nique that calculates the relative efficiency of mul-tiple decision-making units (DMUs) on the basis ofobserved inputs and outputs, which may be expressedwith different types of metrics. The basic concept inDEA is to measure the efficiency of a particular DMUagainst a projected point on an “efficiency frontier”.The usefulness of DEA in evaluating multi-criteriasystems and providing improvement targets for suchsystems is expressed in the large number of its reportedapplications, as described in [5]. Specific DEA modelsfor the context of technology selection or R&D projectevaluation were suggested by Oral et al. [6], Khouja[7], and Baker [8].

The method that we propose in this paper uses anextended DEA model, which quantifies some of thequalitative concepts embedded in the BSC approach.The integrated DEA–BSC model addresses three com-mon goals that firms are trying to accomplish [9,10]:

(1) achieving strategic objectives (effectiveness goal);(2) optimizing the usage of resources in generating de-sired outputs (efficiency goal); and (3) obtaining balance(balance goal). The model is applicable for evaluatingR&D projects in for-profit, private organizations (e.g.,venture capital funds), as well as in not-for-profit or-ganizations, such as government agencies charged withselecting R&D projects.

The contribution of the model that is presented in thispaper is both conceptual—the integration, for the firsttime, of the BSC into the DEA model through balanceconstraints, and theoretical—the introduction of a hier-archical structure of balancing constraints that restrictthe proportions of the total output/input of any givenDMU that are devoted to specific output/input measures.While traditional weight restriction techniques in DEA(see the literature review) focus on restricting the weightflexibility of the individual weights, the model presentedhere focuses on balancing the “importance” attached togroups of measures, and applies it within a hierarchi-cal balance structure. The model is also practical be-cause it supports the evaluation of projects throughouttheir life cycle—during the selection, planning, execu-tion, and termination phases—while relying on the pop-ular BSC measurement system. It also pays attentionto the goals that concern managers—namely, effective-ness, efficiency, and balance.

The rest of the paper is organized as follows: Sec-tion 2 provides a literature review; Section 3 devel-ops a specific BSC for R&D projects. The integratedDEA–BSC model is presented in Section 4, while itsassociated mathematical formulations are given in theappendix. Section 5 discusses a case study that appliesthe DEA–BSC model. Finally, Section 6 presents con-cluding remarks.

2. Literature review

Over the last few decades, the problem of R&Dproject evaluation has attracted significant attentionthat has led to a variety of methods. These meth-ods seek to develop quantitative measures to assessthe performance of R&D projects by systematicallyobtaining and integrating subjective and objective data.The methods range from simple screening proceduresto sophisticated mathematical procedures, and are usu-ally subdivided into the following categories [11,12]:scoring models [13], multi-criteria decision-making(MCDM) models [14–18], comparative approaches[19–21], and economic models [13,22,23]. Compre-hensive reviews of R&D project evaluation methodscan be found at Baker and Freeland [11], Baker and

H. Eilat et al. / Omega 36 (2008) 895–912 897

Pound [24], Danila [25], Schmidt and Freeland [26],and Henriksen and Traynor [27].

More recently, some researchers have proposed DEAas a tool for evaluating R&D projects [7,8]. They cat-egorized the relevant measures of the evaluation aseither inputs or outputs of the DEA model and usedthe efficiency scores to rank order the projects. Lintonet al. [28] used DEA to split a portfolio of projects into“accept”, “consider further” and “reject” groups, as afirst step in a portfolio analysis, and then used a graph-ical analysis approach to complete the evaluation. Oralet al. [6] used DEA to assess cross-efficiencies in col-lective decision-making settings. However, to the bestof our knowledge, weight-restriction techniques, whichconstitute a significant extension of the DEA, have neverbeen applied in the context of R&D project evaluation.

The original DEA model assesses the relative effi-ciency of a DMU as the ratio of weighted outputs toweighted inputs, where the model selects weights foreach DMU so as to present it in the most favorablelight. By doing so it identifies its relative efficiency withrespect to an “efficiency frontier” that is defined by allthe DMUs being assessed. However, in real world ap-plications virtually unconstrained weights are usuallyunacceptable [29]. Likewise, large differences in theweight values for different DMUs may be a concern.Restricted DEA approaches were developed to allowsome control over the weights in the model. A generalapproach for controlling factor weights is the cone-ratio(CR) method [30] that generalizes the original DEAformulation given in [4], by requiring that values forinput and output weights should be restricted withingiven closed cones. Another approach implements theassurance region (AR) principle [31], where the weightof one output/input is used as a basis of comparison forweights of all other outputs/inputs. Further developmentof the AR method can be found in [29], and an exampleof its implementation in [32]. Several other treatmentsof weight restriction have been published, for example,in [33–35]. The method we use and extend in our modeldevelopment was first presented in [36]. This methodpresents the idea of restricting weights based on theuse of proportions of the total output/input of a specificDMU that is devoted to a single output/input measure.

3. Balanced scorecard for R&D projects

To evaluate the attractiveness of project proposals, orthe success of ongoing or completed projects, appro-priate criteria should be determined. At the minimum,it should include criteria that managers feel are mostimportant, and for which they can provide hard data or

firm opinions. It is also important that it be completebut not redundant, and that it be linked to the short- andlong-term objectives of the organization. To determinethe criteria set for R&D project evaluation, we use amodel based on the BSC approach.

The BSC was first proposed by Kaplan and Nor-ton [37] as a methodology aimed at revealing problemareas within organizations and pointing out areas forimprovement. It was also promoted as a tool to alignan organization with its strategy [38], by deriving ob-jectives and measures for specific organizational unitsfrom a top–down process driven by the mission andstrategy of the entire organization. Projects, for the pur-pose of BSC, can be considered “mini-organizations”requiring the same clarifications and benchmarks as theparent organizations that are executing them [3]. In fact,because projects are typically more structured than or-ganizations, they are even more suitable for evaluation.The PMBOK guide [39] provides a structured model ofa project that is helpful in designing a BSC for R&Dprojects. This model includes a series of processes thatare described in terms of their inputs, outputs, and thetools used to transform the inputs into outputs.

The objective of the BSC for R&D projects we pro-pose here, is to support the evaluation process during thedifferent stages of a project’s life cycle. At the selectionphase, where project proposals are evaluated, the BSCcould be useful to clarify and translate the vision andstrategy of the organization, and to set the appropriatecriteria for a project’s attractiveness. Measures in thiscase would usually be forward looking, representingwhat is expected from these projects. At the planningphase, the scorecard might be used to set targets, alignprojects with organizational strategy, and allocate re-sources within and among projects. At the executionphase, the BSC could be instrumental in providing arelative measure of performance, evaluating the value ofthe projects in the face of changing circumstances andpriorities, and communicating the results throughoutthe organization. The measures in this case would be amix of forward-looking measures, as mentioned above,and backward-looking measures that represent what hasalready been accomplished. Finally, at the closingphase, the BSC for R&D projects can be used as amethod of inquiry to identify best practices, and pro-mote continuous learning.

A key component to any BSC is the baseline or bench-mark against which performance is measured. Withouta standard or a baseline, evaluation is impossible. Oncea baseline for evaluation is determined, the evaluationis done against the benchmark and the targeted plans.However, standards are hard to determine and can be

898 H. Eilat et al. / Omega 36 (2008) 895–912

misleading. Since DEA is based on relative analysis,the projects are evaluated against each other. By com-bining the BSC with DEA we overcome one of the ma-jor obstacles of BSC, namely, the need to determinestandards. By using our methodology, the final evalua-tion of the projects after their completion could also setstandards and point towards best practices.

Our proposed BSC for R&D projects looks at thefive perspectives—the four original perspectives of BSC(financial, customer, internal-business processes, learn-ing and growth) and an uncertainty perspective, whichwe added to emphasize its role in R&D projects.

The specific factors of project success and the im-portance of each perspective are indeed situation de-pendent. It is, however, possible to generalize theseperspectives for most R&D projects (e.g., achievinghigh productivity, improving quality, delivering theproject’s outcome at its due date, decreasing cycle timeand increasing market share). Thus, the BSC for R&Dprojects, that we present below, should be consideredas a template for building specific BSC models byapplying a systematic inquiry process, as defined byKaplan and Norton [38].

3.1. The financial perspective

The financial perspective examines the bottom-linecontribution of the project in monetary terms. It re-flects the profitability, cash flow, cost vs. budget, etc.The financial objectives serve as the focus for the objec-tives and measures in all the other scorecard perspec-tives. Every measure in the scorecard should be partof a cause-and-effect relationship to improve financialperformance [1].

Many researchers have criticized what they perceiveas an overemphasis on achieving and maintaining short-term financial results that can cause a bias towards in-vesting in projects with short-term benefits, leading toinsufficient investments in projects with long-term valuecreation, particularly in the intangible and intellectualassets that R&D projects usually produce. To overcomethis obstacle, the BSC presents four other perspectivesthat ensure a more balanced evaluation of the project.

3.2. The customer perspective

The customer perspective of our BSC for R&Dprojects looks at the market value of the project de-liverables, as well as stakeholder satisfaction with thefinal outcomes. The customer is interested in the re-sponsiveness, timeliness, service and quality that theproject provides. This perspective can include measures

taken from customer surveys, focus groups, complaints,delivery statistics, etc. The question to ask is “howsuccessful are the projects from the point of view ofthe customer?” Time to market, quality, and perfor-mance, as well as the way the customer is treated andthe way his expectations are satisfied, are all relevantto evaluate the projects.

3.3. The internal-business processes perspective

This perspective measures the contribution of theproject to the core competencies of the organization.It addresses the degree to which the proposed projectsupports the organization’s mission and strategic ob-jectives. It is assumed that the top management hasdetermined the strategic direction of the organizationbeforehand. The strategic fit can be expressed as a judg-ment ranging from strong to peripheral or it can usemore detailed measures. The question asked is “whatshould the organization excel at?” If the organizationwants to expand its range of core capacities into a newfield, it must establish specific measures to reflect this.When the fit is poor, the R&D project must be rejectedor the strategy must be rethought. Otherwise, the fitlevel, be it strong, good, moderate or only peripheral,must affect the overall measure of the project’s attrac-tiveness.

3.4. The learning and growth perspective

In today’s global competitive environment, organiza-tions are constantly looking for further performance im-provements to keep pace with competition. The objec-tive in the learning and growth perspective is to providethe infrastructure to enable the objectives of the abovethree perspectives. When the evaluation is solely basedon the short-term financial perspective, it is often diffi-cult to sustain investments to enhance the capability ofthe human resources, systems, and organizational pro-cesses. Hence, this perspective looks at the long-rangegrowth impact of the project. The measures it includes(e.g., propriety position) check whether the project isa platform for growth, and look at the durability of itseffects.

3.5. The uncertainty perspective

The uncertainty perspective includes measures suchas the probability of technical success and the probabil-ity of commercial success, which are critical measuresin evaluating R&D projects. These measures are esti-mated either directly in the scale of 0–1 [13, ch. 20], or

H. Eilat et al. / Omega 36 (2008) 895–912 899

indirectly through operational and market-related mea-sures. The probability for technical success includessuch measures as technical “gap,” program complex-ity, technology skill base, and availability of people andfacilities. The probability for commercial success in-cludes such measures as market need, market maturity,competitive intensity, commercial applications devel-opment skills, commercial assumptions and regulatoryimpact.

3.6. The BSC template

A starting point for customizing the BSC to thespecial needs in a specific application can be set byusing key success factors from the literature, the PM-BOK guide [39], and internal organization standardsand benchmarks.

The BSC model that we constructed is presented inTable 1. This model consists of two hierarchical lev-els: the cards and the measures. The model includesthe five cards discussed above with 24 output and inputmeasures. Table 1 also reports on the units of measure-ment used for each input and output. These units spana diverse spectrum of metrics including monetary val-ues, arbitrary (subjective) scales, and probability values.These settings make DEA a suitable modeling venue asit is geared to handle incommensurable metrics.

4. The DEA–BSC evaluation model

DEA consist of a family of models with variousassumptions on the input–output relationships that areexhibited by the DMUs under consideration [2]. Ourfirst task in developing a DEA-based model is to selectthe formulation that best fits the particular environmentin which we are interested. For example, when evalu-ating a set of diverse projects with significantly differ-ent resource requirements which are competing for thesame resources, a variable return-to-scale model (e.g.,the model developed by Banker et al. [28], and com-monly known as the BCC model) would be more appro-priate than one which assumes constant return-to-scale.Conversely, when the projects are more homogeneous,a constant return-to-scale model may be more suitable.In the current presentation, we rely on the original con-stant return-to-scale model of DEA, as developed byCharnes et al. [4] and commonly known as the CCRmodel.1

1 The model extension is similarly applicable to other DEAmodels, and particularly to the BCC model mentioned above.

In the discussion that follows, we assume that thereare n projects. Each project consumes varying amountsof m different inputs and produces s different out-puts. Specifically, project Pj (j = 1, . . . , n) consumesamounts Xj = {xij } of inputs (i = 1, . . . , m) and pro-duces Yj ={yrj } outputs (r =1, . . . , s). We assume thatxij > 0 and yij > 0. The m × n matrix of inputs is de-noted by X and the s × n matrix of outputs is denotedby Y. The corresponding input and output weights (i.e.,the variables of the DEA model) are denoted by thevectors v = {vi} and u = {ur}, respectively.

The model that we present extends the original CCRmodel (see Appendix A) by integrating into it a BSCstructure. All the input and output values for each spe-cific project represent the measures in its BSC andvice versa. The BSC structure is embedded into theDEA model through a set of balance constraints. Theseconstraints are related to “weight restrictions” in DEA[30,31,40,35,29]. Specifically, these constraints followthe proportional weight-restriction method that was pro-posed by Wong and Beasley [36]. However, traditionalweight-restriction methods are applied separately oneach single variable, whereas the balance constraints weintroduce here are applied to sets of variables that areassociated with the cards in the BSC hierarchical struc-ture described above.

4.1. The single-level DEA–BSC model

We shall define a single-level BSC structure as onewith a single partition of the set of inputs and outputs.In what follows, we will focus only on output-balanceconstraints. The discussion regarding the input-balanceconstraints is similar.

Let O1, . . . , OK represent the partition of the set ofoutputs into K “cards”. Consequently, Eq. (1) below istrue by definition:

K∑k=1

⎛⎝ ∑

r∈Ok

uryrj

/∑r

uryrj

⎞⎠ = 1, ∀j . (1)

The value Sk = (∑

r∈Okuryr0)/(

∑ruryr0) is a dimen-

sionless quantity that represents the proportion of thetotal output of project P0 devoted to card Ok . We canregard Sk as the “importance” attached to card Ok byproject P0, since the larger this expression, the more theproject depends upon outputs in Ok in determining itsoverall efficiency score.

In order to reflect the desired balance, a decisionmaker can set limits on what may be regarded as suit-able lower and upper bounds for the relative impor-tance on each card. Formally, we impose the following

900 H. Eilat et al. / Omega 36 (2008) 895–912Ta

ble

1B

SCm

odel

for

R&

Dpr

ojec

tsa

H. Eilat et al. / Omega 36 (2008) 895–912 901

902 H. Eilat et al. / Omega 36 (2008) 895–912

Fig. 1. An example graph representation of a hierarchical BSCstructure.

restrictions for any specific project P0 that is being eval-uated:

Lk �∑r∈Ok

uryr0

/∑r

uryr0 �Uk, ∀k. (2)

The single-level DEA–BSC model adds the set ofconstraints in (2), and the corresponding set associatedwith the inputs, to the CCR model. The linear program-ming formulation of the extended model is given inAppendix B and a theorem that defines the conditionsfor its feasibility is stated and proven in Appendix C.

While our presentation assumes cards with mutuallyexclusive sets of measures, the mathematical extensionto the case of non-mutually exclusive measures is trivial.Different cards may have some common measures, aslong as the cards themselves are not identical—in whichcase the corresponding balance constraints would beredundant. When a certain measure appears in severalcards, its associated scaled output should be added tothe denominator of the constraints in (2) according tothe number of times it appears in the cards.

The balance constraints reflect two kinds of valuejudgments on the factor weights. First, the relative min-imal “importance” of each card is reflected by its lowerbound. Second, the span of variation of each card isgiven by the difference between the lower and upperbounds.

4.2. The multiple-level DEA–BSC model

To incorporate a more general BSC structure withmultiple hierarchical levels, we use a graph representa-tion. Let GO(NO, EO) and GI(NI, EI) be the directedgraphs associated with the set of output and input mea-

sures, respectively. Focusing on the outputs, we let Oi

denote a node i ∈ NO in the graph to represent a card,which includes a subset of output measures. An arc(i, j) ∈ EO in the graph represents an inclusion rela-tionship among the corresponding cards, namely Oj ⊂Oi . Hence, the set of cards with the same direct ances-tor i ∈ NO constitute a partition of Oi , meaning thatthey do not include more than one instance of a sin-gle measure and collectively they create the father card.This description defines a hierarchical structure that isrepresented by a spanning tree.

An example of this representation is shown inFig. 1. The numbers associated with the nodes followa breadth first search (BFS) starting with O0.2 Thepair of numbers adjacent to each node represents thebalance bounds associated with the corresponding card.

Let BO represent the s × 2�O matrix of coefficientsof the output-balance constraints, where s is the numberof outputs and �O the number of output cards excludingO0 (the root card, O0, represents the complete set ofoutput measures, and is not balanced against any othercard). Similarly, let BI represent the r × 2�I matrix ofcoefficients of the input-balance constraints, where r isthe number of inputs and �I the number of input cardsexcluding I0. The matrix BO is composed of two ma-trices: BOL of dimension s × �O for lower bound con-straints and BOL of the same dimension for upper boundconstraints, such that BO = (BOL|BOU). (Using similarnotation, BI = (BIL|BIU).) The corresponding output-balance matrices for the single-level formulation dis-cussed above are presented in the following equations:

BOL=

⎛⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎝

−1+L1 L2 Ln...

......

−1+L1 L2L1 −1+L2...

...

−1+L2L2...

. . ....

Ln

−1+Ln...

......

L1 L2 −1+Ln

⎞⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎠

⊗(�1×Y T0 ),

(3a)

2 The graph starts with card O0 that contains the complete set ofmeasures. The next level includes the cards O1, . . . , Ok0 that rep-resent the partition of O0 into k0 cards. Next, OKi+1, . . . , OKi+1 ,

Ki = ∑ij=0 ki , represent the partition of Oi to ki cards, etc.

H. Eilat et al. / Omega 36 (2008) 895–912 903

BOU=

⎛⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎝

1−U1 −U2 −Un...

......

1 − U1 −U2−U1 1 − U2

......

1 − U2−U2

.... . .

...

−Un

1 − Un...

......

−U1 −U2 1 − Un

⎞⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎠

⊗(�1×Y T0 ).

(3b)

The vector �1 denotes a column vector of ones, and ⊗the entry-by-entry multiplication.

For the multiple-level formulation, we present inAppendix D an algorithm for generating the balancematrix based on the BFS algorithm. The linear for-mulation of the DEA–BSC model for a general datastructure is presented in (4).

maxu,v

s0 = uTY0

s.t.

vTX0 = 1,

uTY − vTX�0,

uTBO �0,

vTBI �0,

uT �� · �1,

vT �� · �1. (4)

Again, as in the single-level case, the bounds appliedto the proportions of the card score should be consistentfor the problem to be feasible. Theorem 2 in AppendixC defines the conditions for feasibility in the multi-level DEA–BSC model that correspond to a multi-levelstructure.

5. Case study

The original impetus for the work presented in thispaper arose out of a study that involves an R&D unitwithin a large organization. The objectives of this unitare defined in terms of the long-range goals of theorganization rather than a specific technical objective.The federal research laboratory discussed in [27] is anexample of such a unit. In order to illustrate and moti-vate the method, we have developed the following casestudy that refers to an industrial research laboratory ina similar setting.

The research laboratory in this study conducts bothbasic and applied research, and may provide technicalsupport for the operations and manufacturing functionsof the organization. It tends to focus on large, high-riskand long-term projects involving leading-edge technolo-gies, and it is directed at fulfilling the objectives andthe missions of the organization.

5.1. The data

The data for the case study are presented in Table 2.In generating the data, we had two objectives in mind:first, to use measures that were originally proposed inthe literature and are part of the BSC described above,and second, to stay as close as possible to data thatare relevant to the R&D unit that motivated the study.Random data fluctuations were inserted to overcomepotential data confidentiality problems.

A typical evaluation scenario in this R&D unitinvolves some 30–60 projects (all within the same tech-nological category). The evaluation of these projectsinvolves many performance aspects; therefore, using11 output measures and two input measures for thisevaluation is quite reasonable. An ordinary DEA modelwould fail to provide sufficient discrimination amongthe projects (i.e., it is likely to declare too many projectsas relatively efficient). We deliberately demonstrate ourcase with just 50 projects (given the number of inputsand outputs) to highlight the added value that we gainwith the DEA–BSC model, which is capable of pro-viding a sharper discriminative ranking of the projects.

5.2. The model implementation

Different organizations may have a different man-agerial approach regarding the selection and controlof their R&D projects. One possible approach is togive the project manager the flexibility to decide onthe desired preferences among conflicting perspectivesof the project’s success, and then to direct the projectaccordingly. By applying this approach, top manage-ment gives project managers the privilege to decidehow their projects would be evaluated. For example,the project manager can decide whether the financialperspective is more important than the internal-businessperspective, and to what extent. Putting more effort intoachieving desirable results in one perspective can be atthe expense of other perspectives.

A different approach would allow only limited flexi-bility within general guidelines set by top management.In this case, top management gives more attention towhat it considers more desirable to the organization

904 H. Eilat et al. / Omega 36 (2008) 895–912Ta

ble

2T

heca

sest

udy

data

a

H. Eilat et al. / Omega 36 (2008) 895–912 90537

7499

754

74

44

47

0.5

3899

153

778

710

107

710

0.7

3975

121

775

77

77

77

0.6

4072

987

604

47

44

40.

641

8114

37

917

77

77

70.

742

9911

27

737

77

47

70.

743

7616

010

9710

710

47

70.

444

7613

17

737

710

77

70.

745

7493

453

47

44

44

0.6

4673

120

773

74

74

44

0.6

4798

108

767

77

77

77

0.6

4882

102

764

47

77

47

0.6

4987

997

724

47

77

70.

650

7513

07

837

77

77

70.

8

a Out

put

valu

esth

atco

rres

pond

tohi

ghes

tra

tios

are

shad

ed.

by setting preferences, and then evaluating the projectsaccordingly. This represents a top–down approach thatstarts with the strategy of the organization, which is thentranslated into actual preferences. The implementationthat follows will demonstrate the implications and com-pare the results of these two approaches.

The managerial approach is expressed in our modelby the balance bounds. We shall use two sets of boundsfor the two approaches presented above, and implementthe corresponding mathematical models. The particularcase involves three kinds of R&D projects: basic re-search projects with relatively low probability of successand high financial return (“Oysters”); applied researchprojects with relatively high probability of success andlower financial return (“Bread and Butter”); and projectsthat provide technical support for operations and man-ufacturing that enhance the internal-business processes.This diversity directed us in determining the balancebounds that are presented in Table 3 .

To implement the first approach discussed above,we assumed maximal tolerance between lower andupper bounds, and allowed tradability between cards.However, within each card we used lower and upperbounds that express the relative importance of eachmeasure in the card. For the second approach, weforced the relative importance of the financial perspec-tive to reside between 30% and 70% and allowed therelative importance of the customer, internal business,and learning and growth perspectives to vary between6% and 46%. Finally, the bounds for the relative im-portance of the uncertainty perspective were set at12% and 52%, respectively. (Note that the differencebetween the upper and lower bounds, which is the tol-erance allowed in the change of preferences, is 40%in all cases.) In setting these values, we express thetradability between the financial and the uncertaintyperspectives—encouraging high financial returns atthe expense of incurring a high level of risk (or lowerprobability of success), and between the financial andinternal-business perspectives—encouraging technicalsupport for operation and manufacturing at the expenseof financial returns.

The mathematical model was implemented througha MATLAB procedure that was developed as a soft-ware decision support tool (DSS). The application pro-cedure gives decision makers user-friendly access tothe model. The software receives the following inputs:(1) a graph representation of the measurement struc-ture (i.e., the BSC), (2) the lower and upper balancingbounds for each card (and for each measure), and (3)the values of the inputs and outputs. Using this data,the procedure automatically constructs an LP model

906 H. Eilat et al. / Omega 36 (2008) 895–912

Table 3Balance bounds used for the implementation of the DEA–BSC model in the case study

Card label Card\measure 1st case 2nd case

Lower bound Upper bound Lower bound Upper bound

O1 Financial perspective 0 1 0.3 0.7Discounter cash flow – – – –

O2 Customer perspective 0 1 0.06 0.46Customer focus group feedback 0.4 0.6 0.4 0.6Performance improvement metrics 0.4 0.6 0.4 0.6

O3 Internal-business perspective 0 1 0.06 0.46Congruence 0.32 0.52 0.32 0.52Importance 0.24 0.44 0.24 0.44Synergy with other operations 0.24 0.44 0.24 0.44

O4 Learning and growth perspective 0 1 0.06 0.46Propriety position 0.4 0.6 0.4 0.6Platform for growth 0.24 0.44 0.24 0.44Durability 0.16 0.36 0.16 0.36

O5 Uncertainty perspective 0 1 0.12 0.52Probability of tech & com success – – – –

I1 Resources 0 1 0 1Investments – – – –

of each project, solves it, and then returns the DEAscores.

Table 4 presents the results of the implementation.The first pair of columns shows the results obtained byapplying the CCR model without balance constraints atall. The second pair of columns shows the outcome ofthe first managerial approach presented above. The thirdpair of columns shows the results obtained by applyingthe second managerial approach. The balance boundsfor the latter two cases are presented in Table 3. Thescore column in each pair presents the DEA efficiencyscore, whereas the rating column presents a rankingwith increasing order numbers for decreasing scores(projects with the same DEA score were assigned thesame rank order).

Fig. 2 shows the distribution of the scores given inTable 4. It is evident that as more balancing restrictionsare applied to the model, its discriminating power in-creases. The group of projects with the score 1 in theCCR results includes 8 projects and the range of scoresfor the other projects is [0.60, 1). After adding all bal-ance constraints to the model, we are left with only threeprojects whose score is 1, and the range of the otherscores is [0.55, 1). The change in the distribution is aresult of the added information expressed by the bal-ance bounds that were obtained from the managers andreflects their managerial approach.

The highest-scoring projects when applying thebalance bounds of the first approach are projects 2,20, 34, and 43. To gain an insight on these efficient

projects, we review the output–input ratios that char-acterize the data of project 2. Recall that in the CCRmodel, the DMU that has the highest output-to-inputratio in any output–input pair is guaranteed to be fullyefficient.3 Output values that correspond to the highestoutput-to-input ratios are shaded in Table 2. Project 2received a score of 1 since it has three such maximalratios: the highest performance improvement to invest-ment ratio, the highest propriety position to invest-ment ratio, and the highest platform for growth scoreto investment ratio. When adding balance restrictionsto the model, project 2 remains with the score 1 due toits relatively high ratios in all the other input–outputpairs (see Table 4). In contrast, project 24 also getsthe score 1 in the CCR model, since it has the highestdurability-to-investment ratio. However, when addingthe balance constraints to the measures within each card(the first approach above), the score of this project fallsto 0.9973. Further, when adding the balance constraintsto the cards as well, the score falls even lower to 0.852.This is due to its relatively poor performance in othermeasures (e.g., its discounted cash flow to investment isfairly low). Project 34 does not have the highest output-to-input ratio in any output-input pair; however, it is stillrated efficient. This project has relatively high output

3 Such DMU can select positive virtual multipliers for this spe-cific pair and minimal values for the multiplier of all other inputsand outputs and thus achieve a score of 1.

H. Eilat et al. / Omega 36 (2008) 895–912 907

Table 4DEA–BSC results

Project # CCR results 1st approach results 2nd approach results

Score Ratinga Score Ratinga Score Ratinga

1 0.7446 32 0.7184 31 0.6538 392 1 1 1 1 1 13 0.7869 26 0.7595 27 0.7174 304 0.8715 10 0.8148 17 0.7927 185 0.7958 23 0.7697 24 0.7513 246 0.7894 25 0.6852 37 0.6764 337 0.7605 30 0.6815 39 0.6727 378 0.8130 19 0.7851 22 0.7486 269 0.8074 21 0.7633 26 0.7437 27

10 1 1 1 1 0.9663 211 0.8789 8 0.8649 11 0.8541 912 0.7652 29 0.7377 28 0.7282 2813 0.8068 22 0.8068 18 0.7638 2114 0.8348 15 0.8066 19 0.7625 2215 0.7432 33 0.6849 38 0.6522 4016 0.8789 9 0.8474 12 0.8283 1517 0.7271 34 0.7054 35 0.6737 3618 0.7823 27 0.7373 29 0.7124 3119 0.9221 5 0.9221 6 0.9031 520 1 1 1 1 1 121 0.6944 37 0.6930 36 0.6304 4322 0.9221 6 0.9164 7 0.8363 1323 0.7932 24 0.7837 23 0.6890 3224 1 1 0.9773 2 0.8520 1125 1 1 0.9708 4 0.8948 626 0.6283 42 0.5706 45 0.5514 4827 0.8697 11 0.8671 10 0.8506 1228 0.6509 39 0.6392 41 0.6345 4229 0.9771 3 0.9747 3 0.8320 1430 0.6358 40 0.6174 42 0.5660 4631 1 1 1 1 0.8605 832 0.6048 43 0.6031 44 0.5629 4733 0.8161 18 0.8161 16 0.7879 1934 1 1 1 1 1 135 0.8241 16 0.8033 20 0.7574 2336 0.8693 12 0.8436 14 0.8261 1637 0.7662 28 0.7184 32 0.6758 3438 0.8113 20 0.7636 25 0.7501 2539 0.8371 14 0.8351 15 0.8247 1740 0.8206 17 0.7989 21 0.7206 2941 0.8882 7 0.8689 9 0.8537 1042 0.6570 38 0.6499 40 0.6159 4443 1 1 1 1 0.9257 344 0.9910 2 0.8990 8 0.8897 745 0.7495 31 0.7178 33 0.6595 3846 0.8457 13 0.8457 13 0.7846 2047 0.6307 41 0.6167 43 0.5993 4548 0.7243 35 0.7187 30 0.6739 3549 0.7110 36 0.7064 34 0.6402 4150 0.9467 4 0.9467 5 0.9168 4

aThe rating assigns increasing order numbers for decreasing scores. Projects with the same score are assigned the same order number (e.g.all projects with the score 1 are assign the order number 1).

908 H. Eilat et al. / Omega 36 (2008) 895–912

0

2

4

6

8

10

12

14

16

18

0.0 -0.1

0.1 -0.2

0.2 -0.3

0.3 -0.4

0.4 -0.5

0.5 -0.6

0.6 -0.7

0.7 -0.8

0.8 -0.9

0.9-1.0

1.0

Num

ber o

f Pro

ject

s

CCR Results

DEA-BSC 1st Case

DEA-BSC 2nd Case

Score

Fig. 2. Histogram of scores.

values in all measures, and hence it remains with thescore 1 when adding the balance constraints.

The tradeoff between the financial and uncertaintyperspectives can be demonstrated through projects 2,10, 20, and 43. While project 20 has the highest dis-counted cash flow-to-investment ratio, its probability oftechnical and commercial success is relatively low (0.5).However, projects 2, 10 and 43 all have a high probabil-ity of technical and commercial success-to-investmentratios, but smaller discounted cash flow to investment(especially project 10 with a ratio of 110

73 = 1.51). Allfour projects, despite their different emphasis, achievethe highest score of 1 in the CCR model and in thefirst approach discussed above. The second approach,however, assigns to only two of these projects (projects2 and 20) a score of 1. The other two are still highlyranked, but their scores are lower than 1 as a resultof their lower discounted cash flow-to-investment ratiothat is important according to the values in Table 3.

These results are communicated throughout the or-ganization, so that project managers receive a clearsignal on their success, by using an objective evaluationmethod. A more proactive approach may even use theseresults as a basis to reallocate resources. Projects withhigh relative ratings would get more resources (sincewe expect these projects to stay efficient), whereasresources to under-performing projects might be cut.The reallocation of resources for maximal overall

organization benefit and technical efficiency may be ac-complished through the method given in [41].

6. Summary and conclusion

This paper presented a multi-criteria approach forR&D project evaluation based on the integration oftwo different innovative managerial methodologies. Wecombined concepts taken from data envelopment anal-ysis (DEA) and balanced scorecard (BSC), which haveproven to be useful measurement and analysis tools inmany practical applications. These concepts were inte-grated into a single DEA–BSC model. Values obtainedthrough this model account for “benefits” (outputs),“costs” (inputs), and preferences. The model discrimi-nates projects according to desired characteristics andranks them consistent with the organization’s intendedemphasis.

The DEA–BSC model advances the individual capa-bilities of DEA and BSC. From the viewpoint of DEA,the model generalizes the standard treatment of the databy splitting the inputs and outputs into subsets (cards),and adding constraints (balancing requirements) thatreflect relationships among the cards. From the view-point of BSC, the model proposes a new approach toevaluate performance by applying quantitative analysisthat combines the measures within each card into a sin-gle value. It also addresses some of the difficulties in

H. Eilat et al. / Omega 36 (2008) 895–912 909

existing BSC applications, namely, reliance on a known(sometimes arbitrarily chosen) baseline against whichperformance is evaluated and the fact that BSC doesnot produce a single, comprehensive measure of perfor-mance.

Relying on the cards’ structure, we introducedmulti-level balance restrictions. We included theserestrictions in the DEA model, creating the integratedDEA–BSC model. This model was initially developedfor the simplified, single-level partition hierarchy andwas extended to the multi-level hierarchy.

We illustrated the implementation of the model in thesettings of an industrial research laboratory charged withR&D projects. In future work, we hope to include an in-terim project evaluation, as well as a retrospective pro-ductivity assessment and post-project impact analysis.

The model is consistent with the BSC methodologythat has been applied in many companies, and can beused for the evaluation of project proposals, and ongoingprojects—in all their progressive stages.

The model presented in this paper could be extendedto deal with portfolio considerations that are differ-ent from those taken in evaluating individual projects(e.g., finding the proper mix of projects that reflects thedesired tradeoff between risk and reward, balancingR&D efforts among different technologies, etc.). Thechallenge is to devise a method that would aggregatethe inputs and outputs of individual projects while tak-ing into account possible interactions among them. Inthis case, the DMUs would represent alternative port-folios that would be evaluated against each other. Thisapproach will require a decomposition of the portfolioanalysis problem into two parts. The first would dealwith the generation of the portfolio (i.e., the combina-tion of projects and their aggregate inputs and outputs),whereas the second would evaluate their relative effi-ciency. This extension is currently being pursued bythe authors.

Appendix A. The CCR model

The CCR model defines the relative efficiency of aspecific project P0 as the ratio between the sum of itsweighted outputs

∑ruryr0 and the sum of its weighted

inputs∑

rvixi0. The variables, ur > 0 and vi > 0, aredefined in a way that allows the project in questionto present itself in the most favorable way. The ratio,which is to be maximized for project P0, leads to thefollowing objective function:

maxu,v

s0 =∑

ruryr0∑ivixi0

. (A.1)

The optimization problem in (A.1) is unbounded.Normalization constraints (one for each project) forcethe ratios of weighted outputs to weighted inputs ofevery project to be less than or equal to 1, as shownbelow:∑

ruryrj∑ivixij

�1, ∀j . (A.2)

The constrained optimization problem defined by(A.1) and (A.2), including the positivity constraints ofthe weights, construct the original CCR model in itsratio form. The ratio formulation was transformed in[4] into an equivalent linear programming formulationknown as the input-oriented CCR model4 shown in(A.3). The constant � is a small positive number thatfunctions as a lower bound for the multipliers.

maxu,v

s0 =∑

r

uryr0

s.t. ∑i

vixi0 = 1,

∑r

uryrj −∑

i

vixij �0 ∀j ,

ur ��,

vi ��. (A.3)

In this formulation, no a priori values are assignedto the input–output weights. This means, for exam-ple, that an alternative, which is a superior producer ofa marginally important output, might be diagnosed asefficient even if it performs poorly with respect to allother outputs. Furthermore, in evaluating different alter-natives, the same factors may be assigned widely differ-ent weights for the same dimension. Hence, the scoresprovided by the CCR model are not necessarily goodindicators for R&D project performance, as they mightnot reflect the desired balance among the outputs of themodel.

Appendix B. The linear programming formulationof the single-level DEA–BSC

The formulation in (B.1) is the single-level DEA–BSClinear formulation. In this formulation, the balancingconstraints are divided into two groups—lower- andupper-bound constraints, for each output card Ok ,k = 1, . . . , KO, and input card Ik , k = 1, . . . , KI. Thelower and upper bounds are denoted [LOk

, UOk] and

4 The CCR model admits both input-oriented and output-orientedmodels. While the input-oriented model focuses on savings of inputs,the output-oriented model concentrates on enhancement of outputs.

910 H. Eilat et al. / Omega 36 (2008) 895–912

[LIk, UIk

] for the outputs and inputs, respectively. Theseconstraints ensure that any score produced by the modelreflects the desired balance among the output cards andamong the input cards.

maxu,v

z0 =∑

r

uryr0

s.t. ∑i

vixi0 = 1,

∑r

uryrj−∑

i

vixij �0 ∀j ,

−∑r∈Ok

uryr0+LOk

∑r

uryr0 �0 ∀k=1, . . . , KO,

∑r∈Ok

uryr0−UOk

∑r

uryr0 �0 ∀k=1, . . . , KO,

−∑i∈Ik

vixi0+LIk

∑i

vixi0 �0 ∀k = 1, . . . , KI,

∑i∈Ik

vixi0 − UIk

∑i

vixi0 �0 ∀k = 1, . . . , KI,

− ur � − � ∀r ,

− vi � − � ∀i. (B.1)

Appendix C. Feasibility conditions for the DEA–BSC model

Theorem 1 refers to the single-level DEA– BSCmodel, whereas Theorem 2 refers to the multi-levelDEA–BSC model. The conditions are expressed us-ing the following general notation for the lower- andupper-bounds, Lk and Uk , and for the number of cards,K, that represent interchangeably the output-balanceconstraints parameters, LOk

, UOkand KO, and the

input-balance constraints parameters, LIk, UIk

and KI.

Theorem 1. The single-level DEA–BSC model is feasi-ble if and only if the following conditions hold for the setof output-balance bounds, {(LOk

, UOk), k=1, . . . , KO},

and the set of input-balance bounds, {(LIk, UIk

), k =1, . . . , KI}, independently: (i)

∑Kk=1Lk �1, (ii)∑K

k=1Uk �1, (iii) Lk �Uk ∀k, and (iv) Uk �0 ∀k (mayequal zero if and only if yr0 = 0 ∀r ∈ Ok for theoutput-balance bounds, or xi0 = 0 ∀i ∈ Ik for theinput-balance bounds).

Proof. We prove the case where only output-balanceconstraints exist. The more general case is proven byrepeating the same proof for the input-bound con-straints.We first show that if the model is feasible allfour conditions must hold. Summing the lower-bound

constraints, we get that (−1 +∑KOk=1LOk

)∑

ruryr0 �0.Since

∑ruryr0 > 0, the first condition follows. Sim-

ilar reasoning applies to the upper-bound constraintsproving that the second condition must hold. Thethird condition is attained by summing correspond-ing lower- and upper-bound constraints obtaining(LOk

−UOk)∑

ruryr0 �0. Finally, the fourth conditionholds since both variables and outputs are non-negative.Hence, all four conditions must hold for the problemto be feasible.

To prove the opposite direction, we refer to the ratioform of the model presented in Section 4. We showthat when all four conditions hold, we have at leastone feasible solution. But under these conditions theremust exist KO numbers, �1, . . . , �KO , LOk

��k �UOk,

such that �1 + · · · + �KO = 1. We may assumethat �k > 0, since Uk > 0 ∀k (Uk = 0 if and onlyif all the outputs involved are equal to zero. Forthis case, the corresponding constraint does not re-strict the variables and may be disregarded). But,O1, . . . , OK are mutually exclusive, and by definition∑

k (∑

r∈Okuryr0/

∑ruryr0) = 1; hence we get that

there exists at least one set of numbers u > 0 that sat-isfies

∑r∈Ok

uryr0/∑

r uryr0 = �k for k = 1, . . . , KO.This means that u satisfies the balance constraints ofsingle-level DEA–BSC model. However, a feasible so-lution must also satisfy the rest of the constraints inthe model, namely:

∑ruryrj /

∑ivixij �1, ∀j . Since∑

r∈Ok�uryr0/

∑r�uryr0 = �k for k = 1, . . . , KO and

for any � > 0,∑

r�uryr0 can take any positive value.This means that we can find � for which �u is a feasiblesolution for the DEA–BSC model. �

The multi-level DEA–BSC model is feasible if andonly if all the conditions in Theorem 1 for the single-level DEA–BSC model hold for all the sub-blocks ofthe balancing constraints in the hierarchical structure. Aformal presentation of this result is stated in Theorem2 below.

Theorem 2. The multi-level DEA–BSC model is fea-sible if and only if the following conditions hold forthe set of output-balance bounds, {(LOk

, UOk), k =

1, . . . , KO}, and the set of input-balance bounds,{(LIk

, UIk), k = 1, . . . , KI}, independently, ∀i ∈

{i� − 1 : Ki < �} (K−1 = 0, Ki = k0 + · · · + ki , i�0,and � representing the highest index of the input/outputcards in the data structure): (i)

∑Ki+1k=Ki+1Lk �1, (ii)∑Ki+1

k=Ki+1Uk �1, (iii) Lk �Uk ∀k = Ki + 1, . . . , Ki+1,and (iv) Uk �0 ∀k =Ki +1, . . . , Ki+1 (may equal zeroif and only if yr0 = 0 ∀r ∈ Ok for the output-balance

H. Eilat et al. / Omega 36 (2008) 895–912 911

bounds, or xi0 = 0 ∀i ∈ Ik for the input-balancebounds).

Proof. The proof follows from the same reasoning pre-sented in Theorem 1, applied here for all subblocks ofthe balancing constraints. �

Appendix D. Algorithm for generating the balancematrix

The formulation of the multi-level DEA–BSC modelrequires the generation of the balance matrix B. We usethe graph representation of the BSC structure, and applyon it the Breadth First Search (BFS) algorithm.

The BFS works as follows. At every intermediatepoint in the execution, it associates each node in thegraph with one of two states: marked or unmarked. Themarked nodes of the graph are known to be reachablefrom the source, and the status of the unmarked nodeshas yet to be determined. The algorithm refers to arc(i, j) as admissible if node i is marked and node j is un-marked. Initially, only the source node (C0) is marked.The algorithm marks subsequent nodes by examiningthe admissible arcs. The algorithm terminates when thegraph contains no admissible arcs. Fig. D1 presents theBFS search algorithm. The set LIST in the algorithmincludes all marked nodes that the algorithm has yet toexamine in the sense that some admissible arcs mightemanate from them. When the nodes in the set LISTare selected in a first-in-first-out (FIFO) fashion, we getthe BFS procedure. By-products of the algorithm arethe function pred that matches the direct predecessor toany marked node and the function order that records theorder in which the nodes were marked.

We define a measures×cards matrix with one row foreach measure, and one column for each card. The value1 is assigned to measures that are included in the card,and the value 0 is assigned to the other measures. Thematrix is built by climbing in the tree that representsthe card’s structure from the leaves upwards using thefunction pred. The measures × cards matrix that corre-sponds to Fig. 1 is given in (D.1). The columns in thematrix correspond to the cards C0, C1, . . . , C11.

C=

⎛⎜⎜⎜⎜⎜⎜⎜⎜⎜⎝

1 1 0 0 0 1 0 0 0 0 0 01 1 0 0 0 0 1 0 0 0 1 01 1 0 0 0 0 1 0 0 0 0 11 1 0 0 0 0 0 1 0 0 0 01 0 1 0 0 0 0 0 0 0 0 01 0 0 1 0 0 0 0 0 0 0 01 0 0 0 1 0 0 0 1 0 0 01 0 0 0 1 0 0 0 0 1 0 0

⎞⎟⎟⎟⎟⎟⎟⎟⎟⎟⎠

. (D.1)

Algorithm search;

begin

unmark all nodes in C;

mark node s=0;

pred(s):=0;

next:=1;

order(next):=s;

LIST:=s;

while LIST not empty do

begin

select a node in LIST;

if node j is incident to an admissible

arc (i,j) then

begin

mark node j;

pred(j):=i;

next:=next + 1;

order(j):= next;

add node j to LIST;

end

else delete node i from LIST;

end

end

Fig. D1. The BFS algorithm.

The measures×cards matrix C and the function predare used to generate the balanced matrix B. Let Ck−1

be the kth column of matrix C. Hence, the column C0

corresponds to the card C0. Since this card is not meantto be balanced against any other card, it does not havea corresponding column in B. Now, to generate thecolumns of B we use the following equations.

BkL = (−Ck + Cpred(k) · Lk) ⊗ Y0,

BkU = (Ck − Cpred(k) · Uk) ⊗ Y0. (D.2)

The sign ⊗ represents the entry-by-entry multiplication.An automatic version of this algorithm was imple-

mented in the MATLAB environment.

References

[1] Kaplan RS, Norton DP. Translating strategy into action: thebalanced scorecard. Boston, MA: Harvard Business SchoolPress; 1996.

[2] Charnes A, Cooper WW, Lewin A, Seiford LM. Dataenvelopment analysis: theory, methodology and applications.Massachusetts: Kluwer Academic Publishers; 1994.

[3] Stewart WE. Balanced scorecard for projects. ProjectManagement Journal 2001;32(1):38–53.

912 H. Eilat et al. / Omega 36 (2008) 895–912

[4] Charnes A, Cooper WW, Rhodes E. Measuring the efficiencyof decision making units. European Journal of the OperationalResearch 1978;2:429–44.

[5] Seiford LM. Data envelopment analysis: the evolution of thestate of the art (1978–1995). Journal of Productivity Analysis1996;7:99–137.

[6] Oral M, Kettani O, Lang P. A methodology for collectiveevaluation and selection of industrial R&D projects.Management Science 1991;37(7):871–85.

[7] Khouja M. The use of data envelopment analysis for technologyselection. Computers and Industrial Engineering 1995;28:123–32.

[8] Baker RC, Talluri S. A closer look at the use of dataenvelopment analysis for technology selection. Computers andIndustrial Engineering 1997;32(1):101–8.

[9] Cooper RG, Edgett SJ, Kleinschmidt EJ. Portfolio managementfor new products. Ontario: McMaster University; 1997.

[10] Cooper RG. Winning at new products: accelerating the processfrom idea to launch. 3rd ed., Cambridge, MA: PerseusPublishing; 2001.

[11] Baker N, Freeland J. Recent advances in R&D benefitmeasurement and project selection methods. ManagementScience 1975;21(10):1164–75.

[12] Hall DL, Nauda A. An interactive approach for selectingR&D projects. IEEE Transactions on Engineering Management1990;37(2):126–33.

[13] Martino JP. Research and development project selection. NewYork: Wiley; 1995.

[14] Golabi K. Selecting a group of dissimilar projects for funding.IEEE Transactions on Engineering Management 1987;EM-34(3):138–45.

[15] Bard JF. Using multicriteria methods in the early stages ofnew product development. Journal of the Operational ResearchSociety 1990;41(8):755–66.

[16] Lockett G, Hetherington B, Yallup P, Stratford M, Cox B.Modelling a research portfolio using AHP: a group decisionprocess. R&D Management 1986;16:151–60.

[17] Melachrinoudis E, Rice K. The prioritization of technologiesin a research laboratory. IEEE Transactions on EngineeringManagement 1991;38(3):269–78.

[18] Lauro GL, Vepsalainen PJ. Assessing technology portfolios forcontract competition: an analytic hierarchy process approach.Socio-Economic Planning Science 1986;30(6):407–15.

[19] Khorramshahgol R, Gousty Y. Delphic goal programming(DGP): a multi-objective cost/benefit approach to R&Dportfolio analysis. IEEE Transactions on EngineeringManagement 1986;EM-33(3):172–5.

[20] Souder WE. System for using R&D project evaluation methods.Research Management 1978;21:29–37.

[21] Cook WD, Seiford LM. R&D project selection in a multi-dimensional environment: a practical approach. Journal of theOperational Research Society 1982;33(5):397–405.

[22] Newton DP, Pearson AW. Application of option pricing theoryto research and development. R&D Management 1994;24(1):83–9.

[23] Silverman BG. Project appraisal methodology: a multi-dimensional R&D benefit/cost assessment tool. ManagementScience 1981;27(7):802–21.

[24] Baker N, Pound WH. R&D project selection: where we stand.IEEE Transactions On Engineering Management 1964;11:124–34.

[25] Danila N. Strategic evaluation and selection for R&D projects.R&D Management 1989;19(1):47–62.

[26] Schmidt RL, Freeland JR. Recent progress in modeling R&Dproject-selection processes. IEEE Transactions on EngineeringManagement 1992;39(2):189–201.

[27] Henriksen AD, Traynor AJ. A practical R&D project-selectionscoring tool. IEEE Transactions on Engineering Management1999;46(2):158–70.

[28] Banker RD, Charnes A, Cooper WW. Some models forestimating technical and scale inefficiencies in data envelopmentanalysis. Management Science 1984;30(9):1078–92.

[29] Roll Y, Golany B. Alternate methods of treating factor weightsin DEA. Omega 1993;21(1):99–109.

[30] Charnes A, Cooper WW, Wei QL, Huang ZM. Cone-ratiodata envelopment analysis and multi-objective programming.International Journal of Systems Science 1989;20:1099–118.

[31] Thompson RG, Langemeier LN, Lee CT, Lee E, Thrall RM. Therole of multiplier bounds in efficiency analysis with applicationto Kansas farming. Journal of Econometrics 1990;46:93–108.

[32] Thomas RR, Barr RS, Cron WL, Slocum JW. A process forevaluating retail store efficiency: a restricted DEA approach.International Journal of Research in Marketing 1998;15:487–503.

[33] Roll Y, Cook W, Golany B. Controlling factor weights in dataenvelopment analysis. IIE Transactions 1991;23(1):2–9.

[34] Cook W, Roll Y, Kazakov A. A DEA model for measurementthe relative efficiency of highway maintenance patrols. INFORS1990;28(2):113–24.

[35] Dyson RG, Thanassoulis E. Reducing weight flexibility indata envelopment analysis. Journal of the Operational ResearchSociety 1988;39:563–76.

[36] Wong Y-HB, Beasley JE. Restricting weight flexibility in dataenvelopment analysis. Journal of the Operational ResearchSociety 1990;41(9):829–35.

[37] Kaplan RS, Norton DP. The balanced scorecard: measuresthat drive performance. Harvard Business Review,1992(January–February): p. 71–9.

[38] Kaplan RS, Norton DP. Using the balanced scorecard asa strategic management system. Harvard Business Review1996(January–February): p. 75–85.

[39] PMI, A guide to the project management body of knowledge.Project management institute; 1996.

[40] Roll Y, Cook W, Golany B. Controlling factor weights in dataenvelopment analysis. IIE Transactions 1991;23(1):2–9.

[41] Golany B, Phillips FY, Rousseau JJ. Models for improvedeffectiveness based on DEA efficiency results. IIE Transactions1993;25(6):2–10.