purchasing performance evaluation: with data envelopment analysis

European Journal of Purchasing & Supply Management 8 (2002) 123–134

Purchasing performance evaluation: with data envelopment analysis

Liane Eastona,*, David J. Murphyb, John N. Pearsonc

aSchool of Business Administration, Penn State Harrisburg, 777 W. Harrisburg Pike, Middletown, PA 17057-4898, USAbAir Force Institute of Technology, Graduate School of Logistics and Acquisition Management, Wright-Patterson AFB, OH 45433-7765 USAcDepartment of Business Administration, College of Business, Arizona State University, Campus Box 874706, Tempe, AZ 85287-4706, USA

Received 26 June 1997; received in revised form 20 August 2001; accepted 8 December 2001

Abstract

Because of supply chain management and other factors, purchasing’s performance is considered an important element of

corporate performance. Nonetheless, the measurement of purchasing performance, and comparing that performance to other

purchasing departments has proven to be very difficult. These difficulties stem from the lack of valid measurement criteria and

adequate methodologies to aggregate individual performance measures into a single index of overall performance. Many

methodologies are unable to account for the relative importance of performance measures, which varies among firms.

This paper examines the application of Data Envelopment Analysis (DEA), which has demonstrated potential as a management

tool to overcome the shortcomings of other techniques and to help purchasing executives improve the efficiency of their operations.

A DEA model was developed to compare the purchasing efficiency of firms in the petroleum industry. The model introduces

one more method by which managers can obtain information to assist in the decision making process. r 2002 Elsevier Science Ltd.

All rights reserved.

Keywords: DEA; Efficiency; Linear programming; Performance evaluation; Purchasing

1. Introduction

In the late 1980s and the early 1990s, purchasingstarted to gain increased recognition due to its directimpact on the final cost of the end product. Purchasedcomponents constituted over 55% of the sales dollar inmany manufacturing industries, which left purchasingwith the significant potential to reduce costs for thecompany.

Technological advances and economic pressurescaused purchasing’s influence to grow. Advancedtechnologies reduced labor and operating cost, whichincreased the percentage of purchased cost in every salesdollar. Economic pressures both domestically andabroad forced companies to reduce costs, allowingmanagement to realize the important role purchasingcan play in any competitive environment.

In more recent years, the concentration on corecompetencies coupled with business practices that focuson the supply chain, have afforded purchasing the

opportunity to play a larger strategic role in the firm(Carr and Smeltzer, 1999; Narasimhan et al., 2001). Thisfocus on core competencies and supply chain manage-ment significantly increases the number and importanceof external relationships. Firms that have recognized thestrategic importance of purchasing, now expect thisfunction to develop and maintain effective and efficientrelationships with suppliers. Purchasing is now lookedupon to lead the way towards analyzing the externalmarket, developing the appropriate relationships, andnegotiating contracts that will increase the profitabilityof the firm and the supply chain (Cox, 1996).Purchasing’s role in supply chain management includescommunicating with suppliers in an effort to decreaseredundancies and increase efficiencies along parts of thesupply chain (Wisner and Tan, 2000). Because ofpurchasing’s strategic impact, the rationale for measur-ing its performance becomes evident. Unfortunately, thepurchasing department is still one of the more difficultfunctional areas to evaluate (Chao, 1989; Hendrick andRuch, 1988; Malec et al., 1991; Millen, 1990; Murphy,1992; Stanley, 1992; Van Weele, 1984). One approach tothe evaluation of the purchasing department is tocompare that department’s performance with the

*Corresponding author. Tel.: +1-717-948-6161; fax: +1-717-948-

6456.

E-mail address: [email protected] (L. Easton).

0969-7012/02/$ - see front matter r 2002 Elsevier Science Ltd. All rights reserved.

PII: S 0 9 6 9 - 7 0 1 2 ( 0 2 ) 0 0 0 0 2 - 3

performance of other firms in the industry. However,there are two major problems with comparisons to otherpurchasing departments. First, purchasing executives inthe past could not compare departments becauseaggregated performance data was hard to obtain, andeven if it was attainable, methods that would provide aneffective comparison of overall performance did notexist. Second, many metrics in purchasing ignore theefficiency or productivity aspect of performance.1

Purchasing measures such as a supplier’s quality ignorethe amount of inputs (e.g. purchasing’s budget) that hadto be put forth in order to achieve that certain level ofoutput. Comparing one department to another on justthe outputs is an incomplete appraisal.

Data Envelopment Analysis (DEA) is a linearprogramming model developed by Charnes et al.(1978) which appears to have the characteristics thatwill enable development of an improved purchasingperformance evaluation system. The primary advantageof DEA is that it will establish a composite index ofoverall performance, allowing for easy comparisonamong departments or firms. DEA also providesinformation on the individual performance measure-ments that made up the aggregate score. This disag-gregated information enables the manager to take actionon specific measures. Moreover, DEA provides ameasure of efficiency (output/inputs), so that both inputsand outputs are accounted for, which makes for a morevalid comparison between purchasing departments.

This study materialized, in part, based upon aninitiative that took place at the Center for AdvancedPurchasing Studies (CAPS). CAPS had undertaken abenchmarking purchasing project and had gathered therequired data from the leading firms in selected industrygroups. Senior purchasing executives met as an ad hoccommittee in order to determine what specific datashould be collected as a basis for benchmarkingpurchasing performance in that industry (NAPM,1991). Each of the different industries, which were tobe benchmarked, established its own measurementcriteria. CAPS then collected the aggregate data, andfirms were able to compare performance against astandard for each purchasing performance measure.Even though the benchmarks were in a ratio analysisformat, the measurement evaluation was incomplete. Afirm could not establish where it stood relative to theother firms, but only where it stood with respect to acomposite standard (typically an average) for eachindividual purchasing performance measure. In addi-tion, there was no aggregate measure of overallpurchasing performance from which a firm couldcompare performance with other industry members.

The purpose of this research is to utilize DEA todevelop an evaluation model that overcomes some ofthe shortcomings of current techniques, and to explorethe potential of DEA as a management tool to helppurchasing executives improve the efficiency of theiroperations. There are two research objectives associatedwith this paper: (1) to design a DEA evaluation systemwhich improves on current purchasing evaluationsystems that compare performance among purchasingdepartments, (2) to establish where DEA fits within aframework of performance measurement.

2. Data envelopment analysis

DEA is a linear programming-based technique thatconverts multiple input and output measures into asingle comprehensive measure of productivity efficiency(Epstein and Henderson, 1989). DEA provides ameasure by which one firm or department can compareits performance, in relative terms, to other homogeneousfirms or departments. DEA is mainly utilized under twodifferent circumstances. First, it can be used when adepartment from one firm wants to compare its level ofefficiency performance against that of a correspondingdepartment in other firms. However, one major assump-tion is that all departments have similar strategic goalsand directions (Metters et al., 1999), which includes theneed for all the firms to be within the same industry.Second, DEA can be used in a longitudinal nature bycomparing the efficiency of a department or firm overtime.

In measuring the relative efficiencies of organizations,the DEA measurement can be defined as the ratio oftotal weighted output to total weighted input. WithDEA each organization can utilize different weights forthe set of performance measures. Weights are selectedthat will maximize the composite efficiency score foreach functional unit. This allows each unit to takeadvantage of their own unique areas of specialization(Sexton, 1986).

This variable weighting allows for the evaluation ofperformance while taking into account differences ingoals, responsibilities, and type of procurement.2 Therange of possible weights is controlled by requiring allweights to be positive, and specifying that if another unitused the same weight, their total efficiency score couldnot exceed the value of one. This ensures that all firmsare evaluated on the basis of relative efficiency. Thetechnique also gives information as to the specific effecteach input or output has on overall efficiency.

1Productivity is the measure of efficiency, which is the ratio of actual

output to actual inputs. For the purposes of this paper, efficiency and

productivity will be used interchangeably.

2 If there are major strategic differences in goals between organiza-

tions or departments (groups) then these groups should not be

compared directly under DEA, because DEA should only be used to

compare groups with similar overall strategies.

L. Easton et al. / European Journal of Purchasing & Supply Management 8 (2002) 123–134124

The information supplied by DEA can provide amajor advantage over benchmarking and other techni-ques where only one measure can be evaluated at atime, gaining no insight into overall efficiency. The singlecomposite measure of DEA allows for the rank orderingof all the firms in terms of their overall performance.

Originally, DEA was developed for the non-profitarena. DEA has been used in situations such as theidentification and measurement of hospital inefficiencies(Sherman, 1986); the assessment of national parksefficiencies (Rhodes, 1986); and the comparison ofresults of DEA techniques with accounting approachesnormally used to evaluate efficiencies in non-profitinstitutions (Ahn et al., 1989). The results of thesestudies are summarized in Table 1.

DEA has also been utilized in the private/for-profitsector. More specifically, DEA has been utilized in

supply chain management research. Weber and Desai(1996) employed it to construct an index of relativesupplier performance. Clarke and Gourdin (1991)applied DEA to the vehicle maintenance activities of17 separate maintenance shops of a large-scale, non-profit logistics system. Two of the studies demonstratehow DEA can be used in longitudinal studies todetermine the progress of one unit or department overtime. For example, Metzger (1993) used DEA toconduct a longitudinal study to measure the effectsof appraisal and prevention costs on productivity.Kleinsorge et al. (1991) utilized DEA to conduct alongitudinal monitoring process of one carrier in aneffort to assess expected performance improvementsover time. More details on these studies can be seen inTable 2. While these studies indicate that DEA can beutilized in supply chain management research, no one,

Table 1

Sample of DEA studies and major results

Author Application Contribution

Sherman (1986) Medical–surgical areas of seven acute care

hospitals

1. Able to identify inefficient units not previously identified by

regression or single ratio analysis

2. Tradeoff analysis of inputs/outputs

3. Locate sources of inefficiency

Rhodes (1986) Eighty national parks and historical sites 1. Able to answer policy questions regarding emphasis of proper

goals

2. Tradeoffs of staffing decisions

3. Tradeoffs of resource allocation

Ahn et al. (1989) Texas state university system 1. Able to account for multiple inputs and outputs

2. Demonstrate differences from single ratios and regression

3. Tradeoff ratios for alternative courses of action

Charnes et al. (1989) Texas electric cooperatives 1. Compared DEA, ratios, and regression

2. Reduced 670 individual ratios to one index

Table 2

Sample of DEA studies in supply chain management

Author Application Contribution

Weber and Desai

(1996)

Comparison of six suppliers’ performances

for a Fortune 500 Co.

1. Able to identify inefficient suppliers for the purpose of

negotiation leverage

2. Presented how parallel coordinates can be used to determine

which aspects of supplier’s performance needs improvement in

order to increase efficiency

Kleinsorge et al.

(1992)

Longitudinal study of one carrier 1. Able to demonstrate the use of DEA as a longitudinal measure of

a carrier’s efficiency and performance

2. Presented the use of intangible measures with DEA

Metzger (1993) Longitudinal study of the effects of quality on

productivity

1. Able to demonstrate the use of DEA as a longitudinal measure of

quality initiatives on productivity

2. Utilized step-wise regression to insure that only significant input

variables were used in DEA

Clarke and Gourdin

(1991)

Comparison of vehicle maintenance activities

of 17 maintenance shops

1. Able to identify inefficient vehicle maintenance facilities

2. Tested the perceived usefulness of DEA (53% of participants

found DEA a useful management tool)

L. Easton et al. / European Journal of Purchasing & Supply Management 8 (2002) 123–134 125

as of yet, has demonstrated its use as a potentialevaluation tool for purchasing performance, which isone of the objectives of this study.

3. Performance measurement

Performance measurement is defined as the process ofquantifying action, or more specifically the process ofquantifying and analyzing effectiveness and efficiency.Effectiveness is defined as the extent to which goals areaccomplished and efficiency is a measure of how well thefirm’s resources are utilized to achieve specific goals(Neely et al., 1995; Mentzer and Konrd, 1991). Thisstudy focuses only on the efficiency aspect of perfor-mance measurement.

The need for performance measurement in purchasinghas long been recognized. Performance measures drivethe actions of managers; therefore, the correct metricsare a critical element of a company’s performance. Earlyconceptual development of performance measurementin purchasing focused on cost issues with two majorconcepts emerging. The first was concerned with theproper utilization of purchasing personnel. The lessoverhead it took to perform the purchasing function, thebetter (Gushee and Boffey, 1928). The second dealt withend product cost as the ultimate measure of purchasingperformance (Lewis, 1933). The more that purchasingcould lower end product cost the more successful thedepartment was. The traditional measures, whichmainly consisted of costs and profits, continued untilthe 1980s (Ghalayini and Noble, 1996).

In the late 1980s and 1990s, traditional performancemeasurements were generally considered outdated forseveral reasons. These traditional measures have beenaccused of contributing to dysfunctional behavior,which in turn impairs the performance of the corpora-tion in the long-term (Neely et al., 1997). Performancemeasurement systems have been blamed of encouragingmanagers to strive for short-term gains while sacrificinglong-term profits, and of influencing managers toimprove their department’s performance to the detri-ment of another department’s performance. Morespecifically, traditional measures have been criticizedbecause they are: (1) based too much on financialmeasures and not enough on operational measures suchas quality; (2) incomplete or unidimensional; (3)contradictory to continuous improvement; (4)based on outdated cost accounting systems; (5)inflexible; (6) lacking integration with otherdepartments and strategic focus; and (7) invalid (Neelyet al., 1997; Ghalayini and Noble, 1996; Caplice andSheffi, 1994).

While DEA will not overcome a poor measurementsystem, it does provide the means to reduce or eliminatesome of the problems associated with the first four

concerns listed above. In other words, DEA coupledwith performance measures that promote sound beha-vior can reduce many of the problems associated withperformance measurement systems. How DEA can helpto improve a performance measurement system insupply chain management will be discussed next.

With DEA, both financial measures and operationalmeasures can be used together to obtain the finalcomposite index/aggregate efficiency score as long asnormalized data, such as indices, ratios and percentages,are not mixed with non-normalized data (Dyson et al.,2001). Therefore, the use of financial measures does notnecessarily preclude the use of operational measures andvice versa. Additionally, Kleinsorge et al. (1991) statethat one of the major advantages of DEA is thatperceived measures can also be used as long as theycan be quantified. However, Dyson et al. (2001) doescaution that this quantified data should be in interval orratio form.

DEA can also help to eliminate the use of unidimen-sional measures that may promote dysfunction beha-vior, because DEA allows for the use of multiple inputsand multiple outputs. These multiple inputs and outputsallow managers to develop a more comprehensivemeasurement system that promotes behavior that isof long-term benefit to the organization. DEA alsoprovides enough detail for managers to take actionbecause it provides an analysis of which individualperformance measures are lagging in comparison toother departments.

DEA can help measure a company’s level ofcontinuous improvement, because it can be used tomeasure a company’s progress over time. A DEAmeasurement can be taken each month or each quarterto assess whether or not productivity is improving.However, great care must still be taken to establishmeasures that motivate employees to improve asopposed to measures which just set norms (Ghalayiniand Noble, 1997). One example of how DEA can beused with continuous improvement is with the continualmeasurement of the effects of an ABC cost accountingsystem.

While DEA has little to do with cost accountingmethods directly, it can help to measure the progressbeing made with an activity based costing (ABC)system. ABC is a cost accounting method, and one ofits major objectives is to use a more accurate method ofassigning overhead cost to end products than traditionalcost accounting systems use. Traditional cost accountingmethods typically use direct labor to assign overhead toproducts, which many consider to be an outdatedmethod (Cooper, 1988a, b; Jeans and Morrow, 1989).3

Instead, ABC looks for more direct links between

3For a more detailed discussion on ABC, please see Cooper

(1988a, b).


overhead cost and certain activities.4 More importantly,one of the greatest assets of ABC, as quoted from aKPMG consultant, is that ABC forces managers tothink about how they can reduce the hidden cost oftransactions (Neely et al., 1995). If costs are reduced, theproductivity of the department is increased as long asthe output is not decreased. DEA, which measuresproductivity, can be used to access the success of anABC initiative over time. Moreover, the input part ofthe DEA equation is very similar to the sources of cost(overhead cost) used in ABC. Consequently, in terms ofthe input part of the equation very little if any extra datais needed above what has already been obtained forABC.

DEA can also be used in conjunction with a total costof ownership (TCO) program to evaluate suppliers.TCO’s aim is to determine the true cost of conductingbusiness with a particular supplier for a specific good orservice (Ellram, 1994). TCO goes beyond price toinclude cost associated with quality, delivery perfor-mance, and cycle-time. Many cost issues used in TCO,such as quality and delivery are operational measuresand are not normally measured in cost form. However,for TCO purposes, these measures should be convertedinto cost. This is difficult for some companies to do andthis is where DEA can be useful. As mentionedpreviously with DEA, financial figures and operationalmeasures can be mixed. Measures, which are typicallynot maintained in financial form, do not have to beconverted. Therefore, if the purpose of TCO is tocompare suppliers who provide the same product or toevaluate one supplier over a period of time to see if thereis improvement, then DEA may be a helpful tool to apurchasing department’s effort to conduct business withthe lowest total cost supplier.

In summation, DEA is a tool that can be used toreduce some of the existing problems with traditionalperformance measurement systems. Additionally, it canalso be utilized to measure the progress of manypurchasing and company initiatives such as ABC,continuous improvement, or TCO. However, DEA isonly as good as the measures that are utilized. Forexample, if short-term as opposed to long-term perfor-mance measurements are used then this will motivatemanagers to be short-term oriented. DEA is limited tocomparing like units. Therefore, even if you have severalpurchasing departments within the industry, but somecompete on low price while others compete on speed,then these two groups should be analyzed separately.

Another shortfall of DEA is that its only a measure ofrelative efficiency. Consequently, a separate measure-ment system will have to be maintained for a depart-ment or firm to measure effectiveness.

4. Research design

A DEA evaluation model, based upon the Charnes–Cooper–Rhodes (CCR) ratio model, was developed andused to compare the performance of purchasing depart-ments of major firms within the petroleum industry. TheDEA model is described in detail by several authors(Ahn et al., 1989; Banker et al., 1989; Sexton, 1986) andis shown in Appendix A. The model derives an efficiencyfrontier which provides an estimate of relative efficiencyfor each purchasing department in the industry set,using input and output variables.

The primary data used in this study were thesame types used to perform the 1991 report entitled‘‘Purchasing Performance Benchmarks For the USPetroleum Industry’’ (NAPM, 1991). The purchasingdepartments in the industry represent a relativelyhomogenous group of decision making units whichoperate under the same broad set of conditions, and arein agreement on the performance measures mostvaluable for comparison among peer firms. Eighteenof the most prominent firms participated.

These 18 firms provided the data required to developa DEA model for the evaluation of purchasingperformance. The 18 firms had a total of $454.63 billionin 1991 sales (85% of industry total), for an average of$25.26 billion per firm. Sales ranged from a low of $794million to a high of $116.94 billion. Neither the identityof the participating firms nor the specific data theyprovided for this study will be revealed. The data wereused to calculate benchmarks and DEA ratings for theparticipating firms. Benchmarks, data items, and thedata-collection instrument itself were validated througha review by all participating firms.

A general model of the purchasing function in thepetroleum industry (Fig. 1) was used to develop theDEA model. Eighteen potential measures (specificmeasurement factors identified in the CAPS benchmarkratios) were considered to represent the major inputsand outputs of the system. Firms were asked to rankthese measures in order to aid in final selection. Thosemeasures considered to be most important by thepurchasing executives in the study were included in thefinal DEA purchasing model.

4.1. Performance measures utilized in study

If purchasing is to play a strategic role in acorporation, then its goals and measures must bealigned with corporate strategic planning (Carr and

4For example, the salaries of purchasing personnel are considered

an overhead cost. This cost exists because the purchasing agents are

responsible for several activities to include the placement of purchase

orders. Hence, with ABC, the purchasing overhead would be assigned

to end product based on the number of POs placed (among other

activities) associated with the end product as opposed to the direct

labor that went into the end product.


Smeltzer, 1997). This link to strategic planning isaccomplished mainly through the use of departmentalperformance measures, which contribute to the corpo-rate goals. Two major factors, cost and technology drivethe petroleum industry’s corporate goals.

Because of the commodity structure, dismal refiningmargins, and increasing competitive retail markets ofthe petroleum industry, the low-cost producer has thecompetitive advantage (Rhodes and Koen, 1996).Hence, petroleum companies are seeking ways todecrease operating costs.

Additionally, petroleum companies are trying to gainadvantage by developing key technologies that willreduce the cost of exploration and production. Thesecompanies are forming alliances and mergers to acquirethe skills and technology necessary to compete effec-tively (McWilliams, 1997).

Therefore, in accordance with the industry’s goals toreduce cost through decreased operating cost andadvanced technologies, purchasing in turn must alsofocus its goals and measures toward reduced cost andimproved technologies.

The purchasing departments in the companies re-searched were responsible for maintenance repair andoperation (MRO) items and not the raw materials suchas oil. Many of the metrics that have been deemedessential in the 1980s and 1990s do not necessarilypertain to MRO items. For instance, while deliveryconcerns and quality are important in MRO items, thelevel of importance is not as great as it is for rawmaterial purchases. MRO items, besides capital equip-ment, cannot directly affect the quality of the product inmost cases because they do not become part of the endproduct. Delivery performance and quality are often not

tracked on MRO items because many of these items arenot purchased frequently enough to justify the measure-ment. With MRO items the major concern tends to becost.

Two outputs and four inputs were selected to measurethe relative efficiency of the 18 petroleum industrypurchasing departments:

Output One: Total purchase dollars by the purchasingdepartment (PURDLRS): This is the aggregate measureof the prime function of the purchasing department, thatis, the purchase of goods and services for the effectiveand efficient operation of the firm.

Output Two: Percent of total company purchasedollars handled by the purchasing department(PCTTOT): Benchmarks are reported on the percentof total company purchased goods and the percentof total company purchased services that are handledby the purchasing department. These variablesprovide an indication of the perceived importanceof the purchasing department within the corporatehierarchy.

Input One: Total purchasing operating expenses(OPEXP): This is a measure of both material andcapital utilization. The availability of supplies andfacilities reflected in this measure should facilitate thepurchase of supplies and services and the developmentof EDI/supplier relations.

Input Two: Total number of purchasing professionalemployees (PROF): The number of purchasing profes-sionals measures the available work force for perform-ing the purchasing function at each firm. This variableprovides a measure of the direct labor available topurchase supplies and services and to manage the supplybase.

OperatingExpense(OPEXP)

Professionals(PROF)

Administration/Clerical(ADMIN)

ActiveSuppliers(SUPPLIERS)

Purchase Dollars(PURDLRS)

Percent ofCompany Dollars(PCTTOT)

PurchasingProcess

Available Resources Major Outputs

Fig. 1. General model of the petroleum industry purchasing process.


Input Three: Total number of administrative purchas-ing employees (ADMIN): The number of administrativeemployees measures the available hourly workforce forperforming support functions needed in the purchasingdepartment (e.g. secretary, procurement clerk, etc.). Thisvariable provides a measure of the department’s abilityto streamline the procurement process and reducepurchase order processing time.

Input Four: Total number of active suppliers (SUP-PLIERS): This variable provides a measure of thesuppliers available to provide required goods andservices. This approach is particularly valid in light ofthe current efforts in purchasing to reduce the supplierbase while improving relationships with the remainingsuppliers. The goal is to reduce the total cost ofconducting business with the supply base.

The first three input measures represent the costsinvolved in operating the purchasing department.Monczka et al. (1979) conducted the most comprehen-sive known effort to develop information for use inimproving purchasing measurement. Two hundred andfifty purchasing practitioners were interviewed in 18 USorganizations. While they found ‘‘no best way ofmeasuring purchasing performance’’ they did establish13 purchasing measures, including ‘‘operating cost perdollar spent’’ to be an effective measure if employedconsistently. In a commodity type industry such aspetroleum, reducing operating cost can provide acompany with a great advantage over its competitors.

Input measure four, the number of active suppliers,has taken on greater significance in the last decade.Reducing the supplier base down to one or two suppliersper item has been associated with higher quality, reliabledeliveries, and overall lower administrative costs (Hand-field, 1993). In a study conducted in Britain, a reducedsupply base was associated with more partnership typerelationships rather than adversarial ones (Hosford,1994). A reduced supply base and/or partnershipsfacilitate the development of more advanced technolo-gies which has been deemed essential for the petroleumindustry. Because of all the positive aspect associatedwith fewer suppliers, a firm may find that the purchasingtask can be accomplished as well or better with fewersuppliers making ‘‘number of suppliers’’ an importantinput to measure.

5. Analysis

Seven of the benchmarks included in the 1991 CAPSstudy of the petroleum industry can be computed withthe six input and output values used in the DEAanalysis. These seven benchmarks were computed foreach of the 18 firms and are listed in Table 3.

The DEA analysis identified six petroleum firms,which were rated as one hundred percent efficient (1.0

rating). This efficiency is relative to the other firms in thesample. These firms could be identified as ‘‘best-in-class’’, for the petroleum industry sample. Ratings ofother firms ranged from a low of 0.12 to a high of 0.77.DEA ratings of overall purchasing performance for eachof the firms are shown in Table 4. Each was assigned anarbitrary three letter code in order to protect its identity.

In addition to the DEA rating of overall purchasingperformance, Table 4 also identifies the EfficiencyReference Set (ERS) for each of the firms ratedinefficient. These firms, and their associated multipliers,will form the Hypothetical Comparison Unit (HCU)which is used for comparing performance with theefficient frontier. Formulation of the HCU and analysisof each firm’s data is discussed in the section addressingthe DEA results.

5.1. Data envelopment analysis

DEA provides a comprehensive evaluation of overallpurchasing performance. This provides managementwith an advantage over comparing performance only onindividual measures. DEA also identifies a subset of thetop performers that could be designated as ‘‘best-in-class’’, which could be further studied to revealprocesses that lead to improved performance. In DEAthis is not an arbitrary classification. These firmsactually have achieved that performance level so it isknown with certainty that it can be reached.

The reason DEA can identify ‘‘best’’ in this fashion isthat it is an external method of analysis. It does notdepend upon the average of performance in the group inorder to derive either its ratings or its definition of‘‘best’’. DEA utilizes the extreme, or top performers intotal productivity (ratio of weighted output to weightedinput) to establish these values. It should be emphasizedthat DEA does not establish an absolute ‘‘best’’, butonly relative to the other purchasing departments thatparticipated in the DEA performance evaluation.

Another difference in the information DEA providesis due to the fact that aggregate ratings of performanceutilize a flexible weighting scheme. DEA establishes theweights that each firm uses to combine the individualinputs and outputs into a single index of relativeefficiency. Consequently, as the relative importance ofeach of the inputs and outputs varies, the weights usedby the purchasing department may also vary. In thisway, firm characteristics (such as functions, responsi-bilities, and goals/objectives) may be effectively ac-counted for.

Additionally, DEA provides insights into the magni-tude of potential improvement which other techniquescannot. Management can identify which inputs andoutputs may be altered, and by how much, in order toimprove overall relative efficiency. This can be done bycomparing the department’s input and output values


with those of the DEA generated HCU. The HCU iscomprised of a set of input and output values, which is aderived composite of the performance of the firms in theERS (see Table 4). Table 5 is an example of acomparison with the HCU for one of the inefficientfirms, GGG.

This analysis will enable the purchasing executive todetermine exactly which input or output levels can bechanged, and by how much, in order to improve relativeefficiency. For example, in the case of firm GGG, if allinput levels are reduced to the amounts indicated thenthe relative efficiency rating would improve to 1.0.

This HCU is derived as a linear combination of theinput and output values from the firms in the ERS. DEAidentifies the firms within the ERS and providesmultipliers (the variable value from the linear program’sdual) to be used in computing the HCU. Table 6provides an example of how the HCU is constructed forfirm GGG. Recall from Table 3, that firms AAA andDDD were identified as the ERS.

Table 3

Benchmark ratings

Firm A Purchase $

per employee

($M)

B Purchase $

per professional

($M)

C Purchase

expense per

purchase $ (%)

d Suppliers

PER employee

E Suppliers per

professional

g Purchase $

PER supplier

($K)

h Percent of total

expenditures

(%)

AAA 10.00 14.29 0.5 45.0 64.3 222.0 100

BBB 23.67 33.14 0.362 642.9 900.0 36.8 87

CCC 15.00 27.00 0.448 111.1 200.0 135.0 83

DDD 10.20 21.39 0.666 37.9 79.6 269.0 100

EEE 8.09 9.38 1.589 55.4 64.2 146.0 72

FFF 14.11 32.92 0.461 157.1 366.7 89.7 88

GGG 2.29 2.75 1.384 12.5 15.0 183.0 18

HHH 5.07 5.99 1.771 38.5 45.4 132.0 95

III 7.95 16.89 1.072 594.1 1262.4 13.4 21

JJJ 3.82 5.90 1.536 32.3 50.0 118.0 47

KKK 8.94 12.58 1.458 468.8 659.3 19.1 33

LLL 4.11 10.28 1.813 64.1 160.2 64.0 73

MMM 6.75 11.17 1.05 229.2 379.3 29.0 58

NNN 2.84 4.03 2.407 36.8 52.2 77.0 40

OOO 2.01 4.43 4.298 64.3 141.7 31.3 55

PPP 2.35 3.87 4.395 175.3 288.1 13.4 34

QQQ 2.09 3.37 3.447 154.8 249.2 13.5 26

RRR 0.97 1.75 7.206 29.9 53.8 32.0 12

Range 0.97/23.67 1.75/33.14 0.362/7.206 12.5/642.9 15/1262.4 13.4/269.0 12/100

Aver. 7.24 12.30 1.992 163.9 279.5 90.2 58

Table 4

DEA ratings

Firm DEA

rating

Efficiency reference set # Times in

effic. set

AAA 1.0 6

BBB 1.0 5

CCC 1.0 6

DDD 1.0 5

EEE 1.0 5

FFF 1.0 0

GGG 0.766784 AAA, DDD

HHH 0.757813 AAA, EEE

III 0.508976 BBB

JJJ 0.471756 AAA, DDD

KKK 0.448756 BBB, CCC, EEE

LLL 0.408673 CCC, DDD

MMM 0.406109 AAA, BBB, CCC, EEE

NNN 0.333977 AAA, DDD

OOO 0.182493 CCC, DDD

PPP 0.152564 BBB, CCC, EEE

QQQ 0.143133 AAA, BBB, CCC, EEE

RRR 0.120604 DDD

Table 5

GGG compared with its efficiency reference set

Outputs and

inputs

GGG:

actual

inputs/

outputs

Derived composite

of efficiency

reference set (AAA

and DDD)

Excess inputs

of GGG vs.

firms AAA

and DDD

PURDLR 27.5 27.5 0

PCTTOT 18 18 0

OPEXP 3806 0.156 0.23206

PROF 10 1.665 8.335

ADMIN 2 1.0630 0.937

SUPPLIER 150 115 35


Management judgment must be used when interpret-ing these hypothetical numbers. It is possible for theDEA program to produce numbers that are not feasible.For example, in some cases the HCU will indicate thatan increase in the output PCTTOT will increase relativeefficiency. That PCTTOT figure is sometimes >100%and it would not be possible to increase the variable tothat level. Also, it would be difficult to employ 1.665purchasing professionals (Table 6).

The DEA model provides management with addi-tional information that may help in selecting a course ofaction for improving performance. DEA assumes thatany point on the efficient frontier is feasible. Thisindicates that efficiency can be reached using combina-tions of inputs other than those indicated by the HCU.

6. Management implications

6.1. Management evaluation of DEA

Fifty-four purchasing executives were asked to com-ment on the acceptability of the existing performanceevaluation techniques and the desired characteristics ofan enhanced evaluation system. Responses showed thatmanagers desire characteristics that are not present incurrent methods of performance evaluation, but areincluded in the DEA evaluation.

Specifically, managers want to improve productivityand efficiency. They think an evaluation system thatexamines several inputs and outputs simultaneously isimportant, and they would like to see a system that takesinto account some of the different firm characteristics.Purchasing managers also believe that too muchemphasis is placed on single performance indexes.Finally, managers desire a system which provides timelyfeedback concerning resource utilization and efficiency.Participants believed that the DEA technique appears tosatisfy the most significant concerns of the purchasingmanagers. Its ability to allow flexible weighting andevaluate several inputs and outputs simultaneouslyaddresses the significant characteristics identified by

managers. The characteristics and feedback potential ofDEA make it a valuable tool for enhancing the qualityof decision making and improving purchasing depart-ment efficiency.

Even though ‘‘ease of use’’ was not assessed in thisstudy, Clarke and Gourdin’s (1991) DEA study on theefficiency of the logistics process surveyed the mangerson their perceptions of DEA. The managers felt that theresults were easy to comprehend, implying that abackground in operations research is not necessary tointerpret DEA.

6.2. Strengths of DEA

DEA provides additional management informationwhich enhances the decision making process. Purchasingexecutives will now be able to evaluate more alter-natives, and immediately see the effects of potentialchanges. DEA’s flexible weighting scheme also gives itthe capability to apply the evaluation measures equi-tably to all firms within the industry group.

The primary strength of DEA lies in two areas: (1) itsflexible weighting scheme which can accommodate thevarying importance of evaluation factors, and (2) itsability to provide more helpful information to managersfor use in improving performance. DEA provides asingle index of overall purchasing performance flexibleenough to account for the different firm characteristicsthat affect the required weights. DEA does a good job ofidentifying inefficiencies; however, it does not provide ameans for distinguishing among the firms rated efficient(1.0).

The information provided to management allowsthem to assess the potential improvement in relativeefficiency and develop several alternative courses ofaction. These alternatives may involve patterningbehavior after the HCU values. However, managersshould use DEA only as an initial indicator of anyproblems or inefficiencies. Further analysis into thesituation must be taken before any action is taken orany behavior modification is initiated (Epstein andHenderson, 1989).

Table 6

Derivation of HCU values, firm GGG

Output and inputs Dual variable for

firm AAA from

DEA

Output–input

vector of AAA

Dual variable for

firm DDD from

DEA

Output–input

vector of DDD

Composite: efficiency

reference set (HCU)

PURDLR 100.0 663.0 27.5

PCTTOT 100.0 18.0 18.0

0.163126 0.016874

OPEXP 0.5 4.413 0.156

PROF 7.0 31.0 1.665

ADMIN 3.0 34.0 1.063

SUPPLIER 450.0 2466.0 115.0


Because DEA provides a unique weighting scheme foreach firm, managers may also explore other possibleuses of resources as those weights indicate the marginalproductivity of each input and output in relation toincreases in relative efficiency. Management can thendetermine which resources will improve efficiency themost and determine exactly what the change inperformance will be.

Lastly, DEA is an excellent tool for measuring theprogress of one DMU (decision making unit). DEA canbe conducted in a longitudinal fashion to determine if aDMU is becoming more efficient. Instead of comparingone DMU to another, the same DMU is compared overtime.

6.3. Limitations of DEA

One potential problem in attempting to implement aDEA system is the availability of data. This researchwas possible because the benchmarking project at theCAPS had collected aggregate purchasing data fromfirms within industry groups. Consequently, most of thepetroleum firms in this study had collected benchmark-ing type data on a regular basis. This type of DEAevaluation system is not possible unless a group ofcomparable firms collect the data consistently, and on aregular basis, which could become an arduous task.Most firms would be unwilling to release such perfor-mance data unless an independent party monitors theDEA system and assures anonymity.

Interpretation of the weights established by DEAmust de done carefully. Management should notinterpret the weights to indicate a value, or measure ofimportance, of the individual performance measure(input/output). DEA establishes the weights in a purelymathematical fashion. The weights allow each firm tomaximize its efficiency rating subject to the constraintsof the problem. Thus, DEA calculates a technicalefficiency without regard to any potential social oreconomic value that each performance measure mayhave. It may be possible, in future research on DEA inpurchasing, to establish a range of possible weights foreach measure. DEA would then establish the technicalweights within the established value bounds.

DEA requires robust metrics, which are widelyaccepted measures, interpreted similarly by differentfirms or DMUs, and can be used for comparison acrossdifferent firms (Caplice and Sheffi, 1994). DEA has alsobeen thought of as a tool that can provide validmeasures because it incorporates multiple inputs andoutputs, providing validity over single performancemeasures because a greater number of metrics areutilized. The greater the number of measures, the morevalid the metric, because it is a more accuraterepresentation of the actual activity performed. Un-fortunately, DEA’s need for robustness is in direct

contrast with validity. In order to obtain true validity,some DMUs may want to incorporate idiosyncraticmeasures reflecting the actual activity of that particularDMU. Nonetheless, these unique measures are incontrast with DEA’s need for robustness. Summarily,DEA metrics can only achieve robustness at the expenseof less valid measures.

Another issue with validity and DEA arises becauseof the need for it to maintain its power to discriminate.As the number of inputs and outputs measuredincreases, which in turn increases validity, DEA’s powerto discriminate decreases. If a group of firms wish tohave both high validity in the measurement and stillallow DEA to discriminate between firms, then a highnumber of participants should be involved. Thesuggested ratio is 2m� s; where m� s is the productof the number of inputs and outputs (Dyson et al.,2001). Therefore, to maintain high validity and dis-crimination power the number of participating unitsshould be large. However, this increases the likelihoodthat all units are not homogeneous.

DEA is a measure of efficiency, not a measure ofeffectiveness. Therefore, an additional evaluation systemis needed to ensure that the company is effective inobtaining its goals. Additionally, DEA is a measure ofrelative efficiency and not absolute efficiency. Compa-nies that lie on the efficiency frontier may be lulled intocomplacency, which is a dangerous place to be in an everchanging business environment.

The final implication for management deals withimplementing DEA recommendations. DEA is only atool. It has no knowledge of the business environment ofthe firm. It cannot use judgment in calculating theevaluation. It is possible that certain values derived byDEA are impossible or undesirable to achieve in reality.In addition, management may choose corporate ordepartmental strategies that require the purchasingdepartment to operate in a relatively inefficient mode.For example, the firm may have embarked on a qualityimprovement program requiring increases in the numberof purchasing professionals employed. Management hasmade a conscious decision on the strategy; however, theDEA results may indicate that this is inefficient.Therefore, DEA only provides management withinformation to aid in decision making, and is not asubstitute for sound management judgment.

7. Future research

DEA has been shown to possess potential forimproving management’s ability to evaluate the perfor-mance of the purchasing department within a singleindustry group. It is not known whether the techniquecan adequately evaluate a less homogenous populationof purchasing departments. One approach to assessing


this capability is to apply a DEA evaluation system toseveral industry groups using the benchmarking typedata. In this way, the ability of DEA to comparepurchasing performance across industry groups could beassessed. If successful, purchasing executives would beable to assess their performance against a much widergroup of purchasing departments.

The ability to include a more diverse sample ofpurchasing departments would also serve to increase thesample size of any resulting study. This would allow forthe analysis of secondary factors in purchasing perfor-mance. Relationships between purchasing’s efficiencyand other firm characteristics such as sales, profits,company size, organizational structure, and industrygroup could be explored. If there is a relationshipbetween any of firm characteristics and purchasingperformance, the outstanding performers could beidentified more easily. They, in turn, could be furtheranalyzed to determine what specific practices andprocedures facilitate superior performance.

Evaluation of public sector’s purchasing departmentsusing DEA is currently being explored by researchingthe performance of Air Force base contracting offices.There are over one hundred of these purchasingorganizations. Each is required to collect the sameperformance data on a regular basis. This availability ofdata would make the analysis easier to accomplish,while at the same time greatly increasing sample size,therefore, enhancing the potential for further secondaryanalysis. Evaluation factors which are consideredimportant to Air Force contracting officers are currentlybeing identified. Many benchmarking type indicatorsare kept in the Air Force-wide data base. Onceappropriate factors are identified, a DEA model forthe Air Force base contracting function will bedeveloped and the performance data analyzed. Thatmodel will then be compared with existing evaluationtechniques and with evaluations previously performed inthe commercial sector. This information should proveextremely valuable in answering remaining questionsregarding the applicability of DEA.

Appendix A. The DEA model

This is the CCR ratio form of DEA, where there aren DMUs, m inputs, and s outputs.

Xij > 0 ¼ amount of input i used by DMUj ;

Yrj > 0 ¼ amount of output r produced by DMUj :

The decision variables in DEA are the unit weights tobe attached to each input and output by DMUj :

Vik ¼ unit weight on input i by DMUj ;

Urk ¼ unit weight on output r by DMUj :

Then, n fractional linear programs are formulated,one for each DMU. The objective function is the ratio oftotal weighted output divided by the total weightedinput. The unit being evaluated under each linearprogram is designated as DMU0.

MaxH0 ¼Ps

r¼1 Ur0Yr0Pmi¼1 Vi0Xi0

ð1Þ

subject toPs

r¼1 Ur0YrjPmi¼1 Vi0Xij

p1; j ¼ 1; 2;y; n

ðincluding DMU0 itselfÞUr0Pm

i¼1 Vi0Xi0Xe; r ¼ 1;y; s;

Vi0Pmi¼1 Vi0Xi0

Xe; i ¼ 1;y;m; ð2Þ

where e>0 represents a non-Archimedean constantsmaller than any real positive number, such that keon

for all real values of k and any real number n > 0:

References

Ahn, T., Arnold, V., Charnes, A., Cooper, W.W., 1989. DEA and

ratio efficiency analysis for public institutions of higher learning in

Texas. Research in Governmental and Nonprofit Accounting 5 (2),

165–185.

Banker, R., Charnes, A., Cooper, W.W., Swarts, J., Thomas, D.A.,

1989. An introduction to data envelopment analysis with some of

its models and their uses. Research in Governmental and Nonprofit

Accounting 5, 125–163.

Caplice, C., Sheffi, Y., 1994. A review and evaluation of logistics

metrics. The International Journal of Logistics Management 5 (2),

11–28.

Carr, A.S., Smeltzer, L.R., 1997. An empirically based operational

definition of strategic purchasing. European Journal of Purchasing

and Supply Management 3 (4), 199–207.

Carr, A.S., Smeltzer, L.R., 1999. The relationship of strategic

purchasing to supply chain management. European Journal of

Purchasing and Supply Management 5 (1), 43–51.

Chao, C.-N., 1989. Purchasing performanceFviews of purchasing

managers, buyers, and internal customers for different industries.

Unpublished Dissertation, Arizona State University, December.

Charnes, A., Cooper, W.W., Rhodes, E., 1978. Measuring the

efficiency of decision-making units. European Journal of Opera-

tional Research 2, 429–444.

Charnes, A., Cooper, W.W., Divine, D., Ruefli, T.W., Thomas, D.,

1989. Comparisons of DEA and existing ratio and regression

systems for effecting efficiency evaluations of regulated electric

cooperatives in Texas. Research in Governmental and Nonprofit

Accounting 5, 187–210.

Clarke, R.L., Gourdin, K.N., 1991. Measuring the efficiency of the

logistics process. Journal of Business Logistics 12 (2), 17–33.

Cooper, R., 1988a. The rise of activity-based costing systems: part 1-

what is an activity-based cost system? Journal of Cost Management

2 (2), 45–54.

Cooper, R., 1988b. The rise of activity-based costing systems: part 2-

when do I need an activity based cost system? Journal of Cost

Management 2 (3), 41–48.


Cox, A., 1996. Relational competence and strategic procurement

management: towards an entrepreneurial an contractual theory of

the firm. European Journal of Purchasing and Supply Management

2 (1), 57–70.

Dyson, R.G., Allen, R., Camanho, A.S., Podinovski, V.V., Sarrico,

C.S., Shale, E.A., 2001. Pitfalls and protocol in DEA. European

Journal of Operational Research 132, 245–259.

Ellram, L.M., 1994. A taxonomy of total cost of ownership models.

Journal of Business Logistics 15 (1), 171–191.

Epstein, M.K., Henderson, J.C., 1989. Data envelopment analysis for

managerial control and diagnosis. Decision Science 20, 90–119.

Ghalayini, A.M., Noble, J.S., 1996. The changing basis of performance

measurement. International Journal of Operations & Production


Ghalayini, A.M., Nobel, J.S., Crowe, T.J., 1997. An Integrated

Dynamic Performance Measurement System for Improving Man-

ufacturing Competitiveness. International Journal of Production

Economics 48 (3), 207–225.

Gushee, E.T., Boffey, L.F., 1928. Scientific Purchasing. McGraw-Hill,

New York, NY.

Handfield, R.B., 1993. The role of materials management in

developing time-based competition. International Journal of

Purchasing and Materials Management 29 (1), 2–10.

Hendrick, T.E., Ruch, W.A., 1988. Determining performance apprai-

sal criteria for buyers. Journal of Purchasing and Materials


Hosford, N., 1994. Partnerships-what’s really going on? Purchasing

and Supply Management, Jul/Aug 38–40.

Jeans, M., Morrow, M., 1989. The practicalities of using activity-based

costing. Management Accounting 67 (10), 42–44.

Kleinsorge, I.K., Schary, P.B., Tanner, R.D., 1991. The shipper-carrier

partnership: a new tool for performance evaluation. Journal of

Business Logistics 12, 35–57.

Kleinsorge, I.K., Schary, P.B., Tanner, R.D., 1992. Data envelopment

analysis for monitoring customer-supplier relationships. Journal of

Accounting and Public Policy 11, 357–372.

Lewis, H.T., 1933. Industrial Purchasing. Prentice-Hall, New York,

NY.

Malec, H.A., McClean, J.D., Crain, T.B., Goulet, J.M., 1991.

Benchmarking Purchasing in the Semiconductor Industry with

Sigma Barometers. CAPS and NAPM, Tempe, AZ.

McWilliams, G., 1997. Technology is what’s driving this business.

Business Week 3, 146–148.

Mentzer, J.T., Brenda, P.K., 1991. An efficiency/effectiveness ap-

proach to logistics performance analysis. Journal of Business

Logistics 12 (1), 33–62.

Metzger, L.M., 1993. Measuring quality cost effects on productivity

using data envelope analysis. Journal of Applied Business Research

9 (3), 69–79.

Metters, R.D., Frei, F.X., Vargas, V.A., 1999. Measurement of

multiple sites in service firms with data envelopment analysis.

Production and Operations Management 8 (3), 264–281.

Millen, A., 1990. Purchasing 2000: how effective is purchasing?

Purchasing 109 (6), 58–62.

Monczka, R.M., Carter, P.L., Hoagland, J.H., 1979. Purchasing

Performance: Measurement and Control MSU Business Studies.

Michigan State University, East Lansing.

Murphy, D.J., 1992. Purchasing performance evaluation: a data

envelopment analysis approach. Unpublished Dissertation, Arizo-

na State University, July.

NAPM, 1991. Purchasing performance benchmarks in the US

petroleum industry. National Association of Purchasing Manage-

ment, Tempe, AZ.

Narasimhan, R., Jayram, J., Carter, J., 2001. An empirical examina-

tion of the underlying dimensions of purchasing competence.

Production and Operations Management 10 (1), 1–15.

Neely, A.D., Gregory, M., Platts, K., 1995. Performance measurement

systems design: a literature review and research agenda. Interna-

tional Journal of Operations & Production Management 15 (4),

80–116.

Neely, A.D., Richards, H., Mills, J., Platts, K., Bourne, M., 1997.

Designing performance measures: a structured approach. Interna-

tional Journal of Operations & Production Management 17 (11),

1131–1152.

Rhodes, E.L., 1986. An exploratory analysis of variations in

performance among US parks. Measuring Efficiency: an Assess-

ment of Data Envelopment Analysis New Directions for Program

Evaluation. Jossey-Bass, San Francisco.

Rhodes, A., Koen, A.D., 1996. U.S. refiners find benefit in joint

ventures with foreign partners. Oil and Gas Journal 94 (30), 16–20.

Sexton, T.R., 1986. The methodology of data envelopment analysis.

Measuring Efficiency: an Assessment of Data Envelopment

Analysis New Directions for Program Evaluation. Jossey-Bass,

San Francisco, Winter.

Sherman, H.D., 1986. Managing productivity of health care organiza-

tions. Measuring Efficiency: an Assessment of Data Envelopment

Analysis New Directions for Program Evaluation. Jossey-Bass, San

Francisco, Winter.

Stanley, L., 1992. Purchasing performance: a structural model of the

effects of strategic supplier relationships and organizational

structure. Proceedings of the 1992 National Purchasing and

Materials Management Research Symposium. National Associa-

tion of Purchasing Management and California State University,

Fresno, March 12–14, pp. 55–70.

Van Weele, A.J., 1984. Purchasing control: performance measurement

and evaluating the industrial purchasing function. Netherlands

Wolters-Noordhoff, Groningen.

Weber, C.A., Desai, A., 1996. Determinants of paths to vendor market

efficiency using parallel coordinates representation: a negotiation

tool for buyers. European Journal of Operational Research 90,

142–155.

Wisner, J.D., Tan, K.C., 2000. Supply chain management and

its impact on purchasing. Journal of Supply Chain Management

36 (4), 33–42.


purchasing performance evaluation: with data envelopment analysis

Documents