benchmarking performance indices

8/11/2019 Benchmarking Performance Indices

1/15

Benchmarking performance indices: pitfalls andsolutions

John Maleyef

The Authors

John Maleyef, Lally School o Management and Technology, Rensselaer PolytechnicInstitute, Hartord, Connecticut, US

Abstract

Many organizations include benchmarking as a component o their perormancemanagement system. Oten, a perormance index is used to quantiy the ability o anorganizational entity to operate successully. Benchmarking a perormance index is doneinappropriately hen statistical methods are employed that ignore sample size efects oruse aggregate perormance data o!er a period during hich changes occurred ithin theorganization. Benchmarking ill also be inefecti!e hen in!alid targets are employed."hen benchmarking is done incorrectly, customer satisaction may actually decline dueto gaming and poor morale among employees. Based on the philosophy o ". #dards

$eming, the techniques o statistical process control %&'(), and basic undergraduatestatistics, a system is described or efecti!ely benchmarking a perormance index.#xamples are presented to illustrate the pitalls that exist in many perormancemanagement systems and to explain the system presented or efecti!e benchmarking.

Article type:*heoretical ith application in practice.Keywords: Benchmarking, &tatistics, &tatistical process control, +uality, 'erormance,$eming.Content Indicators: esearch -mplications 'ractice -mplications Originalityeadability

Introduction

Many organizations address some orm o a benchmarking process as an integral part otheir perormance management system. &uch benchmarking is oten perormed based oninternal quality system requirements %e.g. six/sigma or total quality management %*+M)).Benchmarking may be used, or example, to identiy those manuacturing cells thatachie!e consistently higher yields than cells making similar products, or to identiy callcenter operators ho perorm better or orse than their peers. #xternal requirementsalso moti!ate the need or the de!elopment o a perormance benchmarking system.*hese moti!ations may stem rom competiti!e pressures or rom requirements imposedby certi0cation or accreditation requirements, go!ernmental regulations, or customers.1or example, in the 2&3, *he Joint (ommission on 3ccreditation o 4ealthcareOrganizations %J(34O) requires a benchmarking process or healthcare acilities."ith the exception o manuacturing entities here measurement data are generally

used to analyze perormance to design or customer speci0cations, perormance indicesin the orm o a ratio or proportion are !ery common in most non/manuacturingorganizations. 1or example, the technology employed by call centers may automaticallyrecord, or each call, the abandonment rate o calls recei!ed, expressed as the ratio othe number o callers that hung up hile on hold di!ided by the number o callers. 3hospital may record the mortality rate or a certain disease category %ratio o the numbero deaths to the number o patients). -n manuacturing, this type o perormance index isalso used at times. 3 common example is yield %number o parts conorming tospeci0cations di!ided by the number o parts manuactured).*his paper extends the ork o "alsh %5666)ho addressed the de!elopment o targetsand ho to use statistical methods, including &'(, to assess perormance. 4oe!er,"alsh presented methods o analysis that required a perormance measure rather than aperormance index. Methods used to e!aluate measurements are typically not

appropriate hen dealing ith an index and, in act, are oten mis/applied in these cases.*his paper includes to main components. 1irst, key concepts that in!ol!e the analysis o
http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b27http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b27


2/15

a perormance index are pro!ided and pitalls o some commonly used statisticalapproaches are discussed. &econd, a system is outlined that pro!ides an efecti!emechanism or benchmarking organizational entities based on a perormance index.

Literature review

'erormance benchmarking is the merging o to methodologies, benchmarking andperormance management. Benchmarking has been de0ned as 7the search or and theimplementation o best practices7 %(amp, 899:, p. 8:), and includes the benchmarking oproducts and ser!ices, business processes, and perormance measures. *he goal obenchmarking perormance measures is 7to establish and !alidate ob;ecti!es or the !itale perormance measures that guide the organization7 %(amp, 899:,p. 8industry, based on entities in the same type o business>competiti!e, based on direct competitors> andprocess, based on dissimilar companies employing similar processes %#lmuti and?athaala, 899@).

"hen data are used as part o a benchmarking process, the ocus o the analysis is onthe 7gaps7 beteen the organizationAs data and the benchmark standard, oten ithoutregard or random !ariations %(amp, 899:, (hapter :).'erormance management has been de0ned as 7the use o perormance measurementinormation to efect positi!e change in organizational culture, systems and processes, byhelping to set agreed upon perormance goals, allocating and prioritizing resources,inorming managers to either con0rm or change current policy or program directions tomeet these goals, and sharing results o perormance in pursuing those goals7%'rocurement #xecuti!esA 3ssociation, 8999). 3uthors are careul to distinguishperormance management rom perormance measurement hich in!ol!es thede!elopment o metrics that quantiy the 7eciency and efecti!eness o action7 %Ceelyet al!, 899:).&ome authors ha!e been critical o either perormance management systems or

benchmarking systems. 1or example, perormance management systems ha!e beencriticized or the internal ocus on measures that may not correlate ith the satisactiono external customers %&indell and ?elly, 5666). *he balanced scorecard approachappears to ofer a solution by consolidating the !arious dimensions o perormance, bothinternal and external %Dautreau and ?leiner, 5668). 4oe!er, the de!elopment o directcause/and/efect relationships that link perormance measures ith organizationalsuccess remains a challenge %Mc?enzie and &hilling, 899E). 3nother methodology, qualityunction deployment, has been recommended to help con0rm that the customer!iepoint is being considered during product design %Fairi and Gousse, 899:) or ser!icesystem de!elopment %'un et al!, 5666). Benchmarking has also been criticized. &ome othe criticisms are similar to concerns regarding perormance management systems. 1orexample, 1reytag and 4ollensen %5668)mention the internal ocus, ithout a direct linkto customer satisaction, as a concern along ith the tendency o some organizations to

use benchmarking as a short/term solution to a problem rather than an ongoing process.Fairi and 3hmed %8999)listed cultural diferences as being a concern hen transerringbest practices in global organizations, and Bronlie %5666)addressed the diculty inefecti!ely obtaining useul inormation rom external entities.'erormance benchmarking has become a component o numerous certi0cation andaccreditation systems. 1or example, to be accredited by the J(34O, healthcareorganizations must implement a perormance management system based in part on theuse o a common set o perormance metrics %J(34O, 5666). Duidelines or the BaldrigeCational +uality 'rogram also include perormance benchmarking %C-&*, 5665). -ncon;unction ith this requirement, independent organizations ha!e e!ol!ed that alloparticipating entities to submit perormance data on a periodic basis, ith theorganization returning statistical reports that compare the data to suitable benchmarks.1or example, 'urdue 2ni!ersity has designed extensi!e questionnaires or call centermanagers to input their call center perormance data, hich has been consolidated into a
http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b3http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b3http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b3http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b7http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b7http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b3http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b22http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b19http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b19http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b19http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b19http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b25http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b9http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b18http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b29http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b23http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b23http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b23http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b23http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b8http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b28http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b2http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b11http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b20http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b3http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b3http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b7http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b7http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b3http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b22http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b19http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b19http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b25http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b9http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b18http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b29http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b23http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b8http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b28http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b2http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b11http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b20


3/15

database. -n return, call center managers get reports that compare their perormanceith all participants %http=HH.benchmarkportal.com).*o be efecti!e, perormance benchmarking must be statistically sound, must ensure thatperormance metrics are correlated ith customer needs, and must be part o acontinuous process that results in efecti!e management action. *he lack o astandardized system or perormance benchmarking stems rom the diference among

industries regarding the nature o the benchmarking process as ell as the complexity othe statistical methods in!ol!ed. 1or example, &hah and &ingh %5668) present arameork or perormance benchmarking o supply chains, that in!ol!es !arious metricskeyed to supply chain perormance. Maleyef et al! %5668a) present a system orbenchmarking healthcare acilities using metrics that relate to patient care. *hestatistical sophistication o these systems ranges rom no statistical analysis %here7gaps7 are generally interpreted as a shortall, ithout regard or random !ariation), tomore sophisticated methods such as data en!elop analysis %Madu and ?uei, 899E).

Implications for decision makers

"hen data are used to make decisions, managers must be aare that incenti!es ill be

created and that the incenti!es may conIict ith company mission or the intention o themanager. *he efect o management decision making processes as addressed by ".#dards $eming %8996/899). (onsider $emingAs 7system o proound knoledge7%$eming, 899, chapter K). *he our underpinnings o this system are=

a business system is a series o interconnected processes>e!ery process generates data that beha!e randomly>probability las can orm the basis o interpreting data> andemployees ill beha!e in !ery predictable ays depending in part on hodecisions are made in the presence o uncertainty.

*his last point, hich $eming reerred to simply as 7psychology7, is the source o manyorganizational problems especially hen statistical methods are implementedinappropriately.*hree examples ill be used to illustrate ho seemingly !alid and ell/intended eforts

on the part o management can back0re. -n these examples, a compensation system thatpro!ides incenti!e pay to employees based on the quality o their ork is implemented=

manuacturing cell orkers are rearded an extra bonus on days here the yieldor the cell exceeds 9: percent>dealers o a large companyAs products and ser!ices are ranked quarterly based onthe rate o complaints recei!ed and cash payments are made to managers i theyappear in the top 5: percent o dealers> andteachers in a school district are rearded extra bonus pay hen the standardizedtest scores or students in their class impro!es o!er the score o last yearAs class.

"hat is rong ith rearding employees or superior perormance and punishing themor sub/standard perormanceL -n $emingAs 8K points, he arns against the practice o7management by numbers7, hich can occur hen managers reard or punishemployees based on perormance data %$eming, 89E


4/15

spend an inordinate amount o time 7teaching to the test7 or outright helping students tocheat %*readay, 5666> 4artocollis, 5666).-n the end, management by numbers can cause more harm than good to quality andperormance. 1or example %) abo!e, i bonuses are rearded or ha!ing the loest rateo complaints recei!ed rom customers, it ould be dicult to imagine employees 0ndingays to make it easier or customers to register a complaint. -n the long term, the

company ould be better of i it ere made aare o unhappy customers beore theydeected to other companies, but the management system ould make this aarenessunlikely.

Key statistical concepts and potential pitfalls

*he type o data addressed in this paper is classi0ed as a proportion, de0ned as anumber beteen zero and one that contains a numerator %number o times the e!ent tooccurred) and a denominator %number o opportunities or the e!ent to occur). #xamplesinclude conormance rate o parts, complaint rate o customers, mortality rate opatients, and abandonment rate o callers. 1or proportions, the e!ent cannot occur morethan once or each opportunity. *hat is, a part is either conorming or nonconorming, a

customer complaints or doesnAt complain, etc. 3t times, other orms o data are assumedto be proportions. One case in!ol!es here it is rare or an e!ent to occur more thanonce per opportunity, or example, hospital/acquired inections per patient/day. "hile it isnot impossible or a patient to acquire to inections in one day, the chance o thisoccurring is remote and the rate o inections may be analyzed as i it ere a proportion.-t is possible to con!ert any orm o data to a proportion so that the methods described inthis paper ould apply. 1or example, data rom customer satisaction sur!eys may berecorded as the proportion o customers ho are 7satis0ed7 ith a ser!ice, here allcustomers ho chose 7good7 or 7excellent7 on their sur!ey are combined. Or, dataconsisting o the number o errors made hen processing a mortgage may be recordedas the proportion o mortgages on hich errors hen made during processing. 1inally,data consisting o a part measurement may be recorded as the proportion o parts thatconormed to speci0cations %i.e. the production yield).

"hen analyzing a perormance index, it is common or the analysis to ocus on theproportion ithout regard to the statistical efect o the sample size %e.g. number oparts, number o calls, number o patients). *his !alue appears as the denominator o theindex. 1or example, hen comparing yields o!er a group o manuacturing cells makingthe same product, a simple comparison o their yields is Iaed i the number o partsmanuactured in each cell is not taken into account. 3 comparison o hospital mortalityrates ould be Iaed i the number o patients ser!ed ere not considered as part o theanalysis.

Illustrative example

3 simple example ill be used to illustrate the concepts and methods described in theremainder o this paper. *he example is simplistic in orm so that the ocus on underlyingconcepts can be done efecti!ely. *he concepts illustrated, hoe!er, are applicable toany real orld situation here a perormance index is expressed as a proportion. 3ssumethat tenty indi!iduals are gi!en one coin each and told to Iip their coin a speci0ednumber o times. - all o the coins ere balanced, 7heads7 ill be shon on :6 percent othe tosses. "e do not kno, hoe!er, that e!ery person has been gi!en a balanced coin.*hat is, one or more persons in the group may be gi!en an unbalanced coin thatgenerates 7heads7 at a rate that difers rom :6 percent. O the 56 people, ten are askedto Iip their coin 866 times and ten are asked to Iip their coin :66 times. #ach person isasked to count the number o times their coin shoed 7heads7 and, upon completion otheir tosses, to calculate their proportion o 7heads7. 3 computerized random numbergenerator as used to simulate this exercise, ith the summary results pro!ided in*able-.

Improper approaches
http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b26http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b10http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#1310100101010.tifhttp://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#1310100101010.tifhttp://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b26http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b10http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#1310100101010.tifhttp://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#1310100101010.tif


5/15

3n analyst presented ith the data shon in *able -may be asked to identiy thoseentities that operated in a ashion inconsistent ith an appropriate target, hich in thiscase could be de0ned as a :6 percent probability o obtaining 7heads7. 3lternati!ely, i a0xed target ere not a!ailable, an analyst may ish to identiy those entities thatoperated in a ashion inconsistent ith the other entities. One type o in!alid approach tothis analysis ould be based on here each person ranked compared to the other

persons in the study. *ypically, this approach ould consist o the de!elopment o apercentile score or each entity or the assignment o each entity to a quartile %the loest5: percent o entities, second loest 5: percent o entities, etc.). *able -- pro!idespercentile scores and quartile assignments or each person in the exercise. Based on thisinormation, an analyst may decide that the persons in the loer or upper 86 percentilerange appear to ha!e unbalanced coins. -n this case, those persons in the outerpercentiles ould be persons 8< and 89 %too e 7heads7), and persons 8@ and 8 %toomany 7heads7).3nother popular, but equally improper, approach is the calculation o the means andstandard de!iations o the perormance index, the de!elopment o a inter!al extendingto standard de!iation units abo!e and belo the mean, and the highlighting o entitiesthat all outside this inter!al. -n this case, the mean proportion o 7heads7 is :6.K percent%the a!erage o the data contained in*able -) and the standard de!iation is K.: percent

%the standard de!iation o the data contained in*able -). 2sing this approach, all thosepersons alling outside o the inter!al extending rom K8.K percent to :9.K percent %:6.Kpercent9.6 percent) ould be likely candidates or ha!ing the coins that difered romthe norm. 4ence, person 8< ould be considered likely to ha!e an unbalanced coin thatresults in an unusually lo probability o 7heads7.*here are at least to problems ith the approaches discussed abo!e. *he 0rst problemis that it is inappropriate to compare proportions hen the sample size !aries acrossentities. &tatistical theory, as ell as common sense, dictates that the larger the samplesize, the closer the resulting proportion ill estimate the ability o a process to perorm.*hat is, the more the coin is tossed, the closer the resulting proportion ill be to theactual probability that the coin tossed is 7heads7. &o, reerring to person 8


6/15

3s a result o this phenomenon, it is likely that hen a perormance index is analyzedithout taking sample size into account, either=

an entity ith a smaller sample size is identi0ed as atypical e!en though it is

operating in a ashion that is consistent ith other entities> or

an entity ith a large sample size is identi0ed as typical, e!en though it is

operating in a ashion that is inconsistent ith other entities.

Importance of statistical control

Oten, indi!iduals trained in classical statistics ha!e diculty applying the techniquesthey kno to make efecti!e business decisions. -n "ut o the Crisis %$eming, 89E


7/15

4e belie!ed that these practices led to ad!ersarial relationships, since most o theproblems in an organization are caused by the 7system7 rather than the indi!idualemployees. 4ere again, $eming addresses the allacy o assuming that employees ha!esigni0cant direct control o!er the processes they ork ithin. 2nless undamentalchanges are made to the processes, ho can e expect perormance to impro!eL -t isnot dicult to anticipate the reaction o many orkers to improper targeting.

-n the context o managing perormance, the concept o 7comparing apples to oranges7must be a!oided. 1or example, in hospital administration, a key perormance index ismortality rate. *hese rates ha!e been published in nespapers and other publications inan efort to inorm consumers regarding hospital perormance. 4oe!er, it has beenshon that actors readily a!ailable or analysis are knon to afect mortality rates, andmust be accounted or hen comparing medical acilities %$ubois et al!, 89E@). 4oe!er,e!en hen these actors are accounted or, other actors not readily a!ailable such asmorbidity %magnitude o an illness) and co/morbidity %existence o more than one illness)ill also afect the mortality rate. *hese actors can only be analyzed by checking eachpatient record, hich is oten not practical. 4ence, there may exist some circumstanceshere perormance should not be compared across organizations.3!oiding an apples to oranges comparison is accomplished by ensuring that=

the perormance metric is de0ned and measured in a consistent ay>

the metric in!ol!es the same priority o customer ser!ice in each organizationcompared> and

organizations ould reasonably be expected to perorm similarly gi!en their

!ariety o customers, suppliers, location, etc.

1or example, benchmarking call center abandonment rate ould be appropriate only ithis metric ere de0ned and measured in a consistent ay, the call center perorms asimilar ser!ice in each comparison organization, and the !arious locations, sizes,customers, etc. do not preclude a reasonable expectation o similar perormance.

ummary of key principles

*o summarize the key points made to this point regarding the analysis o a perormanceindex or benchmarking purposes, the olloing principles ha!e been established=

Only organizational entities that are stable o!er the data collection period can becompared to a target or to other stable entities=

stable, or unchanged, entities ill generate perormance data that !ary o!er time>

i an entityAs perormance is not stable, then something changed during the data

collection period and the reason or the change should be determined.

"hen comparing an entityAs perormance index to a target or to other entities,random !ariation o the index, related to its sample size, must be taken intoaccount=

a perormance index cannot be compared ithout knoledge o the !alue o both

the numerator and denominator> the amount o random !ariation in a perormance index ill be in!ersely related to

the sample size %denominator) o the index.

"hen choosing a target against hich to compare an entityAs perormance index,care must be taken to choose a target that corresponds to the specialcharacteristics o the particular organization=

;ust because an entity difers rom a target does not necessary mean that a

problem exists or that an opportunity or impro!ement has occurred>

it may be possible to de!elop an ad;usted target based on the special

characteristics o an organization.

*he setting o targets, the method o comparing entities, and the reaction tobenchmark studies all impact ho employees ill beha!e ithin the organization=
http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b6http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b6http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b6http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b6http://tamino.managementfirst.net/vl=869689/cl=49/nw=1/rpsv/cw/mcb/14635771/v10n1/s1/#b6


8/15

i done improperly, benchmarking can lead to poor morale among employees, andmay cause orkers to act in ays that degrade, rather than impro!e, customersatisaction.

ystem for benchmarking proportions

*hree distinct steps must be incorporated into any system that compares a perormanceindex to a target or that compares perormance indices across organizations. -n thissection, each step in!ol!ing the analysis o a perormance index is described using thedata pro!ided earlier to illustrate the methods. 1or the sake o bre!ity, statistical detailsare kept to a minimum. *hus, readers ithout knoledge o statistical basics may needto re!ie other sources beore attempting implementation.

Analysis of statistical control

3 control chart is a requently used tool o statistical process control %&'() to determine ia manuacturing process is in statistical control. (ontrol charts are also applicable in non/manuacturing applications %Mac(arthy and "asusri, 5665). *he basic structure o a

control chart in!ol!es organizing the entire sample o data into subgroups according tothe time rame during hich the data ere collected. 1or example, data collected during&eptember ould be subgrouped into 6 daily increments. 1or each subgroup, asummary statistic is calculated, such as the proportion o calls per day that ereabandoned. #ach summary statistic is plotted on a display that shos the trend operormance o!er the data collected period. *hen, a center line that corresponds toa!erage perormance o!er the entire study period is added to the display along ith aset o upper and loer control limits. *hese control limits are calculated based on astatistical expectation that a stable process ill generate summary statistics that allithin the limits about 99.@ percent o the time.(ertain diagnostic rules are employed to determine, based on the pattern seen on thecontrol chart, i the process appeared stable %i.e. in statistical control). - the processere not stable, then the special cause o the process change ould be identi0ed and

acted upon. -n this ay, problems are identi0ed and opportunities or impro!ement arehighlighted. Only stable processes are predictable rom one period to the next. 4ence,only stable processes are eligible or comparison ith targets or ith other processes.*he type o control chart used to analyze proportion data is reerred to as a # chart.&tandard control limit ormulas or a#chart are presented in . *he#chart or person 8 inthe coin tossing example is shon as 1igure 8. -n this case, the :66 tosses ereorganized as 56 subgroups o 5: tosses each. #ach point plotted on the chart is theproportion o 7heads7 in each subgroup. - the process ere stable %i.e. the same coinas tossed in the same manner) statistical theory ould suggest that 99.@ percent osubgroup proportions ould be contained ithin the control limits and that, ithin theselimits, a random pattern consistent ith a normal %bell cur!e) pattern ould be expected.3 tutorial on#charts and their associated diagnostic rules is pro!ided by ?aminsky et al!%899@).-n 1igure 8, the center line corresponds to the a!erage proportion o 7heads7 %: percent,hich is also included in*able -) or person 8. *he upper control limit %E percent) andthe loer control limit %5 percent) represent the range o expected proportions or 5:opportunities o an e!ent hose probability o occurrence is : percent. Based on this #chart, it ould be reasonable to assume that the process de0ned by person 8 is stableo!er the data collection period, implying that the characteristics o the coin and ho itas tossed ere unchanged. *he data or each person included in the coin tossingexercise ould be plotted in the same manner.*he importance o the stability requirement is illustrated in the olloing example.(onsider the comparison o complaint rates, by eek, or to departments in a companythat pro!ide a similar ser!ice o!er a 5


9/15

existed o!er the data collection period. *he cost o this inaccurate analysis ould be to/old. 1irst, the company ould ha!e missed an opportunity to determine the cause o theimpro!ement experienced by department 8, hich could ha!e resulted in perormanceimpro!ement i implemented at department 5. &econd, the perormance o department 5has and ill continue to degrade, hich ill likely result ultimately in lost customers orthis company.

Performance comparison

*he target used to benchmark a stable process ill all into one o to categories. 3nabsolute target is one that is deri!ed ithout consideration o the process !ariation. *hatis, in order to be acceptable, each entity must meet some target le!el o perormance.#xamples include design speci0cations, regulatory standards, and goals set bymanagement corresponding to le!els that must be reached to assure competiti!eness. -nthe coin tossing exercise, a 0xed standard o :6 percent ould be used i the goal as todetermine those persons tossing unbalanced coins. *he second orm o targeting is theuse o a relati!e target. -n this case, the analyst ould attempt to highlight those entitiesthat are operating in a ashion inconsistent ith the other entities. elati!e targets are

used hen the standard o perormance is de0ned by other organizations that pro!ide asimilar product or ser!ice. #xamples include abandonment rate o callers and mortalityrate or a certain illness. -n the coin tossing exercise, a relati!e target ould be used ithe goal ere to identiy those persons ho appeared to ha!e coins that difered romthe coins used by the remainder o the group."hen comparing perormance o stable processes using data expressed as a proportion,the ocus becomes comparing the aggregate perormance o each stable entity %de0nedas the center line o the entityAs #chart) to a suitable target, hich ould also beexpressed as a proportion. 3ssuming stability or each person,*able -shos the centerline !alues %aggregate proportion o 7heads7) or the 56 participants. 3t this point, astandard statistical hypothesis test ould be used determine, or each entity, hether ornot their perormance difered rom a target proportion. *hese tests ould take samplesize into account.

1or proportion data, the appropriate hypothesis test is called a one/sample hypothesistest or a proportion %Berenson et al!, 5665, p. 59). *he test in!ol!es the calculation o astandard normal$/score that is compared ith an appropriate set o limits. - the $/scoreexceeds the limits, then it is assumed that the entityAs perormance difers rom thetarget. *hese limits are oten set at 5.66, hich corresponds to an approximate :percent risk that a process operating in a ay consistent ith the target is identi0ed asbeing inconsistent ith the target %idening the limits ould reduce this risk). -n thesetests, the perormance o an entity is only considered to be difering rom the target i theanalyst can be 9: percent con0dant that the diference exists.shos the normalized $/score calculation used to implement a one/sample hypothesistest or proportions. "hen implementing this test, an entityAs aggregate perormanceindex, aggregate sample size, and target proportion are needed. 1or this exercise, anabsolute target o :6 percent could be used, or alternati!ely, a relati!e target o :6.8

percent %the a!erage proportion o 7heads7 or the entire group) could be used.- the absolute target o :6 percent is used, the analysis or person 8 %:.5 percent7heads7 on :66 tosses) ould result in a $/score o 8.K. &ince this $/score does notexceed the 5.66 limit, e do not ha!e sucient e!idence to consider this personAs cointo be unbalanced. - the relati!e target o :6.8 percent ere used, the $/score or person8 ould be 8.9, supporting the assumption that the coin is similar to coins used by otherparticipants.*able ---pro!ides the$/scores or all 56 participants, using both the absoluteand the relati!e target. Only person @ is identi0ed as ha!ing an unbalanced coin %$/scoreo /5.@@). -t should be noted at this time that the random number generator used tode!elop the exercise assumed a balanced coin or all tossers except person @, hose coinas setup to generate 7heads7 only K: percent o the time.*he relati!e target o :6.8 percent as deri!ed based on a total o


10/15

perormers and thus inappropriate or comparison. &econd, the method is appropriatehen the total sample size %


11/15

special characteristics. 3n example o this type o system is the Maryland 4ospital3ssociationAs quality indicator pro;ect %.qipro;ect.org).

!iscussion and future research

*his paper highlighted opportunities and potential pitalls o using a perormance indexor purposes o benchmarking organizational entities. -t as shon that, hen notaccounting or process changes that may occur during a period o data collection,opportunities or impro!ement may be missed, problems my go unaccounted or, andresulting comparisons may mislead analysts. 3lso, entities that operate on a smallerscale run the risk o being identi0ed as difering rom targets, hen in act theirperormance is consistent ith their target le!el. 3lternati!ely, larger entities operating ina ay inconsistent ith a target ill tend not to be highlighted, e!en hen theirperormance !aried signi0cant rom the target le!el. 3s a result o improper analyses,quality ill likely be degraded, gi!en that ay employees are knon to react to arbitraryperormance management systems. *he system presented in this paper pro!ides anefecti!e rameork or properly comparing the perormance index o an organizationalentity to an appropriate target.

*he enhanced ability to collect and disseminate perormance data quickly andinexpensi!ely has alloed perormance benchmarking to e!ol!e to the point here themeans exists or most organizations to obtain comparison perormance data orbenchmarking either internal or external entities. -n con;unction ith these e!ents, manyindependent organizations, both or/pro0t and not/or/pro0t, are routinely obtaining datarom participants and returning statistical reports. 4oe!er, no standard approach existsor the analysis o perormance data in these systems. 3dditionally, examples can beound o incorrect statistical analysis using some o the pitalls described earlier in thisarticle. -n turn, the efecti!e interpretation and appropriate action ocus is likely to becompromised hen indi!idual managers ha!ing little or no statistical expertise arecharged ith the task o responding to these reports. 4ence, an important ocus o utureork ould in!ol!e the standardization o the data analysis routines ith a ocus onmethods that allo or reports to be easily olloed by most managers. -n addition,

managers must be more efecti!ely educated in the basic statistical methods employedithin these standard systems. Cot only ould this ork allo or more efecti!eperormance benchmarking, but ill also loer the resources necessary orimplementation. *his ould allo small and medium sized 0rms, as ell as otherorganizations lacking sucient resources, to bene0t rom perormance benchmarking.

%&uation '

%&uation (

%&uation )

%&uation *


12/15

%&uation +

%&uation

%&uation -

%&uation .

Figure 1# chart or #erson '

Table IResults o simulated e/ercise


13/15

Table IIPercentile and &uartile summaries

Table IIIHy#othesis test results

"eferences

Berenson, M.N., Ne!ine, $.M., ?rehbiel, *.(., 5665, Basic Business &tatistics= (oncepts and3pplications, Eth ed., 'rentice/4all, #ngleood (lifs, CJ.Bronlie, $., 5666, 70enchmar1ing your mar1eting #rocess7, Nong ange 'lanning, 5, 8,

EE/9:.(amp, .(., 899:, Business 'rocess Benchmarking, 3&+( +uality 'ress, Milaukee, "-.$eming, ".#., 89E


14/15

4artocollis, 3., 5666, 78 educators accused o encouraging students to cheat7, Ce Gork*imes, May, BK.Joint (ommission on 3ccreditation o 4ealthcare Organizations, 5666, (omprehensi!e3ccreditation Manual or 4ospitals= *he Ocial 4andbook, J(34O, Oakbrook *errace, -N.?aminsky, 1.(., Maleyef, J., 'ro!idence, &., 'urinton, #., "aryasz, M., 899@, 7Using SPC toanaly$e &uality indicators in a healthcare organi$ation7, Journal o 4ealthcare isk

Management, 8@, K, 8K/55.Mac(arthy, B.N., "asusri, *., 5665, 7 re4ie5 o non7standard a##lications o statistical#rocess control 9SPC: charts7, -nternational Journal o +uality eliability Management,89, , 59:/56.Mc?enzie, 1., &hilling, M., 899E, 74oiding #erormance measurement tra#s3 ensuringefecti4e incenti4e design and im#lementation7, (ompensation and Bene0ts e!ie, 6,K, :@/


15/15

Appendi# &% 'ne(sample hypothesis test for proportions

1or the entity, calculate A%the total sample size o!er all subgroups) as ollos=#quation

benchmarking performance indices

Documents