Why isn’t the customer Why isn’t the customer always right in academia? always right in academia?

The peer review culture and The peer review culture and the complication and the complication and irrelevance problems.irrelevance problems.

Michael Wood (Michael Wood (michael.wood@port.ac.uk))

Paper presented at Portsmouth Business Paper presented at Portsmouth Business School R&KT Conference, 19 June 2012School R&KT Conference, 19 June 2012

http://woodm.myweb.port.ac.uk/presentations.htm

The core products of universities are degree courses The core products of universities are degree courses and academic research published in journals. Unlike and academic research published in journals. Unlike most other products and services, the quality of both most other products and services, the quality of both courses and academic articles are reviewed mainly by courses and academic articles are reviewed mainly by peers, and only to a limited extent by customers. peers, and only to a limited extent by customers. (Student feedback on academic courses is obviously (Student feedback on academic courses is obviously collected, but this generally refers to the delivery not collected, but this generally refers to the delivery not the content – which is considered a matter for the the content – which is considered a matter for the experts, or peers, only.) This may, and often does, lead experts, or peers, only.) This may, and often does, lead to two types of problem from the perspective of to two types of problem from the perspective of customers. Firstly, academic knowledge may become customers. Firstly, academic knowledge may become unnecessarily complicated – peers, being expert in unnecessarily complicated – peers, being expert in their subject, are unlikely to see this as a problem their subject, are unlikely to see this as a problem which should be addressed. Secondly, there is a which should be addressed. Secondly, there is a danger that academic knowledge becomes danger that academic knowledge becomes increasingly irrelevant from the perspective of the end increasingly irrelevant from the perspective of the end user. Obviously, if customers become dissatisfied with user. Obviously, if customers become dissatisfied with the offerings of academia, they may seek equivalent the offerings of academia, they may seek equivalent services from other providers, so there is a strong services from other providers, so there is a strong argument that universities should try to tackle these argument that universities should try to tackle these problems. problems.

Academics are often Academics are often thought to tackle irrelevant thought to tackle irrelevant problems in complicated problems in complicated waysways• Irrelevance often not noticed Irrelevance often not noticed

because of the complicationbecause of the complication

• Complication may be justified by the Complication may be justified by the erroneous assumption of relevanceerroneous assumption of relevance

I think there is a lot of truth in this, and I think there is a lot of truth in this, and the problem is often peer reviewthe problem is often peer review

I think most of what I want to say is I think most of what I want to say is obviously right, but you may obviously right, but you may disagreedisagree

Please heckle as appropriate …Please heckle as appropriate …

A product or service can be A product or service can be reviewed by ... reviewed by ...

• Producer (Producer (but other stakeholders but other stakeholders likely to want a check!)likely to want a check!)

• Users or customers (via market or Users or customers (via market or otherwise)otherwise)

• PeersPeers

• Others Others

A restaurant may be A restaurant may be reviewed by …reviewed by …

• Users (customers)Users (customers)– IndividualsIndividuals– Experts (newspaper columnists, Michelin Experts (newspaper columnists, Michelin

inspectors) on behalf of customersinspectors) on behalf of customers

• Other stakeholders: e.g.Other stakeholders: e.g.– Animals (which might be eaten)Animals (which might be eaten)– The environment …The environment …

Peer review by other chefs only taken seriously in so Peer review by other chefs only taken seriously in so far as this is assumed to mirror customer reviewfar as this is assumed to mirror customer review

Academic courses are Academic courses are reviewed by:reviewed by:• Peers Peers

– Subject expertsSubject experts– Other teachersOther teachers

• Users or customersUsers or customers– StudentsStudents– Businesses (applied courses only)Businesses (applied courses only)

• Other stakeholders … ?Other stakeholders … ?

Course content typically reviewed mainly by subject Course content typically reviewed mainly by subject experts, but businesses may have some input.experts, but businesses may have some input.

Academic research – journal Academic research – journal articles – are reviewed by:articles – are reviewed by:

• Peers … Peers … only!only!

Academic culture seems to Academic culture seems to assume that:assume that:

• Experts (peers) know best Experts (peers) know best

• User / customer / other stakeholder User / customer / other stakeholder review on periphery only (e.g. review on periphery only (e.g. delivery of courses), but not content)delivery of courses), but not content)

But ….But ….

Academic knowledge is a Academic knowledge is a means to an end, and peer means to an end, and peer review might:review might:• Assume inappropriate ends from the Assume inappropriate ends from the

users’ perspective – users’ perspective – the irrelevance the irrelevance problemproblem

• Ignore other stakeholders Ignore other stakeholders

• Make the message too complicated for Make the message too complicated for the user, either in presentation or content the user, either in presentation or content or both – or both – the complication problemthe complication problem

For example …For example …

Who would be interested in Who would be interested in these articles?these articles?

For example …For example …See 4* Management journals: e.g. See 4* Management journals: e.g.

Management Science Management Science

British Journal of Management British Journal of Management

Compare these for usefulness and user-Compare these for usefulness and user-friendliness with two papers by A Einsteinfriendliness with two papers by A Einstein

Ist die Trägheit eines Körpers von seinem Energieinhalt Ist die Trägheit eines Körpers von seinem Energieinhalt abhängig?abhängig?

Zur Elektrodynamik bewegter KörperZur Elektrodynamik bewegter Körper

And what about the usefulness And what about the usefulness and user-friendliness of the and user-friendliness of the message?message?For example ….For example ….

… … in many cases I suspect real in many cases I suspect real simplification may be possible – see simplification may be possible – see http://woodm.myweb.port.ac.uk/nothard.pdf http://woodm.myweb.port.ac.uk/nothard.pdf andand http://www.timeshighereducation.co.uk/story.asp?http://www.timeshighereducation.co.uk/story.asp?

storyCode=171312&sectioncode=26 storyCode=171312&sectioncode=26 and …and …

Part of Table 2 in Glebbeek and Part of Table 2 in Glebbeek and Bax (2004)Bax (2004)

Could easily be more user Could easily be more user friendly:friendly:Results and curvilinear predictions for Region Results and curvilinear predictions for Region

1 and mean absenteeism and age (Model 4)1 and mean absenteeism and age (Model 4)

-50000

0

50000

100000

150000

200000

250000

0 5 10 15 20 25 30 35 40

Turnover (% per year)

Per

form

ance

Journals do pay lip service to Journals do pay lip service to usefulness and user-usefulness and user-friendlinessfriendliness

But users are not consulted for their But users are not consulted for their views. Why not …?views. Why not …?

Academia could produce Academia could produce more useful output. But:more useful output. But:

• Vested interests of peers in Vested interests of peers in preserving mystique of expertisepreserving mystique of expertise

• Within a paradigm it may be difficult Within a paradigm it may be difficult to see possibilities outside to see possibilities outside

• Little culture of simplicityLittle culture of simplicity– E.g. editor’s complaint that my paper was too E.g. editor’s complaint that my paper was too

simple: it did not provide “simple: it did not provide “complex solutions to complex solutions to common problems”common problems”

So …So …

• Infuse some customer / user / wider Infuse some customer / user / wider stakeholder review into the peer review stakeholder review into the peer review culture culture – Not arguing that needs of business should be Not arguing that needs of business should be

paramount – always multiple stakeholdersparamount – always multiple stakeholders

• Encourage reviews from multiple Encourage reviews from multiple perspectives (e.g. see suggestion at perspectives (e.g. see suggestion at http://woodm.myweb.port.ac.uk/PrJournalProbsAlts.ppt http://woodm.myweb.port.ac.uk/PrJournalProbsAlts.ppt and and http://www.timeshighereducation.co.uk/story.asp?http://www.timeshighereducation.co.uk/story.asp?storycode=411308 storycode=411308

The core products of universities are degree courses and The core products of universities are degree courses and academic research published in journals. Unlike most academic research published in journals. Unlike most other products and services, the quality of both courses other products and services, the quality of both courses and academic articles are reviewed mainly by peers, and and academic articles are reviewed mainly by peers, and only to a limited extent by customers. (Student feedback only to a limited extent by customers. (Student feedback on academic courses is obviously collected, but this on academic courses is obviously collected, but this generally refers to the delivery not the content – which is generally refers to the delivery not the content – which is considered a matter for the experts, or peers, only.) This considered a matter for the experts, or peers, only.) This may, and often does, lead to two types of problem from may, and often does, lead to two types of problem from the perspective of customers. Firstly, academic the perspective of customers. Firstly, academic knowledge may become unnecessarily complicated – knowledge may become unnecessarily complicated – peers, being expert in their subject, are unlikely to see peers, being expert in their subject, are unlikely to see this as a problem which should be addressed. Secondly, this as a problem which should be addressed. Secondly, there is a danger that academic knowledge becomes there is a danger that academic knowledge becomes increasingly irrelevant from the perspective of the end increasingly irrelevant from the perspective of the end user. user. Obviously, if customers become Obviously, if customers become dissatisfied with the offerings of academia, dissatisfied with the offerings of academia, they may seek equivalent services from other they may seek equivalent services from other providers, so there is a strong argument that providers, so there is a strong argument that universities should try to tackle these universities should try to tackle these problems. problems.