d colquhoun on how to get good science 2007

8/3/2019 D Colquhoun on How to Get Good Science 2007

http://slidepdf.com/reader/full/d-colquhoun-on-how-to-get-good-science-2007 1/3

Academics, like everyone else, areexpected to do a good job. They arepaid largely by taxpayers, andtaxpayers have every right to demandvalue for their money. The problem isthat it is very hard to measure thevalue of their output. Most of the ideasthat have made life as comfortable as itis in the affluent West have theirorigins in science departments inuniversities, but it isn't possible toplace a monetary value on, say, JamesClerk Maxwell 's equations of

electricity and magnetism, or onBernard Katz's work on synaptictransmission. Still less is it possible tomeasure the contributions of A EHousman, Stanley Spencer or Augustus John (all UCL people, as it happens).

This paper describes one example of what happens when universitieschange from being run by academicsto being run by managers. It describesan effect of corporatisation in themedical school of Imperial CollegeLondon, but the same trends arevisible in universities throughout theworld. The documents on which it isbased were sent to me after I'd written‘All of us who do research (rather thantalk about it) know the disastrouseffects that the Research AssessmentExercise (RAE) has had on research inthe United Kingdom: short-termism,intellectual shallowness, guestauthorships and even dishonesty ’(Colquhoun, 2007).

The problem is not so much the RAE

itself (the last one was done muchbetter than the assessment describedbelow), but rather it is the effect thatthe RAE has had on universitymanagers, who try to shape the wholeuniversity in their misperception aboutits methods. It is another example of Goodhart's law. The problem ariseswhen people with little understandingof scholarship, or of statistics, attemptto measure numerically things thatcannot be so measured. That is aplague of our age (Colquhoun, 2006),

but it is a process loved by politicians,‘ human resources’ people anduniversity managers.

Imagine how you would feel if youwere sent every year a spreadsheetthat showed your publication scoreand financial viability, and showedthese things for all your colleagues too.Well, you may say, there's nothingwrong with knowing how you aredoing. But imagine too that yourpublication score is entirelyautomated, with no attempt tomeasure the quality of what you aredoing. And imagine that if your grantsdon't cover your costs, you are in

danger of being fired. And imaginethat your meetings with seniorcolleagues consist of harassment aboutwhat journals you publish in, and howmany grants you have, not a discussionof your scientific aims. Not so good,you may think. But this is exactly whathas been happening at ImperialCollege Medical School.

Let's take a closer look at howacademics are being assessed.

Imperial's ‘publication score’The publication score that appearsalongside that of your colleagues iscalculated thus.

Multiply the impact factor of the journal by the author position weight,and divide by the number of authors.The author position weight is 5 for thefirst and last author, 3 for the secondauthor, 2 for the third author and 1 forany other position.

This index is clearly the invention of an

uninformed incompetent. That isobvious for a start because it uses theimpact factor. The impact factor is a(superfluous) way of comparing journals. It is the invention of EugeneGarfield, a man who has doneenormous harm to true science. Buteven Garfield has said:

‘ In order to shortcut the work of looking up actual (real) citation countsfor investigators the journal impactfactor is used as a surrogate to

estimate the count. I have alwayswarned against this use.’(Garfield, 1998)

Garfield still hasn't understoodthough. As the examples below show,the citation rate is itself a very dubiousmeasure of quality. Garfield quotesapprovingly:

‘ Impact factor is not a perfect tool tomeasure the quality of articles, butthere is nothing better, and it has theadvantage of already being inexistence and is therefore a goodtechnique for scientific evaluation.’(Hoeffel, 1998)

And you can't get much dumber thanthat. It is a ‘good technique’ becauseit is already in existence? There issomething better. Read the papers.

Try asking an impact factor enthusiastwhy it matters that the distribution of citation numbers for a given journal ishighly skewed, and you will usually bemet with a blank stare. One effect of the skew is that there is no detectablecorrelation between impact factor andcitation rate (see, for example, Seglen,1997; Colquhoun, 2003) . The easiestway to illustrate the numb-skullednature of this assessment is with a fewexamples.

Publication score versus citationTake a selection of 22 my ownpublications (the selection is arbitrary:it spans a range from 15 to 630citations and omits some of the dross).Fig. 1 A shows that the well-known lackof correlation between citations andimpact factor is true for me too. Fig. 1B

shows the same for the publicationscore.

The highest publication score (77.3)was for a two page perspective inScience , with a mere 41 citations(Sivilotti & Colquhoun, 1995). Asperspectives go, it was fine. But itseems that this was 7.2 times morevaluable than my best ever paper (onwhich I was recently asked to write aclassical perspective) which has apublication score of only 10.7 (but 565

citations) (Colquhoun & Sakmann,1985). My lowest publication score (inthis selection) is 2.08. That is for a

Physiology News | No. 69 | Winter 2007 | www.physoc.org

12 PN SOAPBOX

How to get good scienceDavid Colquhoun considers how a university can achieve the best research and teaching, and themost efficient administration – the aim of every university vice-chancellor (president, rector orprovost) in the country



Hawkes et al . (1992) , a mathematicalpaper which provides the methodneeded for maximum likelihood fittingof single channel recordings, withoutwhich most of my experimental workcould not have been done; its

mathematical difficulty may accountfor its modest number of citations (42)but its value for our work has beenenormous after the maths was put intoa computer program that can be usedby the semi-numerate.

Citations versus value: a real lifestoryThe dimwitted nature of thepublication score, and also of usingcitation rates, can be illustrated inanother way. Consider some of the

background to a couple of examples;these are the real life facts that areignored by bean counters.

Colquhoun & Sakmann (1981) got ascore of 73.2 and 278 citations. It wasa three page Nature letter, a first stabat interpretation of the fine structureof single channel openings. It wasn'tbad, but since Nature papers are soshort they mostly can't be thought of as real papers, and 4 years later wepublished the work properly in The Journal of Physiology (Colquhoun &

Sakmann, 1985), the result of 6 yearswork (57 pages, 565 citations). For thisImperial would have awarded me apublication score of a mere 10.7.

Here is another interesting case. If weexclude chapters in Single channel recording (Neher & Sakmann, 1983;1995) which apparently don't count,my most highly cited paper isColquhoun, Neher, Reuter & Stevens(1981). This has 630 citations and apublication score of 36.6 for me,

though only 14.6 for Harald Reuter.The reality behind this paper is asfollows.

In the early days of gigohm seal HaraldReuter decided that he wanted to learnthe method, and to achieve this heinvited three of us who already hadsome experience of the method tospend part of the summer vacation in

Bern. We had a wonderful summerthere, and being somewhatovermanned it was not very stressful. Itwould, I think, be fair to say that allfour of us did much the same amountof work. While recording we noticed atype of channel that was opened byintracellular calcium, like the calcium-activated potassium channel that wasalready well known in 1981. This onewas a bit different because it was notselective for potassium. We hadn'texpected to get a paper out of thevacation job but it seemed novelenough to write up, and 1982 being ayear when anything with ‘singlechannel’ in the title, however trivial,sailed into Nature, and because we hada limited amount of data, we sent itthere. Because we had all contributedmuch the same amount of work, weput the authors in alphabetical order.The analysis of the results, such as itwas, was crude in the extreme (papercharts unrolled on the floor andmeasured with a ruler). If we hadn'tseen this particular channel subtype,

someone else would have done with ayear or two. It just happened to be thefirst one of its type and so has beencited a lot, despite being scientificallytrivial.

This example shows not only theiniquitous uselessness of thepublication score used by Imperial; italso shows dramatically the almostequal uselessness of countingcitations.

How not to get Nobel prizesEmployees of Imperial Medical Schoolare told:

The divisional minimum benchmarksare ‘To have sufficient papers in toprated journals in the speciality toensure four publications in the RAEreview period (2001–2007) with anImpact Factor of at least 5 and withno overlap with co-authors from

Imperial’

The ‘productivity’ target forpublications is "To publish three papersper annum including one in aprestigious journal with an impactfactor of at least 5". Unfortunately,Dr X has published only two papers in2006 ...

Let's see who lives up to their‘productivity’criterion.

Take, for example two scientists whocommand universal respect in my ownfield – Erwin Neher and Bert Sakmann.They got the Nobel Prize for Physiologyor Medicine in 1991. In the 10 yearsfrom 1976 to 1985, Sakmannpublished an average of 2.6 papers peryear (range 0 to 6). In 6 of these 10years he failed to meet the publicationtarget set by Imperial, and thesefailures included the years in which theoriginal single channel paper waspublished (Neher & Sakmann, 1976)

and also the year when Colquhoun &Sakmann (1985) was published.

In 2 of these 10 years he had nopublications whatsoever.

On the other hand, a paper in 1981 in a journal with an ‘unacceptable’ impactfactor of 3.56 has had over 15000citations (Hamill et al . 1981). Thispaper would have earned for Sakmanna publication score of a miserable 0.71,

less than 100th of our perspective in

Science.

All this shows what is obvious toeveryone but bone-headed beancounters. The only way to assess themerit of a paper is to ask a selectionof experts in the field. Nothing elseworks. Nothing.

It seems to have escaped the attentionof bean counters that this is preciselywhat has always been done by goodgrant giving agencies and search andpromotion committees. Academicshave always been assessed. But beforeHR departments and corporate-

SOAPBOX

Figure 1. A, number of citations versus impact factor (log-log scales). B, number of citations versus ‘publication score’ (log-log scales).

PN 13




academics got involved, it was donecompetently. Now a whole branch of pseudo-science has appeared whichdevotes itself to trying to find ways of assessing people without bothering tofind out what they have done.‘Bibliometrics’ is as much witchcraft

as homeopathy. How long, onewonders, will it be before somebodycoins the term ‘bibliomics’? (Oops, aGoogle search shows I'm too late, somenumbskull has already done it).

How to get good scienceUniversities will have to decide whatsort of science they want.

They can bend their policies to everywhim of the RAE; they can bow to thepressures for corporatisation from the

funding council.

Or they can have creative scientistswho win the real honours.

They cannot have both.

If they want to have the latter they willhave to have universities run byacademics. And they will have to avoidcorporate and commercial pressures.They will have to resist the pressures toremove power from their bestresearchers by abolishing eminent

departments and centralising power ata higher level. We have seen what thisapproach has done to the NHS, but it isa characteristic of the corporatisingmentality to ignore or misuse data.They just know they are right.

It is also the box-ticking culture of managerialism that has resulted inapproval of BSc degrees in anti-science(Colquhoun, 2007). Impressivesounding validation committees tickall the boxes, but fail to ask the one

question that really matters: is what isbeing taught nonsense?

The policies described here will resultin a generation of 'spiv' scientists,churning out 20 or even more papers ayear, with very little originality. Theywill also, inevitably, lead to an increasein the sort of scientific malpractice thatwas recently pilloried viciously, butaccurately, in the New York Times, and a

further fall in the public's trust inscience. That trust is alreadydisastrously low, and one reason forthat is, I suggest, pressures like thosedescribed here which lead scientists topublish when they have nothing to say.

I wrote recently (Colquhoun, 2007) ‘Allof us who do research (rather than talkabout it) know the disastrous effectsthat the Research Assessment Exercise

has had on research in the UnitedKingdom: short-termism, intellectualshallowness, guest authorships andeven dishonesty’. Now we can add tothat list bullying, harassment and anincompetent box-ticking style of assessment that tends to be loved byHR departments.

This process might indeed increaseyour RAE score in the short term(though there is no evidence that itdoes even that). But, over a couple of

decades, it will rid universities of potential Nobel prize winners.

David ColquhounDepartment of Pharmacology,University College London, UK

This article is an extended version of comments published in the Times Higher Education Supplement on 1 June 2007.

References

Colquhoun D (2003). Challenging the tyrannyof impact factors. Nature 423 , 479.

Colquhoun D (2006). Playing the numbersgame. A book review, of Does measurementmeasure up? How numbers reveal and conceal

the truth by John M. Henshaw. Nature 442 ,357.

Colquhoun D (2007). Science degrees withoutthe science. Nature 446 , 373–374.

Colquhoun D (2007). What have we learnedfrom single ion channels? J Physiol 581,425–227.

Colquhoun D, Neher E, Reuter H, & Stevens CF(1981). Inward current channels activated byintracellular calcium in cultured cardiac cells.Nature 294 , 752–754.

Colquhoun D & Sakmann B (1981).

Fluctuations in the microsecond time range of the current through single acetylcholinereceptor ion channels. Nature 294 , 464–466.

Colquhoun D & Sakmann B (1985). Fast eventsin single-channel currents activated byacetylcholine and its analogues at the frogmuscle end-plate. J Physiol 369 , 501–557.

Garfield E (1998). http://www.garfield.library.upenn.edu/papers/derunfallchirurg_v101(6)p413y1998english.html.

Hamill OP, Marty A, Neher E, Sakmann B, &Sigworth FJ (1981). Improved patch clamptechniques for high resolution currentrecording from cells and cell-free membranepatches. Pflügers Arch 391, 85–100.

Hawkes AG, Jalali A, & Colquhoun D (1992).Asymptotic distributions of apparent opentimes and shut times in a single channelrecord allowing for the omission of brief events. Philosophical Transactions of the Royal Society London B 337, 383–404.

Hoeffel, C. (1998) Journal impact factors Allergy 53, 1225.

Neher E & Sakmann B (1976). Single channelcurrents recorded from membrane of

denervated frog muscle fibres. Nature 260,799–802.

New York Times. Seehttp://www.dcscience.net/improbable.html#clone

Seglen PO (1997). Why the impact factor of journals should not be used for evaluatingresearch. Br Med J 314, 498–502.

Sivilotti LG & Colquhoun D (1995).Acetylcholine receptors: too many channels,too few functions. Science 269, 1681–1682.


14 PN SOAPBOX

Sakmann in Göttingen, 1980. He andNeher did the work themselves.

Comments on this article can be left at DC’s Goodscience blog ( http://www.dcscience.net/goodscience/?p=4) or at DC’sImprobable Science blog (http://www.dcscience.net)

Note: the goodscience blog has now been merged and

everything is now at http://dcscience.net (but the linksgiven above will be forwarded to the right place)

d colquhoun on how to get good science 2007

Documents