does it matter where we publish?
DESCRIPTION
Does It Matter Where We Publish?. Adam Eyre-Walker University of Sussex. Reasons. Prestige peers employers Impact scientific community society. Measuring impact. General difficult to measure Citations Easy to measure Velvet Underground effect - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/1.jpg)
Does It Matter Where We Publish?
Adam Eyre-WalkerUniversity of Sussex
![Page 2: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/2.jpg)
Reasons
• Prestige• peers• employers
• Impact• scientific community• society
![Page 3: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/3.jpg)
Measuring impact
• General• difficult to measure
• Citations• Easy to measure• Velvet Underground effect
• “only sold 10,000 copies, but everyone who bought it formed a band” (Brian Eno)
![Page 4: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/4.jpg)
Question
Do papers of similar quality garner more citations in better quality journals?
![Page 5: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/5.jpg)
Ideal experiment
• Papers made anonymous• authors• journal• structure
• Assessed for quality by experts• Assessments confidential• Citations tracked
![Page 6: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/6.jpg)
Research Assessment Exercise (RAE)
• Each department submits• 4 publications per academic (75%)• 4 indicators of Esteem (5%)• Environment
• PhD student numbers• Grant income• Research structure• Facilities
• Superceded by REF
![Page 7: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/7.jpg)
RAE dataset
• 1170 papers published 2001 - 2007• Scored by AEW• Scored 1 to 4 (or unclassified)• Subjects : evolutionary biology, genomics,
bioinformatics, ecology, animal behaviour and organismal biology
• Ranked without reference to• journal• citations
![Page 8: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/8.jpg)
RAE scores
RAE score No. of papers
1 74
2 539
3 401
4 103
![Page 9: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/9.jpg)
Journal quality
• Impact factor• Number times papers published in the two
previous years are cited in target year
• Two components• Quality of papers• Effect of the journal on citations
![Page 10: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/10.jpg)
IF categories
IF category IF No of journals
No. of papers
1 IF < 2 41 532 2 < IF < 3 39 1343 3 < IF < 4 33 1594 4 < IF < 5 21 2145 5 < IF < 6 16 1286 6 < IF <10 28 1807 10 < IF < 20 13 1108 IF > 20 8 181
![Page 11: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/11.jpg)
Citations
• No. of citations from ISI Web of Science• October 2008
![Page 12: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/12.jpg)
Data
• For each paper• Year of publication• RAE score (RS)• No. of citations• Impact factor of journal (IF category)• Otherwise anonymous
![Page 13: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/13.jpg)
How good is the assessor?
• Correlation between AEW and co-scorers > 0.7
• Correlation between RS and no. of citations within each IF category and within each year
![Page 14: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/14.jpg)
How good is the assessor?
• Correlation between AEW and co-scorers > 0.7
• Correlation between RS and no. of citations within each IF category and within each year• Average r = 0.27 (p<0.001)• Average rmax = 0.73
![Page 15: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/15.jpg)
RS v citation correlation
r = -0.37p = 0.005
![Page 16: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/16.jpg)
Central question
• Do papers of similar quality accumulate more citations in some journals than others?
![Page 17: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/17.jpg)
RAE score 2 in 2003
1 2 3 4 5 6 7 80
5
10
15
20
25
30
35
40
45
IF category
Ave
rage
no.
of c
itatio
ns
![Page 18: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/18.jpg)
RAE score 2
p < 0.001
1 2 3 4 5 6 7 80
10
20
30
40
50
60
70
80
IF cat
Ave
rage
no.
of c
itatio
ns
![Page 19: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/19.jpg)
RAE score 3
1 2 3 4 5 6 7 80
20
40
60
80
100
120
IF category
Ave
rage
no.
of c
itatio
ns
p < 0.001
![Page 20: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/20.jpg)
ANCOVA
RS grade p-value 30 v 5
1 0.005 2.1 (1.7)
2 0.000 2.5 (0.3)
3 0.000 2.1 (0.4)
4 0.60 -
![Page 21: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/21.jpg)
ANCOVA
RS grade
p-value All30 v 5
IF<1030 v 5
1 0.005 2.1 (1.7) 1.2 (0.4)
2 0.000 2.5 (0.3) 3.0 (0.3)
3 0.000 2.1 (0.4) 5.4 (1.4)
4 0.60 -
![Page 22: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/22.jpg)
Alternative explanation
• Systematically underestimate quality of papers in high ranking journals
• Test• Assume no effect of journal• Consider correlation between RS and
journal (IF) controlling for citations
![Page 23: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/23.jpg)
Partial correlations
Year Correlation2001 0.54***2002 0.49***2003 0.54***2004 0.41***2005 0.31***2006 0.37***2007 0.35***
![Page 24: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/24.jpg)
ANCOVA
RS grade p-value 30 v 5
1 0.005 2.1 (1.7)
2 0.000 2.5 (0.3)
3 0.000 2.1 (0.4)
4 0.60 -
![Page 25: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/25.jpg)
Question
• Impact factors – relative contribution of• Paper quality• Journal effect
![Page 26: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/26.jpg)
Summary
• Assessor scores correlated to citations• strength correlated to age
• Assessor scores influenced by journal• Papers published in higher quality
journals accumulate more citations independent of quality
• F1000 data
![Page 27: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/27.jpg)
Question
• Where will we submit?
![Page 28: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/28.jpg)
Thanks
Nina Stoletzki
![Page 29: Does It Matter Where We Publish?](https://reader035.vdocuments.mx/reader035/viewer/2022062310/568162fa550346895dd37599/html5/thumbnails/29.jpg)
Assessor and journal
Assume citations measure quality no journal effect