from biomedical research to health improvement grant lewison [email protected] and
DESCRIPTION
FROM BIOMEDICAL RESEARCH TO HEALTH IMPROVEMENT Grant Lewison [email protected] and [email protected]. e valua m etrics Evaluation of research through publication metrics. PUTTING BIOMEDICAL RESEARCH INTO PRACTICE. - PowerPoint PPT PresentationTRANSCRIPT
FROM BIOMEDICAL RESEARCH TO HEALTH IMPROVEMENT
Grant Lewison
evaluametrics Evaluation of research through publication metrics
PUTTING BIOMEDICAL RESEARCH INTO PRACTICE
evaluametrics Evaluation of research through publication metrics
Applications
for funds
New
vaccines
Better
diagnosis
New drugs
Less illness
incidence
Better
clinical care
Training of
researchers
Popular
opinions
Pollution
regulations
Lifestyle
choices
Physical
environment
Citations in
papers
Papers in
journals
Patents
Medical
education
Public health
campaignsGovernment
policy
Clinical
guidelines
Decisions
on funding
Conduct of
research
Papers at
conferences
Newspaper
articles
Clinical
trials
Economic
outputOther docs,
books
OTHER CITATIONS: CLINICAL GUIDELINES - 2
• Example: cancer clinical guidelines (BJC, 98, 1944)
• Sample of 43 guidelines (up to 2006)
• Comparison with world cancer research, 1999-2001 because peak PY of cited refs is then:
evaluametrics Evaluation of research through publication metrics
0
50
100
150
200
250
300
350
400
1980 1985 1990 1995 2000 2005Year
No
of r
efer
ence
s
OTHER CITATIONS: CLINICAL GUIDELINES - 3
• UK papers are over-cited by x3, but some other countries’ papers are also over-cited. DE and JP much under-cited:
evaluametrics Evaluation of research through publication metrics
0
0.5
1
1.5
2
2.5
3
UK DK AU IE SE CA IT BE FR NL NO GR US FI CH ES AT DE PT JP
Rat
io o
f % o
f gui
delin
e re
fs to
% o
f onc
olog
y re
sear
ch
OTHER CITATIONS: CLINICAL GUIDELINES - 4
• Comparisons can also be made of impact of papers from other countries, e.g., six Swedish medical schools:
evaluametrics Evaluation of research through publication metrics
0
10
20
30
40
GOT LUN UPP KAR LIN UME
% o
f all
Swed
en
% paps % CCGsGöteborg (GOT) papers are much better cited than ones from Karolinska (KAR), probably because the latter are more basic and the former more clinical
UTE MOU
CER
LIVBLA
PANPRO
OESLEU LYM
OVAMEL
STO
LUN
COL
MAM
0
5
10
15
20
0 5 10 15 20
% of UK cancer DALYs
% o
f ref
s on
UK
CC
Gs
OTHER CITATIONS: CLINICAL GUIDELINES - 5
What cancer sites are represented in papers cited by UK guidelines, and do they correlate with disease burden?
Data on burden for 2004 from WHO. Some cancers (e.g., lung - LUN, prostate - PRO, pancreatic - PAN) appear less among guideline refs than would be expected, and others (breast - MAM, melanoma - MEL) appear more.
evaluametrics Evaluation of research through publication metrics
evaluametrics Evaluation of research through publication metrics
OTHER CITATIONS: POLICY DOCUMENTS - 2
• Example of erpho report:January 2006 Issue 16 ISSN 1477-7304HIV and sexually transmittedinfections in the East of England
Report has 33 references of which 10 journal papers
• Example of report from No 10:Strategy Unit Alcohol Harm Reduction project.
Interim Analytical Report Nov 2003This report had over 200 references of which ~70 journal papers: mainly social sciences (NOT biomedical), and heavily from UK
evaluametrics Evaluation of research through publication metrics
OTHER CITATIONS: POLICY DOCUMENTS - 3
• UK Department of Health has many reports, e.g.:
Sport and Exercise MedicineA Fresh Approach
published May 2012, includes39 refs to journal papers• some only have a few refs, e.g.:
Improving Outcomes: A Strategyfor Cancer January 2011
has 36 refs of which 6 are journal papers, but this is a policy papersigned by Secretary of State so hasmore weight
evaluametrics Evaluation of research through publication metrics
OTHER CITATIONS: TEXTBOOKS - 2
• Three individual chapters analysed, with FR, IT and UK authors
• FR: The effect of cell-matrix interactions and aging on the malignant process (2007; basic; RL = 3.4)
• IT: Lung cancer: does age affect treatment strategy? (2009; clinical; RL = 1.0)
• UK: Supportive, Palliative and End-of-Life Care for Older Breast Cancer Patients (2010; clinical; RL = 1.1)
• Examination of their references: age, national distribution, citations
evaluametrics Evaluation of research through publication metrics
OTHER CITATIONS: TEXTBOOKS - 3
Cumulative %s of papers older than given delay
0
20
40
60
80
100
21 16 11 6 1Delay, years
Cum
ulat
ive
perc
ent
FR %
IT %
UK %
Some references cited by French author are very old (20% are 16+ years). Ones cited by UK author are the youngest – but result may not be typical
evaluametrics Evaluation of research through publication metrics
OTHER CITATIONS: TEXTBOOKS - 4
Percent of cited references from different countries
Each citing author gives preference to own country’s papers, but this is most extreme for France, which over-cites by x 8, more than UK (x 4) and Italy (x3), relative to presence in world cancer research
0
10
20
30
40
50
FR % IT % UK % US %Cited country
% o
f ref
eren
ces
FR IT UK
evaluametrics Evaluation of research through publication metrics
0
20
40
60
80
100
2 5 10 20 50 100 200 500 1000Cites in 5 yrs
% o
f ref
s ex
ceed
ing
So the cited references seem to be carefully selected as high-impact papers
OTHER CITATIONS: TEXTBOOKS - 5
• The references appear to be highly cited papers
• For the clinical papers cited by IT author, mean ACI in 5 years = 83 cites (cf. world mean in cancer of 22)
BBC WEBSITE COVERAGE OF CANCER RESEARCH
evaluametrics Evaluation of research through publication metrics
Story gives the date, the name of the journalist, the institution carrying out the work and the name of the journal – often also one of the leading researchers – so that the cited paper can be identified easily
evaluametrics Evaluation of research through publication metrics
stomach bladder lymphoma
oesophagus mouth
liver
uterus
pancreas
ovary leukaemia
skin
cervix
prostate colorectal
lung
breast
0.1
1
10
100
0.1 1 10 100
Percent of UK cancer DALYs (2002)
Perc
ent o
f BBC
can
cer
rese
arch
sto
ries
COMPARISON OF BBC AND DISEASE BURDEN
Note log-log scalesMany cancers are under-reported but this may reflect the paucity of research papers. Note relative popularity of breast cancer stories and those on skin and cervical cancer research