publication bias in medical informatics evaluation research: is it an issue or not? mag. (fh)...
TRANSCRIPT
Publication Bias in Medical Informatics evaluation research:
Is it an issue or not?
Mag. (FH) Christof Machan, M.Sc.Univ-Prof. Elske Ammenwerth
Dr. Thomas Bodner
Overview
• Background• Detecting publication bias - 2 attempts
– study 1: A simplistic approach– study 2: The funnel plot
• Discussion• Conclusions / Outlook
Why think about publication bias
• evidence-based practice• systematic review as best evidence
– should include as many publications on the question invesigated as possible
– high danger of publication bias leading to wrong conclusions
• Strong evidence for publication bias in other fields (social sciences, biomedicine)
• What about evaluation literature in medical informatics?
Background
Results72 publications reported clear
results (positive or negative)– 60 positive– 12 negative
Anzahl positiver und negativer Arbeiten
60
12
0
10
20
30
40
50
60
70
postive Arbeiten negative Arbeiten
Study 1: Detecting publication bias - a simplistic approach
Positive / negative papers
positive negative
Detecting PB
AssumptionIf there is a remarkable difference between publications reporting positive and publications reporting negative results in a random sample publication bias is a possible reason. (see also Dickersin 1990)
Methods• random sample of 86 MI evaluation publications out of evalDB• Classify / Count / Compare
• graphical depiction of publication bias• easy to understand• scatter plot displaying study quality (e.g. sample size, standard error,...)
and effect size• recommended by Cochrane
Source: http://www.ukl.uni-freiburg.de/med/med8/seminar%2018%2011%2004.ppt#30
effect size
n
Relatives Risiko
3
2
1
,9
,8
,7
,6
,5
,4
,3
Fallzahl n
3000
2000
1000
500400300
200
100
504030
20
10
Publiziert
mehrfach
ja
nein
effect size
n
Detecting PB
Study 2: Detecting publication bias - the funnel plot - basics
Methods• study design: meta-analysis• Search in Medline and Embase for controlled trials on CPOE and medication errors / ADEs• Hand Search in several journals• intervention: CPOE system• effect size: relative risk for medication errors / ADEs between intervention and control group• 26 studies included• assessment of publication bias by funnel plot
Study 2: Effects of CPOE on medication errors / ADEs
Detecting PB
AssumptionMissing publications of low study quality reporting on negative and / or non-significant results may be an indication for publication bias.
Discussion
Study 1:• 60 out of 72 studies positive seems a high number• limitations:
– assumption that ratio of positive / negative studies equals 50:50 not valid
– other biases (e.g. langauge bias)– chance– ...
Discussion
Study 1:• plot indicates sign for publication bias• limitations:
– other selection biases (language, citation, etc.)– poor methodological quality of smaller studies– true heterogenity– chance