ann versporten, ingrid morales, carl suetens
DESCRIPTION
Scientific Institute of Public Health. Data validation study of the National surveillance of nosocomial infections in intensive care units. Ann Versporten, Ingrid Morales, Carl Suetens. IPH, wednesday seminar: May 7, 2003. Overview. Background: overview national surveillance ICU - PowerPoint PPT PresentationTRANSCRIPT
1
Ann Versporten,
Ingrid Morales, Carl Suetens
IPH, wednesday seminar: May 7, 2003
Scientific Institute of Public Health
Data validation study of the National surveillance of
nosocomial infections in intensive care units
2
Overview• Background: overview national surveillance ICU• Reasons for validation• Validation study
– Aims– Methods– Results
• Pneumonia• Bacteraemia
– Discussion– Conclusions– Recommendations
3
Background: National surveillance ICU
• 1996: Start National Surveillance of Hospital Infections (NSIH) : intensive care component (Pneumonia & Bacteraemia) – HELICS-based protocol (Hospitals in Europe
link for Infection Control through Surveillance)– patient-based surveillance: 1 file by patient, +
infection file if ICU-acquired PN or BAC– Nosocomial: infection acquired during hospital
stay (admitted >48h in ICU)
4
Background: National surveillance ICU
• Objective: to follow-up nosocomial-infection rates
• Risk-adjusted infection rates are used as external benchmarks for comparison purposes
5
Methods: Data collection for ICU surveillance
1. Data at admission
2. Day-by-day e.g. central venous catheter, mechanical
ventilation, antibiotic use
3. Infection data e.g. diagnostic criteria of PN, origin of BSI
4. Data at discharge
6
Reasons for validation
• Assessment of the validity of the findings
• Need to evaluate the accuracy of infection data reported to the NSIH program
7
Validation study
8
Main aim
• Validate reported ICU-surveillance data (ICU protocol: PN & Bac) against a reference gold standard
• Evaluate the accuracy of all data reported to the surveillance
• Evaluate the credibility of the surveillance
9
Specific aims
• Exhaustivity (completeness) denominator
• Sensitivity: probability of reporting a true PN & Bac to the ICU-surveillance
• Specificity: probability of reporting a PN & Bac as negative to the ICU-surveillance if the disease is truly absent
• Positive predictive value
• Negative predictive value
11
Methods - 1
• Sampling of hospitals: Systematic sampling of 45 hospitals on the
base of a list of hospital-trimesters
(ICU participation period 01/01/1997 – 31/12/1999)
• Replacement: later period accepted
• Informed consent, voluntary participation
• Retrospective chart review methodology
12
Methods - 2: Research program
Sampling of patient files: All reported PN+ & Bac+ (from surv.) All records with a positive hemoculture reported
on a laboratorium list (for all admitted patients on ICU) (estimation false-neg Bac)
A 20% random sample of the negative files (estimation of false-neg PN)
Estimation of exhaustivity of denominator on the base of administrative lists of ICU-admissions
13
Methods - 3• Calculation Se, Sp and Predictive values
“gold standard” = research team
• Trained data collectors (IPH) Application protocol definitions
validation: uniform & standardised evaluation = blind discrepant infections: reviewed by other
colleague
• Confidential & anonymous treatment of patient data
14
Methods - 4
• National results
• No individual hospital results, only discussion at end validation proccess Quality of dataQuestions
15
Results - 1
• 563 investigated patient files in analysis: pts staying >24h in ICU (23 hospitals)
• Infections reported by hospitals to surveillance: 147 Pneumonia 49 Bacteraemia
• Type of ICU: 91% polyvalent• Size of ICU: mean 10 beds • Length of stay: median = 4,7 days
16
Results - 2
• Exhaustivity of denominators:– For all patients staying >24h in ICU
72,8%
– For all patients staying >48h in ICU81,2%
17
Results - 3: Pneumonia
(106/133)*147=117.2(24/430)*1843=102.9
Validation+ - Total
Surv. + 117 30 147- 103 1740 1843
Total 220 1770 1990
Validation+ - Totaal
Surv. + 106 27 133- 24 406 430
Total 130 433 563
All PN inf.file &/or bdb Freq %
1 147 7.392 1843 92.61
Total 1990 100
Results of validation study for PN (inf. file &/or dbd) Results from surveillance for PN (inf. file &/or dbd)
Results applied on total sample(proportional balancing to files not been validated
18
Results - 4: Bacteraemia
Validation+ - Total
Surv. + 32 17 49- 22 1919 1941
Total 54 1936 1990
Validation+ - Total
Surv. + 32 12 44- 22 497 519
Total 54 509 563
All Bac inf.file &/or bdb Freq %
1 49 2.462 1941 97.54
Total 1990 100
Results of validation study for Bac (inf. file &/or dbd) Results from surveillance for Bac (inf. file &/or dbd)
Results applied on total sample(proportional balancing to files not been validated
19
Results: SE & SP
Se % (95% CI) Sp % (95% CI)
Pneumonia
Infection file 32,7 (25,2-41,2) 98,5 (97,4-99,2)
Inf.file &/or dbd 53,2 (43,5-62,7) 98,5 (97,4-99,0)
Bacteraemia
Infection file 48,1 (29,2-67,6) 99,3 (98,5-99,7)
Inf.file &/or dbd 59,3 (39,0-76,9) 99,1 (98,2-99,6)
20
Results: predictive values
PPV (%)(CI) NPV (%)(CI)
Pneumonia
Infection file 78,6 (65,6-87,9) 85,9 (83,5-87,9)
Inf.file &/or dbd 79,6 (68,3-87,8) 88,9 (86,8-90,8)
Bacteraemia
Infection file 65,0 (40,9-83,7) 97,3 (96,0-98,2)
Inf.file &/or dbd 65,3 (43,6-82,4) 97,3 (96,0-98,2)
21
Discussion - 1
• Exhaustivity denominator: improvement possible – risk of bias, e.g. if only high risk patients included
• Pneumonia: low Se., good Sp.
• Bacteraemia: low Se., good Sp.
22
Discussion – 2
• Possible reasons for lack of sensitivity– 30% of the results originate from 1997 (start
surv. NI in ICU). – 50% of the collected data correspond with the 3
first surveillance trimesters that hospitals participated to our ICU surveillance. = Explanation of lack of accuracy in the
interpretation of the protocol ?
23
Who are those missed patients ??
Why are there so many false negative Pneumonias ?
24
Characteristics false negative PN
Pneumonia N % mort.
mean length of
stay (days)
n ventilation
daysmean
SAPS II
median PN Risk Score
% with micro-org.
% pts with >=1 (other)
missing value
mean stay post infection
(days)
True + PN 93 24.7 20.0 7.1 42.6 47 84 8.6 13.9
False - PN 23 30.4 13.6 8.9 35.1 41 70 21.7 9.3
True - PN 1626 6.7 5.6 1.7 31.3 19 - 15.7 -
28
Factors influencing the Se. & Sp. of the infection data
• Who collects data ?
• Who decides whether a PN should be reported or not ?
• Criteria of bloodculture?
• Adherence to protocol definitions
• Degree of workload (ratio pat.-staff)
• Size of hospital
• …
29
Conclusions
• Exhaustivity varies for each hospital, but remains satisfactory in general
• Bac more accurately reported than PN (Se)
• Seldomly infections reported which were not a nosocomial infection (Sp)
• Absence of a gold standard ! (problem for diagnostic of PN)
30
Conclusions (next)
• Establishing Se & Sp only possible at the end of validation studyPreliminary conclusions:
Sensitivity rather low (identification of a NI through surveillance)Specificity is high (% files truly classified as non-NI)
Low Se has also been reported by the CDC: “The data collectors detected over 2,5 times as many PN, ..” (Emori, Edwards, et al. 1998)
31
Recommendations
• Training of professionals in charge of surveillance (Ehrenkranz, Shultz, et al. 1995)
case definitions (e.g. PN-diagnostic: use of micro-biologic reports & AB-administration)
surveillance-methods
• Simplification of protocol
• Development of electronic surveillance
32
Recommendations (next)
• Validation on continuous basis Training on the field Optimalisation contacts IPH / hospitals