Appendix 1. Supplementary methods and findings

Given the age of the study cohort of reviews, to ascertain the applicability of our conclusions and

proposed guidance to current practice, we examined systematic reviews of test accuracy published

in 2019. The aim of this post hoc assessment was to provide a brief overview of reviews published in

2019, focussing on test comparison methods and the reporting items highlighted in the guidance

proposed in Box 1. A summary of the search and findings is given below.

Search strategy, study selection and data extraction

We performed a search of MEDLINE (Ovid) on 31st July 2019 to identify test accuracy reviews

published between 1st January and 31st July 2019. Since PRISMA-DTA recommends that the title of a

diagnostic test accuracy (DTA) systematic review should identify the report as a systematic review

(meta-analysis) of DTA studies, we used the search strategy outlined below to obtain a snapshot of

reviews published in 2019.

1 systematic review.m_titl.

2 meta-analysis.m_titl.

3 accuracy.m_titl.

4 1 OR 2

5 3 AND 4

6 limit 5 to yr=”2019”

We applied the same eligibility criteria and selection process as described in the manuscript. One of

three assessors (YT, CP and CH) screened titles, abstract and, if necessary, the full text to determine

eligibility. Data extraction was performed by a single assessor (YT or CP). Any uncertainties were

resolved by the assessors through discussion without the need to involve another member of the

author team.

Search results

Of the 151 records identified by the search, 43 reviews assessed the accuracy of more than one

index test (Figure A1). Based on our terminology for classifying the reviews, 37 reviews were judged

to be comparative while the remaining 6 were multiple test reviews.

1

Figure A1. Flow of reviews through the selection process to identify eligible reviews published in 2019*The 37 comparative accuracy reviews met at least one of the following four criteria: (1) clear objective to compare the accuracy of at least two tests; (2) selected only comparative studies; (3) performed statistical analyses comparing the accuracy of all or at least a pair of tests; or (4) performed a direct (head-to-head) comparison of two tests.

2

Findings

The 43 reviews were published in 39 different journals—33 specialist and six general medical

journals. The number of tests evaluated in the reviews ranged between two and 17, with most

reviews evaluating two (18/43; 42%) or three (7/43; 16%) tests.

Test comparison methods are summarised in Table A1 and reporting characteristics in Table A2. In

this 2019 review cohort, 14% of the reviews restricted study selection to only comparative studies

(Table A1), similar to the proportion (13%) for the DARE cohort of reviews (Table 2). We found that

10% of the 2019 cohort included only direct comparisons compared with 13% for the DARE cohort

published between 2008 and 2012. The proportion of reviews using the recommended hierarchical

meta-regression approach for test comparisons was the same (14%) for both review cohorts.

Reporting of several items is still deficient in 2019 reviews (Table A2). For example, the test

comparison strategy was not reported in 76% (Table 4) of the DARE cohort and 74% (Table A2) of the

2019 cohort. Reviews published in 2019 still mainly relied on indirect comparisons yet such reviews

did not typically address the limitations of indirect comparisons; 69% of the DARE cohort and 57% of

the 2019 cohort did not acknowledge limitations.

Table A1. Strategies and methods for test comparisons in a cohort of reviews published in 2019Characteristic Comparative reviews Multiple

test reviews

TotalStatistical analyses to compare test accuracyYes No

Number of reviews* 20 (47) 17 (40) 6 (14) 43 (100)Number of tests evaluated

2 11 (55) 5 (29) 2 (33) 18 (42)3 4 (20) 3 (18) 0 7 (16)4 0 0 0 0≥5 5 (25) 9 (53) 4 (67) 18 (42)

Study typeComparative only 4 (20) 2 (12) 0 6 (14)Any study type 16 (80) 15 (88) 6 (100) 37 (86)

Test comparison strategyDirect comparison only 8 (40) 2 (12) 0 10 (24)Indirect comparison only – comparative studies available

7 (35) 11 (65) 0 18 (42)

Indirect comparison only – no comparative studies available

2 (10) 2 (12) 0 4 (9)

Both direct and indirect comparison 2 (10) 1 (6) 0 3 (7)Unclear 1 (5) 0 0 1 (2)None 0 1 (6) 6 (100) 7 (16)

Method used for test comparisonMeta-regression – hierarchical model 6 (30) 0 0 6 (14)Network meta-analysis† 1 (5) 0 0 1 (2)Univariate pooling of difference in sensitivity and specificity

1 (5) 0 0 1 (2)

Naïve (comparison of pooled estimates from 1 (5) 0 0 1 (2)

3

Characteristic Comparative reviews Multiple test

reviews

TotalStatistical analyses to compare test accuracyYes No

separate meta-analyses) using t-testComparison of area under the curve 4 (20) 0 0 4 (9)Overlapping confidence intervals 2 (10) 0 0 2 (5)Overlapping prediction regions 1 (5) 0 0 1 (2)Narrative 0 14 (82) 0 14 (33)None 0 3 (18) 6 (100) 9 (21)Unclear 4 (20) 0 0 4 (9)

NA = not applicable; SROC = summary receiver characteristic operating.*Numbers in parentheses are row percentages.†A review of 2 index tests stated that network meta-analysis was done.1 However, details were not given and no method was cited. The network plots showed the 2 index tests and the reference standard. Based on the available information it is unclear if this was an appropriate network meta-analysis.

Table A2. Reporting and presentation characteristics of a cohort of reviews published in 2019Characteristic Comparative reviews Multiple

test reviews

TotalStatistical analyses to compare test accuracyYes No

Number of reviews* 20 (47) 17 (40) 6 (14) 43 (100)Reporting guideline used† 13 (65) 9 (53) 2 (33) 24 (56)Clear comparative objective stated 14 (70) 8 (47) 0 22 (51)Role of the tests

Add-on 1 (5) 1 (6) 0 2 (5)Replacement 3 (15) 2 (12) 0 5 (12)Triage 1 (5) 3 (18) 1 (17) 5 (12)Add on or replacement 1 (5) 0 0 1 (2)Unclear 14 (70) 11 (65) 5 (83) 30 (70)

Flow diagram presentedYes – included number of studies per test 4 (20) 2 (12) 0 6 (14)Yes – excluded number of studies per test 16 (80) 14 (82) 6 (100) 36 (84)No 0 1 (6) 0 1 (2)

Comparative studies identifiedYes 7 (35) 2 (12) 1 (17) 10 (23)No 7 (35) 10 (59) 4 (67) 21 (49)No comparative studies in review 6 (30) 5 (29) 1 (17) 12 (28)

Test comparison strategyYes 9 (45) 2 (12) 0 11 (26)No 11 (55) 15 (88) 6 (100) 32 (74)

Method used for test comparisonYes 15 (75) NA NA 15 (75)Unclear 5 (25) NA NA 5 (25)

2x2 data for each study 9 (45) 8 (47) 3 (50) 20 (47)Individual study estimates of test accuracy 17 (85) 14 (82) 5 (83) 36 (84)Forest plot(s) 18 (90) 13 (76) 5 (83) 36 (84)SROC plot

SROC plot comparing summary points or curves for 2 or more tests

11 (55) 0 0 11 (26)

Separate SROC plot per test 7 (35) 9 (53) 3 (50) 19 (44)SROC plot for one test only 1 (5) 3 (18) 1 (17) 5 (12)No SROC plot 1 (5) 5 (29) 2 (33) 8 (19)

Limitations of indirect comparison acknowledged

4

Characteristic Comparative reviews Multiple test

reviews

TotalStatistical analyses to compare test accuracyYes No

Yes 2 (10) 2 (12) 0 4 (9)No 10 (50) 13 (76) 6 (100) 29 (57)No, only comparative studies included 8 (40) 2 (12) 0 10 (23)

NA = not applicable; SROC = summary receiver characteristic operating.*Numbers in parentheses are row percentages.†PRISMA-DTA and PRISMA reporting guidelines were used in six and 18 reviews respectively. The 19 remaining reviews did not state that a reporting guideline was used. One of the reviews that used PRISMA also used the MOOSE (Meta-analysis Of Observational Studies in Epidemiology) guideline.

References

1. Huang R, Jiang L, Xu Y, et al. Comparative Diagnostic Accuracy of Contrast-Enhanced

Ultrasound and Shear Wave Elastography in Differentiating Benign and Malignant Lesions: A

Network Meta-Analysis. Front Oncol. 2019;9:102.

5

Appendix Box 1. Definition of test accuracy terms Term Definition

Comparative accuracy study A primary study that compared the accuracy of two or more index tests in the same study population by randomizing participants to only one of the index tests (randomized design), or by applying all the index tests to each participant (paired or within-subject design).

Comparative accuracy review A diagnostic test accuracy review with a clear objective to compare the accuracy of at least two tests; limited study selection to only comparative studies; performed a direct (head-to-head) comparison of two tests; and/or performed a statistical comparison of test accuracy.

Diagnostic odds ratio (DOR) Ratio of the odds of positivity in those who have the target condition compared to the odds of positivity in those without the condition.

Direct comparison In a comparative diagnostic test accuracy review, a direct comparison includes only studies that have evaluated all the index tests being compared in the test comparison. Thus differences between studies in clinical or methodological characteristics are less likely to confound differences in accuracy.

Index test The new or existing test of interest that is being evaluated.

Indirect comparison In a comparative diagnostic test accuracy review, an indirect comparison includes all eligible studies that have evaluated at least one of the index tests in the test comparison. Thus indirect comparisons maximize use of the available data but are prone to confounding.

Multiple test review A diagnostic test accuracy review that does not have an explicit objective to compare the accuracy of two or more but assessed the accuracy of each test individually without making any formal comparison between tests.

Non-comparative accuracy study

A primary study that evaluated the accuracy of a single index test or the accuracy of only one of the index tests being evaluated in a comparative accuracy review.

Q* Point on the SROC curve where sensitivity is equal to specificity, i.e. the intersection of the summary curve and the line of symmetry (downward diagonal line running from the top left corner of the SROC plot to the bottom right corner).

Reference standard The best way of verifying the presence or absence of the target condition. It may be a single test or a combination of tests and clinical information.

Sensitivity Proportion of those with the target condition who have positive index test results.

Specificity Proportion of those without the target condition who have negative index test results.

SROC plot The summary receiver operating characteristic (SROC) plot is a scatterplot of sensitivity against specificity which displays the results of the studies included in an analysis from a diagnostic test accuracy review as points in receiver operating characteristic (ROC) space.

6

Target condition The condition, clinical state or pathological disease that is to be detected or excluded.

Test accuracy The ability of a test to discriminate between those who have and those who do not have the target condition. Test accuracy is estimated by comparing results of an index test with a reference standard, sometimes known as a ‘gold’ standard.

Appendix Figure 1. Comparison of test accuracy on SROC plots

A. Indirect comparison B. Direct comparison

HRP-2= histidine-rich protein-2; pLDH= plasmodium lactate dehydrogenase; RDT = rapid diagnostic test. For each test on a SROC plot, each symbol represents the pair of sensitivity and specificity from a study. To reflect the precision of sensitivity and specificity in each study, the size of the symbols was scaled by the sample sizes for those with and those without the target condition. Panel A shows an indirect comparison using data from the P. falciparum malaria review by Abba et al 2011.47 The indirect comparison included 75 HRP-2 based RDT studies and 19 pLDH based RDT studies. Of these, nine studies compared both tests in the same patients (I.e. paired or within subject design) and were included in the direct comparison shown in panel B. Point estimates for an HRP-2 based RDT and a pLDH based RDT from the same study are connected by a dotted line.

7

Appendix Table 1. Summary of methodological and reporting characteristics of five exemplar comparative reviewsReview Publication

typeReview objectives Test comparison and meta-analysis Comments

Alldred 201214

Cochrane To estimate and compare the accuracy of second trimester serum markers for the detection of Down’s syndrome, both as individual markers and as combinations of markers.

Comprehensive description of statistical methods for both direct and indirect comparisons, including the strategy for handling multiple thresholds. HSROC meta-regression models were used to formally compare test accuracy. Relative accuracy was expressed in terms of the relative diagnostic odds ratio as appropriate.

A well-structured, large and complex review due to the number of tests, test combinations and thresholds included.

Wang 201115 Cochrane To assess the diagnostic accuracy of non-invasive cardiac screening tests versus coronary angiography in potential kidney transplant recipients. Diagnostic accuracy was compared between screening tests.

Clear and detailed description of test comparison strategy and methods. Both direct and indirect comparisons were planned. Test accuracy was statistically compared in a HSROC meta-regression model.

Included an exemplary flow diagram.

Pennant 201016

Technology assessment report

To assess the incremental diagnostic accuracy of PET and PET/CT compared with existing diagnostic strategies and to compare the diagnostic accuracy of PET and PET/CT for the diagnosis of breast cancer recurrence

Rationale given for the test comparisons performed. For each pairwise comparison of imaging modalities, a bivariate meta-regression model was used to compare test accuracy. Relative accuracy reported using relative sensitivities and relative specificities.

Forest plots showing the pair of test accuracy estimates from each study were presented.

Williams 201017

Specialist To assess whether rapid urine tests were sufficiently sensitive to avoid urine culture in children with negative results and to compare the accuracy of dipsticks with microscopy.

Detailed test comparison strategy and methods. Differences in thresholds for test positivity were accounted for. Direct comparisons were performed using a HSROC model and meta-regression. Where appropriate, the relative diagnostic odds ratio was used to summarise relative accuracy.

A very detailed description of the methods.

Schuetz General To compare CT and MRI for ruling out A bivariate meta-regression model was used to compare Review findings

8

Review Publication type

Review objectives Test comparison and meta-analysis Comments

201018 medical journal

clinically significant coronary artery disease (CAD) in adults with suspected or known CAD.

test accuracy. Both direct and indirect comparisons were done though the test comparison strategy was not specified in the methods.

interpreted with caution due to limited evidence from comparative studies.

9

Appendix Figure 2. Comparison of test accuracy using summary points or summary curves

A. Comparison of summary points B. Comparison of summary curves

HRP-2= histidine-rich protein-2; pLDH= plasmodium lactate dehydrogenase; RDT = rapid diagnostic test. For each test on a SROC plot, each symbol represents the pair of sensitivity and specificity from a study. The size of each symbol was scaled according to the precision of sensitivity and specificity in the study. Panel A shows a comparison of summary points using the P. falciparum malaria review by Abba et al 2011.47 The indirect comparison included 75 HRP-2 based RDT studies and 19 pLDH based RDT studies. The solid circles (summary points) on the SROC plot represent the summary estimates of sensitivity and specificity for each test. Each summary point is surrounded by a dotted line representing the 95% confidence region and a dashed line representing the 95% prediction region (the region within which one is 95% certain the results of a new study will lie). Panel B shows a comparison of summary curves using the bipolar disorder review by Carvalho et al 2014.48 The indirect comparison included 44 studies that evaluated the diagnostic accuracy of the mood disorder questionnaire (30 studies), the bipolar spectrum diagnostic scale (8 studies) and the hypomania checklist (HCL-32, 17 studies) for detection of any type of bipolar disorder in a mental setting. Each summary curve was drawn restricted to the range of specificities from included studies that evaluated the test.

References

47. Abba K, Deeks JJ, Olliaro P, et al. Rapid diagnostic tests for diagnosing uncomplicated P.

falciparum malaria in endemic countries. Cochrane Database of Syst Rev 2011(7):CD008122.

48. Carvalho AF, Takwoingi Y, Sales PM, et al. Screening for bipolar spectrum disorders: A

comprehensive meta-analysis of accuracy studies. J Affect Disord 2014;172C:337-346.

10