the effectiveness of symptom checkers for self-diagnosis ... · confidential: for review only 1 the...
TRANSCRIPT
Confidential: For Review O
nly
The Effectiveness of Symptom Checkers for Self-Diagnosis
and Triage: Beyond “Googling” Symptoms
Journal: BMJ
Manuscript ID: BMJ.2015.025489
Article Type: Research
BMJ Journal: BMJ
Date Submitted by the Author: 17-Feb-2015
Complete List of Authors: Semigran, Hannah; Harvard Medical School, Health Care Policy Linder, Jeffrey; Brigham and Women's Hospital, Medicine; Harvard Medical School, Gidengil, Courtney; RAND Corporation, ; Boston Children's Hospital, Infectious Diseases
Mehrotra, Ateev; Harvard Medical School, Health Care Policy; Beth Israel Deaconess Medical Center, General Internal Medicine and Primary Care
Keywords: Symptom checkers, Internet, Antibiotic prescribing, mHealth
https://mc.manuscriptcentral.com/bmj
BMJ
Confidential: For Review O
nly
1
The Effectiveness of Symptom Checkers for Self-Diagnosis and Triage:
Beyond “Googling” Symptoms
Hannah Semigran,1
Jeffrey A. Linder,1,2
Courtney Gidengil,1,3,4
Ateev Mehrotra,1,5
1 Harvard Medical School, Boston, MA
2 Division of General Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA
3 RAND Corporation, Boston, MA
4 Division of Infectious Diseases, Boston Children’s Hospital, Boston, MA
5 Division of General Internal Medicine and Primary Care, Beth Israel Deaconess Medical Center,
Boston, MA
Corresponding Author:
Ateev Mehrotra, MD, MPH
Harvard Medical School
180 Longwood Avenue
Boston, MA 02115
617-432-3905
This study was funded by the United States’ National Institute of Health, (National Institute of Allergy
and Infectious Disease - Grant # R21 AI097759-01).
Conflict of Interest: All authors are affiliated with Harvard Medical School. Harvard Medical School’s
Family Health Guide is used as the basis for three symptom checkers evaluated. None of the authors
have been or plan to be involved in the development, evaluation, promotion or any other facet of
Harvard Medical School-related symptom checkers.
Word Count: 3,130
Tables: 5
Page 1 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
2
SUMMARY:
Objective: Use of the internet for self-diagnosis is common. Symptom checkers are online tools which
use computer algorithms to help patients with self-diagnosis and/or self-triage. Despite the growth in
use of such tools, performance of symptom checkers has not been assessed. Our objective was to
determine the accuracy of online symptom checkers’ diagnostic and triage advice.
Design: Audit study of all available symptom checkers using 45 standardized patient (SP) vignettes
divided equally into three categories based on problem urgency: emergent care required (e.g.,
pulmonary embolism), non-emergent care reasonable (e.g., otitis media), and self-care reasonable (e.g.,
viral upper respiratory illness).
Main outcome measures: For symptom checkers that provided a diagnosis, our main outcomes were
whether the symptom checker listed the correct diagnosis first or at all in the list of potential diagnoses
(n = 650 SP evaluations). For symptom checkers that provided a triage recommendation, our main
outcomes were whether the symptom checker appropriately recommended emergent care, non-
emergent care, or self-care (n = 516 SP evaluations).
Results: The 20 symptom checkers identified provided the correct diagnosis first in 33% (95% confidence
interval [CI], 30-37) of SP evaluations, listed the correct diagnosis at all in 60% (95% CI 56-63) of SP
evaluations, and provided the appropriate triage advice in 56% (95% CI 51-61) of SP evaluations. Triage
performance varied by condition urgency, with appropriate triage advice provided in 80% (95% CI 74-86)
of emergent cases, 55% (95% CI 47-62) of non-emergent cases, and 34% (95% CI 27-41) of self-care
cases (p<0.001). Performance across individual symptom checkers on appropriate triage advice ranged
from 33% (95% CI 19-48) to 78% (95% CI 64-91) of SP evaluations.
Page 2 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
3
Conclusions: Symptom checkers had deficits in both diagnosis and triage. Triage advice was generally
risk-averse; symptom checkers encouraged users to seek care for conditions where self-care was
reasonable.
Page 3 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
4
Introduction
The public is increasingly using the internet to research their health concerns. More than a third of
adults in the United States regularly use the internet to self-diagnose what ails them,1 and the United
Kingdom’s online patient portal for national health information, NHS Choices, reports over 15 million
visits per month.2 While there is a wealth of online resources to learn about specific conditions, self-
diagnosis for acute conditions is often limited to simply typing a list of symptoms into an internet search
engine.3 Three quarters of online health searchers begin with search engines like Google, Bing, or
Yahoo.1 Between 2008 and 2014 health-related searches using Google UK increased 13.5%.
3 Internet
search engines can lead users to confusing and sometimes unsubstantiated information.4-6
Recently,
there has been a proliferation of more sophisticated programs called symptom checkers that attempt to
more effectively diagnose patients and direct them to the appropriate care setting.
Using computerized algorithms, symptom checkers ask users a series of questions about their symptoms
or require users to input their symptoms themselves. The algorithms vary and may use branching logic,
Bayesian inference, or other methods. Private companies and other organizations, including the NHS,
the American Academy of Pediatrics, and the Mayo Clinic, have launched their own symptom checkers.
One symptom checker, iTriage, reports 50 million uses per year.7 Typically symptom checkers are
accessed via websites, but some are also available as apps for smart phones or tablets.
Symptom checkers serve two main functions: facilitating self-diagnosis and assisting with triage. The
self-diagnosis function provides a list of diagnoses usually rank ordered by likelihood. The diagnosis
function is typically framed as helping educate patients on the range of diagnoses that might fit their
symptoms. The triage function informs patients whether they should seek care at all and, if so, where
(i.e. accident & emergency department, general provider’s (GP) clinic) and with what urgency (i.e.
emergent or within a few days). Symptom checkers may supplement or replace telephone triage lines,
Page 4 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
5
which are common in primary care.8-10
To ensure the safety of medical mobile apps, the United States
Congress is considering regulation of apps that “provide a list of possible medical conditions and advice
on when to consult a health care provider.”11 12
Symptom checkers have several potential benefits. They can encourage patients with a life-threatening
problem to seek emergent care.13
For patients with a non-emergent problem that does not require a
medical visit, these programs can reassure patients by recommending that they stay home. Simple acute
conditions such as viral upper respiratory illness are common and for approximately a quarter of acute
respiratory illness visits, patients do not receive any intervention beyond over-the-counter therapy,14
and over half of patients receive unnecessary antibiotics.15-18
Decreasing unnecessary visits saves
patients’ time and money, deters overprescribing of antibiotics, and may increase general practitioners’
(GP) capacity to manage more patients – a critical issue given GP workload in the United Kingdom
increased by 62% from 1995 to 2008.10
However, there are several key concerns. If patients with a life-threatening problem are misdiagnosed
and told to not seek care, their health could worsen, increasing morbidity and mortality. Alternatively, if
patients with minor illnesses are told to seek care, in particular in an accident & emergency department,
such programs could increase unnecessary visits and therefore result in increased time and costs for
patients and society.
To our knowledge, no previous study has systematically evaluated the diagnostic and triage
performance of symptom checkers. To evaluate the ability of symptom checkers to provide an accurate
diagnosis and appropriate triage advice, we audited all available symptom checkers using 45
standardized patient (SP) vignettes.
Methods
Page 5 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
6
Identifying symptom checkers
We identified all available symptom checkers, which we defined as online internet resources for
personalized self-diagnosis and self-triage, through internet searches, literature review, and
consultation with experts in the field. We identified 28 symptom checkers that were in English, free, and
publicly-available. We excluded symptom checkers that used the same underlying logic or algorithm as
other symptom checkers. Australia’s Health Direct Symptom Checker shares a license and similar triage
output with England’s NHS Symptom Checker. The American Academy of Pediatrics’ Healthy Children
Symptom Checker and the Swedish Kids Symptom Checker both utilize an electronic version of a
common nurse triage protocol.19
Everyday Health uses the same algorithm as FreeMD; MedicineNet and
RxList use the same as WebMD, the AARP uses the same as Healthline; Kaiser Permanente is powered
by Healthwise; finally, both Drugs.com and GenieMD use the same algorithm developed by Harvard
Health Publications. After these exclusions, we evaluated 20 symptom checkers between June 2014 and
December 2014.
Symptom checkers’ characteristics
We categorized symptom checkers by whether they facilitated self-diagnosis, self-triage, or both; type of
organization that operated the symptom checker; the maximum number of diagnoses provided, and
whether they were based on Schmitt or Thompson nurse triage guidelines, which are decision support
protocols commonly used in telephone triage for pediatric and adult consultations, respectively.19 20
We
grouped government and health plans together because both have a financial incentive to deter
unnecessary visits. To estimate the number of visitors to symptom checkers, we used Compete Pro, a
US-based online marketing analysis website. 21
We obtained the number of unique visitors for October
2014. Visit data was not available for symptom checkers that were a part of a larger website (e.g.
mayoclinic.org), in certain countries, or those that were only provided as an app.
Page 6 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
7
Clinical vignettes
To evaluate symptom checkers’ diagnosis and triage performance, we used 45 SP vignettes. We
developed SP vignettes from various sources listed in the appendix, and the correct diagnosis was
provided by the SP vignette. We chose to use clinical vignettes to test performance because they are a
common method to test physicians on their diagnostic ability and management decisions. Though we
focused on more common problems, we purposefully included three rare, serious conditions (Rocky
Mountain spotted fever, tetanus, and malaria) to assess the tools’ ability for diagnosing and triaging
such problems.
Symptom checkers ask users for a list of symptoms or ask a series of questions. Each SP vignette was
simplified into a core set of symptoms for easy entry. In some situations, we supplemented the data
provided by the vignette because a symptom checker asked about a symptom not addressed in vignette.
We used the same set of symptoms, with additions, noted in the appendix for each symptom checker.
We stratified SP vignettes into 3 levels of severity: (1) 15 SP vignettes for which emergent care is
required, (2) 15 SP vignettes for which non-emergent care is reasonable, and (3) 15 SP vignettes for
which a medical visit is generally unnecessary and self-care is sufficient. Categorization of SP vignette
severity was based on consensus clinical judgment from three physicians on the research team.
Assessing diagnosis and triage results
Each SP vignette was entered into each website, and we recorded the resulting diagnoses and triage
advice. In some cases, we could not evaluate a vignette because some symptom checkers only focus on
children or adults or the symptom checker did not list or ask for the key symptom in vignette. We
referred to SP vignettes that successfully yielded an output as “SP evaluations”.
Page 7 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
8
To assess diagnostic accuracy, we noted whether the correct diagnosis was listed first or listed at all. For
several vignettes two symptom checkers presented a large number of diagnoses (up to 99). Because
such a long list of potential diagnoses is unlikely to be useful for a patient, we considered a diagnosis to
be listed at all only if it was within the first 20 diagnoses a symptom checker provided. We also judged
the diagnosis incorrect if the symptom checker indicated that the condition could not be identified. In a
sensitivity analysis, we limited our assessment of accuracy to the first 3 diagnoses instead of the first
diagnosis listed.
We categorized the triage advice into three groups: (1) Emergent, which included advice to call an
ambulance, go to the accident & emergency department, or see your GP immediately; (2) Non-
Emergent, which included advice to go to an urgent care facility, to call your GP, to see your GP, go to a
specialist, go to a retail clinic, or have an e-visit; and (3) Self-Care, which included advice to stay at home
or go to a pharmacy. If multiple triage locations were suggested (e.g. accident & emergency department
or a specialist), the most urgent suggestion was used. We chose to do so because in almost all of the
cases, the most urgent triage suggestion was listed first. Symptom checkers that required users to select
the correct diagnosis before giving triage advice were not included in assessing triage accuracy with the
exception of iTriage, which always suggested emergent triage advice.
Analysis
We calculated summary statistics for diagnostic accuracy and triage advice with 95% confidence
intervals based on binomial distribution using Stata/MP 13.0. Given our focus on symptom checkers as a
whole, we did not make statistical comparisons of accuracy between symptom checkers. We used chi-
square tests to compare the diagnosis and triage accuracy by level and urgency and by type of symptom
checker. We conducted a sensitivity analysis of triage advice excluding several symptom checkers that
always or usually recommended emergent care.
Page 8 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
9
Results
Study sample
The 20 identified symptom checkers were based in the United Kingdom, United States, Netherlands, and
Poland (Table 1). Ten of the symptom checkers provided both diagnoses and triage advice, 6 provided
only diagnoses, and 4 provided only triage advice. Performance was assessed on a total of 650 SP
evaluations for diagnosis and 516 SP evaluations for triage. Across the symptom checkers, 8 did not ask
for demographics (age and gender). For the 8 symptom checkers where we could estimate web traffic,
there were on average 1.25 million visitors in October 2014.
Accuracy of diagnosis being listed first
Overall, the correct diagnosis was listed first in 33% (95% CI 30-37; table 2 and 4) of SP evaluations.
Performance varied by urgency of condition. The correct diagnosis was listed first for 25% (95% CI 19-31)
of emergent SP evaluations and 39% (95% CI 32-45, p=0.004) of self-care SP evaluations (table 4). In a
sensitivity analysis, we found that the symptom checkers listed the correct diagnosis within the first
three diagnoses listed in 52% (95% CI 48-56) of SP evaluations.
Performance varied across symptom checkers and by urgency of vignettes (table 3). Listing the correct
diagnosis first ranged from 5% for MEDoctor (95% CI 0-13) of SP evaluations to 47% for Family Doctor
(95% CI 31-62).
Few differences were observed by symptom checker characteristics (table 5). There was no difference
between symptom checkers that did and did not ask for demographic information (34% [95% CI 29-39]
vs. 33% [95% CI 27-38], p=0.72).
Accuracy of listing correct diagnosis at all
Page 9 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
10
Across all symptom checkers, the correct diagnosis was listed in 60% (95% CI 56-63) of SP evaluations
(table 4). This varied by urgency of SP vignettes 52% (95% CI 45-59) for emergent conditions, 62% (95%
CI 55 to 68) for non-emergent conditions, and 65% (95% CI 59-72) for self-care conditions (p=0.013). The
correct diagnosis was listed at all in 63% (95% CI 60-67) of the common vignettes versus 19% (95% CI 9-
29) for the three rare conditions (p<0.001). There was no significant difference in listing the correct
diagnosis between symptom checkers that listed more than 11 diagnoses versus those that only listed 1-
3 diagnoses (4% [95% CI 57-70] vs. 54% [95% CI 47-61], p=0.15; table 5]. The accuracy of listing the
correct diagnosis by individual symptom checkers ranged from 34% (95% CI 17-52) to 84% (95% CI 73-
95) (table 3).
Accuracy of triage advice
Appropriate triage advice was given in 56% (95% CI 52-61) of SP evaluations and was higher for
emergent care vignettes versus those for which no medical care was necessary (80% [95% CI 74-86] vs.
34% [95% CI 27-41], p<0.001) (table 4). iTriage, Isabel, and Healthwise advised emergent care for 100%
of emergent SP evaluations. iTriage, Symcat, Symptomate and Isabel always suggested users to seek
care and therefore never advised self-care (table 3). Even when excluding these symptom checkers,
there was minimal improvement in the appropriateness of the other tools’ triage advice (61% [95% CI 56
to 66]).
Symptom checkers that used the Schmitt or Thompson nurse-triage protocols were more likely to
provide appropriate triage decisions than those that did not (72% [95% CI 60-84] vs. 55% [95% CI 50-59]
of SP evaluations, p=0.01). Accurate triage advice varied by operator (provider groups and physician
associations 68% [95% CI 58-77], private companies 59% [95% CI 53-65], health plans/governments 43%
[95% CI 34-51], p<0.001).
Discussion
Page 10 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
11
Using SP vignettes, we audited the performance of symptom checkers for self-diagnosis and triage.
Although there was a range of performance across symptom checkers, symptom checkers had clear
deficits in diagnosis and triage accuracy. On average symptom checkers listed the correct diagnosis in
60% of SP evaluations with the best-performing symptom checker listing the correct diagnosis in 84% of
SP evaluations. Our results on diagnostic are roughly similar to work that has focused on a more limited
set of diagnoses and just one symptom checker. A single orthopedic symptom checker listed the correct
diagnosis for knee pain 89% of the time and Boots WebMD listed the correct diagnosis 70% of the time
for ear, nose, and throat symptoms.22 23
In terms of triage, symptom checkers advised the appropriate
level of care about half the time, but this varied by clinical severity. The correct triage decision was more
than two times higher for SP vignettes requiring emergent care (80%) versus those SP vignettes where
no medical care was necessary (34%).
Whether this level of performance for diagnosis and triage is acceptable depends on what is viewed as
the gold standard. Physicians generally have a diagnostic error rate of 10-15% .24
If symptom checkers
are seen as a replacement for seeing a physician, their performance is clearly an inferior alternative.
A more appropriate comparison may be to compare symptom checkers to nurse telephone triage lines,
which are widely used in developed nations. The key component of symptom checkers may be
appropriate triage, as distinguishing between Rocky Mountain spotted fever and meningitis may be less
important than ensuring patients seek emergent care. Telephone triage recommendations, compared to
in-person physician recommendations, range in accuracy from 61% in a study of pediatric abdominal
pain to 69% in a multicenter observational study.25 26
Many nurse phone triage lines use the same
symptom checkers evaluated in this study and the Schmitt and Thompson telephone protocols were the
underlying logic for several symptom checkers.27
Symptom checkers may thus be viewed in general as a
reasonable alternative to telephone triage.
Page 11 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
12
One potential advantage of symptom checkers over telephone triage is cost. Telephone triage has been
promoted as a means of reducing unnecessary office visits.8 9 28 29
However, in a recent large cluster
randomized trial in 42 practices in four centres in the UK, telephone triage did not decrease spending, as
the savings from deterring visits was offset by the labor costs of nurse or physician time to respond to
the telephone calls.10
Because of their negligible costs, symptom checkers could potentially be a more
cost-effective way of providing triage advice—particularly if they can deter unnecessary office visits.
The risk-averse nature of symptom checker advice is a concern. In two-thirds of SP evaluations where
medical attention was not necessary, symptom checkers encouraged care. Four of the symptom
checkers never advised the user that medical attention was not necessary. This is consistent with prior
work where computer-based triage tools have been observed to provide more conservative triage
advice than physician consultations.30
This trend is also found in telephone triage, in which more
conservative triage recommendations has been attributed to organizational reasons and uncertainty
about the patient’s diagnosis.25
This risk-averse approach of symptom checkers may be
overcompensation for the lack of face-to-face care or driven by liability concerns. Some patients
researching health conditions are motivated by fear, and the listing of concerning diagnoses by
symptom checkers and risk-averse triage advice could contribute to hypochondriasis.31-33
The term
“cyberchondria” describes the escalated anxiety associated with self-diagnosis on the internet.34
The symptom checkers in this study represent the first generation of such tools, and there are a number
of potential advancements that may improve their performance in future versions. Incorporating local
epidemiological data may help inform diagnoses. For instance, addition of real-time information about
the local incidence of illness in the community greatly improved the performance of a Group A
Streptococcal pharyngitis diagnostic tool.35
Diagnosis and triage rates could also be improved if
symptom checkers could incorporate population or individual clinical data from medical claims or the
Page 12 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
13
electronic medical record. Demographic information is critical for both diagnostic and triage decisions
for programs like symptom checkers.36
One surprising finding in our study was that symptom checkers
that asked for demographic background information did not perform better. However, it is possible that
this demographic information was not effectively incorporated into the symptom checkers algorithms.
There were key limitations to this study. We cannot be sure we identified all publicly-available symptom
checkers, despite a thorough search including consultation with experts in this field and a search of all
relevant databases. We used clinical vignettes in which the symptoms and diagnoses were typically
clear, and few had comorbid conditions, resulting in a possible overestimation of the true diagnostic
accuracy of symptom checkers.24
When symptom checkers suggested several care sites (e.g. accident &
emergency department or GP office), our triage assessment was based only on the highest acuity site of
care listed and this may contribute to our finding that triage advice is risk-averse. Symptom checkers’
impact will depend on how patients respond to the triage advice, which will need to be evaluated in
future work.
Symptom checkers are part of a larger trend of both patients and physicians using the internet for many
health care tasks and therefore it appears likely that the use of symptom checkers will only increase.
Patients are chatting online with a physicians,37
emailing doctors for medical advice,38
receiving care via
e-visits,39 40
and downloading health apps to smartphones.41
From a physician’s perspective, an
increasing number of their patients are using new internet-based tools like symptom checkers and
physicians should be aware that the diagnosis and triage advice patients receive may often be
inaccurate. For patients, our results imply that in many cases, symptom checkers can give a sense of
possible diagnoses, but they should be cautious as there is a significant rate of diagnostic error and
tendency towards risk aversion. However, if the alternative to using a symptom checker is not seeking
any advice, there may be value in their use. Further evaluations and monitoring of symptom checkers
Page 13 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
14
will be important to assess whether they help users learn more and make better decisions about their
health.
References
1. Fox S, Duggan M. Health Online 2013. Internet and American Life Project. Washington D.C.: Pew
Research Center and California Health Care Foundation, 2013:4.
2. Gann B. Giving patients choice and control: health informatics on the patient journey. Yearb Med
Inform 2012;7(1):70-3.
3. Daily Mail. More Brits are self-diagnosing illness online instead of going to a doctor than ever before...
and searches for 'stress-related' symptoms have DOUBLED, 2014.
4. Black P. The dangers of using Google as a diagnostic aid. British Journal of Nursing 2009;18(19):1157.
5. Zhongbo C, Turner MR. The internet for self-diagnosis and prognostication in ALS. Amyotrophic
Lateral Sclerosis 2010;11:566.
6. Luger TM, Houston TK, Suls J. Older adult experience of online diagnosis: results from a scenario-
based think-aloud protocol. Journal of Medical Internet Research 2014;16(1):e16.
7. Reuters. Aetna Brings New iTriage Employer Technology to Mid-Sized Businesses, 2013.
8. Lattimer V, George S, Thompson F, Thomas E, Mullee M, Turnbull J, et al. Safety and effectiveness of
nurse telephone consultation in out of hours primary care: randomised controlled trial. British
Medical Journal 1998;317(7165).
9. Bunn F, Byrne G, Kendall S. The effects of telephone consultation and triage on healthcare use and
patient satisfaction: A systematic review. British Journal of General Practice 2005;55(521).
10. Campbell J, Fletcher E, Britten N, Green C, Holt T, Lattimer V, et al. Telephone triage for management
of same-day consultation requests in general practice (the ESTEEM trial): a cluster randomised
controlled trial and cost-sequence analysis. Lancet 2014;in press.
11. Lewis TL, Wyatt JC. mHealth and mobile medical Apps: a framework to assess risk and promote safer
use. Journal of Medical Internet Research 2014;16(9):e210.
12. Medical Electronic Data Technology Enhancement for Consumers' Health (MEDTECH). 2nd Session
ed, 2014.
13. Saczynski J, Yarzebski J, Lessard D, Spencer F, Gurwitz J, Gore J, et al. Trends in pre-hospital delay in
patients with acute myocardial infarction (from the Worcester Heart Attack Study). American
Journal of Cardiology 2008;102(12):1591.
14. Mehrotra A, Wang MC, Lave JR, Adams JL, McGlynn EA. Retail clinics, primary care physicians, and
emergency departments: a comparison of patients' visits. Health Aff (Millwood)
2008;27(5):1272-82.
15. Gill J, Fleischut P, Haas S, Pellini B, Crawford A, Nash D. Use of antibiotics for adult upper respiratory
infections in outpatient settings: A national ambulatory network study. Family Medicine
2006;38(5):349, 51.
16. Barnett ML, Linder JA. Antibiotic prescribing for adults with acute bronchitis in the United States,
1996-2010. Jama 2014;311(19):2020-2.
17. Barnett ML, Linder JA. Antibiotic prescribing to adults with sore throat in the United States, 1997-
2010. JAMA Intern Med 2014;174(1):138-40.
18. Gonzales R, Malone DC, Maselli JH, Sande MA. Excessive antibiotic use for acute respiratory
infections in the United States. Clin Infect Dis 2001;33(6):757-62.
Page 14 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
15
19. Little P, Rumsby K, Kelly J, Watson L, Moore M, Warner G, et al. Information leaflet and antibiotic
prescribing strategies for acute lower respiratory tract infection: a randomized controlled trial.
Jama 2005;293(24):3029-35.
20. Schmitt BD. Pediatric Telephone Protocols: Office Version. Elk Grove Village IL: American Academy of
Pediatrics, 2012.
21. Thompson DA. Adult Telephone Protocols, 3rd Edition. Elk Grove Village IL: American Academy of
Pediatrics, 2013.
22. Compete. Site Comparison, 2014.
23. Bisson LJ, Komm JT, Bernas GA, Fineberg MS, Marzo JM, Rauh MA, et al. Accuracy of a computer-
based diagnostic program for ambulatory patients with knee pain. Am J Sports Med
2014;42(10):2371-6.
24. Farmer SE, Bernardotto M, Singh V. How good is Internet self-diagnosis of ENT symptoms using
Boots WebMD symptom checker? Clin Otolaryngol 2011;36(5):517-8.
25. Graber ML. The incidence of diagnostic error in medicine. BMJ Qual Saf 2013;22 Suppl 2:ii21-ii27.
26. Staub GM, von Overbeck J, Blozik E. Teleconsultation in children with abdominal pain: a comparison
of physician triage recommendations and an established paediatric telephone triage protocol.
BMC Med Inform Decis Mak 2013;13:110.
27. Giesen P, Ferwerda R, Tijssen R, Mokkink H, Drijver R, van den Bosch W, et al. Safety of telephone
triage in general practitioner cooperatives: do triage nurses correctly estimate urgency? Qual
Saf Health Care 2007;16(3):181-4.
28. Sadeghi S, Barzi A, Sadeghi N, King B. A Bayesian model for triage decision support. Int J Med Inform
2006;75(5):403-11.
29. Stacey D, Graham I, O'Connor A, Pomey M. Barriers and facilitators influencing call center nurses'
decision support for callers facing values-sensitive decisions: A mixed methods study.
Worldviews on Evidence-Based Nursing 2005;2(4).
30. Richards D, Meakins J, Tawfik J, Godfrey L, Dutton E, Richardson G, et al. Nurse telephone triage for
same day appointments in general practice: multiple interrupted time series trial of effect on
workload and costs. British Medical Journal 2002;325(7374).
31. Poote A, French D, Dale J, Powell J. A study of automated self-assessment in primary care student
health centre setting. Journal of Telemedicine and Telecare 2014;20(3):125.
32. Usborne S. Cyberchondria: The perils of internet self-diagnosis. London: The Independent, February
17, 2009.
33. Hartzband P, Groopman J. Untangling the web - Patients, doctors, and the internet. New England
Journal of Medicine 2010;362(12):1064.
34. Brigo F, Igwe SC, Ausserer H, Nardone R, Tezzon F, Bongiovanni LG, et al. Why do people Google
epilepsy? An infodemiological study of online behavior for epilepsy-related search terms.
Epilepsy Behav 2014;31:67-70.
35. White RW, Horvitz E. Experiences with web search on medical concerns and self diagnosis. AMIA
Annu Symp Proc 2009;2009:696-700.
36. Fine AM, Nizet V, Mandl KD. Participatory medicine: A home score for streptococcal pharyngitis
enabled by real-time biosurveillance: a cohort study. Ann Intern Med 2013;159(9):577-83.
37. DocBot: A novel clinical decision support algorithm. Engineering in Medicine and Biology Society
(EMBC), 2014 36th Annual International Conference of the IEEE; 2014 26-30 Aug. 2014.
38. Eminovic N, Wyatt J, Tarpey A, Murray G, Ingram G. First evaluation of the NHS Direct Online Clinical
Enquiry Service: A nurse-led web chat triage service for the public. Journal of Medical Internet
Research 2004;6(2).
39. Eysenbach G, Diepgen T. Patients looking for information on the internet and seeking teleadvice.
Archives of Dermatology 1999;135(2).
Page 15 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
16
40. Mehrotra A, Paone S, Martich GD, Albert SM, Shevchik GJ. A comparison of care at e-visits and
physician office visits for sinusitis and urinary tract infection. JAMA Intern Med 2013;173(1):72-
4.
41. DeJong C, Santa J, Dudley RA. Websites that offer care over the Internet: is there an access quality
tradeoff? Jama 2014;311(13):1287-8.
42. Edney A. The FDA Sets Its Sights on Medical Apps: Bloomberg Businessweek, September 20, 2013.
Page 16 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
17
Symptom
Checker Description
Maximum
No. of
Diagnoses
Observed
Triage Options Provided
AskMD (USA) Online health and wellness platform from
Sharecare
(https://www.sharecare.com/askmd/get-
started)
15 n/a
Drugs.com
(USA)
Online resource for drug and related health
information; uses content from Harvard
Health Publications
(http://www.drugs.com/symptom-
checker/)
10 ED, primary care doctor, home care
EarlyDoc
(Netherlands)
For triage criteria, uses Dutch College of
General Practitioners (NHG) TriageWijzer
and the Australian Triage Scale (used in
Australia and New Zealand to assess
urgency). (https://www.earlydoc.com/en/)
3 Don't wait and call a doctor now,
call a doctor preferably today, see
your doctor preferably on a
weekday, your complaints don't
seem urgent
Esagil (USA) Provides list of likely diagnoses (based on
the percent of entered symptoms that are
congruent with the diagnosis); the user can
also enter blood and urine analysis results
along with symptoms (http://esagil.org/)
65 n/a
Family Doctor
(USA)
Displays flow chart to track symptoms to a
diagnosis and triage option; produced by
the American Academy of Physicians
(http://familydoctor.org/familydoctor/en/h
ealth-tools/search-by-symptom.html)
7 ER, see your doctor, home care
FreeMD (USA) Takes user through a series of questions in a
"checkup" to finish with "what might be
wrong with you" and "where to go for
care"; owned by DSHI Systems, which
provides triage decision support solutions
from emergency medicine physicians to the
US government (Dep. of Veteran Affairs)
and private sector companies; program
called TriageXpert
(http://www.freemd.com/)
3 ED, urgent care, doctor's office,
doctor e-visit, retail clinic, dentist,
home care
Harvard
Medical School
Family Health
Guide (USA)
From Harvard Health Publications; this tool
is available both online and in print
(published 1999), and the online tool often
refers the user to the book in order to make
a diagnosis and triage decision
(http://www.health.harvard.edu/fhg/sympt
oms/symptoms.shtml)
4 ED, GP, home care
Healthline
(USA)
Health and wellness website that licenses
content to employers, health providers, and
health plans
(http://www.healthline.com/symptom-
checker)
76 n/a
Healthwise Non-profit provider for health content and n/a Call 911 now, seek care now, seek
Table 1|Overview of symptom checkers included in the study
Page 17 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
18
(USA) patient education; symptom checker
licensed to other organizations; we
accessed using the Province of Alberta's
website
(https://myhealth.alberta.ca/health/pages/
symptom-checker.aspx)
care today, try home care
Healthy
Children (USA)
From the American Academy of Pediatrics;
use's Barton D. Schmitt's "Pediatric
HouseCalls Symptom Checker" triage
protocol
(http://www.healthychildren.org/English/ti
ps-tools/symptom-
checker/Pages/default.aspx)
n/a Call 911 now, call your doctor now
(night or day), call your doctor
within 24 hours, call your doctor
during weekday office hours, parent
care at home
Isabel (UK) Created by the Isabel Medical Charity
(http://symptomchecker.isabelhealthcare.c
om/suggest_diagnoses_advanced/landing_
page)
10 Walk in care, family doctor,
emergency services
iTriage (USA) Owned by Aetna; provides clinical sites in
user's region with addresses and phone
numbers (https://www.itriagehealth.com/)
5 Emergency department, urgent
care, retail clinic, family practice,
internal medicine, specialties,
prescription medication, over the
counter medication
Mayo Clinic
(USA)
Health resource website from Mayo Clinic
(http://www.mayoclinic.org/symptom-
checker/select-symptom/itt-20009075)
20 n/a
MEDoctor
(USA)
Free differential diagnosis system from
MEDoctor, Inc.
(https://www.medoctor.com/)
3 n/a
NHS Symptom
Checkers (UK)
Available through England's National Health
Services (NHS) Choices website
(https://www.nhs.uk/symptomcheckers/pa
ges/symptoms.aspx)
n/a Emergency department, general
practitioner, home care
Symcat (USA) Triage tool uses data linking specific patient
symptoms and physician diagnoses across
visits seen in the NAMCS survey
(http://www.symcat.com/)
6 Primary care, retail clinic,
emergency room, urgent care
Symptify (USA) Online self-assessment tool and other
health services including emergency contact
list, consultation list etc.
(https://symptify.com/)
9 ER, urgent care, home care,
inconclusive
Symptom MD
(USA)
iPhone and Android app, provides symptom
care guides from Barton D. Schmitt's
pediatric telephone triage guidelines and
David A. Thompson's adult telephone triage
guidelines (http://www.symptommd.com/)
n/a Call 911 now, go to ER now, Call
doctor now or go to ER, Call doctor
within 24 hours, Call doctor during
office hours, self-care at home
Symptomate
(Poland)
Uses Bayesian network methodology and a
medical database for diagnoses
(https://symptomate.com/)
5 ER, specialist, GP
WebMD (USA) Medical reference and health care resource
website
(http://symptoms.webmd.com/#introView)
99 n/a
Page 18 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
19
Ask
MD
/
Sh
are
care
Dru
gs.
com
Earl
yD
oc
Esa
gil
Fam
ily D
oct
or
Fre
eM
D
HM
S F
am
ily H
ealt
h G
uid
e
He
alt
hli
ne
Isab
el
iTri
age
Mayo
Cli
nic
MED
oct
or
Sym
cat
Sym
pti
fy
Sym
pto
mate
We
bM
D
Dru
gs.
com
Earl
yD
oc
Fam
ily D
oct
or
Fre
eM
D
HM
S F
am
ily H
ealt
h G
uid
e
He
alt
hw
ise
He
alt
hy C
hil
dre
n
Isab
el
iTri
age
NH
S
Sym
cat
Sym
pti
fy
Sym
pto
mate
Sym
pto
mM
D
Standardized Patient Vignette Emergent Care Non Emergent Care Self Care
Require emergent care
Acute liver failure 〇 〇 〇 〇 U
Appendicitis 〇 〇 ⊗ ⊗ ⊗ ⊗ ⊗ 〇 〇 ⊗ ⊗ ▨Asthma attack ⊗ ⊗ ⊗ ⊗ 〇 ⊗ 〇 ⊗ ⊗ 〇 ⊗ 〇
COPD more severe 〇 ⊗ 〇 ⊗ 〇 ▨Deep Vein Thrombosis ⊗ ⊗ ⊗ ⊗ 〇 〇 〇 ⊗ ⊗
Heart attack ⊗ ⊗ ⊗ ⊗ ⊗ ⊗
Hemolytic Uremic Syndrome ⊗ 〇 〇 U
Kidney Stones 〇 ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ 〇 〇 〇 ▨Malaria 〇 〇 N N
Meningitis ⊗ ⊗ ⊗ ⊗ ⊗
Pneumonia (more severe) ⊗ 〇 〇 〇 ⊗ 〇 〇 〇 〇 ⊗ ▨Pulmonary Embolism ⊗ 〇 ⊗ 〇 〇 ⊗ 〇 ⊗ ⊗ ▨Rocky Mountain Spotted
Fever⊗
Stroke 〇 ⊗ ⊗ 〇 〇 ▨Tetanus ⊗
Symptom Checker
Incorrect
diagnosis
Correct
diangosis
listed
Correct
diagnosis
listed first
Process couldn't be
started (ex. Too young)
Diagnosis given Triage Advice Given
Table 2|Diagnosis given and triage advice from each symptom checker, stratified by severity of the standardized patient (SP) vignette
Page 19 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
20
Ask
MD
/
Sh
are
care
Dru
gs.
com
Ea
rly
Do
c
Esa
gil
Fa
mil
y D
oct
or
Fre
eM
D
HM
S F
am
ily
He
alt
h G
uid
e
He
alt
hli
ne
Isa
be
l
iTri
ag
e
Ma
yo
Cli
nic
ME
Do
cto
r
Sy
mca
t
Sy
mp
tify
Sy
mp
tom
ate
We
bM
D
Dru
gs.
com
Ea
rly
Do
c
Fa
mil
y D
oct
or
Fre
eM
D
HM
S F
am
ily
He
alt
h G
uid
e
He
alt
hw
ise
He
alt
hy
Ch
ild
ren
Isa
be
l
iTri
ag
e
NH
S
Sy
mca
t
Sy
mp
tify
Sy
mp
tom
ate
Sy
mp
tom
MD
Symptom Checker
Requires non-emergent care
Acute otitis media ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ▩Potential bacterial
pharyngitisa
〇 ⊗ ⊗ ⊗ 〇 〇 ⊗ 〇 ⊗ ⊗ ▩Potential bacterial
pharyngitisa
⊗ ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ⊗ ⊗ 〇 ⊗ ⊗ N
Acute sinusitis 〇 〇 ⊗ ⊗ ⊗ ⊗ 〇 〇 〇 ⊗ ⊗ ▩Back pain with foot drop ⊗ ⊗ ⊗ 〇 ⊗ ▨Cellulitis ⊗ 〇 〇
COPD flare ⊗ ⊗ ⊗ ⊗ 〇 〇 ▨Influenza ⊗ ⊗ ⊗ 〇 ⊗ ⊗ ⊗ 〇 〇 ⊗ ⊗ ▨Mononucleosis ⊗ 〇 〇 ⊗ 〇 〇 〇 ⊗ ⊗ ▩Peptic Ulcer Disease 〇 〇 ⊗ ⊗ 〇 〇 ⊗ ⊗ 〇 ⊗ ⊗ N
Pneumonia ⊗ 〇 〇 ⊗ ⊗ 〇 〇 ⊗ ▩Salmonella ⊗ 〇 ▩Shingles 〇 ⊗ ⊗ ⊗ 〇 ⊗ 〇 〇 ⊗ N
Urinary tract infection 〇 ⊗ ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ⊗ ⊗ ⊗ ⊗ ▩Vertigo ⊗ 〇 ⊗ ⊗ ⊗ N
Page 20 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
21
Abbreviations: HMS, Harvard Medical School; NHS, National Health Services a There are multiple vignettes for this condition. See the Appendix for each.
Ask
MD
/
Sh
are
care
Dru
gs.
com
Ea
rly
Do
c
Esa
gil
Fa
mil
y D
oct
or
Fre
eM
D
HM
S F
am
ily
He
alt
h G
uid
e
He
alt
hli
ne
Isa
be
l
iTri
ag
e
Ma
yo
Cli
nic
ME
Do
cto
r
Sy
mca
t
Sy
mp
tify
Sy
mp
tom
ate
We
bM
D
Dru
gs.
com
Ea
rly
Do
c
Fa
mil
y D
oct
or
Fre
eM
D
HM
S F
am
ily
He
alt
h G
uid
e
He
alt
hw
ise
He
alt
hy
Ch
ild
ren
Isa
be
l
iTri
ag
e
NH
S
Sy
mca
t
Sy
mp
tify
Sy
mp
tom
ate
Sy
mp
tom
MD
Symptom Checker
Self-Care Appropriate
Acute bronchitisa 〇 〇 ⊗ ⊗ 〇 〇 ▧
Acute bronchitisa 〇 〇 ⊗ ⊗ ⊗ 〇 〇 ⊗ 〇 ⊗ ⊗ N
Acute conjunctivitis ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ◯ ◯ ◯ ⊗ ⊗ ▨Acute viral pharyngitis 〇 ⊗ ⊗ 〇 〇 ⊗ ⊗ ⊗ ⊗ ▧Allergic rhinitis ⊗ 〇 ⊗ ⊗ ⊗ 〇 ⊗ ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ▧Back pain, unremarkable ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ⊗ ⊗ 〇 ▧Bee sting without anaphylaxis 〇 ⊗ 〇 ⊗
Candidal yeast infection ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ▩Canker sore 〇 ⊗ 〇 〇 〇
Constipation ⊗ ⊗ ⊗ ⊗ ⊗ N U
Eczema ⊗ 〇 〇 〇 ⊗ ⊗ ⊗ ⊗ 〇 H
Stye 〇 ⊗ ⊗ 〇 〇 ⊗ 〇 〇 ▧Viral URI
a 〇 ⊗ ⊗ ⊗ ◯ ◯ ⊗ ◯ 〇 〇 ⊗ 〇 〇 ▧Viral URI
a ⊗ 〇 ⊗ ⊗ 〇 〇 〇 〇 〇 ▧Vomiting ⊗ 〇 ⊗ 〇 〇 ⊗ ⊗ 〇 ⊗ N N
Page 21 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
22
% (95% CI)
Symptom Checker
(n=20)
Correct
Diagnosis
Listed First
Correct
Diagnosis
Listed
Appropriate
Triage
Ask MD 43 (26-59) 75 (61-89) - b
Drugs.com 40 (24-55) 58 (43-74) 60 (44-75)
EarlyDoc 32 (9-55) 37 (13-61) 53 (26-79)
Esagil 20 (8-33) 50 (35-65) - b
Family Doctor 47 (31-62) 56 (40-71) 54 (38-70)
FreeMD 36 (22-51) 48 (32-63) 59 (44-74)
HMS Family Health
Guide
34 (18-50) 55 (39-72) 78 (64-91)
Healthline 38 (23-53) 58 (43-73) - b
Healthwise - a -
a 43 (28-58)
Healthy Children - a -
a 73 (48-99)
Isabel 44 (29-60) 84 (73-95) 51 (36-66)
iTriage 36 (22-51) 77 (64-90) 33 (19-48)
Mayo Clinic 17 (5-29) 76 (62-89) - b
MEDoctor 5 (0-13) 43 (26-60) - b
NHS - a -
a 52 (37-68)
Symcat 40 (25-55) 76 (62-89) 44 (29-60)
Symptify 29 (15-43) 44 (29-60) 70 (55-85)
Symptomate 31 (14-48) 34 (17-52) 64 (36-93)
SymptomMD - a -
a 71 (57-86)
WebMD 36 (21-50) 62 (47-77) - b
Table 3|Accuracy of diagnosis decision and triage advice for each symptom
checker
Abbreviations: HMS, Harvard Medical School; NHS, National Health Services a Symptom checker does not provide diagnosis suggestions
b Symptom checker does not provide triage advice
Page 22 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
23
% (95% CI)
No. of
vignettes
(%)
Correct Diagnosis
Listed First P value
Correct
Diagnosis
Listed
P value Appropriate
Triage P value
All Vignettes 45 (100) 33 (30-37) 60 (56-63) 56 (52-61)
Type of SP vignette
Emergent 15 (33) 25 (19-31)
0.004
52 (45-59)
0.013
80 (74-86)
< 0.001 Non-emergent 15 (33) 37 (30-43) 62 (55-68) 55 (47-62)
Self-care 15 (33) 39 (32-45) 65 (59-72) 34 (27-41)
Table 4|Accuracy of diagnosis decision and triage advice for all symptom checkers, stratified by severity of the SP vignette and by
frequency of the condition
Page 23 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
24
No. (%)
% (95% CI)
% (95% CI)
No. (%)
% (95% CI)
All Symptom
Checkers
Symptom
Checkers
that
Diagnose
Correct
Diagnosis
Listed First
P value
Correct
Diagnosis
Listed
P value
Symptom
checkers
that provide
triage
advice
Appropriate
Triage P value
All Symptom
Checkers 20 (100) 16 (100)
33 (30-37)
60 (56-63)
14 (100)
56 (52-61)
Use nurse-triage books (Schmitt or Thompson)?
Yes 3 (15) 0 (0)
- a
- a
- a
- a
12 (86)
72 (60-84) 0.01
No 17 (85) 16 (100)
33 (30-37) 60 (56-63) 2 (14)
55 (50-59)
Asks demographic questions?
Yes 11 (55) 9 (56)
34 (29-39) 0.72
62 (57-66) 0.26
8 (57)
51 (45-56) 0.001
No 9 (45) 7 (44)
33 (27-38) 57 (51-63) 6 (43)
65 (59-72)
Site owner
Health plan or
government 3 (15) 1 (6)
36 (22-51)
0.9
77 (64-90)
0.02
3 (21)
43 (34-51)
< 0.001 Provider group 5 (25) 4 (25)
34 (26-41) 62 (55-70) 3 (21)
68 (58-77)
Private company 12 (60) 11 (69)
33 (29-37) 57 (52-62) 8 (57)
59 (53-65)
Maximum number of diagnoses listed
1-3 5 (25) 5 (25)
33 (26-40)
0.48
54 (47-61)
0.15
4 (29)
52 (44-60)
0.25 4-10 6 (30) 6 (30)
36 (30-42) 60 (54-66) 6 (43)
58 (52-65)
11+ 5 (25) 5 (25)
31 (24-37) 64 (57-70) 4 (29)
- b
Table 5|Accuracy of diagnosis given and triage advice for all symptom checkers given certain characteristics of the tools
a Symptom checker does not provide diagnosis suggestions
b Symptom checker does not provide triage advice
Page 24 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nlyAppendix
This appendix includes more details on the standardize patient (SP) vignettes that were used,
diagnosis and triage accuracy for each symptom checker, and the results of our sensitivity
analyses.
Our SP vignettes were gathered from several sources, which are listed in Supplemental Table 1.
Each vignette provided the age, gender, symptoms, and correct diagnosis for a given condition.
This table also notes where we added additional symptoms if the symptom checkers asked for
them. Added symptoms are italicized. The “simplified” symptoms were those inputted into
each symptom checker.
Supplemental Table 2 has additional information for Table 3 in the manuscript. This includes
the accuracy of the diagnosis decision and triage advice for each symptom checker with the
addition of the stratification by the severity of the SP vignette.
Lastly, we performed sensitivity analyses shown in Supplemental Table 3 to assess the
appropriateness of the triage advice of the symptom checkers by excluding certain symptom
checkers that were not as variable in their triage advice. This includes iTriage, which always
suggested that the user visit an emergency department, and Symcat, Symptomate, and Isabel,
all of which never suggest self-care. Excluding these symptom checkers only had a modest
impact on rates of appropriate triage advice.
Page 25 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
Diagnosis Vignette Simplified (added symptoms)
Acute liver failure¹ A 48-year-old woman with a history of migraine headaches presents to the emergency room with altered mental
status over the last several hours. She was found by her husband, earlier in the day, to be acutely disoriented and
increasingly somnolent. On physical examination, she has scleral icterus, mild right upper quadrant tenderness, and
asterixis. Preliminary laboratory studies are notable for a serum ALT of 6498 units/L, total bilirubin of 5.6 mg/dL, and
INR of 6.8. Her husband reports that she has consistently been taking pain medications and started taking additional
500 mg acetaminophen pills several days ago for lower back pain. Further history reveals a medication list with
multiple acetaminophen-containing preparations.
48 y/o f, confusion,
disorientation, increasingly
drowsy, mild right upper
quadrant tenderness, chronic
tylenol/acetaminophen -
recently took more
Appendicitis¹ A 12-year-old girl presents with sudden-onset severe generalized abdominal pain associated with nausea, vomiting,
and diarrhea. On exam she appears ill and has a temperature of 104°F (40°C). Her abdomen is tense with generalized
tenderness and guarding. No bowel sounds are present.
12 y/o f, sudden onset severe
abdominal pain, nausea,
vomiting, diarrhea, T=104
Asthma¹ A 27-year-old woman with a history of moderate persistent asthma presents to the emergency room with
progressive worsening of shortness of breath, wheezing, and cough over 3 days. She reports prior exposure to a
person who had a runny nose and a hacking cough. She did not receive significant relief from her rescue inhaler with
worsening symptoms, despite increased use. She has been compliant with her maintenance asthma regimen, which
consists of an inhaled corticosteroid and a leukotriene receptor antagonist for maintenance therapy and albuterol as
rescue therapy. Her cough is disrupting her sleep pattern and as a consequence she is experiencing daytime
somnolence, which is affecting her job performance.
27 y/o f, Hx of asthma, mild
shortness of breath,
wheezing, 3 days cough,
symptoms not responsive to
inhalers, recent cold
COPD flare (more severe)¹A 67-year-old woman with a history of COPD presents with 3 days of worsening dyspnea and increased frequency
of coughing. Her cough is now productive of green, purulent sputum. The patient has a 100-pack-year history of
smoking. She has had intermittent, low-grade fever of 100°F (37.7°C) for the past 3 days and her appetite is poor.
She has required increased use of rescue bronchodilator therapy in addition to her maintenance medications to
control symptoms.
67 y/o f, Hx of COPD, 3 days
worsening shortness of
breath, increase coughing,
green sputum, low grade
fever, increase use of rescue
bronchodilator therapy
Deep vein
thrombosis¹
A 65-year-old woman presents with unilateral leg pain and swelling of 5 days' duration. There is a history of
hypertension, mild CHF, and recent hospitalization for pneumonia. She had been recuperating at home but on
beginning to mobilize and walk, the right leg became painful, tender, and swollen. On examination, the right calf is 4
cm greater in circumference than the left when measured 10 cm below the tibial tuberosity. Superficial veins in the
leg are more dilated on the right foot and the right leg is slightly redder than the left. There is some tenderness on
palpation in the popliteal fossa behind the knee.
65 y/o f, 5 days swelling, pain
in one leg, recent
hospitalization, leg painful,
tender, swollen, red
Requires emergent care (n=15)
Supplemental Table 1: The 45 SP vignettes used to judge the symptom checkers’ accuracy and their condensed formats
Page 26 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
Heart Attack² Mr. Y is a 64 year old Chinese male who presents with chest pain for 24 hours. One day prior to presentation, the
patient began to experience 8/10, non-radiating substernal chest pressure associated with diaphoresis and
shortness of breath. The pain intially improved with Tylenol, however over the following 24 hours, his symptoms
worsened. The patient went to his primary physician, where an EKG was performed which showed ST elevation in
leads V2-V6.
64 y/o m, 1 day chest pain
(8/10), non-radiating
substernal chest pressure,
sweating, shortness of
breath, (chest tightness )
Hemolytic uremic
syndrome¹
A 4-year-old boy presents with a 7-day history of abdominal pain and watery diarrhea that became bloody after the
first day. Three days before the onset of symptoms, he had visited the county fair with his family and had eaten a
hamburger. Physical examination reveals a mild anemia
4 y/o m, 7 day Hx of
abdominal pain, bloody
diarrhea, ate hamburger at
fair 3 days ago
Kidney stones¹ A 45-year-old white man presents to the emergency department with a 1-hour history of sudden onset of left-sided
flank pain radiating down toward his groin. The patient is writhing in pain, which is unrelieved by position. He also
complains of nausea and vomiting.
45 y/o m, 1 hour severe left-
sided flank pain radiating into
groin, nausea, vomiting, pain
unrelieved by position
Malaria¹ A 28-year-old man presents to his physician with a 5-day history of fever, chills, and rigors, not improving with
acetaminophen (paracetamol), along with diarrhea. He had been traveling in Central America for 3 months,
returning 8 weeks ago. He had been bitten by mosquitoes on multiple occasions, and although he initially took
malaria prophylaxis, he discontinued it due to mild nausea. He does not know the specifics of his prophylactic
therapy. On examination he has a temperature of 100.4°F (38°C), and is mildly tachycardic with a BP of 126/82
mmHg. The remainder of the examination is normal.
28 y/o m, 5 day Hx of fever,
chills, rigors, diarrhea, recent
travel abroad to area with
malaria, bitten by
mosquitoes, did not take
malaria prophylaxis
consistently
Meningitis¹ An 18-year-old male student presents with severe headache and fever that he has had for 3 days. Examination
reveals fever, photophobia, and neck stiffness.
18 y/o m, 3 days severe
headache, fever,
photophobia, neck stiffness
Pneumonia³ A 65-year-old man with hypertension and degenerative joint disease presents to the emergency department with a
three-day history of a productive cough and fever. He has a temperature of 38.3°C (101°F), a blood pressure of
144/92 mm Hg, a respiratory rate of 22 breaths per minute, a heart rate of 90 beats per minute, and oxygen
saturation of 92 percent while breathing room air. Physical examination reveals only crackles and egophony in the
right lower lung field. The white-cell count is 14,000 per cubic millimeter, and the results of routine chemical tests
are normal. A chest radiograph shows an infiltrate in the right lower lobe.
65 y/o m, Hx of hypertension
and degenerative joint
disease, 3 day Hx of
productive cough and fever
(101)
Page 27 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
Pulmonary embolism A 65-year-old man presents to the emergency department with acute onset of SOB of 30
minutes' duration. Initially, he felt faint but did not lose consciousness. He is complaining of
left-sided chest pain that worsens on deep inspiration. He has no history of cardiopulmonary
disease. A week ago he underwent a total left hip replacement and, following discharge, was
on bed rest for 3 days due to poorly controlled pain. He subsequently noticed swelling in his
left calf, which is tender on examination. His current vital signs reveal a fever of 100.4°F
(38.0°C), heart rate 112 bpm, BP 95/65, and an O2 saturation on room air of 91%.
65 y/o m, shortness of
breath for 30 min,
chest pain that
worsens with
inspiration, recent
surgery, recent bed
rest, swelling in left
calf, which is tender,
fever
Pulmonary embolism. 2014;
https://online.epocrates.com/noFra
me/showPage?method=diseases&M
onographId=116&ActiveSectionId=22
. September 22, 2014.
Rocky Mountain
Spotted Fever
An 8-year-old boy in Oklahoma is brought to the emergency department over the fourth of July
weekend because of fever, chills, malaise, athralgias, and a headache. Physical examination
reveals a maculopapular rash that is most prominent on his wrists and ankles.
8 y/o m, Fever, chills,
joint pain, headache,
rash wrists/ankles
Plantz SH, Adler JN, eds. NMS
Emergency Medicine. Baltimore:
Williams & Wilkins; 1998. National
Medical Series for Independent
Study.
Stroke A 70-year-old man with a history of chronic HTN and atrial fibrillation is witnessed by a family
member to have nausea, vomiting, and right-sided weakness, as well as difficulty speaking and
comprehending language. The symptoms started with only mild slurred speech before
progressing over several minutes to severe aphasia and right arm paralysis. The patient is
taking warfarin.
70 y/o m, nausea,
vomiting, right-sided
weakness, rt arm
paralysis, difficulty
speaking and
comprehension
Hemorrhagic stroke. 2014;
https://online.epocrates.com/noFra
me/showPage?method=diseases&M
onographId=1079&ActiveSectionId=2
2. September 24, 2014.
Tetanus A 63-year-old man sustained a cut on his hand while gardening. His immunization history is
significant for not having received a complete tetanus immunization schedule. He presents
with signs of generalized tetanus with trismus ("lock jaw"), which results in a grimace described
as "risus sardonicus" (sardonic smile). Intermittent tonic contraction of his skeletal muscles
causes intensely painful spasms, which last for minutes, during which he retains consciousness.
The spasms are triggered by external (noise, light, drafts, physical contact) or internal stimuli,
and as a result he is at the risk of sustaining fractures or developing rhabdomyolysis. The
tetanic spasms also produce opisthotonus, board-like abdominal wall rigidity, dysphagia, and
apneic periods due to contraction of the thoracic muscles and/or glottal or pharyngeal
muscles. During a generalized spasm the patient arches his back, extends his legs, flexes his
arms in abduction, and clenches his fists. Apnea results during some of the spasms. Autonomic
overactivity initially manifests as irritability, restlessness, sweating, and tachycardia. Several
days later this may present as hyperpyrexia, cardiac arrhythmias, labile hypertension, or
hypotension.
65 y/o m, cannot open
mouth, contraction of
muscles causing
painful spasms for
minutes, sweating,
tachycardia, cut hand
while gardening, did
not get tetanus shot
Tetanus infection. 2014;
https://online.epocrates.com/noFra
me/showPage?method=diseases&M
onographId=220&ActiveSectionId=22
. September 23, 2014.
Acute otitis media An 18-month-old toddler presents with 1 week of rhinorrhea, cough, and congestion. Her
parents report she is irritable, sleeping restlessly, and not eating well. Overnight she
developed a fever. She attends day care and both parents smoke. On examination signs are
found consistent with a viral respiratory infection including rhinorrhea and congestion. The
toddler appears irritable and apprehensive and has a fever. Otoscopy reveals a bulging,
erythematous tympanic membrane and absent landmarks.
18 mo f, 1 week
rhinorrhea, cough,
congestion, irritable,
lack of appetite, fever,
in daycare
Otitis media. 2014;
https://online.epocrates.com/noFra
me/showPage?method=diseases&M
onographId=39&ActiveSectionId=22.
September 22, 2014.
Requires non-emergent care (n=15)
Page 28 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
Acute pharyngitis¹ A 7-year-old girl presents with abrupt onset of fever, nausea, vomiting, and sore throat. The child denies cough,
rhinorrhea, or nasal congestion. On physical exam, oral temperature is 101°F (38.5°C) and there is an exudative
pharyngitis, with enlarged cervical lymph nodes. A rapid antigen test is positive for group A Streptococcus (GAS).
7 y/o f, fever (101), nausea,
vomiting, sore throat, swollen
lymph nodes, tonsilar
exudate; no cough,
rhinorrhea, or nasal
congestion
Acute pharyngitis5 Mr. A is a 24 year-old man who presents to your office for complaints of sore throat, fever, and headache. His
symptoms started 2 days ago with acute onset of sore throat and fever to 102.2. He has had no cough. His physical
examination is normal, except for the presence of tonsillar exudates and some tender anterior cervical
lymphadenopathy. He is otherwise in good health, and is on no medications except for ibuprofen for fever. He has
no drug allergies. (, Centor score = 4 – treat, or test and treat)
24 y/o m, sore throat, fever
(102.2), headache, no
cough,tonsilar exudates
Acute sinusitis5 Mrs. S is a 35 year-old woman who presents with 15 days of nasal congestion. She has had facial pain and green
nasal discharge for the last 12 days. She has had no fever. On physical examination, she has no fever and the only
abnormal finding is maxillary tenderness on palpation. She is otherwise healthy, except for mild obesity. She is on no
medications, except for an over-the-counter decongestant. She has no drug allergies
35 y/o f, sx for 15 days, nasal
congestion, facial pain, green
nasal discharge, no fever
Back pain6 Consider a 35-year-old man who developed low back pain after shoveling snow 3 weeks ago. He presents to the
office for an evaluation. On examination there is a new left foot drop. In study 82% physicians recommend MRI
(sciatica/sprain)
35 y/o m, back pain following
shoveling, left foot drop,
symptoms 3 weeks of
duration (loss of sensation in
foot)
Cellulitis¹ A 45-year-old man presents with acute onset of pain and redness of the skin of his lower extremity. Low-grade fever
is present and the pretibial area is erythematous, edematous, and tender.
45 y/o m, pain and redness of
skin, low grade fever,
redness, edema, and
tenderness lower leg
COPD flare (milder)¹ A 56-year-old woman with a history of smoking presents to her primary care physician with shortness of breath and
cough for several days. Her symptoms began 3 days ago with rhinorrhea. She reports a chronic morning cough
productive of white sputum, which has increased over the past 2 days. She has had similar episodes each winter for
the past 4 years. She has smoked 1 to 2 packs of cigarettes per day for 40 years and continues to smoke. She denies
hemoptysis, chills, or weight loss and has not received any relief from over-the-counter cough preparations.
56 y/o f, Hx of smoking,
shortness of breath and
cough for several days,
rhinorrhea 3 days ago, white
sputum, no chills
Influenza¹ A 30-year-old woman presents in January with 2-day history of fever, cough, headache, and generalized weakness.
She was in her usual state of health before an abrupt onset of these symptoms. A few viral illnesses have affected
her during the current winter, but not to this severity. She reports sick contacts at work and did not receive the
seasonal influenza vaccine this season.
30 y/o f, 2 day fever, cough,
headache, weakness, did not
get flu shot
Page 29 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
Mononucleosis¹ A 16-year-old female high school student presents with complaints of fever, sore throat, and fatigue. She started
feeling sick 1 week ago. Her symptoms are gradually getting worse, and she has difficulty swallowing. She has had a
fever every day, and she could hardly get out of bed this morning. She does not remember being exposed to
anybody with a similar illness recently. On physical examination she is febrile and looks sick. Enlarged cervical lymph
nodes, exudative pharyngitis with soft palate petechiae and faint erythematous macular rash on the trunk and arms
are found.
16 y/o f, 1 week Hx of fever,
sore throat, fatigue, difficulty
swallowing, fever, enlarged
lymph nodes, exudates,
macular rash on trunk/arms
Peptic Ulcer Disease¹ A 40-year-old man presents to his primary care physician with a 2-month history of intermittent upper abdominal
pain. He describes the pain as a dull, gnawing ache. The pain sometimes wakes him at night, is relieved by food and
drinking milk, and is helped partially by ranitidine. He had a similar but milder episode about 5 years ago, which was
treated with omeprazole. Physical examination reveals a fit, apparently healthy man in no distress. The only
abnormal finding is mild epigastric tenderness on palpation of the abdomen.
40 y/o m, 2 month Hx of
intermittent upper abdominal
pain, dulling and gnawing
ache, wakes at night and is
relieved by food/drinking
milk/ranitidine, prior episode
5 yrs ago
Pneumonia¹ A 6-year-old boy with a medical history significant for mild persistent asthma is brought to the clinic by his mother
with a history of a 5-day cough. His mother reports that the child's fever continues to be elevated despite
acetaminophen therapy. He has missed school for the past 3 days and he has a classmate sick with pneumonia. The
mother reports that the appetite is good for the child. His cough produced yellowish sputum at home. His vitals at
the clinic are: respiratory rate 19 breaths/min, heart rate 80 beats/min, and temperature 101.6°F (38.7°C). He
appears in no respiratory distress. His lung examination reveals bilateral rales and occasional wheeze. CXR reveals
lobar infiltrates without pleural effusions.
6 y/o m, Hx of asthma, 5 days
cough, fever, appetite good,
yellow sputum, t 101.6
Salmonella¹ A 14-year-old boy presents with nausea, vomiting, and diarrhea. Eighteen hours earlier, he had been at a picnic
where he ingested undercooked chicken along with a variety of other foods. He reports moderate-volume,
nonbloody stools occurring 6 times a day. He has mild abdominal cramps and a low-grade fever. He is evaluated at
an acute care clinic and found to be mildly tachycardic (heart rate 105 bpm) with a normal BP and a low-grade
temperature of 100.1°F (37.8°C). His physical exam is unremarkable except for mild diffuse abdominal tenderness
and mild increased bowel sounds. He is able to take oral fluids and is instructed on the appropriate oral fluid and
electrolyte rehydration.
14 y/o m, nausea, vomiting,
non-bloody diarrhea, mild
abdominal cramps (T=100.1),
mild abdominal tenderness,
diarrhea after attending a
picnic and eating
undercooked chicken,
Shingles¹ A 77-year-old man reports a 5-day history of burning and aching pain on the right side of his chest. This is followed
by the development of erythema and a maculopapular rash in this painful area, accompanied by headache and
malaise. The rash progressed to develop clusters of clear vesicles for 3 to 5 days, evolving through stages of
pustulation, ulceration, and crusting.
77 y/o m, 5 day burning and
aching on right side of chest,
erythema, maculopapular
rash, headache, malaise, rash
progressed to clear vesicles
after 3-5 days
Page 30 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
Urinary tract
infection¹
A 26-year-old female newly wed presents complaining of painful urination, feeling of urgent need to urinate, and
more frequent urination for 2 days. She denies any fever, chills, nausea, vomiting, back pain, vaginal discharge, or
vaginal pruritus.
26 y/o f, painful urination,
urgent need to urinate, more
frequent urination for 2 days,
sexually active; no fever,
chills, nausea, vomiting, back
pain, vaginal discharge,
vaginal pruritus
Vertigo¹ A 65-year-old woman presents with a chief complaint of dizziness. She describes it as a sudden and severe spinning
sensation precipitated by rolling over in bed onto her right side. Symptoms typically last <30 seconds. They have
occurred nightly over the last month and occasionally during the day when she tilts her head back to look upward.
She describes no precipitating event prior to onset and no associated hearing loss, tinnitus, or other neurologic
symptoms. Otologic and neurologic examinations are normal except for the Dix-Hallpike maneuver, which is negative
on the left but strongly positive on the right side.
65 y/o f, dizziness, sudden
onset, recurrent, lasts <30
sec, consistent trigger, no
hearing loss, ringing in ears,
muscle weakness, loss of
sensation
Acute bronchitis¹ A 34-year-old woman with no known underlying lung disease 12-day history of cough. She initially had nasal
congestion and a mild sore throat, but now her symptoms are all related to a productive cough without paroxysms.
She denies any sick contacts. On physical examination she is not in respiratory distress and is afebrile with normal
vital signs. No signs of URI are noted. Scattered wheezes are present diffusely on lung auscultation.
34 y/o f, 12 day cough, initial
nasal congestion and sore
throat, cough, no fever
Acute bronchitis5 Mrs. L is a 61 year-old woman who presents with 4 days of a cough productive of yellow sputum. Her symptoms
started 4 days ago with rhinorrhea and productive cough. She initially had fevers as high as 101 for 2 days, but those
have now resolved. In the office, she has normal vital signs and a normal physical examination. She is otherwise
healthy except for high cholesterol for which she is being treated with atorvastatin. She has no drug allergies.
61 y/o f, 4 day cough, yellow
sputum, rhinorrhea, fever
(resolved)
Acute conjunctivitis¹ A 14-year-old boy with no significant past medical history presents 3 days after developing a red, irritated right eye
that spread to the left eye today. He has watery discharge from both eyes and they are stuck shut in the morning. He
reports recent upper respiratory symptoms and that several children at his day camp recently had pink eye. He
denies significant pain or light sensitivity and does not wear contact lenses. On examination, his pupils are equal and
reactive and he has a right-sided, tender preauricular lymph node. Penlight examination does not reveal any corneal
opacity.
14 y/o m, 3 days red, irritated
eye (spread from right to
left), discharge, URI
symptoms, no pain or light
sensitivity
Acute pharyngitis5 Mr. E is a 26 year-old man who presents to your office for complaints of sore throat, headache, and non-productive
cough. His symptoms started 2 days ago with acute onset of sore throat. He has been afebrile. His physical
examination is normal, except for some pharyngeal erythema. He is otherwise in good health, and is on no
medications except for acetaminophen for his sore throat and fever. He has no drug allergies.
26 y/o m, 2 day sore throat,
headache, cough, no fever
Self-care appropriate (n=15)
Page 31 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
Allergic rhinitis¹ A 22-year-old student presents with a 5-year history of worsening nasal congestion, sneezing, and nasal itching.
Symptoms are year-round but worse during the spring season. On further questioning it is revealed that he has
significant eye itching, redness, and tearing as well as palate and throat itching during the spring season. He
remembers that his mother told him at some point that he used to have eczema in infancy.
22 y/o m, 5 year Hx of nasal
congestion, sneezing, nasal
itching worse during spring
season, eye itching, redness,
tearing, palate and throat
itching, Hx of eczema in
infancy
Back pain¹ A 38-year-old man with no significant history of back pain developed acute LBP when lifting boxes 2 weeks ago. The
pain is aching in nature, located in the left lumbar area, and associated with spasms. He describes previous similar
episodes several years ago, which resolved without seeing a doctor. He denies any leg pain or weakness. He also
denies fevers, chills, weight loss, and recent infections. Over-the-counter ibuprofen has helped somewhat, but he
has taken it only twice a day for the past 3 days because he does not want to become dependent on painkillers. On
examination, there is decreased lumbar flexion and extension secondary to pain, but a neurologic exam is
unremarkable.
38 y/o m, acute low back pain
after lifting, no leg pain or
weakness, no fevers, chills,
weight loss, or recent
infections
Bee sting without
anaphylaxis¹
A 9-year-old boy is brought to the ER after being stung by a bee at a picnic. He is crying hysterically. After 15 minutes
of calming him down, exam reveals a swollen tender upper lip but no tongue swelling, no drooling, no stridor, no
rash, and no other complaints.
9 y/o m, bee sting, swollen
and tender upper lip; no
tongue swelling, drooling,
stridor, rash, or other
complaints
Canker sore¹ A 17-year-old male student presents with recurrent mouth ulceration since his early schooldays. He has no
respiratory, anogenital, gastrointestinal, eye, or skin lesions. His mother had a similar history as a teenager. The
social history includes no tobacco use and virtually no alcohol consumption. He has no history of recent drug or
medication ingestion. Extraoral exam reveals no significant abnormalities and specifically no pyrexia; no cervical
lymph node enlargement; nor cranial nerve, salivary, or temporomandibular joint abnormalities. Oral exam reveals a
well-restored dentition and there is no clinical evidence of periodontal-attachment loss or pocketing. He has five 4
mm round ulcers with inflammatory haloes in his buccal mucosae.
17 y/o m with recurrent
mouth ulceration for year, no
respiratory, anogenital,
gastrointestinal, eye, or skin
lesions, mother has similar
Hx, no Hx of recent drugs or
medication
Candidal yeast
infection6
Consider a 40-year-old, monogamous, married woman who calls to report a 2-day history of vaginal itching and
thick white discharge. She has no abdominal pain or fever. (in study 50% recommended physician visit)
40 y/o f, 2 day vaginal itching,
thick white discharge, no
abdominal pain or fever
Page 32 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
Constipation¹ A 5-month-old baby boy presents with difficulty and delay in passing hard stools. His mother reports that he strains
for several hours and may even miss a day, before passing stool with screaming and occasional spots of fresh blood
on the stool or diaper. He has recently been weaned from breastfeeding to cows' milk formula, which he had been
reluctant to drink initially. The child is thriving and now feeding normally. There was no neonatal delay in defecation
and no history of excessive vomiting or abdominal distension.
5 mo m, difficulty/delay in
passing hard stools, strains
for hours, may miss a day,
screams when passes stool
and occasional spots of
blood, weaned from
breastmilk to cows' milk, now
feeding normally
Eczema¹ A 12-year-old female presents with dry, itchy skin that involves the flexures in front of her elbows, behind her knees,
and in front of her ankles. Her cheeks also have patches of dry, scaly skin. She has symptoms of hay fever and has
recently been diagnosed with egg and milk allergy. She has a brother with asthma and an uncle and several cousins
who have been diagnosed with eczema.
12 y/o f, dry, itchy skin in
front of elbows, behind
knees, in front of ankles,
cheeks have patches of dry,
scaly skin, symptoms of hay
fever, egg and milk allergy,
brother has asthma and uncle
and cousins have eczema
Stye¹ A 30-year-old man presents with a painful, swollen right eye for the past day. He reports minor pain on palpation of
the eyelid and denies any history of trauma, crusting, or change in vision. He has no history of allergies or any eye
conditions and denies the use of any new soaps, lotions, or creams. On exam, he has localized tenderness to
palpation and erythema on the midline of the lower eyelid near the lid margin. The remainder of the physical exam,
including the globe, is normal.
30 y/o m, painful, swollen
right eye for past day, no Hx
of trauma, crusting, change in
vision, allergies, or eye
conditions, localized
tenderness, erythema
(redness)
Viral upper respiratory illnessMr. R. is a 56 year-old man who presents to you with 6 days of non-productive cough, nasal congestion, and green
nasal discharge. He has had intermittent fevers as high as 100.8. His physical examination is normal except for
rhinorrhea. He is otherwise healthy, except for chronic osteoarthritis of the right knee. He has no drug allergies.
56 y/o m, 6 day cough, nasal
congestion, green nasal
discharge, fever (100.8),
rhinorrhea
Page 33 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
Viral upper respiratory illness¹A 30-year-old man presents with a 2-day history of runny nose and sore throat. He feels hot and sweaty, has a mild
headache, is coughing up clear sputum and complains of muscle aches. He would like antibiotics as he was
prescribed them last year when he had a similar condition. On examination, he is afebrile, has a normal pulse, a
slightly inflamed pharynx and nontender cervical lymphadenopathy. There is no neck stiffness and his chest is clear.
He has tried over-the-counter cough medications, but has not found these helpful. He smokes 10 cigarettes per day.
30 y/o m, 2 day HX of runny
nose, sore throat, hot,
sweaty, mild headache, cough
with clear sputum, muscle
aches, no fever or neck
stiffness
Vomiting7 Elizabeth’s 2-year-old son has a fever and vomited twice. Elizabeth worries about dehydration, so she gives Jack a
sippy cup of apple juice. He immediately vomits up the juice. Elizabeth debates what to do next. Should she try to
reach Jack’s pediatrician or should she take Jack to the ED? Instead, she calls her triage nurse line. Temperature =
100.5
2 y/o m, low grade fever (T =
100.5), vomited twice, vomits
up juice
Table References
1. Epocrates. https://online.epocrates.com/noFrame/.
2. Lue J. NYU Medical Grand Rounds Clinical Vignette. 2012;
http://www.medicine.med.nyu.edu/education/im-residency-homepage/research-
opportunities/clinical-vignettes. Accessed September 8, 2014.
3. Halm EA, Teirstein AS. Clinical practice. Management of community-acquired pneumonia. N Engl J
Med. Dec 19 2002;347(25):2039-2045.
4. Plantz SH, Adler JN, eds. NMS Emergency Medicine. Baltimore: Williams & Wilkins; 1998. National
Medical Series for Independent Study.
5. Gidengil CA, Linder J, Beach S, Setodjian C, Hunter G, Mehrotra A. Using clinical vignettes to predict
antibiotic prescribing for acute respiratory infections. In review.
6. Sirovich BE, Gottlieb DJ, Welch HG, Fisher ES. Variation in the tendency of primary care physicians
to intervene. Arch Intern Med. Oct 24 2005;165(19):2252-2256.
7. Boroughs DS, Dougherty JA, Goldsmith C. Telephone Triage: Help Is Just a Call Away.
http://ce.nurse.com/RVignette.aspx?TopicId=718. Accessed September 10, 2014.
Page 34 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review Only
OverallRequires
emergent care
Requires non-
emergent care
Self-care
appropriateOverall
Requires
emergent care
Requires non-
emergent care
Self-care
appropriateOverall
Requires
emergent care
Requires non-
emergent care
Self-care
appropriate
AskMD 43 29 47 55 75 64 80 82 - b
- b
- b
- b
Drugs.com 40 29 40 54 58 50 60 70 60 57 60 62
EarlyDoc 32 0 17 83 37 0 17 1 53 67 60 33
Esagil 20 7 27 27 50 57 40 53 b0
b b b
Family Doctor 47 40 50 50 56 40 50 50 54 50 50 60
FreeMD 36 33 33 43 48 40 53 50 59 67 87 21
HMS Family Health Guide 34 36 31 38 55 50 54 62 78 92 79 62
Healthline 38 33 31 39 58 40 60 73 - b
- b
- b
- b
Healthwise - a
- a
- a
- a
- a
- a
- a
- a 43 100 7 21
Healthy Children - a
- a
- a
- a
- a
- a
- a
- a 73 100 100 43
Isabel 44 33 73 27 84 80 87 87 51 100 53 0
iTriage 36 14 47 47 77 76 87 67 33 100 0 0
Mayo Clinic 17 7 21 23 76 57 86 85 - b
- b
- b
- b
MEDoctor 5 8 67 8 43 31 67 33 - b
- b
- b
- b
NHS - a
- a
- a
- a
- a
- a
- a
- a 52 87 20 50
Symcat 40 27 60 33 76 60 73 93 44 53 80 0
Symptify 29 7 20 60 44 33 40 60 70 92 71 50
Symptomate 31 50 30 10 34 58 30 10 64 78 67 0
SymptomMD - a
- a
- a
- a
- a
- a
- a
- a 71 86 71 57
WebMD 36 27 40 40 62 60 53 73 - b
- b
- b
- b
Correct Principle Diagnosis (%) Correct Diagnosis listed (%) Appropriate Triage (%)
Supplemental Table 2: Accuracy of diagnosis decision and triage advice for each symptom checker, stratified by severity of the SP vignette
a Symptom checker does not provide diagnosis suggestions b Symptom checker does not provide triage advice
Page 35 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Confidential: For Review O
nly
Appropriate triage
% (95% CI)
All symptom checkers 56% (52-61)
Without iTriage 59% (54-63)
Without Symcat, Symptomate, and Isabel 58% (53-63)
Without Symcat, Symptomate, Isabel, and iTriage 61% (56-66)
Supplemental Table 3: Sensitivity analysis for appropriateness of triage
advice when symptom checkers that always provide advice to go to the
emergency department are removed (iTriage) and when symptom
checkers that never suggest self-care are removed (Symcat,
Symptomate, and Isabel).
Page 36 of 36
https://mc.manuscriptcentral.com/bmj
BMJ
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960