Alan S. Brody, MD, Editor

Evaluating Online Radiology Information Resources

From the beginning, the Electronic.Resource.Site has

referred readers to lnternet sites for information. In this

column, Paul Guillerman addresses an important issue:

How do we assess the accuracy of Internet information? With some computer expertise, or a little help, anyone can

develop a professional-appearing Web site. The appear-

ance of the site does not necessarily reflect the expertise

(or lack of it) of the person who wrote the content. Dr Guillerman ' s review, and the sites he presents, should be

useful to all radiology Internet users and Web site develop- ers.--A.S.B.

.~UILDELINES FOR WEB SITE5

The past several years have witnessed an explosion in the availability and use of medical information resources on the Internet. According to a 1998 survey conducted by PSL Consulting, over 75% of American physicians access the Internet on a daily or weekly basis to find in- formation on diseases, to read medical journals, or to visit medical association sites (1). Physicians are not alone in their appetite for online medical resources. A recent survey found that 17.5 million Americans use the Internet to search for health information, often due to the lack of information given to them by their health care providers (2).

The enthusiasm for online medical resources must be tempered by the realization that quantity does not neces- sarily beget quality. The danger of inaccurate or misun- derstood online medical information has been well docu- mented in the medical literature and lay press (3-8). To address this risk, several organizations have developed guidelines for the creation and evaluation of online medi- cal resources. Radiologists who use the Internet, whether to browse for medical information or actually author medical Web sites, would be well served by becoming familiar with some of the more widely recognized guide- lines.

The Health on the Net Foundation Code of Conduct (http

://www.hon.ch/HONcode/Conduct_t.html) was devised by a nonprofit international organization headquartered in Geneva, Switzerland. The code is a self-governance initia- tive for Web masters entailing principles that address author' s credentials, date of modification, data confidential- ity, references, funding, and advertising policy.

Criteria for Assessing the Quality of Internet Health Infor- mation (http://hitiweb.mitretek.org/docs/policy.htmI) was de- veloped to address the need for objective, reproducible crite- ria to evaluate the online health information quality. Toward this end, Mitretek Systems' Health Information Technology Institute convened a Health Summit Working Group of health care providers, information science professionals, and Web site developers. The group proposed the following eval- uation criteria: credibility (source [identity, credentials, con- flict of interest, bias], context, currency, relevance/utility, editorial review process); content (accuracy, hierarchy of evidence, original sources, disclaimers, omissions noted); disclosure (site purpose, profiling); links (selection, architec- ture, content, back linkages, descriptions); design (accessibil- ity, navigability, search engine); interactivity (feedback mechanism, chat rooms, tailoring); and caveats.

OMNI Guidelines for Resource Evaluation (http

://omni.ac.uk/agec/evalguid.html) is a gateway to Internet health care resources funded by the UK Electronic Libraries Program. The Organising Medical Networked Information (OMNI) guidelines espouse the criteria of content evaluation (coverage, accuracy, currency, uniqueness); context evalua- tion (scope, audience, authority, provenance); and access evaluation (availability/usability, access restrictions, charg- ing policy, special requirements, software reliability, copy- right, language, design layout/user interface, user support/ documentation).

561

The Evaluation Reporting Template for Interactive

Health Communication Applications (http://www.scipich .org/template/evalintro.htm) was developed by the Science Panel on Interactive Communication and Health, a non- federal panel of medical, public health, health systems en-

gineering, decision science, computer science, and com- munication experts. The template was designed to prevent

developers from producing flawed applications, help users determine which applications are most likely to benefit them, aid clinicians in selecting applications for their pa-

tients, and help policymakers invest in the most cost-effec- tive applications. The template is organized into four sec-

tions: describing the application, formative and process evaluation, outcome evaluation, and evaluator background.

Although published accounts of patient harm as a direct result of erroneous online radiology information

have yet to surface, increased risk of such an event is incurred by the growing prevalence of Internet teaching files, multimedia textbooks, and tutorials that have not undergone formal peer review. Fortunately, radiologists have started to become active in the evaluation of online radiology information resources.

At AJR Web Review (http://www.arrs.org/eajr), Web site reviews are conducted by the American Roentgen Ray

Society with procedures similar to those used to select pa-

pers for publication in the American Journal of Roentgen- ology. The review abstract describes the goal, learning ob- jective, and content of the Web site.

The Radiological Society of North America Launch Pad (http://www. rsna. org/REG/launchpad/launchpad, html) is a meta-link that verifies the quality and relevance of the

external resources linked by the site. However, the method of verification is not detailed, and the site does not claim to peer review the linked resources.

PediatricRadiology.com (http://pediatricradiology.com) is curated by Michael P. D'Alessandro. This Web site fol- lows the Health on the Net Foundation code of conduct

and engages in peer review by accreditation. To be accred- ited, an online resource must be free, be coded in hyper- text markup language format for compatibility with com- mon Web browsers, and adhere to the evaluation criteria of authorship (identity, affiliation, credentials), attribu-

tion (reference listing), disclosure (ownership and spon-

sorship), and currency (date of content posting). Since online informational resources promise to grow

in number and influence, it is imperative that more radi-

ologists become active in the development and application

of evaluation criteria to prevent inaccurate content and poor design from compromising radiology education and clinical practice. This may be accomplished by radiolo- gists learning the method of Web site evaluation and insti-

tuting peer review of existing and future online radiology resources. However, several issues must be addressed be- fore peer review of online radiology resources can be

implemented in a consistent, standardized format to pre- vent the variable quality and utility that plague many ex-

isting evaluation systems (9). For example, what specific evaluation criteria should be

used and how should they be prioritized? What online radi- ology resources should be peer reviewed, how often, and

by whom? How should peer review of online radiology re- sources be encouraged? Once these questions are answered,

online radiology resources can be associated with a level of quality assurance equivalent to that of traditional academic

publications, such as journals and textbooks. More active

participation of radiologists in medical Web site evaluation can uphold the tradition of radiology being at the forefront of computer applications in medicine and expand the sphere of influence of radiologists in the health care field. Most im- portant, this activity can help ensure that no patient is harm- ed as a result of erroneous radiology information obtained on the Internet.

R. Paul Guillerman, MD Children's Hospital Medical Center

Cincinnati, Ohio

1. PSL Consulting. Physician survey. Available at: http://www.cyberatlas .com/Nghlights/health.html. Accessed June 18, 1999.

2. Oeloitte & Touche, VHA. 1999 Environmental assessment: rising to the challenge of a new century. Available at: http://www.cyberatlas.com /market/professional/heaRh_net.html. Accessed June 18, 1999.

3. Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the punic on the World Wide Web: systematic survey ef advice on managing fever in children at home. BMJ 1997; 314:1875- 1879.

4. Weisbord SD, Soule JB, Kimmel PL. Poison on line: acute renal failure caused by oil of wormwood purchased through the Internet. N Engl J Med 1997; 337:825-827.

5. Keoun B. Cancer patients find quackery on the Web. J Natl Cancer Inst 1996; 88:1263-1265.

6. Silberg WM, Lundberg GD, MusaccNo RA. Assessing, controlling, and assuring the quality of medical information on the Internet: caveant lec- tor et viewer--let the reader and viewer beware. JAMA 1997; 277:1244- 1245.

7. Goldwein JW, Benjamin I. Internet-based medical information: time to take charge. Ann Intern Med 1995; 123:152-153.

8. Bulkeley WM. E-mail medicine: untested treatments, cures find strong- hold in on-line services. Wall Street Journal. February 27, 1995:A1.

9. Jadad AR, Gagliardi A. Rating health information on the Internet: navi- gating to knowledge or to Babel? JAMA 1998; 279:611-614.

562