pdrd—a computer-based primary care decision support system
TRANSCRIPT
MEI). INFORM. (1993), VOL. 18, NO. 2, 103-112
PDRD-a computer-based primary care decision support system
J. JOUSIMAAt and I. KUNNAMO Finnish Medical Society Duodecim, University of Kuopio, Department of Community Health and General Practice, Kuopio, Finland
(Received September I YYZ)
Abstract. This article identifies the current stage of development and assessment of computer-assisted decision support systems in the domain of general practice. Physician’s Desk Reference and Database (PDRD), an electronic medical database, is presented, and a plan for assessment of P D R D is briefly discussed.
KeJvwords: Decision making; Computer-assisted; Primary health care.
1. Introduction Medical computerized decision support systems (CDS) have been broadly
defined to be any computer programs designed to help health professionals make clinical decisions [l] . These systems have been further divided into ‘passive’ information management tools and ‘active’ patient-specific systems that provide customized advice combining medical knowledge and patient-specific data [ 1,2]. Some authors reserve the term ‘expert systems’ to the latter group of systems [2], while others regard it as synonymous with CDS [3].
The first articles dealing with the possibility of machines assisting in clinical decision-making appeared in the late 1950s [4]. Since that time the computer world has been revolutionized by the emergence of cheap microprocessors and the abundance of personal microcomputers. These changes have also been reflected in medical computing. In the 1970s major technical innovations used mainframe systems, while in the 1980s and 1990s individually designed microcomputer workstation software and local area networks have been the major directions of progress.
Computers have been adopted in medical care for keeping medical records, enhancing the availability of laboratory data and speeding up routine paperwork. Such systems may contain tools for focusing attention, e.g. flagging abnormal laboratory results and warning of drug interactions [S]. Computer-based systems allowing rapid access to large databases like MEDLINE and electronic textbooks like The Oxford Textbook of Medicine CD-ROM version, are also in clinical use.
The implementation of active CDS systems has been less successful despite active research and development work. Experimental systems or pilot versions have usually concentrated on relatively circumscribed clinical topics such as differential diagnosis of dementia [6], thyroid function tests [7] or nephropathia epidemica [8]. There are only a few systems that are planned to some extent to meet the general practitioners’ needs for information. DXPLAIN [2] is an example of a system that works via modern calls to host computers. MEDITEL adult system [9], Quick
f Address correspondence to: Dr J . Jousimaa, Laurinpuistikko 14 D 18, 80160 Joensuu, Finland.
1)307-7640/93 %10~00 0 1093 ’l’aylor 8; 1:rancis I,td
Info
rm H
ealth
Soc
Car
e D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y Q
UT
Que
ensl
and
Uni
vers
ity o
f T
ech
on 1
1/21
/14
For
pers
onal
use
onl
y.
104 J . Jousimaa and I . Kunnamo
Medical Reference QMR [lo], Oxford System of Medicine OSM [ l l ] and ILIAD [12] are software that require installation on a microcomputer hard disk. The domain in these programs is mainly in the field of internal medicine.
2. Development and assessment of general practice CDS systems In a cross-national study 178 GPs from nine European countries were
interviewed about their attitudes towards data and decision support in general practice [ l 11. The general attitudes were very positive, and the majority of GPs in most countries were in favour of a computer on their consulting-room desk.
Although there are local differences in the role of general practitioners and their working facilities, the need for continuous updating of medical knowledge is urgent. Information overload and rapid progress in diagnostic, and therapeutic procedures makes it difficult for doctors to keep up to date and to be able to handle the whole spectrum of patients’ problems and diseases. In many countries, GPs have a gate-keeper role on access to specialists. With growing cost-consciousness they might be able to limit the number of referrals without risking patient safety or satisfaction.
In a study where the self-reported and observed information sources of British doctors in office practice were analysed, they favoured textbooks, print sources, computerized literature reviews and computers to answer diagnostic and treatment questions during consultations [13]. In the same study it was noted that the textbooks were often out of date, and print sources of poor quality. The need to develop information sources that are easy and fast to use and update is evident. At present software and databases installable on hard disk or stored on a CD-ROM offer the fastest way to obtain information during consultations, though modem-linked connections are useful in cases where connections to large databases or electronic mailboxes are required. The advantages of centralized information sources are easy updating and minimal user investment (software not necessarily required), but their relative slowness prevents their frequent use. The recent development of reasonably priced optical and magneto-optical mass memories with enormous data-storing capacities speaks in favour of personal database use. CDS systems will probably have their greatest acceptance, when linked to workstation programs that take care of the routine office paperwork and consultation statistics [ 11.
The delay of CDS application in medicine, and especially general practice, can be explained by several factors. Possible reasons are the complex and often non-biomedical basis of problems. Also the co-existence of several conditions and patients’ and physicians’ different views and expressions of the nature of the diseases may render the data input and processing by computer logic more difficult. As very specialized diagnostic problems require sophisticated software and large databases, so collecting reliable data to cover most of the problems appearing in primary care is enormously laborious.
Major changes in the traditional ways of solving problems and teaching medical students may be required after the appearance of new CDS systems, and it is likely to cause some resistance and mistrust-even fear of being replaced by machines. Most authors see the role of CDS to serve as tools for the doctor, who retains ultimate responsibility for the diagnostic and therapeutic procedures [3,9,14]. Diagnostic and therapeutic choices can be more individualized, and the medical benefit can be weighed against patients’ preferences. CDS cannot be expected to
Info
rm H
ealth
Soc
Car
e D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y Q
UT
Que
ensl
and
Uni
vers
ity o
f T
ech
on 1
1/21
/14
For
pers
onal
use
onl
y.
PDRD-computer-based decision support system 105
suggest correct diagnoses in all situations, only to prompt the doctor to consider the most likely options and to avoid mistakes.
Physicians’ acceptance of computers and CDS systems has been assessed in many studies [15,16]. Negative attitudes towards computing would not seem to be a major problem according to the cross-national study in Europe [l l] , but the interviewed GPs were leaders and innovators rather than a random sample, which may weaken the generalizability of the results. Some resistance to the introduction of information technology exists in all professions, and has been also reported among doctors [17]. An important observation is also that the guidelines for medical practice did not necessarily move actual practice closer to the behaviours the guidelines recommended [18].
Proper validation of the database should be performed before adopting the system for routine use. Most studies have concentrated on assessing the performance of the system compared with the diagnostic accuracy of specialists or specialist groups. In many cases the computer programs have been found to be equal to, or even better than, specialists in giving accurate suggestions under experimental conditions [6,9, lo]. These studies do not, however, answer the question of whether the systems will really be used in everyday practice.
A critical approach towards CDS systems is still warranted, as proper evaluation studies of existing systems have not yet been performed. In a meta-analysis of the most recently (1 985-1 990) evaluated expert systems, nine articles qualified for review. Of these, all were focused on the performance of the system in experimental settings, only two assessed their influence on the physicians’ differential diagnosis and none of them measured any outcome of the use of the system [19]. In a recent study an expert system was found to improve medical student diagnostic problem-solving and decision making [20]. This study did not, however, use any control group of system non-users, and other factors influencing learning, such as lectures, ward rounds and reading, could not be excluded from having accounted for the learning.
According to Weaver [21], an empirical study to compare systematically the health benefits or costs that result from the use or non-use of any CDS system: (1) should compare the advice of physicians at different levels of expertise; (2) should evaluate the advice in terms of patient outcomes; (3) would need to use the system for all patients; (4) would need the participation of several physicians.
This study has probably not yet been performed, because of the developmental level of the systems to be analysed. If the system will not be used for all patients, analysis of the cases when the CDS system was not used would also be useful.
The need to perform prospective technology assessment studies at the current developmental stage has been emphasized [22]. In a study where computer scientists, physicians, health care administrators, representatives of health insurance organizations and trade unions were interviewed for assessing the potential benefits and risk of expert systems in medicine, a majority (35 of 42) estimated that the systems would contribute to the increase of the quality of care within the next 10-15 years [3].
3. Physician’s Desk Reference and Database (PDRD) 3, l . Technical and user data
The PDRD is an electronic textbook published by the Finnish Medical Society Duodecim [23]. PDRD can be subscribed to as a diskette version or CD-ROM.
Info
rm H
ealth
Soc
Car
e D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y Q
UT
Que
ensl
and
Uni
vers
ity o
f T
ech
on 1
1/21
/14
For
pers
onal
use
onl
y.
106 J , Jousimaa and I . Kunnamo
The diskette version contains the desk reference, parts of the database and a public domain workstation program PAPRUT. The CD-ROM contains in addition to the diskette version the whole database, picture material and other medical public domain and demonstration software. Both versions are updated two or three times yearly. GRATEFUL M E D program must be ordered separately, if the user wants to run automated literature searches.
The software is written with Turbo Pascal and runs on any IBM-compatible microcomputer (286 processor or faster recommended) with a hard disk. DOS 3.3 or newer and 640 kB of base memory are required. PDRD Desk Reference with PAPRUT workstation program requires 8 MB free space on a hard disk.
The technical user interface is based on keyboard commands. Desired functions of the program can be started either by pressing certain letter keys or shadowing the function symbols on the screen with arrow keys. A menu- and icon-based version will be published later. The influence of the program’s technical interface on program learning speed and acceptance would be a subject of further studies.
PDRD has been in experimental use since May 1989. In May 1993 the system had approximately 400 regular users, most of them general practitioners, but other health professionals such as hospital specialists, medical teachers and students, nurses, receptionists and medical factory representatives have been identified among users. The Desk Reference was published as an ordinary textbook in June 1992.
3.2. The contents, quality assurance and editorial staff of P D R D The textbase of PDRD is almost entirely in Finnish, with the exception of a very
limited English demonstration version. The editorial staff has started cooperation with Scandinavian countries in order to employ a more extensive expert group and in time to produce complete versions in other languages.
All the Desk Reference articles have been written especially for this program, although source articles from medical textbooks have naturally been used. This does not only mean ‘cosmetic’ modifications to adapt the computer screen properties, but the diagnostic strategies have been rewritten from the usual hospital-oriented views to primary health care use, keeping cost-effectiveness in mind. Because of the high quality of equipment and investigation facilities in Finnish rural health centres (including endoscopy, ultrasound, minor operations, cardiac monitoring), the texts are suitable to be used in outpatient departments also.
At present the PDRD Desk Reference part contains approximately 1000 articles. The size of the Desk Reference is not planned to increase much from this, but the texts will be modified and updated continuously by an editorial staff, whose work resembles editing a medical journal. The editorial staff is composed mainly of general practitioners, but other experts are employed occasionally to complete or evaluate special topics.
Approximately half of the instructions are written by experienced general practitioners and the other half by hospital or other specialists. T o ensure the quality of the data all the instructions will be repeatedly peer-reviewed by at least two independent experts, and at least one G P will refer the suitability of the instructions to primary care facilities. Continuous feedback is also collected from the end-user level.
The knowledge base of the PDRD database is still very limited. The main attention is focused on reviewing existing literature sufficiently to draw conclusions
Info
rm H
ealth
Soc
Car
e D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y Q
UT
Que
ensl
and
Uni
vers
ity o
f T
ech
on 1
1/21
/14
For
pers
onal
use
onl
y.
PDRD-computer-based decision support system 107
on treatment strategies published in desk reference. Another important topic is reviewing new drug preparations on the market. In the near future the PDRD software will be offered free to all Finnish specialist organizations to enable them to produce their own databases and quality standards that are relevant for use by specialists. The PDRD editorial staff will then use parts of this data in the general medicine database. Other future areas will be patient instructions (proper drug use and self-care), later perhaps an electronic doctor book for family use. The only limit of the database size will be the data-storing capacity of CD-ROM disks, at least 600MB.
3 . 3 . Running PDRD The user may type up to eight different search terms on the screen. There are
AND, OR and N O T options combining search terms, as well as the possibility of giving only the beginning of the search term (e.g. MYO: for myocardial infarction, myocarditis, etc.). A complete list of existing search terms can be brought on the screen to choose the appropriate ones. The user may also search with key words putting the *-mark before search term, and so the search is done only in articles, where the term is a main topic. If the user prefers, searching may also be performed chapter by chapter or searching any desired character-string. If the database article has not been installed on hard disk, the program may automatically search it from a floppy disk or CD-ROM without additional typing. Reading instructions from hard disk will bring appropriate textbook chapters within 1-2 s on the screen; using floppy disk or CD-ROM causes a short time delay in the searching process.
PDRD has a text editor, which enables the user to write his or her own notes in the program. The indexing program just automatically reads through the ASCII or Wordperfect (version 4.2) text and indexes it in the same way as the original PDRD texts have been indexed. The user may also choose to use indexing words other than the ones in the vocabulary, and index the text manually.
MeSH is used as the basic indexing vocabulary. The MeSH vocabulary has been partially modified to meet the needs of Finnish general practitioners; some biochemical terms have been omitted and some non-MeSH terms included, which can be separated from original MeSH terms and linked to other international vocabularies if necessary. The vocabulary has been yearly completed by analysing all the words that appeared in the desk reference or database, but did not appear in the vocabulary, and adding all relevant ones.
The vocabulary of PDRD works as a synonym thesaurus, where each MeSH term has numerous synonyms and related words, mainly Finnish and Latin words used by the doctors in everyday language, also medical jargon and abbreviations. For example, the result of the search would be the same, if the user used the search term ‘blood pressure’, its Finnish counterpart ‘verenpaine’ or simply ‘RR’. This is an important difference from conventional indexing software using only character string indexing.
PDRD works together with the PAPRUT workstation program, which is a public-domain program to be used either by single users or within local area networks. The PAPRUT workstation program contains some tools for focusing attention, such as warnings on drug allergies, etc. A closer cooperation of the two programs, e.g. matching the PDRD searches with individual data in PAPRUT patient register and starting prescriptions directly from PDRD texts, will be essentials of future programming.
Info
rm H
ealth
Soc
Car
e D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y Q
UT
Que
ensl
and
Uni
vers
ity o
f T
ech
on 1
1/21
/14
For
pers
onal
use
onl
y.
108 J . Jousimaa and I . Kunnamo
Table 1 . P D R D levels and data producers
P D R D level Data producer(s)
P D R D desk reference P D R D editorial staff GPs Other specialists
P D R D database Specialists and sub-specialists Other medical information producers
Automated literature search strategies
Local instructions Local health authorities
User’s notes P D R D users
Health librarians
Patient instructions Editorial staff (other than PDIID desk reference editorial)
3.4. The structure of PDRD articles The PDRD has three levels (table 1):
(1) the desk reference contains short, summarized recommendations regarding diagnosis, therapy or disease follow-up;
(2) the database contains review articles as background knowledge on the same topics;
(3) automated searches from MEDLINE and some other databases can be activated by the PDRD software using the GRATEFUL M E D program by the National Library of Medicine, USA.
In addition to these, PDRD may contain other medical information such as user’s notes or patient instructions.
There are three levels of instructions in the desk reference:
(1) Problem-orientated instructions, whose purpose is to lead the PDRD user from the patient’s symptom or problem to the correct diagnosis. In many cases only the diagnostic strategy leading to several possibilities is presented. In the beginning of the text, aims or principles are presented. Their purpose is to:
( a ) help the user t ecognize conditions that require immediate attention to avoid complications (e.g. meningitis as a cause of sudden fever and neck pain);
(6 ) remind the user of conditions which require therapy different from normal empirical symptomatic treatment (e.g. the possibility of coeliac disease as a cause of a prolonged diarrhoea);
(c) guide to deliberated, cost-effective diagnostic strategies avoiding unnecessary laboratory or X-ray investigations or hospital referrals.
After this critical part of the text, epidemiological knowledge of the condition is shortly presented, followed by diagnostic strategies or algorithms and advice on timely referral.
(2) Treatment protocols are used where the correct diagnosis or condition is supposed to be known by the time of consultation. These instructions contain:
Info
rm H
ealth
Soc
Car
e D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y Q
UT
Que
ensl
and
Uni
vers
ity o
f T
ech
on 1
1/21
/14
For
pers
onal
use
onl
y.
PDRD-computer-based decision support system 109
( a ) the most recommendable or best-documented treatment protocols (documentation will be found in the database);
( 6 ) alternative possibilities only if the best treatment is not clearly known or the choice of treatment is strongly dependent on individual factors;
( c ) side-effects and follow-up.
(3 ) Follow-up protocols of certain chronic diseases, which may be primarily treated in hospital (e.g. cancers). They contain:
( a ) appointment schedules; ( b ) routine laboratory tests and X-ray investigations; ( c ) advice to the patient when to contact the doctor before scheduled
appointment; ( d ) factors that would justify consulting a specialist, e.g. need to change
medication under supervision or possibility of relapse.
Evidently there is a certain overlap between the instruction areas, and some cannot be included in any of these groups, but the editors of PDRD continuously emphasize the importance of well-structured, disciplined instructions and work on creating optimal quality standards adapted to the primary care setting.
As the treatment protocols may show great local variations the PDRD editors also consider organizing consensus meetings if serious disagreement between specialists might occur.
Some of the desk reference chapters contain a MEDLINE search strategy planned by professionals in medical informatics. The PDRD program is linked with the search engine function on the GRATEFUL M E D program, which communicates with the Medical Information Centre (MIC) computer at the Karolinska Institute in Stockholm. All the user has to do is to press the (M) key on the keyboard while reading the article, and the search engine will automatically contact the MIC computer, give passwords and perform a literature review of the topic of the article. The user may change the planned search strategy on-line, but this requires some knowledge on the correct searching terms. At present the MEDLINE strategy is inclined in only approx. 5% of the texts, but new strategies will be included in every updating of the PDRD.
Being a model of normal text in electronic form read from the computer screen, PDRD would seem to belong to the group of ‘passive’ C D S systems. The possibilities of finding data from PDRD textbase are, however, more diversified than from studying ordinary textbooks, because PDRD is able to give new associations, that even the authors might not have thought of. For example, search terms ‘FACIAL PARALYSIS’ and ‘DERMATITIS’ yield the article ‘LYME DISEASE’ on the screen, as both terms appear together only in this article.
Another feature favouring the promoting of P D R D computer version is the hypertext function, which enables the user to move from one article to an unlimited number of other articles, and then finally turn back to the site of origin.
4. A project for evaluating the value of PDRD in routine use Since the publication of the first PDRD version in 1989, the program’s technical
user interface has been changed, new MEDLINE searching strategies have been included, the vocabulary has been completed, and the knowledge base has been continuously revised and enlarged. All these changes have been based on the
Info
rm H
ealth
Soc
Car
e D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y Q
UT
Que
ensl
and
Uni
vers
ity o
f T
ech
on 1
1/21
/14
For
pers
onal
use
onl
y.
110 J . Jousimaa and I. Kunnamo
development ideas of the programmers and editorial staff, and the criticism of the text referees. End-user feedback has been relatively infrequent. Among the most common- complaints have been the lack of mouse-support and icons (Windows and MacIntosh users), defective knowledge base and difficulty of finding appropriate search terms. Generally the users have not spontaneously commented on any subjective influence of the program on their working habits, outcomes of its use or the quality of the instructions.
The time for evaluating PDRD is optimal, because:
(1) the originally planned volume of the desk reference has been achieved and possible shortcomings can thus be identified;
(2) there are enough end-users and the program is currently spreading from computer enthusiasts to regular users;
(3) the PDRD desk reference has been published as a book, which enables comparison of the availability and usefulness of the same data in traditional and electronic form.
Data collection by telephone interviews has been started as the first part of the assessment study. All new subscribers in 1992 were interviewed immediately after subscription and will be re-interviewed 1-3 months after sending the program and 12 months thereafter.
This study should answer at least the following questions:
(1) Who will start using PDRD? (2) What kind of computer familiarity (if any) has existed before starting? (3) What is the lag between getting the software and getting to use it? (4) In what situations will the PDRD book prove more useful than the
(5 ) When and where will the PDRD be used: during consultations, between
(6) What are the subjective short- and long-term benefits and problems of using
computer?
consultations, in the library or at home?
the program?
Another project still in the planning stage will be started to study regular PDRD users. The program may include a function that automatically records any use of the PDRD during patient consultations. When the diagnoses of consultations are recorded with the workstation program, data on the use frequency and reasons can be collected. Physician compliance with the instructions will be further studied, checking individual patient records. Automatic data collecting enables long-term follow-up and will yield a relatively large amount of data while it diminishes the biasing awareness of observation of the study subject [24]. These data may be validated by using observers or video-recorded consultations.
Patient health gain as an outcome of PDRD use is problematic because long-term follow-up is required and other information sources cannot be excluded. A study comparing PDRD computer version users, PDRD book users and non-users will be performed, when measurable outcome objects are identified. Objects considered include, for example, laboratory and X-ray examinations, medication, treatment and follow-up protocols of certain common diseases such as sinusitis, hypertension and diabetes, as well as referral frequency and quality.
Info
rm H
ealth
Soc
Car
e D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y Q
UT
Que
ensl
and
Uni
vers
ity o
f T
ech
on 1
1/21
/14
For
pers
onal
use
onl
y.
PDRD-computer-based decision support system 111
5. Summary CDS systems mimicking ‘artificial intelligence’ that are comprehensive enough
to be suitable for use in primary care settings suffer from the fact that most of the patients’ conditions are subject to interpretations, and an enormously large database has to be collected to cover most of these conditions. PDRD is an electronic textbook that avoids some of these problems, because it has been built from conventional texts planned especially for primary care conditions. Computers give the texts new properties that are beyond the limits of traditional textbooks in speed and association of search terms and articles. The final value of this database remains to be seen in an evaluation study, which is now in progress.
References 1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
SHORTIAFFE, E. H. (1987) Computer programs to support clinical decision making. Journal of the American Medical Association, 258, 61-66. BARNETT, G. O., CIMINO, J. J., HUPP, J. A., and HOFFER, E. P. (1987) DXplain-an evolving diagnostic decision-support system. Journal of the American Medical Association, 258, 67-74. POTTHOFF, R., ROTHEML‘ND, M., SCHWEFEL, D., ENGEI-BRECHT, R., and VAN EIMEREN, W. (1988) Expert systems in medicine. International Journal of Technology Assessment in Health Care,
LEDIXY, R. S., and LUSTED, L. B. (1959) Reasoning foundations of medical diagnosis. Science, 130,9-21. TATRO, D. S., BRIGGS, R. L., CHAVEZ-PARDO, R., FEINBERG, L. S., HANNIGAN, J. F., MOORE, T. N., and COHEN’, S. N. (1975) Online drug interaction surveillance. AmericanJournal of Hospital Pharmacology, 32,417-420. PI.CC;~;E, L. A,, VER~IEY, F. R. J., and JOILES, J . (1990) A desktop expert system for the differential diagnosis of dementia. International Journal of Technology Assessment in Health Care, 6, 147-1 56. NYKASES, P., and NCUTILA, P. (1991) Validation and evaluation of a system for thyroid disorders. Expert Systems With Applications, 3, 289-295. FORSSTR~M, J. EKLUSD, P., VISTASEN, H., WAXLAX, J., and LAHDEVINTA, J. (1991) DIAGAID: a connectionist approach to determine the informational value of clinical data. Artificial Intelligence in Medicine, 3, 193-200. WAXMAS, H. S., and WORLEY, W. E. (1990) Computer-assisted adult medical diagnosis: subject review and evaluation of a new microcomputer-based system. Medicine, 69, 125-1 36. BASKOWITZ, R. A., McNnii., M. A., CHAI.I.ISOR, S. M., PARKER, R. C., KAPOOR, W. N., and MII.I.ER, R. A. (1989) A computer-assisted medical diagnostic consultation service. Implementation and prospective evaluation of a prototype. Annals of Internal Medicine, 110, 824-832. GlSzl.ER, M., and PRITCHARD, P. (1990) The Oxford System of Medicine in the context of trans-national markets. Can medical knowledge-based systems cross frontiers? (Biomedical Computing Unit, Imperial Cancer Research Fund: London). BI~RGEROS, B. (1 991) Iliad: a diagnostic consultant and patient simulator. M . D . Computing, 8, 46-53. COVEI.I., D. G., UMAS, C. G., and MASSING, P. R. (1985) Information needs in office practice: are they being met? Annals of Internal Medicine, 103, 596-599. EDITORIAI. (1 987) Computer-assisted decision support: a new available reality. Journal of the American Medical Association, 258, 86. B4RSETT, G. O., CI.\IISO, J . J., HCPP, J. A., and HOIW.X, E. P. (1987) DXplain: experience with knowledge acquisition and program evaluation. Proceedings of the Elewenth Annual Symposium on Computer Applications in Medical Care, pp. 150-1 54. SIS(;I:R, J.. SACKS, H. S., LLWST, F., and CHAI,MERS, T. C. (1983) Physician attitudes toward applications of computer data base systems. Journal of the American Medical Association, 249, 1610-1614. KAPIAS, B. (1987) The medical computing ‘lag’: Perceptions of barriers to the application of computers to medicine. International Journal of Technology Assessment in Health Care, 3, 123-1 36. Lo>i+s, J., ASDI:.RSOS, G . M., DOMSICK-PIIJRRE, K., VAYDA. E., ESKIS, M. W.. and HASSAH, W. J. (1989) Do practice guidelines guide practice? New England Journal of Medicine, 321, 1306-1311. NOEHR, C. (1991) Outcomes assessment, quality assurance and evaluation of expert diagnostic systems. Proceedings of Medical Informatics Europe. TCRSISR, C . W., Wii,i,iA\isos, M., LISCOI.~, M. J., HACG, P. J., BCCHASAS, J., ASDERSOS, C., G w n , M., CCSDICK, R., and WARSIX, H. R. (1990) The effects of Iliad on medical student problem solving. Proceedings of the 14th Symposium on Computer Applications in Medical Care (Washington, DC: IEE Society Press), pp. 478-481.
4, 121-133.
Info
rm H
ealth
Soc
Car
e D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y Q
UT
Que
ensl
and
Uni
vers
ity o
f T
ech
on 1
1/21
/14
For
pers
onal
use
onl
y.
112 J . Jousimaa and I . Kunnamo
21. WEAVER, R. R. (1991) Assessment and diffusion of computerized decision support systems. International Journal of Technology Assessment in Health Care, 7 , 42-50.
2 2 . NOEHR, C. (1991) Computer-assisted decision-support-better health? Aalborg Universitetscenter Institut for Samfundudvikling og Planlaegning.
23. KCNNAMO, I., JOLSIMAA, J., and SALMI, L. (1991) The physician’s desk reference and database with automated literature search facilities in a primary care PC workstation. Newsletter to European Health Librarians, 14, 11-1 2 .
24. RETHANS, J., STCRMANS, F., DROP, R., VAS DP:R VI,P:LTP:N, C., and Hosus, P. (1991) Does competence of general practitioners predict their performance? Comparison between examination setting and actual practice. British Medical Journal, 303, 1377-1 380.
Info
rm H
ealth
Soc
Car
e D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y Q
UT
Que
ensl
and
Uni
vers
ity o
f T
ech
on 1
1/21
/14
For
pers
onal
use
onl
y.