evaluation of electronic medical records - a clinical task perspective presentation of thesis
DESCRIPTION
?. Error. Norwegian University of Science and Technology , Trondheim. Evaluation of electronic medical records - a clinical task perspective Presentation of thesis. Hallvard Lærum. The Kvalis Project. Quality assurance of electronic medical records in hospitals - PowerPoint PPT PresentationTRANSCRIPT
Evaluation of electronic medical records
- a clinical task perspective
Presentation of thesis
Norwegian University ofScience and Technology, Trondheim
Hallvard Lærum
Error ?
The Kvalis Project
• Quality assurance of electronic medical records in hospitals
• Funded by The Research Council of Norway
• Interdisciplinary, three PhD fellows– Sociology: Gro Underland– Informatics: Gunnar Ellingsen– Medicine: Hallvard Lærum
Evaluation of electronic medical records
Descriptive aspectsWhat is the level of functionality of current EMR systems?Where are they found?
Explorative aspectsHow may the effects of various EMR systems on physicians’ clinical practice in various hospitals be evaluated?
InvestigationsHow does the method perform in various contexts?Reliability and validity studies
The 3rd Scandinavian University Course in medical informatics
Building a researchnetwork
Building a researchnetwork
Keeping up a websiteKeeping up a website
Arranging monthly seminarsArranging monthly seminars
Arranging meetingsArranging meetings
Establishing contact withEMR vendors Establishing contact withEMR vendors
Department of Medical Informatics & Clinical Epidemiology, Oregon Health and Science University
Establishing contact withresearch centersinternationally
Establishing contact withresearch centersinternationally
Establishing contact with hospitalsEstablishing contact with hospitals
Descriptive aspects of the thesis
EMR systems are very common in Norwegianhospitals (80% of hospital beds as of 2002)
The systems are essentially document-based, covering the narrative parts of the medical records, lab data and radiology reports.
NoNo systematic evaluations systematic evaluations of Norwegian EMRs of Norwegian EMRs published by 1999published by 1999
Explorative aspects of the thesis 1
Considerations for development of an evaluation method
• Applicable to various EMR systems and hospitals– Not tied to a single system or hospital
• Carried out using limited resources– Fast results
• Used in operational reviews, incorporating organizational issues
– Laboratory simulations less feasible
• Involve measurement close to the site of impact– Involving how clinical work is done
The first task inventory
• Direct observations, 40 h– 20 h St.Olavs Hospital, 20h RiTø (Gunnar E.)– 7h transcribed verbatim– Basis for hierarchical task analysis– Noticed that the PCs were not much in use
• The first task inventory: 104 tasks– Too large for practical use– Task wording rather technical– Covert tasks not included (e.g. cognitive tasks)
D e finep rob lem
G e t m e d ica lkn o w le d ge
G e t p a tie n td a ta
C re a te n ewin fo rm a tion
In fo rmP a tie n t
N e g o tia tew ith p a tie n t
E xecu tein te rve n tion
D o cum e nta c tiv ity
D ia gn o s tic s an d Tre a tm e nt
G e t e x is t ingp lan
G e t in foa b ou t reso urces
N e g otia tew ith p a tie n t
In fo rmP a tie n t
D o cum e ntp lan
P la nn ing
G e ne ra llo g is tics
C o m m u n ica tion
A d m in is tra tive
R e se a rch
Q u a lityC o n tro l
T e a ch ing
P a tien t's re la tionto soc ie ty
O th er
O th er
Explorative aspects 2
Video observations – Flow of information
OntoLog Courtesy of Jon Heggland, IDI NTNU
• Information seeking behaviour– Looking for incidents where needed
information was not found
• Tagging of video recordings
• Stimulated recall* not sucessful– Too time-consuming (and boring!)
for an average clinician– Recruitment problems
• Prohibitively resource intensive– 9 patients, 4.5 h video recordings,
two physicians
• Noticed that PCs were not much in use
* Kushniruk & Patel 1995
Simplification of the task list
• Task list to be used in questionnaires and interview guides
• Considerations for the process– Emphasis on information-related tasks essential for patient
care– Wording adapted to clinicians’ way of speaking– Covert tasks were included– Tasks suitable for most specialties were retained– Tasks representing negligible work were deleted– Gorman’s five information needs were taken into account– Tasks should be supported by functionality found in current
EMR systems or in those expected in the near future
• Multiple iterations by work group (2 physicians, 2 informaticians)
!
A list of relevant tasks for the clinician
Norwegian English translation
1. Få oversikt over pasientens problemstilling Review the patient's problems
2. Lete frem enkeltopplysninger fra pasientjournalen Seek out specific information from patient records
3. Følge resultatene av en bestemt prøve eller us. over tid Follow the results of a test or investigation over time
4. Slå opp svar på nye prøver eller undersøkelser Obtain the results from new tests or investigations
5. Føre daglige og/eller forefallende journalnotater Enter daily notes
6. Få tak i oppl. om prosedyre for utredning eller behandling Obtain information on investigation or treatment procedures
7. Få svar på spørsmål om generell med.-faglig kunnskap Answer questions concerning general medical knowledge
8. Få ut samledata for en gruppe pasienter Produce data reviews for specific patient groups
9. Rekvirere klin-kjem. lab.analyser Order clinical biochemical laboratory analyses
10. Slå opp svar på klin.-kjem. lab.analyser Obtain the results from clinical biochemical laboratory analyses
11. Rekvirere rtg.us., UL eller CT Order X-ray, ultrasound or CT investigations
12. Slå opp svar på rtg., UL eller CT Obtain the results from X-ray, ultrasound, or CT investigations
13. Rekvirere. andre supplerende undersøkelser Order other supplementary investigations
14. Slå opp svar på andre supplerende undersøkelser Obtain the results from other supplemental investigations
15. Henvise pas. til annen avdeling eller spesialist Refer the patient to other departments or specialists
16. Ordinere behandling direkte (med., op. eller annen) Order treatment directly (e.g. medicines, operations etc.)
17. Skrive resept Write prescriptions
18. Skrive sykmelding Complete sick-leave forms
19. Samle inn pasientopplysninger til ulike legeerklæringer Collect patient data for various medical declarations
20. Gi skriftlig individuell info til pasienten Give written individual information to patients
21. Gi skriftlig generell medisinsk-faglig info til pasienten Give written general information to patients about the illness
22. Samle inn opplysninger til epikrise Collect patient information for discharge reports
23. Kontrollere og signere ferdig skrevne diktater Check and sign typed dictations
The first Questionnaire was compiled
• Translated sections (two-way translation validation involving native english speaking translator)– End User Computing Satisfaction scale (Doll & Torkzadeh)– Global user satisfaction questions (SGUS, Anderson & Aydin)– Computer literacy (Brown & Coney)
• New sections– Frequency of PC /EMR use for the 23 clinical tasks– Basic availability of computers
Pilot study (n=22)
Investigations in this thesis
• National survey, 2001
• Local survey in Arendal, 2002
• Validation and reliability studies, 2003
The national survey Feb 2001
• Survey– 314 physicians in 32 hospital units in 19
hospitals– 227 (72.3%) responded after one reminder
(135 respondents were reminded)
• Telephone interviews with key IT personnel in each hospital
– What clinical tasks could be performed using the EMR system?
– A set of minimal functionality requirements, incl. requirements for integration of external systems
The section covering PC/EMR useMissing
responses!
Implemented functions
DIPS Infomedix
1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge8. Produce data reviews for specific patient groups9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv.15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients22. Collect patient information for discharge reports23. Check and sign typed dictations
DocuLive
0 % 50 % 100 % 50 % 100 %50 % 100 %
General
Medical knowledge
Aggregated dataSupplementary investigations
Actions
Information to patient
Communication and verification
Percent of respondents offered minimal functionality for the given task
• There is a considerable difference in implemented functions between systems
Use and implementation of EMR systems
DIPS Infomedix
1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations
DocuLive
0 % 50 % 100 % 50 % 100 %50 % 100 %
General
Medical knowledge
Aggregated dataSupplementary investigations
Actions
Patient info
Communication and flow of information
Percent of respondents offered functionality/ using it
Use both EMR and other software
Use EMR Type of program not stated
Implementation
A lot of implemented functionality appears not to be used
Cut-off: Respondents answering ”half of the
time” or better are users
Use and implementation of EMR systems
DIPS Infomedix
1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes (i.e. progress notes)6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations
DocuLive
0 % 50 % 100 % 50 % 100 %50 % 100 %
General
Medical knowledge
Aggregated dataSupplementary investigations
Actions
Patient info
Communication and flow of information
Percent of respondents offered functionality/ using it
Use both EMR and other software
Use EMR Type of program not stated
Implementation
Some physicians enter daily notes themselves
Use and implementation of EMR systems
DIPS Infomedix
1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations
DocuLive
0 % 50 % 100 % 50 % 100 %50 % 100 %
General
Medical knowledge
Aggregated dataSupplementary investigations
Actions
Patient info
Communication and flow of information
Percent of respondents offered functionality/ using it
Use both EMR and other software
Use EMR Type of program not stated
Implementation
Obtaining lab data (and other results) is popular, order entry is not
Use and implementation of EMR systems
DIPS Infomedix
1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations
DocuLive
0 % 50 % 100 % 50 % 100 %50 % 100 %
General
Medical knowledge
Aggregated dataSupplementary investigations
Actions
Patient info
Communication and flow of information
Percent of respondents offered functionality/ using it
Use both EMR and other software
Use EMR Type of program not stated
Implementation
Big surprise: Nobody is using the EMR system to write prescriptions or complete sick-leave forms
Why this lack of EMR use? – 1Answers provided by the questionnaire is limited
I wish the EMR couldbe more like WordStar
Where’s the PCwhen I need it?Lack of available computers?
Con: Some PCs were available: 93% a computer in their office, 97% had a computer available in other rooms used for clinical work.
Pro: Clinicians’ work is not stationary. No hospitals have yet implemented mobile computing, and no hospitals can afford a PC in every room (personal communications)
Low computer literacy?Con: The physicians had at least a basic knowledge of computers (average score 72.2 ±1.6, 92% owned a computer)
Pro: Specific EMR system training may be needed
Why the lack of EMR use? - 2
I know my pre-scriptions by heart!
I’d like to find allthe information
in one place
Follow-up studies in each hospital are necessary
Low usability of the EMR system as a whole?Prescriptions: Navigating to the prescription module, finding the correct medication, typing dosage, administration route and package size and printing on the correct type of paper may represent more work than doing it by hand.Sick-leave forms: Slow system response times (12-15 sec)
The EMR is not completePro: Until all relevant clinical information is found in the EMR, it will serve a secondary role.Pro: The functionality of current EMR systems is limited
Resistance to new work roles?Pro: The usage patterns found in the national study are conform to traditional work roles.Pro: Work role issues were the most prevalent theme in the answers to the open-ended questions in the validity study (Paper 4).
Typing is a secretary’s job!
Order entry is anurse’s job!
Local study of EMR system in Aust-Agder Hospital, Arendal(Paper 2 and 3)
• The paper-based medical record is scanned and obliterated!
• Questionnaire rev.2 (versions for medical secretaries and nurses developed separately)
– 70 of 80 physicians (88%)– 79 of 85 medical secretaries (93%)– 172 of 235 nurses (73%)
• Interviews– 8-12 representatives of each profession,
0.5-2 h interviews
The EMR in Aust-Agder Hospital
2nd revision of the questionnaire
Physician’s use of the regular EMR at Aust-Agder Hospital
Much higher frequency of EMR use than in other hospitals having the same system
The physicians reported a less frequent use of the scanned document images
Physicians: Change in ease of performing the clinical tasks
Most tasks related to information retrieval are reportedlyeasier using the system, but 33% of internists found task 1 and 2 more difficult
User satisfaction of physicians
The physicians are relatively satisfied with the regular EMR, but not with the use of the scanned document images
Validation and reliability studies3rd revision of questionnaire
• Test-retest reliability study– 37 of 96 physicians (39%) from three hospitals having different EMR
systems completed the questionnaire two times. ”Test” questionnaire: 55.2% (52 of 96), ”Retest” questionnaire. 71% (37/52).
• Content validity study – interviews– 10 physicians, 1 h interviews– Relevance of tasks, estimation of accuracy of answers to task-oriented
questions, themes in open-ended questions.
• Criterion validity using data from local and national studies– Criteria
• General information retrieval vs. tasks involving information retrieval• Overall work performance vs.task performance • Task performance vs. user satisfaction
– Majority of tasks covered, median correlation coefficient 0.445-0.513
Test-retest reliability: Weighted kappa was generally high
Problematic tasks were related to functionality not available locally, and problems in discering the EMR from other software
Themes appearing in the interviews
• Work role issues (8/10 physicians) – resistance to doing ”clerikal” tasks– Tasks 10 and 19 – The third method would be the "ask-the-nurse" method. This is convenient, though,
then I may do other things. [In the future] It could be that it will be so easy to do it, that I could do it myself…if it's really easy, a completely negligible task. But if it takes some time..if I have to wait or something..then I feel that it should be a medical secretary's task, at least in a hospital. (respiratory diseases)
• Various wording problems (7/10 physicians)– Tasks 16, 4 and 21– I don't understand what you mean with "directly"…write orders on the [order entry
form], request or order an operation…one other [example] is requesting treatment by physiotherapist (orthopedy)
• Questions regarding use of non-existent functionality (7/10 physicians)– Task 3– Some questions are difficult to answer, as we can't log on [to the EMR system] and
find results from X-ray investigations (plastic surgery)
• Distinguishing EMR from other software or media (6/10 physicians)– Task 4– Is [the separate lab system] regarded as a part of [the EMR system]? (neurology)
Considerations of self-reported use
• Sources of error in self-reported use– Telescoping and other memory-related effects– Implicit expectations of the questionnaire– Willingness to respond– Strategic responses– Other
– There seem to be more reasons for the user to report too high frequency of use than too low.
• Use of the EMR system may not necessarily benefit the patient– EMR system used the wrong way– Unintended effects of using the EMR system– Logical errors in the EMR system
Application of the task list and the questionnaire
• The questionnaire may be used for screening, focusing the evaluation effort and providing a basis for further exploration– Separating reported use from non-use
• The combination of EMR use and task performance may identifiy problematic aspects of the EMR system.
• The questionnaire should always be combined with a qualitative study to investigate the ”why” (and validate the findings)
Thank you!
Extra information
National study: Inclusion of hospitals
• Inclusion of hospitals– The EMR should at least contain medical narratives (admission reports, progress
notes, discharge reports), directly or indirectly updated by and electronically available to the physicians. (minimal functional requirement for task 1)
National study: Lessons learned in this survey:
• Missing responses– ”Use EMR/Other program” questions frequently not
answered (median 10%, IQR 5-15%)
• Two hospital units (8 physicians) had to be excluded post hoc– 1: Minimal requirements for EMR implementation not met– 2: Two EMR systems in use simultaneously
• Error discovered one year after publication– Task not supported after all in one hospital unit (11
physicians)– Lab results not integrated in the EMR in this unit, available
in stand-alone system only. – Correction sent to BMJ (paper 1) Nov 2002.
Local study: User satisfaction of physicians, nurses and medical secretaries
The medical secretaries are most satisfied, the physicians least satisfied with the system
Local study: Use of hospital information system for individual tasks
Local study: Results of interviews
• All professions found that the patient data were more accessible when stored electronically.
• Physicians (internists) found searching in scanned multiple documents time-consuming and difficult
• The medical secretaries found that generation, handling, fetching and delivery of paper documents and logistics of paper-based patient records had diminished dramatically.
• Nurses were still using pen and paper documenting their activities.
Changes in rev.3 of the questionnaire
• Added ”escape” choices in task-related questions– Task not relevant for me– Task not possible using the local EMR system
Validation: Interviews 1: Accuracy
Discordant interpretationfor 1/10 in task 6, ”Obtain information on investigation or treatment procedures”
validation: Interviews 2: Relevance of tasks
Task 8, ”Produce data reviews for specific patient groups” was not found relevant by a majority of the physicians.(not part of one’s job/ infrequently performed)