Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 1 of 66
Electronic Health Record Usability Testing Report
MDVita version 25
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 2 of 66
EHR Usability Test Report of MDVita version 25
Report based on ISO/IEC 25062:2006(E) Common Industry Format for Usability Test Reports
MDVita EHR version 25
Date of Usability Test: 12/13/2017 – 12/21/2017
Date of Report: 12/26/2017. Last Modified: 01/02/2018
Report Prepared By: HealthCare 2000, Inc
Ileana Pichel, System Analyst
(305) 262-5959 ext. 0
7205 Corporate Center Drive, Suite 401, Miami, FL 33126
Table of Contents 1. EXECUTIVE SUMMARY ........................................................................................................................ 4
2. INTRODUCTION .................................................................................................................................... 8
3. METHOD ................................................................................................................................................ 8
3.1. PARTICIPANTS .............................................................................................................................. 8
3.2. STUDY DESIGN ............................................................................................................................. 9
3.3. TASKS .......................................................................................................................................... 10
3.4. PROCEDURES............................................................................................................................. 11
3.5. TEST ENVIRONMENT ................................................................................................................. 12
3.6. TEST FORMS AND TOOLS ......................................................................................................... 12
3.7. PARTICIPANT INSTRUCTIONS .................................................................................................. 12
3.8. USABILITY METRICS .................................................................................................................. 13
3.9. DATA SCORING........................................................................................................................... 13
4. RESULTS ............................................................................................................................................. 15
4.1. DATA ANALYSIS AND REPORTING .......................................................................................... 15
4.2. DISCUSSION OF THE FINDINGS ............................................................................................... 18
4.2.1. EFFECTIVENESS ................................................................................................................ 18
4.2.2. EFFICIENCY ......................................................................................................................... 18
4.2.3. SATISFACTION .................................................................................................................... 18
4.2.4. MAJOR FINDINGS ............................................................................................................... 18
4.2.5. AREAS FOR IMPROVEMENT ............................................................................................. 19
5. APPENDICES ....................................................................................................................................... 19
5.1. Appendix 1: SAMPLE RECRUITING SCREENER....................................................................... 21
5.2. Appendix 2: PARTICIPANT DEMOGRAPHICS ........................................................................... 23
5.3. Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ............. 24
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 3 of 66
5.4. Appendix 4: MODERATOR’S GUIDE .......................................................................................... 26
5.4.1. EHRUT Usability Test ............................................................................................................. 26
5.5. Appendix 5: Tasks Sheet .............................................................................................................. 28
5.6. Appendix 6: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................... 65
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 4 of 66
EXECUTIVE SUMMARY
A usability test of MDVita v 25 was conducted between December 13th, 2017 and December 21st, 2017
by Health Care 2000, Inc in one-to-one sessions with the test participants. The purpose of this test was to
test and validate the usability of the current user interface, and provide evidence of usability in the EHR
Under Test (EHRUT). During the usability test, 10 healthcare providers matching the target demographic
criteria served as participants and used the EHRUT in simulated, but representative tasks. The participants
are representative of our user base.
This study collected performance data on ten tasks typically conducted on an EHR:
170.315(a)(1) CPOE – Medications
170.315(a)(2) CPOE – Laboratory
170.315(a)(3) CPOE – Diagnostic Imaging
170.315(a)(4) Drug-drug, Drug-allergy Interaction Checks
170.315(a)(5) Demographics
170.315(a)(6) Problem List
170.315(a)(7) Medication List
170.315(a)(8) Medication Allergy List
170.315(a)(9) Clinical Decision Support
170.315(a)(14) Implantable Device List
170.315(b)(2) Clinical Information Reconciliation and Incorporation
During the three hours one-on-one usability test, each participant was greeted by the administrator and
asked to review and sign an informed consent/release form (included in Appendix 3); they were instructed
that they could withdraw at any time. Participants had prior experience with the EHR. The administrator
introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the
EHRUT. During the testing, the administrator timed the test and, along with the data logger(s) recorded
user performance data on paper and electronically. The administrator did not give the participant assistance
in how to complete the task.
The Customized Common Industry Format Template for Electronic Health Record Usability Testing (NISTIR
7742) (http://ws680.nist.gov/publication/get_pdf.cfm?pub_id=907312) is used for usability test reports
(ISO/IEC 25062:2006(E)).
The following types of data were collected for each participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks
Number and types of errors
Path deviations
Participant’s verbalizations
Participant’s satisfaction ratings of the system
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 5 of 66
All participant data was de-identified – no cross reference could be made from the identity of the participant
to the data collected. Following the conclusion of the testing, participants were asked to complete a posttest
questionnaire. Participants were not compensated for their time. Various recommended metrics, in
accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the
Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a
summary of the performance and rating data collected on the EHRUT.
Task
Nbr
Measure Task Success
(%)
(Success/Failure)
Path Deviation
Mean (SD)
Task Time
Deviations (Observed / Optimal)
Errors
Mean (SD)
Task Ratings 5=Easy
Mean (SD)
1 Create Medication Order 100/0 10(0) 74/90 0(0) 5(0)
2 Change Medication Order 100/0 10(0) 71/90 0(0) 5(0)
3 Access Medication Order 100/0 4(0) 11/15 0(0) 5(0)
4 Create Laboratory Order 100/0 11(0) 35/45 0(0) 5(0)
5 Change Laboratory Order 100/0 9(0) 34/45 0(0) 5(0)
6 Access Laboratory Order 100/0 5(0) 14/20 0(0) 5(0)
7 Create Imaging (Radiology) Order
100/0 13(0) 45/60 0(0) 5(0)
8 Change Imaging (Radiology) Order
100/0 9(0) 20/25 0(0) 5(0)
9 Access Imaging (Radiology) Order
100/0 7(0) 13/20 0(0) 5(0)
10 Adjust the severity level for drug-drug notifications
100/0 10(9) 19/30 0(0) 5(0)
11 Act upon drug-drug and drug-allergy interventions
100/0 12.8(0.4) 70/90 0.2(0.4) 5(0)
12 Record patient’s Demographics 100/0 11.8(0.4) 79/90 0.2(0.4) 5(0)
13 Change patient’s Demographics 100/0 8(0) 37/45 0(0) 5(0)
14 Access patient’s Demographics 100/0 4(0) 17/30 0(0) 5(0)
15 Record Problem in Problem List 100/0 7(0) 23/30 0(0) 5(0)
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 6 of 66
16 Change Problem List 100/0 7(0) 22/30 0(0) 5(0)
17 Access Problem List 100/0 5(0) 14/20 0(0) 5(0)
18 Record medication in Medication List
100/0 8(0) 20/30 0(0) 5(0)
19 Change Medication List 100/0 8(0) 16/20 0(0) 5(0)
20 Access Medication List 100/0 5(0) 11/15 0(0) 5(0)
21 Record Medication Allergy 100/0 10(0) 24/30 0(0) 5(0)
22 Change Medication Allergy 100/0 6(0) 15/20 0(0) 5(0)
23 Access Patient’s Allergy List 100/0 4(0) 10/15 0(0) 5(0)
24 Configure CDS Interventions 100/0 9(0) 28/30 0(0) 5(0)
25 Act upon Problem List interventions and Access evidence-based, diagnostic and therapeutic reference resources
100/0 12(0) 27/30 0(0) 5(0)
26 Act upon Medication List interventions and Access evidence-based, diagnostic and therapeutic reference resources
100/0 9(0) 24/30 0(0) 5(0)
27 Act upon Medication Allergy List interventions
100/0 11(0) 29/35 0(0) 5(0)
28 Record patient’s implantable device
100/0 8(0) 47/60 0(0) 5(0)
29 Change patient’s implantable device
100/0 4(0) 15/20 0(0) 5(0)
30 Access patient’s implantable device List
100/0 4(0) 13/20 0(0) 5(0)
31 Reconcile patient medication, problem and medication allergy lists with CCDA and generate resulting CCDA
100/0 27(0) 93/120 0(0) 5(0)
32 CDS. Act upon Demographics intervention. Access evidence-based, diagnostic and therapeutic reference resources and review bibliographic citation, developer, funding source, and release/revision date
100/0 9(0) 50/45 0(0) 5(0)
33 CDS. Act upon Laboratory test result intervention
100/0 8(0) 48/50 0(0) 5(0)
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 7 of 66
34 CDS. Act upon Vital signs intervention
100/0 8(0) 33/35 0(0) 5(0)
35 CDS. Act upon combination (Demographics and Medication) intervention
100/0 10(0) 29/30 0(0) 5(0)
36 CDS. Act upon CCDA reconciliation intervention
100/0 26(0) 127/130 0(0) 5(0)
The results from the System Usability Scale scored the subjective satisfaction with the system based on
performance with these tasks to be: 100.1
In addition to the performance data, the following qualitative observations were made:
- Major findings
All of the tasks were found easy to use by all users. This is due to improvement made to the UI in the
evaluated version 25 vs. the previous attested version 20, based on user feedbacks.
Some users had difficulty finding the tabs and fields to record and complete the whole patient demographics
task. Some users complained that too much information was needed and found the UI data elements too
far apart.
Some users verbally praised how easy was and intuitive to use the reconciliation tasks.
Some users had experience with other EHRs and praised MDVita for being so well-organized and how
easy and complete it is.
- Areas for improvement
Considering the information gathered from the users, the following areas of functionality have been
identified as in need of enhanced usability design:
Record and Change Patient’s Demographics
Demographics MU core components must be grouped close to each other in the UI for the users to identify
and enter faster.
1 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan
Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores
over 80 would be considered above average.
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 8 of 66
INTRODUCTION
The EHRUT tested for this study was MDVita version 25. Designed to present medical information to
healthcare providers in General and Primary Care practices, the EHRUT consists of health care practice
management, electronic health records, clinical notes, orders, patient chart with clinical information,
ePrescribe, and treatment plans. The usability testing attempted to represent realistic exercises and
conditions.
The purpose of this study was to test and validate the usability of the current user interface, and provide
evidence of usability in the EHR Under Test (EHRUT). To this end, measures of effectiveness, efficiency
and user satisfaction, such as time taken to complete task, and the user’s feedbacks, were captured during
the usability testing.
METHOD PARTICIPANTS
A total of 10 participants, that were representative of our user base, were tested on the EHRUT(s).
Participants in the test were one physician, one registered nurse, seven medical assistances and one office
supervisor. Participants were recruited by HealthCare 2000, Inc from its client base and were not
compensated for their time. In addition, participants had no direct connection to the development of or
organization producing the EHRUT(s). Participants were not from the testing or supplier organization.
Participants were given the opportunity to have the same orientation and level of training as the actual end
users would have received.
For the test purposes, end-user characteristics were identified and translated into a recruitment screener
used to solicit potential participants; an example of a screener is provided in Appendix 1.
Recruited participants had a mix of backgrounds and demographic characteristics conforming to the
recruitment screener. The following is a table of participants by characteristics, including demographics,
professional experience, computing experience and user needs for assistive technology. Participant names
were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 9 of 66
Part ID Gender Age Education Occupation/ role Professional
Experience Computer
savvy
Previous
EHR
Experience
Assistive Technology
Needs
1 User 1 F 40-59 High
School
Medical
Assistance
7 years Yes Yes No
2 User 2 F 40-59 College Medical
Assistance
5 years Yes Yes No
3 User 3 M 23-39 Post
Graduate
Medical
Assistance
3 years Yes Yes No
4 User 4 F 23-39 High
School
Medical
Assistant
7 years Yes Yes No
5 User 5 F 40-59 College Registered
Nurse
10 years Yes Yes No
6 User 6 M 40-59 Vocational
Training
Medical
Assistant
8 years Yes Yes No
7 User 7 F 23-39 Associate
degree
Medical
Assistant
7 years Yes Yes No
8 User 8 F 23-39 Vocational
Training
Office
Supervisor
3 years Yes Yes No
9 User 9 M 23-39 Doctorate
Degree,
MD
Medical
Assistant
1 year Yes Yes No
10 User
10
M 40-59 Doctorate
Degree,
MD
Physician 3 years Yes Yes No
All ten participants (matching the demographics in the section on Participants) were recruited and all ten
participated in the usability test. A document was sent to the participants to get their demographic
characteristics as shown in Appendix 1.
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application performed well – that is,
effectively, efficiently, and with satisfaction – and areas where the application failed to meet the needs of
the participants. The data from this test may serve as a baseline for future tests with an updated version
of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this
testing serves as both a means to record or benchmark current usability, but also to identify areas where
improvements must be made.
During the usability test, participants interacted with EHRUT. Each participant used the system via a Laptop
computer, and was provided with the same instructions. The system was evaluated for effectiveness,
efficiency and satisfaction as defined by measures collected and analyzed for each participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks
Number and types of errors
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 10 of 66
Path deviations
Participant’s verbalizations (comments)
Participant’s satisfaction ratings of the system
Additional information about the various measures can be found in Section 3.8 on Usability Metrics.
TASKS
A number of tasks were constructed that would be realistic and representative of the kinds of activities a
user might do with this EHR and were tested in the order of forced-ranked risk-based severity by our end
users, including:
1. 170.315 (a)(1) Computerized Provider Order Entry (CPOE) - Medications
Create Medication Order
Change Medication Order
Access Medication Order
2. 170.315 (a)(2) CPOE - Laboratory
Create Laboratory Order
Change Laboratory Order
Access Laboratory Order
3. 170.315 (a)(3) CPOE - Imaging
Create Imaging (Radiology) Order
Change Imaging (Radiology) Order
Access Imaging (Radiology) Order
4. 170.315 (a)(4) Drug-Drug, Drug Allergy Interaction Checks
Adjust the severity level for drug-drug notifications
Act upon drug-drug and drug-allergy interventions prior medication order completion
5. 170.315 (a)(5) Demographics
Create a Patient entering Date of Birth, Birth Sex, Preferred Language, Race, Ethnicity, Sexual
Orientation and Gender identity
Change demographics
Access Patient’s Demographics
6. 170.315 (a)(6) Problem List
Record a Patient problem using SNOMED
Change Problem Status
Access Problem List
7. 170.315 (a)(7) Medication List
Record medication in the Medication List
Change Medication List
Access Medication List
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 11 of 66
8. 170.315 (a)(8) Medication Allergy List
Record medication allergy
Change medication allergy
Access Patient’s Allergy List
9. 170.315 (a)(9) Clinical Decision Support (CDS)
Configure CDS interventions
Act upon Problem List interventions
Act upon Medication List interventions
Act upon Medication Allergy List interventions
Act upon Demographic interventions
Act upon Laboratory test result intervention
Act upon Vital Signs interventions
Act upon a combination of at least two elements above
Access evidence-based, diagnostic and therapeutic reference resources for Problem List,
Medication List, and Demographics interventions
10. 170.315 (a)(14) Implantable Device List
Record an implantable device
Change an implantable device
Access Patient’s implantable device List
11. 170.315 (b)(2) Clinical Information Reconciliation and Incorporation
Reconcile patient medication list with a CCDA
Reconcile patient problem list with a CCDA
Reconcile patient medication allergy list with a CCDA
Tasks were selected based on their relation to the 2015 Edition Meaningful Use criteria, criticality of
function, and those that may be most troublesome for users.
PROCEDURES
Participants were met at their location as coordinated beforehand in the schedule; their identity was verified
and matched with a name on the participant schedule. Participants were then assigned a participant ID.
Each participant reviewed and signed an informed consent and release form (See Appendix 3) sent by
administrator beforehand.
The usability testing administrator conducting the test was a System Analyst who is also a Registered Nurse
with 10 years of experience in healthcare.
The administrator moderated the session including administering instructions and tasks. The administrator
also monitored task times, obtained post-task rating data, and took notes on participant comments. The
administrator also served as the data logger and took notes on task success, path deviations, number and
type of errors, and comments.
Participants were instructed to perform the tasks (see specific instructions below):
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 12 of 66
As quickly as possible making as few errors and deviations as possible.
Without assistance; administrators were allowed to give immaterial guidance and clarification on
tasks, but not instructions on use.
Without using a think aloud technique.
For each task, the participants were given a written copy of the task a day before the session took place.
Task timing began once the administrator finished reading the question. The task time was stopped once
the participant indicated they had successfully completed the task. Scoring is discussed below in Section
3.9.
Following the session, the administrator gave the participant the post-test questionnaire (e.g., the System
Usability Scale, see Appendix 5), and thanked each individual for their participation.
Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses,
and post-test questionnaire were recorded into a form document.
TEST ENVIRONMENT
The test was performed in a one-on-one session at the participant’s location. The participants started the
session using a Laptop computer provided by the administrator.
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used, including:
1. Informed Consent
2. Moderator’s Guide
3. Post-test Questionnaire
Examples of these documents can be found in Appendices 3-5 respectively. The Moderator’s Guide was
devised so as to be able to capture required data.
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each participant (also see the full moderator’s
guide in Appendix 4):
Thank you for participating in this study. Your input is very important. Our session today will last about 3
hours. During that time you will use an instance of an electronic health record. I will ask you to complete a
few tasks using this system and answer some questions. You should complete the tasks as quickly as
possible making as few errors as possible. Please try to complete the tasks on your own following the
instructions very closely. Please note that we are not testing you we are testing the system, therefore if
you have difficulty all this means is that something needs to be improved in the system. I will be here in
case you need specific help, but I am not able to instruct you or provide help in how to use the application.
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 13 of 66
Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful to
you, and how we could improve it. I did not have any involvement in its creation, so please be honest with
your opinions. All of the information that you provide will be kept confidential and your name will not be
associated with your comments at any time. Should you feel it necessary you are able to withdraw at any
time during the testing.
Following the procedural instructions, participants were shown the EHR and as their first task, were given
time (30 minutes) to explore the system and make comments. Once this task was complete, the
administrator gave the following instructions:
For each task, I will read the description to you and say “Begin.” At that point, please perform the task and
say “Done” once you believe you have successfully completed the task. I would like to request that you not
talk aloud or verbalize while you are doing the tasks. I will ask you your impressions about the task once
you are done.
Participants were then given 31 tasks to complete.
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health
Records, EHRs should support a process that provides a high level of usability for all users. The goal is for
users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this
end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing.
The goals of the test were to assess:
1. Effectiveness of EHRUT by measuring participant success rates and errors
2. Efficiency of EHRUT by measuring the average task time and path deviations
3. Satisfaction with EHRUT by measuring ease of use ratings
DATA SCORING The following table (Table 3) details how tasks were scored, errors evaluated, and the time data analyzed.
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 14 of 66
Measures Rationale and Scoring
Effectiveness:
Task Success
A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis.
The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.
Task times were recorded for successes. Observed task times divided
by the optimal time for each task is a measure of optimal efficiency.
Optimal task performance time, as benchmarked by expert
performance under realistic conditions, is recorded when constructing
tasks. Target task times used for task times in the Moderator’s Guide
must be operationally defined by taking multiple measures of optimal
performance and multiplying by some factor 2.5that allows some time
buffer because the participants are presumably not trained to expert
performance. Thus, if expert, optimal performance on a task was [20
seconds then allotted task time performance was [20 * 1.25] 50
seconds. This ratio should be aggregated across tasks and reported
with mean and variance scores.
Effectiveness:
Task Failures
If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a “Failures.” No task times were taken for errors.
The total number of errors was calculated for each task and then
divided by the total number of times that task was attempted. Not all
deviations would be counted as errors. This should also be expressed
as the mean number of failed tasks per participant.
On a qualitative level, an enumeration of errors and error types should
be collected.
Efficiency:
Task Deviations
The participant’s path (i.e., steps) through the application was
recorded. Deviations occur if the participant, for example, went to a
wrong screen, clicked on an incorrect menu item, followed an incorrect
link, or interacted incorrectly with an on-screen control. This path was
compared to the optimal path. The number of steps in the observed
path is divided by the number of optimal steps to provide a ratio of path
deviation.
It is strongly recommended that task deviations be reported. Optimal
paths (i.e., procedural steps) should be recorded when constructing
tasks.
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 15 of 66
Efficiency:
Task Time
Each task was timed from when the administrator said “Begin” until the
participant said, “Done.” If he or she failed to say “Done,” the time was
stopped when the participant stopped performing the task. Only task
times for tasks that were successfully completed were included in the
average task time analysis. Average time per task was calculated for
each task. Variance measures (standard deviation and standard error)
were also calculated.
Satisfaction:
Task Rating
Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. 2
Common convention is that average ratings for systems judged easy to use should be 3.3 or above.
To measure participants’ confidence in and likeability of the EHRUT
overall, the testing team administered the System Usability Scale
(SUS) post-test questionnaire. Questions included, “I think I would like
to use this system frequently,” “I thought the system was easy to use,”
and “I would imagine that most people would learn to use this system
very quickly.” See full System Usability Score questionnaire in
Appendix 5.3
Table 3. Details of how observed data were scored.
RESULTS
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods specified in the Usability Metrics
section above.
The usability testing results for the EHRUT are detailed below.
2 See Tedesco and Tullis (2006) for a comparison of post-task ratings for usability tests. Tedesco,
D. & Tullis, T. (2006) A comparison of methods for eliciting post-task subjective ratings in usability
testing. Usability Professionals association Conference, June 12 – 16, Broomfield, CO.
3 The SUS survey yields a single number that represents a composite measure of the overall
perceived usability of the system. SUS scores have a range of 0 to 100 and the score is a relative
benchmark that is used against other iterations of the system.
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 16 of 66
Task
Nbr
Measure Task Success
(%)
(Success/Failure)
Path Deviation
Mean (SD)
Task Time
Deviations (Observed / Optimal)
Errors
Mean (SD)
Task Ratings 5=Easy
Mean (SD)
1 Create Medication Order 100/0 10(0) 74/90 0(0) 5(0)
2 Change Medication Order 100/0 10(0) 71/90 0(0) 5(0)
3 Access Medication Order 100/0 4(0) 11/15 0(0) 5(0)
4 Create Laboratory Order 100/0 11(0) 35/45 0(0) 5(0)
5 Change Laboratory Order 100/0 9(0) 34/45 0(0) 5(0)
6 Access Laboratory Order 100/0 5(0) 14/20 0(0) 5(0)
7 Create Imaging (Radiology) Order
100/0 13(0) 45/60 0(0) 5(0)
8 Change Imaging (Radiology) Order
100/0 9(0) 20/25 0(0) 5(0)
9 Access Imaging (Radiology) Order
100/0 7(0) 13/20 0(0) 5(0)
10 Adjust the severity level for drug-drug notifications
100/0 10(9) 19/30 0(0) 5(0)
11 Act upon drug-drug and drug-allergy interventions
100/0 12.8(0.4) 70/90 0.2(0.4) 5(0)
12 Record Patient’s Demographics 100/0 11.8(0.4) 79/90 0.2(0.4) 5(0)
13 Change Patient’s Demographics 100/0 8(0) 37/45 0(0) 5(0)
14 Access Patient’s Demographics 100/0 4(0) 17/30 0(0) 5(0)
15 Record Problem in Problem List 100/0 7(0) 23/30 0(0) 5(0)
16 Change Problem List 100/0 7(0) 22/30 0(0) 5(0)
17 Access Problem List 100/0 5(0) 14/20 0(0) 5(0)
18 Record medication in Medication List
100/0 8(0) 20/30 0(0) 5(0)
19 Change Medication List 100/0 8(0) 16/20 0(0) 5(0)
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 17 of 66
20 Access Medication List 100/0 5(0) 11/15 0(0) 5(0)
21 Record Medication Allergy 100/0 10(0) 24/30 0(0) 5(0)
22 Change Medication Allergy 100/0 6(0) 15/20 0(0) 5(0)
23 Access Patient’s Allergy List 100/0 4(0) 10/15 0(0) 5(0)
24 Configure CDS Interventions 100/0 9(0) 28/30 0(0) 5(0)
25 Act upon Problem List interventions and Access evidence-based, diagnostic and therapeutic reference resources
100/0 12(0) 27/30 0(0) 5(0)
26 Act upon Medication List interventions and Access evidence-based, diagnostic and therapeutic reference resources
100/0 9(0) 24/30 0(0) 5(0)
27 Act upon Medication Allergy List interventions
100/0 11(0) 29/35 0(0) 5(0)
28 Record patient’s implantable device
100/0 8(0) 47/60 0(0) 5(0)
29 Change patient’s implantable device
100/0 4(0) 15/20 0(0) 5(0)
30 Access patient’s implantable device List
100/0 4(0) 13/20 0(0) 5(0)
31 Reconcile patient medication, problem and medication allergy lists with CCDA and generate resulting CCDA
100/0 27(0) 93/120 0(0) 5(0)
32 CDS. Act upon Demographics intervention. Access evidence-based, diagnostic and therapeutic reference resources and review bibliographic citation, developer, funding source, and release/revision date
100/0 9(0) 50/45 0(0) 5(0)
33 CDS. Act upon Laboratory test result intervention
100/0 8(0) 48/50 0(0) 5(0)
34 CDS. Act upon Vital signs intervention
100/0 8(0) 33/35 0(0) 5(0)
35 CDS. Act upon combination (Demographics and Medication) intervention
100/0 10(0) 29/30 0(0) 5(0)
36 CDS. Act upon CCDA reconciliation intervention
100/0 26(0) 127/130 0(0) 5(0)
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 18 of 66
The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based
on performance with these tasks to be: 100. Broadly interpreted, scores under 60 represent systems with
poor usability; scores over 80 would be considered above average.
DISCUSSION OF THE FINDINGS
All participants rated all of the tasks as ‘Easy’.
All participants completed all the tasks. There were minor deviations to the preferred path while entering
patient demographics, due to errors.
EFFECTIVENESS
Software programs may be functional, the successful performance of the required tasks, the manner in
which a user interacts with the software to perform these tasks may be cumbersome and confusing, causing
it to be unusable. In evaluation of the tested EHR, no tasks failed.
Overall all of the individual tasks tested, tested successfully, resulting in a 100% success rate across all
participants.
EFFICIENCY
Based upon observations of task times and deviations, all of the instances of successful tasks, were
completed within the estimated optimal task performance time. Times were benchmarked by expert
performance under realistic conditions, and multiplied by a 2.5 second time buffer to allow for the fact not
all of the participants in this study are expert users.
SATISFACTION
Participants were asked to rate each task individually on a scale of 1= Very Easy to 5= Very Difficult.
Collectively, all of the tasks were rated with a “1= Very Easy”. The System Usability Scale Questionnaire,
as interpreted referencing Jeff Sauro; Measuring Usability, Quantitative Usability, Statistics & Six Sigma,
indicates this application is user friendly and intuitive in all areas tested.
All participants expressed the application was well organized and was rated easy to learn and easy to use.
MAJOR FINDINGS
Participants were overall very pleased with the product. All of the tasks were found easy to use.
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 19 of 66
Some users had difficulty finding the tabs and fields to record and complete the whole patient demographics
task. Some users complained that too much information was needed and found the UI data elements too
far apart.
Some users verbally praised how easy was and intuitive to use the reconciliation tasks.
Some users had experience with other EHRs and praised MDVita for being so well-organized and how
easy and complete it is.
AREAS FOR IMPROVEMENT
Considering the information gathered from the users, the following areas of functionality have been
identified as in need of enhanced usability design:
Record and Change Patient’s Demographics
Demographics MU core components must be grouped close to each other in the UI for the users to identify
and enter faster.
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 20 of 66
APPENDICES
The following appendices include supplemental data for this usability test report. Following is a list of the
appendices provided:
1: Sample recruiting screener
2: Participant demographics
3: Non-Disclosure Agreement (NDA) and Informed Consent Form
4: Example Moderator’s Guide
5: Tasks Sheets
6: System Usability Scale Questionnaire
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 21 of 66
Appendix 1: SAMPLE RECRUITING SCREENER
Recruiting Script
Hello, my name is Ileana Pichel, calling from HealthCare 2000. We are recruiting individuals to participate in a usability study for an electronic health record. We would like to ask you a few questions to see if you qualify and if would like to participate. This should only take a few minutes of your time. This is strictly for research purposes. Can I ask you a few questions?
1. [If not obvious] Are you male or female?
2. Have you participated in a focus group or usability test in the past 12 months? [If yes, Terminate]
3. Do you, or does anyone in your home, work in marketing research, usability research, web design […etc.]?
[If yes, Terminate]
4. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record
software or consulting company? [If yes, Terminate]
5. Would you provide age or age range? _______
6. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-
American, Latino/a or Hispanic, etc.]
7. Do you require any assistive technologies to use a computer? [if so, please describe]
Professional Demographics Customize this to reflect your EHR’s primary audience
1. What is your current position and title? (Must be healthcare provider)
RN: Specialty ________________
Physician: Specialty ________________
Resident: Specialty ________________
Administrative Staff
Other ________________________ 2. How long have you held this position?
3. Describe your work location (or affiliation) and environment? (Recruit according to the intended users of
the application) [e.g., private practice, health system, government clinic, etc.] 4. Which of the following describes your highest level of education? [e.g., high school graduate/GED, some
college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)]. Mark below.
• No high school degree
• High school graduate, diploma or the equivalent (for example: GED)
• Some college credit, no degree
• Trade/technical/vocational training
• Associate degree
• Bachelor’s degree
• Master’s degree
• Doctorate degree (e.g., MD, DNP, DMD, PhD)
Computer Expertise Customize this to reflect what you know about your EHR’s audience
1. Besides reading email, what professional activities do you do on the computer? [e.g., access EHR,
research; reading news; shopping/banking; digital pictures; programming/word processing, etc.] [If no
computer use at all, Terminate]
2. About how many hours per week do you spend on the computer? [Recruit according to the demographics
of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]
3. What computer platform do you usually use? [e.g., Mac, Windows, etc.]
4. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]
5. In the last month, how often have you used an electronic health record?
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 22 of 66
6. How many years have you used an electronic health record?
7. How many EHRs do you use or are you familiar with?
8. MDVita product experience? No ___ Yes ____ how long? ____
9. How does your work environment patient records?
On paper
Some paper, some electronic
All electronic
Contact Information If the person matches your qualifications, ask
Those are all the questions I have for you. Your background matches the people we're looking for.
Would you be able to participate on [date, time]? [If so collect contact information]
May I get your contact information?
Name of participant:
Address:
City, State, Zip:
Daytime phone number:
Evening phone number:
Alternate [cell] phone number:
Email address:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 23 of 66
Appendix 2: PARTICIPANT DEMOGRAPHICS
Following is a high-level overview of the participants in this study.
Gender: Men 4 Women 6 Total Participants 10
Occupation/Role:
Physician 1 Medical Assistance 7 Nurse 1 Office Manager 1
Years of Experience:
(0-5) Years 5 (6-10) Years 5
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 24 of 66
Appendix 3: NON-DISCLOSURE AGREEMENT AND
INFORMED CONSENT FORM
Non-Disclosure Agreement THIS AGREEMENT is entered into as of _ _, 2017, between
_________________________ (“the Participant”) and the testing organization Health Care 2000, Inc.
located at 7205 Corporate Center Drive, Suite 401, Miami, FL 33126.
The Participant acknowledges his or her voluntary participation in today’s usability study may bring the
Participant into possession of Confidential Information. The term "Confidential Information" means all
technical and commercial information of a proprietary or confidential nature which is disclosed by Health
Care 2000, Inc., or otherwise acquired by the Participant, in the course of today’s study.
By way of illustration, but not limitation, Confidential Information includes trade secrets, processes,
formulae, data, know-how, products, designs, drawings, computer aided design files and other computer
files, computer software, ideas, improvements, inventions, training methods and materials, marketing
techniques, plans, strategies, budgets, financial information, or forecasts.
Any information the Participant acquires relating to this product during this study is confidential and
proprietary to Health Care 2000, Inc and is being disclosed solely for the purposes of the Participant’s
participation in today’s usability study.
Participant’s printed name: ___________________________________________
Signature: _____________________________________ Date: ____________________
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 25 of 66
Informed Consent
Health Care 2000, Inc. would like to thank you for participating in this study. The purpose of this study is to evaluate
an electronic health records system. If you decide to participate, you will be asked to perform several tasks using the
prototype and give your feedback. The study will last about 60 minutes.
Agreement
I understand and agree that as a voluntary participant in the present study conducted by Health Care 2000, Inc. I am
free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study
conducted by Health Care 2000, Inc.
I understand and agree that the purpose of this study is to make software applications more useful and usable in the
future.
I understand and agree that the data collected from this study may be shared with outside of Health Care 2000, Inc
and Health Care 2000’s client. I understand and agree that data confidentiality is assured, because only de-identified
data – i.e., identification numbers not names – will be used in analysis and reporting of the results.
I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can
leave at any time.
Please check one of the following:
YES, I have read the above statement and agree to be a participant. NO, I choose not to
participate in this study.
Signature: __________________________________________ Date: ________________
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 26 of 66
Appendix 4: MODERATOR’S GUIDE EHRUT Usability Test
Moderator’s Guide
Administrator ________________________
Data Logger ________________________
Date _____________________________ Time _________
Participant # ________
Location ____________________________
Prior to testing
Confirm schedule with Participants
Ensure EHRUT lab environment is running properly
Ensure lab and data recording equipment is running properly
Prior to each participant:
Reset application
Start session recordings with tool
Prior to each task:
Reset application to starting point for next task
Orientation (10 minutes)
Thank you for participating in this study. Our session today will last 3 hours. During that time you will take a look
at an electronic health record system.
I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy
(or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be
asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or
deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with
anything to do with the system itself. Please save your detailed comments until the end of a task or the end of the
session as a whole when we can discuss freely.
I did not have any involvement in its creation, so please be honest with your opinions.
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 27 of 66
The product you will be using today is describe the state of the application, i.e., production version, early prototype,
etc. Some of the data may not make sense as it is placeholder data.
Do you have any questions or concerns?
Preliminary Questions (10 minutes)
What is your job title / appointment?
How long have you been working in this role?
What are some of your main responsibilities?
Tell me about your experience with electronic health records.
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 28 of 66
Appendix 5: Tasks Sheet
Task 1: Create Medication Order (90 seconds)
Take the participant to the starting point for the task.
While physician is reviewing the Patient’s Medication history, he\she decides to order a Medication for the Patient:
Singulair 10 Mg, take a tablet once a day in the morning. Order 30 units with 6 Refills.
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click “Medication” history
4. Verify Singulair is not one of the active medications
5. Click “Quick Order Medication”
6. Select Patient Insurance, Location (South Dade Medical), Physician, and click Create
7. Click on Diagnosis Tab – Select Diagnosis for medication being ordered (J30.9 – Allergic Rhinitis)
8. Click on Medication Tab - Find Singulair 10 Mg Drug (My Rx – Allergies list) , Click Select
9. Enter SIG: Take 1 Tablet by mouth once a day, in the morning, 30 tablets, 6 refills (Link Med. to DX)
10. Click Save, Acknowledge and Continue, Save and Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 29 of 66
Task 2: Change Medication Order (90 seconds)
Take the participant to the starting point for the task.
After ordering Singulair 10 mg, it is decided to change the medication dose to 5 mg.
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click “Medication” history
4. Verify Singulair 10mg is on the patient’s medication history
5. Click “Quick Order Medication”
6. Select Patient Insurance, Location (South Dade Medical), Physician, and click Create
7. Click on Diagnosis Tab – Select Diagnosis for medication being ordered (J30.9 – Allergic Rhinitis)
8. Click on Medication Tab - Select Stop on Singulair 10 mg, Save, Acknowledge, Save
9. Under Medication Tab - Find (RX) Singulair 5 mg (My Rx – Allergies list) click select
10. Enter SIG: Take 1 Tablet by mouth once a day, in the morning, 30 tablets, 6 refills (Link Med. to DX)
Save, Acknowledge and Continue, Save and Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 30 of 66
Task 3: Access Medication Order (15 seconds)
Take the participant to the starting point for the task.
Accessing medication orders of a patient as part of the daily routine.
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click “Medication” history
4. View medication order history of Singulair drug (+) and Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 31 of 66
Task 4: Create Laboratory Order (45 seconds)
Take the participant to the starting point for the task.
While reviewing a patient chart, the provider orders Lab tests
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Laboratory
4. Click “Quick Order Labs”
5. Select Patient Insurance, Location (South Dade Medical Centers), Physician, and Click Create
6. Click on Diagnosis Tab – Enter 2 Diagnosis for Lab Order (E03.9 – Hypothyroidism, E55.9 Vit. D
Deficiency)
7. Click on Referral/Orders/Lab Tab – Select Lab Tab below
8. Select “Routine” - Click Build
9. Select TSH and Vitamin D Level
10. Click OK – Link to DX’s
11. Click Save and Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 32 of 66
Task 5: Change Laboratory Order (45 seconds)
Take the participant to the starting point for the task.
A Lab order may need update or additional instructions after created.
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Laboratory
4. Click “Instructions” tab
5. Select most recent lab order
6. Click Care Event ID and Edit lab order instructions
7. Click Build
8. Click Urinalysis, Click OK
9. Click OK, Save and Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 33 of 66
Task 6: Access Laboratory Order (20 seconds)
Take the participant to the starting point for the task.
A Lab order may need update or additional instructions after created.
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Laboratory
4. Click “Instructions” tab
5. Review active lab order instructions history
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 34 of 66
Task 7: Create Imaging (Radiology) Order (60 seconds)
Take the participant to the starting point for the task.
After examining a Patient, it is decided to order a chest x-ray
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Orders
4. Click “Quick Order X-Ray”
5. Select Patient Insurance, Location (South Dade Medical Centers), Physician, and click Create
6. Click on Diagnosis Tab and Select Diagnosis for Referral being ordered (R07.9 Chest Pain)
7. Click on Referral\Orders\Lab Tab – Select Referral Tab shown below
8. Select “Routine”
9. Select Radiology “Yes”
10. Click Build
11. Click Chest X-Ray
12. Click OK – Link to DX’s
13. Save and Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 35 of 66
Task 8: Change Imaging (Radiology) Order (25 seconds)
Take the participant to the starting point for the task.
After ordering an X-Ray, the urgency level may need change.
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Referrals
4. Click on Instructions
5. Click Care Event ID and Edit the referral order instructions
6. Click on the Edit Feature for the chest x-ray
7. Select “Urgent” Order Level
8. Click OK
9. Save and Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 36 of 66
Task 9: Access Imaging (Radiology) Order (20 seconds)
Take the participant to the starting point for the task.
Part of the daily routine, access and review Lab Imaging (Radiology) orders
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Referrals
4. Click Instructions
5. Open the most recent Care Event ID = Active Chest X-Ray Ordered
6. View details
7. Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 37 of 66
Task 10: Adjust the severity level of Drug-Drug notifications (30 seconds)
Take the participant to the starting point for the task. Provide the user with credentials with sufficient rights to
change the Drug-Drug level interactions.
A user with the assigned security function may adjust the level of notifications for drug-drug interactions in the
office.
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Logon on MDVita with the User and Password supplied by the administrator
2. Click File
3. Click Options
4. Click System Options
5. Select “Drug Interactions” tab
6. Set Display Level to “Moderate”
7. Check “Exclude Food Interactions”
8. Click Save
9. Click Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 38 of 66
Task 11: Act upon drug-drug and drug-allergy interventions prior medication order completion (90
seconds)
Take the participant to the starting point for the task.
While adding a medication for a patient respond to drug-drug and drug-allergy interventions
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Medications
4. Click Add Medication
5. Find Coumadin tab 1 mg and click Select
6. Review Drug alert interventions
7. Check Acknowledge
8. Click Save
9. Click Add Medication
10. Find Bactrim medication and click Select
11. Review drug Allergic Reaction listed in interventions
12. Click Cancel
13. Review drug was not ordered
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 39 of 66
Task 12: Record Patient’s Demographics (90 seconds)
Take the participant to the starting point for the task.
Need to enter a new Patient:
Joan Arc, DOB: 12/24/1959, Female, Preferred Language: English, Sexual Orientation: Straight, Gender
Identity: Female, Race: White, Ethnicity: Not Hispanic or Latino
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Add Medical Records
2. Enter Patient Name ‘Joan Arc’
3. Enter Address 7205 Corporate Drive, Miami FL 33126
4. Enter date of birth 12/24/1959
5. Enter Gender Female
6. Select Preferred Language English
7. Select Sexual Orientation Straight
8. Select Gender Identity Female
9. Click Tab ‘Additional Info’
10. Select Race1 White
11. Select Ethnicity1 Not Hispanic or Latino
12. Click Save
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 40 of 66
Task 13: Change Patient’s Demographics (45 seconds)
Take the participant to the starting point for the task.
Need to change patient demographics
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Joan Arc’ and click Select
3. Click Profile
4. Change date of birth to 12/24/1958
5. Change Preferred Language to Spanish
6. Click Additional Info tab
7. Change Rac1 to Decline to Specify
8. Click Save
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 41 of 66
Task 14: Access Patient’s Demographics (30 seconds)
Take the participant to the starting point for the task.
Need to access Patient demographics
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Joan Arc’ and click Select
3. Click Profile
4. Review demographics
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 42 of 66
Task 15: Record problem in Problem List (30 seconds)
Take the participant to the starting point for the task.
Need to record patient’s list of current problems
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Problem List
4. Click Add Problem
5. Find G47.00 - Insomnia and click Select
6. Enter 1/1/2017 Start date, Status Active
7. Click Save
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 43 of 66
Task 16: Change Problem List (30 seconds)
Take the participant to the starting point for the task.
Need to modify recorded patient’s list of current problems
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Problem List
4. Find Insomnia diagnosis
5. Change status from Active to Resolved
6. Enter modified date on 11/20/2017
7. Click Save
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 44 of 66
Task 17: Access Problem List (20 seconds)
Take the participant to the starting point for the task.
Patient arrives for a visit and wants to see a list of Active Problems
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Problem List
4. Click Active Only
5. Review problem list
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 45 of 66
Task 18: Record medication in Medication List (30 seconds)
Take the participant to the starting point for the task.
Need to record patient’s list of current medications
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Medications
4. Click Add Medication
5. Find Protonix 20 MG and click Select
6. Enter 1/1/2017 Start date
7. Check Acknowledge Interventions
8. Click Save
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 46 of 66
Task 19: Change Medication List (20 seconds)
Take the participant to the starting point for the task.
Need to modify recorded patient’s list of current medications
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Medications
4. Find Protonix medication and expand history
5. Click Edit in the latest medication instance
6. Update SIG details to “Once daily”
7. Enter Days: 30, Qty:30
8. Check Acknowledge Interventions notifications
9. Click Save
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 47 of 66
Task 20: Access Medication List (15 seconds)
Take the participant to the starting point for the task.
Patient arrives for a visit and wants to see what medications he\she is currently taking.
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Medications
4. Click Active Only
5. Review medication list
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 48 of 66
Task 21: Record medication allergy (30 seconds)
Take the participant to the starting point for the task.
Seeing a patient and need to record patient’s allergies
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Allergies
4. Click Add
5. Select Tab ‘Drugs’
6. Select Penicillin in the list
7. Verify the Status is set to ‘Active’
8. Enter Today’s date in Allergy Date.
9. Select reactions: Respiratory Distress and Swelling
10. Click Save
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 49 of 66
Task 22: Change Medication Allergy (20 seconds)
Take the participant to the starting point for the task.
Seeing a patient and need to change the allergy details
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Allergies
4. Find Penicillin allergy and click Edit
5. Remove ‘Respiratory Distress’ reaction and add ‘Watery eyes’
6. Click Save
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 50 of 66
Task 23: Access Patient’s Allergy List (15 seconds)
Take the participant to the starting point for the task.
The patient arrives for a visit. Verify with the patient that the allergy list is current.
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Allergies
4. Review patient allergies
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 51 of 66
Task 24: Configure CDS interventions (30 seconds)
Take the participant to the starting point for the task. The user account requires specific security in order to
perform this task.
Need to setup users for which CDS interventions are enabled
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. From main menu click ‘File’
2. Click ‘Security’
3. Click ‘Users’
4. Select your user account and Edit
5. Select ‘CDS’ tab
6. Enable the following Clinical CDS rules: Malignant Hypertension (Problem), Prescribing Selegiline
(Medication), Penicillin (Medication Allergy), Elevated bilirubin (Lab Test) , Fall prevention over 90
years of age (Demographics), Hypertension in very elderly (Vitals), and Potentially Inappropriate
Medication use in Older Adults (Combination)
7. Enable ‘Transition of Care’ and ‘Show References’ for rules above
8. Enable ‘Access to Diagnostic and Therapeutic resources’
9. Click Save and Close the screen
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 52 of 66
Task 25: Act upon Problem List interventions (30 seconds)
Take the participant to the starting point for the task.
Review and acting upon CDS alerts while entering a specific problem (Diagnosis)
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘John Smith’ and click Select
3. Click ‘Problem List’
4. Click Add
5. Search for ICD10: I10
6. Click ‘Select’
7. Save
8. Open CDS alert, review intervention/resource information , and click ‘Handled’
9. Click on ‘D&T Resources’
10. Check Problem entered and click ‘Search’
11. Review Diagnosis and Therapeutic materials
12. Click ‘Close’
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 53 of 66
Task 26: Act upon Medication List interventions (30 seconds)
Take the participant to the starting point for the task.
Review and acting upon CDS alerts while entering a specific medication
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘John Smith’ and click Select
3. Click ‘Medications’
4. Click Add
5. Search for ‘Selegiline Tab 5 MG’
6. Click ‘Select’
7. Check ‘Acknowledge and Continue’ for alerts
8. Save
9. Open CDS alert, review intervention/resource information , and click ‘Handled’
10. Click Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 54 of 66
Task 27: Act upon Medication Allergy List interventions (35 seconds)
Take the participant to the starting point for the task.
Review and acting upon CDS alerts while entering a specific medication to which the patient is allergic to.
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘John Smith’ and click Select
3. Click ‘Allergies’
4. Click Add
5. Select Cephalosporins
6. Enter Allergy Date 1/1/2017
7. Select Reaction ‘Agitation’
8. Click ‘Save’
9. Open CDS alert, review intervention/resource information , and click ‘Handled’
10. Click ‘Close’
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 55 of 66
Task 28: Record Patient’s Implantable Device (60 seconds)
Take the participant to the starting point for the task.
Record information about a Patient’s implantable device
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click ‘Implants’
4. Click ‘Add’
5. Enter an implantable device UDI, supplied by proctor
(01)00643169007222(17)160128(21)BLC200461H
6. Click Find
7. Enter today’s date in Start Date
8. Click Save
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 56 of 66
Task 29: Change Patient’s Implantable Device (20 seconds)
Take the participant to the starting point for the task.
Need to change patient demographics
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Implants
4. Change Status to ‘Inactive’
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 57 of 66
Task 30: Access Patient’s Implantable Device (20 seconds)
Take the participant to the starting point for the task.
Need to access Patient demographics
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Jeremy Bates’ and click Select
3. Click Implants
4. Review Implantable Device list
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 58 of 66
Task 31: Reconcile patient medication, problem, and medication allergy lists with a CCDA (120 seconds)
Take the participant to the starting point for the task.
Import patient medication, problem, and medication allergy lists from a CCDA file received from another
provider
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘John Matters’ and click Select
3. Click ‘Import – Patient Information’ from the contextual menu
4. Click Transfer of Care
5. Got to directory ‘D:\MDVita\UsabilityTest’ and select file: MATTERS, JOHN.CDA.xml
6. Review document received
7. Click ‘Save’
8. Answer ‘Yes’ to the question ‘Do you want to incorporate document data?”
9. Allergy List is shown. Reconcile Allergy List selecting ‘Add Record’ from drop down
10. Click Incorporate
11. Review resulting Allergy list
12. Click ‘Confirm and Save’
13. Select Medication List
14. Reconcile Medication selecting ‘Add Record’
15. Click ‘Incorporate’
16. Review the resulting Medication list.
17. Click ‘Confirm and Save’
18. Select Problem List
19. Reconcile Problem List selecting ‘Add Record’
20. Click ‘Incorporate’
21. Review the resulting Problem list
22. Click ‘Confirm and Save’
23. Click Close
24. Open Patient ‘John Matters’ chart
25. Review Medication, Problem and Allergy lists
26. Click Export Medical Record
27. Save resulting CCDA
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 59 of 66
Task 32: Act upon Demographics intervention (45 seconds)
Take the participant to the starting point for the task.
Review and acting upon CDS alerts while entering patient’s demographics
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘John Smith’ and click Select
3. Click Profile
4. Change Date of Birth to 01/01/1924
5. Click Save
6. Open CDS alert, review intervention/resource information ,
7. Review bibliographic citation, developer, funding source, and release/revision date
8. click ‘Handled’
9. Click Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 60 of 66
Task 33: Act upon Laboratory test intervention (50 seconds)
Take the participant to the starting point for the task.
Review and acting upon CDS alerts while entering a specific lab test result
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘John Smith’ and click Select
3. Click Laboratory
4. Click Add
5. Enter a new laboratory result using LOINC 1975-2 (Bilirubin) and value 1.93
6. Click Save
7. Open CDS alert, review intervention/resource information , and click ‘Handled’
8. Click Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 61 of 66
Task 34: Act upon Vital Signs intervention (35 seconds)
Take the participant to the starting point for the task.
Review and acting upon CDS alerts while entering patient vital signs
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘John Smith’ and click Select
3. Click care Events
4. Click Add
5. Enter Blood Pressure values 170/99
6. Click Save
7. Open CDS alert, review intervention/resource information , and click ‘Handled’
8. Click Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 62 of 66
Task 35: Act upon combination (Demographics and Medication) intervention (30 seconds)
Take the participant to the starting point for the task.
Review and acting upon CDS alerts while entering patient Medication and triggering a CDS combined rule of
Medication and Demographics (Age).
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed
Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘John Smith’ and click Select
3. Click ‘Medications’
4. Click Add
5. Search for ‘Benadryl 25 MG’
6. Click ‘Select’
7. Check ‘Acknowledge and Continue’ for alerts
8. Click Save
9. Open CDS alert, review intervention/resource information , and click ‘Handled’
10. Click Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments:
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 63 of 66
Task 36: Act upon CCDA reconciliation interventions (130 seconds)
Take the participant to the starting point for the task.
Import patient medication, problem, and medication allergy lists from a CCDA file received from another
provider and act upon triggered interventions during reconciliation
Success:
Easily completed
Completed with difficulty or help :: Describe below
Not completed Comments:
Task Time: ________ Seconds
Optimal Path:
1. Open Patient Search
2. Search for Patient ‘Aurora Smith’ and click Select
3. Click ‘Import – Patient Information’ from the contextual menu
4. Click Transfer of Care
5. Got to directory ‘D:\MDVita\UsabilityTest’ and select file: CDS.CDA.xml
6. Review document received
7. Click ‘Save’
8. Answer ‘Yes’ to the question ‘Do you want to incorporate document data?”
9. Allergy List is shown. Reconcile Allergy List selecting ‘Add Record’ from drop down
10. Click Incorporate
11. Review resulting Allergy list
12. Click ‘Confirm and Save’
13. Open CDS alert, review intervention/resource information , and click ‘Handled’
14. Select Medication List
15. Reconcile Medication selecting ‘Add Record’
16. Click ‘Incorporate’
17. Review the resulting Medication list.
18. Click ‘Confirm and Save’
19. Open CDS alert, review intervention/resource information , and click ‘Handled’
20. Select Problem List
21. Reconcile Problem List selecting ‘Add Record’
22. Click ‘Incorporate’
23. Review the resulting Problem list
24. Click ‘Confirm and Save’
25. Open CDS alert, review intervention/resource information , and click ‘Handled’
26. Click Close
Correct
Minor Deviations / Cycles (Describe below)
Major Deviations (Describe below)
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: _____
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 64 of 66
Final Questions (15 Minutes)
What was your overall impression of this system?
What aspects of the system did you like most?
What aspects of the system did you like least?
Were there any features that you were surprised to see?
What features did you expect to encounter but did not see? That is, is there anything that is missing in this
application?
Compare this system to other systems you have used.
Would you recommend this system to your colleagues?
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 65 of 66
Appendix 6: SYSTEM USABILITY SCALE QUESTIONNAIRE In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems usability”
known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have elaborated on the SUS over
the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
Strongly Strongly
disagree agree
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
4. I think that I would need the support of
a technical person to be able to use this
system
5. I found the various functions in this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25
Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 Page 66 of 66
9. I felt very confident using the system
9. I needed to learn a lot of things before I could get going with this system
1 2 3 4 5
1 2 3 4 5