Page 1 of 27
Usability Test Report for HealthShare 2013.1
Table of Contents Executive Summary ....................................................................................................................................... 2
Introduction .................................................................................................................................................. 4
Method ......................................................................................................................................................... 4
Participants ............................................................................................................................................... 4
Study Design.............................................................................................................................................. 6
Tasks .......................................................................................................................................................... 7
Procedures ................................................................................................................................................ 7
Test Environment ...................................................................................................................................... 7
Participant Instructions ............................................................................................................................. 7
Data Scoring .............................................................................................................................................. 8
Results ........................................................................................................................................................... 9
Task by Task Results .................................................................................................................................. 9
Major Findings ........................................................................................................................................ 18
Areas for Improvement ........................................................................................................................... 19
Appendix A – Participant Documentation and Feedback Form .................................................................. 20
Appendix B – Complete Numeric Results of HealthShare Usability Test .................................................... 26
Page 2 of 27
Executive Summary A usability test of HealthShare 2013.1, a strategic informatics platform often used to create and view an
aggregated patient record, was conducted on September 10-12, 2013. The test was executed from
various sites by logging into a centrally configuration solution. This ensured participation by a collection
of representative users from various backgrounds and regions.
The study assessed the usability of searching for and identifying a patient, viewing the medication list
and viewing the allergy list. For each step in the process, the time was recorded at each step.
Participants were given an outline, introducing them to the purpose of the test, and were instructed to
complete a series of tests and document their results in a form, found in Appendix A. Participants were
not given assistance in navigating the test.
When beginning the test, the system presented the user with a unique ID that was recorded on their
form. This allowed the administrator to cross reference the recorded observations with the timing
recorded in the test session.
The following data was requested or recorded:
Time to execute each task
Time to complete the entire series of tests
Path deviations
Participant impressions and notes on usability
Participant satisfaction ratings
The following background data was collected on each participant:
Clinical Experience, role and experience level
Prior Experience with this system
Location
Technical details of the solution used for access – internet browser and operating system
The participant’s identity was not associated with the collected data. The system recorded data was
matched to the data recorded by the participant via a unique session ID created during the testing
session. No compensation was given to participants.
A total of twenty (20) participants completed the test.
Based on the established criteria for Effectiveness, Efficiency and Satisfaction, all tasks were considered
successful .
Page 3 of 27
Efficiency Satisfaction Effectiveness
Task Time Goal
Avg Time (s)
Errors Avg Time Reported (s)
Avg Deviations
Average
Login N/A Not applicable 9.05
Patient Search 20 12.59 0 8.95 Patient Select 30 11.08 0.1
5 8.65
View Summary 60 30.52 0 8.4 View Allergies 60 34.05 0.0
5 8.85
View Meds 90 57.47 0.65
8.35 Logout N/A Not applicable 9.35
Figure 1: Summarized Results of Usability Test
0 12.59 0 11.08 0 30.52 0 34.05 0 57.47
Page 4 of 27
Introduction The Electronic Health Record Under Test (EHRUT) in this usability study was HealthShare 2013.1.
Designed to aggregate and present composite health records of a single patient to a clinician,
HealthShare is used in varied clinical settings. Typically used to create an aggregated record on a
regional, state-wide or national level, the system is not used as the day-to-day EHR system by a clinician
but is used to view aggregated records for patients who may be receiving care from multiple settings.
Accordingly, HealthShare does not allow the manual entry of data by clinicians. Recording and updating
of clinical data, including medication allergies and medications, is done electronically by submitting data
to HealthShare in a standard or custom format, such as HL7v2, CCD, CDA, etc. Therefore, usability
testing centered around accessing the data in an effective, efficient and satisfactory manner.
The purpose of the test was to verify that a typical clinical user can access the patient record and access
the specific clinical data for medication allergies and medications. Both subjective and objective
measures were captured in the areas of effectiveness, efficiency and satisfaction.
Method
Participants A total of 20 participants were tested on the EHRUT. Participants in this test were Physicians,
Pharmacists, Nurses and Laboratory or Radiology Technologists, by background. All participants are
employees of InterSystems working in varied offices around the world and on different products.
Recruited participants had a mix of clinical backgrounds and reside in different regions around the
world. Participants were given one day to login and complete the test.
Participants resided in the United States (6), Australia (9), Germany (1), United Arab Emirates (2), United
Kingdom (1) and Brazil (1).
Figure 2: Participant Breakdown by Region
Page 5 of 27
Participants were asked to use their own systems to connect to the server and complete the test.
Eleven (11) of the 20 participants used Windows, while nine (9) use Apple (MacOS) based systems. For
Windows users, Internet Explorer was the predominant browser used by eight (8) of the participants.
The other Windows-based participants used Firefox (2) and Chrome (1). For the MacOS based
participants, Safari was the predominant browser choice, representing five (5) of the responses with the
others using Chrome (2) and Firefox (2).
Figure 3: Client Operating System and Browser Details
The clinical background and years of experience was varied. Participants all had a clinical background as
most users of the system for viewing clinical data would have a clinical background. Participants were
either a physician (8), nurse (7), pharmacist (3) or laboratory/radiology technician (2). All users currently
work in IT, but some still practice as well.
Page 6 of 27
Figure 4: Participant Breakdown by Clinical Background
Years of experience in clinical practice ranged from 0 to 30 years, with the collective average being 15.45
years.
Study Design The goal of this test was to ensure the typical clinical user can intuitively access the system, identify the
patient and access the clinical data for medication allergies and medication allergy list. The study is
designed to uncover any areas of usability challenges which have a negative impact on the effectiveness,
efficiency and user satisfaction of the system.
The data collected here may serve as a baseline for future testing.
Participants executed the tasks from their own environments remotely and without supervision. All
participants were fluent in English and were provided with the same instructions and feedback form,
written in English. The system logged times for execution of each task. Feedback forms were submitted
following the test.
The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected
and analyzed for each participant:
Tasks completed successfully
Time to complete tasks
Number and types of errors
Path deviations
Participant’s comments
Participant’s ratings on effectiveness, efficiency and satisfaction with the system
Page 7 of 27
Tasks
The following tasks were outlined to be representative of the kinds of activities a clinical user would
perform in the EHRUT:
1. Login to the EHRUT
2. Search for a specific patient
3. Select that patient to view the patient record
4. Review the summary chart for the patient
5. View the patient’s allergy list
6. View the patient’s medication list
7. Logout to end the session
Procedures Participants were given instructions and time frames within which to complete the test via email. An
automatically generated ID uniquely identifying the test session is created and provided on login.
Following test completion, participants completed the feedback form and submitted it to the
administrators of the test.
Participants were instructed to perform the outlined tasks as quickly as possible, while taking time to
process and understand the clinical data, as they would in a real care setting.
The Task instructions were provided and participants were advised to print these instructions for
efficient execution.
Participants were advised to complete the feedback form following the completion of all seven tasks.
Participant details, task times, deviations and comments were recorded into a spreadsheet.
Test Environment The EHRUT would typically be accessed from a care setting, such as a hospital or clinic, using a
supported web browser. For this test, the environment was accessed remotely via supported web
browser.
The EHRUT solution was running on Windows, hosted in Amazon Cloud Services, allowing for remote
access by the participants.
The variance of browsers, access locations, and system operating systems is representative of a typical
regional health information exchange which would use the EHRUT. The system performance should also
be representative of a typical setup.
Participant Instructions Participants received a simple email asking them to participate within a day. The instructions for
accessing and completing the test were included in a Word doc supplied electronically to participants.
This contained an explanation of the test, access instructions, the test script and feedback form. This
HealthShare Usability Testing document is included in Appendix A.
Page 8 of 27
Data Scoring
To rate the system for effectiveness and efficiency, time measures were taken as each participant
navigated the tasks.
Effectiveness: Task success was deemed to be the successful completion of the task, without error and
without a reasonable amount of time for that task. Task failure was deemed to be if the participant
abandoned the task, did not succeed at the task within a reasonable amount of time or received and
error.
Efficiency: Task success was deemed to be if the participant was able to complete the task quickly and
efficiently without deviations. Task failure was deemed to be if the participant was not able to complete
the task without deviations quickly. Average and max/min times were calculated.
Satisfaction: For each task, the participant was asked to use a likert scale from 1-10 to indicate their
satisfaction with the solution for that task, with 1 being Not Satisfied and 10 being Very Satisfied. A
score of 7 was considered acceptable.
Page 9 of 27
Results
Task by Task Results Participants were asked to complete seven steps in this usability test.
1. Login to the EHRUT
2. Search for a specific patient
3. Select that patient to view the patient record
4. Review the summary chart for the patient
5. View the patient’s allergy list
6. View the patient’s medication list
7. Logout to end the session
For each step, the system collected data as the participant navigated the system, including time for
completion of each step, as well as any diversions. For each test item, the participant responded to a
feedback form, following the test which allowed them to provide a satisfaction rating (on a likert scale
from 1-10), as well as add addition comments on any difficulties they experienced while executing the
test and any comments on the usability of this portion of the software.
Steps 1 and 7 (Login and Logout) were not scored for effectiveness and efficiency since they were not
related to the clinical use of the system being evaluated. Login and Logout were only scored on
Satisfaction.
The complete spreadsheet of results is attached in Appendix B – Usability Testing Results. In this matrix,
all participants are identified by an anonymous session ID which was created for their session. Users
were presented with this session ID when they accessed the system for the first time and recorded it on
their feedback form. The matrix does not contain a sequential set of IDs as administrative users were
testing/using the system in the interim and generated IDs as well. Those IDs were eliminated and only
those tests from clinical participants who submitted a feedback form were used in the collected data.
Results for Task 1: Login
Only Satisfaction was tracked for the Login task. On a scale of 1-10, Login received an average score of
9.05 with standard deviation of 1.54. The minimum score was 5, while the maximum score was 10.
Average 9.05
Standard Deviation 1.54
Min 5.00
Max 10.00
With a goal of 7 for satisfaction, Login was deemed to be successful.
Page 10 of 27
Results for Task2: Patient Search
In this test, participants were asked to search for a specific patient – John Smith. No training on the use
of the user interface was given. This test was scored on efficiency, effectiveness and satisfaction.
The task was considered to be efficient if it was successfully completed in under twenty (20) seconds
with less than an average of 0.5 deviations across the participant set.
The task was considered to be effective if it was completed without error and in under twenty (20)
seconds. To evaluate this, a combination of automatically calculated data, along with user response
data regarding errors was evaluated.
Below are the complete data, across the twenty participants for the Patient Search test.
Patient Search
Time (Search)
Deviations (Search)
Satisfaction (Search)
11.607 0 10
11.326 0 10
9.641 0 10
15.99 0 8
18.798 0 10
0 7
13.167 0 8
9.89 0 10
11.232 0 10
14.633 0 7
4.056 0 9
11.435 0 10
8.658 0 8
10.436 0 7
17.254 0 10
15.288 0 8
10.312 0 10
9.095 0 10
16.973 0 10
19.329 0 7
Time Deviations Satisfaction
Goals 20.00 0.50 7.00
Average 12.59 0.00 8.95
Standard Deviation
3.95
0.00
1.28
Min 4.06 0.00 7.00
Max 19.33 0.00 10.00
Page 11 of 27
As noted above, the average time to complete the task was 12.59 seconds, well under the goal of 20
seconds. The max across the participants was 19.33 seconds. No deviations were recorded across the
entire participant set.
The Satisfaction scoring showed an average of 8.95, well above the 7.0 goal. Standard deviation on
satisfaction scoring was 1.28 with minimum recorded score of 7 and maximum recorded score at 10.
No errors were reported in completing this task by the twenty participants. A few participants indicated
a slow response time, but this was considered to be related to internet speed and not software
response time as slow response times were reported across the test set for certain participants, but the
system performed very well for others in different locations.
Accordingly, this task was deemed to meet acceptable standards for efficiency, effectiveness and
satisfaction.
Results for Task 3: Patient Select
In this test, participants were asked to view the results of their search and select the specified patient,
John Smith. The test was scored on efficiency, effectiveness and satisfaction.
The task was considered to be efficient if it was successfully completed in under thirty (30) seconds with
less than an average of 0.5 deviations across the participant set. More time was allotted to this task
than the previous task is it is very important for the clinician to review search results and select the
correct patient.
The task was considered to be effective if it was completed without error and in under thirty (30)
seconds. To evaluate this, a combination of automatically calculated data, along with user response
data regarding errors was evaluated.
Patient Select
Time (Select)
Deviations (Select)
Satisfaction (Select)
3.79 0 9
9.344 2 10
2.59 0 9
4.087 0 9
14.087 0 8
12.386 0 5
9.64 0 8
5.46 0 10
5.616 0 10
9.266 0 8
4.805 0 9
8.44 0 10
10.67 0 9
26.099 0 7
12.012 0 10
18.252 1 8
Page 12 of 27
15.631 0 10
3.588 0 9
21.746 0 10
24.102 0 5
Time Deviations Satisfaction
Goal 30.00 7.00
Average 11.08 0.15 8.65
Standard Deviation
7.03
0.49
1.53
Minimum 2.59 0.00 5.00
Maximum 26.10 2.00 10.00
As noted above, the average time to complete the task was 11.08 seconds, well under the goal of 30
seconds. The max across the participants was 26.10 seconds. Across all the participants, only two
participants deviated from the path, giving a deviation average for the test as 0.15.
The Satisfaction scoring showed an average of 8.65, well above the 7.0 goal. Standard deviation on
satisfaction scoring was 1.53 with minimum recorded score of 5 and maximum recorded score at 10.
No errors were reported in completing this task by the twenty participants. A few participants indicated
a slow response time, but this was considered to be related to internet speed and not software
response time as slow response times were reported across the test set for certain participants, but the
system performed very well for others in different locations.
Accordingly, this task was deemed to meet acceptable standards for efficiency, effectiveness and
satisfaction.
Results for Task 4: View Summary
In this test, participants were asked to view the summary of the clinical record for the patient. The test
was scored on efficiency, effectiveness and satisfaction.
The task was considered to be efficient if it was successfully completed in under sixty (60) seconds with
less than an average of one (1) deviations across the participant set. More time was allotted to this task
than the previous task as clinicians were not presented with clinical data previously and may review it
for a longer period of time.
The task was considered to be effective if it was completed without error and in under sixty (60)
seconds. To evaluate this, a combination of automatically calculated data, along with user response
data regarding errors was evaluated.
View Summary
Time (Summary)
Deviations (Summary)
Satisfaction (Summary)
47.861 0 7
11.419 0 10
Page 13 of 27
11.716 0 10
41.855 0 9
19.204 0 10
44.335 0 5
6.849 0 10
38.61 0 10
26.598 0 10
30.155 0 8
3.245 0 9
12.636 0 10
12.293 0 8
0 6
21.107 0 6
37.3 0 8
17.285 0 10
17.846 0 7
22.683 0 10
156.843 0 5
Time Deviations Satisfaction
Goals 60.00 7.00
Average 30.52 0.00 8.40
Standard Deviation
33.33
0.00
1.82
Minimum 3.25 0.00 5.00
Maximum 156.84 0.00 10.00
As noted above, the average time to complete the task was 30.52 seconds, well under the goal of 60
seconds. The max across the participants was 156.84 seconds. The standard deviation was 33.33,
minimum time was 3.25 and maximum time was 156.84. On analyzing across all tests, it was noticed
that the final participant (identified in the results matrix in Appendix B as Participant Session 44) took a
much larger than average amount of time on every test. As there was no active monitoring of the test,
it is unknown if this participant was distracted or multi-tasking. For reference and comparison to the
goal, the time ratings when eliminating Participant 44 and using the data from the other 19 participants
results in the following time data:
Average 23.50
Std Dev 13.61
Min 3.25
Max 47.86
No participants deviated from the test. No errors were reported when executing this task.
Page 14 of 27
Based on the overall average (with all 20 participants data being used in the calculations) of 30.52, as
well as the lack of deviations, this task is considered to be efficient. The time data, along with the lack of
errors reported, indicates this task is effective.
The Satisfaction scoring showed an average of 8.40, well above the 7.0 goal. Standard deviation on
satisfaction scoring was 1.82 with minimum recorded score of 5 and maximum recorded score at 10.
Accordingly, this task was deemed to meet acceptable standards for efficiency, effectiveness and
satisfaction.
Results for Task 5: View Allergy List
In this test, participants were asked to view the allergy list for the patient by clicking on the Allergies and
Alerts tab in the clinical record. The test was scored on efficiency, effectiveness and satisfaction.
The task was considered to be efficient if it was successfully completed in under sixty (60) seconds with
less than an average of one (1) deviations across the participant set. More time was allotted to this task
than the previous task as clinicians are reviewing clinical data.
The task was considered to be effective if it was completed without error and in under sixty (60)
seconds. To evaluate this, a combination of automatically calculated data, along with user response
data regarding errors was evaluated.
View Allergies
Time (Allergies)
Deviations (Allergies)
Satisfaction (Allergies)
44.491 0 8
21.637 0 10
12.433 0 10
80.293 0 9
3.089 0 10
61.76 0 6
14.976 0 10
12.605 0 10
36.535 0 10
52.088 0 6
24.367 0 8
8.175 0 10
30.17 0 10
1 7
11.746 0 10
39.031 0 7
35.583 0 10
5.101 0 10
9.329 0 10
143.567 0 6
Time Deviations Satisfaction
Goals 60.00 7.00
Page 15 of 27
Average 34.05 0.05 8.85
Standard Deviation
33.74
0.22 1.60
Min 3.09 0.00
6.00
Max 143.57 1.00 10.00
As noted above, the average time to complete the task was 34.05 seconds, well under the goal of 60
seconds. The max across the participants was 143.57 seconds. The standard deviation was 33.74,
minimum time was 3.09 and maximum time was 143.57. On analyzing across all tests, it was noticed
that the final participant (identified in the results matrix in Appendix B as Participant Session 44) took a
much larger than average amount of time on every test. As there was no active monitoring of the test,
it is unknown if this participant was distracted or multi-tasking. For reference and comparison to the
goal, the time ratings when eliminating Participant 44 and using the data from the other 19 participants
results in the following time data:
Average 27.97
Std Dev 21.47
Min 3.09
Max 80.29
This resulted in a reduced average of 29.97. One participant was still above our 60 second goal.
However, with all other participants performing the task in significantly less than the allotted time, the
task is considered to have met its time goal.
Only one participant deviated from the test and made only a single deviation. No errors were reported
when executing this task.
Based on the overall average (with all 20 participants data being used in the calculations) of 34.05, as
well as the adjusted average of 27.97 seconds when removing data from Participant 44, the task met its
time goal for efficiency. Combined with the lack of deviations across the participant set (one deviation
in total for 0.05 average), this task is considered to be efficient. The time data, along with the lack of
errors reported, indicates this task is effective.
The Satisfaction scoring showed an average of 8.85, well above the 7.0 goal. Standard deviation on
satisfaction scoring was 1.60 with minimum recorded score of 6 and maximum recorded score at 10.
Accordingly, this task was deemed to meet acceptable standards for efficiency, effectiveness and
satisfaction.
Results for Task 6: View Medication List
In this test, participants were asked to view the medication list for the patient by clicking on the
Medications tab in the clinical record. The test was scored on efficiency, effectiveness and satisfaction.
Page 16 of 27
The task was considered to be efficient if it was successfully completed in under ninety (90) seconds
with less than an average of one (1) deviations across the participant set. More time was allotted for
this task as reviewing Medication History is a lengthier process than allergies. Recent and Historical
Medications are displayed and there is more data to view.
The task was considered to be effective if it was completed without error and in under ninety (90)
seconds. To evaluate this, a combination of automatically calculated data, along with user response
data regarding errors was evaluated.
View Meds
Time (Meds)
Deviations (Meds)
Satisfaction (Meds)
47.861 0 3
48.267 2 8
17.347 0 9
25.085 0 9
13.93 0 10
118.373 0 6
9.796 0 10
23.993 0 10
32.308 0 10
99.294 0 7
30.014 0 8
56.877 0 10
59.171 0 10
1 6
87.158 8 10
42.447 0 8
3.525 1 10
151.196 0 7
21.247 1 10
203.985 0 6
Time Deviations Satisfaction
Goal 90.00 7.00
Average 57.47 0.6
5 8.35
Standard Deviation
53.00
1.81
1.98
Min 3.53 0.00
3.00
Max 203.99 8.00 10.00
As noted above, the average time to complete the task was 57.47 seconds, which is under the goal of 90
seconds. The max across the participants was 203.99 seconds, while the minimum was 3.53. The
standard deviation was 53.00. On analyzing across all tests, it was noticed that the final participant
(identified in the results matrix in Appendix B as Participant Session 44) took a much larger than average
amount of time on every test. As there was no active monitoring of the test, it is unknown if this
Page 17 of 27
participant was distracted or multi-tasking. For reference and comparison to the goal, the time ratings
when eliminating Participant 44 and using the data from the other 19 participants results in the
following time data:
Average 49.33
Std Dev 40.51
Min 3.53
Max 151.20
This resulted in a reduced average of 49.33. Four participants, in total, took longer than 90 seconds to
complete the task, while the other 16 participants completed the task in under the 90 second goal Based
on the average, the task is considered to have met its time goal.
Five out of twenty (5/20) participants deviated from the task, resulting in 0.65 deviations across the
task. No errors were reported when executing this task.
Based on the overall average (with all 20 participants data being used in the calculations) of 57.47, as
well as the adjusted average of 49.33 seconds when removing data from Participant 44, the task met its
time goal for efficiency. Combined with the low number of average deviations across the participant
set, this task is considered to be efficient. The time data, along with the lack of errors reported,
indicates this task is effective.
The Satisfaction scoring showed an average of 8.35, well above the 7.0 goal. Standard deviation on
satisfaction scoring was 1.98 with minimum recorded score of 3 and maximum recorded score at 10.
Accordingly, this task was deemed to meet acceptable standards for efficiency, effectiveness and
satisfaction.
Commentary on the Medication section gave feedback on how medication presentation could be
organized more effectively for different care settings. This is described further in the Areas for
Improvement section.
Results for Task 7: Logout
Only Satisfaction was tracked for the Logout task. On a scale of 1-10, Login received an average score of
9.05 with standard deviation of 1.54. The minimum score was 5, while the maximum score was 10.
Average 9.35
Standard Deviation 0.93
Min 7.00
Max 10.00
With a goal of 7 for satisfaction, Logout was deemed to be successful with an average score of 9.35 and
a minimum score of 7.0.
Page 18 of 27
Major Findings
Based on the collection of feedback, the tasks were all successful and indicated the system is quite
usable for searching and selecting a patient record, viewing medication allergies and viewing a patients
medication list. While minor suggestions were logged (noted in the Areas for Improvement section), the
average satisfaction scores indicated the system is currently usable in a clinical setting.
Page 19 of 27
Areas for Improvement
Written feedback was collected on each task item. Upon reviewing this data, areas of improvement
were identified in some of the tasks. In each of these areas, further analysis with the clinical team to
focus on specific areas will be undertaken.
Patient Search
A few participants commented that there were a large number of potential search fields and suggested
that fewer search options be provided for the average user or that the form be sectioned to draw the
user’s attention to the most commonly used areas for search – e.g. Name or Date of Birth.
As this screen is completely customizable by site, reducing and reorganizing the fields is part of the
system setup and configuration. However, it will be placed under consideration to generally reduce the
default number of fields shown.
View Allergy List
A few participants suggested that allergies for medication category could be highlighted differently than
other recorded allergies (e.g. food allergies). This will be considered for a future enhancement.
View Medication List
Many participants, depending on their care setting, suggested improvements to the Medication List,
including presentation of additional data on the main list (without navigating to details) and different
organization between recent and historical medications. Additional focus on the Medication List with a
clinical focus group will be planned.
Page 20 of 27
Appendix A – Participant Documentation and Feedback Form
HealthShare Usability Testing
Thank you for agreeing to participate in our HealthShare Usability testing. Please read this complete
document, including script and form elements you will be responding to prior to beginning the test.
This test is focused specifically around accessing the medication list and allergy list for a specific patient.
During this test, we will be asking you to perform the following tasks:
8. Login to a HealthShare system
9. Search for a specific patient
10. Select that patient to view the patient record
11. Review the summary chart for the patient
12. View the patient’s allergy list
13. View the patient’s medication list
14. Logout of the session
While executing the tests, we would like you to evaluate the system on how satisfied are you with the
user interface. Were you satisfied with the execution of the task, how the information was presented
and how you would be able to use this data in a real-world setting?
The system will be recording metrics that time tasks and allow us to score the system on effectiveness
(could you complete the task) and efficiency (how much time did each task take).
The feedback form provided will ask you to give an evaluation for each of the tasks.
Additional feedback is welcome on the evaluation form as well.
Page 21 of 27
Test and Task Outline
Now please execute the following tasks. The feedback form is available on the next screen.
VERY IMPORTANT: During the LOGIN task, you will receive a popup providing you with a session ID.
Please record this number on your feedback form.
Start: Access System Please access the HealthShare system at the following link:
http://54.243.145.157:57772/csp/healthshare/hsaccess/HS.UI.PatientSearch.cls
Task 1: Login to the system
Login ID: doctor
Password: doctor
Click Login
Following login, you will be given a session ID. Please record this ID on your feedback form!
Task 2: Search for the Patient
Search for a patient named John Smith
Click Search at the bottom of the search form.
Task 3: Select Patient
Select John Smith to view the patient’s record.
Task 4: View Summary Record for the Patient
You are presented with the summary of the most recent data for John Smith. Please view this screen
and ensure you can locate the allergies and medications.
Task 5: View Allergies for the Patient
Select the Allergies tab to display a chart of the known allergies for the patient.
Is the Allergies tab easy to access and understand?
Is it easy to identify which allergies are drug/medication related?
Task 6: View Medications for the Patient
Select the Medications tab to display a chart of the known allergies for the patient.
Is the Medication tab easy to access and understand?
Task 7: Logout
From the top menu bar, click Logout to end the session.
Page 22 of 27
Feedback Form
You may fill this feedback form out manually and then return to [email protected] (An
electronically filled or a scanned copy is acceptable)
Test Date:
Participant Details
Clinical Role (e.g. Nurse, Doctor, Specialist, etc):
Years in Practice:
Do you have any prior experience with HealthShare?
Where are you located?
Which internet browser are you using (e.g. Firefox, IE, Chrome, Safari)?
What operating system are you using (e.g. Windows, Mac)?
Identification
Please record the Session ID that displayed on the screen following login:
Task Feedback
Please rate each of the tasks for effectiveness, efficiency and satisfaction, with 1 being least and 10
being greatest.
Task 1: Login
1. Were you able to login without difficulty?
2. Did you receive any errors while executing this task (if so, please specify)?
3. How satisfied were you with the system login process?
(Not satisfied) (Satisfied) (Very Satisfied)
1 2 3 4 5 6 7 8 9 10
4. Please provide any additional feedback:
Page 23 of 27
Task 2: Search for Patient (John Smith)
1. Were you able to search for the patient without difficulty?
2. Did you receive any errors while executing this task (if so, please specify)?
3. How satisfied were you with the patient search interface?
(Not satisfied) (Satisfied) (Very Satisfied)
1 2 3 4 5 6 7 8 9 10
4. Please provide any additional feedback:
Task 3: Select Patient (John Smith)
1. Were you able to select for the patient without difficulty?
2. Did you receive any errors while executing this task (if so, please specify)?
3. How satisfied were you with the patient selection interface?
(Not satisfied) (Satisfied) (Very Satisfied)
1 2 3 4 5 6 7 8 9 10
4. Please provide any additional feedback:
Task 4: View Patient Summary
1. Were you able to view the patient summary without difficulty?
2. Did you receive any errors while executing this task (if so, please specify)?
Page 24 of 27
3. How satisfied were you with the patient summary interface?
(Not satisfied) (Satisfied) (Very Satisfied)
1 2 3 4 5 6 7 8 9 10
4. Please provide any additional feedback:
Task 5: Access Allergy List
1. Were you able to access the Allergy List without difficulty?
2. Were you able to clearly identify which allergies pertained to medication/drug reactions?
3. Did you receive any errors while executing this task (if so, please specify)?
4. How satisfied were you with the allergy list access?
(Not satisfied) (Satisfied) (Very Satisfied)
1 2 3 4 5 6 7 8 9 10
5. Please provide any additional feedback:
Task 6: Access Medication List
1. Were you able to access the Medication List without difficulty?
2. Did you receive any errors while executing this task (if so, please specify)?
3. How effective was the medication list access?
(Not effective) (Effective) (Very Effective)
1 2 3 4 5 6 7 8 9 10
4. Please provide any additional feedback:
Page 25 of 27
Task 7: Logout
1. Were you able to access the Logout link without difficulty?
2. Were you successfully logged out and no longer had access to the patient record?
3. Did you receive any errors while executing this task (if so, please specify)?
4. How satisfied were you with the logout process?
(Not satisfied) (Satisfied) (Very Satisfied)
1 2 3 4 5 6 7 8 9 10
5. Please provide any additional feedback:
Appendix B – Complete Numeric Results of HealthShare Usability Test
Login Patient Search Patient Select View Summary View Allergies View Meds
Logout Anonymous
ID Clinical Role Yrs in
Practice Experience
with EHRUT
Location Browser OS Satisfaction (Login)
Time (Search)
Deviations (Search)
Satisfaction (Search)
Time (Select)
Deviations (Select)
Satisfaction (Select)
Time (Summary)
Deviations (Summary)
Satisfaction (Summary)
Time (Allergies)
Deviations (Allergies)
Satisfaction (Allergies)
Time (Meds)
Deviations (Meds)
Satisfaction (Meds)
Satisfaction (Logout)
Total Time
1
Physician
11
1
United States
Safari
MacOS
10
11.607
0
10
3.79
0
9
47.861
0
7
44.491
0
8
47.861
0
3
9
155.61
3 Nurse 22 0 Australia Firefox Windows 10 11.326 0 10 9.344 2 10 11.419 0 10 21.637 0 10 48.267 2 8 10 101.993
5
Pharmacist
10
3
United States
Safari
MacOS
10
9.641
0
10
2.59
0
9
11.716
0
10
12.433
0
10
17.347
0
9
10
53.727
15 Physician 23 2 Brazil Safari MacOS 8 15.99 0 8 4.087 0 9 41.855 0 9 80.293 0 9 25.085 0 9 10 167.31
17
Nurse
25
0
United Kingdom
IE
Windows
9
18.798
0
10
14.087
0
8
19.204
0
10
3.089
0
10
13.93
0
10
10
69.108
18 Lab Tech 15 0 Australia IE Windows 5 0 7 12.386 0 5 44.335 0 5 61.76 0 6 118.373 0 6 8 236.854
19
Midwife, Nurse
12
0
Australia
IE
Windows
10
13.167
0
8
9.64
0
8
6.849
0
10
14.976
0
10
9.796
0
10
8
54.428
20 Physician 30 2 Australia Chrome MacOS 10 9.89 0 10 5.46 0 10 38.61 0 10 12.605 0 10 23.993 0 10 10 90.558
21
Nurse
16
1
United States
Chrome
Windows
10
11.232
0
10
5.616
0
10
26.598
0
10
36.535
0
10
32.308
0
10
10
112.289
26 Physician 18 0 Australia Firefox MacOS 9 14.633 0 7 9.266 0 8 30.155 0 8 52.088 0 6 99.294 0 7 9 205.436
27
Physician
29
2
United States
Chrome
MacOS
9
4.056
0
9
4.805
0
9
3.245
0
9
24.367
0
8
30.014
0
8
9
66.487
28 Nurse 15 0 Germany Safari MacOS 10 11.435 0 10 8.44 0 10 12.636 0 10 8.175 0 10 56.877 0 10 10 97.563
30
Radiation Therapist
6
0
Australia
IE
Windows
10
8.658
0
8
10.67
0
9
12.293
0
8
30.17
0
10
59.171
0
10
10
120.962
31 Pharmacist 6 0 Australia IE Windows 5 10.436 0 7 26.099 0 7 0 6 1 7 1 6 8 36.535
34
Physician
0
1
United States
Safari
MacOS
10
17.254
0
10
12.012
0
10
21.107
0
6
11.746
0
10
87.158
8
10
10
149.277
35
Nurse
16
0
United States
IE
Windows
9
15.288
0
8
18.252
1
8
37.3
0
8
39.031
0
7
42.447
0
8
9
152.318
36 Nurse 29 1 Australia IE Windows 9 10.312 0 10 15.631 0 10 17.285 0 10 35.583 0 10 3.525 1 10 10 82.336
38
Physician
12
0
United Arab Emirites
Firefox
MacOS
10
9.095
0
10
3.588
0
9
17.846
0
7
5.101
0
10
151.196
0
7
10
186.826
40
Pharmacist
8
0
United Arab Emirites
IE
Windows
10
16.973
0
10
21.746
0
10
22.683
0
10
9.329
0
10
21.247
1
10
10
91.978
44 Physician 6 0 Australia Firefox Windows 8 19.329 0 7 24.102 0 5 156.843 0 5 143.567 0 6 203.985 0 6 7 547.826
Goal Average
Experience
15.45
0.65
Goal
7.00
20.00
0.50
7.00
30.00
7.00
60.00
7.00
60.00
7.00
90.00
7.00
7.00
Averages Average 9.05
12.59 0.00 8.95 11.08
0.15 8.65 30.52 0.00 8.40 34.05
0.05 8.85 57.47 0.65
8.35 9.35 138.97
Std Dev Total
Participants
20.00
Std Dev
1.54
3.95
0.00
1.28
7.03
0.49
1.53
33.33
0.00
1.82
33.74
0.22
1.60
53.00
1.81
1.98
0.93
110.44
Min Min 5.00
4.06 0.00 7.00 2.59 0.00 5.00 3.25 0.00 5.00 3.09 0.00
6.00 3.53 0.00
3.00 7.00 36.54
Max
Max
10.00
19.33
0.00
10.00
26.10
2.00
10.00
156.84
0.00
10.00
143.57
1.00
10.00
203.99
8.00
10.00
10.00
547.83
Page 26 of 27
Page 27 of 27