Role Prediction Using Electronic Medical Record System Audits

Wen Zhang1, Carl Gunter3, David Liebovitz4, Jian Tian1 , Bradley Malin1,2

1Dept. of Electrical Engineering & Computer Science, Vanderbilt University2Dept. of Biomedical Informatics, Vanderbilt University

3Dept. of Computer Science, University of Illinois at Urbana Champaign4Dept. of Medicine, Northwestern University

1

Misuse of EMR Systems is Real

• Medical center employees misuse medical record systems to breach privacy

When Where Who

2007 Palisades Medical Center George Clooney

2011 UCLA Various Celebrities

2

• HIPAA Security Rule Access to EMRs should be limited

• The problem is not limited to celebrity snooping

• But how?

Challenges to Security in EMRs

• Basic security principle:– Least privilege– Separation of duty

• Access control technologies have been around since the 1970’s

• Information systems often provide role-based access control (RBAC) capability[1]

– Privileges mapped roles

– Users mapped to privileges

• Roles are hard to define, so EMR systems often provide broad access rights

3[1] R.Sandhu, E.Coyne, H.Feinstein and C.Youman. IEEE computer. 1996.

In “Rare” Cases – Break the Glass

• A user may not sufficient access rights to perform job

• This model allows users to temporarily escalate privilege

• Access is logged and reviewed by administrator

• May require user to specify “reason” for access

4

Rare Cases?• Central Norway Health Region enabled break the glass

• 53,000 of 99,000 patients (54.5%) broken glass

• 5,000 of 12,000 users (42.7%) broke the glass

• Over 295,000 logged breakage events in one month

Role Users Invoked Glass Breaks in Past Month

Nurse 5633 36%

Doctor 2927 52%

Health Secretary 1876 52%

Physiotherapist 382 56%

Psychologist 194 58%

5[3] L. Røstad and N. Øystein. Proceedings of the 2nd International Conference on Availability, Reliability and Security (ARES)

Idea! Refine Access ControlBased on Behavior

• Experience-based Access Management (EBAM)

• Combine static knowledge (RBAC)

with actual actions (access logs) and organizational knowledge for feedback control

6

RBAC

EMR Access Logs

Medical Center Knowledge

Experience-Based Access Management [2]

[2] C.Gunter, D.Liebovitz, B.Malin. IEEE Security and Privacy Magazine. 2011.

• Use audit logs to predict if a user is associated with a role

• Goals:– Determine if expert-defined job titles are reasonable– Provide administrators with a better idea of how to refine roles

The Role Prediction Problem for EBAM

Doctor

NurseRole

Classifier

Biller

….

7

Access Reason

Medical Service

Location of Patient

User Patient Time Service User Position (Role) Reason Locationu1 p1 8/4/10 OBSTETRICS NMH Physician Office - CPOE Attending Phys/Prov Ward Au2 p2 12/14/10 OBSTETRICS NMH Physician - CPOE Patient Care Ward Au23 p3 12/14/10 PEDIATRICS Unit Secretary 2 Unit Secretary Orders Ward B

Evaluation with Cerner EMR of Northwestern Memorial Hospital

• Represent users as <Service, Reason, Location> vectors

• Statistics

8

Users Roles Reasons Services Locations

8095 140 143 43 58

• Example audit logs

• To assist in role management, we worked with organization experts to build a hierarchy (specialized to Northwestern)

• Optimization Tradeoff:• Goal 1: Accuracy (should increase as we step up in hierarchy)• Goal 2: Separation of Duty (will increase as we step down)

Leveraging Role Hierarchies

Employee

DoctorSpecific Clinician

Dietitian

Junior Dietitian

Senior Dietitian

Physician Nurse

…

… …

… … …

General (62 roles)

Conceptual (5 roles)

Specific (140 roles)

9

Basis of a “Role-Up” Algorithm

• General idea: Audit roles at different levels of the hierarchy

1. Score each role in conceptual position & general position

2. Select role with the highest score & generalize its children

3. Repeat 1 & 2 until a threshold score is reached

10

• Allow administrators to balance between the prediction accuracy and separation of duties (number of roles)

Balanced Scoring Function

• R measures the extent to which specificity could be kept by the node

• A measures the extent to which predictablity could be achieved by the node

11

Employee

DoctorSpecific Clinician

Dietary

Junior Dietician

Senior Dietician

Physician Nurse

Nurse 1 Nurse 2Physician

2Physician

1

0.4760.224 0.410

0.453 0.0441

α = 0.5, Threshold = 0.4

12

Employee

DoctorSpecific Clinician

Dietary

Junior Dietician

Senior Dietician

Physician Nurse

Nurse 1 Nurse 2

0.224 0.410

0.4530.0441

13

α = 0.5, Threshold = 0.4

Employee

DoctorSpecific Clinician

DietaryNurse

Nurse 1 Nurse 2

After one iteration, the role set is{Doctor, Nurse 1, Nurse 2, Dietary}

14

α = 0.5, Threshold = 0.4

Training & Testing at the Same Level of the Role Hierarchy

Employee

Specific Clinician

Nurse

Nurse 1

15

Conceptual

General

Specific

82.38%

52.45%

51.34%

AccuracyLevel

Distribution of Accuracy Over the Role Hierarchy

16

Rank Role Accuracy Users1 (tie) AP-Technologist 100% 541 (tie) ED Assistant 100% 261 (tie) ED NMH Physician-CPOE 100% 43

1 (tie) NMH Resident/Fellow ID Clinic-CPOE 100% 10

1 (tie) Patient Care Staff Nurse – Lactation 100% 14

17

Most Predictable Roles

Least Predictable Roles

Rank Role Accuracy Users140 Patient Care Staff Nurse 7.6% 1554139 Rehab OT 14.3% 28138 Transfer 20.0% 20137 View Only PC 3 21.4% 14136 Patient Care Staff Nurse (Pilot) 22.1% 217

18

Number of Users in the Role Can Influence Accuracy

19

Case Study: Most Likely Mispredictions for Patient Care Staff Nurse

Predicted Role PredictionPatient Care Staff Nurse - Lactation 19.6%View Only PC 1 14.3%Radiology – Nurse 14.0%Patient Care Staff Nurse (Pilot) 10.4%SN-RN/Customer Service 5.8%

20

Original Role Predicted Role ProbabilityRehab OT Rehab PT 85.7%Patient Care Staff Nurse - Agency

Patient Care Staff Nurse - Lactation

75.0%

Rehab PT Rehab OT 60.0%

View Only PC 3Patient Care Staff Nurse - Lactation

50.0%

Medical Records - Scanner

Medical Records 47.4%

21

Most Likely Mispredictions

Parameter Bias Trades Between Accuracy and Separation of Duty

• Biased toward Accuracy:• number of roles is small (27)• accuracy is highest (63%)

22

0.1 … 0.8 0.9Number of RolesRecommended

27 … 60 64

Accuracy ofRole Predictions

63.3% … 51.8% 51.3%

• Biased toward Specificity:• number of roles is high (60)• accuracy is lower (52%)

Conclusion and Future Plans

23

• EHR audit logs can be analyzed to determine if the users’ behaviors are consistent with their designated job titles

• Role hierarchies enable automatic discovery of appropriate levels of role management

• Plan to expand Role-“up” to allow for Role-“down” and Role-“over”

• Need to evaluate Role-up with real hospital administrators, to assess its usability and acceptance of results

Acknowledgements

• National Science Foundation– CCF-024422– CNS-0964063

• National Library of Medicine– R01-LM010207

• Office of the National Coordinator for HIT– SHARPS (sharps.org)

24

Questions?

wen.zhang.1@vanderbilt.edu

25