Web Security BreachLessons Learned

Ray FordAssociate Vice President for Information Technology

Gordy PaceDirector of Multimedia Applications

Copyright 2002 - Ray Ford. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

OUTLINE

• The Event• Context - UM-M Web Environment• Web/Clinical Assumptions/Safeguards• Aftermath of the Event• The Investigation and its Results• Long-term Impact and Policies• Summary of Lessons Learned

THE EVENT• 10:30am MST, November 5, 2001: Message from

from news media to University Relations to UM attorney to CIO -- start at UM-M home page, search with “Choices” as keyword, look at the results

• Hits include references to text files plus MS Word and Excel docs marked “confidential” and/or with names suggesting child/family clinical records

• Media: Report scheduled for the evening news -- would we care to comment? (“Not right this minute -- we’ll get back to you …”)

• Then things started happening, which continue to this day …

Context University of Montana-Missoula

• UM: medium sized, PhD/research institution with campuses in Missoula, Butte, Helena, Dillon

• Size: about 16,000 total students; largest campus is Missoula (UM-M) with about 13,000 students

• Programs: range from AA, BA/BS, MS, to PhD, with world class programs in natural resource management, environmental studies, and wildlife

• UM research effort in FY02: over $50M in external funding, including significant amounts in clinical areas

UM-M Web Environment

• Original 1990’s Vision: Aggressive central Web presence supported by ample, highly trained central support staff

• Late 1990’s (Budget-constrained) Reality: Minimal central support staff -- core Web services are centralized and supported, but content creation is widely distributed with people funded by “the units” building and maintaining most site content

• Sever proliferation: Many units (particularly research groups) also run their own Web servers in the “umt.edu” domain.

Distributed Web EnvironmentImplicit Assumptions

• That all servers (a distributed mix of Unix and Wintel platforms) are operated securely

• That all system administrators (a distributed set of people with diverse backgrounds) are knowledgeable and diligent regarding security

• That all contributors (an even more diverse set of people) are knowledgeable and diligent regarding security and acceptable use of accounts

UM Clinical EnvironmentExplicit Safeguards

“Non-filtered” clinical records are to be maintained ONLY on removable media which are handled in a specified secure fashion

• Computers are NOT to be attached to the network when the removable media are loaded

• Non-filtered clinical records are NEVER to be copied to personal accounts, file servers, etc.

Thus clinical records CANNOT, if handling guidelines are followed, end up on a Web server

But …

Return to The Event Immediate Response

• CIO and Computer Center Director agree from the beginning to limit access to all pertinent records to “need to know” basis

• CC Director and 1 staff member “secure” the actual data plus the Web server logs to CDs, delete the information, then hand over CDs to UM attorney

• UM forms a response team, names single spokesman (the CIO), and launches a follow-up investigation

Media Attention

• TV: makes local but not national broadcasts• Newspaper: front page in Missoulian, LA Times,

Minneapolis Star-Tribune; syndication to …• Other: syndicated story picked up by dozens of

other print and on-line media over next several months (including both print and on-line Chronicle of Higher Education)

• On-going: references continue, but with interesting “evolution” of details (e.g., UM mapped to University of Minnesota)

The Investigation - Basics

• Goal: Answers to the familiar who, what, when, where, why, how, … Eventually to be summarized in a public report

• Philosophy: Each file, and even the file name itself, could contain sensitive information, so access to the data and the logs needs to be carefully restricted

• Approach: Assign specific investigation responsibilities, and let the “need to know” from those responsibilities determine information access

The Investigation - Responsibilities and Information Access

• CIO: Primary spokesman, needs access to summary information only

• CC Director: Investigation of logs, needs access to logs only

• Clinical reps: Investigation of file contents, need access to actual files

• University attorney: Coordination of multi-institution concerns, needs access to summary information only

Investigation Results - Summary• Who/What/Why? A collection of over 6000

personal files was copied by a valid user into an area normally used for Web pages associated with a particular program. The user was authorized to do Web postings for the program, but uploaded the whole contents of a portable PC’s hard disk, apparently by mistake.

• How did the files become accessible? The files were not “linked to” by any explicit Web links, but were “discovered” during routine contents indexing.

Investigation Results - Summary

• How were they accessed by Web users? Simple searches “hit” on auto-deduced keyword terms, producing a link that made it possible to view/download the file

• Who actually accessed what confidential data? Good question, but determining the answer requires some detective work …

Investigation Results - Summary• How did the person happen to have confidential files

in possession to begin with? The files were part of non-UM employment with a non-UM clinic, i.e., there was NO connection between the files and UM

• Was there a violation of UM clinical protocol? Of some other clinical protocol? Given that these were not UM/clinical files, there could be no UM violation; whether there was a violation of third party protocol is up to the third party to determine

• Was there any UM violation? Yes, inappropriate use of a Web account for archiving personal data.

Access Logs and Log Analysis

Information accessible only via “search” meant that comprehensive Web access logs were available, showing for each access: who (IP address), how (nature of the query), what (files), and when

• What confidential information was accessed? meant matching Clinical investigation results (what was confidential) with log investigation results (what was accessed)

• Who did the accesses (and why)? involved starting with IP addresses of the accesser, and looking carefully at the whole log

Who Accessed What?

• We identified the set of all accesses for confidential data.

• Several accesses were seemingly “random” with users who did no follow-up -- a file was listed among the hits, but was not subsequent accessed. These could be reasonably discounted.

• Other random accesses were followed by specific follow-up searches -- follow-up required.

• Some accesses started with very specific keywords that suggested non-random access -- hmmm …

Who Accessed What and Why?• The two follow-up accessers were (unknown)

ISP users whose accesses were followed closely in time by the two non-random accesses -- both non-random accessers were from IP addresses assigned to news media organizations

• This circumstantial evidence points to two separate cases of random access, curiosity-based follow-up, then news media “tip”

• News media access pattern was one or more “confirmation access”, then a greater or lesser set of follow-up accesses.

Who Accessed What and Why?• Starting with complete set of accesses

- delete random access with no follow-up- delete news media access -- it is reasonable to assume that their accesses were for investigative use only- what was left were the accesses of the two presumed “tipsters” -- besides notifying the press there is no way to determine what else they might have done with the information

Bottom Line: not “no access”, but not as bad as feared

Investigation Summary

• Careful log analysis: Can bound exposure (show what files were/weren’t accessed) and show IP of accesser(s), but cannot directly tell who did what and why -- it is thus helpful but not conclusive (and works only for search accesses)

• Thank heavens for a free press: Though their stories may have made UM look bad, their rapid investigative follow-up alerted UM to the problem before there was widespread “public” access

Event - Long Term Impact

• Definition/Implementation of Policy: Requirement for training and certification for Web accounts

• Modest infusion of new resources: Approval for new Web director position to develop better content publishing protocols and coordinate UM efforts (but not major staff-up to centralize the efforts)

• Increased interest in centralization: A few servers abandoned in favor of central server (but no mandate requiring centralization)

• Consideration of software “solutions”: Within limits implied by resources and distribution …

Policy For Both Primary and Secondary Web Servers

• “… the person designated as responsible for that program must complete training on the rights and responsibilities of Web content contributors” and “… the user must complete re-certification training every twelve months. ”

• “Web postings shall be the responsibility of a single person. Material may be developed and tested by any number of people off-line, but uploading such material to a ‘live’ Web server shall be the responsibility of a single individual. Under NO circumstances should a program’s user account(s) be shared by multiple users.”

• “Accounts intended for posting information will be authorized only for approved University programs. Requests for such accounts will identify the program and the person designated to post information about that program.”

Lessons Learned The Easy Ones

• Importance of recurring user certification and training, to be offered on-line -- in development -- for example see http://fwp.state.mt.us/bearid

• Importance of care in post mortem information handling -- limiting information access may seem initially like “stone walling”, but is crucial in avoiding further proliferation of sensitive information

Lessons Learned The Not So Easy Ones

• Importance of open and honest dealings with the press -- the press is not your friend but neither is it your enemy (unless you make it so)

• The public/press has little interest in or patience with technicalities, but it is very hard to know what aspects of Web things are technical vs. well-known -- do your best and good luck

Example: if it occurs within your domain, it is on your Web site and you are responsible (no matter who actually operates the server in question)

Lessons LearnedThe Not So Easy Ones

• There is inherent conflict between asking a diverse collection of people to contribute information in a timely manner vs. establishing a system that guarantees that this type of mistake does not occur

• Being prepared to deal with “posting mistakes” when they occur is probably an inherent cost of business in a typical university environment -- be prepared.

Questions and/or Comments?

Thanks for your attention and interest(and I hope we never have the need to talk about this particular topic again)