Evaluation of Hospital Drills: Using the Tool

Amy Kaji, MD, MPH

November 16th, 2005

Acute Care College Medical Student Seminar

Hospital Disaster Drills

Why are drills necessary? Hospitals will be called upon to provide

care to the ill, injured, exposed, and concerned

Joint Commission on Accreditation of Healthcare Organizations (JCAHO) Requirement

May help train employees Tests aspects of hospital response

Hospital Disaster Drills

Types Computer simulation Tabletop exercises Operationalized drills involving specific

victim scenarios Evaluations

Can help maximize the value of the drill Based on accurate observation Benefit of standardization

Johns Hopkins University Evidence-based Practice Center (JHU EPC) Developed an evaluation tool for hospital drills Assess impact of drill as hospital response

progresses and develops Presents topics for evaluation in a systematic

manner Identify strengths and weaknesses in hospital

disaster preparedness Promotes targeted efforts to strengthen

preparedness NOT intended to be used to determine whether a

hospital passes or fails in its planning endeavors

The Evaluation Modules

Developed by a multi-disciplinary team of experts at JHU EPC

Based upon systematic review of published reports on hospital disaster drills and literature relevant to bioterrorism preparedness

Had “expert input” from federal, state, and local agencies

Piloted the modules in two multi-hospital regional disaster drills in the summer of 2003

Guiding Principles in Developing Evaluation Modules Need for observing multiple hospital zones

Evaluation of a disaster drill requires an understanding of drill activities in all areas of the hospital

Four zones identified include: Incident Command Decontamination Triage Treatment

Addendum if Biological or Radiological Scenario Decontamination Zone Module needed for radiation

& chemical drills; not recommended in biological drill

Guiding Principles in Developing the Evaluation Modules Need for documentation of time points

Recording time points of drill activities is a widely accepted method of evaluation

Labor-intensive if excessive number of time points and may detract from overall evaluation

Limited, specific time points thus predetermined

Guiding Principles in Developing the Evaluation Modules Need for documenting clinical care

outcomes Track the volume of victims in each zone

and adequacy of provisions made for them, including space, staff, supplies, etc.

Modules are not intended to collect individual victim level data

Modules monitor the zone and outcome for the zone as a whole, not for each victim

Guiding Principles in Developing the Evaluation Modules Need for debriefing (after-action review)

Obtain feedback from participants, including organizers, staff, and victims

Allows for discussion of issues that span more than one zone

Evaluate and integrate cross-zone issues at a post-drill debriefing session

Guiding Principles in Developing the Evaluation Modules Need for ease and flexibility of care

Designed to be readily understood, easy to use, and applicable to many different drill scenarios

Items on form are arranged by subject, and this is the same across all modules to facilitate analysis

Decision to include a module is dependent upon the specific scenario and drill

Guiding Principles in Developing the Evaluation Modules Need for safety and security

Consider planned drill activities; i.e., use of unfamiliar equipment

Protect actual patients on the premises Safety of drill victims and healthcare

workers Contingency plan to stop the drill in case

of an actual emergency Consider designating a safety officer to

monitor the drill and its participants

Recommended Use of Evaluation Modules and Addenda ( X = USE IT!)Type of Drill Modules Addenda

Fire IC Triage Treatment DECON Biological RAD

Explosion X X X

Natural Disaster (Earthquake, flood, drought)

X X X

Internal

Hospital

Failure

X X X

Biological X X X X

Chemical X X X X

Radiation X X X X X

Internal Structure of Evaluation Modules Zone forms for Incident Command, Decontamination, Triage, and

Treatment have same structure and subject headers: Time points Zone description Personnel Zone operations Communications Information flow Security Victim decontamination and tracking Victim flow Personal protective equipment (PPE) and Safety Equipment and Supplies Rotation of Staff Zone disruption

Internal Structure of Evaluation Modules Numbering of questions on the forms

“C” denotes questions that are common to several zones

Questions that are zone-specific are identified by a unique letter code

120 questions per module Coding on the forms and use of comment boxes

Yes = Y; No = N; Unclear = U; Not applicable = NA NA is not a substitute for missing information,

negative information, or to avoid writing a comment

Color coding for the modules and addenda Assists in organizing and tracking modules and

addenda before and after a drill

Common Sample Questions from an Evaluation Module (Time Points) C1. Time the drill began: ___AM/PM/U C2. Time the hospital disaster plan was initiated

in this zone: ___AM/PM/U/Not initiated C3. Time this zone was ready to accept victims:

___AM/PM/U C4. Time when this zone was notified that

incident command was operational:___AM/PM/U/Not notified

C5. Time the drill ended in this zone:___AM/PM/U

Common Sample Questions from an Evaluation Module (Zone Description) C8. Was the boundary for this zone defined?

Y/N/U C9. If this zone had a defined boundary, how was

it defined? (Check all that apply) Barricade Security personnel Sign Tape Vehicle Wall (permanent or temporary) No boundary Other (specify): __________________

Common Sample Questions from an Evaluation Module (Personnel)

C11. Did someone take charge of this zone? Y / N / U    C12. If someone took charge of this zone, how many

minutes after the drill activities in this zone began did this person take charge? (Check one)   O < 10 min O 10 - 29 min O 30 - 59 min  O 1 - 2 hrs O > 2 hrs O NA   

C13. If someone took charge of this zone, was it the officially designated person? Y / N / U / NA

Common Sample Question from an Evaluation Module (Personnel) C14. How was the person in charge of the zone

identified? (Check all that apply) a. [  ] Arm band b. [  ] Hat c. [  ] Name tag  d. [  ] Verbal statement e. [  ] Vest f.  [  ] Not identified  g. [  ] Other physical identification (specify):

__________________________________________

Common Sample Questions from Evaluation Module (Zone Operations) C29. Was the space allocated for the zone adequate?

Y/N/U C30. If not enough space for the zone, where did zone

activities overflow to? (Check all that apply) Adequate space allotted Conference room Hallways Outside hospital Treatment/victim care areas Waiting rooms No overflow NA Other (specify): ______________________________

C31. Was this zone used for the same functions during non-drill operations? Y/N/U

Common Sample Questions from Evaluation Module (Zone Operations) C33. Did clinical staff interact directly with

families of victims? Y/N/U/NA C34. Were families of victims referred to specially

designated staff? Y/N/U/NA C35. How was victims’ privacy ensured? (Check

all that apply) Curtains Individual areas Privacy screens Not ensured Other (specify): ____________________________

Common Sample Questions from Evaluation Module (Communications) Communication device (s): If device not present , circle “N”

in column “a” and go to the next line. A. Was device present? Y/N/U B. If present, # available: C. If present, was it used in drill? Y/N/U D. Comments (note problems)

C36. 2 way radio/phones C37. Direct line C42. Numeric paging C44. Text paging C45. E-mail and internet access C47. Intercom C49. Runner

Common Sample Questions from Evaluation Module (Information Flow) C54. How was this zone notified of the event? (Check all that

apply) FAX Runner Telephone Not notified Other (Specify):__________

C55. Who notified this zone of the event? Drill organizer Incident command center Media Other hospital staff Outside source Victims arriving Not notified Other (specify):____________

Common Sample Questions from Evaluation Module (Security) C60. Were security present in this zone? Y/N/U C62. If security personnel were present, what

type of security? (Check all that apply and provide approximate numbers) FBI ___ Hospital Security ___ Local Police ___ State Police ___ NA Other (specify): ______________

Common Sample Questions from Evaluation Module (Victim Documentation and Tracking)

C74. Were all incoming victims registered and given a unique identification or medical record number? (check one) Yes, before entering this zone Yes, on entering this zone No, not while in this zone Unclear

C76. Was a central list of victims generated for this zone? Y/N/U

C77. Were the triage markers on the victims clearly visible? Y/N/U

Common Sample Questions from Evaluation Module (Victim Flow) C81. Did a bottleneck develop in this zone? Y/N/U C84. Were the paths leading to the next zone marked?

Y/N/U C86. Were the lowest acuity victims directed by staff to an

area separate from higher acuity victims? Y/N/U C87. What proportion of victims had treatment delayed

because of zone staffing shortage? (Check one) None Less than half At least half (but not all) All Unclear

Common Sample Questions from Evaluation Module (Personal Protective Equipment and Safety)

If needed, were these items for standard precautions available for the healthcare workers? Y/N/U

Used by staff? Y/N/U Adequate supply? Y/N/U

Eye protection Waterproof gowns Isolation gowns Gloves

Common Sample Questions from Evaluation Module (Equipment and supplies)

TX15. Were medications needed for the treatment of victims available within the hospital? Y/N/U/NA

TX16. Were medications requested from and outside source? Y/N/U

Were needed medical supplies available?TX19. Bandages Y/N/U/NATX20. Basic airway equipmentY/N/U/NATX22. Blood drawing supplies Y/N/U/NATX23. Burn Packs Y/N/U/NATX24. Cleaning supplies for contaminated equipmentY/N/U/NATX25. Crash carts Y/N/U/NATX26. Intravenous fluids Y/N/U/NA

Common Sample Questions from Evaluation Module (Rotation of staff) C98. Was there a staff rotation /shift change? Y/N/U C99. If there was a staff rotation, did the officially designated

person in charge of the zone change? Y/N/U/NA C101. What method of shift changing was used? (Check one)

Group shift change Staggered shift change NA Other (specify): _____________

C102. How were incoming staff updated? (Check all that apply) Group briefing Individual briefing Written notes Not updated NA Other (Specify):_____________

Common Sample Questions from Evaluation Module (Zone disruption) C103. Was there a plan to relocate this zone if

necessary? Y/N/U C104. Did this zone close at any time during the

drill? Y/N/U If no, disregard the remainder of this section.

STOP. This zone module is complete… C105. If the zone closed during the drill, what was

the reason for closing? (Check all that apply) Contamination Other safety concerns Space Other (Specify): ________________

Description of Modules and Objectives Pre-drill Module Incident Command Center Zone Module Decontamination Zone Module Triage Zone Module Treatment Zone Module Group Debriefing Module

Pre-drill Module

Should be used in all disaster drills during the planning stages

Form is designed to collect the following: Goals and objectives for the scope of evaluation Background information Information on areas that hospital wishes to

evaluate Resources required If multi-hospital or regional drill, each site must

work closely with overall coordinators

Incident Command Center Zone Module Designed to reliably collect information about

operations of the incident command system (ICS) Should be used in all disaster drills when

evaluating the ICS Form is designed to assess the following:

Command structure in the zone Adequacy of staffing in the ICS Communication and information flow from hospital

areas to the ICS Communication with outside agencies Adequacy of security, safety provisions, and

physical space

Decontamination Zone Module

Designed to collect information re: functioning of decontamination area

Used if scenario involves radiation or chemical exposure and decontamination is needed

Form assesses the following: Command structure in the zone Communication and information flow in the zone Victim and staff safety in the zone Adequacy of staffing and physical space in the

zone Appropriateness of equipment and PPE Victim flow in the zone

Triage Zone Module

Designed to collect information re: functioning of treatment areas

Should be used whenever drill objectives include evaluation of patient care activities beyond triage

Appropriate for use in emergency department-based treatment areas or in other clinical areas

Triage Zone Module

Form designed to assess the following: Command structure in the zone Communication and information flow in the

zone Victim and staff safety in the zone Relation of physical characteristics of zone to

treatment activities Efficacy of treatment operations Adequacy of materials and supplies in the

zone Victim flow in the zone

Debriefing

Debriefing is integral part of drill process Main objective of debriefing is to identify

issues not captured by evaluation modules

Facilitators should create an open, non-judgmental atmosphere

Should occur in all drills to obtain feedback from participants and observers

Debriefing

Different approaches to debriefing exist One method: conduct a session with all

participants and observers present and ask a series of general questions about the drill

Another method: conduct a group debriefing session with the participants from that zone

Group Debriefing Module

Contains open-ended questions designed to facilitate discussion after completion of drill Questions may be added or deleted Designed to cover all issues, including incident

command structure, communications, security, decontamination, triage, and treatment

Documenting the debriefing A scribe should be assigned to record Videotaping / audiotaping debriefing may help

capture comments but should not hinder open exchange

Group Debriefing Module Questions

Did you feel you were notified of the disaster in a timely fashion?

Did the incident command center work effectively?

Did anyone receive incorrect information from the incident command center? If not correct, what specifics do you recall about incorrect information?

Was the information from the incident command center received by other zones in a timely way?

Group Debriefing Module Questions

Were there problems with information flow within the hospital?

Were memorandums of understanding with outside agencies (e.g., police) activated?

Did nurses and physicians respond quickly to the disaster call?

Was the zone set up when the first mock victim arrived?

Was security in place before the first mock victim arrived?

Group Debriefing Module Questions

Did people have a good understanding of their roles, as defined in the disaster plan?

Did the decontamination system work effectively?

Did you have any problems with the decontamination equipment? Functioning properly? Adequate number of units? Participants used correctly?

Were there delays in decontamination? If so, what triggered those delays?

Description of Addenda

Four addenda are part of the hospital disaster drill evaluation: Biological Incident Addendum Radiation Incident Addendum General observation and documentation addendum Victim tracking addendum.

Used to supplement the zone forms Example: for a radiation exposure drill, the

Radiation Incident Addendum is added to Incident Command Center, Decontamination, Triage, and Treatment Zone Modules

Biological Incident Addendum

Designed to collect additional information that address response to a biological incident

Should be added to end of each Incident Command, Triage, and Treatment Zone modules

Should be used in all drills that address a biological incident

Biological Incident Addendum

Form is designed to assess the following: Awareness that biological agent cause of

illness Whether appropriate personnel were contacted Whether health and safety needs of staff were

met Whether health and safety needs of existing

patients were met Whether health and safety needs of victims

were met Availability of special medications and

supplies

Biological Incident Addendum

BA11. Was the Center for Disease Control and Prevention (CDC) informed that a biological illness was present?

BA14. Was occupational health informed that a biological illness was present?

BA22. If the cause of the illness was not known prior to victim arrival, how long after the first victim arrived was the cause of the illness identified? (Check one) <1 hour 1-4 hours 5-8 hours >8 hours Never identified NA

Biological Incident Addendum BA24. What resources were used to make the diagnosis?

(Check all that apply) Consultation with an in-hospital expert Consultation with an expert from another hospital Consultation with an expert from local health

department Consultation with an expert from state health

department Consultation with the CDC History and physical exam by the treating physician Microbiological data Radiological data NA Other (specify): ______________

Biological Incident Addendum

BA26. Was isolation required for the suspected biological illness involved? Y/N/U (Isolation is required for smallpox, plague, viral hemorrhagic fever, certain pneumonias or rashes, and other symptoms suggestive of a contagious infectious outbreak)

BA27. If isolation was required, were victims transported into an isolation room? Y/N/U

BA28. If they were transported to an isolation room, was the room under negative pressure?

Y/N/U/NA

Biological Incident Addendum

BA29. Were there enough isolation rooms? BA30. If not enough isolation rooms, how were

victims isolated? (Check all that apply) Conversion of other rooms/area

(specify):___________ Existing isolation room in other area

(specify):__________ Overflow victims not isolated Victims with the same suspected biological illness

placed in the same isolation room NA Other

Radiation Incident Addendum

Designed to gather information in response to radiation-related incident

Should be added to end of each zone module

Should be used in all drills that address radiation exposure

Radiation Incident Addendum

Form is designed to assess the following: Awareness that radiation was cause of illness Whether appropriate monitoring personnel

were contacted Whether health and safety needs of staff were

met Whether health and safety needs of existing

patients were met Whether health and safety needs of victims

were met Availability of special supplies

General Observation and Documentation Addendum Designed for use by an additional

observer to document detailed activities in a single unit

Example: During a chemical drill, an additional

observer could be assigned to area where PPE is donned to document time required to dress, appropriateness of dress, etc.

Has a front page and a continuation page, which may be copied as necessary

General Observation and Documentation Addendum Instructions: Enter

time and describe any activities in this zone related to drill. Include: Response to patients Information received Any activities (such

as real emergencies) that may delay or prevent drill activities in this zone

Time Activity or Observation

Victim Tracking Addendum

Designed for use by an additional observer to track victims through the drill

May be used within one zone for a large group of victims, OR

Observer can follow victims across zones from beginning to end of drill disposition from medical perspective Useful if there is victim descriptions and

emergency medical procedures are well delineated Has front page & continuation page

Victim Tracking Addendum

Victim ID Time arriving in zone

Time exiting from zone

Disposition

Observers

Value and success of drill, depend on observers, who must be able to understand and record events

Observer selection is therefore critically important

Observers must be trained to use evaluation modules Documentation by observers provides information

for evaluation Record the type and number of victims, as well

as the care given or not given to victims May record personal views but should note it

as opinion

Role of Observer

Observers must not have any role other than that of evaluating the drill

Individual who normally works in zone may function as an observer, but during the drill, he/she may not assist with any activities

Must not respond to questions from drill participants about the drill

To qualify as observers, volunteers drawn from outside the hospital must have knowledge of hospital functions

Observers

Background knowledge required General knowledge of operations of zone Specific medical knowledge not required

Observers

Number of observers needed in each zone One observer should be present in each

zone continuously Additional observers may be needed to

observe numerous staff or victims Should use victim tracking or general

observation and documentation addendum

Observers

Number of observers needed in each zone If extensive time point data collection is

needed, an additional observer should be assigned to this task

Specific time points and reasons for collection should be outlined before drill

If clinical process or outcome data is needed, additional observers will be needed

Observers must have sufficient clinical knowledge to report on decision making

Observer Training

Training session Receive zone assignments Review relevant zone modules Achieve familiarity with content of

evaluation modules and zone configuration Explain all questions and response sets Address questions about forms Instruct how to be an effective observer

Final Observer Training Points

All observations are confidential Observer should evaluate drill without

obstructing flow of drill Observers may ask participants questions

to clarify observations Questions should be asked in an

unobtrusive manner Observers should refrain from asking

questions that may alter drill actions

Final Observer Training Points

Observers must not participate in drill activities

If questioned about a drill issue by a drill participant, the observer should state that they are evaluating and unable to answer

Each question on each module should have a response The response NA should be indicated

only when the question does not apply

References

Cosgrove SE, Jenckes MW, Kohri D, Hsu EB, Green G, Feuerstein CJ, Catlett CL, Robinson KA, Bass EB. Evaluation of Hospital Disaster Drills: A Module-Based Approach. Prepared by Johns Hopkins University Evidence-based Practice Center under Contract No. 290-02-0018. AHRQ Publication No. 04-0032. Rockville, MD: Agency for Healthcare Research and Quality. April 2004.