Facilitator/Evaluator Orientation/BriefingMARC Mass Fatality TTX

May 21, 2013

2

• Roles and Responsibilities• Evaluation Methodology Overview

– Exercise Evaluation Guide (EEG) Concept• Evaluation Process

– Evaluator Responsibilities• Assessment, Review, and Analysis of the

Exercise

Briefing Topics

Roles and Responsibilities

• Facilitators provide situation updates and moderate discussions. They also provide additional information or resolve questions as required.

• Evaluators use EEGs to measure and assess performance, capture unresolved issues, and analyze exercise results. Evaluators passively assess and document players’ performance against established emergency plans and exercise evaluation criteria, in accordance with HSEEP standards. Evaluators have a passive role in the exercise and only note the actions/decisions of players without interfering with exercise flow. For this exercise, evaluators may provide a limited coaching role of stimulating thought process.

4

Facilitators Evaluators• Chris Kates – UC• John Linstrom- MF• Justin Sorg – EM/Logs• Joan Harding– Health

and Medical• Dwight Graves PIO • Dana Hall VIC

• UC – Joe Lynch • MF – Chuck Smith• EM Logs – Ricky

Carroll• Health & Medical – Erin

Lynch• PIO Dwight Graves• VIC – Lisa Ladue

5

• Validate strengths, and identify improvement opportunities

• Validate plans, procedures, and protocols

• Improve processes and plans• Determine level of capability in regard to

the exercise’s core capabilities

Evaluation Goals

• The Homeland Security Exercise and Evaluation Program (HSEEP) Volume III provides evaluation and improvement planning policy.

• Evaluations should be objective-based to remain in line with:

–National Preparedness Goal–Core Capabilities List–Jurisdiction and agency plans

• HSEEP EEG templates provide streamlined process for standardized evaluation concept, not necessarily content.

EEG Concept

6

• Allows evaluators to review validation of plan elements– Plans used will depend on participating agencies

• Includes references to plan sections, tables, and charts– Allows evaluators to better prepare for exercise– Provides detail for what will be expected of players

• Remains focused on exercise objectives as they relate to the selected core capability

EEG Function

7

• Evaluation team should use EEGs in conjunction with full plans to prepare

• Use EEGs as guides for observation, but do not focus on completing them during the exercise

Use EEGs for data analysis after exercise and in preparation for the evaluator debriefing

Use measured results to make actionable improvement recommendations within the After Action Report (AAR)/ Improvement Plan (IP)

Use of EEGs

8

HSEEP Methodology Steps

Evaluation Tools

Timeline

Exercise objectives,

actual plans and

procedures

Evaluator log EEG analysis sheet

AAR data input form

Through Final Planning

Conference (FPC) planning

During exercise

Controller/ Evaluator

(C/E) Debriefing

Post exercise, before AAR

draft development

Use of Evaluation Tools

9

Evaluator Notes and Data• Evaluators should capture player

discussions, decisions, actions, and activity during exercise play. Use back of EEGs for notes

• The EEGs provided should serve as guides for documentation of exercise activity

10

EEG Analysis Form• Each evaluator will be

provided with EEGs that will give specific guidance on what data to collect during the exercise, how to record it, and how to analyze it

• These forms should be completed at the conclusion of the exercise, using data gathered during the exercise

11

EEG AAR Data Input Form• Each EEG will include a

second form to be completed following the C/E Debriefing

• This form will serve as an expansion to the notes captured in the EEG analysis form, including an area to provide recommendations.

• Evaluators should complete one AAR input form for each identified strength or area for improvement

12

After Action Report (AAR)/Improvement Plan (IP)

• Data from the EEGs and debriefing will be compiled into the AAR/IP.

IP

13

• Be on time• Wear appropriate attire• Make sure you have the tools necessary

to perform task• Understand the exercise concept and

scenario • Read the sit man

Facilitator Evaluator Responsibilities

14

• Be familiar with the assigned EEGs• Collect information• Make observations on player

discussions, knowledge, and intended impact of decisions

• Analyze actions and develop post exercise AAR/IP input

Evaluator Responsibilities

15

• Review appropriate plans, procedures, and protocols

• Participate in C/E Briefing• Review appropriate exercise materials to be

familiar with scenario, objectives, and core capabilities

• Review EEGs and other supporting materials for your area of responsibility

Pre-exercise Responsibilities

16

• Monitor and document player performance• Use EEGs to confirm objectives are being met• Focus on critical tasks, as specified in the

EEGs and plans being exercised• Do not prompt players or interfere with

exercise play• Limited coaching is allowed

Exercise Responsibilities

17

At the start of the exercise, evaluators should:• Position themselves to appropriately hear

discussions• Be prepared to move between break out groups

if necessary to follow various discussionsEvaluators should not:• Focus solely on individuals• Interfere with exercise discussion or action

Evaluator Guidelines

18

Evaluators must keep an accurate written record of important issues that take place during the exercise: Were participants knowledgeable of plans and

procedures to meet exercise objectives? Did discussions validate that operations would occur

in accordance with plans and procedures? Did participants validate that sufficient personnel are

equipped, trained, and assigned to perform tasks?

Observing the Exercise

19

• Evaluators should record events such as:–Discussed actions of players–Key decisions–Discussions that denote deviations from plans and procedures

–Player discussion of key movements and activities–Module responses on easel pad–Hotwash responses

Recording Important Events

20

Postexercise Analysis Responsibilities

• Participate in the Hot Wash, and take notes on findings identified by players

• Summarize your notes and prepare for the C/E Debriefing• Have your summary ready for the lead evaluator, including

bulleted strengths and areas for improvement from activity analysis

• Use data to determine whether the exercise capabilities and plans were effectively implemented– Identify observations as strengths or areas for

improvement

21

• Allows players to participate in a self-assessment of the exercise play

• Allows evaluators to clarify additional points and collect missing information

• Identify three sustains and three improves• Make sure sustains and improves are

listed on the easel pad

Player Hot Wash

22

• Work to complete EEG analysis forms for submission

• Discuss degree to which each objective was completed

• Provide strengths, best practices, lessons learned, and possible IP recommendations

• Begin outlining issues to be included in AAR/IP on AAR data input forms

Controller and Evaluator (C/E) Debriefing

23

AAR/IP Input• Use the EEG AAR data input form• Complete one form for each strength and area for improvement.

– Observation. Write a one-sentence statement that identifies the strength or area for improvement as noted in the activity analysis tables.

– References. Reference relevant plans, policies, procedures, laws, or regulations that could be used to facilitate implementing recommendations for improvement or capturing strengths.

– Analysis. Document detailed data on what you observed during the exercise specific to this individual observation.

– Recommendations. Suggest steps to allow the participants to further improve the plans or share strengths.

• All EEG input is due to Joe or Lou no later than departure from the exercise site on the exercise day.

24

Evaluation MatrixObjective Met Objective Not Met

Players followed plan procedures

Plan worksDocument best practices and strengths

Capture why plan didn’t workDocument how to improve relevant plan elements

Players did not follow plan procedures

Capture how objective was metRecommend successful actions and procedures be included in plan rewrite

Capture why plan was not usedMay require plan training and familiarization

25

Evaluator Deliverables• Before you leave

–Completed EEGs–Notes–Pictures of easel pads in your

section (Joe has camera and will do this but easel sheets are important to retain)

26

Questions/Discussion

27