© 2007 expedia, inc. all rights reserved expedia business resiliencyprepared by expedia for pep...
TRANSCRIPT
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All R
ights ReservedPage 1
© 2007 Expedia, Inc.
All rights reserved
Headquarters Closed: Now What? Live-Fire ExerciseLessons Learned
Howard Mannella, CBCPPrincipal Resiliency Strategist
Expedia, Inc.
Prepared for:Partners in Emergency Preparedness
April 14, 2009
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 2
What we’ll cover
• What we did/how we did it• Success factors• What we learned• What we didn’t learn
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 3
Who is Expedia?
• World’s leading OLTC• A global retailer of travel• Broadest reach in the category
– Some of the most popular sites on the Web
• Operations in Americas, EMEA, APAC
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 4
Exercise regimes
Walk-through
TableTop
FunctionalSimulation
DisruptiveLive-Fire
Competency Building
Degree of Disruption
Pause/Rewind/Replay
Degree of Validation
Low
Low
High
Low
High
High
Low
HighIndependent variables: scope, element of surprise
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 5
Expedia’s Exercises
“Virtual IT”• Exercise regime: Semi-
announced disruptive ‘live-fire’
• Scope: All IT Staff in the Bellevue HQ(with exceptions)
• Scenario: “Building unavailable and inaccessible, travel to Bellevue not an option”
• Timeframe: SOD Thurs - EOD Wed
“Europe Interrupted”• Exercise regime: Semi-
announced disruptive ‘live-fire’
• Scope: All-Hands in the Paris and Munich offices(with very limited exceptions)
• Scenario: “Building closed due to limited fire”
• Timeframe: 4pm Day 1 – EOD Day 2
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 6
Exercise objectives
• Assess ability of staff to work and be managed remotely• Assess effectiveness of communication protocols and channels• Assess effect on interactions with business areas, partners,
other locations• Opportunity for managers to think through and act upon an
emergency scenario and manage unexpectedly-dispersed staff• Opportunity for teams to think about and work under
emergency conditions in a safe low-risk environment• (For Paris) Technical validation of new Emergency Work Center• Identify gaps and weaknesses through physical failure• Forcing function for updating playbooks and other emergency
documentation and validating content
Objectives unique to Disruptive Exercises noted with emphasis
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 7
Considerations
• Exercises are forcing functions to help/make teams ready– They generate work – before, during, after– Pre-announcements and prep are not ‘cheating’
• Exercises are not supposed to go smoothly– Issues are expected
(and welcomed)– It’s not “Business as Usual”– They are not “Pass/Fail”
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 8
Quotes from the Post-Exercise Survey
• Were there business impacts during the exercise?“Yes, but in a positive way.”
• Did you come onto the campus?“Yes, but I didn’t do any IT-related tasks!”
• “Which of CIO’s direct reports do you report to?”“I’m not sure.”
• “I thought the exercise was very valuable!” • “The most ridiculous exercise I ever saw!”
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 9
Success factors
Tools and Enablers
Sponsorship and Support
Planning
Building the Buzz
Metrics
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 10
Sponsorship and Support
• CIO and Europe Leadership solidly behind it• Provided communications to divisions, briefings to
executive leadership
Sponsorship and Support
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 11
Questions to ask
• What are we trying to accomplish?• Who are the audiences? How do we engage them?• What will we measure? How?
“Do a disruptive exercise” is not a question, it’s an answer
Sponsorship and Support
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 12
Objective-based selling
Exercise objectives Walk
Thru
Table
Top
Func
Sim
Disrupt
L.F.
Recovery strategy technically sound? X X X X
Do notification/escalation procedures work? X X X X
Is exercise realistic? X X X
Where are the gaps and weaknesses in the plan? X X X
Are business areas adequately prepared? x X X
Business feasible in emergency conditions? x X
Do all staff know their roles ATOD? X
Do all tools and enablers actually work under load? X
“Acid Test”? X
Sponsorship and Support
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 13
BCP isNotDR
BCP isNot
a checklist
BCP isNot
“WFH-WFS”
Educating managementSponsorship and Support
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 14
Planning
• Over six weeks elapsed time per exercise• Working sessions with IT and remote location area
heads• Briefing session at HQ Admin all-hands• Awareness of cultural issues
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 15
Tools and enablers
• SharePoint site• Discussion board• Pre-and post-exercise survey• Emergency Notification
for English divisions(phone/email blast)
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 16
Building the buzz
• Signage• Staff meetings• Emails• Test phone blasts
(with a challenge)
Enlist the communications experts
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 17
Metrics
• Help Desk tickets• Email volumes• IM volumes• Badge access• Problem email distros• Official email thread• Command Center log (multiple)• “Closing the circuit” on escalations and communications
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 18
Communication Effectiveness
0%
20%
40%
60%
80%
100%
0:00 0:15 0:30 0:45 1:00 1:15 1:30 1:45 2:00
Phone/Email Blast:• Two-thirds of staff confirmed within 6 minutes• 80% confirmed within 30 minutesNormal Expedia Email:• Confirmation estimated by email replies
(unsolicited)• Actual confirmation not easily tracked for
Exchange aliases
Emergency Notification Service (phone/email blasts)
Normal Expedia channels (email aliases)
Time (minutes)
IT S
taff
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 19
Positives• Creativity• Technology and other enablers worked
for the most part• Participation: teams were engaged
and “played” the exercises realistically and professionally
• Teams put significant work intoupdating playbooks, briefing staffand pre-exercise planning
• The Paris Emergency Work Center (EWS)passed Proof of Concept and contract acceptance
• Provided an opportunity for experience, practice and competency, assuring crisper future responses
• Surfaced many gap items and weaknesses for resolution before they are needed
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 20
Issues
• Vendor management• Inter-group choreography• Rumor management• Human factors• Business judgment• Cultural differences• WFH/WFS not a complete solution• Human factor gaps need (constant) remediation• There will always be technology “gotcha’s”• Education through failure more effective than
education through jawboningWe learned many things almost impossible to learn from tabletops or simulations
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 21
The Bottom Line
Expedia demonstrated
the ability to keep
divisions operating
during short-term
disruptions
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 22
Best practices and Lessons Learned
Best Practices:• Make it clear that it’s not about
individual’s performance• Work the communication plan• Begin with the end in mind• Publish success
Lessons Learned:• Big difference between “a
person can work from home” vs. “a division can operate under disrupted conditions”
• Issues increase exponentially with scope
• Live-fire exercises will uncover things that tabletops and simulations cannot
Lessons Not [Yet] Learned:• Socialization of results could have been better• Follow-up could have been better• Make it part of a program
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 23
Questions?
Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 24
Thank you!
Howard [email protected]