business continuity /disaster recovery panel sta annual conference 2006
TRANSCRIPT
Business Continuity /Disaster Recovery Panel
STA Annual Conference 2006
2
A High Level Summary of the Lessons Learned
from Hurricane Wilma
by Franklin Templeton Investments
Wayne BehrensDirectorBusiness Continuity Planning
Prepared for theSecurities Transfer Assocation
Amelia Island Meeting on Oct 20, 2006
Franklin Templeton Investments
Who was Wilma
Impact of Wilma
What Went Right
Crisis Management Lessons
Business Continuity Lessons
Technology Lessons
Facility Lessons
Wilma Agenda
Franklin Templeton Investments
1. Parent: Franklin Resources Inc.
2. Approx 500 Billion in Assets Under Management
3. Major BrandsA. FranklinB. TempletonC. Mutual SeriesD. Fiduciary TrustE. Darby OverseasF. Bisset (Canada)
4. Employees: Approx 8,000 in 29 countries. In the Florida Area:A. St. Petersburg (1,200+)B. Ft. Lauderdale (466)C. Nassau (49)D. Miami (16)
Who the Heck was Wilma?
1. Hurricane Wilma is the lesser known cousin of Katrina. However, Wilma was the most intense hurricane ever recorded in the Atlantic basin. It devastated parts of the Yucatán Peninsula as well as southern Florida.
2. There were 62 deaths attributed to Wilma and damage is estimated at $12.2 billion in the U.S., making Wilma the sixth costliest storm in U.S history.
3. When Wilma reached Ft. Lauderdale, she was a category 2 hurricane with sustained winds of 110 mph to 130 MPH.
Impact of Wilma on Franklin
1. Wilma reached Ft. Lauderdale early Monday morning, October 24th. By Monday afternoon, we had received preliminary damage assessments indicating that hundreds of windows primarily on the north west section of the building were blown out.
2. We were unable to occupy the building from October 24 through November 18.
3. No loss of life and no injuries to employees. Some damage to a few employee’s homes. Many employee’s homes were without power for days or weeks.
4. 500 employees were impacted, with over 230 being relocated to other sites: Toronto, St. Petersburg, Miami, New York, Short Hills, San Mateo, and Rancho Cordova.
What Went Right
1. In general, our Crisis Management, Business Continuity and Technology Disaster Recovery plans worked well.
2. Wilma caused virtually no disruption to our customers.
3. The Ft. Lauderdale Emergency Management Team took charge of the situation in Ft. Lauderdale.
4. All business units were able to follow their business continuity plans and recover their operations.
5. Technology operations were restored to include two critical applications.
Crisis Management Lessons
1. When the local Emergency Management Teams (EMT’s) are in the midst of the incident and working literally in the dark, they do not always have the ability to fully coordinate the recovery of business operations away from the site.
2. Each of the major roles in our crisis management teams need to have a specific checklist.
3. We need to put in place a pre-plan to track and deal immediately with the relocation of employees to other sites.
4. You cannot over communicate. Despite the fact that we tried very hard to be proactive on communicating to our employees, we still heard a number of complaints in this area.
Business Continuity Lessons
1. Over 40 laptops were left in the FTL office when the hurricane hit. There had been so many hurricane threats during the year that many employees did not feel that Wilma would really hit or cause this much damage if it did.
2. The standard for Business Continuity Plans needs to be expanded to address a month long outage:A. Shifting work to alternate sites for
the first 3-5 days without moving employees
B. Plan for an incident to last over a month to include a month end
3. A number of issues arose in regards to employees:A. Pay during the outage B. Childcare and school closuresC. Bonus for extraordinary effortsD. Relocation of children & elders
Technology Lessons Learned
1. We were surprised by the number of business units that still relied upon hard copies of faxed documents to stand alone fax machine. We have worked on a better process and documentation of FAX rerouting requirements.
2. Better written procedures for forwarding 800 numbers.
3. There is a desire from the business for a better disaster recovery solution for email and Blackberry servers.
4. Environmental monitoring of server rooms needs to be tied into a central control point to insure it is remotely accessible.
Facilities Lessons Learned
1. We should have drilled more rigorously on damage assessments. For the first couple of days, the damage assessments were verbal and led us to believe the damage was much more extensive than it was. It turned out only 15% to 20% of the work areas were damaged.
2. Conversely, our repair and re-occupancy time estimates were wildly optimistic. Based on initial reports, we planned
for a week long disruption. We were out for almost a month.
3. Need to have working knowledge prior to the incident of what local agencies will require to re-occupy a building.
4. Keep in mind the fire marshal and the building inspectors are not always in sync.
Questions?
Wilma Closing
In the end it was the knowledge, flexibility and perseverance of our people who really carried the day and made the recovery a success.
Franklin’s Approach to Planning for a Pandemic
Pandemic Agenda
• Goal of this presentation
• Likelihood of a Pandemic
• Basic Business Continuity Strategy
• Why plan
• How will a pandemic differ
• Basic elements of our plan
– Crisis Management
– Business Continuity Planning
– Technology
– General Services
– Human Relations
– Corporate Communications
• What our plan does not cover
Pandemic Background
3. Strategy: Our basic disaster planning strategy is to shift our operations to other sites for 3 to 5 days, after which we will then need to start to shift people to alternate sites. However, this will not work in a pandemic situation.
1. Goal: To provide an overview of Franklin’s current approach and thinking in regards to planning for a possible Pandemic.
2. Likelihood of a pandemic occurring: The question is more like earthquakes in California. It is not a question of if, but rather when and how bad. Some data points such as the 1918, 1957 & 1968 pandemics and SAR’s. But, no good data on frequency or severity.
Pandemic Background (Continued)
4. Why plan if civilization is going to collapse:
We used a “reasonable worst case scenario”. A scenario which we think is likely to occur. This is not a worst case scenario which anticipates the general breakdown of society and services.
5. How will a pandemic differ from our “normal incidents:
- Many sites are likely to be impacted at approximately the same time - Will not be able to shift people between sites - Site might be impacted, but not incapacitated- Sites affected for months not hours- Affects people directly not facilities or IT.- Employees may choose not to come to work.- No clear beginning or end.
Pandemic Plan Outline
Our pandemic plan is broken down into:
1. Crisis Management:
- Framework to address a pandemic
- Tabletop exercises
2. Business Continuity Planning:
- Guidance to business units on how to review their business continuity strategies and workflows against a
pandemic type scenario
3. Technology
- Steps to reduce the impact of a
pandemic on our data centers
- Increased remote work capabilities
- Reviewing other strategies
Pandemic Plan Outline (Continued)
4. General Services– Best practices for employee hygiene program and procedures for facility
managers to follow in the event of a pandemic
5. Human Relations– Global HR policy framework to provide recommendations to local HR
groups to address issues that are likely to arise in a pandemic
6. Corporate Communications
– Integrated communication plan
7. Plan Does not currently include:
- PPE such as Masks gloves, Etc.
- Antiviral Drugs such as Tamiflu- Vaccines
Questions?
Pandemic Closing
2020
All Disasters are Local:All Disasters are Local:Regionalizing Business Regionalizing Business ContinuityContinuity
Securities Transfer AssociationSecurities Transfer Association
2006 Annual Conference2006 Annual Conference
Brian TishukBrian TishukChicagoFIRST Executive DirectorChicagoFIRST Executive DirectorOctober 20, 2006October 20, 2006
Critical Critical InfrastructuresInfrastructures
Critical Critical InfrastructuresInfrastructures
Water
Transportation
FinancialServices
Energy
Emergency Services
Government Services
Telecommunications
Food
Defense Industrial Base
ChemicalIndustry
Postal & Shipping
Agriculture
PublicHealth
The The FinancialFinancial Sector Sector
OVERVIEW OF THE US FINANCIAL SYSTEMAs of June 3, 2002
Financial marketssecurities and futures
markets, OTC markets , etc.
Financial intermediariesbanks, insurance companies,
savings institutions,broker/dealers, FCMs, etc.
Lenders/Investorsindividuals, firms,
government
Borrowers/Issuersindividuals, firms,
government
Supervision &Regulation,Treasury
and CentralBank functions
Treasury,Fed, SEC,
OCC, CFTC,FDIC, OTS,NCUA, Stateinsurance &
financialmarket
authorities,SROs, etc.
Financial utilities: payment, clearing & settlement.Service providers
Critical public utilities and services: Telecommunications, power, transportation, public safety,insurance companies as recovery agents
funds
funds
returns
returns
returns
returns
funds
funds
Financial instrumentssecurities, futures,
annuities,loans, derivatives,
CP, FX, etc.
Financial sectorassociations and
trade groups
Private sectorchecks andbalances
audit,rating agencies,
public disclosure,etc.
Federal Financial Federal Financial PartnershipPartnership
Financial and Banking Information Financial and Banking Information Infrastructure Committee (FBIIC)Infrastructure Committee (FBIIC)
(formed (formed January 2002)January 2002)
Financial Services Sector Financial Services Sector Coordinating Council (FSSCC)Coordinating Council (FSSCC)
(formed June 2002)(formed June 2002)
President’s Working Groupon Financial Markets
PUBLIC SECTOR PRIVATE SECTOR
US TreasuryAssistant Secretary forFinancial Institutions
FBIIC CHAIR
Financial and Banking InformationInfrastructure Committee
(FBIIC)
US Treasury DepartmentCommodity Futures Trading Commission
Conference of State Bank SupervisorsFederal Deposit Insurance Corporation
Federal Housing Finance BoardFederal Reserve Board of Governors
Homeland Security CouncilNational Association of Insurance
CommissionersNational Credit Union Administration
New York Federal Reserve BankOffice of the Comptroller of the Currency
Office of Federal Housing Enterprise OversightOffice of Thrift Supervision
Securities and Exchange Commission
Financial Services Sector Coordinating Council
for CIP/HLS
Financial Services Trade Associations & Institutes
New York Stock ExchangeThe Clearinghouse
FS/ISACSecurities Industry Automation Corporation
The Options Clearing CorporationChicagoFIRST
NASDAQAMEXASIS
FBIIC FSSCC
Assistant Secretary forFinancial InstitutionsSECTOR LIAISON
Rhonda MacLeanSECTOR COORDINATOR
Treasury - Lead Agency(PDD 63)
The Role for Regional The Role for Regional
Public/Private Public/Private PartnershipsPartnerships
5/19/2003 2
Business Continuity Planning
Local/Regional
Coordination with city and state authorities Region specific table-top/simulation exercises for threat assessment Telecom traffic assessment
Protecting the financial services sector requires collaboration at alllevels of business.
Company
Protect intellectual capita l Provide contingency workspace Protect executive leadership Perform threat and vulnerability
assessments Prioritize recovery of a ll business
functions Test business continuity response
and recover plans Consider alternate sites for
operations Harden physical security Consider sharing recovery solutions
with trusted partners
Industry
Increase awareness amongstparticipants
Coordinate per iodic test of back-upnetworks and facil ities
Coordinate emergency contactinformation
Share best practices amongstmembers
Provide regulatory and legisla tivesupport for industry issues
Re-organize security of nation under theDepartment of Homeland Security
Allocate resources to protect nationalinfrastructure
Provide intelligence to private industrythrough ISACs
National
The Missing The Missing PiecePiece
To increase the To increase the resilience of resilience of financial services financial services in the event of a in the event of a regional disaster in regional disaster in collaboration with collaboration with the city, state, and the city, state, and federal agencies. federal agencies.
All Disasters are All Disasters are LocalLocal
How will that jurisdiction prevent, prepare for, and How will that jurisdiction prevent, prepare for, and respond to incidents?respond to incidents?
Do your business continuity plans Do your business continuity plans incorporate government response plans?incorporate government response plans?
How can coordination be fostered among How can coordination be fostered among jurisdictions?jurisdictions?
Regional partnerships can strengthen the Regional partnerships can strengthen the business continuity plans of participating business continuity plans of participating firmsfirms
Regional Partnerships:Regional Partnerships:Formed and FormingFormed and Forming
Miami (FloridaFIRST)Miami (FloridaFIRST) San Francisco (BARC FIRST; Bay Area Response Coalition)San Francisco (BARC FIRST; Bay Area Response Coalition) Los Angeles (SoCal FIRC; Financial Institutions Recovery Los Angeles (SoCal FIRC; Financial Institutions Recovery
Coalition)Coalition) Minneapolis (MN-ISAC; MN Security Board)Minneapolis (MN-ISAC; MN Security Board) Tampa Bay Region of FloridaFIRSTTampa Bay Region of FloridaFIRST HoustonFIRSTHoustonFIRST PhiladelphiaFIRSTPhiladelphiaFIRST ColumbusFIRSTColumbusFIRST Alabama Recovery Coalition for the Financial SectorAlabama Recovery Coalition for the Financial Sector Chicago (ChicagoFIRST)Chicago (ChicagoFIRST) Washington, DCWashington, DC DetroitDetroit AlaskaAlaska SeattleSeattle JacksonvilleJacksonville Las VegasLas Vegas New Orleans (still thinking about it)New Orleans (still thinking about it)
RPC FIRSTRPC FIRST
Fostering Collaboration among PartnershipsFostering Collaboration among Partnerships
ChicagoFIRST leading the formation of a CouncilChicagoFIRST leading the formation of a Council Council would share best practicesCouncil would share best practices Council would help one another with administrative Council would help one another with administrative
questionsquestions Council can plug into FSSCCCouncil can plug into FSSCC
RPC = RPC = Regional Partnership CouncilRegional Partnership Council FIRST = FIRST = Financial Industry Resilience, Security, and TeamworkFinancial Industry Resilience, Security, and Teamwork Formed in early 2006Formed in early 2006
The ChicagoFIRST ApproachThe ChicagoFIRST Approach(formed May (formed May
2003)2003)
ChicagoFIRST’s Primary ChicagoFIRST’s Primary ObjectivesObjectives
Obtain a seat at Chicago's 911 Center in the event of a Obtain a seat at Chicago's 911 Center in the event of a crisis that affects Chicago's financial communitycrisis that affects Chicago's financial community
LaSalle Bank/ABN AMROLaSalle Bank/ABN AMRO
Create permits/passes for essential personnel to safely Create permits/passes for essential personnel to safely access business facilities in the event of a general access business facilities in the event of a general evacuation of the city (credentialing)evacuation of the city (credentialing)
Northern Trust BankNorthern Trust Bank
Develop and communicate standard evacuation procedures Develop and communicate standard evacuation procedures for industry personnel to exit city limits in the event of a for industry personnel to exit city limits in the event of a disasterdisaster
JP Morgan Chase JP Morgan Chase
MembersMembers ABN AMRO / LaSalle BankABN AMRO / LaSalle Bank Allstate Insurance CompanyAllstate Insurance Company AonAon ArchipelagoArchipelago Ariel Capital ManagementAriel Capital Management Bank of AmericaBank of America Chicago Board Options Chicago Board Options
ExchangeExchange Chicago Board of TradeChicago Board of Trade Chicago Federal Home Loan Chicago Federal Home Loan
BankBank Chicago Mercantile ExchangeChicago Mercantile Exchange Chicago Stock Exchange Chicago Stock Exchange Fidelity National FinancialFidelity National Financial
Global Electronic Trading Global Electronic Trading CompanyCompany
Harris BankHarris Bank JP Morgan ChaseJP Morgan Chase Man FinancialMan Financial Mesirow FinancialMesirow Financial Mizuho Securities USAMizuho Securities USA Northern TrustNorthern Trust The Options Clearing The Options Clearing
CorporationCorporation PrivateBank and TrustPrivateBank and Trust UBSUBS Washington MutualWashington Mutual William Blair & CompanyWilliam Blair & Company
Strategic Partners (pg. 1 of 2)Strategic Partners (pg. 1 of 2)
Chicago Office of Emergency Management and Chicago Office of Emergency Management and CommunicationsCommunications
Chicago Police Department Chicago Police Department Commodity Futures Trading CommissionCommodity Futures Trading Commission FBI / InfraGardFBI / InfraGard Federal Deposit Insurance CorporationFederal Deposit Insurance Corporation Federal Emergency Management AgencyFederal Emergency Management Agency Federal Reserve Bank of ChicagoFederal Reserve Bank of Chicago Financial and Banking Information Infrastructure Financial and Banking Information Infrastructure
CommitteeCommittee Financial Services Information Sharing and Analysis Financial Services Information Sharing and Analysis
CenterCenter Financial Services Roundtable / BITSFinancial Services Roundtable / BITS Financial Services Sector Coordinating CouncilFinancial Services Sector Coordinating Council Futures Industry AssociationFutures Industry Association
Strategic Partners (pg. 2 of 2)Strategic Partners (pg. 2 of 2)
Great Lakes PartnershipGreat Lakes Partnership Illinois Department of Financial and Professional Illinois Department of Financial and Professional
RegulationRegulation Illinois Emergency Management AgencyIllinois Emergency Management Agency Illinois State PoliceIllinois State Police Illinois Terrorism Task ForceIllinois Terrorism Task Force National Futures AssociationNational Futures Association Office of the Comptroller of the CurrencyOffice of the Comptroller of the Currency Securities and Exchange CommissionSecurities and Exchange Commission Securities Industry AssociationSecurities Industry Association United States Attorney’s Office for the Northern United States Attorney’s Office for the Northern
District of IllinoisDistrict of Illinois United States Department of Homeland SecurityUnited States Department of Homeland Security United States Department of the TreasuryUnited States Department of the Treasury United States Secret ServiceUnited States Secret Service
Achieving Our GoalsAchieving Our Goals
Formal 911 Center Formal 911 Center SeatSeat Obtained seat at 911 Center in fall 2003Obtained seat at 911 Center in fall 2003
Primarily for government agenciesPrimarily for government agencies May use seat when Center is activatedMay use seat when Center is activated
Enhancements to seat at emergency operations centerEnhancements to seat at emergency operations center Set of individuals to staff the seat (with Chicago Fed Set of individuals to staff the seat (with Chicago Fed
help)help) Handbook with protocols for using the seat, Handbook with protocols for using the seat,
activating our crisis communicator, and contact activating our crisis communicator, and contact informationinformation
Private component of web site created and Private component of web site created and configured to provide a message board for posting configured to provide a message board for posting and recording critical informationand recording critical information
Information about the membership, including Information about the membership, including critical locations and essential employees, on the critical locations and essential employees, on the computer at the seatcomputer at the seat
Informal Information Informal Information SharingSharing Seat at 911 Center will be used rarelySeat at 911 Center will be used rarely
But the relationships with the city and state But the relationships with the city and state are invaluableare invaluable
Spring 2004 information about leaning Spring 2004 information about leaning transmission towertransmission tower
August 1, 2004 threats against financial August 1, 2004 threats against financial institutionsinstitutions
LaSalle Bank fire, December 2004LaSalle Bank fire, December 2004
Credentialing and Credentialing and EvacuationsEvacuations
CredentialingCredentialing Discovered city and state each seeking Discovered city and state each seeking
credentialing systems, but not coordinatingcredentialing systems, but not coordinating
City adopted credentialing pilot in which City adopted credentialing pilot in which ChicagoFIRST participatesChicagoFIRST participates
EvacuationsEvacuations Illinois Department of Transportation tabletops Illinois Department of Transportation tabletops
in 2004, 2005, and 2006in 2004, 2005, and 2006
September 7, 2006 evacuation drill in the LoopSeptember 7, 2006 evacuation drill in the Loop
Additional Additional AchievementsAchievements
Working GroupsWorking Groups Security Working GroupSecurity Working Group
Coordinating training needs and opportunitiesCoordinating training needs and opportunities
Coordinating physical security and optionsCoordinating physical security and options
Piloted NC4 Situation Awareness ServicePiloted NC4 Situation Awareness Service
Power Working GroupPower Working Group Understanding electricity in multi-tenant Understanding electricity in multi-tenant
buildingsbuildings
Sharing ComEd information among membersSharing ComEd information among members
Working GroupsWorking Groups Telecommunications Working GroupTelecommunications Working Group
Educating membershipEducating membership– GETSGETS– TSPTSP– SBC call forwardingSBC call forwarding
Surviving a central office failureSurviving a central office failure– TeleContinuityTeleContinuity– LEMKOLEMKO– Sprint IP networkSprint IP network
Working GroupsWorking Groups
Pandemic Planning Working GroupPandemic Planning Working Group Free exchange of HR, legal, & BCP information, without Free exchange of HR, legal, & BCP information, without
NDAsNDAs
Coordinating with state and local health departmentsCoordinating with state and local health departments
Coordinating with sector-wide effortsCoordinating with sector-wide efforts
Evaluating hiring a public health advisor for Evaluating hiring a public health advisor for ChicagoFIRSTChicagoFIRST
Tabletop scheduled for November 2, 2006Tabletop scheduled for November 2, 2006
Working GroupsWorking Groups Public Relations Working GroupPublic Relations Working Group
Single point of contact for the mediaSingle point of contact for the media Firms leverage membership with pressFirms leverage membership with press ChicagoFIRST increases media understandingChicagoFIRST increases media understanding
Crisis Communications Working GroupCrisis Communications Working Group Quarterly tests of the 911 Center proceduresQuarterly tests of the 911 Center procedures Quarterly tests of DialogicQuarterly tests of Dialogic (notification (notification
data)data) Quarterly tests of TeleContinuity and GETSQuarterly tests of TeleContinuity and GETS
2004 Milestones2004 Milestones Testified before House Financial Services Testified before House Financial Services
Committee on ChicagoFIRST as a partnershipCommittee on ChicagoFIRST as a partnership
9/11 Commission legislation identifies 9/11 Commission legislation identifies ChicagoFIRST as a modelChicagoFIRST as a model
GAO Report on Financial Market Preparedness GAO Report on Financial Market Preparedness praises ChicagoFIRSTpraises ChicagoFIRST
Treasury handbook identifies ChicagoFIRST as Treasury handbook identifies ChicagoFIRST as modelmodel
Tabletop on city’s response to Chicago Tabletop on city’s response to Chicago financial communityfinancial community
2005 Milestones2005 Milestones Tabletop focused on futures and options Tabletop focused on futures and options
marketsmarkets
Public television features ChicagoFIRSTPublic television features ChicagoFIRST
Fund Illinois Terrorism Task Force (ITTF) video Fund Illinois Terrorism Task Force (ITTF) video for the citizens of Illinoisfor the citizens of Illinois
Co-chair Private Sector Committee of the ITTFCo-chair Private Sector Committee of the ITTF
2006 Activities2006 Activities Mutual aid among the membersMutual aid among the members
Credentialing critical supplies like cashCredentialing critical supplies like cash
Evacuation drillEvacuation drill
City of Chicago camera programCity of Chicago camera program
Provided testimony on pandemic Provided testimony on pandemic preparedness to the House Financial preparedness to the House Financial Services CommitteeServices Committee
ChicagoFIRST ChicagoFIRST Model WorksModel Works The The model model is the partnership approach, not the goals is the partnership approach, not the goals
or organization of ChicagoFIRSTor organization of ChicagoFIRST FloridaFIRST covers the entire state, with FloridaFIRST covers the entire state, with
several regionsseveral regions BARC FIRST and SoCal FIRC split CaliforniaBARC FIRST and SoCal FIRC split California MN-ISAC has Target, Best Buy, 3M as membersMN-ISAC has Target, Best Buy, 3M as members
Leverage Leverage partnership to encourage public sector partnership to encourage public sector information sharing and improvementsinformation sharing and improvements
Seats in EOCsSeats in EOCs CredentialingCredentialing Access protocols for critical suppliesAccess protocols for critical supplies
The Value The Value PropositionProposition LaSalle Bank fireLaSalle Bank fire
Mizuho futures Mizuho futures
Cooperation vs. competition on employee safety and Cooperation vs. competition on employee safety and business continuity (mutual aid established after the business continuity (mutual aid established after the fire)fire)
Government appreciates single point of contactGovernment appreciates single point of contact
NC4 and TeleContinuity NC4 and TeleContinuity
Contact Contact InformationInformationBrian TishukBrian TishukExecutive DirectorExecutive DirectorChicagoFIRSTChicagoFIRST
312-322-4441 312-322-4441
[email protected]@chicagofirst.org
www.chicagofirst.orgwww.chicagofirst.org