new clerk academy · for minor disasters for important business functions, the recovery strategy is...
TRANSCRIPT
New Clerk Academy
May 8, 2014
OVERVIEW
Disaster Recovery
3
Presentation Agenda
Introduction and Definitions
DR Motivators and Drivers
Recovery Challenges
Scope of Disasters
Components of Recovery Plans
Business Continuity Planning Process
FACCSG Data Vaulting Solution
4
What is a disaster?
Definitions:
The interruption of business due to the loss or denial
of the information assets required for normal
operations.
A sudden, unexpected emergency requiring
immediate action.
An event that renders the system inoperable.
A loss or interruption to business functions.
A failure within the infrastructure which interrupts
business resulting in unplanned downtime.
5
DR Motivators and Drivers
Audit requirements
Government mandates
Insurance
requirements
Business impact
Legal liabilities
Protection of business
assets
Protection of business
relationships
Impact to customers
Recent disasters
Increased regulation
from external sources
6
Recovery Challenges
Why do planning efforts fail?
Degree of dependence not understood
Potential impact not recognized
Lack of management commitment
The “won’t happen here” syndrome
DR is never a priority
7
Recovery Challenges (cont’d)
Shrinking recovery windows
Seamless or transparent recoveries
Growth of E-commerce
Internet dependencies
“Too big to recover” syndrome
Regional and global recoveries
8
Why Move Forward?
Potential Impact
Tangible costs:
Lost revenue
Lost productivity
Legal implications
and fines
Lost wages
Intangible costs:
Political image
Reputation
Credibility
Employee morale
9
Planning Questions
What types of interruptions could we have?
Which of our business functions are critical?
How are we defining critical?
How would a disaster affect the quality of our
service?
If you had 15 minutes to vacate your office, do
you know what you would take with you?
10
Planning Questions (cont’d)
How long could your organization survive in
the event of a disaster?
Are recovery tasks adequately organized and
assigned?
What happens if there’s a disaster at 2:00
pm on a holiday weekend?
11
• terrorism
• theft
• tidal wave
• tornado
• volcano
• water damage
blizzard
bomb threat
chemical spill
civil disturbance
contractual failure
dam collapse
earthquake
epidemic
hardware failure
explosion
fire
flooding
freezing
heat
high winds
hostage taking
hurricane
network failure
nuclear war
power failure
riot
sabotage
strike
telecommunications failure
Types of Disasters
12
Region/Area
Building/Site
Data-Center
Network
System
Data
Application database
Scope – Layers of Protection
Emergency Operations Center (EOC) Crisis management
Internal hot-site
Commercial hot-site Remote fail-over Work area recovery
Redundant networks Reroutable networks Network backups
Database backups Database logging
Component redundancies Local fail-over systems
Data back-ups Off-site media storage Disc mirroring (remote) Electronic vaulting(remote)
13
Components of an IT Recovery Plan
System Recovery
Database Recovery
Application Recovery
Data Recovery
Network Recovery
Telecommunications Recovery
LAN Recovery
User Work Area Recovery
14
Components of a Business Continuity or Emergency Response Plan
Employee Safety and Awareness
Emergency Response
Crisis Management
Site or Facilities Recovery
Relocation of business units
Vendor-supplier recovery
Public Relations/Media Response
IT Recovery is a component of the BCP
15
Disaster Recovery Plan
Definitions
A predefined, tested set of procedures to
implement in the event of a disaster.
Documented procedures which outline the who,
why, what, and how to plan and recover from a
disaster.
Purpose is to increase the chances of survival
and to decrease the amount of loss.
16
Does provide:
Blueprint for survival of the business after a
disaster
Does not:
Duplicate a normal business environment
Provide business-as-usual, in survival mode
A Disaster Recovery Plan …
17
Business Continuity Planning Process
Management Sponsorship
Scope and Objectives
Risk Management and Disaster Avoidance
Recovery Requirements
Design and Development of the Plan
Rehearsal, Maintenance, and Reviews
18
Management Sponsorship
Strategies for selling senior management
Identify risks and vulnerabilities
Describe the benefits of having a plan in place
Identify impact in not having a plan
Substantiate downtime in dollars
Clarify insurance coverage boundaries
Increase awareness
19
Scope and Objectives
Defining scope What are you attempting to protect? Business continuity versus IT recovery Define scope of plan
Defining objectives
Define how much you are willing to lose Define disaster scenarios Classify disasters: minor, major, catastrophic Define acceptable, tolerable downtime Define project management plan Define project team members Define methodology to be used
20
Operating
System Database
System
Recovery Team
Site/ Building
Security
Clean-up / Restoration
Facilities
Tape
Librarian
Data Entry
Data Control
Personnel,
Insurance, Public
Relations, Legal
Petty Cash,
Legal Purchasing, Audit
Administration
Users
Application
Support
Application Development
User-Liaison
Business Continuity Recovery Coordinator
Outside Authorities Business Continuity Manager Senior Management
Data
Communications
Voice
Communications
LAN
Communication
Disaster Recovery Project Team Structure - Sample
21
BACKUP
ALTERNATE
PROCESSING
RECOVERY
PROCEDURES DISASTER
RECOVERY PLAN
REVISIONS AND UPDATES
TESTING
TRAINING
MAINTENANCE
DR Methodology - Sample
PROJECT
MANAGEMENT
CRITICAL
APPLICATIONS
MANAGEMENT PROCEDURES
INVENTORY
EVENT
DETECTION
22
Risk Management and Disaster Avoidance
Identify risks, threats, and exposures
Assess current environment
Conduct disaster avoidance review
Assess cost-benefit ratio for accepting risk
versus implementing controls
Implement controls to mitigate risk and
potential disasters
23
Recovery Requirements
Define recovery needs and requirements
Conduct Business Impact Analysis (BIA) define key functions
develop and validate questionnaire
conduct interviews
identifies tolerable, maximum amount of downtime
identifies potential workarounds
Categorize and prioritize business functions vital, critical, important, deferrable
disaster classifications: minor, major, catastrophic
Conclusions from the analysis
24
Recovery Requirements (cont’d)
Define Recovery Time Objectives (RTOs)
Define recovery strategies to support potential disasters
For example:
For minor disasters for important business functions, the recovery strategy is to do nothing for a period of 4 hours or less, not to exceed 24 hours.
For minor disasters for critical business functions, the strategy will be to alert stand-by processing and wait for 1 hour or less.
For major disasters affecting vital business functions, the strategy is to assemble the disaster management team and to invoke the recovery plan within the hour.
25
DISASTER
ELAPSED TIME
Backup Facility Activated
Full Recovery
Alternate Processing Interim
Processing Normal
Recovery Timeline
26
* If resolved, write incident report.
* If NOT resolved within allowable time,
escalate to next level.
?
OPERATOR:
- Attempt to diagnose
and resolve problem
REFER, AS NEEDED, TO:
- Shift Supervisor
- Operations Manager
15 MIN
PROBLEM
DECLARE DISASTER
INITIATE RECOVERY PLAN
DISASTER
NOTIFY:
- Dir, Computer Ctr Opns
- Mgr, Computer Ctr Opns
- Mgr, Info Sys Network
CALL IN:
- Outside resources
CONTINUE:
- Diagnosis and
problem solving
4 HRS
ALERT 1
NOTIFY:
- VP, Information Systems
- Mgr, Change Management
- User Liaison Team Leader
REPORT ON-SITE ASAP:
- Dir, Computer Ctr Opns
- Mgr, Op Supp & Control
- Disaster Recovery Manager
- Mgr, Computer Ctr Opns
- Mgr, Info Sys Network
CONTINUE:
- Resolution efforts
EVALUATE CRITICALITY
AND DECIDE ON
INITIATING RECOVERY
PLAN
MAXIMUM
TOLERABLE
DOWNTIME
(24 HRS)
ALERT 2
Problem Escalation Process
27
Define backup alternatives, options, workarounds for potential scenarios. Depending upon the type of disaster, will dictate which backup option is appropriate.
Some of include: do nothing
recover at time of disaster
recover using manual processing, service bureau
recover using stand-alone, PC-based system
recover at another company owned facility
recover at a commercial hot-site
recover at an internal hot-site
NONE OF THESE ARE MUTUALLY EXCLUSIVE combination of the above
Define Backup Alternatives or Options
28
Design and Develop Recovery Plan
Plan to include: Scope and objectives
Escalation and notification procedures
Recovery scenarios
Project team roles and responsibilities
Recovery priorities
Recovery procedures
Alternate processing
Notify key business units owners
Notify key vendor and supplier contacts
“Return to Normal” procedures
29
RECOVERY
DIRECTIVE
DAMAGE/IMPACT
ASSESSMENT
MANAGEMENT PROCEDURES
DISASTER MANAGEMENT TEAM
NOTIFICATION
& CONTROL
CENTER
ALTERNATE
PROCESSING
APPLICATION TEAMS
RECOVERY
PROCEDURES
INTERIM
PROCESSING
RECOVERY TEAMS
FULL
RECOVERY
DISASTER
EMERGENCY
PROCEDURES
Disaster Recovery Process Flow
30
Rehearsals, Maintenance and Reviews
Rehearsals Practice-practice-practice
Design test plans define evaluation criteria for success
define areas to be rehearsed (people, procedures)
Planned rehearsals versus unannounced
Use variety of techniques: partial to full-scale
evacuation
phone lines/email
table-top exercises
1-2 applications or business units
full scale rehearsals
31
Maintenance
Schedule regular updates
Incorporate feedback from the rehearsals
Incorporate into change control process
Reviews
Validate against Service Level Objectives (SLAs)
Conduct pre-audit of existing recovery plans
Conduct objective review
Rehearsals, Maintenance and Reviews (cont’d)
Civitek Data Vaulting Solution
Civitek has partnered with Hayes Computer Systems and EMC to provide a data vaulting solution to Clerks
Each participating Clerk will purchase or lease a local storage appliance manufactured by EMC
The local device will be networked with the central vault in Alpharetta, Ga
This device can be easily integrated into the existing IT environment
How does the service work?
Data is replicated from Clerks local EMC
appliance to a secure CAT 3 rated facility in
Alpharetta, Ga
Utilizes Clerk’s current WAN connected to all
67 counties throughout Florida
Data replication requiring more network
bandwidth can utilize a secure vpn
connection
34
Benefits
Improve Disaster Recovery- data off site,
replace tape based DR with replication
Reduce backup costs
Reduce backup and recovery risks
Speed up your backups
Simplify backup and recovery operations
Benefits
Improve Disaster Recovery- data off site,
replace tape based DR with replication
Reduce backup costs
Reduce backup and recovery risks
Speed up your backups
Simplify backup and recovery operations
Disaster Recovery
Replicate data offsite
Replace Tape based
DR
Reduce Backup Costs
Replace costly media
Tape drives
Backup application
licensing costs
Reduce Backup and Recovery Risks
Replace unreliable
media
Tapes get lost
Tapes get damaged
High percentage fail
RTO
RPO
Speed up backups
Efficient technology
DD Boost software
EMC Networker
EMC Avamar
NetBackup
Backup Exec
Dell NetVault
Dell vRanger
HP Data Protector
Simplify backup and recovery
Scalability
Less to manage
Less infrastructure
Options
Buy – server sized for your environment and
projected growth.
Lease- server sized for your environment ,
lease to buy, replace as growth requires
Cloud protect- server sized for your
environment, replaced with larger server as
needed, charged for amount of data stored.
Contacts
David Porter -Marketing
Earl Donaldson – Network Engineer
Tab Bradford- TAP Administrator