25e - book a test with your dr plan a test with your dr plan.pdfthe role of the plan manager during...

Presenter: Richard Dolewski

© 2016 Denovo. Confidential Trade Secret.

25E - Book a Test with your DR Plan

Send an Invitation & connect on LinkedIn

Receive Tip & Techniques Monthly

Industry Best Practices to improve your

Business Continuity so you: Don’t Fall with the Fallen!

www.linkedin.com/in/richarddolewski

Richard Dolewski

© 2017 Confidential Trade Secret.

According to Gartner...

It is estimated that 50% of all DR tests fail and 75% of all companies never fully complete a

DR test!

• Many companies simply don’t have the manpower or financial resources to put toward testing.

• Their technology staff is focused on developing and maintaining business applications and managing everyday IT support issues

Testing

Prepares you not only for a disaster, but for problems that

can occur back home.

© 2017 Confidential Trade Secret.


It’s not all bad news when your plan fails during a test…

…frequent testing identifies gaps in your recovery plan to ensure the Plan works when

you need it in a Disaster !


Companies Lack Recovery Confidence

✓ 16 % - DR solution complete, and tested

✓ 20 % - Only 90% confident of a recovery

✓ 64 % - CIO(s) Expressed concerns, as they

rated their current recovery confidence

Source: 4500 IT Professional Sampling

The Information Availability Institute (IAI), The State of Resilience

30%

25%

20%

15%

10%

5%

0%

100% 90% 75% 50% 25% No Plan

Success Level in a Recovery


Validation Exercise

A logical Assessment of the validity of a plan and its assumptions.

• Focus on the Recovery

• Ability of the Internal Stake-holder

• Ability of the External Stake-holders

Validation Exercise

1. Review all SLA’s with the business

2. Review flow of the plan

3. Ensure all Assumptions of the plan are valid

4. Review 3rd party providers

5. Review Communication Carrier SLA



Practice makes Perfect

Primary Objective in testing a DR Plan

▪ Is to learn what does not work

▪ Should have worked….but….you know

▪ Remediation

▪ Immediately repair what did not work


The role of the Plan Manager during a Technical test is to:

• Set a date for the Test – Passive and Active Test

• Set Testing objectives to meet desired business outcomes

• Develop Recovery Scenarios

• Ensure that every objective is fully realized

• Ensure that each test participant follows the procedures

• Record issues and their resolutions ( RCA ) as they arise

• Record the duration of each of the recovery procedures

• Liaison with the recovery provider staff.

Plan Manager Activities


Ensure Your Objectives are Realistic

• Your Objectives should not exceed nor extend your capabilities

• Its better to under-estimate & Achieve your objectives

• Credibility is very tough to recover…much tougher then any system

• Walk do not run

Testing -Know Your Responsibilities

• Know who is responsible for the Test

• Define key responsibilities

• Ownership

• Measure and report on roles performed by your staff and/or your partners



Executives are not typically involved and should NOT be.

Why:

• Executive Pressure is hindering

• Intimidating

• Often Nasty

• Glorified Experts

The team must provide regular status reports & the executives should be accessible when required.

The Role of Executives


Be Involved

• Work closely with your task owners on any issues that arise

• Be involved in the decisions

• Spend as much time as required on testing, then spend some more

• Use external sources for guidance

Ensure Plan represents the currency of your Business


DR Planning is an Ongoing Process

Business

Requirements Plan

ImplementTest

…to be prepared, you must regularly

go through the cycle

Disaster Recovery Plan Audit

• Annual audit of Systems Environment

• Includes the review of the documented plan

• Validate compliance to the Corporate Policy & Standards

• Identify areas for improvement

• Analyze identified critical business processes

• Recovery sequence

• Recovery time frame

• Examine backup and recovery process

• Review implementation, test and maintenance plan

• Evaluate recovery effectiveness

Are these servers still in DR Scope ?

Tier 1 - Availability RTO: 2 RPO: LSTPlatform Server Hardware Criticality Core Application

IBMi CUSDB100 Physical A JDE EnterpriseOne

Windows CUSWE1WB100 Virtual A JDE Web Server

Unix CUSUE1AP100 Virtual A JDE Logic Server

Windows CUSWIFDC100 Virtual A Domain ControllerTier 2 - Protect RTO: 12 RPO: 1

Platform Server Hardware Criticality Core Application

Windows CUSWE1DB100 Virtual A DSI Database

Windows CUSWE1DP100 Virtual A Deployment Server

Windows CUXWE1AP101 Virtual B DSI Server

Windows CUSWE1AP102 Virtual B Print Server

Not In Scope RTO: None RPO: NonePlatform Server Hardware Criticality Core Application

Windows CUSWIFEX100 Virtual None Exchange Server

Windows CUSWIFLS100 Virtual None Skype Server

Tier 3 - Recover RTO: 48 RPO: 24Platform Server Hardware Criticality Core Application

Windows CUSWIFFS100 Virtual A File Server

Windows CUSWE1AP100 Virtual A 3rd Party Application

Windows CUSWE1AP103 Virtual B CreateForms

Network Recovery Considerations

Key Takeaways

Data Center

HO SiteGCV Site

Private Wide Area Network

Plants

Dynamic Failover / Alternate Protected Path WAN Connectivity

Internet

Publc Owned IP SpaceMulti Downstream ISP BGP

Advertisement

Connectivity Migration• Fiber Network Connectivity to Remote Sites• Protected Connectivity at GVC & Hospital• Data Replication POC (sync vs. async)• Public IP Space Conversion to ARIN Assigned

Range• Isolated network App/site testing


Current App Landscape: High-Level

Plan Elements

+ Employees

+ Vendors & Service Providers

+ Contacts (Internal & External)

+ Specialized Supplies

+ Vital Records & Documents

+ Locations

+ Recovery Teams & Tasks

+ Manual Workaround Procedures

+ Applications, Software & Equipment

Recovery Script Issues

This procedure will pre-determine your company’s course of action:

• When do I inform management?

• When do I put the Vender(s) on Alert or Declare?

• What time - What actions ???

• Who will execute them ???


Have your Recovery Metrics Changed

RPO:

Recovery Point Objective

RTO:

Recovery Time Objective

Disaster recovery deliverables are based on:



Tape Storage – Current and Archived


Validation Exercise

• Evaluating Vendors

• Evaluating vendor Contracts - Legal aspects

• Arrangements for drop shipment of critical equipment

• Advanced arrangements with primary and alternate suppliers of critical equipment and supplies


+ Support staff and availability

+ Experience

+ Test time

+ Numbers of customers

+ Cost

+ Additional services

+ Declaration fees

+ What’s included and what’s not?

Evaluating Hot site Vendors

TestingPractice like the Pros

Elements of Surprise

• Fire Drill

• Audit your tapes & Offsite Tape

• Switch Roles for recovery activities


Implementation Review

Review Documentation:

✓ Organization

✓ Consistency

✓ Is it Clear?

✓ Staffing

✓ Lack of Documentation ( Missing )


✓ Passive Testing

✓ Active Testing

Test because your business depends on it!


Passive Testing - Format

This type of test is conducted in a conference room with a very large table:

• The Length of this test is generally 1.5 - 2 Hours

• Include everyone listed in the DR plan

• Bring Food and Refreshments



• Participants bring their copy of the DR plan

• Plan Manager reviews objectives of the test

• Plan Manager starts the test

• The Recovery team discusses the scenario

• Recovery team executes the Recovery tasks


Passive Testing

The exercise should:

State the objectives of the walk through

List the participants

Select a scenario relative to your company

Include the scenario in definition handouts

Summarize the changes for the Computer Contingency

Plan and schedule for their completion



• Team member Name

• Home Address (Street address, not a PO Box number)

• Home telephone number

• Pager number, if available

• Cellular telephone number

• Personal Email ( not the corporate email address )

• Alternate telephone number ( ie : cottage )

Table Top Testing

Reduce the team!!!

Reduce the Recovery team by 20%.

Examine Logistics

Put the logistics into action

Can we make it happen as written???

Telephoning vendors after normal business

hours to ensure that their hotline and

service numbers are correct and manned


The Plan Manager is also responsible for writing the summary

report for the test.

• Review the objectives of the technical test.

• Summarize the changes for the DR Plan and distribute.

• Summarize any recommendations resulting from the test.

• Post test meeting with Participants.

• State the schedule for the next test.

Plan Manager Activities


Test because your

your Business

depends on IT

ActiveDR Plan Testing

Validates the Recovery Plan in terms of:

1. Recovery capabilities and abilities

2. Alternate site configuration

3. Network recovery

4. Offsite records

5. Identify weaknesses in the plan

6. Provides training

This all equals success during a disaster.

DR Simulation Exercises

IT Active Testing

• A hands on exercise focus to determine how well the plan works.

• Tests the Business on how impacted they are in a disaster.


The first contact person notified of a potential disaster will assume the responsibilitiesof performing the tasks listed below.

• Name of the person making the alert

• Nature of the problem

• Initial assessment, personnel injuries and physical damage.

• Building accessibility

• Immediate dangers or restrictions

• Contact number for person performing the ALERT

Disaster Verification


Active Testing

It’s a Validation Process

✓ Test until you get the first one at 100%

✓ Management expects a working solution “Yesterday”

✓ Test several times a year to ensure all is well

Sample Active Test Objectives

• Complete application recovery vs hardware topologies

• All communications interfaces tested

• Communication file retransmissions with vendors

• Mid-Day failure recovery for Online systems

• Validation of data recovery

• Draft of DR plan presented to Team for review



Review the Recovery Script

This procedure will pre-determine your company’s course of action:

• When do I inform management?

• What time - What actions ???

• Who will execute them ???

• Can we meet Timeline Expectations

Review Hot-Site Activation

✓ Declaring a Disaster

✓ Disaster Declaration Personnel

✓ Directions to primary Hotsite

✓ Directions to Alternate Hotsite

✓ Travel Information

✓ Recalling Offsite Tape

✓ Hot-Site Opening

✓ High Availability Role – Swap



Active Testing

Simulated Realism is extremely difficult

• Requires Tremendous effort & Preparation

• Realism in testing is absolutely necessary


Testing Summary

• Recovering from a Disaster is overwhelming

• This can lead to frustration & loss of motivation

Frequent Testing results in:

✓ People that are comfortable in familiar situations

✓ Visually see the recovery goal & efforts

✓ Confidence already in place


Active Testing

1. At least sixty days in advance, schedule the test with the Hotsite. Notify computer recovery team members of the chosen date

2. Meet with computer recovery team to establish test objectives thirty days prior to the test date

3. One week Thirty days prior to the test, publish the test plan to participants

4. Confirm your test date with all Stakeholders

5. Initiate the transfer of tapes from your offsite storage office to the Hotsite facility


Active Testing

There are a wide variety of active tests that can be performed. Examples include:

1. A full technical test of restoration of production applications

2. A technical test of local or remote network recovery

3. Rebuilding the Server from scratch

4. Executing the notification, escalation and assembly tasks on a non-business day.


Testing

Testing your recovery strategy ensures you have a good back up strategy!

• Your backup is only as good as your recovery

• Your recovery is only as good as your backups

Hint: Design recovery strategy before your

backup strategy

Plan Maintenance


Plan Maintenance

Should be part of your Change Control Process.

Major changes in personnel should be reflected in the plan as soon as they happen.


Drivers of Change

• Have you kept up to date with your IT integrations?

• DR Planning not implemented into your change control

• You no longer have qualified staff

• Expectations of your DR Plan are unrealistic

• Optimistic - In Reality - Not prepared

Changes May Impact …

• Hardware/Infrastructure

• IBM Power, VIOS, Virtual partitions

• Operating System

• IBMi, Active Directory

• Business Software

• Fixes, data refreshes, third party products

• Security

• Network, system access, Firewalls

Plan Maintenance

Appendix

+ Vendor agreements and contracts

+ Telephone calling tree

+ Team notification procedures

+ Recovery and resumption sites, addresses, telephone numbers


Plan Maintenance

Quarterly:

Circulate copies to each computer recovery team member.

Send a copy of the Personnel report to the Human Resources department to review for changes and deletions. Deletions must be replaced by the new job holders.

Semi-Annually:

Circulate copies of the recovery procedures to each computer recovery team member. The teammembers should note their changes on the procedures directly and return them to the PlanManager.

Annually:

Circulate copies of the Disaster Recovery Plan to members of the computer recovery team for review.


Recovery Planning Requires Teamwork

• “Planning assures recovery success. Your business depends on IT.

• DR Planning is a partnership, and is ONLY successful when everyone works together.”


Successful Testing Requires Teamwork

“The desire to recover is High. The time to test frequently is not there.

Testing is a partnership and ONLY successful

when everyone works together.”

Best Practices


Introducing Murphy

Does this sound familiar?

✓ Testing only done from full backups

✓ Special backups performed to ensure success

✓ Backup tapes pre-shipped

✓ Same staff perform recovery steps – Alternates back running the shop

✓ Never test the whole thing – Communications, All servers in Enterprise


• Publicise chain of command

+ Identify clearly the chain of command that will be in effect if a disaster is declared

+ Be very clear

+ Be very clear (this repeat is intentional)

• Shift work – This is now a 24 hour shop

+ Plan for rotation shifts

+ Allow team members to rest

+ No time for heroes

Planning Best Practices


Planning Best Practices

• Build recovery teams with leaders

+ These leaders understand the systems

+ Strong technical and business knowledge of systems they are responsible for recovering

• Practice makes perfect

+ Allow team members ample opportunity to practice recovery scenarios

+ Primary and alternatives equally trained

+ Develop skillset and develop confidence


Testing / Hotsite Best Practices

✓ Examine your hardware schedule in your DR contract

• Check for back level compatibility

• Check capacity

• Do you need additional capacity

• Performance Profile

• Travel

✓ Avoid Testing from special backups

✓ Train your alternates


Prevent a Disaster

Arrive at Hot-site without Keys to Tape Boxes.

• #1 Recovery Problem Today, Incomplete Backups!!!

• Objects starting with Q in QUSRSYS contain critical Recovery Data

• Know your Network Attributes

Do your backups assist you in both a weekend loss and a mid-week loss?


DR Team Travel - Last Minute


Tips

• Test because your Business Depends on it.

• Keep Passwords and Security current

• Tapes should be audited

• Pre-Arrange for serial dependent software Keys

• Ensure software & PTF’s are current

• Test a Mid-Week scenario

• Install latest Backup/Recovery group PTFs


Tips

• Keep your hot-site aware of your hardware profile

• Know where your LIC CD is?

• Have you performed a FULL save since the last upgrade?

• Changed OS/400 commands to run custom programs ie: RSTLIB, RST etc.

• QSECOFR User profile has been modified

• Get to know your Backup & Recovery Guide

• It’s that “Really big fat book”


+ Poor Protection of Infrastructure

+ No Recovery / Communication Plan

+ Unrealistic Recovery Goals

+ DR Plans are in accurate / Non Existent

+ Inadequate Staff to recover

Top Reasons Recoveries fail


+ Mitigated wrong risks or no budget to address risks

+ Unclear Roles and Responsibilities

+ Misaligned priorities in recovery

+ Poorly mapped Interdependencies

+ Alternative Recovery services failed

Top Reasons Recoveries fail


Where to Spend on DR

+ Invest with the RPO/RTO services level you need

+ Invest according to the business value of each application

+ Invest in your own recovery expertise and Procedures

+ Or, Completing outsource your DR


#1 – Ignoring the Problem

Many people are choosing to ignore Disaster Recovery Planning & Testing

• Nothing has happened to my system yet…

• Next year’s Budget !!!!


Reasons Why …Not enough…

• TIME

• EXPERTISE

• MONEY

Must be high enough business priority to take corrective action


Make a commitment to at least one disaster recovery item this year.

Fortune Favors The Prepared.

Book your

next test

now!


The nd, Thank You!