Disaster Planning Items Even Experienced Planners OverlookDisaster Planning Items Even

Experienced Planners Overlook

Steven Lewis, Ph.D., CISA

The Systems Audit Group, Inc.Edwards Disaster Recovery Directorytm

25 Ellison Rd, Newton, MA 02459 Tel: 617-332-3496 FAX: 617-332-4358

DRYP@RCN.comwww.Risk-Help.com

2

Introduction – 20 years into DR/BC, what have we learned

Everybody forgets something

Usually because it’s right underfoot

But also because changes have happened to the way we work

3

Introduction - organizations have changed in major ways:

Mergers/acquisitions, new offices

New equipment, local/wide-area networks, advanced in-house equipment

New functions & on-line services including home-banking & other automated services, move into “7x24” operations

INTERNET-based services & products bring new vulnerabilities

4

Introduction - What hasn’t changed: Three Dimensions of Disaster

Loss of INFORMATION• (eg, due to virus, hacker, software bugs, etc)

Loss of ACCESS• (eg, due to flood, storm, quarantine etc.)

Loss of PEOPLE• (eg, due to illness, reserve call-up, fear of

pandemic, etc)

5

Introduction - three recovery time periods

Immediate: (up to a week)• set up command center• alert & begin transfer to hotsite or transfer of

mobile units (if applicable)• service to customers via ATM’s (if available)• service to customers at other offices (if any)

6

Introduction - three recovery time periods, cont.

Short-term: (up to a month)• re-assign personnel to most crucial tasks• arrange for long-term-temporary space• begin dial-up service to hotsite (if applicable)• order new computer & network equipment

7

Introduction - three recovery time periods, cont.

Long-term: (up to 6 months)• re-establish all functions• refurbish and prepare permanent site• install new computer & network equipment

8

So, what are we forgetting:So, what are we forgetting:

9

Telephone terminators vulnerable Telephone terminators vulnerable to falling waterto falling water

• Exposed wall panels where telephone wires enter the building – usually in basement.

• Falling water follows the cables and “burns out” the connectors.

• Frequently occurs in windstorms and flooding from nearby fire department activity.

• No heat buildup, so only need to enclose, similar to electric “boxes.”

10

••Vulnerability of network Vulnerability of network ““racksracks””to sprinklersto sprinklers

• Most computer networking racks are exposed in order to reduce heat buildup.

• Often located under a bathroom.

• However, they are not shielded from nearby sprinkler heads or water coming from above the ceiling and following the cables down to the rack.

• Need to provide both cover and airflow.

• At least, hang cables in a “U” away from rack.

11

Doors that need a key to open Doors that need a key to open from the insidefrom the inside

• Doors with deadbolts - usually in older buildings.

• Organizations with “public/non-public” hours (banks, govnts., etc): at the end of “public” hours, employees are still inside for many hours and can be trapped in back areas.

• Every exit door needs to be operable from the inside without a key.

12

Vulnerable documents in nonVulnerable documents in non--fireproof safes/displaysfireproof safes/displays

• Often use non-fireproof safes to protect key legal, personnel, currency and other documents.

• Safes are burglar-proof; most are not insulated, and in a fire, the contents are incinerated.

13

Vulnerable documents in nonVulnerable documents in non--fireproof safes/displays, cont.fireproof safes/displays, cont.

• Artwork on walls

• Antique displays

14

Not providing everyNot providing every--day, day, ““lowlow--techtech”” paper itemspaper items

The two most important paper documents in any disaster are:

– company checks and purchase orders.

A supply must be kept off-site, ready to purchase critical equipment and supplies.

Also checks for main-office-based subsidiaries like Trust Operations

15

Unique Unique ““decorationsdecorations”” -- not not protected protected andand not insured!not insured!

• Many organizations have valuable paintings, displays, currency collections, and antiques on display in their facilities.

• Often these are not covered by insurance riders for danger or loss.

16

Helping employees in their Helping employees in their personal planningpersonal planning

• If they haven’t prepared for disaster themselves, they won’t be there to help you in yours!

• Do they live in a flood zone? Need special medicine from home, have elders/pets to care for, etc, etc

17

EmployeesEmployees’’ relevant personalrelevant personal--life life situationssituations

Often, require key employees to relocate to a computer hotsite or company command center for the duration of an emergency.

Can they do that?• Single parents• Caregivers• No car/depend on public transportation or

carpool

18

EmployeesEmployees’’ relevant personalrelevant personal--life life situations, cont.situations, cont.

Need to know, but need to maintain privacy

• Yearly, have employees sign an affidavit

• May have to change the plan to get them to sign.

19

Monitoring AMonitoring Atypical situationstypical situations

Temporarily disabled employees • Need help exiting• Mobile but unable to carry key materials

Short-term construction projects close exits

Need reporting process to identify & cope with these situations.

20

Skipping Skipping ““backgroundbackground””departmentsdepartments

All organizations depend on a series of support functions, including:• mail delivery, • check printing,• voicemail, • janitorial, • Exterminators,• Personnel/payroll, etc.

21

Skipping Skipping ““backgroundbackground””departmentsdepartments, cont., cont.

Can overlook their dependencies, such as• Couriers & other delivery agencies• Storage of dangerous chemicals• Voicemail retrieval• Payroll input processing

22

Planning for afterPlanning for after--hours hours operationsoperations

Many organizations offer services around the clock:

• example: bank night depositories for large commercial customers

Often, there are no offsite lists of contact numbers to notify users if facilities becomes unavailable.

23

Outside emergency organizations Outside emergency organizations --do they know you?do they know you?

Due to consolidation, many new far-flung locations. Local fire and police departments may not have up-to-date:• contact information • copies of building plans, etcplanners should obtain yearly confirmation that

all local site-relevant information has been given to the local authorities.

24

Outside emergency organizations Outside emergency organizations --do you know them?do you know them?

- Who will issue your local bird-fluquarantine?

- Who are your contacts at:red crosscounty emerg. mgmnt. agencyState emerg. mgmnt agencylocal police & fire, etc, etc

25

Skipping a Skipping a ““second set of eyessecond set of eyes””reviewreview

Internal “stakeholders” finally agree on the planbut results are often myopic & too inward-oriented.

Planners need to get an informal “outside audit” by knowledgeable people from outside of the organization.• Former employees• Parent organization• Other franchisers, colleagues etc.

26

Not locating outNot locating out--ofof--region backregion back--up up vendorsvendors

When large-scale disasters strike, the vendors of many common items & services become overwhelmed:• electric generators • “pumping-out flooded basements”• Print new forms, etc.

Planners need to have out-of-region backup second-sources located beforehand.

27

Not locating regionally diverse Not locating regionally diverse alternate vendorsalternate vendors

1995 Kobe earthquake - many companies had both primary and secondary vendors in the Kobe area.

28

Not absorbing lessons from past Not absorbing lessons from past crisescrises

All organizations have had major disasters that have entered into their corporate lore (eg., cars in the lobby, total blackouts, broken sump pumps, etc)

However, these organizations typically fail to document and implement the ‘fixes’ for:• went wrong, • what went right, • what they've learned, • and what they need to change for the future.

29

Ignoring nearby risks and Ignoring nearby risks and limitationslimitations

Near focus of demonstrations or potential targets of violence?

Located on an Army base, or site which can be closed to the non-military public.

Planners should acquire a wider perspective on these sensitive sites from region-wide contingency management organizations• NEDRIX (New England Disaster Recovery Information

Exchange), • Assn. of Contingency Planners, etc.

30

Not asking if your Not asking if your ““outsourcingoutsourcing””provider is a terror target?provider is a terror target?

In this age of outsourcing, many organizations have moved call centers and other operations to foreign countries.

A popular outsourcing city turned up on a terror organizations “attack list.”

31

Not considering that all employees Not considering that all employees may not survive the disastermay not survive the disaster

After 9/11, many organizations had provided for backup operations, but their employees had all perished.

You need to know where to get replacement employees or contract workers who can take up the slack in such a situation.

If you can’t, then you’ve identified another key vulnerability for your organization.

32

Assuming youAssuming you’’re important to the re important to the utility companiesutility companies

Many planners simply assume that because they are a bank/nursing home/food retailer, etc, the utility companies will assign them a high priority in their recovery operations.

During the 2004 hurricane devastation in Florida, many found out that wasn’t true.

33

Assuming your generatorAssuming your generator’’s fuel s fuel supply wonsupply won’’t run outt run out

Most diesel-powered generators only have about a three-day supply of fuel in their supply tanks.

Often, during a disaster, areas become inaccessible to fuel trucks for longer periods of time than that, and alternate plans need to be drawn up for that eventuality.

34

Assuming that airplanes will Assuming that airplanes will always be availablealways be available

Many organizations are dependent upon air deliveries of their products (example, drug companies running evaluation tests.

Others are dependent upon air deliveries of replacement computers and other parts in a disaster–just the time when airports might be closed.

35

Not specifying Not specifying ““dead in the waterdead in the water””timestimes in your plansin your plans

How long are you willing to not:• Answer the phone• Provide Internet banking, if you’re a bank• Provide Internet product ordering• Have a main office, • etc.

36

Not reading your 3Not reading your 3rdrd-- partyparty--vendors disaster plansvendors disaster plans

- Payroll,- Operations service bureau for banks,- Internet Service Providers (ISP’s),- etc.

37

Not reading your 3Not reading your 3rdrd-- partyparty--vendors disaster plans, cont.vendors disaster plans, cont.

Often will have a stated response time that you can’t live with – like a week, when you need 2 days

Sometimes are unrealistic in how quickly they will really get back up and running

- often takes 2 days just to assess damages

38

Not Not ““proving that your disaster proving that your disaster plan will workplan will work

Four types of tests”• “Blink” test• “Expert” audit and scenario walk-through’s• Component testing• “Pull-the-plug” testing

39

Neglecting to test the plan and do Neglecting to test the plan and do what it sayswhat it says

A well thought-out plan still has to be exercised to make sure all the parts are in place.

Actually buy the back-up generator listed in the plan.

Be sure the off-site copies of the computer back-ups are tested on a regular basis.

Also, be sure your overall plan is distributed to all the peoplewho need it.

40

Ignoring Internet considerations

How’s your web presence doing…

This can create a PR disaster.

Needs to be a process in place to monitor this so you know first and fast.

41

42

43

INTERNET considerations, cont:

violation of regulatory restrictions, including:• Inadvertently soliciting business where not

licensedNeed reporting scheme to regularly check for &

cope with these situations.

44

Forgetting “Magic Number 23”

There’s only a 23% probability of the disaster happening during business hours.

What if can’t locate or reach people?Hurricane Hugo took some groups a week to contact.

What if they’ve been affected themselves?What if you can’t get a dial tone?

People need to know what to do, even if they’re notcontacted!

45

Ignoring pandemic planning

Go to: www.pandemicflu.govwww.pandemicflu.govAssign someone to check on employees

vaccinationsEvaluate offering vaccinationsReview policies on med. co-pays & sick leaveBut, be cautious about loss of

credibility/swine flu

46

Pre-approved press releases

During disaster isn’t the time to compose creative communications

Need to assure the public immediately that you’re still in business!

The top person won’t be there to approve the wording when disaster strikes

47

In conclusion

• Really “scenario” it through

• In-advance decision-making

• Learn from your own & others’ experiences

• Know where to get the help you’ll need . . .

www.EdwardsInformation.com