office of the vice president of information technology “ i’ve seen this movie before and i...

Office of the Vice President of Information Technology

Disaster Recovery/Event Management PerspectivesDéjà Vu All Over Again

“ I’ve seen this movie before and I didn’t like it then either!”“ I’ve seen this movie before and I didn’t like it then either!”

Brian D. VossVice President and Chief Information OfficerUniversity of Maryland


Getting a few things straight up front ….

•Linda and the folks in Vermont are the story of this EDUCAUSE Live! Session

•I may be the ‘ghost of disasters past’ but I didn’t experience one, this time, worth discussing in its own right

•My experiences with Irene were mostly of a personal nature – my campus had no issues worth mentioning

•I’m going to remind you, though, that if this one didn’t get you … the next one might! the next one might! So if you haven’t prepared yet, now is another opportunity!


My Calamitous Louisiana Experience

Hurricanes •Katrina (Aug/Sept 2005)•Rita (Sept 2005)•Gustav (Sept 2008 – I was in Alaska)Oil Spill (2010)Mississippi River 100-year Flood Threat (2011 – I was in Paris)Budget Crisis (2009-2011 – I felt was in a Fellini movie!)

I decided not to stick around for fiery hail, frogs, boils, locusts …. Not to mention the urge to always be checking on my daughter (first born) in Chicago daily. So I beat feet to Maryland!


Welcome to Maryland!

Town Hall Meeting with UMD OIT Staff, August 23, 2011 1:51pm EDT

Let’s watch a little video …


An Earthquake???

Yup … An Earthquake!


Thursday Morning 25 August – Still discussing earthquake

Uhh …. Psssssst …. Don’t look now BUT THERE’S A HURRICANE COMING!


Personal Perspectives

Personally•“You’ve got to be kidding me … I just LEFT Hurricane Country!”•Conjured up feelings of personal dread•Just moved into a house … surrounded by 100-ft tall trees100-ft tall trees!•Felt the campus wasn’t getting agitated [enough] about this•Remembered how LSU began significant preparations in earnest 3-4 days ahead of landfall•UMD seemed more focused on the potential disruption and inconvenience of move-in weekend and other eventsProfessionally•New guy – not really well positioned to make waves nor understanding how/where/when to make them•Concerned about coming off as “The BIG [Mr. Knowitall] Expert” and causing a bad impression in the infancy of my time at UMD


Professional Practice

ActionsBuck up and go my colleague, Linda Clement, VP Student Affairs and expressed Buck up and go my colleague, Linda Clement, VP Student Affairs and expressed my concerns as gently and respectfully as possiblemy concerns as gently and respectfully as possible•Discovered …

•Great deal of pre-planning was in place•While mood was not alarmed, it was dedicated, cautious, and circumspect •Key University elements – student life, academic affairs, facilities, campus safety/police, communications and university relations – were already well into their ‘playbooks’

Returned to IT to see how my new organization was preparingReturned to IT to see how my new organization was preparing•Discovered

•Well documented, rehearsed, and detailed plan was being rolled-out and implemented•Disaster Response efforts well structured •Contingencies were well described and emergency action plans ready


And then the storm HIT!!!!

•While Eastern/Coastal Maryland and the area around the Chesapeake Bay certainly had some significant damage and impact, the College Park area primarily had some wind, some rain, and very minor street flooding.•Trees were certainly down around town (and even on my street in Wheaton) but it was not calamitous •Power outages were sporadically impacting specific areas or residences/businesses; but nothing widespread•We were spared•Life returned to normal by Monday morning


We became occupied with other pressing matters ….


Lessons Learned – Revisiting the Past

Post-Katrina Disaster Recovery ThinkingPost-Katrina Disaster Recovery Thinking

Traditional Disaster RecoveryTraditional Disaster RecoveryWhat if my data center is lostWhat if my data center is lost

Broader Disaster RecoveryBroader Disaster RecoveryWhat if my campus is lostWhat if my campus is lostWhat if the city where my campus is located is lostWhat if the city where my campus is located is lost

Survivor Disaster RecoverySurvivor Disaster RecoveryWhat if I’m fine … but everyone around me is notWhat if I’m fine … but everyone around me is not

Traditional Disaster RecoveryTraditional Disaster RecoveryYou’re down, everything else is fineYou’re down, everything else is fine

Do you have a workable DR plan?Do you have a workable DR plan?Do you know where on campus you’ll go?Do you know where on campus you’ll go?Did you take necessary back-ups and do you have them ready to Did you take necessary back-ups and do you have them ready to

re-produce production files?re-produce production files?What vendors will you need to tap – and for what?What vendors will you need to tap – and for what?How will you quickly re-establish network connectivity? Phone How will you quickly re-establish network connectivity? Phone

service? Web presence? E-mail? Mission critical information service? Web presence? E-mail? Mission critical information systems?systems?

Lessons:Lessons:It’s the data, stupid.It’s the data, stupid. Hardware can be replaced; data can not be Hardware can be replaced; data can not beIf you lose just your data center, the need for rapid response will be acute If you lose just your data center, the need for rapid response will be acute

and immediate; your institution can not operate without ITand immediate; your institution can not operate without IT

Broader Disaster RecoveryBroader Disaster RecoveryYou and everyone around you is downYou and everyone around you is down

Are your off-sites conveniently (and perhaps tragically) close?Are your off-sites conveniently (and perhaps tragically) close?Do you have arrangements to get key services restored at a distanceDo you have arrangements to get key services restored at a distance

Web, E-mail, Financial/HR, Student Information, LMSWeb, E-mail, Financial/HR, Student Information, LMSHot-sites may be too much $$$$ – but can you find suitable raised Hot-sites may be too much $$$$ – but can you find suitable raised

floor/HVAC/power to ‘re-build’floor/HVAC/power to ‘re-build’Can you support your administration “in exile?”Can you support your administration “in exile?”

Internet access, computers, cell phones, e-mail, IMInternet access, computers, cell phones, e-mail, IMIs your ‘life-boat’ plan portable over larger distances?Is your ‘life-boat’ plan portable over larger distances?Can you grab your key people? Can you care for them?Can you grab your key people? Can you care for them?Lessons:Lessons:

People are your most key resource – but expect them to be burdened with People are your most key resource – but expect them to be burdened with other prioritiesother priorities

Knowing what you’ll need to do and having it organized is more important Knowing what you’ll need to do and having it organized is more important than knowing all about ‘how’ you’ll do it when you get there (wherever than knowing all about ‘how’ you’ll do it when you get there (wherever ‘there’ might be)‘there’ might be)

Survivor Disaster RecoverySurvivor Disaster RecoveryYou’re the last ones standingYou’re the last ones standing

Dealing with unimaginable demandsDealing with unimaginable demandsStart imagining itStart imagining it

Do you have a stock of equipment to set up a large support Do you have a stock of equipment to set up a large support operation in short-order?operation in short-order?Networking gear, computers, cables, supplies, telephone serviceNetworking gear, computers, cables, supplies, telephone service

Value of a flexible and capable staffValue of a flexible and capable staffThey’ll see things no one should have to seeThey’ll see things no one should have to see

Consider how you’ll do all this on top of your normal jobs, as Consider how you’ll do all this on top of your normal jobs, as campus life resumes and student enrollment increasescampus life resumes and student enrollment increases

How ready is your campus administration to take on the role of How ready is your campus administration to take on the role of disaster response center?disaster response center?Facilities, public safety/police, communications, academic affairsFacilities, public safety/police, communications, academic affairsIs the CEO (Chancellor, or President) prepared?Is the CEO (Chancellor, or President) prepared?

Those who cannot learn from history Those who cannot learn from history are doomed to repeat it.are doomed to repeat it.

-George Santayana-George Santayana

CIOs can no longer say they can’t imagine what could happen – CIOs can no longer say they can’t imagine what could happen – because it just did. because it just did. {AGAIN}{AGAIN}

Next time, you may not be watching it on CNN – you may be Next time, you may not be watching it on CNN – you may be living itliving it

Now is the time to think, plan, and take action – later it will be Now is the time to think, plan, and take action – later it will be too latetoo late

office of the vice president of information technology “ i’ve seen this movie before and i...

Documents

voss vice president

umd slide

little video slide

didnt experience

university relations

campus safetypolice

personal nature

landfall umd