lu - acuta 2009 conference presentation
DESCRIPTION
TRANSCRIPT
Lessons Learned From the Texas Gulf Coast
ACUTA 2009Palm Springs, California
About Your SpeakersTexas State University System
Dr. Michael E. Dobe, Sr., Associate Vice President, Chief Information Officer & Adjunct Professor at Lamar University (Beaumont, Texas)
http://www.linkedin.com/in/mdobe Texas A & M System
Mr. Steven Conway, Director of Computing and Info Services at Texas A&M University at Galveston (Galveston, Texas)
http://www.linkedin.com/pub/5/b7/24a
About the Checklist
Steven Conway
Texas A&M University at Galveston (TAMUG)
ACUTA/ATIS Hurricane Preparedness Project
Formed Committee May 2007Reviewed and Modified ATIS documentPublished ACUTA Hurricane Checklist Oct
2007
http://www.acuta.org/?2026
Hurricane Preparedness
Checklist
ACUTA - The Association for Information Communications Technology Professionals in Higher Education
Developed in cooperation withATIS – The Alliance for Telecommunications Industry Solutions
Hurricane Katrina, August 2005, Terra MODIS True Color Image
Courtesy of Louisiana State University
Committee Amy Phillips, Louisiana State University Health Sciences Center Frank O'Quinn, Louisiana State University Sandra Thompson, University of West Florida James Shea, Boston University Jeanne Spinosa, Johnson & Wales University Garret Yoshimi, University of Hawaii George Denbow, University of Texas at Austin Michele Narcavage, University of Pennsylvania Steve Ellis, University of Notre Dame Steve Conway, Texas A&M University at Galveston Walt Magnussen, Texas A&M University - Task Force Chair Veronica Lancaster, ATIS Brad Ruark, ATIS /AT&T Jeri Semer, ACUTA Executive Director Aaron Fuehrer, ACUTA Staff Liaison
ACUTA Hurricane Preparedness Checklist
Using the Checklist As An After-Action Review Tool
Perspectives From Beaumont
Dr. Michael Dobe, Sr.Lamar University
About Lamar UniversityLocated in Beaumont, TX (Approx 40 Minutes
from Louisiana Border)Central Data Center in Beaumont Serves
Three Campuses:Lamar UniversityLamar Institute of TechnologyLamar State College- Orange
Total Students Served – Approximately 16,000
Proximity of Beaumont and Galveston
119 mi – about 1 hour 57 mins
Rita, Edouard, Gustav, IkeHurricane Activity in SE Texas
Rita – September 2005 (CAT 4 With Major Damage to Beaumont)
2006 and 2007 Atlantic Hurricane Seasons Quiet in SE Texas
Edouard – August 2008 (Missed SE Texas)Gustav – September 1, 2008 (Minor Impact in
SE Texas)Ike Hit SE Texas Less Than Two Weeks After
Gustav…
Hurricane Ike’s Path
http://en.wikipedia.org/wiki/File:Ike_2008_track.png
Hurricane Ike Timeline Lamar evacuated campus 10 Sept 2008Hurricane Ike landfall Evening of 12-13 Sept 2008Low Lying Areas of Beaumont Flooded, Extensive
Flooding in Orange County, Bridge City Hardest HitLamar University Campus in Beaumont Did Not FloodLibrary, Family and Consumer Sciences, and
Montaigne Center Buildings Had Roof Damage24 Sept 2008 Essential Staff Returned to Campus25 Sept 2008 Classes Resumed at Lamar University
Assessing Services During IkeLU Lessons LearnedStorm FactsContact ListsEmergency Sites and
LodgingAdministration,
Communications, and Manpower
Safety, Security and Training
InventoryPhysical AssetsPower and FuelNetwork ServiceMain Computer Center
and Key Telecommunications Hubs
Long-Term PreparationsPost-Storm ActivitiesDamage Assessment and
Post-Storm Activities
Storm Facts NOAA WebsiteMajor News Media: CNN, Weather Channel
and Storm Pulse (http://www.stormpulse.com/)
Importance of Local News Media: KFDM and Entergy
Government Contacts: Jefferson County and FEMA
Local Contacts: State Police and Beaumont Police
Contact Lists Updated and Distributed Internal Contacts
For GustavLeadership Team Phone ListPocket Card for Satellite Phone NumbersITS Contact List
External Contact Lists Maintained By Staff Sections
Emergency Sites and Lodging Emergency Control Site Remained on
Campus in BeaumontTemporary Lodging and Shelter Arranged by
IndividualsDisaster Recovery/Business Continuity Site at
San Marcos State University
Texas State University Systemhttp://www.tsus.edu/
TSUS Institutions (East Texas Locations in Bold)Lamar University Sam Houston State University Sul Ross State University Sul Ross State University Rio Grande
College Texas State University-San Marcos Lamar Institute of Technology Lamar State College-Orange Lamar State College-Port Arthur
Sam Houston State in Hunstville
153 mi – about 2 hours 29 mins
TOO CLOSE FOR COMFORT
Stephen F. Austin State
128 mi – about 2 hours 36 mins
NOT MEMBER OF TSUS
San Marcos State University
251 mi – about 4 hours 15 mins
During Ike – Approx. 16 hrs
LAMAR UNIVERSITY BC Site
Administration, Communications and Manpower Importance of Clear Chain of Command
and Clear Delineation of ResponsibilitiesITS Set Up Two Teams: Offsite B/C Team
and Onsite Command Center TeamCommunication with Employees via LU
Website and ConnectED (Now Blackboard Connect) to E-Mail and Phones.
Communications Methods: Cell Phones, Satellite Phones, Land Lines and SIP Phones for Key Employees
ACUTA NOTE: Effective Leadership Was Key to Success
ACUTA Note: Importance of Coordination of Operations Functions
ACUTA NOTE: Importance of Coordination of the Academic Response
Administration, Communications and Manpower ITS Hunkered Down With LUPD in Beaumont
Daily Meetings for Coordination in Beaumont on Campus
Reporting and Notification Lead By President and Leadership Team
Conference Bridges For Coordination With Remote Staff Managed by Network Services
Public Coordination Handled by LU PDManaged Through Office of VP for Finance and Amin
Supplemental PersonnelAccountability and Time ReportingCosts
LU IT Services Staff DispositionLocal Core Team Stayed on Campus
ThroughoutRemote Core Team Relocated to San Marcos
Area – Worked on Ordering Replacement Systems
Remaining Staff Dispersed With FamiliesAT&T Conference Bridge to Maintain Contact
With Dispersed Staff
Onsite IT Service Lead: John Genuardi, Customer Support Director
Onsite Network Support: Patrick Stewart, Manager of Network Services
Onsite Data Center Lead: Srinivas “Sri” Vandaraj, Senior Security Analyst
Safety, Security and Training President’s Weekly Security Meeting
Initiated as Outcome of RitaSystem Access Managed By Onsite Core ITS
Team (Issues With Passwords: Need One-Time Password System)
Physical Re-Entry Credentials Provided by Chief of Police
ITS and PD Coordinated Security Cameras Shutdown and Start-upTraining for ConnectedED
Inventory and Physical AssetsImpact of Key Staff ResignationPlanning: AVP (IT Infrastructure and Security)
Responsible for Availability of Emergency Supplies and Work Supplies
AVP Had Resigned Between Gustav and Ike!Single Point of Staff Failure – Transitioned
Responsibilities Quickly …Equipment Inventory, To Include Network and
Data Center Backup and Recovery Plans Needed to Be Transitioned ON THE FLY! (New Role of Onsite Lead)
Note on Vehicle Planning: Golf Carts Proved Critical After Storm
Power and Fuel: Handled By Facilities ManagementStaff Continuity Maintained - Gustav to Ike Communication with Local Fuel VendorsCommunication and Coordination With Power
CompaniesEngine/Genset (portable and permanent)Fuel TanksBatteriesRectifiersStaging and Deployment
Network ServicePreventative Measures:
No Redundant Routes for Network (Big Exposure) LEARN Connectivity Initiative Q 1 of 2009Half-Completed VoIP Migration (Half Cisco/Half
Avaya)Priority Areas:
Network Services Spread Between Windows Services Group (DHCP) and Unix Group (DNS).
Failure to Standardize Led to Confusion and LAN Service Outages During the Storm (Premature AD Shutdown).
Failure to Document All Services Creates Risk
Significant Telecomm EventsGOT LUCKY -- LU OC-3 Circuit to Houston Stayed Up
Throughout Event – NO INTERNET CONNECTIVITY OUTAGE (Gigaman to Orange Also Stayed Up)
Onsite Team Kept Campus Services Running Until Night of Landfall (Sept. 12-13)
Throughout Storm VoIP access via WAN using SIP Phones (Ordered More Broadband Cards After Storm)
Cell Service Unreliable Beaumont to Remote SitesResidential lines out for over several weeksSatellite Phones Proved Difficult to Use
Main Computer Center and Key Telecommunications Hubs Success Story: No Systems Shutdown
Sequence in Place For Data Centers. Planning Initiated in Gustav Continued Into Ike Lead Up
Downside to Not Running Your Own Power:Power Failure Exposure Due to Unmet Power
Requirements in Data CenterConcerns re: Lack of Fuel Led Local Team to Bring
Down Systems After EventTape Backups: Contract for Off Site Had Just
Lapsed! Can You Sing “Do the Hustle”?Wind and High Water: Failure to Stock Pile
Sandbags
Long Term Preparations Maintain Year-Round
Example of Successful Planning: Replication of Website and Content Management Service
Criticality of ERP Disaster Recovery/BC Site at San Marcos
Hardware In Place in San Marcos Software to Be Configured With Data Replication (Tested)
By Hurricane Season 2009Long-Term Plans/Prevention
Move to the Cloud: Portal and Mail Outsourcing Initiatives Importance of Service Continuity: Redesign of Network
Services (Standardization Under Network Services Group)
Post-Storm Activities Led By Onsite Team Lead Employee SafetyContactsConference BridgesCostsTemporary Lodging/ShelterExtra Work ForceSafetyEmergency Supply AvailabilityPower and Fuel
Flooding in Low-Lying Areas
Damage to People’s Homes
Damage to Local Businesses
Damage Assessment and Service Restoration Announcement Via ConnectED on Return of
Staff to Campus (September 24th), Return of Students to Campus (September 25th)
Class Schedule Readjusted by Office of the Provost, With Resumption of Classes on September 25th
President’s Faculty/Staff Convocation on September 25th
Perspectives From Galveston
Steven Conway
Texas A&M University at Galveston (TAMUG)
Hurricane Ike TAMUG evacuated campus 10 Sept 2008Hurricane Ike landfall 12 Sept 200875% of Galveston was floodedTAMUG did not flood, 2 buildings had roof
damageTAMUG restarted classes at TAMU in College
Station TX Wed 24 Sept91% of the students returned20 Jan 2008 Classes resume in Galveston
Significant Telecomm EventsCell service stayed up throughout storm-Cat 2UTMB Gigaman Circuit to Houston stayed upTAMUG-UTMB Gigaman available 24 Sept
when power returned on campusT1’s out for over a monthResidential lines out for over 6 weeksVoIP access to College Station via WAN
Hurricane Ike Flooding Galveston
A house burns uncontrolled in a flooded neighborhood as Hurricane Ike approaches the Texas coast, Friday, Sept. 12, 2008, in Galveston, Texas. (AP Photo/Matt Slocum)
TAMUG Boat on Causeway