municipal league joint venture disaster recovery project
DESCRIPTION
Municipal League Joint Venture Disaster Recovery Project. Lee M. Mandell, Ph.D. Director of Information Technology and Research/CIO North Carolina League of Municipalities. What is Disaster Recovery?. - PowerPoint PPT PresentationTRANSCRIPT
Municipal League Joint Venture Disaster Recovery Project
Lee M. Mandell, Ph.D.Director of Information Technology and Research/CIO North Carolina League of Municipalities
What is Disaster Recovery?
Disaster Recovery is the ability to recover from a loss of the IT facilities (or access to those facilities) and infrastructure used to conduct daily business.
Why Would You Need It?
The loss of the use of IT facilities can occur for many possible reasons:
hurricanes, floods, fires, tornados, chemical spills, earthquakes, power grid failures, HVAC failures, ice and snow storms, pandemics, terrorism, computer viruses, structural failures, vandalism, and the list can probably go on
DR Is Not BC
A disaster recovery solution should be
part of a comprehensive business continuity planHaving a hardware/software/ communications alternative to a data center is not enoughThis presentation is only about one piece of the DR/BC puzzle and needs to be put into the context of the broader problem and solution
The DR Center Project
The NC League of Municipalities, in partnership with the Municipal Association of SC, and its technical partner VC3, has developed a jointly-owned Disaster Recovery facility for Wintel and compatible systems
Our thanks to HP for technical and financial assistance on this project
Project Overview
Although initially serving only two municipal organizations, the project determined that a methodology and solution existed that would allow multiple Leagues to share the initial cost and overhead of a scalable, centralized solution and allow recovery of business functions in the event of a disaster.
Project Overview
The DR solution developed for this joint project leverages: technology changes that have
occurred and matured in the past few years (server virtualization and imaging software)
the public Internet, and sweat equity
to dramatically reduce the cost and increase the operational effectiveness of a DR facility
Physical Server
Virtual Machines
ESX Server
Server Virtualization Basics
Deploy multiple virtual machines on a single physical server
Unmodified Application
Unmodified OS
Virtual Hardware
Benefits Increase hardware
utilization by sharing hardware resources across a large number of virtual machines.
Each Virtual Machine is a
complete system encapsulated in a
set of software files
Options for Disaster Recovery
Physical to Physical Today’s scenario without
Virtual Infrastructure Requires identical DR site
Physical to Virtual Reduces cost, improves time
to recovery, and increases flexibility
Virtual to Virtual Greatest flexibility, lowest
cost, best time to recovery
Activities to Date
Two sub-committees formed: Technology Committee Business Plan Committee
Technology Committee: Documented initial technical
requirements Designed a technical framework Completed the Proof of Concept
Phase Completed the Pilot Phase Entered the Production Phase
Technology Committee AssumptionsSupport for Windows NT 4.0, 2000 & 2003 ServersDR Site is a “very warm” standby; NOT a live HOT site Data synchronized on a daily basis (in some cases more frequently)Images transferred and access to restored servers over public InternetDR Site initially sized to support one organization at a timeFull recovery achieved within 24 hours remotely; targeted goal and expectation is less than 12 hoursDR site must accommodate firewall needsDR site must accommodate backups after production services have been transferredTwo tests per year per organization at the DR site
Phase Objectives
Proof of Concept Phase Validate the technology Verify data transfer and conversion timings Refine budget and processes
Pilot Phase Utilize live production systems Build a functional environment with a sub-
set of production servers from each League Validate sustained functionality and
reliabilityProduction Phase
Implement final solution with production services from all Leagues
The History of the Solution
August 2005: MASC and NCLM decide to form a joint venture to solve their own DR needsSeptember 2005: High level solution architectedNovember 2005: Proof of concept phase successfully completedJanuary 2006: Initial pilot phase successfully completedApril 2006: Hardware and software refined and orders placedMay 2006: Data Center space identified in Charlotte, NCJune 2006: Data Center space securedJuly 2006: MASC went live with basic DR Solution; recovery test completed successfully
Key Technology ComponentsHardware
Three HP dual processor servers (shared)
• 2.6 Ghz• 16 GB • 1.8 TB
VPN concentrator (shared) Firewall appliance (shared) Gigabit switch (shared) Console KVM Switch (shared) Image Server (low cost/high capacity
data storage) Software
VMWare – Server Virtualization Suite Symantec LiveState – Imaging Software
DR Server Replication
DR Process
Nightly transfers/replication of incremental data and system configuration images to off-site facility: Secure FTP with 3DES encryption
Image production and transfer process automated
Images validated after creation and transfer
Monthly Disk Transports of full images (base size 300 GB): 128 bit AES encryption with encryption at the data and file system level
Remote connection to restored servers at DR Center through Citrix
DR Center Metrics
Creating LiveState images of 11 MASC core virtualized servers: Initial Image: 2 to 2.5 hours 300 GB Daily Incremental Image: 2 to 2.5
hours 3 to 4 GB
Transferring incremental images nightly to DR Site: Incremental Image : 2 Mb/sec (10 Mbs
burst) 3.5 to 4 hours
Remote recovery of all 11 systems from incremental images: 6 to 8 hours (parallel recovery)
DR Cost ComparisonsTraditional Vendor-Provided DR Services Costs:$3,000 to $4,000 per monthCosts per League for five-year planning horizon:$180,000 to $240,000
DR Center Costs:initial shared setup costs at $40,800; hosting charges of $1,150 per month; pre-funded replacement charges of $680 per month; local setup costs at $20,400 to $51,900Costs per League (2) for five-year planning horizon:$95,700 to $127,200
Key Benefits of Basic DR Center Solution
Captures all data and all system state informationAllows remote restoration of all serversUses public Internet for both image transfer and access to restored systemsScales more cost effectively as data grows as compared to other solutionsShares costs and resources for little-used but crucial capabilityEasy to plan and budget the continuing costsDemonstrates that a partnership among like organizations to support joint DR solution can work (formal working/ownership agreement developed and signed)
NCLM Enhanced DR Solution
Reduce the cost and complexity and increase the flexibility and robustness of business continuity by storing entire database and system files on a storage area network (SAN) that can be replicated and restored at DR CenterProvide near real-time, byte-level replication and point-in-time rollback recovery
Provide additional level of data recovery beyond nightly backups at the file/email message levelDirect broadband connection to DR CenterLocate server room appliances at DR CenterWould add around $92,000 to five-year costs, net
Disaster Recovery Using SAN/iQ Remote Copy
Perform disaster recovery using Remote Copy No need to upgrade secondary site server
hardware in lock-step with the primary site Easy to automate and no need for bare metal
recovery tools Virtual image and data all protected under one
DR planPrimary
SiteRecovery Site
The Rest of the Future
Expand basic or enhanced version to other southern municipal leaguesThe basic DR solution also has the promise of being able to provide shared DR services to cities and towns in NC and SC that have historically been cost prohibitive for them to purchase, allowing the recovery of their critical business functions in the event of a disaster
Thank You!