data center consolidation considerations & best practices
TRANSCRIPT
Data CenterConsolidations
Webinar 09.16.11
Bethann Canada, VirginiaPeter Tamayo, WashingtonBaron Rodriguez, PTAC
Data Center Consolidation Considerations & Best Practices
Agenda
• Overview of data center consolidation considerations.
• Virginia’s data consolidation lessons learned – Bethann Canada
• Washington’s data center consolidation effort – Peter Tamayo
• Questions: Please send your questions to AEM Host through the chat box feature.
2
3
Political Reality
• Dwindling state resources
• Political drivers
• Savings?
• Increased security?
• Increased reliability?
• Centralized control of state assets
• Increased capacity needs as a result of SLDS initiatives
Issues to consider…
4
• Physical & Logical Access• Data & System Inventories• Consolidated Data Center Staffing• Business Continuity/Disaster
Recovery Plans
Security Policies
5
• Data Access & Use Policy – This policy specifies data access controls and defines appropriate data use. Such policies often break down specific roles and job responsibilities related to educational data.
• Non-Disclosure Policy – Educational agencies should require individuals who have access to personally identifiable education records to sign affidavits of non-disclosure.
• Data Retention/Destruction Policy – The data center provider generally is responsible for backup and archival of electronic systems. Therefore, a data retention and destruction policy should be crafted based on a careful review of local, state and federal policies.
• Data Breach Policy – This policy outlines procedures to follow in the event of a data breach. It should include references to any applicable state, local, and federal laws and provide appropriate escalation procedures to follow in the event of a data breach.
Other considerations…
• Access/Audit Log Review
• Legal Ramifications
6
“It’s like building your house with your own hands and then watching someone take it apart piece-by-piece with a crowbar.”
Bethann Canada on VDOE’s infrastructure consolidation
2003-20091. Personnel2. Assets3. Network4. Re-IP5. Messaging6. Desktop7. Server
Legislation requires Executive Branch agencies to turn over IT infrastructure to a central IT Agency.
Services are then outsourced to Northrup Grumman.
Governor directs agencies to “play nice”.
The Emperor is still naked…
Service Center Offerings 24x7x365 Security Desk All movement in/out through main lobby Automated access control Controlled visitor access Personnel badges, camera surveillance,
camera analytics Support from local fire and police Backup data center in Southwest VA
Security Offerings Firewalls Intrusion Detection/Prevention Web Proxy for Internet usage and Anti-Virus Email Proxy for Spam and Malware Security Information and Event
Management system Vulnerability and compliance scanning
Network Offerings Single and Dual-factor VPN Site-to-Site VPN Encryption
Router to router Remote Access Site-to-Site VPN E-mail Full Disk Web Proxy
Moving to a consolidated data center: Runbook TOC
Change Log Vendor and Agency Contacts (work, home, cell) Issues Register Resource Plan Application and Database Overview Application Dependencies Backup Overview Server Details Storage Details Network Details Server Runbook Hour-by-Hour Rollback
VDOE’s server move was outsourced to EMC.
Planning, documentation, and execution were extremely thorough and the move was carried out flawlessly.
Moving to a consolidated data center – Hour-by-Hour Plan
Task Posted start time
Estimated duration
Posted completion
time
Estimated Start Time Actual start Percent
completetime In process
Time Remaining
Estimate completion
Actual completion
Application Shutdown
10/16/09 18:00 0:15 10/16/2009
18:1510/16/2009
18:0010/16/09
18:00 100% 2:57 0:00complete 10/16/09 18:15
Database Shutdown
10/16/09 18:00 0:10 10/16/2009
18:1010/16/2009
18:1510/16/09
18:00 100% 2:57 0:00complete 10/16/09 18:15
Server Backup
10/16/09 18:00 10:00 10/17/2009
4:0010/16/2009
18:1510/16/09
18:30 100% 2:27 0:00complete 10/17/09 4:45
Server Shutdown
10/17/09 5:00 4:25 10/17/2009
9:2510/17/2009
4:4510/17/09
4:45 100% 16:12 0:00complete 10/17/09 5:18
Server derack
10/17/09 6:05 3:00 10/17/2009
9:0510/17/2009
5:1810/17/09
5:18 100% 15:39 0:00complete 10/17/09 6:45
load into truck
10/17/09 6:05 3:00 10/17/2009
9:0510/17/2009
3:4510/17/09
6:15 100% 14:42 0:00complete 10/17/09 7:30
Transport equipment to CESC
10/17/09 8:20 1:00 10/17/2009
9:2010/17/2009
7:3010/17/09
7:30 100% 13:27 0:00complete 10/17/09 8:00
Rack equipment
10/17/09 10:20 3:30 10/17/2009
13:5010/17/2009
9:0010/17/09
8:50 100% 12:07 0:00complete 10/17/09 12:02
Start Servers10/17/09 13:50 3:00 10/17/2009
16:5010/17/2009
12:0210/17/09
12:02 100% 8:55 0:00complete 10/17/09 19:24
Start Databases
10/17/09 16:40 0:30 10/17/2009
17:1010/17/2009
19:2410/17/09
14:45 100% 6:12 0:00complete 10/17/09 15:29
Start Applications
10/17/09 17:10 1:30 10/17/2009
18:4010/17/2009
15:2910/17/09
15:50 100% 5:07 0:00complete 10/17/09 18:54
Test Applications
10/17/09 18:40 6:00 10/18/2009
0:4010/17/2009
18:5410/17/09
17:33 100% 3:24 0:00complete 10/17/09 20:24
Moving to a consolidated data center: Hour-by-Hour Rollback Plan Failed lift and ship (truck crashes,
servers destroyed) Database failure Storage failure
3 days Larry Pathrow Request new Storage device 4 hoursLarry Pathrow Install and prepare new array . (initialize the replacement,
carve up the SAN Storage and reallocate the NAS Storage) 1 hour Larry Pathrow Verify/update the zoning on the switches if there are new
WWNs on the storage processors. BackUp Task(s) Karen Lusk Restore Data from Backup Tapes (4 TB data) Application Task(s)
5 mins Application Owner Startup Application
15 mins
Application Owner Test Application
Fast-forward two years… No MOU or SLAs No evidence that backups are taking place 7-month turnaround on work requests for
new servers 2-month turnaround on work requests for
additional storage No knowledge of who has access to data Annual audit points DR cost-prohibitive
Costs 399 PC’s and Laptops $37,422 “help desk” $5,354 357 Mailboxes $5,244 30 Network Printers $1,569 26 Servers $17,776 Storage $28,340
Total monthly bill for June, 2011 $90,969
Benefits Relief – no servers in building
COOP advantage No midnight calls when the AC fails
Shift in focus to business needs Information Management Data Stewardship Accountability Systems Longitudinal Data System
PTAC Webinar, September 16, 2011
18
Fast Facts◦ Number of Schools / Districts / ESDs: 2,253 / 295 / 9◦ Number of K-12 students: 1,023,000◦ Number of K-12 school staff: 90,000◦ Number of SEA employees: 400 staff located in a building built
in 1895◦ State Funding in K-12: More than $6B annually◦ LEAs have Local control◦ Elected Superintendent◦ Dedicated K-20 network and State Government Network◦ Multiple billion dollar budget gaps over the past several years
19September 2011
Why? ◦ IT asset value in excess of $1B◦ State and local governments have to live within their means◦ No growth in near future◦ Opportunities for improving service to constituents◦ Need to be more “Green”
How?◦ Leverage economies of scale to drive down costs◦ Fully utilize assets◦ Standardize infrastructure◦ Consolidate data centers◦ Use the state’s buying power to purchase commoditized products◦ Cloud computing
20September 2011
New WA State Data Center (SDC) and Office Building◦ $255 million dollar construction cost◦ Consolidates at least 32 data centers◦ RFI in Summer 2010 for vendor information on a “turnkey”
approach for data center operations◦ New Data Center will have four data halls within the 50,000 sq.
ft. SDC◦ Office Building completed in June 2011◦ SDC is partially operational serving building staff◦ Plans to migrate agencies to the new data center have lost
momentum due to the reorganization of the state’s IT department
21September 2011
Engage with agency leadership, legal and financial staff and legislators early and often with the K-12 requirements and business cases
Discuss with WA LEAs and other SEAs on lessons learnedo NCES Forum, MIS
Participate with the other WA state agencies and the central IT department in the planning process
o Lift and Shift or “End of Life” replacement migrationo Server virtualization projecto Negotiate preliminary agency migration sequenceo Review and provide feedback to preliminary documents
Utilize sound and rigorous project management practices
22September 2011
23September 2011
Key items on the project plan◦ Review policy, security and legal requirements and perform gap
analysis WA State Policies and Standards: Personnel Security, Physical
Security, Data Security, Network Security, Access Security, Records Retention
◦ Stakeholder Management Coordinate with the K-12 application development staff Coordinate with OSPI program staff and user community Peer communication in a shared environment
◦ Patch hardware with the latest software updates◦ Clean up folders and files◦ Freeze environment
24September 2011
Key items on the project plan (continued)◦ Update IT Portfolio and other as-is and to-be documentation◦ Conduct proof-of-concept projects, pilots or other phased-in
approaches◦ Develop test plans◦ Develop cut-over and support plan with updated SLAs◦ Update and exercise Business Continuity Plans ◦ Implement a safety net
Full backups Develop roll-back plans Get another set of eyes on the plans and documentation
25September 2011
Peter Tamayo, Chief Information Officer◦[email protected]◦(360) 725-6134
26September 2011
Data Center Consolidations – Final Thoughts
• Efforts are partnerships!
• Work with your state attorney general to ensure compliance with federal, state, and local laws.
• Contact the Family Policy Compliance Office should you need assistance.
27
PTAC Help Desk & Website
28
Send PTAC your questions on privacy, confidentiality, and data security related to longitudinal data systems (LDSs)
Contact the Help Desko [email protected]
o Toll Free Phone: 855-249-3072
o Toll Free FAX: 855-249-3073
Get copies of PTAC resources, join our email list @ http://nces.ed.gov/programs/ptac
Data Center Consolidation
Webinar 09.16.11
Data Center Consolidation Considerations & Best Practices
Thank You for Participating