Server Virtualization:Navy Network Operations Centers

Lucas TaberJuly 13th, 2006

University of San DiegoGSBA 573 IT Project Management

Introduction

The Navy maintains several Network Operations Centers (NOCs) that are critical to their mission.

The NOCs are overwhelmed by the rapid growth of information system: Space Heat Power

This project will address this issue through the application of server virtualization technology.

Problem

Navy NOC #1 has reached capacity ahead of schedule.

Plant facilities require immediate and expensive upgrades to support additional systems.

Modernized systems must be installed to reduce costs and improve capabilities, but existing systems cannot be immediately decommissioned.

Operating at capacity is an operational risk to the Navy.

Project Objectives Implement server virtualization technology

to reduce the number of physical servers:

Eliminate the risk of systems failure Overheating Power Failure

Eliminate the cost of interim fixes and schedule delay Keep Projects on Schedule

Reduce the cost of future installations. Future Server Deployment Future Facility Upgrades (NOC #2)

Stakeholders

The facilities management team at NOCs 1 and 2. The operations staff. The engineering team. The shore modernization team. The Fleet Readiness Certification Board (FRCB). The security Certification and Accreditation (C&A)

team. The Navy Designated Approval Authority (DAA). All Navy Customers.

Project Justification

Current server utilization is very low. Industry estimate = 5% on average. Additional capacity is wasted and wastes

resources. Servers can be consolidated on a 10 to 1

ratio (maximum). Navy estimate = 4 to 1

Consolidation can recover as much as 75% of the current facilities resources. Cost Avoidance Risk Reduction

Project Requirements

1. The project must achieve a minimum server consolidation ratio of 2 to 1.

2. Consolidated server hardware resource utilization will not exceed 80%.

3. Navy Designated Approval Authority (DAA) must approve the security certification and accreditation package.

4. Fleet Readiness Certification Board (FRCB) must approve the installation.

5. Consolidation efforts must not degrade the level of service provided by the servers or increase long-term operational risk.

6. Overall maintenance costs must decrease within the following fiscal year.

7. Virtualization technology will only be deployed at the two primary NOCs.

8. No server installations will be performed to achieve the consolidation, only existing server hardware can be used.

9. Additional Storage Area Network (SAN) materials can be procured if cost savings can still be achieved.

10. Planned system downtime will not exceed three hours.

Deliverables

Project Management-Related Deliverables Work Breakdown Structure (WBS) Plan of Action and Milestones (POA&M) Project Budget Weekly Activity Reports Bill of Materials

Product Related Deliverables Concept of Operations Site Survey Report Test Plan Document Test Plan Results System Security Authorization Agreement (SSAA) Package Installation Guide Training Curriculum Implementation Plan Recommended Standard Operating Procedures (RSOP) System Operation Verification Test (SOVT) Trip Report / Lessons Learned Virtualized Server Systems

Success Criteria

Success Criteria The project is completed by February of

2007 (prior to the next scheduled installation).

The facilities team for NOC 1 can avoid an emergency facilities upgrade.

Operational risk is the same or lower. No additional manpower is required to

operate the new software.

WBS: Phase 1 - Concept

Concept Hold Kick-Off Meeting Identify Potential Solutions

Prepare Analysis of Alternatives Contact Product Vendors Research on the Internet

Generate Preliminary Cost Estimate Write Concept of Operations

Write Document Perform Document Q & A

WBS: Phase 2 - Development Development

Engineering Select Software Product and Software Version Select Server Hardware or Identify Existing Resources Perform Site Survey of NOC Identify Server Virtualization Candidates Generate Site Survey Report

Write Document Perform Document Q & A

Develop Implementation Plan Write Document Perform Document Q & A Review Implementation Plan With Site Obtain Implementation Plan Approval From Site

Write Installation Guide Write Document Perform Document Q & A

Write Recommended Standard Operating Procedures (RSOP) Write Document Perform Document Q & A

Write Systems Operation Verification Test (SOVT) Write Document Perform Document Q & A

Lab Testing Download and Assemble Software Write Test Plan

Write Document Perform Document Q & A

Install Software in Lab Execute Test Plan Produce Test Results and Analysis

WBS: Phase 2 - Development Security

Perform independent security evaluation Generate security test results and analysis Write SSAA

Write Document Perform Document Q & A

Submit SSAA to DAA Obtain Authority To Operate (ATO)

Training Develop Training Curriculum

Write Document Perform Document Q & A

Train Site Operators Obtain FRCB Fielding Approval

Generate CASE Form Write Document Perform Document Q & A

Procurement Generate Budgetary Cost Estimate Obtain Funding Procure Software and Hardware Receive and Process Software and Hardware

Record Products in Configuration Management Database Deliver Software to Installers Ship Hardware to Site.

WBS: Phase 3 - Implementation

Implementation Travel to NOC Perform In-brief with Site Hardware and Software Implementation

Measure Existing Resource Utilization Install New Hardware Install Virtualization Software Virtualize Existing Servers and Migrate to New

Hardware Monitor Performance

Site Acceptance Conduct SOVT Obtain Site Acceptance

Perform Out-brief with Site Return Travel

WBS: Phase 4 – Close Out

Close-Out Generate Trip Report / Lessons

Learned Write Document Perform Document Q & A

Transition Program to Operations

Staffing

Project Manager Lead Engineer Test Engineer Technical Writer Security Analyst Security Tester Trainer Buyer Installer Logistician

Schedule

Communication Management

Daily Coordination Email

Weekly Progress Monitoring Staff Meeting / Teleconference Weekly Action Report

Project Details / Requirements Formalized Documentation (e.g.,

CONOPS, Test Plan, SOVT).

Risk Management

The project will use a risk register to monitor risk areas and apply mitigation strategies.

Rank Risk Probability Mitigation 1 Project Fails Security Testing 15% C&A team is being consulted on 2 Schedule Overrun 10% Weekly status meetings to ID problems 3 Sponsor cuts funding 5% Provide regular progress updates to

sponsor to address sponsor concerns. 4 Product does not scale as

expected 3% Perform load testing during the lab test

to ensure the product scales under load.

Expected Benefits

Short-term benefits

Recovered rack space*

Recovered server hardware (addl. spares) *

Reduction in thermal output *

Eliminate delays in current system fielding *

Avoid emergency upgrade to NOC 1 $500,000.00

Long-term benefits

Reduced monthly power costs$250,000.00/yr.

Reduce the cost of upgrading NOC 2 *

Reduced future deployment costs *

Expected Costs

Short-term Costs

Project Labor*

Government Pass-Through Fees *

New Hardware $50,000

Virtualization Software (100 processors) $575,000.00

Long-term Costs

Software Maintenance$120,750.00/yr

Summary

An rapid deployment of server virtualization technology will provide significant benefit to the Navy. Cost Avoidance: Present and Future Risk Reduction Increased Capacity