high performance network analysis

© 2011 Cisco and/or its affiliates. All rights reserved. 1

Enterprise Operate PracticeCisco Services

Andrew Wojtkowiak – Network Consulting Engineer

High Performance Network Analysis


Getting Started• Background

Cisco Services performed an assessment of the wired infrastructure to serve as a holistic health check of the University Corporation of Atmospheric Research network

• Goal of the assessmentTo identify immediate remediation needsProvide Opportunities for network improvement


Discussion Flow

High Level Findings

Strengths and Concerns

BackgroundAnd

Key Areas Assessed

Executive Level

Findings

Encompassing Projects

RemediationSteps

Looking Forward


Background• The High Performance Network Analysis (HPNA) was performed to assure the stability of

the core routing and switching infrastructure

• Performed as a holistic network health check

• Emphasis placed on Availability and Resiliency with the Campus environments

• On-site interviews and data collection

• Analyzed ~80 devices as part of the HPNA

• Collected detailed network data such as topology diagrams, software, network standards, protocols, etc…


Key Areas Addressed• Network Topology

• Protocol Resiliency

• Network Service Resiliency

• Hardware and Software


Analysis Findings


Strengths• Dedicated and professional network staff

Everyone we worked with was very open, professional and accommodating

• Excellent Hardware and Software replacement strategiesHardware and Software is kept up to date and staff is knowledgeable of bugs and vulnerabilities

• Change Management ProcessWell documented and followed change management process

• Individualized tools for Network ManagementTools for deployments, configurations, backups, and management


Concerns• Single Points of Failure

Increased risk of a pervasive network incident; scalability and availability concerns

• Process DocumentationLack of formal process to follow. No repeatable steps that all team members can use.

• Global Configuration TemplatesTemplates will help reduce configuration inconsistencies and ensure services are configured according to policy

• Configuration InconsistenciesIncreased time to repair due to troubleshooting overhead; decreased network security; compliance risk


• A few single points of failure

TCOM switch for internet connectivity

Foothills Lab secondary switch

NWSC second switch

• Major risk with TCOMHigher latency backup

• Foothills under construction, second switch in move

• NWSC secondary switch is being considered

• Foothills and NWSC would limit connectivity from those locations to the rest of the network.

Current State Network Risks Financial Risks

Single Points of Failure


Example Single Point of Failure


• Processes are well defined by the individuals who perform the tasks

Software and Hardware replacement

Standards for implementing new devices

• No actual defined documentation

• Only certain people are well versed in processes

• Not easily reproducible

• No defined steps for changes

• Allocate time to turn processes into documentation

• Allocate someone to review the documents

• Keep them up to date as they change.

Current State Network Risks Recommendations

Process Documentation


1 23

1

5

2

15

Cisco 6500 Series Switches IOS

12.2(33)SXH3

12.2(33)SXH4

12.2(33)SXH8

12.2(33)SXI4a

12.2(33)SXI5

12.2(33)SXI6

12.2(33)SXI8a

Software Resiliency Findings

All CatOS has reached End of SW Maintenance, and will no longer receive attention with regards to defect or security vulnerability patching

8.667%

8.433%

Total CatOS Summary

8.6

8.4


• Configuration standards are adhoc; without formal documentation

• No way to perform configuration compliance to a template*

• Number of configuration inconsistencies and errors (Protocol, Service, Security)

• Network unpredictability

• Potential increased troubleshooting overhead and operational difficulty

• Prolonged loss of connectivity and service interruption to critical applications

• Increased exposure to security vulnerabilities

• Increased cost associated with operating the network


Global Configuration Templates


• HSRP inconsistencies

• Partially configured advanced spanning tree features

• Optimize/Standardize Spanning-tree priorities

• OSPF passive interface

• Some routers do not have a peer

• Possible loops or rouge switches influencing the network

• Routing updates are not limited

• Implement changes to the network to remediate the smaller configuration inconsistencies

• The standard templates will assist in ensuring fewer deviations from standard.


Configuration Inconsistencies


Other Considerations


• Three buildings connected in a partial mesh topology

• Collapsed connections to each other

• Port density growth at N*(N-1) rate for every new building

• Lack of modularity and scalability

• Large fault domains across all buildings

• Network disruption and outages

• Increased troubleshooting overhead

• Quantifiable cost increase in both capital and operational expenditure


Implement a Standalone Network Core

Additional Capital Expenditure associated with running fiber

Nx(N-1) = 12 Ports (6 Links)

Cost to Add 4th Building

Additional Operational Expenditure associated with design complexity


Need for CoreCurrent Topology - No Core• Fully-meshed distribution layers• Physical cabling requirement• Routing complexity


Recommended Design

This leading practice hierarchical design has been proven to:

Promote easy growth and ease of troubleshooting

Reduce capital and operational expenditure

Create small fault domains

Promote deterministic traffic flows

Enable logical and physical topology mapping

Center Green

Dedicated WAN / Internet Switch Block

Mesa Lab Foothills

New Location

TCOM/FRGPResearch Networks

FirewallsInternet

Dedicated Core


• Monitoring facing the Internet

Intrusion Prevention

SPAN Sessions to security team

• Extensive ACLs on core switches

• No Control Plane Policing to protect devices

• Limited methods to log and account for network incidents

• Increased CPU usage on switches

• Create method to evaluate internal ACLs routinely

• Consider Control Plane Policing for basic router/switch services

Routing

Switching


Implement Network Security


GAP Prioritization – Where To Focus

Correlating business impact (risk reduction) to ease of execution and exemplar implementation time

Project List:1) Remediate single points of failure

2) Create, utilize and maintain global configuration standard templates

3) Create, utilize and maintain process documentation

4) Remediate configuration inconsistencies within the network

More complex to implement

Low priority

0-6 months9months> year

Easy to implement

High priority

1Must Do – Reduce Risk

Very Hard

Quick Wins – High Business Impact

Easy But Low Return

2

3

4


Q & A

Thank you.

high performance network analysis

Documents

cisco andor

network managementtools

network improvement

network standards

analysis findings

switchmajor risk

movenwsc secondary switch

holistic health check