virtualizing your network divide and conquer educause & internet2 security professionals...

Virtualizing Your Network Virtualizing Your Network Divide and ConquerDivide and Conquer

EDUCAUSE & Internet2 SecurityEDUCAUSE & Internet2 SecurityProfessionals Conference Professionals Conference April 10-12, 2007April 10-12, 2007

Copyright Robert E. Neale 2007. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

www.ts.vcu.edu

Agenda

• VCU’s “State of the network 2005”

• Why “virtualize” the network?

• Implementation at VCU – New network architecture and design– Network security enhancements– Implementation time frame

• Summary

www.ts.vcu.edu

VCU Background Information

• Virginia Commonwealth University

• Located in Richmond, Va.

• Two campuses

• 30,300 students – 4,500 in VCU housing

• 9,000 faculty/staff

• http://www.vcu.edu

http://www.vcu.edu/maps/acmap/acmapov/

www.ts.vcu.edu

Network Environment at the end of 2005

• Nine Routers across both campuses

• 1800 layer two Cisco switches

• Internet firewall with many holes

• Router ACL security for subnets & servers.

• Rapid growth in network – new buildings– New connections– wireless

www.ts.vcu.edu

Business

Cabell

Sanger

900 E Main

McGuire

MasseyMain

Hospital

C6506

C6506

C6506

C6506C6506 C6506

C6506

C6506

C6506

Hibbs

Lafayette

C6506

C2950

C2950

C3550 IDS(Enterasys)

7513

Why “virtualize” the network?Why “virtualize” the network?

www.ts.vcu.edu

Growing Complexity of Network

• Over 140 Buildings, 1800+ switches

• Need to segment users for better network security management

• DHCP vs Static IP issues

• Requirements for multiple VLANS across the network backbone infrastructure

www.ts.vcu.edu

Mandate for Better Security

• New laws for protection of sensitive data

• Threats are becoming more sophisticated

• Resources stretched in attempting to address security problems

• Need to improve management of network to protect less secure systems

www.ts.vcu.edu

QoS & Future Requirements

• QoS needed for Voice over IP project

• QoS planned for video locations

• Possible separate research VLANs

• Business Continuity - data replication

• Internal network SLA’s

www.ts.vcu.edu

Implementation at VCUOutside Resources Partnerships

• Sycom

– Network engineering resources

– Architecture design with our staff

• Cisco Systems

– Architectural review

– Proof of Concept Center

www.ts.vcu.edu

Design Goals

• Translation to private MPLS L3 VPN hierarchical design– Existing technology preserved and

integrated into new design

• Allow for phased implementation• Shared services virtualization and

network segmentation• Introduction of QoS into new design• Simplification of security model

www.ts.vcu.edu

Introduction: Terminology

MPLS-VPN Backbone

CE

Site IGP

C PE

PE

Access Layer

P

PE-CE Routing Protocol

Virtual Routing and Forwarding (VRF)

Tables

• C – Customer Router that sits at the Customer Site that peers with only other Customer devices.

• CE – Customer Edge Device that sits at the access layer. The CE could be a layer 2 or layer 3 device

• Site IGP – Site Interior Gateway Protocol. The IGP run at the Customer Site.• PE – Provider Edge Router. The Provider Edge Router sits at the edge of the MPLS

backbone. • P – Provider Router that resides in the MPLS backbone.• PE-CE Link – The connection between the PE and CE. • PE-CE Routing Protocol – The dynamic routing protocol (EBGP, RIPv2, EIGRP, OSPF) or

static routing protocol (Static, Connected) run over the PE-CE link • VRF –A Virtual Routing and Forwarding table that exists at the PE that is used to provide a

separate routing table for different customers or customer business units.

CE

www.ts.vcu.edu

High Level Backbone Logical Design

SiSi

SiSi

SiSi

SiSi

MPLS Backbone10GB fully meshed

OSPF + LDP

1900

ResnetVrf blue

SiSi

INTERNET

ResnetVrf blue

3560

Admin DataVlan 200Vrf red

Admin voiceVlan 100Vrf green

3560

Admin DataVlan 200Vrf red

Admin voiceVlan 100Vrf green

3560

Network Services VRF Grey

Cat6K with dual Sup720 and FWSM

Cat6K with single Sup720




PE1

PE2

PE3 PE4

PE5

PE6

PE8PE9PE7

LabsVrf orange

All 9 PE routers are part of same iBGP

mesh P1

P2 P3

P4

LabsVrf orange

SiSi

Cat6K with dual Sup720

SiSi


SiSi


SiSi


SiSi


SiSi


SiSi


Inter VRF Routing Here

10GB

1GB

SiSi

Cat6K with dual Sup720 and FWSM

www.ts.vcu.edu

FANSTATUS

1

2

3

4

5

6

7

8

9

Power Supply 1 Power Supply 2

Catalyst 6500 SERIES

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B

SUPERVISOR 720 WITH INTEGRATED SWITCH FABRIC

EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-X6704-10GE

4 PORT 10 GIGABIT ETHERNET

PORT1

TX RX

PORT2

TX RX

PORT3

TX RX

PORT4

TX RX

WS-X6724-SFP24 PORT GIGABIT ETHERNET SFP

STATUS

1 2 3 4 21 22 23 2417 18 19 2013 14 15 169 10 11 125 6 7 8

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-X6704-10GE


PORT1

TX RX

PORT2

TX RX

PORT3

TX RX

PORT4

TX RX


STATUS

1 2 3 4 21 22 23 2417 18 19 2013 14 15 169 10 11 125 6 7 8

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-X6704-10GE


PORT1

TX RX

PORT2

TX RX

PORT3

TX RX

PORT4

TX RX


STATUS

1 2 3 4 21 22 23 2417 18 19 2013 14 15 169 10 11 125 6 7 8

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-X6704-10GE


PORT1

TX RX

PORT2

TX RX

PORT3

TX RX

PORT4

TX RX


STATUS

1 2 3 4 21 22 23 2417 18 19 2013 14 15 169 10 11 125 6 7 8

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

Sanger-CoreLoopback0: 172.16.1.128/32

Loopback1: 172.17.0.1/32

Cabell-CoreLoopback0: 172.16.2.128/32

Loopback1: 172.17.0.3/32

Business-CoreLoopback0: 172.16.2.129/32

Loopback1: 172.17.0.4/32UCC-Core

Loopback0: 172.16.1.129/32Loopback1: 172.17.0.2/32

4/1 4/14/2

4/2 4/1 4/1

4/2

4/2

4/3 4/3

4/3 4/3

172.16.0.8/30

172.16.0.16/30

172.16.1.0/30 172.16.2.0/30

172.16.0.4/30172.16.0.12/30

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

Cabell DistributionLoopback0: 172.16.2.130/32Loopback1: 172.17.0.16/32

WS-X6408

STATUS

8 PORT GIGABIT ETHERNETLINK

1

LINK

2

LINK

3

LINK

4

LINK LINK

6

LINK LINK

85 7

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

Siegel DistributionLoopback0: 172.16.2.131/32Loopback1: 172.17.0.17/32

WS-X6408

STATUS


1

LINK

2

LINK

3

LINK

4

LINK LINK

6

LINK LINK

85 7

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

Lafayette DistributionLoopback0: 172.16.2.132/32Loopback1: 172.17.0.18/32

WS-X6408

STATUS


1

LINK

2

LINK

3

LINK

4

LINK LINK

6

LINK LINK

85 7

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

Business DistributionLoopback0: 172.16.2.133/32Loopback1: 172.17.0.19/32

WS-X6408

STATUS


1

LINK

2

LINK

3

LINK

4

LINK LINK

6

LINK LINK

85 7

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

Sanger DistributionLoopback0: 172.16.1.130/32Loopback1: 172.17.0.32/32

WS-X6408

STATUS


1

LINK

2

LINK

3

LINK

4

LINK LINK

6

LINK LINK

85 7

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

Main Hospital DistributionLoopback0: 172.16.1.131/32Loopback1: 172.17.0.33/32

WS-X6408

STATUS


1

LINK

2

LINK

3

LINK

4

LINK LINK

6

LINK LINK

85 7

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

West Hospital DistributionLoopback0: 172.16.1.132/32Loopback1: 172.17.0.34/32

WS-X6408

STATUS


1

LINK

2

LINK

3

LINK

4

LINK LINK

6

LINK LINK

85 7

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

1

2

3

4

5

6

FANSTATUS



WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

McGuire DistributionLoopback0: 172.16.1.133/32Loopback1: 172.17.0.35/32

WS-X6408

STATUS


1

LINK

2

LINK

3

LINK

4

LINK LINK

6

LINK LINK

85 7

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

WS-X6324-SM

24 PORT 100FX

4321 8765 1211109 16151413 20191817 24232221

UCC DistributionLoopback0: 172.16.1.134/32Loopback1: 172.17.0.36/32

STATUS

PHONE

WS-X6548-RJ-454748

4546

4344

4142

3940

3738

3536

3334

3132

2930

2728

2526

2324

2122

1920

1718

1516

1314

1112

910

78

56

12

34

37 38 39 40 41 42 43 44 45 46 4847363533 3431 3229 3027 2825 26242321 2219 2017 1815 1613 1412119 107 85 63 41 2

172.16.2.4/30

172.16.2.8/30

172.16.2.12/30

172.16.2.16/30

172.16.2.20/30

172.16.2.24/30

172.16.2.28/30

172.16.2.32/30

172.16.1.4/30

172.16.1.8/30

172.16.1.12/30

172.16.1.16/30

172.16.1.20/30

172.16.1.24/30

172.16.1.28/30

172.16.1.32/30

172.16.1.36/30

172.16.1.40/30

STATUS

PHONE

WS-X6548-RJ-454748

4546

4344

4142

3940

3738

3536

3334

3132

2930

2728

2526

2324

2122

1920

1718

1516

1314

1112

910

78

56

12

34

37 38 39 40 41 42 43 44 45 46 4847363533 3431 3229 3027 2825 26242321 2219 2017 1815 1613 1412119 107 85 63 41 2

WS-SUP720-3B


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

Port-Channel 1Gig 1/4, 1/5

10 G BASE - LR SERIAL 1310NM OPTICAL INTERFACE MODULE

WS-G6488

LINK

TX RX

10 G BASE - LR SERIAL 1310NM OPTICAL INTERFACE MODULE

WS-G6488

LINK

TX RX

Gig 1/1

Physical Network Layout

Sanger Cabell

Computer Center

School of

Business

Sanger

Main Hospital

West Hospital

McGuire

Computer Center

Cabell

Siegel

Lafayette

School of Business

www.ts.vcu.edu

Network Management VRF

PE

PEVlan 200

Ip vrf forwarding mgmt172.17.1/22

CE CE CE

802.1q

802.1q

802.1q

P P

P P

NMS

Ip vrf forwarding mgmt

Access switch management interfaces in VLAN 200

PE and P router Loopback1 mgmt

interfaces in Global routing table (OSPF)

global

Vrf mgmt

Globalcontext

Mgmt context

Vrf common

Forwarding between vrf mgmt and P / PE router management

interfaces done at PE with FWSM

PE router with FWSM

Vrf tempvcutempvcu context

PE

Legacy network placed in VRF

“Tempvcu” during migration

www.ts.vcu.edu

User Access To MPLS Network

PE

VRF

VRF

VRFInterface vlan 200

Interface vlan 100

Interface vlan 50

Data VRFVlan 200

Voice VRFVlan 100

Access CE L2 Switch

Mgt addressVlan 50

interface GigabitEthernet3/5 Trunk connection to access switch switchport switchport trunk encapsulation dot1q switchport mode trunk no ip address mls qos trust dscp!interface Vlan50 description SVI supporting the nms VRF ip VRF forwarding mgmt ip address 172.17.36.2 255.255.252.0 !address-family ipv4 VRF mgmt redistribute connected no auto-summary no synchronization exit-address-family

www.ts.vcu.edu

NetworkMgmt

SECNet

RESNet

VoIP

SVRNet

DMZNet

FWContext

FWContext

FWContext

FWContext

FWContext

FWContext

CommonArea Internet

TempVCUFW

Context

www.ts.vcu.edu

QoS Design – High Level• A five class model supported in the MPLS-Backbone

– • Voice – • Business Critical data – • Video/UDP – – • Best Effort Data – – • Scavenger Data –

• Access switches classify/mark traffic • Edge Routers perform the following QoS Functions

– Classification/Marking– Microflow policing and remarking to Scavenger– Copying IP DSCP into MPLS EXP

• Core interfaces have congestion management enabled

www.ts.vcu.edu

Improve Management of Network Security Policies

• Problems with management of ACLs– Poor scalability– Increased management requirements– Increased troubleshooting complexity

• Solution is replacement of ACLs in routers with FWSMs

www.ts.vcu.edu

Logical Representation of ACL Consolidation

Switch Interfaces with ACLs

Firewall ACL

Clients

Traffic to other VRFs or Internet

www.ts.vcu.edu

Existing ACL ImplementationEdge Router Active ACL’s Line Count

Gallium 176 10,246

Poca 10 2497

Cabell 22 284

Seigel 7 61

Lafayette 10 111

Business 33 334

Sanger 29 212

Main Hospital 13 161

West Hospital 17 150

McGuire 35 232

Total 352 14,288

www.ts.vcu.edu

New FSM ImplementationName Active ACL Line Count

Email 4 4

MCVH 2 8

Network 5 106

RESNet 2 102

ServNet 4 16

TempVCU 3 25

VCUSecure 3 10

Voice 5 100

Wireless 4 41

Total 32 412

www.ts.vcu.edu

RESNet Example

• Total lines of RESNet ACL prior to consolidation: 2146

• Total lines of RESNet ACL post-consolidation: 248

• Percent of original line count: 11.6%

• Line reduction ratio: 8.65 times

• * Note that this data was extremely viable for consolidation due to the commonalities that could be extracted from all ACLs. This data should be viewed as a better-case scenario.

www.ts.vcu.edu

Enhanced Network Security

• Groups of users:– Sensitive data PC network separate from PC labs and

public PC connections.– Different policies applied according to groups of PC

users.

• Groups of services:– IP Phone Services– Video Conferencing– Wireless

• VLANs implemented at switch port

www.ts.vcu.edu

Other Network Security Enhancements

• Cisco Clean Access – RESNet, SECNet, Public Access, Wireless

• Implementation of IPS for ServerNet.

• Cisco VPN remote access to internal VCU server resources only.

• Implementing Cisco MARS – Monitoring, Analysis, and Response System

• Split DNS implementation

www.ts.vcu.edu

Timeframe

• 12/05 Initial partnership, planning, purchases• 2/06 Design reviewed by Cisco Adv Svrc• 3/06 Cisco CPOC in Raleigh NC• 5/06 Core & Distribution switches in MPLS • 5/06 Started VoIP data remediation• 7/06 Basic VRF’s created• 9/06 RESNet VRF – Route ACL to FWSM• 4/07 ServerNet and DMZNet migration started• 9/07 Planned completion of ACL to FWSM• 11/07 Planned completion of VoIP project

www.ts.vcu.edu

Summary

• Virtualization of your network can:– Provides flexibility– Reduce network complexity– Improve network management – Enhance network security

• Lessons learned:– Virtualize the network before VoIP– Assess your organizations ability to change– Outside resources critical to success

Questions?Questions?

virtualizing your network divide and conquer educause & internet2 security professionals...

Documents

management of network

network divide

network environment

complexity of network

vcu new network architecture

customer site

pece link

pe provider edge router