metafabric™ architecture virtualized data center: design and implementation guide

MetaFabric™ Architecture VirtualizedData Center

Design and Implementation Guide

Release

1.0

Published: 2014-03-18

Copyright © 2014, Juniper Networks, Inc.

Juniper Networks, Inc.1194 North Mathilda AvenueSunnyvale, California 94089USA408-745-2000www.juniper.net

Copyright © 2014, Juniper Networks, Inc. All rights reserved.

Juniper Networks, Junos, Steel-Belted Radius, NetScreen, and ScreenOS are registered trademarks of Juniper Networks, Inc. in the UnitedStates and other countries. The Juniper Networks Logo, the Junos logo, and JunosE are trademarks of Juniper Networks, Inc. All othertrademarks, service marks, registered trademarks, or registered service marks are the property of their respective owners.

Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify,transfer, or otherwise revise this publication without notice.

[Insert Series Title] [Insert Book Title]Copyright © 2014, Juniper Networks, Inc.All rights reserved.

The information in this document is current as of the date on the title page.

YEAR 2000 NOTICE

Juniper Networks hardware and software products are Year 2000 compliant. Junos OS has no known time-related limitations through theyear 2038. However, the NTP application is known to have some difficulty in the year 2036.

ENDUSER LICENSE AGREEMENT

The Juniper Networks product that is the subject of this technical documentation consists of (or is intended for use with) Juniper Networkssoftware. Use of such software is subject to the terms and conditions of the End User License Agreement (“EULA”) posted athttp://www.juniper.net/support/eula.html. By downloading, installing or using such software, you agree to the terms and conditions ofthat EULA.

Copyright © 2014, Juniper Networks, Inc.ii

http://www.juniper.net/support/eula.html

Table of Contents

Part 1 MetaFabric™ArchitectureVirtualized ITDataCenterDesignandImplementation Guide

Chapter 1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

MetaFabric Architecture Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Validated Solution Design and Implementation Guide Overview . . . . . . . . . . . . . . 8

MetaFabric 1.0 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Solution Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Compute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Class of Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Network Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Chapter 2 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Design Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Design Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Design Topology Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Design Highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Solution Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Compute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Virtual Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Hypervisor Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Blade Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Access and Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Core Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Edge Routing and WAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Compute Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Network Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Network Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Business-Critical Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

iiiCopyright © 2014, Juniper Networks, Inc.

High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Hardware Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Software Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Class of Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Application Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Perimeter Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Secure Remote Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Network Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Out-of-Band Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Network Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Security Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Performance and Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Summary of Key Design Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Chapter 3 MetaFabric 1.0 High Level Testing and Validation Overview . . . . . . . . . . . . . 61

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Key Characteristics of Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

POD1 (QFX3000-M QFabric) Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 63

POD2 (QFX3000-M QFabric) Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Core Switch (EX9214) Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Edge Firewall (SRX3600) Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Edge routers (MX240) Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Compute (IBM Flex chassis) Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 65

OOB-Mgmt (EX4300-VC) Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Chapter 4 Transport (Routing and Switching) Configuration . . . . . . . . . . . . . . . . . . . . . 73

Network Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Configuring theNetworkBetween theDataCenterEdgeand theDataCenter

Core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Implementing MC-LAG Active/Active with VRRP . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Summary of Implementation Details for MC-LAG Active/Active . . . . . . . . . . 85

MC-LAG Configuration for Better Convergence . . . . . . . . . . . . . . . . . . . . . . . . 86

Configuring the Network Between the Data Center Core and the Data Center

PODs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Routing Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Configuring BGP Between the EDGE and Service Provider . . . . . . . . . . . . . . 106

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

Configuring OSPF in the Data Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

Copyright © 2014, Juniper Networks, Inc.iv

MetaFabric™ Architecture Virtualized Data Center

Chapter 5 High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

High Availability Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Hardware Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Software Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

QFabric-M Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Configurint the Core and Edge Router . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Configuring the Perimeter Firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Chapter 6 Class-of-Service Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Class-of-Service Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Configuring Class-of-Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Configuring Class-of-Service (POD Level) . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Configuring Data Center Bridging and Lossless Ethernet . . . . . . . . . . . . . . . . . . . 130

Configuring Class-of-Service (POD Level) . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Chapter 7 Security Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Perimeter Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Configuring Chassis Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

Configure Chassis Clustering Data Fabric . . . . . . . . . . . . . . . . . . . . . . . . 138

Configuring Chassis Clustering Groups . . . . . . . . . . . . . . . . . . . . . . . . . . 139

Configuring Chassis Clustering Redundancy Groups . . . . . . . . . . . . . . . 139

Configuring Chassis Clustering Data Interfaces . . . . . . . . . . . . . . . . . . . 140

Configuring Chassis Clustering – Security Zones and Security Policy . . . 141

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Configuring Network Address Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Configure Source NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Configure Destination NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

Configuring Intrusion Detection and Prevention . . . . . . . . . . . . . . . . . . . . . . 148

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Host Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Configuring the Firefly Host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

Chapter 8 Data Center Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

Data Center Services Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

Configuring Compute Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

Compute Hardware Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

Configuring Compute Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

Configuring Compute Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

Configuring POD to Pass-thru Chassis Compute Nodes . . . . . . . . . . . . . 172

Configuring the CNA Fabric Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

vCopyright © 2014, Juniper Networks, Inc.

Table of Contents

Configuring the 10Gb CNA Module Connections . . . . . . . . . . . . . . . . . . . 181

Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

Virtualization Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

Configuring LACP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

Configuring VMware Clusters, High Availability, and Dynamic Resource

Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

Configuring VMware Enhanced vMotion Compatibility . . . . . . . . . . . . . . . . . 197

Mounting Storage Using the iSCSI Protocol . . . . . . . . . . . . . . . . . . . . . . . . . 200

Configuring Fault Tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

Configuring VMware vMotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

EMC Storage Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204

Configuring EMC Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204

Configuring EMC FAST Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

Configuring FAST Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

Configuring Storage Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

Configuring Logical Unit Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

Enabling Storage Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

Configuring the Network File System . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

Configuring VNX Snapshot Replicas . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

Configuring Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

Configuring the Link and Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229

Configuring VIP and Server Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

Load-Balanced Traffic Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235

Microsoft Exchange Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235

Installation Checklist and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236

Deploying Network for Exchange VM . . . . . . . . . . . . . . . . . . . . . . . . . . . 236

Configuring Storage for Exchange VM . . . . . . . . . . . . . . . . . . . . . . . . . . 242

Enabling Storage Groups with Unisphere . . . . . . . . . . . . . . . . . . . . . . . . 245

Provisioning LUNs to ESXi Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248

Configuring vMotion Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

Chapter 9 Network Management and Orchestration . . . . . . . . . . . . . . . . . . . . . . . . . . . 267

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267

Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267

Configuring Junos Space with Network Director . . . . . . . . . . . . . . . . . . . . . . 268

Configuring VM Orchestration in the Network Director 1.5 Virtual View . . . . 269

Network Director Provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270

Configuring Class of Service Using Network Director . . . . . . . . . . . . . . . . . . . 272

Creating VLANs Using Network Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274

Setting Up QFabric Using Network Director . . . . . . . . . . . . . . . . . . . . . . . . . . 275

Setting Up QFabric Using Network Director – Ports and VLAN . . . . . . . . . . . 277

Setting Up a QFabric System Using Network Director – Create Link

Aggregation Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

Copyright © 2014, Juniper Networks, Inc.vi


Network Director – Downloading and Upgrading Software Images . . . . . . . 283

Network Director – Monitoring the QFabric System . . . . . . . . . . . . . . . . . . . 285

Configuring Security Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Discovery and Basic Configuration Using Security Director . . . . . . . . . . 288

Resolving DMI Mismatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290

Object Builder (Using Security Director) . . . . . . . . . . . . . . . . . . . . . . . . . 291

Creating Firewall Policy Using Security Director . . . . . . . . . . . . . . . . . . . 292

Creating NAT Policy Using Security Director . . . . . . . . . . . . . . . . . . . . . . 294

Jobs Workspace in Security Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296

Audit Logs in Security Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297

Chapter 10 Solution Scale and Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

Overview of Solution Scale Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300

Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300

viiCopyright © 2014, Juniper Networks, Inc.

Table of Contents

List of Figures



Figure 1: Applications Drive IT Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Figure 2: Data Center Before MetaFabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Figure 3: Data Center After MetaFabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Figure 4: MetaFabric – Putting It All Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Figure 5: MetaFabric Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Figure 6: Juniper Networks Virtualized IT Data Center - Sizing Options . . . . . . . . . . 9

Figure 7: Juniper Networks Virtualized IT Data Center – Solution

Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Figure 8: Network Management Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Chapter 2 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Figure 9: Virtualized IT Data Center Ecosystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Figure 10: Virtualized IT Data Center Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Figure 11: Virtual Machine Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Figure 12: Server Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Figure 13: VMware Distributed Virtual Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Figure 14: VMware Network I/O Control Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Figure 15: Sample Blade Switch, Rear View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Figure 16: Juniper Networks QFabric Systems Enable a Flat Data Center

Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Figure 17: Core Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Figure 18: Core Switching Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Figure 19: Edge Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Figure 20: Edge Routing Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Figure 21: Storage Lossless Ethernet Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Figure 22: Storage Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Figure 23: Virtualized IT Data Center Solution Software Stack . . . . . . . . . . . . . . . 38

Figure 24: MC-LAG – ICCP and ICL Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Figure 25: VRRP and MC-LAG – Active/Active Option . . . . . . . . . . . . . . . . . . . . . . 43

Figure 26: MC-LAG – MAC Address Synchronization Option . . . . . . . . . . . . . . . . . 44

Figure 27: MC-LAG – Traffic Forwarding Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Figure 28: MC-LAG – ICCP Down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Figure 29: MC-LAG – ICL Down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Figure 30: MC-LAG – Peer Down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Figure 31: Class of Service – Classification and Queuing . . . . . . . . . . . . . . . . . . . . 47

Figure 32: Class of Service – Buffer and Transmit Design . . . . . . . . . . . . . . . . . . . 48

Figure 33: Physical Security Compared to Virtual Network Security . . . . . . . . . . . 49

ixCopyright © 2014, Juniper Networks, Inc.

Figure 34: Application Security Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Figure 35: Physical Security Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Figure 36: Remote Access Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Figure 37: Seven Tier Model of Network Management . . . . . . . . . . . . . . . . . . . . . . 54

Figure 38: Out of Band Management Network Design . . . . . . . . . . . . . . . . . . . . . . 56

Figure 39: Out of Band Management – Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57


Figure 40: The End to End Lab Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

Figure 41: MC-LAG Active/Active Logical Topology . . . . . . . . . . . . . . . . . . . . . . . . 69

Figure 42: Topology of Core-to-POD Roles in the Data Center . . . . . . . . . . . . . . . . 71


Figure 43: Configuration of RETH Interfaces and MC-LAG Between Core and

Perimeter (Right) Compared to Configuration of RETH Interfaces and AE

(Left) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Figure 44: Interface Configuration Between Edge, Perimeter, and Core . . . . . . . . . 75

Figure 45: MetaFabric 1.0 Routing Configuration and Topology . . . . . . . . . . . . . . 105

Figure 46: OSPF Area Configuration Between Edge and Core (Including

Out-of-Band Management) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

Figure 47: OSPF Area Configuration Between Core and PODs . . . . . . . . . . . . . . . 112

Figure 48: Loop-Free Alternate Convergence Example . . . . . . . . . . . . . . . . . . . . . 114


Figure 49: The VDC POD and Compute/Storage Topology . . . . . . . . . . . . . . . . . . 131

Chapter 7 Security Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Figure 50: Logical View of Juniper Networks Firefly Host Installation . . . . . . . . . . 155

Figure 51: An Example dvPort Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

Figure 52: Configure an Application Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

Figure 53: The Annotation Allows Firefly Host to Detect Related VMs . . . . . . . . . 158

Figure 54: Define Security Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

Chapter 8 Data Center Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

Figure 55: Compute and Virtualization as Featured in the MetaFabric 1.0

Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

Figure 56: IBM x3750 M4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

Figure 57: IBM Flex System Enterprise Chassis (Front View) . . . . . . . . . . . . . . . . 166

Figure 58: IBM Flex System (Rear View) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

Figure 59: IBM Flex System Fabric CN4093 10Gb/40Gb Converged Scalable

Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

Figure 60: IBM Flex System EN4091 10Gb Ethernet Pass-thru Module . . . . . . . . 169

Figure 61: IBM Flex System x220 Compute Node . . . . . . . . . . . . . . . . . . . . . . . . . 170

Figure 62: IBM Pure Flex Pass-thru Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

Figure 63: POD1 Topology with the IBM Pure Flex Chassis + 40Gbps CNA

Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

Figure 64: POD 2 Topology Using the IBM Pure Flex System Chassis with the

10-Gbps CNA I/O Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

Figure 65: VMware vSphere Client Manages vCenter Server Which in Turn

Manages Virtual Machines in the Data Center . . . . . . . . . . . . . . . . . . . . . . . . 187

Figure 66: VMWare vSphere Distributed Switch Topology . . . . . . . . . . . . . . . . . . 188

Copyright © 2014, Juniper Networks, Inc.x


Figure 67: VMware vSphere Distributed Switch Topology . . . . . . . . . . . . . . . . . . 189

Figure 68: Log In to vCenter Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

Figure 69: vCenter Web Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

Figure 70: Click Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

Figure 71: Click Related Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

Figure 72: Click Uplink Ports and Select a Port . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

Figure 73: Enable LACPMode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

Figure 74: Infra Cluster Hosts Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

Figure 75: POD1 Cluster Hosts Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

Figure 76: POD2 Cluster Hosts Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

Figure 77: INFRA Cluster VMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Figure 78: POD1 Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

Figure 79: POD2 Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

Figure 80: Port Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

Figure 81: Port Group and NIC Teaming Example . . . . . . . . . . . . . . . . . . . . . . . . . 199

Figure 82: Configure Teaming and Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

Figure 83: POD1 PG-STORAGE-108 Created for iSCSI . . . . . . . . . . . . . . . . . . . . . 201

Figure 84: VMware Fault Tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

Figure 85: VMware Fault Tolerance on POD1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

Figure 86: VMware vMotion Enables Virtual Machine Mobility . . . . . . . . . . . . . . 203

Figure 87: VMware vMotion Configured in the Test Lab . . . . . . . . . . . . . . . . . . . . 204

Figure 88: EMC FAST Cache Configuration (Select System, then Properties in

the Drop-Down) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

Figure 89: EMC FAST Cache Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

Figure 90: Pool 1 - Exchange-DB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207

Figure 91: Selected Storage Pool Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207

Figure 92: Storage Pool Disks Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

Figure 93: Storage Pool Properties, Advanced Tab . . . . . . . . . . . . . . . . . . . . . . . 208

Figure 94: VM-Pool Selected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

Figure 95: VM-Pool Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

Figure 96: VM-Pool Disk Membership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

Figure 97: Exchange-DB-LUN Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

Figure 98: LUN Created for All ESX Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

Figure 99: The Selected Pool Was Created for MS Exchange Logs . . . . . . . . . . . . 213

Figure 100: Exchange Logs the LUN Created . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

Figure 101: Example Storage Group Properties Window . . . . . . . . . . . . . . . . . . . . 215

Figure 102: LUN Added to Storage Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

Figure 103: ESXi Hosts Added to Storage Group . . . . . . . . . . . . . . . . . . . . . . . . . . 217

Figure 104: Add LUNs to Storage Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

Figure 105: NFS Pool Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

Figure 106: LUN Created on the New Storage Pool . . . . . . . . . . . . . . . . . . . . . . . . 219

Figure 107: NFS Pool Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

Figure 108: NFS Export Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

Figure 109: Snapshot Configuration Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

Figure 110: Select Source Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

Figure 111: Select Snapshot Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

Figure 112: Select Source LUNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

Figure 113: Select Snapshot Storage Overhead . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

Figure 114: ChooseWhen to Create LUN Snapshot . . . . . . . . . . . . . . . . . . . . . . . . 225

xiCopyright © 2014, Juniper Networks, Inc.

List of Figures

Figure 115: Assign Snapshot to a Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226

Figure 116: Summary of Snapshot Wizard Configuration . . . . . . . . . . . . . . . . . . . 227

Figure 117: Load Balancing Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

Figure 118: Configure nPath . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

Figure 119: Verify Objects during nPath Configuration . . . . . . . . . . . . . . . . . . . . . . 233

Figure 120: Configure and Verify VIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233

Figure 121: Load-Balancing Traffic Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

Figure 122: Home > Inventory > Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239

Figure 123: Create New Port Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

Figure 124: Modify Teaming Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Figure 125: PG-STORAGE-108 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Figure 126: PG-STORAGE-208 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242

Figure 127: EMC Unisphere Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243

Figure 128: Create Storage Pool Name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243

Figure 129: FAST Cache enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244

Figure 130: Exchange-DB LUN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245

Figure 131: Storage Group Created . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246

Figure 132: Storage Group Properties - LUNs Tab . . . . . . . . . . . . . . . . . . . . . . . . . 247

Figure 133: Hosts Allowed to Access the Storage Group . . . . . . . . . . . . . . . . . . . 248

Figure 134: Add LUN to Storage Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248

Figure 135: Manage Virtual Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

Figure 136: Add New VMkernel Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250

Figure 137: Select VMkernel as Adapter Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250

Figure 138: Select Port Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251

Figure 139: VMkernel IP Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251

Figure 140: Install iSCSI Software Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

Figure 141: iSCSI Initiator Is Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

Figure 142: iSCSI Initiator Network Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 254

Figure 143: Add iSCSI Server Location in Dynamic Discovery . . . . . . . . . . . . . . . . 255

Figure 144: LUN Present on the Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255

Figure 145: Add Storage from vSphere Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256

Figure 146: Select Disk/LUNfor Storage Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256

Figure 147: Select LUN to Mount . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

Figure 148: Select VMFS-5 as a File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

Figure 149: Name the Datastore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

Figure 150: Datastore Creation Complete . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

Figure 151: Create New VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

Figure 152: VM Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260

Figure 153: Give the VM a Name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261

Figure 154: Select Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262

Figure 155: Select Operating System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

Figure 156: Configure Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264

Figure 157: Select Virtual Disk Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265

Figure 158: Virtual Machine with Additional Disks and Network Adapters . . . . . 266

Chapter 9 Network Management and Orchestration . . . . . . . . . . . . . . . . . . . . . . . . . . . 267

Figure 159: The OOB Management Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268

Figure 160: Select IP address, IP Range, IP-Subnet, or HostName . . . . . . . . . . . 269

Figure 161: Configure Virtual Network target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

Copyright © 2014, Juniper Networks, Inc.xii


Figure 162: Enable Orchestration Mode in Network Director . . . . . . . . . . . . . . . . 270

Figure 163: Configure Device Common Settings . . . . . . . . . . . . . . . . . . . . . . . . . . 271

Figure 164: Change in Pending Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

Figure 165: Change in Pending Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

Figure 166: Select the Data Center Switching Device Family . . . . . . . . . . . . . . . . 272

Figure 167: Select the Profile "Hierarchal Port Switching (ELS)" . . . . . . . . . . . . . 273

Figure 168: Enable PFC Code-point and Queue for NO-LOSS Behavior . . . . . . . 273

Figure 169: COS Profile Deployed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274

Figure 170: Create VLAN-ID and VLAN Name . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274

Figure 171: Configure Layer 2 Filters and MAC Move Limit . . . . . . . . . . . . . . . . . . . 275

Figure 172: VLAN Profile ND-Test1Created . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275

Figure 173: Select Setup QFabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276

Figure 174: Configure Device Aliases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276

Figure 175: Configure Node Group Type RNSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277

Figure 176: Configure Center Switching Non ELS . . . . . . . . . . . . . . . . . . . . . . . . . . 277

Figure 177: Configure VLAN Service, Port, CoS, and so on . . . . . . . . . . . . . . . . . . . 278

Figure 178: Port Profile Created (NDTestport) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278

Figure 179: Assign Port Profile to Available Port . . . . . . . . . . . . . . . . . . . . . . . . . . 279

Figure 180: Assign Port Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279

Figure 181: Click Assign . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280

Figure 182: New Physical Port Added to Port Profile List . . . . . . . . . . . . . . . . . . . 280

Figure 183: Port Profile Created Successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

Figure 184: Check to Confirm Port Profile Is Pending . . . . . . . . . . . . . . . . . . . . . . . 281

Figure 185: Select Deploy Now . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

Figure 186: Add New Port Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

Figure 187: Select Devices to Add as LAG Member Links . . . . . . . . . . . . . . . . . . . 283

Figure 188: Links Selected to Be LAG Member Links . . . . . . . . . . . . . . . . . . . . . . 283

Figure 189: Network Director Image Repository . . . . . . . . . . . . . . . . . . . . . . . . . . 284

Figure 190: Image Staging on Network Director . . . . . . . . . . . . . . . . . . . . . . . . . . 284

Figure 191: Stage Image to Device for Install or for Later Installation . . . . . . . . . . 285

Figure 192: Select Image to Stage to Remote Device . . . . . . . . . . . . . . . . . . . . . . 285

Figure 193: Device Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286

Figure 194: QFabric Traffic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286

Figure 195: Hardware Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Figure 196: Confirmation of Run Fabric Analyzer Operation . . . . . . . . . . . . . . . . . 287

Figure 197: DMI Mismatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289

Figure 198: DMI Schema Repository Requires Authentication . . . . . . . . . . . . . . . 290

Figure 199: Security Zone Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290

Figure 200: Address Object Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292

Figure 201: New Rule Created (Test-1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294

Figure 202: Add New Source Address to Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294

Figure 203: Example NAT Policies in Security Director . . . . . . . . . . . . . . . . . . . . . 296

xiiiCopyright © 2014, Juniper Networks, Inc.

List of Figures

List of Tables



Table 1: Juniper Networks Virtualized IT Data Center – Details of Sizing

Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Chapter 2 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Table 2: MetaFabric 1.0 Solution Design Highlights . . . . . . . . . . . . . . . . . . . . . . . . 20

Table 3: Comparison of Pass-Through Blade Servers andOversubscribed Blade

Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Table 4: Core Switch Hardware - Comparison of the EX9200 and EX8200

Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Table 5: Core Switch Forwarding - Comparison of MC-LAG and Virtual

Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Table 6: Comparison of Storage Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Table 7: Application Security Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Table 8: Data Center Remote Access Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Table 9: Summary of Key Design Elements – Virtualized IT Data Center

Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58


Table 10: Hardware and Software deployed in solution testing . . . . . . . . . . . . . . . 66

Table 11: Software deployed in MetaFabric 1.0 test bed . . . . . . . . . . . . . . . . . . . . . 67

Table 12: Networks and VLANs Deployed in the Test Lab . . . . . . . . . . . . . . . . . . . . 67

Table 13: Applications Tested in the MetaFabric 1.0 Solution . . . . . . . . . . . . . . . . . 68

Table 14: MC-LAG Configuration Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Table 15: IRB, IP Address Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72


Table 16: MC-LAG Settings Between Core 1 and Edge 1 . . . . . . . . . . . . . . . . . . . . . 76

Table 17: MC-LAG Between Core 1 and Edge 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77


Table 18: MetaFabric 1.0 Class-of-Service Queues . . . . . . . . . . . . . . . . . . . . . . . . 130

Chapter 10 Solution Scale and Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

Table 19: Application Scale Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300

xvCopyright © 2014, Juniper Networks, Inc.

PART 1

MetaFabric™ Architecture Virtualized ITData Center Design and ImplementationGuide

• Overview on page 3

• Design on page 17

• MetaFabric 1.0 High Level Testing and Validation Overview on page 61

• Transport (Routing and Switching) Configuration on page 73

• High Availability on page 123

• Class-of-Service Configuration on page 129

• Security Configuration on page 137

• Data Center Services on page 161

• Network Management and Orchestration on page 267

• Solution Scale and Known Issues on page 299

1Copyright © 2014, Juniper Networks, Inc.

CHAPTER 1

Overview

Thebenefits of virtualization are driving data center operators to rethink their legacy data

center networks and look for newways to reduce costs and improve efficiency in the

data center. Moving from a legacy network to a state-of-the-art solution allows you to

deploy new applications in seconds rather than days, weeks, or months. If you want to

harness the power of virtualization in your data center network, this guide will help you

to achieve your goal.

• MetaFabric Architecture Overview on page 3

• Domain on page 5

• Goals on page 6

• Audience on page 7

• Validated Solution Design and Implementation Guide Overview on page 8

• MetaFabric 1.0 Overview on page 8

• Solution Overview on page 10

MetaFabric Architecture Overview

Cloud,mobility, andbigdataaredrivingbusinesschangeand IT transformation. Enterprise

businesses and service providers across all industries are constantly looking for a

competitive advantage, and reliance on applications and the data center have never

been greater (Figure 1 on page 3).

Figure 1: Applications Drive IT Transformation

Traditional networks are physically complex, difficult to manage, and not suited for the

dynamicapplicationenvironmentsprevalent in today’s datacenters. Becauseofmergers,


acquisitions, and industry consolidation, most businesses are dealing with data centers

that are distributed across multiple sites and clouds, which adds evenmore complexity.

Additionally, the data center is so dynamic because the network is constantly asked to

domore, becomemore agile, and support new applications while ensuring integration

with legacyapplications.Consequently, thisdynamicenvironment requiresmore frequent

refresh cycles.

The network poses two specific problems in the data center:

1. Impedes time to value—Network complexity gets in the way of delivering data center

agility.

2. Low value over time—Every time a new application, technology, or protocol is

introduced, the network needs to be ripped out and replaced.

The growing popularity and adoption of switching fabrics, new protocols, automation,

orchestration, security technologies, and software-defined networks (SDNs) are strong

indicators of the need for a more agile network in the data center. Juniper Networks has

applied its networking expertise to the problems of today’s data centers to develop and

deliver the MetaFabric™ architecture—a combination of switching, routing, security,

software, orchestration, and SDN—all working in conjunction with an open technology

ecosystem to accelerate the deployment and delivery of applications for enterprises and

service providers.

With legacy data center networks, you needed to create separate physical and virtual

resources at your on-premises data center, your managed service provider, your hosted

service provider, and your cloud provider. All of these resources required separate

provisioning andmanagement (Figure 2 on page 4).

Figure 2: Data Center Before MetaFabric

Now, implementing aMetaFabric architecture allows you to combinephysical and virtual

resources across boundaries to provision andmanage your data center efficiently and

holistically (Figure 3 on page 5).



Figure 3: Data Center After MetaFabric

The goal of the MetaFabric architecture is to allow you to connect any physical network,

with any combination of storage, servers, or hypervisors, to any virtual network, andwith

any orchestration software (Figure 4 on page 5). Such an open ecosystem ensures that

you can add new equipment, features, and technologies over time to take advantage of

the latest trends as they emerge.

Figure 4: MetaFabric – Putting It All Together

The MetaFabric architecture addresses the problems common in today’s data center by

delivering a network and security architecture that accelerates time to value, while

simultaneously increasing value over time. The MetaFabric 1.0 virtualized IT data center

solutiondescribed in this guide is the first implementationof theMetaFabric architecture.

Future solutionsandguidesareplanned, includinga larger scale virtualized ITdata center,

IT as a service (ITaaS), and amassively scalable cloud data center.

Domain

This guide addresses the needs that enterprise companies have for an efficient and

integrateddatacenter. It discusses thedesignand implementationaspects for acomplete

suite of compute resources, network infrastructure, and storage components that you

need to implement and support a virtualized environment within your data center. This

guide also discusses the key customer requirements provided by the solution, such as

business-critical applications (such as Microsoft Exchange and SharePoint), high

availability, class of service, security, and network management.


Chapter 1: Overview

Goals

The primary goal of this solution is to enable data center operators to design and

implement an IT data center that supports a virtualized environment for large Enterprise

customers. The data center scales up to 2,000 servers and 20,000 virtual machines

(VMs) that run business-critical applications.

The MetaFabric 1.0 solution provides a simple, open, and smart architecture and solves

several challenges experienced in traditional data centers:

• Complexity—Typically, legacy data centers have been implemented in an incremental

fashionwithwhatever vendorgave themthebestdeal. The result is that thearchitecture

provides no end-to-end services or management. The solution is to reduce the

complexity andmake the data center simple to operate andmanage.

• Cost—The cost of managing a complex data center can be high. The solution is to

create an open data center to drive operational efficiencies and reduce cost.

• Rigidity—Building a data center based on incremental demands ultimately results in

an architecture that is too rigid and not able to adapt to newworkloads or provide the

agility that anevolvingbusinessdemands. The solution is to createa smart architecture

from the beginning that can adapt and be agile to new demands.

Figure 5: MetaFabric Architecture

Examples of how the MetaFabric architecture solves real-world problems include:

• Simple—This solution uses two QFabric systems. Each QFabric system acts like a

single, very large switch and only requires onemanagement IP address for 16 racks of

equipment. In effect, management tasks are reduced by over 90%.

• Open—Juniper Networks devices use standards-based Layer 3 protocols and interact

with VMware vCenter APIs. In addition, this solution includes interoperability with

ecosystem partners such as VMware, EMC, IBM, and F5 Networks.

• Smart—In this solution, smart workloadmobility with automated orchestration and

template-based provisioning is provided by using Network Director.

The features in a simple, open, and smart architecture in your data center include:



• Integrated solution—By designing a data center with integration inmind, you can blend

heterogeneous equipment and software frommultiple vendors into a comprehensive

system. This enables your network to interact efficiently with compute and storage

components that work well together.

• Seamless VMMobility—By designing an architecture that supports the movement of

VMs from one location in the data center to another, VMs can be stopped,moved, and

restarted in a different location in the data center with flexibility and ease.

• Network visibility—By designing a data center to provide VM visibility, you can connect

the dots between the virtual and physical components in your system. You will know

how your VMs are connected to switches and understand the vMotion history of a VM.

• Scale and virtualization—The solution scales to 20,000 VMs and can support either a

100 percent virtualized compute environment or a mixed physical and virtual

environment.

Benefits of the solution include:

• Peace ofmind—Knowing that a solution has been tested and validated reduces the

anxietyof implementinganew ITproject. This solutionprovidespeaceofmindbecause

it has been thoroughly tested by the Juniper Networks Solutions Validation team.

• Reduce deployment rime—Integrating products frommultiple vendors takes time and

effort, resulting in lost productivity caused by interoperability issues. This solution

eliminates such issues because the interoperability and integration has already been

verified by the Juniper Networks Solutions Validation team.

• Reduce CAPEX—Capital expenditures go up when different equipment is added in a

piecemeal fashion and needs to be replaced or upgraded to achieve new business

goals. This solution factors in the goals and scalability ahead of time, resulting in lower

cost of ownership.

• Best of breed—Another pitfall of buying equipment in an incremental fashion is that

legacy equipment often cannot scale to the same levels as newer equipment. This

solution selects cutting-edge equipment that is designed towork together seamlessly

and in harmony.

• Pre-packaged solution—Having to design, evaluate, and test a data center

implementation from a variety of vendors is a lot of work. This solution takes the

guesswork out of such an effort and provides a cohesive set of products designed to

meet your business needs for your data center.

Audience

This MetaFabric 1.0 solution is designed for enterprise IT departments that wish to build

a complete end-to-end data center that contains compute, storage, and network

components optimized for a virtualized environment. The enterprise IT data center

segment represents the majority of Fortune 500 companies.

The primary audience for this guide includes the following technical staff members:


Chapter 1: Overview

• Network/data center/cloud architects—Responsible for creating the overall design of

the network architecture that supports their company’s business objectives.

• Datacenterengineers—Responsible forworkingwitharchitects, planners, andoperation

engineers to design and implement the solution.

Validated Solution Design and Implementation Guide Overview

Juniper Networks creates end-to-end solutions in conjunction with select third-party

partners, such as VMware and IBM. These integrated solutions enable our customers to

implement comprehensive IT projects to accomplish business goals. Our reference

architectures are designed by subject matter experts and verified through in-house

solution testing, which uses a detailed framework to validate the solution from both a

network and an application perspective. Testing andmeasuring applications at scale

verify the integration of the network, compute, storage, and related components.

Juniper Networks validated solutions are complete, purpose-built, domain architectures

that:

• Solve specific problems

• Have undergone end-to-end validation testing

• Are thoroughly documented to provide clear deployment guidance

Juniper Networks solution validation labs subject all solutions to extensive testing using

bothsimulationand livenetworkelements toensurecomprehensivevalidation.Customer

use cases, common domain examples, and field experience are combined to generate

prescriptive configurations and architectures to inform customer and partner

implementations of Juniper Networks solutions. A solution-based approach enables

partners and customers to reduce time to certify and verify new designs by providing

tested, prescriptive configurations to use as a baseline. Juniper Networks solution

validation provides the peace of mind and confidence that the solution behaves as

described in a real-world production environment.

This guide is intended to be the first in a series of guides that enable our customers to

build effective data centers to meet specific business goals.

MetaFabric 1.0 Overview

To provide flexibility to your implementation of the virtualized IT data center, there are

several sizes of the MetaFabric 1.0 solution. As seen in Figure 6 on page 9, you can start

with a small implementation and grow your data center network into a large one over

time. The reference architecture tested and documented in this guide uses the large

topology option with two QFX3000-MQFabric points of delivery (PODs) instead of six.



Figure 6: Juniper Networks Virtualized IT Data Center - Sizing Options

The small option shown in Figure 6 on page 9 uses two QFX3600 switches for

aggregation and six QFX3500 switches for access. Two 40-Gigabit Ethernet ports on

theQFX3500switchareusedasuplinks,while theother twoare split into four40-Gigabit

Ethernet server ports. As a result, each QFX3500 switch has 56 network ports and

implements 7:1 oversubscription. Themedium option is a single QFX3000-MQFabric

systemwith 64 network ports and 768 server ports, resulting in 3:1 oversubscription. The

large option uses 7:1 oversubscription and consists of 6 QFX3000-MQFabric systems.

NOTE: A fourth option not shown in the diagramwould be to replace the 6QFX3000-MQFabric systemswithoneQFX3000-GQFabric systemtobuilda data center containing 6144 ports.

The different sizing options solution offer different port densities to meet the growing

needs of the data center. The predefined configuration and provisioning options that

cover the small,medium, and largedeployment scenarios are shown inTable 1 onpage9.

Table 1: Juniper Networks Virtualized IT Data Center – Details of Sizing Options

LargeMediumSmall

1446412Network Ports

4032768336Server Ports

6 (QFX3000-MQFabric)1 (QFX3000-MQFabric)8Switches

128208Rack Units


Chapter 1: Overview

Solution Overview

This MetaFabric 1.0 solution identifies the key components necessary to accomplish the

specifiedgoals. Thesecomponents includecompute, network, andstorage requirements,

as well as considerations for business-critical applications, high availability, class of

service, security, and network management (Figure 7 on page 10). As a result of these

requirements and considerations, it is critical that all components are configured,

integrated, and tested end-to-end to guarantee service-level agreements (SLAs) to

support the business.

Figure 7: Juniper Networks Virtualized IT Data Center – SolutionComponents

The following sections describe the general requirements you need to include in a

virtualized IT data center.

• Compute on page 10

• Network on page 11

• Storage on page 12

• Applications on page 12


• Class of Service on page 13

• Security on page 13

• Network Management on page 14

Compute

Because this solution is focused on a virtualized IT environment, naturally many of the

requirements are driven by virtualization itself. Compute resourcemanagement involves

the provisioning andmaintenance of virtual servers and resources thatmust be centrally

managed. The requirements for compute resources within a virtualized IT data center

include:



• Workloadmobility andmigration for VMs—Applications must be able to bemigrated

to other virtual machines when resource contention thresholds are reached.

• Location independence for VMs—An administrator must be able to place the VMs on

any available compute resource andmove them to any other server as needed, even

between PODs.

• VM visibility—An administrator must be able to view where the virtual machines are

located in the data center and generate reports on VMmovement.

• High availability—Compute resources must be ready and operational to meet user

demands.

• Fault tolerance—If VMs fail, there should be ways for the administrator to recover the

VMs or move them to another compute resource.

• Centralized virtual switchmanagement—Keeping themanagement for VMs and virtual

switches in one place alleviates the hassle of logging into multiple devices to manage

dispersed virtual equipment.

Network

The network acts as the glue that binds together the data center services, compute, and

storage resources. To support application and storage traffic, you need to consider what

is requiredat theaccessandaggregationswitching levels, core switching, andedge router

tiers of your data center. These are the areas that Juniper Networks understands best,

so we can help you in selecting the correct networking equipment to support your

implementation of the virtualized IT data center.

The requirements for a virtualized IT data center network include:

• 1-Gigabit, 10-Gigabit, and 40-Gigabit Ethernet Ports—This requirement covers themost

common interface types in the data center.

• Convergeddataandstorage—By sending data and storage traffic over a single network,

this reduces the cost required to build, operate, andmaintain separate networks for

data and storage.

• Load balancing—By distributing and alternating the traffic over multiple paths, this

ensures an efficient use of bandwidth and resources to prevent unnecessary

bottlenecks.

• Applicationqualityofexperience—Bydesigningclassof service requirements fordifferent

traffic queues, this ensures prioritization for mission-critical traffic (such as storage

and business-critical applications) and best effort handling for routine traffic (such as

e-mail).

• Networksegmentation—Breaking thenetwork intodifferentportions lowers theamount

of traffic congestion, and improves security, reliability, and performance.

• Traffic isolation and separation—By carefully planning traffic flows, you can keep

East-to-West and North-to-South data center traffic separate from each other and

prevent traffic from traveling across unnecessary hops to reach its destination. This


Chapter 1: Overview

allowsmost traffic to flow locally, which reduces latency and improves application

performance.

• Time synchronization—This requirement ensures that a consistent time stamp is

standardized across the data center for management andmonitoring purposes.

Generally speaking, you need to determine which Layer 2 and Layer 3 hardware and

software protocols meet your needs to provide a solid foundation for the traffic that

flows through your data center.

Storage

There are two primary types of storage: local storage and shared storage. Local storage

is generally directly attached to a server or endpoint. Shared storage is a shared resource

in the data center that provides storage services to a set of endpoints. The MetaFabric

1.0 solution focuses primarily on shared storage as it is the foundation for all of the

endpoint storagewithinadatacenter. Sharedstoragecanbebrokendown into sixprimary

roles: controller, front end, back end, disk shelves, RAID groups, and storage pools.

Although there are many different types of shared storage that vary per vendor, the

architectural building blocks remain the same. Each storage role has a very specific role

and function in order to deliver shared storage to a set of endpoints.

The requirements for storage within a virtualized IT data center include:

• Scale—The storage component must be able to handle sufficient input/output

operations per second (IOPS) to support business-critical applications.

• Lossless Ethernet—This is a requirement for converged storage.

• Boot from shared storage—The advantages of this requirement include easier server

maintenance, more robust storage (such as more disks, more capacity, and faster

storage processors), and easier upgrade options.

• Multiple protocol storage—The storage device must be able to support multiple types

of storage protocols, such as Internet Small Computer System Interface (iSCSI),

Network File System (NFS), and Fiber Channel over Ethernet (FCoE). This provides

flexibility to the administrator to integrate different types of storage as needed.

Applications

For your applications, you need to consider the user experience and plan your

implementation accordingly. Business-critical applications provide the main reason for

the existence of the data center. The other data center components (such as compute,

network, and storage) serve to ensure that these applications are hosted securely in a

manner that can provide a high-quality user experience.Web services, e-mail, database,

and collaboration tools are housed in the data center – these tools form the basis for

business efficiency andmust deliver application performance at scale. As such, the data

center architecture should focus on delivering a high-quality user experience through

coordinated operation across all tiers of the data center.

For example, can theWeb, application, and database tiers communicate properly with

eachother? If youplan toallowVMmotion tooccuronlywithinanaccessandaggregation



POD, you can include Layer 3 integrated routing and bridging (IRB)within the access and

aggregation layer. However, if you choose to move VMs from one POD to another, you

need to configure the IRB interface at the core layer to allow the VM to reach theWeb,

application, and database servers that are still located in the original POD. Factoring in

such design aspects ahead of time prevents headaches to the data center administrator

in the months and years to come.

The requirements for applications within a virtualized IT data center include:

• Business-critical applications—The solution must address common data center

applications.

• Highperformance—Applicationsmustbedelivered tousers ina timely fashion toensure

smooth operations.

High Availability

Keeping your equipment up and running so that traffic can continue to flow through the

data center is a must to ensure that applications run smoothly for your customers. You

should strive to build a robust infrastructure that can withstand outages, failover, and

software upgrades without impacting your end users. High availability should include

both hardware and software components, along with verification. Key considerations

for high availability in an virtualized IT data center include:

• Hardware redundancy—At least two redundant devices should be placed at each layer

of the data center to ensure resiliency for traffic. If one device fails, the other device

should still be able to forward data and storage packets to their destinations. The data

center requires redundant network connectivity and the ability for traffic to use all

available bandwidth.

• Software redundancy—Features such as nonstop software upgrade, Virtual Router

Redundancy Protocol (VRRP), graceful restart, MC-LAG, and graceful Routing Engine

switchover (GRES) are needed tomaintain device uptime, providemultiple forwarding

paths, and ensure stability in the data center.

Class of Service

Because of the storage requirements in the virtualized IT data center, you must include

lossless Ethernet transport in your design tomeet the needs for converged storage in the

solution.Also, youmust consider thevarying levelsof classof servicenecessary to support

end-to-end business-critical applications, virtualization control, network control, and

best-effort traffic.

Security

Another important task is to secure your data center environment from both external

and internal threats.Because this solutioncontainsbothphysical andvirtual components,

youmust secure both the applications and traffic that flow through the heart of the data

center (oftenacrossVMs)aswell as theperimeter of thedatacenter (consistingprimarily

of physical hardware, such as an edge firewall). Youmust also provide secure remote

access to the administrators who are managing the data center.


Chapter 1: Overview

Security requirements for this solution include:

• Perimeter security—Using hardware-based security provides services such as Network

Address Translation (NAT), encrypted tunnels, and intrusion detection to prevent

attacks and prohibit unauthorized access.

• Application security—Use of a software solution for application security provides

network segmentation, robust policies, and intrusion detection.

• Remote access—Implementing a secure access method provides two-factor

authentication and Role-Based Access Control (RBAC) to allow access to authorized

data center administrators.

NetworkManagement

The final challenge is connecting the dots between physical and virtual networking;

bridging this gap enables the data center engineer to quickly troubleshoot and resolve

issues. For network management in a virtualized IT data center, you need to consider

management of fault, configuration, accounting, performance and security (FCAPS) in

your network (Figure 8 on page 14).

Figure 8: Network Management Requirements

For more information about FCAPS (the ISOmodel for network management), see

ISO/IEC 10040.

Network management requirements for the solution include:

• Virtual and physical—Youmust be able tomanage all types of components in the data

center network, regardless if they are hardware-based or virtualized.

• Fault—Errors in the network must be isolated andmanaged in the most efficient way

possible. You should be able to recognize, isolate, correct, and log faults that occur in

your network.

• Configuration—You should be able to provision your network flexibly from a central

location andmanage configurations for the devices in your data center.

• Accounting—Youmust be able to gather network usage statistics, and establish users,

passwords, and permissions.



• Performance—You should be able to themonitor throughput, network response times,

packet loss rates, link utilization, percentage utilization, and error rates to ensure the

network continues to perform at acceptable levels.

• Security—Youmust be able to control access to network components through use of

authorization, encryption, and authentication protocols.


Chapter 1: Overview

CHAPTER 2

Design

• Design Considerations on page 17

• Design Scope on page 18

• Design Topology Diagram on page 19

• Design Highlights on page 20

• Solution Design on page 21

• Summary of Key Design Elements on page 58

• Benefits on page 58

Design Considerations

As seen in the Overview, designing a virtualized IT data center requires careful

consideration of the three key segments of compute, network, and storage, along with

their related subareas:

• Compute

• Virtual machines

• Servers

• Hypervisor switch

• Blade switch

• Network

• Access

• Aggregation

• Core switching

• Edge routing

• WAN

• Storage


The design must also include careful planning of other architectural considerations:

• Applications

• High availability

• • Class of service

• Security

• Network management

In general, the design for the solutionmust satisfy the following high-level requirements:

• The entire data center must have end-to-end convergence for application traffic of

under one second from the point of view of the application.

• Compute nodes must be able to use all available network links for forwarding.

• Traffic must be able to travel between the points of delivery PODs.

• Virtual resources must be able to bemoved within a POD.

• The out-of-band (OOB)management network must be able to survive the failure of

the data plane within a POD.

Design Scope

This MetaFabric 1.0 solution covers the areas shown in Figure 9 on page 18. Juniper

Networks supplies products that appear in the blue portions of the diagram, while open

ecosystempartner products appear in the black portion. The ecosystempartners for this

solution include IBM (Compute), EMC (Storage), F5 Networks (Services), and VMware

(Virtualization).

Figure 9: Virtualized IT Data Center Ecosystem



Design Topology Diagram

Figure 10 on page 19 shows the general layout of the hardware components included in

the MetaFabric 1.0 solution architecture.

Figure 10: Virtualized IT Data Center Topology


Chapter 2: Design

Design Highlights

Table 2 on page 20 shows the key features of the MetaFabric 1.0 solution and how they

are implementedwith hardware and software from JuniperNetworks andour third-party

ecosystem partners.

Table 2: MetaFabric 1.0 Solution Design Highlights

ImplementationFeature

IBM Flex System servers, VMware vSphere 5.1, vCenterCompute and virtualization

MX240 routers, EX9214 switchesCore and edge network

QFX3000-MQFabric systemAccess and aggregation

OSPF, BGP, IRB, and VLANsLayer 2 and Layer 3 protocols

EMC VNX5500 unified storageStorage

Microsoft SharePoint, Microsoft Exchange, andWikiMedia run at scaleApplications

Nonstopsoftwareupgrade, in-servicesoftwareupgrade,SRXJSRPcluster,MC-LAG Active/Active with VRRP

High availability

Lossless Ethernet, end-to-end application class of serviceClass of service

SRX3600, Firefly HostSecurity

Junos Pulse Gateway SARemote access

Junos Space Network Director 1.5, Security DirectorNetwork management

EX4300 Virtual ChassisOut-of-bandmanagement network

F5 LTM Load BalancerApplication load balancer



Solution Design

This section explains the compute resources, network infrastructure, and storage

components required to implement the MetaFabric 1.0 solution. It also discusses the

softwareapplications, highavailability, classof service, security, andnetworkmanagement

components of this solution.

The purpose of the data center is to host business-critical applications for the enterprise.

Each role in the data center is designed and configured to ensure the highest quality user

experience possible. All of the functional roleswithin the data center exist to support the

applications in the data center.

• Compute on page 21

• Network on page 26

• Storage on page 35



• Class of Service on page 47

• Security on page 48

• Network Management on page 53

• Performance and Scale on page 57

Compute

In the compute area, you need to select the physical and virtual components that will

host your business-critical applications, network management, and security services.

This includes careful selection of VMs, servers, hypervisor switches, and blade switches.

Virtual Machines

A virtual machine (VM) is a virtual computer that is made up of a host operating system

and applications. A hypervisor is software that runs on a physical server, emulating

physical hardware forVMs.TheVMoperateson theemulatedhardwareof thehypervisor.

The VM believes that it is running on dedicated, physical hardware. This layer of

abstraction enables the benefit of presentation to the operating system; regardless of

changes to the hardware, the operating system sees the same set of logical hardware.

This enables operators to make changes to the physical environment without causing

issues on the servers hosted in the virtual environment, as seen in Figure 11 on page 22.


Chapter 2: Design

Figure 11: Virtual Machine Design

Virtualization also enables flexibility that is not possible on physical servers. Operating

systems can bemigrated from one set of physical hardware to another with very little

effort. Completeenvironments, to include theoperatingsystemand installedapplications,

can be cloned in a virtual environment, enabling complete backups of the environment

or, in somecases, youcancloneor recreate identical serversondifferentphysical hardware

for redundancy or mobility purposes. These clones can be activated upon primary VM

failure and enable an easy level of redundancy to exist at the data center application

layer. An extension to the benefit of cloning is that newoperating systems can be created

from these clones very quickly, enabling faster service rollouts and faster time to revenue

for new services.

Servers

The server in the virtualized IT data center is simply the physical compute resource that

hosts the VMs. The server offers processing power, storage, memory, and I/O services

to the VMs. The hypervisor is installed directly on top of the servers without any sort of

host operating system, becoming a bare-metal operating system that provides a

framework for virtualization in the data center.

Because the server hosts the revenue generating portion of the data center (the VMs

and resident applications), redundancy is essential at this layer. A virtualized IT data

center server must support full hardware redundancy, management redundancy, the

ability to upgrade softwarewhile the server is in service, hot swapping of power supplies,

cooling, andother components, and theability tocombinemultiple serverorbladechassis

into a single, logical management plane.

The server chassismust be able to provide transport between the physical hardware and

virtual components, connect to hosts through 10-Gigabit Ethernet ports, use 10-Gigabit

Ethernet or 40-Gigabit Ethernet interfaces to access the POD, consolidate storage, data,

andmanagement functions, provide class of service, reduce theneed for physical cables,

and provide active/active forwarding.



Figure 12: Server Design

As seen in Figure 12 on page 23, this solution includes 40-Gigabit Ethernet connections

between QFabric system redundant server Node groups and IBM Flex servers that host

up to 14 blade servers. Other supported connection types include 10-Gigabit Ethernet

oversubscribed ports and 10-Gigabit Ethernet pass-through ports. The solution also has

two built-in switches per Flex server and uses MC-LAG to keep traffic flowing through

the data center.

Hypervisor Switching

The hypervisor switch is the first hop from the application servers in the MetaFabric 1.0

architecture. Virtual machines connect to a distributed virtual switch (dvSwitch) which

is responsible formappinga setof physical network cards (pNICs)acrossa set of physical

hosts into a single logical switch that can be centrally managed by a virtualization

orchestration tool suchasVMware vCenter (Figure 13 onpage 23). ThedvSwitch enables

intra-VM traffic on the same switching domain to pass between the VMs locally without

leaving the blade server or virtual environment. The dvSwitch also acts like a Virtual

Chassis, connectsmultipleESXi hosts simultaneously, andoffersport group functionality

(similar to a VLAN) to provide access between VMs.

Figure 13: VMware Distributed Virtual Switch

This poses an interesting security challenge on the hypervisor switch, as traditional,

appliance-based firewallsdonothavevisibility into thehypervisor switchingenvironment.


Chapter 2: Design

In cases where restrictions must be placed on VM-to-VM traffic, security software can

be installed on the hypervisor to perform firewall functions between VMs.

The hypervisor switch is a critical piece of the MetaFabric 1.0 architecture. As such, it

should support functions that enable class of service and SLA attainment. Support for

IEEE802.1p is required to support class of service. Support for link aggregation of parallel

links (IEEE 802.3ad) is also required to ensure redundant connection of VMs. As in the

other switching roles, support for SLA attainment is also a necessity at this layer. The

hypervisor switch should support SNMPv3, flow accounting and statistics, remote port

mirroring, and centralizedmanagement and reporting to ensure that SLAs can be

measured and verified.

To complete the configuration for the hypervisor switch, provide class of service on flows

for IP storage, vMotion, management, fault tolerance, and VM traffic. As shown in

Figure 14 on page 24, this solution implements the following allocations for network

input/output (I/O) control shares: IP storage (33.3 percent), vMotion (33.3 percent),

management (8.3 percent), fault tolerance (8.3 percent), and VM traffic (16.6 percent).

These categories have beenmaximized for server-level traffic.

Figure 14: VMware Network I/O Control Design

Blade Switching

The virtualized IT data center features virtual appliances that are often hosted on blade

servers, or servers that supportmultiple interchangeable processing blades that give the

blade server the ability to host large numbers of VMs. The blade server includes power

and cooling modules as well as input/output (I/O) modules that enable Ethernet

connection into the blade server (Figure 15 on page 25). Blade switching is performed

between the physical Ethernet port on the I/Omodule and the internal Ethernet port on

the blade. In some blade servers, a 1:1 subscriptionmodel (one physical port connects to

one blade) is used (this is called pass-thru switching), with one external Ethernet port

connecting directly to a specific blade via an internal Ethernet port. The pass-through

model offers the benefit of allowing full line bandwidth to each blade server without

oversubscription. Thedownside to this approach is oftena lackof flexibility inVMmobility

and provisioning as VLAN interfaces need to bemoved on the physical switch and the

blade switch when amove is required.



Figure 15: Sample Blade Switch, Rear View

Another mode of blade switch operation is where the blade switch enables

oversubscription to the blade servers. In this type of blade server, there may be only 4

external ports that connect internally to 12 separate blade servers. This would result in

3:1 oversubscription (three internal ports to every one external port). The benefit to this

mode of operation is that it minimizes the number of connected interfaces and access

switch cabling per blade server, even though the performance of oversubscribed links

and their connected VMs can degrade as a result. While this architecture is designed for

data centers that utilize blade servers, the design works just as well in data centers that

do not utilize blade servers to host VMs.

Table 3 on page 26 shows that both pass-through blade servers and oversubscribed

blade servers are acceptable choices for this solution in your data center network. In

some cases, youmight need the faster speed provided by the 40-Gigabit Ethernet

connections to support newer equipment, while in others you would prefer the line-rate

performance offered by a pass-through switch. As a result, all three blade server types

are supported in this design.


Chapter 2: Design

Table 3: Comparison of Pass-Through Blade Servers and Oversubscribed Blade Servers

40G Chassis SW10G Chassis SWPass-Through SWAttribute

YesYesYesTransport

YesYesYes10-Gigabit Ethernet host interface

YesNoNo40-Gigabit Ethernet uplink interface

YesYesYesConsolidate storage, data, andmanagement

YesYesYesClass of service

Yes (2:14)Yes (12:14)NoCable reduction

3.5:11.2:11:1Oversubscription

YesYesYesActive/Active

To provide support for compute and virtualization in the virtualized IT data center, this

solution uses:

• Virtualmachines—VMsrunningWindowsandapplications, suchasMicrosoftSharePoint,

Microsoft Exchange, andWikiMedia

• Servers—IBM x3750 and IBM Flex System chassis

• Configure an IBMFlex Systemserverwithmultiple ESXi hosts supporting all theVMs

running business-critical applications (SharePoint, Exchange, and MediaWiki).

• Configure a distributed vSwitch betweenmultiple physical ESXi hosts configured

on the IBM servers.

• Hypervisor—VMware vSphere 5.1 and vCenter

• Blade switches—IBM EN4091 and CN4093

This design for the compute and virtualization segment of the data center meets the

requirements of this solution for workloadmobility andmigration for VMs, location

independence for VMs, VM visibility, high availability, fault tolerance, and centralized

virtual switch management.

Network

The network is often themain focus of the data center as it is built to pass traffic to, from,

and between application servers hosted in the data center. Given the criticality of this

architectural role, and the various tierswithin the data center switching block, it is further

broken up into access switching, aggregation switching, core switching, edge routing,

andWAN connectivity. Each segment within the data center switching role has unique

design considerations that relate back to business criticality, SLA requirements,

redundancy, and performance. It is within the data center switching architectural roles



that the network must be carefully designed to ensure that your data center equipment

purchases maximize network scale and performance while minimizing costs.

Access and Aggregation

The access layer consists of physical switches that connect to servers and end hosts.

Access switching typically focuses on implementing Layer 2 switches, but can include

Layer 3 components (such as IRB) to supportmore robust VMmobility. Access switching

should also support high availability. In a multi-chassis or virtual chassis environment,

where multiple physical switches can be combined to form a single, logical switch,

redundancy can be achieved at the access layer. This type of switch architecture is built

with control plane redundancy, MC-LAG, and the ability to upgrade individual switches

while they are in service. Additionally, the access switching role should support storage

traffic, or the ability to pass data traffic over Ethernet via iSCSI and Fiber Channel over

Ethernet (FCoE). Data Center Bridging (DCB) should also be supported by the access

switching role to enable full support of storage traffic. Within DCB, support for

priority-based flow control (PFC), enhanced transmission selection (ETS), and data

center bridging exchange (DCBX) should also be supported as these features enable

storage traffic to pass properly between all servers and storage devices within a data

center segment.

The aggregation switch acts as amultiplexing point between the access and the core of

the data center. The aggregation architectural role serves to combine a large number of

smaller interfaces from the access into high bandwidth trunk ports that can bemore

easily consumed by the core switch. Redundancy should be a priority in the design of the

aggregation role as all Layer 2 flows between the data center and the core switch are

combined and forwarded by the data center aggregation switch role. At this layer, a

switching architecture that supports the combination of multiple switches into a single,

logical systemwith control and forwarding plane redundancy is recommended. This

switchingarchitectureenables redundancy featuressuchasMC-LAG, loop-free redundant

paths, and in-service software upgrades to enable data center administrators to

consistently meet and exceed SLAs.

One recommendation is to combine the access and aggregation layers of your network

by using a QFabric system. Not only does a QFabric system offer a single point of

provisioning,management, and troubleshooting for thenetworkoperator, it alsocollapses

switching tiers for any-to-any connectivity, provides lower latency, andenables all access

devices to be only one hop away from one another, as shown in Figure 16 on page 28.


Chapter 2: Design

Figure 16: Juniper Networks QFabric Systems Enable a Flat Data CenterNetwork

To implement the access and aggregation switching portions of the virtualized IT data

center, this solutionuses theQFX3000-MQFabric system.Thereare twoQFabric systems

(POD1 and POD2) in this solution to provide performance and scale. The QFabric PODs

support 768 ports per POD and feature low port-to-port latency, a single point of

management per POD, and lossless Ethernet to support storage traffic. The use of

predefined POD configurations enables the enterprise to more effectively plan data

center rollouts by offering predictable growth and scale in the solution architecture. Key

configuration steps include:

• Configure the QFX3000-MQFabric systems with 3 redundant server Node groups

(RSNGs) connected to 2 IBM Flex System blade servers to deliver application traffic.

• The first IBM Flex System server uses a 40-Gigabit Ethernet converged network

adapter (CNA) connected to a QFabric system RSNG containing QFX3600 Node

devices (RSNG4).

• The second IBM Flex System server has 10-Gigabit Ethernet pass throughmodules

connected to RSNG2 and RSNG3 on the second QFabric system.

• Connect the EMC VNX storage platform to the QFabric systems for storage access

using iSCSI and NFS.

• Connect the QFabric systems with the EX9214 core switch by way of a network Node

group containing 2 Node devices which use four 24-port LAGs configured as trunk

ports.

• ConfigureOSPF in thePODs (within theQFabric systemnetworkNodegroup) towards

the EX9214 core switch andplace these connections in Area10 as a totally stubby area.

Core Switching

The core switch is often configured as a Layer 3 device that handles routing between

various Layer 2 domains in the data center. A robust implementation of the core switch

in the virtualized IT data center will support both Layer 2 and Layer 3 to enable a full

range of interoperability and service provisioning in amultitenant environment.Much like

in the edge role, the redundancy of core switching is critical as it too is a traffic congestion



pointbetween thecustomerand theapplication.Aproperlydesigneddatacenter includes

a fully redundant core switch layer that supports a wide range of interfaces (1-Gigabit,

10-Gigabit, 40-Gigabit, and 100-Gigabit Ethernet) with high density. The port density in

the core switching role is a critical factor as the data center core should be designed to

support futureexpansionwithout requiringnewhardware (beyond linecardsand interface

adapters). The core switch role should also support a wide array of SLA statistics

collection, andshouldbeservice-aware tosupport collectionof service-chainingstatistics.

The general location of the core switching function in this solution is shown in

Figure 17 on page 29.

Figure 17: Core Switching

Table 4 on page 30 shows some of the reasons for choosing an EX9200 switch over an

EX8200switch toprovide core switching capabilities in this solution. TheEX9200switch

providesa significantly larger numberof 10-Gigabit Ethernetports, support for 40-Gigabit

Ethernetports, ability tohostmoreanalyzer sessions, firewall filters, andBFDconnections,

and critical support for in-service software upgrade (ISSU) andMC-LAG. These reasons

make the EX9200 switch the superior choice in this solution.


Chapter 2: Design

Table 4: Core Switch Hardware - Comparison of the EX9200 and EX8200 Switches

DeltaEX9200EX8200Solution Requirement

+88%240128Line-rate 10G

YesNo40G

+815%647Analyzer Sessions

+375%256K54KACLs

+415%900175BFD

YesNo(NSSU)ISSU

YesNoMC-LAG

Table 5 on page 30 shows some of the reasons for choosing MC-LAG as the forwarding

technology over Virtual Chassis in this solution. MC-LAG provides dual control planes, a

non-disruptive implementation, support for LACP, state replication across peers, and

support for ISSUwithout requiring dual Routing Engines.

Table 5: Core Switch Forwarding - Comparison of MC-LAG and Virtual Chassis

MC-LAGVirtual ChassisAttribute

21Control Planes

NoYesCentralized Management

22Maximum Chassis

Non-disruptiveDisruptiveImplementation

YesNoRequire IEEE 802.3ad (LACP)

ICCPKernelState Replication

NoYesRequire Dual Routing Engines

YesNoISSU

To implement the core switching portion of the virtualized IT data center, this solution

uses two EX9214 switches with the following capabilities and configuration:

• Keyfeatures—240Gbps line rateper slot for 10-Gigabit Ethernet, support for 40-Gigabit

Ethernet ports, 64 analyzer sessions, scalable to 256,000 firewall filters, and support

for bidirectional forwarding detection (BFD), in-service software upgrade (ISSU), and

MC-LAG groups.



• Key configuration steps (Figure 18 on page 31)

• Configure Layer 2 MC-LAG active/active on the EX9214 towards the QFabric PODs,

the F5 load balancer, and theMX240 edge router (byway of the redundant Ethernet

link provided by the SRX3600 edge firewall) to provide path redundancy.

• Configure IRB and VRRP for all MC-LAG links for high availability.

• Configure IRB on the EX9214 and the QFabric PODs to terminate the Layer 2/Layer

3 boundary.

• Configure a static route on the core switches to direct traffic from the Internet to the

load balancers.

• o Configure OSPF to advertise a default route to the totally stubby areas in the

QFabric PODs. EachQFabric PODhas its ownOSPFarea. Also, configure the EX9214

core switches as area border routers (ABRs) that connect all three OSPF areas, and

designatebackbonearea0over aggregated linkae20between the twocore switches

Figure 18: Core Switching Design

Edge Routing andWAN

Edge Routing

Theedge is thepoint in thenetwork thataggregatesall customerand Internet connections

into and out of the data center. Although high availability and redundancy are important

considerations throughout the data center, it is at the edge that they are the most vital;

the edge serves as a choke point for all data center traffic and a loss at this layer renders

the data center out of service. At the edge, full hardware redundancy should be

implemented using platforms that support control plane and forwarding plane

redundancy, link aggregation, MC-LAG, redundant uplinks, and the ability to upgrade the

software and platformwhile the data center is in service. This architectural role should

support a full range of protocols to ensure that the data center can support any

interconnect type that may be offered. Edge routers in the data center require support

for IPv4 and IPv6, as well as ISO and MPLS protocols. As the data center might be

multi-tenant, the widest array of routing protocols should also be supported, to include

static routing,RIP,OSPF,OSPF-TE,OSPFv3, IS-IS, andBGP.With large scalemulti-tenant

environments in mind, it is important to support Virtual Private LAN Service (VPLS)


Chapter 2: Design

through the support of bridge domains, overlapping VLAN IDs, integrated routing and

bridging (IRB), and IEEE802.1Q(QinQ).Theedgeshould support acomplete setofMPLS

VPNs, includingL3VPN, L2VPN(RFC4905andRFC6624, orMartini andKompelladrafts,

respectively), and VPLS.

Network Address Translation (NAT) is another factor to consider when designing the

data center edge. It is likely thatmultiple customers serviced by the data center will have

overlapping private network address schemes. In environments where direct Internet

access to the data center is enabled, NAT is required to translate routable, public IP

addresses to the private IP addressing used in the data center. The edgemust support

Basic NAT 44, NAPT44, NAPT66, Twice NAT44, and NAPT-PT.

Finally, as the edge is the ingress and egress point of the data center, the implementation

should support robust data collection to enable administrators to verify and prove strict

service-level agreements (SLAs) with their customers. The edge layer should support

collection of average traffic flows and statistics, and at a minimum should support the

ability to report exact traffic statistics to include the exact number of bytes and packets

that were received, transmitted, queued, lost, or dropped, per application.

Figure 19 on page 33 shows the location of the edge routing function in this solution.



Figure 19: Edge Routing

WAN

TheWAN role provides transport between end users, enterprise remote sites, and the

data center. There are several differentWAN topologies that can be used, depending on

the business requirements of the data center. A data center can simply connect directly

to the Internet, utilizing simple IP-based access directly to servers in the data center, or

a secure tunneled approach using generic routing encapsulation (GRE) or IP Security

(IPsec).Manydata centers serve awidebaseof customers and favorMultiprotocol Label

Switching (MPLS) interconnection via the service provider’s managed MPLS network,

allowing customers to connect directly into the data center via the carrier’s MPLS

backbone. Another approach to theWAN is to enable direct peering between customers

and the data center; this approach enables customers to bypass transit peering links by

establishingadirect connection (for example, viaprivate leased line) into thedatacenter.

Depending on the requirements of the business and the performance requirements of

the data center hosted applications, the choice ofWAN interconnection offers the first

choice indeterminingperformanceandsecurity of thedatacenter applications. Choosing

a private peering or MPLS interconnect offers improved security and performance at a

higher expense. In cases where the hosted applications are not as sensitive to security


Chapter 2: Design

and performance, or where application protocols offer built-in security, a simple Internet

connected data center can offer an appropriate level of security and performance at a

lower cost.

To implement the edge routing andWAN portions of the virtualized IT data center, this

solution uses MX240 Universal Edge routers. Because the MX240 router offers dual

Routing Engines and ISSU at a reasonable price point, it is the preferred option over the

smaller MX80 router. The key connection and configuration steps are:

• Connect the MX240 edge routers to the service provider networks to provide Internet

access to the data center.

• Configure the two edge routers to be EBGP peers with 2 service providers to provide

redundant Internet connections.

• Configure IBGP between the 2 edge routers and applying a next-hop self export policy.

• Configure BGP local preference on the primary service provider to offer a preferred exit

point to the Internet.

• Export a dynamic, condition-based, default route to the Internet into OSPF on both

edge routers toward the edge firewalls and core switches to provide Internet access

for the virtualized IT data center devices (Figure 20 on page 34).

• Configure both edge routers in Area 1 for OSPF.

• Enable Network Address Translation (NAT) to convert private IP addresses into public

IP addresses.

Figure 20: Edge Routing Design

This design for the network segment of the data center meets the requirements of this

solution for 1-Gigabit, 10-Gigabit, and 40-Gigabit Ethernet ports, converged data and

storage, load balancing, quality of experience, network segmentation, traffic isolation

and separation, and time synchronization.



Storage

The storage role of theMetaFabric 1.0 architecture is to provide centralized file and block

data storage so that all hosts inside of the data center can access it. The data storage

can be local to a VM, such as a database that resides within a hosted application, or

shared, such as a MySQL database that can reside on a storage array to serve multiple

different applications. TheMetaFabric 1.0 architecture requires the use of shared storage

to enable compute virtualization and VMmobility.

One of the key goals of the virtualized IT data center is to converge both data and storage

onto the same network infrastructure to reduce the overall cost andmake operations

and troubleshooting easier. There are several different options when converging storage

traffic: FCoE, NFS, and iSCSI. One of themost recent trends in building a green-field data

center is to use IP storage and intentionally choose not to integrate legacy Fibre Channel

networks.Additionally, because iSCSIhasbetterperformance, lower read-write response

times, lower cost, and full application support, iSCSI offers the better storage network

choice over NFS. Additionally, storage traffic is very latency and drop sensitive, so it is

critical that the network infrastructure provide a lossless Ethernet service to correctly

prioritize all storage traffic. As a result, this solution uses both iSCSI and NAS for storage,

and provides a lossless Ethernet service to guarantee the delivery of storage traffic.

Table 6onpage 35 showsa comparison of FCoE, NFS, and iSCSI. BecauseNFSand iSCSI

meet the same requirements provided by FCoE, plus the ability to scale to 10-Gigabit

Ethernet and beyond, the NFS and iSCSI storage protocols are the preferred choice for

the MetaFabric 1.0 solution.

Table 6: Comparison of Storage Protocols

iSCSINFSFCoERequirement

YesYesYesLossless Ethernet

YesYesNo10GE and beyond

YesYesYesConverged data and storage

YesYesYesLess than 3p end-to-end latency

Figure 21 onpage36 shows thepathof storage traffic as it travels through thedata center

and highlights the benefit of priority queuing to provide lossless Ethernet transport for

storage traffic. By configuringPriority FlowControl (PFC), the storage device canmonitor

storage traffic in the storage VLAN and notify the server when traffic congestion occurs.

The server can pause sending additional storage traffic until after the storage device has

cleared the congested receive buffers. However, other queues are not affected and

uncongested traffic continues flowing without interruption.


Chapter 2: Design

Figure 21: Storage Lossless Ethernet Design

The packet flow for storage traffic is as follows:

1. The server transmits storage traffic to the QFabric system.

2. The QFabric system classifies the traffic.

3. Traffic is queued according to priority.

4. The QFabric system transmits the traffic to the storage array.

5. The storage array receives the traffic.

6. The storage array transmits traffic back to the QFabric system.

7. The QFabric system classifies the traffic.

8. Traffic is queued according to priority.

9. The QFabric system transmits the traffic to the servers and VMs.

10. The server receives the traffic.

To implement the storage portion of the virtualized IT data center, this solution uses EMC

VNX5500 unified storage with a single storage array. This storage is connected to the

QFabric PODs, which in turn connect to the servers and VMs, as seen in

Figure 22 on page 37. The design assumes that the data center architect wishes to save

on cost initially by sharing a single storage array with multiple QFabric PODs. However,

the design can evolve to allocating one storage array per oneQFabric POD, as usage and

demand warrant such expansion.



Figure 22: Storage Design

This solution also implements Data Center Bridging (DCB) to enable full support of

storage traffic. Within DCB, support for priority-based flow control (PFC), enhanced

transmission selection (ETS), and Data Center Bridging Capability Exchange (DCBX)

enables storage traffic to pass properly between all servers and storage devices within

a data center segment and to deliver a lossless Ethernet environment.

This design for the storage segment of the virtualized IT data center meets the

requirements of this solution for scale, lossless Ethernet, the ability to boot from shared

storage, and support for multiple protocol storage.

Applications

Applications in the virtualized IT data center are built as Virtual Machines (VMs) and are

hosted on servers, or physical compute resources that reside on the blade server. This

design for applications meets the requirements of this solution for business-critical

applications and high performance.

The MetaFabric 1.0 solution supports a complete software stack that covers four major

application categories: computemanagement, networkmanagement, network services,

and business-critical applications (Figure 23 on page 38). These applications run on top

of IBM servers and VMware vSphere 5.1.


Chapter 2: Design

Figure 23: Virtualized IT Data Center Solution Software Stack

ComputeManagement

VMware vCenter is a virtualizationmanagement platform that offers centralized control

and visibility into compute, storage, and networking resources. Data center operators

use the de facto, industry-standard vCenter on a daily basis to manage and provision

VMs. VMware vCloud Director allows the data center manager to create an in-house

cloud service and partition the virtualization environment into segments that can be

administered by separate business units or administrative entities. The pool of resources

can now be partitioned into virtual data centers which can offer their own independent

virtualization services. Use of vCenter and vCloud Director offers the first element of

software application support for the MetaFabric 1.0 solution.

NetworkManagement

The MetaFabric 1.0 solution uses Junos Space Management Applications to provide

networkprovisioning, orchestration, and inventorymanagement. Theapplications include

Network Director for management of wired and wireless data center networks, and

Security Director for security policy administration.

Network Services

Network load balancing is a common network service. There are twomethods to provide

network load balancing: virtual and hardware-based. The virtual load balancer operates

in thehypervisor asaVM.Oneof thebenefits of a virtual loadbalancer is rapidprovisioning

of additional load-balancing power. Another benefit is that the administration of the

virtual loadbalancer canbedelegated toanother administrative entitywithout impacting

other applications and traffic.

However the drawback to a virtual load balancer is that the performance is limited to

the number of compute resources that are available. Hardware load balancers offer

muchmore performance in traffic throughput and SSL encryption and decryption with

dedicated security hardware.

The MetaFabric 1.0 solution uses the local traffic manager (LTM) from F5 Networks.



The load balancers provide the following services:

• Advertise the existence of the application

• Distribute the traffic across a set of servers.

• Leverage features such as SSL acceleration and compression.

• Provide additional Layer 7 features.

Business-Critical Applications

Software applications are made of multiple server tiers; the most common areWeb,

application, anddatabase servers. Each server has its owndiscrete set of responsibilities.

TheWeb tier handles the interaction with the users and the application. The application

tier handles all of the application logic and programming. The database tier handles all

of the data storage and application inventory.

The following software applications were tested as part of the MetaFabric 1.0 solution:

• Microsoft SharePoint

The SharePoint application requires three tiers: Web, application, and database. The

Web tier usesMicrosoft IIS to handleWeb tracking and interactionwith end users. The

application tier uses Microsoft SharePoint and Active Directory to provide the file

sharing and content management software. Finally, the database tier uses Microsoft

SQL Server to store and organize the application data.

• Microsoft Exchange

TheExchangeapplication requires two tiers: aWeb tier, andasecond tier that combines

the application and the database into a single tier.

• MediaWiki Application

The MediaWiki application requires two tiers: a combinedWeb and application tier,

and a database tier. Apache httpd is combinedwith the hypertext preprocessor (PHP)

to render and present the application, while the data is stored on the database tier

with MySQL.

High Availability

Thisdesignmeets thehighavailability requirementsofhardware redundancyandsoftware

redundancy.

Hardware Redundancy

To provide hardware redundancy in the virtualized IT data center, this solution uses:

• Redundant server hardware—Two IBM 3750 standalone servers and two IBM Pure Flex

System Chassis

• Redundant access and aggregation PODs—TwoQFX3000-MQFabric systems

• Redundant core switches—Two EX9214 switches

• Redundant edge firewalls—Two SRX3600 Services Gateways


Chapter 2: Design

• Redundant edge routers—TwoMX240 Universal Edge routers

• Redundant storage—Two EMC VNX5500 unified storage

• Redundant load balancers—Two F5 LTM 4200v load balancers

• Out-of-bandmanagement switches use Virtual Chassis technology—Four EX4300

switches

Software Redundancy

To provide software redundancy in the virtualized IT data center, this solution uses:

• Graceful restart—Helper routers assist restarting devices in restoring routing protocols,

state, and convergence.

• Graceful Routing Engine switchover—Keeps the operating system state synchronized

between themaster and backup Routing Engines in a Juniper Network device.

• In-service software upgrade(for the core switches and edge routers)—Enables the

network operating system to be upgraded without downtime.

• MC-LAG—Enables aggregated Ethernet interface bundles to contain interfaces from

more than one device.

• Nonstop active routing—Keeps the Layer 3 protocol state synchronized between the

master and backup Routing Engines.

• Nonstop bridging—Keeps the Layer 2 protocol state synchronized between themaster

and backup Routing Engines.

• Nonstopsoftwareupgrade—(for theQFX3000-MQFabric systemPODs)—Enables the

network operating system to be upgraded with minimal impact to forwarding.

• Virtual Router Redundancy Protocol (VRRP)—Provides a virtual IP address for traffic

and forwards the traffic to one of two peer routers, depending on which one is

operational.

MC-LAGDesign Considerations

To allow all the links to forward traffic without using Spanning Tree Protocol (STP), you

can configureMC-LAG on edge routers and core switches. The edge routers useMC-LAG

toward the edge firewalls, and the core switches useMC-LAG toward eachQFabric POD,

application load balancer (F5), and out-of-band (OOB)management switch.

Multichassis link aggregation group (MC-LAG) is a feature that supports aggregated

Ethernet bundles spreadacrossmore thanonedevice. LinkAggregationControl Protocol

(LACP) supportsMC-LAGand is used for dynamic configuration andmonitoring on links.

The available options for MC-LAG include Active/Standby (where one device is active

and the other assists if the active device fails) or Active/Active (where both devices

actively participate in the MC-LAG connection).

For this solution, MC-LAG Active/Active is preferred because it provides link-level and

node-level protection for Layer 2 networks and Layer 2/Layer 3 combined hybrid

environments.



Highlights of MC-LAG Active/Active

MC-LAG Active/Active has the following characteristics:

• Both core switches have active aggregated Ethernet member interfaces and forward

the traffic. If one of the core switches fails, the other core switchwill forward the traffic.

Traffic is load balanced by default, so link-level efficiency is 100 percent.

• The Active/Active method has faster convergence than the Active/Standbymethod.

Fastconvergenceoccursbecause information isexchangedbetweenthe routersduring

operations. After a failure, the remaining operational core switch does not need to

relearn any routes and continues to forward the traffic.

• Routing protocols (such as OSPF) can be used over MC-LAG/IRB interfaces for Layer

3 termination.

• If you configure Layer 3 protocols in the core, you can use an integrated routing and

bridging (IRB) interface to offer a hybrid Layer 2 and Layer 3 environment at the core

switch.

• Active/Active also offers maximum utilization of resources and end-to-end load

balancing.

To extend a link aggregation group (LAG) across two devices (MC-LAG):

• Both devices need to synchronize their aggregated Ethernet LACP configurations

• Learned MAC address and ARP entries must be synchronized.

The above MC-LAG requirements are achieved by using the following

protocols/mechanisms as shown in Figure 24 on page 41:

1. Interchassis Control Protocol (ICCP)

2. Interchassis Link Protection Link (ICL-PL)

Figure 24: MC-LAG – ICCP and ICL Design


Chapter 2: Design

1. ICCP

• ICCP is a control plane protocol for MC-LAG. It uses TCP as a transport protocol

and Bidirectional Forwarding Detection (BFD) for fast convergence. When you

configure ICCP, youmust also configure BFD.

• ICCP synchronizes configurations and operational states between the twoMC-LAG

peers.

• ICCP also synchronizes MAC address and ARP entries learned from one MC-LAG

node and shares themwith the other peer.

• • Peering with the ICCP peer loopback IP address is recommended to avoid any

direct link failurebetweenMC-LAGpeers.As longas the logical connectionbetween

the peers remains up, ICCP stays up.

• Although you can configure ICCP on either a single link or an aggregated bundle

link, an aggregated Ethernet LAG is preferred.

• • You can also configure ICCP and ICL links on a single aggregated Ethernet bundle

undermultiple logical interfacesusing flexibleVLANtagging supportedonMXSeries

platforms.

2. ICL-PL

• ICL is a special layer 2 link for Active-Active only between the MC-LAG peers

• ICL-PL is needed to protect connectivity of MC-LAG in case of failure of all core

facing links corresponding to one MC-LAG node.

• If the traffic receiver is single homed to one of the MC-LAG nodes (N1), ICL is used

to forward thepackets receivedbywayof theMC-LAG interface to theotherMC-LAG

nodes (N2).

• Split horizon is enabled to avoid loop on the ICL

• There is no data plane MAC learning over ICL.

MC-LAG Specific Configuration Parameters

Redundancy group ID—ICCP uses a redundancy group to associate multiple chassis that

performsimilar redundancy functions.A redundancygroupestablishesacommunication

channel so that applications on ICCP peers can reach each other. A redundancy group

ID is similar to a mesh group identifier.

MC-AE ID—Themulti-chassis aggregated Ethernet (MC-AE) ID is a per-multi-chassis

interface. For example, if one MC-AE interface is spread across multiple core switches,

you should assign the same redundancy group ID. When an application wants to send a

message to a particular redundancy group, the application provides the information and

ICCP delivers it to the members of that redundancy group.

Service ID—Anewservice IDobject for bridgedomainsoverridesanyglobal switchoptions

configuration for the bridge domain. The service ID is unique across the entire network

for agivenservice toallowcorrect synchronization. For example, a service IDsynchronizes

applications like IGMP, ARP, andMACaddress learning for a given service across the core



switches. (Note: Both MC-LAG peers must share the same service ID for a given bridge

domain.)

MC-LAG Active/Active Layer 3 Routing Features

MC-LAGActive/Active is a Layer 2 logical link. IRB interfaces are used to create integrated

Layer 2 and Layer 3 links. As a result, you have two design options when assigning IP

addresses across MC-LAG peers:

• Option 1: VRRPMC-LAGActive/Active provides common virtual IP andMACaddresses

and unique physical IP and MAC addresses. Both address types are needed if you

configure routing protocols on MC-LAG Active/Active interfaces. The VRRP data

forwarding logic has beenmodified in Junos OS if you configure both MC-LAG

Active/ActiveandVRRP.Whenconfiguredsimultaneously, both theMC-LAGandVRRP

peers forward traffic and load-balance the traffic between them, as shown in

Figure 25 on page 43.

Figure 25: VRRP andMC-LAG – Active/Active Option

Data packets received by the backup VRRP peer on the MC-LAGmember link are

forwarded to the core link without sending them to themaster VRRP peer.

• Option 2: MAC address synchronization Figure 26 on page 44 provides a unique IP

address per peer, but shares a MAC address between the MC-LAG peers. You should

use option 2 if you do not plan to configure routing protocols on the MC-LAG

Active/Active interfaces.


Chapter 2: Design

Figure 26: MC-LAG –MACAddress Synchronization Option

• You configure the same IP address on the IRB interfaces of both node.

• The lowest MAC address is selected as the gateway MAC address.

• • The peer with the higher IRB MAC address learns the peer’s MAC address through

ICCP and installs the peer MAC address as its ownMAC address.

• OnMX Series platforms, configuremcae-mac-synchronize in the bridge domain

configuration.

• On EX9214 switches, configuremcae-mac-synchronize in a VLAN configuration.

We recommendOption 1 as the preferred method for the MetaFabric 1.0 solution for the

following reasons:

• The solution requires OSPF as the routing protocol between the QFabric PODs and

the core switches on the MC-LAG IRB interfaces and only Option 1 supports routing

protocols.

• Layer 3 extends to the QFabric PODs for some VLANs for hybrid Layer 2/Layer 3

connectivity to the core.

MC-LAG Active/Active Traffic Forwarding Rules

Figure 27: MC-LAG – Traffic Forwarding Rules

As shown in Figure 27 on page 44, the following forwarding rules apply to MC-LAG

Active/Active:



• Traffic received on N1 fromMCAE1 could be flooded to the ICL link to reach N2. When

it reaches N2, it must not be flooded back to MCAE1.

• Traffic received on SH1 could be flooded to MCAE1 and ICL by way of N1. When N2

receives SH1 traffic across the ICL link, it must not be again flooded to MCAE1. N2 also

receives the SH1 traffic by way of the MC-AE link.

• When receiving a packet from the ICL link, the MC-LAG peers forward the traffic to

all local SH links. If the corresponding MCAE link on the peer is down, the receiving

peer also forwards the traffic to its MCAE links.

NOTE: ICCP is used to signal MCAE link state between the peers.

• When N2 receives traffic from the ICL link and the N1 core link is up, the traffic should

not be forwarded to the N2 core link.

MC-LAG Active/Active High Availability Events

ICCP is down, when ICL is up:

Figure 28: MC-LAG – ICCP Down

Here are the actions that happen when the ICCP link is down and the ICL link is up:

• By default, if the ICCP link fails, as shown in Figure 28 on page 45, the peer defaults to

its own local LACP system ID and the links for only one peer (whichever one negotiates

with the customer edge [CE] router first) are attached to the bundle. Until LACP

converges with a new system ID, there will be minimum traffic impact.

• One peer stays active, while the other enters standbymode (but this is

nondeterministic).

• The access switch selects a core switch and establishes LACP peering.

To optimize for this condition, include the prefer-status-control-active statement on the

active peer.

• With the prefer-status-control-activestatement configured on the active peer, the peer

remains active and retains the same LACP system ID.


Chapter 2: Design

• With the force-icl-down statement, the ICL link shuts down when the ICCP link fails.

• By configuring these statements, traffic impact isminimized during an ICCP link failure.

ICCP is up and ICL goes down:

Figure 29: MC-LAG – ICL Down

Here are the actions that happen when the ICCP link is up and the ICL link is down:

• If you configure a peer with the prefer-status-control-standby statement, the MC-AE

interfaces shared with the peer and connected to the ICL go down.

• This configuration ensures a loop-free topology because it does not forward duplicate

packets in the Layer 2 network.

Active MC-LAG node downwith ICCP loopback peering with prefer-status-control-active

on both peers:

Figure 30: MC-LAG – Peer Down

Here are the actions that happen when both MC-LAG peers are configured with the

prefer-status-control-active statement and the active peer goes down:

• When you configure MC-LAG Active/Active between SW1/SW2 and the QFabric POD,

SW1 becomes active and SW2 becomes standby. During an ICCP failure event, if SW1

has the prefer-status-control active statement and it fails, SW2 is not aware of the



ICCP or SW1 failures. As a result, SW2mcae-id switches to the default LACP system

ID, which causes the MC-LAG link to go down and up, and results in long traffic

reconvergence times.

• To avoid this situation, configure the prefer-status-control-active statement on both

SW1andSW2.Also, youshouldprevent ICCP failuresbyconfiguring ICCPona loopback

interface.

• Configure backup-liveness-detection on both the active and standby peers. BFD helps

to detect peer failures and enable sub-second reconvergence.

The design for high availability in theMetaFabric 1.0 solutionmeets the requirements for

hardware redundancy and software redundancy.

Class of Service

Key design elements for class of service in this solution include network control (OSPF,

BGP, and BFD), virtualization control (high availability, fault tolerance), storage (iSCSI

andNAS), business-critical applications (Exchange,SharePoint,MediaWiki, andvMotion)

and best-effort traffic. As seen in Figure 31 on page 47, incoming packets are sorted,

assigned to queues based on traffic type, and transmitted based on the importance of

the traffic. For example, iSCSI lossless Ethernet traffic has the largest queue and highest

priority, followed by critical traffic (fault tolerance and high availability), business-critical

application traffic (including vMotion), andbulkbest-effort trafficwith the lowestpriority.

Figure 31: Class of Service – Classification and Queuing

As seen in Figure 32 on page 48, the following percentages are allocated for class of

service in this solution: network control (5 percent), virtualization control (5 percent),

storage (60 percent), business-critical applications (25 percent) and best-effort traffic

(5 percent). These categories have beenmaximized for network-level traffic, as the

network supportsmultiple serversandswitches.Asa result, storage traffic andapplication

traffic are the most critical traffic types in the network, and these allocations have been

verified by our testing.


Chapter 2: Design

Figure 32: Class of Service – Buffer and Transmit Design

To provide class of service in the virtualized IT data center andmeet the design

requirements, this solution uses:

• Lossless Ethernet for storage traffic

• Five allocations to differentiate traffic

• A queue for business-critical applications

• Best-effort traffic for data traffic

Security

Security is a vital component of any network architecture and the virtualized IT data

center is no exception. There are various areas within the data center where security is

essential. At the perimeter, security is focused on securing the edge of the data center

fromexternal threatsandwithprovidinga securegateway to the Internet. Remoteaccess

is another area where security is vital in the data center. Operators will often require

remote access to the data center to performmaintenance or new service activations.

This remote accessmust be secured andmonitored to ensure that only authorized users

are permitted access. Robust authentication, authorization and accounting (AAA)

mechanisms should be in place to ensure that only authorized operators are allowed.

Given that the data center is a cost and revenue center that can house the critical data

andapplicationsofmanydifferententerprises,multi-factorauthentication isanabsolute

necessity to properly secure remote access.

Software application security in the virtualized IT data center is security that is provided

between VMs. A great deal of inter-VM communication occurs in the data center and

controlling this interactivity is a crucial security concern. If a server is supposed to access

a database residing on another server, or on a storage array, a virtual security appliance

should be configured to limit the communication between those resources to allow only

those protocols that are necessary for operation. Limiting the communication between

resources prevents security breaches in the data center andmight be a requirement

dependingon the regulatory requirementsof thehostedapplications (HIPPA, for instance,



can dictate security safeguards that must exist between patient and business data). As

discussed in the VirtualMachine section, security in the virtual network, or betweenVMs,

differs fromsecurity that canbe implementedonaphysical network. Inaphysical network,

a hardware firewall can connect to different subnets, security zones, or servers and

provide security between those devices (Figure 33 on page 49). In the virtual network,

the physical firewall does not have the ability to see traffic between the VMs. In these

cases, a virtual hypervisor security appliance should be installed to enable security

between VMs.

Figure 33: Physical Security Compared to Virtual Network Security

Application Security

When securing VMs, you need a comprehensive virtualization security solution that

implements hypervisor security with full introspection; includes a high-performance,

hypervisor-based stateful firewall; uses an integrated intrusion detection system (IDS);

provides virtualization-specific antivirus protection; and offers unrivaled scalability for

managing multitenant cloud data center security. The Juniper Networks Firefly Host

(formerly vGW) offers all these features and enables the operator to monitor software,

patches, and files installed on a VM from a central location. Firefly Host is designed to

be centrally managed from a single-pane view, giving administrators a comprehensive

view of virtual network security and VM inventory.

Table 7 on page 50 shows the relativemerits of three application security design options:

vSRX,SRX,andFireflyHost.Becauseotherchoices lack intrusiondetectionandprevention,

quarantine capabilities, andmission-critical line-rate performance and scalability, Firefly

Host is the preferred choice for this solution. Additionally, Firefly Host is integrated into

all VMs and provides every endpoint with its own virtual firewall.


Chapter 2: Design

Table 7: Application Security Options

Firefly HostSRXvSRXRequirement

YesYesYesStateful security policies

YesYesYesCentralizedmanagement

YesYesYesIntrusion detection and prevention

YesNoNoQuarantine

YesNoNo10G line-rate performance at scale

To provide application security in the virtualized IT data center, this solution uses the

Juniper Networks Firefly Host to provide VM-to-VM application security. Firefly Host

integrates with VMware vCenter for comprehensive VM security andmanagement.

Figure 34: Application Security Design

In Figure 34 on page 50, the following sequence occurs for VM-to-VM traffic:

1. A VM sends traffic to a destination VM.

2. The Firefly Host appliance inspects the traffic.

3. The traffic matches the security policy.

4. The ESXi host transmits the traffic.

5. The second ESXi host receives the traffic.

6. Firefly Host inspects the traffic.

7. The traffic matches the security policy and permits the traffic to continue to the

destination.

8. The destination VM receives the traffic.



Perimeter Security

Edge firewalls handle security functions such as Network Address Translation (NAT),

intrusion detection and prevention (IDP), security policy enforcement, and virtual private

network (VPN) services. As shown in Figure 35 on page 51, there are four locationswhere

you could provide security services for the physical devices in your data center:

1. Firewall filters in the QFabric system PODs

2. Firewall filters in the core switches

3. Dedicated, stateful firewalls (like the SRX3600)

4. Physical firewalls connected to the QFabric system PODs

Figure 35: Physical Security Design

This solution implements option 3, which uses a stateful firewall to protect traffic flows

travelling between the edge routers and core switches. Anything below the POD level is

protected by the Firefly Host application.


Chapter 2: Design

To provide perimeter security in the virtualized IT data center, this solution uses the

SRX3600 Services Gateway as an edge firewall. This firewall offers up to 55-Gbps of

firewall performance, which can easily support the VM traffic generated by this solution.

The key configuration tasks include:

• Configure the SRX gateways as an active/backup cluster.

• Place redundant Ethernet group reth1 (configured toward the edge routers) in the

non-trust zone.

• Place reth0 (configured toward the core switches) in the trust zone.

• Configure a security policy for traffic coming from the non-trust zone to allow only

access to data center applications.

• Configure Source Network Address Translation (SNAT) for Internet access to the

application servers (private address) to provide Internet access.

• Configure Destination Network Address Translation (DNAT) for remote access to the

data center by translating the Pulse gateway internal IP address to an

Internet-accessible IP address.

• Configure the edge firewalls in OSPF area 1.

Secure Remote Access

The MetaFabric 1.0 solution requires secure remote access into the data center

environment. Such access must provide multifactor authentication, granular security

controls, and user scale that give multitenant data centers the ability to provide access

to administrators and access to many thousands of users.

The secure remote access application must be accessible through the Internet; capable

of providing encryption, RBAC, and two-factor authentication services; able to access a

virtualized environment; and scale to 10,000 users.

Table8onpage52showsacomparisonof theMAGgatewayand the JunosPulsegateway

options. For this solution, the Junos Pulse gateway is superior because it offers all the

capabilities of the MAG gateway as well as being a virtualized application.

Table 8: Data Center Remote Access Options

Virtual Pulse GatewayMAGGatewayRequirement

YesYesInternet accessible

YesYesEncryption

YesYesTwo-factor authentication

YesYesScale to 10,000 users

YesNoVirtualized



To provide secure remote access to and from the virtualized IT data center, this solution

uses the Juniper Networks SASeries SSLVPNAppliances as remote access systems and

the Junos Pulse gateway.

Figure 36: Remote Access Flow

As shown in Figure 36onpage53, the remote access flow in the virtualized IT data center

happens as follows:

1. The user logs in from the Internet.

2. The user session is routed to the firewall.

3. Destination NAT is performed on the session.

4. The authorized user matches the security policy.

5. The traffic is forwarded to the Junos Pulse gateway.

6. Traffic arrives on the Untrust interface.

7. Trusted traffic permits a local address to be assigned to the user.

8. The user is authenticated and granted access through RBAC.

Thisdesign for security in theMetaFabric 1.0 solutionmeets the requirements forperimeter

security, application security, and secure remote access.

NetworkManagement

Network management is often reduced to its basic services: fault, configuration,

accounting, performance, and security (FCAPS). In the virtualized ITdata center, network

management ismore than a simple tool that facilitates FCAPS: it is an enabler to growth

and innovation that provides end-to-end orchestration of all data center resources.

Effective network management provides a single-pane view of the data center. This

single-pane view enables visibility andmobility and enables the data center operator to

monitor and change the environment across all data center tiers. Networkmanagement

in thevirtualized ITdatacenter canbebrokendown into seven tiers (Figure37onpage54).


Chapter 2: Design

Figure 37: Seven Tier Model of Network Management

It is the combinationof these tiers thatprovides completeorchestration in thedata center

andenablesoperators to turnupnewservicesquickly, andchangeor troubleshootexisting

services using a single-pane view of the data center. The user interface is responsible for

interactingwith the data center operator. This is the interface fromwhich the data center

single-pane view is presented. From the user interface, an operator can view, modify,

delete, or add network elements and new services. The user interface acts as a single

role-based access control (RBAC) policy enforcement point, allowing an operator

seamless access to all authorized devices while protecting other resources from

unapproved access. The application programming interface (API) enables single-pane

management by providing a common interface and language to other applications,

support tools, anddevices in thedatacenternetwork (RESTAPI isanexamplecommonly

used in network management). The API enables the single-pane view by abstracting all

support elements and presenting them through a single networkmanagement interface

– the user interface.

The network management platform should have the capability to support specialized

applications. Applications in the network management space are specifically designed

to solve a specific problem in themanagement of the data center environment. A single

application on the network management platform can be responsible for configuring

andmonitoring the security elements in the data center, while another application is

designed to manage the physical and virtual switching components in the data center.

Again, the abstraction of all of these applications into a single-pane view is essential to

data center operations to ensure simplicity and a commonmanagement point in the

data center.

The next tier of data center network management is the global network view. Simply

put, this is the tier where complete view of the data center and its resources can be

assembled and viewed. This layer should support topology discovery, the automatic

discovery of not only devices, but how those devices are interconnected to one another.



Theglobalnetworkviewshouldalsosupportpathcomputation(the linkdistancebetween

network elements as well as the set of established paths between those network

elements). The resourcevirtualization tier ofnetworkmanagementenablesmanagement

of the various endpoints in the data center and acts as an abstraction layer that allows

the operator to manage endpoints that require different protocols such as OpenFlow or

Device Management Interface (DMI).

Thecommondataservices tierofnetworkmanagementenables thevariousapplications

and interfaceson thenetworkmanagementsystemtoshare relevant informationbetween

the layers.Anapplication thatmanagesasetof endpointsmight requirenetwork topology

details in order to map and potentially push changes to those network devices. This

requires that the applications within the network management system share data; this

is enabled by the common data services layer.

Managed devices in the network management role are simply the endpoints that are

managedby thenetworkmanagement system.Thesedevices includephysical andvirtual

switches, routers, VMs, blade servers, and security appliances, to name a few. The

managed devices and the orchestration of services between those devices is the prime

purpose of the network management system. Network management should be the

answer to the question, ”how does a data center operator easily stand up andmaintain

services within the data center?” The network management system orchestrates the

implementation and operation of the managed devices in the data center

Finally, integrationadaptersare requiredwithinacompletenetworkmanagement system.

As every device in the data center might not bemanageable by a single network

management system, other appliances or services might be required to manage the

entiredatacenter.The integrationandcoordinationof thesevariousnetworkmanagement

tools is the purpose of this layer. Some data center elements such as Virtual Machines

might require VMware ESXi server to manage the VMs and hypervisor switch, while

another network management appliancemonitors environmental and performance

conditions on the host server. A third systemmight be responsible for configuring and

monitoring the network connections between the blade servers and the rest of the data

center. Integration adapters enable each of these components to talk to one another

and, in many cases, allow a single network management system to control the entire

network management footprint from a single pane of glass.

Out-of-BandManagement

The requirements for out-of-bandmanagement include:

• Administration of the compute, network, and storage segments of the data center.

• Separation of the control plane from the data plane so themanagement network

remains accessible.

• Support for 1-Gigabit Ethernet management interfaces.

• Provide traffic separation across compute, network, and storage segments.

• Enable administrators access to the management network.

• Denymanagement-to-management traffic.

Some of the key elements of this design are seen in Figure 38 on page 56.


Chapter 2: Design

Figure 38: Out of BandManagement Network Design

To provide out-of-bandmanagement in the virtualized IT data center, this solution uses

two pairs of EX4300 switches configured as a Virtual Chassis (Figure 39 on page 57).

The key connection and configuration steps include:

• • Connect all OOB network devices to the EX4300 Virtual Chassis (100-Megabit Fast

Ethernet and 1-Gigabit Ethernet).

• Configure the EX4300 Virtual Chassis OOBmanagement system in OSPF area 2.

• Connect the 2 IBM 3750 standalone servers that host themanagement VMs (vCenter,

Junos Space, Network Director 1.5, domain controller, and Junos Pulse gateway) to the

EX4300 Virtual Chassis.

• • Create four VLANs to separate storage, compute, network, andmanagement traffic

from each other.

• Manage andmonitor the VMs on the test bed using VMware vSphere and Network

Director 1.5.



Figure 39: Out of BandManagement – Detail

Network Director

To provide network configuration and provisioning in the virtualized IT data center, this

solution uses Juniper Networks Network Director. Network Director 1.5 is used tomanage

network configuration, provisioning, andmonitoring

Security Director

Toprovide security policy configuration in the virtualized IT data center, this solution uses

Juniper Networks Security Director. Security Director is used to manage security policy

configuration and provisioning.

This designmeets the networkmanagement requirements ofmanaging both virtual and

physical components within the data center and handling the FCAPS considerations.

Performance and Scale

• The solution must support 20,000 virtual machines and scale up to 2,000 servers.

• The solution must support a total of 30,000 users.

• 10,000Microsoft Exchange users

• 10,000Microsoft SharePoint user transactions

• 10,000MediaWiki user transactions

• The solutionmust offer less than 3μ latency between servers and 21μ latency between

PODs

• The solution must provide high availability.

• Less than one second convergence*

• No single point of failure


Chapter 2: Design

Summary of Key Design Elements

To recap the key components of the design for this solution, see Table 9 on page 58.

Table 9: Summary of Key Design Elements – Virtualized IT Data Center Solution

SoftwareHardware PlatformRole

13.2R1.7MX480Edge Router

12.1X44-D30.4SRX3600Edge Firewall

13.2R3.2EX9214Core Switch

13.1X50-D15QFX3000-MPOD 1

13.1X50-D15QFX3000-MPOD 2

13.2X50-D10EX4300-VCOOBManagement

7.4R1.0vSecure AccessRemote Access

1.5Network DirectorNetworkManagement

7.1.56-5EMC VNX 5500Storage

2PET10KIBM Flex SystemCompute

10.2.5 Build 591.0F5 VIPRION 4480Load Balancer

5.1VMware vSphereVirtualization

5.5Firefly HostApplication Security

Benefits

There are several benefits to the virtualized IT data center described in this guide:

• Agility—The solution enables rapid deployment of new VMs and services.

• VMmotion—Virtual machines can bemoved and redeployed quickly anywhere in the

data center.

• Visibility of physical and virtual elements—The solution offers insight for the

administrator into theentire compute, network, andstoragesystemforboth thephysical

components and the virtual machines and switches.

• Simplifiedmanagement—The solution implements several components, such as the

QFabric system, Network Director, vCenter, and Firefly Host, that provide single points

ofmanagement that reduce thenumber of logins that are required for anadministrator

to monitor andmaintain the data center.



• Systemuptime—Highavailability features, suchasMC-LAG,nonstopsoftwareupgrades,

in-service software upgrades, VRRP, and hardware redundancy, are included in the

solution to keep your data center up and running.

• Customer confidence—Through the efforts of the Juniper Networks Solution Validation

team, the validated solution in this guide can provide quicker understanding, adoption,

and implementation of the solution itself to provide your virtualized IT data centerwith

a competitive edge.


Chapter 2: Design

CHAPTER 3

MetaFabric 1.0 High Level Testing andValidation Overview

The following section provides a high-level overview of the MetaFabric 1.0 solution.

Specific implementation details for each operational role begin in Chapter 4.


• Key Characteristics of Implementation on page 62

Overview

Topology

This solution document focuses on a deployment scenario that provides a

performance-oriented fabricwith 3:1 over-subscription, supporting up to 768x10-Gigabit

portsand thedeploymentofup to7PODs.Thesolution features low-latencydeterministic

traffic forwarding using the Juniper Networks QFX3000-MQFabric System in the access

and aggregation roles (at the POD level). The end-to-end lab testing topology is shown

in Figure 40 on page 62.


Figure 40: The End to End Lab Topology

Key Characteristics of Implementation

The MetaFabric 1.0 solution was verified in the Juniper Networks solution validation labs

using the following set of design elements and features:

• Transport: MC-LAG Active/Active with VRRP and SRX JSRP, IRB and VLAN, Lossless

Ethernet

• Protocols: OSPF, BGP

• High availability: NSSU, ISSU, SRX Cluster

• Security: Perimeter - SRX3600, App Security - Firefly Host

• Remote access: SA network and VM appliance configuration



• OOB: EX4300-VC

• Compute and virtualization: IBM Flex chassis, VMware 5.1, vCenter

• Network management: Juno Space, Network Director 1.5

• Application load balancer: F5 DSR LB implementation

• Quality of service: Lossless Ethernet, PFC, DCBX

• Scale and performance: SharePoint, Exchange, Wikimedia scale with Shenick Tester

• POD1 and POD2 are configured with the QFX3000-MQFabric system as an access

and aggregation switch.

POD1 (QFX3000-MQFabric) Configuration

ThePOD1 JuniperNetworks®QFX3000-MQFabricSystem isconfiguredwith the following

elements:

• Three redundant server node groups (RSNG) connected to two IBM Flex blade servers

• IBMFlex-1 has 40-Gigabit CNA connected to anRSNGwithQFX3600nodes (RSNG4)

• IBM-Flex-2 has 10-Gigabit pass-thru modules connected to RSNG2 and RSNG3

• EMC VNX storage is connected to the QFabric for storage access though iSCSI and

NFS

• QFX3000-MQFabric system is also configured with one network node group (NNG)

with two nodes connected to EX9214 core-switch using 4 X 24port link aggregation

groups (LAGs) configured as trunk ports

• POD1 (QFabric NNG toward EX9214): Area10 (totally stubby area)

POD2 (QFX3000-MQFabric) Configuration

The Juniper Networks®QFX3000-MQFabric System deployed in POD2 is configured

with the following elements:

• Three RSNG connected to two IBM Flex blade servers

• IBM IBM-Flex-2 has 10-Gigabit pass-thru modules connected to RSNG2 and RSNG3

• EMC VNX storage is connected to the QFabric for storage access though iSCSI and

NFS

• QFX3000-MQFabric system is also configured with one NNG node-group with two

nodes connected to EX9214 core-switch using 4 X 32 port LAGs configured as trunk

ports

• POD2 (QFabric NNG toward EX9214): Area11 (totally stubby area)


Chapter 3: MetaFabric 1.0 High Level Testing and Validation Overview

Core Switch (EX9214) Implementation

The core role deployed in the solution verification lab features the Juniper Networks®

EX9214 Ethernet Switch with the following configuration elements:

• Layer 2 MC-LAG Active/Active is configured on EX9214 toward QFabric-m, F5 LB,

MX240 toward SRX3600

• IRB is configured on EX9214 and QFabric and QFX-VC to terminate the Layer 2/Layer

3 boundary

• Static route is configured to core-switch the traffic to load balancer (LB) from Internet

• OSPF is configured to send only default routes to the NSSA areas toward POD1 and

POD2

• IRB and VRRP are configured for all MC-LAG links

• Core-switch is configured as ABR having all three areas connected

• OSPF area 0 is ae20 between the two core-switches

Edge Firewall (SRX3600) Implementation

The edge firewall role was tested and verified featuring the Juniper Networks®SRX3400

Services Gateway. The edge firewall implementation was configured with the following

elements:

• SRX Active/Backup cluster configured

• reth1 configured toward edge routers in untrust zone

• reth0 configured toward core-switch in trust zone

• Security policy is configured for traffic from untrust zone to allow only access to DC

applications

• S-NAT is configured for Internet access to application servers (private address) to

provide Internet access

• D-NAT is configured for remote access to the data center for Pulse gateway internal

IP address to Internet accessible IP address

• Firewall is configured in OSPF area 1

Edge routers (MX240) Implementation

Theedge routing role in theMetaFabric 1.0 solution features the JuniperNetworks®MX240

3DUniversal Edge Router. The edge routing was configuredwith the following elements:

• MX240 pair configured as edge routers connected to service provider network for DC

Internet access

• Both the edge-r1 and edge-r2 EBGP peering with SP1 and SP2

• IBGP is configured between edge-r1 and edge-r2 with next-hop self export policy



• Local-preference is configured on SP1 as a preferred exit point to Internet

• Conditionbased(basedon Internet route)default route injection intoOSPF isconfigured

on both the edge-r1 and edge-r2 toward the firewall/core-switches for Internet access

to VDC devices

• Edge routers (two MX240s): Area1

Compute (IBM Flex chassis) Implementation

The computing role in the solution test labswas built using compute hardware from IBM,

including the IBMFlex Chassis. This role in the solutionwas configuredwith the following

elements:

• o IBM Flex server is configured with multiple ESXi hosts hosting all the VMs running

the business-critical applications (SharePoint, Exchange, Media-wiki, andWWW)

• o Distributed vSwitch is configured betweenmultiple physical ESXi hosts configured

in IBM hosts

OOB-Mgmt (EX4300-VC) Implementation

The entire solution ismanagedout-of-band (OOB) featuring JuniperNetworks®EX4200

Ethernet Switches with Virtual Chassis technology. The OOBmanagement role was

configured and tested with the following elements:

• All the network device OOB connections are plugged into the EX4200-VC (100mand

1 Gbps)

• OOB-MGMT (EX4300-vc): OSPF Area 0

• Two X IBM 3750 standalone servers are connected to the EX4300-VC hosting all the

managementVMs(vCenter, JunosSpace,ND1.5, domaincontroller, andPulsegateway)

• VMware vSphere/Network Director 1.5 used to orchestrate the VMs on the test bed

• Network Director 1.5 is used to configure or orchestrate network configuration and

provisioning

• JuniperNetworksVGWgateway is configuredonaVMtoprovideVM-to-VMapplication

security



Hardware and Software Requirements

This implementation guide employs the hardware and software components shown in

Table 10 on page 66:

Table 10: Hardware and Software deployed in solution testing

FeaturesSoftwareHardware

VLANS, LAG, NNG, RSNG, RVI, OSPF13.1X50-D15QFX3000-MQFabric system

MC-LAG (Active/Active), OSPF, VLANs IRB13.2R3.2EX9208

Clustering, NAT, firewall rules12.1X44-D30.4SRX 3600

MC-LAG ( Active/Active), OSPF, BGP13.2R1.7MX480

DSRM load balancing (direct server return mode)10.2.5 Build 591.0F5 VIPRION 4480

Compute nodes 10-Gigabit and 40-Gigabit CNA and10-Gigabit pass-thru

VMware ESXi 5.1IBM Flex

Standalone serverVMware ESXi 5.1IBMx3750

7.1.56-5EMC-VNX

Application security (VMs)5.5Juniper (Firefly Host)

SA VM appliance for remote access security7.4R1.0SA



In addition, Table 11 on page 67 provides an overview of the network management and

provisioning tools used to validate the solution.

Table 11: Software deployed in MetaFabric 1.0 test bed

FeaturesVersionHardware installedApplication

Virtual-view (VMprovisioning/monitoring)

1.5VMsNetwork Director

Provisioning andmonitoring SRX3600

13.1R1VMsSecurity Director

13.1 R1VMsJunos space

5.1VMsVmware vCenter

Not supportedVMsSecurity Design

13.1 R1VMsService Now

The solution is configured with IP addressing as shown in Table 12 on page 67:

Table 12: Networks and VLANs Deployed in the Test Lab

Vlan-NameVLAN-IDGatewayNetworkNetwork Subnets

Network VLAN80410.94.47.3010.94.47.0/27Network Devices

Security-VLAN80110.94.47.6210.94.47.32/27Security Devices

Storage-VLAN80310.94.47.7810.94.47.64/28Unused

compute-vlan80010.94.47.9410.94.47.80/28Storage Devices

compute-VLAN80010.94.47.12610.94.47.96/27IBMCompute nodeConsoleIP

compute-VLAN80010.94.47.25410.94.47.128/25ESX Compute NodeManagement IP

10.94.63.25410.94.63.0/24VMs

Internet Routable Subnets

10.94.127.3010.94.127.0/27VDC App Server Internet IP(source NAT pool)

10.94.127.6210.94.127.32/27SA IP address



Table 12: Networks and VLANs Deployed in the Test Lab (continued)

Vlan-NameVLAN-IDGatewayNetworkNetwork Subnets

Unrestricted address space fortester ports connected inside theVDC. No security policy for theaddress space.

10.94.127.12610.94.127.64/26Unused

Publicly available applications inVDC.

10.94.127.19010.94.127.128/26Server VIP address

Address space for VMs on theexternal network for simulation ofclient traffic

10.94.127.22210.94.127.192/27LAN client VM and TrafficGenerator address

Address space further subdividedfor point-to-point links inside SP1cloud. No gateway.

10.94.127.224/28SP1 address

Address space further subdividedfor point-to-point links inside SP2cloud. No gateway.

0.94.127.240/28SP2 address

Applications tested as part of the solution were configured with address space shown

in Table 13 on page 68:

Table 13: Applications Tested in theMetaFabric 1.0 Solution

GatewayVlan-IDInternal addressExternal addressApplication

OOB-MGMT81010.94.63.2410.94.127.33SA

POD1-SW1104172.16.4.10 -1210.94.127.181Exchange

POD2-SW1102172.16.2.11 -1410.94.127.180SP

POD1-SW1103172.16.3.1110.94.127.182WM

Multi-chassis LAG is used in the solution between core and aggregation or access to

enable always-up, loop-free, and load-balanced traffic between the switching roles in

the data center (Figure 41 on page 69).



Figure 41: MC-LAG Active/Active Logical Topology



Table 14 on page 70 shows the configuration parameters used in the configuration of

MC-LAG between the VDC-core-sw1 and the edge-r1 nodes. These settings are used

throughout the configuration and are aggregated here.

Table 14: MC-LAG Configuration Parameters

chassis-idprefer-statusIRBinterfaceLACP-idmc-ae-idinterfaceMC-LAG client

MC-LAGNode

1activeirb.000:00:00:00:00:011ae1VDC-edge-fw0VDC-edge-r2


1activeirb.5000:00:00:00:00:011ae0VDC-pod1-sw1VDC-core-sw2






activeirb.1000:00:00:00:00:077ae6VDC-edge-fw0VDC-core-sw2

1activeirb.1000:00:00:00:00:088ae7VDC--edge-fw1VDC-core-sw2

1activeirb.2000:00:00:00:00:099ae8VDC-oob-mgmtVDC-core-sw2

1activeNA00:00:00:00:00:1111ae10VDC-lb1-L2-Int-standbyVDC-core-sw2

1activeirb.1500:00:00:00:00:1212ae11VDC-lb1-L3-Ext-activeVDC-core-sw2

1activeirb.1500:00:00:00:00:1313ae12VDC-lb1-L3-Ext-standbyVDC-core-sw2

1activeNA00:00:00:00:00:1414ae13VDC-lb1-L2-Int-activeVDC-core-sw2

The physical and logical configuration of the core-to-POD roles in the data center are

shown in Figure 42 on page 71. The connectivity between these layers features 24-link

AE bundles (4 per pod with a total of 96 AEmember interfaces between each POD and

the core). The local topology within each data center role are detailed in later sections

of this guide.



Figure 42: Topology of Core-to-POD Roles in the Data Center



The configuration of integrated routing and bridging (IRB) interfaceswithin this segment

of the data center is outlined in Table 15 on page 72.

Table 15: IRB, IP Address Mapping

DescriptionVRRP IPTransportVLANIP address

ClientInterfaceMC-LAG clientIP address

IRBinterface

Untrust-Edge-fw192.168.26.25411192.168.26.3reth1VDC-edge-fw0192.168.26.1irb.0

Untrust-Edge-fw11192.168.26.3reth1VDC-edge-fw1192.168.26.1irb.0

POD1-uplink-1192.168.50.25450192.168.50.3nw-ng-0:ae0VDC-pod1-sw1192.168.50.1irb.50




POD2-uplink-1192.168.54.25454192.168.54.3ae0VDC-pod2-sw1192.168.54.1irb.54

POD2-uplink-2192.168.55.25455192.168.55.3ae1VDC-pod2-sw1192.168.55.1irb.55

Trust-Edge-fw192.168.25.25410192.168.25.3reth0VDC-edge-fw0192.168.25.1irb.10

Trust-Edge-fw192.168.25.25410192.168.25.3reth0VDC--edge-fw1192.168.25.1irb.10

OOB-MGMT-sw192.168.20.25420192.168.20.3ae0VDC-oob-mgmt192.168.20.1irb.20

Layer-2-server-facingcore-swVDC-lb1-L2-Int-standbyNA

layer3-External-link192.168.15.25415192.168.15.5ExternalVDC-lb1-L3-Ext-active192.168.15.1irb.15

Layer3-External-link192.168.15.25415192.168.15.5ExternalVDC-lb1-L3-Ext-standby192.168.15.1irb.15

Layer-2-server-facingcore-swVDC-lb1-L2-Int-activeNA



CHAPTER 4

Transport (Routing and Switching)Configuration

This section covers the configuration of all network elements between the edge and the

data center Point of Delivery (POD) The section includes the following configuration and

verification areas:

• Network configuration

• Configuring edge, perimeter, and core

• Configuring core to PODs

• Routing protocol configuration

• Network Configuration on page 73

• Verification on page 81

• Implementing MC-LAG Active/Active with VRRP on page 85

• Configuring the Network Between the Data Center Core and the Data Center

PODs on page 86


• Routing Configuration on page 104

Network Configuration

Overview

Configuration of the solution starts with the configuration of the perimeter security;

integration between the edge, perimeter and the data center core; and then continues

with configuration of the access and aggregation roles in the data center (in this solution,

those roles are collapsed into theQFabric POD). Finally, the networkmust be configured

in the virtual switching role.

Configuring the Network Between the Data Center Edge and the Data Center Core

This configuration includeselementsofhighavailability as theconfigurationandoperation

of the solution are heavily reliant on the use of Juniper Networks Virtual Chassis and

employmulti-chassis link aggregation (MC-LAG) between each data center operational

role.


SRX chassis clustering provides high availability and redundancy by grouping two SRX

Series services gateways (must be the samemodel) into a cluster. The cluster consists

of a primary node and a secondary node. These nodes provide backup for each other in

the event of software, hardware, or network failures. Session state is synchronized

between thenodes in theSRXcluster to ensure that established sessionsaremaintained

during failover and reversion. The two nodes synchronize configuration, processes, and

services utilizing two Ethernet links: a control link is established to enable control plane

synchronization and a fabric link is established to enable data plane communication

(traversal of network traffic between cluster nodes).

Redundant Ethernet Trunk Group LAGs (RETH interfaces) can be established across

nodes in a chassis cluster (Figure 43 on page 74). Link aggregation allows a redundant

Ethernet interface (knownas aRETH interface in theCLI) to addmultiple child interfaces

frombothnodesof anSRXcluster, creatinga single, virtual interfaceoverwhichupstream

and downstream devices can communicate. This solution features active/standby SRX

cluster configuration: all active links are located on one SRX, and all standby links are on

the other SRX. In an SRX active/backup cluster, LAGmember links from the active node

will forward data traffic. Link Aggregation Control Protocol (LACP) is enabled on the

redundantEthernet interfacesimilar toanyaggregatedEthernet (AE) interfaceconfigured

in other routers/ or switches. The SRXRETH interface configuration includes allmember

interfaces from both the active and backup node.

Figure 43: Configuration of RETH Interfaces andMC-LAG Between Coreand Perimeter (Right) Compared to Configuration of RETH Interfacesand AE (Left)

Topology

The topology used in this section of the configuration is shown in Figure 44 on page 75.



Figure 44: Interface Configuration Between Edge, Perimeter, and Core


Chapter 4: Transport (Routing and Switching) Configuration

Table 16 on page 76 shows the configuration parameters used in the configuration of



Table 16: MC-LAG Settings Between Core 1 and Edge 1

chassis-idprefer-statusIRBInterfaceLACP-idmc-ae-idInterfaceMC-LAG Client

MC-LAGNode









0activeirb.1000:00:00:00:00:077ae6VDC-edge-fw0VDC-core-sw1









Table 17 on page 77shows the configuration parameters used in the configuration of



Table 17: MC-LAG Between Core 1 and Edge 1

chassis-idprefer-statusIRBInterfaceLACP-idmc-ae-idInterfaceMC-LAG Client

MC-LAGNode









1activeirb.1000:00:00:00:00:077ae6VDC-edge-fw0VDC-core-sw2









To configure the network between the data center edge and the data center core, follow

these steps:

1. Configure theSRXreth1 interfaceandmembers towardVDC-edge-r1andVDC-edge-r2.

set chassis cluster reth-count 4

2. Configure the MC-LAG bundle (ae1 and ae3) on VDC-edge-r1 toward the SRX .

set interfaces xe-1/1/0 gigether-options 802.3ad ae1set interfaces xe-1/1/1 gigether-options 802.3ad ae1##Configure ae1 MC-LAG A/A L2 Interface ####set interfaces ae1 description To-Firewall-reth1set interfaces ae1 flexible-vlan-taggingset interfaces ae1 encapsulation flexible-ethernet-servicesset interfaces ae1 aggregated-ether-options lacp activeset interfaces ae1 aggregated-ether-options lacp system-priority 100set interfaces ae1 aggregated-ether-options lacp system-id 00:00:00:00:00:01set interfaces ae1 aggregated-ether-options lacp admin-key 1set interfaces ae1 aggregated-ether-options mc-ae mc-ae-id 1set interfaces ae1 aggregated-ether-options mc-ae redundancy-group 1set interfaces ae1 aggregated-ether-options mc-ae chassis-id 0set interfaces ae1 aggregated-ether-options mc-ae mode active-activeset interfaces ae1 aggregated-ether-options mc-ae status-control activeset interfaces ae1 aggregated-ether-options mc-ae events iccp-peer-downforce-icl-downset interfaces ae1 aggregated-ether-options mc-ae events iccp-peer-downprefer-status-control-activeset interfaces ae1 unit 0 encapsulation vlan-bridgeset interfaces ae1 unit 0 vlan-id 11set interfaces ae1 unit 0 multi-chassis-protection 192.168.168.2 interfaceae0.1###Configured 2Member links for ae3###set interfaces xe-1/2/0 gigether-options 802.3ad ae3set interfaces xe-1/2/1 gigether-options 802.3ad ae3##Configure ae1 MC-LAG A/A L2 Interface ####set interfaces ae3 description To-Firewall-Standbyset interfaces ae3 flexible-vlan-taggingset interfaces ae3 encapsulation flexible-ethernet-servicesset interfaces ae3 aggregated-ether-options lacp activeset interfaces ae3 aggregated-ether-options lacp system-priority 100set interfaces ae3 aggregated-ether-options lacp system-id 00:00:00:00:00:03set interfaces ae3 aggregated-ether-options lacp admin-key 3set interfaces ae3 aggregated-ether-options mc-ae mc-ae-id 3set interfaces ae3 aggregated-ether-options mc-ae redundancy-group 1set interfaces ae3 aggregated-ether-options mc-ae chassis-id 0set interfaces ae3 aggregated-ether-options mc-ae mode active-activeset interfaces ae3 aggregated-ether-options mc-ae status-control activeset interfaces ae3 aggregated-ether-options mc-ae events iccp-peer-downforce-icl-downset interfaces ae3 aggregated-ether-options mc-ae events iccp-peer-downprefer-status-control-activeset interfaces ae3 unit 0 encapsulation vlan-bridgeset interfaces ae3 unit 0 vlan-id 11set interfaces ae3 unit 0 multi-chassis-protection 192.168.168.2 interfaceae0.1

3. Configure the bridge domain and IRB interfaces on VDC-edge-r1.

set bridge-domains bd1 domain-type bridgeset bridge-domains bd1 vlan-id 11set bridge-domains bd1 interface ae1.0 ###MC-LAG interface##set bridge-domains bd1 interface ae0.1 ### MC_LAG ICL link ###



set bridge-domains bd1 interface ae3.0 ###MC-LAG interface##set bridge-domains bd1 routing-interface irb.0 ### L2/L3 routing interface######Configure the IRB interface for L2/L3 routing ##set interfaces irb unit 0 family inet address 192.168.26.1/24 arp 192.168.26.2l2-interface ae0.1set interfaces irb unit 0 family inet address 192.168.26.1/24 arp 192.168.26.2mac 50:c5:8d:87:af:f0set interfaces irb unit 0 family inet address 192.168.26.1/24 arp 192.168.26.2publishset interfaces irb unit 0 family inet address 192.168.26.1/24 vrrp-group 1virtual-address 192.168.26.254set interfaces irb unit 0 family inet address 192.168.26.1/24 vrrp-group 1priority 250set interfaces irb unit 0 family inet address 192.168.26.1/24 vrrp-group 1fast-interval 100set interfaces irb unit 0 family inet address 192.168.26.1/24 vrrp-group 1preemptset interfaces irb unit 0 family inet address 192.168.26.1/24 vrrp-group 1accept-dataset interfaces irb unit 0 family inet address 192.168.26.1/24 vrrp-group 1authentication-type md5set interfaces irb unit 0 family inet address 192.168.26.1/24 vrrp-group 1authentication-key "$9$WKVXVYaJDkqfoaFnCA0O"

4. Configure ICCP and ICL links for MC-LAG on vdc-edge-r1.

set interfaces ae0 flexible-vlan-taggingset interfaces ae0 encapsulation flexible-ethernet-servicesset interfaces ae0 aggregated-ether-options lacp activeset interfaces ae0 aggregated-ether-options lacp periodic slowset interfaces xe-1/0/0 hold-time up 100set interfaces xe-1/0/0 hold-time down 15000set interfaces xe-1/0/0 gigether-options 802.3ad ae0set interfaces xe-1/0/1 hold-time up 100set interfaces xe-1/0/1 hold-time down 15000set interfaces xe-1/0/1 gigether-options 802.3ad ae0

NOTE: Hold-down timer configured higher than the BFD timer(1 sec) forbetter convergence

set interfaces ae0 unit 0 description "ICCP Link between edge-r1 and edge-r2"set interfaces ae0 unit 0 vlan-id 4000set interfaces ae0 unit 0 vlan-id 4000set interfaces ae0 unit 0 family inet address 192.168.1.1/30set interfaces ae0 unit 1 description "ICL Link to edge-r2-vlan-11"set interfaces ae0 unit 1 encapsulation vlan-bridgeset interfaces ae0 unit 1 vlan-id 11

5. Configure the Inter-Control Center Communications Protocol (ICCP) on vdc-edge-r1.

set protocols iccp local-ip-addr 192.168.168.1set protocols iccp peer 192.168.168.2 redundancy-group-id-list 1set protocols iccp peer 192.168.168.2 liveness-detection minimum-interval 500set protocols iccp peer 192.168.168.2 liveness-detection multiplier 2set protocols iccp peer 192.168.168.2 liveness-detection detection-timethreshold 2000

NOTE: The BFD timer is configured as 1 second in this solution testing.This setting provided sub-2-second convergence in testing.



6. Configure the MC-LAG bundle (ae1 and ae3) on VDC-edge-r2 toward the SRX.

set interfaces xe-1/1/0 gigether-options 802.3ad ae1set interfaces xe-1/1/1 gigether-options 802.3ad ae1set interfaces ae1 description To-Firewall-reth1set interfaces ae1 flexible-vlan-taggingset interfaces ae1 encapsulation flexible-ethernet-servicesset interfaces ae1 aggregated-ether-options lacp activeset interfaces ae1 aggregated-ether-options lacp system-priority 100set interfaces ae1 aggregated-ether-options lacp system-id 00:00:00:00:00:01set interfaces ae1 aggregated-ether-options lacp admin-key 1set interfaces ae1 aggregated-ether-options mc-ae mc-ae-id 1set interfaces ae1 aggregated-ether-options mc-ae redundancy-group 1set interfaces ae1 aggregated-ether-options mc-ae chassis-id 1set interfaces ae1 aggregated-ether-options mc-ae mode active-activeset interfaces ae1 aggregated-ether-options mc-ae status-control standbyset interfaces ae1 aggregated-ether-options mc-ae events iccp-peer-downforce-icl-downset interfaces ae1 unit 0 encapsulation vlan-bridgeset interfaces ae1 unit 0 vlan-id 11set interfaces ae1 unit 0 multi-chassis-protection 192.168.168.1 interfaceae0.1set interfaces xe-1/2/0 gigether-options 802.3ad ae3set interfaces xe-1/2/1 gigether-options 802.3ad ae3set interfaces ae3 description To-Firewall-reth1set interfaces ae3 flexible-vlan-taggingset interfaces ae3 encapsulation flexible-ethernet-servicesset interfaces ae3 aggregated-ether-options lacp activeset interfaces ae3 aggregated-ether-options lacp system-priority 100set interfaces ae3 aggregated-ether-options lacp system-id 00:00:00:00:00:03set interfaces ae3 aggregated-ether-options lacp admin-key 3set interfaces ae3 aggregated-ether-options mc-ae mc-ae-id 3set interfaces ae3 aggregated-ether-options mc-ae redundancy-group 1set interfaces ae3 aggregated-ether-options mc-ae chassis-id 1set interfaces ae3 aggregated-ether-options mc-ae mode active-activeset interfaces ae3 aggregated-ether-options mc-ae status-control standbyset interfaces ae3 aggregated-ether-options mc-ae events iccp-peer-downforce-icl-downset interfaces ae3 unit 0 encapsulation vlan-bridgeset interfaces ae3 unit 0 vlan-id 11set interfaces ae3 unit 0 multi-chassis-protection 192.168.168.1 interfaceae0.1

7. Configuring the bridge domain and IRB interface on VDC-edge-r2.

set bridge-domains bd1 domain-type bridgeset bridge-domains bd1 vlan-id 11set bridge-domains bd1 interface ae1.0set bridge-domains bd1 interface ae0.1set bridge-domains bd1 interface ae3.0set bridge-domains bd1 routing-interface irb.0set interfaces irb unit 0 family inet address 192.168.26.2/24 arp 192.168.26.1l2-interface ae0.1set interfaces irb unit 0 family inet address 192.168.26.2/24 arp 192.168.26.1mac 50:c5:8d:87:87:f0set interfaces irb unit 0 family inet address 192.168.26.2/24 arp 192.168.26.1publishset interfaces irb unit 0 family inet address 192.168.26.2/24 vrrp-group 1virtual-address 192.168.26.254set interfaces irb unit 0 family inet address 192.168.26.2/24 vrrp-group 1priority 125set interfaces irb unit 0 family inet address 192.168.26.2/24 vrrp-group 1fast-interval 100



set interfaces irb unit 0 family inet address 192.168.26.2/24 vrrp-group 1preemptset interfaces irb unit 0 family inet address 192.168.26.2/24 vrrp-group 1accept-dataset interfaces irb unit 0 family inet address 192.168.26.2/24 vrrp-group 1authentication-type md5set interfaces irb unit 0 family inet address 192.168.26.2/24 vrrp-group 1authentication-key "$9$WKVXVYaJDkqfoaFnCA0O"

8. Configure the ICCP and ICL link for the MC-LAG on vdc-edge-r2.

###Configure LACP parameters for ICL/ICCP link ##set interfaces ae0 flexible-vlan-taggingset interfaces ae0 encapsulation flexible-ethernet-servicesset interfaces ae0 aggregated-ether-options lacp activeset interfaces ae0 aggregated-ether-options lacp periodic slow## LAGmember link configuration ###set interfaces xe-1/0/0 hold-time up 100set interfaces xe-1/0/0 hold-time down 15000set interfaces xe-1/0/0 gigether-options 802.3ad ae0set interfaces xe-1/0/1 hold-time up 100set interfaces xe-1/0/1 hold-time down 15000set interfaces xe-1/0/1 gigether-options 802.3ad ae0

NOTE: Hold-down timer configured higher than the BFD timer (1 sec) toget improved convergence if prefer-status-control active is configured onboth MC-LAG nodes.

### ICCP Logical link ###set interfaces ae0 unit 0 description "ICCP link between edge-r2 to edge-r1"set interfaces ae0 unit 0 vlan-id 4000set interfaces ae0 unit 0 family inet address 192.168.1.2/30### ICL Logical Link ###set interfaces ae0 unit 1 description "ICL Link to edge-r2-vlan-11"set interfaces ae0 unit 1 encapsulation vlan-bridgeset interfaces ae0 unit 1 vlan-id 11

9. Configure the ICCP protocol for MC-LAG on vdc-edge-r2.

set protocols iccp local-ip-addr 192.168.168.2set protocols iccp peer 192.168.168.1 redundancy-group-id-list 1set protocols iccp peer 192.168.168.1 liveness-detection minimum-interval 500set protocols iccp peer 192.168.168.1 liveness-detection multiplier 2set protocols iccp peer 192.168.168.1 liveness-detection detection-timethreshold 2000

Verification

The following verification commands (with sample output) can be used to confirm that

the transport, clustering, and MC-LAG configuration were successful.

Verification

Purpose The following verification commands (with sample output) can be used to confirm that

the transport, clustering, and MC-LAG configuration were successful.

Results

Verify that MC-LAG is up on the edge routers.



1. This output shows an active state for the MC-LAG connections on the edge routers

and confirms that the twoMC-LAG bundles are operational. In a failure state, the

typical error showing a broken configuration will be “Exchange error” on this output,

which implies there is misconfiguration on the ICCP/MC-AE configuration.

root@VDC-edge-r01-re0>show interfacesmc-aeMember Link : ae1 Current State Machine's State: mcae active state Local Status : active Local State : up Peer Status : active Peer State : up Logical Interface : ae1.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.2 ae0.1 up

Member Link : ae3 Current State Machine's State: mcae active state Local Status : active Local State : up Peer Status : active Peer State : up Logical Interface : ae3.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.2 ae0.1 up{master}

root@VDC-edge-r01-re0>

2. Verify the reth0 interface on the edge-firewall.

root@VDC-edge-fw01-n1>show interfaces reth0

Physical interface: reth0, Enabled, Physical link is Up Interface index: 128, SNMP ifIndex: 628 Description: Trust Zone toward POD Link-level type: Ethernet, MTU: 9188, Speed: 40Gbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 0 Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x0 Current address: 00:10:db:ff:10:00, Hardware address: 00:10:db:ff:10:00 Last flapped : 2013-10-17 17:05:37 PDT (5d 17:34 ago) Input rate : 20352 bps (28 pps) Output rate : 4480 bps (8 pps)

Logical interface reth0.0 (Index 68) (SNMP ifIndex 647)

Flags: SNMP-Traps 0x0 VLAN-Tag [ 0x8100.10 ] Encapsulation: ENET2 Statistics Packets pps Bytes bps Bundle: Input : 111252547290 28 13795224589923 20352 Output: 112746013568 8 14431770429808 4480 Security: Zone: trust Allowed host-inbound traffic : bootp bfd bgp dns dvmrp igmp ldp msdp nhrp ospf pgm pim rip router-discovery rsvp sap vrrp dhcp finger ftp tftp



ident-reset http https ike netconf ping reverse-telnet reverse-ssh rlogin rpm rsh snmp snmp-trap ssh telnet traceroute xnm-clear-text xnm-ssl lsping ntp sip r2cp Protocol inet, MTU: 9170 Flags: Sendbcast-pkt-to-re Addresses, Flags: Is-Preferred Is-Primary Destination: 192.168.25/24, Local: 192.168.25.3, Broadcast: 192.168.25.255 Protocol multiservice, MTU: Unlimited Flags: Is-Primary

Logical interface reth0.32767 (Index 67) (SNMP ifIndex 662)

Flags: SNMP-Traps 0x0 VLAN-Tag [ 0x0000.0 ] Encapsulation: ENET2Statistics Packets pps Bytes bpsBundle: Input : 0 0 0 0 Output: 0 0 0 0Security: Zone: NullProtocol multiservice, MTU: Unlimited Flags: None

3. Verify that the edge router is selecting the active firewall node for traffic forwarding.

This selection is done based on the gratuitous ARP request sent by the active SRX

firewall.

a. Check the route for the firewall reth0 IP address.

root@VDC-edge-r01-re0>show route 192.168.26.3

inet.0: 66 destinations, 77 routes (66 active, 0 holddown, 0 hidden)+ = Active Route, - = Last Active, * = Both

192.168.26.0/24 *[Direct/0] 1w6d 23:12:12 > via irb.0

b. Check the forwarding table to see if the next hopand interfaceare chosen correctly

Active firewall node (Node 2 is active)root@VDC-edge-r01-re0> show route forwarding-table destination 192.168.26.3

Routing table: default.inetInternet:Destination Type RtRef Next hop Type Index NhRef Netif192.168.26.3/32 dest 0 0:10:db:ff:10:1 ucst 597 39 ae3.0Routing table: __master.anon__.inetInternet:Destination Type RtRef Next hop Type Index NhRef Netifdefault perm 0 rjct 519 1



NOTE: Whena failover occurs, the secondary nodemust announce to thepeer device that it is now owner of the MAC address associated with theRETH interface (the RETHMAC is shared between nodes). It does thisusing gratuitous ARP, or an ARPmessage that is broadcast without anARP request. Once a gratuitous ARP is sent, the local switch udpates itsMAC table tomap the newMAC/port pairing. By default, the SRX sendsfour gratuitous ARPs per RETH on a failover. These are sent from thecontrol plane and through the data plane.

4. Verify both LAGs on the edge router (ae1 and ae3). –Note that even though both the

LACPLAGs appear in an “up” state, only the LAG link toward the active cluster firewall

node will forward traffic; the standby node will remain up and ready to take over in

case of failure.

root@VDC-edge-r01-re0>show lacp interfaces ae1

Aggregated interface: ae1 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity xe-1/1/0 Actor No No Yes Yes Yes Yes Fast Active xe-1/1/0 Partner No No Yes Yes Yes Yes Fast Active xe-1/1/1 Actor No No Yes Yes Yes Yes Fast Active xe-1/1/1 Partner No No Yes Yes Yes Yes Fast Active LACP protocol: Receive State Transmit State Mux State xe-1/1/0 Current Fast periodic Collecting distributing xe-1/1/1 Current Fast periodic Collecting distributing

{master}

root@VDC-edge-r01-re0>show lacp interfaces ae3

Aggregated interface: ae3 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity xe-1/2/0 Actor No No Yes Yes Yes Yes Fast Active xe-1/2/0 Partner No No Yes Yes Yes Yes Fast Active xe-1/2/1 Actor No No Yes Yes Yes Yes Fast Active xe-1/2/1 Partner No No Yes Yes Yes Yes Fast Active LACP protocol: Receive State Transmit State Mux State xe-1/2/0 Current Fast periodic Collecting distributing xe-1/2/1 Current Fast periodic Collecting distributing



ImplementingMC-LAG Active/Active with VRRP

To allow all the links to forward traffic without being blocked by spanning-tree,

multi-chassis link aggregation (MC-LAG) is configured on the edge routers and core

switches. The edge routers useMC-LAG toward the edge firewalls, and the core switches

use MC-LAG toward each POD switch, application load balancer (F5), and OOB

management switch. MC- is a feature that supports aggregated Ethernet (AE) LAG

bundles spread across more than one device. LACP is used for dynamic configuration

andmonitoring on links.

Summary of Implementation Details for MC-LAG Active/Active

MC-LAG is a key component of the MetaFabric 1.0 solution architecture. MC-LAG is

configured using the following design considerations:

• Do not mix Layer 2 next generation CLI syntax (L2NG, or family ethernet-switching)

and non-l2ng (flexible-ethernet-serivces) syntax on the same interface.

• Mac-learning is disabled on inter-chassis link (ICL).

• Arp learning is disabled on ICL.

• Static arp is required for integrated routing and bridging (IRB)-to-IRB connectivity

across the ICL.

This is configured to support OSPF over the Inter-Control Center Communications

Protocol (ICCP).

• Load balancing betweenmc-lag peers is 100 percent local bias by default.

• Load balancing within local peer is the same as normal lag hashing.

• Two possible options for Layer 3 gateway is Virtual Router Redundancy Protocol

(VRRP) based or irb-mac-sync.

• If irb-mac-sync is used, routing protocols on IRB are not supported.

• In a VRRP-based Layer 3 solution, even the VRRP backup node forwards traffic.

• Prefer separate link aggregation group (LAG) links for ICL and ICCP.

• ICCP peering with loopback IP is preferred to use all available links through interior

gateway protocol (IGP).

• Configure backup-liveness detection to get sub-second traffic loss during MC-LAG

peer switch reboots.

• Spanning Tree Protocol (STP) is not supported on ICL or MC-LAG interfaces.

• Access security features are not supported on ICL or MC-LAG interfaces.

• Configuremcaewith “prefer status control active” on both provider edge routers (PEs)

to avoid lacp system ID flap during active node (SW) reboot.



MC-LAG Configuration for Better Convergence

To improve the network convergence in this solution, the following configuration

statements are configured:

• prefer-status-control active on both the MC-LAG nodes for all MC-AE interfaces. With

this configuration, LACPsystem-idwill be retainedonboth ICCP/ICL failures to improve

convergence.

• Loopback IP peering is configured for the ICCPprotocol. The ICCPpeer can be reached

over protocols in case of direct ICCP link failure.

• The 1-secondBFDtimer is configured for the ICCPprotocol andall IRB/VRRP interfaces.

• More than 1-second hold-down timer is configured on the ICL links to prevent ICL link

start-up before the ICCP during failure events.

• init-delay-time is configured on EX9214 core switches to delay the start-up of the

MC-AE interfaces before the protocol convergence. This knob removes any packet

loss during the recovery of failed links and devices.

Configuring the Network Between the Data Center Core and the Data Center PODs

The steps required to configure network between the data center core and data center

PODs are shown in the following section:

• Configure MC-LAG on the VDC POD to the Core

• Configure MC-LAG at Core Switch 1

• Configure MC-LAG at Core Switch 2

• Verify the configuration

To configure the network between the data center core and the data center PODs, follow

these steps:

1. ConfigureMC-LAGae0betweenVDC-pod1-sw1 toVDC-core-sw1andVDC-core-sw2.

a. Enable the Ethernet bundle on the chassis.

[edit]set chassis node-group NW-NG-0 aggregated-devices ethernet device-count10Configure the DSCP BA classifier for IPv6.

b. Configure the AE bundle toward Core-sw1 and Core-sw2 as a single AE bundle in

VDC-pod1-sw1.

[edit]set interfaces NW-NG-0:ae0 description POD1-to-core-MC-LAG-UPLINKset interfaces NW-NG-0:ae0 aggregated-ether-optionsminimum-links 1set interfaces NW-NG-0:ae0 aggregated-ether-options link-speed 10gset interfaces NW-NG-0:ae0 aggregated-ether-options lacp activeset interfaces NW-NG-0:ae0 unit 0 family ethernet-switching port-mode trunk

c. Enable all the application VLANs on the POD switches toward the core-switches.



[edit]set interfacesNW-NG-0:ae0unit 0 family ethernet-switching vlanmembersMGMTset interfaces NW-NG-0:ae0 unit 0 family ethernet-switching vlanmembers Infraset interfacesNW-NG-0:ae0unit0familyethernet-switchingvlanmembersTera-VMset interfaces NW-NG-0:ae0 unit 0 family ethernet-switching vlanmembersSecurity-Mgmt

set interfacesNW-NG-0:ae0unit0familyethernet-switchingvlanmembersVmotionset interfacesNW-NG-0:ae0unit 0 family ethernet-switching vlanmembersVM-FTset interfaces NW-NG-0:ae0 unit 0 family ethernet-switching vlanmembersRemote-Access

set interfaces NW-NG-0:ae0 unit 0 family ethernet-switching vlanmembersCore-transport-1

d. Configure the member links connected to Core-sw1 and Core-sw2 under the AE

bundle.

NOTE: n0:xe-0/0/[0-11] is connected to VDC-core-sw1.

n1:xe-0/0/[0-11] is connected to VDC-core-sw2.

[edit]set interfaces interface-rangeMC-LAG-ae0-membersmember "n0:xe-0/0/[0-11]"set interfaces interface-range MC-LAG-ae0-members member "n1:xe-0/0/[0-11]"set interfaces interface-range MC-LAG-ae0-members description "MC-LAG toCore-sw ae0"

set interfaces interface-range MC-LAG-ae0-members ether-options 802.3adNW-NG-0:ae0

e. Configure MC-LAG ae0 between VDC-pod1-sw1 to VDC-core-sw1 and

VDC-core-sw2 and enable Ethernet bundle on the chassis:

[edit]set chassis node-group NW-NG-0 aggregated-devices ethernet device-count10Configure the DSCP BA classifier for IPv6.

f. Configure the AE bundle toward Core-sw1 and Core-sw2 as a single AE bundle in

VDC-pod1-sw1.


g. Enable all the applications VLAN on the POD switches toward the core-switches.

[edit]set interfacesNW-NG-0:ae0unit 0 family ethernet-switching vlanmembersMGMTset interfaces NW-NG-0:ae0 unit 0 family ethernet-switching vlanmembers Infraset interfacesNW-NG-0:ae0unit0familyethernet-switchingvlanmembersTera-VMset interfaces NW-NG-0:ae0 unit 0 family ethernet-switching vlanmembersSecurity-Mgmt

set interfacesNW-NG-0:ae0unit0familyethernet-switchingvlanmembersVmotionset interfacesNW-NG-0:ae0unit 0 family ethernet-switching vlanmembersVM-FT



set interfaces NW-NG-0:ae0 unit 0 family ethernet-switching vlanmembersRemote-Access

set interfaces NW-NG-0:ae0 unit 0 family ethernet-switching vlanmembersCore-transport-1

h. Configure the member links connected to Core-sw1 and Core-sw2 under the AE

bundle.

i. [edit]set interfaces interface-rangeMC-LAG-ae0-membersmember "n0:xe-0/0/[0-11]"set interfaces interface-range MC-LAG-ae0-members member "n1:xe-0/0/[0-11]"set interfaces interface-range MC-LAG-ae0-members description "MC-LAG toCore-sw ae0"

set interfaces interface-range MC-LAG-ae0-members ether-options 802.3adNW-NG-0:ae0

NOTE: n0:xe-0/0/[0-11] is connected to VDC-core-sw1

n1:xe-0/0/[0-11] is connected to VDC-core-sw2

2. Configure MC-LAG at VDC-core-sw1 (EX9214-1).

a. Specify the number of aggregated Ethernet interfaces to be created.

[edit]set chassis aggregated-devices ethernet device-count 30

b. Specify the members to be included within the aggregated Ethernet bundle ae0.


c. ConfigureLACPparameterswith static system-idandadmin-keyon theaggregated

Ethernet bundle.

[edit]set interfaces ae0 description "MC-LAG to VDC-pod1-sw1-nng-ae0"set interfaces ae0 aggregated-ether-options lacp activeset interfaces ae0 aggregated-ether-options lacp system-priority 100set interfaces ae0 aggregated-ether-options lacp system-id 00:00:00:00:00:01set interfaces ae0 aggregated-ether-options lacp admin-key 1

d. Configure MC-AE interface parameters.

[edit]set interfaces ae0 aggregated-ether-optionsmc-aemc-ae-id 1set interfaces ae0 aggregated-ether-optionsmc-ae redundancy-group 1set interfaces ae0 aggregated-ether-optionsmc-ae chassis-id 0set interfaces ae0 aggregated-ether-optionsmc-aemode active-activeset interfaces ae0 aggregated-ether-optionsmc-ae status-control activeset interfaces ae0 aggregated-ether-optionsmc-ae init-delay-time 520set interfaces ae0 aggregated-ether-optionsmc-ae events iccp-peer-downforce-icl-down



set interfaces ae0 aggregated-ether-optionsmc-ae events iccp-peer-downprefer-status-control-active

set interfaces ae0 unit 0multi-chassis-protection 192.168.168.5 interface ae9.0

NOTE: Please review the following caveats and guidelines:

• Themulti-chassis aggregated Ethernet identification number(mc-ae-id) specifies the link aggregation group to which theaggregated Ethernet interface belongs. The ae0 interfaces onVDC-core-sw1 and VDC-core-sw2 are configured withmc-ae-id 1.The ae1 interfaces on VDC-core-sw1 and VDC-core-sw2 areconfigured withmc-ae-id 2.

• The redundancy-group 1 statement is used by ICCP to associatemultiple chassis that perform similar redundancy functions and toestablish a communication channel so that applications on peeringchassis cansendmessages toeachother.Theae0andae1 interfaceson VDC-core-sw1 and VDC-core-sw2 are configured with the sameredundancy group redundancy-group 1.

• The chassis-id statement is used by LACP for calculating the portnumber of theMC-LAG's physicalmember links. VDC-core-sw1 useschassid-id 0 to identify both its ae0 and ae1 interfaces.VDC-core-sw2uses chassis-id 1 to identify both its ae0 and ae1 andother interfaces.

• Themode statement indicates whether an MC-LAG is inactive-standbymode or active-activemode. Chassis that are in thesame groupmust be in the samemode. Here we have configuredactive-active.

• Status-control must be active on one PE and standby on the othernode.

• mc-ae events iccp-peer-down prefer-status-control-active is needed

to pre-configure the active node during ICCP failover scenarios. Bydefault one device will be active and other node will be standby

• Toget better convergenceonnode reboot scenarios, both nodes canbe configured as prefer-status-control-active. We have tomake sure

that the ICCP session goes down due to physical link failures byconfiguring ICCP peering with the loopback address.

• mc-ae init-delay-time is configured to be a higher value,which delays

the start-up of the MC-AE links during device recovery. This allowsthe protocol convergence to complete and be ready to forwardpackets coming fromMC-AE to the core-switch.

e. Configure allowed VLANS on this MC-LAG link.

[edit]set interfaces ae0 unit 0 family ethernet-switching interface-mode trunkset interfaces ae0 unit 0 family ethernet-switching vlanmembers MGMT



set interfaces ae0 unit 0 family ethernet-switching vlanmembers Infraset interfaces ae0 unit 0 family ethernet-switching vlanmembers Tera-VMset interfaces ae0 unit 0 family ethernet-switching vlanmembers Security-Mgmtset interfaces ae0 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces ae0 unit 0 family ethernet-switching vlanmembers VM-FTset interfaces ae0 unit 0 family ethernet-switching vlanmembers Remote-Accessset interfacesae0unit0 family ethernet-switching vlanmembersPOD1-Transport-1

f. Configure ae bundle (ae20) connected between the core switches as a Layer 3

link to enable ICCP.

[edit]set interfaces ae20 vlan-taggingset interfaces ae20 aggregated-ether-options lacp activeset interfaces ae20 unit 0 description " ICCP link between core-sw1 and core-sw2"set interfaces ae20 unit 0 vlan-id 4000set interfaces ae20 unit 0 family inet address 192.168.2.1/30

g. Configure another ae bundle (ae9) connected between the core switches as a

Layer 2 link . Thiswill functionas themulti-chassis protection link (ICL-PL)between

the core switches.

[edit]set interfaces ae9 aggregated-ether-options lacp activeset interfaces ae9 aggregated-ether-options lacp periodic fastset interfaces ae9 unit 0 description "ICL Link for all VLANS"set interfaces ae9 unit 0 family ethernet-switching interface-mode trunk

h. Configure the ICL link members with a hold-time value higher than the configured

BFD timer(1s) to have zero loss convergence during recovery of failed devices.

[edit]set interfaces xe-0/3/6 hold-time up 100set interfaces xe-0/3/6 hold-time down 3000set interfaces xe-0/3/7 hold-time up 100set interfaces xe-0/3/7 hold-time down 3000set interfaces xe-1/3/6 hold-time up 100set interfaces xe-1/3/6 hold-time down 3000set interfaces xe-1/3/7 hold-time up 100set interfaces xe-1/3/7 hold-time down 3000set interfaces xe-3/3/6 hold-time up 100set interfaces xe-3/3/6 hold-time down 3000set interfaces xe-3/3/7 hold-time up 100set interfaces xe-3/3/7 hold-time down 3000set interfaces xe-5/3/6 hold-time up 100set interfaces xe-5/3/6 hold-time down 3000set interfaces xe-5/3/7 hold-time up 100set interfaces xe-5/3/7 hold-time down 3000

i. Configure allowed VLANs on this bundle.

[edit]set interfaces ae9 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae9 unit 0 family ethernet-switching vlanmembers Infraset interfaces ae9 unit 0 family ethernet-switching vlanmembers SQLset interfaces ae9 unit 0 family ethernet-switching vlanmembers SharePointset interfaces ae9 unit 0 family ethernet-switching vlanmembers Firewallset interfaces ae9 unit 0 family ethernet-switching vlanmembers MGMT



set interfaces ae9 unit 0 family ethernet-switching vlanmembersWikimediaset interfacesae9unit 0 family ethernet-switching vlanmembersExchange-Clusterset interfaces ae9 unit 0 family ethernet-switching vlanmembers Tera-VMset interfacesae9unit0 familyethernet-switchingvlanmembersLoad-balancer-Extset interfaces ae9 unit 0 family ethernet-switching vlanmembers Security-Mgmtset interfaces ae9 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces ae9 unit 0 family ethernet-switching vlanmembers VM-FTset interfaces ae9 unit 0 family ethernet-switching vlanmembers Remote-Accessset interfaces ae9 unit 0 family ethernet-switching vlanmembers OOB-Transportset interfaces ae9 unit 0 family ethernet-switching vlanmembersLoad-balancer-Ext-Tera-VM

set interfaces ae9 unit 0 family ethernet-switching vlanmembersTrafficGenerator-502



set interfacesae9unit 0 family ethernet-switching vlanmembersPOD1-Transport-1set interfacesae9unit0 familyethernet-switchingvlanmembersPOD1-Transport-2set interfacesae9unit0 familyethernet-switchingvlanmembersPOD1-Transport-3set interfacesae9unit0 familyethernet-switchingvlanmembersPOD1-Transport-4set interfacesae9unit0 familyethernet-switchingvlanmembersPOD2-Transport-1set interfacesae9unit0 familyethernet-switchingvlanmembersPOD2-Transport-2

j. Configure IRB on both the core-sw1 and core-sw2 and enable VRRP on the IRBs.

[edit]set interfaces irb unit 50 family inet address 192.168.50.1/24 arp 192.168.50.2l2-interface ae9.0

set interfaces irb unit 50 family inet address 192.168.50.1/24 arp 192.168.50.2 mac4c:96:14:68:83:f0

set interfaces irbunit50family inetaddress 192.168.50.1/24arp 192.168.50.2publishset interfaces irb unit 50 family inet address 192.168.50.1/24 vrrp-group 1virtual-address 192.168.50.254

set interfaces irb unit 50 family inet address 192.168.50.1/24 vrrp-group 1 priority 125set interfaces irbunit 50 family inetaddress 192.168.50.1/24vrrp-group 1 fast-interval100

set interfaces irb unit 50 family inet address 192.168.50.1/24 vrrp-group 1 preemptset interfaces irbunit50 family inetaddress 192.168.50.1/24vrrp-group 1accept-dataset interfaces irb unit 50 family inet address 192.168.50.1/24 vrrp-group 1authentication-typemd5

set interfaces irb unit 50 family inet address 192.168.50.1/24 vrrp-group 1authentication-key "$9$Asx6uRSKvLN-weK4aUDkq"

k. Configure VLAN and enable “domain-type bridge” to configure IRB as the

routing-interface under this bridge domain.

[edit]set vlans POD1-Transport-1 vlan-id 50set vlans POD1-Transport-1 l3-interface irb.50set vlans POD1-Transport-1 domain-type bridge



NOTE: The enhanced Layer 2 software (ELS) CLI doesn’t need anMC-AE interface, ICL under the bridge configuration. It picks theappropriate MC-AE interface from the VLAN association on theconfigured links.

l. Configure ICCP protocol parameters.

[edit]set protocols iccp local-ip-addr 192.168.168.4set protocols iccp peer 192.168.168.5 redundancy-group-id-list 1set protocols iccp peer 192.168.168.5 backup-liveness-detection backup-peer-ip192.168.168.5

set protocols iccp peer 192.168.168.5 liveness-detectionminimum-interval 500set protocols iccp peer 192.168.168.5 liveness-detectionmultiplier 2

m. Configure the switch service-id for MC-LAG.

[edit]set switch-options service-id 1

NOTE: Youmustconfigurethesameuniquenetwork-wideconfigurationfor a service in the set of PE routers providing the service. This serviceID is required if themulti-chassis aggregated Ethernet interfaces arepart of a bridge domain.

3. Configure MC-LAG at VDC-core-sw2 (EX9214-2).

a. Specify the number of aggregated Ethernet interfaces to be created.

[edit]set chassis aggregated-devices ethernet device-count 30

b. Specify the members to be included within the aggregated Ethernet bundle ae0.

[edit]set interfaces interface-rangePOD1-MC-LAG-ae0-membersmember"xe-0/0/[0-7]"set interfaces interface-rangePOD1-MC-LAG-ae0-membersmember"xe-0/1/[0-3]"set interfaces interface-range POD1-MC-LAG-ae0-members description "MC-LAGto POD1 ae0"

set interfaces interface-rangePOD1-MC-LAG-ae0-membersether-options802.3adae0

c. ConfigureLACPparameterswith static system-idandadmin-keyon theaggregated

Ethernet bundle.

[editset interfaces ae0 description "MC-LAG to VDC-pod1-sw1-nng-ae0"set interfaces ae0 aggregated-ether-options lacp activeset interfaces ae0 aggregated-ether-options lacp system-priority 100set interfaces ae0 aggregated-ether-options lacp system-id 00:00:00:00:00:01set interfaces ae0 aggregated-ether-options lacp admin-key 1

d. Configure MC-AE interface parameters.

[edit]



set interfaces ae0 aggregated-ether-optionsmc-aemc-ae-id 1set interfaces ae0 aggregated-ether-optionsmc-ae redundancy-group 1set interfaces ae0 aggregated-ether-optionsmc-ae chassis-id 1set interfaces ae0 aggregated-ether-optionsmc-aemode active-activeset interfaces ae0 aggregated-ether-optionsmc-ae status-control standbyset interfaces ae0 aggregated-ether-optionsmc-ae init-delay-time 520set interfaces ae0 aggregated-ether-optionsmc-ae events iccp-peer-downforce-icl-down

set interfaces ae0 aggregated-ether-optionsmc-ae events iccp-peer-downprefer-status-control-active

set interfaces ae0 unit 0multi-chassis-protection 192.168.168.4 interface ae9.0

e. Configure allowed VLANS on this MC-LAG link.

[edit]set interfaces ae0 unit 0 family ethernet-switching interface-mode trunkset interfaces ae0 unit 0 family ethernet-switching vlanmembers MGMTset interfaces ae0 unit 0 family ethernet-switching vlanmembers Infraset interfaces ae0 unit 0 family ethernet-switching vlanmembers Tera-VMset interfaces ae0 unit 0 family ethernet-switching vlanmembers Security-Mgmtset interfaces ae0 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces ae0 unit 0 family ethernet-switching vlanmembers VM-FTset interfaces ae0 unit 0 family ethernet-switching vlanmembers Remote-Accessset interfacesae0unit0 family ethernet-switching vlanmembersPOD1-Transport-1

f. Configure ae bundle (ae20) connected between the core switches as a Layer 3

link to enable ICCP.


g. Configure ae bundle (ae9) connected between the core switches as a Layer 2 link.

This will function as the multi-chassis protection link between the core switches.

[edit]set interfaces ae9 aggregated-ether-options lacp activeset interfaces ae9 aggregated-ether-options lacp periodic fastset interfaces ae9 unit 0 description "ICL Link for all VLANS"set interfaces ae9 unit 0 family ethernet-switching interface-mode trunkset interfaces ae9 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae9 unit 0 family ethernet-switching vlanmembers Infraset interfaces ae9 unit 0 family ethernet-switching vlanmembers SQLset interfaces ae9 unit 0 family ethernet-switching vlanmembers SharePointset interfaces ae9 unit 0 family ethernet-switching vlanmembers Firewallset interfaces ae9 unit 0 family ethernet-switching vlanmembers MGMTset interfaces ae9 unit 0 family ethernet-switching vlanmembersWikimediaset interfacesae9unit 0 family ethernet-switching vlanmembersExchange-Clusterset interfaces ae9 unit 0 family ethernet-switching vlanmembers Tera-VMset interfacesae9unit0 familyethernet-switchingvlanmembersLoad-balancer-Extset interfaces ae9 unit 0 family ethernet-switching vlanmembers Security-Mgmtset interfaces ae9 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces ae9 unit 0 family ethernet-switching vlanmembers VM-FTset interfaces ae9 unit 0 family ethernet-switching vlanmembers Remote-Access



set interfaces ae9 unit 0 family ethernet-switching vlanmembers OOB-Transportset interfaces ae9 unit 0 family ethernet-switching vlanmembersLoad-balancer-Ext-Tera-VM




set interfacesae9unit 0 family ethernet-switching vlanmembersPOD1-Transport-1set interfacesae9unit0 familyethernet-switchingvlanmembersPOD1-Transport-2set interfacesae9unit0 familyethernet-switchingvlanmembersPOD1-Transport-3set interfacesae9unit0 familyethernet-switchingvlanmembersPOD1-Transport-4set interfacesae9unit0 familyethernet-switchingvlanmembersPOD2-Transport-1set interfacesae9unit0 familyethernet-switchingvlanmembersPOD2-Transport-2

h. Configure the ICL link members with a hold-time value higher than the configured

BFD timer(1s) to have zero loss convergence during recovery of failed devices.

[edit]set interfaces xe-0/3/6 hold-time up 100set interfaces xe-0/3/6 hold-time down 3000set interfaces xe-0/3/7 hold-time up 100set interfaces xe-0/3/7 hold-time down 3000set interfaces xe-1/3/6 hold-time up 100set interfaces xe-1/3/6 hold-time down 3000set interfaces xe-1/3/7 hold-time up 100set interfaces xe-1/3/7 hold-time down 3000set interfaces xe-3/3/6 hold-time up 100set interfaces xe-3/3/6 hold-time down 3000set interfaces xe-3/3/7 hold-time up 100set interfaces xe-3/3/7 hold-time down 3000set interfaces xe-5/3/6 hold-time up 100set interfaces xe-5/3/6 hold-time down 3000set interfaces xe-5/3/7 hold-time up 100set interfaces xe-5/3/7 hold-time down 3000

i. Configure IRB on both the core-sw1 and core-sw2 and enable VRRP.

[edit]set interfaces irb unit 50 family inet address 192.168.50.2/24 arp 192.168.50.1l2-interface ae9.0

set interfaces irb unit 50 family inet address 192.168.50.2/24 arp 192.168.50.1 mac4c:96:14:6b:db:f0

set interfaces irbunit50family inetaddress 192.168.50.2/24arp 192.168.50.1publishset interfaces irb unit 50 family inet address 192.168.50.2/24 vrrp-group 1virtual-address 192.168.50.254

set interfaces irb unit 50 family inet address 192.168.50.2/24 vrrp-group 1 priority250

set interfaces irbunit50family inetaddress 192.168.50.2/24vrrp-group 1 fast-interval100

set interfaces irb unit 50 family inet address 192.168.50.2/24 vrrp-group 1 preemptset interfaces irbunit50family inetaddress 192.168.50.2/24vrrp-group 1accept-dataset interfaces irb unit 50 family inet address 192.168.50.2/24 vrrp-group 1authentication-typemd5




j. Configure LACP on the AE links.

[edit]set interfaces ae0 description "MC-LAG to VDC-pod1-sw1-nng-ae0"set interfaces ae0 aggregated-ether-options lacp activeset interfaces ae0 aggregated-ether-options lacp system-priority 100set interfaces ae0 aggregated-ether-options lacp system-id 00:00:00:00:00:01set interfaces ae0 aggregated-ether-options lacp admin-key 1

k. Configure VLAN and enable “domain-type bridge” to configure IRB as the

routing-interface under this bridge domain.

[edit]set vlans POD1-Transport-1 vlan-id 50set vlans POD1-Transport-1 l3-interface irb.50set vlans POD1-Transport-1 domain-type bridge

NOTE: The ELS CLI doesn’t need anMC-AE interface ICL under thebridge configuration. It picks the appropriateMC-AE interface from theVLAN association on the configured links.

l. Configure ICCP protocol parameters.

[edit]set protocols iccp local-ip-addr 192.168.168.5set protocols iccp peer 192.168.168.4 redundancy-group-id-list 1set protocols iccp peer 192.168.168.4 backup-liveness-detection backup-peer-ip192.168.168.4

set protocols iccp peer 192.168.168.4 liveness-detectionminimum-interval 500set protocols iccp peer 192.168.168.4 liveness-detectionmultiplier 2

m. Configure switch service-id for MC-LAG.

[edit]set switch-options service-id 1

NOTE: Youmustconfigurethesameuniquenetwork-wideconfigurationfor a service in the set of PE routers providing the service. This serviceID is required if themulti-chassis aggregated Ethernet interfaces arepart of a bridge domain.

Verification

Purpose The following verification section (with verification commands and sample output) can

be used to confirm that MC-LAG Active/Active between the core and PODs has been

configured correctly.

Results



1. Verify that ICCP is configured and is showing as Up.

root@VDC-core-sw1-re0#run show iccp

Redundancy Group Information for peer 192.168.168.5 TCP Connection : Established Liveliness Detection : UpBackup liveness peer status: Up Redundancy Group ID Status 1 Up <<<< ICCP protocol is UP for RG ID 1>>>Client Application: l2ald_iccpd_client <<< L2 Forwarding joined the Redundancy group >>> Redundancy Group IDs Joined: 1 Client Application: lacpd Redundancy Group IDs Joined: 1 <<<< redundancy group ID

2. Verify that the ICL is configured with all VLANS and that the ICL is up. Verify that all

VLANS configured on the MC-AE interfaces are properly allowed over the ICL link.

root@VDC-core-sw1-re0#show interfaces ae9

lacp {active;periodic fast;}

}unit 0 {description "ICL Link for all VLANS";family ethernet-switching {interface-mode trunk;vlan {members[Exchange InfraSQLSharePointFirewallCompute-MGMTWikimediaExchange-ClusterTera-VMLoad-balancer-ExtSecurity-MgmtVmotionVM-FTRemote-Access OOB-Transport Load-balancer-Ext-Tera-VMTrafficGenerator-502 TrafficGenerator-503 TrafficGenerator-504POD1-Transport-1 POD1-Transport-2 POD1-Transport-3 POD1-Transport-4POD2-Transport-1 POD2-Transport-2 ];

}}

}

root@VDC-core-sw1-re0#run show interfaces ae8 extensive

Physical interface: ae8 (MC-AE-9, active), Enabled, Physical link is Up Interface index: 136, SNMP ifIndex: 517, Generation: 139 Description: MC-LAG to OOB Link-level type: Ethernet, MTU: 9192, Speed: 20Gbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 0 Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x4000 Current address: 4c:96:14:6b:db:c8, Hardware address: 4c:96:14:6b:db:c8 Last flapped : 2013-10-22 13:16:00 PDT (18:45:32 ago) Statistics last cleared: Never Traffic statistics: Input bytes : 1407792089 10280 bps Output bytes : 1984111643 5464 bps Input packets: 11931513 4 pps Output packets: 5950574 5 pps



IPv6 transit statistics: Input bytes : 0 Output bytes : 0 Input packets: 0 Output packets: 0 Dropped traffic statistics due to STP State: Input bytes : 0 Output bytes : 0 Input packets: 0 Output packets: 0 Input errors: Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Giants: 0, Policed discards: 0, Resource errors: 0 Output errors: Carrier transitions: 0, Errors: 0, Drops: 0, MTU errors: 0, Resource errors: 0 Ingress queues: 8 supported, 4 in use Queue counters: Queued packets Transmitted packets Dropped packets 0 best-effort 0 0

0 1 expedited-fo 0 0

0 2 assured-forw 0 0

0 3 network-cont 0 0

0 Egress queues: 8 supported, 4 in use Queue counters: Queued packets Transmitted packets Dropped packets 0 best-effort 5568474 5568474

0 1 expedited-fo 0 0

0 2 assured-forw 0 0

0 3 network-cont 1352698 1352698

0 Queue number: Mapped forwarding classes 0 best-effort 1 expedited-forwarding

2 assured-forwarding 3 network-control

Logical interface ae8.0 (Index 356) (SNMP ifIndex 540) (Generation 165)

Flags: SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge Statistics Packets pps Bytes bps Bundle: Input : 11896496 4 1397993319 10280 Output: 5915576 5 1989022843 5160 Link: xe-3/1/0.0



Input : 8765149 2 901071300 2728 Output: 3542312 5 1126682587 4712 xe-3/1/1.0 Input : 3131347 2 496922019 7552 Output: 2408262 0 872699254 448 LACP info: Role System System Port Port Port priority identifier priority number

key xe-3/1/0.0 Actor 100 00:00:00:00:00:09 127 86

9 xe-3/1/0.0 Partner 127 88:e0:f3:1f:f0:a0 127 1

1 xe-3/1/1.0 Actor 100 00:00:00:00:00:09 127 87

9 xe-3/1/1.0 Partner 127 88:e0:f3:1f:f0:a0 127 4

1 LACP Statistics: LACP Rx LACP Tx Unknown Rx Illegal Rx xe-3/1/0.0 485041 485547 0 0 xe-3/1/1.0 485095 485495 0 0 Marker Statistics: Marker Rx Resp Tx Unknown Rx Illegal Rx xe-3/1/0.0 0 0 0 0 xe-3/1/1.0 0 0 0 0 Protocol eth-switch, MTU: 9192, Generation: 199, Route table: 5 Flags: Trunk-Mode

3. Verify the MC-AE interface status.

root@VDC-core-sw1-re0#run show interfacesmc-ae

Member Link : ae0Current State Machine's State: mcae active state >>> Active/ActiveLocal Status : activeLocal State : up >>> Should be upPeer Status : activePeer State : up Logical Interface : ae0.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 up >>>Should show up with the correct ICL interface.>>Member Link : ae1Current State Machine's State: mcae active stateLocal Status : activeLocal State : upPeer Status : activePeer State : up Logical Interface : ae1.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 upMember Link : ae2Current State Machine's State: mcae active stateLocal Status : activeLocal State : up



Peer Status : activePeer State : up Logical Interface : ae2.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 upMember Link : ae3Current State Machine's State: mcae active stateLocal Status : activeLocal State : upPeer Status : activePeer State : up Logical Interface : ae3.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 upMember Link : ae4Current State Machine's State: mcae active stateLocal Status : activeLocal State : upPeer Status : activePeer State : up Logical Interface : ae4.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 upMember Link : ae5Current State Machine's State: mcae active stateLocal Status : activeLocal State : upPeer Status : activePeer State : up Logical Interface : ae5.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 upMember Link : ae6Current State Machine's State: mcae active stateLocal Status : activeLocal State : upPeer Status : activePeer State : up Logical Interface : ae6.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 up Member Link : ae7 Current State Machine's State: mcae active state Local Status : active Local State : up Peer Status : active Peer State : up Logical Interface : ae7.0 Topology Type : bridge Local State : up



Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 up Member Link : ae8 Current State Machine's State: mcae active state Local Status : active Local State : up Peer Status : active Peer State : up Logical Interface : ae8.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 up Member Link : ae10 Current State Machine's State: mcae active state Local Status : active Local State : up Peer Status : active Peer State : up Logical Interface : ae10.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 up Member Link : ae11 Current State Machine's State: mcae active state Local Status : active Local State : up Peer Status : active Peer State : up Logical Interface : ae11.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 up Member Link : ae12 Current State Machine's State: mcae active state Local Status : active Local State : up Peer Status : active Peer State : up Logical Interface : ae12.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 up Member Link : ae13 Current State Machine's State: mcae active state Local Status : active Local State : up Peer Status : active Peer State : up Logical Interface : ae13.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 up Member Link : ae14 Current State Machine's State: mcae active state Local Status : active Local State : up



Peer Status : active Peer State : up Logical Interface : ae14.0 Topology Type : bridge Local State : up Peer State : up Peer Ip/MCP/State : 192.168.168.5 ae9.0 up

4. Verify that the ICL and MC-AE interfaces are in the same broadcast domain.

root@VDC-core-sw1-re0#run show vlans POD1-Transport-2

Routing instance VLAN name Tag Interfacesdefault-switch POD1-Transport-2 51 ae1.0* ae9.0*

5. Verify that BFD is configured on all IRB interfaces with an “Up” status. Also verify that

appropriate timers are configured. (A 6-second timer is supported on EX9200 for

MC-LAG Active/Active.)

root@VDC-core-sw1-re0>show bfd session

Detect TransmitAddress State Interface Time Interval Multiplier192.168.2.2 Up ae20.0 6.000 2.000 3

192.168.20.2 Up irb.20 6.000 2.000 3

192.168.20.3 Up irb.20 6.000 2.000 3 192.168.25.2 Up irb.10 6.000 2.000 3 192.168.25.3 Up irb.10 6.000 2.000 3 192.168.50.2 Up irb.50 6.000 2.000 3 192.168.50.3 Up irb.50 6.000 2.000 3 192.168.51.2 Up irb.51 6.000 2.000 3 192.168.51.3 Up irb.51 6.000 2.000 3 192.168.52.2 Up irb.52 6.000 2.000 3 192.168.52.3 Up irb.52 6.000 2.000 3 192.168.53.2 Up irb.53 6.000 2.000 3 192.168.53.3 Up irb.53 6.000 2.000 3 192.168.54.2 Up irb.54 6.000 2.000 3 192.168.54.3 Up irb.54 6.000 2.000 3 192.168.55.2 Up irb.55 6.000 2.000 3 192.168.55.3 Up irb.55 6.000 2.000 3 192.168.168.5 Up 6.000 2.000 3

6. Verify the status of VRRP.

root@VDC-core-sw1-re0>show vrrp summary

Interface State Group VR state VR Mode Type Address irb.10 up 1 backup Active lcl 192.168.25.1

vip 192.168.25.254 irb.15 up 1 backup Active lcl 192.168.15.1












vip 172.16.1.254

irb.106 up 1 backup Active lcl 172.16.6.252

vip 172.16.6.254


vip 172.16.7.254


vip 172.16.9.254


vip 10.30.3.254

root@VDC-core-sw2-re0>show vrrp summary

Interface State Group VR state VR Mode Type Address irb.10 up 1 master Active lcl 192.168.25.2

vip 192.168.25.254 irb.15 up 1 master Active lcl 192.168.15.2

vip 192.168.15.254



irb.16 up 1 master Active lcl 192.168.16.2









vip 172.16.1.254


vip 172.16.6.254


vip 172.16.7.254


vip 172.16.9.254


vip 10.30.3.254



Routing Configuration


• Topology on page 104

• Configuring BGP Between the EDGE and Service Provider on page 106


• Configuring OSPF in the Data Center on page 111


Overview

TheMetaFabric 1.0solution features routingconfiguredbetween theedgeand theservice

provider, as well as routing of traffic from the edge and core roles. BGP is implemented

at the edge. OSPF is used as the IGP in the solution.

Topology

The routing topology and configuration are illustrated in Figure 45 on page 105.



Figure 45: MetaFabric 1.0 Routing Configuration and Topology

The configuration of routing in this solution is outlined in the following sections:

• BGP implementation

• Configuring simulated service provider connectivity

• Configuring VDC-Edge-R1

• Configuring VDC-Edge-R2

• Verifying the configuration

• OSPF implementation

• Configuration overview and parameters

• Link-free alternate overview



• Configuration examples

• Verifying the configuration

Configuring BGP Between the EDGE and Service Provider

In VDC 1.0, Internet connectivity for the data center is achieved by establishing EBGP

peering with multiple service providers. As shown in Figure 45 on page 105, EBGP is

configured from Edge-R1 to SP1 and Edge-R2 to SP2. Internet routes are simulated using

testing tools connected to SP1 and SP2. SP1 and SP2 advertise the same Internet routes

to the edge routers.

The next element of routing in the solution is the configuration of EDGE R1 and EDGE R2

peering via iBGP with an export policy to enable "next-hop self". BGP local preference

is configured to prefer the SP1.

• The edge routers must advertise the data center's business-critical applications’

(SharePoint, Exchange, andWikimedia) public address space into the Internet for the

Internet users to access the data center resources. To support redundancy, each edge

router is advertising the same prefix into the Internet.

• Application server Internet access is provided using Source NAT on the edge firewall

and forwarded to the edge routers for Internet access to service provider networks.

• Remote access users connecting from Internetwill use the JunosPulse gateway public

IP address for the VPN connection. The SA appliance VM hosting the pulse gateway

service IP address is advertised to the Internet using an export policy.

To configure BGP between the edge and the service provider, follow these steps:

1. Configure the Simulated Service Provider (1).

a. Configure the AE interface to Edge-R1.

[edit]set interfaces xe-0/0/20 ether-options 802.3ad ae1set interfaces xe-0/0/22 ether-options 802.3ad ae1

set interfaces ae1 description "To VDC Edge R1"set interfaces ae1 aggregated-ether-options lacp activeset interfaces ae1 aggregated-ether-options lacp periodic fastset interfaces ae1 unit 0 family inet address 10.94.127.229/30

b. Configure EBGP peering with Edge-R1.

[edit]set protocols bgp group EDGE-R1 local-address 10.94.127.229set protocols bgp group EDGE-R1 neighbor 10.94.127.230 peer-as 64512

NOTE: Step 1 is provided for completeness. In a real-world scenario, theservice provider configuration is outside of administrator control. Thesolution validation lab simulated a service provider connection as shownin step 1.



2. Configure VDC-Edge-R1.

a. Configure routing-options and EBGP to SP1.

[edit]set routing-options autonomous-system 64512set protocols bgp group SP1 export Export-VDC-Subnetsset protocols bgp group SP1 neighbor 10.94.127.229 peer-as 100

b. Configure iBGP peering with EDGE-R2.

[edit]set protocols bgp group EDGE-R2 local-address 192.168.168.1set protocols bgp group EDGE-R2 neighbor 192.168.168.2 peer-as 64512

c. Configure route export “next-hop-self”.

[edit]set protocols bgp group EDGE-R2 export next-hop-self

d. Configure the BGP export policy to advertise the applications’ public prefix

[edit]set policy-options policy-statement Export-VDC-Subnets term App-Server-VIPfrom protocol ospf

set policy-options policy-statement Export-VDC-Subnets term App-Server-VIPfrom route-filter 10.94.127.128/26 exact accept

e. Configure the remote secure access prefix export policy

[edit]set policy-options policy-statement Export-VDC-Subnets term Secure-Acces-IPfrom protocol ospf

set policy-options policy-statement Export-VDC-Subnets term Secure-Acces-IPfrom route-filter 10.94.127.32/27 exact accept

f. Enable Internet access for application servers.

[edit]set policy-options policy-statement Export-VDC-Subnets termServer-Internet-NAT-IP from protocol ospf

set policy-options policy-statement Export-VDC-Subnets termServer-Internet-NAT-IP from route-filter 10.94.127.0/27 exact accept

g. Configure the next-hop-self policy.

[edit]set policy-optionspolicy-statement next-hop-self term 1 then local-preference200set policy-options policy-statement next-hop-self term 1 then next-hop selfset policy-options policy-statement next-hop-self term 1 then accept

3. Configure VDC-Edge-R2.

a. Configure EBGP and export policy to SP2.

[edit]set protocols bgp group T0-B6-Gateway neighbor 10.94.127.241 peer-as 300set protocols bgp group EDGE-R2 export from-ospfset protocols bgp group EDGE-R2 neighbor 10.94.127.246 peer-as 64512

b. Configure BGP export of VDC subnets.

[edit]



set protocols bgp group SP2 export Export-VDC-Subnetsset protocols bgp group SP2 neighbor 10.94.127.245 peer-as 200

c. Configure iBGP peering with VDC-Edge-R1.

[edit]set protocols bgp group EDGE-R1 local-address 192.168.168.2set protocols bgp group EDGE-R1 export next-hop-selfset protocols bgp group EDGE-R1 neighbor 192.168.168.1 peer-as 64512

d. Configure routing policy.

[edit]set policy-options policy-statement Export-VDC-Subnets term App-Server-VIPfrom protocol ospf

set policy-options policy-statement Export-VDC-Subnets term App-Server-VIPfrom route-filter 10.94.127.128/26 exact accept

set policy-options policy-statement Export-VDC-Subnets term Secure-Acces-IPfrom protocol ospf

set policy-options policy-statement Export-VDC-Subnets term Secure-Acces-IPfrom route-filter 10.94.127.32/27 exact accept

set policy-options policy-statement Export-VDC-Subnets termServer-Internet-NAT-IP from protocol ospf

set policy-options policy-statement Export-VDC-Subnets termServer-Internet-NAT-IP from route-filter 10.94.127.0/27 exact accept

set policy-options policy-statement Export-VDC-Subnets term Tera-VM-Serverfrom route-filter 10.20.127.0/24 exact accept

set policy-options policy-statement Export-VDC-Subnets term TrafficGeneratorfrom protocol ospf

set policy-options policy-statement Export-VDC-Subnets term TrafficGeneratorfrom route-filter 10.30.2.0/24 exact accept



e. Configure BGP “next-hop self”.

[edit]set policy-options policy-statement next-hop-self term 1 from protocol bgpset policy-options policy-statement next-hop-self term 1 then local-preference 100set policy-options policy-statement next-hop-self term 1 then next-hop selfset policy-options policy-statement next-hop-self term 1 then accept

Verification

Purpose The following verification commands (with sample output) can be used to confirm BGP

configuration.

Results



1. Verify on Edge-R1 that an EBGP session with SP1 exists. Also verify the iBGP session

with Edge-R2.

root@VDC-edge-r01-re0>show bgp summary

Groups: 2 Peers: 2 Down peers: 0Table Tot Paths Act Paths Suppressed History Damp State Pendinginet.0 16 7 0 0 0 0Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn

State|#Active/Received/Accepted/Damped...10.94.127.229 100 46637 46471 0 0 2w0d12h 7/16/16/0 0/0/0/0192.168.168.2 64512 46210 47057 0 0 2w0d12h 0/0/0/0 0/0/0/0

2. Verify on the Edge-R2 that EBGP peering with SP2 and iBGP peering with Edge-R1 is

successful.

root@VDC-edge-r2-re0>show bgp summary

Groups: 2 Peers: 2 Down peers: 0Table Tot Paths Act Paths Suppressed History Damp State Pendinginet.0 75 8 0 0 0 0Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn

State|#Active/Received/Accepted/Damped...10.94.127.245 200 46378 46671 0 0 2w0d12h 0/16/16/0 0/0/0/0192.168.168.1 64512 47062 46215 0 0 2w0d12h 8/59/59/0 0/0/0/0

3. Verify the routing table on Edge-R1 by showing routes received via BGP.

root@VDC-edge-r01-re0>show route receive-protocol bgp 10.94.127.229

inet.0: 65 destinations, 76 routes (65 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path 0.0.0.0/0 10.94.127.229 100 I* 10.10.0.0/16 10.94.127.229 2 100 I* 10.40.0.0/16 10.94.127.229 2 100 I 10.93.222.0/23 10.94.127.229 100 I 10.94.47.0/27 10.94.127.229 100 I 10.94.63.18/32 10.94.127.229 100 I 10.94.63.19/32 10.94.127.229 100 I* 10.94.127.192/27 10.94.127.229 2 100 I* 10.94.127.224/30 10.94.127.229 100 I 10.94.127.228/30 10.94.127.229 100 I* 10.94.127.240/30 10.94.127.229 2 100 I* 12.12.12.12/32 10.94.127.229 1 100 I 172.17.0.0/16 10.94.127.229 100 I 172.22.0.0/16 10.94.127.229 100 I 172.23.0.0/16 10.94.127.229 100 I* 192.168.253.180/30 10.94.127.229 2 100 I



4. Verify the routing table on Edge-R2 by showing routes received via BGP.


inet.0: 66 destinations, 135 routes (66 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path 0.0.0.0/0 10.94.127.245 200 I 10.10.0.0/16 10.94.127.245 2 200 I 10.40.0.0/16 10.94.127.245 2 200 I 10.93.222.0/23 10.94.127.245 200 I 10.94.47.0/27 10.94.127.245 200 I 10.94.63.18/32 10.94.127.245 200 I 10.94.63.19/32 10.94.127.245 200 I 10.94.127.192/27 10.94.127.245 2 200 I 10.94.127.224/30 10.94.127.245 2 200 I 10.94.127.240/30 10.94.127.245 200 I 10.94.127.244/30 10.94.127.245 200 I 12.12.12.12/32 10.94.127.245 1 200 I 172.17.0.0/16 10.94.127.245 200 I 172.22.0.0/16 10.94.127.245 200 I 172.23.0.0/16 10.94.127.245 200 I 192.168.253.180/30 10.94.127.245 2 200 I

5. Verify that only the configured prefixes on Edge-R1 are advertised to the service

providers as per the configured policy.

root@VDC-edge-r01-re0>show route advertising-protocol bgp 10.94.127.229

inet.0: 65 destinations, 76 routes (65 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path* 10.20.127.0/24 Self 0 I* 10.30.2.0/24 Self 3025 I* 10.30.3.0/24 Self 2025 I* 10.30.4.0/24 Self 3025 I* 10.94.127.0/27 Self 1025 I* 10.94.127.32/27 Self 0 I* 10.94.127.128/26 Self 0

6. Verify that only the configured prefixes on Edge-R2 are advertised to the service

provider as per the configured export policy.


inet.0: 66 destinations, 135 routes (66 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path 0.0.0.0/0 10.94.127.245 200 I 10.10.0.0/16 10.94.127.245 2 200 I 10.40.0.0/16 10.94.127.245 2 200 I 10.93.222.0/23 10.94.127.245 200 I 10.94.47.0/27 10.94.127.245 200 I 10.94.63.18/32 10.94.127.245 200 I 10.94.63.19/32 10.94.127.245 200 I 10.94.127.192/27 10.94.127.245 2 200 I 10.94.127.224/30 10.94.127.245 2 200 I 10.94.127.240/30 10.94.127.245 200 I 10.94.127.244/30 10.94.127.245 200 I 12.12.12.12/32 10.94.127.245 1 200 I 172.17.0.0/16 10.94.127.245 200 I 172.22.0.0/16 10.94.127.245 200 I 172.23.0.0/16 10.94.127.245 200 I 192.168.253.180/30 10.94.127.245 2 200 I



Configuring OSPF in the Data Center

The MetaFabric 1.0 solution uses OSPF as the IGP because of widespread familiarity of

the protocol. Both edge routers and core switches are configured with Layer 2 MC-LAG

(Active/Active).

Layer 3 connectivity for the edge firewalls and POD switches is enabled using a bridge

domain and IRB interfaces. OSPF is also enabled on POD switches on the IRB toward

the core switch for hybrid Layer 2 and Layer 3 connectivity to the core.

This section contains configurations, configuration parameters, and topology overviews

in the following areas:

• Configuration overview and parameters

• Conditional default route advertising to OSPF

• Link-free alternate overview and configuration

• OSPF configuration examples

• Verification

Configuration Overview and Parameters

OSPF was configured in the MetaFabric 1.0 solution validation lab using the following

design considerations and configuration elements:

• Three OSPF areas configured to localize the failure with the area boundary

• Edge routers and firewalls are MC-LAG/IRB Layer 3 interfaces in area 1.

• OOBmanagement network is in area 0 connected to the core-switch.

• Link between core-switches are in area 0.

• Link between core-switch andmanagement switch is in area 0.

• POD1 is configured as totally stubby area 10.

• POD2 is configured as totally stubby area 11.

• Reference bandwidth 1000g.

• No-active-backbone is enabled toward POD switches.

• Each core switch and edge router needs to be configured with an OSPF priority of 255

and 254 to strictly enforce that the core switch and edge routers will always become

the designated router and backup designated router for that bridge domain.

• All IRBsandVRRPaddressesshouldbeadvertised intoOSPFaspassiveso that sessions

don’t get established with a server or other devices, but the Layer 3 connected routes

are advertised into OSPF.

• Conditional-baseddefault aggregate routes fromedge routersare re-distributed toward

the core and POD switches for Internet connectivity.

• Loop-free Alternate (LFA) is configured on all the OSPF links to improve convergence.



Figure 46: OSPF Area Configuration Between Edge and Core (IncludingOut-of-BandManagement)

ThePODswitchesareconnected to thecore switchesusingLAG.ThePODsareconfigured

as OSPF areas 10 and 11 (Figure 47 on page 112).

Figure 47: OSPF Area Configuration Between Core and PODs

Configuring Conditional Default Route Advertising to OSPF Based on a Service Provider

Internet Route

In this configuration, if a route exists (the example given is a route to 12.12.12.12), an

aggregate default route (0.0.0.0/0) will be created and exported into OSPF. If the route

to 12.12.12.12 does not exist, due to both SP links being down or the EBGP peering with

the SP being down, the gendefault policy will fail. This policy failure means that the

default aggregate route is not created or exported to OSPF. Configuration of conditional

default route advertising is covered in this section.



To configure conditional route advertising to OSPF based on the presence of a service

provider Internet route, follow these steps:

1. On the edge routers, check to verify that the route to 12.12.12.12 has been learned via

EBGP(fromanyserviceprovider using thepolicy). Createagendefault policymatching

a route from SP 12.12.12.12.

[edit]set policy-options policy-statement gendefault term upstreamroutes from route-filter12.12.12.12/32 exact

set policy-options policy-statement gendefault term upstreamroutes from protocolbgp

set policy-options policy-statement gendefault term upstreamroutes then acceptset policy-options policy-statement gendefault term deny then reject

2. If the policy matches, then an aggregate default route will be created.

[edit]set routing-options generate route 0.0.0.0/0 policy gendefault

3. Create a routing policy to export the aggregated route into OSPF.

[edit]setpolicy-optionspolicy-statementexport-default-route term1 fromprotocolaggregateset policy-options policy-statement export-default-route term 1 from route-filter0.0.0.0/0 exact

set policy-options policy-statement export-default-route term 1 then external type 1set policy-options policy-statement export-default-route term 1 then accept

4. Apply the policy to OSPF.

[edit]set protocols ospf export export-default-route

Configuring Loop-Free Alternate

Loop-free alternate (LFA) is a mechanism used to pre-calculate the loop-free backup

paths and program the forwarding table to quickly use in case of failure for a given prefix.

The IGP SPF database is used to calculate the LFA route. After a failure is detected using

BFD or OAM, the PFE-complex signals the failure up to the control plane, where the

routingdaemon (RPD) resides. Pleasenote that at this stage, theonly thing that is known

is a failed link. Once this happens, the router now has two tasks:

• Calculating a shortest path to the destination and choosing a next hop to circumvent

the failed link. Typically a link-state protocol is being used as IGP, such as OSPF and

IS-IS. Both protocols will have to perform a Dijkstra-algorithm to calculate a loop-free

topology again. This is known as shortest-path first (SPF) calculation.

• Informing theneighbors of this link failure toallowother routers calculatinganupdated

loop-free topology as well.

AfterRPDcalculatesall newnexthops forall givenprefixes, this newnext-hop information

needs to be pushed down to the Packet Forwarding Engine. Once the Packet Forwarding

Engine is updated, traffic is reroutedand flowsagain. At this stage, local-repair is achieved,

as this node has done all calculations and rerouting. Other nodes in the network might

not yet have received the updates regarding the link failure. Once all nodes in the network



have received the IGP updates regarding the failure and updated their next hops, global

convergence is reached.

As seen in Figure 48 on page 114, convergence of LFA does take time.

Figure 48: Loop-Free Alternate Convergence Example

The goal of LFA is to pre-programPacket Forwarding Engine backup paths for the known

prefixes in a loop-free manner to speed up convergence without the need to wait for

either local repair start or global repair.

LFA is changing the event flow behavior described in Figure 48 on page 114. Rerouting

beginsmuch earlier, immediately after detecting the link failure. There is no need to wait

for the routing protocol process (rpd) to complete the SPF calculation or wait for

convergence of other routers in the network (global convergence). As a result of being

able to reroute after the Packet Forwarding Engine is aware of the link error, LFA shows

better failover time in the test bed. For LFA to work, the Packet Forwarding Engine must

have a backup next hop preinstalled in it. Because it is preinstalled, there is no need to

wait for an SPF-calculation done by the routing protocol process (rpd). This backup next

hop gets a new name here: the loop-free alternate (LFA). The loop-free alternate next

hop will be used at once if the primary next hop is going down. LFA does not need any

additional protocol. LFA is self-contained and does not rely on any helper node to work

properly; as such, rollout is be done in small steps. LFA preinstalls a backup next hop in

for the forwarding plane. This backup next hop is elected by running multiple SPF

calculations , each with a different neighbor as the root of the tree. Upon link failure, the

backup next hop can be immediately selected, as this will provide loop-free forwarding

for a given destination node.

To configure Loop-free Alternate, follow these steps:

1. Configure node-link-protection under theOSPF interface configuration to enable LFA.

[edit]set protocols ospf area 0.0.0.1 interface irb.0 node-link-protection

2. Configure per-packet load balancing (PPLB) to allow the Packet Forwarding Engine

to retain the LFA backup next hops in the Packet Forwarding Engine.

[edit]set policy-options policy-statement pplb then load-balance per-packetset policy-options policy-statement pplb then accept

Configuring OSPF in the Data Center



OSPF configuration in the data center is covered in the following section. The following

nodes and solution elements are configured below:

• Edge router configuration

• Firewall configuration

• Core switch configuration

• POD switch configuration



To configure OSPF in the data center, follow these steps:

1. Configure the edge router.

a. Configure and apply export policy, SPF reference bandwidth, LFA, and priority.

[edit]set protocols ospf export export-default-routeset protocols ospf reference-bandwidth 1000gset protocols ospf area 0.0.0.1 interface irb.0 node-link-protectionset protocols ospf area 0.0.0.1 interface irb.0 priority 254

b. Enable BFD protection on the IRB.

[edit]set protocols ospf area 0.0.0.1 interface irb.0 bfd-liveness-detectionminimum-interval 500

set protocols ospf area 0.0.0.1 interface ae0.0 node-link-protectionset protocols ospf area 0.0.0.1 interface ae0.0 priority 254set protocols ospf area 0.0.0.1 interface ae0.0 bfd-liveness-detectionminimum-interval 500

set protocols ospf area 0.0.0.1 interface ae0.0 bfd-liveness-detectionmultiplier 2set protocols ospf area 0.0.0.1 interface lo0.0

c. Configure the condition policy for the OSPF default route based on the BGP route.

[edit]set policy-options policy-statement export-default-route term 1 from protocolaggregate

set policy-options policy-statement export-default-route term 1 from route-filter0.0.0.0/0 exact

set policy-options policy-statement export-default-route term 1 then external type1

set policy-options policy-statement export-default-route term 1 then acceptset policy-optionspolicy-statement gendefault termupstreamroutes fromprotocolbgp

setpolicy-optionspolicy-statementgendefault termupstreamroutesfromroute-filter12.12.12.12/32 exact

set policy-options policy-statement gendefault term upstreamroutes then acceptset policy-options policy-statement gendefault term deny then rejectset routing-options generate route 0.0.0.0/0 policy gendefault

d. Configure PPLB.

[edit]set policy-options policy-statement pplb then load-balance per-packetset policy-options policy-statement pplb then acceptset routing-options forwarding-table export pplb

2. Configure OSPF and BFD on the firewalls.

[edit]set protocols ospf export OSPF-Export-SAset protocols ospf reference-bandwidth 1000gset protocols ospf area 0.0.0.1 interface reth1.0 node-link-protectionsetprotocolsospfarea0.0.0.1 interface reth1.0bfd-liveness-detectionminimum-interval500

set protocols ospf area 0.0.0.1 interface reth1.0 bfd-liveness-detectionmultiplier 2set protocols ospf area 0.0.0.1 interface reth0.0 node-link-protection



set protocols ospf area 0.0.0.1 interface reth0.0 bfd-liveness-detectionminimum-interval 500

set protocols ospf area 0.0.0.1 interface reth0.0 bfd-liveness-detectionmultiplier 2set protocols ospf area 0.0.0.1 interface fxp0.0 disable

3. Configure OSPF and BFD in the core switches

[edit]set protocols ospf export export-ospfset protocols ospf reference-bandwidth 1000gset protocols ospf area 0.0.0.10 stub default-metric 100set protocols ospf area 0.0.0.10 stub no-summariesset protocols ospf area 0.0.0.10 interface fxp0.0 disableset protocols ospf area 0.0.0.10 interface irb.50 node-link-protectionset protocols ospf area 0.0.0.10 interface irb.50 priority 255setprotocolsospfarea0.0.0.10 interface irb.50bfd-liveness-detectionminimum-interval500

set protocols ospf area 0.0.0.10 interface irb.50 bfd-liveness-detectionmultiplier 2set protocols ospf area 0.0.0.10 interface irb.51 node-link-protectionset protocols ospf area 0.0.0.10 interface irb.51 priority 255setprotocolsospfarea0.0.0.10 interface irb.51bfd-liveness-detectionminimum-interval500


set protocols ospf area 0.0.0.10 interface irb.52 bfd-liveness-detectionmultiplier 2set protocols ospf area 0.0.0.10 interface irb.53 priority 255setprotocolsospfarea0.0.0.10 interface irb.53bfd-liveness-detectionminimum-interval500

set protocols ospf area 0.0.0.10 interface irb.53 bfd-liveness-detectionmultiplier 2set protocols ospf area 0.0.0.0 interface lo0.0 passiveset protocols ospf area 0.0.0.0 interface irb.107 passiveset protocols ospf area 0.0.0.0 interface irb.20 node-link-protectionsetprotocolsospfarea0.0.0.0 interface irb.20bfd-liveness-detectionminimum-interval500

set protocols ospf area 0.0.0.0 interface irb.20 bfd-liveness-detectionmultiplier 2

set protocols ospf area 0.0.0.0 interface irb.101 passiveset protocols ospf area 0.0.0.0 interface irb.109 passiveset protocols ospf area 0.0.0.0 interface irb.106 passiveset protocols ospf area 0.0.0.0 interface ae20.0 node-link-protectionset protocols ospf area 0.0.0.0 interface ae20.0 priority 255set protocols ospf area 0.0.0.0 interface ae20.0 bfd-liveness-detectionminimum-interval 500

set protocols ospf area 0.0.0.0 interface ae20.0 bfd-liveness-detectionmultiplier 2set protocols ospf area 0.0.0.0 interface irb.20 node-link-protectionset protocols ospf area 0.0.0.0 interface irb.20 priority 255setprotocolsospfarea0.0.0.0 interface irb.20bfd-liveness-detectionminimum-interval500

set protocols ospf area 0.0.0.0 interface irb.52 bfd-liveness-detectionmultiplier 2

set protocols ospf area 0.0.0.0 interface irb.503 passiveset protocols ospf area 0.0.0.11 stub default-metric 100set protocols ospf area 0.0.0.11 stub no-summaries



set protocols ospf area 0.0.0.11 interface irb.54 node-link-protectionset protocols ospf area 0.0.0.11 interface irb.54 priority 255setprotocolsospfarea0.0.0.11 interface irb.54bfd-liveness-detectionminimum-interval500



set protocols ospf area 0.0.0.1 interface irb.52 bfd-liveness-detectionmultiplier 2set protocols ospf area 0.0.0.1 interface irb.14 node-link-protectionset protocols ospf area 0.0.0.1 interface irb.14 passiveset protocols ospf area 0.0.0.1 interface irb.14 priority 255set protocols ospf area 0.0.0.1 interface irb.15 node-link-protectionset protocols ospf area 0.0.0.1 interface irb.15 passiveset protocols ospf area 0.0.0.1 interface irb.15 priority 255

4. Configure OSPF in the PODs.

[edit]set protocols ospf reference-bandwidth 1000gset protocols ospf area 0.0.0.10 stub no-summariesset protocols ospf area 0.0.0.10 interface vlan.104 passiveset protocols ospf area 0.0.0.10 interface vlan.108 passiveset protocols ospf area 0.0.0.10 interface vlan.103 passiveset protocols ospf area 0.0.0.10 interface vlan.502 passiveset protocols ospf area 0.0.0.10 interface vlan.503set protocols ospf area 0.0.0.10 interface vlan.50 node-link-protectionset protocols ospf area 0.0.0.10 interface vlan.50 bfd-liveness-detectionminimum-interval 500

set protocols ospf area 0.0.0.10 interface vlan.50 bfd-liveness-detectionmultiplier 2set protocols ospf area 0.0.0.10 interface vlan.51 node-link-protectionset protocols ospf area 0.0.0.10 interface vlan.51 bfd-liveness-detectionminimum-interval 500



set protocols ospf area 0.0.0.10 interface vlan.53 bfd-liveness-detectionmultiplier 2set protocols ospf area 0.0.0.10 interface vlan.501 passive

Verification

Purpose The following verificationcommands (with sampleoutput) canbeused toconfirmOSPF

configuration in the data center.

Results



1. Verify that all OSPF sessions are up (command outputs provided for all configured

routers).

root@VDC-edge-r01-re0>show ospf neighbor

Address Interface State ID Pri Dead192.168.1.2 ae0.0 Full 192.168.168.2 254 38192.168.26.3 irb.0 Full 192.168.168.3 128 39192.168.26.2 irb.0 Full 192.168.168.2 254 37

root@VDC-edge-r2-re0>show ospf neighbor


root@VDC-edge-fw01-n1>show ospf neighbor

Address Interface State ID Pri Dead192.168.25.2 reth0.0 Full 192.168.168.5 255 34192.168.25.1 reth0.0 Full 192.168.168.4 255 35192.168.26.2 reth1.0 Full 192.168.168.2 254 35192.168.26.1 reth1.0 Full 192.168.168.1 254 36

root@VDC-core-sw1-re0>show ospf neighbor

Address Interface State ID Pri Dead192.168.2.2 ae20.0 Full 192.168.168.5 255 33192.168.25.2 irb.10 Full 192.168.168.5 255 39192.168.25.3 irb.10 Full 192.168.168.3 128 39192.168.50.2 irb.50 Full 192.168.168.5 255 35192.168.50.3 irb.50 Full 192.168.168.6 128 37192.168.51.2 irb.51 Full 192.168.168.5 255 36192.168.51.3 irb.51 Full 192.168.168.6 128 35192.168.52.2 irb.52 Full 192.168.168.5 255 38192.168.52.3 irb.52 Full 192.168.168.6 128 31192.168.53.2 irb.53 Full 192.168.168.5 255 37192.168.53.3 irb.53 Full 192.168.168.6 128 33192.168.54.2 irb.54 Full 192.168.168.5 255 34192.168.54.3 irb.54 Full 192.168.168.7 128 36192.168.55.2 irb.55 Full 192.168.168.5 255 33192.168.55.3 irb.55 Full 192.168.168.7 128 35192.168.20.2 irb.20 Full 192.168.168.5 255 39192.168.20.3 irb.20 Full 192.168.168.20 128 39

root@VDC-core-sw2-re0>show ospf neighbor

Address Interface State ID Pri Dead192.168.2.1 ae20.0 Full 192.168.168.4 255 38192.168.25.1 irb.10 Full 192.168.168.4 255 32192.168.25.3 irb.10 Full 192.168.168.3 128 31192.168.50.3 irb.50 Full 192.168.168.6 128 38192.168.50.1 irb.50 Full 192.168.168.4 255 34192.168.51.1 irb.51 Full 192.168.168.4 255 39192.168.51.3 irb.51 Full 192.168.168.6 128 37192.168.52.1 irb.52 Full 192.168.168.4 255 35192.168.52.3 irb.52 Full 192.168.168.6 128 33192.168.53.1 irb.53 Full 192.168.168.4 255 39192.168.53.3 irb.53 Full 192.168.168.6 128 34192.168.54.1 irb.54 Full 192.168.168.4 255 35192.168.54.3 irb.54 Full 192.168.168.7 128 38



root@VDC-pod1-sw1>show ospf neighbor

Address Interface State ID Pri Dead192.168.50.2 vlan.50 Full 192.168.168.5 255 34192.168.50.1 vlan.50 Full 192.168.168.4 255 39192.168.51.2 vlan.51 Full 192.168.168.5 255 35192.168.51.1 vlan.51 Full 192.168.168.4 255 39192.168.52.2 vlan.52 Full 192.168.168.5 255 36192.168.52.1 vlan.52 Full 192.168.168.4 255 35192.168.53.2 vlan.53 Full 192.168.168.5 255 37192.168.53.1 vlan.53 Full 192.168.168.4 255 38192.168.55.1 irb.55 Full 192.168.168.4 255 32192.168.55.3 irb.55 Full 192.168.168.7 128 37192.168.20.1 irb.20 Full 192.168.168.4 255 34192.168.20.3 irb.20 Full 192.168.168.20 128 31

root@VDC-pod2-m00-me4>show ospf neighbor

Address Interface State ID Pri Dead192.168.54.2 irb.54 Full 192.168.168.5 255 35192.168.54.1 irb.54 Full 192.168.168.4 255 31192.168.55.2 irb.55 Full 192.168.168.5 255 33192.168.55.1 irb.55 Full 192.168.168.4 255 33

2. Verify OSPF conditional default route advertisement.

root@VDC-core-sw1-re0>show route 0/0


0.0.0.0/0 *[OSPF/150] 15:58:16, metric 1025, tag 0 > to 192.168.25.3 via irb.10

root@VDC-core-sw2-re0>show route protocol ospf 0/0


0.0.0.0/0 *[OSPF/150] 15:55:51, metric 1025, tag 0 > to 192.168.25.3 via irb.10

root@VDC-pod1-sw1>show route protocol ospf 0/0

inet.0: 24 destinations, 24 routes (24 active, 0 holddown, 0 hidden)Restart Complete+ = Active Route, - = Last Active, * = Both

0.0.0.0/0 *[OSPF/10] 15:57:29, metric 1100 > to 192.168.50.1 via vlan.50 to 192.168.50.2 via vlan.50 to 192.168.51.1 via vlan.51 to 192.168.51.2 via vlan.51 to 192.168.52.1 via vlan.52 to 192.168.52.2 via vlan.52 to 192.168.53.1 via vlan.53 to 192.168.53.2 via vlan.53

root@VDC-pod2-m00-me4>show route protocol ospf 0/0




0.0.0.0/0 *[OSPF/10] 15:57:30, metric 1100 to 192.168.54.1 via irb.54 > to 192.168.54.2 via irb.54 to 192.168.55.1 via irb.55 to 192.168.55.2 via irb.55

3. Verify OSPF LFA routes.

root@VDC-edge-r01-re0>show ospf backup coverage

Topology default coverage:

Node Coverage:

Area Covered Total Percent Nodes Nodes Covered0.0.0.1 2 4 50.00%

Route Coverage:

Path Type Covered Total Percent Routes Routes CoveredIntra 4 7 57.14%Inter 0 32 0.00%Ext1 1 1 100.00%Ext2 1 3 33.33%All 6 43 13.95%

root@VDC-edge-r2-re0%cli

{master}

root@VDC-edge-r2-re0>show ospf backup coverage


Node Coverage:


Route Coverage:


root@VDC-core-sw1-re0>show ospf backup coverage


Node Coverage:

Area Covered Total Percent Nodes Nodes Covered0.0.0.0 0 1 0.00%0.0.0.1 0 4 0.00%0.0.0.10 2 2 100.00%



0.0.0.11 2 2 100.00%0.0.0.2 0 2 0.00%

Route Coverage:


root@VDC-core-sw2-re0>show ospf backup coverage


Node Coverage:

Area Covered Total Percent Nodes Nodes Covered0.0.0.0 0 1 0.00%0.0.0.1 0 4 0.00%0.0.0.10 2 2 100.00%0.0.0.11 2 2 100.00%0.0.0.2 0 2 0.00%

Route Coverage:


root@VDC-pod2-m00-me4>show ospf backup coverage


Node Coverage:


Route Coverage:




CHAPTER 5

High Availability

• High availability overview

• Hardware redundancy

• Software redundancy

High Availability Overview

The MetaFabric 1.0 solution is designed with both hardware and software redundancy

throughout the data center.

Hardware Redundancy

The following hardware redundancy options are configured in the VDC 1.0 solution:

• Node-level physical redundancy, featuringedge routers, redundant core switches, POD

switches, and an SRX firewall cluster

• Redundant FRUs ( power supply, fans)

• Redundant Routing Engine on edge routers, core-switches, POD switches

• Redundant switch fabric on edge router and core switches

Software Redundancy

The following software redundancy featuresare configured in theMetaFabric 1.0 solution:

QFabric-M Configuration

The QFabric-M features the following software redundancy configurations:

• Link/node-level redundancy usingmultichassis LAGsonedge router and core switches

• Redundant server node groups (RSNG) on POD1 and POD 2 (QFX3000-MQFabric

system). This is configured on the PODs using the following configuration commands:

set fabric resources node-group RSNG2 node-device n2set fabric resources node-group RSNG2 node-device n3

• OSPF LFA feature to enable backup next hop during failure events


• QFX3000-MQFabric system built-in architecture supports hardware and software

redundancy

• Non-step software upgrade (NSSU) is supported on theQFX3000-MQFabric system

• Protocol graceful restart is configured using the following command:

set groups global routing-options graceful-restart

NOTE: WhenNSR is enabled, graceful protocol restart is not supported.NSRis not currently supported on the QFX3000-MQFabric system.

Configurint the Core and Edge Router

The core switches (EX9200) and edge routers (MX240) feature software redundancy

configured as shown here:

• Graceful Routing Engine switchover (GRES) on Routing Engine hardware failure. This

is configured on MX Series platforms using the following commands:

set groups global chassis redundancy routing-engine 0 masterset groups global chassis redundancy routing-engine 1 backupset groups global chassis redundancy failover on-loss-of-keepalivesset groups global chassis redundancy graceful-switchover

• Nonstop software upgrade (NSSU) is supported on the QFX3000-MQFabric system

and MX240

• In-service software upgrade (ISSU) is supported (EX9200)

NOTE: ISSU is supported only with the presence of 1-Gbps line cardsavailable in the chassis (EX9200).

• Nonstop active routing (NSR) is supported. This is configured using the following

command:

set groups global routing-options nonstop-routing

NOTE: When NSR is enabled, graceful protocol restart is not supported.NSR is not supported on the QFX3000-MQFabric system.

• Nonstop bridging (NSB) is enabled using the following command:

set protocols layer2-control nonstop-bridging



NOTE: Nonstopbridgingoperatesbysynchronizingallprotocol informationfor NSB-supported Layer 2 protocols between themaster and backupRouting Engines. If the switch has a Routing Engine switchover, theNSB-supported Layer 2 protocol sessions remain active because they arealready synchronized on the backup Routing Engine. The Routing Engineswitchover is transparent to neighbor devices, which do not detect anychanges related to the Layer 2 protocol sessions on the switch.

• Graceful protocol restart is also supported at the core and edge. Configuration of this

feature is performed using this command:

set groups global routing-options graceful-restart

Configuring the Perimeter Firewall

Theedge firewalls (SRX3600) feature the following software redundancyconfigurations:

• Edge firewall (SRX3600)chassis cluster configuration is performedusing the following

commands:

set groups global protocols layer2-control nonstop-bridgingset chassis cluster reth-count 4set chassis cluster redundancy-group 0 node 0 priority 129set chassis cluster redundancy-group 0 node 1 priority 128set chassis cluster redundancy-group 1 node 0 priority 129set chassis cluster redundancy-group 1 node 1 priority 128

• Fabric links between the SRX chassis are configured using the following commands:

set interfaces fab0 fabric-options member-interfaces ge-5/0/15set interfaces fab1 fabric-options member-interfaces ge-18/0/15

Verification

The following verification commands (with sample output) can be used to confirm the

configuration and function of high availability features.

Results

1. Verify that all the protocols sessions are up in the backup Routing Engine. This

command output verifies that NSR is configured properly in the EX9200:

root@VDC-edge-r01-re0> show ospf neighbor


root@VDC-edge-r2-re0> show ospf neighbor


root@VDC-edge-fw01-n1> show ospf neighbor


Chapter 5: High Availability

Address Interface State ID Pri Dead192.168.25.2 reth0.0 Full 192.168.168.5 255 34192.168.25.1 reth0.0 Full 192.168.168.4 255 35192.168.26.2 reth1.0 Full 192.168.168.2 254 35192.168.26.1 reth1.0 Full 192.168.168.1 254 36

root@VDC-core-sw1-re0> show ospf neighbor

Address Interface State ID Pri Dead192.168.2.2 ae20.0 Full 192.168.168.5 255 33192.168.25.2 irb.10 Full 192.168.168.5 255 39192.168.25.3 irb.10 Full 192.168.168.3 128 39192.168.50.2 irb.50 Full 192.168.168.5 255 35192.168.50.3 irb.50 Full 192.168.168.6 128 37192.168.51.2 irb.51 Full 192.168.168.5 255 36192.168.51.3 irb.51 Full 192.168.168.6 128 35192.168.52.2 irb.52 Full 192.168.168.5 255 38192.168.52.3 irb.52 Full 192.168.168.6 128 31192.168.53.2 irb.53 Full 192.168.168.5 255 37192.168.53.3 irb.53 Full 192.168.168.6 128 33192.168.54.2 irb.54 Full 192.168.168.5 255 34192.168.54.3 irb.54 Full 192.168.168.7 128 36192.168.55.2 irb.55 Full 192.168.168.5 255 33192.168.55.3 irb.55 Full 192.168.168.7 128 35192.168.20.2 irb.20 Full 192.168.168.5 255 39192.168.20.3 irb.20 Full 192.168.168.20 128 39

root@VDC-core-sw2-re0> show ospf neighbor

Address Interface State ID Pri Dead192.168.2.1 ae20.0 Full 192.168.168.4 255 38192.168.25.1 irb.10 Full 192.168.168.4 255 32192.168.25.3 irb.10 Full 192.168.168.3 128 31192.168.50.3 irb.50 Full 192.168.168.6 128 38192.168.50.1 irb.50 Full 192.168.168.4 255 34192.168.51.1 irb.51 Full 192.168.168.4 255 39192.168.51.3 irb.51 Full 192.168.168.6 128 37192.168.52.1 irb.52 Full 192.168.168.4 255 35192.168.52.3 irb.52 Full 192.168.168.6 128 33192.168.53.1 irb.53 Full 192.168.168.4 255 39192.168.53.3 irb.53 Full 192.168.168.6 128 34192.168.54.1 irb.54 Full 192.168.168.4 255 35192.168.54.3 irb.54 Full 192.168.168.7 128 38

root@VDC-pod1-sw1> show ospf neighbor

Address Interface State ID Pri Dead192.168.50.2 vlan.50 Full 192.168.168.5 255 34192.168.50.1 vlan.50 Full 192.168.168.4 255 39192.168.51.2 vlan.51 Full 192.168.168.5 255 35192.168.51.1 vlan.51 Full 192.168.168.4 255 39192.168.52.2 vlan.52 Full 192.168.168.5 255 36192.168.52.1 vlan.52 Full 192.168.168.4 255 35192.168.53.2 vlan.53 Full 192.168.168.5 255 37192.168.53.1 vlan.53 Full 192.168.168.4 255 38192.168.55.1 irb.55 Full 192.168.168.4 255 32192.168.55.3 irb.55 Full 192.168.168.7 128 37192.168.20.1 irb.20 Full 192.168.168.4 255 34192.168.20.3 irb.20 Full 192.168.168.20 128 31



2. 2. Verify thatNSR is configured properly. This is done by confirming that all OSPF

sessions are in a “Full” state in the backup Routing Engine. The command belowwas

run on the MX240:

root@vdc-edge-r2-re1> show ospf neighbor


3. 3. Verify that GRES is configured properly. This is done by confirming that the backup

Routing Engine is ready for switchover. The command belowwas run on the MX240:

root@vdc-edge-r2-re1> show system switchover

Graceful switchover: OnConfiguration database: ReadyKernel database: SynchronizingPeer state: Steady State


Chapter 5: High Availability

CHAPTER 6

Class-of-Service Configuration

The following section covers class-of-service (CoS) configuration in the MetaFabric 1.0

solution. The CoS configuration covers the following areas:

• Class-of-service overview

• Class- of-service configuration in the PODs

• Class- of-service configuration

• Data Center Bridging (DCB) and lossless Ethernet configuration

• Verification

• Class-of-Service Overview on page 129

• Configuring Class-of-Service on page 130

• Configuring Data Center Bridging and Lossless Ethernet on page 130

Class-of-Service Overview

Requirements

End-to-end class of service is required in a data center environment to ensure a high

quality user experience for users of business-critical applications. Themost important

or high priority applications and services should always have priority over other traffic

types. The other type of class of service required is configuration and support for DCB

and the provisioning of Lossless Ethernet to enable communication between storage

arrays as a high priority, lossless mediumwithout causing blocking or interruption of

non-storage traffic.

End-to-end traffic and CoS in this solution have been classified into the following five

forwarding classes (sorted from highest priority to lowest priority, top to bottom):

• Network control traffic

• Virtualization control

• Storage (Llossless)

• Business applications

• Best effort (everything else)


The solution queues (buffers and transmit rates) are shown in Table 18 on page 130.

Table 18: MetaFabric 1.0 Class-of-Service Queues

QueueLoss PriorityTransmit RateSchedulers

111High5%Network-control

110High5%VM_Control

001High25%Business_Applications

100Low (PFC enabled)60%no-loss

000High5%Best-effort.

Configuring Class-of-Service

This section covers the configuration of the following solution components:

• Class of service in the Points of Delivery (PODs).

• DCB in the PODs

• Lossless Ethernet configuration

Configuring Class-of-Service (POD Level)


integration between the edge, perimeter, and the data center core; and then continues

with configuration of the access and aggregation roles in the data center (in this solution,

those roles are collapsed into theQFabric POD). Finally, the networkmust be configured

in the virtual switching role.

Configuring Data Center Bridging and Lossless Ethernet

This section covers the configuration of the following solution components:

• Class of service in the PODs

• Data center bridging (DCB¬) in the PODs

• Lossless Ethernet configuration

Configuring Class-of-Service (POD Level)


integration between the edge, perimeter, and the data center core; and then continues

with configuration of the access and aggregation roles in the data center (these roles

are collapsed into the QFabric POD in the MetaFabric 1.0 solution). Finally, the network

must be configured in the virtual switching role.



Topology

Figure 49 on page 131 shows the topology of the data center PODs and the connections

to the compute and storage farms in the testing lab. This MetaFabric 1.0 solution uses

iSCSI, and Network File System (NFS) as a storage protocol. EMC VNX is configured as

storage array. All the VM hard drives are mounted on the EMC storage using the iSCSI

transport. The NFS partition is used for file storage. Storage traffic requires lossless

transport end-to-end when traversing the Ethernet network. Incoming traffic destined

for thedatacenter applications is classifiedat theedge router toQueue 111 and forwarded

to the VDC network through the vdc-perimeter firewall.

Figure 49: The VDC POD and Compute/Storage Topology

Lossless Ethernet and DCB are configured in the following section.

To configure Lossless Ethernet and DCB, follow these steps:

1. ConfigureDCBXandLLDPonthevdc-pod1-sw1andvdc-pod2-sw1 for theserver-facing

and storage-facing interfaces. DCBX and LLDP are needed to exchange the peer

lossless Ethernet capabilities. The important parameters are:

1. Lossless-queue number

2. Willing or non-willing (Juniper Networks switches always operate in non-willing

mode),which implies that the switch configuration is givenprecedence for lossless

behavior

[edit]set protocols dcbx interface allset protocols lldp interface all

2. Configure forwarding class mapping for all queues.

[edit]set class-of-service forwarding-classes class BestEffort queue-num0set class-of-service forwarding-classes class Business_Applications queue-num 1


Chapter 6: Class-of-Service Configuration

set class-of-service forwarding-classes class no-loss queue-num 4set class-of-service forwarding-classes class no-loss no-lossset class-of-service forwarding-classes class VM_Control queue-num 6set class-of-service forwarding-classes class Network_Control queue-num 7

3. Configure 802.1p classifiers for the iSCSi traffic and assign code-points as 4.

[edit]set class-of-service classifiers ieee-802.1 802.1-classifier forwarding-class BestEffortloss-priority low code-points 000

set class-of-service classifiers ieee-802.1 802.1-classifier forwarding-classBusiness_Applications loss-priority low code-points 001

set class-of-service classifiers ieee-802.1 802.1-classifier forwarding-class VM_Controlloss-priority low code-points 110

set class-of-service classifiers ieee-802.1 802.1-classifier forwarding-class no-lossloss-priority low code-points 100

set class-of-service classifiers ieee-802.1 802.1-classifier forwarding-classNetwork_Control loss-priority low code-points 111

set class-of-service host-outbound-traffic forwarding-class Network_Control

4. Configure forwarding class-sets for iSCSI and Ethernet traffic (representing different

forwarding classes).

[edit]set class-of-service forwarding-class-sets no-loss class no-lossset class-of-service forwarding-class-sets VDC-Lan class BestEffortset class-of-service forwarding-class-sets VDC-Lan class Business_Applicationsset class-of-service forwarding-class-sets VDC-Lan class Network_Controlset class-of-service forwarding-class-sets VDC-Lan class VM_Control

5. Configure thecongestionnotificationprofile toenablePFCforqueue3(no-lossqueue).

This configuration ismandatory, enabling lossless behavior on queue 3. This enforces

priority-flow-control for the queue.

[edit]set class-of-service congestion-notification-profile cnp input ieee-802.1 code-point100 pfc

6. 6. Configure the transmit rate and priority for each scheduler to allow for bandwidth

sharing.

[edit] set class-of-service schedulers BestEffort transmit-rate percent 5set class-of-service schedulers BestEffort priority lowset class-of-service schedulers Business_Applications transmit-rate percent 25set class-of-service schedulers Business_Applications priority lowset class-of-service schedulers VM_Control transmit-rate percent 5set class-of-service schedulers VM_Control priority lowset class-of-service schedulers no-loss transmit-rate percent 60set class-of-service schedulers no-loss priority lowset class-of-service schedulers Network_Control transmit-rate percent 5set class-of-service schedulers Network_Control priority low

7. Configure scheulder-maps for iSCSI. This configuration binds the scheduler to the

forwarding class sets.

[edit]set class-of-service scheduler-maps VDC-Lan forwarding-class BestEffort schedulerBestEffort

set class-of-servicescheduler-mapsVDC-Lan forwarding-classBusiness_Applications



scheduler Business_Applicationsset class-of-service scheduler-maps VDC-Lan forwarding-class VM_Control schedulerVM_Control

set class-of-service scheduler-maps VDC-Lan forwarding-class Network_Controlscheduler Network_Control

setclass-of-servicescheduler-mapsno-loss forwarding-classno-lossschedulerno-loss

8. Configure traffic control profiles. These profiles bind the scheduler maps.

[edit]set class-of-service traffic-control-profiles no-loss scheduler-map no-lossset class-of-service traffic-control-profiles no-loss guaranteed-rate percent 60set class-of-service traffic-control-profiles VDC-Lan scheduler-map VDC-Lanset class-of-service traffic-control-profiles VDC-Lan guaranteed-rate percent 40

9. 9. Apply the classifier, forwarding class set, and CNP profile to the server-facing and

storage-facing interfaces. In the verification lab, these ports are all facing

TrafficGenerator test equipment.

a. Configure the server-connected ports.

[edit]set class-of-service interfaces RSNG2:ae0 forwarding-class-set no-lossoutput-traffic-control-profile no-loss

set class-of-service interfaces RSNG2:ae0 forwarding-class-set VDC-Lanoutput-traffic-control-profile VDC-Lan

set class-of-service interfaces RSNG2:ae0 congestion-notification-profile cnpsetclass-of-service interfacesRSNG2:ae0unit*classifiers ieee-802.1802.1-classifierset class-of-service interfaces RSNG2:ae0 unit * rewrite-rules ieee-802.1802.1-rewrite

b. Configure the storage-connected ports.

[edit]set class-of-service interfaces n1:xe-0/0/14 forwarding-class-set VDC-Lanoutput-traffic-control-profile VDC-Lan

set class-of-service interfaces n1:xe-0/0/14 forwarding-class-set no-lossoutput-traffic-control-profile no-loss

set class-of-service interfaces n1:xe-0/0/14 congestion-notification-profile cnpset class-of-service interfaces n1:xe-0/0/14 unit * classifiers ieee-802.1802.1-classifier

set class-of-service interfaces n1:xe-0/0/15 forwarding-class-set VDC-Lanoutput-traffic-control-profile VDC-Lan



set class-of-service interfaces n1:xe-0/0/16 forwarding-class-set VDC-Lanoutput-traffic-control-profile VDC-Lan



set class-of-service interfaces n1:xe-0/0/16 unit * rewrite-rules ieee-802.1802.1-rewrite



10. Copy the class-of-service configuration to all interfaces.

[edit]copy class-of-service interfaces RSNG2:ae0 to RSNG2:ae1copy class-of-service interfaces RSNG2:ae0 to RSNG2:ae2copy class-of-service interfaces RSNG2:ae0 to RSNG2:ae4copy class-of-service interfaces RSNG2:ae0 to RSNG3:ae3copy class-of-service interfaces RSNG2:ae0 to RSNG3:ae4copy class-of-service interfaces RSNG2:ae0 to RSNG3:ae5copy class-of-service interfaces RSNG2:ae0 to RSNG4:ae0copy class-of-service interfaces RSNG2:ae0 to n4:xe-0/0/39copy class-of-service interfaces RSNG2:ae0 to n5:xe-0/0/39

Verification


the DCB and lossless Ethernet configurations are operating as expected.

Results

1. Verify VLANmembership.

root@vdc-pod2-sw1> show vlans PFC

Name Tag InterfacesPFC 308 n1:xe-0/0/14.0*, n1:xe-0/0/15.0*, n1:xe-0/0/16.0*, n1:xe-0/0/18.0*

2. Verify that PFC frames are sent during periods of congestion.

root@vdc-pod2-sw1> show interfaces n1:xe-0/0/14 extensive |find “MAC Priority”

MAC Priority Flow Control Statistics: Priority : 0 0 0 Priority : 1 0 0 Priority : 2 0 0 Priority : 3 0 0 Priority : 4 0 116448 Priority : 5 0 0 Priority : 6 0 0 Priority : 7 0 0 Packet Forwarding Engine configuration: Destination slot: 0Logical interface n1:xe-0/0/14.0 (Index 103) (SNMP ifIndex 1215824492) (Generation 172) Flags: SNMP-Traps 0x0 Encapsulation: ENET2 Traffic statistics: Input bytes : 0 Output bytes : 5405 Input packets: 0 Output packets: 23 Local statistics: Input bytes : 0 Output bytes : 5405 Input packets: 0 Output packets: 23 Transit statistics: Input bytes : 0 0 bps Output bytes : 0 0 bps



Input packets: 0 0 pps Output packets: 0 0 pps Protocol eth-switch, MTU: 9192, Generation: 200, Route table: 0 Flags: Trunk-Mode

3. Verify that packets in other queues are dropped during congestion. Also verify that

queue 4 (lossless queue) is showing no drops.

root@pod2-sw1> show interfaces queue n1:xe-0/0/16

Physical interface: xe-0/0/16, Enabled, Physical link is Up Interface index: 49199, SNMP ifIndex: 1215824460Forwarding classes: 16 supported, 7 in useEgress queues: 12 supported, 7 in use

Queue: 0, Forwarding classes: BestEffort

Queued: Packets : 0 0 pps Bytes : 0 0 bps Transmitted: Packets : 17848847 422311 pps Bytes : 2284652416 432446792 bps Tail-dropped packets : Not Available Total-dropped packets: 71401120 1689149 pps Total-dropped bytes : 9139343360 1729689096 bps

Queue: 1, Forwarding classes: Business_Applications


Queue: 3, Forwarding classes: fcoe


Queue: 4, Forwarding classes: no-loss




Queue: 6, Forwarding classes: VM_Control


Queue: 7, Forwarding classes: Network_Control


Queue: 8, Forwarding classes: mcastQueued: Packets : 0 0 pps Bytes : 0 0 bps Transmitted: Packets : 0 0 pps Bytes : 0 0 bps Tail-dropped packets : Not Available Total-dropped packets: 0 0 pps Total-dropped bytes : 0 0 bps



CHAPTER 7

Security Configuration

This section covers the configuration of security in the MetaFabric 1.0 solution. The

operational roles covered by this section are:

• Perimeter security

• Configure chassis clustering

• Control and data fabric configuratio

• Node-specific configuration

• Security configuration (zones, address books, policies)

• Network Address Translation (NAT)

• Intrusion detection and prevention (IDP)

• • Host security

• Application security (Virtual Gateway Appliance)

• Perimeter Security on page 137

• Host Security on page 155

Perimeter Security



• Configuring Network Address Translation on page 144


• Configuring Intrusion Detection and Prevention on page 148


Overview

The Juniper Networks SRX3600 is deployed in this solution as the edge firewall and

provides perimeter security for the virtualized data center network residing between the

edge router andcore switch. TheSRX3600 is configured in active/backupchassis cluster

mode.Active/backuphighavailability is themost commontypeofhighavailability firewall


deployment and consists of two firewall members of a cluster, one of which actively

provides routing, firewall, NAT, VPN, and security services, alongwithmaintaining control

of the chassis cluster. In case of chassis cluster failover, the backup firewall will become

the active firewall and the active firewall will become the backup.

Configuring Chassis Clustering

Configuring chassis cluster requires a minimum of two devices. Here, we are using

SRX3600, which has similar hardware and Junos OS software version. Since we only

have a single cluster, the solution uses only cluster-id 1 SRX 3600-1 acting as node 0.

SRX 3600-2 is configured as node 1. These commands are the only commands where it

matters which chassis member you apply them to because the setting is stored in the

NVRAM rather than in the configuration itself. The command will also cause the cluster

member to reboot, which is common to all current versions of Junos OS.

To configure chassis clustering, youmust first configure the cluster-id and node ID on

each cluster member as shown in the following steps:

1. Configure SRX 3600-1.

set chassis cluster cluster-id 1 node 0 reboot


set chassis cluster cluster-id 1 node 1 reboot

NOTE: This set of commandsmust be run as an operational commandand not in configurationmode.

Control port configuration: Once the chassis members have rebooted, the SRX3600

uses two designated, labeled ports as control ports.

Configure Chassis Clustering Data Fabric

Once the SRX3600s are configured as a chassis cluster and control ports have been

assigned, the fabric (data) ports of the cluster must be configured. These fabric ports

are used to pass real-time objects (RTOs) in Active/Passive mode. RTOs are messages

that the nodes use to synchronize information between chassis members of a chassis

cluster.

To configure the data fabric, youmust configure two fabric interfaces (one on each

chassis) as shown in the following steps. These interfaces are connected to each other

to form the fabric link.


set interfaces fab0 fabric-optionsmember-interfaces ge-5/0/15


set interfaces fab1 fabric-optionsmember-interfaces ge-18/0/15



Configuring Chassis Clustering Groups

Since the SRX cluster configuration is held within a single common configuration, we

need away to assign some elements of the configuration to a specificmember only. This

is done in Junos OSwith the node-specific configurationmethod called groups. The last

command uses the node variable to define how the groups are applied to the nodes

(each node will recognize its number and accept the configuration accordingly).We also

configure out-of-bandmanagement on the fxp0 interface of the SRXwith separate IP

addresses for the individual control planes of the cluster. Node-specific configuration is

covered in the next configuration example.

To configure chassis clustering groups, including the host name, backup-router, and

interface addressing, follow these steps:


set groups node0 system host-name vdc-edge-fw01-n0set groups node0 system backup-router 10.94.47.62set groups node0 interfaces fxp0 unit 0 family inet address 10.94.47.33/27


set groups node1 system host-name vdc-edge-fw01-n1set groups node1 system backup-router 10.94.47.62set groups node1 interfaces fxp0 unit 0 family inet address 10.94.47.34/27

3. Configure apply groups

set groups nodeset apply-groups "${node}"

Configuring Chassis Clustering Redundancy Groups

The next step in configuring chassis clustering is to configure redundancy groups.

RedundancyGroup0 is always for the control plane,while redundancy group 1+ is always

for the data plane ports. Because active/backupmode allows only one chassis member

tobeactiveat a time,weonlydefineRedundancyGroups0and 1.Wemust also configure

howmany redundant Ethernet groups we will have active on the device (so that the

system can allocate the appropriate resources for it). This is similar to aggregated

Ethernet.

We will also need to define which device has priority (in JSRP, high priority is preferred)

for the control plane, as well as which device is preferred to be active for the data plane.

Remember that the control planecanbeactiveonadifferent chassis than thedataplane

in active/passive (there isn’t anything wrong with this from a technical standpoint, but

many administrators prefer having both the control plane and data plane active on the

same chassis member).

To configure redundancy groups and priority, see the following example:

1. Configure redundancy groups and priority.

set chassis cluster reth-count 2set chassis cluster redundancy-group 0 node 0 priority 129set chassis cluster redundancy-group 0 node 1 priority 128


Chapter 7: Security Configuration

set chassis cluster redundancy-group 1 node 0 priority 129set chassis cluster redundancy-group 1 node 1 priority 128

Configuring Chassis Clustering Data Interfaces

The next step in chassis cluster configuration is to define the actual data interfaces on

the platform so that in the event of a data plane failover, the other chassis member will

be able to take over the connection seamlessly. This configuration involves defining the

membership information of themember interfaces to theRETH interface, definingwhich

redundancygroup theRETH interfacewill beamemberof (inActive/Passive itwill always

be 1,) and finally defining the RETH interface information, such as the IP address of the

interface.

NOTE: Redundant Ethernet interface LAGs are configured toward the edgefirewall and core switch.

To configure redundant data interfaces on the chassis cluster, follow these steps:

1. Configure redundant Ethernet LAG interface reth0 toward core switches used as the

trust interface.

[edit] set interfaces reth0 description "Trust Zone toward POD"set interfaces reth0 vlan-taggingset interfaces reth0 redundant-ether-options redundancy-group 1set interfaces reth0 redundant-ether-optionsminimum-links 1set interfaces reth0 redundant-ether-options lacp activeset interfaces reth0 redundant-ether-options lacp periodic fastset interfaces reth0 unit 0 vlan-id 10set interfaces reth0 unit 0 family inet address 192.168.25.3/24

2. Configure member links for the reth0 from node 0. (Once the chassis cluster is

configured, everything canbeconfigured fromtheprimarynodeas thecluster behaves

as a single, physical chassis.)

[edit]set interfaces xe-3/0/0 gigether-options redundant-parent reth0set interfaces xe-3/0/1 gigether-options redundant-parent reth0set interfaces xe-4/0/0 gigether-options redundant-parent reth0set interfaces xe-4/0/1 gigether-options redundant-parent reth0….

set interfaces xe-16/0/0 gigether-options redundant-parent reth0set interfaces xe-16/0/1 gigether-options redundant-parent reth0set interfaces xe-17/0/0 gigether-options redundant-parent reth0set interfaces xe-17/0/1 gigether-options redundant-parent reth0

3. Configure redundant Ethernet LAG interface reth1 toward edge routers used as the

untrust interface.

[edit]set interfaces reth1 description "Untrust Zone toward Edge-routers"set interfaces reth1 vlan-taggingset interfaces reth1 redundant-ether-options redundancy-group 1set interfaces reth1 redundant-ether-optionsminimum-links 1set interfaces reth1 redundant-ether-options lacp active



set interfaces reth1 redundant-ether-options lacp periodic fastset interfaces reth1 unit 0 vlan-id 11set interfaces reth1 unit 0 family inet address 192.168.26.3/24set interfaces reth1 unit 0 family inet address 10.94.127.30/27

4. Configure redundant member links for reth0 (can be done from node0).

[edit]set interfaces xe-1/0/0 gigether-options redundant-parent reth1set interfaces xe-1/0/1 gigether-options redundant-parent reth1set interfaces xe-2/0/0 gigether-options redundant-parent reth1set interfaces xe-2/0/1 gigether-options redundant-parent reth1….

set interfaces xe-14/0/0 gigether-options redundant-parent reth1set interfaces xe-14/0/1 gigether-options redundant-parent reth1set interfaces xe-15/0/0 gigether-options redundant-parent reth1set interfaces xe-15/0/1 gigether-options redundant-parent reth1

Configuring Chassis Clustering – Security Zones and Security Policy

Oncechassis clustering is completelyconfigured, theRETH interfacesshouldbeassigned

to the appropriate security zones and virtual routers. Going forward, the configuration

will referenceRETH interfaces instead of individualmember interfaces. For this example,

we are simply going to leave theRETH0andRETH1 interfaces in the default virtual router

inet.0, which does not require any additional configuration. Additional configuration of

the perimeter security includes the following steps:

• Zone and address book configuration

• Security policy configuration

To configure security on the SRX chassis cluster, follow these steps:

1. Configure security zones and address books.

[edit]set security zones functional-zonemanagement host-inbound-traffic system-servicesssh

set security zones functional-zonemanagement host-inbound-traffic system-serviceshttps

set security zones functional-zonemanagement host-inbound-traffic protocols allset security zones security-zone untrust address-book address TVM-Client-Subnet10.10.0.0/16

setsecurityzonessecurity-zoneuntrustaddress-bookaddressTrafficGenerator-External10.40.0.0/16

set security zones security-zone untrust host-inbound-traffic protocols allset security zones security-zone untrust interfaces reth1.0 host-inbound-trafficsystem-services all

set security zonessecurity-zoneuntrust interfaces reth1.0host-inbound-trafficprotocolsall

set security zones security-zone trust address-bookaddressSA-server1 10.94.63.24/32set security zonessecurity-zone trustaddress-bookaddressSP-Server 10.94.127.180/32set security zones security-zone trust address-book address Exchange-Server10.94.127.181/32



set security zones security-zone trust address-book address MediaWiki-Server10.94.127.182/32

set security zones security-zone trust address-book address TVM-Server-Subnet10.20.127.0/24

set security zones security-zone trust address-book addressTrafficGenerator-Internal-502 10.30.0.0/16





set security zones security-zone trust host-inbound-traffic protocols allset security zones security-zone trust interfaces reth0.0 host-inbound-trafficsystem-services all

set security zones security-zone trust interfaces reth0.0 host-inbound-traffic protocolsall

2. Configure outbound security policy for traffic sourcing from the trust zone (reth0) to

the untrust zone (reth1).

[edit]set security policies from-zone trust to-zone untrust policy Internet-accessmatchsource-address any

set security policies from-zone trust to-zone untrust policy Internet-accessmatchdestination-address any

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication junos-http

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication junos-https

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication junos-http-ext

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication junos-ntp

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication junos-dns-udp

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication ICMP

set security policies from-zone trust to-zone untrust policy Internet-access thenpermit

3. Configure inbound security policies for traffic sourcing from the untrust zone (reth1)

to the trust zone (reth0).

[edit]set security policies from-zone untrust to-zone trust policy remote-accessmatchsource-address any

set security policies from-zone untrust to-zone trust policy remote-accessmatchdestination-address SA-server1

set security policies from-zone untrust to-zone trust policy remote-accessmatchapplication junos-https

set security policies from-zone untrust to-zone trust policy remote-access then permit



set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchsource-address any

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchdestination-address Exchange-Server

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication junos-imap

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication junos-pop3

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication junos-ms-rpc-msexchange

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication junos-http

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication junos-https

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication junos-http-ext

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication junos-ms-rpc-msexchange-directory-nsp

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication junos-ms-rpc-msexchange-directory-rfr

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication junos-ms-rpc-msexchange-info-store

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication Exchange

set security policies from-zone untrust to-zone trust policy Exchange-Accessmatchapplication junos-smtp

set security policies from-zone untrust to-zone trust policy Exchange-Access thenpermit

set security policies from-zone untrust to-zone trust policy MediaWiki-Accessmatchsource-address any

set security policies from-zone untrust to-zone trust policy MediaWiki-Accessmatchdestination-address MediaWiki-Server

set security policies from-zone untrust to-zone trust policy MediaWiki-Accessmatchapplication junos-http

set security policies from-zone untrust to-zone trust policy MediaWiki-Accessmatchapplication junos-https

set security policies from-zone untrust to-zone trust policy MediaWiki-Accessmatchapplication junos-http-ext

set security policies from-zone untrust to-zone trust policy MediaWiki-Access thenpermit

set security policies from-zone untrust to-zone trust policy SharePoint-Accessmatchsource-address any

set security policies from-zone untrust to-zone trust policy SharePoint-Accessmatchdestination-address SP-Server

set security policies from-zone untrust to-zone trust policy SharePoint-Accessmatchapplication junos-http

set security policies from-zone untrust to-zone trust policy SharePoint-Accessmatchapplication junos-https

set security policies from-zone untrust to-zone trust policy SharePoint-Accessmatchapplication junos-http-ext

set security policies from-zone untrust to-zone trust policy SharePoint-Accessmatchapplication SharePoint

set security policies from-zone untrust to-zone trust policy SharePoint-Access thenpermit

set security policies from-zone untrust to-zone trust policy ICMP-allowmatchsource-address any



set security policies from-zone untrust to-zone trust policy ICMP-allowmatchdestination-address any

setsecuritypolicies from-zoneuntrust to-zonetrustpolicy ICMP-allowmatchapplicationICMP

set security policies from-zone untrust to-zone trust policy ICMP-allow then permit

Verification


the SRX chassis cluster is configured properly.

Results

1. Verify chassis cluster configuration.

root@vdc-edge-fw01-n0> show chassis cluster status

Cluster ID: 1 Node Priority Status Preempt Manual failover

Redundancy group: 0 , Failover count: 1 node0 129 primary no no node1 128 secondary no no

Redundancy group: 1 , Failover count: 1 node0 129 primary no no node1 128 secondary no no

Configuring Network Address Translation

Network Address Translation (NAT) is a method for modifying or translating network

address information in packet headers. Either or both source and destination addresses

in a packet can be translated. NAT can include the translation of port numbers as well

as IP addresses. In this solution, we are using Source and Destination NAT.

Configure Source NAT

Source NAT is the translation of the source IP address of a packet leaving the Juniper

Networks device. Source NAT is used to allow hosts with private IP addresses to access

a public network. Here, we have defined translation of the original source IP address to

an IP address from a user-defined address pool with Port Address Translation. The

association between the original source IP address to the translated source IP address

is dynamic. The configuration uses a source network (172.16.0.0/16) which is translated

to the public pool address range (10.94.127.1 to 10.94.127.10). Proxy ARP is a required

element of the solution for the address range 10.94.127.1/32 to 10.94.127.11/32 on interface

reth1.0.ProxyARPallows the JuniperNetworks securitydevice to respond toARP requests

received on the interface for the translated addresses (rather than only responding to

ARP requests destined for the IP address of the firewall’s logical interfaces).

Source NAT configuration is outlined in the following configuration example.

To configure source NAT on the SRX chassis cluster, follow these steps:

1. Configure the source NAT pool.



[edit]set security nat source pool public-pool address 10.94.127.1/32 to 10.94.127.10/32

2. Configure the source NAT rule set.

[edit]set security nat source rule-set Internet-access from zone trustset security nat source rule-set Internet-access to zone untrustset security nat source rule-set Internet-access rule datacenter match source-address172.16.0.0/16

set security nat source rule-set Internet-access rule datacenter matchdestination-address 0.0.0.0/0

set security nat source rule-set Internet-access rule datacenter then source-nat poolpublic-pool

3. ConfigureproxyARPon theoutboundNAT interface (reth1, or untrust, in this example).

[edit]set security nat proxy-arp interface reth1.0 address 10.94.127.1/32 to 10.94.127.10/32

Configure Destination NAT

Destination NAT is the translation of the destination IP address of a packet entering the

Juniper Networks device. Destination NAT is used to redirect traffic destined to a virtual

host (identified by the original destination IP address) to the real host (identified by the

translated destination IP address). Destination NAT allows connections to be initiated

only for incoming network connections—for example, from the Internet to a private

network. The following configuration parameters were used in the below example:

1. Destination NAT pool dst-nat-SA-pool1 contains the IP address 10.94.63.24/32. This

device is a Juniper Networks SA Series SSL VPN Appliance (Remote Access Server).

This device can also be provisioned as a virtual appliance.

2. DestinationNAT rule setSAwith ruleSA-rule1 tomatchpackets received todestination

IPaddress 10.94.127.33/32. Formatchingpackets, thedestinationaddress is translated

to the address in the dst-nat-pool-1 pool.

3. Proxy ARP for the address 10.94.127.33/32 on interface reth1.0. This allows the Juniper

Networks security device to respond to ARP requests received on the interface for

that address.

4. Security policies to permit traffic from the untrust zone to the translated destination

IP address in the trust zone.

NOTE: When destination NAT is performed, the destination IP address istranslated according to configured destination NAT rules and then securitypolicies are applied.

To configure destination NAT on the SRX chassis cluster, follow these steps:

1. Configure the destination NAT pool.

[edit]set security nat destination pool dst-nat-SA-pool1 address 10.94.63.24/32



2. Configure the destination NAT rule set.

[edit]set security nat destination rule-set SA from zone untrustset security nat destination rule-set SA rule SA-rule1 match destination-address10.94.127.33/32

set security nat destination rule-set SA rule SA-rule1 then destination-nat pooldst-nat-SA-pool1

3. Configure the proxy ARP on the inbound NAT interface.

[edit] set security nat proxy-arp interface reth1.0 address 10.94.127.33/32

Verification


the NAT is configured properly.

Results

1. Verify the source NAT pool creation.

root@vdc-edge-fw01-n1> show security nat source pool all

node0:--------------------------------------------------------------------------Total pools: 1

Pool name : public-poolPool id : 4Routing instance : defaultHost address base : 0.0.0.0Port : [1024, 63487] port overloading : 1Total addresses : 10Translation hits : 0Address range Single Ports Twin Ports 10.94.127.1 - 10.94.127.10 12 0

node1:--------------------------------------------------------------------------Total pools: 1

Pool name : public-poolPool id : 4Routing instance : defaultHost address base : 0.0.0.0Port : [1024, 63487] port overloading : 1Total addresses : 10Translation hits : 25470Address range Single Ports Twin Ports 10.94.127.1 - 10.94.127.10 15 0

2. Verify the source NAT rule set configuration.

root@vdc-edge-fw01-n1> show security nat source

node0:--------------------------------------------------------------------------Total port number usage for port translation pool: 645120



Maximum port number for port translation pool: 268435456Total pools: 1Pool Address Routing PAT Total

Name Range Instance Addresspublic-pool 10.94.127.1-10.94.127.10 default yes 10

Total rules: 1Rule name Rule set From To Actiondatacenter Internet-access trust untrust public-pool

node1:--------------------------------------------------------------------------Total port number usage for port translation pool: 645120Maximum port number for port translation pool: 268435456Total pools: 1Pool Address Routing PAT Total

Name Range Instance Addresspublic-pool 10.94.127.1-10.94.127.10 default yes 10

Total rules: 1Rule name Rule set From To Actiondatacenter Internet-access trust untrust public-pool

3. Verify source NAT rules, match conditions, actions, and rule order .

root@vdc-edge-fw01-n1> show security nat source rule all

node0:--------------------------------------------------------------------------Total rules: 1Total referenced IPv4/IPv6 ip-prefixes: 2/0

source NAT rule: datacenter Rule-set: Internet-access Rule-Id : 1 Rule position : 1 From zone : trust To zone : untrust Match Source addresses : 172.16.0.0 - 172.16.255.255 Destination addresses : 0.0.0.0 - 255.255.255.255 Destination port : 0 - 0 Action : public-pool Persistent NAT type : N/A Persistent NAT mapping type : address-port-mapping Inactivity timeout : 0 Max session number : 0 Translation hits : 0

node1:--------------------------------------------------------------------------Total rules: 1 Total referenced IPv4/IPv6 ip-prefixes: 2/0source NAT rule: datacenter Rule-set: Internet-access Rule-Id : 1 Rule position : 1 From zone : trust To zone : untrust



Match Source addresses : 172.16.0.0 - 172.16.255.255 Destination addresses : 0.0.0.0 - 255.255.255.255 Destination port : 0 - 0 Action : public-pool Persistent NAT type : N/A Persistent NAT mapping type : address-port-mapping Inactivity timeout : 0 Max session number : 0 Translation hits : 25621

{primary:node1}

Configuring Intrusion Detection and Prevention

The Junos OS intrusion detection and prevention (IDP) policy enables you to selectively

enforce various attack detection and prevention techniques on network traffic passing

through an IDP-enabled device. It allows you to define policy rules to match a section of

traffic based on a zone, network, and application, and then take active or passive

preventive actions on that traffic. An IDP policy defines how your device handles the

network traffic. It allowsyou toenforcevariousattackdetectionandprevention techniques

on traffic traversing your network.

A policy is made up of rulebases, and each rulebase is comprised of a set of rules. You

define ruleparameters, suchas trafficmatchconditions, action, and logging requirements,

and then add the rules to rule bases. After you create an IDP policy by adding rules in one

ormore rulebases, you can select that policy to be the active policy on your device. Junos

OS allows you to configure multiple IDP policies, but a device can have only one active

IDP policy at a time. You can install the same IDP policy onmultiple devices, or you can

install a unique IDP policy on each device in your network. A single policy can contain

only one instance of any type of rulebase.

For transit traffic to pass through IDP inspection, you configure a security policy and

enable IDP application services on all traffic that you want to inspect. Security policies

contain rules defining the types of traffic permitted on the network and how the traffic

is treated inside the network. Enabling IDP in a security policy directs traffic thatmatches

the specified criteria to be checked against the IDP rulebases.

NOTE: Theactionset in thesecuritypolicyactionmustbepermit.Youcannotenable IDP for traffic that the device denies or rejects.



To install and configure IDP on the SRX chassis cluster, follow these steps:

Install an IDP license to enable IDP signature updates. In order to download and use the

predefined attack signatures in a policy, the IDP licensemust be installed. If you are using

only customsignatures, youdonotneedan IDP license.Onceyour license file is purchased

and available, install the license using the Junos OS terminal.

1. root@vdc-edge-fw01-n1> request system license add terminal

[Type ^D at a new line to end input, enter blank line between each license key]Serial No : AB0813AA0021Model : SRX3600Features : SRX3600-APPSEC-A-1 0Issue Date : 17-Dec-2013Expiration Date : 16-Dec-2014License Id : JUNOS466173License Key : JUNOS466173 aeaqea qmifbd aobrgn aucmbq giyqqb qcdw7l rqbea4 ujbpu2 q4esq2 sucbpr wrroiw w5kgvv 35oxsq ne4ynp ljbecm c5ug52 3s6cbj ldpuqj xny

2. Once you install the license, check for feature “idp-sig”.

root@vdc-edge-fw01-n0> show system license

License usage: Licenses Licenses Licenses Expiry Feature name used installed needed idp-sig 1 1 0 2014-12-15 16:00:00 PST appid-sig 0 2 0 2014-12-15 16:00:00 PST logical-system 1 1 0 permanentLicenses installed: License identifier: JUNOS466166 License version: 2 Valid for device: AB0813AA0014 Features: idp-sig - IDP Signature date-based, 2013-12-16 16:00:00 PST - 2014-12-15 16:00:00 PST

License identifier: JUNOS466167 License version: 2 Valid for device: AB0813AA0014 Features: appid-sig - APPID Signature date-based, 2013-12-16 16:00:00 PST - 2014-12-15 16:00:00 PST License identifier: JUNOS466168 License version: 2 Valid for device: AB0813AA0014 Features: appid-sig - APPID Signature date-based, 2013-12-16 16:00:00 PST - 2014-12-15 16:00:00 PST



NOTE: If configuring a firewall cluster, a firewall clustering license isrequired on both nodes of the cluster. The license is device specific.

3. Download and install the signature database. After the IDP license is installed, the

iDP signature database can be downloaded and installed by performing the following

steps:

• Makesure thedevicehas thenecessary configuration for connectivity to the Internet.

A name server must be configured.

• Configure the signature database URL.

set security idp security-package url https://services.netscreen.com/cgi-bin/index.cgi

4. Verify the version of the signature database in the Signature DB server. Look for

“successfully retrieved”. In this example, the version in the server is 2327.

root@vdc-edge-fw01-n0> request security idp security-package download check-server

node0:--------------------------------------------------------------------------Successfully retrieved from(https://services.netscreen.com/cgi-bin/index.cgi).Version info:2345(Detector=12.6.140140207, Templates=2345)

{primary:node0}

5. Download the signature database (operational command, not configuration

command).

root@vdc-edge-fw01-n0> request security idp security-package download

node0:--------------------------------------------------------------------------Will be processed in async mode. Check the status using the status checking CLI

{primary:node0}root@vdc-edge-fw01-n0> request security idp security-package download status node0:--------------------------------------------------------------------------In progress:platforms.xml.gz 100 % 250 Bytes/ 250 Bytes

{primary:node0}root@vdc-edge-fw01-n0> request security idp security-package download status

node0:--------------------------------------------------------------------------Done;Successfully downloaded from(https://services.netscreen.com/cgi-bin/index.cgi)and synchronized to backup.Version info:2345(Wed Feb 12 19:13:53 2014 UTC, Detector=12.6.140140207)

{primary:node0}

6. Verify the progress of the IDP signature download.

root@vdc-edge-fw01-n0> request security idp security-package download status



node0:--------------------------------------------------------------------------In progress:platforms.xml.gz 100 % 250 Bytes/ 250 Bytes

{primary:node0}root@vdc-edge-fw01-n0> request security idp security-package download status

node0:--------------------------------------------------------------------------Done;Successfully downloaded from(https://services.netscreen.com/cgi-bin/index.cgi)and synchronized to backup.Version info:2345(Wed Feb 12 19:13:53 2014 UTC, Detector=12.6.140140207)

7. Install the IDP database.

request security idp security-package install

8. Monitor the status of the install command.

root@vdc-edge-fw01-n0> request security idp security-package install status

node0:--------------------------------------------------------------------------In progress:Updating with new attack or detector for existing running policy...

node1:--------------------------------------------------------------------------Done;Attack DB update : successful - [UpdateNumber=2345,ExportDate=Wed Feb 12 19:13:53 2014 UTC,Detector=12.6.140140207] Updating control-plane with new detector : successful Updating data-plane with new attack or detector : successful (The last known good detector link has been updated with the new detector)

9. Once the security policy is configured and the action is set to “permit”, enable IDP

under “application services”. This redirects traffic that matches the security policy to

the IDP service for inspection. Below is an example of traffic flowing from the trust to

the untrust Internet-access security policy.

root@vdc-edge-fw01-n0# show security policies from-zone trust to-zone untrust policyInternet-access | display set

set security policies from-zone trust to-zone untrust policy Internet-accessmatchsource-address any

set security policies from-zone trust to-zone untrust policy Internet-accessmatchdestination-address any

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication junos-http

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication junos-https

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication junos-http-ext

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication junos-ntp

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication junos-dns-udp

set security policies from-zone trust to-zone untrust policy Internet-accessmatchapplication ICMP



set security policies from-zone trust to-zone untrust policy Internet-access thenpermitapplication-services idp

10. Enable IDP for inbound traffic (flowing from the untrust security zone to the trust

security zone). Once IDP is enabled in a security policy, the IDP policy should be

activated, monitored for effectiveness, and tuned. The command used to activate

the IDP policy in this example is:

set security idp active-policy HTTP-inspection

NOTE: There can be only one active IDP policy. The active IDP policy canbe applied tomultiple rules.

11. The following display set configuration shows a complete policy called

HTTP-inspection on the perimeter firewall. In this example, two rules are created. The

R1 rule is from the trust security zone to the untrust security zone. TheR2 rulemonitors

traffic from the untrust security zone to the trust security zone. The IDP rulebase is

configured to matchWeb-based attacks. Finally, the policy is activated as shown in

Step 10 using the command set security idp active-policy HTTP-inspection.

root@vdc-edge-fw01-n0# show security idp | display set

set security idp idp-policy HTTP-inspection rulebase-ips rule R1match from-zone trustset security idp idp-policy HTTP-inspection rulebase-ips rule R1match source-addressany

set security idp idp-policy HTTP-inspection rulebase-ips rule R1match to-zone untrustset security idp idp-policy HTTP-inspection rulebase-ips rule R1 matchdestination-address any

set security idp idp-policy HTTP-inspection rulebase-ips rule R1 match applicationdefault

set security idp idp-policy HTTP-inspection rulebase-ips rule R1 match attackspredefined-attack-groups "Critical - HTTP"

set security idp idp-policy HTTP-inspection rulebase-ips rule R1 match attackspredefined-attack-groups "Major - HTTP"

set security idp idp-policy HTTP-inspection rulebase-ips rule R1 then actiondrop-connection

set security idp idp-policy HTTP-inspection rulebase-ips rule R1 then notificationlog-attacks

set security idp idp-policy HTTP-inspection rulebase-ips rule R1 then severity criticalset security idp idp-policy HTTP-inspection rulebase-ips rule R2match from-zoneuntrust

set security idp idp-policyHTTP-inspection rulebase-ips rule R2match source-addressany

set security idp idp-policy HTTP-inspection rulebase-ips rule R2match to-zone trustset security idp idp-policy HTTP-inspection rulebase-ips rule R2matchdestination-address any

set security idp idp-policy HTTP-inspection rulebase-ips rule R2match applicationdefault

set security idp idp-policy HTTP-inspection rulebase-ips rule R2match attackspredefined-attack-groups "Critical - HTTP"

set security idp idp-policy HTTP-inspection rulebase-ips rule R2match attackspredefined-attack-groups "Major - HTTP"

set security idp idp-policy HTTP-inspection rulebase-ips rule R2 then actiondrop-connection



set security idp idp-policy HTTP-inspection rulebase-ips rule R2 then notificationlog-attacks

set security idp idp-policy HTTP-inspection rulebase-ips rule R2 then severity criticalset security idp active-policy HTTP-inspectionset security idp security-package url https://services.netscreen.com/cgi-bin/index.cgi

Verification


IDP is configured and working properly.

Results

1. The show security idp status command output verifies that IDP is configured and

running.

root@vdc-edge-fw01-n1> show security idp status

node0:--------------------------------------------------------------------------State of IDP: Default, Up since: 2014-02-10 19:51:58 PST (3d 23:29 ago)

Packets/second: 1 Peak: 4658 @ 2014-02-13 22:34:57 PSTKBits/second : 1 Peak: 1459 @ 2014-02-13 22:34:57 PSTLatency (microseconds): [min: 0] [max: 0] [avg: 0]

Packet Statistics: [ICMP: 0] [TCP: 146727] [UDP: 523] [Other: 0]

Flow Statistics: ICMP: [Current: 0] [Max: 42 @ 2014-02-13 09:14:29 PST] TCP: [Current: 2] [Max: 48 @ 2014-02-12 12:31:09 PST] UDP: [Current: 0] [Max: 30 @ 2014-02-12 06:00:33 PST] Other: [Current: 0] [Max: 0 @ 2014-02-10 19:51:58 PST]

Session Statistics: [ICMP: 0] [TCP: 1] [UDP: 0] [Other: 0]

Number of SSL Sessions : 0

Policy Name : HTTP-inspection Running Detector Version : 12.6.140140207

Forwarding process mode : regular

node1:

--------------------------------------------------------------------------State of IDP: Default, Up since: 2014-02-11 10:40:32 PST (3d 08:40 ago)

Packets/second: 0 Peak: 0 @ 2014-02-11 10:40:32 PSTKBits/second : 0 Peak: 0 @ 2014-02-11 10:40:32 PSTLatency (microseconds): [min: 0] [max: 0] [avg: 0]

Packet Statistics: [ICMP: 0] [TCP: 0] [UDP: 0] [Other: 0]

Flow Statistics:



ICMP: [Current: 0] [Max: 0 @ 2014-02-11 10:40:32 PST] TCP: [Current: 0] [Max: 0 @ 2014-02-11 10:40:32 PST] UDP: [Current: 0] [Max: 0 @ 2014-02-11 10:40:32 PST] Other: [Current: 0] [Max: 0 @ 2014-02-11 10:40:32 PST]

Session Statistics: [ICMP: 0] [TCP: 0] [UDP: 0] [Other: 0]

Number of SSL Sessions : 0

Policy Name : HTTP-inspection Running Detector Version : 12.6.140140207

2. Verify that the IDP attack table is configured and running on the primary node.

root@vdc-edge-fw01-n1> show security idp attack table

node0:--------------------------------------------------------------------------

node1:--------------------------------------------------------------------------

{primary:node1}

3. Verify that the IDP application statistics are incrementing based on the configured

IDP rule set. (output is truncated to show the relevant packet counter on node1.)

root@vdc-edge-fw01-n1> show security idp application-statistics

node0:--------------------------------------------------------------------------IDP applications:

application type packet countDNS 0HTTP 0LDAP 0SSL 0MSRPC 0MSSQL 0MYSQL 0BGP 0

node1:--------------------------------------------------------------------------IDP applications:

application type packet countDNS 36HTTP 4147LDAP 0SSL 747MSRPC 0MSSQL 0MYSQL 0



Host Security


• Configuring the Firefly Host on page 156


Overview

Juniper Networks Firefly Host is a virtualized firewall that runs on VMware ESX/ESXi for

to secure intra-virtual machine (VM) and inter-VM traffic. Juniper Firefly Host has three

main components:

• Firefly Host Security Design VM (SVM)—This provides a central management server.

It provides charts, tables, and graphs, and collects the logs from the security policy

which helps to adjust the virtualized environment.

• Firefly Host Security VM—This is installed on each host of VMware ESX/ESXi to be

secured. Firefly Host Security VM acts as a conduit to the Firefly Host kernel module

that it inserts into the hypervisors of hosts. The Firefly Host Security VMmaintains the

policy and logging information.

• Firefly Host kernelmodule—Virtualized network traffic is secured and analyzed against

the security policy for all VM on the ESX/ESXi host in the Firefly Host kernel module

installed on the host. All connections are processed and firewall security policy is

enforced in the Firefly Host Series kernel module.

FireflyHostprotects theVMaswell as thehypervisor.When it is deployed into theVMware

environment, Firefly Host Security VM is installed on VMware ESX/ESXi host

(Figure50onpage 155), it inserts theFireflyHost kernelmodule into thehost’s hypervisor

between the virtual network interface card (NIC) and virtual switch (vSwitch) or

distributed virtual switch (DvSW).

Firefly Host supports vMotion, enabling mobility of both the VM and the Firefly Host. In

cases where a VM is moved to a different virtual machine, the security policy assigned

to that VMmoves along with the virtual machine. Because Firefly Host is supported by

vMotion, this VMmobility does not require any additional configuration.

Figure 50: Logical View of Juniper Networks Firefly Host Installation



Firefly Host Security Design VMalsomanages the entire Security VirtualMachine (SVM),

defining security policies, configuring antivirus, IDS, and so on. To secure ESX/ESXi hosts

and VMs, we need to deploy SVM on the ESX/ESXi hosts first. As soon as you have

deployedSVMon each ESX/ESXi host, it will be secured and insert the Firefly Host kernel

on the ESX/ESXi hypervisor.

Firefly Host Security Design VM can bemanaged through aWeb GUI that enables you

to define firewall security policy for all the VMs, similar to how you configure a physical

SRX firewall. Traffic can be controlled between two VMs running on one ESX/ESXi host,

andmultiple VMs running onmultiple ESX/ESXi hosts.

Firefly Host Security Design VM pushes the firewall security policy to the SVM kernel

module. When traffic enters through a physical network adapter on an ESX/ESXi host,

it travels to the virtual switch or distributed switch first, then visits the Firefly Host kernel

module before being forwarded to the appropriate VM. As the security policy resides in

the kernel module and is based on the security policies, traffic is allowed or denied to or

from the VM.

Configuring the Firefly Host

When you install SVMon the hosts, all the VMs are unsecured by default. Before defining

security policies, youmust secure the VM environment.

Step-by-Step Procedure



To configure Firefly Host, follow these steps:

1. The first step in configuration is to log in to the Firefly Host to select the VMs that

should be secured. The example below contains several ESXi hosts under Unsecured

Network and Secured Network. On the left side (under Unsecured Network),

Win2012-Exch02 VM is not secured. On the right side (under Secured Network),

Win2012-Exch06 VM is secured. To secure or unsecure VM, you need to select or

deselect the check box in front of the VM and click on Secure or Unsecure in the

Settings tab. You also need to secure the port groupwhen securing a VM (this is done

similarly by selecting Secure in the Settings tab for a dvPort Group).

Figure 51: An Example dvPort Group

2. Configureagroup foronesetofapplications. Theexamplebelowshowsanapplication

name (MediaWiki) that represents a single group. Additional application groups can

be created using Add Smart Group under Security Settings, Group tab in Firefly Host.

Define vi.notes which contains the keyword MediaWiki in the Firefly Host. By doing

this, it will detect all VMs that have the keyword MediaWiki in an annotation of VM.

Beforedefining security policies, it is a good idea to survey theexistingVMenvironment

to obtain a list of the applications hosted in the data center. Creating Smart Groups

initially will save time during security policy configuration.

Figure 52: Configure an Application Group



3. Once groups are defined in Firefly Host, an additional step is required on the vCenter

Server.At theMediaWikiVMsummary tabunder vCenterServer, add thesamekeyword

you used in vi notes in the Annotations field in Step 2. This is required to enable the

FireflyHost toproperlydetect thevirtualmachine. In thebelowexample, theMediaWiki

Group in the Firefly Host will detect all VMs that are properly annotated with the tag

MediaWiki.

Figure 53: The Annotation Allows Firefly Host to Detect Related VMs

4. Next, define security policies in the Firewall area of the Firefly Host. Also define an

initial, Global rule under Global Policy in Policy Group. This rule creation applies to all

VMs in the environment, enabling security even if an application group isn’t properly

created. To create specific rules, navigate to Policy Groups in the left pane. You will

notice that the policy groups contain both Inbound andOutbound rules. Inbound rule

means traffic is coming into the VM and Outbound rule means traffic is originating

from the VM. Below is an example rule that allows HTTP, HTTPS, and ICMP inbound

to the MediaWiki application VM.



Figure 54: Define Security Policies

Verification

Many network administrators are required to monitor security status in the data center.

The administrators must be able to see details on allowed traffic, as well as blocked or

anomalous traffic. This information is found in the Logswindow. Logging can be enabled

or disabled on a per policy basis. You can also enable logging for all the policies. Please

keep in mind that enabling logging for all policies can have an effect on CPU utilization

and can introduce network congestion or packet drops. Because of this, we do not

recommend enabling logging for all policies.

To see policy logs, you need to enable logging per policy. Once logging is enabled, the

FireflyHost can filter by source IPaddress, destination IPaddress, or protocol. This filtering

is performed in the advanced view of the logging screen.

For more information on Firefly Host configuration, troubleshooting, and best practices,

see the Firefly Host Administration Guide at:

Juniper Networks Firefly Host - Installation and Administration Guide for VMware



https://www.juniper.net/techpubs/en_US/vgw5.5/information-products/topic-collections/vgw-vmware/vgw-vmware-install-admin.pdf

CHAPTER 8

Data Center Services

• Data Center Services Overview on page 161

• Configuring Compute Hardware on page 162

• Virtualization on page 186

• EMC Storage Overview on page 204

• Load Balancing on page 227


Data Center Services Overview

This section covers the configuration of services in the virtualized IT data center. The

areas covered in this chapter include:

• Compute

• Compute overview

• Hardware overview

• Compute configuration

• Configure management

• Configure switching

• Virtualization

• Virtualization overview

• Configure virtualization

• EMC storage configuration

• Load balancing

• Overview

• Configuration

• Applications



Configuring Compute Hardware


• Requirements on page 162


Overview

The Juniper MetaFabric 1.0 solution is designed around optimizing network, security,

virtualization, mobility, and visibility in the data center environment. To that end, all of

the data center network, security, and resiliency features are designed to support the

hosting of applications in such a way that provides the highest quality user experience.

This section covers the configuration of the physical compute hosts that reside in the

data center.

Requirements

The solution requirements that guided testing and validation include:

• Solution must support lossless Ethernet.

• Solution must provide redundant network connectivity using all available bandwidth.

• Solution must support moving a virtual machine (VM) between hosts.

• Solution must support high availability of a VM.

• Solutionmust support virtual network identificationusingLinkLayerDiscoveryProtocol

(LLDP).

• Solution must provide physical and virtual visibility and reporting of VMmovements.

• Solution must support lossless Ethernet for storage transit.

• Solution must support booting a server from the storage array.

The IBM Flex Chassis was selected as the physical compute host. High-level

implementation details include:

• IBM Flex server is configured with multiple ESXi hosts hosting all the VMs running the

business-critical applications (SharePoint, Exchange, Media-wiki, andWWW).

• Distributed vSwitch is configured betweenmultiple physical ESXi hosts configured in

IBM hosts.

Topology

The topologyused in thedatacenter compute, virtualization, andstoragedesign is shown

in Figure 55 on page 163.



Figure 55: Compute and Virtualization as Featured in theMetaFabric 1.0Solution

Compute Hardware Overview

The IBM System x3750M4 is a 4-socket server that features a streamlined design,

optimized for performance. This solution uses two IBM standalone system 3750s as an

Infra Cluster. Infra Cluster is hosting all infrastructure-related VMs such as Junos Space,

Firefly Host Security Design VM, and Virtual Center server. The IBM System 3750 has

dedicated twomanagement ports, which are connected to amanagement switch as a

LAG. Figure 56 on page 163 shows the IBM System x3750M4.

Figure 56: IBM x3750M4


Chapter 8: Data Center Services

The configuration of out-of-bandmanagement (OOB) is required to properly manage

the computing hardware featured in this solution. The configuration of OOB is covered

in this section.

To configure the IBM System x3750M4 in the OOB role, follow these steps:

1. Configure twoLAG(ae11 andae12) interfaces for each IBMsystemon themanagement

switch.

[edit]set interfaces ge-1/0/44 ether-options 802.3ad ae11set interfaces ge-1/0/45 ether-options 802.3ad ae11set interfaces ge-1/0/46 ether-options 802.3ad ae12set interfaces ge-1/0/47 ether-options 802.3ad ae12set interfaces ae11 description "connection to POD1 Standalone server"set interfaces ae11 aggregated-ether-optionsminimum-links 1set interfaces ae11 unit 0 family ethernet-switching vlanmembers Compute-VLANset interfaces ae12 description "connection to POD2 standalone server"set interfaces ae12 aggregated-ether-optionsminimum-links 1set interfaces ae12 unit 0 family ethernet-switching vlanmembers Compute-VLANset vlans Compute-VLAN vlan-id 800

2. Configure LAG on the IBM system. This configuration step is performed as part of the

virtualization configuration section.

NOTE: Each server has four 10-Gigabit Ethernet NIC ports connected tothe QFX3000-MQFabric system as a data port for all VM traffic. Eachsystem is connected to each POD for redundancy purposes. The IBMSystem3750 isconnectedtoPOD1using4x 10-GigabitEthernet.AsecondIBM System 3750 connects to POD2 using 4 x 10-Gigabit Ethernet. Theuse of LAG provides switching redundancy in case of a POD failure.

3. Configure POD1 to connect to the IBM System 3750 server. Four ports of data traffic

are configured as a LAGand carry several VLANs that are required for the Infra Cluster.

[edit]set interfaces interface-range POD1-Standalone-server member n2:xe-0/0/8set interfaces interface-range POD1-Standalone-server member n3:xe-0/0/8set interfaces interface-range POD1-Standalone-server member n3:xe-0/0/9set interfaces interface-range POD1-Standalone-server member n2:xe-0/0/9set interfaces interface-range POD1-Standalone-server ether-options 802.3adRSNG2:ae0

set interfaces RSNG2:ae0 description POD1-Standalone-serverset interfaces RSNG2:ae0 unit 0 family ethernet-switching port-mode trunkset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers MGMTset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers Infraset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembersWikimediaset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers SQLset interfacesRSNG2:ae0unit0 familyethernet-switchingvlanmembersStorage-POD1set interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembersExchange-Cluster

set interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers SharePointset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers Tera-VM



set interfacesRSNG2:ae0unit0familyethernet-switchingvlanmembersSecurity-Mgmtset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers VM-FTset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembersRemote-Access

4. Configure POD2 for connection to the second IBM System 3750.

[edit]set interfaces interface-range IBM-Standalonemember "n3:xe-0/0/[26-27]"set interfaces interface-range IBM-Standalonemember "n5:xe-0/0/[26-27]"set interfaces interface-range IBM-Standalone ether-options 802.3ad RSNG3:ae1set interfaces RSNG3:ae1 description POD2-IBM-Standaloneset interfaces RSNG3:ae1 unit 0 family ethernet-switching port-mode trunkset interfaces RSNG3:ae1 unit 0 family ethernet-switching vlanmembers MGMTset interfacesRSNG3:ae1unit0 familyethernet-switchingvlanmembersStorage-POD2set interfaces RSNG3:ae1 unit 0 family ethernet-switching vlanmembers Infraset interfaces RSNG3:ae1 unit 0 family ethernet-switching vlanmembers SQLset interfaces RSNG3:ae1 unit 0 family ethernet-switching vlanmembers SharePointset interfaces RSNG3:ae1 unit 0 family ethernet-switching vlanmembersExchange-cluster

set interfaces RSNG3:ae1 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces RSNG3:ae1 unit 0 family ethernet-switching vlanmembersWikimediaset interfaces RSNG3:ae1 unit 0 family ethernet-switching vlanmembers Tera-VMset interfacesRSNG3:ae1unit0 familyethernet-switchingvlanmembersSecurity-Mgmtset interfaces RSNG3:ae1 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces RSNG3:ae1 unit 0 family ethernet-switching vlanmembers VM-FTset interfacesRSNG3:ae1unit0familyethernet-switchingvlanmembersRemote-Access

The MetaFabric 1.0 solution utilizes a second set of compute hardware as well. The IBM

Flex System Enterprise Chassis is a 10U next-generation server platform that features

integrated chassis management. It is a compact, high-density, high-performance, and

scalable rack-mount system. It supports up to 14 one-bay compute nodes that share

common resources, such as power, cooling, management, and I/O resources within a

single Enterprise chassis. The IBM Flex System can also support up to seven 2-bay

compute nodes or three 4-bay compute nodes when the shelves are removed. You can

mix andmatch 1-bay, 2-bay, and4-bay compute nodes tomeet specific hardware needs.

Themajor components of Enterprise Chassis (Figure 57 on page 166) are:

• Fourteen 1-bay compute node bays (can also support seven 2-bay or three 4-bay

compute nodes with the shelves removed).

• Six 2500W power modules that provide N+N or N+1 redundant power. Optionally, the

chassis can be ordered through the configure-to-order (CTO) process with six 2100W

power supplies for N+1 redundant power.

• Ten fanmodules.

• Four physical I/Omodules.

• Awide variety of networking solutions that include Ethernet, Fibre Channel, FCoE, and

• InfiniBand.

• Two IBM Chassis Management Module (CMMs). The CMM provides single-chassis

management support.



Figure 57: IBM Flex System Enterprise Chassis (Front View)

The following components can be installed into the rear of the chassis

(Figure 58 on page 166):

• Up to two CMMs.

• Up to six 2500W or 2100W power supply modules.

• Up to six fan modules that consist of four 80-mm fanmodules and two 40-mm fan

modules.

• Additional fan modules can be installed for a total of 10 modules.

• Up to four I/Omodules.

Figure 58: IBM Flex System (Rear View)

The IBMFlexSystem includesaChassisManagementModule (CMM).TheCMMprovides

a single point of chassis management as well as the network path for remote keyboard,

video, andmouse (KVM) capability for compute nodes within the chassis. The IBM Flex

System chassis can accommodate one or two CMMs. The first is installed into CMMBay



1, the second into CMM bay 2. Installing two CMMs provides control redundancy for the

IBM Flex System.

The CMM provides these functions:

• Power control

• Fanmanagement

• Chassis and compute node initialization

• Switch management

• Diagnostics

• Resource discovery and inventory management

• Resource alerts andmonitoring management

• Chassis and compute node power management

• Network management

The CMM has the following connectors:

• USBconnection:Canbeused for insertionofaUSBmediakey for tasks suchas firmware

updates.

• 10/100/1000-Mbps RJ45 Ethernet connection: For connection to amanagement

network. The CMM can bemanaged through this Ethernet port.

Configuring Compute Switching

The IBM Flex System also offers modular switching options to enable various levels of

switching redundancy, subscription (1:1 vs oversubscription), and switched or Pass-thru

modesofoperation.The first of thesemodulesused in thesolution is the IBMFlexSystem

Fabric CN4093 10Gb/40Gb Converged Scalable Switch. The IBM Flex System Fabric

CN4093 10Gb/40Gb Converged Scalable Switch provides unmatched scalability,

performance, convergence, and network virtualization, while also delivering innovations

to help address a number of networking concerns and providing capabilities that help

you prepare for the future.

The switch offers full Layer 2/3 switching and FCoE Full Fabric and Fibre Channel NPV

Gateway operations to deliver a converged and integrated solution, and it is installed

within the I/Omodule bays of the IBM Flex System Enterprise Chassis. The switch can

help youmigrate to a 10-Gb or 40-Gb converged Ethernet infrastructure and offers

virtualization features.



Figure 59: IBM Flex System Fabric CN4093 10Gb/40Gb ConvergedScalable Switch

The CN4093 switch is initially licensed for fourteen 10-GbE internal ports, two external

10-GbE SFP+ ports, and six external Omni Ports enabled.

The base switch and upgrades are as follows:

• 00D5823 is the part number for the physical device, which comes with 14 internal

10-GbE ports enabled (one to each node bay), two external 10-GbE SFP+ ports that

are enabled to connect to a top-of-rack switch or other devices, and six Omni Ports

enabled to connect to either Ethernet or Fibre Channel networking infrastructure,

depending on the SFP+ cable or transceiver used.

• 00D5845 (Upgrade 1) can be applied on the base switch when you needmore uplink

bandwidthwith two 40-GbEQSFP+ ports that can be converted into 4x 10-GbE SFP+

DAC linkswith theoptionalbreak-outcables.Thisupgradealsoenables 14more internal

ports, for a total of 28 ports, to provide more bandwidth to the compute nodes using

4-port expansion cards.

• 00D5847 (Upgrade2)canbeappliedon thebaseswitchwhenyouneedmoreexternal

Omni Ports on the switch or if you want more internal bandwidth to the node bays.

The upgrade enables the remaining 6 external Omni Ports, plus 14more internal 10-Gb

ports, for a total of 28 internal ports, to providemore bandwidth to the compute nodes

using four-port expansion cards.

Further ports can be enabled:

• Fourteenmore internalportsand twoexternal40GbEQSFP+uplinkportswithUpgrade

1

• Fourteenmore internal ports and six more external Omni Ports with the Upgrade 2

license options.

• Upgrade 1 andUpgrade 2 can be applied on the switch independently from each other

or in combination for full feature capability.

The CNAmodule has amanagement and console port. There are two different

command-line interface (CLI) modes on IBM/BNT network devices: IBMNOSmode and

ISCLI (Industry Standard CLI) mode. The first time you start the CN4093, it boots into

the IBMNetworking OS CLI. To access the ISCLI, enter the following command and reset

the CN4093.



1. Reset the CN4093.

Router (config)# boot cli-mode ibmnos-cli

The switch retains your CLI selection, even when you reset the configuration to factory

defaults. The CLI boot mode is not part of the configuration settings. If you downgrade

the switch software to an earlier release, it will boot into menu-based CLI. However, the

switch retains the CLI boot mode, and will restore your CLI choice.

The secondmodular switching option deployed as part of the solution is the IBM Flex

System EN4091 10 Gb Ethernet Pass-thru Module (Figure 60 on page 169). The EN4091

10-GbEthernetPass-thruModuleoffers aone-for-oneconnectionbetweenasinglenode

bay and an I/Omodule uplink. It has nomanagement interface, and can support both

1-Gbps and 10-Gbps dual-port adapters that are installed in the compute nodes. If

quad-port adapters are installed in the compute nodes, only the first two ports have

access to the Pass-thru module ports.

The necessary 1-GbE or 10-GbEmodule (SFP, SFP+, or DAC)must also be installed in

the external ports of the pass-thru module. This configuration supports the speed (1 Gb

or 10 Gb) andmedium (fiber-optic or copper) for adapter ports on the compute nodes.

Figure 60: IBM Flex System EN4091 10Gb Ethernet Pass-thru Module

The EN4091 10Gb Ethernet Pass-thru Module has the following specifications:

• Internal ports - 14 internal full-duplex Ethernet ports that can operate at 1-Gb or 10-Gb

speeds.

• External ports - 14 ports for 1-Gb or 10-Gb Ethernet SFP+ transceivers (support for

1000BASE-SX, 1000BASE-LX, 1000BASE-T, 10GBASE-SR, or 10GBASE-LR) or SFP+

DAC.

• Unmanageddevice that has no internal Ethernetmanagement port. However, it is able

to provide its VPD to the secure management network in the Chassis Management

Module.

• Allowsdirect connection fromthe 10-GbEthernetadapters thatare installed incompute

nodes in a chassis to an externally located top-of-rack switch or other external device.

NOTE: The EN4091 10-Gb Ethernet Pass-thru Module has only 14 internalports. As a result, only two ports on each compute node are enabled, one foreachof the twomodules thatare installed in thechassis. If four-port adaptersare installed in the compute nodes, ports 3 and 4 on those adapters are notenabled.



Configuring Compute Nodes

The Juniper MetaFabric 1.0 solution utilizes the IBM Flex System servers as the primary

compute nodes. The lab configuration utilized 5 compute nodes (of a possible 14) in

each IBM Pure Flex System. The IBM Flex System portfolio of compute nodes includes

Intel Xeon processors and IBM POWER7 processors. Depending on the compute node

design, nodes can come in one of these form factors:

• Half-widthnode:Occupiesonechassisbay, half thewidthof thechassis (approximately

215 mm or 8.5 in.). An example is the IBM Flex System x220 Compute Node.

• Full-width node: Occupies two chassis bays side-by-side, the full width of the chassis

(approximately 435mm or 17 in.). An example is the IBM Flex System p460 Compute

Node.

Thesolution labutilized the IBMFlexSystemx220computenode (Figure61onpage 170).

The IBM Flex System x220 Compute Node, machine type 7906, is the next generation

cost-optimized compute node that is designed for less demanding workloads and

low-density virtualization. The x220 is efficient and equipped with flexible configuration

options and advancedmanagement to run a broad range of workloads. The IBM Flex

System x220 Compute Node is a high-availability, scalable compute node that is

optimized to support the next-generation microprocessor technology. With a balance

of cost andsystemfeatures, the x220 is an ideal platform for general businessworkloads.

The x220 is a half-wide compute node and requires that the chassis shelf is installed in

the IBM Flex System Enterprise Chassis. The IBM Flex System x220 Compute Node

features the Intel Xeon E5-2400 series processors. The Xeon E5-2400 series processor

has models with either 4, 6, or 8 cores per processor with up to 16 threads per socket.

The x220 supports LP DDR3memory LRDIMMs, RDIMMs, and UDIMMs. The x220 server

has two 2.5-inch hot-swap drive bays accessible from the front of the blade server. On

standardmodels, the two2.5-inchdrivebaysareconnected toaServeRAIDC105onboard

SATA controller with software RAID capabilities.

The applications that are installed on the compute nodes can run natively on adedicated

physical server or they can be virtualized (in a virtual machine that is managed by a

hypervisor layer). All the compute nodes are using theVMware ESXi 5.1 operating system

as a baseline for virtualization, and all the enterprise applications are running as virtual

machines on top of the ESXi 5.1 Operating System.

Figure 61: IBM Flex System x220 Compute Node

This solution implementation utilizes two compute PODs. Two IBM Pure Flex Systems

are connected to the QFX3000-MQFabric system POD1, and two Flex Systems are

connected to POD2 (also utilizing QFX3000-M).



The POD1 and POD2 topologies use similar hardware to run the virtual servers.

POD1 includes the following compute hardware:

• Two IBM Pure Flex Systems with x220 compute node

• IBM Pure Flex Chassis 40Gb CNA Card

• IBM Pure Flex Chassis 10Gb Pass-thru (P/T) I/O Card

POD2 includes the following compute hardware:

• Two IBM Pure Flex Systems with x220 compute node

• IBM Pure Flex Chassis 10Gb CNA Card

• IBM Pure Flex Chassis 10Gb Pass-thru (P/T) I/O Card

Beforemovingon to configuring the virtualization, a short overviewof switchingoperation

andconfiguration in the IBM flex system is required. Figure62onpage 171 shows thePOD1

network topology utilizing the IBM Pure Flex System Pass-thru (P/T) chassis.

Figure 62: IBM Pure Flex Pass-thru Chassis

IBM Pure Flex Pass-thru chassis has four 10-Gb Ethernet I/O Cards. Each I/O card has 14

10-Gb Ethernet network ports for each compute node. That means each compute node

will have four network adapters on the physical connection. Eachmodule has 14 external

network portswhich are internally linkedwith 14Compute nodes through the back plane.



• I/Omodule 1 of port number 1 will connect to compute node 1




The 14 Compute nodes have connectivity to all 4 I/Omodules, and each compute node

has 4 network ports. All four network ports are connected to different node of RSNG of

the QFX3000-MQFabric system, which gives full redundancy. Also LAG is configured

between the servers and access switches. This configuration ensures utilization of all

the links while providing full redundancy.

The next section shows a sample configuration for connection from the QFX3000-M

(POD1) and the QFX3000-M (POD2) to the two pass-thru chassis compute nodes.

Configuring POD to Pass-thru Chassis Compute Nodes

Toconfigure theconnectionbetween thePODsand thecompute role, follow these steps:

1. ConfigurePOD1 (QFabricQFX3000-M).Pleasenote that youmustuseanMTUsetting

of 9192. Enabling jumbo frames in the data center generally enables better

performance.

[edit]set interfaces interface-range IBM-FLEX-2-CN1-passthroughmember"n2:xe-0/0/[30-31]"

set interfaces interface-range IBM-FLEX-2-CN1-passthroughmember"n3:xe-0/0/[30-31]"

set interfaces interface-range IBM-FLEX-2-CN1-passthrough ether-options 802.3adRSNG2:ae1




set interfaces RSNG2:ae1 description "IBM Flex-2 Passthrough-CN1"set interfaces RSNG2:ae1 unit 0 family ethernet-switching port-mode trunkset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers MGMTset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers Infraset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembersWikimediaset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers SQLset interfacesRSNG2:ae1unit0 familyethernet-switchingvlanmembersStorage-POD1set interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembersExchange-Cluster

set interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers SharePointset interfacesRSNG2:ae1unit0 familyethernet-switchingvlanmembersSecurity-Mgmtset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers VM-FTset interfacesRSNG2:ae1unit0familyethernet-switchingvlanmembersRemote-Accessset interfaces RSNG2:ae2 description "IBM Flex-2 Passthrough-CN2"set interfaces RSNG2:ae2 unit 0 family ethernet-switching port-mode trunkset interfaces RSNG2:ae2 unit 0 family ethernet-switching vlanmembers MGMT



set interfaces RSNG2:ae2 unit 0 family ethernet-switching vlanmembers Infraset interfaces RSNG2:ae2 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces RSNG2:ae2 unit 0 family ethernet-switching vlanmembersWikimediaset interfaces RSNG2:ae2 unit 0 family ethernet-switching vlanmembers SQLset interfacesRSNG2:ae2unit0 familyethernet-switchingvlanmembersStorage-POD1set interfaces RSNG2:ae2 unit 0 family ethernet-switching vlanmembersExchange-Cluster

set interfaces RSNG2:ae2 unit 0 family ethernet-switching vlanmembers SharePointset interfacesRSNG2:ae2unit0 familyethernet-switchingvlanmembersSecurity-Mgmtset interfaces RSNG2:ae2 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces RSNG2:ae2 unit 0 family ethernet-switching vlanmembers VM-FTset interfacesRSNG2:ae2unit0familyethernet-switchingvlanmembersRemote-Accessset vlans Exchange vlan-id 104 set vlans Exchange l3-interface vlan.104set vlans Exchange-Cluster vlan-id 109 set vlans Infra vlan-id 101set vlans MGMT vlan-id 800 set vlans Remote-Access vlan-id 810set vlans SQL vlan-id 105 set vlans Security-Mgmt vlan-id 801set vlans SharePoint vlan-id 102 set vlans Storage-POD1 vlan-id 108set vlans Storage-POD1 l3-interface vlan.108 set vlans VM-FT vlan-id 107set vlans Vmotion vlan-id 106 set vlansWikimedia vlan-id 103set vlansWikimedia l3-interface vlan.103

2. Configure POD 2 (QFabric QFX3000-M). Please note the use of an MTU setting of

9192.Enabling Jumboframes in thedatacentergenerallyenablesbetterperformance.

set groups Jumbo-MTU interfaces <*ae*>mtu 9192 set interfaces interface-rangeIBM-FLEX-2-CN1-passthroughmember "n3:xe-0/0/[34-35]"






set interfaces RSNG3:ae0 description IBM-FLEX-2-CN-1-Passthroughset interfaces RSNG3:ae0 unit 0 family ethernet-switching port-mode trunkset interfaces RSNG3:ae0 unit 0 family ethernet-switching vlanmembers MGMTset interfacesRSNG3:ae0unit0 familyethernet-switchingvlanmembersStorage-POD2set interfaces RSNG3:ae0 unit 0 family ethernet-switching vlanmembers Infraset interfaces RSNG3:ae0 unit 0 family ethernet-switching vlanmembers SQLset interfaces RSNG3:ae0 unit 0 family ethernet-switching vlanmembers SharePointset interfaces RSNG3:ae0 unit 0 family ethernet-switching vlanmembersExchange-cluster

set interfaces RSNG3:ae0 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces RSNG3:ae0 unit 0 family ethernet-switching vlanmembersWikimediaset interfacesRSNG3:ae0unit0familyethernet-switchingvlanmembersSecurity-Mgmtset interfaces RSNG3:ae0 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces RSNG3:ae0 unit 0 family ethernet-switching vlanmembers VM-FTset interfaces RSNG3:ae0 unit 0 family ethernet-switching vlanmembersRemote-Access

set interfaces RSNG2:ae1 description IBM-FLEX-2-CN2-passthroughset interfaces RSNG2:ae1 unit 0 family ethernet-switching port-mode trunk



set interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers MGMTset interfacesRSNG2:ae1unit0 familyethernet-switchingvlanmembersStorage-POD2set interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers Infraset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers SQLset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers SharePointset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembersExchange-cluster

set interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembersWikimediaset interfacesRSNG2:ae1unit0 familyethernet-switchingvlanmembersSecurity-Mgmtset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces RSNG2:ae1 unit 0 family ethernet-switching vlanmembers VM-FTset interfacesRSNG2:ae1unit0familyethernet-switchingvlanmembersRemote-Accessset vlans Exchange vlan-id 104set vlans Exchange-cluster vlan-id 109set vlans Infra vlan-id 101set vlans MGMT vlan-id 800set vlans Remote-Access vlan-id 810set vlans SQL vlan-id 105set vlans SQL l3-interface vlan.105set vlans Security-Mgmt vlan-id 801set vlans SharePoint vlan-id 102set vlans SharePoint l3-interface vlan.102set vlans Storage-POD2 vlan-id 208set vlans Storage-POD2 l3-interface vlan.208set vlans VM-FT vlan-id 107set vlans Vmotion vlan-id 106set vlansWikimedia vlan-id 103

TheMetaFabric 1.0 solution also utilizes the 40-Gb Ethernet CNA I/Omodule (in POD1).

A short overview of the operation and configuration of this module is required.

Figure63onpage 175shows thePOD1network topologyutilizing the IBMPureFlexSystem

Chassis with the 40-Gb Ethernet CNA I/Omodule.



Figure 63: POD1 Topologywith the IBMPure Flex Chassis + 40Gbps CNAModule

Figure 63 on page 175 is an example of the IBM Pure Flex System Compute Node 1. All

computenodes inan IBMPureFlexSystemutilizing the 10-GbCNAor40-GbCNAmodules

will havea similar physical lookandconnectivity. Actually, I/OModuleSwitch is integrated

into the IBM Pure Flex System. By looking at the IBM Pure Flex System, you will only see

EXT ports physically. INT ports are not visible behind the chassis. INT ports connect to

backplane of the CNA Fabric switch I/O Module. EXT ports are connected to external

switches to the QFX3000-MQFabric system. Ethernet LAG is also configured between

the QFX POD1 and the compute nodes in POD1. EXT Ports (3 and 7) from each I/O port

are connected to Node (6 and 7) of the QFX3000-MQFabric system.We have created

RSNG4:ae0 LAG between the I/O Module Switch and the QFX3000-MQFabric system.

Without a license, the CNAmodule has one network port (INT A port) which is internally

linked with an external port (EXT) through the chassis backplane. As shown in the

example, Compute Node 1 will see only one network port (INTA). INTA port will be visible

only to VMware ESXi which is running on the compute node. EXT ports are connected to

external switches where physical cables connect to another layer of switches. After you

install the advanced license for the 40-Gb CNA Fabric Switch I/O Module, an additional

internal port is activated. After installing the license, you will see two ports in each I/O

Module for the compute node.

For instance, Compute Node 1 has two ports (INTA1 and INTB1) on each 40-Gb CNA

Fabric Switch I/O Module. As we have two CNA Fabric Switch I/O Modules, Compute

Node 1 will have four internal network ports. The other 40-Gb CNA Fabric Switch I/O

Module will have the same naming convention (such as INTA1 and INTB1 which is on the

different CNA Fabric Switch I/O Module 1 or I/O Module 2). Once an expanded license is

installed, external ports EXT3, EXT4, EXT5, and EXT6becomea single EXT340-Gbport;

andEXT7, EXT8, EXT9, andEXT10becomea single EXT740-Gbport on eachCNAFabric

Switch I/Omodule.



NOTE: Simply creating RSNG between the CNA Fabric Switch andQFX3000-MQFabric system is not an effective configuration. Thisconfiguration can cause intermittent packet loss because both I/OmodulesfromtheCNAFabric switchwork independently. Intermittent packet losswillhappen if you configure LAG/RSNGonly on theQFX3000-MQFabric systemswitch. To resolve this issue, LAGmust also be configured on the CNA FabricSwitch I/Omodule.

Configuration of LAG on CNA Fabric Switches is covered below. In this solution example,

we have cross-connected the EXT1 and EXT2 ports of CNA Fabric Switch I/O Module 1

and 2. This is referred to as an ISL onCNAFabric Switches. This is themajor configuration

required to work LAG efficiently on the internal and external side. LAG is configured on

INT ports and EXT ports. This LAG is configured using the LACP protocol as a trunk port

and carrying multiple VLAN application traffic.

Configuring the CNA Fabric Switches

To configure LAG on the CNA Fabric Switches, follow these steps:

1. Configure the Fabric Switch I/O Module1 on the IBM Pure Flex System 40-Gbps CNA.

interface port INTA1taggingexit

!interface port INTA2taggingexit!interface port INTA3taggingexit

!interface port INTA4taggingexit


!interface port INTB1taggingexit



!



interface port INTB4taggingexit


!interface port EXT1taggingexit




!interface port INTA1lacpmode activelacp key 1001





!interface port INTB1lacpmode activelacp key 1001



!



interface port INTB3lacpmode activelacp key 1003



!interface port EXT1lacpmode activelacp key 200




!vlan 101enablename "INFRA"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

!vlan 102enablename "SharePoint"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

!vlan 103enablename "WM"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

!vlan 104enablename "EXCHANGE"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

!vlan 105enablename "SQL"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

!vlan 106enablename " Vmotion"



member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7!vlan 107enablename "VM-FT"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

!vlan 108enablename "Storage-iSCSI"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

!vlan 109enablename "Exchange DAG"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

!vlan 800enablename "VDCMgmt"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

!vlan 801enablename "Security-MGMT"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

!vlan 4094enablename "VLAN 4094"member EXT1-EXT2!

!vlag enablevlag tier-id 10vlag isl vlan 4094vlag isl adminkey 200vlag adminkey 1000 enablevlag adminkey 1005 enablevlag adminkey 1003 enablevlag adminkey 1001 enablevlag adminkey 1002 enable

vlag adminkey 1004 enable

NOTE: Similar configuration is required on CNA Fabric switch I/OModule2 because of the different I/Omodule integrated into the single IBM PureFlex System. In this configuration, INTA1 and INTB1 use LACP key 1001 tocreate a LAG, EXT3 and EXT7 use LACP key 1000, and EXT1 and 2 useLACP key 200. As a result, there are a total of three LAGs. EXT1 and 2 actas an ISL link and carry traffic for LACP LAGs 1000 and 1001 (which isinternal and external traffic).



2. Configure QFX3000-MQFabric System connectivity to the IBM Pure Flex System

40Gb CNA I/OModule.

[edit]set chassis node-group RSNG4 node-device n6 pic 1 xle port-range 4 15set chassis node-group RSNG4 node-device n7 pic 1 xle port-range 4 15set chassis node-group RSNG4 aggregated-devices ethernet device-count 10set interfaces interface-range IBM-FLEX-1-40G-IO-1-2-VLAGmember n6:xle-0/1/6set interfaces interface-range IBM-FLEX-1-40G-IO-1-2-VLAGmember n7:xle-0/1/6set interfaces interface-range IBM-FLEX-1-40G-IO-1-2-VLAGmember n6:xle-0/1/8set interfaces interface-range IBM-FLEX-1-40G-IO-1-2-VLAGmember n7:xle-0/1/8set interfaces interface-range IBM-FLEX-1-40G-IO-1-2-VLAG ether-options 802.3adRSNG4:ae0

set interfaces RSNG4:ae0 description "40G CNA to IBM-FLEX-1-IO-1"set interfaces RSNG4:ae0mtu 9192set interfaces RSNG4:ae0 aggregated-ether-options lacp activeset interfaces RSNG4:ae0 unit 0 family ethernet-switching port-mode trunkset interfaces RSNG4:ae0 unit 0 family ethernet-switching vlanmembers MGMTset interfaces RSNG4:ae0 unit 0 family ethernet-switching vlanmembers Infraset interfaces RSNG4:ae0 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces RSNG4:ae0 unit 0 family ethernet-switching vlanmembersWikimediaset interfaces RSNG4:ae0 unit 0 family ethernet-switching vlanmembers SQLset interfacesRSNG4:ae0unit0 familyethernet-switchingvlanmembersStorage-POD1set interfaces RSNG4:ae0 unit 0 family ethernet-switching vlanmembersExchange-Cluster

set interfaces RSNG4:ae0 unit 0 family ethernet-switching vlanmembers SharePointset interfacesRSNG4:ae0unit0familyethernet-switchingvlanmembersSecurity-Mgmtset interfaces RSNG4:ae0 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces RSNG4:ae0 unit 0 family ethernet-switching vlanmembers VM-FTset interfaces RSNG4:ae0 unit 0 family ethernet-switching vlanmembersRemote-Access

NOTE: In this configuration, two node devices (N6 and N7) are part ofNodegroupRSNG4.FourXLEportsareconfigured inaLAGasRSNG4:ae0with theLACPactiveprotocol.RSNG4:ae0 isconfiguredasa trunkcarryingmultiple VLANs.

TheMetaFabric 1.0 solutionalsoutilizes the IBMPureFlexSystemChassiswith the 10-Gb

Ethernet CNA I/OModule (in POD2). A short overviewof the operation and configuration

of thismodule is required. Figure64onpage 181 shows thePOD2network topologyutilizing

the IBM Pure Flex System Chassis with the 10-Gb Ethernet CNA I/OModule.



Figure 64: POD 2 Topology Using the IBM Pure Flex SystemChassis withthe 10-Gbps CNA I/OModule

EXT Ports 1 and 2 of I/O Modules 1 and 2 are connected to each other, respectively. This

creates an interswitch link (ISL) between the two I/Omodules. The ISL creation enables

both I/Omodules to act as a single switch. EXT ports 11, 12, and 16 are connected to the

QFX3000-MQFabric PODs. POD2 also has an RSNG node group that is connected to

servers. Figure 64 on page 181 shows an example of three RSNG node groups in a

QFX3000-MQFabric systemconnectedtoan IBMPureFlexSystemchassis.Thediagram

above features the IBM Pure Flex System chassis with a 10-Gb CNA I/OModule. This

configuration was only used for Compute Node 1. Configuration details for the compute

node connected to POD2 are below.

Configuring the 10Gb CNAModule Connections

To configure the IBM 10Gb CNAmodule connectivity to POD 2, follow these steps:

1. Configure the CNA Fabric Switch I/Omodule on the IBMPure Flex System 10-GbCNA

I/OModule.

interface port INTA1taggingexit

!interface port INTA2rmontaggingexit






!interface port EXT1taggingpvid 4094exit

!interface port EXT2taggingpvid 4094exit







!vlan 101enablename "Infra"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

!vlan 102enablename "SharePoint"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

!vlan 103enablename "WikiMedia"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

!vlan 104enable



name "Exchange"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

!vlan 105enablename "SQL"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

!vlan 106enablename " Vmotion"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

!vlan 107enablename "FT"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

!vlan 108enablename "Storage-iSCSI"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

!vlan 800enablename "VDCMgmt"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

!vlan 801enablename "Security-Mgmt"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

!vlan 4094enablename "VLAN 4094"member EXT1-EXT2





!interface port INTA5lacpmode active



lacp key 1005!interface port EXT1lacpmode activelacp key 200







!interface port EXT16lacpmode activelacp key 1000!!

!vlag enablevlag tier-id 10vlag isl vlan 4094vlag isl adminkey 200vlag adminkey 1000 enablevlag adminkey 1001 enablevlag adminkey 1002 enablevlag adminkey 1003 enablevlag adminkey 1004 enablevlag adminkey 1005 enable

2. Configure QFX3000-MQFabric System connectivity to the IBM Pure Flex System

10-Gb CNA I/OModule.

[edit]set interfaces interface-range IBM-FLEX-1-10G-CNA-IO-1-2-VLAGmember"n1:xe-0/0/[24-27]"

set interfaces interface-range IBM-FLEX-1-10G-CNA-IO-1-2-VLAGmember"n2:xe-0/0/[30-31]"



set interfaces interface-range IBM-FLEX-1-10G-CNA-IO-1-2-VLAGether-options802.3adRSNG2:ae0

set interfaces RSNG2:ae0 description IBM-FLEX-1-10G-CNAset interfaces RSNG2:ae0 unit 0 family ethernet-switching port-mode trunkset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers MGMTset interfacesRSNG2:ae0unit0 familyethernet-switchingvlanmembersStorage-POD2set interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers Infraset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers SQLset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers SharePointset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembersExchange-cluster

set interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembersWikimediaset interfacesRSNG2:ae0unit0familyethernet-switchingvlanmembersSecurity-Mgmtset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers Vmotionset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembers VM-FTset interfaces RSNG2:ae0 unit 0 family ethernet-switching vlanmembersRemote-Access

set vlans Exchange vlan-id 104set vlans Exchange-cluster vlan-id 109set vlans Infra vlan-id 101set vlans MGMT vlan-id 800set vlans Remote-Access vlan-id 810set vlans SQL vlan-id 105set vlans SQL l3-interface vlan.105set vlans Security-Mgmt vlan-id 801set vlans SharePoint vlan-id 102set vlans SharePoint l3-interface vlan.102set vlans Storage-POD2 vlan-id 208set vlans Storage-POD2 l3-interface vlan.208set vlans VM-FT vlan-id 107set vlans Vmotion vlan-id 106set vlansWikimedia vlan-id 103

Verification

The following verification commands (with sample output) can be used to confirm the

configuration of compute and compute switching resources in the data center.

Results

1. Verify POD1 internal switch VLAN status.

POD1-Flex1-40G-IO-1> show vlan

VLAN Name Status MGT Ports---- -------------------------------- ------ --- -------------------------1 Default VLAN ena dis INTA1-INTB14 EXT1-EXT16101 INFRA ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT7102 SharePoint ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT7103 WM ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT7104 EXCHANGE ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT7105 SQL ena dis INTA1-INTA5 INTB1-INTB5



EXT1-EXT3 EXT7106 Vmotion ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT7107 VM-FT ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT7108 Storage-iSCSI ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT7109 Exchange DAG ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT7800 VDC Mgmt ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT7801 Security-MGMT ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT7900 Tera-VM ena dis INTA1-INTA5 INTB1-INTB5 EXT1-EXT3 EXT74094 VLAN 4094 ena dis EXT1 EXT24095 Mgmt VLAN ena ena EXTM MGT1

2. Verify POD2 internal switch VLAN status.

POD2-Flex1-10G-IO-1> show vlan

VLAN Name Status MGT Ports---- -------------------------------- ------ --- -------------------------1 Default VLAN ena dis INTA1-INTA14 EXT1 EXT2 EXT11-EXT16101 Infra ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16102 SharePoint ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16103 WikiMedia ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16104 Exchange ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16105 SQL ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16106 vMotion ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16107 FT ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16108 VLAN 108 dis dis empty109 Exchange DAG ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16208 Storage - iSCSI ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16800 VDC Mgmt ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16801 Security-Mgmt ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT16900 Tera-VM ena dis INTA1-INTA5 EXT1 EXT2 EXT11-EXT164094 VLAN 4094 ena dis EXT1 EXT24095 Mgmt VLAN ena ena EXTM MGT1

Virtualization

• Virtualization Overview on page 187

• Configuring LACP on page 190



• Configuring VMware Clusters, High Availability, and Dynamic Resource

Scheduler on page 193

• Configuring VMware Enhanced vMotion Compatibility on page 197

• Mounting Storage Using the iSCSI Protocol on page 200

• Configuring Fault Tolerance on page 201

• Configuring VMware vMotion on page 203

Virtualization Overview

In theMetaFabric 1.0 solution, all compute nodes are installed into a virtual environment

featuring the VMware ESXi 5.1 operating system. VMware ESXi provides the foundation

for building a reliable data center. VMware ESXi 5.1 is the latest hypervisor architecture

fromVMware. ESXi, vSphere client, and vCenter are components of vSphere. ESXi server

is the most important part of vSphere. ESXi is the virtualization server. All the virtual

machines or Guest OS are installed on the ESXi server.

To install, manage, and access those virtual servers which sit above the ESXi server, you

will need another part of the vSphere suite called vSphere client or vCenter. The vSphere

client allows administrators to connect to ESXi servers and access or manage virtual

machines, and is used from the clientmachine to connect to the ESXi server andperform

management tasks.

The VMware vCenter server is similar to the vSphere client, but it is a server with even

more power. The VMware vCenter server is installed on aWindows or Linux server. In this

solution, the vCenter server is installed on aWindows 2008 server that is running as a

virtual machine (VM). The VMware vCenter server is a centralizedmanagement

application that lets youmanage virtual machines and ESXi hosts centrally. VMware

vSphere client is used to access vCenter Server and ultimately manage ESXi servers

(Figure 65 on page 187). VMware vCenter server is compulsory for enterprises to have

enterprise features such as vMotion, VMwareHighAvailability, VMwareUpdateManager,

and VMware Distributed Resource Scheduler (DRS). For example, you can easily clone

an existing virtual machine by using vCenter server. vCenter is another important part of

the vSphere package.

Figure65:VMwarevSphereClientManagesvCenterServerWhich inTurnManages Virtual Machines in the Data Center

In Figure66onpage 188, all the computenodesarepart of adata center and theVMware

HA Cluster is configured on compute nodes. All compute nodes are running ESXi 5.1 OS,

which is a host operating system to all the data center VMs running business-critical

applications. With vSphere Client, you can also access ESXi hosts or the vCenter Server.

The vSphere Client is used to access the vCenter Server andmanage VMware enterprise

features.



A vSphere Distributed Switch (VDS) functions as a single virtual switch across all

associatedhosts (Figure66onpage 188). This enables you to set network configurations

that span across all member hosts, allowing virtual machines to maintain a consistent

network configuration as they migrate across multiple hosts. Each vSphere Distributed

Switch is a network hub that virtualmachines can use. A vSphere Distributed Switch can

forward traffic internally between virtual machines or link to an external network by

connecting to physical Ethernet adapters, also known as uplink adapters. Each vSphere

DistributedSwitch can also have one ormore dvPort groups assigned to it. dvPort groups

groupmultiple ports under a common configuration and provide a stable anchor point

for virtual machines connecting to labeled networks. Each dvPort group is identified by

anetwork label,which is unique to thecurrentdata center. VLANsenablea singlephysical

LAN segment to be further segmented so that groups of ports are isolated from one

another as if theywere on physically different segments. The standard is 802.1Q. A VLAN

ID, which restricts port group traffic to a logical Ethernet segment within the physical

network, is optional.

Figure 66: VMWare vSphere Distributed Switch Topology

VMware vSphere distributed switches can be divided into two logical areas of operation:

thedataplaneand themanagementplane.Thedataplane implementspacket switching,

filtering, and tagging. Themanagementplane is thecontrol structureusedby theoperator

to configure data plane functionality from the vCenter Server. The VDS eases this

management burden by treating the network as an aggregated resource. Individual

host-level virtual switches are abstracted into one large VDS spanningmultiple hosts at

the data center level. In this design, the data plane remains local to each VDS but the

management plane is centralized.



The first step in configuration is to create a vSphere distributed switch on a vCenter

Server. After you have created a vSphere distributed switch, youmust add hosts, create

dvPort groups, and edit vSphere distributed switch properties and policies.

With thedistributedswitch feature,VMwarevSpheresupportsprovisioning, administering,

andmonitoring of virtual networking across multiple hosts, including the following

functionalities:

• Central control of thevirtual switchport configuration, port groupnaming, filter settings,

and so on.

• LinkAggregationControlProtocol (LACP) thatnegotiatesandautomatically configures

link aggregation between vSphere hosts and access layer switches.

• Network health-check capabilities to verify vSphere with the physical network

configuration.

Additionally, the distributed switch functionality supports (Figure 66 on page 188):

• Distributed port—A port on a vSphere distributed switch that connects to a host’s

VMkernel or to a virtual machine’s network adapter.

• Distributed virtual port groups (DVPortgroups)—Port groups that specify port

configuration options for eachmember port. DVportgroups is a set of DV ports.

Configuration is inherited from dvSwitch to dvPortgroup.

• Distributed virtual uplinks (dvUplinks)—dvUplinks provide a level of abstraction for

the physical NICs (vmnics) on each host.

• PrivateVLANs(PVLANs)—PVLANsupport enablesbroader compatibilitywithexisting

networking environments using the technology.

Figure 67: VMware vSphere Distributed Switch Topology



Figure 67 on page 189 shows an illustration of two compute nodes running ESXi 5.1 OS

with multiple VMs deployed on the ESXi hosts. Notice that two physical compute nodes

are running VMs in this topology, and the vSphere distributed switch (VDS) is virtually

extended across all ESXi hostsmanaged by the vCenter server. The configuration of VDS

is centralized to the vCenter Server.

A LAG bundle is configured between the access switches and ESXi hosts. As mentioned

in the compute node section, an RSNG configuration is required on the QFX3000-M

QFabric systems.

ESXi 5.1 supports LACP protocol for the LAG, which can be enabled by connecting the

vCenter Server Web GUI only.

NOTE: Link Aggregation Control Protocol (LACP) can only be configured viathe vSphereWeb Client.

Configuring LACP

To enable or disable LACP on an uplink port group:

NOTE: All port groups using the Uplink Port Group enabled with LACPmusthave the load-balancing policy set to IP hash load balancing, network failuredetection policy set to link status only, and all uplinks set to active.

1. Log in to the vCenter Server Web on port 9443.

Figure 68: Log In to vCenter Server

2. Select vCenter under the Home radio button from the left tab.



Figure 69: vCenterWeb Client

3. Click Networking under vCenter on the left side.

Figure 70: Click Networking

4. Locate anUplinkPortGroup in the vSphereWebClient. To locate anuplink port group:

a. Select a distributed switch and click the Related Objects tab.



Figure 71: Click Related Objects

b. Click Uplink Port Groups and select an uplink port group from the list.

Figure 72: Click Uplink Ports and Select a Port

5. Select the dvSwitch-DVUplinks and click settings from the Actions tab.

6. Click Edit.

7. In the LACP section, use the drop-down box to enable or disable LACP.

Figure 73: Enable LACPMode

8. When you enable LACP, a Mode drop-downmenu appears with these options:

• Active—The port is in an active negotiating state, in which the port initiates

negotiations with remote ports by sending LACP packets.



• Passive—The port is in a passive negotiating state, in which the port responds to

LACP packets it receives but does not initiate LACP negotiation.

Set this option to passive (disable) or active (enable). The default setting is passive.

NOTE: Step 8 is optional.

9. ClickOK.

Configuring VMware Clusters, High Availability, and Dynamic Resource Scheduler

VMware clusters enable the management of multiple host systems as a single, logical

entity, combining standalone hosts into a single virtual devicewith pooled resources and

higher availability. VMware clusters aggregate the hardware resources of individual ESX

Server hosts but manage the resources as if they resided on a single host. Now, when

you power on a virtual machine, it can be given resources from anywhere in the cluster,

rather than from a specific physical ESXi host.

VMware high availability (HA) allows virtual machines running on specific hosts to be

restartedautomatically usingother host resources in the cluster in the caseof host failure.

VMware HA continuously monitors all ESX Server hosts in a cluster and detects failures.

The VMware HA agent placed on each host maintains a heartbeat with the other hosts

in the cluster. Each server sends heartbeats to theother servers in the cluster at 5-second

intervals. If any servers loseheartbeatover three consecutiveheartbeat intervals, VMware

HA initiates the failover action of restarting all affected virtual machines on other hosts.

VMware HA also monitors whether sufficient resources are available in the cluster at all

times in order to be able to restart virtual machines on different physical host machines

in theevent of host failure. Safe restart of virtualmachines ismadepossibleby the locking

technology in the ESX Server storage stack, which allowsmultiple ESX Server hosts to

have simultaneous access to the same virtual machine files.

VMwareDynamicResourceScheduler (DRS)automaticallyprovides initial virtualmachine

placementandmakesautomatic resource relocationandoptimizationdecisionsashosts

are added or removed from the cluster. DRS also optimizes based on virtual machine

load, managing resources in events where the load on individual virtual machines goes

up or down. VMware DRS also makes cluster-wide resource pools possible.

For more information on configuration of VMware HA clusters, see:

VMware vSphere 5.1 HA Documentation

TheMetaFabric 1.0 solution utilized VMware clusters in both POD1 and POD2. Below are

overview screenshots that illustrate the use of clusters in the solution.

TheMetaFabric 1.0 solution test bedcontains three clusters: Infra (Figure 74onpage 194),

POD1(Figure75onpage 194),andPOD2(Figure76onpage 194).All clustersareconfigured

with HA and DRS.



http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.avail.doc%2FGUID-7525F8DD-9B8F-4089-B020-BAA4AC6509D2.html&resultof=%2522%2566%2561%2575%256c%2574%2522%20%2522%2574%256f%256c%2565%2572%2561%256e%2563%2565%2522%20%2522%2574%256f%256c%2565%2572%2522%20%2522%2563%256f%256e%2566%2569%2567%2575%2572%2569%256e%2567%2522%20%2522%2563%256f%256e%2566%2569%2567%2575%2572%2522

Figure 74: Infra Cluster Hosts Detail

Figure 75: POD1 Cluster Hosts Detail

Figure 76: POD2 Cluster Hosts Detail

The Infra cluster (Figure 77 on page 195) is running all VMs required to support the data

center infrastructure. The Infra cluster is hosted on two standalone servers (IBM System

x3750M4). The VMs hosted on the Infra cluster are:

• Windows 2K8 Server with vCenter Server VM

• Windows 2K8 domain controller VM

• Windows 2K8 SQL database server VM

• Junos Space Network Director

• Remote Secure Access (SA)

• Firefly Host Management (also referred to as vGWManagement)

• Firefly Host SVM – Hosts (also referred to as vGWSVM – Hosts)

• Windows 7 VM - For NOC (Jump station)



Figure 77: INFRA Cluster VMs

The POD1 cluster (Figure 78 on page 196) hosts the VMs that run all enterprise

business-critical application in the test bed. POD1 is hosted on one IBM Flex pass-thru

chassis and one 40-Gb CNAmodule chassis. POD1 contains the following

applications/VMs:

• Windows Server 2012 domain controller

• Exchange Server 2012 CAS



• Exchange Mailbox server



• MediaWiki Server

• vGWSVM - All compute nodes



Figure 78: POD1 Cluster

The POD2 cluster (Figure 79 on page 196) hosts the VMs that run all enterprise

business-critical applications in the test bed. POD2 has one IBM Flex pass-thru chassis

and one 10-Gb CNAmodule chassis. POD2 contains the following applications/VMs:

• Windows Server 2012 secondary domain controller

• SharePoint Server (Web-front end, six total VMs)

• SharePoint Application Server (two of these)

• SharePoint Database Server

• vGWSVM – All compute nodes

Figure 79: POD2 Cluster



Configuring VMware Enhanced vMotion Compatibility

VMware Enhanced vMotion Compatibility (EVC) configures a cluster and its hosts to

maximize vMotion compatibility. Once enabled, EVC will ensure that only hosts that are

compatible with those in the cluster can be added to the cluster. This solution uses the

Intel Sandy Bridge Generation option for enhanced vMotion compatibility that supports

the baseline feature set.

To configure a vSphere distributed switch on a vCenter server, perform following steps.

• Add a vSphere distributed switch

• Add hosts to a vSphere distributed switch

• Add a distributed port group (dvPG) configuration

For more details on configuration of VMware EVC, see:

VMware vSphere 5.1 Documentation - Enable EVC on an Existing Cluster

In the MetaFabric 1.0 solution, EVC is configured as directed in the link provided. A short

overview of the configuration follows.

Each ESXi host in the POD hosts multiple VMs and is part of a different port group. VMs

running on the PODs include Microsoft Exchange, MediaWiki, Microsoft SharePoint,

MySQL database, and Firefly Host (VM security). Because traffic is flowing to and from

many different VMs, multiple port groups are defined on the distributed switch:

• Infra = PG-INFRA-101

• SharePoint = PG-SP-102

• MediaWiki = PG-WM-103

• Exchange = PG-XCHG-104

• MySQL Database for SharePoint = PG-SQL-105

• vMotion = PG-vMotion-106

• Fault Tolerance = PG-Fault Tolerance-107

• Exchange Cluster = PG-Exchange-Cluster-109

• iSCSI POD1 = PG-STORAGE-108

• iSCSI POD2 = PG-STORAGE-208

• Network MGMT = PG-MGMT-800

• Security (vGW) = PG-Security-801

• Remote Access = PG-Remote-Access-810

These port groups are configured as shown in Figure 80 on page 198. In this scenario, a

port group naming convention was used to ease identification andmapping of VM and

its function (for example, Exchange, SharePoint) to a VLAN ID. For instance, one VM is

connected to PG104 running an Exchange application while another VM is is connected



http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.vsphere.vcenterhost.doc/GUID-DE575FB9-24AE-4453-966B-5340B8147FC9.html?resultof=%2522%2545%2556%2543%2522%2520%2522%2565%2576%2563%2522%2520

to PG103 running a MediaWiki application on the same ESXi host. Port group naming

convention is also used in this scenario to identify theVLAN ID towhich the host belongs.

For instance, PG-XCHG-104 is using VLAN ID 104 on the network. (The 104 in the name

is the same as the host VLAN ID.) The use of different port groups and VLANs enables

the use of vMotion, which in turn enables fault tolerance in the data center.

Figure 80: Port Groups

NIC teaming is also deployed in the solution. NIC teaming is a configuration of multiple

uplink adapters that connect to a single switch to form a team. A NIC team can either

share the load of traffic between physical and virtual networks among some or all of its

members, or provide passive failover in the event of a hardware failure or a network

outage. All the port groups (PG) except for iSCSI protocol storage groups are configured

with a NIC teaming policy for failover and redundancy. All the compute nodes have four

active adapters as dvUplink in the NIC teaming policy. This configuration enables load

balancing and resiliency. The IBM Pure Flex Systemwith a 10-Gb CNA card has two

network adapters on each ESXi host. Consequently, that system has only two dvUplink

adaptersperESXihost. Figure81onpage 199 isanexampleofoneportgroupconfiguration.

Other port groups are configured similarly (with the exception being the storage port

group).



Figure 81: Port Group and NIC Teaming Example

Figure 82: Configure Teaming and Failover



NOTE: An exception to the use of NIC teaming is an iSCSI port group. TheISCSI protocol doesn’t support multi-channeling or bundling (LAG). Whendeploying iSCSI, insteadofconfiguring fouractivedvUplinks, asingledvUplinkshould be used. In this solution, QFX3000-MQFabric POD1 uses one portgroup (PG-storage-108) and QFX3000-MQFabric POD2 uses another portgroup (PG-storage-208). These port groups are connected to the storagearray utilizing the iSCSI protocol. Figure 82 on page 199 shows the iSCSI portgroup (PG-storage-108). Port group storage 208 is configured in the sameway.

TheVMkernel TCP/IPnetworking stack supports iSCSI, NFS, vMotion, and fault tolerance

logging. The VMkernel port enables these services on the ESX server. Virtual machines

run their own system TCP/IP stacks and connect to the VMkernel at the Ethernet level

through standard and distributed switches. In ESXi, the VMkernel networking interface

provides network connectivity for the ESXi host and handles vMotion and IP storage.

Moving a virtual machine from one host to another is calledmigration. VMware vMotion

enables the migration of active virtual machines with no down time.

Management of iSCSI, vMotion, and fault tolerance is enabled by the creation of four

virtual kernel adapters. These adapters are bound to their respective distributed port

group. Formore informationoncreatingandbinding virtual kernel adapters todistributed

port groups, see:

http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.vsphere.networking.doc/GUID-59DFD949-A860-4605-A668-F63054204654.html

Mounting Storage Using the iSCSI Protocol

Tomount the storage using iSCSI protocol, perform following steps:

• Create a single VMkernel adapter for iSCSI.

• Change the port group policy for the iSCSI VMkernel adapter.

• Bind iSCSI adapters with VMkernel adapters.

• Set up Jumbo frames with iSCSI.

• Configure dynamic discovery addresses for iSCSI adapters.

• Re-scan storage on iSCSI adapters.

NOTE: The ESXi hostmust have permission to access the storage array. Thisis discussed further in the storage section of this guide.

For informationaboutconfiguringandmountingof iSCSI storageconnection toavSwitch

(either vSwitch or distributed switch), see:

Configuring and Troubleshooting iSCSI Storage



http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.vsphere.networking.doc/GUID-59DFD949-A860-4605-A668-F63054204654.html

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008083

Figure 83 on page 201 shows an example of an ESXi host deployed in POD1. The port

gropus PG-Storage-108 and PG-Storage-208 dvPG have been created for POD1 and

POD2, respectively. (The example shows PG-Storage-108.) VMkernel is configured to

use the 172.16.8.0/24 subnet for hosts in POD1 and the 172.20.8.0/24 subnet for hosts in

POD2 to bind with the respective storage port group to access the EMC storage.

• EMC storage iSCSI IP for POD1 = 172.16.8.1 and 172.16.8.2

• EMC storage iSCSI IP for POD2 = 172.20.8.1 and 172.20.8.2

As mentioned earlier, the iSCSI protocol doesn’t support multichannel (LAG) but can

support multipath; you will see only one physical interface bind with the storage port

group.Toachievemultipath, separate storageport groupandnetwork subnetare required

to access EMC storage as a backup link.

Figure 83: POD1 PG-STORAGE-108 Created for iSCSI

Configuring Fault Tolerance

VMware vSphere fault tolerance provides continuous availability for virtual machines by

creating andmaintaining a secondary VM that is identical to, and continuously available

to replace, the primary VM in the event of a failure. The feature is enabled on a per virtual

machine basis. This virtual machine resides on a different host in the cluster, and runs in

virtual lockstep with the primary virtual machine. When a failure is detected, the second

virtual machine takes the place of the first one with the least possible interruption of

service. Because the secondary VM is in virtual lockstep with the primary VM, it can take

overexecutionatanypointwithout interruption, therebyproviding fault tolerantprotection.



Figure 84: VMware Fault Tolerance

TheprimaryandsecondaryVMscontinuouslyexchangeheartbeats. This exchangeallows

thevirtualmachinepair tomonitor the statusofoneanother toensure that fault tolerance

is continually maintained. A transparent failover occurs if the host running the primary

VM fails, inwhich case the secondary VM is immediately activated to replace the primary

VM. A new secondary VM is started and fault tolerance redundancy is reestablished

within a few seconds. If the host running the secondary VM fails, it is also immediately

replaced. In either case, users experience no interruption in service and no loss of data.

VMware vSphere HAmust be enabled before you can power on fault tolerant virtual

machines or add a host to a cluster that already supports fault tolerant virtualmachines.

Only virtual machines with a single vCPU are compatible with fault tolerance.

For configuration instructions for VMware fault tolerance, see:

Preparing Your Cluster and Hosts for Fault Tolerance

The MetaFabric 1.0 solution test bed features VMware fault tolerance

(Figure85onpage202).Thiswas testedaspartof thesolutionon theportgroup“PG-Fault

tolerance-107”. VMkernel is bound to this port group. Once fault tolerance is enabled on

a VM, a secondary VM is automatically created.

Fault tolerance is also enabled onWindows domain controller VM (running on one

compute node in the POD1 cluster).

Figure 85: VMware Fault Tolerance on POD1



http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.avail.doc_50%2FGUID-38A72660-296D-49BB-BE6E-E1EB948FD67B.html

Configuring VMware vMotion

The VMware VMotion feature, part of VirtualCenter, allows you tomigrate running virtual

machines from one physical machine to another with no perceivable impact to the end

user (Figure86onpage203). YoucanuseVMotion toupgradeand repair serverswithout

any downtime or disruptions and also to optimize resource pools dynamically, resulting

inan improvement in theoverall efficiencyofadatacenter. Toensure successfulmigration

andsubsequent functioningof thevirtualmachine, youmust respect certain compatibility

constraints. Complete virtualization of all components of amachine, such as CPU, BIOS,

storage disks, networking, andmemory, allows the entire state of a virtual machine to

be captured by a set of data files. Therefore, moving a virtual machine from one host to

another is nothing but data transfer between two hosts.

Figure 86: VMware vMotion Enables Virtual MachineMobility

VMware vMotion benefits data center administrators in critical situations, such as:

• Hardware maintenance: VMotion allows you to repair or upgrade the underlying

hardware without scheduling any downtime or disrupting business operations.

• Optimizing hardware resources: VMotion lets youmove virtual machines away from

failing or underperforming hosts.

• This can be done automatically in combination with VMware Distributed Resource

Scheduler (DRS). VMware DRS continuously monitors utilization across resource

pools and allocates resources among virtual machines based on current needs and

priorities. When virtual machine resources are constrained, DRSmakes additional

capacity available by migrating live virtual machines to a less utilized host using

VMotion.

The requirements for vMotion include:

• Datastore compatibility: The source and destination hosts must use shared storage.

You can implement this shared storage using a SAN or iSCSI. The shared storage can

use VMFS or shared NAS. Disks of all virtual machines using VMFSmust be available

to both source and target hosts.

• Network compatibility: VMotion itself requiresaGigabit Ethernetnetwork.Additionally,

virtual machines on source and destination hosts must have access to the same

subnets, implying that network labels for each virtual Ethernet adapter shouldmatch.

You should configure these networks on each ESX host.

• CPU compatibility: The source and destination hosts must have compatible sets of

CPUs.



VMwarevMotion isconfiguredonallMetaFabric 1.0hosts (Figure87onpage204).VMware

vMotion is using a separate port group called PG-vMotion-106, and VMkernel is bound

to this port group. Network and storage is unique on all hosts, which is a requirement for

vMotion.OncevMotionconfiguration is completed, activeVMswill bemovedanyavailable

host where resources are free. DRS can also kick in the vMotion feature if one of the ESX

hosts shows high resource utilization (CPU, memory). You can also manually trigger

vMotion if the need arises to move a VMwithin the data center.

Figure 87: VMware vMotion Configured in the Test Lab

For more information on configuration of VMware vMotion, see:

Creating a VMkernel port and enabling vMotion on an ESXi/ESX host

EMC Storage Overview

• Configuring EMC Storage on page 204

• Configuring FAST Cache on page 205

Configuring EMC Storage

The MetaFabric 1.0 solution features EMC VMX series storage controllers. The EMC VNX

series implements a modular architecture that integrates hardware components for

Block, File, and Object with concurrent support for native NAS, Internet Small Computer

System Interface (iSCSI), Fiber Channel, and Fibre Channel over Ethernet (FCoE)

protocols. The VNX series is based on Intel Xeon-based PCI Express 2.0 processors and

delivers File (NAS) functionality via two to eight Data Movers and Block (iSCSI, FCoE,

and FC) storage via dual storage processors using a full 6-Gb/s SAS disk drive topology.

Configuring EMC storage for control station can be done by the vendor only. Once you

configuremanagement for EMC, it is accessedusing theEMCUnisphere tool (viaHTTPS).



http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2054994

Configuring EMC FAST Cache

The solution was tested with EMC Fully Automated Storage Tiering (FAST) Cache. A

caching tier is a large capacity secondary cache using enterprise flash drives positioned

between the storage processor’s DRAM-based primary cache and hard disk drives. This

feature is calledFASTCache.At a system level, FASTCachehelpsmake themost efficient

use of flash drive capacity. FAST Cache does this by using flash drives for the most

frequently accessed data in the storage system instead of dedicating flash drives to a

particular application. FAST Caches are also enabled on storage pools and allow you to

leverage the lower response timeandbetter IOPSof flash driveswithout dedicating flash

drives to a specific application. Because of this implementation, flash drives need not be

dedicated to specific applications. Instead, their performance is leveraged in the caching

tier, improving performance for any application accessing the storage array. FAST Cache

adjusts to a hot spot anywhere in the array so you no longer need to analyze specific

application requirements. It providesbetter performance toall applications in the storage

systems while using fewer dedicated flash drives. A set of flash drives is selected either

automatically by the storage system or manually by the user. By default, after you have

installed the FAST Cache enabler, FAST cache is enabled on new RAID group LUNs and

storage pools. When a FAST cache has been created, the storage system automatically

promotes frequently accessed chunks of data to the FAST cache to improve application

performance.

Configuring FAST Cache

To create a FAST Cache, click the FAST Cache tab in the Storage System Properties

window to view FASTCache information (Figure 88 on page 205). If the FASTCache has

not been created on the storage system, the Create button in the bottom of the dialog

box is enabled. The Destroy button is enabled when the FAST Cache has been created.

Fast Cache is enabled on EMC VNX5500 storage.

Figure88:EMCFASTCacheConfiguration(SelectSystem,thenPropertiesin the Drop-Down)

FAST Cache can be created in certain configurations, depending on the storage system

model, and number and size of flash drives installed in the storage system. These criteria

are used to present you with the available options for your configuration. For example,



if an insufficient numberof flashdrivesareavailable,Unispheredisplaysanerrormessage

and FAST Cache cannot be created. The number of flash drives can also bemanually

selected. The bottom portion of the screen shows the flash drives that will be used for

creatingFASTCache.Youcanchoose thedrivesmanuallybyselecting theManualoption.

If the LUN is created in a RAID group, you can enable or disable FAST Cache at the LUN

level. It is enabled by default if the FASTCache enabler is installed on the storage system.

In this example (Figure 89 on page 206), Fast Cache is enabled with four disks.

Figure 89: EMC FAST Cache Configuration

Configuring Storage Pools

A storage pool is a general term used for RAID groups and pools. A pool is a set of disks,

all with the same redundancy (RAID 5, RAID 6, and RAID 1/0) on which you create one

or more Logical Unit Numbers (LUNs). Two storage pools are created: Exchange DB and

Exchange Logs with RAID 1/0 type redundancy near-line SAS (NL SAS) drives. NL SAS

drives are almost 1-TBdrive capacity. Storage pools are created by following these steps:

1. Log in with the EMC Unisphere tool to access EMC storage.

2. Select a storage system.

3. Go to Storage >Storage Configuration >Storage Pools. In the Pools tab, click Create.



Figure 90 on page 207 shows an example of the EMC Unisphere management tool. The

storage pool selected is Pool 1– Exchange-DB.

Figure 90: Pool 1 - Exchange-DB

Figure 91 on page 207 shows the properties of the selected storage pool (Pool 1 –

Exchange-DB). This screen shows the physical and virtual capacity of the storage pool.

The top tabs also enable viewing andmodification of disk assignment, advanced

properties, and storage tiering.

Figure 91: Selected Storage Pool Properties

Figure92onpage208showsthecontentsof theDisks tab(under storagepoolproperties).

Fromthis screen, additional disks canbeadded to the storagepool. This tabalsodisplays

the physical and operational properties of each disk assigned to the storage pool.



Figure 92: Storage Pool Disks Properties

In the storagepool propertiesAdvanced tab (Figure93onpage208), youcanset thealert

threshold for the storage pool (the percentage utilization that will trigger an alarm) and

enable or disable FAST Cache.

Figure 93: Storage Pool Properties, Advanced Tab

Now thatwe have viewed the settings of an individual application storage pool, let’s look

at combining multiple application storage pools into an aggregated storage pool.

Figure 94 on page 209 shows the aggregated storage pool.



Figure 94: VM-Pool Selected

Figure 95 on page 209 shows the properties of the selected storage pool.

Figure 95: VM-Pool Properties

Figure 96 on page 210 shows disk membership of the aggregated storage pool.



Figure 96: VM-Pool Disk Membership

Configuring Logical Unit Numbers

Once the aggregate VMpool has been created, LUNsmust be defined. On a storage pool

(or aggregate storage pool), the LUN acts as the disk identifier for application or

server-specific storage. Different storage pools can be createdwith different application

and requirement-specific RAID (performance-oriented, availability-oriented, or a mix of

the two) and storage capacity. To create and allocate LUNs, follow these steps:

1. Select the VNX system using the Unisphere tool.

2. Select Storage, then LUN.

3. In the Create LUN dialog, under Storage Pool Properties:

• Select Pool.

• Select a RAID type for the LUN.

• For Pool LUNs, only RAID 6, RAID 5, and RAID 1/0 are valid. RAID 5 is the default

RAID type.

If available, the software populates the storage pool for the new LUNwith a list of pools

that have the specifiedRAID type, or displays thenameof the selectedpool. TheCapacity

section displays information about the selected pool. If there are no pools with the

specified RAID type, click New to create a new one.

1. In LUN Properties, select the Thin check box if you are creating a thin LUN.

2. Assign a User Capacity and ID to the LUN you want to create.

3. If youwant tocreatemore thanoneLUN, selectanumber inNumberofLUNs tocreate.



Formultiple LUNs, the software assigns sequential IDs to the LUNs as they are available.

For example, if you want to create five LUNs starting with LUN ID 11, the LUN IDs might

be 11, 12, 15, 17, and 18.

1. In LUNName, either specify a name or select to automatically assign LUN IDs as LUN

names. Choose one of the following:

a. Click Apply to create the LUNwith the default advanced properties, or

b. Click the Advanced tab to assign the properties yourself.

2. Assign optional advanced properties for the LUN:

a. Select a default owner (SP A or SP B) for the new LUN or accept the default value

of Auto.

b. Set the FAST tiering policy option.

3. Click Apply to create the LUN, and then click Cancel to close the dialog box. An icon

for the LUN is added to the LUNs view window.

The LUN created for Exchange DB is a single LUN in the Exchange storage Pool

(Figure 97 on page 212).



Figure 97: Exchange-DB-LUN Properties

Figure 98 on page 213 shows the LUN assigned to all the ESX hosts. This LUN is created

so that ESX hosts can access this LUN andmount it as a datastore.



Figure 98: LUN Created for All ESX Hosts

A pool was also created for use as a storage destination for Microsoft Exchange logs


Figure 99: The Selected PoolWas Created for MS Exchange Logs

Once the pool is created, the LUN can be created (Figure 100 on page 214).



Figure 100: Exchange Logs the LUN Created

Enabling Storage Groups

In events where you choose to connect multiple servers to the storage system, storage

groupsmust be configured and enabled. The Storage Groups option lets you place LUNs

into groups that are known as storage groups. These LUNs are accessible only to the

host that is connected to the storage group.

To create a storage group:

1. Select All Systems>VNX System.

2. Select Hosts>Storage group.

3. Create storage group, then clickOK to save changes and close the dialog box. You

can also click Apply to apply the changes without closing the dialog box.



NOTE: Once you enable Storage Groups for a storage system, any hostcurrently connected to the storage systemwill no longer be able to accessdata on the storage system. To the host, it will appear as if the LUNswereremoved. In order for the host to access the storage data, youmust addLUNs to Storage Group and then connect the host to the Storage Group.

Figure 101 on page 215 shows the properties screen of the ESX-StorageGroup created

in the MetaFabric test lab.

Figure 101: Example Storage Group PropertiesWindow

Once the storage group is created, LUNs can be added to the storage group




Figure 102: LUN Added to Storage Group

4. Youmust also add any ESXi hosts to the storage group if those hosts need to access

any data housed on the storage group (Figure 103 on page 217).



Figure 103: ESXi Hosts Added to Storage Group

5. If LUNs are already created, they can be added directly to the storage group by

navigating to the Storage tab, selecting the LUN, and clicking theAdd to storage group

button (Figure 104 on page 217).

Figure 104: Add LUNs to Storage Group

The prior storage sections havemoved you to a point where logical disks now exist

on the storage array. These disks are not formatted and are unusable by the operating



systems until they have been formatted andmounted. These operations are covered

in the next sections.

Configuring the Network File System

The VNX is a multiprotocol machine that provides access to data through the Network

File System (NFS) protocol. NFS is a client/server distributed file service that provides

file sharing in network environments. The NFS protocol enables the VNX to assume the

functions of an NFS server. NFS environments typically include:

• Native UNIX clients

• Linux clients

• Windows systems configured with third-party applications that provide NFS client

services

When a VNX is configured as an NFS server, file systems are mounted on a Data Mover

and a path to that file system from the Data Mover is exported. Exported file systems

are then available across the network and can bemounted by remote users.

NFS pools are created from the storage pool section of EMC Unisphere. In this example,

we are going to use a storage pool called NFS Pool (Figure 105 on page 218).

Figure 105: NFS Pool Properties

You will recall that a LUNmust be created on top of the storage pool. The LUN is used

as a logical disk identifier that enables the use of the storage (Figure 106 on page 219).



Figure 106: LUN Created on the New Storage Pool

Next, youmust define the NFS pool to enable ESXi hosts to access the storage. To do

this, follow these steps:

1. Select the VNX system in Unisphere.

2. Go to Storage.

3. Select the Storage Configuration.

4. Go to the Storage Pools for file.

5. Create the NFS Pool with storage capacity.

Figure 107onpage220shows theNFSpool properties once thepool hasbeencreated for

server access.



Figure 107: NFS Pool Properties

Once the NFS pool is created, youmust export the pool in order to make the file system

or directories available to NFS clients. To do this:

1. Select Storage >Shared Folders > NFS, and click Create.

2. From the Choose Data Mover list, select a Data Mover fromwhich to export the file

system.

3. From the File System list, select the file system or checkpoint that contains the

directory to export.

4. To export a subdirectory, add the rest of the path to the string in the field.

Figure 108 on page 221 shows the configuration of NFS export. Note that access to hosts

is assigned on a per-subnet basis and can be assigned as read-only, read/write, root

access, or operator access permissions.



Figure 108: NFS Export Configuration

Once you export NFS, the directories contained in the NFSwill be available for mounting

on the application servers.

Configuring VNX Snapshot Replicas

VNX Snapshots is a software feature that creates point in-time data copies. VNX

Snapshots are used for data backups, software development and testing, re-purposing,

data validation, and local rapid restores. VNXSnapshots do not consume large amounts

of pool capacity. As a result, this feature is recommended for use as front-line data

management and recovery.

To configure the snapshot replicas for LUN:

1. Select the VNX system in EMC Unisphere.

2. Go to Dataprotection > Snapshots.

3. Select theSnapshotConfigurationWizard fromthe right side (Figure 109onpage221).

Figure 109: Snapshot ConfigurationWizard

4. Click Next, then select the server host where the LUN is configured andmounted




Figure 110: Select Source Server

5. Click Next, then select the target VNX storage system (Figure 111 on page 222), then

click Next.

Figure 111: Select Snapshot Target



6. Select the source LUN (the LUN you wish to snapshot). In this example,

SharePoint-SQL-DB is used as the source LUN and added to the list

(Figure 112 on page 223). Once you have selected the source LUN, click Next.

Figure 112: Select Source LUNs

7. Finally, you can select the default settings or uncheck the Accept Snapshot overhead

values check box andmodify the default settings (Figure 113 on page 224). Once you

have the settings you want, click Next.



Figure 113: Select Snapshot Storage Overhead

8. Choose to create the Snapshot LUN or choose to not create the Snapshot LUN at the

current time. For this example,wecreated theSnapshot LUN(Figure 114onpage225).

Click Next.



Figure 114: ChooseWhen to Create LUN Snapshot

9. Choose to attach the snapshot now, or save it for later attachment

(Figure 115 on page 226). The recommendation is to assign the snapshot LUN to a

different server than the source LUN.



Figure 115: Assign Snapshot to a Server

10. Click Finish. Configuration of snapshot LUN is complete (Figure 116 on page 227).



Figure 116: Summary of SnapshotWizard Configuration

Load Balancing


Overview

This section provides the implementation details of the F5 load balancer deployed in the

MetaFabric 1.0 solution test lab. This section explains the following topics:

• Load-balancer topology

• Redundancy

• Link and network configuration

• VIP and server pool configuration

• Traffic flow

Topology

The topologyused in testing theF5 load-balancingelementof theMetaFabric 1.0 solution

is shown in Figure 117 on page 228. The MetaFabric 1.0 data center solution uses F5 to

load-balance traffic between servers. The solution testing featured twoVIPRIONC4480

hardware chassis; each chassis is configured with one B4300 blade for 10-Gigabit

connectivity. The VIPRION chassis are running this software package: BIG-IP 10.2.4 Build

591.0 Hotfix HF2 image.



Figure 117: Load Balancing Topology

In this solution, twoVIPRION systems are connected to the core switches using LAG. The

F5 systems are configured with virtual IPs (VIPs) and server pools to provide

load-balancing services to SharePoint, Wikimedia, and Exchange traffic. SharePoint,

Wiki, and Exchange servers are connected to POD switches on VLAN 102,103, and 104,

respectively. DSRmode is configured in F5 to bypass return traffic from server for all the

VIPs.

Configuring Redundancy

Each VIPRION system is configured as a cluster in this topology, although they can also

be configured as single devices. In this solution, the twoVIPRION systems are configured

as two clusters (one cluster per chassis), deployed in active/standbymode for

redundancy. This means that one cluster is active and processing network traffic, while

the other cluster is up and available to process traffic, but is in a standby state. If the

active cluster becomes unavailable, the standby cluster automatically becomes active,

and begins processing network traffic.

For redundancy, a dedicated failover link is configured between two VIPRION systems

as a LAG interface. Interfaces 1/1.3 and 1/1.4 in LAG are configured as failover links on

both systems. The following steps are required to configure redundancy (failover):

1. Create an interface trunk dedicated for failover.

2. Create a dedicated failover VLAN.

3. Create a self IP address and associate the self IP with the failover VLAN.



4. Define a unicast entry specifying the local and remote self IP addresses.

5. Define amulticast entry using the management interface for each VIPRION system.

For more information on redundancy and failover configuration of the F5 Load Balancer,

see:

http://support.f5.com/kb/en-us/solutions/public/11000/900/sol11939.html

For more information on redundant cluster configuration, see:

http://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/VIPRION_configuration_guide_961/clustered_systems_redundant.html

Configuring the Link and Network

The F5 load-balancing topology deployed in this solution features two LAG interfaces.

OneLAG isconfigured forexternal connectivity (receivesservice request toVIPsconfigured

on F5 from the Internet) andone LAG is configured for internal connectivity to the servers

connected to the PODs. Both the F5 systems are configured with external and internal

connectivity and those interfaces on both systemswill be in theUP state. Only the active

F5systemprocesses traffic - the standbyonlyprocesses traffic in caseswhere theprimary

experiences a failure.

Each LAG in F5 systemhas twomember links. Onemember link connects to Core Switch

1 and the other connects to Core Switch 2. MC-LAG is configured on the core switches.

To the F5, it appears that the LAG is connecting to a single system. LACP is used as

control protocol for creating LAG between F5 and core switches.

The following configurations are performed on both F5 systems to enable external

connectivity:

• Create a LAG named External on both F5 systems, and assign interfaces 1/1.5 and 1/1.6

as members of that LAG.

• Create VLAN 15 and name it External on both F5 systems and the VLAN assigned to

the External LAG.

• Configure a self IP address of 192.168.15.3 on the active F5 system, and 192.168.15.4 on

the standby F5 system for VLAN 15.

• Create a floating IP address 192.168.15.5 on the active F5 system. This floating IP

address is active on the active F5 cluster. If the active load balancer fails, this floating

IP address is used on new active load balancer

Configure internal connectivity with the following steps:

• Create a LAG named core-sw on both F5 systems, and assign interfaces 1/1.1 and 1/1.2

as members of that LAG.

• VLANs 102, 103, and 104 are named Core-Access, Wikimedia-Access, and

Exchange-Access, respectively. These VLANs were created on both the F5 systems

and the VLANs, and are assigned to the core-sw LAG. As per their Access names, the

SharePoint, Wikimedia, and Exchange servers are located in VLANs 102, 103, and 104,

respectively.



http://support.f5.com/kb/en-us/solutions/public/11000/900/sol11939.html

http://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/VIPRION_configuration_guide_961/clustered_systems_redundant.html

• Create a self IP address of 172.16.2.25, 172.16.3.25, and 172.16.4.25 for VLANs 102, 103,

and 104

Internal connections to the servers are configured as a Layer 2 connection through the

POD switches (that are connected to the core switches).

NOTE: For external connections, static routes are advertised from the coreswitch for VIPs configured in F5 for clients in the Internet to send requests tothe VIP for specific services like Exchange, Wikimedia, and SharePoint. Forthe static route, we configured a floating IP address created for “external”VLAN 192.168.15.5 as the next hop to reach the VIPs.When the active clusterfails, thenewactiveuses theconfigured floating IPaddress, sendsagratuitousARP for this floating IP address, and begins receiving traffic.

Configuring VIP and Server Pool

A virtual server IP address (VIP) is a traffic-management object on the F5 system that

is represented by an IP address and a service. Clients on an external network can send

application traffic to a virtual server, which then directs the traffic according to the VIP

configuration. Themain purpose of a virtual server is to balance traffic load across a pool

of servers on an internal network. Virtual servers increase the availability of resources for

processing client requests by monitoring server load and distributing the load across all

available servers for a particular service.

NOTE: In this solution testing, nPath routing (DSR, or Direct Server Return)is used to bypass the F5 for return path traffic from servers, routing trafficdirectly to the destination from the application servers.

It is recommended to use the nPath template in the F5 configuration GUI (Template and

wizards window) to configure VIP and server pools in DSRmode. The following link

provides greater detail regarding configuration of nPath in F5 systems:

http://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm_implementations_guide_10_1/sol_npath.html

In this solution, configure three VIP addresses 10.94.127.180, .181, and .182, and assign

server pools to these VIPs using the nPath template to service SharePoint, Exchange,

andWikimedia services.

The following tasks need to be completed in order to configure the BIG-IP system to use

nPath routing:

• Create a custom Fast L4 profile.

• Create a pool that contains the content servers.

• Defineavirtual serverwithport andaddress translationdisabledandassign thecustom

Fast L4 profile to it.

• Configure the virtual server address on each server loopback interface.



http://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm_implementations_guide_10_1/sol_npath.html

• Set the default route on your servers to the router’s internal IP address.

• Ensure that the BigIP configuration key connection.autolasthop is enabled.

Alternatively, on each content server, you can add a return route to the client.

• SharePoint VIP services are mapped 10.94.127.180:8080 (TCP port 8080).

• MediaWiki services are mapped to 10.94.127.182:80.

• Exchange services are mapped to:

• 10.94.127.181:993 (IMAP4)

• 10.94.127.181:443 (OutlookWeb Access)

• 10.94.127.181:0 (any port for RPC)

• 10.94.127.181:995 (POP3)

The following illustrations and steps explain the creation of nPath routing for IMAP4 for

exchange. These examples can be used to guide creation of nPath for other services by

substituting the VIP address and port number.

To configure nPath routing for IMAP4 and Exchange, follow these steps:

1. Using thenPath template createaVIP, server pool, andmonitor. This template creates

a Fast L4 profile and assigns it to the VIP address.

Figure 118: Configure nPath

2. The above window shows the configuration of nPath routing using the configuration

template.



a. Assign a unique prefix name (my_nPath_IMAP) for the F5 system to name the

server pool, monitor, and other objects.

b. To create the IMAP4 VIP as part of the Exchange service, specify the VIP address

10.94.127.181, TCP port 993.

c. This template gives a choice for creating a new server pool or using an existing

pool. By default, it creates a new server pool using the prefix name shown

(my_nPath_IMAP_pool). Add servers one at a time with the IP address and port

number, as shown.

d. This templategivesachoice for creatinganewmonitor or usinganexistingmonitor.

By default, it creates a newmonitor using the prefix name shown

(my_nPath_IMAP_TCP_pool). By default, it uses TCPmonitoring and the user can

change themonitoring type.Thedefault interval for thehealthcheck is 30seconds,

and the timeout value is 91 seconds.

e. ClickFinished to createnPath routing for an IMAPservicewithVIP 10.94.127.181:993.

NOTE: The default TCPmonitor, with no Send string or Receive stringconfigured, tests a service by establishing aTCP connectionwith the poolmember on the configured service port and then immediately closes theconnectionwithoutsendinganydataon theconnection.Thiscausessomeservices such as telnet and ssh to log a connection error, and fills up theserver logs with unnecessary errors. To eliminate the extraneous logging,youcanconfigure theTCPmonitor to send just enoughdata to theservice,or just use the tcp_half_openmonitor. Depending on your monitoringrequirements, youmight also be able tomonitor a service that expectsempty connections, such as tcp_echo (by using the default tcp_echomonitor). NOTE: Each server has four 10-Gb NIC ports connected to theQFX3000-MQFabric PODs as a data port for all VM traffic. Each systemisconnected toeachPODfor redundancypurposes.The IBMSystem3750is connected to POD1 using 4 x 10-Gigabit Ethernet. A second IBMSystem3750 connects to POD2 using 4 x 10-Gigabit Ethernet. The use of a LAGprovides switching redundancy in case of a POD failure.

3. Create or verify other objects (Pool, Profile, or Monitor) using the template created

with the VIP. As you can see, nPath routing created Fast L4 Profile as needed.



Figure 119: Verify Objects during nPath Configuration

4. Verify the VIP by selecting Virtual Servers under the Local Traffic tab. The VIP in this

example is namedmy_nPath_IMAP_virtual_server with an assigned IP address of

10.94.127.181 and TCP port 993. As per nPath requirement Performance (Layer 4), the

profile also known as Fast L4 profile is assigned to this VIP.

Figure 120: Configure and Verify VIP

Note that the SNAT pool is disabled for this VIP as per the nPath requirement. When

traffic enters the F5 system for this VIP, the F5 does not perform SNAT; it simply

forwards the traffic to the server as is without modifying the source or destination

address. This VIP simply performs load balancing and sends the traffic to the

destination serverwith theclient IPaddressas the sourceaddressand theVIPaddress

(10.94.127.181) as its destination.



NOTE: Servers should be configured to process these packets with theVIP address (10.94.127.181) as the destination address. A loopbackinterfaceadapter shouldbe installed andconfiguredwith theVIPaddresson theserver toenableprocessingofapplication traffic.Aproper IP routingentry should also be present in the servers to route these packets directlyto the client destination address.

Load-Balanced Traffic Flow

This section explains the packet flow from the client in the Internet to the server through

the F5 system and from the server back to the client, bypassing the F5 system


Figure 121: Load-Balancing Traffic Flow



The flow of traffic to and from the load balancers flows in the following manner:

1. As described in Figure 121 on page 234, three VIPs are created in the F5 system for

SharePoint, Wikimedia, and Exchange. Assume that the client sends the request to

theWikimedia server. In this case, the client sends the request to the VIP address of

10.94.127.182 and the source IP address is the client’s IP address. The destination IP

address will be the VIP IP address (10.94.127.182) and the destination port will be 80.

As described in the previous section, the core switch advertises the VIP address in the

network. As a result, the edge router knows the route to reach the VIP address of

10.94.127.182.

2. This packet arrives on the active F5 via external LAG. Because of the nPath

configuration, theWikimedia VIP address load-balances the traffic and sends it to

one of the servers without modifying the source or destination address.

3. Because of the nPath configuration, theWikimedia VIP address load-balances the

traffic and sends it to one of the servers as is without modifying the source or the

destination address. The F5 system reaches theWikimedia servers by way of a Layer

2 connection on VLAN 103. An internal LAG connection is a trunk port carrying VLANs

102, 103, and 104 to reach all the servers.

4. TheWikimedia server receives the traffic on the loopback address (configured with

VIP IP 10.94.127.182) and processes it.

5. TheWikimedia server sends this packet back to the client by way of a router and

bypassing the F5 system.

6. The return packet reaches the client.

Applications


• Microsoft Exchange Implementation on page 235

Overview

The Juniper MetaFabric 1.0 solution featured several business-critical applications in the

test and verification lab. This section will cover implementation details for the following

applications:


Microsoft Exchange Implementation

This section describes the design, planning, and instructions for deploying a highly

available Microsoft Exchange Server 2012 cluster for client access service andmailbox

database server using VMware high availability (HA). It covers configuration guidelines

for the VMware vSphere HA cluster parameters for the cluster and best practice

recommendation. This guide does not cover a full installation of Microsoft Exchange

Server. This section covers the following topics:

• Installation checklist and scope



• Network for Exchange VM

• Storage for Exchange VM

Installation Checklist and Scope

This section contains detailed instructions on configuring network and storage. We

assume that the following elements are already installed:

• ESXi 5.1 hypervisor on the IBM Flex chassis compute node.

• vCenter Server to manage ESXi 5.1 hosts.

This deployment example assumes that VMware HA is configured on ESXi hosts using

vCenter Server. Virtualmachines that are runningonanESXi host at the timeof complete

failure will be automatically migrated.

VMware vSphere HA requirements:

• All hosts inavSphereHA-enabledclustermusthaveaccess to the samesharedstorage

location used by the VM on the cluster. This includes any Fibre channel, FCoE, iSCSI,

and NFS datastores used by the VM. In this solution, we are using iSCSI and NFS

datastores.

• All hosts in a vSphere HA cluster should have an identical virtual networking

configuration. In this solution, all hosts are participating the same vSphere Distributed

Switch (vDS).

Deploying Network for Exchange VM

Microsoft Exchange is a two-tier application that includes a Client Access Server (CAS)

andmailbox database server. Exchange was deployed in POD1 during testing. CAS is a

front-end server for user mailboxes; the mailbox database is the back-end server. The

entire user’s mailbox is accessed from themailbox database server. The cluster is

configured on back-end servers called the Exchange database availability group (DAG).

10,000usersarecreatedonanLDAPserver (Windowsdomaincontroller),ActiveDirectory

using the DATACENTER domain. All of the tested Microsoft Enterprise applications

(including Microsoft Exchange) were integrated with theWindows domain controller

(DATACENTER). In this solution, all applications are using separate VLANs to ensure

traffic isolation.

Deployment of Microsoft Exchange requires the following steps:

• Define the network prefix.

• Define VLANs.



NOTE: TheWindows domain controller (DATACENTER) is considered aninfrastructure prefix and is assigned to a separate network. This design andimplementation assumes that this VLAN (and prefix/gateway) has alreadybeen defined on the switches.

TheWindows domain controller subnet is 172.16.1.0/24, and the defaultgateway is 172.16.1.254.

TwoWindows domain controllers were installed and configured, one is inPOD1 and other is in POD2 for redundancy. The assigned IP addresses forboth domain controllers are 172.16.1.11 and 172.16.1.10, respectively.

1. Define network subnets for assignment to Exchange server elements.

a. ExchangeServer is 172.16.4.0/24, the default gateway is 172.16.4.254. The exchange

server is scaled to serve 10,000 users, and 3 client access servers (CAS) have been

installed and configured. Server IP addresses are 172.16.4.10, 172.16.4.11, 172.16.4.12,

and 172.16.4.13.

b. Exchange DAG Cluster is 172.16.9.0/24 and the default gateway is 172.16.9.254.

Thecluster is configured tohost threemailboxdatabase servers. Theusers primary

mailbox database is active on one server with the backupmailbox databases

available on the other two servers. Themailbox database servers communicate

over VLAN-109 (The VLAN over which the DAG Cluster is configured).

2. Define VLANs for Exchange elements (enables traffic isolation).

a. Exchange server = VLAN and vlan-id 104.

b. Exchange Mailbox cluster = VLAN and vlan-id 109.

3. Configure theVLANandgatewayaddressonQFabric switch (POD1)as thisapplication

is located only in POD1.

[edit]set vlans Exchange vlan-id 104set vlans Exchange l3-interface vlan.104set interfaces vlan unit 104 family inet address 172.16.4.254/24set protocols ospf area 0.0.0.10 interface vlan.104 passiveset vlans Exchange-Cluster vlan-id 109

4. Configure VLANs on all IBM Pure Flex system CNAmodules.

vlan 104enablename "EXCHANGE"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

vlan 109enablename "Exchange DAG"member INTA1-INTA5,INTB1-INTB5,EXT1-EXT3,EXT7

Below configuration shows of 10Gb CNA I/OModule 1 and 2.



vlan 104enablename "Exchange"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

vlan 109enablename "Exchange DAG"member INTA1-INTA5,EXT1-EXT2,EXT11-EXT16

NOTE: This configuration is not required if you are using IBM Flex SystemPass-thrumodules. This configuration is an example of the 40-Gb CNAI/OModule (module 1 and 2).

5. Configure VLANs on LAVC switches.

[edit]set interfaces ae1 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae2 description "IBM Standalone server"set interfaces ae2 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae2 unit 0 family ethernet-switching vlanmembers Exchange-clusterset interfaces ae3 description IBM-FLEX-10-CNA-VLAG-BNTset interfaces ae3 unit 0 family ethernet-switching vlanmembers Exchange-clusterset interfaces ae3 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae4 description IBM-FLEX-2-CN-1-Passthroughset interfaces ae4 unit 0 family ethernet-switching vlanmembers Exchange-clusterset interfaces ae4 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae5 description IBM-FLEX-2-CN-2set interfaces ae5 unit 0 family ethernet-switching vlanmembers Exchange-clusterset interfaces ae5 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae6 description IBM-FLEX-2-CN-3set interfaces ae6 unit 0 family ethernet-switching vlanmembers Exchange-clusterset interfaces ae6 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae7 description IBM-FLEX-2-CN-4set interfaces ae7 unit 0 family ethernet-switching vlanmembers Exchange-clusterset interfaces ae7 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae8 description IBM-FLEX-2-CN5set interfaces ae8 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae8 unit 0 family ethernet-switching vlanmembers Exchange-clusterset vlans Exchange vlan-id 104set vlans Exchange-cluster vlan-id 109

6. Allow same VLANs and configure Layer 3 gateway for Exchange-Cluster on both core

switches Core1 and Core2.

[edit]set interfaces ae1 description "MC-LAG to vdc-pod1-sw1-nng-ae1"set interfaces ae1 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae2 description "MC-LAG to vdc-pod1-sw1-nng-ae2"set interfaces ae2 unit 0 family ethernet-switching vlanmembers Exchange-Clusterset interfaces ae4 description "MC-LAG to vdc-pod2-sw1-ae0"set interfaces ae4 unit 0 family ethernet-switching vlanmembers Exchange-Clusterset interfaces ae5 description "MC-LAG to vdc-pod2-sw1-ae1"set interfaces ae5 unit 0 family ethernet-switching vlanmembers Exchange



set interfaces ae9 unit 0 description "ICL Link for all VLANS"set interfaces ae9 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces ae9 unit 0 family ethernet-switching vlanmembers Exchange-Clusterset interfaces ae10 description Layer2-internal-link-MC-LAG-core-sw-to-LB2-standbyset interfaces ae10 unit 0 family ethernet-switching vlanmembers Exchangeset interfaces irb unit 109 description Exchange-Clusterset interfaces irbunit 109 family inetaddress 172.16.9.252/24arp 172.16.9.253 l2-interfaceae9.0

set interfaces irb unit 109 family inet address 172.16.9.252/24 arp 172.16.9.253mac4c:96:14:68:83:f0

set interfaces irb unit 109 family inet address 172.16.9.252/24 arp 172.16.9.253 publishset interfaces irbunit 109 family inetaddress 172.16.9.252/24vrrp-group 1virtual-address172.16.9.254

set interfaces irb unit 109 family inet address 172.16.9.252/24 vrrp-group 1 priority 125set interfaces irb unit 109 family inet address 172.16.9.252/24 vrrp-group 1 preemptset interfaces irb unit 109 family inet address 172.16.9.252/24 vrrp-group 1 accept-dataset interfaces irb unit 109 family inet address 172.16.9.252/24 vrrp-group 1authentication-typemd5


set protocols ospf area 0.0.0.0 interface irb.109 passiveset vlans Exchange vlan-id 104set vlans Exchange-Cluster vlan-id 109set vlans Exchange-Cluster l3-interface irb.109

7. As Exchange is being deployed in a virtual environment, you next need to create an

Exchange and Exchange-Cluster port group on the virtual distributed switch (VDS)

using vCenter server. Log in to vCenter Server (10.94.63.29) using the vSphere Client.

TocreateanExchangeandExchange-Cluster port group, navigate toHome> Inventory

> Networking.

Figure 122: Home > Inventory > Networking

8. Right-click dvSwitch, then click NewPort Group.



Figure 123: Create NewPort Group

9. ClickNext, then Finish. Once the Exchange port group is created, you can edit the port

group by right--clicking and thenmodifying the teaming policy.



Figure 124: Modify Teaming Policy

10. Repeat Steps 1 through 9 to create the port group for the Exchange-Cluster,

Storage-108, andStorage-208.Anexampleof thePG-Storage-108port group follows.

Figure 125: PG-STORAGE-108 Settings



An example of PG-Storage-208 follows.

Figure 126: PG-STORAGE-208 Settings

NOTE: The storage port group using the iSCSI protocol doesn’t supportport bonding (LAG). In the case of iSCSI, there is only one active uplink.

Configuring Storage for Exchange VM

The Exchange VM is connecting to storage via the iSCSI protocol. This section details

the creation of the storage for the Exchange VM.

To create storage via iSCSI protocol for connection to the Exchange VM, follow these

steps:

1. Log in with the EMC Unisphere tool to access EMC storage.

2. Select a storage system.

3. Navigate to Storage > Storage Configuration > Storage Pools. In the Pools tab, click

Create.



Figure 127: EMCUnisphere Tool

4. Provide a name for the storage pool (in this example, Pool 1 – Exchange-DB).

Figure 128: Create Storage Pool Name

5. Ensure that FAST Cache is enabled (under the Advanced tab).



Figure 129: FAST Cache enabled

6. Create and allocate LUN to the storage pool. Select the VNX system using the

Unisphere tool.

7. Navigate to Storage > LUNs.

8. In the Create LUN dialog, under Storage Pool Properties:

a. Select Pool.

b. Select a RAID type for the LUN: For Pool LUNs, only RAID 6, RAID 5, and RAID 1/0

are valid. RAID 5 is the default RAID type.

c. If available, the software populates the storage pool for the new LUNwith a list of

pools that have the specified RAID type, or displays the name of the selected pool.

The Capacity section displays information about the selected pool. If there are no

pools with the specified RAID type, click New to create a new one.

9. In LUN Properties, select the Thin checkbox if you are creating a thin LUN.

10. Assign a User Capacity and ID to the LUN you want to create.

11. To create more than one LUN, select a number in Number of LUNs to create. For

multiple LUNs, the software assigns sequential IDs to the LUNs as they are available.

For example, to create five LUNs starting with LUN ID 11, the LUN IDs might be 11, 12,

15, 17, and 18.

12. In LUN Name, either specify a name or select automatically assign LUN IDs as LUN

Names.

13. Choose one of the following options:

a. Click Apply to create the LUNwith the default advanced properties, or

b. Click the Advanced tab to assign the properties yourself.



14. Assign optional advanced properties for the LUN:

a. Select a default owner (SP A or SP B) for the new LUN or accept the default value

of Auto.

b. Set the FAST tiering policy option.

15. Click Apply to create the LUN, and then click Cancel to close the dialog box. An icon

for the LUN is added to the LUN viewwindow. Below is an example of the Exchange

LUN that was created.

Figure 130: Exchange-DB LUN

Enabling Storage Groups with Unisphere

Youmust enable storage groups using Unisphere if only one server is connected to the

system and you want to connect additional servers to the system. The Storage Groups

option lets you place LUNs into groups that are known as storage groups. These LUNs

are accessible only to the host that is connected to the storage group. To enable storage

groups with Unisphere:



1. Select All Systems > VNX System.

2. SelectHosts>StorageGroup. (Once you enable StorageGroups for a storage system,

any host currently connected to the storage systemwill no longer be able to access

data on the storage system. To the host, it will appear as if the LUNs were removed.

In order for the host to access the storage data, youmust add LUNs to the Storage

Group and then connect the host to the Storage Group.)

3. ClickOK to save changes and close the dialog box, or click Apply to save changes

without closing the dialog box. Figure 131 on page 246 shows the storage group that

was created. Any new LUNs added will be added to this storage group.

Figure 131: Storage Group Created

Figure 132 on page 247 shows the LUNs tab of the storage group properties. You can

see all LUNs that have been added to the storage group.



Figure 132: Storage Group Properties - LUNs Tab

4. From the Hosts tab (Storage Group Properties), you can select hosts to add to the

storage pool (which hosts are able to access the pool).



Figure 133: Hosts Allowed to Access the Storage Group

Once the storage group is created, LUNs can be added directly to the storage pool

from the Storage > LUNs screen.

Figure 134: Add LUN to Storage Group

Provisioning LUNs to ESXi Hosts

TheLUNscreated in theprevious stepsnowneed tobeaddedandmountedasdatastores

on the appropriate ESXi hosts. To do this, youmust first configure the VMware kernel



network. ESXi uses VMkernel ports for systemmanagement and IP storage. VMkernel

IP storage interfaces provide access to one or more EMC VNX iSCSI network portals or

NFS servers.

To configure VMkernel:

1. Log in to the VMware vSphere client to the vCenter Server.

2. Navigate to Home > Inventory > Hosts and Clusters.

3. Select the hosts and on the right side, click Configuration.

4. Under Networking, vSphere Distributed Switch, clickManage Virtual Adapters.

Figure 135: Manage Virtual Adapters

5. Click Add to create a new VMkernel port, then click Next.



Figure 136: Add NewVMkernel Port

6. Select a virtual adapter type (VMkernel) and click Next.

Figure 137: Select VMkernel as Adapter Type

7. Select the port group (created in previous steps for POD1). Click Next.



Figure 138: Select Port Group

8. Configure IP address settings for the VMkernel virtual adapter, click Next, and then

click Finish.

Figure 139: VMkernel IP Settings



9. Before configuring the VM, make sure that the EMC VNX storage is reachable. You

can do this from the ESXi server shell using vmping.

~ # esxcfg-vmknic -l

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type vmk0 17 IPv4 10.94.47.131 255.255.255.0 10.94.47.255 00:90:fa:1c:8a:04 9000 65535 true STATIC vmk0 17 IPv6 fe80::290:faff:fe1c:8a04 64 00:90:fa:1c:8a:04 9000 65535 true STATIC, PREFERRED vmk1 130 IPv4 172.16.8.27 255.255.255.0 172.16.8.255 00:50:56:60:0a:0b 9000 65535 true STATIC vmk1 130 IPv6 fe80::250:56ff:fe60:a0b 64 00:50:56:60:0a:0b 9000 65535 true STATIC, PREFERRED vmk3 1498 IPv4 172.16.6.31 255.255.255.0 172.16.6.255 00:50:56:67:9b:5c 9000 65535 true STATIC vmk3 1498 IPv6 fe80::250:56ff:fe67:9b5c 64 00:50:56:67:9b:5c 9000 65535 true STATIC, PREFERRED vmk4 1370 IPv4 172.16.7.31 255.255.255.0 172.16.7.255 00:50:56:68:53:ee 9000 65535 true STATIC vmk4 1370 IPv6 fe80::250:56ff:fe68:53ee 64 00:50:56:68:53:ee 9000 65535 true STATIC, PREFERRED

~ # ping 172.16.8.1

PING 172.16.8.1 (172.16.8.1): 56 data bytes64 bytes from 172.16.8.1: icmp_seq=0 ttl=128 time=0.383 ms64 bytes from 172.16.8.1: icmp_seq=1 ttl=128 time=0.215 ms64 bytes from 172.16.8.1: icmp_seq=2 ttl=128 time=0.231 ms

--- 172.16.8.1 ping statistics ---3 packets transmitted, 3 packets received, 0% packet lossround-trip min/avg/max = 0.215/0.276/0.383 ms

~ # ping 172.16.8.2

PING 172.16.8.2 (172.16.8.2): 56 data bytes64 bytes from 172.16.8.2: icmp_seq=0 ttl=128 time=0.451 ms64 bytes from 172.16.8.2: icmp_seq=1 ttl=128 time=0.243 ms64 bytes from 172.16.8.2: icmp_seq=2 ttl=128 time=0.224 ms

--- 172.16.8.2 ping statistics ---3 packets transmitted, 3 packets received, 0% packet lossround-trip min/avg/max = 0.224/0.306/0.451 ms~ #

10. From vCenter, click Storage Adapters. If the iSCSI software adapter is not installed,

click Add and install the adapter.



Figure 140: Install iSCSI Software Adapter

11. Once installed, right-click the iSCSI softwareadapter andselectProperties. You should

see that the software is enabled.

Figure 141: iSCSI Initiator Is Enabled

12. Click the Network Configuration tab to verify that the storage is configured and

connected.



Figure 142: iSCSI Initiator Network Configuration

13. Click the Dynamic Discovery tab and click Add. Enter the IP and port of your EMC VNX

storage.



Figure 143: Add iSCSI Server Location in Dynamic Discovery

14. ClickOK and Close. When prompted to rescan the HBA, click Yes. You see a LUN

presented on the server.

Figure 144: LUN Present on the Server

15. From the vSphere client, select the Exchange-Logs server and Add Storage.



Figure 145: Add Storage from vSphere Client

16. Select Disk/LUN as the storage type. Click Next.

Figure 146: Select Disk/LUNfor Storage Type

17. Select the Disk/LUN youwant tomount. Verify that you aremounting the proper LUN

using the LUN ID, capacity, and path ID. Click Next.



Figure 147: Select LUN toMount

18. Select VMFS 5.0which is supported in ESXi 5.1. VMFS 5.0 also supports 2TB+. Click

Next.

Figure 148: Select VMFS-5 as a File System

19. Notice that the hard disk is blank under Current Disk Layout. Click Next.

20.Enter a name for the datastore and click Next.



Figure 149: Name the Datastore

21. Select themaximumcapacity for thisdatastore. (Themaximumcapacity is thedefault

option.) Click Next and Finish. Click Properties on the created datastore to see an

output similar to the following

Figure 150: Datastore Creation Complete



Configuring vMotion Support

This implementation supports vMotion. Wemust make sure, however, that in EMC

Unisphere, all ESXi hosts have been given access to all LUNs. Once this is configured and

enabled, other hosts will attempt to mount the same datastore once you start rescan

on the other hosts. Similarly, all LUNsmust bemounted on all appropriate ESXi hosts

as this is a requirement for vMotion. Network and storagemust be provisioned to all ESXi

hosts. Thenext sectionwill showhowtoaddnewVMoraddanexistingVMforapplication

scaling. This guide does not cover the installation or configuration of the Exchange

applications.

To configure vMotion support for an ESXi host, follow these steps:

1. Log in to Virtual Center Server using vSphere client.

2. Select the cluster to create a new VM. Click Create a new virtual machine.

Figure 151: Create NewVM

3. Click Typical and then click Next.



Figure 152: VM Type

4. Enter a name for your virtual machine. Click Next.



Figure 153: Give the VM aName

5. Tomount storage to the VM, select the datastore POD1-VMFS-LUN3created earlier,

and click, Next.



Figure 154: Select Storage

6. Select an operating system (Windows 2012 64-bit was used in this scenario). Click

Next.



Figure 155: Select Operating System

7. Exchange CAS requires only one NIC. The Exchangemailbox requires two NICs. You

can add a new NIC here or wait until the VM is created to add another NIC. For now,

leave the default and click Next.



Figure 156: Configure Network

8. Select the virtual disk size for the operating system. (This will be the C:/ drive in the

OS.) Click Finish.



Figure 157: Select Virtual Disk Size

9. The current example is used to create a new VM that can bemodified based on your

requirements. For instance, an Exchangemailbox requires additional disks and an

additional network adapter for use in Exchange clusters. An example of a modified

VM is shown below.



Figure 158: Virtual Machine with Additional Disks and Network Adapters

10. Once you have provisioned all of the VM resources, you can start installation by

mounting the installation ISO as a CD. In this case you would first install and update

MicrosoftWindowsServer 2012. Once the operating system is installed, the Exchange

installation (and all the dependencies, such as AD integration) can be performed.



CHAPTER 9

NetworkManagement andOrchestration

The following section covers the configurationof services in theVirtualized ITDataCenter.

The areas covered by this chapter include:

• Junos Space

• Overview

• Configure Junos Space and Network Director

• Implement and configure Security Director


• Configuring Security Director on page 287

Overview

The MetaFabric 1.0 solution lab employed Junos Space along with Network Director (a

Junos Space application) in the provisioning and testing of the solution. Specifically,

Junos Space Release 13.1R1 was used along with Network Director 1.5 (with Virtual View

application) for implementation andmanagement of the network. Virtualmachine (VM)

orchestration was also controlled by the Junos Space implementation.

Junos Space is installed on a VM on the IBM 3750 standalone server and serves the

out-of-band (OOB)management role in the data center topology. Two IBM3750 servers

are configured in a single ESX cluster for redundancy and fault tolerance. In case of one

ESXnode failure, the JunosSpaceVMcanbemoved to theotherESXi hostusingvMotion.

NetworkDirector 1.5 includessupport for theQFX3000-Msystemtoprovisionandmonitor

the entire solution. Network Director Virtual View is enabled for orchestration to enable

tracking of VMmovement within the data center test topology.

Security Director is also a component of themanagement and orchestration of the data

center solution. Security Director is an application installed on Junos Space. It is used to

configure and operate the perimeter security elements of the solution.

Topology

Figure 159 on page 268 illustrates the topology of the out-of-band (OOB)management

network tested in the MetaFabric 1.0 lab.


Figure 159: The OOBManagement Topology

Configuring Junos Spacewith Network Director

This section covers the configuration of the solution using Junos Space with Network

Director.

1. Install Junos Space Release 13.1R1 (or later) into the VM environment. (In this test lab,

this VMwas hosted on the IBM standalone ESXi cluster.)

2. Downloadand installNetworkDirector 1.5onto JunosSpace.This shouldautomatically

install as a Space application.

3. Configure the Network Devices to bemanaged with the proper SNMP community,

trap groups, and NETCONF.

set snmp community public authorization read-onlyset snmp community public clients 10.94.63.18/32set snmp community public clients 10.94.63.19/32set snmp trap-group space version v2set snmp trap-group space targets 10.94.63.19set system services netconf ssh

4. From the logical view of Network Director (Figure 160 on page 269), select Discover

Devices, and enter the IP address (or range) to enable device discovery. The IP

addresses used in this lab were configured as the OOBmanagement addresses on

each network node.



Figure 160: Select IP address, IP Range, IP-Subnet, or HostName

5. Select the virtual-viewon theNDGUI and enter the IP address of theVMware vCenter

and authentication details.

Figure 161: Configure Virtual Network target

NDwill pull all the data from the vCenter to the virtual view to enable viewing of all

provisioned virtual machines.

Configuring VMOrchestration in the Network Director 1.5 Virtual View

This section covers the configuration of VMOrchestration using Network Director 1.5

Virtual-View


Chapter 9: Network Management and Orchestration

1. Enable Orchestration in NDVirtual View. Thismust be enabled so that that VLAN and

port changesperformedbyvMotionwill be trackedandconfiguredbyNetworkDirector

on the physical network.

Figure 162: Enable OrchestrationMode in Network Director

1. The following network-level configurations must bemade prior to VMOrchestration

activation in Network Director:

• Configure EX/QFX ports in trunk mode.

• Enable LLDP on the VMware Distributed Switch as well as on the QFX3000-M

QFabric PODs.

• NDwill configure the groups config on all devices with the necessary VLANs once

orchestration is enabled.

• During a vMotion event, ND automatically assigns the new port to the VLAN on the

destination switch.

NOTE: Orchestration is not supported on LAG interfaces configured towardservers on the access switches.

Network Director Provisioning

This section covers the configuration of network elements using Network Director.

1. SelectDevice common settings underBuild > Logical View and configure the following

parameters:

a. Username and password

b. Services: FTP,HTTP, and so on

c. Login banner



d. Logs

e. STP

f. DCBX/LLDP

g. DNS server settings

Figure 163: Configure Device Common Settings

2. Once the wizard is completed with all the parameters, the selected devices will be in

Pending Deployment mode.

Figure 164: Change in Pending Deployment

3. Once you are ready to commit the changes, click Deploy.



Figure 165: Change in Pending Deployment

Configuring Class of Service Using Network Director

This section covers the configuration of class of service using Network Director.

1. SelectCoS from the Profile and ConfigurationManagement drop-downmenu (found

in Logical View) and click Add from the list of devices families. Then select the Data

Center Switching device family.

Figure 166: Select the Data Center Switching Device Family

2. Select Hierarchical Port Switching (ELS) for QFabric-QFX devices if you want to

configure PFC/ETS CoS. The default CoS-Profile is displayed. Modify the default

traffic settings parameters, if needed.



Figure 167: Select the Profile "Hierarchal Port Switching (ELS)"

3. Enable the PFC code point and queue for NO-LOSS behavior. In this example, Queue

3 is chosen as the no-loss queue.

Figure 168: Enable PFC Code-point and Queue for NO-LOSS Behavior

4. This CoS profile will be referenced while creating port configuration in the next steps.



Figure 169: COS Profile Deployed

Creating VLANs Using Network Director

This section covers the creation and installation of VLANs using Network Director.

1. Select a VLAN and enter the VLAN-ID and VLAN-Name (ND-Test-VLAN in this

example).

Figure 170: Create VLAN-ID and VLANName

2. Click Next to go to Advanced Settings and configure Layer 2 filters and MAC address

move limits.



Figure 171: Configure Layer 2 Filters andMACMove Limit

3. Click Done to create the VLAN Profile ND-Test1.

Figure 172: VLAN Profile ND-Test1Created

Setting UpQFabric Using Network Director

This section covers the configuration of a QFabric system using Network Director.

1. InNetworkDirector, navigate toBuildMode>DeviceManagement. SelectSetupQFabric.



Figure 173: Select Setup QFabric

2. Configure aliases for the node and interconnect devices by clicking the default aliases

shown in the GUI as NODE-1 and NODE-2.

Figure 174: Configure Device Aliases

3. Configure node devices in a redundant server node group (RSNG), using the aliases

configured in Step 2 (NODE-1 and NODE-2).



Figure 175: Configure Node Group Type RNSG

Setting UpQFabric Using Network Director – Ports and VLAN

This section covers the configuration ofQFabric ports andVLANsusingNetworkDirector.

1. Click Port to manage the port profile and select Data Center Switching Non ELS.

Figure 176: Configure Center Switching Non ELS

2. Configure the following parameters:

a. VLAN name.

b. Service type (select server port if the port will be connected to a server).

c. Family type (select “switching” and “trunk”).

d. Add VLAN and select the profile ND-test1.



Figure 177: Configure VLAN Service, Port, CoS, and so on

3. Click Done to create the port profile.

Figure 178: Port Profile Created (NDTestport)

4. Assign the port profile to the physical interface. ClickAssign to list the available ports.



Figure 179: Assign Port Profile to Available Port

5. Select vdc-pod1-sw1 (QFabric QFX3000-M) and select RSNG2 to select the port

connection.

Figure 180: Assign Port Profile

6. Click Assign to assign the physical port.



Figure 181: Click Assign

7. The physical port is now added to the port profiles (assignments) list.

Figure 182: New Physical Port Added to Port Profile List

8. The port profile is created successfully for vdc-core-sw1.



Figure 183: Port Profile Created Successfully

9. Deploy the port profile to the vdc-core-sw1. Go to Deploy Mode and check pending

configuration deployments.

Figure 184: Check to Confirm Port Profile Is Pending

10. Deploy the changes to the device using the Deploy Now option.



Figure 185: Select Deploy Now

Setting Up a QFabric SystemUsing Network Director – Create Link Aggregation Groups

This section covers the configuration of LAGs by using Network Director.

NOTE: MC-LAG is only supported forQFX standalone systems. (This featurewas not supported in the MetaFabric 1.0 test bed.)

1. Select Build > DeviceManagement >Manage Port Groups. Select Add new port group.

Figure 186: Add NewPort Group

2. Select a device from the left pane, then clickSelectDevices to select the LAGmember

links.



Figure 187: Select Devices to Add as LAGMember Links

Selected links are shown below, which will belong to a LAG (port group).

Figure 188: Links Selected to Be LAGMember Links

Network Director – Downloading and Upgrading Software Images

This section covers software and imagemanagement using Network Director.



1. Select Deploy > ImageManagement > .

Figure 189: Network Director Image Repository

2. Browse and select the image you would like to upload to the ND repository. Select

Upload.

Figure 190: Image Staging on Network Director

3. Deploy the downloaded image to the physical devices. Select one of the following

three options:

a. Staging only (Download to remote device but do not install)

b. Upgrade only (Install previously staged image to device)

c. Staging and Upgrade (Download and install image)



Figure 191: Stage Image to Device for Install or for Later Installation

4. Click “Next” to select the devices for the upgrade and select the downloaded image

per device family

Figure 192: Select Image to Stage to Remote Device

Network Director –Monitoring the QFabric System

This section covers software and imagemanagement using Network Director.



1. SelectMonitor>SelectDevices>vdc-pod1-sw1. This opensup thedashboard toenable

viewing of the QFabric System status.

Figure 193: Device Dashboard

2. Select Traffic to see traffic statistics on the QFabric system.

Figure 194: QFabric Traffic Statistics

3. Select Equipment to enable viewing of CPU andmemory utilization.



Figure 195: Hardware Dashboard

4. Click Run Fabric Analyzer (in the right pane) to get the fabric internal link status.

Figure 196: Confirmation of Run Fabric Analyzer Operation

Configuring Security Director


Overview

Junos Space Security Director is one of the Junos Spacemanagement applications and

helpsorganizations improve the reach, ease, andaccuracyof securitypolicyadministration

with a scalable, GUI-basedmanagement tool. It automates security provisioning through

one centralizedWeb-based interface to help administrators manage all phases of the



security policy life cyclemore quickly and intuitively, from policy creation to remediation.

Security Director provides the following management efficiencies:

1. Scale security policy acrossmultiple Juniper Networks SRXSeries ServicesGateways,

or managemultiple logical system (LSYS) instances on a single SRX Series device.

2. Centrally configure andmanage application security (for example, AppSecure),

firewall, VPN, IPS,andNATsecuritypolicy throughonescalablemanagement interface.

3. Define and enforce policies for controlling usage of specific applications such as

Facebook, instant messaging, and embedded social networking widgets through

included AppFWmanagement.

4. Reuse security policies within Junos Space Security Director for improved security

enforcement accuracy, consistency, and compliance.

5. Build the infrastructure for furthermanagement innovationacross thenetwork through

open and secure Junos Space Network Management Platform integration.

When you finish creating and verifying your security configurations, you canpublish these

configurationsandkeep themready tobepushed to the securitydevices. SecurityDirector

helps you deploy all the security configurations to the devices all at once by providing a

single interface that is intuitive. You can select all security devices that you are using on

the network and push all security configurations to these devices.

The Security Director application is divided into sevenworkspaces, which include Object

Builder, Firewall Policy, NAT Policy, VPN, Downloads, IPS Management, and Security

Director Devices.

• Object Builder—Workspace to create objects used for firewall policy, NAT policy, and

VPN configurations.

• FirewallPolicy—Workspace tocreateandpublish firewall policiesonsupporteddevices.

• NAT Policy—Workspace to create and publish NAT policies on supported devices.

• VPN—Workspace to create site-to-site, hub-and-spoke, and full-mesh IPsec VPNs.

• Downloads—Workspace to download and install signatures.

• IPS Management—Workspace to create andmanage IPS signatures, signature sets,

and IPS policies.

• Security Director Devices—Workspace to update the configurations on the devices.

Discovery and Basic Configuration Using Security Director

Discovery is the process of finding a device and then synchronizing the device inventory

and configuration with the Junos Space database. To use device discovery, Junos Space

must be able to connect to the device.



NOTE: When you initiate discovery on a device, Junos Space automaticallyenables SSH and the NETCONF protocol over SSH by pushing the followingcommands to the device:

• set system services ssh protocol-version v2

• set system services netconf ssh

To discover network devices, Junos Space uses the SSH and SNMP protocols. Device

authentication is handled through administrator login SSH v2 credentials and SNMP

v1/v2c settings, which are part of the device discovery configuration. You can specify a

single IP address, a DNS hostname, an IP range, or an IP subnet to discover devices on a

network. During discovery, Junos Space connects to the physical device and retrieves the

active configuration and the status information of the device. To connect with and

configure devices, Junos Space uses Juniper Network’s Device Management Interface

(DMI), which is an extension to the NETCONF network configuration protocol. When

discovery succeeds, Junos Space creates an object in the Junos Space database to

represent the physical device andmaintains a connection between the object and the

physical device so their information is linked.

Once you have added the device, youmight a get mismatched DMI version

(Figure 197onpage289).DMIversionmismatch requires that theDMIbeupdatedtoensure

that the management schema is compatible between Junos Space and themanaged

devices.

Figure 197: DMI Mismatch



Resolving DMIMismatch

To resolve DMI mismatch issues on Junos Space, navigate to the Network Management

Platform, clickDMISchema under Administration, clickUpdateSchema, and select either

Archive or Repository. An Archive schema is one already installed on the network

management platform. A Repository schemamust be downloaded to the network

management platform. Repository details (URL, authentication) must be provided as

shown in Figure 198 on page 290. To retrieve from the repository, provide your Juniper

Networks customer or partner account username and password. Test the connection

before saving it. Thiswill show recommended schemas for your devicewhichwill resolve

the mismatch version of schemas.

Figure 198: DMI Schema Repository Requires Authentication

Once devices are discovered, navigate to the Security Director’s Device pane. Here, you

can see the status of security devices on the network. To sync the device settings with

Junos Space and Security Director, right-click a security device and click Update. This

will import the configuration from the security device and sync the configuration with

the Security Director database.

NOTE: Tocreatezoneson JuniperNetworkssecuritydevices, use theNetworkManagementPlatform.Asof thecurrent versionofSecurityDirector (13.1P1.14asof test completion), zonecreation isnot supportedwithinSecurityDirector.This might be fixed in future versions of Security Director.

The configuration of security devices appears in the directory hierarchy.

Figure 199 on page 290 shows the creation of security zones fromwithin the Network

Management Platform.

Figure 199: Security Zone Creation



Object Builder (Using Security Director)

Use the Object Builder workspace in Security Director to create objects used by firewall,

VPN, and NAT policies. These objects are stored in the Junos Space database. You can

reuse these objects with multiple security policies, VPNs, and NAT policies across an

entire device or network footprint. This approachmakes the design of services more

structured and avoids the need to create the objects during the service design. You can

use the Object Builder workspace to create, modify, clone, and delete the following

objects:

• Addresses and address groups

• Services and service groups

• Application signatures

• Extranet devices

• NAT pools

• Policy profiles

• VPN profiles

• Variables

• Template and template definitions

Figure 200 on page 292 is an example of address object creation using Security Director.

Custom services, NAT pools, devices, and so on can be created in a similar fashion. To

create an object, follow these basic steps:

1. Go to the Object Builder under Security Director.

2. Go to Addresses.

3. Click on the Plus sign to create a new address on the right side.

4. Click on the Pen side to modify an existing address.



Figure 200: Address Object Creation

NOTE: New address objects can also be created under Firewall Policy.

Creating Firewall Policy Using Security Director

Security Director provides you with the following types of firewall policies:

• All Devices—Predefined firewall policy that is available with Security Director. You can

add pre-rules and post-rules. When all the device policy configuration information is

updated on the devices, the rules are updated in the following order:

• All devices pre-rules

• Group pre-rules

• Device-specific rules

• Group post-rules

• All devices post-rules

An All Devices policy enables rules to be enforced globally to all the devices managed

by Security Director.

• Group—Type of firewall policy that is shared with multiple devices. This type of policy

is used when you want to update a specific firewall policy configuration to a large set

of devices. You can create group pre-rules, group post-rules, and device rules for a

group policy. When a group firewall policy is updated on the devices, the rules are

updated in the following order:

• Group pre-rules

• Device-specific rules

• Group post-rules

• Device Policy—Type of firewall policy that is created per device. This type of policy is

usedwhen youwant to push a unique firewall policy configuration per device. You can

create device rules for a device firewall policy.



• Device-Exception Policy—Type of firewall policy that is created when a device is

removed from a group policy.

• Global Policy—Global Policy Rules are enforced regardless of ingress or egress zones;

they are enforced on any device transit. Any objects defined in the Global Policy Rules

must be defined in the global address book.

The basic settings of a firewall policy are obtained from the policy profile in Security

Director. The basic settings include log options, firewall authentication schemes, and

traffic redirection options.

All device pre-rules and post-rules are applicable to all security devices. Once pre-rules

are published, these rules are applied to all managed security devices. Security Policy

post-rulesarepublished to thesecuritydeviceandcanbeused tooverwritedevice-specific

post-rules already deployed on the security device.

The general steps that must be followed to create a new security policy are:

1. Under Firewall policy, click to create a new policy.

2. Create the policy created for a device or a group

3. Once the policy is created, assign the policy to the device.

4. Click on Create.

The new policy created is displayed in the right pane. An option is also included to save

the policy and validate the configuration (ensure that the configuration does not contain

errors). After policy creation, you need to publish or publish and update the policy to the

security device.

• Publish policy will push the policy to the Junos Space database. This will also validate

the configuration.

• Publish and update will push the policy to the security device. This is often preferred

as ameans of provisioningmultiple devices during shortmaintenancewindows as this

feature publishes the device to the Junos Space database, validates that the

configurationwill have no errors, and then updates themanaged security devices with

the new configuration.

Figure201onpage294showsanexampleofpolicycreationafter importingaconfiguration

from amanaged security device. The steps that were followed in this example are:

1. Navigate to the Firewall policy under Security Director.

2. The right pane shows a list of policy name and all existing policies.

3. Click on Lock to lock the policy. (Policy must be locked to edit.)

4. Click on the plus (+) sign to add a new rule or the (-) sign to delete an existing rule. In

this example, we are adding a new rule (Test-1). Click the +sign.



Figure 201: New Rule Created (Test-1)

5. Whilemodifying theaddress, select anaddress fromtheexistingaddressbookor click

on the plus (+) sign to add a new address.

Figure 202: Add NewSource Address to Rule

6. Once you create the rule, use the up or down arrow tomove the rules up or down.

7. Click Save to save the configuration to the Security Director database.

8. To verify the configuration, click Validate.

Creating NAT Policy Using Security Director

Network Address Translation (NAT) is a form of network masquerading that replaces

private IP addresses (or addresses you wish to hide from public routing) with other

addresses (most often a public, routable IP address). NAT can also be used to provide

public access toprivate resources in thedatacenterby translatingapublic IP (an incoming

HTTP GET to aWeb server, for instance) to a private IP address. In Junos OS security

devices, NAT is largely zone-based. A trust zone is a segment of the network where

security measures are applied (sometimes referred to the “inside” of the network). It is

usually assigned to the internal LAN. An untrust zone is most often facing the Internet

(can also be facing any insecure network such as a partner or customer). NATmodifies

the IP addresses of the packets moving between these security zones.



Junos Space Security Director supports three types of NAT:

• Source NAT—Translates the source IP address of a packet leaving the trust zone

(outbound traffic). It translates the traffic originating from the device in the trust zone.

Using source NAT, an internal device can access the network by using the IP addresses

specified in the NAT policy.

• Destination NAT—Translates the destination IP address of a packet entering the trust

zone (inbound traffic). It translates the traffic originating from a device outside the

trust zone. Using destination NAT, an external device can send packets to a hidden

internal device.

• Static NAT—Always translates a private IP address to the same public IP address. It

translates traffic from both sides of the network (both source and destination). For

example, aWeb server with a private IP address can access the Internet using a static,

one-to-one address translation.

Junos Space Security Director provides you with a workflow where you can create and

apply NAT policies on devices in a network. To create a NAT policy:

1. Select Security Director > NAT Policy.

2. Click Create NAT Policy from the left pane. You can create a group policy or a device

policy.

3. Tocreateagrouppolicy, enter nameof thegrouppolicy, adescription, and theassigned

device for which policies have been configured.

You can also search for the devices by entering the device name, device IP address, or

device tag in the Search field in the Select Devices section. The above steps can also be

used to create device NAT policies. The Validate button will check the NAT policies for

errors. If any errors are found during the validation, a red warning icon is shown for the

policy or policies containing errors. In the case of NAT policies, incomplete rules and

duplicate rule names should flag as errors during validation. Please note that an existing

policy must be locked before any changes can be attempted. NAT policies can also be

rearranged (moved up or down) using the arrow keys.

Once you define a NAT policy, it must be published.

To publish a NAT policy:

1. Select Security Director > NAT Policy.

2. Right-click thepolicyon the right side that youwant topublishandclickAssigndevices.

3. Select Schedule at a later time check box if you want to schedule and publish the

configuration later.

4. Click Next.

5. To preview the configuration changes that will be pushed to the device, click View

Configuration in XML format > XML configuration tab.

6. Click Close.



7. Click Publish if you want to only publish the configuration.

8. Click Publish and Update if you want to publish and update the devices with the

configuration.

You can view any job under Jobs for the status. Figure 203 on page 296 shows two NAT

policies configured on the selected firewall.

Figure 203: Example NAT Policies in Security Director

JobsWorkspace in Security Director

The Jobsworkspace lets youmonitor the status of all jobs that have been run in all Junos

Space applications. A job is a user-initiated action that is performed on a Junos Space

Network Management Platform object, such as a device, service, or customer. All

scheduled jobs can bemonitored.

Typical jobs in Junos Space Network Management Platform include device discovery,

deploying services, pre-staging devices, and performing functional and configuration

audits. Jobs canbescheduled tooccur immediately or in the future. For all jobs scheduled

in Junos Space Network Management Platform, you can view job status from the Jobs

workspace. JunosSpaceNetworkManagementPlatformmaintainsahistoryof job status

for all scheduled jobs. When a job is scheduled from aworkspace, Junos Space Network

Management Platform assigns a job ID that serves to identify the job (along with the job

type) in the Manage Jobs inventory page.

You can perform the following tasks from the Jobs workspace:

• View status of all scheduled, running, canceled, and completed jobs.

• Retrieve details about the execution of a specific job.

• View statistics about average execution times for jobs, types of jobs that are run, and

success rate

• Cancel a scheduled job or in-progress job (when the job has stalled and is preventing

other jobs from starting).

• Archiveold jobsandpurge themfromthe JunosSpaceNetworkManagementPlatform

database.



Audit Logs in Security Director

Audit logs provide a record of Junos Space Network Management Platform login history

and user-initiated tasks that are performed from the user interface. From the Audit Logs

workspace, youcanmonitoruser login/logoutactivityover time, trackdevicemanagement

tasks, view services thatwere provisioned on devices, and so forth. Junos SpaceNetwork

Management Platform audit logging does not record non-user initiated activities, such

as device-driven activities, and is not designed for debugging purposes. User-initiated

changes made from the Junos Space CLI are logged but are not recorded in audit logs.

Administrators can sort and filter on audit logs to determinewhich users performedwhat

actions on what objects at what time. For example, an Audit Log administrator can use

audit log filtering to track the user accounts that were added on a specific date, track

configuration changes across a particular type of device, view services that were

provisioned on specific devices, or monitor user login/logout activity over time. To use

the audit log service to monitor user requests and track changes initiated by users, you

must be assigned the Audit Log Administrator role.

Over time, theAudit LogAdministratorwill archivea large volumeof JunosSpaceNetwork

Management Platform log entries. Such log entries might or might not be reviewed, but

they must be retained for a period of time. The Archive Purge feature helps youmanage

your Junos Space Network Management Platform log volume, allowing you to archive

log files and then purge those log files from the Junos Space Network Management

Platform database. For each Archive Purge operation, the archived log files are saved in

a single file, in CSV format. The audit logs can be saved to a local server (the server that

functions as the active node in the Junos Space Network Management Platform fabric),

or a remote network host ormedia.When you archive data to a local server, the archived

log files are saved to the default directory /var/lib/mysql/archive.

The Audit Logs Export feature enables you to download audit logs in CSV format so that

you can view the audit logs in a separate application or save them on another machine

for further use, without purging them from the system.



CHAPTER 10

Solution Scale and Known Issues

• Overview of Solution Scale Known Issues on page 299

• Scale on page 300

• Known Issues on page 300

Overview of Solution Scale Known Issues

This section provides a high-level overview of test results and a summary of target and

actual scale and performance vectors achieved during the solution testing.

At a high level, the solution met or exceeded each of the following solution scale and

performance goals:

General

• Solution must support end-to-end convergence within 2 seconds.

• Solution must allow compute nodes to use all available network links for forwarding.

• Solution must support inter-POD transport.

• Solution must support orchestration andmovement of virtual resources between the

PODs.

• Solution must support an out-of-band (OOB)management network that can survive

the failure of the data plane within a POD.

Security

• Solution must support perimeter security.

• Solution must support a flexible method of applying application security.

• Solution must support secured remote access.

Compute, Virtualization, and Storage

• Solution must support lossless Ethernet.

• Solution must provide redundant network connectivity using all available bandwidth.

• Solution must support moving a VM between hosts.

• Solution must support high availability of a VM.


• Solution must support virtual network identification using LLDP.

• Solution must provide physical and virtual visibility and reporting of VMmovements.

• Solution must support lossless Ethernet for storage transit.

• Solution must support booting a server from the storage array.

Application Scaling

• Solution must support 10,000Microsoft Exchange users.

• Solution must support 10,000Microsoft SharePoint user transactions.

• Solution must support 10,000MediaWiki user transactions.

NetworkManagement

• Solution must support monitoring and provisioning using Junos Space and Network

Director 1.5.

• Solution must support Orchestration of VM and network configurations by using

Network Director 1.5.

There were some exceptions to the solution requirements that were discovered during

testing. An overview of these exceptions can be found in the “Known Issues” section of

this chapter.

Scale

Table 19: Application Scale Targets

RemarksScaleApplications

Passed2000 Users transactionsMicrosoft SharePoint

Passed2000 Users transactionsMicrosoft Exchange

Passed2000MediaWiki

Passed2000Web requests

Passed10GSchinick background traffic

Known Issues

There were several persistent issues in testing that caused the test results to fall outside

of the solution requirements. These issues include:

• iSCSI LAG is not supported. Only 10-Gbps of forwarding can be achieved.



• This is a limitation of the storage array and the iSCSI protocol. iSCSI does support

multi-path in the event of failure.

• IPv6 on MC-LAG (Active/Active) is not currently supported.

• In-service software upgrade (ISSU) is not supported by the MX Series if the chassis is

populated with all 10-Gbps line cards. A 1-Gbps line card is required in order to start

ISSU.

• Graceful Routing Engine switchover (GRES) requires a 6-second BFD timer to work

properly on both the MX and EX Series. This is a known issue on MX Series (and now

on EX/QFabric).

• A 96-member LACP LAG is not supported by MX Virtual Chassis. A workaround was

put in place to utilize 4 sets of 24-member LAG interfaces.

• VMOrchestration by Junos Space is not supported if LAG is configured toward the

target servers.

• Junos Space / Network Director does not support Layer 3 provisioning.

• Junos Firefly Host does not support Application Layer Gateways (ALGs).


Chapter 10: Solution Scale and Known Issues

metafabric™ architecture virtualized data center: design and implementation guide

Technology