implementing next generation performance routing...

Implementing Next Generation Performance

Routing – PfRv3Jean-Marc Barozet – Technical Leader, IWAN

BRKRST-2362

• IWAN Introduction

• PfR Principles

• Domain Discovery

• Performance Monitoring

• Enterprise Deployment

• IWAN Management

• Key Takeaways

Agenda

Cisco Intelligent WANSolution Components

MPLS

Unified

Branch

3G/4G-LTE

Internet

PrivateCloud

VirtualPrivateCloud

PublicCloud

Application Optimization

Enhanced Application

Visibility and Performance

Secure Connectivity

Comprehensive

Threat Defense

Intelligent Path Control

Application

Aware Routing

TransportIndependent

Simplified

Hybrid WAN

Management Automation

IWAN: Intelligent Path ControlLeveraging the Internet

Branch

MPLS

Internet

Virtual PrivateCloud

Private Cloud

• PfR monitors network performance and routes applications

based on application performance policies

• PfR load balances traffic based upon link utilization levels

to efficiently utilize all available WAN bandwidth

Other traffic is load

balanced to maximize

bandwidth

Voice/Video/Critical will be

rerouted if the current path

degrades below policy thresholds

Voice/Video/Critical take

the best delay, jitter, and/or

loss path


Load Balancing

Policy-Based Path Selection

Network Availability

Secure Connectivity

Scalable, Strong Encryption

App-Aware Threat Defense

Cloud Web Security

Application Optimization

Application Visibility

App Acceleration

Intelligent Caching

Hybrid WAN

Application-centric Design

Common Operational Model

Deployment Flexibility

Cisco Intelligent WANSolution Components for SPs

SP-IWAN: Intelligent Path ControlLeveraging Multiple Paths with Different SLA

MPLS

INET

MC/BR PE

PE

PE

PE

BR1

BR2

MC1

• Developing IWAN Within the SP’s

• IWAN As a Service

• Delivering Value Within the Service Provider Network

path green path-id 1

path blue path-id 2

PfR

Branch

Transit

SP CloudServices

PfR

PfR

SG

Can the Internet Deliver Enterprise Apps?Verizon Booth - Insights from Client Designs and Lab Testing

Opportunities

• Application-level path selection

improves utilization and

resiliency of hybrid WAN

• Beyond MPLS + Internet, IWAN

also enables other architecture

designs such as dual-MPLS

• Economic justification for

increased Internet bandwidth at

branch offices enables new

options

• Some applications (esp “as a

service” comms apps) may

require split tunneling /

centralized provisioning

• Backhaul or breakout? Internet at

branches changes security

requirements

• Centralized policy orchestration

requires unified global QoS

standard

• Always-on transport

requirements and wireless

network design parameters

ConsiderationsHybrid WAN Designs with Cisco

IWAN

• MPLS + Internet

• MPLS + MPLS

• Internet + Internet

• MPLS + Wireless or Satellite

• MPLS Cloud Gateway +

Internet for Cloud Diversity

Network Professional Services

Getting Started with IWAN? www.verizonciscocollaboration.com/HybridWANflightplan

IWAN Layers

MPLS Routing Internet Routing

Overlay Routing Protocol (BGP, EIGRP)

Transport Independent Design (DMVPN)

PfRAVC QoS

Infrastructure Routing

Transport Overlay

Overlay routing

over tunnels

Intelligent Path

Selection

ZBFW

CWS

Why a Transport Independent Design?

MPLS

Branch

Central

Hybrid

Multiple routing domains, Multiple access technologies, multiple paths

INET INET

Branch

Central

Internet

INET

MPLS

Branch

Central

MSP

INETeBGP EIGRP

DMVPN

EIGRP EIGRP eBGP

ezVPN

DMVPN

Transport Independent Design – Overlay Model

DMVPN1

Branch

Central

IWAN

DMVPN2

• Simplified configuration

• Active/Active WAN Paths

• One Overlay – DMVPN

• One WAN Routing Domain – BGP, EIGRP

Consistent VPN Overlay Enables Security Across Transition

iBGP iBGP

iBGP iBGP

PfRv3

• Centralized provisioning

• AVC Infrastructure

• VRF Awareness

• Blackout ~ 1s

• Brownout ~ 2s

• Scale 2000 sites

• Hub config only

PfRv2

• Policy simplification

• App Path Selection

• Blackout ~6s

• Brownout ~9s

• Scale 500 sites

• 10s of lines of config


Today

Performance Routing Evolution

IWAN 2.0

PfR/OER

• Internet Edge

• Basic WAN

• Provisioning per

site per policy

• 1000s of lines of

config

PfRv3

• Multiple Data Centers

• Multiple Next Hop per

DMVPN cloud

DC1 DC2

Supporting Advanced Topology

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

R10 R11 R12 R13

BR2

MC

IWAN POP1 IWAN POP2

MC

DMVPN1MPLS

DMVPN2INET

BR1 BR4BR3 BR6BR5 BR8BR7

10.8.0.0/1610.9.0.0/16 10.8.0.0/16

10.9.0.0/16

• Support for multiple BRs per cloud

• Horizontal scaling

• Support for Multiple POPs

• Different Prefix

• Common Prefix

DCIWAN Core

Horizontal Scaling Architecture

• Requirements

• Multiple DMVPN Hubs per cloud for redundancy and scaling

• HA

- If the current exit/channel to a remote site fails, converge over to an alternate exit/channel on the same (DMVPN1) network. Else, converge over to the alternate (DMVPN2) network.

• Scale

- Distribute traffic across multiple BRs/exits on a single (DMVPN) to utilize all WAN and router capacity.

- Convergence across hubs/pops should only occur when all exits/channels in a hub/pop fail or reach max-bw limits.

BR1 BR2

MC

DMVPN1 DMVPN1

IWAN POP

• Multiple path to

the same DMVPN

• Multiple next hops

in the same

DMVPN

Multiple POPs – Common Prefixes

• Requirements:

– 2 (or more) Transit Sites advertise the very same set of prefixes

– Datacenter may not be collocated with the Transit Sites

– DCs/DMZs are reachable across the WAN Core for each Transit Site

– Branches can access any DC or DMZ across either POP(hub). And, DC/DMZs can reach any branch across multiple Transit Sites (hubs).

– Multiple BRs per DMVPN per site may be required for crypto and bandwidth horizontal scaling

DC1

DCIWAN Core

DC2

BR1 BR3

MC

BR5 BR7

MC

DMVPN1 DMVPN1DMVPN2 DMVPN2

10.8.0.0/16 10.8.0.0/16

IWAN POP1 IWAN POP2

PfRv3 Principles

Define IWAN Traffic Policies

Voice

Interactive Video

Critical Data

Preferred Path

Fallback Path

Delay threshold

Loss threshold

Jitter threshold

Preferred Path

Fallback Path

Delay threshold

Loss threshold

Jitter threshold

Preferred Path

Fallback Path

Delay threshold

Loss threshold

Default N/A Load Balancing

Admin Policies Performance Policies

DSCP

Application

DSCP

Application

DSCP

Application

Default class

Performance Routing

Branch

MC/BR MC/BR MC/BR BR

BR1 BR2 BR3 BR4

MC1

IWAN POP1 IWAN POP2

MC2

DMVPNMPLS

DMVPNINET

DCIWAN Core

DC1 DCn

Branch Branch

Define your Traffic Policy

Centralized on a Domain Controller Application or DSCP based Policy

Preferred Path Load Balancing Performance thresholds (loss, delay and Jitter)

Performance Measurement

Learn the flows and monitor performance Passive Monitoring with Unified Performance

Monitor on the CPE, aka Border Routers

Smart Probing Reports to a local Master Controller

Path Control

Decision on the local Master Controller Path enforcement on the Border Routers

Routing tables unchanged

SITE1

Performance Monitoring

MPLS

INET

Bandwidth on egressPer Traffic Class

(dest-prefix, DSCP, AppName)

CPE1 CPE11

CPE12

CPE10

CPE2

Passive Monitoring

Performance Monitor• Collect Performance Metrics

• Per Channel- Per DSCP

- Per Source and Destination Site

- Per Interface

SITE3Single CPE

SITE2Dual CPE


MPLS

INET

Integrated Smart Probes• Traffic driven – intelligent on/off

• Site to site and per DSCP

Performance Monitor• Collect Performance Metrics

• Per Channel- Per DSCP

- Per Source and Destination Site

- Per Interface

CPE1 CPE11

CPE12

CPE10

CPE2

Smart Probing

SITE3Single CPE

SITE2Dual CPE

SITE1

SITE1

Performance Violation

MPLS

INET

ALERT• From Destination site

• Sent to source site

• Loss, delay, jitter, unreachable

CPE1 CPE11

CPE12

CPE10

CPE2

SITE3Single CPE

SITE2Dual CPE

SITE1

Policy Decision

MPLS

INET

CPE1 CPE11

CPE12

CPE10

CPE2

• Reroute Traffic to a Secondary Path

SITE3Single CPE

SITE2Dual CPE

Domain Discovery

IWAN Domain

Branch


BR1 BR2 BR3 BR4

MC1

IWAN POP1 IWAN POP2

MC2

DMVPNMPLS

DMVPNINET

DCIWAN Core

DC1 DCn

• A collection of sites sharing the same set of policies

• Each site runs Performance Routing components

• They exchange services through the Enterprise Domain Peering framework

• Centralized configuration from a Domain Controller

• Overlay network per Transport for flexibility and simplification

IWAN

Domain

Controller

Branch Branch

TransitTransit

PfR Components

The Decision Maker: Master Controller (MC)

Apply policy, verification, reporting No packet forwarding/ inspection required

Standalone of combined with a BR VRF Aware

The Forwarding Path: Border Router (BR)

Gain network visibility in forwarding path (Learn, measure)

Enforce MC’s decision (path enforcement) VRF aware

BR1 BR2 BR3 BR4

MC1

DMVPNMPLS

DMVPNINET

MC2


PfR Sites

Transit Sites

Enterprise POPs or Hubs Transit to DC or spoke to spoke

• Site Definition:

– Controlled by a local Master Controller (MC)

– Site ID – the IP address of the MC loopback

– One/Multiple BRs

– Each BR one/multiple links

Branch Sites

Stub

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

BR1 BR2 BR3 BR4

MC1

POP1 - TRANSITSite ID = 10.8.3.3

POP2 - TRANSITSite ID = 10.9.3.3

DMVPNMPLS

DMVPNINET

MC2

BRANCH SITESite10Site ID = 10.2.10.10


Hub Master Controller

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

BR1 BR2 BR3 BR4

DC/MC1

HUB SITESite ID = 10.8.3.3

TRANSIT SITESite ID = 10.9.3.3

DMVPNMPLS

DMVPNINET

MC2

BRANCH SITESite10Site ID = 10.2.10.10

Hub MC Transit MC


• One of the MC is assigned the Domain Controller (DC) role

– DC + MC = Hub Master Controller

• Central point of provisioning for Domain policies

• Each POP is allocated an unique POP-ID in the entire domain.

– Hub MC POP-ID = 0

POP-ID 0 POP-ID 1

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

Hub MC

BR1 BR2 BR3 BR4

MC1

Transit Master Controller

• Introduce “Transit Master Controller" concept for the 2nd Transit site

• Behaves like a Hub MC without provisioning

• Each POP is allocated an unique POP-ID in the entire domain

Transit MCMC2

DMVPNMPLS

DMVPNINET




IOS-XE 3.15

IOS 15.5(2)T

POP-ID 1POP-ID 0

Branch Sites

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

BR1


• Hub MC listening for incoming requests

• Branch MC connects to Hub MC

• Service Exchange

– Timers

– Policies and Monitor configurations

– Site Prefixes

Hub MC

BR1 BR3 BR4

MC1Transit MC

MC2



MC Peering

DMVPNMPLS

DMVPNINET

Policies

Monitors

Policies

Monitors

Policies

Monitors

WAN Interface Discovery

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

BR1


Hub MC

BR1 BR3 BR4

MC1Transit MC

MC2



• Transit BRs have path names manually defined, ie MPLS and INET

• Transit BRs send Discovery Packet with path names from to all discovered sites

• Path Discovery from the Hub Border RoutersPath MPLSPath-id 1

Path INETPath-id 2

Path INETPath-id 2

Path MPLSPath-id 1

DMVPNMPLS

DMVPNINET

POP-ID 0 POP-ID 1

WAN Path is detected on the branch

- Path Name

- POP-ID

- Path-Id

- DSCP

WAN Interface – Performance Monitors

• Apply 3 Performance Monitors instances (PMI) over external interfaces

• Monitor1 – Site Prefix Learning (egress direction)

• Monitor2 – Aggregate Bandwidth per Traffic Class (egress direction)

• Monitor3 – Performance measurements (ingress direction)

• Creates a Channel (see later)

BR

2 31 2 31

Site Prefix Discovery

• Every MC in the domain owns a Site Prefix database

• Gives the mapping between site and prefixes

• 2 options:

– Static

– Automatic LearningINETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

TRANSIT SITE

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

Site Prefix – Automatic Learning

1Source Destination DSCP App

10.1.10.200 10.8.1.200 AF41 AppXY

R10

MC

Site-Pfx Mask

10.1.10.0 /24

SAF - Site 10

10.1.10.0/24

SAF - Site 10

10.1.10.0/24

SAF - Site 10

10.1.10.0/24

• Source Prefix and Mask collected from Performance Monitor

• Monitor interval is 30 sec

• BR send to its local MC

• MC send information to all peers via Peering

SAF - Site 10

10.1.10.0/24

IWAN POP1

Site Prefix Discovery

Site Prefix List

Hub 10.8.0.0/16

R10 10.1.10.0/24

R11 10.1.11.0/24

R12 10.1.12.0/24

R12 10.1.13.0/24

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

TRANSIT SITE

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

10.1.10.0/24

MC/BR

BR1 BR2 BR3 BR4

MC1 MC2

Shared Prefixes (M)• Prefix (10.8.0.0/16 in this example) can

belong to multiple Sites.

• Prefix associated with a list of site-ids

• Marked with 'M' flag now in the Site Prefix Database

• A TC may be associated with more than 1 site

DMVPNMPLS

DMVPNINET

SITE-ID PREFIXES FLAGS

10.8.3.3 10.8.0.0/16 S,C,M

10.9.3.3 10.8.0.0/16 S,C,M



R10

Hub MC Transit MC

POP ID 0 POP ID 1

10.8.0.0/16 10.8.0.0/16

IOS-XE 3.15

IOS 15.5(2)T

What is a Traffic Class?

INETMPLS

10.1.10.0/24 10.1.11.0/24 10.1.12.0/24

IWAN POP

MC1

BR1 BR2

R10 R11 R12 R13

Traffic with EF, AF41, AF31 and 0DSCP Based Policies

Prefix DSCP AppID Dest SiteNext-Hop

10.1.11.0/24 EF N/A Site 11 ?

10.1.11.0/24 AF41 N/A Site 11 ?

10.1.11.0/24 AF31 N/A Site 11 ?

10.1.11.0/24 0 N/A Site 11 ?

10.1.10.0/24 EF N/A Site 10 ?

10.1.10.0/24 AF41 N/A Site 10 ?

10.1.10.0/24 AF31 N/A Site 10 ?

10.1.10.0/24 0 N/A Site 10 ?

10.1.12.0/24 EF N/A Site 10 ?

10.1.12.0/24 AF41 N/A Site 10 ?

10.1.12.0/24 AF31 N/A Site 10 ?

10.1.12.0/24 0 N/A Site 10 ?

Traffic Class

Destination Prefix

DSCP Value

Application (N/A when DSCP policies used)

What is a Traffic Class?

INETMPLS

10.1.10.0/24 10.1.11.0/24

IWAN POP

MC1

BR1 BR2

R10 R11 R12 R13

Traffic with EF, AF41, AF31 and 0App1, App2, etc

Application based Policies

Prefix DSCP AppID Dest Site Next-Hop

10.1.11.0/24 EF N/A Site 11 ?

10.1.11.0/24 AF41 App1 Site 11 ?

10.1.11.0/24 AF41 App2 Site 11 ?

10.1.11.0/24 AF31 N/A Site 11 ?

10.1.11.0/24 0 N/A Site 11 ?

10.1.10.0/24 EF N/A Site 10 ?

10.1.10.0/24 AF41 N/A Site 10 ?

10.1.10.0/24 AF31 N/A Site 10 ?

10.1.10.0/24 0 N/A Site 10 ?

10.1.12.0/24 EF N/A Site 10 ?

10.1.12.0/24 AF41 N/A Site 10 ?

10.1.12.0/24 AF31 N/A Site 10 ?

10.1.12.0/24 0 N/A Site 10 ?

10.1.12.0/24Traffic Class

Destination Prefix

DSCP Value

Application (N/A when DSCP policies used)

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

IWAN POP

MC1

BR1 BR2

R10 R11 R12 R13

Source Site: Collecting TC Bandwidth

• Traffic going outside a source site

• Initially based on Routing Information

• Captured by a Performance Monitor on the external interface on egress

• BR reports to its local MC

• Monitor interval is 30 sec (fixed)

Hub MC10.8.3.3/32

Source Destination DSCP App

10.8.1.200 10.1.10.200 AF41 APP1

2

MC1 Dst-Site-Pfx App DSCP Dst-Site-

Id

State BW BR Exit

10.1.10.0 APP1 AF41 10.2.10.1

0

CN 24 BR1 Tu10

DMVPNMPLS

DMVPNINET

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

Destination Site: Collecting Performance MetricsActual User Traffic

• Traffic flow captured on the destination site

• Performance Monitor collects Performance Metrics

• Per Channel

• Default Monitor interval is 30 sec (configurable)

R10 MCChannel Dst-Site-id Path DSCP BW Delay Jitter Loss

5 Hub Tu1 AF41 24 51 2 1

3

IWAN POP


10.8.1.200 10.1.10.200 AF41 APP1

DMVPNMPLS

DMVPNINET

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

Destination Site: Collecting Performance MetricsSmart Probes

• Without actual traffic

• 20 pps for channel without traffic

• IOS-XE: BR sends 10 probes spaced 20ms apart in the first 500ms and another similar 10 probes in the next 500ms

• IOS: BR sends one packet every 50ms

• With actual traffic

• Lower frequency when real traffic is observed over the channel

• Probes sent every 1/3 of [Monitor Interval], ieevery 10 sec by default

• Measured by Performance Monitor just like other data traffic

3

IWAN POP

SmartProbes

What is a Channel?

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

Present Channel 3

• Site 10

• DSCP AF41

• MPLS

• Path 1

Backup Channel 4

• Site 10

• DSCP AF41

• INET

• Path 2

IWAN POP

• A Channel is a unique combination of Interface, sites, next-hop and path

• Created based on real traffic observed on the BRs

• Added every time there is a new DSCP or a new interface or a new site added to the prefix database.

• Smart Probe is received

• On all exits

What is a Channel?

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

Present Channel 13

• Site 11

• DSCP EF

• MPLS

Backup Channel 14

• Site 11

• DSCP EF

• INET

Between Any Pair

of Sites that has

traffic!

IWAN POP

Monitoring Channel

• Monitoring performance per channel

• Channel per destination prefix, DSCP and Path Id

• Include all sites advertising that prefix

• Load balance maybe done between POPs if prefix is shared between multiple transit sites

• Track individual BR performance on the hub

• A PfR-label uniquely identify a path between sites across clouds

10.1.10.0/24

Hub MC

MC/BR

BR1 BR2 BR3 BR4

MC1

Path MPLSId 1

Path MPLSId 1

Transit MCMC2

DMVPNMPLS

DMVPNINET

POP ID 0 POP ID 1

Path INETId 2

Path INETId 2

POP-ID PATH-ID POP-ID PATH-ID

10.8.0.0/1610.9.0.0/16

10.8.0.0/1610.9.0.0/16

10.8.0.0/1610.9.0.0/16

IOS-XE 3.15

IOS 15.5(2)T

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

Enterprise HQ

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

Performance Violation

• Performance notification exported ONLY when there is a violation on a specific channel

• Generated from ingress monitor attached on BRs to the source site MC

• Based on Monitor interval (30 sec default, configurable)

• Via all available external interfaces.

3

R10

Channel Dst-Site-id DSCP Path BW Delay Jitter Loss

5 Hub AF41 Tu1 24 250 2 1

R10

TCA Delay

DSCP AF41

Path MPLS

Performance Violation – Detected on Dst Site

R10

TCA Delay

DSCP AF41

Path MPLS

Dst-Site-Pfx Dst-Site-id App DSCP State BR Exit

10.1.10.0 R10 APP1 AF41 CN BR1 Tu1

10.1.10.0 R10 N/A AF41 CN BR1 Tu1

10.1.10.0 R10 N/A AF31 CN BR1 Tu1

10.1.10.0 R10 N/A 0 CN BR2 Tu2

10.1.11.0 R11 N/A EF CN BR1 Tu1

10.1.11.0 R11 N/A AF31 CN BR1 Tu1

10.1.11.0 R11 N/A 0 CN BR2 Tu2

10.1.12.0 R12 N/A 0 CN BR2 Tu2

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

Enterprise HQ

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

3

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

Enterprise HQ

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

3

Policy Decision – Reroute TC

• MC computes a new path for each impacted TC

• MC tells the BRs to enforce the new path

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

Enterprise HQ

MC1

BR1 BR2

R10 R11 R12 R13

Hub MC10.8.3.3/32

3

Reroute TC – Path Enforcement

• Dataplane forwarding

• Activated on all but external interfaces

• Lookup per packet - output-if/next hop retrieved

• Packet Forwarded

• If no entry – Uses FIB entry

• TC flows redirected to the new path over the auto mGRE tunnels between the BRs

• No change in the routing table

Deploying IWAN Intelligent Path Control

Performance Routing – Platform Support

Cisco ISR G2 family

3900-AX2900-AX1900-AX

890

Cisco ISR 4000

44004300

Cisco ASR-1000

Cisco CSR-1000

MCBR

MCBR

MCBR

MCBR(1)

(1) Future

IWAN Deployment – DMVPN

• Transport Independent Design based on DMVPN

• Branch spoke sites establish an IPsec tunnel to and register with the hub site

• Data traffic flows over the DMVPN tunnels

• WAN interface IP address used for the tunnel source address (in a Front-door VRF)

• One tunnel per user inside VRF

• Overlay Routing

• BGP or EIGRP are typically used for scalability

• IP routing exchanges prefix information for each site

• Per-tunnel QOS is applied to prevent hub site oversubscription to spoke sites

• Optional: Performance Monitoring (Advanced IWAN)

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

R10 R11 R12 R13

R84 R85 R94 R95

R83

IWAN POP1 IWAN POP2

R93

MPLSDMVPN

INETDMVPN

DCIWAN Core

http://docwiki.cisco.com/wiki/PfR3:Solutions:IWA

N

10.8.0.0/1610.9.0.0/16

10.8.0.0/1610.9.0.0/16

Front-door VRF

MPLS VRF

default

INSIDE

OUTSIDE

MPLS VRF

default

MPLS

VPN-DMZ

Internet Edge

INTERNET VRF

default

Internet default

INSIDE

OUTSIDE default

INTERNET

VRF default

EIG

RP

Internet

Branch

DMVPN

Hub

R84 R85

R10

DMVPN Configuration – FVRF

vrf definition IWAN-TRANSPORT-1

!

address-family ipv4

exit-address-family

!

Front-door VRF definition for

MPLS Transport

TRANSIT SITE

R83

R84 R85

172.16.84. 4 172.16.85.5

MPLS INTERNET

10.1.10.0/24

R10

172.16.101.10 172.16.102.10

vrf definition IWAN-TRANSPORT-2

!

address-family ipv4

exit-address-family

!

For YourReference

DMVPN Configuration – IPSeccrypto ikev2 keyring DMVPN-KEYRING-1

peer ANY

address 0.0.0.0 0.0.0.0

pre-shared-key c1sco123

!

!

crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-1

match fvrf IWAN-TRANSPORT-1

match identity remote address 0.0.0.0

authentication remote pre-share

authentication local pre-share

keyring local DMVPN-KEYRING-1

!

crypto ipsec security-association replay window-size 512

!

crypto ipsec transform-set GCM256/SHA/TRANSPORT esp-gcm 256

mode transport

!

crypto ipsec profile DMVPN-PROFILE-1

set transform-set AES256/SHA/TRANSPORT

set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-1

crypto ikev2 dpd 40 5 on-demand Set DPD timers for Branch

Configs ONLY!

!

R83

R84 R85

10.0.100.84 10.0.200.85

10.1.10.0/24

R10

10.0.100.10

Maximise window size to

eliminate future anti-replay issue

10.0.200.10

TRANSIT SITE

For YourReference

DMVPNMPLS

DMVPNINTERNET

DMVPN Hub Configuration – Interfaces & Routinginterface GigabitEthernet0/0/3

description MPLS-TRANSPORT

vrf forwarding IWAN-TRANSPORT-1

ip address 172.16.84.4 255.255.255.0

!

interface Tunnel100

bandwidth 100000

ip address 10.0.100.84 255.255.255.0

no ip redirects

ip mtu 1400

ip pim nbma-mode

ip pim sparse-mode

ip nhrp authentication cisco123

ip nhrp map multicast dynamic

ip nhrp network-id 100

ip nhrp holdtime 600

ip nhrp redirect

ip tcp adjust-mss 1360

tunnel source GigabitEthernet0/0/3

tunnel mode gre multipoint

tunnel key 101

tunnel vrf IWAN-TRANSPORT-1

tunnel protection ipsec profile DMVPN-PROFILE-1

!

ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 172.16.84.8

Tunnel endpoint is in Front-door VRF

MPLS TRANSPORT – R84

Put Transport Interface into

MPLS Front-door VRF

Define the bandwidth

R83

R84 R85

Map to Physical Interface

Set DMVPN Ph3

DMVPN Network ID: MPLS

Default route for Tunnel endpoints

10.0.100.84 10.0.200.85

TRANSIT SITE

For YourReference

DMVPNMPLS

DMVPNINTERNET

DMVPN Spoke Configuration – Interfaces & Routinginterface GigabitEthernet0/1

vrf forwarding IWAN-TRANSPORT-1

ip address 172.16.101.10 255.255.255.0

!

interface Tunnel100

bandwidth 1000

ip address 10.0.100.10 255.255.255.0

no ip redirects

ip mtu 1400

ip pim dr-priority 0

ip pim sparse-mode

ip nhrp authentication cisco123

ip nhrp network-id 100

ip nhrp holdtime 600

ip nhrp nhs 10.0.100.84 nbma 172.16.84.4 multicast

ip nhrp nhs 10.0.100.94 nbma 172.16.94.4 multicast

ip nhrp registration no-unique

ip nhrp shortcut

ip tcp adjust-mss 1360

if-state nhrp

tunnel source GigabitEthernet0/1

tunnel mode gre multipoint

tunnel key 101


tunnel protection ipsec profile DMVPN-PROFILE-1

!

ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 172.16.101.8

Set DMVPN Ph3

Tunnel endpoint is in Front-

door VRF 10.1.10.0/24

R10

10.0.100.10

Put Transport Interface into

MPLS Front-door VRF

Instantiate DMVPN Tunnel

DMVPN Network ID: MPLS

Multiple DMVPN Hub for

Resiliency

Default route for Tunnel endpoints

Tunnel state based on NHRP

Note: ip nhrp registration no-unique

Should be use only on dynamically addressed spokes (usually over the Internet

10.0.200.10

For YourReference

Overlay Routing Protocols

• IWAN Profiles are based upon BGP and EIGRP for scalability and optimal Intelligent Path Control

• Intelligent Path Control:

• PfR can be used with any routing protocols by relying on the routing table (RIB). • Requires all valid WAN paths be ECMP so that each valid path is in the RIB.

• For BGP and EIGRP, PfR can look into protocol’s topology information to determine both best paths and secondary paths thus, ECMP is not required.

• PfRv3 always checks for a parent route before being able to control a Traffic Class. Parent route check is done as follows:

• Check to see if there is an NHRP shortcut route

• If not – Check in the order of BGP, EIGRP, Static and RIB

• Make sure that all Border Routers have a route over each external path to the destination sites PfR will NOT be able to effectively control traffic otherwise.

Which protocol should I use?

Routing Deployment – EIGRP

• Single EIGRP process for Branch, WAN and POP/hub sites

• Extend Hello/Hold timers for WAN

• Adjust tunnel interface “delay” to ensure WAN path preference

• MPLS primary, INET secondary

• Hubs

• Route tag filtering to prevent routing loops across DMVPNs

• Branch prefix summary route forspoke-to-spoke tunnels

• Spokes

• EIGRP Stub for scalability10.1.10.0/24 10.1.11.0/24

10.1.12.0/2410.1.13.0/24

Delay 1000

Delay2000

Delay 1000

Set InterfaceDelay to

influence best path

EIGRPStub R10 R11 R12 R13

TAGDMVPN-1

TAGDMVPN-2

IWAN POP1

R83

IWAN POP2

R84 R85

10.8.0.0/1610.9.0.0/16

10.8.0.0/1610.9.0.0/16

INETMPLS10.0.200.0/2410.0.100.0/24

Routing Deployment – BGP

• A single iBGP routing domain is used

• Set BGP Hello/Hold timers for IWAN (20, 60)

• POP (Hub) Sites

• DMVPN hub are also BGP route-reflectors for the spokes

• BGP dynamic peering configured on route-reflectors

• Default and Internal Summary routes to spokes

• Set Community and Local Pref for all prefixes

• Redistribute BGP into local IGP

• Branch (Spokes) Sites:

• peer to each DMVPN/BGP route reflector in each POP

• Local Pref set at POPs to prefer MPLS over INET Paths

• Branches are Stubs and only advertise local prefixes• Redistribute - connected, IGP routes

INETMPLS

10.7.10.0/24 10.7.11.0/2410.7.12.0/2410.7.13.0/24

IWAN POP1

iBGPCommunity 6:34 Community 6:36Local Pref 1000 Local Pref 200

BR1 BR2

R10 R11 R12 R13

10.4.0.0/1610.6.32.0/24

10.6.36.0/2310.6.34.0/23

IWAN POP2 10.5.0.0/1610.6.33.0/24

IGP IGP

IGP

10.4.0.0/1610.0.0.0/80.0.0.0

10.4.0.0/1610.0.0.0/80.0.0.0

10.1.12.0/2410.1.13.0/2410.1.12.254/32

PfR Deployment – Hubdomain IWAN

vrf default

master hub

source-interface Loopback0

enterprise-prefix prefix-list ENTERPRISE_PREFIX

site-prefixes prefix-list DC_PREFIX

domain IWAN

vrf default

border

master 10.8.3.3


!

interface Tunnel100

description -- Primary Path --

domain IWAN path MPLS path-id 1

domain IWAN

vrf default

border

master 10.8.3.3


!

interface Tunnel200

description – Secondary Path --

domain IWAN path INET path-id 2

R83

R84

R85

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

Hub MC

R10 R11 R12 R13

R84 R85 R94 R95

R83

Path MPLSId 1

Path INETId 2

R93

DMVPNMPLS

DMVPNINET

POP ID 0


• Enterprise Prefix: summary prefix for the entire domain

• Site Prefix: no automatic learning – Mandatory

• POP Id unique per domain

• Path ID unique per Site

Enterprise Prefix List• The main use of the enterprise prefix list is to determine the

enterprise boundary.

• With enterprise-prefix

• If a prefix doesn't match any site-prefix but matches enterprise-prefix then the prefix belongs to a site that is not participating in PfRv3 but it does belong to the enterprise.

• PfR will not influence traffic towards sites that has NOT enabled PFR.

• Without enterprise-prefix

• All the traffic that would be going towards a spoke that is NOT PfR enabled will be learnt as internet traffic class and therefore subjected to load balancing.

HUBMaster MC

domain IWAN

vrf default

master hub


enterprise-prefix prefix-list ENTERPRISE_PREFIX

!

ip prefix-list ENTERPRISE_PREFIX seq 10 permit 10.0.0.0/8

HMC

Site Prefixes – Static Configuration

• This allows configuring site-prefix manually instead of learning.

• This configuration should be used at the site if the site is used for transit.

• For example, Site A reaches Site B via Hub-Site, where Hub-Site is transit site. The configuration is used to prevent learning of Site A prefix as Hub-Site prefix when it is transiting from Hub.

domain IWAN

vrf default

master hub


site-prefixes prefix-list DC1_PREFIX

!

ip prefix-list DC1_PREFIX seq 10 permit 10.8.0.0/16

!

MC1

BR1 BR2


10.1.10.200 10.1.11.200 AF41 AppXY

TRANSIT SITE

IWAN 2.0 – HUB-MC Scaling

ISR 443150 sites

ASR 1001-X1000 sites

ISR 4451200 sites

ASR 1002-X2000 sites

CSR1000v1 vCPU

200 sites

CSR1000v 2 vCPU

500 sites

IWAN Policies – DSCP or App Baseddomain IWAN

vrf default

master hub

load-balance

class MEDIA sequence 10

match application <APP-NAME1> policy real-time-video

match application <APP-NAME2> policy custom

priority 1 one-way-delay threshold 200

priority 2 loss threshold 1

path-preference MPLS fallback INET

class VOICE sequence 20

match dscp <DSCP-VALUE> policy voice


class CRITICAL sequence 30

match dscp af31 policy low-latency-data

• Policies:

– DSCP or Application Based Policies (NBAR2)

– DSCP marking can be used with NBAR2 on the LAN interface (ingress on BR)

• Default Class is load balanced

R83

• Pre-defined thresholds

• Custom thresholds

Built-in Policy Templates

Pre-defined

Template

Threshold Definition

Voice priority 1 one-way-delay threshold 150 threshold 150 (msec)

priority 2 packet-loss-rate threshold 1 (%)

priority 2 byte-loss-rate threshold 1 (%)

priority 3 jitter 30 (msec)

Real-time-video priority 1 packet-loss-rate threshold 1 (%)


priority 2 one-way-delay threshold 150 (msec)

priority 3 jitter 20 (msec)

Low-latency-data priority 1 one-way-delay threshold 100 (msec)



Pre-

defined

Template

Threshold Definition

Bulk-data priority 1 one-way-delay threshold 300 (msec)



Best-effort priority 1 one-way-delay threshold 500 (msec)



scavenger priority 1 one-way-delay threshold 500 (msec)



For YourReference

IWAN Deployment – Recommended Policies

domain IWAN

vrf default

master hub

load-balance


match dscp ef policy voice


class INTERACTIVE-VIDEO sequence 20

match dscp cs4 policy real-time-video

match dscp af41 policy real-time-video

match dscp af42 policy real-time-video


class CRITICAL-DATA sequence 30

match dscp af21 policy low-latency-data


R83

CVD IWAN 2.-0 – DSCP Based Policies recommended

• When load balancing is enabled, PfRv3 adds a

“default class for match all DSCP (lowest priority

compared to all the other classes)” and PfRv3

controls this traffic.

• When load balancing is disabled, PfRv3 deletes this

“default class” and as a part of that frees up the TCs

that was learnt as a part of LB – they follow the

routing table

• Pre-defined thresholds

• Custom thresholds can also be used

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

R83

R84 R85

R10 R11 R12 R13

Redundant MC – Anycast IP

Backup Hub MC10.8.3.3/31R83b

Hub MC10.8.3.3/32

HUB SITE

• What happens when a MC fails?

• Traffic forwarded based on routing information –ie no drop

• What happens when the Hub MC fails?

• Branch MCs keep their configuration and policies

• Continue to optimize traffic

• A backup MC can be defined on the hub.

• Using the same IP address as the primary

• Routing Protocol is used to make sure BRs and branch MC connect to the primary

• Stateless redundancy

• Backup MC will re-learn the traffic

PfR Deployment – Transit Sitedomain IWAN

vrf default

master transit 1


site-prefixes prefix-list DC_PREFIX

hub 10.8.3.3

domain IWAN

vrf default

border

master 10.9.3.3


!

interface Tunnel100

description -- Primary Path --


domain IWAN

vrf default

border

master 10.9.3.3


!

interface Tunnel200

description – Secondary Path --

domain IWAN path INET path-id 2

R93

R94

R95

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

R10 R11 R12 R13

R84 R85 R94 R95

R83

Path MPLSId 1

Path INETId 2

Transit MCR93

DMVPNMPLS

DMVPNINET

POP ID 1


• Site Prefix: no automatic learning – Mandatory

• POP Id unique per domain

• Path ID unique per Site

• Peering with Hub MC

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

R10 R11 R12 R13

R84 R85 R94 R95

R83 R93

DMVPNMPLS

DMVPNINET



PfR Deployment – Single CPE Branch

• Single CPE Branch Sites

• Branch MCs connect to the Hub

domain IWAN

vrf default

master branch


hub 10.8.3.3

border

master local


R10

R10 R11 R12 R13

PfR Deployment – Dual CPE Branch

• Dual CPE Branch Sites

• Branch MCs connect to the Hub

domain IWAN

vrf default

border

master 10.2.12.12


R13

domain IWAN

vrf default

master branch


hub 10.8.3.3

border

master local


R12

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

R84 R85 R94 R95

R83 R93

DMVPNMPLS

DMVPNINET



R10 R11 R12 R13

R10 R11 R12 R13

IWAN Peering

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

R85

R10 R11 R12 R13

10.2.10.10

Hub MC

R84 R94 R95

R83Transit MC

R93



R83#sh eigrp service-family ipv4 neighbors

EIGRP-SFv4 VR(#AUTOCFG#) Service-Family Neighbors for AS(59501)

H Address Interface Hold Uptime SRTT RTO Q Seq

(sec) (ms) Cnt Num

5 10.2.10.10 Lo0 513 01:17:12 65 390 0 39

4 10.2.11.11 Lo0 582 01:17:01 59 354 0 40

3 10.2.12.12 Lo0 510 01:17:04 61 366 0 78

2 10.8.4.4 Lo0 538 01:17:04 1 100 0 15

1 10.8.5.5 Lo0 562 01:17:05 4 100 0 18

0 10.9.3.3 Lo0 546 01:17:15 5 100 0 40

MC1#

• Hub MC gets all remote MC IP Addresses => Site-Ids

Branch Sites

10.1.10.0/24

R10

• External interfaces discovered

• Branch MC receives and applies the monitors

R10#show domain IWAN master status

Master VRF: Global

Instance Type: Branch

Instance id: 0

Operational status: Up

Configured status: Up

[SNIP]

Minimum Requirement: Met

[SNIP]

R10#show domain IWAN master status

Borders:

IP address: 10.2.10.10

Version: 2

Connection status: CONNECTED (Last Updated 00:47:53 ago )

Interfaces configured:

Name: Tunnel100 | type: external | Service Provider: MPLS | Status: UP | Zero-SLA: NO

Number of default Channels: 2

Path-id list: 1:1 0:0 0:1

Name: Tunnel200 | type: external | Service Provider: INET | Status: UP | Zero-SLA: NO

Number of default Channels: 2

Path-id list: 1:2 0:0 0:2

Tunnel if: Tunnel0

R83#shmtcs

APP - APPLICATION, TC-ID - TRAFFIC-CLASS-ID, APP-ID - APPLICATION-ID

SP - SERVICE PROVIDER, PC = PRIMARY CHANNEL ID,

BC - BACKUP CHANNEL ID, BR - BORDER, EXIT - WAN INTERFACE

UC - UNCONTROLLED, PE - PICK-EXIT, CN - CONTROLLED, UK - UNKNOWN

Dst-Site-Pfx Dst-Site-Id APP DSCP TC-ID APP-ID State SP PC/BC BR/EXIT

10.1.12.0/24 10.2.12.12 N/A default 3 N/A CN MPLS 4/NA 10.8.4.4/Tunnel100

10.1.12.0/24 10.2.12.12 N/A ef 2 N/A CN INET 8/5 10.8.5.5/Tunnel200

10.1.12.0/24 10.2.12.12 N/A af31 4 N/A CN MPLS 6/7 10.8.4.4/Tunnel100

10.1.11.0/24 10.2.11.11 N/A default 8 N/A CN MPLS 10/NA 10.8.4.4/Tunnel100



10.1.10.0/24 10.2.10.10 N/A default 7 N/A CN INET 11/12 10.8.5.5/Tunnel200



Total Traffic Classes: 9 Site: 9 Internet: 0

R83#

Traffic Classes Summary

Traffic Class – Site11 - Critical TC Id Controlled Path Information - Channels

Dst-Site-Prefix: 10.1.12.0/24 DSCP: ef [46] Traffic class id:2

Clock Time: 20:49:31 (CET) 06/01/2015

TC Learned: 01:03:18 ago

Present State: CONTROLLED

Current Performance Status: in-policy

Current Service Provider: MPLS path-id:1 since 00:01:31

Previous Service Provider: INET pfr-label: 0:0 | 0:2 [0x2] for 3439 sec

BW Used: 14 Kbps

Present WAN interface: Tunnel100 in Border 10.8.4.4

Present Channel (primary): 5 MPLS pfr-label:0:0 | 0:1 [0x1]

Backup Channel: 8 INET pfr-label:0:0 | 0:2 [0x2]

Destination Site ID bitmap: 0

Destination Site ID: 10.2.12.12

Class-Sequence in use: 10

Class Name: VOICE using policy User-defined

priority 2 packet-loss-rate threshold 5.0 percent

priority 1 one-way-delay threshold 150 msec

priority 2 byte-loss-rate threshold 5.0 percent

BW Updated: 00:00:00 ago

Check Traffic Classes Details

Check Traffic Class

Voice for site 10

Check Channels

used (Primary and

Backup)

Class name and

Policies

Active Path used

Reason for Latest Route Change: Backup to Primary path preference transition

Route Change History:

Date and Time

Previous Exit

Current Exit

Reason

1: 20:48:00 (CET) 06/01/2015

INET/10.8.5.5/Tu200 (Ch:8)

MPLS/10.8.4.4/Tu100 (Ch:5)

Backup to Primary path preference transition

2: 19:50:41 (CET) 06/01/2015

MPLS/10.8.4.4/Tu100 (Ch:5)

INET/10.8.5.5/Tu200 (Ch:8)

One Way Delay : 252 msec*

3: 19:46:44 (CET) 06/01/2015

None/0.0.0.0/None (Ch:0)

MPLS/10.8.4.4/Tu100 (Ch:5)

Uncontrolled to Controlled Transition

Check Traffic Classes Details (Cont’d)

Not the actual output – reformatted for better reading

Last 5 reasons for change

Channel Id: 5 Dst Site-Id: 10.2.12.12 Link Name: MPLS DSCP: ef [46] pfr-label: 0:0 | 0:1 [0x1] TCs: 1

Channel Created: 01:13:07 ago

Provisional State: Initiated and open

Operational state: Available

Channel to hub: FALSE

Interface Id: 11

Supports Zero-SLA: Yes

Muted by Zero-SLA: No

Estimated Channel Egress Bandwidth: 15 Kbps

Immitigable Events Summary:

Total Performance Count: 0, Total BW Count: 0

ODE Stats Bucket Number: 1

Last Updated : 00:14:40 ago

Packet Count : 40

Byte Count : 3360


Loss Rate Pkts: 0.0 %

Loss Rate Byte: 0.0 %

Jitter Mean : 43000 usec

Unreachable : FALSE

Check Channel after TCA

TCA Statistics:

Received:801 ; Processed:801 ;

Unreach_rcvd:0

Latest TCA Bucket

Last Updated : 00:14:42 ago


Loss Rate Pkts: NA

Loss Rate Byte: NA

Jitter Mean : NA

Unreachability: FALSE

On Demand Export

(ODE)

Threshold Crossing

Alert (TCA)

Load Balancing

• Current Situation

- Load balancing works on physical links only

- both R84 and R94 share the same physical link(both the MPLS NH's converge into the same physical link on the spoke) and so PfR will think of both of them having the same Physical link BW

• Default Classes TCs

- Load balancing at any time (not only at creation time).

- TC will be moved to ensure bandwidth on all links is within the defined range

• Performance TCs

- Initial load-balancing while placing the TCs, on a per TC basis. PfR does not account for the TCs getting fatter.

Hub MC

R84 R85 R94 R95

R83

Path MPLSId 1

Path INETId 2

R93

POP ID 0

Transit MC

POP ID 1

Path MPLSId 1

Path INETId 2

R10

MPLS INET

Failover Time

10.1.10.0/24

Hub MC

MC/BR

BR1 BR2 BR3 BR4

MC1

Path MPLSId 1

Path MPLSId 1

Transit MCMC2

DMVPNMPLS

DMVPNINET

POP ID 0 POP ID 1



Path INETId 2

Path INETId 2

• Channel Unreachable: ~1 sec

• PfRv3 considers a channel reachable as long as the site receives a PACKET on that channel

• A channel is declared unreachable in both direction if

• There is NO traffic on the Channel, probes are our only way of detecting unreachability. So if no probe is received within 1 sec, we detect unreachability.

• When there IS traffic on the channel, if we don’t see any packet for more than a second on a channel we detect unreachability.

• Ingress Performance Violation detected

• Delay, loss or jitter thresholds

• Based on Monitor-interval

domain IWAN

vrf default

master hub

monitor-interval 2 dscp ef

monitor-interval 2 dscp af41

monitor-interval 2 dscp cs4

monitor-interval 2 dscp af31

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

BR1 BR2 BR3 BR4

MC1

Direction from POPs to Spokes

• Each POP is a unique site by itself and so it will only control traffic towards the spoke on the WAN’s that belong to that POP.

• PfRv3 will NOT be redirecting traffic between POP across the DCI or WAN Core. If it is required that all the links are considered from POP to spoke, then the customer will need to use a single MC.

Path MPLSId 1

Path INETId 2

Path MPLSId 1

Path INETId 2

MC2

DMVPNMPLS

DMVPNINET




Direction from Spoke to HUB

• The spoke considers all the paths (multiple NH’s) towards the POPs and will be maintaining a list of Active/Standby candidate next hops per prefix and interface, that will be derived based on the routing configuration.

• Active next hop: A next hop is considered active for a given prefix if it has the best metric.

• Standby next hop: A next hop is considered standby for a given prefix if it advertises a route for the prefix but does not have the best metric.

• Unreachable next hop: A next hop is considered unreachable for a given prefix if it does not advertise any route for the prefix.

Direction from Spoke to HUB

• With Path Preference

• PfR will be giving Path Preference more priority and so our candidate channels for a particular traffic-class will first consider all the links belonging to the preferred path preference (i.e it will include the active and then the standby links belonging to the preferred path) and will then go to the fallback provider links.

• Without Path Preference

• PfR will give preference to the active channels and then the standby channels (active/standby will be per prefix) with respect to the performance and policy decisions.

• Note that the Active and Standby channels per prefix will span across the POP’s.

• Spoke will randomly (hash) choose the active channel

10.1.10.0/24

R10

R85 R94 R95

R83 R93

Example

DMVPNMPLS

DMVPNINET

10.8.0.0/16 10.8.0.0/16



10.0.100.84 10.0.100.94

R84

• PfR Policies:

– Voice, Video and Business App with Path Preference DMVPN-MPLS

– Default Class Load Balanced

• Routing Configuration will determine the use of possible next-hops

10.8.0.0/16

10.0.0.0/8

BGP

10.8.0.0/16

10.0.0.0/8

BGP

R11

10.1.11.0/24

10.0.200.85 10.0.200.95

Path Selection – Transit Site Affinity

10.1.10.0/24

R10

R85 R94 R95

R83 R93

DMVPNMPLS

DMVPNINET

10.8.0.0/16 10.8.0.0/16

DC1Site ID = 10.8.3.3

DC2Site ID = 10.9.3.3

R84

LP201 LP200LP151 LP150

PREFIX INTERFACENEXT-

HOPSBGP LP Status

10.8.0.0/16

With PfR Path

Preference

R84

R94

R85

R95

201

200

151

150

Preferred – Active

Preferred – Standby

Fallback - Active -

Fallback – Standby

Without PfR

Path Preference

R84

R85

R94

R95

201

151

200

150

Active

Active

Standby

Standby

• Routing policies

– MPLS DMVPN best path

– In each DMVPN cloud, DC1 preferred (R84 and R85)

• PfR Policies:

– MPLS DMVPN preferred

– INET DMVPN fallback

• Important Note:

– Single NH used in each tunnel – failover to the standby NH if OOP

– Load balancing is only between external interfaces, not within a DMVPN cloud.

Path Selection – No Transit Site Affinity

10.1.10.0/24

R10

R85 R94 R95

R83 R93

DMVPNMPLS

DMVPNINET

10.8.0.0/16 10.8.0.0/16

R84

LP200 LP200LP150 LP150

PREFIX INTERFACENEXT-

HOPSBGP LP Status

10.8.0.0/16

With PfR Path

Preference

R84

R94

R85

R95

200

200

150

150



Fallback - Active -

Fallback – Active

Without PfR

Path Preference

R84

R85

R94

R95

200

150

200

150

Active

Active

Active

Active

• Routing policies

– MPLS DMVPN best path

– In each DMVPN cloud, equal cost between all next-hops

• PfR Policies:

– MPLS DMVPN preferred

– INET DMVPN fallback

DC1Site ID = 10.8.3.3

DC2Site ID = 10.9.3.3

• Important Note:

– Multiple NH active on a tunnel but still only one is used for traffic forwarding

– Load balancing is only between external interfaces, not within a DMVPN cloud.

Direct Internet Access Routing with F-VRF

• Direct Internet Access managed outside of PfRv3 with a combination of route-maps and default route

• Internet Traffic forwarded to the external interface pointing toward Internet

• No option to forward a subset of the traffic except the use of PBR and IP addresses

• Path Control:

• PBR or Default Route

• NAT

• Return Traffic: leak from F-VRF to Global

Forward Traffic to

get to the Internet

directly to the

physical interfaceInternet

G0/0

VRF INET-PUBLIC1

Global Table

CPE

MPLS VPN

VRF INET-PUBLIC2

IOS NAT/FW

IOS NAT/FW

DM

VP

N

DM

VP

N

Policy Route for

10.0.0.0/8 traffic

Set next-hop VRF to

Global Table

Deploying with Multiple VRFs

Deploying with user VRFs

• DMVPN Tunnel per VRF

• Over the top routing per VRF

• SAF Peering per VRF

Enterprise Branch Sites

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

MC1

R84 R85

R10 R11 R12 R13

vrf definition TEST1

!

address-family ipv4

exit-address-family

!

vrf definition TEST2

!

address-family ipv4

exit-address-family

!

1 2 1 2

1 2 1 2 1 2 1 2

interface Tunnel 101

vrf forwarding TEST1

tunnel key 101


!

interface Tunnel 102


tunnel key 102


TRANSIT SITE

Deploying with VRF – Hub MC


INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

MC1

R84 R85

R10 R11 R12 R13

interface Loopback1


!

interface Loopback2


domain IWAN

vrf TEST1

master hub


!

vrf TEST2

master hub


GLOBAL: 10.8.3.3VRF TEST1: 11.8.3.3VRF TEST2: 12.8.3.3

TRANSIT SITE

Deploying with VRF – Hub MC Policies

domain IWAN

vrf TEST1

master hub

load-balance




class VIDEO sequence 20

match dscp af41 policy voice



match dscp af31 policy low-latency-

data

[Cont’d]

vrf TEST2

master hub

load-balance





match dscp af31 policy low-latency-

data

Deploying with VRF – Hub BR


INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

MC1

R84 R85

R10 R11 R12 R13

domain IWAN

vrf TEST1

border

master 11.8.3.3


!

vrf TEST2

border

master 12.8.3.3


!

interface Tunnel101

description -- Primary Path –



!

interface Tunnel102

description -- Primary Path –



Tu101Tu102


TRANSIT SITE

Deploying with VRF – Branch MC/BR


INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

MC1

R84 R85

R10 R11 R12 R13

Tu101Tu102

domain IWAN

vrf TEST1

master branch


hub 11.8.3.3

border

master local


!

vrf TEST2

master branch


hub 12.8.3.3

border

master local


R10


TRANSIT SITE

PfRv3 Management

PfRv3 Exporter Configuration

• Enable exporter on the Hub MC

• Distributed through SAF to all MCs and BRs in the domain

• Cisco Prime Infrastructure 3.0

• LiveAction 4.3

• All records available at:

• http://docwiki.cisco.com/wiki/PfRv3:Reporting

domain IWAN

vrf default

master hub

collector 10.151.1.95 port 2055

MC1

INETMPLS

10.1.10.0/24 10.1.11.0/2410.1.12.0/2410.1.13.0/24

Hub MC10.8.3.3/32

IWAN POP

MC1

BR1 BR2

R10 R11 R12 R13

http://docwiki.cisco.com/wiki/PfRv3:Reporting

PfRv3 Syslogs

• Syslog messages for all major PfRv3 events

• Use cisco standard format (Facility-Severity-Mnemonic) for all syslogs with common Facility name 'DOMAIN’

• Add TCA-ID to all syslog to allow correlation of TCA syslog to PFR reaction syslog. If PFR action is not related to TCA then TCA-ID will be 0

• Command '[no] logging' in domain submode default is syslog on

• Distributed through SAF to all MCs and BRs in the domain

• http://docwiki.cisco.com/wiki/PfRv3:Syslogs

• DOMAIN-2-IME

• DOMAIN-2-IME_DETAILS

• DOMAIN-4-MC_SHUTDOWN

• DOMAIN-5-TCA

• DOMAIN-6-TC_CTRL

• DOMAIN-5-TC_PATH_CHG

• DOMAIN-3-PLR_INT_CFG

• DOMAIN-5-MC_STATUS

IOS-XE 3.16

IOS 15.5(3)M

*Jun 1 18:50:41.104: %DOMAIN-5-TC_PATH_CHG: Traffic class Path

Changed. Details: Instance=0: VRF=default: Source Site ID=10.8.3.3:

Destination Site ID=10.2.11.11: Reason=Delay: TCA-ID=4: Policy

Violated=VOICE: TC=[Site id=10.2.11.11, TC ID=6, Site

prefix=10.1.11.0/24, DSCP=ef(46), App ID=0]: Original Exit=[CHAN-

ID=14, BR-IP=10.8.4.4, DSCP=ef[46], Interface=Tunnel100,

Path=MPLS[label=0:0 | 0:1 [0x1]]]: New Exit=[CHAN-ID=13, BR-

IP=10.8.5.5, DSCP=ef[46], Interface=Tunnel200, Path=INET[label=0:0 |

0:2 [0x2]]]

http://docwiki.cisco.com/wiki/PfRv3:Syslogs

Management Solutions for Cisco Infrastructure

• Customer wants advanced

provisioning, life cycle

management, and

customized policies

• Multi-tenant

• System-wide network

consistency assurance

• Lean IT OR IT Network team

On-Prem

Prime

Infrastructure

• Customer needs feature

configurable enterprise

network management and

end-to-end monitoring

• One Assurance across

Cisco portfolio from Branch

to Datacenter

• IT Network team

Enterprise Network

Mgmt and Monitoring

Cloud-Based

IWAN App

• Customer wants massive

simplicity and operational

automation

• Highly consistent network

requirement with prescriptive

Cisco Validated Designs

• Lean IT

Prescriptive

Policy Automation

• Customer looking for

advanced monitoring and

visualization

• Network troubleshooting and

QoS/ PfR/ AVC configuration

• Real-time analytics and

flow/device scalability

• IT Network team

Application Aware

Performance Mgmt

Advanced

Orchestration

Two Deployment Modes for SDN led Provisioning

Application Policy Infrastructure Controller (APIC) – Enterprise Module (EM)

FEATURE CONFIGURABLE NMS with APIC-EM

Cisco Prime Infrastructure

Prime Infra NMS integrated with APIC-EM

providing full GUI based configuration and

FCAPS management orchestrated by the

System of Automation

Customer,

Partner or 3rd

party

developed

Automation

Custom apps utilizing feature programmability via Prime NB APIs for configuration and data

POLICY PRESCRIPTIVE APPS on APIC-EM

IWAN Access Wireless.

.

CollabSegme

ntation

Threat

Defense

.

.

Cisco developed modular, policy

automated management apps with

common UI/UX framework with and

embedded service automation

Customer,

Partner or 3rd

party

developed

Apps

Custom apps utilizing

policy programmability

via APIC-EM NB

REST APIs

Control and Customization Business Driven

PKI Service PnP Service

IWAN App

Key Takeaways

Performance Routing Phases – Summary PfR/OER version 1

IOS 12.3(8)T, XE 2.6.1

PfR version 2

IOS 15.2(3)T, IOS-XE

3.6

PfR version 3

IOS 15.4(3)M

IOS-XE 3.13

PfR version 3

IOS 15.5(1)T

IOS-XE 3.14

PfR version 3

IOS 15.5(2)T

IOS-XE 3.15

Per Device provisioning

Passive monitoring with

Traditional NetFlow (TNF)

Active monitoring with IP

SLA

Manual provisioning jitter

probes

1000’s lines of

configuration (pfr-map per

site)

Per Device provisioning

Target Discovery (TD)

Automatic provisioning of

jitter probes

Passive monitoring with

Traditional NetFlow (TNF)

Active monitoring with IP

SLA

10’s lines of configuration

• PfR Domain

• One touch provisioning

• Auto Discovery of sites

• NBAR2 support

• Passive Monitoring

(performance monitor)

• Smart Probing

• VRF Awareness

• IPv4/IPv6 (Future)

• <10 lines of

configuration and

centralized

• Zero SLA

• WCCP Support

• Transit Sites

• Multiple Next Hop per

DMVPN

• Multiple POPs

• Show last 5 TCA

Blackout 6 seconds

Brownout 9 seconds

Limited scalability due to

provisioning (~ tens of

sites)

Blackout 6 seconds

Brownout 9 seconds

Scale 500 sites

• Blackout ~ sub second

• Brownout ~ 2 sec

• Scale 2000 sites

Key Takeaways

• IWAN Intelligent Path Control pillar is based upon Performance Routing (PfR)

• Maximizes WAN bandwidth utilization

• Protects applications from performance degradation

• Enables the Internet as a viable WAN transport

• Provides multisite coordination to simplify network wide provisioning.

• Application-based policy driven framework and is tightly integrated with existing AVC components.

• Smart and Scalable multi-sites solution to enforce application SLAs while optimizing network resources utilization.

• PfRv3 is the 3rd generation Multi-Site aware Bandwidth and Path Control/Optimization solution for WAN/Cloud based applications.

• Available now on ISR-G2, ISR-4000, CSR1000v, ASR1k

More Information• Cisco.com IWAN and PfRv3 Page:

• http://www.cisco.com/go/iwan

• http://www.cisco.com/go/pfr

• DocWiki

• http://docwiki.cisco.com/wiki/PfRv3:Home

• CVD IWAN 2.0

• IWAN Technical Design Guide: http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Jan2015/CVD-IWANDesignGuide-JAN15.pdf

• Intelligent WAN Config Files:http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Jan2015/CVD-IWANConfigurationFilesGuide-JAN15.pdf

• IWAN Security for Remote Site DIA and Guest Wirelesshttp://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Mar2015/CVD-IWAN-DIADesignGuide-Mar15.pdf

• IWAN Application Optimization using Cisco WAAS and Akamai Connecthttp://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Mar2015/CVD-IWAN-WAASDesignGuide-Mar15.pdf

http://www.cisco.com/go/iwan

http://www.cisco.com/go/pfr

http://docwiki.cisco.com/wiki/PfRv3:Home

http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Jan2015/CVD-IWANDesignGuide-JAN15.pdf

http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Jan2015/CVD-IWANConfigurationFilesGuide-JAN15.pdf

http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Mar2015/CVD-IWAN-DIADesignGuide-Mar15.pdf

http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Mar2015/CVD-IWAN-WAASDesignGuide-Mar15.pdf

Participate in the “My Favorite Speaker” Contest

• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)

• Send a tweet and include

• Your favorite speaker’s Twitter handle @jbarozet

• Two hashtags: #CLUS #MyFavoriteSpeaker

• You can submit an entry for more than one of your “favorite” speakers

• Don’t forget to follow @CiscoLive and @CiscoPress

• View the official rules at http://bit.ly/CLUSwin

Promote Your Favorite Speaker and You Could Be a Winner

http://bit.ly/CLUSwin

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect.

Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

• Related sessions

Thank you

implementing next generation performance routing...

Documents