openflow on asr9k - cisco...1.1 openflow is an application of onepk from asr9k point of view,...

Anju Dey, ASR9k Tech Lead (Author)

Eddie Ruan, ASR9k Principal Engineer (Author)

Javed Asghar, ASR9k TME (Reviewer) The document covers the details of OpenFlow support on ASR9000 Series routers.

C i s c o S y s t e m s

OpenFlow on ASR9K

OPENFLOW ON ASR9K .......................................................................................................................4

1 INTRODUCTION ...........................................................................................................................4

1.1 OPENFLOW IS AN APPLICATION OF ONEPK ............................................................................................... 4

1.2 HYBRID MODE VS. PURE MODE .............................................................................................................. 4

2 MAIN FUNCTIONALITIES...............................................................................................................4

2.1 OPENFLOW TABLE TYPES ...................................................................................................................... 4

2.2 OPENFLOW MATCHES .......................................................................................................................... 5

2.3 OPENFLOW ACTIONS ............................................................................................................................ 7

2.4 CISCO EXTENSION ACTIONS ................................................................................................................... 9

2.5 COUNTERS .......................................................................................................................................... 9

2.6 OPENFLOW CHANNEL ......................................................................................................................... 11

2.7 FLOW TABLE SCALE ............................................................................................................................ 11

2.8 SUPPORTED INTERFACE TYPES .............................................................................................................. 11

2.8.1 Supported ................................................................................................................................ 11

2.8.2 Not Supported ......................................................................................................................... 12

2.9 HARDWARE SUPPORT ......................................................................................................................... 12

2.9.1 Supported:............................................................................................................................... 12

2.9.2 Not Supported: ........................................................................................................................ 12

3 OPENFLOW CONFIGURATION ON ASR9K .................................................................................... 12

3.1 IMAGES ............................................................................................................................................ 12

3.2 ONEPK CONFIG (MANDATORY) ............................................................................................................ 12

3.3 L2 ONLY SWITCH ................................................................................................................................ 13

3.4 L2 + L3 SWITCH ................................................................................................................................. 13

3.5 L3_V4 SWITCH .................................................................................................................................. 13

3.5.1 L3_V4 switch using VRF .......................................................................................................... 13

3.5.2 L3_V4 switch using Layer 3 interfaces .................................................................................... 14

3.6 L3 DUAL STACK SWITCH ...................................................................................................................... 14

3.6.1 L3_DS switch using VRF .......................................................................................................... 14

3.6.2 L3_DS switch using Layer 3 interfaces .................................................................................... 14

3.7 NETFLOW ......................................................................................................................................... 14

4 SHOW/DEBUG COMMANDS ....................................................................................................... 15

4.1 OPENFLOW SHOW COMMANDS ............................................................................................................ 15

4.2 SHOW POLICY-MAP COMMANDS ........................................................................................................... 15

4.3 PBR PLATFORM SHOW COMMANDS ...................................................................................................... 16

4.4 DEBUG COMMANDS FOR OPENFLOW AGENT .......................................................................................... 16

4.5 DEBUG COMMANDS FOR POLICY MANAGER ........................................................................................... 16

4.6 DEBUG COMMANDS FOR PD PBR ......................................................................................................... 16

5 USE CASES ................................................................................................................................. 18

5.1 L3 NETWORK ..................................................................................................................................... 18

5.2 L2 NETWORK ..................................................................................................................................... 19

5.3 RICE UNIVERSITY USE CASE ................................................................................................................. 19

6 SUMMARY ................................................................................................................................. 22

APPENDIX A FAQ .............................................................................................................................. 23

APPENDIX B OUTPUT OF COMMONLY USED SHOW COMMANDS ...................................................... 25

OPENFLOW AGENT COMMANDS: ................................................................................................................... 25

POLICY COMMANDS ..................................................................................................................................... 29

ASR9K PLATFORM SPECIFIC COMMANDS ........................................................................................................ 30

TCAM COMMANDS ..................................................................................................................................... 31

TABLE 1: TABLE TYPES ........................................................................................................................................................ 5

TABLE 2: OPENFLOW MATCHES ............................................................................................................................................ 6

TABLE 3: OPENFLOW ACTIONS ............................................................................................................................................. 7

TABLE 4: OPENFLOW "SET FIELD" ACTIONS ............................................................................................................................. 9

TABLE 5: CISCO EXTENSION ACTIONS ..................................................................................................................................... 9

TABLE 6: COUNTERS ......................................................................................................................................................... 10

TABLE 7: SCALE ................................................................................................................................................................ 11

OpenFlow on ASR9K

1 Introduction ASR9K starts to support OpenFlow 1.3 from XR-5.1.1 release, which is an EFT release for this feature.

The GA release is from XR-5.1.2 onwards. ASR9K can connect to both 1.0 and 1.3 OpenFlow controllers.

OpenFlow is supported on Typhoon line cards in release XR-5.1.1. It will be supported on future line

cards as well in future releases. It will not be supported on Trident and Thor (SIP-700) line cards.

1.1 OpenFlow is an application of onePK From ASR9K point of view, OpenFlow is an application running natively in IOS-XR, on top of onePK.

OpenFlow agent runs on RSP and is responsible to connect to an external OpenFlow controller and

convert OpenFlow messages to corresponding onePK APIs. Usage of onePK allows ASR9K to use a

common infrastructure to support all SDN related features including CLI based PBR, OpenFlow, onePK

and BGP-FS.

1.2 Hybrid mode vs. Pure Mode OpenFlow-only mode: In this mode only OpenFlow operations are supported. All packets are processed

by the OpenFlow pipeline, and can not be processed otherwise.

OpenFlow-hybrid mode: Both OpenFlow and normal swithcing/routing operations such as L2 Ethernet

switching, L3 routing etc. are supported. Packets can be processed either by the OpenFlow pipeline,

normal pipeline or both.

It is recommended to use ASR9k in hybrid mode.

2 Main Functionalities

2.1 OpenFlow Table Types An OpenFlow flow table consists of a set of flows. Each flow contains a set of matches and actions. A table has a set of capabilities in terms of supported matches & actions. Capabilities of different supported table types are covered in the sections 2.2 and 2.3 in detail. Just like a policy-map, a table can be applied to a set of targets. It is applied only in ingress direction. Hence, OpenFlow matches and actions are applied to incoming traffic only.

An OpenFlow pipeline of an OpenFlow switch on ASR9K will have only one flow table in release XR-5.1.1.

The following table types are supported on ASR9K:

Table Type

L2_ONLY Supports L2 header matches. Supports L2 actions. Can be applied to ingress L2 interfaces.

L2_L3 Supports L2 & L3 (IPv4/IPv6) header matches. Supports L2 actions. Can be applied to ingress L2 interfaces.

L3_V4 Supports L3 IPv4 header matches. Supports L3 IPv4 actions. Can be applied to ingress L3 interfaces.

L3_DS (L3 Dual Stack) Supports L2 and L3 (IPv4/IPv6) header matches. Supports L3 (IPv4/IPv6) actions. Can be applied to ingress L3 interfaces.

Table 1: Table Types

2.2 OpenFlow Matches

Matches are supported on ingress port and various packet headers depending upon the packet type.

Flows can have priorities. Hence, the highest priority flow entry that matches the packet gets selected.

Following table shows the list of matches supported on ASR9K for various table types, in release XR-

5.1.1 with respect to OpenFlow version 1.3.

OpenFlow Matches OpenFlow Switch Types supported on ASR9K

Applied to L2 Bridge domain

Applied to L3 or L3 vrf interface

OXM Flow match field type for OpenFlow basic class

Description L2 only L2_L3 L3_V4 L3_DS

OFPXMT_OFB_IN_PORT Switch input port Yes Yes Yes Yes

OFPXMT_OFB_IN_PHY_PORT Switch physical port No No No No

OFPXMT_OFB_METADATA

Metadata passed between tables.

No No No No

OFPXMT_OFB_ETH_DST

Ethernet destination address

Yes Yes No Yes

OFPXMT_OFB_ETH_SRC Ethernet source address Yes Yes No Yes

OFPXMT_OFB_ETH_TYPE Ethernet frame type Yes Yes No Yes

OFPXMT_OFB_VLAN_VID VLAN id Yes Yes No Yes

OFPXMT_OFB_VLAN_PCP VLAN priority Yes Yes No Yes

OFPXMT_OFB_IP_DSCP IP DSCP (6 bits in ToS field) No Yes Yes Yes

OFPXMT_OFB_IP_ECN IP ECN (2 bits in ToS field) No No No No

OFPXMT_OFB_IP_PROTO IP protocol No Yes Yes Yes

OFPXMT_OFB_IPV4_SRC IPv4 source address No Yes Yes Yes

OFPXMT_OFB_IPV4_DST IPv4 destination address No Yes Yes Yes

OFPXMT_OFB_TCP_SRC TCP source port No Yes Yes Yes

OFPXMT_OFB_TCP_DST TCP destination port No Yes Yes Yes

OFPXMT_OFB_UDP_SRC UDP source port No Yes Yes Yes

OFPXMT_OFB_UDP_DST UDP destination port No Yes Yes Yes

OFPXMT_OFB_SCTP_SRC SCTP source port No No No No

OFPXMT_OFB_SCTP_DST SCTP destination port No No No No

OFPXMT_OFB_ICMPV4_TYPE ICMP type No No No No

OFPXMT_OFB_ICMPV4_CODE ICMP code No No No No

OFPXMT_OFB_ARP_OP ARP opcode No No No No

OFPXMT_OFB_ARP_SPA ARP source IPv4 address No No No No

OFPXMT_OFB_ARP_TPA ARP target IPv4 address No No No No

OFPXMT_OFB_ARP_SHA ARP source hardware addr No No No No

OFPXMT_OFB_ARP_THA ARP target hardware addr No No No No

OFPXMT_OFB_IPV6_SRC IPv6 source address No Yes No Yes

OFPXMT_OFB_IPV6_DST IPv6 destination address No Yes No Yes

OFPXMT_OFB_IPV6_FLABEL IPv6 Flow Label No No No No

OFPXMT_OFB_ICMPV6_TYPE ICMPv6 type No No No No

OFPXMT_OFB_ICMPV6_CODE ICMPv6 code No No No No

OFPXMT_OFB_IPV6_ND_TARGET

Target address for ND No No No No

OFPXMT_OFB_IPV6_ND_SLL Source link-layer for ND No No No No

OFPXMT_OFB_IPV6_ND_TLL Target link-layer for ND No No No No

OFPXMT_OFB_MPLS_LABEL MPLS label No No No Yes

OFPXMT_OFB_MPLS_TC MPLS TC No No No Yes

OFPXMT_OFP_MPLS_BOS MPLS BoS bit No No No Yes

OFPXMT_OFB_PBB_ISID PBB I-SID No No No No

OFPXMT_OFB_TUNNEL_ID Logical Port Metadata No No No No

OFPXMT_OFB_IPV6_EXTHDR IPv6 Extension Header pseudo-field

No No No No

Table 2: OpenFlow Matches (Highlighted items are planned for XR-5.4 and higher releases)

2.3 OpenFlow Actions Only “Apply-actions” instruction (OFPIT_APPLY_ACTIONS) of OpenFlow 1.3 is supported.

Pipeline processing instructions that allow packets to be sent to subsequent tables for further

processing are not supported in this release. Group tables and Meter tables are not supported either.

Packet forwarding and packet modification types of actions are supported. The lists of actions are

always immediately applied to the packet.

Following table shows the list of action types supported on ASR9K for various table types, in release XR-

5.1.1, with respect to OpenFlow version 1.3.

OpenFlow Actions OpenFlow Switch Types supported on ASR9K



OXM Flow action field type for OpenFlow basic class


OFPAT_OUTPUT Output to switch port. Yes Yes No No

OFPAT_COPY_TTL_OUT Copy TTL "outwards" No No No No

OFPAT_COPY_TTL_IN Copy TTL "inwards" No No No No

OFPAT_SET_MPLS_TTL MPLS TTL No No Yes Yes

OFPAT_DEC_MPLS_TTL Decrement MPLS TTL No No No No

OFPAT_PUSH_VLAN Push a new VLAN tag Yes Yes No No

OFPAT_POP_VLAN Pop the outer VLAN tag Yes Yes No No

OFPAT_PUSH_MPLS Push a new MPLS tag No No Yes Yes

OFPAT_POP_MPLS Pop the outer MPLS tag No No Yes Yes

OFPAT_SET_QUEUE Set queue id when outputting to a port

No No No No

OFPAT_GROUP Apply group No No No No

OFPAT_SET_NW_TTL IP TTL No No No No

OFPAT_DEC_NW_TTL Decrement IP TTL No No No No

OFPAT_SET_FIELD Set a header field using OXM TLV format

Yes Yes Yes Yes

OFPAT_PUSH_PBB Push a new PBB service tag (I-TAG)

No No No No

OFPAT_POP_PBB Pop the outer PBB service tag

No No No No

Table 3: OpenFlow Actions

Please note that with respect to OFPAT_OUTPUT action, forwarding to physical ports, switch-defined

logical ports and all reserved ports except ALL, TABLE, IN_PORT, ANY, LOCAL and FLOOD is supported.

The following table shows the “Set field” actions supported by ASR9K.

OpenFlow“Set Field” actions OpenFlow Switch Types supported on ASR9K



OXM Flow field type for OpenFlow basic class


OFPXMT_OFB_ETH_DST

Ethernet destination address

Yes Yes No No

OFPXMT_OFB_ETH_SRC Ethernet source address Yes Yes No No

OFPXMT_OFB_ETH_TYPE Ethernet frame type No No No No

OFPXMT_OFB_VLAN_VID VLAN id (outer) Yes Yes No No

OFPXMT_OFB_VLAN_PCP VLAN priority Yes Yes No No

OFPXMT_OFB_IP_DSCP IP DSCP (6 bits in ToS field) No No Yes Yes

OFPXMT_OFB_IP_ECN IP ECN (2 bits in ToS field) No No No No

OFPXMT_OFB_IP_PROTO IP protocol No No No No

OFPXMT_OFB_IPV4_SRC IPv4 source address No No Yes Yes

OFPXMT_OFB_IPV4_DST IPv4 destination address No No Yes Yes

OFPXMT_OFB_TCP_SRC TCP source port No No Yes Yes

OFPXMT_OFB_TCP_DST TCP destination port No No Yes Yes

OFPXMT_OFB_UDP_SRC UDP source port No No Yes Yes

OFPXMT_OFB_UDP_DST UDP destination port No No Yes Yes

OFPXMT_OFB_SCTP_SRC SCTP source port No No No No

OFPXMT_OFB_SCTP_DST SCTP destination port No No No No

OFPXMT_OFB_ICMPV4_TYPE ICMP type No No No No

OFPXMT_OFB_ICMPV4_CODE ICMP code No No No No

OFPXMT_OFB_ARP_OP ARP opcode No No No No

OFPXMT_OFB_ARP_SPA ARP source IPv4 address No No No No

OFPXMT_OFB_ARP_TPA ARP target IPv4 address No No No No

OFPXMT_OFB_ARP_SHA ARP source hardware addr No No No No

OFPXMT_OFB_ARP_THA ARP target hardware addr No No No No

OFPXMT_OFB_IPV6_SRC IPv6 source address No No No No

OFPXMT_OFB_IPV6_DST IPv6 destination address No No No No

OFPXMT_OFB_IPV6_FLABEL IPv6 Flow Label No No No No

OFPXMT_OFB_ICMPV6_TYPE ICMPv6 type No No No No

OFPXMT_OFB_ICMPV6_CODE ICMPv6 code No No No No

OFPXMT_OFB_IPV6_ND_TARGET

Target address for ND No No No No

OFPXMT_OFB_IPV6_ND_SLL Source link-layer for ND No No No No

OFPXMT_OFB_IPV6_ND_TLL Target link-layer for ND No No No No

OFPXMT_OFB_MPLS_LABEL MPLS label No No No No

OFPXMT_OFB_MPLS_TC MPLS TC No No Yes Yes

OFPXMT_OFP_MPLS_BOS MPLS BoS bit No No Yes Yes

OFPXMT_OFB_PBB_ISID PBB I-SID No No No No

OFPXMT_OFB_TUNNEL_ID Logical Port Metadata No No No No

OFPXMT_OFB_IPV6_EXTHDR IPv6 Extension Header pseudo-field

No No No No

Table 4: OpenFlow "Set field" actions (Highlighted items are planned for XR-5.4 and higher releases)

2.4 Cisco Extension Actions Following table shows the list of actions added by Cisco to support some extra features on ASR9K.

Cisco proprietary actions OpenFlow Switch Types supported on ASR9K




Set Ipv4 Nexthop Set ipv4 nexthop address No No Yes Yes

Set Ipv6 Nexthop Set ipv6 nexthop address No No No Yes

Set Forward Class ID Set forward class ID No No Yes Yes

Set VRF ID Set VRF ID No No Yes Yes

Table 5: Cisco Extension Actions (Highlighted items are planned for XR-5.4 and higher releases)

Apart from adding above-mentioned “set actions”, Netflow extension to enable/disable Netflow feature

on an interface has also been added. Please refer to section 3.7 for the corresponding configuration.

2.5 Counters

Counter Description Show command

Per Flow Table counters Per Flow Table counters such as “active entries”, “packet lookups” and “packet matches”.

RP/0/RSP0/CPU0:OFA2#show openflow switch 1 stats Logical Switch Id: 1 OFPST_PORT reply (xid=0x0):Total ports: 1 Port 1: rx pkts=56587983, bytes=4661065366, drop=9635, errs=0, tx pkts=8619, bytes=401668, drop=0, errs=0,

Logical Switch Id: 1 OFPST_TABLE reply (xid=0x0):Total tables: 1 Table 0: classifier Wildcards = 0x3fffff Max entries = 50000 Active entries = 1 Number of lookups = 0 Number of matches = 0

Per Flow counters Per flow counters such as “received packets”, “received bytes” and duration.

RP/0/RSP0/CPU0:OFA2#show openflow switch 1 flows Logical Switch Id: 1 Total flows: 1 Flow: 1 Match: ipv6,ipv6_dst=192:1::2 Actions: output_nh(ipv6=2001:20::2) Priority: 0 Table: 0 Cookie: 0x1 Duration: 196.353s Number of packets: 0 Number of bytes: 0

Per Port counters Per Port counters such as “received packets”, “received bytes”, “transmitted packets”, “transmitted bytes” and errors.

RP/0/RSP0/CPU0:OFA2#show openflow switch 1 stats Logical Switch Id: 1 OFPST_PORT reply (xid=0x0):Total ports: 1 Port 1: rx pkts=56587983, bytes=4661065366, drop=9635, errs=0, tx pkts=8619, bytes=401668, drop=0, errs=0, Logical Switch Id: 1 OFPST_TABLE reply (xid=0x0):Total tables: 1 Table 0: classifier Wildcards = 0x3fffff Max entries = 50000 Active entries = 1 Number of lookups = 0 Number of matches = 0

Table 6: Counters

2.6 OpenFlow Channel Any physical interface including the Management interface on ASR9k can be used to connect to the

OpenFlow controller. The OpenFlow channel on ASR9K is encrypted using TLS by default. OpenFlow

channel to the controller may be configured to use a specific VRF. Any TCP port can be used to connect

to the controller (wherever the controller is listening), 6653 is the default (and assigned by IANA).

2.7 Flow Table Scale The following scale is supported on ASR9K in XR-5.1.1 release.

16 OpenFlow switches

50,000 flows per system.

L2_ONLY switch and L3_V4 switch use 20-byte key size. L2+L3 switch and L3_DS switch use 80-byte key

size. In XR-5.1.1, we support 50K flows per system. This 50K could be achieved via one NP for L2_ONLY

and L3_V4 table types. Flow table scale target will be raised in future releases.

Please note that with default TCAM configuration, 60% of the TCAM space is allocated to 20-byte keys

and remaining 40% to 80-byte keys. Following CLI can be used to change this ratio of allocation.

admin-config)#hw-module profile tcam tcam-part-30-70 location <location>

Table below shows approximate number of flows supported per NP. The number of flows supported

per line card will depend upon how many NPs it has. Please note that the flow numbers listed here are

based on TCAM capacity as 1-D hardware limit.

TCAM partitioning L2_ONLY/L3_V4 switch L2+L3/L3_DS switch

Default TCAM partitioning 89000 flows 14000 flows

30-70 TCAM partitioning 40000 flows 26000 flows

Table 7: Scale

2.8 Supported Interface Types

2.8.1 Supported

Physical interfaces such as GigabitEthernet, TenGig, HundredGig.

Gig/TenGig/HuGig sub-interfaces

Bundle interfaces

Bundle sub interfaces

BVI (supported only for L3_V4 and L3_DS)

Pseudo-wire Head End sub-interfaces (supported only for L2 and L2_L3 tables)

2.8.2 Not Supported

Satellite interfaces

GRE interfaces

Tunnel-TE interfaces

2.9 Hardware support

2.9.1 Supported:

Typhoon line card

All Chassis types

2.9.2 Not Supported:

Cluster

SIP-700 line card

Trident line card

3 OpenFlow Configuration on ASR9K

3.1 Images

3.2 OnePK config (Mandatory)

RP/0/RSP0/CPU0:ios#config t

RP/0/RSP0/CPU0:ios(config)#onep

RP/0/RSP0/CPU0:ios(config)#datapath transport vpathudp sender-id 1

asr9k-mini-px.vm

asr9k-mpls-px.vm (required for L3VPN, L2VPN)

asr9k-k9sec-px.pie (required for OpenFlow)

3.3 L2 only switch An L2 only OpenFlow switch is attached to a bridge-domain as follows. Pipeline 129 is used for a

L2_ONLY switch.

3.4 L2 + L3 switch An L2_L3 OpenFlow switch is attached to a bridge-domain as follows. Pipeline 130 is used for a L2+L3

switch.

3.5 L3_V4 switch L3_V4 switch can be attached either to a VRF or directly to layer 3 interfaces under global VRF. In case of

VRF, all the interfaces in that VRF become part of the OpenFlow switch. Pipeline 131 is used for a L3_V4

switch.

3.5.1 L3_V4 switch using VRF

openflow

switch 1 pipeline 129

tls trust-point local tp1 remote tp1

bridge-group SDN-2 bridge-domain OF-2

controller ipv4 5.0.1.200 port 6653 security tls

openflow


tls trust-point local tp1 remote tp1

bridge-group SDN-2 bridge-domain OF-2

controller ipv4 5.0.1.200 port 6653 security tls

Openflow


vrf IPv4

controller ipv4 5.0.1.200 port 6653 security none

!

3.5.2 L3_V4 switch using Layer 3 interfaces

3.6 L3 Dual Stack switch L3_DS switch can be attached either to a VRF or directly to layer 3 interfaces under global VRF. In case of

VRF, all the interfaces in that VRF become part of the OpenFlow switch. Pipeline 132 is used for a L3_DS

switch.

3.6.1 L3_DS switch using VRF

3.6.2 L3_DS switch using Layer 3 interfaces

3.7 Netflow A Netflow switch can be used to enable Netflow feature on layer 3 interfaces. A Cisco extension as

mentioned in section 2.4 has been added to enable/disable netflow.

Openflow


interface Bundle-Ether2.1

interface GigabitEthernet0/1/0/6.4


Openflow


vrf IPv4


!

Openflow


interface Bundle-Ether2.1



!

4 Show/debug commands

4.1 Openflow show commands

4.2 Show policy-map commands

Openflow

switch 100 netflow

flow monitor mmap sampler smap

interface Bundle-Ether1



show openflow switch <>

show openflow switch <> controllers | stats

Show openflow switch <> ports

Show openflow switch stats

Show openflow switch flows | brief/summary

Show openflow interface switch <>

show openflow hardware capabilities pipeline <>

show table-cap table-type <>

Show policy-map transient list type pbr

Show policy-map transient type pbr pmap-name <>

Show policy-map transient targets summary

4.3 PBR platform show commands

4.4 Debug commands for OpenFlow Agent

4.5 Debug commands for Policy Manager

4.6 Debug commands for PD PBR

show pbr-pal ipolicy <policy_name | all> location <loc>

show pbr-pal ipolicy <policy_name> iclass <iclass_handle | all> vmr-info

location <loc>

show pbr-pal ipolicy <policy_name> iclass <iclass_handle | all> stats

[clear-on-read] location <loc>

show prm server tcam entries <table> vmr-id <> 100 np0 loc <>

show prm server tcam summary <table> PBR all loc <>

debug openflow switch ovs module ofproto level debug

debug openflow switch ovs module ofproto-plif level debug

debug openflow switch ovs module plif-onep level debug

debug openflow switch ovs module plif-onep-util level debug

debug openflow switch ovs module plif-onep-wt level debug

debug policymgr all

debug policymgr trace

debug policymgr lib all

debug policymgr lib trace

debug pbr-pal all loc

5 Use cases

5.1 L3 network

Problem definition: Three different flows from 3 different sites connected to PE1 are trying to send 350

mbps of traffic each to PE2. The bandwidth of the shortest link Path-2, between PE1 and PE2 is only 1

Gigabit. Hence Path-2 gets congested as soon as the third site begins to send traffic.

OpenFlow solution: OpenFlow controller can be used to install rules on PE1 as follows:

Match on Flow 1 (destined to Video server) and redirect traffic to Path-2

Match on Flow 2 (destined to Web server) and redirect traffic to Path-1

Match on Flow 3 (destined to File transfer server) and redirect traffic to Path-3

Hence utilizing the network bandwidth effectively by redirecting destination specific traffic using

OpenFlow rules.

© 2012 Cisco and/or its affiliates. All rights reserved. 3

PE1

Controller

PE1

PE2

PE1 PE1 PE1 P2

P1

PE1 PE1 PE2 PE1

1

2

3

Video Server

Web

Server

FTP

Server

Traffic Patterns

• Flow 1 - Site 1 Sends 350 mbps of traffic to Video

Server

• Flow 2 - Site 2 Sends 350 mbps of traffic to Web

Server

• Flow 3 - Site 3 Sends 350 mbps of traffic to File

transfer Server

1 Gig

SDN Open flow Components

• Open flow Controller – Runs as an application in any VMs

• OF Agent in CISCO routers or switches creates a TCP connection to

controller and uses Open flow protocol to communicate.

• User or application Installs Open flow rules from controller

• Rule 1 : Site 1 to Video Server – Path 2

• Rule 2 : Site 2 to Web Server – Path 1

• Rule 3 : Site 3 to FTP Server – Path 3

• Traffic Flow now

Path -2

Path -1

Path -3

Path -1

Path -3

Problem : Adding Flow 3 congests 1 Gige link between PE1 and PE2. Which is the

Shortest path.

Solution : Open flow would be used to

install flows and redirect traffic

Data CEnter

Solution : Thus we are able to efficiently utilize bandwidth by redirecting

Destination Specific Traffic using

openFlow rules

5.2 L2 network

Problem definition: Enterprise Data Center needs to perform data backup to multiple other backup sites

based on the Traffic flow. The Main DC is in Vlan 100 and Backup sites are at VLAN 1000,1001,1002.

These Sites are interconnected through L2VPN. In this topology if customer needs to selectively

determine Backup traffic destination site he/she needs to send them in separate VLANS.

OpenFlow solution: With Open flow we can match on any Layer 2 header fields (in this example we

have taken priority bits) and steer the traffic to go on any L2 interconnect and also rewrite the VLANs

appropriately.

5.3 Rice University Use Case Rice University's campus network is a traditional MPLS VPN network. There are multiple affinity

networks running on top of this campus network. These affinity networks are separated via different

VRFs from MPLS/VPN point of view. In some cases, they need to grant some temporary access from one

affinity network to another affinity network. In the example below, a staff which belongs to staff

network wants to access some stuff in student network.

© 2012 Cisco and/or its affiliates. All rights reserved. 4

Controller

PE1

PE2

PE3

PE4

Data Center Backup

Server 1

L2 Interconnect -1

L2 Interconnect -2

L2 Interconnect -3

Enterprise Data

Center

Data Center Backup

Server 2

Data Center Backup

Server 3

Vlan 100

Vlan 1000

Vlan 1001

Vlan 1002

SDN Open flow Components

• Open flow Controller – Runs as an application in any VMs

• OF Agent in CISCO routers or switches creates a TCP connection to controller and uses

Open flow protocol to communicate.

• User or application Installs Open flow rules from controller

• Rule 1 : Traffic from enterprise DC coming with VLAN 100 and TOS value 1 should

be steered to L2 interconnect 1 and VLAN rewritten as 1000

• Rule 2 : Traffic from enterprise DC coming with VLAN 100 and TOS value 2 should

be steered to L2 interconnect 2 and VLAN rewritten as 1002

• Rule 3 : Wild card match send it to Open flow controller ( PACKET_IN/ PUNT ).

• Controller will react to PACKET_IN message with vlan 100 and TOS 3 by

pushing down a flow to steer traffic to L2 interconnect 3 and rewrite vlan to

1002.

• The idle time out is set to 300 sec

• Traffic would be Steered to L2 interconnect 3

L2 Interconnect –X – L2VPN PW

This type of requests' handling is normally done via PBR or static route to provide VRF leaking from one

to another. Rice University has about 11 such kind of affinity networks. They maintain a 11x11 policy

matrix to decide if VRF leaking is allowed or not.

It is not a trivial job to maintain this 11x11 matrix logic consistently across multiple PEs. Rice University

looks for a simple solution via SDN framework to solve this problem.

Rice University likes Open Flow's flexibility to steer specific traffic. But they don't like pure Open Flow

approach. In pure Open Flow approach, they need to take care of basic ping handling as well. It is too

much tax for them to get this kind of flexibility. Therefore, they feel Open Flow hybrid mode is a better

choice for them.

6 Summary

ASR9k implements OpenFlow 1.0 and OpenFlow 1.3 in XR-5.1.1 release. Section 2 describes the table

types, matches, actions and counters supported in this release. Interface and hardware support is also

documented in section 2. ASR9K plans to continue adding support for other matches and actions such

as match on ICMP v4/v6 type/code, MPLS label etc. in upcoming releases.

Section 5 describes some of the use cases in detail. Various combinations of matches and actions can

lead to many other use cases.

Appendix A FAQ

Q. What are Cisco extensions and what can they be used for?

A. The following Cisco extensions to the list of actions are supported.

1. Set ipv4 nexthop

2. Set ipv6 nexthop

3. Set FCID

4. Enable/disable netflow

Set ipv4/ipv6 nexthop actions are used to redirect an ipv4/ipv6 packet to the specified nexthop address,

instead of using the destination address in the packet. This provides ABF (ACL Based Forwarding) kind of

functionality using OpenFlow. However, VRF support and nexthop tracking as supported by CLI based

ABF feature is not supported in this release.

“Set FCID” action can be used to support PBTS (Policy Based Tunnel Selection) functionality using

OpenFlow.

Enable/disable netflow is used to enable or disable netflow on an interface using OpenFlow.

Q. What are the supported controller types?

A. All OpenFlow controllers should work. However, IXIA, ODL, and POX controllers were used for testing

on ASR9K.

Q. Are the flow priorities supported?

A. Flow priorities are supported. Lowest priority is zero and highest is 32768.

Q. Is there a limit on the number of actions per flow?

A. There is as such no limit on the number of actions supported per flow. However, if the data

associated with multiple actions happens to be large such as source MAC address, destination address,

set ipv6 nexthop etc. we may hit the hardware limit. In such cases, an error will be returned and the

flow will not be programmed in the hardware. Only a single output action is supported in XR-5.1.1.

24

Q. Do we need to use only Management Ethernet port to connect to the OpenFlow controller?

A. Any supported physical interface including Management Ethernet interface can be used to connect to

the controller. However for HA purposes, only physical port should be used to connect to the controller.

Q. Does ASR9K support OpenFlow channel auxiliary connections?

A. Auxiliary connections are not supported on ASR9K.

Q. Can multiple controllers be connected to the same ASR9K?

A. Yes, multiple controllers can be supported on ASR9K. Maximum of 8 controllers per switch can be

supported.

Q. What is the periodicity of stats collection by default? Can it be changed?

A. Flow statistics are collected by default at the rate of 50 flows per second. However, this periodicity

can be changed using the following command.

Openflow

Switch <switch_id> pipeline <pipeline number>

statistics collection-period <period>

Q. Does ASR9K support idle-timeouts and hard timeouts?

A. Yes, idle timeouts and hard timeouts are supported on ASR9K. The OpenFlow controller can set idle

timeouts as well as hard timeouts. Idle-timeouts are implemented based on stats collection; hence the

granularity of idle-timeouts is related to the stats-collection interval. Command “sh openflow switch

<switch_id> flows” can be used to display these timeouts.

Appendix B Output of commonly used show commands

OpenFlow Agent commands:

RP/0/RSP0/CPU0:OFA2#show run openflow switch

openflow


interface GigabitEthernet0/1/0/10

dataplane-default secure

statistics collection-period 0


!

!

RP/0/RSP0/CPU0:OFA2#show openflow switch 1 controller

Logical Switch Id: 1

Total Controllers: Not available

Controller: 1

Address : 56.1.10.2:6633

Protocol : tcp

VRF : default

Local Trustpoint: : Not available

Remote Trustpoint: : Not available

Connected : Yes

Role : Master

last_error : Connection timed out

state : ACTIVE

sec_since_connect : 57

RP/0/RSP0/CPU0:OFA2#show openflow switch 1 ports


Port Interface Name Config-State Link-State Features

1 Gi0/1/0/10 PORT_UP LINK_UP 0

26

RP/0/RSP0/CPU0:OFA2#show openflow switch 1 stats


OFPST_PORT reply (xid=0x0):Total ports: 1

Port 1: rx pkts=56587983, bytes=4661065366, drop=9635,

errs=0,

tx pkts=8619, bytes=401668, drop=0, errs=0,


OFPST_TABLE reply (xid=0x0):Total tables: 1

Table 0: classifier

Wildcards = 0x3fffff

Max entries = 50000

Active entries = 1

Number of lookups = 0

Number of matches = 0

RP/0/RSP0/CPU0:OFA2#show openflow switch 1 flows


Total flows: 2

Flow: 1

Match: ipv6,ipv6_dst=192:1::2

Actions: output_nh(ipv6=2001:20::2)

Priority: 0

Table: 0

Cookie: 0x1

Duration: 196.353s

Number of packets: 0

Number of bytes: 0

Flow: 2

Match:

Actions: NORMAL

Priority: 0

Table: 0

Cookie: 0x0

Duration: 223.331s

Number of packets: 0

Number of bytes: 0

27

RP/0/RSP0/CPU0:OFA2#show openflow switch flows summary

Total Forwarding flow count: 2

Total Netflow flow count : 0

Total flow count : 2


Switch flow count : 2

RP/0/RSP0/CPU0:OFA2#show openflow switch flows brief


Total flows: 2

Flow: 1 Match: ipv6,ipv6_dst=192:1::2 Actions:

output_nh(ipv6=2001:20::2)

Priority: 0, Table: 0, Cookie: 0x1, Duration: 333.131s,

Packets: 0, Bytes: 0

Flow: 2 Match: Actions: NORMAL

Priority: 0, Table: 0, Cookie: 0x0, Duration: 360.109s,

Packets: 0, Bytes: 0

Total flow count: 2

28

RP/0/RSP0/CPU0:OFA2#show openflow switch 1

Logical Switch Context Id: 1

Switch type: Forwarding

Pipeline ID: 132

Data plane: secure

Fallback: normal

Config state: no-shutdown

Working state: enabled

Rate limit: 0

Burst limit: 0

Max backoff (sec): 8

Probe interval (sec): 5

TLS local trustpoint: NONE

TLS remote trustpoint: NONE

Stats coll period (sec): disabled

Logging flow changes: Not available

OFA Description:

Manufacturer: Cisco Systems, Inc.

Hardware: ASR-9006-AC V01

Software: 5.1.2.11I of_agent 1.0

Serial Num: FOX1333GVJS

DP Description: OFA2:sw1

OF Features:

DPID: 0001002651cdcc8e

Number of tables: 1

Number of buffers: 256

Capabilities: FLOW_STATS TABLE_STATS

PORT_STATS

Actions: SET_NW_SRC SET_NW_DST

SET_TP_SRC SET_TP_DST SET_FCID SET_NH

Controllers:

56.1.10.2:6633, Protocol: TCP, Openflow 1.3, last

alive ping: 2014-02-21 07:32:00

Bridge domain: NONE

VRF: default-00001

Interfaces:

GigabitEthernet0/1/0/10

Policy commands

RP/0/RSP0/CPU0:OFA2#show policy-map transient list type pbr

1) PolicyMap: onep-pmap-7416-4 Type: pbr (transient)

Total Flows : 2

RP/0/RSP0/CPU0:OFA2#show policy-map transient type pbr pmap-

name onep-pmap-7416-4

policy-map type pbr onep-pmap-7416-4

handle:0x34000006

table description: L3 IPv4 and IPv6

class handle:0x74000007 sequence 65535

match ethertype ipv6

match destination-address ipv6 192:1::2/128

ipv6 next-hop 2001:20::2

!

class handle:0xf4000006 sequence 4294967295 (class-default)

transmit

!

end-policy-map

!

RP/0/RSP0/CPU0:OFA2#show policy-map transient targets type

pbr pmap-name onep-pmap-7416-4

1) Policymap: onep-pmap-7416-4 Type: pbr

Targets (applied as main policy):

GigabitEthernet0/1/0/10 input

Total targets: 1

ASR9K Platform specific commands

RP/0/RSP0/CPU0:OFA2#show pbr-pal ipolicy onep-pmap-7416-4

detail location 0/1/CPU0

policy name : onep-pmap-7416-4

number of iclasses : 2

number of VMRs : 2

ucode format : 23

vmr id for NP0 : 71

interface count : 1

interface list : Gi0/1/0/10

RP/0/RSP0/CPU0:OFA2#show pbr-pal ipolicy onep-pmap-7416-4

iclass all vmr-info location 0/1/cpu0

iclass handle : 0x74000007

ifh : x

protocol : x

source ip addr : x

dest ip addr : 192:1::2/128

source port : x

dest port : x

DSCP : x

ethertype : 0x86dd

vlan id : x

vlan cos : x

source mac : x

dest mac : x

result :

1100000000000001a4000000020000000000000000200100200000000000000

0

iclass handle : 0xf4000006

ifh : x

protocol : x

source ip addr : x

dest ip addr : x

source port : x

dest port : x

DSCP : x

ethertype : x

vlan id : x

vlan cos : x

source mac : x

dest mac : x

result :

110000000000000000000000000000000000000000000000000000000000000

0

31

TCAM commands

RP/0/RSP0/CPU0:OFA2#show pbr-pal ipolicy onep-pmap-7416-4 iclass

all stats location 0/1/CPU0

iclass packets/bytes drop

packets/drop bytes

74000007 22376/1924336 0/0

f4000006 6/524 0/0

RP/0/RSP0/CPU0:OFA2#show prm server tcam summary 576-LT pBR all

location 0/1/CPU0

Node: 0/1/CPU0:

----------------------------------------------------------------

TCAM summary for NP0:

TCAM Logical Table: TCAM_LT_ODS8 (3), free entries: 14756,

resvd 127

ACL Common Region: 448 entries allocated. 448 entries free

Application ID: NP_APP_ID_PBR (5)

Total: 1 vmr_ids, 2 active entries, 2 allocated entries.

TCAM summary for NP1:

TCAM Logical Table: TCAM_LT_ODS8 (3), free entries: 14757,

resvd 128

ACL Common Region: 448 entries allocated. 448 entries free

Application ID: NP_APP_ID_PBR (5)

Total: 0 vmr_ids, 0 active entries, 0 allocated entries.

32

RP/0/RSP0/CPU0:OFA2#show prm server tcam entries 576-LT vmr-id

71 10 all location 0/1/CPU0

Node: 0/1/CPU0:

----------------------------------------------------------------

ODS NP: 0, LT: 3, AppId: 5, VmrId 71, Offset 0, Entries

2, Shadow 0

60e00 V: a171 0000 0000 0000 0000 0000 0000 0000 0000 0000 23

0000 0000 0000 0000 0000 0000 0000 0000 0200 0000

0000 0000 0000 0000 0100 9201 0000 0000 0000 0000

0000 0000 86dd 0000 0000 0000 0000 0000 0000 0000

M: 0000 ffff ffff ffff ffff ffff ffff ffff ffff ffff

ffff ffff ffff ffff ffff ffff ffff ffff 0000 0000

0000 0000 0000 0000 0000 0000 ffff ffff ffff ffff

ffff ffff 0000 ffff ffff ffff ffff ffff ffff ffff

R: 110000d0 18610001 a4000000 02000000 00000000 00200100

20000000 00000000

7ee08 V: a170 0000 0000 0000 0000 0000 0000 0000 0000 0000 23

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

M: 0003 ffff ffff ffff ffff ffff ffff ffff ffff ffff

ffff ffff ffff ffff ffff ffff ffff ffff ffff ffff



R: 110000c0 18610000 00000000 00000000 00000000 00000000

00000000 00000000

openflow on asr9k - cisco...1.1 openflow is an application of onepk from asr9k point of view,...

Documents