openflow on asr9k - cisco · pdf fileopenflow on asr9k ... openflow is supported on typhoon...
TRANSCRIPT
Anju Dey, ASR9k Tech Lead (Author)
Eddie Ruan, ASR9k Principal Engineer (Author)
Javed Asghar, ASR9k TME (Reviewer) The document covers the details of OpenFlow support on ASR9000 Series routers.
C i s c o S y s t e m s
OpenFlow on ASR9K
OPENFLOW ON ASR9K .......................................................................................................................4
1 INTRODUCTION ...........................................................................................................................4
1.1 OPENFLOW IS AN APPLICATION OF ONEPK ............................................................................................... 4
1.2 HYBRID MODE VS. PURE MODE .............................................................................................................. 4
2 MAIN FUNCTIONALITIES...............................................................................................................4
2.1 OPENFLOW TABLE TYPES ...................................................................................................................... 4
2.2 OPENFLOW MATCHES .......................................................................................................................... 5
2.3 OPENFLOW ACTIONS ............................................................................................................................ 7
2.4 CISCO EXTENSION ACTIONS ................................................................................................................... 9
2.5 COUNTERS .......................................................................................................................................... 9
2.6 OPENFLOW CHANNEL ......................................................................................................................... 11
2.7 FLOW TABLE SCALE ............................................................................................................................ 11
2.8 SUPPORTED INTERFACE TYPES .............................................................................................................. 11
2.8.1 Supported ................................................................................................................................ 11
2.8.2 Not Supported ......................................................................................................................... 12
2.9 HARDWARE SUPPORT ......................................................................................................................... 12
2.9.1 Supported:............................................................................................................................... 12
2.9.2 Not Supported: ........................................................................................................................ 12
3 OPENFLOW CONFIGURATION ON ASR9K .................................................................................... 12
3.1 IMAGES ............................................................................................................................................ 12
3.2 ONEPK CONFIG (MANDATORY) ............................................................................................................ 12
3.3 L2 ONLY SWITCH ................................................................................................................................ 13
3.4 L2 + L3 SWITCH ................................................................................................................................. 13
3.5 L3_V4 SWITCH .................................................................................................................................. 13
3.5.1 L3_V4 switch using VRF .......................................................................................................... 13
3.5.2 L3_V4 switch using Layer 3 interfaces .................................................................................... 14
3.6 L3 DUAL STACK SWITCH ...................................................................................................................... 14
3.6.1 L3_DS switch using VRF .......................................................................................................... 14
3.6.2 L3_DS switch using Layer 3 interfaces .................................................................................... 14
3.7 NETFLOW ......................................................................................................................................... 14
4 SHOW/DEBUG COMMANDS ....................................................................................................... 15
4.1 OPENFLOW SHOW COMMANDS ............................................................................................................ 15
4.2 SHOW POLICY-MAP COMMANDS ........................................................................................................... 15
4.3 PBR PLATFORM SHOW COMMANDS ...................................................................................................... 16
4.4 DEBUG COMMANDS FOR OPENFLOW AGENT .......................................................................................... 16
4.5 DEBUG COMMANDS FOR POLICY MANAGER ........................................................................................... 16
4.6 DEBUG COMMANDS FOR PD PBR ......................................................................................................... 16
5 USE CASES ................................................................................................................................. 18
5.1 L3 NETWORK ..................................................................................................................................... 18
5.2 L2 NETWORK ..................................................................................................................................... 19
5.3 RICE UNIVERSITY USE CASE ................................................................................................................. 19
6 SUMMARY ................................................................................................................................. 22
APPENDIX A FAQ .............................................................................................................................. 23
APPENDIX B OUTPUT OF COMMONLY USED SHOW COMMANDS ...................................................... 25
OPENFLOW AGENT COMMANDS: ................................................................................................................... 25
POLICY COMMANDS ..................................................................................................................................... 29
ASR9K PLATFORM SPECIFIC COMMANDS ........................................................................................................ 30
TCAM COMMANDS ..................................................................................................................................... 31
TABLE 1: TABLE TYPES ........................................................................................................................................................ 5
TABLE 2: OPENFLOW MATCHES ............................................................................................................................................ 6
TABLE 3: OPENFLOW ACTIONS ............................................................................................................................................. 7
TABLE 4: OPENFLOW "SET FIELD" ACTIONS ............................................................................................................................. 9
TABLE 5: CISCO EXTENSION ACTIONS ..................................................................................................................................... 9
TABLE 6: COUNTERS ......................................................................................................................................................... 10
TABLE 7: SCALE ................................................................................................................................................................ 11
OpenFlow on ASR9K
1 Introduction ASR9K starts to support OpenFlow 1.3 from XR-5.1.1 release, which is an EFT release for this feature.
The GA release is from XR-5.1.2 onwards. ASR9K can connect to both 1.0 and 1.3 OpenFlow controllers.
OpenFlow is supported on Typhoon line cards in release XR-5.1.1. It will be supported on future line
cards as well in future releases. It will not be supported on Trident and Thor (SIP-700) line cards.
1.1 OpenFlow is an application of onePK From ASR9K point of view, OpenFlow is an application running natively in IOS-XR, on top of onePK.
OpenFlow agent runs on RSP and is responsible to connect to an external OpenFlow controller and
convert OpenFlow messages to corresponding onePK APIs. Usage of onePK allows ASR9K to use a
common infrastructure to support all SDN related features including CLI based PBR, OpenFlow, onePK
and BGP-FS.
1.2 Hybrid mode vs. Pure Mode OpenFlow-only mode: In this mode only OpenFlow operations are supported. All packets are processed
by the OpenFlow pipeline, and can not be processed otherwise.
OpenFlow-hybrid mode: Both OpenFlow and normal swithcing/routing operations such as L2 Ethernet
switching, L3 routing etc. are supported. Packets can be processed either by the OpenFlow pipeline,
normal pipeline or both.
It is recommended to use ASR9k in hybrid mode.
2 Main Functionalities
2.1 OpenFlow Table Types An OpenFlow flow table consists of a set of flows. Each flow contains a set of matches and actions. A table has a set of capabilities in terms of supported matches & actions. Capabilities of different supported table types are covered in the sections 2.2 and 2.3 in detail. Just like a policy-map, a table can be applied to a set of targets. It is applied only in ingress direction. Hence, OpenFlow matches and actions are applied to incoming traffic only.
An OpenFlow pipeline of an OpenFlow switch on ASR9K will have only one flow table in release XR-5.1.1.
The following table types are supported on ASR9K:
Table Type
L2_ONLY Supports L2 header matches. Supports L2 actions. Can be applied to ingress L2 interfaces.
L2_L3 Supports L2 & L3 (IPv4/IPv6) header matches. Supports L2 actions. Can be applied to ingress L2 interfaces.
L3_V4 Supports L3 IPv4 header matches. Supports L3 IPv4 actions. Can be applied to ingress L3 interfaces.
L3_DS (L3 Dual Stack) Supports L2 and L3 (IPv4/IPv6) header matches. Supports L3 (IPv4/IPv6) actions. Can be applied to ingress L3 interfaces.
Table 1: Table Types
2.2 OpenFlow Matches
Matches are supported on ingress port and various packet headers depending upon the packet type.
Flows can have priorities. Hence, the highest priority flow entry that matches the packet gets selected.
Following table shows the list of matches supported on ASR9K for various table types, in release XR-
5.1.1 with respect to OpenFlow version 1.3.
OpenFlow Matches OpenFlow Switch Types supported on ASR9K
Applied to L2 Bridge domain
Applied to L3 or L3 vrf interface
OXM Flow match field type for OpenFlow basic class
Description L2 only L2_L3 L3_V4 L3_DS
OFPXMT_OFB_IN_PORT Switch input port Yes Yes Yes Yes
OFPXMT_OFB_IN_PHY_PORT Switch physical port No No No No
OFPXMT_OFB_METADATA
Metadata passed between tables.
No No No No
OFPXMT_OFB_ETH_DST
Ethernet destination address
Yes Yes No Yes
OFPXMT_OFB_ETH_SRC Ethernet source address Yes Yes No Yes
OFPXMT_OFB_ETH_TYPE Ethernet frame type Yes Yes No Yes
OFPXMT_OFB_VLAN_VID VLAN id Yes Yes No Yes
OFPXMT_OFB_VLAN_PCP VLAN priority Yes Yes No Yes
OFPXMT_OFB_IP_DSCP IP DSCP (6 bits in ToS field) No Yes Yes Yes
OFPXMT_OFB_IP_ECN IP ECN (2 bits in ToS field) No No No No
OFPXMT_OFB_IP_PROTO IP protocol No Yes Yes Yes
OFPXMT_OFB_IPV4_SRC IPv4 source address No Yes Yes Yes
OFPXMT_OFB_IPV4_DST IPv4 destination address No Yes Yes Yes
OFPXMT_OFB_TCP_SRC TCP source port No Yes Yes Yes
OFPXMT_OFB_TCP_DST TCP destination port No Yes Yes Yes
OFPXMT_OFB_UDP_SRC UDP source port No Yes Yes Yes
OFPXMT_OFB_UDP_DST UDP destination port No Yes Yes Yes
OFPXMT_OFB_SCTP_SRC SCTP source port No No No No
OFPXMT_OFB_SCTP_DST SCTP destination port No No No No
OFPXMT_OFB_ICMPV4_TYPE ICMP type No No No No
OFPXMT_OFB_ICMPV4_CODE ICMP code No No No No
OFPXMT_OFB_ARP_OP ARP opcode No No No No
OFPXMT_OFB_ARP_SPA ARP source IPv4 address No No No No
OFPXMT_OFB_ARP_TPA ARP target IPv4 address No No No No
OFPXMT_OFB_ARP_SHA ARP source hardware addr No No No No
OFPXMT_OFB_ARP_THA ARP target hardware addr No No No No
OFPXMT_OFB_IPV6_SRC IPv6 source address No Yes No Yes
OFPXMT_OFB_IPV6_DST IPv6 destination address No Yes No Yes
OFPXMT_OFB_IPV6_FLABEL IPv6 Flow Label No No No No
OFPXMT_OFB_ICMPV6_TYPE ICMPv6 type No No No No
OFPXMT_OFB_ICMPV6_CODE ICMPv6 code No No No No
OFPXMT_OFB_IPV6_ND_TARGET
Target address for ND No No No No
OFPXMT_OFB_IPV6_ND_SLL Source link-layer for ND No No No No
OFPXMT_OFB_IPV6_ND_TLL Target link-layer for ND No No No No
OFPXMT_OFB_MPLS_LABEL MPLS label No No No Yes
OFPXMT_OFB_MPLS_TC MPLS TC No No No Yes
OFPXMT_OFP_MPLS_BOS MPLS BoS bit No No No Yes
OFPXMT_OFB_PBB_ISID PBB I-SID No No No No
OFPXMT_OFB_TUNNEL_ID Logical Port Metadata No No No No
OFPXMT_OFB_IPV6_EXTHDR IPv6 Extension Header pseudo-field
No No No No
Table 2: OpenFlow Matches (Highlighted items are planned for XR-5.4 and higher releases)
2.3 OpenFlow Actions Only “Apply-actions” instruction (OFPIT_APPLY_ACTIONS) of OpenFlow 1.3 is supported.
Pipeline processing instructions that allow packets to be sent to subsequent tables for further
processing are not supported in this release. Group tables and Meter tables are not supported either.
Packet forwarding and packet modification types of actions are supported. The lists of actions are
always immediately applied to the packet.
Following table shows the list of action types supported on ASR9K for various table types, in release XR-
5.1.1, with respect to OpenFlow version 1.3.
OpenFlow Actions OpenFlow Switch Types supported on ASR9K
Applied to L2 Bridge domain
Applied to L3 or L3 vrf interface
OXM Flow action field type for OpenFlow basic class
Description L2 only L2_L3 L3_V4 L3_DS
OFPAT_OUTPUT Output to switch port. Yes Yes No No
OFPAT_COPY_TTL_OUT Copy TTL "outwards" No No No No
OFPAT_COPY_TTL_IN Copy TTL "inwards" No No No No
OFPAT_SET_MPLS_TTL MPLS TTL No No Yes Yes
OFPAT_DEC_MPLS_TTL Decrement MPLS TTL No No No No
OFPAT_PUSH_VLAN Push a new VLAN tag Yes Yes No No
OFPAT_POP_VLAN Pop the outer VLAN tag Yes Yes No No
OFPAT_PUSH_MPLS Push a new MPLS tag No No Yes Yes
OFPAT_POP_MPLS Pop the outer MPLS tag No No Yes Yes
OFPAT_SET_QUEUE Set queue id when outputting to a port
No No No No
OFPAT_GROUP Apply group No No No No
OFPAT_SET_NW_TTL IP TTL No No No No
OFPAT_DEC_NW_TTL Decrement IP TTL No No No No
OFPAT_SET_FIELD Set a header field using OXM TLV format
Yes Yes Yes Yes
OFPAT_PUSH_PBB Push a new PBB service tag (I-TAG)
No No No No
OFPAT_POP_PBB Pop the outer PBB service tag
No No No No
Table 3: OpenFlow Actions
Please note that with respect to OFPAT_OUTPUT action, forwarding to physical ports, switch-defined
logical ports and all reserved ports except ALL, TABLE, IN_PORT, ANY, LOCAL and FLOOD is supported.
The following table shows the “Set field” actions supported by ASR9K.
OpenFlow“Set Field” actions OpenFlow Switch Types supported on ASR9K
Applied to L2 Bridge domain
Applied to L3 or L3 vrf interface
OXM Flow field type for OpenFlow basic class
Description L2 only L2_L3 L3_V4 L3_DS
OFPXMT_OFB_ETH_DST
Ethernet destination address
Yes Yes No No
OFPXMT_OFB_ETH_SRC Ethernet source address Yes Yes No No
OFPXMT_OFB_ETH_TYPE Ethernet frame type No No No No
OFPXMT_OFB_VLAN_VID VLAN id (outer) Yes Yes No No
OFPXMT_OFB_VLAN_PCP VLAN priority Yes Yes No No
OFPXMT_OFB_IP_DSCP IP DSCP (6 bits in ToS field) No No Yes Yes
OFPXMT_OFB_IP_ECN IP ECN (2 bits in ToS field) No No No No
OFPXMT_OFB_IP_PROTO IP protocol No No No No
OFPXMT_OFB_IPV4_SRC IPv4 source address No No Yes Yes
OFPXMT_OFB_IPV4_DST IPv4 destination address No No Yes Yes
OFPXMT_OFB_TCP_SRC TCP source port No No Yes Yes
OFPXMT_OFB_TCP_DST TCP destination port No No Yes Yes
OFPXMT_OFB_UDP_SRC UDP source port No No Yes Yes
OFPXMT_OFB_UDP_DST UDP destination port No No Yes Yes
OFPXMT_OFB_SCTP_SRC SCTP source port No No No No
OFPXMT_OFB_SCTP_DST SCTP destination port No No No No
OFPXMT_OFB_ICMPV4_TYPE ICMP type No No No No
OFPXMT_OFB_ICMPV4_CODE ICMP code No No No No
OFPXMT_OFB_ARP_OP ARP opcode No No No No
OFPXMT_OFB_ARP_SPA ARP source IPv4 address No No No No
OFPXMT_OFB_ARP_TPA ARP target IPv4 address No No No No
OFPXMT_OFB_ARP_SHA ARP source hardware addr No No No No
OFPXMT_OFB_ARP_THA ARP target hardware addr No No No No
OFPXMT_OFB_IPV6_SRC IPv6 source address No No No No
OFPXMT_OFB_IPV6_DST IPv6 destination address No No No No
OFPXMT_OFB_IPV6_FLABEL IPv6 Flow Label No No No No
OFPXMT_OFB_ICMPV6_TYPE ICMPv6 type No No No No
OFPXMT_OFB_ICMPV6_CODE ICMPv6 code No No No No
OFPXMT_OFB_IPV6_ND_TARGET
Target address for ND No No No No
OFPXMT_OFB_IPV6_ND_SLL Source link-layer for ND No No No No
OFPXMT_OFB_IPV6_ND_TLL Target link-layer for ND No No No No
OFPXMT_OFB_MPLS_LABEL MPLS label No No No No
OFPXMT_OFB_MPLS_TC MPLS TC No No Yes Yes
OFPXMT_OFP_MPLS_BOS MPLS BoS bit No No Yes Yes
OFPXMT_OFB_PBB_ISID PBB I-SID No No No No
OFPXMT_OFB_TUNNEL_ID Logical Port Metadata No No No No
OFPXMT_OFB_IPV6_EXTHDR IPv6 Extension Header pseudo-field
No No No No
Table 4: OpenFlow "Set field" actions (Highlighted items are planned for XR-5.4 and higher releases)
2.4 Cisco Extension Actions Following table shows the list of actions added by Cisco to support some extra features on ASR9K.
Cisco proprietary actions OpenFlow Switch Types supported on ASR9K
Applied to L2 Bridge domain
Applied to L3 or L3 vrf interface
Description L2 only L2_L3 L3_V4 L3_DS
Set Ipv4 Nexthop Set ipv4 nexthop address No No Yes Yes
Set Ipv6 Nexthop Set ipv6 nexthop address No No No Yes
Set Forward Class ID Set forward class ID No No Yes Yes
Set VRF ID Set VRF ID No No Yes Yes
Table 5: Cisco Extension Actions (Highlighted items are planned for XR-5.4 and higher releases)
Apart from adding above-mentioned “set actions”, Netflow extension to enable/disable Netflow feature
on an interface has also been added. Please refer to section 3.7 for the corresponding configuration.
2.5 Counters
Counter Description Show command
Per Flow Table counters Per Flow Table counters such as “active entries”, “packet lookups” and “packet matches”.
RP/0/RSP0/CPU0:OFA2#show openflow switch 1 stats Logical Switch Id: 1 OFPST_PORT reply (xid=0x0):Total ports: 1 Port 1: rx pkts=56587983, bytes=4661065366, drop=9635, errs=0, tx pkts=8619, bytes=401668, drop=0, errs=0,
Logical Switch Id: 1 OFPST_TABLE reply (xid=0x0):Total tables: 1 Table 0: classifier Wildcards = 0x3fffff Max entries = 50000 Active entries = 1 Number of lookups = 0 Number of matches = 0
Per Flow counters Per flow counters such as “received packets”, “received bytes” and duration.
RP/0/RSP0/CPU0:OFA2#show openflow switch 1 flows Logical Switch Id: 1 Total flows: 1 Flow: 1 Match: ipv6,ipv6_dst=192:1::2 Actions: output_nh(ipv6=2001:20::2) Priority: 0 Table: 0 Cookie: 0x1 Duration: 196.353s Number of packets: 0 Number of bytes: 0
Per Port counters Per Port counters such as “received packets”, “received bytes”, “transmitted packets”, “transmitted bytes” and errors.
RP/0/RSP0/CPU0:OFA2#show openflow switch 1 stats Logical Switch Id: 1 OFPST_PORT reply (xid=0x0):Total ports: 1 Port 1: rx pkts=56587983, bytes=4661065366, drop=9635, errs=0, tx pkts=8619, bytes=401668, drop=0, errs=0, Logical Switch Id: 1 OFPST_TABLE reply (xid=0x0):Total tables: 1 Table 0: classifier Wildcards = 0x3fffff Max entries = 50000 Active entries = 1 Number of lookups = 0 Number of matches = 0
Table 6: Counters
2.6 OpenFlow Channel Any physical interface including the Management interface on ASR9k can be used to connect to the
OpenFlow controller. The OpenFlow channel on ASR9K is encrypted using TLS by default. OpenFlow
channel to the controller may be configured to use a specific VRF. Any TCP port can be used to connect
to the controller (wherever the controller is listening), 6653 is the default (and assigned by IANA).
2.7 Flow Table Scale The following scale is supported on ASR9K in XR-5.1.1 release.
16 OpenFlow switches
50,000 flows per system.
L2_ONLY switch and L3_V4 switch use 20-byte key size. L2+L3 switch and L3_DS switch use 80-byte key
size. In XR-5.1.1, we support 50K flows per system. This 50K could be achieved via one NP for L2_ONLY
and L3_V4 table types. Flow table scale target will be raised in future releases.
Please note that with default TCAM configuration, 60% of the TCAM space is allocated to 20-byte keys
and remaining 40% to 80-byte keys. Following CLI can be used to change this ratio of allocation.
admin-config)#hw-module profile tcam tcam-part-30-70 location <location>
Table below shows approximate number of flows supported per NP. The number of flows supported
per line card will depend upon how many NPs it has. Please note that the flow numbers listed here are
based on TCAM capacity as 1-D hardware limit.
TCAM partitioning L2_ONLY/L3_V4 switch L2+L3/L3_DS switch
Default TCAM partitioning 89000 flows 14000 flows
30-70 TCAM partitioning 40000 flows 26000 flows
Table 7: Scale
2.8 Supported Interface Types
2.8.1 Supported
Physical interfaces such as GigabitEthernet, TenGig, HundredGig.
Gig/TenGig/HuGig sub-interfaces
Bundle interfaces
Bundle sub interfaces
BVI (supported only for L3_V4 and L3_DS)
Pseudo-wire Head End sub-interfaces (supported only for L2 and L2_L3 tables)
2.8.2 Not Supported
Satellite interfaces
GRE interfaces
Tunnel-TE interfaces
2.9 Hardware support
2.9.1 Supported:
Typhoon line card
All Chassis types
2.9.2 Not Supported:
Cluster
SIP-700 line card
Trident line card
3 OpenFlow Configuration on ASR9K
3.1 Images
3.2 OnePK config (Mandatory)
RP/0/RSP0/CPU0:ios#config t
RP/0/RSP0/CPU0:ios(config)#onep
RP/0/RSP0/CPU0:ios(config)#datapath transport vpathudp sender-id 1
asr9k-mini-px.vm
asr9k-mpls-px.vm (required for L3VPN, L2VPN)
asr9k-k9sec-px.pie (required for OpenFlow)
3.3 L2 only switch An L2 only OpenFlow switch is attached to a bridge-domain as follows. Pipeline 129 is used for a
L2_ONLY switch.
3.4 L2 + L3 switch An L2_L3 OpenFlow switch is attached to a bridge-domain as follows. Pipeline 130 is used for a L2+L3
switch.
3.5 L3_V4 switch L3_V4 switch can be attached either to a VRF or directly to layer 3 interfaces under global VRF. In case of
VRF, all the interfaces in that VRF become part of the OpenFlow switch. Pipeline 131 is used for a L3_V4
switch.
3.5.1 L3_V4 switch using VRF
openflow
switch 1 pipeline 129
tls trust-point local tp1 remote tp1
bridge-group SDN-2 bridge-domain OF-2
controller ipv4 5.0.1.200 port 6653 security tls
openflow
switch 1 pipeline 130
tls trust-point local tp1 remote tp1
bridge-group SDN-2 bridge-domain OF-2
controller ipv4 5.0.1.200 port 6653 security tls
Openflow
switch 11 pipeline 131
vrf IPv4
controller ipv4 5.0.1.200 port 6653 security none
!
3.5.2 L3_V4 switch using Layer 3 interfaces
3.6 L3 Dual Stack switch L3_DS switch can be attached either to a VRF or directly to layer 3 interfaces under global VRF. In case of
VRF, all the interfaces in that VRF become part of the OpenFlow switch. Pipeline 132 is used for a L3_DS
switch.
3.6.1 L3_DS switch using VRF
3.6.2 L3_DS switch using Layer 3 interfaces
3.7 Netflow A Netflow switch can be used to enable Netflow feature on layer 3 interfaces. A Cisco extension as
mentioned in section 2.4 has been added to enable/disable netflow.
Openflow
switch 11 pipeline 131
interface Bundle-Ether2.1
interface GigabitEthernet0/1/0/6.4
controller ipv4 5.0.1.200 port 6653 security none
Openflow
switch 12 pipeline 132
vrf IPv4
controller ipv4 5.0.1.200 port 6653 security none
!
Openflow
switch 12 pipeline 132
interface Bundle-Ether2.1
interface GigabitEthernet0/1/0/6.4
controller ipv4 5.0.1.200 port 6653 security none
!
4 Show/debug commands
4.1 Openflow show commands
4.2 Show policy-map commands
Openflow
switch 100 netflow
flow monitor mmap sampler smap
interface Bundle-Ether1
interface GigabitEthernet0/1/0/6.5
controller ipv4 5.0.1.1 port 6653 security none
show openflow switch <>
show openflow switch <> controllers | stats
Show openflow switch <> ports
Show openflow switch stats
Show openflow switch flows | brief/summary
Show openflow interface switch <>
show openflow hardware capabilities pipeline <>
show table-cap table-type <>
Show policy-map transient list type pbr
Show policy-map transient type pbr pmap-name <>
Show policy-map transient targets summary
4.3 PBR platform show commands
4.4 Debug commands for OpenFlow Agent
4.5 Debug commands for Policy Manager
4.6 Debug commands for PD PBR
show pbr-pal ipolicy <policy_name | all> location <loc>
show pbr-pal ipolicy <policy_name> iclass <iclass_handle | all> vmr-info
location <loc>
show pbr-pal ipolicy <policy_name> iclass <iclass_handle | all> stats
[clear-on-read] location <loc>
show prm server tcam entries <table> vmr-id <> 100 np0 loc <>
show prm server tcam summary <table> PBR all loc <>
debug openflow switch ovs module ofproto level debug
debug openflow switch ovs module ofproto-plif level debug
debug openflow switch ovs module plif-onep level debug
debug openflow switch ovs module plif-onep-util level debug
debug openflow switch ovs module plif-onep-wt level debug
debug policymgr all
debug policymgr trace
debug policymgr lib all
debug policymgr lib trace
debug pbr-pal all loc
5 Use cases
5.1 L3 network
Problem definition: Three different flows from 3 different sites connected to PE1 are trying to send 350
mbps of traffic each to PE2. The bandwidth of the shortest link Path-2, between PE1 and PE2 is only 1
Gigabit. Hence Path-2 gets congested as soon as the third site begins to send traffic.
OpenFlow solution: OpenFlow controller can be used to install rules on PE1 as follows:
Match on Flow 1 (destined to Video server) and redirect traffic to Path-2
Match on Flow 2 (destined to Web server) and redirect traffic to Path-1
Match on Flow 3 (destined to File transfer server) and redirect traffic to Path-3
Hence utilizing the network bandwidth effectively by redirecting destination specific traffic using
OpenFlow rules.
© 2012 Cisco and/or its affiliates. All rights reserved. 3
PE1
Controller
PE1
PE2
PE1 PE1 PE1 P2
P1
PE1 PE1 PE2 PE1
1
2
3
Video Server
Web
Server
FTP
Server
Traffic Patterns
• Flow 1 - Site 1 Sends 350 mbps of traffic to Video
Server
• Flow 2 - Site 2 Sends 350 mbps of traffic to Web
Server
• Flow 3 - Site 3 Sends 350 mbps of traffic to File
transfer Server
1 Gig
SDN Open flow Components
• Open flow Controller – Runs as an application in any VMs
• OF Agent in CISCO routers or switches creates a TCP connection to
controller and uses Open flow protocol to communicate.
• User or application Installs Open flow rules from controller
• Rule 1 : Site 1 to Video Server – Path 2
• Rule 2 : Site 2 to Web Server – Path 1
• Rule 3 : Site 3 to FTP Server – Path 3
• Traffic Flow now
Path -2
Path -1
Path -3
Path -1
Path -3
Problem : Adding Flow 3 congests 1 Gige link between PE1 and PE2. Which is the
Shortest path.
Solution : Open flow would be used to
install flows and redirect traffic
Data CEnter
Solution : Thus we are able to efficiently utilize bandwidth by redirecting
Destination Specific Traffic using
openFlow rules
5.2 L2 network
Problem definition: Enterprise Data Center needs to perform data backup to multiple other backup sites
based on the Traffic flow. The Main DC is in Vlan 100 and Backup sites are at VLAN 1000,1001,1002.
These Sites are interconnected through L2VPN. In this topology if customer needs to selectively
determine Backup traffic destination site he/she needs to send them in separate VLANS.
OpenFlow solution: With Open flow we can match on any Layer 2 header fields (in this example we
have taken priority bits) and steer the traffic to go on any L2 interconnect and also rewrite the VLANs
appropriately.
5.3 Rice University Use Case Rice University's campus network is a traditional MPLS VPN network. There are multiple affinity
networks running on top of this campus network. These affinity networks are separated via different
VRFs from MPLS/VPN point of view. In some cases, they need to grant some temporary access from one
affinity network to another affinity network. In the example below, a staff which belongs to staff
network wants to access some stuff in student network.
© 2012 Cisco and/or its affiliates. All rights reserved. 4
Controller
PE1
PE2
PE3
PE4
Data Center Backup
Server 1
L2 Interconnect -1
L2 Interconnect -2
L2 Interconnect -3
Enterprise Data
Center
Data Center Backup
Server 2
Data Center Backup
Server 3
Vlan 100
Vlan 1000
Vlan 1001
Vlan 1002
SDN Open flow Components
• Open flow Controller – Runs as an application in any VMs
• OF Agent in CISCO routers or switches creates a TCP connection to controller and uses
Open flow protocol to communicate.
• User or application Installs Open flow rules from controller
• Rule 1 : Traffic from enterprise DC coming with VLAN 100 and TOS value 1 should
be steered to L2 interconnect 1 and VLAN rewritten as 1000
• Rule 2 : Traffic from enterprise DC coming with VLAN 100 and TOS value 2 should
be steered to L2 interconnect 2 and VLAN rewritten as 1002
• Rule 3 : Wild card match send it to Open flow controller ( PACKET_IN/ PUNT ).
• Controller will react to PACKET_IN message with vlan 100 and TOS 3 by
pushing down a flow to steer traffic to L2 interconnect 3 and rewrite vlan to
1002.
• The idle time out is set to 300 sec
• Traffic would be Steered to L2 interconnect 3
L2 Interconnect –X – L2VPN PW
This type of requests' handling is normally done via PBR or static route to provide VRF leaking from one
to another. Rice University has about 11 such kind of affinity networks. They maintain a 11x11 policy
matrix to decide if VRF leaking is allowed or not.
It is not a trivial job to maintain this 11x11 matrix logic consistently across multiple PEs. Rice University
looks for a simple solution via SDN framework to solve this problem.
Rice University likes Open Flow's flexibility to steer specific traffic. But they don't like pure Open Flow
approach. In pure Open Flow approach, they need to take care of basic ping handling as well. It is too
much tax for them to get this kind of flexibility. Therefore, they feel Open Flow hybrid mode is a better
choice for them.
6 Summary
ASR9k implements OpenFlow 1.0 and OpenFlow 1.3 in XR-5.1.1 release. Section 2 describes the table
types, matches, actions and counters supported in this release. Interface and hardware support is also
documented in section 2. ASR9K plans to continue adding support for other matches and actions such
as match on ICMP v4/v6 type/code, MPLS label etc. in upcoming releases.
Section 5 describes some of the use cases in detail. Various combinations of matches and actions can
lead to many other use cases.
Appendix A FAQ
Q. What are Cisco extensions and what can they be used for?
A. The following Cisco extensions to the list of actions are supported.
1. Set ipv4 nexthop
2. Set ipv6 nexthop
3. Set FCID
4. Enable/disable netflow
Set ipv4/ipv6 nexthop actions are used to redirect an ipv4/ipv6 packet to the specified nexthop address,
instead of using the destination address in the packet. This provides ABF (ACL Based Forwarding) kind of
functionality using OpenFlow. However, VRF support and nexthop tracking as supported by CLI based
ABF feature is not supported in this release.
“Set FCID” action can be used to support PBTS (Policy Based Tunnel Selection) functionality using
OpenFlow.
Enable/disable netflow is used to enable or disable netflow on an interface using OpenFlow.
Q. What are the supported controller types?
A. All OpenFlow controllers should work. However, IXIA, ODL, and POX controllers were used for testing
on ASR9K.
Q. Are the flow priorities supported?
A. Flow priorities are supported. Lowest priority is zero and highest is 32768.
Q. Is there a limit on the number of actions per flow?
A. There is as such no limit on the number of actions supported per flow. However, if the data
associated with multiple actions happens to be large such as source MAC address, destination address,
set ipv6 nexthop etc. we may hit the hardware limit. In such cases, an error will be returned and the
flow will not be programmed in the hardware. Only a single output action is supported in XR-5.1.1.
24
Q. Do we need to use only Management Ethernet port to connect to the OpenFlow controller?
A. Any supported physical interface including Management Ethernet interface can be used to connect to
the controller. However for HA purposes, only physical port should be used to connect to the controller.
Q. Does ASR9K support OpenFlow channel auxiliary connections?
A. Auxiliary connections are not supported on ASR9K.
Q. Can multiple controllers be connected to the same ASR9K?
A. Yes, multiple controllers can be supported on ASR9K. Maximum of 8 controllers per switch can be
supported.
Q. What is the periodicity of stats collection by default? Can it be changed?
A. Flow statistics are collected by default at the rate of 50 flows per second. However, this periodicity
can be changed using the following command.
Openflow
Switch <switch_id> pipeline <pipeline number>
statistics collection-period <period>
Q. Does ASR9K support idle-timeouts and hard timeouts?
A. Yes, idle timeouts and hard timeouts are supported on ASR9K. The OpenFlow controller can set idle
timeouts as well as hard timeouts. Idle-timeouts are implemented based on stats collection; hence the
granularity of idle-timeouts is related to the stats-collection interval. Command “sh openflow switch
<switch_id> flows” can be used to display these timeouts.
Appendix B Output of commonly used show commands
OpenFlow Agent commands:
RP/0/RSP0/CPU0:OFA2#show run openflow switch
openflow
switch 1 pipeline 132
interface GigabitEthernet0/1/0/10
dataplane-default secure
statistics collection-period 0
controller ipv4 56.1.10.2 port 6633 security none
!
!
RP/0/RSP0/CPU0:OFA2#show openflow switch 1 controller
Logical Switch Id: 1
Total Controllers: Not available
Controller: 1
Address : 56.1.10.2:6633
Protocol : tcp
VRF : default
Local Trustpoint: : Not available
Remote Trustpoint: : Not available
Connected : Yes
Role : Master
last_error : Connection timed out
state : ACTIVE
sec_since_connect : 57
RP/0/RSP0/CPU0:OFA2#show openflow switch 1 ports
Logical Switch Id: 1
Port Interface Name Config-State Link-State Features
1 Gi0/1/0/10 PORT_UP LINK_UP 0
26
RP/0/RSP0/CPU0:OFA2#show openflow switch 1 stats
Logical Switch Id: 1
OFPST_PORT reply (xid=0x0):Total ports: 1
Port 1: rx pkts=56587983, bytes=4661065366, drop=9635,
errs=0,
tx pkts=8619, bytes=401668, drop=0, errs=0,
Logical Switch Id: 1
OFPST_TABLE reply (xid=0x0):Total tables: 1
Table 0: classifier
Wildcards = 0x3fffff
Max entries = 50000
Active entries = 1
Number of lookups = 0
Number of matches = 0
RP/0/RSP0/CPU0:OFA2#show openflow switch 1 flows
Logical Switch Id: 1
Total flows: 2
Flow: 1
Match: ipv6,ipv6_dst=192:1::2
Actions: output_nh(ipv6=2001:20::2)
Priority: 0
Table: 0
Cookie: 0x1
Duration: 196.353s
Number of packets: 0
Number of bytes: 0
Flow: 2
Match:
Actions: NORMAL
Priority: 0
Table: 0
Cookie: 0x0
Duration: 223.331s
Number of packets: 0
Number of bytes: 0
27
RP/0/RSP0/CPU0:OFA2#show openflow switch flows summary
Total Forwarding flow count: 2
Total Netflow flow count : 0
Total flow count : 2
Logical Switch Id: 1
Switch flow count : 2
RP/0/RSP0/CPU0:OFA2#show openflow switch flows brief
Logical Switch Id: 1
Total flows: 2
Flow: 1 Match: ipv6,ipv6_dst=192:1::2 Actions:
output_nh(ipv6=2001:20::2)
Priority: 0, Table: 0, Cookie: 0x1, Duration: 333.131s,
Packets: 0, Bytes: 0
Flow: 2 Match: Actions: NORMAL
Priority: 0, Table: 0, Cookie: 0x0, Duration: 360.109s,
Packets: 0, Bytes: 0
Total flow count: 2
28
RP/0/RSP0/CPU0:OFA2#show openflow switch 1
Logical Switch Context Id: 1
Switch type: Forwarding
Pipeline ID: 132
Data plane: secure
Fallback: normal
Config state: no-shutdown
Working state: enabled
Rate limit: 0
Burst limit: 0
Max backoff (sec): 8
Probe interval (sec): 5
TLS local trustpoint: NONE
TLS remote trustpoint: NONE
Stats coll period (sec): disabled
Logging flow changes: Not available
OFA Description:
Manufacturer: Cisco Systems, Inc.
Hardware: ASR-9006-AC V01
Software: 5.1.2.11I of_agent 1.0
Serial Num: FOX1333GVJS
DP Description: OFA2:sw1
OF Features:
DPID: 0001002651cdcc8e
Number of tables: 1
Number of buffers: 256
Capabilities: FLOW_STATS TABLE_STATS
PORT_STATS
Actions: SET_NW_SRC SET_NW_DST
SET_TP_SRC SET_TP_DST SET_FCID SET_NH
Controllers:
56.1.10.2:6633, Protocol: TCP, Openflow 1.3, last
alive ping: 2014-02-21 07:32:00
Bridge domain: NONE
VRF: default-00001
Interfaces:
GigabitEthernet0/1/0/10
Policy commands
RP/0/RSP0/CPU0:OFA2#show policy-map transient list type pbr
1) PolicyMap: onep-pmap-7416-4 Type: pbr (transient)
Total Flows : 2
RP/0/RSP0/CPU0:OFA2#show policy-map transient type pbr pmap-
name onep-pmap-7416-4
policy-map type pbr onep-pmap-7416-4
handle:0x34000006
table description: L3 IPv4 and IPv6
class handle:0x74000007 sequence 65535
match ethertype ipv6
match destination-address ipv6 192:1::2/128
ipv6 next-hop 2001:20::2
!
class handle:0xf4000006 sequence 4294967295 (class-default)
transmit
!
end-policy-map
!
RP/0/RSP0/CPU0:OFA2#show policy-map transient targets type
pbr pmap-name onep-pmap-7416-4
1) Policymap: onep-pmap-7416-4 Type: pbr
Targets (applied as main policy):
GigabitEthernet0/1/0/10 input
Total targets: 1
ASR9K Platform specific commands
RP/0/RSP0/CPU0:OFA2#show pbr-pal ipolicy onep-pmap-7416-4
detail location 0/1/CPU0
policy name : onep-pmap-7416-4
number of iclasses : 2
number of VMRs : 2
ucode format : 23
vmr id for NP0 : 71
interface count : 1
interface list : Gi0/1/0/10
RP/0/RSP0/CPU0:OFA2#show pbr-pal ipolicy onep-pmap-7416-4
iclass all vmr-info location 0/1/cpu0
iclass handle : 0x74000007
ifh : x
protocol : x
source ip addr : x
dest ip addr : 192:1::2/128
source port : x
dest port : x
DSCP : x
ethertype : 0x86dd
vlan id : x
vlan cos : x
source mac : x
dest mac : x
result :
1100000000000001a4000000020000000000000000200100200000000000000
0
iclass handle : 0xf4000006
ifh : x
protocol : x
source ip addr : x
dest ip addr : x
source port : x
dest port : x
DSCP : x
ethertype : x
vlan id : x
vlan cos : x
source mac : x
dest mac : x
result :
110000000000000000000000000000000000000000000000000000000000000
0
31
TCAM commands
RP/0/RSP0/CPU0:OFA2#show pbr-pal ipolicy onep-pmap-7416-4 iclass
all stats location 0/1/CPU0
iclass packets/bytes drop
packets/drop bytes
74000007 22376/1924336 0/0
f4000006 6/524 0/0
RP/0/RSP0/CPU0:OFA2#show prm server tcam summary 576-LT pBR all
location 0/1/CPU0
Node: 0/1/CPU0:
----------------------------------------------------------------
TCAM summary for NP0:
TCAM Logical Table: TCAM_LT_ODS8 (3), free entries: 14756,
resvd 127
ACL Common Region: 448 entries allocated. 448 entries free
Application ID: NP_APP_ID_PBR (5)
Total: 1 vmr_ids, 2 active entries, 2 allocated entries.
TCAM summary for NP1:
TCAM Logical Table: TCAM_LT_ODS8 (3), free entries: 14757,
resvd 128
ACL Common Region: 448 entries allocated. 448 entries free
Application ID: NP_APP_ID_PBR (5)
Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
32
RP/0/RSP0/CPU0:OFA2#show prm server tcam entries 576-LT vmr-id
71 10 all location 0/1/CPU0
Node: 0/1/CPU0:
----------------------------------------------------------------
ODS NP: 0, LT: 3, AppId: 5, VmrId 71, Offset 0, Entries
2, Shadow 0
60e00 V: a171 0000 0000 0000 0000 0000 0000 0000 0000 0000 23
0000 0000 0000 0000 0000 0000 0000 0000 0200 0000
0000 0000 0000 0000 0100 9201 0000 0000 0000 0000
0000 0000 86dd 0000 0000 0000 0000 0000 0000 0000
M: 0000 ffff ffff ffff ffff ffff ffff ffff ffff ffff
ffff ffff ffff ffff ffff ffff ffff ffff 0000 0000
0000 0000 0000 0000 0000 0000 ffff ffff ffff ffff
ffff ffff 0000 ffff ffff ffff ffff ffff ffff ffff
R: 110000d0 18610001 a4000000 02000000 00000000 00200100
20000000 00000000
7ee08 V: a170 0000 0000 0000 0000 0000 0000 0000 0000 0000 23
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
M: 0003 ffff ffff ffff ffff ffff ffff ffff ffff ffff
ffff ffff ffff ffff ffff ffff ffff ffff ffff ffff
ffff ffff ffff ffff ffff ffff ffff ffff ffff ffff
ffff ffff ffff ffff ffff ffff ffff ffff ffff ffff
R: 110000c0 18610000 00000000 00000000 00000000 00000000
00000000 00000000