A proposal of the OpenFlow controller development support tool - using Logic and Flow ID label -

Yutaka Yasuda, Kyoto Sangyo University

Introduction

•OpenFlow

Switch control mechanism by software program

•Problem of controller programming

Programmer does not have any method to confirm how reflected their code to the flow control, directly and intuitively

“This code, how does work on... which flow?”“This flow, which code does make it?”

host1

host4host2

host3sw1 sw3

sw2 sw4

Goal of today : development support tool proposal

Controller

• bind code to flow control information• enable to trace the affected flow

step 1:　Binding Code and Flow Entry

ex. : simple 2 ports switch and typical code for that

def packet_in_callback(dpid, inport, reason, len, bufid, packet): flow = extract_flow(packet) flow[core.IN_PORT] = inport if inport == 1 :

1 2(inport=1) [output=2]

(inport=2) [output=1]convention of here : (match fields) [actions]

NOX like pseudo code:callback

make match fields

set flow entry

make actions

(inport=1) [output=2]

(inport=2) [output=1]

outport = 2 else: outport = 1 actions = [[openflow.OFPAT_OUTPUT, [0, outport]]] inst.install_datapath_flow(dpid, flow,..., actions, ...) return CONTINUE

Logic ID label setting

(inport=1) [output=2]

(inport=2) [output=1]

def packet_in_callback(dpid, inport, reason, len, bufid, packet): flow = extract_flow(packet) flow[core.IN_PORT] = inport if inport == 1 : outport = 2 else: outport = 1 actions = [[openflow.OFPAT_OUTPUT, [0, outport]]] inst.install_datapath_flow(dpid, flow,..., actions, ...) return CONTINUE

You can see the correspondence relation between code and flow entry

Logic ID label setting

(inport=1) [output=2]

(inport=2) [output=1]

def packet_in_callback(dpid, inport, reason, len, bufid, packet): flow = extract_flow(packet) flow[core.IN_PORT] = inport if inport == 1 :

outport = 2 else:

outport = 1 actions = [[openflow.OFPAT_OUTPUT, [0, outport]]] inst.install_datapath_flow(dpid, flow,..., actions, ...) return CONTINUE

inst.markLogicPoint( 101 )

inst.markLogicPoint( 102 )

#101

#102

Insert the function to set the Logic ID label to flow entry

Logic ID label setting

(inport=1) [output=2]

(inport=2) [output=1]

def packet_in_callback(dpid, inport, reason, len, bufid, packet): flow = extract_flow(packet) flow[core.IN_PORT] = inport if inport == 1 :

outport = 2 else:

outport = 1 actions = [[openflow.OFPAT_OUTPUT, [0, outport]]] inst.install_datapath_flow(dpid, flow,..., actions, ...) return CONTINUE

inst.markLogicPoint( 101 )

inst.markLogicPoint( 102 )

#101

#102

Match Fields Counters Actionsflow entry: cookie(64bits)

( inport=1 ) [ outport=2 ]#102

Cookie field is good to keep the label information

Result : cross reference of code and flow entriesswitch

def getOutport(dpid, vport, packet): if not OPORTS.has_key(dpid): log.err('Unknown dpid=%x on getoutput' % (dpid),system='paniersw') return False if not OPORTS[dpid].has_key(vport): log.err('Invalid (dpid,vport)=(%x,%d) on getOutport' % (dpid, vport),system='paniersw') return False ops=OPORTS[dpid][vport] for (rule, outport, cookie) in ops: if rule == RULE80 and testTCPport(80, packet): inst.setLogicMark( 101 ) return (outport, cookie) elif rule == RULE443 and testTCPport(443, packet): inst.setLogicMark( 102 ) return (outport, cookie ) elif rule == RULE25 and testTCPport(25, packet): return (outport, cookie) elif rule == ANY: return (outport, cookie) return False

def forward_l2_packet(dpid, inport, vport, packet, buf, bufid): dstaddr = packet.dst.tostring() if not ord(dstaddr[0]) & 1 and inst.st[dpid].has_key(dstaddr): prt = inst.st[dpid][dstaddr] if prt[0] == vport: inst.setLogicMark( 122 ) log.err('**warning** learned port = inport', system="paniersw") myFlood(dpid, bufid, buf, vport, packet) else: inst.setLogicMark( 123 ) log.msg('installing flow for ' + str(packet), system="paniersw") flow = extract_flow(packet) flow[core.IN_PORT] = inport (outport, cookie) = getOutport(dpid, prt[0], packet) actions = [[openflow.OFPAT_OUTPUT, [0, outport]]] inst.install_datapath_flow(dpid, flow, CACHE_TIMEOUT, openflow.OFP_FLOW_PERMANENT, actions, bufid, openflow.OFP_DEFAULT_PRIORITY, inport, buf, cookie) else: inst.setLogicMark( 124 ) myFlood(dpid, bufid, buf, vport, packet)

flow entries

controller program

Reference from flow to code can tell : “This flow entry comes from this code”

Reference from code to flow can tell : “Which flow entries are affected by this code”

1 6

2

3

4

5

7

8

9

10

cookie match fields, etc..

(inport=1) [ outport=6 ]

101 …………………

…………………

…………………

…………………

flow entries

Single branch : OKswitch

def getOutport(dpid, vport, packet): if not OPORTS.has_key(dpid): log.err('Unknown dpid=%x on getoutput' % (dpid),system='paniersw') return False if not OPORTS[dpid].has_key(vport): log.err('Invalid (dpid,vport)=(%x,%d) on getOutport' % (dpid, vport),system='paniersw') return False ops=OPORTS[dpid][vport] for (rule, outport, cookie) in ops: if rule == RULE80 and testTCPport(80, packet): inst.setLogicMark( 101 ) return (outport, cookie) elif rule == RULE443 and testTCPport(443, packet): inst.setLogicMark( 102 ) return (outport, cookie ) elif rule == RULE25 and testTCPport(25, packet): return (outport, cookie) elif rule == ANY: return (outport, cookie) return False

def forward_l2_packet(dpid, inport, vport, packet, buf, bufid): dstaddr = packet.dst.tostring() if not ord(dstaddr[0]) & 1 and inst.st[dpid].has_key(dstaddr): prt = inst.st[dpid][dstaddr] if prt[0] == vport: inst.setLogicMark( 122 ) log.err('**warning** learned port = inport', system="paniersw") myFlood(dpid, bufid, buf, vport, packet) else: inst.setLogicMark( 123 ) log.msg('installing flow for ' + str(packet), system="paniersw") flow = extract_flow(packet) flow[core.IN_PORT] = inport (outport, cookie) = getOutport(dpid, prt[0], packet) actions = [[openflow.OFPAT_OUTPUT, [0, outport]]] inst.install_datapath_flow(dpid, flow, CACHE_TIMEOUT, openflow.OFP_FLOW_PERMANENT, actions, bufid, openflow.OFP_DEFAULT_PRIORITY, inport, buf, cookie) else: inst.setLogicMark( 124 ) myFlood(dpid, bufid, buf, vport, packet)

flow entries

controller program

A recorded Logic ID directly means the branch path that the code actually passed through.(Programmer puts the Logic ID record function to the important and memorable points of the code. )

1 6

2

3

4

5

7

8

9

10

cookie match fields, etc..

(inport=1) [ outport=6 ]

101 …………………

…………………

…………………

…………………

flow entries

Multiple branches : needs the past recorddef getOutport(dpid, vport, packet): if not OPORTS.has_key(dpid): log.err('Unknown dpid=%x on getoutput' % (dpid),system='paniersw') return False if not OPORTS[dpid].has_key(vport): log.err('Invalid (dpid,vport)=(%x,%d) on getOutport' % (dpid, vport),system='paniersw') return False ops=OPORTS[dpid][vport] for (rule, outport, cookie) in ops: if rule == RULE80 and testTCPport(80, packet): inst.setLogicMark( 101 ) return (outport, cookie) elif rule == RULE443 and testTCPport(443, packet): inst.setLogicMark( 102 ) return (outport, cookie ) elif rule == RULE25 and testTCPport(25, packet): return (outport, cookie) elif rule == ANY: return (outport, cookie) return False

def forward_l2_packet(dpid, inport, vport, packet, buf, bufid): dstaddr = packet.dst.tostring() if not ord(dstaddr[0]) & 1 and inst.st[dpid].has_key(dstaddr): prt = inst.st[dpid][dstaddr] if prt[0] == vport: inst.setLogicMark( 122 ) log.err('**warning** learned port = inport', system="paniersw") myFlood(dpid, bufid, buf, vport, packet) else: inst.setLogicMark( 123 ) log.msg('installing flow for ' + str(packet), system="paniersw") flow = extract_flow(packet) flow[core.IN_PORT] = inport (outport, cookie) = getOutport(dpid, prt[0], packet) actions = [[openflow.OFPAT_OUTPUT, [0, outport]]] inst.install_datapath_flow(dpid, flow, CACHE_TIMEOUT, openflow.OFP_FLOW_PERMANENT, actions, bufid, openflow.OFP_DEFAULT_PRIORITY, inport, buf, cookie) else: inst.setLogicMark( 124 ) myFlood(dpid, bufid, buf, vport, packet)

flow entries1 6

2

3

4

5

7

8

9

10

cookie match fields, etc..

(inport=1) [ outport=6 ]

101 …………………

…………………

…………………

…………………

flow entries 101102

122123

125123

If memorable points are (multiple) exists and located them the isolated branch, it needs to record the both of them. (in right figure, 101 and 123 are.)It is not good to overwrite by the recent single label (123).

overwriting

Queuing for multiple label recording

flow entries1 6

2

3

4

5

7

8

9

10

cookie match fields, etc..

(inport=1) [ outport=6 ]

101 …………………

…………………

…………………

…………………

flow entries

123

1 6

2

3

4

5

7

8

9

10

cookie match fields, etc..

(inport=1) [ outport=6 ]

101,123 …………………

…………………

102,124 …………………

…………………

... ... ... ... ... ... 101 123

123

queuing38

(8 x 8 bits)

To keep the past record, Logic ID label will be queuing to Cookie field (64bits). The right figure shows the typical case, 8bits label queuing in 8 stages. The combination of bit width and the number of stages are flexible to fit the target controller program.

overwriting queuing

overwriting

Restrictions : binding code and flow entry

•Some controller SDKs may use Cookie for their purpose

Floodlight uses 32bits (12bit AppID, 20bit reserve)

• Increase switch loads to collect information

“In short, the existing statistics mechanisms are (both) relatively high overhead” (J. Mogul et al., 2010)

J. Mogul et al. “DevoFlow: Cost-Effective Flow Management for High Performance Enterprise Networks”, In Proc. ACM SIGCOMM HotNets-IX, 2010

step 2: 　Tracing Flow Over the Switch

Flow tracing (idea)

•Demand : want to see what happen on the flow from here on

Now, flow entries and code are ready to refer each other by this proposal

Next, want to observe the behavior of flows after then

•Problem : Flow entries have insufficient info to reconstruct the original path

•Reason : Wildcards and Maskbits

There are many cases which cannot figure out the correct path from flow entry info on switches

Flow merging by wildcards / maskbits

Wildcard fail case

(1.0.0.1, *, *) [#4]

(1.0.0.2, *, *) [#4]

sw1

1

2

3

4

5

6

(*, 80, *) [#4]

(*, 25, *) [#5]

sw2

1

2

3

4

5

6

to http proxy

to smtp gateway

host1

host2

convention : ( IP, src port, dst port ) [ outport ]

note:If all flow entries specify the same match fields set, it is possible to check up completely. Typical case of that is “all flows specify all match fields exactly”.

•Typical case to fail to trace the flow pathThere are wildcards in different match fields and it makes fail to reconstruct the original flow path correctly, to check up flow entries in two switches

Ex: The upper flow of sw1 (to 1.0.0.1) will be split into multi flows on sw2. (red lines)And flow entries of sw2 includes the other flows. (blue lines)Nobody knows how many packets will go upper flow of sw1 to upper flow (port 80) of sw2.

Flow tracing (how to)

•Set the label to packet and bring it by packet themselves to the next hop

•Label setting and detecting should be done by switch

It means label should be on the field of flow entry

•Set the label by “Action” then send to the next switch

•Detect the labeled flow by “Match Fields” then treat it separately from unlabeled flow

Flow separation under wildcards / maskbits

Wildcard fail case

•Separation the flow by the labelThe separated flow entry will be made in the next switch, then it is fully traceable

Ex : The case a flow will be split on the next switch. (red line)All “5” labeled packets will make only “5” matching flow entries on sw2And “5” labeled flows on sw2 does not include any unlabeled flow.

(1.0.0.1, *, *, 0) [5, #4]

(1.0.0.2, *, *, 0) [=, #4]

sw1

1

2

3

4

5

6

(*, 80, *, 5) [ =, #4]

(*, 80, *, 0) [ =, #4]

sw2

1

2

3

4

5

6

to http proxyhost1

host2to smtp gateway

(*, 25, *, 5) [ =, #5]

(*, 25, *, 0) [ =, #5]

convention : ( IP, src port, dst port, label ) [ label, outport ]

Flow tracing (the structure)

(to servers)

(to servers)

flow table

flow tableflow table

flow table

Controller

cookie Match Counter Action101, 123 IP(dst)=X out=6

cookie Match Counter Action????? ????? out=3

Packets have info for tracing and it will be a key to check up flow entries of switches

1

2

3

4

5

6

Ex : select outport by IP and the code for it

def packet_in_callback(dpid, inport, reason, len, bufid, packet): match = extract_flow(packet) iph = packet.find('ipv4') if ip_to_str(iph.dstip) == "1.0.0.1" : outport = 4 else outport = 5 actions = [[openflow.OFPAT_OUTPUT, [0, outport]]] inst.install_datapath_flow(dpid, flow,..., actions, ...) return CONTINUE

#4](1.0.0.1, *, * ) [convention : (IP, port-dst, port-src) [ #outport ]　

Here is a switch that selects output port depends on the destination IP address of the packet

Flow ID Label settingdef packet_in_callback(dpid, inport, reason, len, bufid, packet): match = extract_flow(packet) iph = packet.find('ipv4') if ip_to_str(iph.dstip) == "1.0.0.1" : outport = 4 inst.markFlowLabel( 5 ) else outport = 5 actions = [[openflow.OFPAT_OUTPUT, [0, outport]]] inst.install_datapath_flow(dpid, flow,..., actions, ...) return CONTINUE

Match Fields Counters Actionsflow entry:

( label=0 ) [ set label=5 ]

, 0 5, #4](1.0.0.1, *, * ) [

Insert the function to set the Flow ID Label and the label will be set by Action feature of the switch

The place to record the Flow ID label

•Use ToS field (6bits) for IPv4

OpenFlow supports ToS as Match Field and Action

Not regularly used than other fields

• It is possible to use VLAN id or MPLS label but....

Very commonly used for OpenFlow applicable field, such as datacenter

unexpected VLAN or MPLS tag insertion may occur some malfunctions to existing controller programs

• for IPv6, Flow Label (20bits) is good to use

Label overlapping : assigning to bit

•Overlapping : set different labels to single flow by multiple switches

Cannot overwrite existent label info•Label as bit pattern

label=n means ToS=2^(n-1)independently writable for overlapping case

•Very few labels to able to set (ToS has only 6 bits!)Use string label in the code, dynamically assign the number when it turned onTurn on/off by remote operation of controller program

Restriction : Tracing flow over the switch

•Diverted field does not usable for original purpose( v4: ToS / v6: flowlabel)

•Need to remove the label on the edge

To prevent fail-reaction by the ToS value at the outside of the target Open Flow network

There are some topology investigation researches. OFDP : OpenFlow Discovery Protocol by GENI. Topology module by Trema (LLDP base)

•Not applicable for not IP protocols

For not IP protocols?

• Ideally...”Should propose to add flow-id-field as OF std.”

It's a chicken and egg situation

New spec might not applicable on existent switch

•Realistically... “not a big issue?”

OpenFlow is focusing IP (v4) strongly

In L2 use case, processed by exact match of MACs(label does not required, just checking them up is enough)

Full picture

Controller

cookie Match Counter Action

88, 101 tos=0 tos=16

markLogicPoint( 101 )markFlowLabel( 5 )

markLogicPoint( 121 )markFlowLabel( 3 )

if dpid == 25

else if dpid == 11

cookie Match Counter Action

99, 121 tos=16 tos=20

cookie Match Counter Actiontos=16 ( keep tos )

20=2^(5-1) +2^(3-1)

dpid 11

dpid 25

16=2^(5-1)

System layout

Controller

flow table

flow table

flow table

flow table

flow table aggregator

visualizer

statistics data

Controller SDK

tool

Controller App

OpenFlow standard !

operation console

labelon/off

red color means “this part should be implemented”

Conclusion

•Proposed the OpenFlow Controller development support tool•Basic design of the mechanism for binding code and flow

It enables cross referencing the logic and flow each otherIt enables tracing the flow over switches

•Required preparationNeed to add some functions to controller SDK

•To use this featureInsert the function call to your code lines, but that’s allNo special requirement for switch, just OpenFlow capability

Conclusion

•Future work• Finishing up the NOX implementation• Confirming the applicability for other SDK by porting• Visualization• Proposing a new “flow id field” as OF switch standard

operation console

flow table aggregatorvisualizer

statistics data

Controller SDKtool

Controller App

flow table

flow table

flow table

flow tableon/off

Thank you.

if you have any interest of this idea,porting to the other controller,

visualization, making a commercial product,

or joint research,

feel free to contact me please.