Ryu: SDN framework and Python experience

Isaku Yamahata <yamahata@private.email.ne.jp> <isaku.yamahata@gmail.com>

Pycon APAC 2013, September 14

Agenda

● Introduction● Ryu: SDN framework● Ryu Internals

– Openstack support

● Ryu development● Python experience through Ryu

This presentation represents my personal view/opinion. Not Ryu project nor any companies.

Who am I?

● My background is OS/virtualization/cloud– Not network guy

● Programming language– C/C++/assembler

● Projects I've contributed– Linux

– Virtualization● Qemu, KVM, Xen

– OpenStack● Nova, neutron(formarly quantum) Ryu plugin

– Open vSwitch

● My python experience had begun with OpenStack and Ryu

Introduction

What's SDN? And why?

● Software Defined Networking– Making network programmable

– http://www.opennetsummit.org/why-sdn.html● SDN is a disruptive technology that is making

networks programmable by ordinary programmers using ordinary software running on ordinary operating systems in ordinary servers. With SDN, the introduction of new features becomes less manual, less prone to error, and faster to implement.

– [Paraphrased from the HotSDN ‘13 Solicitaion] ● Software Defined Networking (SDN) is a refactoring

of the relationship between network devices and the software that controls them.

● Motivations behind SDN– Academic research

● Allow researchers to program/modify switches freely

– Industry technology trends● Virtualization/cloud technology ● Network is behind those technology

progress● Networkig virtualization/automation

From http://opennetsummit.org/why.html

Separating data/control plan

From http://www.opennetsummit.org/why-sdn.html

SDN architecture

Openflow

datapath(hardware)

controlpath(software)

datapath(hardware)

controlpath(software) openflow

Openflow controller

Openflow protocol(tcp/ssl)

Ethernet switch Openflow ethernet switch

● protocol to control switches

Flow table and match/action

Openflow controller

Flow table

MACsrc

MACdst

IPsrc

IPdst

TCPsrc

TCPdst ... action

Packet in eventWhen entry miss

* * * * * 80outputport N

port PacketportPacketPacket match Action

Ryu, a component-based software-defined networking

framework

What's Ryu?

流 龍Flow

Oriental Dragion,A god of water

Manages flow control to enable intelligent networking

http://ja.wikipedia.org/wiki/%E8%B5%A4%E7%9B%AE%E5%9B%9B%E5%8D%81%E5%85%AB%E6%BB%9D

What's Ryu?

● a component-based software-defined networking framework

● License: Apache 2.0 ● Fully written in Python● Supports various protocols for managing

network devices– OpenFlow, Netconf, OF-config, SNMP etc.

● Official site http://osrg.github.io/ryu/● MLhttps://lists.sourceforge.net/lists/listinfo/ryu-devel● Download https://github.com/osrg/ryu● Documentation http://ryu.readthedocs.org/en/latest/● Wiki https://github.com/osrg/ryu/wiki

Supported features/protocols

● Openflow protoocol– OF-wire: 1.0,1.2, 1.3, Nicira extension

– OF-config 1.1

● Non-openflow protocols– Netconf, OVSDB, netflow, sflow,

VRRP, SNMP● Snmp: Enterprise OID: 41786

– Ryu can configure Open vSwitch directly without ovs-vsctl, ovsdb-client

Some features are under development. The patches can be found on ML archive.

● RyuApp, library– Packet library– STP, LACP

– Sample apps, etc...

– Conversion from/to JSON representation from/to OF

– RPC to communicate/control Ryu

● Integration with other project– OpenStack– HA with Zookeeper

– IDS(Intrusion Detection System) with snort

OF/firewall/router REST API

● OF REST (ofctl_reset)– Insert/delete openflow rule

● Firewall (rest_firewall)– Configure firewall

– https://github.com/osrg/ryu/wiki/Third-Party-Tools,-Etc.

● Router(rest_router)– Configure router

Ryu

REST

OF REST API add a flow entry POST http://example.org/stats/flownetry/add delete flow entries DELETE http://example.org/stats/flowentry/deleteget flow statsGET http://example.org/stats/flow/{dpid}

allow

drop

firewall

OF switch

Ryu

REST

OF switch

router

Topology Viewer

● Discover topology by LLDP and show topology/flows dynamically

HA support

● Centralized controller is single pointer of failure(SPOF)

● Ryu + Zookeeper can be used to address SPOF

Ryu Ryu

zookeeper

OF switch

master slave

failover

IDS support

● Snort integration

https://github.com/osrg/ryu/wiki/Snort-Integration

Ryu

OF switch

SortControl app IDS(snort)

1. L1-L4 matching

2. send patcket to IDS

3. alert4. take actionse.g. loggingt

Ryu Internals

Ryu implementation

● Quite normal python program from the point of implementation view – It doesn't use any special tricks

● Event driven– Event source/dispatcher/sink– Core(= Event dispatcher) is very small– It is so generic that Ryu can be used without OpenFlow

● Component based– Event source/sink are created as components

● Even OpenFlow related codes are so

– Message passing via events, not directly communite.

Ryu architecture

● Follows standard SDN architecture

OpenFlow switch OpenFlow switch Network device

SDN appsWell defined API(REST, RPC...)

Open protocols(OpenFlow, OF-config,NETConfig, OVSDB...)

SDN apps SDN apps

Ryu SDN framework

OpenFlowParser/serializer

Event dispatcher

Ryu built-in app(tenant isolation,

Topology discovery, firewall )Ryu App

libraries

Protocol support(OVSDB, VRRP, ...)

Ryu App...

operator openstack User app

Control layer

Application layer

Aio/thread

● Uses eventlet– Like OpenStack

– gevent was used before

– switched to eventlet for pypy

● twisted was not adopted for simplicity● eventlet(or gevent) is cooperative threading, so

some cautions are needed– This is different from preemptive threading like pthread

Event Dispatcher● class AppManager and class RyuApp● The guts of Ryu● Decouples event sources/sinks

– Event sources generate whatever events

– Event sinks register handlers dynamically

● Dispatches events based on class of events– To event sinks that want class of events

– Class is a first class object in Python

● knows which methods are interested in which event by methods attributes– Methods are also first class object in Python

RyuAppRyuApp

queue

BRICKSEvent

Determin which RyuApp to deliverBased on class of event

dispatch

Events are read only becauseIt is shared with many RyuApps

Event sink

Event dispatcher

RyuAppRyuApp

queue

RyuAppRyuApp

queueEvent source

EventEvent source

Event source/sink

● source– Call methods of the event dispatcher to generate events

● sink– Subclass of class RyuApp

● Event dispatcher knows which methods are interested in which events

– Event handlers are invoked in its own thread context of each RyuApp

– To avoid race condition

– Direct queuing is also possible

RyuApp

queue

Event threadConsuming events

Event

Event request/reply

● request/reply messaging between RyuApps for easy programming

RyuApp

Event thread

RyuAppeventqueue

RyuApp

Event thread

RyuAppeventqueue

replyqueue

replyqueue

requestevent

replyevent

1. queue request event

3. process request

4. queue back result

5. wake up waiting event threadIf necessary

2. wait for reply ifsynchronous

OpenFlow parser and its event

● Only controller part is supported● OF events are created automatically on startup

– Introspection is used

● “Where EventOFPxxx is defined?” is FAQ

ofproto_v1_N_parser

OFPxxx EventOFPxxx

ofp_event

Connection to OpenFlow switch

● class OpenFlowController, class Datapath● Receiving loop and sending loop

OpenFlow switch

Receiving threadGenerates OFPEvents

Sending threadSerialize and sendOF packets

Send queueEventOFP

message

RyuDatapath

RyuAppRyuApp

queue

Event sink

OpenStack support

OpenStack Component

● Composed of Many component

● Neutron– Plugin architecture

– Able to support many network technology

service Openstack project

compute nova

storage swift(object)

glance(image)

cinder(block)

identity keystone

network neutron

... ...

Ryu Plugin for Neutron

● L2 isolation● Multi tenant w/o or w/ VLAN

– Mac address based

– VLAN

– GRE tunnel

Overview of Ryu plugin

Compute-node

Vif driver

CreateOVS port

Ryuagent OVS

OVSinitialization

OVSRyu

agent

OVSinitialization

L3 agent

Neutron Node

Neutron DB(Network ID, key)

Ryu server(Network ID, key)

Neutron API

Ryu node

r

Ryu REST

OpenFlow & OVSDB

Neutron server

Ryuplugin

Network node

OpenStack L2 isolation: logical view

VM VM VM VM VM VM

Tenant X Tenant Y

OpenStack L2 isolation: physical view

ComputeNode

ComputeNode

Compute/network

OVS

OVS

Tenant XVM

Tenant YVM

Tenant XVM

Tenant YVM

Tenant XVM

Tenant YVM

Tenant => GRE key

GRE tunnel

OpenFlow

Tenant XGRE key = M

Tenant YGRE key = N

L2 over L3 with GRE tunnel- Able to span over network segments (l2 segment can over multi data centers)- can coexists withConventional networktechnology

Set GRE keyDeliver packets based on GRE key

En/de-cupsel packetInto/from GRE packet

Ryunw-gw

nw-gw OVS

Table 0 Table 1 Table 2

Src table Tunnel out Local out

VM port

match action

in_portsrc mac

set_tunnelgoto table 1

in_port drop

match action

tunnel_iddst mac

output(tunnel)goto table 2

match action

tunnel_iddst mac

output(vm)tunnel_id goto table 2

tunnel_id dropTunnelport in_port

tunnel_idgoto table 2

in_port drop

OVS

VM1

VM2GRE tunnel

tunnelport

VM port

In port

Flow Table Usage

Nicira extension is used for GRE tunnel

GRE tunneling with OpenStack

● Composed of several RyuApps● Network tenant creation

– Assign GRE key

– Create gateway

● Guest VM instance creation– Create port

● Tenant ↔ key ↔ port relationship

– Set flow to the VM port

● Tunnel port management– Create/delete tunnel port

● Track physical compute node

– Set flow to the tunnel port

rest_quantum

gre_tunneltunnel_port_updater

quantum_adapter

REST

OVS

ovsdbOpenFlow

Neutron

Ryu

quantum: former name of neutron project

Ryu development

Development process

● Open development● Linux style● Discuss on Mailing List openly● Send/review patches on Mailing List

– git format-patch

– git send-patch

– No pull request on github

● Evolution– Ryu has evoleved from

very small program

http://dir.gmane.org/gmane.network.ryu.devel

Python experience through Ryu

Python

● Good things– Easy/fast to learn/use

– Many useful features● Dynamic language, first class everything, decorator, introspection...

– Especially introspection is very useful

– Decorator is handy

– Many useful libraries

● Bad things– Hard to debug

● debugger(pdb) is unstable● Debugger isn't compatible with eventlet

– Magic attributes(__xxx___)

– Many similar libraries: which to use?

AIO libraries

● Gevent → Eventlet● In general, monky-patching is ugly hack and

very fragile● Monkey patching of gevent/eventlet works

stably● Hit some issues and patches are proposed.● epoll is removed by monkey patching

Threading

● eventlet(or gevent) is cooperative threading– Needs special care for protection

● Starvation● Thread scheduling

– Different from native threading like pthread● Synchronization primitives

● Hard to debug– When debugger(pdb) tries to stop, the thread is switched to other thread

● Need to consider– Native vs green

– Giant Interpreter Lock

– What context to deliver events?

Performance

● Gevent performs slightly better than eventlet– But it's very slight.

– Needs other approach for more performance boost

● Pypy– Needs patch for eventlet

● Mulit process?

Network programming

● For IGMP with VRRP● Needs to read Cpython code or C-module code

– Much better than unsupported, though

Thank you

Questions?

http://osrg.github.com/ryu/