ONOS:Open Network Operating System

Pankaj Berde

Open Networking Laboratory

ONOS: Distributed Network OS for Large Scale Networks

With Scale-out, High Availability and High Performance

Why? How? Results?

2

Example Large Networks: Service Provider Networks WAN core backbone

Multiprotocol Label Switching (MPLS) with Traffic Engineering (TE)

Metro networks Metro cores for access networks

Cellular access network LTE for a metro area

Wired access/aggregation Access network for homes DSL/Cable

Core

Cellular Metro

Wired Access

200-500 routers5K- 10K ports

20K-100K devices100K-10M+ ports

10K-50K routers2M-3M+ ports

10K-50K devices 100K- 1M+ ports

3

Key Performance Requirements for Core Backbone Network

Network

ONOS

Application ApplicationHigh Throughput ~500K - 1M path

setups/sec

Low Latency~10 - 100ms

4

~200GB - 1TBAnd will grow…

We decided to build a distributed NOSto meet these performance requirements

4

Global Network View/State ~3-6M Network State op/sec

Takeaways

Large networks require distributed network OS with scale-out, high performance and high availability.

Here distributed does not mean geo-distributed, but a single cluster of 5-25 co-located physical servers

5

Can One Build Distributed Network OS Stacking Open-Source Blocks?

Yes, one can. We did it. BUT …

6

Distributed Network OS withSimple Scale-Out

Controller Instance 2 Instance 3

Instance 1

Data plane

Network GraphGlobal network view

Control Application Control Application

Interconnection (e.g. 10G Ethernet)

Interconnection (e.g. 10G Ethernet)

7

Cassandra Distributed Key-Value Store

Titan Graph DB

Network Graph (Eventually consistent)

ONOS Architecture April 2013

Instance 1

OpenFlow Manager+

Instance 2

OpenFlow Manager+

Instance 3

OpenFlow Manager+

+ Floodlight Drivers

Control Application Control Application

Scale-out

Dis

trib

uted

Re

gist

ry(S

tron

gly

Cons

iste

nt)

Zook

eepe

r

Coor

dina

tion

Distributed Network

Graph/State

Applications Blueprints API

8

Host

Host

Host

Can One Build Distributed Network OS Stacking Open-Source Blocks?

Good for rapid prototypingBUT …

Lacks performance and performance visibility 9

Why Is It Hard To Get Performance Using Off-The-Shelf Software?

Bloated off-the-shelf open source components

Difficult to get visibility under the hood

Difficult to tweak for custom optimizations

10

Standard Ideas for Scalability and their Challenges

Partition data & processing so each instance can run independently

How do you partition optimally and specifically for NOS?

Replicate data to eliminate remote memory access

How do you keep replicas consistent as network state changes?

Make processing stateless to make failure recovery easy

How do you make state durable and where do you persist it?

11

Takeaways

Large networks require distributed network OS

Distributed network OS that can scale and have high performance with fault-tolerance poses many hard problems

12

ONOS: A Key SDN Building Block

13

Robust, High Performance Foundation

Cassandra Distributed Key-Value Store

Titan Graph DB

Network Graph (Eventually consistent)In-memory Network Graph (eventually consistent)

Host

Host

Host

Instance 1 Instance 2 Instance 3

OpenFlow Manager+

OpenFlow Manager+

OpenFlow Manager+

+ Floodlight Drivers

Control Application

Dis

trib

uted

Re

gist

ry(S

tron

gly

Cons

iste

nt)

Zook

eepe

r Control Application

Scale-out

Coor

dina

tion

Distributed Network

Graph/State

Applications ONOS Graph API

IndexingONOS Graph Abstraction

RAMCloudUltra-low latency distributed data store in DRAM

Even

t Noti

ficati

ons

Haz

elca

st

ONOS Today: Performance Focused

14

ONOS-Specific Data Model2013 Today: ONOS Optimized

ONOS Network Graph

Generic Graph

Generic Data Model ONOS-Specific Data Model

Generic Graph

Generic Data Model

ONOS Network Graph

ONOS Network Graph

• Data model optimized for ONOS• Direct access to data via known keys• Reduced read/write operations

• Network graph built on top of generic graph model

• Extra read/write operations due to indirection

15

How Did We Solve Hard Performance Challenges

Used lightweight open-source components which we can tweak like RAMCloud

Partitioning based on switches to exploit locality of data

Some state cached and replicated with state machine replication for sequential consistency

Used kernel-bypass for low-latency remote communication

16

Performance MetricsBasic network state update latency

Switch, port, and link discovery

Network event processing Rerouting of paths in response to link failures

Application path setup throughput

Important note:

These are relevant network OS performance metrics compared to what we measure using micro-benchmarks like cbench.

17

Performance Goals

Large Network

ONOS

Application Application

High Throughput: 500K – 1M setups/sec

Low Latency:10 – 100ms

Global Network View/State (200GB – 1TB+)

0.01

0.1

1

10

100

1000

Kernel Bypassw/ Infiniband

Latency [ms] (log scale)

(*)

(*)

Add Switch (w/ 4 ports)

Add Link

Network State Updates

0.0750.0990.150.2440.722

22.205

18

Network Event Processing

~40ms - ONOS latency~60ms – w/ FlowMods

~10k path reroutes/sec

Latency to reroute 1000 x 6-hop paths:

Path rerouting throughput:

19

Application Path Setup

~25ms – ONOS latency~60ms – w/ FlowMods

~20k path setups/sec

Latency to setup 1000 x 6-hop paths:

Path setup throughput:

20

ONOS Journey (so far…)

Oct, 2012 Apr, 2013 Dec, 2013 Today

Open-source building blocksHorizontally scalable

Highly availablePartition tolerant

Network graph abstraction

Proactive flow setupSDN-IP peering as an app

Demo app for service provider

New data modelIn-memory cache

Low-latency store (RAMCloud)Northbound API

Northbound events

10-100x Performance

NotificationsReactive Forwarding

Proxy ARP

CI and QA processImproved performance

Development VMIndustry standard tools/process

Proof of Concept Limited Release Performance

21

Start

Takeaways

Large networks require distributed network OS.

Distributed network OS poses many hard problems.

ONOS is on track to deliver a distributed OS with scale-out, high

availability and high performance.

22

Build your SDN using ONOS

ON.Lab will do an open source release with several use cases - Q4 2014

23

Team (We are hiring )

24

ONOS Demos at ONS 2014

ONOS on Internet2’s SDN network (Showcase)Key features on a real wide-area network

ONOS performance demo at ON.Lab boothLatest performance results w/ experts to answer questions!

25

Backup