simplifying the openstack and kubernetes network stack with romana

Simplifying the network stack with Romana

Pani NetworksOpenStack / Kubernetes Meetup, Wellington, May 2016

romana.io Simplifying the network stack with Romana @romanaproject

Agenda

● “Cloud native”, why does it matter?● A better network for cloud native architectures● Demos


About us

● Team background:– Data center networks

– Low-level traffic management

● Created L2 overlay network startup– Bought by Cisco

● OpenStack networking● There's got to be a better way

– Time is right

What is 'cloud native'?


The past: Enterprise networking

● Full control● Applications need L2 and L3

– May need hard-wired IP addresses

– Broadcasts

● Servers are pets, not cattle: “Careful!”– VM migration

● Complex!


Cloud native applications

● Automate all the things!– Infrastructure as code

– Cattle, not pets: “Meh... just kill it.”

– Workloads come and go quickly

– Build for resiliance

● IP is all you need– No hardcoded IP addresses, discovery

– No special network requirements

– Basic IP connectivity

The problem


We have a mismatch

● Building cloud native applications…● … on top of enterprise networking

– SDN controllers use overlay L2 domains

– VLAN, VXLAN, OVS, etc.

● Complexity and brittleness– Lose benefits of simplicity

– Lose performance (encap, blinded hardware)

– Difficult to maintain and trouble shoot


The price you pay: Complexity

VXLAN Decap

VXLAN Decap

VXLAN Encap

VXLAN Encap

2 Top of Rack Round Trips

East/West Traffic

Per Instance Security


The price you pay: Performance

Router

Endpoint A Endpoint B

Router

L2 overlay A

L2 overlay B

VRouter


Why do we do this to ourselves?

● We don't need any L2 features● Except maybe traffic segmentation

– Multi tenancy

– Tiers and policies

The solution


Networking the way it was intended

● Use native L3 capabilities● No overlays● De-emphasize IP address ranges● Still provide segmentation, multi tenancy● Simple, clear and scalable network setup


Truly cloud native networking

● Project Romana● Open source● Apache 2.0 license● Mostly written in Go● Kubernetes and OpenStack


Truly cloud native networking

● Use only IP routing– No overlays

– All workload addresses are 'real'

– Simplicity!

● Use smart addressing– Encode tenant or segment in IP address

– Assign “virtual” addresses with host prefixes

– Massive (!) collapse of route table

● Routes are static– No route updates, no broadcasts for new endpoint


Romana Architecture

● On each host: Agent– Configures routes– Connects endpoint interfaces– Sets policy implementations

● Controller: Cooperating microservices– Each service with RESTful interface– Specialized for different tasks

● Environment: Different integration points– APIs, drivers for various parts of OpenStack or

Kubernetes


Romana Architecture

Host A Host B Host C

Agent Agent Agent

Tenant

Topology

IPAM

Root

Environment (OpenStack or Kubernetes)

Policy

Beautifully simple networking


Routing and route aggregation

Host A

eth0:192.168.8.11

Host B

eth0:192.168.8.22

Host C

eth0:192.168.8.33



Host A

eth0:192.168.8.11

romana-gw:10.0.0.1/16

Host B

eth0:192.168.8.22

romana-gw:10.1.0.1/16

Host C

eth0:192.168.8.33

romana-gw:10.2.0.1/16



Host A

eth0:192.168.8.11

romana-gw:10.0.0.1/16

10.0.0.5

10.0.1.7

10.0.1.19

10.0.5.3

Host B

eth0:192.168.8.22

romana-gw:10.1.0.1/16

10.1.3.52

10.1.9.2

Host C

eth0:192.168.8.33

romana-gw:10.2.0.1/16

10.2.0.16

10.2.3.81

10.2.4.6



Host A

eth0:192.168.8.11

romana-gw:10.0.0.1/16

10.0.0.5

10.0.1.7

10.0.1.19

10.0.5.3

Routes:10.1/16 → 192.168.8.2210.2/16 → 192.168.8.33

Host B

eth0:192.168.8.22

romana-gw:10.1.0.1/16

10.1.3.52

10.1.9.2

Routes:10.0/16 → 192.168.8.1110.2/16 → 192.168.8.33

Host C

eth0:192.168.8.33

romana-gw:10.2.0.1/16

10.2.0.16

10.2.3.81

10.2.4.6

Routes:10.0/16 → 192.168.8.1110.1/16 → 192.168.8.22


Larger network: L2 under ToR

Host B1

Host B2

Host B3

Host B4

Host A1

ToR A ToR B

spine network

192.168.1.200 192.168.2.200

192.168.1.1

Host A2

192.168.1.2

Host A3

192.168.1.3

Host A4

192.168.1.4

192.168.2.1

192.168.2.2

192.168.2.3

192.168.2.4

Rack A Rack B



Host B1

Host B2

Host B3

Host B4

Host A1

ToR A ToR B

spine network

192.168.1.200 192.168.2.200

192.168.1.1

Host A2

192.168.1.2

Host A3

192.168.1.3

Host A4

192.168.1.4

10.68/14

10.72/14

10.76/14

10.80/14

192.168.2.1

192.168.2.2

192.168.2.3

192.168.2.4

10.132/14

10.136/14

10.140/14

10.144/14

Rack A Rack B

10.64/10 10.128/10



Host B1

Host B2

Host B3

Host B4

Host A1

ToR A ToR B

spine network

192.168.1.200 192.168.2.200

192.168.1.1

Host A2

192.168.1.2

Host A3

192.168.1.3

Host A4

192.168.1.4

10.68/14

10.72/14

10.76/14

10.80/14

192.168.2.1

192.168.2.2

192.168.2.3

192.168.2.4

10.132/14

10.136/14

10.140/14

10.144/14

Rack A Rack B

10.64/10 10.128/10

Host A2 Routes

0.0.0.0 192.168.1.200→10.68/14 192.168.1.1→10.76/14 192.168.1.3→10.80/14 192.168.1.4→



Host B1

Host B2

Host B3

Host B4

Host A1

ToR A ToR B

spine network

192.168.1.200 192.168.2.200

192.168.1.1

Host A2

192.168.1.2

Host A3

192.168.1.3

Host A4

192.168.1.4

10.68/14

10.72/14

10.76/14

10.80/14

192.168.2.1

192.168.2.2

192.168.2.3

192.168.2.4

10.132/14

10.136/14

10.140/14

10.144/14

Rack A Rack B

10.64/10 10.128/10

ToR A Routes

10.128/10 192.168.2.200→10.68/14 192.168.1.1→10.72/14 192.168.1.2→10.76/14 192.168.1.3→10.80/14 192.168.1.4→

Host A2 Routes

0.0.0.0 192.168.1.200→10.68/14 192.168.1.1→10.76/14 192.168.1.3→10.80/14 192.168.1.4→


Larger network: Full L3

Host B1

Host B2

Host B3

Host B4

Host A1

ToR A ToR B

spine network

192.168.1.200 192.168.2.200

192.168.1.1

Host A2

192.168.1.2

Host A3

192.168.1.3

Host A4

192.168.1.4

10.68/14

10.72/14

10.76/14

10.80/14

192.168.2.1

192.168.2.2

192.168.2.3

192.168.2.4

10.132/14

10.136/14

10.140/14

10.144/14

Rack A Rack B

10.64/10 10.128/10

ToR A Routes

10.128/10 192.168.2.200→10.68/14 192.168.1.1→10.72/14 192.168.1.2→10.76/14 192.168.1.3→10.80/14 192.168.1.4→

Host Routes

0.0.0.0 192.168.1.200→

Scalable distributed firewalland

traffic policies


Romana: Traffic segmentation

● Tenant traffic separated:– Tenants don't get whole CIDR prefix or L2 domain

– But fully isolated from other tenants' traffic

● Tenants can define segments:– Like tiers, provide isolation and policies

● Use segment and tenant bits in IP addresses:– Apply policies (iptables) based on that

– Segments can stretch across hosts


Semantic and topological addressing

31

30

29

28

27

26

25

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9 8 7 6 5 4 3 2 1 0

0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1

10

Network prefix bitsThe network prefix. In this example, we are using the 10/8

address space.

6

Host ID Segment IDWe currently

store tenant ID in upper bits of segment ID.

4 67

Endpoint ID

Widths are configurable, don't have to use byte boundaries.


Semantic and topological addressing

31

30

29

28

27

26

25

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9 8 7 6 5 4 3 2 1 0

0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1

10

Network prefix bitsThe network prefix. In this example, we are using the 10/8

address space.

6

Host ID Segment IDWe currently

store tenant ID in upper bits of segment ID.

4 67

Endpoint ID

Widths are configurable, don't have to use byte boundaries.

Encode thetenant ID


Host BHost A

Allowing traffic within tenant

10.0.0.5 10.1.0.12

iptables:check src/dst addrs“tenant/segment bits

must match”

Src: 10.0.0.5Dst: 10.1.0.12

Same tenant/segment bits


Host BHost A

Isolating tenant traffic: Default

10.0.0.5 10.1.128.9

iptables:check src/dst addrs“tenant/segment bits

must match”

Src: 10.0.0.5Dst: 10.1.128.9

Different tenant/segment bits

Differenttenant


Host BHost A

Apply network policy between segments (full isolation as default)

10.0.0.5 10.1.1.9

iptables:Does policy chain

exist?Otherwise: DROP

Src: 10.0.0.5Dst: 10.1.1.9

Same tenant, different segment

policy-chain:From segment 0?Protocol TCP?To port 80?

Demo 1:

Kubernetes + Romana clusteron top of Catalyst OpenStack cloud


Demo 1 - Overview


Demo 1 - Overview

bar-1 bar-2foo

Jump host withpublic IP address


Demo 1 - Overview

bar-1 bar-2foo


Demo 1 - Overview

bar-1 bar-2foo

Install OpenStackcommand line tools


Demo 1 - Overview

bar-1 bar-2foo

$ neutron port-update \ e925b70e-031e-4ef7-a27c-583b4b775290 \ --allowed-address-pairs type=dict list=true \ mac_address=fa:16:3e:e1:df:59,ip_address=10.0.0.0/8


Demo 1 - Overview

bar-1 bar-2foo

$ git clone https://github.com/romana/romana$ cd romana/romana-install$ ./romana-setup -p static -i my-inventory -s kubernetes install


Demo 1 - Overview

bar-1 bar-2foo

Romanainstaller


Demo 1 - Overview

bar-1 bar-2foo

Kubernetes + Romana

Romana clusteraddress range:

10/8


Demo 1 - Overview

bar-1 bar-2foo

Kubernetes + Romana

Podswith containers.

Pods have RomanaIP addresses.


Demo 1 - What you will see

● Creation of pods● Network configuration● Application of network policies

Demo 2:

Mixing containers with legacy workloads


Demo 2 - Overview

bar-1 bar-2foo

Kubernetes + Romana


Demo 2 - Overview

bar-1 bar-2foo

Kubernetes + Romana

vm-workload

Legacy applicationin VM


Demo 2 - Overview

bar-1 bar-2foo

Kubernetes + Romana

vm-workload

Direct connection:- No gateway- No encap/decap- No NAT



● Creation of pods● Contact pod from VM● See the packet route

Demo 3:

Romana + Kubernetes clusteron top of Romana + OpenStack cluster


Baking layered cakes

● Kubernetes on OpenStack? Why?– On demand clusters

– Full tenant isolation

● Really nice with fully routed networking– No double encapsulation

– Logical, efficient packet forwarding

● Not all workloads fit into containers– Seamless connection between pods and VMs


Demo 3 - Overview

HW1 HW2 HW3 HW4


Demo 3 - Overview

HW1 HW2 HW3 HW4

$ ./romana-setup -p static -i hw-inventory -s devstack install


Demo 3 - Overview

HW1 HW2 HW3 HW4

OpenStack + Romana

Romana cluster 1address range:

10/8


Demo 3 - Overview

VM2 VM3VM1

HW1 HW2 HW3 HW4

OpenStack + Romana

OpenStack VMs

VMs haveIP addresses

ofRomana cluster 1


Demo 3 - Overview

VM2 VM3VM1

HW1 HW2 HW3 HW4

OpenStack + Romana

$ ./romana-setup -p static -i vm-inventory -s kubernetes install


Demo 3 - Overview

VM2 VM3

Kubernetes + Romana

VM1

HW1 HW2 HW3 HW4

OpenStack + Romana

Romana cluster 2address range:

172.16/12


Demo 3 - Overview

VM2 VM3

Kubernetes + Romana

VM1

HW1 HW2 HW3 HW4

OpenStack + Romana

Podswith containers.

Pods haveIP addresses

ofRomana cluster 2


OpenStack + Romana

Kubernetes + Romana

Demo 3 - Overview

VM2 VM3VM1

HW1 HW2 HW3 HW4


OpenStack + Romana

Kubernetes + Romana

Demo 3 - Overview

VM2 VM3VM1

HW1 HW2 HW3 HW4

Remember this one?

2 Top of Rack Round Trips

East/West Traffic

Per Instance Security

Without pure L3 networklayered clusters

would be even morecomplex.


OpenStack + Romana

Kubernetes + Romana

Demo 3 - Overview

VM2 VM3VM1

HW1 HW2 HW3 HW4

But with Romana, networkingeven in layered clusters becomes

really easy...



● Creation of pods● Pods and VMs with fully routable addresses● Ease of use showcase: Trouble shooting


Conclusion

● Cloud native architectures simplify things● Need cloud native networking to enjoy benefits● Romana:

– Cloud native without compromises

– Native network performance

– Mostly static config: Solid network

– Very easy to work with and understand

● Easy to try:– Simple installers for Kubernetes and OpenStack


Thank you!

● Romana Links– http://romana.io - Project home

– http://romana.io/blog - Blog

– https://github.com/romana/romana - Sources

● Contact– @romanaproject - Twitter

– [email protected] - Email

– https://romana.slack.com/ - Slack channel

http://romana.io/

http://romana.io/blog

https://github.com/romana/romana

mailto:[email protected]

https://romana.slack.com/

simplifying the openstack and kubernetes network stack with romana

Software