container communication on lattice #2
TRANSCRIPT
![Page 1: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/1.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
2016/2/10第31回PaaS勉強会@sinohara
Container Communication on lattice #2
![Page 2: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/2.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
⾃⼰紹介
@sinoharaNTTでNWの研究開発してます
PaaSたのしい
第28回PaaS勉強会でもLTさせてもらいました!
![Page 3: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/3.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
![Page 4: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/4.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
What is lattice?
• コンテナ管理システム(ルーティング、HM、ロギングetc)• Pivotal社のOSS• Cloud Foundryのdocker-image特化版
![Page 5: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/5.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
今⽇のトピック
latticeでコンテナ間通信
• コンテナ間の通信は未サポート• Dockerはコンテナ間通信の仕組みが沢⼭あるが・・
→latticeはdockerではなくgarden-linux上で動く
![Page 6: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/6.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
(前回)• 同⼀Cell(VM)内でコンテナ間通信は成功• Cell跨りは間に合わず・・
Container #1 Container #2 Container #3 Container #4
LATTICE ARCHITECTURE
Cell-1Cell-0
eth010.0.1.21
w0m6uagc6vub-0
w0m6uagc6vub-110.254.0.1
w0m6uagc6vuc-0
w0m6uagc6vuc-110.254.0.5
wb-0m6uf716cb1010.254.0.2
w0m6uf716lku-0
w0m6uf716lku-110.254.0.1
wb-0m6uf716cb2010.254.0.6
w0m6uf716lkv-0
w0m6uf716lkv-110.254.0.5
Eth010.0.1.193
54.92.69.17854.92.69.172
Region = ap-northeast-1
Virtual Machine Container Network Interfacedescription Network Tunnel
wb-0m6uagc6m51010.254.0.2
wb-0m6uagc6m50010.254.0.6
kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster
V0.2.5
Cell-brain
54.92.69.42
HTTP_GET 54.92.69.42.xip.io/index.php?ip=10.254.0.5
(php) http_get(http://10.254.0.5/phpinfo.php)
疎通確認方法
![Page 7: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/7.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
今⽇のトピック
latticeでコンテナ間通信
• コンテナ間の通信は未サポート• Dockerはコンテナ間通信の仕組みが沢⼭あるが・・
→latticeはdockerではなくgarden-linux上で動く• Cell間やってきました!
![Page 8: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/8.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
⽅針
• L2 over L3でつなぐ
• Ubuntuと相性の良いCanonical Fanを使⽤
• latticeのソース読むのはしんどいので動いてるモノを触って出来る範囲でトライ
![Page 9: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/9.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
FanNetworking
•同⼀のClass B-NW内でOverlay + Tunneling•コンテナIPのネゴシエーションがVM内で完結•Overlayは/8占有。Cell毎に/24を割り当て•コンテナのIPの共有は別途必要•現在v0.3.0
172.16.0.0/16172.16.0.0/16
172.16.3.5
172.16.23.37
10.3.5.x/24
10.23.37.x/24
VM#1
VM#2
Overlay-NW
Class-B NW
https://wiki.ubuntu.com/FanNetworking
https://launchpad.net/ubuntu/+source/ubuntu-fanhttps://insights.ubuntu.com/2015/06/24/ubuntu-fan-images/
Overlay-NW
![Page 10: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/10.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
流れを追って説明します
![Page 11: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/11.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
Container #1 Container #2
LATTICE ARCHITECTURE
Cell-0 eth0(cell0)10.0.1.21
Veth-host#1
Veth-guest#110.254.0.1
Veth-host#2
Veth-guest#210.254.0.5
Region = ap-northeast-1
Virtual Machine Container Network Interfacedescription Network Tunnel
Vbr#110.254.0.2
Vbr#210.254.0.6
tutum/apache-php tutum/apache-php
lattice=V0.6.0
Fan
Container #1 Container #2
Cell-0 eth0(cell0)10.0.1.21
Veth-host#1
Veth-guest#110.254.0.1
Veth-host#2
Veth-guest#210.254.0.5
Vbr#110.254.0.2
Vbr#210.254.0.6
tutum/apache-php tutum/apache-php
10.0.1.0/24
Step 0. Lattice(初期状態)
![Page 12: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/12.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
Container #1 Container #2
Cell-0 eth0(cell0)10.0.1.21
Veth-host#1
Veth-guest#110.254.0.1
Veth-host#2
Veth-guest#210.254.0.5
Region = ap-northeast-1
Vbr#110.254.0.2
Vbr#210.254.0.6
tutum/apache-php tutum/apache-php
lattice=V0.6.0
fan-br#1250.1.21.1
tunl0250.1.21.1
Container #1 Container #2
Cell-0 eth0(cell0)10.0.1.21
Veth-host#1
Veth-guest#110.254.0.1
Veth-host#2
Veth-guest#210.254.0.5
Vbr#110.254.0.2
Vbr#210.254.0.6
tutum/apache-php tutum/apache-php
fan-br#1250.1.21.1
tunl0250.1.21.1
10.0.1.0/24
L2 over L3 tunnel250.0.0.0/8
Step 1. FanをCellにセットアップ
• カーネル更新(3.19.0-41)• ubuntu-fanのinstall• Fan⽤Overlay-NW作成(250/8)
LATTICE ARCHITECTUREVirtual Machine Container Network Interfacedescription Network Tunnel
Fan
![Page 13: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/13.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
Container #1 Container #2
Cell-0 eth0(cell0)10.0.1.21
Veth-host#1
Veth-guest#110.254.0.1
Veth-host#2
Veth-guest#210.254.0.5
Region = ap-northeast-1
Vbr#110.254.0.2
Vbr#210.254.0.6
tutum/apache-php tutum/apache-php
lattice=V0.6.0
fan-br#1250.1.21.1
fan-veth-h#1 fan-veth-h#2
fan-veth-g#1250.1.21.2
fan-veth-g#2250.1.21.3
tunl0250.1.21.1
Container #1 Container #2
Cell-0 eth0(cell0)10.0.1.21
Veth-host#1
Veth-guest#110.254.0.1
Veth-host#2
Veth-guest#210.254.0.5
Vbr#110.254.0.2
Vbr#210.254.0.6
tutum/apache-php tutum/apache-php
fan-br#1250.1.21.1
fan-veth-h#1 fan-veth-h#2
fan-veth-g#1250.1.21.2
fan-veth-g#2250.1.21.3
tunl0250.1.21.1
10.0.1.0/24
L2 over L3 tunnel250.0.0.0/8
Step 2. 仮想NIC作成
LATTICE ARCHITECTUREVirtual Machine Container Network Interfacedescription Network Tunnel
Fan
• netnsで仮想NICをコンテナ内に繋ぎ込み• コンテナ内のルーティングテーブルに250/8の設定追加
![Page 14: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/14.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
Container #1 Container #2
Cell-0 eth0(cell0)10.0.1.21
Veth-host#1
Veth-guest#110.254.0.1
Veth-host#2
Veth-guest#210.254.0.5
Region = ap-northeast-1
Vbr#110.254.0.2
Vbr#210.254.0.6
tutum/apache-php tutum/apache-php
lattice=V0.6.0
fan-br#1250.1.21.1
fan-veth-h#1 fan-veth-h#2
fan-veth-g#1250.1.21.2
fan-veth-g#2250.1.21.3
tunl0250.1.21.1
Container #1 Container #2
Cell-0 eth0(cell0)10.0.1.21
Veth-host#1
Veth-guest#110.254.0.1
Veth-host#2
Veth-guest#210.254.0.5
Vbr#110.254.0.2
Vbr#210.254.0.6
tutum/apache-php tutum/apache-php
fan-br#1250.1.21.1
fan-veth-h#1 fan-veth-h#2
fan-veth-g#1250.1.21.2
fan-veth-g#2250.1.21.3
tunl0250.1.21.1
10.0.1.0/24
L2 over L3 tunnel250.0.0.0/8
完成!• IPinIP protocol(4)→許容
LATTICE ARCHITECTUREVirtual Machine Container Network Interfacedescription Network Tunnel
Fan
![Page 15: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/15.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
DEMO
riak clusterを動かしてみた
![Page 16: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/16.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
Cell-0
DEMO概要
Router(lattice-brain)
Cell-1
250.0.0.0/8
10.0.0.0/8
sinatra
riak
sinatra
riakclustering
• 250/8でriak cluster構築•外から250/8に直接アクセス不可→Sinatraでインタワークし、latticeのエンドポイントからriak clusterを直接操作
http://52.1.2.3.xip.io:8098/
http://250.1.2.2:8098http://250.1.3.4:8098 InterWork InterWork
tcp-route 8098:8080tcp-route 8098:8080
![Page 17: Container communication on lattice #2](https://reader033.vdocuments.mx/reader033/viewer/2022052705/586f86831a28ab54768b5533/html5/thumbnails/17.jpg)
Copyright@2016 NTT corp. All Rights Reserved.
まとめ
Lattice上でのコンテナ間通信(L2 over L3)の紹介• PoCレベル• netnsとかCellでの作業があります• /8のprivate address(10/8)をlatticeが占有し
ちゃってる問題(IANA未使⽤の250/8で代替)riak cluster on latticeのデモ
Latticeでクラスタリング、あなたなら何に使いますか?