devops in action...devops in action yuta imai solutions architect, amazon data services japan, k.k....
TRANSCRIPT
![Page 1: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/1.jpg)
DevOps in action
Yuta Imai
Solutions Architect, Amazon Data Services Japan, K.K.
![Page 2: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/2.jpg)
今日のテーマ
• いままで見てきたお客様や海外事例のなかで「これはおもしろいなー、すごいなー」と感じたDevOpsっぽい話を紹介していきます。
![Page 3: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/3.jpg)
Agenda: Case Studies!
• NetFlix 1/2 : Service Oriented Architecture and
Origanization
• A Global Gaming : Application integrated deploy
• MMO RPG : Application integrated deploy
• NetFlix 2/2 : Monitor everything
![Page 4: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/4.jpg)
Case study: Netflix 1/2Service Oriented Architecture and
Origanization
![Page 5: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/5.jpg)
NetflixMeetup in Kyoto
http://connpass.com/event/9837/
![Page 6: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/6.jpg)
Netflix’s way
• Service Oriented Architecture
• Appropriate delegation
• No dependency among teams
http://techblog.netflix.com/2015/02/a-microscope-on-microservices.html
![Page 7: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/7.jpg)
Components in each Services
Service
Manager
DAO
Data
![Page 8: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/8.jpg)
Build
Source .war .deb .ami
SBT Gradle Aminator
![Page 9: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/9.jpg)
Deploy
1)
2)
3)
dev
mastertest
deploybuild
![Page 10: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/10.jpg)
Deploy with ASGARD
![Page 11: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/11.jpg)
Deploy with ASGARD
![Page 12: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/12.jpg)
More Netflix OSS..
http://netflix.github.io/
![Page 13: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/13.jpg)
You can try with ready made docker containers.
http://techblog.netflix.com/2014/11/zerotodocker-easy-way-to-evaluate.html
![Page 14: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/14.jpg)
Case study: A Global GamingApplication integrated deploy
![Page 15: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/15.jpg)
http://bit.ly/verizon-latencyhttp://bit.ly/superdata-latency
![Page 16: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/16.jpg)
VPC Subnet
VPC Subnet
Game API Stack
Availability Zone A Availability Zone B
VPC Subnet
VPC Subnet
Auto Scaling group
Amazon
CloudFront
WEB
VPC Subnet
WEB
JOBS
Queue
Cognito
SNS Mobile
Push
SES
S3: DLC,
Assets
![Page 17: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/17.jpg)
![Page 18: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/18.jpg)
100+ms 100+ms
100+ms100+ms
![Page 19: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/19.jpg)
Loosely Coupled Pods
Tokyo
Oregon
Frankfurt
Virginia
![Page 20: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/20.jpg)
Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
EC2
Game Flow
EC2
EC2
![Page 21: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/21.jpg)
Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
④ Connect to Server
⑤ Hack Apart Your Friends
⑥ Game Over
Game Flow
EC2
EC2
![Page 22: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/22.jpg)
Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
④ Connect to Server
⑤ Hack Apart Your Friends
⑥ Game Over
⑦ Write via HTTP API
Game Flow
EC2
EC2
![Page 23: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/23.jpg)
VPC Private Subnet
VPC Public Subnet
Game Server Pods
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
![Page 24: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/24.jpg)
VPC Private Subnet VPC Private Subnet
RabbitMQ + Elastic Load Balancing
Availability Zone A Availability Zone B
10.1.0.13 10.2.0.16
TCP 5672
rabbitmq-node1 rabbitmq-node2
TCP 4369
& 25672
![Page 25: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/25.jpg)
VPC Private Subnet
VPC Public Subnet
CloudFormation + Chef
Availability Zone A
GAME GAME GAME
Auto Scaling group
AWS CloudFormation:
Launch, Bootstrap
Chef: Configure, Install
![Page 26: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/26.jpg)
Where Are Servers?
Tokyo
Oregon
Frankfurt
Virginia
?
?
![Page 27: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/27.jpg)
VPC Subnet
Server Registration
Availability Zone A Availability Zone B
VPC Subnet
Auto Scaling group
WEB WEB
Oregon
Tokyo
VPC Subnet
Cleanup
EC2 API
Start / Stop
Instances
JOBS
![Page 28: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/28.jpg)
Global Services
China
Oregon
Frankfurt
LoginChina
Leaderboard
![Page 29: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/29.jpg)
Case study: MMO RPGApplication integrated deploy
![Page 30: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/30.jpg)
World
Island A Island B
Usual way of MMO implementation: Single big server
Server A
InteractionInteraction
![Page 31: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/31.jpg)
World
Island A Island B
Distribute users by geo location in game map
Server A Server B
no interaction
![Page 32: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/32.jpg)
World
Island A Island B
Distribute hot area with more game servers
no sync
Low density
AreaHigh density
Area
![Page 33: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/33.jpg)
User density differentiated by map’s popularity
Area Aless monsters
low experience points
poor item drop rate
Area Bmedium experience points
enough gold drops
avg. item drop rate
Area Cboss monsters
high chance of get items
![Page 34: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/34.jpg)
When an area’s population get higher
Area Cboss monsters
high chance of get items
Server C-1
- newly created
Server C-2
- newly created
Server C
- already existed
Geo size
3
![Page 35: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/35.jpg)
Case study: Netflix 2/2Monitor everything
![Page 36: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/36.jpg)
Monitor everything
• Let’s monitor everything– User behavior
– Application log
– Access log
• Then make it available!
![Page 37: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/37.jpg)
SPS : the Pulse of Netflix Streaming
• Starts Per Second
• They use SPS to
detect issues
• Deviation modeling
is important.
Static
Exponential Smoothing
Double Exponential Smoothing
http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-streaming.html
![Page 38: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/38.jpg)
A/B Testing
A
B
![Page 39: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/39.jpg)
A/B Testing
A
B
A
B
![Page 40: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/40.jpg)
A/B Testing
A
B
A
B
A
B
![Page 41: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/41.jpg)
A/B Testing
A
B
C
D
E
F
G
Test1
A
B
C
D
E
F
G
Test2
A
B
C
D
E
F
G
Test3
A
B
C
D
E
F
G
Test4
A
B
C
D
E
F
G
Test5
A
B
C
D
E
F
G
Test6
A
B
C
D
E
F
G
Test7
![Page 42: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/42.jpg)
Huge numbers in A/B testing
• 2,097,152 unique experience across 7 tests
• Hundreds of new A/B tests per year
• 2,105 Templates, 566 CSS, 685 JS
• 2.5 million unique css/js packages per push cycle
This cannot be handled manually!
![Page 43: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/43.jpg)
A/B tests should be automatically convergent
A
B
C
D
E
F
G
Test1
A
B
C
D
E
F
G
Test2
A
B
C
D
E
F
G
Test3
A
B
C
D
E
F
G
Test4
A
B
C
D
E
F
G
Test5
A
B
C
D
E
F
G
Test6
A
B
C
D
E
F
G
Test7
![Page 44: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/44.jpg)
• Monitor metrics
• Then– Deploy multiple versions of application
– Configuration
A/B tests should be automatically convergent: How?
![Page 45: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/45.jpg)
Atlas: Telemetry Platform
http://techblog.netflix.com/2014/12/introducing-atlas-netflixs-primary.html
![Page 46: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/46.jpg)
Let’s build data stream/pipeline
App1
App2
App3
App5
App4
Expose API to users for:
• Adding new data source
• Adding new application
![Page 47: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/47.jpg)
Conclusion
![Page 48: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ](https://reader035.vdocuments.mx/reader035/viewer/2022081613/5fbc450015d92a4f4e47ad8a/html5/thumbnails/48.jpg)
Conclusion
• Infra team should become ‘Tool team’
• Many things to go other than Test, Build, Deploy
• DevOps affect software architecture