Download - Cloud Foundry Summit 2017
![Page 1: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/1.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
2017年6月21日
The Road to "JYU-BAI"- Adopting Cloud Foundry at Yahoo! JAPAN -
2017年6月20日
![Page 2: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/2.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
About me
2
Software Engineer
Manager
Yahoo! JAPAN
Yasuhiko Kubono
![Page 3: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/3.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .3
• Introducing Cloud Foundry into our
services
-Yasuhiko Kubono
• How do we Actually Operate
-Yusuke Kondo
![Page 4: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/4.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Introducing Cloud Foundry
into our services
![Page 5: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/5.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Agenda
5
• About Yahoo! JAPAN
• Why we use Cloud Foundry?
• Introducing Cloud Foundry into our services
• Case study
![Page 6: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/6.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
About
Yahoo! JAPAN
![Page 7: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/7.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Outline
Yahoo Japan Corporation(SoftBank Group)
Businesses: Internet Advertising - e-Commerce
- Member services - Others
Headquarters: Tokyo Japan
Founded: January 31, 1996
# of Employees: 5,826 (As of March 31, 2017)
7
![Page 8: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/8.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
# of Engineers & Designers
2,500
![Page 9: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/9.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
More than 100
Web Services
![Page 10: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/10.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
39,89M Active User IDs
1Month※2017年1-3月の平均
67,4B Page Views
Total requests
![Page 11: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/11.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Why we use
Cloud Foundry?
![Page 12: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/12.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Why we use Cloud Foundry?
Speed up development time
JYU-BAI increase productivity by 10 times
12
![Page 13: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/13.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Adoption Plan
13
2016
Initial introduction to
a few services
2017
Full-scale
implementation
2018
Expand
implementation
Here
![Page 14: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/14.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Introducing
Cloud Foundry
into our Services
![Page 15: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/15.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
More than 100
Web Services
![Page 16: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/16.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Programming Languages
16
C, Perl, C++,
PHP, Node.js, Java...
![Page 17: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/17.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Architecture Differs by Web Service
17
e.g. Travel tips e.g. Yahoo! Auction
Small-scale web services
list
cart
logic
logic
logic
Large-scale web services
Search
API Gateway
![Page 18: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/18.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Obstacles
18
Same architecture does not fit
in each web services
![Page 19: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/19.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Solutions
Enroll CF Coach in each web servicesaround 20 staff / 15 services
Coaches role:Promote cloud design methods that suit for each web services
19
![Page 20: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/20.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Role map
20
Core Team
CF Coach
For
Shopping
Shopping
engineer
Shopping
engineer
CF Coach
For
Auction
Auction
engineer
CF Coach
For
Media
Media engineer
…
![Page 21: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/21.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Case study
![Page 22: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/22.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Where we started from
![Page 23: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/23.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
List Necessary Functions
23
Service A Service B Service C Service D Service E Service F Service G
MySQL ● ● ● ●
Oracle ● ● ●
KVS
ObjectStrage ●
C/C++ ●
PHP ● ● ● ●
Node.js ● ● ● ● ●
Java ● ● ●
advertisement ● ● ● ● ●
beacon ● ●
![Page 24: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/24.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Challenges we encountered
Functions that can’t be used in the cloud because of
complicated dependency
Internal security polices are not suited for cloud
environment
Most of our web services were stateful design24
![Page 25: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/25.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
How we started
We selected one web service, and started
by preparing the necessary functions for
that service
Resolve issues each time they occur
25
![Page 26: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/26.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
So, which web servicedid we start with?
![Page 27: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/27.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Criteria for the web service
1. Simplicity
• Service with limited functions and external PF that can be used
2. Actively developed
• Web services that actively developedso that the effectiveness of introducing CFcan be measured
27
![Page 28: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/28.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
First target : CS tool
• Constructed with few servers in OpenStack environment
• WebServer: apache
• Apache Traffic Server (ATS) : Reverse Proxy
28
Characteristics• Language: PHP
• Framework: cakephp
• Uses REST API
• MySQL
Server Configuration
ATS
CS tool(apache)
API
ATS
MySQL
HTTPS
HTTP MySQL
CS tool(apache)
(our auction service)
![Page 29: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/29.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Partial release using ATS (entry points)
Partially diverted entry point to CF apps using ATS:
• So that CF or OpenStack can be switched in entry points
29
ATS
CS tool(apache)
CS tool(apache)
API
HTTPS
HTTP
Some entry points
ATS
CS tool(CF)
![Page 30: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/30.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Lessons learned from the first target
• How to Implement in Production
• Development method based on OSS
• How to make service stateless on CF
30
![Page 31: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/31.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Adopting & Expanding to other services
31
Decide
target
Investigate
issues
Adopting
knowledge Solve issues
Knowhow
accumulated
![Page 32: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/32.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Next Presentation,
How do we Actually Operate
Photo by: Aflo
![Page 33: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/33.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Hello CF Summit 2017!
Yusuke Kondo or @konfoo
Responsible to...• operating Cloud Foundry & Concourse on IaaS• increasing engineers’ productivity by providing
tools and best practices around CI/CD
![Page 34: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/34.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Overview of Yahoo! JAPAN proprietary Infrastructure
More than four DCs in Japan
More than 90,000 VM running on OpenStack
34
![Page 35: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/35.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Cluster Spec
35
dev production
Load Balancer Software Hardware x2
IaaS Openstack Openstack
Hypervisor # 40 40
![Page 36: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/36.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Current Status (As of Jun. 9, 2017)
36
dev production
Cluster # 1 1
Cell # per Cluster 40 30
Org # 136 38
App Instance # approx. 2,000 approx. 400
Rps at peak time N/A approx. 2,000
![Page 37: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/37.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Future Plan (As of Jun. 9, 2017)
37
dev production
Cluster # 1 1 => 6
Cell # per Cluster 40 30 => 100
Org # 136 38
App Instance # approx. 2,000 approx. 400
Rps at peak time N/A approx. 2,000
![Page 38: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/38.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration
with
Backend Services
![Page 39: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/39.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
App Role Based ACL
MQ
Service
RDB
Object Storage
Key Value Store
Cache Service
FaaS
Existent Platforms
39
![Page 40: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/40.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration with Existent Platforms
• Cookie off-loading Route Service
• On-demand MySQL (OpenStack Trove API)
• Distributed pubsub service (Pulsar)
40
![Page 41: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/41.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Marketplace Dashboard
41
Goal: Providing all PFs in CF Marketplace
![Page 42: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/42.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Issues we faced
Platform ACL is based on IP address or hostname
=> Requesting for exceptional permission for accessing via IP Range with a limited term.
=> Migrating from Host-based ACL to Role-Based ACL in the long term
42
![Page 43: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/43.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration
with
Logs and Metrics
![Page 44: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/44.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
What we already have
In-house Monitoring & Alerting PF based on Apache Kafka, Hbase,
Splunk, an enterprise log analytics platform
44
![Page 45: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/45.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
User-side Logs and Metrics
45
PCF Cluster-1
VM
APP
VM
App
VM
VM
APP
App
Loggregator
Splunk
Monitoring PF
App AppApp
APPAPP
APPAPP
No action is needed for App developers
![Page 46: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/46.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
What we prepared
Firehose Nozzle and Relay Server
• Nozzle filters and formats the App logs streamed by Firehose
• Relay Server forwards the log stream to specific index
46
![Page 47: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/47.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Issue we faced
High log traffic. 900 lines per sec ! (as of Jun. 8, 2017)
=> Provided users with CF friendly logger
47
![Page 48: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/48.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Operator-side logs and Metrics
Splunk
• Platform logs such as CF component syslog
Prometheus
• Bosh metrics, VM metrics, Firehose metrics
• Emitting alerts to our smartphone
48
![Page 49: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/49.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .49
![Page 50: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/50.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration
with
other System
![Page 51: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/51.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration with package monitoring tool
51
Application Source Code
Dependent packages
Runtime Buildpack version
Vulnerable Package
Monitoring Tool
Track the buildpack version which the App are staged with and report outdated apps.
![Page 52: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/52.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration with package monitoring tool
52
Application Source Code
Dependent packages
Runtime
Scan package version
Scan whole source code
![Page 53: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/53.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration with Concourse
We use Concourse for
• deploying new Cloud Foundry release
• updating buildpacks
• syncing employee accounts with UAA
• backup database to object storage
• ...
53
![Page 54: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/54.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Lessons learned
![Page 55: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/55.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
We are still on the way to change mind
Changing your organization mind is the most essential part.
• Educate not only users, but also platform division where you belong.
• Work closely with your security paranoid team. Involve them to update the policy
55
![Page 56: Cloud Foundry Summit 2017](https://reader034.vdocuments.mx/reader034/viewer/2022052117/5a647b077f8b9a57568b4829/html5/thumbnails/56.jpg)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .