cloud foundry summit 2017
TRANSCRIPT
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
2017年6月21日
The Road to "JYU-BAI"- Adopting Cloud Foundry at Yahoo! JAPAN -
2017年6月20日
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
About me
2
Software Engineer
Manager
Yahoo! JAPAN
Yasuhiko Kubono
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .3
• Introducing Cloud Foundry into our
services
-Yasuhiko Kubono
• How do we Actually Operate
-Yusuke Kondo
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Introducing Cloud Foundry
into our services
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Agenda
5
• About Yahoo! JAPAN
• Why we use Cloud Foundry?
• Introducing Cloud Foundry into our services
• Case study
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
About
Yahoo! JAPAN
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Outline
Yahoo Japan Corporation(SoftBank Group)
Businesses: Internet Advertising - e-Commerce
- Member services - Others
Headquarters: Tokyo Japan
Founded: January 31, 1996
# of Employees: 5,826 (As of March 31, 2017)
7
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
# of Engineers & Designers
2,500
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
More than 100
Web Services
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
39,89M Active User IDs
1Month※2017年1-3月の平均
67,4B Page Views
Total requests
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Why we use
Cloud Foundry?
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Why we use Cloud Foundry?
Speed up development time
JYU-BAI increase productivity by 10 times
12
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Adoption Plan
13
2016
Initial introduction to
a few services
2017
Full-scale
implementation
2018
Expand
implementation
Here
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Introducing
Cloud Foundry
into our Services
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
More than 100
Web Services
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Programming Languages
16
C, Perl, C++,
PHP, Node.js, Java...
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Architecture Differs by Web Service
17
e.g. Travel tips e.g. Yahoo! Auction
Small-scale web services
list
cart
logic
logic
logic
Large-scale web services
Search
API Gateway
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Obstacles
18
Same architecture does not fit
in each web services
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Solutions
Enroll CF Coach in each web servicesaround 20 staff / 15 services
Coaches role:Promote cloud design methods that suit for each web services
19
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Role map
20
Core Team
CF Coach
For
Shopping
Shopping
engineer
Shopping
engineer
CF Coach
For
Auction
Auction
engineer
CF Coach
For
Media
Media engineer
…
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Case study
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Where we started from
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
List Necessary Functions
23
Service A Service B Service C Service D Service E Service F Service G
MySQL ● ● ● ●
Oracle ● ● ●
KVS
ObjectStrage ●
C/C++ ●
PHP ● ● ● ●
Node.js ● ● ● ● ●
Java ● ● ●
advertisement ● ● ● ● ●
beacon ● ●
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Challenges we encountered
Functions that can’t be used in the cloud because of
complicated dependency
Internal security polices are not suited for cloud
environment
Most of our web services were stateful design24
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
How we started
We selected one web service, and started
by preparing the necessary functions for
that service
Resolve issues each time they occur
25
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
So, which web servicedid we start with?
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Criteria for the web service
1. Simplicity
• Service with limited functions and external PF that can be used
2. Actively developed
• Web services that actively developedso that the effectiveness of introducing CFcan be measured
27
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
First target : CS tool
• Constructed with few servers in OpenStack environment
• WebServer: apache
• Apache Traffic Server (ATS) : Reverse Proxy
28
Characteristics• Language: PHP
• Framework: cakephp
• Uses REST API
• MySQL
Server Configuration
ATS
CS tool(apache)
API
ATS
MySQL
HTTPS
HTTP MySQL
CS tool(apache)
(our auction service)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Partial release using ATS (entry points)
Partially diverted entry point to CF apps using ATS:
• So that CF or OpenStack can be switched in entry points
29
ATS
CS tool(apache)
CS tool(apache)
API
HTTPS
HTTP
Some entry points
ATS
CS tool(CF)
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Lessons learned from the first target
• How to Implement in Production
• Development method based on OSS
• How to make service stateless on CF
30
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Adopting & Expanding to other services
31
Decide
target
Investigate
issues
Adopting
knowledge Solve issues
Knowhow
accumulated
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Next Presentation,
How do we Actually Operate
Photo by: Aflo
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Hello CF Summit 2017!
Yusuke Kondo or @konfoo
Responsible to...• operating Cloud Foundry & Concourse on IaaS• increasing engineers’ productivity by providing
tools and best practices around CI/CD
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Overview of Yahoo! JAPAN proprietary Infrastructure
More than four DCs in Japan
More than 90,000 VM running on OpenStack
34
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Cluster Spec
35
dev production
Load Balancer Software Hardware x2
IaaS Openstack Openstack
Hypervisor # 40 40
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Current Status (As of Jun. 9, 2017)
36
dev production
Cluster # 1 1
Cell # per Cluster 40 30
Org # 136 38
App Instance # approx. 2,000 approx. 400
Rps at peak time N/A approx. 2,000
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Future Plan (As of Jun. 9, 2017)
37
dev production
Cluster # 1 1 => 6
Cell # per Cluster 40 30 => 100
Org # 136 38
App Instance # approx. 2,000 approx. 400
Rps at peak time N/A approx. 2,000
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration
with
Backend Services
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
App Role Based ACL
MQ
Service
RDB
Object Storage
Key Value Store
Cache Service
FaaS
Existent Platforms
39
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration with Existent Platforms
• Cookie off-loading Route Service
• On-demand MySQL (OpenStack Trove API)
• Distributed pubsub service (Pulsar)
40
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Marketplace Dashboard
41
Goal: Providing all PFs in CF Marketplace
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Issues we faced
Platform ACL is based on IP address or hostname
=> Requesting for exceptional permission for accessing via IP Range with a limited term.
=> Migrating from Host-based ACL to Role-Based ACL in the long term
42
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration
with
Logs and Metrics
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
What we already have
In-house Monitoring & Alerting PF based on Apache Kafka, Hbase,
Splunk, an enterprise log analytics platform
44
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
User-side Logs and Metrics
45
PCF Cluster-1
VM
APP
VM
App
VM
VM
APP
App
Loggregator
Splunk
Monitoring PF
App AppApp
APPAPP
APPAPP
No action is needed for App developers
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
What we prepared
Firehose Nozzle and Relay Server
• Nozzle filters and formats the App logs streamed by Firehose
• Relay Server forwards the log stream to specific index
46
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Issue we faced
High log traffic. 900 lines per sec ! (as of Jun. 8, 2017)
=> Provided users with CF friendly logger
47
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Operator-side logs and Metrics
Splunk
• Platform logs such as CF component syslog
Prometheus
• Bosh metrics, VM metrics, Firehose metrics
• Emitting alerts to our smartphone
48
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .49
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration
with
other System
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration with package monitoring tool
51
Application Source Code
Dependent packages
Runtime Buildpack version
Vulnerable Package
Monitoring Tool
Track the buildpack version which the App are staged with and report outdated apps.
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration with package monitoring tool
52
Application Source Code
Dependent packages
Runtime
Scan package version
Scan whole source code
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Integration with Concourse
We use Concourse for
• deploying new Cloud Foundry release
• updating buildpacks
• syncing employee accounts with UAA
• backup database to object storage
• ...
53
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
Lessons learned
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .
We are still on the way to change mind
Changing your organization mind is the most essential part.
• Educate not only users, but also platform division where you belong.
• Work closely with your security paranoid team. Involve them to update the policy
55
Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .