continuous delivery for linux/windows/hadoop...beta cluster hadoop jobtracker jenkins slave hadoop...
Post on 14-Aug-2020
41 Views
Preview:
TRANSCRIPT
Continuous Delivery for Linux/Windows/Hadoop
Wisely Chen
Wednesday, October 23, 13
Agenda
• Background
• Problem
• Solution
• Demo
• Q & A
Wednesday, October 23, 13
Who I am
• Wisely Chen ( thegiive@gmail.com )
• Release manager of Yahoo! Taiwan shopping and data team
• Enjoys promoting open source tech
• Hadoop Summit 2013 San Jose
• Ruby and Rails (All in Taiwan): Coscup 2006, Ubisunrise 2007, OSDC 2007
• Release Practice (All in Taiwan): Webconf 2013, Coscup 2012,2013, PHPConf 2012 , RubyConf 2012
Wednesday, October 23, 13
Team
Sanching YuWindows
Neal LeeHadoop
Wednesday, October 23, 13
Yahoo! Taiwan Shopping
• Top B2C site in Taiwan
• Monday.com acquired in 2008
• 8 engineering teams
• 15K files changed monthly
• 100K lines of code changed monthly
Wednesday, October 23, 13
PlatformBackend
(Window)Frontend(Linux)
Data(Hadoop)
Catalog
Mobile
Member
Payment
Cash Flow
CMS
Logistic
Supplier
Machine Learning
CRM
Business Intelligence
Data Mart
Wednesday, October 23, 13
Problem
FrontendTeam
BackendTeam
DB
DataTeam
HadoopWebService API
Wednesday, October 23, 13
We need a multi-platform build and release solution
Wednesday, October 23, 13
Solution
• Create B&R process
• Build CI infrastructure
• Automate everything
Wednesday, October 23, 13
Solution
• Create B&R process
• Build CI infrastructure
• Automate everything
Wednesday, October 23, 13
DeployCheckPoint
PerfTest
IntegrationTest
UnitTest
Build and Release Process
CommitUATTest
Wednesday, October 23, 13
DeployCheckPoint
PerfTest
IntegrationTest
UnitTest
Hadoop Build and Release Process
CommitUATTest
Wednesday, October 23, 13
Original DifferenceFrontend Team Backend Team Data Team
Process Scrum Waterfall Waterfall
OS Linux Windows Linux
Language PHP/JAVA .NET Pig/MapReduce
Issue Tracking Bugzilla Custom System N/A
Repos Subversion VSS N/A
CI N/A N/A N/A
Int Test N/A N/A N/A
Release Management
Routine RE Another RE SDE
Wednesday, October 23, 13
Multiplatform Complexity Reduction
Frontend Team Backend Team Data Team
Process Scrum(not finish yet)Scrum(not finish yet)Scrum(not finish yet)
OS Linux Windows Linux
Language PHP/JAVA .NET Pig/MapReduce
Issue Tracking BugzillaBugzillaBugzilla
Repos Subversion(migrate to git)Subversion(migrate to git)Subversion(migrate to git)
CI JenkinsJenkinsJenkins
Int Test SeleniumSeleniumSelenium
Release Management
Delicated Release ManagerDelicated Release ManagerDelicated Release Manager
Wednesday, October 23, 13
Communication Cost
巨大Huge Large
Wednesday, October 23, 13
Solution
• Create B&R process
• Build CI infrastructure
• Automate everything
Wednesday, October 23, 13
WebSite CI InfrastructureJenkins Master
WindowsJenkins Slave
Farm
SeleniumServer Farm
LinuxJenkins Slave
Farm
Beta Server Farm
WindowsFarm
LinuxFarm
Staging Server Farm
WindowsFarm
LinuxFarm
Wednesday, October 23, 13
Hadoop CI Infrastructure
Jenkins Master
Alpha
Jenkins Slave
Beta Cluster
HadoopJobTracker
JenkinsSlave Hadoop
nodeHadoop
nodeHadoop
nodeHadoop
nodeSlave Node
Prod. ClusterGateway
PigServer
Wednesday, October 23, 13
DeployCheckPoint
PerfTest
IntegrationTest
UnitTest
Build and Release Process
CommitUATTest
Wednesday, October 23, 13
PigUnit
• A simple xUnit framework
• No cluster set up is required in local mode
• We change default PigUnit behavior
• All PigUnit tests run on single pig server
• Speed up PigUnit
Wednesday, October 23, 13
DeployCheckPoint
PerfTest
IntegrationTest
UnitTest
Build and Release Process
CommitUATTest
Wednesday, October 23, 13
Vaidya
• Rule based performance diagnosis of M/R jobs
• Extendable framework
• You can add your own rules
• Write complex rules using existing rules
Wednesday, October 23, 13
Vaidya Performance Test
Pig Job
Pig Job Conf
Sampling Data
Wednesday, October 23, 13
Vaidya Performance Test
Pig JobPig Job History
Pig Job Conf
Sampling Data
Wednesday, October 23, 13
Vaidya Performance Test
Pig JobPig Job History
Vaidya
VaidyaRulePig Job
Conf
Sampling Data
Wednesday, October 23, 13
Vaidya Performance Test
Pig JobPig Job History
Vaidya
VaidyaRulePig Job
Conf
NotifyUser
Sampling Data
Wednesday, October 23, 13
Vaidya Performance Test
Pig JobPig Job History
Vaidya
VaidyaRulePig Job
Conf
NotifyUser
Performance Result
Next CI Stage
Sampling Data
Wednesday, October 23, 13
<Diagnos)cTest>
<Title><![CDATA[Balanaced Reduce Partitioning]]></Title>
//////<ClassName>//////////<[CDATA[org.apache......BalancedReducePar))oning]]>///////</ClassName>///////<Descrip)on>///////////<![CDATA[This/rule/tests/........]]>///////</Descrip)on>///////<Importance><![CDATA[High]]></Importance>///////<SuccessThreshold><![CDATA[0.20]]></SuccessThreshold>///////<Prescrip)on><![CDATA[advice]]></Prescrip)on></Diagnos)cTest>
See/if/the/reduce/job/is/balanced/or/not/
Vaidya Rule Example
Rule/Importance
Diagnosis/success/threshold
Wednesday, October 23, 13
Solution
• Create B&R process
• Build CI infrastructure
• Automate everything
Wednesday, October 23, 13
DeployCheckPoint
PerfTest
IntegrationTest
UnitTest
Build and Release Process
CommitUATTest
Wednesday, October 23, 13
Before Deploy
• Check bug status
• Documention
• Create a git tag
• Auto doc generating on bugzilla
• Deploy to production cluster
• Change bug status
Wednesday, October 23, 13
Auto comment in bugzilla
Repos urlRelease Note
Issue statuschange
Wednesday, October 23, 13
Auto create git tag
Release Note[Bug xxx] log....
Git Tag
Wednesday, October 23, 13
Application Logic Monitor
0
25
50
75
100
2-4 4-6 6-8 8-10 10-12 12-14 14-16 16-18 18-20 20-22 22-24 24-2
Today Same day in last week Same day in last month
Monitor today’s sales revenue with baseline
Notify user of something wrong
SalesNum
Time
Wednesday, October 23, 13
Result
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
11Q4: Original
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
11Q4: Original
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
High Bug Rate(72%)
Low Throughput
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
11Q4: Original
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
12Q1 : Create B&R Process...
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
12Q1 : Create B&R Process...
巨大 bug rate drop
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
12Q1 : Create B&R Process...
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
12Q2 : Build CI Infrastructure...
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
12Q3 : Build CI Infrastructure...
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
12Q4 : Build CI Infrastructure...
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
12Q4 : Build CI Infrastructure...
Dramatic throughput increase
Bug rate down to 22%
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
12Q4 : Build CI Infrastructure...
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
13Q1 : Automate Everything...
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
13Q2 : Automate Everything...
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
13Q3: Automate Everything...
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
13Q3: Automate Everything...
Deploy time decrease
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Time
13Q3: Automate Everything...
Wednesday, October 23, 13
Hadoop CI Demo
• Demo1 : Unit test
• Demo2 : Performance test
• Demo3 : Deploy
Wednesday, October 23, 13
Wednesday, October 23, 13
Wednesday, October 23, 13
Conclusion
• This process can reduce bug rate
• This process can increase productivity
• Jenkins can fit in any platform
Wednesday, October 23, 13
謝謝大家, 祝你有美好的一天
Wednesday, October 23, 13
Appendix
Wednesday, October 23, 13
0
22.5
45
67.5
90
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
ReleaseBugRate ReleaseThroughput DeployCostTime
On Demand Bi-weekly Weekly
Wednesday, October 23, 13
Web Site CI Process
Backend3 Team
(Window)
Cash Flow
Frontend2 Team(Linux)
Catalog
Mobile
Member
Payment
Logistic
Supplier
CMS
Data2 Team
(Hadoop)MachineLearning
BusinessIntelligence
Data Mart
CRM
Wednesday, October 23, 13
Build and release processPHP CodeCommit
PHP Unit Test
DeployLinux Beta
BetaAuto Test
Deploy PHP Staging
StagingAuto Test
StagingUAT
Check Test Status
Release Meeting
Code Freeze
Release
.NET CodeCommit
.NET Unit Test
DeployWin Beta
Deploy .NET Staging
Deploy PHP Prod
Deploy .NET Prod
ProdAuto/UAT
DONE
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Cost Time
Throughput increased dramaticly
Bug rate down to 22%
Build CI infrastructure
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Cost Time
Significant bug rate drop
12Q1 : Create B&R Process
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Cost Time
Throughput increased dramaticly
Bug rate down to 22%
Build CI infrastructure
Wednesday, October 23, 13
0
25
50
75
100
11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3
Release Bug Rate Release Throughput Deploy Cost Time
Deploy cost time drop
Automate Everything
Wednesday, October 23, 13
Two B&R ProcessesFrontend
Team
BackendTeam
DataTeam
Website Process
Data Process
Wednesday, October 23, 13
Automate Everything
Check PointScript
Bugzilla
Source ReposLog
DeployScript
Notify UserScript
Bug List Yes
No
Check Bug Status
Auto Manual
JenkinsStyle/Static/Similarity Analysis
Wednesday, October 23, 13
top related