Download - Coscup 2013 : Continuous Integration on top of hadoop

Continuous Integration on top of Hadoop

Wisely Chen and Neal Lee

Saturday, August 3, 13

Agenda

• Who I am

• Problem

• Solution

• Demo

• Q&A


Who I am

• Wisely Chen ( [email protected] )

• Release manager of Yahoo![Taiwan] shopping and data team

• Loves to promote open source tech in Taiwan

• Hadoop Summit 2013 San Jose

• Ruby and Rails : Coscup 2006, Ubisunrise 2007, OSDC 2007

• Puppet : PHPConf 2012 , RubyConf 2012

• Release Practice : Webconf 2013, Coscup 2012


mailto:[email protected]

mailto:[email protected]

Who I am

• Neal Lee (@neal_lee)

• Data Engineer at Yahoo![Taiwan] data team

• Aims to build a easy to use self-service BI platform connecting to Hadoop.


EC Data Team拍賣/商城/購物中心

站台流量/點擊/使用者行為追蹤

Transactional data Tracking data

DataHighway

Data Warehouse/Data Mart

DataInfra BI

Platform

Report

RecommendationAPI

Machine Learning

Serve


Problem : Debug


Problem : Performance


Solution


Continuous Integration


Continuous Integration

• A software engineering practice • Maintain code repos

• Automate the build

• Make the build self-testing

• Everyone commit to the baseline everyday

• Every commit should be a build

• Test in a clone of production environment

• Make it easy to get the latest deliverables

• Everyone can see the result of latest build

• Automate deployment


We focus on• A software engineering practice

• Maintain code repos

• Automate the build

• Make the build self-testing

• Everyone commits to the baseline everyday

• Every commit should be a build

• Test in a clone of production environment

• Make it easy to get the latest deliverables

• Everyone can see the result of latest build

• Automate deployment


CI on Hadoop Flow

CodeUnitTest

PerformanceTest Deploy Doc Execution


One Click Deploy

CommitUnitTest



Toolset

CommitUnitTest


Vaidya

BASH


System diagram

CI Master

GitHub

Alpha

CI Slave

Beta Cluster

HadoopJobTracker

CI Slave Hadoop nodeHadoop

nodeHadoop

nodeHadoop

nodeSlave Node

Prod ClusterGateway


Unit Test

CommitUnitTest



PigUnit

• A simple xUnit framework

• No cluster set up is required in local mode

• Unit testing, regression testing, and rapid prototyping on the fly


Using PigUnit

• After

• Coding

• Write PigUnit test case

• Run local PigUnit test

• Push to cluster

• Run Pig on cluster

• Get right result !

• Before

• Coding

• Manual local test

• Push to cluster

• Run Pig on cluster

• Get right result !


Unit test is live doc

• Unit test is runnable live doc

• Pass test case and meet previous requirement


Flexible

• Pig can use PigUnit

• MapReduce can use MapUnit

• Hive can use hive_test


Performance Test

CommitUnitTest



Vaidya

• Rule based performance diagnosis of M/R jobs

• Extensible framework

• You can add your own rules

• Write complex rules using existing rules


Performance Test

Pig JobPig Job History

Vaidya

VaidyaRule

4

Pig Job Conf

NotifyUser

3

Performance result

Next CI Stage

1

1

22

2

5

1. Exec pig job with sampling data on beta server2. Vaidya read job history,conf,rule to check performance problem3. If ok, create performance result4. If job has performance issue, notify user5. Go to next CI stage

Sampling data

1


Vaidya Rule<Diagnos)cTest>

<Title><![CDATA[Balanaced Reduce Partitioning]]></Title>

<ClassName> <![CDATA[ org.apache.hadoop.vaidya.postexdiagnosis.tests.BalancedReducePar77oning ]]></ClassName>

<Descrip)on> <![CDATA[This rule tests as to how well the input to reduce tasks is balanced]]></Descrip)on>

<Importance><![CDATA[High]]></Importance>

<SuccessThreshold><![CDATA[0.20]]></SuccessThreshold>

<Prescrip)on><![CDATA[advice]]></Prescrip)on>

</Diagnos)cTest>

See if the reduce job is balance or not

Rule importance

Diagnose success threshold

Test Java Class


Deploy

CommitUnitTest



Deploy

• Deploy to production cluster

• Easy to rollback

• Create a git tag

• Auto doc generating

• Each release should map to a ticket

• Auto comment in Bugzilla


Auto comment in bugzilla

Repo url

Release Note

Issue statuschange


Auto create git tag

Release Note[Bug xxx] log....

Git Tag


DEMO


Demo

• Demo1 : Unit test fail

• Demo2 : Unit test success

• Demo3 : Check performance test

• Demo4 : Auto generate Doc

• Demo5 : Notify user


Demo


Conclusion

• CI will revolutionize your workflow

• CI will boost your productivity


Logic Debug

• Map/Reduce job oJen takes a lot of )me for execu)on

• Repeated Map/Reduce execu)on cost a lot of )me during logic debugging phase

• Need a way to find out logic problem before execu)on produc)on job

•Coding

ManualTest

Exec

Get Bug


Performance

• Map/Reduce performance is hard to es)mate before execu)on

• Production Grid computing resource is shared by all Yahoos

• Bad performance will affect other Yahoos Grid jobs

• Putting bad performance code on production grid is guilty

• We manually investigate the job performance before we actually execute it on production Grid

CodingManual

TestManual

investgateGet Bug


Download - Coscup 2013 : Continuous Integration on top of hadoop

Top Related