1

Continuous IntegrationFor

Business IntelligenceAnd Analytics

Alex MeadowsPrincipal ConsultantCSpring

2

Agenda

● About Me

● Waterfall vs Agile

● Continuous Integration

● Software Release Process

● References

Feel free to ask questions at any point in the presentation!

3

Alex Meadows

● Worked in various industries (manufacturing, social/email marketing, consulting, software, agriculture)

● Open source agilest

● Loves data :)

4

Let's Talk Releases...

●What's your team's release candance?●How many bug releases go out afterwards?●How much new development gets cut?

5

Waterfall Approach

Source: http://covalentmarketing.com/blog/2013/02/25/software-development-lifecycle-waterfall-vs-agile/

6

Agile Approach

Source: http://www.mountaingoatsoftware.com/system/asset/file/17/ScrumLargeLabelled.png

7

Continuous Integration

● Release Early, Release Often!

● Bundle work together into cohesive “builds”

– Server Configuration

– Code

– Database

– Tests, tests, and more tests

– Approval Process

8

Continuous Integration

● Agile XP principal since 1994

● Used in software everywhere*

– Google● Tests 100 million test cases every day● Entire code base changes 50% monthly

– Amazon, Yahoo, Microsoft, etc.

* Used in mature agile software shops

9

Software Release Process

Source: http://www.360logica.com/blog/2014/06/continuous-integration.html

10

Plan

● Break tasks out into manageable chunks

– Can that ETL code be broken down by:● Source/Target● Table● Process

– Can that dashboard be broken down by:● Object● Type of data/visual/interaction

● Business must prioritize, team must commit

11

BI Challenges

● Work is difficult to break apart

– Most tools are designed for waterfall

– Complex moving components

12

Code

● Team works and develops “artifacts”

– “Code” - Reports, ETL, Visualizations, Database changes, etc.

– Documentation

– Research

13

Build

● Developed work is

– Run through unit tests

– Peer reviewed

– “Compiled” - tested for readiness for QA

– Pushed to QA

14

BI Challenges

● Coordination across disparate tools, components

● Version control doesn't coordinate between tools

● Database version control is hard to introduce

15

Test

● Verify all functionality, edge cases, and user requirements

● 10% manual, 90% automation work

● Automated gate keeper (pass/fail the build)

16

BI Challenges

● Test tools not written for many components

– Database – exists, but minimalist

– ETL – database tools and etlTest (shameless plug ;)

– Visualization – Selenium

– Reports – mix of the above

– Analytics – mix of the above

17

Release/Deploy

● Coordinate and push to next environment

● Approvals act as gate keeping mechanisim

● Cycle starts back and the code phase as needed

18

BI Challenges

● Coordination is possible, but challenging

● Tools might not provide integration points

19

Operate/Monitor

● Release into Production

● Monitor processes, logs, etc for aberrations

● Bugs and other issues are added into backlog for future releases

20

References

21

Sources

● http://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/2013-03-24%20Continuous%20Integration%20at%20Google%20Scale.pdf

●