© 2012 whamcloud, inc. whamcloud and quality chris gearing & mike stok software engineers...

© 2012 Whamcloud, Inc.

Whamcloud and Quality

• Chris Gearing & Mike StokSoftware EngineersWhamcloud, Inc

Lustre User GroupAustin TXApril 2012

Version 1.0

© 2012 Whamcloud, Inc.2

• Whamcloud’s View Of Quality

• A Year’s Progress

• The Year Ahead

• Maloo ‘Your Window onto Test’ – Mike Stok

Agenda

© 2012 Whamcloud, Inc.3

Whamcloud’s View Of Quality

© 2012 Whamcloud, Inc.4

Eric vs. Non Delivery

‟ The only thing I’m non-Linear about is

Whamcloud not delivering what it says

it will deliver ”

Whamcloud’s View Of Quality

Eric BartonCTO Whamcloud

© 2012 Whamcloud, Inc.5

• Whamcloud’s view of quality development

Whamcloud’s View Of Quality

StabilityQualityFeat

ures

Perf

orm

ance

Time

© 2012 Whamcloud, Inc.6

• Whamcloud is investing money, time and expertise in Quality

• …is continuing to develop tools and invest in infrastructure to enhance the Lustre community

• The whole of its engineering team is orientated around quality principles

Whamcloud’s View Of Quality

© 2012 Whamcloud, Inc.7

A Year’s Progress

© 2012 Whamcloud, Inc.8

• For landing testing we have carried out;– 687 days of landing testing– 2381 landing sessions– 2092758 individual tests– 2087607 passed

• Statistically– 1425 passes – 956 failures– 59.85% Session passed

Landing Test Performance

© 2012 Whamcloud, Inc.9

• This chart shows the percentage of landings that passed all tests since LUG 2011

Landing Test Performance

05/2

011

06/2

011

07/2

011

08/2

011

09/2

011

10/2

011

11/2

011

12/2

011

01/2

012

02/2

012

03/2

012

30.00%

35.00%

40.00%

45.00%

50.00%

55.00%

60.00%

65.00%

70.00%

75.00%

80.00%

Pass RateLinear (Pass Rate)

© 2012 Whamcloud, Inc.10

Distributed Test

Source code

repository

PublicSector

ISVs

Resellers

© 2012 Whamcloud, Inc.11

Distributed Test

Source code

repository

PublicSector

ISVs

PublicSectorResellers

Resellers

But!

© 2012 Whamcloud, Inc.12

• Juelich financed a fully equipped cluster• Used for testing all head releases• Specialises in failover testing

– 36TB of multi attached storage

• Good for performance regression tests– Private network with no contention so results are repeatable

• Also used for manual large Lun testing for 2.2 release

• I’d like to thank Juelich and in particular Frank Heckes for making this happen

Juelich Supercomputer Centre

© 2012 Whamcloud, Inc.13

• Indiana provided a 36 node cluster for the development and rollout of a backup test system

• Used for secondary 2.2 release testing• Enabled the transfer of the primary test cluster

from the West Coast to Colorado without any break in the Lustre test and landing processes

• Again I’d like to thank Indiana and Steve Simms for enabling this

Indiana University

© 2012 Whamcloud, Inc.14

The Year Ahead

© 2012 Whamcloud, Inc.15

Development vs. Landing Test• Which is landing test?• Which is development test?

05/2

011

06/2

011

07/2

011

08/2

011

09/2

011

10/2

011

11/2

011

12/2

011

01/2

012

02/2

012

03/2

012

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Pass RateLinear (Pass Rate)

© 2012 Whamcloud, Inc.16

• Distinct paths for development vs. landing– Similar process but distinct purpose

• Provide for maximum flexibility in development testing– Encourage systematic test to be part of the development process– Test during development does improve product quality

• Encourage peer review as the code is written– Earlier review leads to better code and more opportunity for

education

• Be auditable– Improvement requires knowledge of the past– Development test part of the landing collateral

• 100% pass rate for landing test– Developers should push tried and tested code for landing

Development vs. Landing Test

© 2012 Whamcloud, Inc.17

Development Test Cloud

© 2012 Whamcloud, Inc.18

The Goal

0%

20%

40%

60%

80%

100%

Development Test

FailPass

0%

20%

40%

60%

80%

100%

Landing Results

FailPass

© 2012 Whamcloud, Inc.19

• Whamcloud’s Quality Approach

• A Review Of The Last Year

• Our Plans For The Coming Year

Summary

© 2012 Whamcloud, Inc.20

Maloo ‘Your Window onto Test’

© 2012 Whamcloud, Inc.21 LUG Austin, TX - April 2012

• What is Maloo?• Recent changes to Maloo

– High points since last LUG

• Development priorities– Tool quality– Easy access to timely, accurate data

• Some planned features– User preferences– Automated scanning of incoming log files for “interesting” data

Maloo Agenda

© 2012 Whamcloud, Inc.22 LUG Austin, TX - April 2012

• A repository for lustre test result data– Collects the test results and the logs generated– Allows users to query the database– Contains about 1TB of log files

• https://maloo.whamcloud.com

What is Maloo?

© 2012 Whamcloud, Inc.LUG Austin, TX - April 201223

Recent changes to Maloo

© 2012 Whamcloud, Inc.24

Release report

LUG Austin, TX - April 2012

https://maloo.whamcloud.com/reports

© 2012 Whamcloud, Inc.25

Node utilization report

LUG Austin, TX - April 2012

https://maloo.whamcloud.com/reports/show_node_utilization_report

© 2012 Whamcloud, Inc.26

Internal changes

• Invisible work– Development practices– Testing– Packaging– Deployment– …

LUG Austin, TX - April 2012

© 2012 Whamcloud, Inc.LUG Austin, TX - April 201227

Development priorities

© 2012 Whamcloud, Inc.28 LUG Austin, TX - April 2012

• Tool quality– Mechanics of Maloo and its development– Visibility into the tool– Usability

Development priorities

© 2012 Whamcloud, Inc.29 LUG Austin, TX - April 2012

• Data quality– Accurate– Timely– Accessible

Development priorities

© 2012 Whamcloud, Inc.LUG Austin, TX - April 201230

New features

© 2012 Whamcloud, Inc.31 LUG Austin, TX - April 2012

Log file scanning

• Automates a tedious task

New features

© 2012 Whamcloud, Inc.

Maloo Footprints

© 2012 Whamcloud, Inc.33

Maloo Footprints

© 2012 Whamcloud, Inc.34

Maloo Footprints

© 2012 Whamcloud, Inc.35

Maloo Footprints

© 2012 Whamcloud, Inc.36 LUG Austin, TX - April 2012

User profiles

• One size needn’t fit all

New features

© 2012 Whamcloud, Inc.37 LUG Austin, TX - April 2012

What was covered• Changes in Maloo since last year• Our development priorities• A couple of the planned changes

Wrap up and questions

© 2012 Whamcloud, Inc.

Thank you very much

38