reduce release cycle time: nine months to a week - nice!

BW5 Concurrent Session 11/7/2012 2:15 PM

"Reduce Release Cycle Time: Nine Months to a Week Is Jolly Good!"

Presented by:

Mathew Bissett UK Government

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

Mathew Bissett UK Government

Mathew Bissett has been working for Her Majesty’s Government for more than six years, having been recruited straight from University. Deployed into the integration and test team of his department’s main engineering group, Mathew demonstrated strong leadership qualities to quickly progress from test analyst through test team lead to test management. Currently the test manager responsible for the integration and testing of his area’s flagship system, he has driven through delivery process improvements to enable weekly deliveries. Mathew has delivered many talks within his government department and has presented at a British Computer Society Special Interest Group in Software Testing conference.

Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!

Better Software Conference East 2012 v0.13 1

1

Reduce Release Cycle TimeFrom Nine Months to a Week – Nice!

M h Bi

03/09/2012BSC East 2012 v0.13

Mathew BissettTest ManagerHer Majesty’s [email protected]

Introduction

03/09/2012BSC East 2012 v0.13

2

• I joined Her Majesty’s Government in late 2006 from University

• I worked my way up to Test Management within our Integration & Test Team

b ( )

© UK Crown Copyright 2012

• Have been (E2E) Test Manager for my Department’s flagship product since early 2011

• I also run a bit



Show of Hands

03/09/2012BSC East 2012 v0.13

3

• Would you call yourself a:▫ Developer▫ Tester▫ Project Manager▫ Business Analyst

h


▫ Other

BSC East 2012 v0.13

4

03/09/2012

Context SettingDelivery CyclesConclusions & Future




Context Setting

BSC East 2012 v0.13

5

03/09/2012


Processed Data Flow

Context Setting

BSC East 2012 v0.13

6

03/09/2012

CCom

mand &

Control




Context Setting

BSC East 2012 v0.13

7

03/09/2012


System Under Test

Context Setting

BSC East 2012 v0.13

8

03/09/2012

• “[Testing’s] role changes from verifying to prevention by concurrent engineering with both acceptance- and unit- test-driven development –and that blurs the distinction between test, requirements analysis, and design, so th t t ti i l t ti ”


that…testing is no longer testing.”[LV10]



BSC East 2012 v0.13

9

03/09/2012



BSC East 2012 v0.13

10

03/09/2012

9 Months




9 Months - Process

BSC East 2012 v0.13

11

03/09/2012

• Based on RUP & CMMI▫ Heavyweight documentation up front

• Implemented more like Waterfall▫ Defined by serial workflows

• User Acceptance left until the end


▫ Once product was nearing completion

9 Months - Process

BSC East 2012 v0.13

12

D eve lopm en t In teg ra tion I& T R e fe rence O perations

03/09/2012

O peva l

Q ua lity R eview(C he cklis t)

Ins ta ll R e lease

B uild V erifica tio nT est

D e ve lop cod eA p p ly (D E V ) lab e l

T est

B u ild ap p lica tio n

T est B u ild

Q uality R eview(C he cklis t)

Ins ta ll R e lease

B uild V erifica tio nT est

S ystem Te st

A ccep tance T est

1

1

1

2

2

Delive

ry Proc

ess

Delive

ry Proc

ess

Delive

ry Proc

ess

T est2


K e yD ev/I& T P ro cess

TT O P ro cess

C reate R e le aseA pp ly (IN T ) lab e l

Y e sN o N o

1

D efau lt P ath

Fau lt P ath

E labo ra tion /C o nstructio nTrans is tio n

Ite ra tive ly

T estin g C om ple te T estin g C om ple te

Q ua lity R evie w(C h e cklis t)

3

C rea te R e lease3

F it fo rR e lease

F it fo rR e lease Y e s

C reate R e le aseA p p ly (R E L ) la b el

2



9 Months – Process

03/09/2012BSC East 2012 v0.13

13

Planning


TestingDeveloping

9 Months - Pros

BSC East 2012 v0.13

14

03/09/2012

• Time-boxed process▫ Iterations were a fixed duration

• Code freezes▫ After which all functionality should have been

delivered




9 Months - Cons

BSC East 2012 v0.13

15

03/09/2012

• Feature-bound delivery▫ More iterations were added until the product was

fit for delivery - resulting in endless Transition phases

• Subsystems were produced independently▫ It was not possible to test major integration risks


p j guntil late in the delivery cycle

▫ Manual regression testing meant that integrating and re-testing fixes was very costly

• Users not consulted until the product neared completion

BSC East 2012 v0.13

16

03/09/2012

6 Weeks – The Heartbeat




9 Months – 6 Weeks

BSC East 2012 v0.13

17

03/09/2012

• Intermediary ‘Stepping Stones’ process▫ Heavy Systems Engineering involvement▫ Defined integration points

• Challenge lead by Senior User▫ Changed risk appetite in exchange for a delivery

i hi k


within 22 weeks▫ Successfully met challenge after 20 weeks but

wasted final 2 weeks

6 Weeks - Process

BSC East 2012 v0.13

18

03/09/2012

• Backlog driven & ‘light-weight RUP’ based▫ Customer lead prioritisation▫ Planned for Product rather than Subsystems

• Delivery scope was advertised up front on a Wiki▫ Based on customer priority


• Introduced concurrent test phases▫ Start of automated regression testing▫ Continuous Subsystem Integration▫ User Trials undertaken in parallel to development



6 Weeks - Process

BSC East 2012 v0.13

19

03/09/2012


6 Weeks – Process

03/09/2012BSC East 2012 v0.13

20

Planning


TestingDeveloping



6 Weeks - Pros

BSC East 2012 v0.13

21

03/09/2012

• Time-boxed process▫ Delivery cycles were fixed to 6 week

• Integration risks assessed faster▫ Invested in Subsystem and Product automated

regression testingi l h l d


• User Trials helped User Acceptance▫ Prior-proven business benefit

6 Weeks - Cons

BSC East 2012 v0.13

22

03/09/2012

• Feature-bound delivery▫ Great pressure to deliver desired functionality

• Requirements/Designs were often not produced▫ Lack of upfront Systems Engineering effort

• Subsystems still produced independently


▫ Usually only managed one or two integration points

▫ Often made to work by I&T in the final week



BSC East 2012 v0.13

23

03/09/2012

1 Week – The Pulse


6 Weeks – 1 Week

BSC East 2012 v0.13

24

03/09/2012

• Challenge lead by Integration & Test Team▫ Needed to assess integration risks earlier▫ Enabled by previous investment in automation▫ Intention was to remove the pressure from the

end of the delivery cycle




1 Week - Process

BSC East 2012 v0.13

25

03/09/2012

• Customer Backlog and Integration Logic driven▫ Technical lead prioritisation

• Delivery scope was more flexible▫ Reduced the planning horizon to ~3 weeks

• Removed team independence by introducing CI


▫ Subsystem integration responsibility moved into Development Teams

▫ Enforced by Quality Gates

1 Week - Process

BSC East 2012 v0.13

26

03/09/2012

Weekly Regression

Weekly E2E & UATDaily CIE Regression

TestReportsCDD’s

Build Logic


Weekly DQ Regression

Builds& Docs

Builds

VDDs

TestReports



1 Week - Process

BSC East 2012 v0.13

27

03/09/2012

WednesdayWednesdayWednesdayWednesday ThursdayThursday FridayFriday Sat/SunSat/Sun MondayMonday TuesdayTuesday ThursdayThursday FridayFriday Sat/SunSat/Sun MondayMonday TuesdayTuesday

E2EE2E

CIECIE

TRR TRR TRR TRR TRR TRR TRR TRR TRR TRR

CIE CIE CIE CIE CIE CIE CIE CIE CIE

E2E Agility

E2E Regression

TRR

E2E Agility

E2E Regression

TRR

CIE


Transition Transition to to DeploymentDeployment

DQ Regression DQ Regression

Build

TRR

VDDs

Test Reports

Build

VDD

Test Reports

TRR

GOPRM PRM GO

1 Week - Process

BSC East 2012 v0.13

28

WedWedWedWed ThuThu FriFri Sat/SunSat/Sun MonMon TueTue ThuThu FriFri Sat/SunSat/Sun MonMon TueTueMonMon TueTue WedWed ThuThu

03/09/2012

PlanningPlanning

E2EE2E

CIECIE

22 22 22 22

CIECIE CIECIE CIECIE

E2E AgilityE2E Agility

33

11

44

CIECIE CIECIE

33

Test RptTest Rpt22 22

- Pulse Planning Meeting (PPM)- Pulse Planning Meeting (PPM)11

C

D

A

B B

C

G

HE E


Supplier #1Supplier #1

Supplier #2Supplier #2

E2E RegressionE2E Regression

DQ RegressionDQ Regression

- Pulse Planning Meeting (PPM)- Daily Test Readiness Review (TRR) (1200hrs)- Product Handover Meeting (1230hrs)- Pulse Go/No Go meeting (1630hrs)

- Create RTC Tickets- Update RTC Ticket- Build and Test- Test results analysis and report to TRR- Possible new builds following TRR- Continuous analysis of test results and report defects- Produce Test Report (1400hrs)- Review Problem Tickets (1500hrs)

- Pulse Planning Meeting (PPM)- Daily Test Readiness Review (TRR) (1200hrs)- Product Handover Meeting (1230hrs)- Pulse Go/No Go meeting (1630hrs)

- Create RTC Tickets- Update RTC Ticket- Build and Test- Test results analysis and report to TRR- Possible new builds following TRR- Continuous analysis of test results and report defects- Produce Test Report (1400hrs)- Review Problem Tickets (1500hrs)

11

22

33

44

ABCDEFG

F

F

H



1 Week – Process

03/09/2012BSC East 2012 v0.13

29

Planning


TestingDeveloping

1 Week - Pros

BSC East 2012 v0.13

30

03/09/2012

• Non-feature bound time-boxed process▫ Delivery cycles were fixed to 1 week and delivered

only what was ready▫ Regular release candidates based on Product

integration logic• Risk identified up front


p▫ Testing defined in the take-on-work process▫ Enforced by quality gates between delivery phases

• User Acceptance became an input into delivery quality gate



1 Week – Cons

BSC East 2012 v0.13

31

03/09/2012

• Software is constantly changing▫ Manual continuous integration is expensive▫ Manual configuration management is hard across

many integration environments▫ Delivery process required improvement to meet 1

k d li i


week deliveries

BSC East 2012 v0.13

32

03/09/2012





Conclusions

BSC East 2012 v0.13

33

03/09/2012

• Identify Risks Early• Concurrency• Planning Horizon


Identify Risks Early

BSC East 2012 v0.13

34

03/09/2012

• Introduce risk identification into your take on work process

• Have many risk assessment opportunities and assess as soon as practical▫ Share mitigation of risk between teams


• Introduce Quality Gates and enforce them



Identify Risks Early

BSC East 2012 v0.13

35

03/09/2012

Phase Who Environment Exit CriteriaTest Tool Development Automation Team Tool Development Env. Test tools available

Development Testing Development Team Development Env. x% code coverage

System Testing Test Team System Test Env. x% feature coverage

Feature Trials Project Team Trial Environment Initial User validationInitial E2E verification

Quality Gate #1


System Integration Testing

Test Team System IntegrationEnvironment

Verify integration to delivery baseline

Quality Gate #2

End-to-End Testing Test Team End-to-End Test Environment

End-to-End data flow verification successful

User Acceptance Testing

Test Team End-to-End Test Environment

Final User validation successful

Concurrency

BSC East 2012 v0.13

36

03/09/2012

• Introduce concurrency where practical▫ Which test phases can be performed in parallel?▫ Use different test environments for different

purposes• Give yourself more opportunities to succeed

l l did


▫ Regular release candidates• Enforce process when pulling through bugfixes



Concurrency

BSC East 2012 v0.13

37

03/09/2012

18.5

101214161820

Time to Market

Weekly Delivery


1

6

2.5

0

7

2

02468

P12 P13 P14 P15 P16 P17 HB21+

y y6-Weekly Delivery

Concurrency

BSC East 2012 v0.13

38

03/09/2012

11.5

8 5

13.5

9.5

12.5 12.511

23.5

14.5

17.5 17.5

8

2018.5

10

15

20

25Delivery Capacity


5

8.5 8

0

5

10



Planning Horizon

BSC East 2012 v0.13

39

03/09/2012

• Only plan as far as practical▫ Let the technical community define the best path

to meet your customers’ requirements▫ Reduce your planning horizon to save re-work

• Advertise your planning horizon to your


customers▫ Negotiate flexibility with your customers

• Be flexible to unexpected change▫ Embrace the opportunity to deliver fixes quickly

BSC East 2012 v0.13

40

03/09/2012

© UK Crown Copyright 2012Known Predicted Make Believe



Future

BSC East 2012 v0.13

41

03/09/2012

• Automated CI & Build tool• Replace E2E Environment with OpEval

Environment• Empower Users to drive & test system and

enable direct feedback loops


Future

BSC East 2012 v0.13

42

03/09/2012

Volumetric

RegressionFuRe

Op Evaluation

OpRelease

Build & Test

Queue

Dev Component TestDev Component

Test Package Sys TestComponent Int. Test

Dev Component TestDev Component

Test Package Sys TestComponent Int. Test

Dev Component Test Package Sys Test Static

ReviewComponent Int. Test


Dev Component Test

Static Review

unctional egression

Supplier Capability Team

Platform Team CI Tools Delivery

Team



BSC East 2012 v0.13

43

03/09/2012



References

BSC East 2012 v0.13

44

03/09/2012

• Bibliography▫ Ambler10a

Ambler, S., 2010. Agile Testing and Quality Strategies: Discipline Over Rhetoric

▫ Ambler10bAmbler S 2010 Introduction to Disciplined Agile


Ambler, S., 2010. Introduction to Disciplined Agile Delivery

▫ CG09Crispin, L., Gregory, J., 2009. Agile Testing: A Practical Guide for Testers and Agile Teams



References

BSC East 2012 v0.13

45

03/09/2012

• Bibliography▫ Leffingwell10

Leffingwell, D., 2010. Scaling Software Agility: Rearchitecting Enterprise Class systems – An Agile Enterprise Trifecta

▫ LV10


▫ LV10Larman, C., Vodde, B., 2010. Practices for Scaling Lean & Agile Development: Large, Multisite, and Offshore Product Development with Large-Scale Scrum

BSC East 2012 v0.13

46

03/09/2012


Mathew BissettTest Manager

Her Majesty’s [email protected]

reduce release cycle time: nine months to a week - nice!

Technology