Setting test process in AgileSetting test process in Agile

Alex Yermakovich

Part 1: understanding the game

Which are Agile benefits?Which are Agile benefits?

Flexibility/Iterative Speed to Market Encourages Change Release software early and often Builds in quality throughout the development process

What’s Agile for testing?What’s Agile for testing?

Minimal Requirements Minimal Up front Planning Continuous Integration and Regression Testing Each

Iteration Test Automation

Part 2: approach

What is actually Agile?What is actually Agile?

Agile is number of practices!

Agile testing practicesAgile testing practices

Continuous Integration- nightly builds Test Driven Development - test first and Unit tests

coverage Refactoring Coding Standards - code reviews …

Rules for the game: baselineRules for the game: baseline

Every one is responsible for the quality Product/project manager is quality oriented: understand

quality requirements Quality is based on Vision rather then on requirements High level communications and integration of SQE and Dev

team: SQE code and Dev test

Rules for the game: planningRules for the game: planning

SQE is independent and plans for itself but taking priorities from PM and Dev team

Engineers plans iterations on their own and responsible for it

Statistics gets gathered and analyzed Plan with priorities 8 hours working day

Part 3: process

Process descriptionProcess description

Development only

Testing features from iteration 1

Test matrix

Dev & bug fixingStart Finish

Iteration 1 Iteration 2 Iteration 3 Hardening

Sanity Sanity Sanity

Testing features from iteration 2 and regression

Testing features from iteration 3 and full regression

Dev & bug fixing Testing & bug fixing

Test matrix update

Test matrix update

Sanity

Process description: iterationsProcess description: iterations

Quality iteration follows after development iteration. Build must pass Sanity test and then is tested during the

iteration. Rebuilding is allowed only for fixing Sanity bugs, crashes and blocks.

If Sanity test failed new build must be created. Build is delivered by dev team on Kick Off day and Sanity

tested on the first day of iteration. It’s recommenced to create automation Sanity test

Process description: hardeningProcess description: hardening

Only two activities: testing and bugfixing SQE lead is responsible for everything Dev team tests when there is no bugfixing Builds are delivered daily, until all bugs are fixed. In the end final build is packed by dev lead and tested by

SQE lead.

Docs for the processDocs for the process

Testing approach

Start Finish

Iteration 1 Iteration 2 Iteration 3 Hardening

Sanity Sanity Final Sanity

Test matrix update

Final Test matrix with estimations and priorities update

Sanity

Initial Test matrix

1 month

Final features listNew features list

New features list

Quality expectation: iterationQuality expectation: iteration

Defects with severity of Crash or Block must be fixed in the iteration they are found.

Defects with the severity of Major found in iteration must be fixed in the following iteration

Verification of fixed/resolved issues will occur in the following iteration.

Quality expectation: hardeningQuality expectation: hardening

Final Quality Debt negotiation between the Product Manager will occur no later than one week before Hardening prior to the RTM Date.

All features must be tested during hardening. Deepness will depend on resources.

Quality Debt at the beginning of the Hardening Phase will include no more than Minor defects and defects with the severity of Major found in the last iteration

No more than 5 minor defect must be before build is released

Part 4: More practices

Unit tests and nightly buildsUnit tests and nightly builds

No unit tests no quality No ninthly builds no quality Email notification for builds Unit test coverage is controlled by SQE: usually it’s

recommended 75-80%

RefactoringRefactoring

It’s ongoing activity Helps to improve code and resolve potential issues You must have good enough unit test coverage!!!

Quality in flight: monitoring debtQuality in flight: monitoring debt

Finish

Iteration 1 Iteration 2 Iteration 3 Hardening

1 month

45 70

10

0Start

Defect Severity Weight

Feature 0

Minor 2

Major 5

Crash 10

Block 25

Gather priorities for hardeningGather priorities for hardening

11111

Attribute contracts

4.253455

Users scenarious

11111

Check no passwords in logs

AverageConfidence in QualityComplexityDefect History

ImpactChange

impactFeature

11111

Attribute contracts

4.253455

Users scenarious

11111

Check no passwords in logs

AverageConfidence in QualityComplexityDefect History

ImpactChange

impactFeature

0.691.751Attribute contracts

3.004.754.25Users scenarious

0.5011Check no passwords in logs

AverageSQE leadDev leadFeature

0.691.751Attribute contracts

3.004.754.25Users scenarious

0.5011Check no passwords in logs

AverageSQE leadDev leadFeatureGather priorities from the team

Crate summary and use it in test matrix

Test MatrixTest Matrix

Priorities set in the Matrix and features tested by them It is good to monitor time which is spent on testing Essential to put who when or on which build Bugs statistic helps better understand potential problems

Session based testingSession based testing

Create charter: task for 1-1.5 hour Give it to developer, manager, customer Get and document results

Part 5: Summary

Practices summaryPractices summary

Continuous Integration- nightly builds Test Driven Development - test first and Unit tests

coverage Refactoring Coding Standards - Code reviews In flight quality debt – constant monitoring Using priorities Hardening: time to get good build Automation! Use Wiki for improving communication

Question?