!"

#

$

%

&'

$./run

{

Created by Fabrício Epaminondas ©2015-2020

ContinuousTESTING

Testing in an agile environment

• Continuous Testing • Agile principles • Why Continuous Testing? • Creating your first test project in minutes • How it works • Test process • Effective Continuous Testing

15/10/2015 - 20:30 Location: Cinema didático - Facisa

Agenda

The goal

AGILE PRINCIPLES

The Big Picture of General SDLC 4

3

21

SDLC

Requirement Initiation

Analysis & Specification

Risks

Delivery User Acceptance (UAT)

Deployment

Maintenance and Support

Operations

4

4 }

( '

Planning Scope, Time and Cost

Prioritization

Development Design / Prototype

Implementation

Integration

Testing

Documentation

Release

Software-Development-Life-Cycle

5Agile process: Scrum

SPRINT REVIEW /RETROSPECTIVE

2-4 Weeks

1 2 3

4

6

“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”

The Agile Manifesto, First Principle

WHY CONTINUOUS

TESTING?The metaphor

8Software-Development-Life-Cycle Metaphor

1.Jack is a happy programmer, he has just done his first commit

2. After a few days, he receives the test report from QA Team reproving his changes and pointing a new BUG X in production.

3.Jack is sad now, but he fixes the bug and submit his changes again.

4. Now it is the week of a new release and Jack receives other report confirming the BUG X is fixed

5. But the report also contains new BUGS: A, B, C.

9

6.Jack tries to figure out if his changes caused the other bugs

7.But Jack is not alone, he tries to figure out if someone else changes caused the other bugs

8.Jack tries a new solution to get done and re-tested in all environments before the deadline

9.Poor, poor, Jack… it is too late… the deadline has come.

Software-Development-Life-Cycle Metaphor

10

What does Jack do?

Software-Development-Life-Cycle Metaphor

11

Let’s think for a moment…

What are bugs? What do they eat? Where do they live? How to avoid new bugs?

What if bugs were living entities…

What are they thinking?

Bugs….

12

"I WILL BE BACK"(The Bug)

)

13

TRUST ME, I CAN DO IT.. ( Developer in a good day)

14

• To control side effects • To avoid software defect snowball• To efficiently support multiple platforms,

environments and release versions • To get fast results in a short time cycle • To anticipate risks• To create an agile, reliable and repeatable process• Release in hours not months

Use Continuous Testing…

CREATING YOUR FIRST PROJECT

IN MINUTES

Hands-On

16Hands-On

• Let’s create our first project for continuous testing using an amazing and totally free web hosting platform with continuous integration support:

Let’s start from the test project template:

https://gitlab.com/fabricioepa/gitlab-ci-quickstart

gitlab.com

17Hands-On•You can learn how to implement your own test platform

in the training:

https://fabricioepa.wordpress.com/2015/02/02/quickstart-for-continuous-integration/

•For more advanced continuous integration practices see the link:

https://fabricioepa.wordpress.com/2015/02/03/continuous-integration-practices-to-improve-the-software-quality/

HOW IT WORKS

Overview

19Overview

• Test Environment• Test Data Management • Automatic Deployment • Multiple environment configuration profiles •Virtualization (production-like) •Dynamic Infra-structure

•Test Automation • Orchestration

*Continuos Testing

20Related Topics

• Continuous Integration (CI) • Continuous Delivery • Continuous Deployment • See 3 high levels of continuous testing adoption

*Continuos Testing

Design development pipelines 21

Commit Stage Unit Tests

Code Analysis

Code Review

'

Triggers

/branches/product-1.0

Development Unit Tests

Code Analysis

Development pipeline

Build Package

Sanity Tests

Verify Metrics

Publishing Test Report

Unit Test Coverage

Project Metrics

Release Package

Fast Builds pattern ,

Artifact Repository$

01

22

Setup Environment

Infrastructure & Data

QA automation pipeline

Deploy Install packages

Automated Test Plan Integration,

Functional, Regression,

Performance, etc.

Publishing Test Report

Test Coverage

Performance Benchmark

○ Push-buttonsScheduled buildsRepository changes{

' /branches/product-1.0

Triggers

Design QA pipelines

Self-service " + Nightly-builds {

Artifact Repository$

02

23

Setup Environment

Infrastructure

Data

Testing Manual

Exploratory

○

Deploy Install packages

Health check

○

. QA Team

UAT

Self-service " deployments

Design delivery and deployment pipelines

Delivery / Deployment pipelines

User Acceptance Demonstrations

Consumer Preview

Beta testing

GA releases

03

Artifact Repository$

AGILE TEST PROCESS

The real deal

25Main Testing Stages

Planning Scope, Time, Cost

Process guideline

Specification Test Scenarios

Test Cases

Prioritization and

Classification

Execution Manual

Automatic

Closure Test Report

Bug Report

Qualification

Documentation

1 2 3 4

26Test plan

A good test plan should answer at least 3 main questions:

• What? • How? • When?

01

s27Test plan: What?

Guideline for macro activities•Test execution

• Defect detection (Bugs) • Enhancement areas

)

01

• Strategies and methodology• Environment and tools

28Test plan: How?01

• Traceability

29Test plan: How?01

30

Schedule

Test plan: When?

/" 0Team and Resources

Test Scope and Risks

4

01

31

•Black-box / White-box •Unit / Component /

Integration •Smoke / Sanity •Acceptance / Functional •Regression •System / End-To-end •Exploratory

•Load / Stress / Performance

•Parallel / Concurrent •Usability •Alpha / Beta •Compatibility •Destructive / Recovery •Internationalization /

Localization

Choose the right test techniques01

32Test specification

• Requirements study •Test Scenarios and Estimative of Test Cases •Test Case Creation, Review and Maintenance •Prioritization and Classification

•Automatic vs. Manual •Others: Acceptance, Functional, Performance,

etc. •This information will be used for creating

execution test plans.

02

33Executing test plans

•Manual execution •Automatic execution (by QA test pipeline) •Whatever the way you take, you must choose a

strategy to compute all test results in the final analysis

•To use a tool like TestLink may help you to manage all tests.

03

34Closure

•Reports • Test Execution Status • Defects by Priority • Requirements Test Coverage • Other quality aspects:

•Performance, accuracy, security issues, etc.

04

35Closure

•Final report and Qualification: •Approval Status of Deliverables

04

1

X1

Feature A

Feature B

Feature C

L Partial release approval

The real value

EFFECTIVE CONTINUOUS

TESTING

• Let’s remember the agile principle:

37Continuous Quality Improvement

“…continuous delivery of valuable software.” The Agile Manifesto, First Principle

• Testing != Software Quality Assurance • You must understand how to use tests to enhance

the process, reduce rework, increases the quality and business value of the software product.

• This is Continuous Improvement, one of the most important QA practices.

38Continuous Improvement in 4 Steps

Plan Targets, Methodology,

Process, etc…

Do Process execution

Verification / Study Measure Results

Study evidences and deviations

from original plan

Action Diagnostic

Corrective and Preventive Action Points

Improvement Areas

Notes: Continuous Improvement and Agile

• Retrospectives • Learned Lessons • Action Points • Use the PDCA Cycle 2

40Effective Continuous Testing and Agile

•Objective •For each sprint the deliverable must potentially

be production-ready •Quality metrics and daily feedback

•Build status and reports •Statistics and Trends

•Automatic Tests report •Code violations •Code coverage •Project metrics

•Dev&QA teams must understand and maintain the quality metrics

3 Displays + Dashboards

% Email notifications

Continuous Monitoring

Effective Continuous Testing and Agile

42

•Balance test levels according to the project needs on each sprint

Effective Continuous Testing and Agile

Unit Test

Integration Test

System Test

Acceptance Test

Test individual component

Test integrated system

Test component groups

Test final system stack

V-model

43

•Continuously improve your automated testsEffective Continuous Testing and Agile

Unit

Integration

E2E

• Recommended test distribution pyramid

Cheaper automation

Complex Automation

UI / Manual / Exploratory Tests

Reference: [1] http://googletesting.blogspot.com.br/2015/04/just-say-no-to-more-end-to-end-tests.html

44

•Unrelated activities (to the backlog) •Implementing automation scripts •Status monitoring •Multiple environment maintenance: Testing,

Staging, Production… •Sprint Retrospective Tasks

•Use CI feedback to continuous improve the process and the product quality

•Diagnosing pitfalls and registering sprint action points

Continuous Testing and Agile Planning

45Business Value

•The cost of adoption has a fast return on investment

•Rework reduction •Increase team productivity. •Release in days not months •Reduce at about 50% of development, testing

and deployment costs.

!"

#

$

%

&'

$./run

{

Criado por Fabrício Epaminondas ©2015

ContactFABRICIO EPAMINONDAS 4 linkedin.com/in/fabricioepa

5 fabricioepa.wordpress.com

6 fabricioepa

7 @fabricioepa