From 0 To 100 000 Test Cases Per

DayGuide to successful Quality marriage

@HaapalaJani

Me... In Scale

• Jani Haapala

• Dad, Husband, Thinker, Learner, Developer, Tester

• Senior Quality Consultant @ Qentinel

• Focus on software process improvements and automation

• DevOps and Lean advocate

• 15+ years in the field

• Nuts about measurements, data and feedback

@HaapalaJani

Scaling Automation…

@HaapalaJani

Manual Testing

Test Automatio

n

Continuous Delivery

Waterfall

Agile DevOps

Dedicated To People Flow

KONE

@HaapalaJani

DatingFinding Your (Test Automation) partner

@HaapalaJani

$ pep8 --first optparse.pyoptparse.py:69:11: E401 multiple imports on one lineoptparse.py:77:1: E302 expected 2 blank lines, found 1optparse.py:88:5: E301 expected 1 blank line, found 0optparse.py:222:34: W602 deprecated form of raising exceptionoptparse.py:347:31: E211 whitespace before '('optparse.py:357:17: E201 whitespace after '{'optparse.py:472:29: E221 multiple spaces before operatoroptparse.py:544:21: W601 .has_key() is deprecated, use 'in's

Why Do We TestPROBLEM SPACE VS SOLUTION SPACE

@Test

public void pageTitleAfterSearchShouldBeginWithDrupal() throws IOException {

assertEquals("The page title should equal Google at the start of the test.",

"Google", driver.getTitle());

WebElement searchField = driver.findElement(By.name("q"));

searchField.sendKeys("Drupal!");

searchField.submit();

assertTrue("The page title should start with the search string after the search.",

(new WebDriverWait(driver, 10)).until(new ExpectedCondition() {

public Boolean apply(WebDriver d) {

return d.getTitle().toLowerCase().startsWith("drupal!");}}));}

-----------------------------------------------------------

-----------------------------------------------------------

Open Page www.google.com

Insert Text searchbox Drupal!

Click Button Google-Search

Verify Title Starts With Drupal!

-----------------------------------------------------------

Customer must be able to buy shoes

Customer needs to put shoes in basket and complete payment

With browser xx and url yyopen. Icon zz must be clicked and then button ii pressed. If valid information is given into dd, transaction should be seen from db xx with query ss.

OR

@HaapalaJani

Our ”He”, The Test Automation

www.robotframework.org

@HaapalaJani

Moving TogetherBuilding solid (Infrastructure) Foundation To Live In

@HaapalaJani

Infrastructure as Code == SafetyNet

Static checksDevelopment

Stakeholders

Unit tests

Smoke tests

Integration tests

System tests

Release

candidate

Commit

Acceptance tests

Release

1min

10min

20min

30min

60min

90min

10min

@HaapalaJani

Complexity Explosion

Dev

Test

DevOps

+ +

@HaapalaJani

Our ”SHE”, The Infrastructure (as Code)

Bare Metal Server

Virtual Machine

Artifactory

Jenkins

Nodes

Results

Metrics

Monitoring

AWS

EC2

Docker

Jenkins

Nodes

Results

Metrics

Monitoring

NodesNodes

Influxdb

MySQL

Inhouse Tools

Puppet Kubernetes

Docker Compose

Docker

Puppet

KVM CloudFormation

@HaapalaJani

Server Room Cloud

Physical Layer

Application Layer

Executor Layer

Status Layer

”Physical” Layer

Executor Layer

Having KidsSpreading Quality DNA (Scaling Automation) To All

@HaapalaJani

1

Testing Escalation…squared

0

50

100

150

200

250

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

MIN

UT

ES

MONTHS

TPT TPF

Create 100TC -> 200min -> 3h Debug 100TC -> 9700min -> 162h -> 20 days

0

100

200

300

400

500

600

0

1000

2000

3000

4000

5000

6000

1 3 5 7 9 1113151719212325272931333537394143454749

TE

ST

S F

AIL

ING

TE

ST

S C

RE

AT

ED

MONTHS

Tests Failures

@HaapalaJani

0

50

100

150

200

250

300

350

400

450

500

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Co

st

Solutions

10 % 12 % 14 % 16 % 20 % 25 %

Infrastructure Explosion…squared

@HaapalaJani

CSI Concept

Sw n

CSI TEAM

Producti

on

Support and Tooling

System QA

Team n

Team 1

Integration

Sw b

Team n

Team 1

Integration

Sw a

Team n

Team 1

Integration

Team 1

Team n

Team 1

Team n

Team 1

Team n

Integration

Integration

Integration

System QA

@HaapalaJani

Happily Ever After?Happy Kingdom (With 100 000 Test Cases Per Day) For All?

@HaapalaJani

Critical Red

@HaapalaJani

Regression

Jail

Investigation

Court

Rehabilitation

Probation

Case Denied

Collateral Damage

RemoveTest

New TestBug In Test

Bad Environment

New Feature

Actual Bug

Flaky Test

Jailing Tests

@HaapalaJani

Something To Be Proud Of

@HaapalaJani

Communi

ty

Commitment

Co-

operation

CultureQualit

y

Q&A...I Congratulate You, New (DevOps) Marriage Counselors

@HaapalaJani