reaching for your quality stretch goals: testing at realtor.com

© 2015 Move, Inc. All rights reserved. Do not copy or distribute.

Reaching your Quality Stretch@ realtor.comKlaus SalchnerVP Engineeringhttps://www.linkedin.com/in/ksalchner@ksalchner

April/5, 2017

https://www.linkedin.com/in/ksalchner

https://www.linkedin.com/in/ksalchner

© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 2

Agenda

• A/B Testing to measure customer impact – 5-6 min

• Group activity – 5-6 min

• Where in the stack to invest in test automation – 5-6 min


• Testing for reliability, resilience und recovery – 5-6 min


• Q&A – 5-6 min


A/B Testing to measure customer impact

• End-to-end quality goes beyond shipping software

• It is about outcome, value for the customer

• How do you measure whether you solved a real customer need?



• Start at the story level, instead of WHAT you build you talk about

what customer need you solve and WHY that matters

• Come up with a Hypothesis and what you will measure to make it

quantative, meaningful?

E-commerce industry is a great example



By VWO Blog



Optimizey Blog



By VWO Blog


Some of the A/B Testing tools

• Adobe Test & Target

• Optimizely

• Maxymizely

• and more https://conversionsciences.com/blog/ab-testing-tools

• or home-grown

https://conversionsciences.com/blog/ab-testing-tools

https://conversionsciences.com/blog/ab-testing-tools


The 8 rules of A/B Testing by Search Engine Watch

• Hypothesis

• One Variable at a time

• Clear and aligned success criteria

• Volume and statistical significance (also risk)https://www.optimizely.com/optimization-glossary/statistical-significance

• Test Groups and Splits (can be 50/50, 10/90, etc.)

• Randomization

• Always be testing, with common sense

• Documentation

https://www.optimizely.com/optimization-glossary/statistical-significance

https://www.optimizely.com/optimization-glossary/statistical-significance


Q&Ahttp://www.topproducer.com/careers

12


Agenda







• Q&A – 5-6 min


Where in the stack to invest in test automation

• Always go to the lowest levels

• Always write small, reusable, fast tests



Experiencebehavior

Functionalbehavior

Databehavior



Services


Test automation best practices

• Test Early and Test often

Have strong acceptance criteria – is an outline for your testing

Write your unit testing as components are created and iterate – full

or lightweight TDD

Integrate the tests early on into your CI/CD so they run all the time

Run them all the time!



• Create good quality test data

Managing test data, especially at scale, is a hard problem to

manage

Many times there is a pre-set environment with pre-set data which

lives for long-times and as time passes gets messy and in the way

for teams to be efficient



• Create good quality test data

The nirvana is that test data gets generated as tests run, and

cleaned up afterwards

And that you have sets of test data you can load into an

environment as it gets stood up

And that you can spin up and tear down environments as needed

– really hard to achieve



• Create UI tests which are resistant to change

Use names and class IDs to identify objects in the UI

Never use text labels, they always change

Never use positions, they always change



• Decide what test cases to automate, like

Frequently used (key) features

Tests which tend to cause human errors

Repetitive tests run with each CI/CD build / deployment

Tests you need to run on multiple platforms / browsers

Tests which are very time consuming



• Review your test cases with PO / PM and Engineers

Get many eyes on it and understand the main flows and edge

cases

Iterate on the testing like you iterate on the development

Get it out to customers for validation and use the feedback



• BDD – Behavior Driven Dev

Test the behavior, business use case

• TDD – Test Driven Dev

Be pragmatic, define the test for a component and then write the

code, not the whole thing

https://codeutopia.net/blog/2015/03/01/unit-testing-tdd-and-bdd






• What are other doing or not doing?



26


Agenda







• Q&A – 5-6 min


Testing for reliability, resilience and recovery

• Reliability is a concern owned by your Services & APIs

Clustering, fail-over

Elastic scaling (up and down)

Error handling

More


Basic steps for reliability testing…

• Set up a test environment where ideally all services are set up for

elastic scaling (cloud)

• Run a set of automated tests to establish a baseline (can be really

any automated tests like Selenium or JMeter)

• Scale up and down the load and observe how the various services

will start to scale up and down

• It will also tell you which services are the most stressed parts of

your application


Basic steps for reliability testing – expected results

• You want to see that your various services scale up – if they don’t

then they will break (unless they are super-fast/efficient)

• You want to see that the services scale down again – if not then

your costs will go up and up

• During the scale up and down you want to see that the functionality

doesn’t break (validates things like state-less)


More reliability testing…

• You can also set the maximum for the scaling up and down and

observe how the system behaves as it reaches the maximum load

• It will point out the weaknesses in your architecture and where to

optimize



• Resilience is a concern owned by the consumer / client

Error handling

Graceful degradation of features, so if one or more services are

down the user can still use your application:

For example: If the Ratings & Review service is down I can still

find and view products, or purchase products, without delays



By Nextflix Blog



• The “golden standard” is Hystrix, former circuit-breaker, by Netflix

https://github.com/Netflix/Hystrix/wiki

• Calls to services are wrapped in commands, and you can define

thresholds to these commands





By Nextflix Blog


Basics of resilience testing…

• Run automated tests, like Selenium or JMeter, and while these run

start shutting off non-essential services

• Then observe that your experience still functions, fast and reliable

• And observe that some of the features depending on these services

degrade – can be either not showing up or showing relevant

messages to users



• Recovery testing is the exercise of what it takes to recover from a

major fault or an outage

Troubleshooting – like Logs, instrumentation, etc.

Restoring services

Worst case, restoring data

Root causing is part of it


Basics of recovery testing…

• Run automated tests, like Selenium or JMeter, and while these run

start shutting off essential services

• Then observe that your experience stops functioning or slows down

for users

• Then run through the manual and automated steps to recover,

without making the situation worse for users



41


Agenda







• Q&A – 5-6 min



43

reaching for your quality stretch goals: testing at realtor.com

Technology