Testing Distributed Systems

Presented by Miguel Silva, QA Team Leader

Meet Feedzai

https://feedzai.wistia.com/medias/toz8ny7ep7

Buzzwords

Numbers

99.95 availabilityMax 250ms latency

$2B of transactions processed per day3 deployments in fortune 500 companies

Growing 2 person / week (Hiring!)

A typical deployment

Pulse serverApp engineSQL Databases

ZookeeperRabbitMqCassandra

18 machines

The largest deployment

2 Datacenters 6 environments 150 Servers 13Tb of memory 3000 cores 250Tb of storage

Go beyond unit tests

End to end functionalityFault-toleranceHigh-availabilityMachine / datacenter synchronization

Docker and bash

Hard to maintainHard to expandHard to CIHard to test scriptsHard to script tests

System-tests framework

System-tests framework

System-tests framework

Write system-tests like unit tests

System-tests framework

Write system-tests like unit tests

Feature highlights

Set a multi-container, multi-datacenter test scenario in couple of lines

Easy to assert cluster states, connections, applications, etc.

Make deployment-like scenario system-tests like you do a unit test

Feature highlights

Abstracted container creation, setup, volumes complexity from the tests

Integrated in CI environment (jenkins)

Developers easily implement system-tests

Bonus – production firewalls

All containers are blocked by a firewall

When testing, open only the necessary ports

Ensures functionality with production security

Bumps along the way

Docker demon may have some concurrency problems and some calls fail in some tests

Containers would not be removed

Solved by doing retries on some docker calls

Bumps along the way

Docker-java leaves TCP IP sockets stuck (TCP-WAIT)

No sockets available during a build run

Workaround by enabling socket reuse at kernel level

Bumps along the way

Before 1.9, container IP would change when containers are stopped/etc/hosts updated on every docker run

After 1.9, IPs are static and more predictableContainers always keep their IP

Current status

128Gb RAM24 CPU coresover SSD

315 tests3h build7 parallel tests runningUp to 100 parallel containers running

bottleneck

Next steps

Integrate more images

Docker swarm integration

Distributed test execution

Thank You

Visit www.feedzai.com/wickedsmart

LIVE EVENTS | MEET THE PEOPLE BEHIND THE MACHINE