Technical Testing

Alexey Zverev

EXTENT February 2011

Contents

• Technical Testing

• Our world of trading systems

• Approach – Performance Testing

• Approach – Stability Testing

• Approach – Operability Testing

• Approach – Debugging

• Tasks – Test Tools

• Tasks – Test Preparation/Load Shape

• Tasks – Test Execution

• Tasks – Result analysis

Technical Testing

• Technical Testing or NFT – Validation of Non-functional Requirements

• Performance• Stability• Operability

• These requirements are usually applied to complex distributed server

applications– Supports many interactive users – Processing high volume of data coming from and disseminated to several

sources

• Example: Web Sites or Web Services

• Our world – Trading Systems, Exchanges, SOR, DMS, Brokerage, Trading

Desks

Our world of Trading Systems

TradingSystem

Broker/Trader

Market Data Provider

Downstream Systems

ALGO

Bad ALGO

DMA/OMS

SOR

Transaction StorageBackofficePost Trade

Market Data Distributor

The Approach to Performance Testing

• Load Test – Emulate load– Measure KPIs (Latency, Throughput, Capacity)– Determine the load, under which the KPIs become worse than required

• Stress Test– Emulate extreme load– Monitor system behaviour– Ensure that we sustain it

• Soak Test (Long Running Test)– Emulate load for a long time– Check whether we have any deterioration

Approach

• Stability Testing – Apply Load

• kill• unplug• do something crazy

– Operability Testing• Manage the environment

– Debugging• Help developers identify issues• Complex vs. Simple load

Tasks Test Tools

• Load injector

• PCAP analyzer

• These are some of the most essential pieces of any NFT project: – Correctly emulate the connections, implement protocol, use API– Make sure that KPI measurements are valid and accurate– Make sure that a tool does not experience performance issues itself and

doesn’t influence KPI measurements– Developers whose application you are testing will blame your tools! You

will require a proof of your figures based on valid sources (as the developers think)

Tasks Test Preparation Load Shape

• Live production system is the ultimate source of information

Tasks Test Execution

• Automated system restart and test execution is a good idea

• Our system is a complex look at what’s going on

• Be ready to run tests again and again

Tasks Result Analysis

• Monitoring hardware reconciles your latency and hardware results

• Be ready to advocate test results

• Perform in-depth analysis

• Make sure results are repeatable – unicorns are non-existent– Most of your odd results are probably invalid test runs

Questions & Answers

Thank you.