Download - Exactpro: Non-functional testing approach
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
1
Non-functional testing approach
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49www.exactpro.com
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
2
Contents
• NFT Team structure• Tests preparation • Test tools• Latency measurements• Capacity tests• DLC test• Failover tests• Other NFT tests
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
3
NFT team structure
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
4
LSEG Non-Functional Testing Team
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
5
Test plan
Sample of an Non-Functional Test plan
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
6
Load Generators Classification
Measurement-Based Model-Based
Closed-Cycle Open-Cycle
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
7
Asymmetric Response Model
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
88 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
Item Description
Exactpro Test Strategy Step(s)
Load generation in trading systems
Testing Type Active Load and Non Functional Testing
Target SUT Trading Platforms, Market Data Delivery and Post-Trade Systems and their combinations
SUT Interface Back-end (typically connected to message gateways / APIs; data streams generation: mcast/ucast); GUI Testing Capabilities not supported
SUT Interaction Method
Inputs and outputs are generated based on the configured load shapes, parameters and templates. Captured messages can be viewed and analyzed post-factum using the DB queries (Shsha) or/and performance calculator tool (also developed by Exactpro)
Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, ITCH, etc.) and Proprietary (MIT, SAIL, HSVF, RTF, RV, Quant House, etc.) protocols. New plug-ins for additional protocols developed by request
Test Scripts Capable to stress the system with high rate of transactions including microbursts. Used for Throughput, Bandwidth, Latency tests. Can be used for support of fault tolerance (Failover) tests
Test Management, Execution and Reporting
Simulation of multiple client connections with specified load shape for each connection or group of connections (configure number of connections, messages templates, Load Shape for each connection or group of connections, messages distribution for each connection or group of connections), throughput up to 75,000 msg per core per secondSimulation of market data streams with required SLAs
Platform requirements Linux on 64-bit platform
Primary Competitor VeriFix TestPilot, HP Load Runner, IBM Rational Performance Tester, JMeter, Yandex.Tank
Load Injector test tool
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
9
Exchange testing common scheme
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
10
Tests preparation
• System configuration:- Production setup (Instruments, trading parameters, etc)
• Data for building test load:
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
11
Tests preparation. Load injector configuration files
Load injector configuration files structure
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
12
Tests preparation
Load shape:
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
13
Latency results End-to-end
Do we actually send what we thought we send?
• Evaluation of message rate ‘per millisecond’ unit and order mix balance:
Number of msgs per millisecond
% SamplesInbound (into MIT)
Outbound (from MIT)
<5 55.64% 55.01%5-8 3.67% 4.05%8-10 2.60% 2.77%10-15 5.32% 5.39%15-20 5.88% 5.95%20-80 26.85% 26.78%>80 0.05% 0.05%
Partition 1 Message
TypeME cores
Total0 1 2 3Order 3.74% 3.02% 2.00% 4.14% 12.89%Cancel 3.56% 2.89% 1.93% 4.02% 12.39%Amend 0.60% 0.53% 0.34% 0.68% 2.16%Quote 0.32% 0.11% 0.16% 0.27% 0.85%Trades 0.24% 0.18% 0.13% 0.29% 0.84%
Order mix and load balance across ME cores:Message rate per millisecond:
• Internal system stats arbitration (rates per second and total amount of transactions)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
14
Latency results End-to-end
%Native
avg max100 82 518
99.99 82 40899.9 82 139
99 80 103
12 31 48 76 320
687
2527
4005
1874
237
869 30 45 62 181
546
1299
3009
1292
829
990 29 32 61 77 436
791
2545
4977
2540
4
0
100
200
300
400
500
600
max_100max_99_9max_99_0
rate, mps
Late
ncy,
us
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101
106
111
116
121
126
131
136
141
146
151
156
161
166
171
176
050000
100000150000200000250000300000350000400000450000500000
Totalnew_oamend_onew_qcancel_o
Latency, us
Num
ber o
f sam
ples
Latency distribution:
Latency percentiles:
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
15
Other NFT tests
• Capacity test
– Check all external interfaces:• Traiding FIX• Trading Native• PostTrade• DropCopy• ITCH Replay/Recovery
– Check hardware resources consumption by system components:• CPU• Memory• Disk• Network
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
16
Other NFT tests
• DLC test– Pass system through Production like schedule:
• All trading cycles• All scheduled sessions
– Apply appropriate load during various cycles to trigger extensions, special sessions, etc– Perform some functional tests under load
– Data consistency check • All expected messages are disseminated via ITCH during session transitions• All Downstream systems connected and confirmed daily checks
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
17
Other NFT tests
• Failover tests– Application failovers– Hardware outages
• Application failovers– Simulation of various outage types (SIGKILL, SIGSTOP, Stop via SysGuard)– Failovers during different system state (at startup / trading day / EOD)
• Hardware failovers– Network equipment failovers (Switches, Ports, NICs) – impact on Application– Server isolations– Oracle instances
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
18
Other NFT tests
• Rapid user actions tests (connect-disconnect, logon-logout)– System should sustain against such user behavior– HW resources consumption should not grow up
• Slow consumer tests– System should handle such users and should has a protection against them– HW resources consumption should not grow up
• Intensive usage of recovery channels–System should be able to handle high number of requests on recovery channels and
should be able satisfy them
• Massive actions from Market Operations (mass order cancels, mass trade cancels, mass instrument halts)– System should handle Market operations’ actions like mass cancel of 10k active orders
or trades
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
19
Other NFT tests
TQ Special latency test – Main market Reference prices processing delay for TQ dark book:
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
20
Thank You!