performance testing of a road tolling system · testing of web services • webservice mocking...

37
Performance Testing of a Road Tolling System Oliver Baendl Siegfried Goeschl 1 Donnerstag, 27. September 12

Upload: others

Post on 25-May-2020

5 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Performance Testing of a Road Tolling System

Oliver BaendlSiegfried Goeschl

1Donnerstag, 27. September 12

Page 2: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Gauteng Open Road TollingSouth Africa

2Donnerstag, 27. September 12

Page 3: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

42 Tolling Points3Donnerstag, 27. September 12

Page 4: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

1.500.000 registered road-user4Donnerstag, 27. September 12

Page 5: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

11 Satellite Centers5Donnerstag, 27. September 12

Page 6: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

41 E-Toll Shops

6Donnerstag, 27. September 12

Page 7: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

20 Mobile Payment Units7Donnerstag, 27. September 12

Page 8: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Central Operation Center8Donnerstag, 27. September 12

Page 9: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

750 Concurrent CRM Client Users

9Donnerstag, 27. September 12

Page 10: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

162 vehicle passges per second10Donnerstag, 27. September 12

Page 11: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

400 concurrent web users11Donnerstag, 27. September 12

Page 12: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Two Data Centers

12Donnerstag, 27. September 12

Page 13: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Master Control Program

13Donnerstag, 27. September 12

Page 14: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Project Overview

• Vehicles are passing through a tolling point

• The vehicle is identified either using the license plate number (via OCR) or a tag

• Vehicle passage are transmitted to Open Road Tolling Back Office (ORTBO)

• 90% video passages with three images

• 10% tag passages

14Donnerstag, 27. September 12

Page 15: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Project Overview

• Vehicle passages are collected and assigned to road-user accounts in the Transaction Clearing House (TCH)

• TCH responsible for billing the customer

• Passages of unregistered or black-listed cars are processed by the Violation Processing Center (VPC)

15Donnerstag, 27. September 12

Page 16: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Project Overview

• Overall system contains multiple Oracle RACs and MSSQL clusters

• Interfaces between the subsystems mostly web services

• Backup data center replicates the “Central Operation Center” over dark fiber

16Donnerstag, 27. September 12

Page 17: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Schedule Phase One

• Performance testing of CRM Server simulating 771 concurrent & active CRM client sessions

• Performance testing of public-facing web portal simulating 400 concurrent & active web user sessions

• On success the CRM subsystem goes live and has real user accounts

17Donnerstag, 27. September 12

Page 18: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Schedule Phase Two

• Simulating up to 162 vehicle passage per second

• Validating the end-to-end processing under load (passage processing, customer billing and violation processing)

• On success the overall system goes live

18Donnerstag, 27. September 12

Page 19: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Performance Test Scope

• End to end performance acceptance test

• Mandatory and client-witnessed test

• Strict acceptance criteria regarding

‣ throughput

‣ number and severity of errors

‣ validation of test runs

19Donnerstag, 27. September 12

Page 20: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Performance Test Types

Baseline Test One hour average load

Stress Test One hour peak load

Endurance Test 24 hours maximum load

20Donnerstag, 27. September 12

Page 21: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Performance Test Hardware

• Six dedicated load injectors

‣ Windows 2008 Server

‣ 3 physical and 3 virtual boxes

‣ Dual quad-cores with 8 GB RAM

• No direct access outside the data center

‣ Citrix over RDP over RDP

21Donnerstag, 27. September 12

Page 22: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Performance Test Tools• All performance test tools have issues

‣ Make sure that they work for you

• We prefer JVM-based test tools

‣ Easy to migrate between different OS

‣ We are Java developers

• We prefer command-line invocation

‣ RDP over slow network is really slow

22Donnerstag, 27. September 12

Page 23: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Performance Test Tools

• We use a non-distributed installation

‣ Complex network and routing

‣ Firewall not under our control

• Check the license and price tags

‣ Virtual users can be expensive

‣ Connectivity to license server?

23Donnerstag, 27. September 12

Page 24: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

The Test Tools We Used

Subsystem Interface Planned Delivered

CRM Server WebService

Public Web Portal HTML

Vehicle PassageProcessing WebService

24Donnerstag, 27. September 12

Page 25: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Apache JMeter

• Open Source and free

• Implemented a new reporting backend to scope with huge result files

• Rock solid - only minor issues

• Extensible using scripting and Java libraries

• Rough GUI compared to commercial tools

• Unsuitable for complex ASP.NET web sites

25Donnerstag, 27. September 12

Page 26: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

WAPT Pro

• Excellent value for money

• Good support

• ASP.NET module saved my day

• Extensible through Javascript

• One major issue with generation of test values under high load (requires restart)

26Donnerstag, 27. September 12

Page 27: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

soapUI Pro

• Excellent for prototyping and functional testing of web services

• WebService mocking saved my day

• Extensible using Groovy and Java libraries

• Various issues during load testing

• soapUI was replaced with JMeter for performance tests

27Donnerstag, 27. September 12

Page 28: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

• No software installation

• Take two (or more)

• Automation is your friend

• Performance test for everyone

• Performance tests are a valuable asset

Some More Thoughts

28Donnerstag, 27. September 12

Page 29: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

No Software Installation

• Portable Apps on USB Stick

‣ Run tests directly from USB Stick

‣ Clone USB Stick to hard disk

• Have all your tools on the USB stick

‣ Java, JMeter, Editor, Git, ...

29Donnerstag, 27. September 12

Page 30: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Take Two (or more)

• A minimum of two share-nothing load injectors are required to prove that the production servers is causing the performance bottleneck (and not your test tool)

• We used a maximum of five share-nothing load injectors simultaneously

30Donnerstag, 27. September 12

Page 31: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

• Performance test were run mostly at night and over the weekend

• Test execution managed by Hudson

• Test failures triggers email notification

• Test protocols are copied to FTP server

Automation Is Your Friend

31Donnerstag, 27. September 12

Page 32: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Performance Tests for Everyone

• All test scenarios are configured as Hudson jobs

‣ Baseline, stress & endurance test

‣ Some other internal tests

• Everyone can start and monitor a performance test scenario over the web browser

• Even developers can run performance tests

32Donnerstag, 27. September 12

Page 33: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

• Performance tests can be used in creative and un-planned ways

‣ Smoke test for new deployment

‣ Testing database failover

‣ Testing data center replication & failover

‣ Testing different database setups

Performance Tests as Asset

33Donnerstag, 27. September 12

Page 34: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

The few things you should take home

34Donnerstag, 27. September 12

Page 35: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Lessons Learned

• Tests tools are buggy

• Client-witnessed test are hard

• Performance tests are a valuable asset

• Creation and maintenance of complex performance test suite is a project on its own

35Donnerstag, 27. September 12

Page 36: Performance Testing of a Road Tolling System · testing of web services • WebService mocking saved my day • Extensible using Groovy and Java libraries • Various issues during

Questions & Answers

36Donnerstag, 27. September 12