test trend analysis: towards robust reliable and timely tests
TRANSCRIPT
![Page 1: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/1.jpg)
@hughleo01 Liberty Information Technology
Liberty IT, Belfast and Dublin
Test trend analysis: Towards robust, reliable and timely tests
Hugh McCamphill
![Page 2: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/2.jpg)
@hughleo01 Liberty Information Technology2Liberty IT @hughleo01
Indianapolis Dover Belfast GS LIU DCs Europe RDC Russia DRDC
Redmond DC
Global DRDC
& Utilities
Kansas City DC
Latin America RDC
GS LIU DRDC
Portsmouth DC
GS LIU DC
Turkey DCs India DCs Asia RDC China DCs
Wausau
Fairfield
Boston
Seattle Blanchardstown
11 major IT offices
7 country data centers (DCs)DRDC = Disaster Recovery DC
3 US domestic data centers (DCs)
4 international regional data centersLIU = Liberty International Underwriters
More than 5,000+ IT
employees around the globe
Australia
About Us
![Page 3: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/3.jpg)
3Liberty IT
Hugh McCamphill
Principal Software Engineer in Test
Writing Automation since 2004
@hughleo01
![Page 4: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/4.jpg)
@hughleo01 Liberty Information Technology4Liberty IT @hughleo01
Which tests are slow
Which steps are slow
By Martin Lewison from Forest Hills, NY, U.S.A. - Cedar Point and Oberlin's Commencement Uploaded by Astros4477, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=27339553
![Page 5: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/5.jpg)
@hughleo01 Liberty Information Technology5Liberty IT @hughleo01
Tumbling Dice : Decorated Anti-tank Blocks near Harkess Rocks, Bamburgh, NorthumberlandImage Copyright Richard West. This work is licensed under the Creative Commons Attribution-Share Alike 2.0 Generic Licence
Which tests are intermittently passing
What are common failures across tests
![Page 6: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/6.jpg)
@hughleo01 Liberty Information Technology6Liberty IT @hughleo01
“Create commons cute robot mural, Toronto" by https://www.flickr.com/photos/margonaut/ is licensed under CC BY 2.0
Where are the issues?
![Page 7: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/7.jpg)
@hughleo01 Liberty Information Technology7Liberty IT @hughleo01
![Page 8: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/8.jpg)
@hughleo01 Liberty Information Technology8Liberty IT @hughleo01
![Page 9: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/9.jpg)
@hughleo01 Liberty Information Technology9Liberty IT @hughleo01
~ Henry David Thoreau
![Page 10: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/10.jpg)
@hughleo01 Liberty Information Technology10Liberty IT @hughleo01
![Page 11: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/11.jpg)
@hughleo01 Liberty Information Technology11Liberty IT @hughleo01
Example dashboard
![Page 12: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/12.jpg)
@hughleo01 Liberty Information Technology12Liberty IT @hughleo01
RunTests ResultsPost
Results
A Simple Process Flow
![Page 13: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/13.jpg)
@hughleo01 Liberty Information Technology13Liberty IT @hughleo01
A time series database is a
software system that is optimized
for handling time series data,
arrays of numbers indexed
by time
Elasticsearch is a tool for storing, searching, and
analyzing structured and unstructured data
![Page 14: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/14.jpg)
@hughleo01 Liberty Information Technology14Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
![Page 15: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/15.jpg)
@hughleo01 Liberty Information Technology15Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
![Page 16: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/16.jpg)
@hughleo01 Liberty Information Technology16Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
![Page 17: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/17.jpg)
@hughleo01 Liberty Information Technology17Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
curl -XPUT “http://localhost:9200/test-result” –
d @mapping_schema_framework.json
![Page 18: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/18.jpg)
@hughleo01 Liberty Information Technology18Liberty IT @hughleo01
{"index":{"_index":"test-result","_type":"functional" }}
{"project":“TestProject","environment":"QA","classname":“TestProject.Foo.Bar","name":"ShouldBeAbleToAddItem","duration":"47.12","created":"2016-09-14 21:35:53","status":"Passed"}
curl -XPOST “http://localhost:9200/test-
result/_bulk?pretty” --data-binary @testResults.json
Posting of results
![Page 19: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/19.jpg)
@hughleo01 Liberty Information Technology19Liberty IT @hughleo01
A Simple Process Flow
RunTests ResultsPost
Results Visualize
![Page 20: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/20.jpg)
@hughleo01 Liberty Information Technology20Liberty IT @hughleo01
Kibana is an open source data visualization plugin for ElasticSearch
![Page 21: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/21.jpg)
@hughleo01 Liberty Information Technology21Liberty IT @hughleo01
![Page 22: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/22.jpg)
@hughleo01 Liberty Information Technology22Liberty IT @hughleo01
New visualization results in a single bucket
![Page 23: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/23.jpg)
@hughleo01 Liberty Information Technology23Liberty IT @hughleo01
Splitting error messages into buckets of top occurring error messages
![Page 24: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/24.jpg)
@hughleo01 Liberty Information Technology24Liberty IT @hughleo01
Showing top ten most occurring error messages
![Page 25: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/25.jpg)
@hughleo01 Liberty Information Technology25Liberty IT @hughleo01
Splitting error messages into buckets of tests affected by that error
![Page 26: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/26.jpg)
@hughleo01 Liberty Information Technology26Liberty IT @hughleo01
Error Messages
Individual Tests
Splitting error messages into buckets of tests affected by that error
![Page 27: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/27.jpg)
@hughleo01 Liberty Information Technology27Liberty IT @hughleo01
Top Failure Reasons By Test
![Page 28: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/28.jpg)
@hughleo01 Liberty Information Technology28Liberty IT @hughleo01
Top Slowest Tests
![Page 29: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/29.jpg)
@hughleo01 Liberty Information Technology29Liberty IT @hughleo01
Top Failing Tests
![Page 30: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/30.jpg)
@hughleo01 Liberty Information Technology30Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
![Page 31: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/31.jpg)
@hughleo01 Liberty Information Technology31Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
![Page 32: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/32.jpg)
@hughleo01 Liberty Information Technology32Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
![Page 33: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/33.jpg)
@hughleo01 Liberty Information Technology33Liberty IT @hughleo01
Example Dashboard With Time Filter
![Page 34: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/34.jpg)
@hughleo01 Liberty Information Technology34Liberty IT @hughleo01
References
http://martinfowler.com/articles/nonDeterminism.htm
https://watirmelon.blog/2015/11/11/your-tests-arent-flaky/
![Page 35: Test trend analysis: Towards robust reliable and timely tests](https://reader031.vdocuments.mx/reader031/viewer/2022030313/58b8a41a1a28abc06d8b5a59/html5/thumbnails/35.jpg)
@hughleo01 Liberty Information Technology35Liberty IT @hughleo01
Thank you
Hugh McCamphill@hughleo01
Run Tests ResultsPost
Results Visualize
In Summary