test trend analysis : towards robust, reliable and timely tests

@hughleo01 Liberty Information Technology

Liberty IT, Belfast and Dublin

Test trend analysis: Towards robust, reliable and timely tests

Hugh McCamphill

@hughleo01 Liberty Information Technology2Liberty IT @hughleo01

Indianapolis Dover Belfast GS LIU DCs Europe RDC Russia DRDC

Redmond DC

Global DRDC

& Utilities

Kansas City DC

Latin America RDC

GS LIU DRDC

Portsmouth DC

GS LIU DC

Turkey DCs India DCs Asia RDC China DCs

Wausau

Fairfield

Boston

Seattle Blanchardstown

11 major IT offices

7 country data centers (DCs)DRDC = Disaster Recovery DC

3 US domestic data centers (DCs)

4 international regional data centersLIU = Liberty International Underwriters

More than 5,000+ IT

employees around the globe

Australia

About Us


Principal Software Engineer in Test

Belfast Selenium Meetup Organiser

Writing Automation Since 2004

WebDriver user since 2011


Which tests are slow

Which steps are slow

By Martin Lewison from Forest Hills, NY, U.S.A. - Cedar Point and Oberlin's Commencement Uploaded by Astros4477, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=27339553


Tumbling Dice : Decorated Anti-tank Blocks near Harkess Rocks, Bamburgh, NorthumberlandImage Copyright Richard West. This work is licensed under the Creative Commons Attribution-Share Alike 2.0 Generic Licence

Which tests are intermittently passing

What are common failures across tests


“Create commons cute robot mural, Toronto" by https://www.flickr.com/photos/margonaut/ is licensed under CC BY 2.0

Where are the issues?

http://creativecommons.org/licenses/by/2.0/


~ Henry David Thoreau


Example dashboard


RunTests ResultsPost

Results

A Simple Process Flow


A time series database is a

software system that is optimized

for handling time series data,

arrays of numbers indexed

by time

Elasticsearch is a tool for storing, searching, and

analyzing structured and unstructured data


Setting up a mapping

{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}




curl -XPUT “http://localhost:9200/test-result” –

d @mapping_schema_framework.json


{"index":{"_index":"test-result","_type":"functional" }}

{"project":“TestProject","environment":"QA","classname":“TestProject.Foo.Bar","name":"ShouldBeAbleToAddItem","duration":"47.12","created":"2016-09-14 21:35:53","status":"Passed"}

curl -XPOST “http://localhost:9200/test-

result/_bulk?pretty” --data-binary @testResults.json

Posting of results


A Simple Process Flow

RunTests ResultsPost

Results Visualize


Kibana is an open source data visualization plugin for ElasticSearch


New visualization results in a single bucket


Splitting error messages into buckets of top occurring errors


Showing top ten most occurring error messages


Splitting error messages into buckets of tests affected by that error


Error Messages

Individual Tests

Splitting error messages into buckets of tests affected by that error


Top Failure Reasons By Test


Top Slowest Tests


Top Failing Tests


Step Time Versus Number Of Times Executed


Example Dashboard With Time Filter


References

http://martinfowler.com/articles/nonDeterminism.htm

https://watirmelon.blog/2015/11/11/your-tests-arent-flaky/


Thank you

Hugh McCamphill@hughleo01

Run Tests ResultsPost

Results Visualize

In Summary