1LibreOffice Aarhus 2015: State of Automated Testing

State of Automated Testing

Michael Stahl, Red Hat, Inc.2015-09-23

2LibreOffice Aarhus 2015: State of Automated Testing

Overview

1. Introduction

2. Regression Tests Developers Run Every Day

3. Even More Tests

3LibreOffice Aarhus 2015: State of Automated Testing

Introduction

Every release: ~10.000 commits, ~1 million LOC changedRegressions?Automated testingGoal: developers find bugs before they push them to master

“Your developers—or worst case, test organization—produce tests.”

— Michael Klepikov

4LibreOffice Aarhus 2015: State of Automated Testing

Requirements

Goal: developers find bugs before they push them to masterRequirements

Standard unit test librariesMust be run as part of standard build – make check

Reliable – avoid false positivesFastGood defect localizationDebuggable

5LibreOffice Aarhus 2015: State of Automated Testing

CppUnit Tests

C++Standard CppUnit library (Thanks to Markus Mohrhard)In-process (except smoketest)

Kinds:Unit tests

C++ classesComponents via UNO API

Integration testsFilter crash-tests (CVE-tests)Filter testsEtc.

System tests: smoketest

6LibreOffice Aarhus 2015: State of Automated Testing

Filter Tests

Filter test:Import fileCheck some properties were imported correctlyExport file, import againCheck properties were round-tripped correctly

--with-export-validation validates every exported file

ODF ValidatorOOXML Validator

7LibreOffice Aarhus 2015: State of Automated Testing

CppUnit Test Growth

3.3 3.4 3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0

2000

4000

6000

8000

10000

12000

14000

16000

18000

0

50

100

150

200

250

CppUnit Tests

CPPUNIT_TESTCPPUNIT_ASSERTCppunitTest*.mk

LO version

8LibreOffice Aarhus 2015: State of Automated Testing

JUnit Tests

JavaStandard JUnit libraryKinds:

Unit tests – in-processURE Java binding

“Complex” tests – out-of-process – Remote UNOUnit tests of componentsIntegration tests

9LibreOffice Aarhus 2015: State of Automated Testing

JUnit Test Growth

3.3 3.4 3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0

500

1000

1500

2000

2500

3000

3500

4000

0

5

10

15

20

25

30

35

JUnit Tests

JUnit @TestassertJunitTest*.mk

LO version

10LibreOffice Aarhus 2015: State of Automated Testing

qadevOOo “unoapi” Tests

JavaCustom test frameworkOut-of-process – Remote UNOObscure test code“black-box”, overly generic testsKinds

Unit tests – components via UNO API

11LibreOffice Aarhus 2015: State of Automated Testing

qadevOOo Test Non-Growth

3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0

200

400

600

800

1000

1200

1400

1600

1800

0

5

10

15

20

25

30

35

qadevOOo UnoApi Tests

unoapi componentsunoapi interfacesunoapi*.mk

LO version

12LibreOffice Aarhus 2015: State of Automated Testing

Python Tests

PythonStandard unittest libraryIn-processThanks to David Ostrovsky Kinds:

Unit tests – PyUNO bindingUnit tests – components via UNO API

13LibreOffice Aarhus 2015: State of Automated Testing

Python Test Growth

3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0

50

100

150

200

250

300

350

0

1

2

3

4

5

6

7

8

9

Python unittest Tests

Python test*assertPythonTest*.mk

LO version

14LibreOffice Aarhus 2015: State of Automated Testing

Tests Run by make check

3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0

500

1000

1500

2000

2500

3000

3500

4000

"make check"

CPPUNIT_TESTJUnit @Testunoapi interfacesPython test*

LO version

15LibreOffice Aarhus 2015: State of Automated Testing

Requirements Checklist

Std. libs make check

Reliable Fast Defect localization

Debug-able

AutoTest / testtool ✘ ✘ ✘ ✘ ✘ ?

CppUnit„unit-test“ ✔ ✔ ✔ ✔ ✔ ✔

CppUnit „filter-test” ✔ ✔ (-) ✔ ✘ ✔

JUnit „complex“ ✔ ✔ (-) ✔ ✔ ✘

qadevOOo „unoapi“ ✘ ✔ (-) / ✘ (-) / ✔ ✘ ✘

Pythonunittest ✔ ✔ ✔ / ? ✔ ✔ (-)

16LibreOffice Aarhus 2015: State of Automated Testing

Better Testing With Assertions

Use assert() liberally in the product code to detect invalid statesAssertion failure → abort and test failureBut not necessarily good defect localization

17LibreOffice Aarhus 2015: State of Automated Testing

assert() Growth

3.4 3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0

1000

2000

3000

4000

5000

6000

assert()

assert

LO version

18LibreOffice Aarhus 2015: State of Automated Testing

Code Coverage

GCC -fprofile-arcs -ftest-coverage

LCOVhttp://lcov.libreoffice.org/Updated dailyThanks to Maarten Hoes

19LibreOffice Aarhus 2015: State of Automated Testing

LCOV Code Coverage

01.04.2014 01.09.2014 01.02.2015 01.07.20150

500000

1000000

1500000

2000000

2500000

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

LCOV Code Coverage

Lines HitLines TotalLines Coverage

20LibreOffice Aarhus 2015: State of Automated Testing

More Tests

“Crash Testing”Import and export 80k documentsSee talk by Caolán McNamara

Performance testCallgrind profilingmake perfcheck

http://perf.libreoffice.org/Additional out-of-tree tests in test-files.git repoThanks to Matúš Kukan, Laurent Godard, Norbert Thiebaud

21LibreOffice Aarhus 2015: State of Automated Testing

All text and image content in this document is licensed under the Creative Commons Attribution-Share Alike 3.0 License (unless otherwise specified). "LibreOffice" and "The Document Foundation" are registered trademarks. Their respective logos and icons are subject to international copyright laws. The use of these therefore is subject to the trademark policy.

Now … go forth and write more tests!

Thanks for listening.