ASSESSINGTHEEFFECTOFDATATRANSFORMATIONSONTESTSUITE

COMPILATIONPanagiotisStratis,VanyaYaneva,AjithaRajan

12October2018ESEM'18,Oulu,Finland

SOFTWAREISEVERYWHERE

SAFETYANDCORRECTNESSARECRUCIALTESTINGISCRITICAL

TESTINGCANBEEXTREMELYTIMECONSUMING.

FUNCTIONALTESTING

Testingframeworks:

GoogleTest(C++)JUnit(Java)Mocha(JavaScript)

...and100sothers

HOWDOWEIMPLEMENTTESTING?

Repeatthecycle:

everycommiteverymergeTest-drivendevelopment(TDD)

everyday(overnightbuilds)Continuousintegration(CI)

Testingtakesasignificantportionofthedevelopmenttime.

WHENDOWETEST?

Allthetime!

1. Write/generatetests-✔Automatedtestgeneration,paralleltestgeneration

2. Build(compile)-?

3. Runtests-✔Testsuiteminimisation,testcaseprioritisation,paralleltestexecution

SPEEDINGUPTESTINGEXISTINGRESEARCH

REDUCINGCOMPILATIONTIMEISIMPORTANT

Largetestsuitestakealongtimetocompile."...Comparabletorunningtime."[CodeplaySoftware]Weneedtocompilenotonlythesystemcode,butalsothetestcode.

Compileroptimisationsincreasecompilationtime.-O1,-O2,-O3

Testcodeneedstobecompiledoften.

CONTRIBUTIONS

Codetransformationstargetingtestcode,resultinginshortercompilationtimes.

Empiricalevaluationusing15programsfromEEMBC&SPECand1largeindustryprogram.

Speedupincompilationtime:1.3xto69x.

ContainsmanyfunctioncallstotheFUT-oneforeachtestinput.

Highcompilationoverheadforfunctioninliningandinstructionselection.

Thisnumbergrowsasmoretestsareadded.

OURAPPROACH-EXAMPLE

ContainsmanyfunctioncallstotheFUT-oneforeachtestinput.

Highcompilationoverheadforfunctioninliningandinstructionselection.

Thisnumbergrowsasmoretestsareadded.

Hypothesis:ReducingthenumberofcallstotheFUT,atcompiletime,willreducecompilationtime.

OURAPPROACH-EXAMPLE

OURAPPROACHDATATRANSFORMATIONOFTHETESTS

EVALUATION-RESEARCHQUESTIONS

RQ1:CompilationspeedupDoesthetransformationspeedupcompilationtime?

RQ2:ScalabilityDoesthetransformationallowustocompilelargertestsuites?

RQ3:ExecutiontimeandcorrectnessoftestingDoesthetransformationimpacttheexecutiontimeandcorrectnessoftesting?

EVALUATION-SUBJECTS

EEMBCindustry-standardembeddedsystemsapplicationsused5telecomprograms&5automotiveprograms

SPECwell-knownbenchmarkssuiteofcomputeintensiveapplicationsused5programs,includingbzip2,libquantum

Tests:10KrandomlygeneratedtestinputsCompilers:gcc,clang

ComputeCPP,CodeplaySoftwareimplementationoftheSYCLheterogeneousprogrammingmodel

Tests:2testsuites-imageTS&bufferTSeachhasapprox.10Ktests,producedbyCodeplaydevelopers

Compilers:2customcompilers,basedonClangoneforhost&onefordevice

EVALUATION-SUBJECTS

Maxspeedup1.5x(avg.1.3x)

Maxspeedup1.8x(avg.1.4x)

RQ1:SPEEDUP-EEMBC

Maxspeedup15x(avg.7.9x)

Maxspeedup20.2x(avg.12x)

RQ1:SPEEDUP-SPEC

SPEEDUPVARIATIONACROSSPROGRAMS

-ftime-report:showstimespentcompilingindividualfilesandfunctions.

Comparedtimetocompiletestcodevswholecode.

SPEEDUPVARIATIONACROSSPROGRAMS

SPEEDUPVARIATIONACROSSPROGRAMS

Averagecompilationtimeoftestcodeaspercentageoftotalcompilationtime.

SPEEDUPVARIATIONACROSSPROGRAMS

Averagecompilationtimeoftestcodeaspercentageoftotalcompilationtime.

Usingmodulardesignwithpre-compiledlibrariesgreatlyimprovesspeedup.

SPEEDUPANALYSISForEEMBCandSPEC,

Analysedassemblycodegeneratedbythecompiler.Analysedtimespentbyeachcompilerpass,using-ftime-report.

SPEEDUPANALYSIS

Beforetransformation:

SeparatecallstotheFUTemittedforeachtest.

47%oftestcodecompilationspentininstructionselectionfunctioninliningcombineredundantinstructions

Aftertransformation:

1calltotheFUTemitted.

Only13.6%ofcompilationtimespentinthosepasses.

ForEEMBCandSPEC,

Analysedassemblycodegeneratedbythecompiler.Analysedtimespentbyeachcompilerpass,using-ftime-report.

Maxspeedup69.5x(avg.42x)

Orig.time[s] Newtime[s]

bufferTS 257 4

imageTS 434 29

RQ1.SPEEDUP-COMPUTECPP

RQ2.SCALABILITY

Beforetransformation:

Clangandgcccrashwhen1milliontestsarereached.

Aftertransformation:

Clangandgccsuccessfullycompile10milliontests.

ForEEMBCandSPEC,

wegenerated10millionrandomtestinputs.

forhighestoptimisationlevel-O3.

RQ3.EXECUTIONTIMEANDCORRECTNESS

ForEEMBC,SPECandComputeCpp,

ExecutiontimeOurtransformationdoesnotslowdowntheexecutionofthetestcode.

CorrectnessOurtransformationpreservesthecorrectnessofthetestexecution.

TAKEAWAYTestcodecanaddsignificantcompilationoverhead,contributingtototaltestingtime.

Itcanbereducedwithcodetransformationsonthetestcode.

Thisisparticularlyeffectivewhenusingmodulardesignandpre-compiledlibraries.