decrease test build time, not test quality with ca blazemeter

World®’16

DecreaseTestBuildTimeNOTTestQualityRefaelBotbol- ServicesClientManager- CATechnologies

DO5X53E

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

Today,yourbuildtimeconsistsofmuchmorethanjustthetimeittakesJenkinstobuildyourapplication.Italsoincludesregressiontests,deploymenttodifferentenvironments,andmaybeotherapplicationspecifictasksyouwanttoautomateaspartofyour“build”.

Yourbuildtimecansignificantlyinfluencehowoftenandwhenyoudecidetorunyourbuilds(i.e.whetheryourunyourbuildsweekly,nightly,twiceaday,orjustpercommit).Allofthishasadirectimpactonyourproductivityandefficiency.

Buthowcanyoudecreaseyourtestbuildtimewithouttriggeringadropinproductquality- oreventhenumberoftestsyourun?TakealookatanewapproachtotestautomationwithintheCIprocess,andlearnhowto:• Increasethequantityandtypeoftestsrunwhilereducingyourbuildtime• IncludeyourperformancetestsaspartofyourCIprocesstoidentifyperformance

degradationissuesearlyintheprocess• Shiftleftallthewaytothedeveloper.SeehowdeveloperscancreatetestswithDSLusing

theirexistingJSON/YAMLknowledge

Thissessionwillstartoffwithaninteractivediscussionbeforeturningintoademonstration/hands-onworkshop.Attendeeswantingtoparticipateinhands-onwillrequiretobringtheirownlaptopstoexecutetheexercises.

RefaelBotbol

CATechnologiesServicesClientManager


Agenda

SAVETIME:BYPARALLELING YOURTESTS

DELEGATE:ENABLEEACH PERSONACONTRIBUTETOTHEEFFORT

FOCUS:CONCENTRATEONBUILDING NOTONMAINTAININGTESTS

SCALE:EXECUTEBUILDSASMANYASYOURBUSINESSREQUIRES

1

2

3

4

REDUCETHEBUILDTESTINGPHASEWHILEINCREASINGTESTCOVERAGE

DEMONSTRATION/WORKSHOP5


Whichtypeoftestshouldwerun?

§ Functional(API/UI)

§ Regressions

§ CriticalBusinessProcesses

§ Performance

§ Smoketests


OpenSourceToolsyouprobablyuseforTesting

§ Functional(API/UI)– Selenium/JMeter/PhantomJS

§ Regressions – Selenium/JMeter/PhantomJS

§ CriticalBusinessProcesses– Gatling/JMeter/Selenium

§ Performance – Gatling/JMeter/Locust/Tsung

§ Smoketests– Selenium/JMeter/Gatling


Agenda

SAVETIME:BYPARALLELING YOURTESTS

DELEGATE:ENABLEEACHPERSONACONTRIBUTETOTHEEFFORT

FOCUS:CONCENTRATEONBUILDINGNOTONMAINTAININGTESTS


1

2

3

4




TestingLayers• Monitoring• Smoketests• FullscaleperformancetestsProd

• Criticalbusiness processes• PerformancetestsStaging

• FunctionalUI• Functionalbackend• Regressiontests• Criticalbusiness processes

Integration


Divide,Parallel&Conquer

§ Planyourtestsperlayerandparallelasmuchaspossible

§ Stopthebuildonceadesignatedcriticaltesthasfailedand…– Notifythedeveloperregardingtheissue– Freeresourcesforotherbuilds

§ Buildonce,runmany(evenwhenaddingnewtests)


DSLbasedTestCreationusingYAMLinTaurus


DescribeTestExecutionScenario


SetPass/FailCriteria


SAVETIME:BYPARALLELINGYOURTESTS

DELEGATE:ENABLEEACH PERSONACONTRIBUTETOTHEEFFORT



1

2

3

4




Whocancontributetothebuild’stests?

§ Coders – ByprovidinglowleveltestsforAPI/UIandregressions

§ Testers – Byprovidingsystemwidetestsandregressions

§ Operations – Byprovidingcriticalkeypointsandcreatingtestswithpass/failcriteriaforthem


Whocancontributetothebuild’stests?

§ Product– Byprovidingpass/failcriteriaforbusinessprocesses

§ Marketers– Bycompletingtheproductviewandprovidingmarketingscenarios

§ Performance – Byimplementingperformancetests


Thisishowwe(can)doit

§ Coders– UsingcodebasedenginesorevenYML/JSONDSL

§ Testers – TheywouldprobablytellyouhowJ

§ Operations– Automatingthescriptsandprovidingpass/failcriteriatomakesurenothingbreaks


TheHow– SurprisingPart

§ Product – Providingpass/failcriteriaforstagingandproductionenvironments(andafterseveralruns,evenfortheintegration..)

§ Marketers – Launchinganewevent?- WhynottestisbyusingaDSLwiththesepages?

§ Performance – Theseguysalreadyhavethescriptsandtheproductionvalues.Theonlythingthatisleft,istoincludeit.


Coders/TestersExamples

§ AsimpleAPIexample

§ Acomplexscenario

§ Java/Pythonseleniumtests

§ JMeter/Gatlingtests


Product/OperationsExamples

§ Createtestsandsetcriteria

§ Setpass/failcriteriaperenvironment

§ Easilyincludeitaspartofyourtestexecution


MarketingExample


AddPerformanceTeststoyourbuild


Summary

§ First,takeadeepbreath…youhavemadeitsofar,youdeserveitJ

§ WecanuseDSLbasedonYML/JSONtocreatetests

§ Wecanchoosetheflavoroftoolswhichbestsuitsus,andreuseourexistingscripts

§ WecanalsousetheDSLtosetpass/failcriteriaanddescribethetestexecution




FOCUS:CONCENTRATEONBUILDING NOTONMAINTAININGTESTS


1

2

3

4




BuildOnce– UseMany

§ Option1:Separateeverything– Haveyourscriptsrepresentonetestatatime– HaveoneormoreDSLdescribingtheexecutionperbuildper

environment– HaveoneormoreDSLdescribingthepass/failcriteriaperenvironment

Thisgivesyoumaximumagilityoncontrollingyourbuildtests– Addingtestsisfast– Criteriaareappliedtotestsautomaticallyperenvironment


BuildOnce– UseMany

§ Option2:Haveclustersoftests– Groupyourteststogetherbaseontype,functionalityorany

characteristicsthatfitsyou– Attachtherightexecutionandcriteriapergroup

Thisisthebestoptionforspecialprojectssuchas– Anewcampaign– Satelliteprojectswhicharen’tpartofeverybuild– Performancetests


Option1ExecutionExample


Option2ClusteredTestExample






1

2

3

4




UsetheForce CloudtoScale

§ SometesttypescanconsumeallofyourJenkinsWorker’sresources

§ Sometimes,theamountoftestsissimplytoohighforoneWorker

§ Youshortenedthebuildtimeandnoweveryonewantstobuild…


OneWorkertoTestThemAll

§ TheJenkinsWorkerholdsthefilesandconfigurations

§ TheWorkerstartthetestsusingcloudresourcestoscaleit

§ ResultsaretransmittedtobacktotheWorker

§ BasedontheresultstheWorkerdecides:– Lunchingthenextpackoftests– Stoppingthebuildduetobadresults– Markthebuildassuccessfulandcontinuetothenextstep


UsetheForce CloudtoScale


Seleniumgeo-locationscale


OneSmallThingBeforeWeGo






1

2

3

4




GettingTaurus

§ Overall,weinstallTaurusbyusingpip.

§ InformationperOScanbefoundhere:http://gettaurus.org/install/Installation/


GetyourTaurusupandrunning

§ Plugyourflashdrive

§ InstallVirtualBox

§ Installvagrant(.dmg file)

§ Openterminalandcreateafolder

§ Gotothatfolderanddovagrantboxaddmy-Taurus-boxfile:///d:/path/to/taurus/box.box


GetyourTaurusupandrunning

§ Onthatfolderdovagrantinit my-Taurus-box

§ Vagrantup

§ Vagrantssh

§ Onceyou’reinsidethevagrant,docdTaurusScript

§ Do:bzt TaurusScenario.yml


TheChallenge

1. RunaJMeterscriptwithTaurus

2. Createyourown(simple)YMLDSLfileandrunitwithTaurus

3. RunaGatlingtestwithTaurus

4. ParallelthemJ

5. RunyourYMLDSLwith3differentexecutors(JMeter,GatlingandSelenium)andseethedifferencebetweenthem

6. Use–reporttoseetheBlazeMeter’sreport


Questions?DO5X53E


RecommendedSessions

SESSION# TITLE DATE/TIME

DO5T03PLeadershipPanel:ContinuousDeliveryintheFinancialServicesIndustry

11/16/2016at04:30pm

DO5T14SAnalystKeynote:ContinuousDelivery:MakingDevOpsAwesome

11/17/2016at10:30am

DO5X56SNext-GenerationPerformanceTestingwithBlazeMeter,ServiceVirtualizationandTestDataManagement

11/17/2016at04:30pm


MustSeeDemos

ModernizeAppDeliveryIntegratedCDTheater5- DOV501P

DeliverTestDataFasterTestDataManagerTheater5- DOV511P

DeliverBetterAppsServiceVirtualizationTheater5- DOV507P

OrchestrateYourReleaseReleaseAutomationTheater5- DOV513P

decrease test build time, not test quality with ca blazemeter

Technology