Download - selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests
Copyright©DeNACo.,Ltd.AllRightsReserved.Copyright©DeNACo.,Ltd.AllRightsReserved.
MethodstoSustainLongTermOpera5onsofE2EAutomatedTests
SeleniumCommi<erDay2017July1st2017
SWETGroupToshiyaKomoda(@toshiya_komoda)
Copyright©DeNACo.,Ltd.AllRightsReserved.
Bio
■ ToshiyaKomoda
⁃ Serversideengineer⁃ TestEngineer
■ developedgamepla?orms,usingautomatedtests
■ currentlyworkinginSWETgroup
⁃ SWET-SoFwareEngineerinTest
⁃ dedicatedteamfortestengineering
Copyright©DeNACo.,Ltd.AllRightsReserved.
mobagegameplaMorm
■ Mainlyabrowsergamepla?orms⁃ tensofmillionsusersplaygamesonthepla?orm
■ Avarietyofservicesfor3rdpartydevelopers⁃ OpenIDConnect⁃ paymentservice⁃ Communityservice
• friend,chat⁃ andmore
Copyright©DeNACo.,Ltd.AllRightsReserved.
HistoryofmobagegameplaMorms
■ BusinesstrendshiFsto⁃ browsertonaNve⁃ pla?ormtogamepublisher
■ SNll⁃ alotofusersplaygamesonourpla?orms⁃ theprofitishuge
■ Ourgoal⁃ maintainhighavailabilityandstrictbackwardcompaNbility⁃ withminimummaintenancecost
automatedE2Etestsuiteshasbeenkeytoachievethisgoal
Copyright©DeNACo.,Ltd.AllRightsReserved.
Somesta5cs
■ 5⁃ yearsoperaNons
■ 22⁃ testsuites⁃ groupedbasedonthefuncNonality
■ 700⁃ browsertestcases
■ 60000⁃ webAPItestcases
■ enabledeveloperstoreleaseservercodeswithoutmanualQA
⁃ reducelead5me
⁃ reduceQACost
Copyright©DeNACo.,Ltd.AllRightsReserved.
Lifecycleoftestsuites
■ DevelopmentPhase
⁃ buildE2EtestsasQAforservicedevelopment
■ HealthyPhase⁃ alltestsuitesarestableenough
■ BrokenPhase⁃ environmentalchanges,testdatacorrupNon
⁃ increaseoftheraNoofunstabletests…
■ DeadPhase⁃ noonecanpasstheE2Etests⁃ QAcostandleadtimeincreasealot
Copyright©DeNACo.,Ltd.AllRightsReserved.
Lifecycleoftestsuites
■ DevelopmentPhase
⁃ buildE2EtestsasQAforservicedevelopment
■ HealthyPhase⁃ alltestsuitesarestableenough
■ BrokenPhase⁃ environmentalchanges,testdatacorrupNon
⁃ increaseoftheraNoofunstabletests…
■ DeadPhase⁃ noonecanpasstheE2Etests⁃ QAcostandleadtimeincreasealot
Gain > maintenance cost
Copyright©DeNACo.,Ltd.AllRightsReserved.
Methods to keep tests healthy
Copyright©DeNACo.,Ltd.AllRightsReserved.
1.Periodicexecu5onofalltestsuites
■ Goal⁃ improvestabilityoftestsuites⁃ minimizecosttofixbrokentests
■ executealltestsuitesinallenvironments
■ weexecutestestsonceinaday
■ teststabilityisrequired
■ findbrokentestsasmuchaspossible
Copyright©DeNACo.,Ltd.AllRightsReserved.
2.Defineclearprocesstoexcludetestcases
■ Goal⁃ Controlcostandcoveragetrade-off
■ AllowtoexcludeandreNretestcases⁃ highlyflakytestcases⁃ brokentestcasesthatarehardtofix
■ Exampleprocess⁃ idenNfythereasonofthetestfailure
• offcourse,ifthereisabuginproducNonservice,fixit.
⁃ ifthereasonisnotabuginproducNonservice⁃ makeJIRANcketsandexcludethetestcases
Copyright©DeNACo.,Ltd.AllRightsReserved.
3.Improvetestdebuggability
■ Goal⁃ minimizetestmaintenancecost⁃ maximizethenumberofengineerswhocanseetests
■ Testlanguage⁃ shouldbefamiliarlanguagetoservicedevelopers
■ TestInfrastructureshouldprovide⁃ performance⁃ retrymechanism⁃ precisedebuglog
Copyright©DeNACo.,Ltd.AllRightsReserved.
3.Improvetestdebuggability
■ LivepreviewandvideorecordsforUItests■ zalenium
⁃ h]ps://github.com/zalaxndo/zalenium⁃ developedbyzaland⁃ Aseleniumgridextension
Copyright©DeNACo.,Ltd.AllRightsReserved.
4.MaketestslooselycoupledwithSUT(systemundertest)
■ Goal⁃ maketestshardtobreak⁃ maketestsrobusttochangesofSUT
■ WebAPItestsarerelaNvelyeasytomaintain.Why?⁃ clearinterfacebetweentestsandSUT
■ UItestsarenot.Why?⁃ hard-codedpagelinkinformaNon⁃ hard-codedDOMidorothera]ributesvalues
■ CanwemakeUItestslooselycoupledwithSUTasAPItests?⁃ 2ongoingprojectswillbeshown
Copyright©DeNACo.,Ltd.AllRightsReserved.
4.aVisualregressiontes5ngwithsitecrawler
■ Goal⁃ removehard-codedlinkinformaNonfromUItestcodes
■ usecrawlertechniquesinregressiontests⁃ detectlinksinwebpagesautomaNcally⁃ takescreenshotsofallpagesinthesite
2-up swipe
Copyright©DeNACo.,Ltd.AllRightsReserved.
4.bInputtopiciden5fica5onwithmachinelearning
■ Goal⁃ removehard-codedDOMa]ributesvaluesfromUItestcodes
■ usenaturallanguageprocessingtechniquetohandleinputforms⁃ basedonaresearchpaper「UsingSemanNcSimilarityforInputTopicIdenNficaNoninCrawling-basedWebApplicaNonTesNng」,Jun-WeiLin,FarnWang
• h]ps://arxiv.org/abs/1608.06549
■ ourprototypeshowspromisingresults
⁃ canesNmate80%ofinputforms
⁃ experimentsforloginanduserregisterforms
■ canbeusedwiththeprevioussitecrawler
Copyright©DeNACo.,Ltd.AllRightsReserved.
Summary:MethodstoSustainLongTermOpera5onsofE2EAutomatedTests
1. PeriodicexecuNonofalltestsuites
2. Defineclearprocesstoexcludetestcases
3. Improvetestdebuggability
4. maketestslooselycoupledwithSUT(systemundertest)