selenimu commiter day 2017: methods to sustain long term operations of e2e automated tests

Copyright©DeNACo.,Ltd.AllRightsReserved.Copyright©DeNACo.,Ltd.AllRightsReserved.

MethodstoSustainLongTermOpera5onsofE2EAutomatedTests

SeleniumCommi<erDay2017July1st2017

SWETGroupToshiyaKomoda(@toshiya_komoda)

Copyright©DeNACo.,Ltd.AllRightsReserved.

Bio

■  ToshiyaKomoda

⁃  Serversideengineer⁃  TestEngineer

■  developedgamepla?orms,usingautomatedtests

■  currentlyworkinginSWETgroup

⁃  SWET-SoFwareEngineerinTest

⁃  dedicatedteamfortestengineering


mobagegameplaMorm

■  Mainlyabrowsergamepla?orms⁃  tensofmillionsusersplaygamesonthepla?orm

■  Avarietyofservicesfor3rdpartydevelopers⁃  OpenIDConnect⁃  paymentservice⁃  Communityservice

•  friend,chat⁃  andmore


HistoryofmobagegameplaMorms

■  BusinesstrendshiFsto⁃  browsertonaNve⁃  pla?ormtogamepublisher

■  SNll⁃  alotofusersplaygamesonourpla?orms⁃  theprofitishuge

■  Ourgoal⁃  maintainhighavailabilityandstrictbackwardcompaNbility⁃  withminimummaintenancecost

automatedE2Etestsuiteshasbeenkeytoachievethisgoal


Somesta5cs

■  5⁃  yearsoperaNons

■  22⁃  testsuites⁃  groupedbasedonthefuncNonality

■  700⁃ browsertestcases

■ 60000⁃ webAPItestcases

■ enabledeveloperstoreleaseservercodeswithoutmanualQA

⁃  reducelead5me

⁃  reduceQACost


Lifecycleoftestsuites

■  DevelopmentPhase

⁃  buildE2EtestsasQAforservicedevelopment

■  HealthyPhase⁃  alltestsuitesarestableenough

■  BrokenPhase⁃  environmentalchanges,testdatacorrupNon

⁃  increaseoftheraNoofunstabletests…

■  DeadPhase⁃  noonecanpasstheE2Etests⁃  QAcostandleadtimeincreasealot


Lifecycleoftestsuites

■  DevelopmentPhase

⁃  buildE2EtestsasQAforservicedevelopment

■  HealthyPhase⁃  alltestsuitesarestableenough

■  BrokenPhase⁃  environmentalchanges,testdatacorrupNon

⁃  increaseoftheraNoofunstabletests…

■  DeadPhase⁃  noonecanpasstheE2Etests⁃  QAcostandleadtimeincreasealot

Gain > maintenance cost


Methods to keep tests healthy


1.Periodicexecu5onofalltestsuites

■ Goal⁃  improvestabilityoftestsuites⁃ minimizecosttofixbrokentests

■ executealltestsuitesinallenvironments

■ weexecutestestsonceinaday

■ teststabilityisrequired

■ findbrokentestsasmuchaspossible


2.Defineclearprocesstoexcludetestcases

■  Goal⁃ Controlcostandcoveragetrade-off

■  AllowtoexcludeandreNretestcases⁃ highlyflakytestcases⁃ brokentestcasesthatarehardtofix

■ Exampleprocess⁃  idenNfythereasonofthetestfailure

•  offcourse,ifthereisabuginproducNonservice,fixit.

⁃  ifthereasonisnotabuginproducNonservice⁃ makeJIRANcketsandexcludethetestcases


3.Improvetestdebuggability

■  Goal⁃  minimizetestmaintenancecost⁃  maximizethenumberofengineerswhocanseetests

■  Testlanguage⁃  shouldbefamiliarlanguagetoservicedevelopers

■  TestInfrastructureshouldprovide⁃  performance⁃  retrymechanism⁃  precisedebuglog


3.Improvetestdebuggability

■  LivepreviewandvideorecordsforUItests■  zalenium

⁃  h]ps://github.com/zalaxndo/zalenium⁃  developedbyzaland⁃  Aseleniumgridextension


4.MaketestslooselycoupledwithSUT(systemundertest)

■  Goal⁃  maketestshardtobreak⁃  maketestsrobusttochangesofSUT

■ WebAPItestsarerelaNvelyeasytomaintain.Why?⁃  clearinterfacebetweentestsandSUT

■  UItestsarenot.Why?⁃  hard-codedpagelinkinformaNon⁃  hard-codedDOMidorothera]ributesvalues

■  CanwemakeUItestslooselycoupledwithSUTasAPItests?⁃  2ongoingprojectswillbeshown


4.aVisualregressiontes5ngwithsitecrawler

■  Goal⁃  removehard-codedlinkinformaNonfromUItestcodes

■  usecrawlertechniquesinregressiontests⁃  detectlinksinwebpagesautomaNcally⁃  takescreenshotsofallpagesinthesite

2-up swipe


4.bInputtopiciden5fica5onwithmachinelearning

■  Goal⁃  removehard-codedDOMa]ributesvaluesfromUItestcodes

■  usenaturallanguageprocessingtechniquetohandleinputforms⁃  basedonaresearchpaper「UsingSemanNcSimilarityforInputTopicIdenNficaNoninCrawling-basedWebApplicaNonTesNng」,Jun-WeiLin,FarnWang

•  h]ps://arxiv.org/abs/1608.06549

■  ourprototypeshowspromisingresults

⁃  canesNmate80%ofinputforms

⁃  experimentsforloginanduserregisterforms

■  canbeusedwiththeprevioussitecrawler


Summary:MethodstoSustainLongTermOpera5onsofE2EAutomatedTests

1.  PeriodicexecuNonofalltestsuites

2.  Defineclearprocesstoexcludetestcases

3.  Improvetestdebuggability

4.  maketestslooselycoupledwithSUT(systemundertest)

selenimu commiter day 2017: methods to sustain long term operations of e2e automated tests

Technology