embedding performance engineering in cicd · 2018-10-30 · •run paralleltests •run continuous...
TRANSCRIPT
![Page 1: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/1.jpg)
T16 Performance Testing Thursday, May 3rd, 2018 1:30 PM
Embedding Performance Engineering into the CI/CD Pipeline
Presented by:
Anjeneya Dubey
McGraw-Hill Education
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-- -268- - -8770 ·· 904- --278-- -0524 - [email protected] - http://www.stareast.techwell.com/
![Page 2: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/2.jpg)
Anjeneya Dubey McGraw-Hill Education Anjeneya Dubey is the director of performance engineering for McGraw-Hill Education, a learning science company that delivers personalized learning experiences. His responsibilities include ensuring that every product built is high performing, highly scalable, highly available, highly reliable, and fault tolerant. In his past five years with McGraw-Hill, Anjeneya has built automated performance engineering frameworks that detect performance and scalability issues early on in a fast-paced agile environment. Previously he was a technology consultant, focused on providing enterprise quality and performance engineering solutions. Anjeneya has worked with large institutions to set-up enterprise performance and quality engineering solutions.
![Page 3: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/3.jpg)
3/13/18
1
Embedding Performance Engineering Into Continuous Integration & Continuous Delivery Pipeline
By – Anjeneya Dubey
LittleContextaboutMcGraw-HillEducationandMe
AnjeneyaDubeyDirectorofPerformanceEngineeringAnjeneya.dubey@mheducation.com
SoftwareengineeringPerformanceEngineeringCapacityEngineeringInfrastructurePlanningandimplementationAWSCloudArchitecture&OperationsSiteReliabilityEngineering
2
![Page 4: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/4.jpg)
3/13/18
2
Agenda
• ContinuousIntegrationandContinuousDelivery• WhatdoesitmeantoincludeperformanceengineeringintotheCICDPipeline• Challenges• Whatdidwedotoincludeperformanceengineeringinthepipeline
• Processchanges• Performancetesttypes• TestEnvironmentmanagement• TestDatamanagement• ToolsandTechnologiesweuse• Pass/failDecisionMaking
• SelfServicePerformanceEngineering• UsingAIinproduction• Do’sandDon’ts• Summary
3
• AutomatedbuildprocessandbuildverificationtestsforeachenvironmentinContinuousintegration• ExtendContinuousintegrationbyrapidlydeployingcapabilitiestouserstogaincompetitiveadvantage• Reducetestcycletime&timetomarket• Highlyautomatedtesting&release/roll-back• Quickerautomateddecisionmaking&feedbackloop
ContinuousIntegration/ContinuousDelivery
4
![Page 5: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/5.jpg)
3/13/18
3
EmbeddingPerformanceintothePipelineYourpipelineascode• Dev->Test->Prod• Dev->Test->Performance->Prod
Whatdoesitmean?AddingPerformanceenvironmentintothepipelinemeansthatnowtheperformancetestsareblockingyourcodepromotion
Dev Test Production
CodePromotion
Feedback
Performance
5
Challenges- Cultural
• Performanceisanafterthought• Isnotpartoftheagileteams• Isnotpartofthequalityteams• Donotgetincludedintheagileceremonies• Createawarenessonperformancetasks• Empowerdevtotest
6
![Page 6: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/6.jpg)
3/13/18
4
Challenges- Technical
• Automatingtheperformancetestingandanalysis• ReducingTimetoprepareandexecutetest• Quicklyreactingtoperformancemetrics• AutomaticPass/Fail• Scalingtheloadtesttoolforvarietyoftests• Keepthetestingenv/dataconsistent• WhattoShiftleftwhattoshiftright• Costofrunningperformancetestoneverybuild
7
Howdowedoit@MHE?
8
![Page 7: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/7.jpg)
3/13/18
5
Processchanges
• Makenonfunctional/Performancerequirementaspartofthefunctionalrequirement• APIcontracts• Includeperformanceaspartofdefinitionofdoneforsprints• Cleardefinitionofperformancereadyproduct• DiscussPerformanceresultsaspartofthesprintdemoswithallstakeholders
9
PerformanceRequirementsWorkflow
Storieswithacceptancecriteriathatincludesclearperformancerequirements• APIXmusthandleloadofxxtransactionspersecwith95%ileresponsetimeas100ms• AllStoriesmustbeevaluatediftheyrequireperformancecriteria• Performancetestsshouldbecreatedtovalidatethecriteriawithinthesprints• PoorPerformance=FunctionalBug
10
![Page 8: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/8.jpg)
3/13/18
6
TypicalPEProcess
11
TestEnvironment
• Useproductionlikeperformanceenv• Spinuponlywhenyouruntesttosavecost• RefreshDBsfortestdatamanagement
12
![Page 9: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/9.jpg)
3/13/18
7
• Canexpandandcontract- Autoscaling• Infrastructureascode– Terraform,Puppet• Creating&destroyingenvs atease• Createparallelenvs forparallelexecutions
Cloudmakesiteasier
13
Spinupparallelenvs forparallelexecutions
PerfEnv A
PerfEnv B
• Productioncapacityinstance
• ProtocolLevelfullloadtest
• UIperformancetestusingfunctionaltestscripts• Scaleddownenv
• Stress/capacitytest
PerfEnv C
• Testingoutsidethepipeline
• Troubleshooting• Benchmarking/baselining
tests
14
![Page 10: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/10.jpg)
3/13/18
8
PerformanceTesttypesinCICD
• UserExperience- Browsersideperformance• Loadtests• Capacity/StressTests
15
SingleuserperformanceGoodUX=CustomerHappyHowdowemeasurethat?• Collectsingleuserbrowsersideresponsetimes• Leveragefunctionaltestscripts(selenium)• Createscenariosthatyouwanttomeasurethroughourselfserviceautomationframework• AllMethodsinthescriptshavethesnippetthatcollectstheresponsetimes• Executedfromvariousgeolocations• Usabletimevslastbyte• CollectingHAR&Creatingvideosofthetestsforofflineanalysis• UploadthedatatoS3• MHEPerformancePlatformtakesoverfromthere
16
![Page 11: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/11.jpg)
3/13/18
9
Loadtests
• Fullloadtests• ScaledDowntests• Stresstesttofindcapacity
17
FeatureFlags
Whattodowhenyoufindperformanceissues?• Blocktherelease• TurnOffthefeaturethatcreatestheperformanceissue
18
![Page 12: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/12.jpg)
3/13/18
10
TestDatamanagement
• Makeourtestsselfcontained• Create&destroydataaspartofthetestasmuchaspossible
• Fortheonesyoucantcreateduringthetestyoucreateaspartoftheenvironmentbuildout• SpinupparallelAuroraRDSwithpreseededtestdatatospeedupenv buildout
19
Tools&Technologiesweuse
20
![Page 13: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/13.jpg)
3/13/18
11
PerformanceEngineeringPlatform
• Singularplatformtomanageperformancelifecycleforallofourproducts• PowersCICDforPerformanceengineering• Centralrepoforallmetrics• Dynamicthresholds• Passfaildecisionmaking• PowersSelfServicePerformanceEngineering
21
PEPlatformOverview
CollectorServiceAggregatorService
CentralRepository
Reporting&AlertingService
Developer
22
![Page 14: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/14.jpg)
3/13/18
12
PEPlatform– Performancetesttypes
23
Trending– Performancegraphforeachbuild
Build3383on2/28isfailedfor
anAPI
Build3362on2/26isbrokenforanAPIastheresponsetime
degradedalmost30%
24
![Page 15: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/15.jpg)
3/13/18
13
ContainerizeJMeter
• WeuseJMeterheavilyforthealltheCICDtesting• Distributedloadtesting– weneed1master&Nnumberofslavestogeneratehugeload• ScalingtheJMeterforthousandsofuserswasachallenge• Dockerize JMetergivesthescaleneeded• Speedsuptheprovisioning• Partoftheinfraascode– whichmeanswhenthecodegetsdeployedautomaticallyJMeterfarmgetsprovisionedwherethetestgetsexecuted
25
AutomatedPass/Fail
Basedon3basicrules
• Simple&Easy• Implementable• Dependentonthroughput,responsetimesandsystemKPIs
26
![Page 16: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/16.jpg)
3/13/18
14
Thresholdsforpass/fail
• StaticBusinessresponsetimesSLAs• Dynamicuserexperience/APIlevelResponsetimesthresholds• DynamicSystemResourceutilizationthresholds• BasedonhistoricaltrendforeachAPIandalertsifitdeviatedfromlastntests• AllowsseparatethresholdforeachAPI• Doesn’tallowslippageevenwithinthecontract
27
Self-ServicePerformanceEngineering
• Youdon’tneedtobeperformanceengineertoruntest• Automatetheentireperformancecycle• ScriptCreationthroughaUI• ExecutetestaspartofCICDorExecuteitondemandthroughvoiceenabledAlexaorachatBot• AnalysisthroughAPMandMHEbuiltPerformancePlatform• AutomatedNotificationthroughHipchat/Email/PagerDuty• AutomatedDefectcreationwithdetailsinjira
Tester Devops
Developer
PerformanceEngineer
28
![Page 17: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/17.jpg)
3/13/18
15
Self-ServicePerformanceEngineeringTest
Creation
CICD Alexa ChatBots
APM MHEPEPlatform
Hipchat Email Pagerduty
Execution
Analysis
Notificatio
nDe
fect
Jira29
Notifications
• Automateddefectcreation
• Summaryofthetestresult
• APMdashboardlinkswithdrilldown
• Automatedrealtimehipchatnotifications
• WithJiralinkanddetails
30
![Page 18: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/18.jpg)
3/13/18
16
ShiftRight- Anomalydetection
• TwitterAnomalyDetection• Twitter’sBreakoutDetection• PearsonCorrelationAlgorithm• K-MeansClustering• NewRelicRadar
31
Do’s&Don’ts
• Startwithsimple• Perfectitlater• Removefalsepositives- Getitrightfromthebeginning• KnowyourapplicationsKPIs• Runparallel tests• Runcontinuoustests
• Don’trunbenchmark&endurancetestinCI• Dont removethefailingteststopassthroughCD• Don’tkeepincreasingthethresholdstopasstests• Don’treinventyourPEframeworkratherseehowyoucanleverageyourexistingtoolsandframeworkinCICD
32
![Page 19: Embedding Performance Engineering in CICD · 2018-10-30 · •Run paralleltests •Run continuous tests •Don’t run benchmark & endurance test in CI •Dontremove the failing](https://reader036.vdocuments.mx/reader036/viewer/2022062919/5ee0135bad6a402d666b5432/html5/thumbnails/19.jpg)
3/13/18
17
Summary
• IncludeperformanceengineeringinyourCICDpipeline• Automateautomate&automate• Makeyourtestsrepeatable• Collectmetricsalongtheway• Avoidfalsepositives• Keepanalysis&decisionmakingsimple• Empowerdevs totest
33
Questions?
34