agile test automation
TRANSCRIPT
![Page 1: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/1.jpg)
Bringing Agility in Test Automations
Hemant Kumar JaggiSenior Manager, MakeMyTrip.com
![Page 2: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/2.jpg)
LET’S BE
AGILE !
Heard this before?
![Page 3: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/3.jpg)
![Page 4: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/4.jpg)
Thoughts on test automations the “Agile” way
To keep up the pace with agile application development
Traditional Test Automation approach isn’t working
Best practices must be incorporated everywhere
Ideally suited to an Ecommerce (or similar) environment
![Page 5: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/5.jpg)
An Ecommerce looks like
![Page 6: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/6.jpg)
Only thing that doesn’t change is “change”
![Page 7: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/7.jpg)
Building Test Automations the “Agile” way
![Page 8: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/8.jpg)
WHY?
![Page 9: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/9.jpg)
BECAUSE TRADITIONAL
AUTOMATION APPROACH JUST DIDN’T
WORK FOR US
![Page 10: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/10.jpg)
WHY? Challenges we face @ MakeMyTrip
Fast PaceReleases in a week?Monthly unique visitors?Monthly page views?
High Maintenance Cost & Test cases not always in “shape”
![Page 11: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/11.jpg)
Challenges (contd.)
Complex Business RulesHotel promotion combinations?Cheapest Flights search?
Multiple GDS connections
Hotel Search engine?~10,500 Domestic Hotels~250,000 hotels across world
It’s impossible to reverse-engineer!
![Page 12: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/12.jpg)
Challenges (contd.)
Execution issues:Automation execution may take 2-3
days to completeGolden data set takes ~7 days to
buildAutomation teams don’t know
domainFinally .. W
e don’t
trust
Autom
ation c
ode
![Page 13: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/13.jpg)
Life of a QA Engineer is like
![Page 14: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/14.jpg)
WHAT?
![Page 15: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/15.jpg)
WHAT it should be?
Faster time to develop Automation
LOW maintenance costAgnostic to business logic“QA team” friendlyLesser complexityFaster execution time
“It just works ! ”
![Page 16: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/16.jpg)
HOW?
![Page 17: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/17.jpg)
![Page 18: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/18.jpg)
HOW? In lines with Agile Manifesto..
Break into smaller storiesPrioritize them basis
Functional ImportanceFrequency of executionsDependencies
Work “with” QA teamDevelop a story and deliver
![Page 19: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/19.jpg)
Create Buckets
Sanity testsEnd to end user flowsRegression tests*
Miscellaneous flows
![Page 20: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/20.jpg)
Sanity Test
Breadth wise test coverage.MATS – Minimal Acceptance Test
SuiteRelease checklistDetermines whether build is
stable.
![Page 21: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/21.jpg)
User Flows
Covers end to end user flowsDetailed breadth wise coverageE.g. different combinations of
booking flows.
![Page 22: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/22.jpg)
But how to simplify regression tests?
Break them further ..
Into smaller flows, e.g. page wise test suites
Comparative TestingInteractive Testing
![Page 23: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/23.jpg)
Comparative Testing
![Page 24: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/24.jpg)
Comparative Testing
Works on two builds:“QA” build (i.e. build under test)“Production” build
Deploy them in same environment.
Read all ‘to-be-tested’ values from front end
![Page 25: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/25.jpg)
Comparative Testing
Execute automation script on both builds
Dump them into XML (or excel)Excel template has formulae to
compare values from two buildsOr use an XML comparator (in
case of XML output format)
![Page 26: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/26.jpg)
Comparative Testing
IF (Actual == Expected) THEN PASS
ELSE FAIL* Actual Value = value(QA build)* Expected Value = value(live build)
Take XML(or excel) as input to reporting framework, generate HTML reports
![Page 27: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/27.jpg)
Let’s discuss
Pros & Cons of
Comparative Testing
![Page 28: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/28.jpg)
Advantages of Comparative Testing
Faster turn around to developAutomation script does just what it is
supposed to doAutomation becomes a “dumb”
executionVery LOW maintenance costKeep it simple
![Page 29: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/29.jpg)
Advantages of Comparative Testing
Agnostic to business layersChanges in business rules don’t
matter
High level of accuracyMakes it “thorough”Enhances confidence in QA team
![Page 30: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/30.jpg)
Advantages of Comparative Testing
Overall progress is fasterMore dependency on Test Scenarios
than Test CasesLess/No domain knowledge required
for Automation team“Win-Win” for Automation and QA
team
![Page 31: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/31.jpg)
Disadvantages of Comparative Testing
Any “live” issues won’t be caught
Execution requiresEither double the time (if sequential)Or double the resources (if parallel)
![Page 32: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/32.jpg)
Interactive Testing
![Page 33: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/33.jpg)
Interactive Testing
Complements comparative testing
Features that require user interaction
E.g. Filters, Sorting etc.
Comparative + Interactive equals “Regression”
![Page 34: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/34.jpg)
Automation Coverage Tracking
Assign weights to each bucket, say
Automation Score = sum([(Tn/An)*100]*[Wn/(W1+W2+W3+W4+W5)]
Tn=Total Automated TC’s for a bucketAn=Total Automation feasible TC’s for a bucketWn=Weighted score of a bucket
![Page 35: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/35.jpg)
Automation Coverage Tracking (Sample)
![Page 36: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/36.jpg)
Computing efficiency gains (Sample)
![Page 37: Agile test automation](https://reader036.vdocuments.mx/reader036/viewer/2022081512/5585289ad8b42a86388b4578/html5/thumbnails/37.jpg)
Take Away
The “Agile” way of building Test Automations
We resolved “challenges” of traditional automation approach
Simplifying regressionRegression=Comparative + Interactive
Its working @ MakeMyTrip