the what, why and how of web analytics testing
DESCRIPTION
Slides from my talk in UNICOM's Next Generation Testing Conference on 13th December in Bangalore on "The What, Why and How of Web Analytics Testing". This is based on my open-source tool - WAAT. More information about the talk is available here: http://goo.gl/FxISG Information about WAAT is available here: http://goo.gl/oUNHUTRANSCRIPT
![Page 1: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/1.jpg)
The What, Why & How of
Web Analy3cs Tes3ng
Anand Bagmar So#ware Quality Evangelist
Principal Consultant, ThoughtWorks
![Page 2: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/2.jpg)
Why do we do Tes3ng?
![Page 3: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/3.jpg)
What is Web Analy3cs?
![Page 4: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/4.jpg)
Why do we need it?
![Page 5: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/5.jpg)
Web Analy*cs Solu*on Providers
Adobe Marke*ng Cloud / Omniture
SiteCatalyst
WebTrends
AWStats
Google Analy*cs
StatCounter
…
![Page 6: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/6.jpg)
How to test Web Analy3cs?
![Page 7: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/7.jpg)
The end report
![Page 8: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/8.jpg)
![Page 9: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/9.jpg)
![Page 10: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/10.jpg)
Pros
• Ensure report is setup correctly
Cons • Licensing • Reports not yet setup • Validate all requests are sent / captured
Tes3ng at the report level
![Page 11: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/11.jpg)
At the origin!
![Page 12: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/12.jpg)
![Page 13: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/13.jpg)
![Page 14: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/14.jpg)
The problem
![Page 15: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/15.jpg)
Manual
![Page 16: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/16.jpg)
The challenge
![Page 17: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/17.jpg)
Repea3ng over-‐and-‐over again
![Page 18: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/18.jpg)
![Page 19: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/19.jpg)
The solu3on
![Page 20: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/20.jpg)
What does WAAT do for me?
• Minimal changes to exisFng test framework • Does not depend on junit / testng / … • Web AnalyFc tool independent • UI Driver framework independent • Browser independent
![Page 21: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/21.jpg)
Flavors
![Page 22: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/22.jpg)
Architecture
![Page 23: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/23.jpg)
• Original flavor • Source code + jar • Supports 3 plugins • Omniture Debugger
• HUpSniffer • JsSniffer
• Available on github
WAAT -‐ Java
![Page 24: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/24.jpg)
• Ruby gem implemented over WAAT-‐Java
• Uses RJB – Ruby-‐Java-‐Bridge • Supports 2 plugins • HUpSniffer • JsSniffer
• Available on github and rubygems.org
WAAT -‐ Ruby
![Page 25: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/25.jpg)
Omniture Debugger
![Page 26: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/26.jpg)
Omniture Debugger How does it work? Pros • OS independent • Run using the regular test-‐user Cons • Browser dependent – need to implement
ScriptRunner for the UI-‐driver in use • Web-‐AnalyFc soluFon dependent – Adobe MarkeFng
Cloud / Omniture SiteCatalyst
![Page 27: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/27.jpg)
HHpSniffer How does it work? Pros • Web-‐analyFc soluFon independent • Browser independent • UI-‐driver independent Cons • 3rd party libraries are OS Dependent • HTTPs not supported out-‐of-‐the-‐box • Run tests as ‘root’
![Page 28: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/28.jpg)
JsSniffer How does it work? Pros • Web-‐analyFc soluFon independent • Browser independent • HTTPs supported out-‐of-‐the-‐box • No dependency on any 3rd party library Cons • Need to write javascript to get the URL from the
browser • UI-‐driver dependent
![Page 29: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/29.jpg)
• Download zip from github / rubygems.org
• Setup
• jpcap / ScriptRunner setup
• Update exis*ng tests
How to use WAAT?
![Page 30: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/30.jpg)
Update exis3ng Tests • Import relevant packages / gems
• Define & provide parameter values
• Ini*alize Engine
• Enable Web Analy*cs Tes*ng
• Define Test Data
• Verify Web Analy*cs repor*ng using WAAT
• Valida*ons
• Disable Web Analy*cs Tes*ng
![Page 31: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/31.jpg)
<Section !!!actionName="OpenUpcomingPage_HttpSniffer"!!numberOfEventsTriggered="1"!
!tagList="pageName=upcoming:index||!! ! !cc=USD||!! ! !ch=upcoming||!! ! !events=event2||"/>!
Define Test Data
![Page 32: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/32.jpg)
Ini3alize Engine
!engine = getInstance(!! ! !webAnalyticTool, !! ! !inputFileType, !! ! !keepLoadedFileInMemory, !!! ! !log4jPropertiesAbsoluteFilePath!! ! !);!
![Page 33: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/33.jpg)
Enable Web Analy3cs Tes3ng
engine.enableWebAnalyticsTesting();!
![Page 34: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/34.jpg)
Verify Web Analy3cs Repor3ng
!Result verificationResult = !!! !engine.verifyWebAnalyticsData (!! ! !inputDataFileName, !! ! !actionName, !! ! !urlPatterns, !! ! !minimumNumberOfPackets);!
Cart.checkout();!
![Page 35: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/35.jpg)
Sample Valida3ons assertNotNull(verificationResult.getVerificationStatus(), "Verification status should NOT be NULL"); !!assertNotNull(verificationResult.getListOfErrors(), "Failure details should NOT be NULL"); logVerificationErrors(verificationResult); !!assertEquals(verificationResult.getVerificationStatus(), Status.PASS, "Verification status should be PASS"); !!assertEquals(verificationResult.getListOfErrors().size(), 0, "Failure details should be empty");!
![Page 36: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/36.jpg)
Disable Web Analy3cs Tes3ng
engine.disableWebAnalyticsTesting();!
![Page 37: The what, why and how of web analytics testing](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554f41f4b4c905cd048b54ba/html5/thumbnails/37.jpg)
hUp://essenceo#esFng.blogspot.com/search/label/waat
hUps://github.com/anandbagmar/
hUp://rubygems.org/gems/WAAT
[email protected] [email protected]
@BagmarAnand