youtube ads and video - headspin studies/youtube/youtube-ads-vi… · kpi and qoe metrics. ......
TRANSCRIPT
![Page 1: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/1.jpg)
YouTube Ads and Video KPI and QoE Metrics
![Page 2: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/2.jpg)
Test Environment
• MotorolaE2inIndia(Bangalore,OuterRingRoad,1pmIST)1. Lenovo-Motorolahas9%marketshareinIndia2. Android5.0.13. Vodafone4G(Vodafonehas18%marketshareinIndia)
• Nexus6inIndia(Bangalore,OuterRingRoad,1pmIST)1. Android7.1.12. Airtel3G(Airtelhas25%marketshareinIndia)
• SamsungS7inUnitedStates(MountainViewDowntown,1pmPST)1. S7hasa10%marketshareinUSA2. AT&TLTE(AT&Thas33%marketshareinUSA)3. Android7.0
• NumberofIterations- 10 HeadSpin, Inc. Conf ident ial
![Page 3: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/3.jpg)
Test Case
• LoadaYouTubeVideothathasanembeddedAD• MeasurethetimetoloadtheAD• MeasurethetimetoloadthevideofollowingtheAD• MeasuretheVideoQoE MetricsforAD• MeasuretheVideoQoE MetricsforVideo• MeasuretheAudioQoE MetricsforAD• MeasuretheAudioQoE MetricsforVideo
HeadSpin, Inc. Conf ident ial
![Page 4: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/4.jpg)
Test Results
• AllTestresultscollectedandstoredinMySql• AlltheresultsexportedtoGoogleSheets• ResultsAvailableat:https://goo.gl/fVWj96• TestSamplesinGoogleDrive:https://goo.gl/kbse5W
HeadSpin, Inc. Conf ident ial
![Page 5: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/5.jpg)
Video Quality of Experience Measurements
• UsingPEVQ(PerceptualEvaluationforVideoQuality)forendtoendmeasurement
• Forthisproofofconcepttests,thefollowingvideometricswerecaptured• MeanOpinionScore(MOS)• DifferentialMOS• FrameFreeze(%)• Jerkiness• Blur• Blockiness• PeakNoisetoSignalRation(PSNR)
HeadSpin, Inc. Conf ident ial
![Page 6: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/6.jpg)
Audio Quality of Experience Measurements
• UsingPOLQA(PerceptualObjectiveListeningQualityAnalysis)forendtoendmeasurement
• Forthisproofofconcepttests,thefollowingaudiometricswerecaptured• MOS-LQ0• R-Factor• SNR(SignaltoNoiseRatio)
HeadSpin, Inc. Conf ident ial
![Page 7: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/7.jpg)
HeadSpin, Inc. Conf ident ial
Video Mean Opinion Score and Differential MOS
• MOSScoreisonascaleof• 1Unacceptable• 2Poor• 3Fair• 4Good• 5Excellent
• DifferentialMOSisthedifferencebetweenreferenceandprocessedMOS.Thescoreisonascaleof
• 3.1- 4.0MostUsersDissatisfied• 2.1- 3.0ManyUsersDissatisfied• 1.1- 2.0SomeUsersSatisfied• 0.7- 1.0MostUsersSatisfied• 0.0- 0.6VerySatisfied
![Page 8: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/8.jpg)
© 2018 HeadSpin - Wipro Confidential
Audio Mean Opinion Score and R-Factor
• MOS-LQ0 Scoreisonascaleof• 0– Worst• 4.5- Best
• R-Factorisonthefollowingscale• <60Unacceptable• 0– Bad• 100- Best
![Page 9: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/9.jpg)
Test Device/Carrier MOS DMOS Frame Freeze (%)
Jerkiness
Moto E2 Vodafone 4G (Video) 2.627 2.373 0 1.522Moto E2 Vodafone 4G (Ads) 3.122 1.614 96 3.968
Moto E2 Vodafone 3G (Video) 2.799 2.201 32 3.102Moto E2 Vodafone 3G (Ads) 2.893 2.107 12.8 0.428
Nexus 6 Airtel 3G (Video) 2.723 2.277 124 3.419Nexus 6 Airtel 3G (Ads) 2.958 2.042 78.4 1.007
Video Mean Opinion Score (MOS), Differential MOS, Frame Freeze and Jerkiness (India)
HeadSpin, Inc. Conf ident ial
PoorexperiencehighlightedinRED
![Page 10: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/10.jpg)
© 2018 HeadSpin - Wipro Confidential
Video Example (India)
Blurry Video Quality
![Page 11: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/11.jpg)
© 2018 HeadSpin - Wipro Confidential
Ad Example (India)
Blurry Video
![Page 12: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/12.jpg)
© 2018 HeadSpin - Wipro Confidential
Ad Example (India)
Grainy Video with Jerkiness
![Page 13: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/13.jpg)
Test Device/Carrier MOS DMOS Frame Freeze (%)
Jerkiness
S7 AT&T 4G (Video) 3.752 1.248 124 2.65S7 AT&T 4G (Ads) 3.799 1.201 0 1.115
S7 AT&T HSPAP (Video) 3.386 1.614 96 3.968S7 AT&T HSPAP (Ads) 3.631 1.369 152.35 1.899
Video Mean Opinion Score (MOS), Differential MOS, Frame Freeze and Jerkiness (USA)
HeadSpin, Inc. Conf ident ial
![Page 14: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/14.jpg)
© 2018 HeadSpin - Wipro Confidential
Video Example (USA)
Grainy Video
![Page 15: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/15.jpg)
© 2018 HeadSpin - Wipro Confidential
Video Example (USA)
Grainy and Blurry Video
![Page 16: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/16.jpg)
What is Blockiness, Blur and PSNR?
• Blockinessisoftentheresultofalowbitratecodingthatusesablockmatchingalgorithmforthemotionestimationandacoarsequantizationfortheimageblocks.
• Blurisadistortioncharacterizedbyreducedsharpnessofcontouredgesandspatialdetail.
• PeakSignaltoNoiseRatio(PSNR):ToallowforacoarseanalysisofdistortionsindifferentdomainsthePSNRisprovidedfortheY,CbandCrcomponentsseparately.
• Source- http://www.opticom.de/technology/pevq.php
HeadSpin, Inc. Conf ident ial
![Page 17: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/17.jpg)
Test Device/Carrier MOS-LQ0 R-Factor
Moto E2 Vodafone 4G (Video) 2.9778 58Moto E2 Vodafone 4G (Ads) 1.0064 7
Moto E2 Vodafone 3G (Video) 1.0161 8Moto E2 Vodafone 3G (Ads) 1 0
Nexus 6 Airtel 3G (Video) 2.32 45Nexus 6 Airtel 3G (Ads) 1 0
Audio QoE Scores (India)
HeadSpin, Inc. Conf ident ial
PoorexperiencehighlightedinRED
![Page 18: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/18.jpg)
Test Device/Carrier MOS-LQ0 R-Factor
S7 AT&T 4G (Video) 1.12255 15S7 AT&T 4G (Ads) 4.2314 86
S7 AT&T HSPAP (Video) 1.0744 13S7 AT&T HSPAP (Ads) 4.5 100
HeadSpin, Inc. Conf ident ial
PoorexperiencehighlightedinRED
Audio QoE Scores (USA)
![Page 19: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/19.jpg)
Test Device/Carrier Time to load Ad (Seconds)
Time to load Video (Seconds)
S7 AT&T 4G 3 4S7 AT&T HSPAP 6 5
Moto E2 Vodafone 4G 4 3Moto E2 Vodafone 3G 5 3
Nexus 6 Airtel 3G 4 4
YouTube Ads and Video KPI
HeadSpin, Inc. Conf ident ial
![Page 20: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/20.jpg)
YouTube Ads and Video KPI
HeadSpin, Inc. Conf ident ial
![Page 21: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/21.jpg)
HeadSpin, Inc. confidential
q Test and measure QoE KPI/Metrics across multiple iOS and Android SKUs in different cities§ Devices, OS and Carriers coverage based on market share %
q Test every build, code change and releaseq Identify problems before launching appq Supercharge Development and troubleshooting
§ Use USBoverIP software bridge to import devices into XCODE and Android Studioq Automatically get issues from HeadSpin’s AI Engine
§ Collect data to identify the problems with Image management, slow backend and third partyq Run the tests continuously in multiple locations on Stable hardware
§ HeadSpin has POPs in 150+ major worldwide citiesq Proactive alerts when the production app QoE issuesq Find bugs early, launch high quality version of the app!q Compare with other apps like FB Video, Netflix, Amazon Prime Video, Disney etc.
FutureWork
![Page 22: YouTube Ads and Video - HeadSpin studies/YouTube/YouTube-Ads-Vi… · KPI and QoE Metrics. ... (Airtel has 25% market share in India) • Samsung S7 in United States (Mountain View](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8d9fa260e2106d4941e4b1/html5/thumbnails/22.jpg)
https://blogs.akamai.com/2017/04/the-state-of-mobile-app-performance.html
2017 State of Mobile Performance Report