measuring and improving user experience - bdti.com · •measuring: speed, cost, energy efficiency...
TRANSCRIPT
![Page 1: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/1.jpg)
1 Copyright © 2015 BDTI
Jeff Bier | February 5, 2015
Measuring and Improving User Experience
VPQM 2015
![Page 2: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/2.jpg)
2 Copyright © 2015 BDTI
• 1994: Benchmarking processor cores for digital
signal processing applications
• Measuring: Speed, cost, energy efficiency
• But what about ease of use, support, roadmap
risk?
• 2014: Benchmarking smartphones
• Measuring: User experience
• But do we really know what
“user experience” is?
My Benchmarking Journey
![Page 3: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/3.jpg)
3 Copyright © 2015 BDTI
The performance (or energy efficiency, or user experience, or…) of a
system is not equal to the sum of the performance of the individual
elements
Pitfall: Whole vs. Parts
www.treehugger.com
![Page 4: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/4.jpg)
4 Copyright © 2015 BDTI
Pitfall: No Free Lunch
www.environmentteam.com
![Page 5: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/5.jpg)
5 Copyright © 2015 BDTI
For Example
![Page 6: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/6.jpg)
6 Copyright © 2015 BDTI
• Benchmarks should mirror the tasks that users actually perform
• Benchmarks should measure the metrics that users actually observe
• I.e., we should benchmark user experience
How Should We Benchmark Smartphones?
![Page 7: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/7.jpg)
7 Copyright © 2015 BDTI
• There are many types of users with different use patterns and
preferences… and regional variations
• Popular apps are generally not suited to use as benchmarks
• Building realistic proxy apps is expensive and time-consuming
• “Performance” is not a simple metric
• Results from different tests must be combined very careful to obtain
meaningful aggregate metrics
Easier Said Than Done…
![Page 8: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/8.jpg)
8 Copyright © 2015 BDTI
• Educate consumers on UX • Deliver UX ratings at scale
• Performance, battery and display
• Lab-run tests • Ratings weighted by UX data • All ratings certified by BDTI
• User research data • Consumer usage
patterns • By region
Consumer Research Benchmarking Consumers
Certimo: User Experience Ratings For Smart Devices A Unique Approach
![Page 9: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/9.jpg)
9 Copyright © 2015 BDTI
Consumer usage data
• Develop benchmark tests that reflect actual use
cases
• Weight benchmark results based on usage data
Key factors important to user experience
• Performance
• Battery Life
• Display Quality
Measure at the system level
• ~40 system-level tests
• Not individual components in isolation
• Sum of the parts NOT equal to the whole
Certimo Benchmark Design Philosophy
![Page 10: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/10.jpg)
10 Copyright © 2015 BDTI
Example: Certimo Photo Scroller Test
![Page 11: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/11.jpg)
11 Copyright © 2015 BDTI
Certimo Architecture Overview
Battery Life Display
Usage Data
Overall Performance Battery Display
Browsing Infotainment Messaging Multimedia
Benchmark
Test Suites
Ratings Tool
Ra
tin
gs
Performance
Certimo Ratings Tool
![Page 12: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/12.jpg)
12 Copyright © 2015 BDTI
Example Certimo Published Results
![Page 13: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/13.jpg)
13 Copyright © 2015 BDTI
• “Forming preferences is akin to riding a bicycle; we can do it easily but
cannot easily explain how.” [Wilson and Schooler, 1991]
• “The colour of the mug was shown to influence participants’ rating of the
coffee.” [Van Doorn et al., 2014]
• “… reasoning interferes with better initial choice among non-experts.”
[Lopes, 2014]
User Experience
www.iseekgolf.com
![Page 14: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/14.jpg)
14 Copyright © 2015 BDTI
0
5000
10000
15000
20000
25000
30000
1996 1998 2000 2002 2004 2006 2008 2010 2012
MM
AC
s/s
eco
nd
Year
DSP Performance: High-end, Single-core DSPs from Texas Instruments
Source: BDTI Analysis
10 GMACs/
second
Meanwhile, Processor Performance
Advances
![Page 15: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/15.jpg)
15 Copyright © 2015 BDTI
Computer vision: research and fundamental
technology for extracting meaning from images
Machine vision: factory applications
Embedded vision: thousands of applications
• Consumer, automotive, medical, defense, retail,
gaming, security, education, transportation, …
• Embedded systems, mobile devices,
PCs and the cloud
The Evolution of Vision Technology
![Page 16: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/16.jpg)
16 Copyright © 2015 BDTI
Embedded Vision Will Change User Experience
Abound Labs
www.incrediblethings.com
![Page 17: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/17.jpg)
17 Copyright © 2015 BDTI
How Embedded Vision Can Help Us
Understand User Experience
Philips Living Website Technologies
Fraunhofer HomeHealthTesting.com
![Page 18: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/18.jpg)
18 Copyright © 2015 BDTI
The Embedded Vision Alliance (www.Embedded-Vision.com) is a
partnership of 47 leading embedded vision technology suppliers
Mission: Inspire and empower product creators to incorporate
visual intelligence into their products
• The Alliance provides low-cost, high-quality technical
educational resources for engineers
• Member companies position themselves as leaders to
thousands of product creators via the Alliance web site and
conferences
• Members meet quarterly to develop business partnerships and
gain insights into markets and technologies
• We secure frequent press coverage on embedded vision topics,
gaining exposure for our members as thought leaders
Empowering Product Creators to
Harness Embedded Vision
![Page 19: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/19.jpg)
19 Copyright © 2015 BDTI
Alliance Member Companies
For more information, visit www.Embedded-Vision.com
![Page 20: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/20.jpg)
20 Copyright © 2015 BDTI
The only industry event focused on enabling
engineers to create “machines that see”
• “Good balance of technical content
and application-driven examples.”
Embedded Vision Summit 2015 highlights:
• Inspiring keynotes by leading innovators
• Full day of high-quality, practically-oriented technical
talks
• Demos of the latest apps and technologies
• In-depth pre- and post-Summit workshops
Registration open at www.EmbeddedVisionSummit.com
Embedded Vision Summit:
May 12, 2015 — Santa Clara, CA
Ren Wu, Baidu
Mike Aldred, Dyson
![Page 21: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/21.jpg)
21 Copyright © 2015 BDTI
• We need a better understanding of what drives user satisfaction
• Embedded vision can help with this…
• While also complicating it, by enabling new types of products, capabilities
and interactions
• Benchmarks must be thoughtfully designed and carefully executed
• Benchmark design should be driven by user experience
Conclusions
![Page 22: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/22.jpg)
22 Copyright © 2015 BDTI
Recommended Reading
![Page 23: Measuring and Improving User Experience - bdti.com · •Measuring: Speed, cost, energy efficiency •But what about ease of use, support, roadmap risk? •2014: Benchmarking smartphones](https://reader034.vdocuments.mx/reader034/viewer/2022042400/5f0e71487e708231d43f448b/html5/thumbnails/23.jpg)
23 Copyright © 2015 BDTI 23
THANK YOU
Berkeley Design Technology, Inc. 1646 North California Blvd., Suite 220, Walnut Creek, CA 94596 USA | t: +1 (925) 954-1411 | f: +1 (925) 954-1423