human testing of mobile apps for vistacon 2013 conference 25 april 2013 julian harty contact me:...
TRANSCRIPT
HUMAN TESTING OF MOBILE APPSFOR VISTACON 2013 CONFERENCE
25 APRIL 2013
JULIAN HARTYContact me: [email protected]: 29 April 2013
Creative Commons LicenseHow to design your mobile apps by Julian Harty is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
http://creativecommons.org/licenses/by-sa/3.0/deed.en_US
AUTOMATION
DRIVE FOR EFFICIENCY
STATE OF AFFAIRS
The industry is driving towards automated tests
• ‘Manual testing’ is downgraded
• Agile projects need automated tests to provide fast & frequent feedback
• Development environments for Mobile Apps (SDKs) include Test Automation frameworks
AUTOMATED TESTS
Unit Tests
Continuous Builds
UI Test Automation
Scale out to run tests on multiple devices
• Virtual devices, remote devices, local devices
Complex, sometimes fragile, ecosystems
• Challenges: device security models, limitations in frameworks, connectivity, immature domain, lack of support for multi-modal interfaces, etc.
PEOPLE NEED PEOPLE TOO
HUMAN-CENTRIC TESTING
BEYOND THE REACH OF AUTOMATED TESTSTest frameworks focus on code, API, or GUI,
Not:• Movement: acceleration, rotation, location
• Tactile, Haptic, audio Inputs or Outputs
• Aesthetics: look & feel, transitions
• User Experience (UX)
• Highly dynamic updates: video, audio output
• Complex inputs: Gestures, multi-touch, shaking
• Light: e.g. camera as an input device
So how do we test these?
QOX
Quality Of Experience (QOX)
From the standpoint of a user’s holistic experience of using the mobile app.
Essentially subjective
http://en.wikipedia.org/wiki/Quality_of_experience
UX Research, Evaluation, and Testing: strategies, tips and tools Mark J. Williams
I’VE GOT FEELINGS TOO
6 hats of software testing
• The ‘red’ hat – emotions, intuitions, feelings
I've Got a Feeling: Emotions in Testinghttp://www.stickyminds.com/s.asp?F=S17922_COL_2
Here's an article I wrote a few years ago on the six hats of software testing http://archive.newsweaver.com/qualtech/newsweaver.ie/qualtech/e_article000666359.html and if you really have lots of time on your hands, here's a video of a talk I gave on the same topic at StarWest 2008 http://www.stickyminds.com/Media/Video/Detail.aspx?WebPage=117
“An inconsistency between the product and the document might be surprising, frustrating, or annoying to that person, but what really matters is inconsistency between the product and the desire.”
WHAT DO WE NEED TO DO?
SO…
FAST, EFFICIENT INTERACTIVE TESTING
Bugfests
Move and rotate the phone
Be mobile
Bugfest: A group of people get together to test in parallel and in common. Developers on-hand for direct feedback.
LUNCH OUTTESTING A VIDEO APP• Around 2 hours
• Around 10 testers
• Test the app while going out for lunch
Examples of what the testers did
• Tourist trip on the top of a double-decker bus
• London Underground
• Out for a walk with sandwiches
ESTABLISH STRONG FOUNDATIONS
Learn:
• The platforms, the norms, what users expect
• How the platform was designed to be used
• The application lifecycle
• The devices
• Common, and unusual problems
INSIGHTFUL TESTINGRAPID software
• Heuristics
• Mnemonics to guide the testing
Session-Based Test Management techniques
• Charter
• Time-boxed
• Simple, lightweight test reporting
• Debrief
http://moolya.com/blog/2012/04/18/test-mobile-applications-with-cop-who-flung-gun/
EXTENDED TESTING
EXTENDED TESTING
Remote devices
Remote testing
REMOTE TESTERS
Credit: "Ken Banks, kiwanja.net"
http://www.kiwanja.net/gallery/texting/kiwanja_kenya_texting_17.jpghttp://qz.com/49916/indian-smartphone-users-are-gobbling-up-mobile-data/http://www.imedicalapps.com/2012/09/researchers-smartphone-spirometer-mobile-app/
CHALLENGES OF EXTENDED TESTING
Loss of information
• Sensory deprivation
• Missing or incomplete data
Loss of fidelity
• Lags, delays
Loss of control
How do we test rich interactive apps unless you have the device in your hand?
“In theory, theory and practice are the same.
In practice, they are not.” A device in your hand is worth 2 in the cloud, and 100 virtual devices.
MORE CONTEXT
MORE CONTEXT
Additional considerations
• Cultural norms & differences
• Generations
• What’s important to them?
• How can we empathize?
Mobile Apps & Platforms
InternationalC
ultu
re?
Cu
lture
?
Culture?
GSMA REVEALS FEARS OVER MOBILE PRIVACY ARE HOLDING BACK THE GROWTH OF MOBILE APPS IN LATIN AMERICA
http://www.gsma.com/newsroom/gsma-reveals
A South Korean woman using her mobile phone to buy a product at the virtual retail shop in Seoul.www.guardian.co.uk
EXAMPLES OF WHAT CAN GO WRONG“What this means is that unlike the iPhone, which is most certainly the closest competitor on the market to this phone, the Storm's UI is not custom built for touch navigation -- touch navigation is added after the fact. Things which flow naturally on an iPhone -- flicking through lists, scrolling for a contact, moving around in a webpage or looking through photos -- feel inelegant and uncomfortable on the Storm. There's no inertia to movement, no assurance that your finger is the lynchpin to control of the device. The screen is sensitive enough, surely, but how its software reacts to those touches makes all the difference, and here the feeling is that you're never completely in charge of the phone.”
http://www.engadget.com/2008/11/19/blackberry-storm-review/Image: http://worldwide.blackberry.com/gettingstarted/storm-9500.png
! XOR[*]
PRACTICAL OUTCOMES
[*] Not Exclusive OR
COMBINING STRENGTHS
Automated Tests
• Fast feedback, technology and developer centric
Interactive Tests
• Exploratory, rich interactions and dynamics
• Remote testing e.g. in other countries
Combined
• Augmented, Cyborg testing e.g. automated preparation, data collection, etc.
Learn to delegate & learn when to take control
SOME FURTHER READINGS
http://www.wipconnector.com/download/GuideToTheParallelUniverse_3rdEdition.pdf
http://www.enough.de/fileadmin/uploads/dev_guide_pdfs/Guide_12thEdition_WEB.pdf
Testing and Test Automation for Mobile Apps
Julian Harty
Summer 2013CRC Press
Development Marketing Testing(2009) Testing(2013)
Q & A ?
IF YOU WANT THE DRAFT BOOK
Draft book available for review & feedback
Email me [email protected]
• Agree not to share or distribute
• Comment to get the next available draft