fast, strong & nimble mobile performance testing
TRANSCRIPT
1
Presents:
Webinar
2
Expert Perspectives on Mobile Performance Testing
TODAY’S SOASTA PRESENTERS
Mark Tomlinson: West Evergreen Consulting, former LoadRunner PMDan Bartow: SOASTA VP Product ManagementModerator: Brad Johnson
Agenda: • Poll question• Embracing mobile performance testing – what's new and what's
not• Building a performance test plan to address what really matters• Testing and measuring for "Fast" and "Strong" all at once for
mobile and web• Implementing a solution that does it all with agility to match the
new normal
Questions: Submit in the question box during event
3
o Sheer Number of Devices (953M Smartphones)
o Different Operating Systems
o Scale of Global Customers (6B)
o Dynamic Content (Video, Animation, …)
o Emerging People to Machine
Interfaces
Mobile performance at scale can not be tested manually.
4
CPU
Battery
Memory
Native Apps
Mobile Web AppsApp
Server
Database
App Server
Web Server
CacheLoad
Balancer Web
ServerMobile
browserWeb
trafficHTTP(S)
Shared web & mobile infrastructure
Web browser users
Mobile browser users
Hybrid Mobile AppsApp
Server
Database
App Server
Web Server
CacheLoad
Balancer Web
ServerNative App Web
traffic
Dedicated or shared mobile infrastructure
Real devices & large scale mobile traffic
mUX is most affected by back
end load and network
performance
Mobile User Experience (mUX)
is affected by app & device
performance only
mUX is affected by native app, device,
back-end and network
performance Fast AND Strong
5
o Development teams test at a unit and component level (as always)
o Test teams load test with web-based traffic (if load testing at all)
o Ops teams monitor with services built pre-Smartphone
o No one communicates, yet agile dev is all about communication!
6
7
o 20 years in testing
o Performance Guy
o Small tests & Big tests
o Small & Big companies
o …speed freak.
8
- Old applications migrated to new mobile formats – “transformation”
- New Applications added-on to existing systems – “bolt-on”
- Entire new business units created for mobile consumer experience
- Device manufacturers are investing and expanding – alarmingly!!
- “The number of Smartphone users worldwide is predicted to exceed 1 billion by 2014” (Parks Associates, 2010)
- “Tablets to hit 100M shipments in 2012” (Source: ABI Research, 2012)
“Every customer I’ve worked with in the last 18 months is prioritizing mobile performance testing.”
-Mark Tomlinson
9
WHAT’S NEW?
oDevices – lot’s of ‘em
oBattery Life & Heat
oGeographic Diversity
oLatencies – insanely high
oVariable Processor Frequency
WHAT’S NOT!
oThe risks of poor performance
oResponse Time & Volumes
oRoot-cause Analysis
oCapacity Estimation
oTest Planning
oBugs…lot’s of ‘em
10
A real story about a customer situation:
1.1,000 requests per second into the services infrastructure
2.Systems were fine-tuned, low-latency (< 250 ms)
3.Memory (in .NET CLR) was also optimized with perfection
4.They introduced a new mobile experience to the end-user
5.10% of the company started using the mobile app
6.What happened next…???
YOU MAKE THE CALL…
11
Enhance existing plans for mobile:
• Specific response-time goals for mobile
• Connected/disconnected test scenarios
• Mobile test lab setup & configuration
• Matrix of devices to be tested/supported
• Define end-user location and conditions
• Monitors for device’s physical resources
• Monitor client-side performance
Let’s review an example test strategy…
12
Ensure that new mobile plans include:
• Mobile Performance Objectives (time)
• Defined scope for device types
• Detailed explanation of end-user behavior
• Inter-app test conditions
• Carrier-specific test conditions
• Failure and recovery conditions
• Back-end system dependency and impacts
Let’s review a new mobile test plan…
13
Dan Bartow
14
Any mix of traffic at any scale
Hybrid Mobile AppsApp
Server
Database
App Server
Web Server
CacheLoad
Balancer Web
ServerNative App Web
traffic
Dedicated or shared mobile infrastructureReal
devices & large scale mobile traffic
Global Web browser users
Global Mobile browser users
Measure what Matters
o Dev teams test early and continuously for FREE with CTLite
o Test teams respond quickly with complete mobile coverage
o Ops team validate and tune production infrastructure
15
Thanks & Q&A
Contact SOASTA:www.soasta.com/cloudtest/[email protected] us:
twitter.com/cloudtesttwitter.com/cloudtest
facebook.com/cloudtestfacebook.com/cloudtest
White Papers, Webinar Recordings, Case Studieswww.soasta.com - Knowledge Center
Next Webinar: Sept. 27, 10 a.m. PST“Automated Testing & Continuous Deployment
for Mobile Apps in the Cloud”(SOASTA & CloudBees)
Register at www.soasta.com/knowledge-center/webinars
Contact [email protected]@mtomlins
Contact [email protected]@PerfDan