the cloud: a game changer to test, at scale and in production, soa based web and mobile...
DESCRIPTION
Today in retail, financial services, media, telecommunications and a host of other industries, more and more business is transacted through consumer web sites and mobile applications. With new channels creating spikes in traffic, highly complex system architectures, and internet-savvy customers, websites and web applications must be tested at scale to maximize business results and avoid a catastrophic crash. However, whether due to time or cost or other reasons, upwards of 90 percent of web applications are not fully tested before launching. If testing is done, many times it's with a small percent of expected traffic, which is then extrapolated for an estimation of performance. Cloud computing is changing the game for testing web applications. Cloud testing enables, for the first time, performance testing that complements the lab and accounts for the conditions in a production environment, such as traffic spikes, network latency, firewalls, and other factors. And it can be done far more affordably than traditional testing methods, as part of agile development cycles, and without an army of highly skilled performance engineers. Presentation given in Rome for the International SOA conference - Moving SOA into the Cloud in Rome, May 2011TRANSCRIPT
The Cloud A game changer to test, at scale and in production,
SOA based web and mobile applications. Fred Beringer – VP Business Development Europe – SOASTA
International SOA Conference 2011- Rome
PERFORMANCE
matters more than ever before
Social media is highly viral and drives massive amount of
traffic.
#1
500 million active users spending 700 billion minutes per month
13 million hours uploaded in 2010
2 billion video viewed per day
106 million accounts
10 billion tweets in 2010
Your new marketing mix
Web Architecture is complex
#2
CDN Content
Switch Ad Server
Analytics
Shopping Cart CMS Content
Search Session Info
Databases
Load balancers
Media Server
Payment Server
Inside and outside the firewall
Application servers
Content is king
#3
1980
1992
1996
2011
Access devices explosion
#4
Different performance on each device
HIGHER EXPECTACTIONS
#5
of consumers say speed is more important than functionality 39%
Source: Forrester Research
Expect a web page to load in two seconds or less 47%
Source: Forrester Research
Will abandon a web page if it takes more than three seconds
to load 40%
Source: Forrester Research
Low performance and unreliability brings
MASSIVE BUSINESS IMPACT
#6
PayPal down 45 minutes in 2009 $45 Millions lost revenue
Amazon loses 1% of their sales for each 100ms of latency
Google loses 20% of search traffic every 500ms of additional latency
60% of companies admit that they do not have any performance management processes in place or what they have is ineffective.
DynaTrace study- 2009
50%
Of companies having performance management processes admit that they are doing it only in reactive way when problem occurs.
DynaTrace study- 2009
50% DynaTrace study- 2009
of application problems are found in production – when those problems have impacted end users and are most expensive to resolve.
What is required today to test effectively
performance of web applications?
SCALE #1
Real time analysis
of performance data
#2
SPEED
#3
AFFORDABILITY #4
Ready for some pain?
Traditional Performance Testing doesn’t work anymore
2006
PERFORMANCE TESTING
REVOLUTION
“ The massive resources of the cloud have changed the game for performance testing. Cloud Testing enables organizations to test their consumer facing Web and mobile applications fast, affordably and at scale. There’s now no excuse for the type of Web site disasters we’ve seen in the past that cost businesses millions in revenues and perhaps more in reputation. ”
Tom Lounibos CEO SOASTA
Cloud Testing
What does it bring us?
Affordable platform for infinite load
#1
Billions possible points of failure
The new testing challenge
Real-time analysis brings wisdom
#2
Global and realistic traffic
#3
Where is the needle?
#4
In your code
In your infrastructure
Outside your firewall
s
Memory leaks
Scale of Test
Sta
ge /
Team
Dev & Test
Extending Performance Beyond the Lab Discovering Issues across teams, time and scale
Release & Deploy
Network & Ops
CDN file placement
Load Balancer configuration
Network bandwidth
Network configuration
DNS routing
Inadequate server resources
Default configuration settings
Unbalanced web servers Auto-scaling failures
Latency between systems
Slow third-party plug-ins
Garbage collection
Database thread counts
Inefficient database queries Slow pages
Conflict with other apps
Test Lab (<10k Vus) Staging (10-100k) Production (> 100k)
Search technology limits
Method-level tuning
Max sockets exceeded
Firewall max capacity
Global latency variance
Security bottlenecks
Memory leaks
CloudTest Live Real-world Issue Resolution
Company Profile Company Toy Manufacturer
Size Fortune 500
Project Profile Environment Lab and Staging
Application Drupal (Apache, PHP, MySQL)
Location Private Hosted Cloud
Objective Identify Breaking Points
Project Length Five Days (non-consecutive)
Staging Lab
Lab
• First test cycle in the lab • 239 hits per second • 40.5 MBit/second bandwidth
• Some servers hitting 100% CPU • Bottleneck in throughput
CloudTest Live Real-world Issue Resolution
Lab
• Second test cycle in the lab • Higher concurrency target • Bandwidth stayed the same
• Some servers hitting 100% CPU • Performance problem over time • Load balancer flapping servers
CloudTest Live Real-world Issue Resolution
Staging
• Bandwidth (238Mbits/s) – ouch! • CPU at 100%
CloudTest Live Real-world Issue Resolution
Staging
• 767 Home pages served at 650 megs total transfer. • That’s an 850kb page weight! Yikes!
CloudTest Live Real-world Issue Resolution
• Poor throughput on the initial tests • CPU hits 100% • Bandwidth cap identified in the lab • Database connections leaking over time • Load balancer dropping servers
• Massive page weight on the home page • Application unresponsive at ~2000 users
• Customer happy with the breaking point identified • Optimized web assets in the home page • Offloaded assets to the CDN • Load balancer algorithm optimizations
Staging
Lab
CloudTest Live Real-world Issue Resolution
Company Profile Company Advanced Technology
Size Fortune 50
Project Profile Environment Production
Application Java (Struts), Oracle
Location Multiple Hosted Private Data Centers
Objective 172,000 Concurrent
Project Length Two Months
Production
CloudTest Live Real-world Issue Resolution
Recurring error pattern during scenario completion
Production
CloudTest Live Real-world Issue Resolution
Awesome outcome in Production!
CloudTest Live Real-world Issue Resolution
• Load balancer firmware bug with 2048 bit cert • Database issues with the login process
• Worked with hardware LB manufacturer on a fix • Really low error rate • Successful launch!
Production
CloudTest Live Real-world Issue Resolution
AGILE WORLD Fit for an #5
Real-time collaboration
Continuous performance testing
Cloud Testing helps companies
• Every day at noon is a holiday spike in traffic.
• Pages must render to the user in under 1 second.
• 50% of all revenue from Gilt sales comes in less than 1 hour after sale starts.
Cloud Testing helps dolls
Cloud Testing helps taxes
Cloud Testing helps phones
CHALLENGES Testing SOA in the cloud
It gets crowded
Loose SLAs
Harder to predict
(slightly)Different strategy
Same challenges • Services with no user interface • Data driven • Services outside the organization/cloud • End-to-end scenario • Test ownership
Same Bottom-up performance testing approach • Individual services • Set (integration) of services • End-to-end
Deal with Cloud unpredictability • Varying performance encourage continuous performance testing/monitoring. • Infrastructure upgrade could impact performance. • Expect to baseline on premise application and run benchmark in the cloud
after migration. • Test the full bottom-up cycle for increased confidence.
Let’s see how this website performs!
Questions/Discussion
The Cloud A game changer for web performance testing
twitter.com/fredberinger
http://www.fredberinger.com
Photos Credits: Slide 2: http://www.sxc.hu/photo/1219246 Slide 18, 19, 20: http://www.sxc.hu/photo/1267744 Slide 21: http://www.sxc.hu/photo/1181194 Slide 22,23,24: http://www.sxc.hu/photo/396185 Slide 25: http://www.flickr.com/photos/tsoumplekas/4495778365/ Slide 32: http://www.sxc.hu/photo/1160562 Slide 34: http://www.sxc.hu/photo/1331143 Slide 46: http://www.sxc.hu/photo/995000 Slide 48: http://www.sxc.hu/photo/560250 Slide 54: http://www.sxc.hu/photo/561622 Slide 59: http://www.flickr.com/photos/caitlinator/4006198703/ Slide 60: http://www.sxc.hu/photo/1221951 Slide 62: http://www.sxc.hu/photo/1019388