http://www.evolveimages.com/media/b65376d8-3b6a-11e1-a064-8546d2b4117d-aerial-view-of-complex-highway-interchange!
User Experience and the Cloud: If the Users Can’t Use It Then It Doesn’t Work!
!Andrew has fifteen years of experience designing and managing the deployment of systems monitoring and Event Management software. Andrew has held various roles including leading the monitoring organization of a Fortune 100 company and developing solutions as a consultant for a wide variety of organizations, including the Mexican Secretaría de Hacienda y Crédito Público, Telmex, Wal-Mart of Mexico, JP Morgan Chase, Nationwide Insurance and the US Navy Facilities and Engineering Command.!
!Andrew White!User Experience Fanatic!
http://weheartit.com/entry/12433848!
Follow Us: #ITSMSummit!
GROUND RULES FOR THIS SESSION…!
1. If you can’t tell if I am trying to be funny…!!GO AHEAD AND LAUGH!!
2. Feel free to text, tweet, yammer, or whatever. People gotta hear this! Use !
3. If you have a question, no need to wait until the end. Just interrupt me. Seriously… I don’t mind.!
I have a lot of experience leading !Systems and Event Management teams !
My name is Andrew White!
Latency!I am here today to share some of what I have learned about!
User Experience!
And more importantly, I am here today to talk about!
What do I mean by latency and user experience?!
Definitions:!
Follow Us: #ITSMSummit!
LA�TEN�CY – [LEYT-N-SEE]!-noun, plural -cies!1. The state of being latent!2. The time that elapses between a stimulus
and the response to it!3. The state of being not yet evident or active!
http://www.flickr.com/photos/25822731@N02/4644128723/!
Follow Us: #ITSMSummit!
EX�PE�RI�ENCE – [IK-SPEER-EE-UH’NS]!-noun!1. The apprehension of an object, thought,
or emotion through the senses or mind!2. Direct personal participation or
observation; actual knowledge or contact!3. A particular incident, feeling, etc., that a
person has undergone!-verb!4. To be emotionally or aesthetically moved
by; to feel!5. To learn by perceiving, understanding, or
remembering!
http://www.flickr.com/photos/51035626620@N01/170061976/sizes/l/in/photostream/!
Follow Us: #ITSMSummit!
When you put them together we get:!!The ultimate measure of success for any system is the perception of its performance. The less interactive a system becomes the more likely its performance will be perceived to be poor.!
Latency is the mother of inactivity!!
Follow Us: #ITSMSummit!
THE TWO DIMENSIONS OF LATENCY…!
Internal Latency vs. External Latency!
Actual Latency vs. Perceived Latency!
This is what user experience is all about!!In other words: Perceived = Fn(Internal+External)Variation )!
*Among adults who accessed the internet with a mobile phone in the past 12 months (n=1,001) – Gomez Mobile Web Experience Survey conducted by Equation Research!
58% of mobile phone users expect websites to load as quickly, almost as quickly or faster on their mobile phone, compared to the computer they use at home*!
http://www.flickr.com/photos/lucianbickerton/3858380291/sizes/l/!
*Among adults who accessed the internet with a mobile phone in the past 12 months (n=1,001) – Gomez Mobile Web Experience Survey conducted by Equation Research!
60% of mobile web users have had a problem in the past year when accessing a website on their phone*!
http://www.flickr.com/photos/rickyromero/1357938629/sizes/l/!
*Among adults who accessed the internet with a mobile phone in the past 12 months (n=602) – Gomez Mobile Web Experience Survey conducted by Equation Research!
Slow load time was the number on issue, experience by almost 75% of them*!
http://bighugelabs.com/onblack.php?id=2497744197&size=large!
Is 5 seconds really bad?!
Start…!
Start…!
Observed Maximum:!
90th Percentile:!5.44 seconds…!
15.4 seconds…!
Start…!
Start…!
Observed Maximum:!
90th Percentile:!DONE!!5.44 seconds…!
15.4 seconds…!
Start…!
Start…!
Observed Maximum:!
90th Percentile:!DONE!!
DONE!!
5.44 seconds…!
15.4 seconds…!
If you were the one on the phone with one of those customers…!
how would you fill that silence?!
We need to recognize when we have problems to solve!
Why does any of this matter?!
Follow Us: #ITSMSummit!
No complaint… is more common than that of a scarcity of money!-Adam Smith, Wealth of Nations!
QUESTION:!What value does your IT organization create for your business?!
If you can’t answer this question, how can you be sure you are doing the right things and doing them well…!
HINT: “We provide infrastructure or applications the business uses” is not a value statement!
Follow Us: #ITSMSummit!
We all want to make sure of one thing…!
The services we offer are our brand!
And the thing about brand is…!
Brand is not what you say…!
Brand is what THEY say!
• Who are we?!• What do we do?!• Why should people care?!
So it is important to know:!
Would anyone miss us!if we were not here?!
And especially…!
BRAND MATTERS!
How does IT preserve the brand value it creates?!
• 100% Uptime*!
• Scalability*!• Performance*!• Agility*!
• Good UX*!!
*To the best of our ability!
How well would THEY say you are doing?!
Maybe.!Let me show you why this is important…!
http://www.flickr.com/photos/anneacaso/3693155059/sizes/l/in/photostream/!
Objective #1: Users Love Our IT Systems…
Constructive Criticism!
If no critique is given, no improvement will happen.!
Follow Us: #ITSMSummit!
OUR PROBLEM STATEMENT:!
The business needs to reliably reach its customers and users regardless of where they may be located.
Latency forces close geographic proximity of the components and limits the quality of service provided
to geographically distributed customers. !!
If the users can’t use it, it doesn’t work.!
Follow Us: #ITSMSummit!
OUR CONSTRAINTS!At the same time, there are a few inescapable facts we face:!1. Today’s users demand a reasonably well
performing system to do their work!2. Our systems mirror the complexity of the
businesses they support!3. Business continuity requires diversity in our
deployments!4. The speed of light isn’t changing any time soon!
When all of these happen at the same time…!
Ug…!
http://www.flickr.com/photos/gregphoto/4881356366/sizes/l/in/photostream/!
Bad Experience!!!!
http://www.wealthforteens.com/wp-content/uploads/2011/07/happiness_boy.jpg!
We need to be giving more of this!
Follow Us: #ITSMSummit!
BEING USER FOCUSED…!
In addition to monitoring for system availability, we are here to help manage latency.!
The Formula…!1. Continually map, monitor, and categorize all
sources of latency!2. Help identify and remove all sources that are
found!
OK.!So now what?!
I am going to talk today about latency in the cloud and some new ideas that will help both them and you.!
Is it the infrastructure or the application?!The perennial problem….!
Follow Us: #ITSMSummit!
ANATOMY OF AN OUTAGE!
Corporate!LANs & VPNs!
Load Balancer!
Firewall!
Web!Servers!
Message!Queue!
zOS!CICS!
WAS!
Database!
WAS!Database!
zOS!MQ!
DB2!
IM01109089: P0 - Affecting Multiple apps & Internet Sales West!!!!!
4!
!!!!!!
3!
!!!!!!1!
5:45-ish pm: CICS ABENDS start flooding MainView but not high enough to ticket!
!!!!!!2!
6:00-ish pm: MQ flows start are interrupted and are alerting in Flow Diagnostics!
6:04pm: Synthetic transactions fail at and 6:14 the Ops Center confirms the issue and creates a P0 Incident!
6:54pm: Support teams investigate the interrupted flows and determine it is a “back-end” problem!
10:29pm: Support teams investigate MQ and ultimately and rule it out and ultimately decide to reset CICS to resolve the issue!
!!!!
5!
Follow Us: #ITSMSummit!
“How did you go bankrupt?" “Two ways. Gradually, then suddenly.” -Ernest Hemmingway, The Sun Also Rises!
HOW DID WE GET HERE?!
Follow Us: #ITSMSummit!
DRIVING THE RIGHT KIND OF ACTION!
Application!
End User Experience!
Gainesville!
Transaction 1!
Transaction 2!
Transaction N!
San Antonio!
Transaction 1!
Transaction 2!
Transaction N!
Des Moines!
Transaction 1!
Transaction 2!
Transaction N!
Columbus!
Transaction 1!
Transaction 2!
Transaction N!
Infrastructure!
Network!
KPI 1!
KPI 2!
KPI N!
Mainframe!
KPI 1!
KPI 2!
KPI N!
Storage!
KPI 1!
KPI 2!
KPI N!
Linux!
KPI 1!
KPI 2!
KPI N!
Middleware!
KPI 1!
KPI 2!
KPI N!
Database!
KPI 1!
KPI 2!
KPI N!
Follow Us: #ITSMSummit!
Application!
End User Experience!
Gainesville!
Transaction 1!
Transaction 2!
Transaction N!
San Antonio!
Transaction 1!
Transaction 2!
Transaction N!
Des Moines!
Transaction 1!
Transaction 2!
Transaction N!
Columbus!
Transaction 1!
Transaction 2!
Transaction N!
Infrastructure!
Network!
KPI 1!
KPI 2!
KPI N!
Mainframe!
KPI 1!
KPI 2!
KPI N!
Storage!
KPI 1!
KPI 2!
KPI N!
Linux!
KPI 1!
KPI 2!
KPI N!
Middleware!
KPI 1!
KPI 2!
KPI N!
Database!
KPI 1!
KPI 2!
KPI N!
DRIVING THE RIGHT KIND OF ACTION!
Follow Us: #ITSMSummit!
Application!
End User Experience!
Gainesville!
Transaction 1!
Transaction 2!
Transaction N!
San Antonio!
Transaction 1!
Transaction 2!
Transaction N!
Des Moines!
Transaction 1!
Transaction 2!
Transaction N!
Columbus!
Transaction 1!
Transaction 2!
Transaction N!
Infrastructure!
Network!
KPI 1!
KPI 2!
KPI N!
Mainframe!
KPI 1!
KPI 2!
KPI N!
Storage!
KPI 1!
KPI 2!
KPI N!
Linux!
KPI 1!
KPI 2!
KPI N!
Middleware!
KPI 1!
KPI 2!
KPI N!
Database!
KPI 1!
KPI 2!
KPI N!
DRIVING THE RIGHT KIND OF ACTION!
Follow Us: #ITSMSummit!
Application!
End User Experience!
Gainesville!
Transaction 1!
Transaction 2!
Transaction N!
San Antonio!
Transaction 1!
Transaction 2!
Transaction N!
Des Moines!
Transaction 1!
Transaction 2!
Transaction N!
Columbus!
Transaction 1!
Transaction 2!
Transaction N!
Infrastructure!
Network!
KPI 1!
KPI 2!
KPI N!
Mainframe!
KPI 1!
KPI 2!
KPI N!
Storage!
KPI 1!
KPI 2!
KPI N!
Linux!
KPI 1!
KPI 2!
KPI N!
Middleware!
KPI 1!
KPI 2!
KPI N!
Database!
KPI 1!
KPI 2!
KPI N!
DRIVING THE RIGHT KIND OF ACTION!
Follow Us: #ITSMSummit!
Application!
End User Experience!
Gainesville!
Transaction 1!
Transaction 2!
Transaction N!
San Antonio!
Transaction 1!
Transaction 2!
Transaction N!
Des Moines!
Transaction 1!
Transaction 2!
Transaction N!
Columbus!
Transaction 1!
Transaction 2!
Transaction N!
Infrastructure!
Network!
KPI 1!
KPI 2!
KPI N!
Mainframe!
KPI 1!
KPI 2!
KPI N!
Storage!
KPI 1!
KPI 2!
KPI N!
Linux!
KPI 1!
KPI 2!
KPI N!
Middleware!
KPI 1!
KPI 2!
KPI N!
Database!
KPI 1!
KPI 2!
KPI N!
DRIVING THE RIGHT KIND OF ACTION!
Follow Us: #ITSMSummit!
WHAT GOOD MONITORING LOOKS LIKE!
Corporate!LANs & VPNs!
Load Balancer!
Load Balancer!
Firewall!
Switch!
Web Server Farm!
Database!
Data Power!Mainframe!
Middleware!
Load Balancer!
1. System Availability!2. Operating System Performance!3. Hardware Monitoring!4. Service/Daemon and Process Availability!5. Error Logs!6. Application Resource KPIs!7. End-to-End Transactions!8. Point of Failure Transactions!9. Fail-Over Success!10. “Activity Monitors” and “Reverse Hockey
Stick”!
Elements of Good Monitoring!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!3!2! 4! 5! 6!1!!!!!
7!
!!!!!!!!!!!!!!!!!!8!
!!!!!!!!!!!!!!!!!!!!
9! !!!!!!
10!
Follow Us: #ITSMSummit!
Application “Think Time”…!
These two are the network!
WE HAVE A PROBLEM HOUSTON!
Here is the real problem…!
Follow Us: #ITSMSummit!
COGNITIVE DISSONANCE!
Corporate!LANs & VPNs!
Distributed!
Database!
Mainframe!
Network!
Middleware!
Storage!
ISP!Connection!
DNS & Internet!Services!
Content Mgmt!System!
Social Network!Widgets!
Site Tracking!& Analytics!
Banner Ads & !Revenue Generators!
Multimedia &!CDN Content!
Home Wireless!& Broadband!
Mobile Broadband!
The Part You Control!
The Part They Experience!
…meanwhile the user is NOT happy
All our systems look great, SLA’s are being met…
You Have More Control Here
Than You Think!
Follow Us: #ITSMSummit!
SOMETIMES WE MISS WHAT IS GOING ON!
Say… what’s a mountain goat doing all the way up here in a cloud bank?!
Failure comes when you don’t listen. You can’t put something out there and assume it’s great. It’s up to us to make sure we’re listening to improve our chances for success—if not this time, next time !-Alan Lewis, eBay Technical Evangelist!
<TANGENT>!
All You Need!
Love!
</TANGENT>!
http://info.streamdatacenters.com/Portals/165393/Gallery/Album/6624/Richardson%20Aerial-01.png!
This is no longer the way we should think about monitoring!
Monitoring Happens Here!
Cloud Monitoring Happens Here!
Is it the infrastructure or the application?!The perennial problem….!
Follow Us: #ITSMSummit!
COGNITIVE DISSONANCE!
Corporate!LANs & VPNs!
Distributed!
Database!
Mainframe!
Network!
Middleware!
Storage!
ISP!Connection!
DNS & Internet!Services!
Content Mgmt!System!
Social Network!Widgets!
Site Tracking!& Analytics!
Banner Ads & !Revenue Generators!
Multimedia &!CDN Content!
Home Wireless!& Broadband!
Mobile Broadband!
The Part You Control!
The Part They Experience!
…meanwhile the user is NOT happy
All our systems look great, SLA’s are being met…
You Have More Control Here
Than You Think!
The Cloud!
Follow Us: #ITSMSummit!
DRIVING THE RIGHT KIND OF ACTION!
Application!
End User Experience!
Gainesville!
Transaction 1!
Transaction 2!
Transaction N!
San Antonio!
Transaction 1!
Transaction 2!
Transaction N!
Des Moines!
Transaction 1!
Transaction 2!
Transaction N!
Columbus!
Transaction 1!
Transaction 2!
Transaction N!
Infrastructure!
Network!
KPI 1!
KPI 2!
KPI N!
Mainframe!
KPI 1!
KPI 2!
KPI N!
Storage!
KPI 1!
KPI 2!
KPI N!
Linux!
KPI 1!
KPI 2!
KPI N!
Middleware!
KPI 1!
KPI 2!
KPI N!
Database!
KPI 1!
KPI 2!
KPI N!
The Cloud!
Follow Us: #ITSMSummit!
SOME THINGS NEVER CHANGE!
Corporate!LANs & VPNs!
ISP!Connection!
DNS & Internet!Services!
Content Mgmt!System!
Social Network!Widgets!
Site Tracking!& Analytics!
Banner Ads & !Revenue Generators!
Multimedia &!CDN Content!
Home Wireless!& Broadband!
Mobile Broadband!
Is It My Cloud Provider?!• Configuration errors!• Application design issues!• Code defects!• Insufficient infrastructure!• Oversubscription Issues!• Poor routing optimization!• Low cache hit rate!
Is It a Service Provider Problem?!• Non-optimized mobile content!• Bad performance under load!• Blocking content delivery!• Incorrect geo-targeted content!
Is it an ISP Problem?!• Peering problems!• ISP Outages! Is it My Code or a Browser Problem?!
• Missing content!• Poorly performing JavaScript!• Inconsistent CSS rendering!• Browser/device incompatibility!• Page size too big!• Conflicting HTML tag support!• Too many objects!• Content not optimized for device!
The Cloud!
We are going to borrow some old tactics but apply them in a new way!
We aren’t going back to square one!
Follow Us: #ITSMSummit!
That is great but we need more…!
Follow Us: #ITSMSummit!
WHAT DO YOU WANT TO ACCOMPLISH?!
Your monitoring should help you answer:!
• How will we know if the users are getting the experience they are expecting?!
• How much capacity do we need during normal and peak times to ensure user expectations are met?!
• How quickly can the provider we select ramp up to meet our needs if we find that the service is underperforming?!
• How fast do we need to be able to access additional capacity once it is ready for us?!
Follow Us: #ITSMSummit!
THE CRITICAL PATH OF PERFORMANCE!
Browser Worksta?on OS
Worksta?on Hardware Client LAN Corporate
WAN Datacenter
LAN Etc.
Web Server Web Server OS
Web Server Hardware
Datacenter LAN
Middleware Server
Hardware Middleware Server OS
Middleware Applica?on Etc.
Database Server
Database Server OS
Database Server HBA
SAN Fabric Switch
Array Hardware
Array Controller
Hardware Cache
Disk Drives Etc.
Client Node
Middleware
Database
Follow Us: #ITSMSummit!
COMPOSITE APPLICATIONS!
Site Content!Search!
Session!Information!
User Login!& Identity Mgmt!
Content Mgmt!System!
Social Network!Widgets!
Site Tracking!& Analytics!
Banner Ads & !Revenue Generators!
Multimedia &!CDN Content!
Follow Us: #ITSMSummit!
COMPOSITE APPLICATIONS ARE EVERYWHERE!
• ATG (Oracle) – Shopping Cart!• Estara – Click to Chat!• Twitter Widget – Social Networking!• Gigya – Social Networking!• Google Maps API – GeoLocation!• Facebook Widget – Social Networking!• Google Analyics – User Tracking!
hJp://www.sgstool.com/images-‐dev/_products/drill_group.png
People don’t want to buy quarter-inch-drill-bits…!They want to buy quarter-inch-holes.!-Ted Levitt, Marketing Professor, Harvard University!
Follow Us: #ITSMSummit!
GAINING PERSPECTIVE REQUIRES BALANCE!
Packet Capture!
Synthetic Transactions!
Client Monitoring!
Client Monitoring!
Synthetic Transactions!
Server Probe!
1. Client to the Server!2. Server to the Client!3. “3rd Party” Vantage Point!4. Synthetic Transactions!
Four Perspectives of User Experience!
Follow Us: #ITSMSummit!
WHY MULTIPLE PERSPECTIVES?!
Know Your Customer:!§ What they do? !
§ Customers care about completing tasks NOT whether the homepage is available!
§ Where they do it from?!§ Your customers don’t live in the cloud, test from their
perspective!
§ When they do it?!§ Test at peak and normal traffic levels, to find all the
problems!
§ What expectations do customers have?!§ Is 5 seconds fast enough or does it have to be quicker?!
Follow Us: #ITSMSummit!
SEEING IS BELIEVING!
Real User Monitoring Would Report 94ms!Response Time. !
The page seemed “done” to me !
1.2 seconds later!
The time spent rendering represented 93% of the
user experienced latency!
Follow Us: #ITSMSummit!
HOW PERCEPTION IS INFLUENCED!
Source: Gomez Real-‐User Monitoring
0
5
10
15
20
25
Second
s
Load Time Perceived Render
Follow Us: #ITSMSummit!
Gomez recommends the following test criteria…!Capacity!§ Test vendors to 15-20% past estimated capacity goals!Elasticity!§ Baseline end-user performance before & after testing!§ Test during pre-deployment and in production!§ Ramp elasticity testing to peak levels!Burstability!§ Isolate the cloud elements from other
infrastructure to test!§ Test the “failover process”!
PERFORMANCE TESTING CLOUD CAPABILITIES!
Follow Us: #ITSMSummit!
DON’T CONFUSE BANDWIDTH AND SPEED!
737-‐900ER 747-‐400ER
Maximum Number of Passengers 215 524
Maximum Crusing Speed (mph) 511 570
A 737 and a 747 both travel around 500 mph but the 747 carries twice as many people. Would you say it is twice as fast?
Follow Us: #ITSMSummit!
NOT ALL CLOUDS PERFORM EQUALLY!Average Response Time of Reference Application, as measured from US backbone locations!
Source: Gomez Real-‐User Monitoring
Follow Us: #ITSMSummit!
75th Percentile!
50th Percentile!
25th Percentile!
WHY AVERAGES DON’T WORK!
0.5! 0.7! 0.9! 1.8! 2.5! 2.5! 2.6! 2.9! 3.3! 3.5!
Average!
Follow Us: #ITSMSummit!
BEST PRACTICES FOR QUALITY EXPERIENCES!1. Everyone is responsible for UX!2. Don’t worry about the infrastructure unless it is in
the critical path!3. You can control a lot more outside the firewall
than you think!4. Monitor the user experience from multiple
perspectives and multiple geographies!5. Test like you monitor!6. Monitor at a frequency to ensure you can resolve
issues before end-users are impacted!7. Establish a baseline for historical analysis and
benchmark yourself against the competition!!
Positive Feedback from others is important!
Follow Us: #ITSMSummit!
The IT Culture is driven to technology for solutions. Leverage your monitoring and testing tools to help practice failure scenarios. Work on tracking potential points of failure by creating monitoring and report the rate of occurrence to the developers at the start of each new iteration.!
PLAYING TO OUR STRENGTHS!
Follow Us: #ITSMSummit!
OH… REMEMBER THE OPS USE CASE!
• Security!• Backups!• High Availability!• Upgradability!• Deployment Process!• Scaling and Elasticity!• Anticipated Performance Under Load!• Known Defects!
Follow Us: #ITSMSummit!
LET’S KEEP THE CONVERSATION GOING…!
ReverendDrew!
SystemsManagementZen.Wordpress.com!
systemsmanagementzen.wordpress.com/feed/!
@SystemsMgmtZen!
ReverendDrew!
614-306-3434!