![Page 1: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/1.jpg)
1
![Page 2: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/2.jpg)
Did you all get a chance to read that? As a public company we need to have Vista is the only solution built from the ground up on true enterprise
2
our disclosure statement before all presentations. If you have any questions
on what it means please speak with our General Counsel.
Vista is the only solution built from the ground up on true enterprise
technology --- allowing you to ensure that you continue to provide your
faculty and students an outstanding experience
![Page 3: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/3.jpg)
3
![Page 4: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/4.jpg)
4
![Page 5: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/5.jpg)
5
![Page 6: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/6.jpg)
6
![Page 7: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/7.jpg)
7
![Page 8: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/8.jpg)
Search Google and you will be amazed how little meaningful information you get by the words “performance forensics” in the context of computers and
8
get by the words “performance forensics” in the context of computers and software. One paper by Bob Sneed from Sun Microsystems (http://www.sun.com/blueprints/1203/817-4444.pdf) is out there, but very little else.
So you will have to trust me in my primitive definition of performance forensics. You might even offer to help make it better.
Performance forensics is like any other forensics process. It begins with collective evidence. If you are lucky and have a lot of tools in place you will have a starting point of data to sift through. More often then not, the data is not there. You are not always lucky to have the data when you need it and/or it might not be in the best format for getting to the root cause of a problem.
Evidence as we will discuss later can be collected after the fact. Techniques such as discrete simulation can be used to re-enact an incident. When that does happen, you have the ability to capture all of the data you want. You simply need to know what data to collect. It’s a circuitous loop of sorts…mainly because you might not know what data to collect to begin with.
It’s like when I look under the hood of my car. I have no idea what I’m looking at…Maybe it’s that smoking gun I’m in search of. Yeah, I guess if I see some kind of corrosive, smoke or leak it might be painfully obvious…but it never is. Not with today’s cars…Computers and software specifically are the same. Rarely is there that smoking gun sitting in front of your face waiting to be found. Thus evidence is critically important to the process. Interviewing is a big part of evidence gathering. It’s not a separate activity. As I will discuss interviewing is an art. You have to be able to assemble questions that will return meaningful answers. Equally, you have to be able to avoid diagnosis bias and value attribution that are often part of human nature.
![Page 9: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/9.jpg)
Source: http://www.flickr.com/photos/turkguy19/1018419391/
9
![Page 10: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/10.jpg)
http://www.flickr.com/photos/wwarby/3297205226/
10
![Page 11: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/11.jpg)
11
![Page 12: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/12.jpg)
Source: http://www.flickr.com/photos/t_squared/152270386/
12
![Page 13: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/13.jpg)
http://www.flickr.com/photos/7563125@N08/2830710184/
13
![Page 14: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/14.jpg)
14
![Page 15: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/15.jpg)
Source: http://www.flickr.com/photos/turkguy19/1018419787/
15
![Page 16: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/16.jpg)
Source: http://farm2.static.flickr.com/1330/3174009125_ec49351a6d_m.jpg
16
![Page 17: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/17.jpg)
Source: http://www.flickr.com/photos/ale2000/1275120868/
17
![Page 18: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/18.jpg)
18
![Page 19: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/19.jpg)
19
![Page 20: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/20.jpg)
20
![Page 21: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/21.jpg)
21
![Page 22: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/22.jpg)
Source: http://farm4.static.flickr.com/3396/3507282396_3756634f01_m.jpg
22
![Page 23: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/23.jpg)
Source: http://www.flickr.com/photos/psilver/412264230/
23
![Page 24: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/24.jpg)
http://crackerjackonlinemarketing.com/blog/wp-
content/uploads/2008/07/working-woman-with-octopus-hands.jpg
24
![Page 25: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/25.jpg)
Source: http://www.flickr.com/photos/nickbush/450151862/
25
![Page 26: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/26.jpg)
Problems are not always easily identifiable. When I say that I feel off or sick, I leave the listener desiring more information. They might infer that I have a stomach pain, a cold or a
26
listener desiring more information. They might infer that I have a stomach pain, a cold or a headache. It could be that I am tired or I have a broken arm. A more related example that I often hear is that my system is slow. What defines slow? Can you show me? Can I experience the slowness?
Is it always slow every single day and every minute? Are all of the components that make up the physical architecture necessarily slow? Are particular use cases experiencing latency? Do they always experience latency or is it at specific times? Is it specific users who experience latency? Are the users different is some kind of fashion? Does the problem happen after a particular interaction pattern? Does it happen with a particular piece of data?
When a problem is easily identifiable, define a clear, intelligible problem statement. The problem statement is used to aid the investigation so the forensics process can focus on collecting meaningful data to get to root cause analysis.
Narrowing down to a problem statement from the unknown can be an exhaustive effort. Start with questioning (not formal interviewing) in which your goal is to exclusively narrow down the chasm of possibilities. Start with the “Lassie Question: Can you show me?” Experiencing the problem first hand provides basic context. If the problem can’t be reproduced, try to provide supporting clues so that the unpredictable can become more predictable. You can’t necessarily replicate the performance problem at will. Do you have supporting data about your experience? Can you explain what happened to you? Do you know when it happened (smallest time window)? Has it happened before? If so when? Try to get down to the exact minute if possible. Has it happened to anyone else? What were they doing? Did it happen to them at the same time as you?
It comes off like you are asking dozens and dozens of questions, but in reality you are not. You are gathering basic context: Who, What, Where and When.
Be unwilling to announce a problem statement until you have confidence in the development of the problem statement (not the cause of the issue). Remember we are not diagnosing, we are simply collecting and announcing symptoms.
![Page 27: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/27.jpg)
27
![Page 28: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/28.jpg)
I’m not the creator of this methodology. I’m quite sure that others who are far more knowledgeable on the subject would tell you I’m possibly missing a step or that I am drawing out the process too far. A picture is truly worth a thousand words.
28
would tell you I’m possibly missing a step or that I am drawing out the process too far. A picture is truly worth a thousand words.
I will breakdown each element of the methodology in subsequent slides. I’ve designed a circular visualization for the obvious conclusion that I’ve come to over the years in which the process must revolve in order to come to root cause analysis.
Performance forensics doesn’t necessarily begin with evidence collection. Rather, it potentially begins long before an incident occurs. Let’s take an abstract example such as a person complains about chest pain. The person tells their spouse that at times they have unbearable pains, but eventually it goes away. It doesn’t happen enough and the pain isn’t so severe that it’s worth the time or the effort to go to the doctor. The process of convincing yourself that the symptoms you are experiencing is not what you really have is called diagnosis bias. I will talk about this in greater detail later.
This pain might go on and on for quite some time until it progresses. Analysis could be initiated at any point. More often then not, the complaints go unrealized and forensics is placed on hold. It comes back later on. The question is when. Typically when a terrible even occurs. It could be a heart attack or sadly a loss of life. The forensic engineer is tasked with tracing back why it happened, was foul play suspected and could it have been avoided.
I propose that at any time the methodology can be initiated. No major issue has to occur for performance forensics to begin. Symptoms do not necessarily have to show-up for the process to begin. You can call this what you want, but basically the collection of evidence, interviewing, modeling/visualizing and planning for the future is most commonly referred to as capacity planning. It’s not the much different from what we are trying to accomplish with performance forensics. The key difference is proactive behavior versus reactive behavior.
The methodology begins with the collection of data. We can call this data evidence. Evidence is collected in two ways: intended data collection and simulated data collection. When data is not available, we often go through the process of putting data collectors in place. The thought behind this is that if something happened once, it’s bound to happen again.
Interviewing is incorporated into the methodology. I will discuss techniques for interviewing. Understand that when humans are involved and asked to participate, you run the greatest chance for diagnosis bias and value attribution (two topics I will present in greater detail).
Next I will discuss why modeling and visualizing a problem can be critical at getting to the root cause of a performance issue.
![Page 29: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/29.jpg)
29
![Page 30: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/30.jpg)
30
![Page 31: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/31.jpg)
http://www.stevesouders.com/blog/2009/06/30/firefox-35-at-the-top/
31
http://assets.en.oreilly.com/1/event/29/The%20User%20and%20Business%20Impact%20of%20Server%20Delays,%20Additional%20Bytes,%20and%20HTTP%20Chunking%20in%20Web%20Search%20Presentation.pptx
![Page 32: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/32.jpg)
Source: http://www.flickr.com/photos/kaptainkobold/83359336/
32
![Page 33: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/33.jpg)
http://www.phpied.com/image-optimization-7-mistakes/
33
![Page 34: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/34.jpg)
Using pngrewrite to optimize this image
(http://entropymine.com/jason/pngrewrite/)
Cost Savings: 3KB or roughly 15%
34
![Page 35: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/35.jpg)
Using optipng to optimize this image
(http://sourceforge.net/projects/optipng/)
Cost Savings: 4MB or roughly 47.50%
35
![Page 36: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/36.jpg)
http://code.google.com/speed/page-speed/
36
http://developer.yahoo.com/yslow/
http://www.fiddler2.com/Fiddler2/version.asp
Great add-on to Fiddler2 is neXpert
http://videos.visitmix.com/MIX09/T53F
![Page 37: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/37.jpg)
Great presentation highlighting differences between tools: http://assets.en.oreilly.com/1/event/29/Website%20Performance%20Analysi
37
http://assets.en.oreilly.com/1/event/29/Website%20Performance%20Analysis%20Presentation.ppt
![Page 38: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/38.jpg)
38
![Page 39: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/39.jpg)
39
![Page 40: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/40.jpg)
40
![Page 41: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/41.jpg)
41
![Page 42: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/42.jpg)
42
![Page 43: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/43.jpg)
43
![Page 44: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/44.jpg)
44
![Page 45: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/45.jpg)
Source: http://www.flickr.com/photos/pollyann/2877940383/
45
![Page 46: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/46.jpg)
46
![Page 47: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/47.jpg)
http://www.fiddler2.com/Fiddler2/version.asp
47
Demo videos: http://www.fiddler2.com/Fiddler/help/video/default.asp
Example of use as a reverse proxy: http://blogs.msdn.com/nexpert/archive/2009/06/04/capturing-http-with-fiddler-as-a-reverse-proxy.aspx
![Page 48: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/48.jpg)
WebCast of neXpert: http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventI
48
http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventID=1032398774&EventCategory=5&culture=en-US&CountryCode=US
![Page 49: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/49.jpg)
49
![Page 50: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/50.jpg)
50
![Page 51: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/51.jpg)
51
![Page 52: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/52.jpg)
Can also consider using Microsoft VRTA: http://www.microsoft.com/downloads/details.aspx?FamilyID=119f3477-dced-
52
http://www.microsoft.com/downloads/details.aspx?FamilyID=119f3477-dced-41e3-a0e7-d8b5cae893a3&displaylang=en
![Page 53: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/53.jpg)
53
![Page 54: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/54.jpg)
54
![Page 55: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/55.jpg)
55
![Page 56: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/56.jpg)
56
![Page 57: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/57.jpg)
57
![Page 58: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/58.jpg)
JConsole on Steroids
58
Great presentation: http://www.javapassion.com/javase/VisualVM.pdf
Another Great Presentation: http://weblogs.java.net/blog/mandychung/archive/VisualVM-BOF-2007.pdf
![Page 59: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/59.jpg)
59
![Page 60: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/60.jpg)
60
![Page 61: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/61.jpg)
61
![Page 62: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/62.jpg)
62
![Page 63: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/63.jpg)
63
![Page 64: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/64.jpg)
64
![Page 65: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/65.jpg)
65
![Page 66: Shorter Version of BbWorld 09 Forensics Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54c1bc574a7959dd048b456e/html5/thumbnails/66.jpg)
66