![Page 1: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/1.jpg)
Finding the Needle in the Haystack
(Troubleshooting Distributed Systems)
Anthony MolinaroErlang Factory 2014
Wednesday, March 12, 14
![Page 2: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/2.jpg)
Web Services have gotten more complex
Wednesday, March 12, 14
![Page 3: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/3.jpg)
1 Tier (AKAClient/Server)
Wednesday, March 12, 14
![Page 4: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/4.jpg)
2 Tier
Wednesday, March 12, 14
![Page 5: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/5.jpg)
2 Tier Clustered
Wednesday, March 12, 14
![Page 6: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/6.jpg)
2 Tier - SAAS
Wednesday, March 12, 14
![Page 7: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/7.jpg)
3 Tier
Wednesday, March 12, 14
![Page 8: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/8.jpg)
3 Tier Clustered
Wednesday, March 12, 14
![Page 9: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/9.jpg)
3 Tier - SAAS
Wednesday, March 12, 14
![Page 10: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/10.jpg)
N Tier/SOA
Wednesday, March 12, 14
![Page 11: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/11.jpg)
N Tier/SOA - SAAS
Wednesday, March 12, 14
![Page 12: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/12.jpg)
Troubleshootingfor most
• Looking at logs for errors
• Capturing and viewing performance metrics, looking for visual patterns
• Try to reproduce errors based on a vague ticket description or a log line
• This becomes harder when you have dozens to hundreds of different systems to look through
Wednesday, March 12, 14
![Page 13: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/13.jpg)
Troubleshooting Distributed Systems• Perform an internet search for
"Troubleshooting Distributed Systems"
• "Traditional Approach"
• Geared towards overall system performance monitoring
• use NTP/synced ids/log everything you can/do something smart with it
Wednesday, March 12, 14
![Page 14: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/14.jpg)
The Trouble with the "Traditional Approach"• Large data volumes (can be mitigated with
sampling)
• Overhead on all requests (can be mitigated by going low level and forking packets)
• Geared towards general system performance and not application specific issues
Wednesday, March 12, 14
![Page 15: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/15.jpg)
What areApplication Issues?• A developer is trying to debug a web
request bridging multiple subsystems
• A customer calls a support number and describes an issue
• A QA engineer is seeing unexpected results with a new feature or bug fix
• A sales engineer notices as issue while demoing a product
Wednesday, March 12, 14
![Page 16: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/16.jpg)
What could cause those issues?
• Actual bugs
• Data discrepancies
• Partially failed components or services
• PEBKAC
Wednesday, March 12, 14
![Page 17: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/17.jpg)
A Possible Solution
• Cross language tracing of requests
• trigger a trace with an external input
• log lots of extra stuff for that request to a central location via UDP
• provide a way to view the data and drill down to the unexpected part
Wednesday, March 12, 14
![Page 18: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/18.jpg)
Evolution of a Solution
• 3 use cases
• 2000: Search Advertising (Goto/Overture)
• 2004: Content Match (Yahoo)
• 2010: Display Advertising (OpenX)
Wednesday, March 12, 14
![Page 19: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/19.jpg)
Search Advertising• Given some keywords sent to a search
engine
• Pick some Ads
• Include those ads in front of algorithmic results.
• Goto.com pioneered this in the late 90s
• Overture turned this into a service around 2000
Wednesday, March 12, 14
![Page 20: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/20.jpg)
2 Tier Distributed
Wednesday, March 12, 14
![Page 21: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/21.jpg)
Use Case
• Customer account manager gets a call from a customer who asks "Why am I not getting ads?"
Wednesday, March 12, 14
![Page 22: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/22.jpg)
Tools Available
• Light Weight Event System - lwes
• http://www.lwes.org/
• cross language event system
• UDP
• fire and forget messages (low overhead on clients)
Wednesday, March 12, 14
![Page 23: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/23.jpg)
Solution
• isotope
• demarcate a request via secret keywords
• identify the request via the timestamp
• if an isotope request, send lwes events containing perl data structures to a centralized server
• dump to a file and serve up in a browser
Wednesday, March 12, 14
![Page 24: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/24.jpg)
Isotope
Wednesday, March 12, 14
![Page 25: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/25.jpg)
Lessons Learned
• timestamps can lead to conflicts, so need to add some other sort of id
• structured data can be useful
• making data accessible via an internal web service can be useful
Wednesday, March 12, 14
![Page 26: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/26.jpg)
Content Match
• Given the content of a web page
• Determine the subject
• Pick ads relevant to the subject
• Built this at Yahoo in the mid-2000s
Wednesday, March 12, 14
![Page 27: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/27.jpg)
N Tier
Wednesday, March 12, 14
![Page 28: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/28.jpg)
Use Cases
• Developers wonder "Where did my request go?"
• which machines did it hit
• what data did it use to make it's decision
• Customer support gets asked "Why am I not getting ads?"
Wednesday, March 12, 14
![Page 29: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/29.jpg)
Tools
• lwes again
• multicast UDP
• command line listener of events
• similar to lwes-event-printing-listener in lwes C distribution
• able to filter based on an id
Wednesday, March 12, 14
![Page 30: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/30.jpg)
Solution• llog
• demarcate request with a secret query arg which accepted a non-zero positive integer
• id was passed through all communications between components
• when id is non-zero send extra information via multicast lwes to network
• view trace in terminal
Wednesday, March 12, 14
![Page 31: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/31.jpg)
But what about customer support?• Customer support couldn't use the
command line tool
• traces turned on for some number of requests
• captured via multicast lwes and put into database
• reports are generated
Wednesday, March 12, 14
![Page 32: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/32.jpg)
Trace
Wednesday, March 12, 14
![Page 33: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/33.jpg)
Lessons Learned
• Real time listening was useful for debugging, but there were many hacked together scripts to process trace information, and the output was not standardized so hard to parse
• Keeping around traces in a database for some time was very useful, but a relational database was limiting
Wednesday, March 12, 14
![Page 34: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/34.jpg)
Display Advertising
• Given a location on a webpage
• Pick the best ad for the user and webpage
• Currently doing this with OpenX
Wednesday, March 12, 14
![Page 35: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/35.jpg)
N Tier FTW!
Wednesday, March 12, 14
![Page 36: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/36.jpg)
Use Cases
• Why is my ad not showing?
• Where did my request go?
• How do I test a change to a subsystem?
• How do I find replication issues?
Wednesday, March 12, 14
![Page 37: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/37.jpg)
Tools• lwes
• mondemand (http://www.mondemand.org/)
• added structured output of stats/logs/traces on top of lwes
• mondemand-server
• collects traces as JSON objects
• simple UI for viewing
Wednesday, March 12, 14
![Page 38: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/38.jpg)
Solution
• demarcate request with a cookie containing two ids, an owner id and a trace id
• pass ids through to all services
• send trace messages to centralized server
• server captures and stores messages and provides UI for viewing
Wednesday, March 12, 14
![Page 39: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/39.jpg)
Mondemand
Wednesday, March 12, 14
![Page 40: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/40.jpg)
Lessons Learned
• A single id is not enough, you need at least 2 and possibly more
• The tool is useful for everyone from developers to QA to customer support
• Capture as much state as possible when tracing, you'll need it someday
Wednesday, March 12, 14
![Page 41: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/41.jpg)
Basic Examples
Wednesday, March 12, 14
![Page 42: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/42.jpg)
Erlang
mondemand:send_trace ( webserver, % identify program sending trace "trace_owner", % owner of trace "trace_id", % id for trace "received request", % message []) % extra data
Wednesday, March 12, 14
![Page 43: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/43.jpg)
Java
// identify program sending traceclient = new Client ("webserver");
HashMap<String, String> tmp = new HashMap<String, String> ();
client.traceMessage ( "trace_owner", // owner of trace "trace_id", // id for trace "received request", // message tmp); // extra data
Wednesday, March 12, 14
![Page 44: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/44.jpg)
Command Line
mondemand-tool -o lwes::127.0.0.1:20502 \ # identify program sending trace \ -p webserver \ # Owner of trace : id for trace : message \ -T "trace_owner:trace_id:received request"
Wednesday, March 12, 14
![Page 45: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/45.jpg)
Mondemand JSON
{ "SenderIP": "127.0.0.1", "SenderPort": 52823, "ReceiptTime": 1392874916206, "EventName": "MonDemand::TraceMsg", "mondemand.src_host": "renym.local", "mondemand.prog_id": "webserver", "mondemand.owner": "trace_owner", "mondemand.trace_id": "trace_id", "mondemand.message": "received request"}
Wednesday, March 12, 14
![Page 46: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/46.jpg)
Examples with Embedded JSON
Wednesday, March 12, 14
![Page 47: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/47.jpg)
Erlang
mondemand:send_trace ( webserver, % identify program sending trace "trace_owner", % owner of trace "trace_id", % id for trace "received request", % message [ { extra, % extra data can contain "{\"key\":\"value\"}" % json strings } ])
Wednesday, March 12, 14
![Page 48: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/48.jpg)
Java// identify program sending traceclient = new Client ("webserver");
HashMap<String, String> tmp = new HashMap<String, String> ();tmp.put ("extra", // extra data can contain "{\"key\":\"value\"}" // json strings );
client.traceMessage ( "trace_owner", // owner of trace "trace_id", // id for trace "received request", // message tmp); // extra data
Wednesday, March 12, 14
![Page 49: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/49.jpg)
Command Line
mondemand-tool -o lwes::127.0.0.1:20502 \ # identify program sending trace \ -p webserver \ # Owner of trace : id for trace : message \ -T "trace_owner:trace_id:received request" \ # extra data can contain json strings -t "extra:{\"key\":\"value\"}"
Wednesday, March 12, 14
![Page 50: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/50.jpg)
Mondemand JSON{ "SenderIP": "127.0.0.1", "SenderPort": 64613, "ReceiptTime": 1392875074968, "EventName": "MonDemand::TraceMsg", "mondemand.src_host": "renym.local", "mondemand.prog_id": "webserver", "mondemand.owner": "trace_owner", "mondemand.trace_id": "trace_id", "mondemand.message": "received request", "extra": { "key": "value" }}
Wednesday, March 12, 14
![Page 51: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/51.jpg)
Demo of UI
Wednesday, March 12, 14
![Page 52: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/52.jpg)
Final Thoughts
• When building new systems
• add the ability to add ids to a request in some ad hoc manner
• pass the ids throughout the system
• this lays the foundation for any number of tracing setups
Wednesday, March 12, 14
![Page 53: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/53.jpg)
Limitations/Future Work
• Large objects in traces
• UDP packet limits trace sizes
• QueAsy system for feeding traces back into a system as test cases
Wednesday, March 12, 14
![Page 54: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/54.jpg)
Questions?
Wednesday, March 12, 14
![Page 55: Finding the Needle in the Haystack (Troubleshooting ...€¦ · Finding the Needle in the Haystack (Troubleshooting Distributed Systems) Anthony Molinaro Erlang Factory 2014 ... a](https://reader034.vdocuments.mx/reader034/viewer/2022050210/5f5cd9b8b303f71b8916c629/html5/thumbnails/55.jpg)
Thanks!
• http://www.lwes.org/
• http://www.mondemand.org/
• http://github.com/djnym
Wednesday, March 12, 14