Python vs JLizard.... a python logging experience

Download Python vs JLizard.... a python logging experience

Post on 19-May-2015




0 download


The presentation will give a general overview of logging and why it is becoming more important for cloud based systems. In particular it will focus on limitations in PAAS logging infrastructures and will outline how more powerful logging can be achieved on python platforms such as app engine giving details on python/app engine plug in libraries that have been developed. The talk will also outline how JLizard has made wide use of python as part of its log management as a service technology (


  • 1. Log Management As A Service Oct 2011 Python vs. JLizard

2. Outline A Logging Story About Us Data Data Everywhere (i.e. Logs Logs Everywhere) Logging in the Cloud & PAAS Python vs JLizard 3. A Logging Story 4. Can you see the error?25 log events in this window 5. Can you see the error?Consider 10,000 events per second 6. 7. JLizard & Logentries UCD spin out company Founded by Viliam Holub & Trevor Parsons Based on UCD/IBM Research (Enterprise Ireland) Participated in National Digital Research Centres Launchpad program(styled on Ycombinator) Nova UCD spin out award EI Funded Company Dogpatch Europe Company Logentries Launched in Q3 2011 8. About Me Trevor Parsons Tech Background Enterprise Software Design, Monitoring Tools, Data Mining Java Java Java Performance & Testing Tools Startups Crovan: Training & Consultancy JLizard 9. Data Data Everywhere(i.e. Logs Logs Everywhere) Estimates are that 150 billion gigabytes (exabytes) of data was created in2005; This year about 8 times that amount (1,200 exabytes) will be created- The Economist Feb 2010 The amount of enterprise data will grow about 650% over the next fiveyears, the vast majority of it unstructured, or not included in any database. - Gartner 2009 Log data is the fasted-growing data source at large organizations -CNET magazine 2010 Many organizations are currently producing terabytes of log data per month- CIO.Net 2009 10. Log Maths100,000 log messages / second x 300 bytes / logmessage ~ 28.6 MB x 3600 seconds ~ 100.6 GB / hour x 24 hours ~ 2.35 TB / day x 365 days ~ 860.5 TB / year x 3 years ~ 2.52 PBFrom Anton Chuvakins Blog Aug 2010 11. Typical Log VolumesCustomer TypeLog Volumes Events per Second Events per DayLarge Cloud Provider 50 Terabyes per Day 2,000,000 172,000,000,000Large Social Media 25 Terabytes per Day 1,000,000OrganisationTelecom Middleware/1 Terabyte per Day50,0000ApplicationsLarge Organisation 300 GB Per Day15,000(>1000 employees)Online Marketing Org 100 GB per day5,000 432,000,000Small10 GBs per Day500Data CentreSAAS Educational 5Gbs Per Day250ToolsSingle IBM Test Team 2 GBs per Day 100Online Multimedia700Mbs Per Day35Early Stage Start up 50Mbs Per Day 252,000,000 12. ConsequencesHacked System Slow Web SiteSystem Crash Every minute that Facebook is down costs Zynga $10,000, September 2010 13. Logging - A Cloud RequirementServer to Admin Ratio Increasing Significantly, from 10:1 to 500:1, July 2010 Compliance and Security Legislation Driving Requirement forLog Management Where Is My Elastic Log Management Service? & networkworld, September 2010 14. PAAS Logging Limited Logging Infrastructure App Engine limited log buffers Heroku 500 events for free No/Limited access to File System No Log Storage How to debug without any logs???? 15. Python vs. JLizardPython @logentries Logentries Agent Web interface Django App Engine Plug ins 16. Logentries Python AgentWhy Python? Light weight agent Runs on Linux Out Of The Box Windows support created an exe 17. Logentries Python Agent Features:Available on github: -Authenticate with Logentries Account -Register your machine -Configure your logs -Monitor Logs -Export LogsWhat it does: -Push Logs Communicates with Logentries API -Filter Logs Uses SSL -Configure your logs (clusters /apps) Data compressed on the wire -Navigate and Manage your account -Windows Event Logs Other Stuff: Run as daemon Run as windows service 18. Django Web Site Notifications User Authentication Landing Pages/Sign up Billing (Recurly Python Interface) Application UI 19. App Engine LoggingLogging on App Engine: Nice interface Limited Buffers Can download logs periodically No Trend Analysis No Business Intelligence App Engine Roadmap - Improvements 20. App Engine LoggingLogging to Logentries: In-Process Logging Push-Queue Logging Pull-Queue Logging. 21. Use Cases & Platforms Prevent System Crash Windows/Linux/Mac Reduce Log Analysis Time Syslog Forwarding Business Analytics Developer Libs Ruby, Java, C#, Python, Upload Saved LogsNode.js Engineyard Instant Access to All Logs Heroku App Engine Mobile Device Logs Cloudbees Secure Log Storage Mobile Devices Android 22. Logentries 23. Logentries 24. Logentries 25. Logentries 26. Logentries 27. Logentries 28. QuestionsGet a free