python vs jlizard.... a python logging experience
DESCRIPTION
The presentation will give a general overview of logging and why it is becoming more important for cloud based systems. In particular it will focus on limitations in PAAS logging infrastructures and will outline how more powerful logging can be achieved on python platforms such as app engine giving details on python/app engine plug in libraries that have been developed. The talk will also outline how JLizard has made wide use of python as part of its log management as a service technology (Logentries.com).TRANSCRIPT
Log Management As A Service
Oct 2011
Python vs. JLizard
Outline
• A Logging Story
• About Us
• Data Data Everywhere (i.e. Logs Logs Everywhere)
• LogEntries.com
• Logging in the Cloud & PAAS
• Python vs JLizard
A Logging Story
Can you see the error?
25 log events in this window
Can you see the error?
Consider 10,000 events per second
logentries.com
• UCD spin out company
• Founded by Viliam Holub & Trevor Parsons
• Based on UCD/IBM Research (Enterprise Ireland)
• Participated in National Digital Research Centre’s Launchpad program
(styled on Ycombinator)
• Nova UCD spin out award
• EI Funded Company
• Dogpatch Europe Company
• Logentries Launched in Q3 2011
JLizard & Logentries
• Trevor Parsons
– Tech Background
• Enterprise Software Design, Monitoring Tools, Data Mining
• Java Java Java
• Performance & Testing Tools
– Startups
• Crovan: Training & Consultancy
• JLizard
About Me
Data Data Everywhere (i.e. Logs Logs Everywhere)
• Estimates are that 150 billion gigabytes (exabytes) of data was created in
2005; This year about 8 times that amount (1,200 exabytes) will be created
- The Economist Feb 2010
• “The amount of enterprise data will grow about 650% over the next five
years, the vast majority of it unstructured, or not included in any database. “
- Gartner 2009
• Log data is the fasted-growing data source at large organizations -
CNET magazine 2010
• Many organizations are currently producing terabytes of log data per month
- CIO.Net 2009
100,000 log messages / second x 300 bytes / log
message ~ 28.6 MB
x 3600 seconds ~ 100.6 GB / hour
x 24 hours ~ 2.35 TB / day
x 365 days ~ 860.5 TB / year
x 3 years ~ 2.52 PB
From Anton Chuvakin’s Blog Aug 2010
http://chuvakin.blogspot.com/
Log Maths
Customer Type Log Volumes Events per Second Events per Day
Large Cloud Provider 50 Terabyes per Day 2,000,000 172,000,000,000
Large Social Media
Organisation
25 Terabytes per Day 1,000,000
Telecom Middleware/
Applications
1 Terabyte per Day 50,0000
Large Organisation
(>1000 employees)
300 GB Per Day 15,000
Online Marketing Org 100 GB per day 5,000 432,000,000
Small
Data Centre
10 GBs per Day 500
SAAS Educational
Tools
5Gbs Per Day 250
Single IBM Test Team 2 GBs per Day 100
Online Multimedia 700Mbs Per Day 35
Early Stage Start up 50Mbs Per Day 25 2,000,000
Typical Log Volumes
Consequences
…Hacked System …Slow Web Site …System Crash
“Every minute that Facebook is down costs Zynga $10,000” SocialMediaInfluence.com, September 2010
Logging - A Cloud Requirement
Server to Admin Ratio Increasing Significantly, from 10:1 to 500:1 Computerworld.com, July 2010
Compliance and Security Legislation Driving Requirement for Log Management
Where Is My Elastic Log Management Service? securecloudreview.com & networkworld, September 2010
PAAS Logging
• Limited Logging Infrastructure
– App Engine – limited log buffers
– Heroku – 500 events for free
• No/Limited access to File System
• No Log Storage
• How to debug without any logs????
Python vs. JLizard
Python @logentries
• Logentries Agent
• Web interface – Django
• App Engine Plug ins
Logentries Python Agent
Why Python?
• Light weight agent
• Runs on Linux Out Of The Box
• Windows support
– created an exe
Available on github:
www.github.com/logentries
What it does:
– Communicates with Logentries API
– Uses SSL
– Data compressed on the wire
Logentries Python Agent
Features:
- Authenticate with Logentries Account
- Register your machine
- Configure your logs
- Monitor Logs
- Export Logs
- Push Logs
- Filter Logs
- Configure your logs (clusters /apps)
- Navigate and Manage your account
- Windows Event Logs
Other Stuff:
– Run as daemon
– Run as windows service
Django
• Web Site
• Notifications
• User Authentication
• Landing Pages/Sign up
• Billing (Recurly Python Interface)
• Application UI
App Engine Logging
Logging on App Engine:
• Nice interface
• Limited Buffers
• Can download logs periodically
• No Trend Analysis
• No Business Intelligence
• App Engine Roadmap - Improvements
App Engine Logging
Logging to Logentries:
• In-Process Logging
• Push-Queue Logging
• Pull-Queue Logging.
Use Cases & Platforms
• Windows/Linux/Mac
• Syslog Forwarding
• Developer Libs
• Ruby, Java, C#, Python,
Node.js
• Engineyard
• Heroku
• App Engine
• Cloudbees
• Mobile Devices
• Android
• Prevent System Crash
• Reduce Log Analysis Time
• Business Analytics
• Upload Saved Logs
• Instant Access to All Logs
• Mobile Device Logs
• Secure Log Storage
Logentries
Logentries
Logentries
Logentries
Logentries
Logentries
Questions
Get a free account: www.logentries.com