python vs jlizard.... a python logging experience

29
Log Management As A Service Oct 2011 Python vs. JLizard

Upload: python-ireland

Post on 19-May-2015

541 views

Category:

Technology


0 download

DESCRIPTION

The presentation will give a general overview of logging and why it is becoming more important for cloud based systems. In particular it will focus on limitations in PAAS logging infrastructures and will outline how more powerful logging can be achieved on python platforms such as app engine giving details on python/app engine plug in libraries that have been developed. The talk will also outline how JLizard has made wide use of python as part of its log management as a service technology (Logentries.com).

TRANSCRIPT

Page 1: Python vs JLizard.... a python logging experience

Log Management As A Service

Oct 2011

Python vs. JLizard

Page 2: Python vs JLizard.... a python logging experience

Outline

• A Logging Story

• About Us

• Data Data Everywhere (i.e. Logs Logs Everywhere)

• LogEntries.com

• Logging in the Cloud & PAAS

• Python vs JLizard

Page 3: Python vs JLizard.... a python logging experience

A Logging Story

Page 4: Python vs JLizard.... a python logging experience

Can you see the error?

25 log events in this window

Page 5: Python vs JLizard.... a python logging experience

Can you see the error?

Consider 10,000 events per second

Page 6: Python vs JLizard.... a python logging experience

logentries.com

Page 7: Python vs JLizard.... a python logging experience

• UCD spin out company

• Founded by Viliam Holub & Trevor Parsons

• Based on UCD/IBM Research (Enterprise Ireland)

• Participated in National Digital Research Centre’s Launchpad program

(styled on Ycombinator)

• Nova UCD spin out award

• EI Funded Company

• Dogpatch Europe Company

• Logentries Launched in Q3 2011

JLizard & Logentries

Page 8: Python vs JLizard.... a python logging experience

• Trevor Parsons

– Tech Background

• Enterprise Software Design, Monitoring Tools, Data Mining

• Java Java Java

• Performance & Testing Tools

– Startups

• Crovan: Training & Consultancy

• JLizard

About Me

Page 9: Python vs JLizard.... a python logging experience

Data Data Everywhere (i.e. Logs Logs Everywhere)

• Estimates are that 150 billion gigabytes (exabytes) of data was created in

2005; This year about 8 times that amount (1,200 exabytes) will be created

- The Economist Feb 2010

• “The amount of enterprise data will grow about 650% over the next five

years, the vast majority of it unstructured, or not included in any database. “

- Gartner 2009

• Log data is the fasted-growing data source at large organizations -

CNET magazine 2010

• Many organizations are currently producing terabytes of log data per month

- CIO.Net 2009

Page 10: Python vs JLizard.... a python logging experience

100,000 log messages / second x 300 bytes / log

message ~ 28.6 MB

x 3600 seconds ~ 100.6 GB / hour

x 24 hours ~ 2.35 TB / day

x 365 days ~ 860.5 TB / year

x 3 years ~ 2.52 PB

From Anton Chuvakin’s Blog Aug 2010

http://chuvakin.blogspot.com/

Log Maths

Page 11: Python vs JLizard.... a python logging experience

Customer Type Log Volumes Events per Second Events per Day

Large Cloud Provider 50 Terabyes per Day 2,000,000 172,000,000,000

Large Social Media

Organisation

25 Terabytes per Day 1,000,000

Telecom Middleware/

Applications

1 Terabyte per Day 50,0000

Large Organisation

(>1000 employees)

300 GB Per Day 15,000

Online Marketing Org 100 GB per day 5,000 432,000,000

Small

Data Centre

10 GBs per Day 500

SAAS Educational

Tools

5Gbs Per Day 250

Single IBM Test Team 2 GBs per Day 100

Online Multimedia 700Mbs Per Day 35

Early Stage Start up 50Mbs Per Day 25 2,000,000

Typical Log Volumes

Page 12: Python vs JLizard.... a python logging experience

Consequences

…Hacked System …Slow Web Site …System Crash

“Every minute that Facebook is down costs Zynga $10,000” SocialMediaInfluence.com, September 2010

Page 13: Python vs JLizard.... a python logging experience

Logging - A Cloud Requirement

Server to Admin Ratio Increasing Significantly, from 10:1 to 500:1 Computerworld.com, July 2010

Compliance and Security Legislation Driving Requirement for Log Management

Where Is My Elastic Log Management Service? securecloudreview.com & networkworld, September 2010

Page 14: Python vs JLizard.... a python logging experience

PAAS Logging

• Limited Logging Infrastructure

– App Engine – limited log buffers

– Heroku – 500 events for free

• No/Limited access to File System

• No Log Storage

• How to debug without any logs????

Page 15: Python vs JLizard.... a python logging experience

Python vs. JLizard

Python @logentries

• Logentries Agent

• Web interface – Django

• App Engine Plug ins

Page 16: Python vs JLizard.... a python logging experience

Logentries Python Agent

Why Python?

• Light weight agent

• Runs on Linux Out Of The Box

• Windows support

– created an exe

Page 17: Python vs JLizard.... a python logging experience
Page 18: Python vs JLizard.... a python logging experience

Available on github:

www.github.com/logentries

What it does:

– Communicates with Logentries API

– Uses SSL

– Data compressed on the wire

Logentries Python Agent

Features:

- Authenticate with Logentries Account

- Register your machine

- Configure your logs

- Monitor Logs

- Export Logs

- Push Logs

- Filter Logs

- Configure your logs (clusters /apps)

- Navigate and Manage your account

- Windows Event Logs

Other Stuff:

– Run as daemon

– Run as windows service

Page 19: Python vs JLizard.... a python logging experience

Django

• Web Site

• Notifications

• User Authentication

• Landing Pages/Sign up

• Billing (Recurly Python Interface)

• Application UI

Page 20: Python vs JLizard.... a python logging experience

App Engine Logging

Logging on App Engine:

• Nice interface

• Limited Buffers

• Can download logs periodically

• No Trend Analysis

• No Business Intelligence

• App Engine Roadmap - Improvements

Page 21: Python vs JLizard.... a python logging experience

App Engine Logging

Logging to Logentries:

• In-Process Logging

• Push-Queue Logging

• Pull-Queue Logging.

Page 22: Python vs JLizard.... a python logging experience

Use Cases & Platforms

• Windows/Linux/Mac

• Syslog Forwarding

• Developer Libs

• Ruby, Java, C#, Python,

Node.js

• Engineyard

• Heroku

• App Engine

• Cloudbees

• Mobile Devices

• Android

• Prevent System Crash

• Reduce Log Analysis Time

• Business Analytics

• Upload Saved Logs

• Instant Access to All Logs

• Mobile Device Logs

• Secure Log Storage

Page 23: Python vs JLizard.... a python logging experience

Logentries

Page 24: Python vs JLizard.... a python logging experience

Logentries

Page 25: Python vs JLizard.... a python logging experience

Logentries

Page 26: Python vs JLizard.... a python logging experience

Logentries

Page 27: Python vs JLizard.... a python logging experience

Logentries

Page 28: Python vs JLizard.... a python logging experience

Logentries

Page 29: Python vs JLizard.... a python logging experience

Questions

Get a free account: www.logentries.com