logging in python for large applications

Why use logging Basics When to use what? Logbook Error reporting The end

Logging in PythonFor complex applications

Fayaz Yusuf Khan

20 December 2013


Why use logging?

• All errors happen in runtime.

• Testing can never be 100%


import logging

logging.warning(’Watch out!’) # will print a message

logging.info(’I told you so’) # will not print anything

Output:

WARNING:root:Watch out!


Logging to a file

import logging

logging.basicConfig(filename=’example.log’,

level=logging.DEBUG)

logging.debug(’This message should go to the log file’)

logging.info(’So should this’)

logging.warning(’And this , too’)

Output:

DEBUG:root:This message should go to the log file

INFO:root:So should this

WARNING:root:And this, too


Initializing

logger = logging.getLogger(__name__)


print()

Display console output for ordinary usage of a command line scriptor program


logging.info()

Report events that occur during normal operation of a program(e.g. for status monitoring or fault investigation)


logging.debug()

Very detailed output for diagnostic purposes


warnings.warn()

In library code if the issue is avoidable and the client applicationshould be modified to eliminate the warning.


logging.warning()

If there is nothing the client application can do about thesituation, but the event should still be noted


Raise an exception

Report an error regarding a particular runtime event


logging.error() or logging.exception()

Report suppression of an error without raising an exception (e.g.error handler in a long-running server process)


logging.critical()

A serious error, indicating that the program itself may be unable tocontinue running.


Logbookpythonhosted.org/Logbook

Logbook is a logging sytem for Python that replaces the standardlibrary’s logging module.


>>> from logbook import Logger

>>> log = Logger(’Logbook ’)

>>> log.info(’Hello , World!’)

[2010 -07 -23 16:34] INFO: Logbook: Hello , World!


Context managers everywhere!

from logbook import SyslogHandler

error_handler = SyslogHandler(’logbook example ’,

level=’ERROR’)

with error_handler.applicationbound ():

# whatever is executed here and an error is logged to

# the error handler

...


Getting your libraries and old code to behave

from logbook.compat import redirect_logging

redirect_logging ()


Injecting context

import os

from logbook import Processor

def inject_cwd(record ):

record.extra[’cwd’] = os.getcwd ()

with my_handler.applicationbound ():

with Processor(inject_cwd ). applicationbound ():

# everything logged here will have the current

# working directory in the log record.

...


Formatting in Logging

import logging

logging.basicConfig(format=’%( levelname)s:%( message)s’,

level=logging.DEBUG)


Formatting in Logbook

>>> from logbook import StderrHandler

>>> handler = StderrHandler(

... format_string=’{record.channel }: {record.message}’)

>>> handler.formatter

<logbook.handlers.StringFormatter object at 0x100641b90 >


Custom formatters

>>> def my_formatter(record , handler ):

... return record.message

...

>>> handler.formatter = my_formatter

Tip:

Or use Jinja!


Extra fields

handler = StderrHandler(format_string=

’{record.channel }: {record.message) [{ record.extra[cwd]}]’)


Redis

import logbook

from logbook.queues import RedisHandler

handler = RedisHandler ()

l = logbook.Logger ()

with handler:

l.info(’Your log message ’)


RabbitMQ

Handler

handler = RabbitMQHandler(’amqp :// guest:guest@localhost //’,

queue=’my_log ’)

Subscriber

with target_handler:

subscriber = RabbitMQSubscriber(

’amqp :// guest:guest@localhost //’, queue=’my_log ’)

subscriber.dispatch_forever ()


ZeroMQ

Don’t use it. (It’s broken)


Minimize logging

>>> from logbook import FingersCrossedHandler , StderrHandler

>>> handler = FingersCrossedHandler(StderrHandler ())

>>> with handler.applicationbound ():

... logbook.info(’Hello’)

...

...

>>> handler = FingersCrossedHandler(StderrHandler ())

>>> with handler.applicationbound ():

... logbook.info(’Hello’)

... logbook.error(’Bye’)

...

...

[2013 -12 -16 11:12] INFO: Generic: Hello

[2013 -12 -16 11:12] ERROR: Generic: Bye


TestHandler

>>> from logbook import TestHandler , Logger

>>> logger = Logger(’Testing ’)

>>> handler = TestHandler ()

>>> handler.push_thread ()

>>> logger.warn(’Hello World’)

>>> handler.records

[<logbook.base.LogRecord object at 0x100640cd0 >]

>>> handler.formatted_records

[u’[WARNING] Testing: Hello World ’]


Github

github.com/mitsuhiko/logbook


Error reporting

Emails?


Airbrake

Airbrake is the leading exception reporting service, currentlyproviding error tracking for 50,000 applications with support for 18

programming languages.


Sign up

airbrake.io


Flask

from flask import Flask , request

from airbrake import AirbrakeErrorHandler

import gevent

@app.errorhandler (500)

def internal_error(error ):

if app.config[’EXCEPTION_LOGGING ’]:

handler = AirbrakeErrorHandler(

api_key=app.config[’AIRBREAK_API_KEY ’],

env_name=ENV , request_url=request.url ,

request_path=request.path ,

request_method=request.method ,

request_args=request.args ,

request_headers=request.headers)

gevent.spawn(handler.emit , error)


Django

AIRBRAKE = {

’API_KEY ’: ’your_api_key_here ’,

’USE_SSL ’: True ,

’TIMEOUT ’: 5,

’ENVIRONMENT ’: ’unique_name_for_environment ’,

}


Errbit

github.com/errbit/errbit


Sentrygetsentry.com

Sentry is a realtime event logging and aggregation platform. At itscore it specializes in monitoring errors and extracting all the

information needed to do a proper post-mortem without any of thehassle of the standard user feedback loop.


Detail


More detail


Install the server

sentry.readthedocs.org


Sentry + Logbookraven.readthedocs.org/en/latest/config/logbook.html

Install raven

pip install raven

Your app code will have this at the entry point

from raven.handlers.logbook import SentryHandler

client = Client (...)

sentry_handler = SentryHandler(client)

with sentry_handler.applicationbound ():

# everything logged here will go to sentry.

...


Pay attention to formatting!Sentry will intelligently group messages if you use proper string formatting.

logger.error(’There was some {} error’, ’crazy’)

logger.error(’There was some {} error’, ’fun’)

logger.error(’There was some {} error’, 1)


Tagging

logger.error(’Could not do that’,

tags={’user email ’: ’foo@bar ’})


TicketingHandlerCreate unique tickets for log records and keep track of the number of times these log

records were created

from logbook import ERROR

from logbook.ticketing import TicketingHandler

handler = TicketingHandler(’postgres :// localhost/database ’,

level=ERROR)

with handler:

# everything in this block and thread will be handled by

# the ticketing database handler

...


Phew! It’s finally over!

Reach me at fayaz at dexetra dot com.

logging in python for large applications

Technology