logging in python for large applications

Why use logging Basics When to use what? Logbook Error reporting The end

Logging in PythonFor complex applications

Fayaz Yusuf Khan

20 December 2013

Why use logging Basics When to use what? Logbook Error reporting The end

Why use logging?

• All errors happen in runtime.

• Testing can never be 100%

Why use logging Basics When to use what? Logbook Error reporting The end

Why use logging?

• All errors happen in runtime.

• Testing can never be 100%

Why use logging Basics When to use what? Logbook Error reporting The end

Why use logging?

• All errors happen in runtime.

• Testing can never be 100%

Why use logging Basics When to use what? Logbook Error reporting The end

import logging

logging.warning(’Watch out!’) # will print a message

logging.info(’I told you so’) # will not print anything

Output:

WARNING:root:Watch out!

Why use logging Basics When to use what? Logbook Error reporting The end

import logging

logging.warning(’Watch out!’) # will print a message

logging.info(’I told you so’) # will not print anything

Output:

WARNING:root:Watch out!

Why use logging Basics When to use what? Logbook Error reporting The end

Logging to a file

import logging

logging.basicConfig(filename=’example.log’,

level=logging.DEBUG)

logging.debug(’This message should go to the log file’)

logging.info(’So should this’)

logging.warning(’And this , too’)

Output:

DEBUG:root:This message should go to the log file

INFO:root:So should this

WARNING:root:And this, too

Why use logging Basics When to use what? Logbook Error reporting The end

Logging to a file

import logging

logging.basicConfig(filename=’example.log’,

level=logging.DEBUG)

logging.debug(’This message should go to the log file’)

logging.info(’So should this’)

logging.warning(’And this , too’)

Output:

DEBUG:root:This message should go to the log file

INFO:root:So should this

WARNING:root:And this, too

Why use logging Basics When to use what? Logbook Error reporting The end

Initializing

logger = logging.getLogger(__name__)

Why use logging Basics When to use what? Logbook Error reporting The end

print()

Display console output for ordinary usage of a command line scriptor program

Why use logging Basics When to use what? Logbook Error reporting The end

logging.info()

Report events that occur during normal operation of a program(e.g. for status monitoring or fault investigation)

Why use logging Basics When to use what? Logbook Error reporting The end

logging.debug()

Very detailed output for diagnostic purposes

Why use logging Basics When to use what? Logbook Error reporting The end

warnings.warn()

In library code if the issue is avoidable and the client applicationshould be modified to eliminate the warning.

Why use logging Basics When to use what? Logbook Error reporting The end

logging.warning()

If there is nothing the client application can do about thesituation, but the event should still be noted

Why use logging Basics When to use what? Logbook Error reporting The end

Raise an exception

Report an error regarding a particular runtime event

Why use logging Basics When to use what? Logbook Error reporting The end

Raise an exception

Report an error regarding a particular runtime event

Why use logging Basics When to use what? Logbook Error reporting The end

logging.error() or logging.exception()

Report suppression of an error without raising an exception (e.g.error handler in a long-running server process)

Why use logging Basics When to use what? Logbook Error reporting The end

logging.critical()

A serious error, indicating that the program itself may be unable tocontinue running.

Why use logging Basics When to use what? Logbook Error reporting The end

Logbookpythonhosted.org/Logbook

Logbook is a logging sytem for Python that replaces the standardlibrary’s logging module.

Why use logging Basics When to use what? Logbook Error reporting The end

>>> from logbook import Logger

>>> log = Logger(’Logbook ’)

>>> log.info(’Hello , World!’)

[2010 -07 -23 16:34] INFO: Logbook: Hello , World!

Why use logging Basics When to use what? Logbook Error reporting The end

Context managers everywhere!

from logbook import SyslogHandler

error_handler = SyslogHandler(’logbook example ’,

level=’ERROR’)

with error_handler.applicationbound ():

# whatever is executed here and an error is logged to

# the error handler

...

Why use logging Basics When to use what? Logbook Error reporting The end

Getting your libraries and old code to behave

from logbook.compat import redirect_logging

redirect_logging ()

Why use logging Basics When to use what? Logbook Error reporting The end

Injecting context

import os

from logbook import Processor

def inject_cwd(record ):

record.extra[’cwd’] = os.getcwd ()

with my_handler.applicationbound ():

with Processor(inject_cwd ). applicationbound ():

# everything logged here will have the current

# working directory in the log record.

...

Why use logging Basics When to use what? Logbook Error reporting The end

Formatting in Logging

import logging

logging.basicConfig(format=’%( levelname)s:%( message)s’,

level=logging.DEBUG)

Why use logging Basics When to use what? Logbook Error reporting The end

Formatting in Logbook

>>> from logbook import StderrHandler

>>> handler = StderrHandler(

... format_string=’{record.channel }: {record.message}’)

>>> handler.formatter

<logbook.handlers.StringFormatter object at 0x100641b90 >

Why use logging Basics When to use what? Logbook Error reporting The end

Custom formatters

>>> def my_formatter(record , handler ):

... return record.message

...

>>> handler.formatter = my_formatter

Tip:

Or use Jinja!

Why use logging Basics When to use what? Logbook Error reporting The end

Custom formatters

>>> def my_formatter(record , handler ):

... return record.message

...

>>> handler.formatter = my_formatter

Tip:

Or use Jinja!

Why use logging Basics When to use what? Logbook Error reporting The end

Extra fields

handler = StderrHandler(format_string=

’{record.channel }: {record.message) [{ record.extra[cwd]}]’)

Why use logging Basics When to use what? Logbook Error reporting The end

Redis

import logbook

from logbook.queues import RedisHandler

handler = RedisHandler ()

l = logbook.Logger ()

with handler:

l.info(’Your log message ’)

Why use logging Basics When to use what? Logbook Error reporting The end

RabbitMQ

Handler

handler = RabbitMQHandler(’amqp :// guest:guest@localhost //’,

queue=’my_log ’)

Subscriber

with target_handler:

subscriber = RabbitMQSubscriber(

’amqp :// guest:guest@localhost //’, queue=’my_log ’)

subscriber.dispatch_forever ()

Why use logging Basics When to use what? Logbook Error reporting The end

ZeroMQ

Don’t use it. (It’s broken)

Why use logging Basics When to use what? Logbook Error reporting The end

Minimize logging

>>> from logbook import FingersCrossedHandler , StderrHandler

>>> handler = FingersCrossedHandler(StderrHandler ())

>>> with handler.applicationbound ():

... logbook.info(’Hello’)

...

...

>>> handler = FingersCrossedHandler(StderrHandler ())

>>> with handler.applicationbound ():

... logbook.info(’Hello’)

... logbook.error(’Bye’)

...

...

[2013 -12 -16 11:12] INFO: Generic: Hello

[2013 -12 -16 11:12] ERROR: Generic: Bye

Why use logging Basics When to use what? Logbook Error reporting The end

TestHandler

>>> from logbook import TestHandler , Logger

>>> logger = Logger(’Testing ’)

>>> handler = TestHandler ()

>>> handler.push_thread ()

>>> logger.warn(’Hello World’)

>>> handler.records

[<logbook.base.LogRecord object at 0x100640cd0 >]

>>> handler.formatted_records

[u’[WARNING] Testing: Hello World ’]

Why use logging Basics When to use what? Logbook Error reporting The end

Github

github.com/mitsuhiko/logbook

Why use logging Basics When to use what? Logbook Error reporting The end

Error reporting

Emails?

Why use logging Basics When to use what? Logbook Error reporting The end

Error reporting

Emails?

Why use logging Basics When to use what? Logbook Error reporting The end

Error reporting

Emails?

Why use logging Basics When to use what? Logbook Error reporting The end

Airbrake

Airbrake is the leading exception reporting service, currentlyproviding error tracking for 50,000 applications with support for 18

programming languages.

Why use logging Basics When to use what? Logbook Error reporting The end

Sign up

airbrake.io

Why use logging Basics When to use what? Logbook Error reporting The end

Flask

from flask import Flask , request

from airbrake import AirbrakeErrorHandler

import gevent

@app.errorhandler (500)

def internal_error(error ):

if app.config[’EXCEPTION_LOGGING ’]:

handler = AirbrakeErrorHandler(

api_key=app.config[’AIRBREAK_API_KEY ’],

env_name=ENV , request_url=request.url ,

request_path=request.path ,

request_method=request.method ,

request_args=request.args ,

request_headers=request.headers)

gevent.spawn(handler.emit , error)

Why use logging Basics When to use what? Logbook Error reporting The end

Django

AIRBRAKE = {

’API_KEY ’: ’your_api_key_here ’,

’USE_SSL ’: True ,

’TIMEOUT ’: 5,

’ENVIRONMENT ’: ’unique_name_for_environment ’,

}

Why use logging Basics When to use what? Logbook Error reporting The end

Errbit

github.com/errbit/errbit

Why use logging Basics When to use what? Logbook Error reporting The end

Sentrygetsentry.com

Sentry is a realtime event logging and aggregation platform. At itscore it specializes in monitoring errors and extracting all the

information needed to do a proper post-mortem without any of thehassle of the standard user feedback loop.

Why use logging Basics When to use what? Logbook Error reporting The end

Detail

Why use logging Basics When to use what? Logbook Error reporting The end

More detail

Why use logging Basics When to use what? Logbook Error reporting The end

Install the server

sentry.readthedocs.org

Why use logging Basics When to use what? Logbook Error reporting The end

Sentry + Logbookraven.readthedocs.org/en/latest/config/logbook.html

Install raven

pip install raven

Your app code will have this at the entry point

from raven.handlers.logbook import SentryHandler

client = Client (...)

sentry_handler = SentryHandler(client)

with sentry_handler.applicationbound ():

# everything logged here will go to sentry.

...

Why use logging Basics When to use what? Logbook Error reporting The end

Pay attention to formatting!Sentry will intelligently group messages if you use proper string formatting.

logger.error(’There was some {} error’, ’crazy’)

logger.error(’There was some {} error’, ’fun’)

logger.error(’There was some {} error’, 1)

Why use logging Basics When to use what? Logbook Error reporting The end

Tagging

logger.error(’Could not do that’,

tags={’user email ’: ’foo@bar ’})

Why use logging Basics When to use what? Logbook Error reporting The end

TicketingHandlerCreate unique tickets for log records and keep track of the number of times these log

records were created

from logbook import ERROR

from logbook.ticketing import TicketingHandler

handler = TicketingHandler(’postgres :// localhost/database ’,

level=ERROR)

with handler:

# everything in this block and thread will be handled by

# the ticketing database handler

...

Why use logging Basics When to use what? Logbook Error reporting The end

Phew! It’s finally over!

Reach me at fayaz at dexetra dot com.