background processing - pycon my 2015

BACKGROUND PROCESSING TO IMPROVE RESPONSE TIMEby Chew Kok Hoor kokhoor@solutionx.com.my

Python @ SolutionX

Django Web Applications Django Mobile Apps backend Django-based Point-of-Sales system (POS) Device Connectors

Issue

Issue: Webpages / Views that takes time to response Contact Us Page Registration Page

Possible causes: Sending email Slow writes to database, business rules etc.

What happens next

Users / Visitors / Customer complains

Boss ask us to resolve it, ASAP.

Short Term Resolution We spin while we can

What happens next after what happened next earlier

More Users / Visitors / Customers

Timeout errors

Why timeout?

In Linux, we use NGINX – GUNICORN - DJANGO There is a limit of exact concurrent request that

a server can handle. General formula for no. of workers is:

(2 x [number of cores]) + 1 Therefore, 8 cores = (2 x 8) + 1 = 17

Slow response = handle less requests per second

Possible solution

Background Processing

Pros n Cons Disadvantages

New process required On failure unable to notify users thru response to

request. Worse case, lost contact to user who submitted request. Need to monitor tasks not done / failed

Advantages Timeout issue resolved Users get quicker response Can choose to process N tasks at a time Can retry tasks etc., background process can afford

slower response

What should be done?

1. Break down tasks of slow responding requests

2. Process tasks in background

Preparation Stage

Perform slow running task in Background

Submit task for

processing

Tasks BreakdownPreparation Stage record necessary

information for task to execute properly or for user to track status.

activities to prevent inconsistency or breaking of business rules

preferably a means of contacting user when it is done (email, mobile notification, etc.)

Slow Running tasks Email sending Create and setup

new database Setup new user

login roles, etc. Complex

calculations or setup

Choosing Task Queue softwareSoftware Pro Consdjango-background-tasks(https://pypi.python.org/pypi/django-background-tasks)

- Very easy setup- Easy to use- Good for small volume

- Uses Django ORM for backend- Absolutely terrible formedium-to-high volume

RQ(http://python-rq.org/)(https://github.com/ui/django-rq)

- Flexible, powered by Redis- Lower memory footprint than Celery- Great for high volume

- Not as many features as Celery- Medium difficulty setup- Redis only option for storage

Celery(http://www.celeryproject.org/)

- De facto Django standard- Manydifferent storage types- Flexible,- Full-featured- Great for high volume

- Challenging setup- overkill for slower traffic sites

From Two Scoops of Django - Best Practices For Django 1.8 by Daniel Roy Greenfeld & Audrey Roy Greenfeld (http://twoscoopspress.org/)

Choosing Task Queue Software

Django Background Tasks- background processing code

INSTALLATION

pip install django-background-tasks

settings.py

INSTALLED_APPS += ( ‘background_task’,)

Django Background Tasks- background processing code

import time

from background_task import background

@background(schedule=0)def dbt_task(data): print "[START] django-background-tasks for: " + data.get("name","") time.sleep(10) print "[DONE] django-background-tasks for: " + data.get("name","")

Django Background Tasks- initiate background task

from . import taskstasks.dbt_task(request.POST.copy())

Django Background Tasks- start worker (only 1 allowed)

To start background processing process:

python manage.py process_tasks

django-rq- background processing codeINSTALLATION

pip install django-rq

settings.py

INSTALLED_APPS += ( 'django_rq',)

RQ_QUEUES = { 'default': { 'HOST': 'localhost', 'PORT': 6379, 'DB': 0, 'DEFAULT_TIMEOUT': 360, }}

django-rq- background processing code

import time

def rq_task(data): print "[START] rq for: " + data.get("name","") time.sleep(10) print "[DONE] rq for: " + data.get("name","")

django-rq- initiate background task

from . import rq_tasksimport django_rq

django_rq.enqueue(rq_tasks.rq_task, request.POST.copy())

django-rq- start worker(s)

First, start redis (e.g. /usr/local/bin/redis-server)

To start background processing process (supports more than 1 worker(s))

python manage.py rqworker default

Demo

Other Practical Usage of Background ProcessingNot just for response time

Scheduling (require additional components, e.g. RQ Scheduler)

Enforce a separation between front-end logic and back-end logic

Batch processing (daily import / export of data)

Reports generation Maintenance - expiring old sessions, sweeping

caches

FAQ

Chew Kok HoorDirector

SolutionX Software Sdn Bhdkokhoor@solutionx.com.my