Using Celery with Django

Introduction

In a recent Django project I needed to issue some HTTP requests to a website, save the results and then display the information to the user. I could have issued those HTTP requests in a Django view, however the requests could take up to 10 seconds to return, and I didn’t want the page hanging that long for the user. I ended up using Celery, an asynchronous task queue written in Python, to execute the requests.

Installation

Celery
Celery is in pypi so installation is simple.

easy_install celery

pip install celery

Or download the source code, extract it and

python setup.py install

Django + Celery
In a high traffic production environment Celery should be paired with the RabbitMQ backend. However for my low traffic application I went with the django-kombu backend. django-kombu uses the Django database as a message store and requires very little configuration compared to other backends. We also need the djcelery package.

easy_install django-kombu

easy_install django-celery

Configuration

Before we can start using Celery we need configure our Django project. In settings.py make sure you add djkombu and djcelery to the INSTALLED_APPS tuple. Also make sure your database is configured and working.

Also add the following lines to settings.py.

import djcelery
djcelery.setup_loader()

BROKER_BACKEND = "djkombu.transport.DatabaseTransport"
#celery
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
BROKER_VHOST = "/"

Defining Tasks

All Celery tasks should be defined in PROJECT/APPNAME/tasks.py. You can define your tasks as methods with a Celery decorator, or as classes which inherit from a base Celery Task class. I prefer the class definitions. Each task must define a run method (which is called when the task is executed). Finally, each task must be registered with Celery so that they can be found. Below is a modified version of one of my Tasks.

import pycurl
from celery.task import Task
from celery.registry import tasks
from website.models import Website

class CheckWebsiteTask(Task):

	def run(self, ip, **kwargs):

		p, created = Website.objects.get_or_create(ip=ip)

		try:
			c = pycurl.Curl()
			c.setopt(pycurl.URL, ip)
			c.setopt(pycurl.TIMEOUT, 10)

			c.perform()

			p.alive = True

		except Exception, e:
			print e
			p.alive = False

		p.save()

tasks.register(CheckWebsiteTask)

Calling Tasks

Typically you will want to call your tasks somewhere in your view code. Just import your task and call it’s delay method, passing in the arguments that you specified in the run method of your task. The task will then be added to the queue within 5 or 10 seconds and the queue will then start to clear itself.

from django.http import HttpResponseRedirect

from website.tasks import CheckWebsiteTask
from website.models import Website

def check(request,id):

	website = Website.objects.get(id=id)

	CheckWebsiteTask.delay(website.ip)

	return HttpResponseRedirect('/')

Starting Celery

python manage.py celeryd -l info --settings=settings

Celery In The Real World

If you want to learn more about Celery then I recommend you read up on the various articles and tutorials on the Celery website. There is a list of Django apps that use Celery in one way or another, you may be able to use some of these in your own applications. Finally you can browse the source code of celery and django-celery on github to learn how they work under the hood.