Celery in production

Here are some helpful tips on how to run a large Celery application in a production environment.

The first suggestion is to use a configuration module for your Celery application rather than configuring the Celery app in your worker code. Assuming that your configuration file is called config.py, you can pass it to a Celery application as follows:

import celery
app = celery.Celery('mergesort')
app.config_from_object('config')

Then, together with any other configuration directive that might be relevant to the specific application being developed, put the following code in config.py:

BROKER_URL = 'amqp://HOST1'
CELERY_RESULT_BACKEND = 'redis://HOST2'

Probably, the main performance-related suggestion would be to use more than one queue so that tasks can be prioritized and/or separated based on their expected runtime. Using multiple queues and routing tasks to the appropriate queue is a simple way to assign more horsepower (that is, workers) to one group of tasks. Celery...