A worker queue example with Python and Celery

Where the CronJob is well positioned to run repeated tasks at a specific schedule, another common need is to process a series of work items more or less constantly. A job is well oriented to running a single task until it is complete, but if the volume of things you need to process is large enough, it may be far more effective to maintain a constant process to work on those items.

A common pattern to accommodate this kind of work uses a message queue, as shown here:

With a message queue, you can have an API frontend that creates the work to be run asynchronously, move that into a queue, and then have a number of worker processes pull from the queue to do the relevant work. Amazon has a web-based service supporting exactly this pattern of processing called Simple Queue Service (SQS). A huge benefit of this pattern is decoupling the workers from the request, so you can scale each of those pieces independently, as required.

You can do exactly the same...