Introduction

Celery is an asynchronous task/job queue framework that allows you to create distributed systems, where tasks (execution units) are executed concurrently on multiple workers using multiprocessing. It also supports scheduling and scales really well since you can horizontally scale workers.

Celery feature benefits

Celery is great at firing both synchronous and asynchronous tasks such as sending emails, processing credit cards, writing transactions to a general ledger, etc. One of its most beneficial features is the ability to chain multiple tasks to create workflows.

Celery features include:

• Monitoring
• Workflows
• Time & Rate Limits
• Scheduling
• Auto reloading
• Auto scaling
• Resource Leak Protection
• User Components

How does Celery work

Celery requires a message broker. This broker acts as a middleman sending and receiving messages to Celery workers which in turn process tasks as they receive them. Usually, Celery’s recommended message broker is RabbitMQ and Celery beat is used for scheduling that kicks off tasks at regular intervals, which are then executed by the worker nodes available in the cluster.

Here are some points to understand RabbitMQ and Celery Flower:

RabbitMQ is a complete and highly reliable enterprise messaging system based on the emerging AMQP (Advanced Message Queuing Protocol) standard and a proven platform, offering exceptionally high reliability, availability and scalability.

Celery Flower is a real time monitoring tool used to monitor celery events like Task progress, Task details, Task statistics etc., and also acts as a remote control for the celery process.

The following diagram illustrates the RabbitMQ process flow:

Use Case

Let’s continue with the use case of celery with RabbitMQ as a messaging broker:

• Install Celery and Celery dependencies
• Install RabbitMQ and Start RabbitMQ server
• Create a Celery instance using Python
• Start Celery beat
• Start Celery workers
• Install Celery flower

Solution

Install Celery and Celery dependencies

Install Celery

Use the pip command to install Celery:

$ pip install celery

Install Celery dependencies

Use the pip command to install Celery dependencies:

$ pip install pytz
$ pip install Billiard

Install RabbitMQ and Start RabbitMQ server

Install RabbitMQ

Download and install RabbitMQ from this link:

http://www.rabbitmq.com/download.html

Start RabbitMQ server

We need to configure RabbitMQ for message broker services before running the celery. Once the RabbitMQ is successfully started it can be checked using the web UI located at:

http://localhost:15672/

Note: Username and Password is default.

Create a Celery instance using Python

Create a file called tasks.py

from celery import Celery
celery = Celery('tasks')
celery.config_from_object('celeryconfig')

Creating tasks

@celery.task
def add(x, y):
    return x + y

@celery.task
def mul(x, y):
    return x * y

Start Celery Beat

Create a celeryconfig.py file for configuration and scheduling. You can schedule task execution, for example, at a particular time of day or day of the week, using the crontab schedule type given below:

from celery.schedules import crontab

BROKER_URL = 'amqp://guest:guest@localhost//'

CELERYBEAT_SCHEDULE = {
    'every-minute_add': {
        'task': 'app.add',
        'schedule': crontab(minute='*/1'),
        'args':(2,2),
    },
    'every-minute_mul': {
        'task': 'app.mul',
        'schedule': crontab(minute='*/1'),
        'args':(3,4),
    },

}

The following command starts Celery beat:

$ celery -A tasks beat

Start Celery Workers

We can now start worker processes that will be able to accept connections from applications. It will use the file we just created to learn about the tasks it can perform.

Starting a worker instance is as easy as calling out the application name with the celery command.

$ celery -A tasks worker --loglevel=info

The above screen shot shows tasks queue with task list and “Celery ready” console for RabbitMQ connection states.

Once we have successfully started RabbitMQ with Celery, we should login RabbitMQ for checking the “Overview”, “Connections” and “Channels”.

The screen shot below explains the celery process with RabbitMQ messaging broker services GUI:

The screen shot also shows “queued messages” and “message rates” with analytic information by clicking the “Overview” tab. Other tab options include “Connections” to show task names, Protocol, Client (From and to), Timeout, “Channels” to show the Channels list, Virtual host information and State results, “Exchanges”, “Queues” and “Admin”.

Install Celery Flower

Installation

The following command installs Celery Flower:

$pip install flower

Usage

Once the Celery Flower is successfully started we can check using the web UI located at:

http://localhost:5555

The following are the output screens:

We can also see Workers tab with Celery names and status (Online or Offline), Concurrency, Completed tasks, Running Tasks and queues.

The screen shot below depicts Celery monitoring with RabbitMQ showing succeeded tasks and Task times analytics process:

Conclusion

• In this blog, we have discussed Celery as a message processor with RabbitMQ.
• We have created how to schedule the real time task using Celery beat.
• We can easily monitor the real time Celery process using Celery Flower.

References

• http://celery.readthedocs.org/en/latest/getting-started/first-steps-with-celery.html
• http://celery.readthedocs.org/en/latest/index.html