Celery With Supervisor

Reading Time : ~ .

Celery:

Celery is a task queue with focus on real-time processing,while also supports task scheduling. Task queues are used  as  mechanisms to distribute work across multiple threads or machines. A task queues input is a unit of work called a task,dedicated worker processes and constantly moniter the queue for new work to perform. Celery communicates via messages using a  broker to mediate between workers and clients.To initiate a task client puts a message on the queue, then the broker delivers that message to a worker.

Note: You can get more about celery http://celery.readthedocs.org/en/latest/django/first-steps-with-django.html    

To run celery in virtual environment need to type the following command in your virtual environment

export C_FORCE_ROOT="true"

Running the worker with supervisor

In production you will want to run the worker in the background as a daemon and some times there may be a chance of stopping of celery worker automatically then it should be restarted automatically. To do thes tasks you need to use the tools provided like supervisord.

First, you need to install supervisor in your virtualenv and generate a configuration file.

    $ pip install supervisor

    $ cd /path/to/your/project

    $ echo_supervisord_conf > supervisord.conf

Next, just add the following section in configuration file:

    [program:celeryd]

    command=python manage.py celery worker -l info

    stdout_logfile=/path/to/your/logs/celeryd.log

    stderr_logfile=/path/to/your/logs/celeryd.log

    autostart=true

    autorestart=true

    startsecs=10

    stopwaitsecs=600

It's a simplified version of the Celery supervisor configuration file, adapted to work with virtualenvs.

Usage

Just run supervisord in your project directory.

    $ supervisord

Running supervisor during startup or booting time

create a file /etc/init.d/supervisord and configure your actual supervisord.conf in which celery is configured in DAEMON_ARGS as follows

    DAEMON_ARGS="-c /path/to/supervisord.conf"

to run it

    sudo chmod +x /etc/init.d/supervisord

and to automatically schedule it, do

    sudo update-rc.d supervisord defaults

To Stop and Start the service

    service supervisord stop

    service supervisord start

Running supervisor during startup or booting time using upstart(For Ubuntu users)

Create a new file /etc/init/supervisor.conf. Its content should look like this:

    description "supervisor"

    start on runlevel [2345]

    stop on runlevel [!2345]

    respawn

    chdir /path/to/supervisord

    exec supervisord

Note that we’re using the same supervisord configuration file we used before. No changes there…

We can now start and stop supervisord with the following commands

    $ sudo stop supervisor

    $ sudo start supervisor 
    By Posted On
SENIOR DEVELOPER at MICROPYRAMID

Need any Help in your Project?Let's Talk

Latest Comments
Related Articles
HTTP2 and SPDY Protocols - Make HTTP Faster and Safer Dinesh Deshmukh

HTTP/2, next version of HTTP/1, http/1 can not handle the present web which has become more resource intensive, it cannot processes multiple requests in an ...

Continue Reading...
Continuous Integration And Continuous Delivery With Gitlab And Docker Jagadeesh V

In this blog post, we will learn how to setup and use gitlab container registry to implement, test build and deploy your code automatically with ...

Continue Reading...
Web Hooks for Gitlab using PHP and Shell Scripts Jagadeesh V

Web-hooks play vital role if you are in Continuous Integration(CI). Higher Level organizations follow GitLab for CI purposes if they operate on open source solutions ...

Continue Reading...

Subscribe To our news letter

Subscribe to our news letter to receive latest blog posts into your inbox. Please fill your email address in the below form.
*We don't provide your email contact details to any third parties