Best Practices of Using Django Celery in Django Social Auth

Introduction to 'Celery': 

In development environment sometimes we need to execute some works asynchronously(irrespective of current program flow). For example in real time development we may have to send mails for 1000 members. Let us suppose if the time required to send single mail is 1 sec then it will take 1000 secs to send mails to 1000 persons, here the program flow will halt for 1000 seconds, which is a bad experience for the user. 'Celery' is an asynchronous task queue which is written in python provides to run the tasks asynchronously irrespective of current program flow.

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task a client puts a message on the queue, the broker then delivers the message to a worker.

https://github.com/MicroPyramid/KnolBits/tree/master/Celery (for more about celery).

Problems over 3rd party interface integration:

Generally social networks are 3rd party interfaces which are hard to integrate with. These may lead to some problems as following.

  • Speed.

    • Much slower than local data.

    • Users may still expect near-immediate results.

  • Rate limits.

    • Different rules for every service.

    • Need to handle reactive & proactive as some don't publish rates.

  • Instability.

    • Outages (there is a chance of third party servers does go down).

    • Random failures.

To overcome the above problems we'll go with Celery.

Note: Always use RabbitMQ as your message broker and don't use RabbitMq to store the result store(can use mongodb, django, redis etc.,)

Using 'Celery ' in social integration.

1. Task organisation and Distribution.

  • Managing pagination.

    • Minimize the number of api calls when possible.

    • Avoid long-running tasks by setting a timeout ceiling.

    • Avoid the temptation to pass API data to dependent tasks.

  • B. Tracking task dependencies ("Done?" is difficult for distributed systems).

    • Use an external backend to store a dependency tree.

    • Subclass ResultSet to evaluate the task state of the tree.

    • Requires ignore_result=False.

2. Rate Limiting.

  • Problems.

    • Celery's rate limiting doesn't do what you think it does.

    • 3rd party rate limits depend on many factors.

  • Solution.

    • For services with known rate limits:

      • Use an external backend to store rate limit counters.

      • Increment counters based on rate limit factors per api call.

    • For services with unknown rate limits:

      • Use an external backend to store rate limit backoff counters.

      • Ramp up / ratchet down call rate by power law as api calls fail/succeed.

3. Failover problems.

  • Celery failovers:

    • Celery failovers are nothing but some internal problems, which lead to shutdown of celery. This can be overcome by running the worker with supervisor.

  • 3rd party failovers:

    • There are so many reasons for 3rd party server failovers. The only solution for this is nothing but rerunning of tasks.

    By Posted On
SENIOR DEVELOPER at MICROPYRAMID

Latest Comments
Related Articles
How to customize the admin actions in list pages of Django admin? Chaitanya Kattineni

Django by default provides automatic admin interface, that reads metadata from your models to provide a beautiful model interface where trusted users can manage content ...

Continue Reading...
Hosting Django Application with Nginx and UWSGI Ashwin Kumar

Django is a python based web- application development framework. Setting up a sample app and running it as easy as pie. Nginx is a webserver ...

Continue Reading...
How to Create your own e-commerce shop using Django-Oscar. Shirisha Gaddi

Oscar is an open-source ecommerce framework for Django. Django Oscar provides a base platform to build an online shop. Oscar is built as a highly ...

Continue Reading...

Subscribe To our news letter

Subscribe to our news letter to receive latest blog posts into your inbox. Please fill your email address in the below form.
*We don't provide your email contact details to any third parties