Best Practices of Using Django Celery in Django Social Auth

Reading Time : ~ .

Introduction to 'Celery': 

In development environment sometimes we need to execute some works asynchronously(irrespective of current program flow). For example, in real time development we may have to send emails to 1000 members. Let us suppose if the time required to send single mail is 1 sec then it will take 1000 secs to send emails to 1000 persons, here the program flow will halt for 1000 seconds, which is a bad experience for the user. 'Celery' is an asynchronous task queue which is written in python provides to run the tasks asynchronously irrespective of current program flow.

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task a client puts a message on the queue, the broker then delivers the message to a worker.

https://github.com/MicroPyramid/KnolBits/tree/master/Celery (for more about celery).

Problems over 3rd party interface integration:

Generally, social networks are 3rd party interfaces which are hard to integrate with. These may lead to some problems as following.

  • Speed.

    • Much slower than local data.

    • Users may still expect near-immediate results.

  • Rate limits.

    • Different rules for every service.

    • Need to handle reactive & proactive as some don't publish rates.

  • Instability.

    • Outages (there is a chance of third party servers does go down).

    • Random failures.

To overcome the above problems we'll go with Celery.

Note: Always use RabbitMQ as your message broker and don't use RabbitMq to store the result store(can use MongoDB, Django, Redis etc.,)

Using 'Celery ' in social integration.

1. Task organization and Distribution.

  • Managing pagination.

    • Minimize the number of API calls when possible.

    • Avoid long-running tasks by setting a timeout ceiling.

    • Avoid the temptation to pass API data to dependent tasks.

  • B. Tracking task dependencies ("Done?" is difficult for distributed systems).

    • Use an external backend to store a dependency tree.

    • Subclass ResultSet to evaluate the task state of the tree.

    • Requires ignore_result=False.

2. Rate Limiting.

  • Problems.

    • Celery's rate limiting doesn't do what you think it does.

    • 3rd party rate limits depend on many factors.

  • Solution.

    • For services with known rate limits:

      • Use an external backend to store rate limit counters.

      • Increment counters based on rate limit factors per API call.

    • For services with unknown rate limits:

      • Use an external backend to store rate limit backoff counters.

      • Ramp up / ratchet down call rate by power law as API calls fail/succeed.

3. Failover problems.

  • Celery failovers:

    • Celery failovers are nothing but some internal problems, which lead to the shutdown of celery. This can be overcome by running the worker with a supervisor.

  • 3rd party failovers:

    • There are so many reasons for 3rd party server failovers. The only solution for this is nothing but rerunning of tasks.

    By Posted On
SENIOR DEVELOPER at MICROPYRAMID

Need any Help in your Project?Let's Talk

Latest Comments
Related Articles
django Payu Payment gateway Integration Shirisha Gaddi

In this blog, we will see how to integrate Django and PayU Payment Gateway. To integrate with PayU, we have package called "django-payu" - a ...

Continue Reading...
Basics of Django templates Siva Chittamuru

Django template engine comes as part of django framework and its very easy, extensible and handy to develop complex web applications with simplest efforts. Lets ...

Continue Reading...
Integration Of GitHub API with python django Nikhila Mergu

Using Github integration, we can get the user verified email id, general information, git hub URL, id, disk usage, public, private repo's, gists and followers, ...

Continue Reading...

Subscribe To our news letter

Subscribe to our news letter to receive latest blog posts into your inbox. Please fill your email address in the below form.
*We don't provide your email contact details to any third parties