Best Practices of Using Django Celery in Django Social Auth

Reading Time : ~ .

Introduction to 'Celery': 

In development environment sometimes we need to execute some works asynchronously(irrespective of current program flow). For example, in real time development we may have to send emails to 1000 members. Let us suppose if the time required to send single mail is 1 sec then it will take 1000 secs to send emails to 1000 persons, here the program flow will halt for 1000 seconds, which is a bad experience for the user. 'Celery' is an asynchronous task queue which is written in python provides to run the tasks asynchronously irrespective of current program flow.

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task a client puts a message on the queue, the broker then delivers the message to a worker.

https://github.com/MicroPyramid/KnolBits/tree/master/Celery (for more about celery).

Problems over 3rd party interface integration:

Generally, social networks are 3rd party interfaces which are hard to integrate with. These may lead to some problems as following.

  • Speed.

    • Much slower than local data.

    • Users may still expect near-immediate results.

  • Rate limits.

    • Different rules for every service.

    • Need to handle reactive & proactive as some don't publish rates.

  • Instability.

    • Outages (there is a chance of third party servers does go down).

    • Random failures.

To overcome the above problems we'll go with Celery.

Note: Always use RabbitMQ as your message broker and don't use RabbitMq to store the result store(can use MongoDB, Django, Redis etc.,)

Using 'Celery ' in social integration.

1. Task organization and Distribution.

  • Managing pagination.

    • Minimize the number of API calls when possible.

    • Avoid long-running tasks by setting a timeout ceiling.

    • Avoid the temptation to pass API data to dependent tasks.

  • B. Tracking task dependencies ("Done?" is difficult for distributed systems).

    • Use an external backend to store a dependency tree.

    • Subclass ResultSet to evaluate the task state of the tree.

    • Requires ignore_result=False.

2. Rate Limiting.

  • Problems.

    • Celery's rate limiting doesn't do what you think it does.

    • 3rd party rate limits depend on many factors.

  • Solution.

    • For services with known rate limits:

      • Use an external backend to store rate limit counters.

      • Increment counters based on rate limit factors per API call.

    • For services with unknown rate limits:

      • Use an external backend to store rate limit backoff counters.

      • Ramp up / ratchet down call rate by power law as API calls fail/succeed.

3. Failover problems.

  • Celery failovers:

    • Celery failovers are nothing but some internal problems, which lead to the shutdown of celery. This can be overcome by running the worker with a supervisor.

  • 3rd party failovers:

    • There are so many reasons for 3rd party server failovers. The only solution for this is nothing but rerunning of tasks.

    By Posted On
SENIOR DEVELOPER at MICROPYRAMID

Need any Help in your Project?Let's Talk

Latest Comments
Related Articles
Extract text with OCR for all image types in python using pytesseract Shirisha Gaddi

Optical Character Recognition(OCR) is the process of electronically extracting text from images or any documents like PDF and reusing it in a variety of ways ...

Continue Reading...
Django - Database access optimization Anjaneyulu Batta

Django - Database access optimization, Django Queryset is generally lazy in nature. It will not hit the database until it evaluates the query results.

Continue Reading...
Integration of 2checkout with django Nikhila Mergu

Payment Gateways which facilitate communication within banks and Security is an integral component of all payment gateways, as sensitive data such as Credit Card Numbers ...

Continue Reading...

Subscribe To our news letter

Subscribe to our news letter to receive latest blog posts into your inbox. Please fill your email address in the below form.
*We don't provide your email contact details to any third parties