Introduction to 'Celery': 

In development environment sometimes we need to execute some works asynchronously(irrespective of current program flow). For example in real time development we may have to send mails for 1000 members. Let us suppose if the time required to send single mail is 1 sec then it will take 1000 secs to send mails to 1000 persons, here the program flow will halt for 1000 seconds, which is a bad experience for the user. 'Celery' is an asynchronous task queue which is written in python provides to run the tasks asynchronously irrespective of current program flow.

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task a client puts a message on the queue, the broker then delivers the message to a worker.

https://github.com/MicroPyramid/KnolBits/tree/master/Celery (for more about celery).

Problems over 3rd party interface integration:

Generally social networks are 3rd party interfaces which are hard to integrate with. These may lead to some problems as following.

  • Speed.

    • Much slower than local data.

    • Users may still expect near-immediate results.

  • Rate limits.

    • Different rules for every service.

    • Need to handle reactive & proactive as some don't publish rates.

  • Instability.

    • Outages (there is a chance of third party servers does go down).

    • Random failures.

To overcome the above problems we'll go with Celery.

Note: Always use RabbitMQ as your message broker and don't use RabbitMq to store the result store(can use mongodb, django, redis etc.,)

Using 'Celery ' in social integration.

1. Task organisation and Distribution.

  • Managing pagination.

    • Minimize the number of api calls when possible.

    • Avoid long-running tasks by setting a timeout ceiling.

    • Avoid the temptation to pass API data to dependent tasks.

  • B. Tracking task dependencies ("Done?" is difficult for distributed systems).

    • Use an external backend to store a dependency tree.

    • Subclass ResultSet to evaluate the task state of the tree.

    • Requires ignore_result=False.

2. Rate Limiting.

  • Problems.

    • Celery's rate limiting doesn't do what you think it does.

    • 3rd party rate limits depend on many factors.

  • Solution.

    • For services with known rate limits:

      • Use an external backend to store rate limit counters.

      • Increment counters based on rate limit factors per api call.

    • For services with unknown rate limits:

      • Use an external backend to store rate limit backoff counters.

      • Ramp up / ratchet down call rate by power law as api calls fail/succeed.

3. Failover problems.

  • Celery failovers:

    • Celery failovers are nothing but some internal problems, which lead to shutdown of celery. This can be overcome by running the worker with supervisor.

  • 3rd party failovers:

    • There are so many reasons for 3rd party server failovers. The only solution for this is nothing but rerunning of tasks.

Subscribe To our news letter

Subscribe to our news letter to receive latest blog posts into your inbox. Please fill your email address in the below form.
*We don't provide your email contact details to any third parties
Latest Comments
Related Articles
Sending emails using sendgrid on heroku for a Django App Vinisha Naladala

Integrate Sendgrid API to your Heroku app to deliver simplified emails like any notification emails, user signups etc.

Continue Reading...
Improving page speed score in Google Page Score test- PART1 Chaitanya Kattineni

Google's Page Speed Insights measures the performance of a page for mobile devices and desktop devices. It fetches the url twice, once with a mobile ...

Continue Reading...
Django Model managers and properties Sandeep Jagata

Django model is the single, definitive source of data about your data. It contains the essential fields and behaviors of the data you’re storing. Generally, ...

Continue Reading...