|
|
@ -1,24 +1,27 @@ |
|
|
|
Celery Based Background Tasks |
|
|
|
Celery Background Tasks |
|
|
|
============================= |
|
|
|
======================= |
|
|
|
|
|
|
|
|
|
|
|
Celery is a task queue for Python with batteries included. It used to |
|
|
|
If your application has a long running task, such as processing some uploaded |
|
|
|
have a Flask integration but it became unnecessary after some |
|
|
|
data or sending email, you don't want to wait for it to finish during a |
|
|
|
restructuring of the internals of Celery with Version 3. This guide fills |
|
|
|
request. Instead, use a task queue to send the necessary data to another |
|
|
|
in the blanks in how to properly use Celery with Flask but assumes that |
|
|
|
process that will run the task in the background while the request returns |
|
|
|
you generally already read the `First Steps with Celery |
|
|
|
immediately. |
|
|
|
<http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html>`_ |
|
|
|
|
|
|
|
guide in the official Celery documentation. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Installing Celery |
|
|
|
Celery is a powerful task queue that can be used for simple background tasks |
|
|
|
----------------- |
|
|
|
as well as complex multi-stage programs and schedules. This guide will show you |
|
|
|
|
|
|
|
how to configure Celery using Flask, but assumes you've already read the |
|
|
|
|
|
|
|
`First Steps with Celery <http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html>`_ |
|
|
|
|
|
|
|
guide in the Celery documentation. |
|
|
|
|
|
|
|
|
|
|
|
Celery is on the Python Package Index (PyPI), so it can be installed with |
|
|
|
Install |
|
|
|
standard Python tools like :command:`pip` or :command:`easy_install`:: |
|
|
|
------- |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Celery is a separate Python package. Install it from PyPI using pip:: |
|
|
|
|
|
|
|
|
|
|
|
$ pip install celery |
|
|
|
$ pip install celery |
|
|
|
|
|
|
|
|
|
|
|
Configuring Celery |
|
|
|
Configure |
|
|
|
------------------ |
|
|
|
--------- |
|
|
|
|
|
|
|
|
|
|
|
The first thing you need is a Celery instance, this is called the celery |
|
|
|
The first thing you need is a Celery instance, this is called the celery |
|
|
|
application. It serves the same purpose as the :class:`~flask.Flask` |
|
|
|
application. It serves the same purpose as the :class:`~flask.Flask` |
|
|
@ -36,15 +39,18 @@ This is all that is necessary to properly integrate Celery with Flask:: |
|
|
|
from celery import Celery |
|
|
|
from celery import Celery |
|
|
|
|
|
|
|
|
|
|
|
def make_celery(app): |
|
|
|
def make_celery(app): |
|
|
|
celery = Celery(app.import_name, backend=app.config['CELERY_RESULT_BACKEND'], |
|
|
|
celery = Celery( |
|
|
|
broker=app.config['CELERY_BROKER_URL']) |
|
|
|
app.import_name, |
|
|
|
|
|
|
|
backend=app.config['CELERY_RESULT_BACKEND'], |
|
|
|
|
|
|
|
broker=app.config['CELERY_BROKER_URL'] |
|
|
|
|
|
|
|
) |
|
|
|
celery.conf.update(app.config) |
|
|
|
celery.conf.update(app.config) |
|
|
|
TaskBase = celery.Task |
|
|
|
|
|
|
|
class ContextTask(TaskBase): |
|
|
|
class ContextTask(celery.Task): |
|
|
|
abstract = True |
|
|
|
|
|
|
|
def __call__(self, *args, **kwargs): |
|
|
|
def __call__(self, *args, **kwargs): |
|
|
|
with app.app_context(): |
|
|
|
with app.app_context(): |
|
|
|
return self.run(*args, **kwargs) |
|
|
|
return self.run(*args, **kwargs) |
|
|
|
|
|
|
|
|
|
|
|
celery.Task = ContextTask |
|
|
|
celery.Task = ContextTask |
|
|
|
return celery |
|
|
|
return celery |
|
|
|
|
|
|
|
|
|
|
@ -53,11 +59,12 @@ from the application config, updates the rest of the Celery config from |
|
|
|
the Flask config and then creates a subclass of the task that wraps the |
|
|
|
the Flask config and then creates a subclass of the task that wraps the |
|
|
|
task execution in an application context. |
|
|
|
task execution in an application context. |
|
|
|
|
|
|
|
|
|
|
|
Minimal Example |
|
|
|
An example task |
|
|
|
--------------- |
|
|
|
--------------- |
|
|
|
|
|
|
|
|
|
|
|
With what we have above this is the minimal example of using Celery with |
|
|
|
Let's write a task that adds two numbers together and returns the result. We |
|
|
|
Flask:: |
|
|
|
configure Celery's broker and backend to use Redis, create a ``celery`` |
|
|
|
|
|
|
|
application using the factor from above, and then use it to define the task. :: |
|
|
|
|
|
|
|
|
|
|
|
from flask import Flask |
|
|
|
from flask import Flask |
|
|
|
|
|
|
|
|
|
|
@ -68,26 +75,27 @@ Flask:: |
|
|
|
) |
|
|
|
) |
|
|
|
celery = make_celery(flask_app) |
|
|
|
celery = make_celery(flask_app) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@celery.task() |
|
|
|
@celery.task() |
|
|
|
def add_together(a, b): |
|
|
|
def add_together(a, b): |
|
|
|
return a + b |
|
|
|
return a + b |
|
|
|
|
|
|
|
|
|
|
|
This task can now be called in the background: |
|
|
|
This task can now be called in the background:: |
|
|
|
|
|
|
|
|
|
|
|
>>> result = add_together.delay(23, 42) |
|
|
|
result = add_together.delay(23, 42) |
|
|
|
>>> result.wait() |
|
|
|
result.wait() # 65 |
|
|
|
65 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Running the Celery Worker |
|
|
|
Run a worker |
|
|
|
------------------------- |
|
|
|
------------ |
|
|
|
|
|
|
|
|
|
|
|
Now if you jumped in and already executed the above code you will be |
|
|
|
If you jumped in and already executed the above code you will be |
|
|
|
disappointed to learn that your ``.wait()`` will never actually return. |
|
|
|
disappointed to learn that ``.wait()`` will never actually return. |
|
|
|
That's because you also need to run celery. You can do that by running |
|
|
|
That's because you also need to run a Celery worker to receive and execute the |
|
|
|
celery as a worker:: |
|
|
|
task. :: |
|
|
|
|
|
|
|
|
|
|
|
$ celery -A your_application.celery worker |
|
|
|
$ celery -A your_application.celery worker |
|
|
|
|
|
|
|
|
|
|
|
The ``your_application`` string has to point to your application's package |
|
|
|
The ``your_application`` string has to point to your application's package |
|
|
|
or module that creates the `celery` object. |
|
|
|
or module that creates the ``celery`` object. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Now that the worker is running, ``wait`` will return the result once the task |
|
|
|
|
|
|
|
is finished. |
|
|
|