Browse Source

explain when to use a task queue

remove deprecated abstract attr from celery
add explanation of example task
[ci skip]
pull/2055/head
David Lord 8 years ago
parent
commit
d5a88bf0d3
No known key found for this signature in database
GPG Key ID: 7A1C87E3F5BC42A8
  1. 78
      docs/patterns/celery.rst

78
docs/patterns/celery.rst

@ -1,24 +1,27 @@
Celery Based Background Tasks
=============================
Celery Background Tasks
=======================
Celery is a task queue for Python with batteries included. It used to
have a Flask integration but it became unnecessary after some
restructuring of the internals of Celery with Version 3. This guide fills
in the blanks in how to properly use Celery with Flask but assumes that
you generally already read the `First Steps with Celery
<http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html>`_
guide in the official Celery documentation.
If your application has a long running task, such as processing some uploaded
data or sending email, you don't want to wait for it to finish during a
request. Instead, use a task queue to send the necessary data to another
process that will run the task in the background while the request returns
immediately.
Installing Celery
-----------------
Celery is a powerful task queue that can be used for simple background tasks
as well as complex multi-stage programs and schedules. This guide will show you
how to configure Celery using Flask, but assumes you've already read the
`First Steps with Celery <http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html>`_
guide in the Celery documentation.
Celery is on the Python Package Index (PyPI), so it can be installed with
standard Python tools like :command:`pip` or :command:`easy_install`::
Install
-------
Celery is a separate Python package. Install it from PyPI using pip::
$ pip install celery
Configuring Celery
------------------
Configure
---------
The first thing you need is a Celery instance, this is called the celery
application. It serves the same purpose as the :class:`~flask.Flask`
@ -36,15 +39,18 @@ This is all that is necessary to properly integrate Celery with Flask::
from celery import Celery
def make_celery(app):
celery = Celery(app.import_name, backend=app.config['CELERY_RESULT_BACKEND'],
broker=app.config['CELERY_BROKER_URL'])
celery = Celery(
app.import_name,
backend=app.config['CELERY_RESULT_BACKEND'],
broker=app.config['CELERY_BROKER_URL']
)
celery.conf.update(app.config)
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
class ContextTask(celery.Task):
def __call__(self, *args, **kwargs):
with app.app_context():
return self.run(*args, **kwargs)
celery.Task = ContextTask
return celery
@ -53,11 +59,12 @@ from the application config, updates the rest of the Celery config from
the Flask config and then creates a subclass of the task that wraps the
task execution in an application context.
Minimal Example
An example task
---------------
With what we have above this is the minimal example of using Celery with
Flask::
Let's write a task that adds two numbers together and returns the result. We
configure Celery's broker and backend to use Redis, create a ``celery``
application using the factor from above, and then use it to define the task. ::
from flask import Flask
@ -68,26 +75,27 @@ Flask::
)
celery = make_celery(flask_app)
@celery.task()
def add_together(a, b):
return a + b
This task can now be called in the background:
This task can now be called in the background::
>>> result = add_together.delay(23, 42)
>>> result.wait()
65
result = add_together.delay(23, 42)
result.wait() # 65
Running the Celery Worker
-------------------------
Run a worker
------------
Now if you jumped in and already executed the above code you will be
disappointed to learn that your ``.wait()`` will never actually return.
That's because you also need to run celery. You can do that by running
celery as a worker::
If you jumped in and already executed the above code you will be
disappointed to learn that ``.wait()`` will never actually return.
That's because you also need to run a Celery worker to receive and execute the
task. ::
$ celery -A your_application.celery worker
The ``your_application`` string has to point to your application's package
or module that creates the `celery` object.
or module that creates the ``celery`` object.
Now that the worker is running, ``wait`` will return the result once the task
is finished.

Loading…
Cancel
Save