Contributors Guide to the Code

    • Readability is more important than Convention

    • Convention is more important than Performance

      • …unless the code is a proven hotspot.

    More important than anything else is the end-user API. Conventions must step aside, and any suffering is always alleviated if the end result is a better API.

    • Follows PEP 8.

    • Class names must be CamelCase.

    • but not if they are verbs, verbs shall be lower_case:

    • Factory functions and methods must be CamelCase (excluding verbs):

      3.     def consumer_factory(self):     # BAD
      5.     def Consumer(self):             # GOOD
      6.         ...

    Default values

    Class attributes serve as default values for the instance, as this means that they can be set by either instantiation or inheritance.

    Example:

    1. class Producer(object):
    2.     active = True
    3.     serializer = 'json'
    5.     def __init__(self, serializer=None):
    6.         self.serializer = serializer or self.serializer
    8.         # must check for None when value can be false-y
    9.         self.active = active if active is not None else self.active

    A subclass can change the default value:

    and the value can be set at instantiation:

    1. >>> producer = TaskProducer(serializer='msgpack')

    Exceptions

    Custom exceptions raised by an objects methods and properties should be available as an attribute and documented in the method/property that throw.

    This way a user doesn’t have to find out where to import the exception from, but rather use help(obj) and access the exception class from the instance directly.

    Example:

    2.     pass
    4. class Queue(object):
    5.     Empty = Empty
    7.     def get(self):
    8.         """Get the next item from the queue.
    10.         :raises Queue.Empty: if there are no more items left.
    12.         """
    13.         try:
    14.             return self.queue.popleft()
    15.         except IndexError:
    16.             raise self.Empty()

    Composites

    Similarly to exceptions, composite classes should be override-able by inheritance and/or instantiation. Common sense can be used when selecting what classes to include, but often it’s better to add one too many: predicting what users need to override is hard (this has saved us from many a monkey patch).

    Example:

    In the beginning Celery was developed for Django, simply because this enabled us get the project started quickly, while also having a large potential user base.

    Therefore the app concept was introduced. When using apps you use ‘celery’ objects instead of importing things from celery submodules, this sadly also means that Celery essentially has two API’s.

    Here’s an example using Celery in single-mode:

    1. from celery import task
    3. from .models import CeleryStats
    5. @task
    6. def write_stats_to_db():
    7.     stats = inspect().stats(timeout=1)
    8.     for node_name, reply in stats:
    9.         CeleryStats.objects.update_stat(node_name, stats)

    and here’s the same using Celery app objects:

    1. from .celery import celery
    2. from .models import CeleryStats
    4. @app.task
    5. def write_stats_to_db():
    6.     stats = celery.control.inspect().stats(timeout=1)
    7.     for node_name, reply in stats:
    8.         CeleryStats.objects.update_stat(node_name, stats)

    In the example above the actual application instance is imported from a module in the project, this module could look something like this:

    • celery.app

      This is the core of Celery: the entry-point for all functionality.

    • celery.loaders

      Every app must have a loader. The loader decides how configuration is read, what happens when the worker starts, when a task starts and ends, and so on.

      The loaders included are:

      • app

        Custom celery app instances uses this loader by default.

      • default

      Extension loaders also exist, like django-celery, celery-pylons and so on.

    • celery.worker

      This is the worker implementation.

    • celery.backends

      Task result backends live here.

    • celery.apps

    • celery.bin

      Command-line applications. setup.py creates setuptools entrypoints for these.

    • celery.concurrency

      Execution pool implementations (prefork, eventlet, gevent, threads).

    • celery.db

    • celery.events

      Sending and consuming monitoring events, also includes curses monitor, event dumper and utilities to work with in-memory cluster state.

    • celery.execute.trace

      How tasks are executed and traced by the worker, and in eager mode.

    • celery.security

      Security related functionality, currently a serializer using cryptographic digests.

    • celery.task

      single-mode interface to creating tasks, and controlling workers.

    • celery.tests

      The unittest suite.

    • celery.utils