From 72110d6c352507695f76c4100867eaebaeb677c2 Mon Sep 17 00:00:00 2001 From: Radoslav Georgiev Date: Sun, 3 Nov 2019 11:58:07 +0200 Subject: [PATCH] Add section about periodic tasks --- README.md | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) diff --git a/README.md b/README.md index ebd190b..20189ef 100644 --- a/README.md +++ b/README.md @@ -1031,6 +1031,79 @@ Meaning, you can end up with `tasks/domain_a.py` and `tasks/domain_b.py`. All yo The general rule of thumb is - split your tasks in a way that'll make sense to you. +### Periodic Tasks + +Managing periodic tasks is quite important, especially when you have tens, or hundreds of them. + +We use [Celery Beat](https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html) + `django_celery_beat.schedulers:DatabaseScheduler` + [`django-celery-beat`](https://github.com/celery/django-celery-beat) for our peridoic tasks. + +The extra thing that we do is to have a management command, called `setup_periodic_tasks`, which holds the definition of all periodic tasks within the system. This command is located in the `tasks` app, discussed above. + +Here's how `project.tasks.management.commands.setup_periodic_tasks.py` looks like: + +```python +from django.core.management.base import BaseCommand +from django.db import transaction + +from django_celery_beat.models import IntervalSchedule, CrontabSchedule, PeriodicTask + +from project.app.tasks import some_periodic_task + + +class Command(BaseCommand): + help = f""" + Setup celery beat periodic tasks. + + Following tasks will be created: + + - {some_periodic_task.name} + """ + + @transaction.atomic + def handle(self, *args, **kwargs): + print('Deleting all periodic tasks and schedules...\n') + + IntervalSchedule.objects.all().delete() + CrontabSchedule.objects.all().delete() + PeriodicTask.objects.all().delete() + + periodic_tasks_data = [ + { + 'task': some_periodic_task + 'name': 'Do some peridoic stuff', + # https://crontab.guru/#15_*_*_*_* + 'cron': { + 'minute': '15', + 'hour': '*', + 'day_of_week': '*', + 'day_of_month': '*', + 'month_of_year': '*', + }, + 'enabled': True + }, + ] + + for periodic_task in periodic_tasks_data: + print(f'Setting up {periodic_task["task"].name}') + + cron = CrontabSchedule.objects.create( + **periodic_task['cron'] + ) + + PeriodicTask.objects.create( + name=periodic_task['name'], + task=periodic_task['task'].name, + crontab=cron, + enabled=periodic_task['enabled'] + ) +``` + +Few key things: + +* We use this task as part of a deploy procedure. +* We always put a link to [`crontab.guru`](https://crontab.guru) to explain the cron. Otherwhise it's unreadable. +* Everything is in one place. + ## Inspiration The way we do Django is inspired by the following things: