Add section about periodic tasks

This commit is contained in:
Radoslav Georgiev 2019-11-03 11:58:07 +02:00
parent b9c741b15a
commit 72110d6c35

View File

@ -1031,6 +1031,79 @@ Meaning, you can end up with `tasks/domain_a.py` and `tasks/domain_b.py`. All yo
The general rule of thumb is - split your tasks in a way that'll make sense to you. The general rule of thumb is - split your tasks in a way that'll make sense to you.
### Periodic Tasks
Managing periodic tasks is quite important, especially when you have tens, or hundreds of them.
We use [Celery Beat](https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html) + `django_celery_beat.schedulers:DatabaseScheduler` + [`django-celery-beat`](https://github.com/celery/django-celery-beat) for our peridoic tasks.
The extra thing that we do is to have a management command, called `setup_periodic_tasks`, which holds the definition of all periodic tasks within the system. This command is located in the `tasks` app, discussed above.
Here's how `project.tasks.management.commands.setup_periodic_tasks.py` looks like:
```python
from django.core.management.base import BaseCommand
from django.db import transaction
from django_celery_beat.models import IntervalSchedule, CrontabSchedule, PeriodicTask
from project.app.tasks import some_periodic_task
class Command(BaseCommand):
help = f"""
Setup celery beat periodic tasks.
Following tasks will be created:
- {some_periodic_task.name}
"""
@transaction.atomic
def handle(self, *args, **kwargs):
print('Deleting all periodic tasks and schedules...\n')
IntervalSchedule.objects.all().delete()
CrontabSchedule.objects.all().delete()
PeriodicTask.objects.all().delete()
periodic_tasks_data = [
{
'task': some_periodic_task
'name': 'Do some peridoic stuff',
# https://crontab.guru/#15_*_*_*_*
'cron': {
'minute': '15',
'hour': '*',
'day_of_week': '*',
'day_of_month': '*',
'month_of_year': '*',
},
'enabled': True
},
]
for periodic_task in periodic_tasks_data:
print(f'Setting up {periodic_task["task"].name}')
cron = CrontabSchedule.objects.create(
**periodic_task['cron']
)
PeriodicTask.objects.create(
name=periodic_task['name'],
task=periodic_task['task'].name,
crontab=cron,
enabled=periodic_task['enabled']
)
```
Few key things:
* We use this task as part of a deploy procedure.
* We always put a link to [`crontab.guru`](https://crontab.guru) to explain the cron. Otherwhise it's unreadable.
* Everything is in one place.
## Inspiration ## Inspiration
The way we do Django is inspired by the following things: The way we do Django is inspired by the following things: