django-clickhouse/docs/synchronization.md

106 lines
5.0 KiB
Markdown
Raw Normal View History

2019-12-01 11:33:43 +03:00
# Synchronization
2020-02-06 11:39:56 +03:00
2020-02-07 11:05:19 +03:00
## Design motivation
Read [here](motivation.md#sync-over-intermediate-storage).
## Algorithm
<!--- ![General scheme](https://octodex.github.com/images/yaktocat.png) --->
1. [Celery beat](https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html) schedules `django_clickhouse.tasks.clickhouse_auto_sync` task every second or near.
2. [Celery workers](https://docs.celeryproject.org/en/latest/userguide/workers.html) execute `clickhouse_auto_sync`.
It searches for `ClickHouseModel` subclasses which need sync (if `Model.need_sync()` method returns `True`).
2. `django_clickhouse.tasks.sync_clickhouse_model` task is scheduled for each `ClickHouseModel` which needs sync.
3. `sync_clickhouse_model` saves sync start time in [storage](storages.md) and calls `ClickHouseModel.sync_batch_from_storage()` method.
4. `ClickHouseModel.sync_batch_from_storage()`:
* Gets [storage](storages.md) model works with using `ClickHouseModel.get_storage()` method
* Calls `Storage.pre_sync(import_key)` for model [storage](storages.md).
This may be used to prevent parallel execution with locks or some other operations.
* Gets a list of operations to sync from [storage](storages.md).
* Fetches objects from relational database calling `ClickHouseModel.get_sync_objects(operations)` method.
* Forms a batch of tuples to insert into ClickHouse using `ClickHouseModel.get_insert_batch(import_objects)` method.
* Inserts batch of tuples into ClickHouse using `ClickHouseModel.insert_batch(batch)` method.
* Calls `Storage.post_sync(import_key)` method to clean up storage after syncing batch.
This method also removes synced operations from storage.
* If some exception occurred during execution, `Storage.post_sybc_failed(import_key)` method is called.
Note, that process can be killed without exception, for instance by OOM killer.
And this method will not be called.
## Configuration
Sync configuration can be set globally using django settings.py parameters or redeclared for each `ClickHouseModel` class.
`ClickHouseModel` configuration is prior to settings configuration.
### Settings configuration
* [CLICKHOUSE_CELERY_QUEUE](configuration.md#clickhouse_celery_queue)
Defaults to: `'celery'`
A name of a queue, used by celery to plan library sync tasks.
* [CLICKHOUSE_SYNC_STORAGE](configuration.md#clickhouse_sync_storage)
Defaults to: `'django_clickhouse.storages.RedisStorage'`
An [intermediate storage](storages.md) class to use. Can be a string or class.
* [CLICKHOUSE_SYNC_BATCH_SIZE](configuration.md#clickhouse_sync_storage)
Defaults to: `10000`
Maximum number of operations, fetched by sync process from [intermediate storage](storages.md) per sync round.
* [CLICKHOUSE_SYNC_DELAY](configuration.md#clickhouse_sync_storage)
Defaults to: `5`
A delay in seconds between two sync rounds start.
### ClickHouseModel configuration
Each `ClickHouseModel` subclass can define sync arguments and methods:
* `django_model: django.db.models.Model`
Required.
Django model this ClickHouseModel class is synchronized with.
* `django_model_serializer: django.db.models.Model`
Defaults to: `django_clickhouse.serializers.Django2ClickHouseModelSerializer`
[Serializer class](models.md#serializers) to convert DjangoModel to ClickHouseModel.
* `sync_enabled: bool`
Defaults to: `False`.
Is sync for this model enabled?
* `sync_batch_size: int`
Defaults to: [CLICKHOUSE_SYNC_BATCH_SIZE](configuration.md#clickhouse_sync_storage)
Maximum number of operations, fetched by sync process from [storage](storages.md) per sync round.
* `sync_delay: float`
Defaults to: [CLICKHOUSE_SYNC_DELAY](configuration.md#clickhouse_sync_storage)
A delay in seconds between two sync rounds start.
* `sync_storage: Union[str, Storage]`
Defaults to: [CLICKHOUSE_SYNC_STORAGE](configuration.md#clickhouse_sync_storage)
An [intermediate storage](storages.md) class to use. Can be a string or class.
Example:
```python
from django_clickhouse.clickhouse_models import ClickHouseModel
from django_clickhouse.engines import ReplacingMergeTree
from infi.clickhouse_orm import fields
from my_app.models import User
class ClickHouseUser(ClickHouseModel):
django_model = User
sync_enabled = True
sync_delay = 5
sync_batch_size = 1000
id = fields.UInt32Field()
first_name = fields.StringField()
birthday = fields.DateField()
visits = fields.UInt32Field(default=0)
engine = ReplacingMergeTree('birthday', ('birthday',))
```
## Fail resistance
Fail resistance is based on several points:
1. [Storage](storages.md) should not loose data in any case. It's not this library goal to keep it stable.
2. Data is removed from [storage](storages.md) only if import succeeds. Otherwise import attempt is repeated.
3. It's recommended to use ReplacingMergeTree or CollapsingMergeTree [engines](models.md#engines)
instead of simple MergeTree, so it removes duplicates if batch is imported twice.
4. Each `ClickHouseModel` is synced in separate process.
If one model fails, it should not affect other models.