django-clickhouse/docs/migrations.md

Migrations

Migration system allows to make migrate ClickHouse table schema based on ClickHouseModel. Library migrations are based on infi.clickhouse_orm migration system, but makes it a little bit more django-like.

File structure

Each django app can have optional clickhouse_migrations package. This is a default package name, it can be changed with CLICKHOUSE_MIGRATIONS_PACKAGE setting.

Package contains py files, starting with 4-digit number. A number gives an order in which migrations will be applied.

Example:

my_app
>> clickhouse_migrations
>>>> __init__.py
>>>> 0001_initial.py
>>>> 0002_add_new_field_to_my_model.py
>> clickhouse_models.py
>> urls.py
>> views.py

Migration files

Each file must contain a Migration class, inherited from django_clickhouse.migrations.Migration. The class should define an operations attribute - a list of operations to apply one by one. Operation is one of operations, supported by infi.clickhouse-orm.

from django_clickhouse import migrations
from my_app.clickhouse_models import ClickHouseUser

class Migration(migrations.Migration):
    operations = [
        migrations.CreateTable(ClickHouseUser)
    ]

MigrationHistory ClickHouseModel

This model stores information about applied migrations.
By default, library uses django_clickhouse.migrations.MigrationHistory model, but this can be changed using CLICKHOUSE_MIGRATION_HISTORY_MODEL setting. For instance, if you want to make it replicated, you have to redeclare tables engine.

MigrationHistory model is stored in default database.

Automatic migrations

When library is installed, it tries applying migrations every time, you call django migrate. If you want to disable this, use CLICKHOUSE_MIGRATE_WITH_DEFAULT_DB setting.

By default migrations are applied to all CLICKHOUSE_DATABASES, which have no flags:

• 'migrate': False
• 'readonly': True

Note: migrations are only applied, with django default database.
So if you call python manage.py migrate --database=secondary they wouldn't be applied.

Admin migration command

In order to make migrations separately from django's manage.py migrate command, this library implements custom manage.py command clickhouse_migrate.

Usage:

python manage.py clickhouse_migrate [--help] [--database <db_alias>] [--verbosity {0,1,2,3}] [app_label] [migration_number]

Parameters

• app_label: Optional[str] - If set, migrates only given django application
• migration_number: Optional[int] - If set, migrate django app with app_label to migration with this number
  Important note: Library currently does not support unapplying migrations. If already applied migration is given - it will do noting.
• --database: Optional[str] - If set, migrates only this database alias from CLICKHOUSE_DATABASES config parameter
• --verbosity: Optional[int] = 1 - Level of debug output. See here for more details.
• --help - Print help

Migration algorithm

• Get a list of databases from CLICKHOUSE_DATABASES setting. Migrate them one by one.
  • Find all django apps from INSTALLED_APPS setting, which have no readonly=True attribute and have migrate=True attribute. Migrate them one by one.
    • Iterate over INSTAALLED_APPS, searching for clickhouse_migrations package
    • If package was not found, skip app.
    • Get a list of migrations applied from MigrationHistory model
    • Get a list of unapplied migrations
    • Get Migration class from each migration and call it apply() method
    • apply() iterates operations, checking if it should be applied with router
    • If migration should be applied, it is applied
    • Mark migration as applied in MigrationHistory model

Security notes

1. ClickHouse has no transaction system, as django relational databases. As a result, if migration fails, it would be partially applied and there's no correct way to rollback. I recommend to make migrations as small as possible, so it should be easier to determine and correct the result if something goes wrong.
2. Unlike django, this library is enable to unapply migrations. This functionality may be implemented in the future.