carrotquest/django-clickhouse
1
1
Fork 0
mirror of https://github.com/carrotquest/django-clickhouse.git synced 2025-10-24 20:51:07 +03:00
Code Issues Packages Projects Releases Wiki Activity
211 Commits 3 Branches 11 Tags 492 KiB
d8455fc389
Commit Graph

155 Commits

Author SHA1 Message Date
M1ha
13a226501f Fix need_sync period 2019-01-24 11:14:19 +05:00
M1ha
2d2ec9c055 Added debug logs for need_sync 2019-01-24 11:02:44 +05:00
M1ha
c2a1ecc688 Trying to understand invalid sync period 2019-01-24 10:53:54 +05:00
M1ha
da21e9939a Moved sync start mark to the beggining of the task 2019-01-24 10:40:09 +05:00
M1ha
75fcdeeec0 Made redis storage a singleton in order to reduce number of connections 2019-01-23 17:01:10 +05:00
M1ha
ed3b3909ef Fix 2019-01-23 16:30:04 +05:00
M1ha
fae4247e79 Added lock timeouts monitoring 2019-01-23 16:28:27 +05:00
M1ha
42f823d3a5 Fixed a bug in many_init 2019-01-23 15:51:35 +05:00
M1ha
6db5f2b5bb Speed up init of multiple clickhouse models in a row 2019-01-23 15:43:46 +05:00
M1ha
e8549b9509 Removed not needed lock to speed up 2019-01-23 13:47:34 +05:00
M1ha
1a4ed0ddf6 Trying to speed up getting objects from multiple databases 2019-01-23 13:32:18 +05:00
M1ha
241618d8ca Test if post_save is called 2019-01-23 12:48:18 +05:00
M1ha
cac5fe9d52 Removed monitoring, as it has strange avg results for all queries 
and models
 2019-01-23 10:40:37 +05:00
M1ha
c934ff4df5 Bugfix 2019-01-23 10:34:27 +05:00
M1ha
882930a3a8 Added query time statsd timer 2019-01-23 10:32:11 +05:00
M1ha
780b13a42e Optimization 2019-01-22 11:58:41 +05:00
M1ha
056aa0c5f1 Removed infi.clickhouse_orm insert batching 2019-01-22 11:40:24 +05:00
M1ha
de25af223a Added statsd counters for ClickHouse insert batch size 2019-01-22 09:57:59 +05:00
M1ha
6517c087a0 Added statsd counters for ClickHouse insert batch size 2019-01-22 09:52:05 +05:00
M1ha
69c2c5504b MultiModelConverter now processes sub_models in parallel threads 2019-01-21 14:32:12 +05:00
M1ha
480a857fe8 Fix error on migration, 
if MigrationHistory database is readonly and has not been created yet
 2019-01-21 12:35:44 +05:00
M1ha
59f92d5b6e Fix in counter 2019-01-18 18:26:28 +05:00
M1ha
d8dccf209d Fix in counter 2019-01-18 18:01:18 +05:00
M1ha
9bc6c6154b Changed post operation guage, as sometimes statsd can "skip" metrics 
and graph holds invalid counters
 2019-01-18 17:59:46 +05:00
M1ha
6a4bf14ea6 1) Fixed get_final_versions statsd timer key 
2) Added individual model operations statsd monitoring for ClickHouseMultiModel
 2019-01-18 10:47:26 +05:00
M1ha
96b625ecff Speed up of get_sync_objects (parallel execution on multiple databases) 2019-01-18 10:36:48 +05:00
M1ha
073e002125 Багфикс в версиях вставки данных в кликхаус 2019-01-16 17:17:48 +05:00
M1ha
43815723df Added tests for get_final_versions on date and datetime 2019-01-14 12:45:46 +05:00
M1ha
ec72e31986 Tests written for get_final_version 2019-01-14 11:32:21 +05:00
M1ha
ba3f2f6c1b Fixed getting final versions if DateTimeField is used 2019-01-14 10:56:54 +05:00
M1ha
1f3dcedf55 Added ability to get final rows version with the help of version_col 2019-01-11 13:00:42 +05:00
M1ha
ec2f71fc78 Refactoring 2019-01-10 17:29:41 +05:00
M1ha
33dea53aa2 Bugfix in getting empty sync models 2019-01-09 15:36:56 +05:00
M1ha
682c004ba4 Bugfix in getting sync models 2019-01-09 15:14:53 +05:00
M1ha
dbe54da5e3 Refactoring 2018-12-25 12:53:23 +05:00
M1ha
e8a15e4330 Fixed statsd counters 2018-12-19 16:15:17 +05:00
M1ha
6a1ffbadd3 Divided separate get_sync_query_set method for easy reusage 2018-12-19 15:11:17 +05:00
M1ha
76b01c13de Fix multiple model registration 2018-12-19 13:15:42 +05:00
M1ha
1a3d512c57 Fix statsd counters 2018-12-19 13:02:14 +05:00
M1ha
13b67ed77a Fix time between batches count 2018-12-19 12:58:34 +05:00
M1ha
36123db73e Fix time between batches count 2018-12-19 12:52:47 +05:00
M1ha
2dc8e7a270 Fix choosing connection on syncing 2018-12-19 12:37:05 +05:00
M1ha
0829e5bacd Added some statsd counters for sync process 2018-12-19 12:32:27 +05:00
M1ha
6c9e217161 Added some statsd counters for sync process 2018-12-19 12:21:28 +05:00
M1ha
38db27626c Fixes in statsd keys 2018-12-19 12:11:55 +05:00
M1ha
578fc6c939 Bug fix with rank 2018-12-19 11:59:55 +05:00
M1ha
95bf0ab520 Bug fixes 2018-12-19 11:57:05 +05:00
M1ha
d0cfd234de Fix import errors 2018-12-19 11:46:44 +05:00
M1ha
59f0efae15 Fix module import on auto sync 2018-12-19 11:34:00 +05:00
M1ha
b94195c1f1 Fix creating MigrationHistory table 2018-12-18 18:41:00 +05:00
M1ha
d830b74bcb Created separate from infi MigrationHistory in order to: 
1) inherit it from this library models
2) Use some extra functionality
 2018-12-18 18:28:23 +05:00
M1ha
0268cc230a Fixed error in migrate. Wrote tests 2018-12-18 17:34:30 +05:00
M1ha
66ec7dd5db Fixed error in migrate. Wrote tests 2018-12-18 17:33:34 +05:00
M1ha
2d4c8554cd Changed migrations routing, dividing it into replicated and not replicated operations 2018-12-18 15:45:10 +05:00
M1ha
2126bc25da Fix for readonly connections in migrations 2018-12-18 15:01:02 +05:00
M1ha
a9ff8fb8b6 Fix for readonly connections in migrations 2018-12-18 14:59:56 +05:00
M1ha
7415d16b4f Added logging of migration process 2018-12-18 14:51:12 +05:00
M1ha
2d84e5da04 Added logging of migration process 2018-12-18 14:47:56 +05:00
M1ha
b7972fb079 Bugfix 2018-12-13 11:59:34 +05:00
M1ha
2cc742214a Bugfix 2018-12-13 11:55:26 +05:00
M1ha
9ab38c464d Bugfix 2018-12-13 11:53:36 +05:00
M1ha
71e839167a Refactoring 2018-12-13 10:52:38 +05:00
M1ha
7e502ee8a8 1) Fixed SyncKillTest 
2) Added some logs
 2018-12-03 16:41:34 +05:00
M1ha
762c1aa10d 1) Fixed SyncKillTest 
2) Added some logs
 2018-11-29 19:10:41 +05:00
M1ha
9742abbaec Added test for killing sync process safety. 
Test is not passing =)
 2018-11-29 15:19:25 +05:00
M1ha
166d23ca7c Refactoring 2018-11-28 15:33:01 +05:00
M1ha
abe0ae45c5 Added QuerySet that determines db with router help. 
Added using(), create(), bulk_create(), all() methods to QuerySet
 2018-11-28 15:30:19 +05:00
M1ha
ce1a206b04 Moved model_to_dict to utils 2018-11-27 13:09:20 +05:00
M1ha
d300d6e46e Фиксы по тестам 2018-11-26 19:17:58 +05:00
M1ha
745b7b7788 Fixed some bugs 2018-11-26 17:37:31 +05:00
M1ha
0e4dd6b69f Fixed some bugs 2018-11-26 17:28:03 +05:00
M1ha
e81a6b5e4a Bugfix in ClickHouseMultiModel sync 2018-11-26 15:38:10 +05:00
M1ha
ecff04aaf2 Removed database connections from router 2018-11-26 15:13:18 +05:00
M1ha
382655224b Fix 2018-11-26 12:40:32 +05:00
M1ha
e8161ee7ce Fixes in migration system to pass database directly 2018-11-26 11:56:50 +05:00
M1ha
e1d7db1525 Bugfix 2018-11-23 13:16:35 +05:00
M1ha
5add74cc39 Bugfix 2018-11-23 12:05:16 +05:00
M1ha
a5bbc9ba00 Made get_database for ClickHouseModel a class method 2018-11-23 11:58:44 +05:00
M1ha
68914a20f0 Infi https://github.com/Infinidat/infi.clickhouse_orm/issues/89 fix 
some refactoring
 2018-11-23 11:04:31 +05:00
M1ha
cea0f3cab5 import all infi modules in migrations 2018-11-22 17:01:57 +05:00
M1ha
5455d6d9b4 Fixed bug in migrations, if migrations directory doesn't exist in module 2018-11-22 16:59:39 +05:00
M1ha
ea5d9a7dfd Added ClickHouseMultiModel 2018-11-22 16:22:34 +05:00
M1ha
6261d69821 Added extra method in model to get insert batch 2018-11-22 12:54:51 +05:00
M1ha
e4b2958895 Changed serializer init to concrete model fields 2018-11-22 12:33:51 +05:00
M1ha
da755da09d Added wrappers for Infi querysets 2018-11-22 11:18:53 +05:00
M1ha
9b02a49bc9 Added all MergeTree engines supported in infi.clickhouse_orm 2018-11-22 11:09:38 +05:00
M1ha
084c218d2a Added tests on 2018-11-21 17:10:35 +05:00
M1ha
0dbfa8fff1 Added tests on 2018-11-21 16:51:19 +05:00
M1ha
2dab8fcccb Added tests on 2018-11-21 16:50:54 +05:00
M1ha
a37cd25453 Removed deprecated comment 2018-11-20 17:25:03 +05:00
M1ha
6e0bc424d7 1) Finished sync celery tasks. 
2) RedisStorage now prevents parallel sync with RedisLock
 2018-11-20 17:24:15 +05:00
M1ha
6e4a5d7723 1) Исправил счетчик автоответов по баге https://carrot.atlassian.net/browse/RED-5318 
2) Фиксы по тестам
 2018-11-19 16:58:53 +05:00
M1ha
61066e0ec7 statsd metrics set up 2018-11-16 17:41:54 +05:00
M1ha
e0fe50d534 Расставил метрики statsd 2018-11-16 17:36:04 +05:00
M1ha
d4e9a705d1 1) Refactoring 
2) Bug fixes
3) Simple tests on syncing MergeTree and CollapsingMergeTree
 2018-11-16 15:16:36 +05:00
M1ha
4480c1ae1d Added migration system. 2018-11-16 13:14:40 +05:00
M1ha
cdc8a2cd26 Added tests for utils and serializers 2018-11-15 17:41:18 +05:00
M1ha
235b690f35 Added tests for utils and serializers 2018-11-15 17:37:58 +05:00
M1ha
8e9963ad3b Added sync mechanism without tasks, but not tested as no migrations 2018-11-15 16:50:38 +05:00
M1ha
7c39bd5c84 BUG comment on update 2018-11-15 13:03:24 +05:00