adrianeboyd 3bf111585d Update Japanese tokenizer config and add serialization (#5562) ... * Use `config` dict for tokenizer settings * Add serialization of split mode setting * Add tests for tokenizer split modes and serialization of split mode setting Based on #5561		2020-06-08 16:29:05 +02:00
..
__init__.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_lemmatization.py	Add Japanese Model (#5544)	2020-06-04 19:15:43 +02:00
test_serialize.py	Update Japanese tokenizer config and add serialization (#5562)	2020-06-08 16:29:05 +02:00
test_tokenizer.py	Update Japanese tokenizer config and add serialization (#5562)	2020-06-08 16:29:05 +02:00