1 Commits

Author	SHA1	Message	Date
adrianeboyd	3bf111585d	Update Japanese tokenizer config and add serialization (#5562) ... * Use `config` dict for tokenizer settings * Add serialization of split mode setting * Add tests for tokenizer split modes and serialization of split mode setting Based on #5561	2020-06-08 16:29:05 +02:00