Adriane Boyd 11e195d3ed Update ChineseTokenizer ... * Allow `pkuseg_model` to be set to `None` on initialization * Don't save config within tokenizer * Force convert pkuseg_model to use pickle protocol 4 by reencoding with `pickle5` on serialization * Update pkuseg serialization test		2020-09-27 14:00:18 +02:00
..
__init__.py	Rework Chinese language initialization and tokenization (#4619)	2019-11-11 14:23:21 +01:00
test_serialize.py	Update ChineseTokenizer	2020-09-27 14:00:18 +02:00
test_text.py	Merge branch 'develop' into master-tmp	2020-05-21 18:39:06 +02:00
test_tokenizer.py	Improve CLI config validation with latest Thinc	2020-09-26 13:13:57 +02:00