Adriane Boyd 11e195d3ed Update ChineseTokenizer ... * Allow `pkuseg_model` to be set to `None` on initialization * Don't save config within tokenizer * Force convert pkuseg_model to use pickle protocol 4 by reencoding with `pickle5` on serialization * Update pkuseg serialization test		2020-09-27 14:00:18 +02:00
..
__init__.py	Update ChineseTokenizer	2020-09-27 14:00:18 +02:00
examples.py	Tidy up and auto-format	2020-02-18 15:38:18 +01:00
lex_attrs.py	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
stop_words.py	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00