Adriane Boyd c5de9b463a Update custom tokenizer APIs and pickling (#8972) ... * Fix incorrect pickling of Japanese and Korean pipelines, which led to the entire pipeline being reset if pickled * Enable pickling of Vietnamese tokenizer * Update tokenizer APIs for Chinese, Japanese, Korean, Thai, and Vietnamese so that only the `Vocab` is required for initialization		2021-08-19 14:37:47 +02:00
..
__init__.py	Update custom tokenizer APIs and pickling (#8972)	2021-08-19 14:37:47 +02:00
examples.py	Tidy up and auto-format	2020-02-18 15:38:18 +01:00
stop_words.py	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
syntax_iterators.py	Tidy up and move noun_chunks, token_match, url_match	2020-07-22 22:18:46 +02:00
tag_bigram_map.py	Tidy up and auto-format	2020-06-21 22:38:04 +02:00
tag_map.py	Merge branch 'develop' into master-tmp	2020-06-20 15:52:00 +02:00
tag_orth_map.py	Tidy up and auto-format	2020-06-21 22:38:04 +02:00