Adriane Boyd c5de9b463a Update custom tokenizer APIs and pickling (#8972) ... * Fix incorrect pickling of Japanese and Korean pipelines, which led to the entire pipeline being reset if pickled * Enable pickling of Vietnamese tokenizer * Update tokenizer APIs for Chinese, Japanese, Korean, Thai, and Vietnamese so that only the `Vocab` is required for initialization		2021-08-19 14:37:47 +02:00
..
__init__.py
test_serialize.py	Update custom tokenizer APIs and pickling (#8972)	2021-08-19 14:37:47 +02:00
test_tokenizer.py