2 Commits

Author	SHA1	Message	Date
Adriane Boyd	a2a0e1abf1	Switch to mecab-ko as default Korean tokenizer ... Switch to the (confusingly-named) mecab-ko python module for default Korean tokenization. Maintain the previous `natto-py` tokenizer as `spacy.KoreanNattoTokenizer.v1`.	2022-08-11 13:49:23 +02:00
Adriane Boyd	c5de9b463a	Update custom tokenizer APIs and pickling (#8972) ... * Fix incorrect pickling of Japanese and Korean pipelines, which led to the entire pipeline being reset if pickled * Enable pickling of Vietnamese tokenizer * Update tokenizer APIs for Chinese, Japanese, Korean, Thai, and Vietnamese so that only the `Vocab` is required for initialization	2021-08-19 14:37:47 +02:00