Adriane Boyd 1dd38191ec Convert custom user_data to token extension format for Japanese tokenizer (#5652) ... * Convert custom user_data to token extension format Convert the user_data values so that they can be loaded as custom token extensions for `inflection`, `reading_form`, `sub_tokens`, and `lemma`. * Reset Underscore state in ja tokenizer tests		2020-06-29 14:20:26 +02:00
..
__init__.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_lemmatization.py	Add Japanese Model (#5544)	2020-06-04 19:15:43 +02:00
test_serialize.py	Update Japanese tokenizer config and add serialization (#5562)	2020-06-08 16:29:05 +02:00
test_tokenizer.py	Convert custom user_data to token extension format for Japanese tokenizer (#5652)	2020-06-29 14:20:26 +02:00