Adriane Boyd 1d59fdbd39

Update Vietnamese tokenizer (#8099) ...

* Adapt tokenization methods from `pyvi` to preserve text encoding and
whitespace
* Add serialization support similar to Chinese and Japanese

Note: as for Chinese and Japanese, some settings are duplicated in
`config.cfg` and `tokenizer/cfg`.

2021-05-17 18:16:20 +10:00

0 lines

Python

Raw Blame History