spaCy

mirror of https://github.com/explosion/spaCy.git synced 2024-09-21 03:19:13 +03:00

History

Adriane Boyd 1d59fdbd39 Update Vietnamese tokenizer (#8099 ) * Adapt tokenization methods from `pyvi` to preserve text encoding and whitespace * Add serialization support similar to Chinese and Japanese Note: as for Chinese and Japanese, some settings are duplicated in `config.cfg` and `tokenizer/cfg`.	2021-05-17 18:16:20 +10:00
..
3rd_party_licenses.txt	Update Vietnamese tokenizer (#8099 )	2021-05-17 18:16:20 +10:00

* Adapt tokenization methods from `pyvi` to preserve text encoding and
whitespace
* Add serialization support similar to Chinese and Japanese

Note: as for Chinese and Japanese, some settings are duplicated in
`config.cfg` and `tokenizer/cfg`.

2021-05-17 18:16:20 +10:00

3rd_party_licenses.txt

Update Vietnamese tokenizer (#8099 )

2021-05-17 18:16:20 +10:00