Adriane Boyd 1d59fdbd39 Update Vietnamese tokenizer (#8099) ... * Adapt tokenization methods from `pyvi` to preserve text encoding and whitespace * Add serialization support similar to Chinese and Japanese Note: as for Chinese and Japanese, some settings are duplicated in `config.cfg` and `tokenizer/cfg`.		2021-05-17 18:16:20 +10:00
..
am	Tidy up and auto-format	2021-01-15 11:57:36 +11:00
ar	Remove POS, TAG and LEMMA from tokenizer exceptions	2020-07-22 23:09:01 +02:00
bg	Bulgarian tokenizer exceptions (#7114)	2021-02-19 19:19:19 +01:00
bn	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
ca	Remove POS, TAG and LEMMA from tokenizer exceptions	2020-07-22 23:09:01 +02:00
cs	Remove unicode declarations and update language data	2020-09-04 13:19:16 +02:00
da	Merge remote-tracking branch 'upstream/master' into chore/update-develop-from-master-rc3	2021-01-14 11:49:58 +01:00
de	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
el	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
en	raise NotImplementedError when noun_chunks iterator is not implemented (#6711)	2021-01-17 19:56:05 +08:00
es	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
eu	Merge branch 'develop' into master-tmp	2020-05-21 18:39:06 +02:00
fa	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
fi	Merge branch 'master' into tmp/sync	2020-03-26 13:38:14 +01:00
fr	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
ga	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
gu	Remove unicode declarations and tidy up	2020-06-21 22:34:10 +02:00
he	Merge branch 'develop' into master-tmp	2020-09-04 13:15:36 +02:00
hi	Auto-format [ci skip]	2020-10-15 10:08:53 +02:00
hu	Tidy up and auto-format	2020-03-25 12:28:12 +01:00
hy	Remove unicode declarations and tidy up	2020-06-21 22:34:10 +02:00
id	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
it	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
ja	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
ko	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
ky	Tidy up and auto-format	2021-01-30 12:52:33 +11:00
lb	Remove POS, TAG and LEMMA from tokenizer exceptions	2020-07-22 23:09:01 +02:00
lt	Merge branch 'master' into tmp/sync	2020-03-26 13:38:14 +01:00
mk	Tidy up and auto-format	2021-01-05 13:41:53 +11:00
ml	Remove unicode declarations and tidy up	2020-06-21 22:34:10 +02:00
nb	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
ne	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
nl	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
pl	Merge branch 'develop' into master-tmp	2020-05-21 18:39:06 +02:00
pt	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
ro	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
ru	Tidy up tests and docs	2020-09-21 20:43:54 +02:00
sa	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
sr	Un-xfail passing tests	2019-12-25 18:02:20 +01:00
sv	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
th	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
ti	Tidy up and auto-format	2021-01-15 11:57:36 +11:00
tr	Tidy up and auto-format	2021-01-05 13:41:53 +11:00
tt	Merge branch 'master' into develop	2020-02-18 14:47:23 +01:00
uk	unskipping tests with python >= 3.6	2020-12-30 18:46:43 +01:00
ur	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
vi	Update Vietnamese tokenizer (#8099)	2021-05-17 18:16:20 +10:00
yo	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
zh	Tidy up and auto-format	2020-10-03 17:20:18 +02:00
__init__.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_attrs.py	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
test_initialize.py	Fix punctuation settings and add to initialize tests	2021-02-13 11:51:21 +11:00
test_lemmatizers.py	Add Spanish rule-based lemmatizer (#6833)	2021-01-27 19:21:35 +08:00