Duygu Altinok 0e55f806dd Turkish tokenization improvements (#6268) ... * added single and paired orth variants * added token match * added long text tokenization test * inverted init * normalized lemmas to lowercase * more abbrevs * tests for ordinals and abbrevs * separated period abbvrevs to another list * fiex typo * added ordinal and abbrev tests * added number tests for dates * minor refinement * added inflected abbrevs regex * added percentage and inflection * cosmetics * added token match * added url inflection tests * excluded url tokens from custom pattern * removed url match import		2020-10-29 09:43:17 +01:00
..
ar
bn
ca
cs	Adding num_like test for Czech (#5946)	2020-08-21 17:06:33 +02:00
da	Reduce stored lexemes data, move feats to lookups (#5238)	2020-05-19 15:59:14 +02:00
de	Tidy up and auto-format	2020-05-21 14:14:01 +02:00
el	Tidy up and auto-format	2020-05-21 14:14:01 +02:00
en	English: adds ordinal numbers (#5830)	2020-07-29 20:22:47 +02:00
es	Tidy up and auto-format	2020-05-21 14:14:01 +02:00
eu
fa
fi
fr	Tidy up and auto-format	2020-05-21 14:14:01 +02:00
ga
gu	Tidy up and auto-format	2020-05-21 14:14:01 +02:00
he	Hebrew like num (#5952)	2020-08-24 14:30:05 +02:00
hi	Hindi: Adds tests for lexical attributes (norm and like_num) (#5829)	2020-10-07 10:23:32 +02:00
hu
hy	Add missing declaration	2020-05-21 17:30:05 +02:00
id	Tidy up and auto-format	2020-05-21 14:14:01 +02:00
it
ja	Revert "Convert custom user_data to token extension format for Japanese tokenizer (#5652)" (#5665)	2020-06-29 14:34:15 +02:00
ko
lb	Reduce stored lexemes data, move feats to lookups (#5238)	2020-05-19 15:59:14 +02:00
lt
mk	Include Macedonian language (#6230)	2020-10-15 15:55:01 +02:00
ml	Tidy up and auto-format	2020-05-21 14:14:01 +02:00
nb	Tidy up and auto-format	2020-05-21 14:14:01 +02:00
ne	Add Nepali Language (#5622)	2020-06-22 10:25:46 +02:00
nl
pl	Update Polish tokenizer for UD_Polish-PDB (#5432)	2020-05-19 15:59:55 +02:00
pt
ro
ru
sa	Added support for Sanskrit language (#5956)	2020-08-25 10:56:29 +02:00
sr
sv	Tidy up and auto-format	2020-05-21 14:14:01 +02:00
th
tr	Turkish tokenization improvements (#6268)	2020-10-29 09:43:17 +01:00
tt
uk
ur
yo
zh	Tidy up and auto-format	2020-05-21 14:14:01 +02:00
__init__.py
test_attrs.py
test_initialize.py