Antti Ajanki e1f777b151 Improvements for Finnish tokenizer (#4985) ... * don't split on a colon. Colon is used to attach suffixes for abbreviations * tokenize on any of LIST_HYPHENS (except a single hyphen), not just on -- * simplify infix rules by merging similar rules		2020-02-10 20:32:43 -05:00
..
__init__.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_text.py	Improvements to the Finnish language data (#4738)	2019-12-03 12:55:28 +01:00
test_tokenizer.py	Improvements for Finnish tokenizer (#4985)	2020-02-10 20:32:43 -05:00