2 Commits

Author	SHA1	Message	Date
Antti Ajanki	e1f777b151	Improvements for Finnish tokenizer (#4985) ... * don't split on a colon. Colon is used to attach suffixes for abbreviations * tokenize on any of LIST_HYPHENS (except a single hyphen), not just on -- * simplify infix rules by merging similar rules	2020-02-10 20:32:43 -05:00
Antti Ajanki	e626a011cc	Improvements to the Finnish language data (#4738) ... * Enable lex_attrs on Finnish * Copy the Danish tokenizer rules to Finnish Specifically, don't break hyphenated compound words * Contributor agreement * A new file for Finnish tokenizer rules instead of including the Danish ones	2019-12-03 12:55:28 +01:00