5 Commits

Author	SHA1	Message	Date
Ines Montani	0894b8c0ef	Don't split tokens with digits and "/" infixes (resolves #740)	2017-01-12 22:58:26 +01:00
Ines Montani	eef94e3ee2	Split off period after two or more uppercase letters (fixes #483)	2017-01-08 22:28:25 +01:00
Ines Montani	347c4a2d06	Reorganise and reformat global tokenizer prefixes, suffixes and infixes	2017-01-08 20:37:39 +01:00
Ines Montani	eaa3b1319d	Fix formatting	2016-12-18 15:36:53 +01:00
Ines Montani	e47ee94761	Split punctuation into its own file	2016-12-08 19:46:43 +01:00