7 Commits

Author	SHA1	Message	Date
Matthew Honnibal	b5b869366b	* Adjust hyphenation rule in tokenizer	2015-06-28 06:18:58 +02:00
Matthew Honnibal	45ec92243a	* Add hyphenation rule to infix.txt for tokenizer	2015-06-06 05:56:00 +02:00
Jordan Suchow	3a8d9b37a6	Remove trailing whitespace	2015-04-19 13:01:38 -07:00
Matthew Honnibal	0c25001325	* Fix specials.json	2015-04-12 04:45:41 +02:00
Matthew Honnibal	056c672caf	* Bug fixes to tokenization, and support for times	2015-03-26 16:44:48 +01:00
Matthew Honnibal	13520e6cf0	* Add i.e. to specials.json	2015-03-26 16:44:45 +01:00
Matthew Honnibal	5e27bd0c4c	* Add en language data, for tokenizer etc	2015-02-25 17:10:32 -05:00