Matthew Honnibal
|
454c1996d0
|
* Add tokenizer rule to fix numeric range tokenization
|
2015-10-17 15:49:51 +11:00 |
|
Matthew Honnibal
|
3cfe3d8c1c
|
* Revert bad infix change
|
2015-07-26 17:32:37 +02:00 |
|
Matthew Honnibal
|
bd608559bc
|
* Fix infix-period tokenization
|
2015-07-26 17:14:52 +02:00 |
|
Matthew Honnibal
|
94f314c271
|
* Fix tokenization of email addresses.
|
2015-07-26 16:38:08 +02:00 |
|
Matthew Honnibal
|
14e9e6ec6c
|
* Fix ... tokenization, and correct orth inconsistencies in specials.json
|
2015-07-20 12:10:56 +02:00 |
|
Matthew Honnibal
|
b5b869366b
|
* Adjust hyphenation rule in tokenizer
|
2015-06-28 06:18:58 +02:00 |
|
Matthew Honnibal
|
45ec92243a
|
* Add hyphenation rule to infix.txt for tokenizer
|
2015-06-06 05:56:00 +02:00 |
|
Matthew Honnibal
|
5e27bd0c4c
|
* Add en language data, for tokenizer etc
|
2015-02-25 17:10:32 -05:00 |
|