Ines Montani
|
fb9d3bb022
|
Revert "Merge remote-tracking branch 'origin/master'"
This reverts commit d3b181cdf1, reversing
changes made to b19cfcc144.
|
2017-01-03 18:21:36 +01:00 |
|
Matthew Honnibal
|
3ba7c167a8
|
Fix URL tests
|
2016-12-30 17:10:08 -06:00 |
|
Matthew Honnibal
|
3e8d9c772e
|
Test interaction of token_match and punctuation
Check that the new token_match function applies after punctuation is split off.
|
2016-12-31 00:52:17 +11:00 |
|
Gyorgy Orosz
|
1748549aeb
|
Added exception pattern mechanism to the tokenizer.
|
2016-12-21 23:16:19 +01:00 |
|
Ines Montani
|
d1c1d3f9cd
|
Fix tokenizer test
|
2016-12-18 16:55:32 +01:00 |
|
Ines Montani
|
07f0efb102
|
Add test for tokenizer regular expressions
|
2016-12-07 20:33:28 +01:00 |
|
Matthew Honnibal
|
b6b01d4680
|
Remove deprecated tokens_from_list test.
|
2016-11-02 23:47:21 +01:00 |
|
Matthew Honnibal
|
cc8bf62208
|
* Fix Issue #360: Tokenizer failed when the infix regex matched the start of the string while trying to tokenize multi-infix tokens.
|
2016-05-09 13:23:47 +02:00 |
|
Matthew Honnibal
|
b4bfc6ae55
|
* Add test for Issue #351: Indices off when leading whitespace
|
2016-05-04 15:53:17 +02:00 |
|
Matthew Honnibal
|
6f82065761
|
* Fix infixed commas in tokenizer, re Issue #326. Need to benchmark on empirical data, to make sure this doesn't break other cases.
|
2016-04-14 11:36:03 +02:00 |
|
Matthew Honnibal
|
04d0209be9
|
* Recognise multiple infixes in a token.
|
2016-04-13 18:38:26 +10:00 |
|
Matthew Honnibal
|
b1fe41b45d
|
* Extend infix test, commenting on limitation of tokenizer w.r.t. infixes at the moment.
|
2016-03-29 14:31:05 +11:00 |
|
Matthew Honnibal
|
9c73983bdd
|
* Add test for hyphenation problem in Issue #302
|
2016-03-29 14:27:13 +11:00 |
|
Henning Peters
|
c12d3dd200
|
add __init__.py to empty package dirs
|
2016-03-14 11:28:03 +01:00 |
|
Henning Peters
|
9d8966a2c0
|
Update test_tokenizer.py
|
2016-02-10 19:24:37 +01:00 |
|
Matthew Honnibal
|
7f24229f10
|
* Don't try to pickle the tokenizer
|
2016-02-06 14:09:05 +01:00 |
|
Matthew Honnibal
|
515493c675
|
* Add xfail test for Issue #225: tokenization with non-whitespace delimiters
|
2016-01-19 13:20:14 +01:00 |
|
Matthew Honnibal
|
223d2b3484
|
* Add test for Issue #154: Additional whitespace introduced when string ends with a whitespace token.
|
2016-01-16 17:08:07 +01:00 |
|
Matthew Honnibal
|
3fbfba575a
|
* xfail the contractions test
|
2015-12-31 13:16:28 +01:00 |
|
Matthew Honnibal
|
4b4eec8b47
|
* Fix Issue #201: Tokenization of there'll
|
2015-12-29 18:09:09 +01:00 |
|
Matthew Honnibal
|
4e16f9e435
|
* Move tests underneath spacy/
|
2015-10-26 00:07:31 +11:00 |
|