spaCy/spacy/tests/tokenizer
Adriane Boyd 29b058ebdc
Fix spacy when retokenizing cases with affixes (#6475)
Preserve `token.spacy` corresponding to the span end token in the
original doc rather than adjusting for the current offset.

* If not modifying in place, this checks in the original document
(`doc.c` rather than `tokens`).
* If modifying in place, the document has not been modified past the
current span start position so the value at the current span end
position is valid.
2020-12-08 14:25:56 +08:00
..
__init__.py Revert #4334 2019-09-29 17:32:12 +02:00
sun.txt Revert #4334 2019-09-29 17:32:12 +02:00
test_exceptions.py Tidy up and auto-format 2020-02-18 15:38:18 +01:00
test_explain.py Refactor pipeline components, config and language data (#5759) 2020-07-22 13:42:59 +02:00
test_naughty_strings.py Merge branch 'develop' into master-tmp 2020-09-04 13:15:36 +02:00
test_tokenizer.py Fix spacy when retokenizing cases with affixes (#6475) 2020-12-08 14:25:56 +08:00
test_urls.py Merge branch 'develop' into master-tmp 2020-06-20 15:52:00 +02:00
test_whitespace.py Merge branch 'develop' into master-tmp 2020-09-04 13:15:36 +02:00