mirror of
https://github.com/explosion/spaCy.git
synced 2024-12-25 01:16:28 +03:00
39815513e2
* Add split one token into several (resolves #2838) * Improve error message for token splitting * Make retokenizer.split() tests use a Token object Change retokenizer.split() to use a Token object, instead of an index. * Pass Token into retokenize.split() Tweak retokenize.split() API so that we pass the `Token` object, not the index. * Fix token.idx in retokenize.split() * Test that token.idx is correct after split * Fix token.idx for split tokens * Fix retokenize.split() * Fix retokenize.split * Fix retokenize.split() test |
||
---|---|---|
.. | ||
__init__.pxd | ||
__init__.py | ||
_retokenize.pyx | ||
doc.pxd | ||
doc.pyx | ||
printers.py | ||
span.pxd | ||
span.pyx | ||
token.pxd | ||
token.pyx | ||
underscore.py |