Grivaz 39815513e2 Add split one token into several (resolves #2838) (#3253) ... * Add split one token into several (resolves #2838) * Improve error message for token splitting * Make retokenizer.split() tests use a Token object Change retokenizer.split() to use a Token object, instead of an index. * Pass Token into retokenize.split() Tweak retokenize.split() API so that we pass the `Token` object, not the index. * Fix token.idx in retokenize.split() * Test that token.idx is correct after split * Fix token.idx for split tokens * Fix retokenize.split() * Fix retokenize.split * Fix retokenize.split() test		2019-02-15 01:27:13 +11:00
..
__init__.pxd	* Break up tokens.pyx into tokens/doc.pyx, tokens/token.pyx, tokens/spans.pyx	2015-07-13 20:20:58 +02:00
__init__.py	Tidy up and document Doc, Token and Span	2017-10-27 15:41:45 +02:00
_retokenize.pyx	Add split one token into several (resolves #2838) (#3253)	2019-02-15 01:27:13 +11:00
doc.pxd	Fix issue 2396 (#3089)	2018-12-29 18:02:26 +01:00
doc.pyx	Bugfix/get lca matrix (#3110)	2019-01-06 19:07:50 +01:00
printers.py	check if argument flat is true or not (#3156)	2019-01-14 23:47:05 +01:00
span.pxd	Add Span.to_array method	2017-08-19 12:20:45 +02:00
span.pyx	Correct docs of Token.subtree and Span.subtree (issue #3122) (#3124)	2019-01-09 03:11:15 +01:00
token.pxd	💫 New system for error messages and warnings (#2163)	2018-04-03 15:50:31 +02:00
token.pyx	Correct docs of Token.subtree and Span.subtree (issue #3122) (#3124)	2019-01-09 03:11:15 +01:00
underscore.py	Don't raise error if set_extension has getter and setter (closes #2177)	2018-04-03 18:30:17 +02:00