2 Commits

Author	SHA1	Message	Date
Ines Montani	3d577b77c6	Auto-formatting	2019-02-14 19:56:38 +01:00
Grivaz	39815513e2	Add split one token into several (resolves #2838) (#3253) ... * Add split one token into several (resolves #2838) * Improve error message for token splitting * Make retokenizer.split() tests use a Token object Change retokenizer.split() to use a Token object, instead of an index. * Pass Token into retokenize.split() Tweak retokenize.split() API so that we pass the `Token` object, not the index. * Fix token.idx in retokenize.split() * Test that token.idx is correct after split * Fix token.idx for split tokens * Fix retokenize.split() * Fix retokenize.split * Fix retokenize.split() test	2019-02-15 01:27:13 +11:00