explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2025-10-30 23:47:31 +03:00
Code Issues Packages Projects Releases Wiki Activity

extend suffixes from first to last

Browse Source 
reverse suffix list in `tokenizer_pseudo_code()` so the order of returned tokens matches input order
This commit is contained in:
Yuval Pinter 2017-05-22 10:56:03 -04:00 committed by GitHub
parent 6c67482f34
commit af3d121ec9
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
website/docs/usage/customizing-tokenizer.jade
View File

@ -113,7 +113,7 @@ p
else:
tokens.append(substring)
substring = ''
tokens.extend(suffixes)
tokens.extend(reversed(suffixes))
return tokens
p