explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2025-01-13 18:56:36 +03:00
Code Issues Packages Projects Releases Wiki Activity

Restore missing normalization from gold align

Browse Source 
PR #4526 missed extra lower-casing and spacing normalization.
This commit is contained in:
Matthew Honnibal 2019-10-27 13:47:08 +01:00
parent 554850206c
commit bddfbc7e1b
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
spacy/gold.pyx
View File

@ -77,6 +77,7 @@ _NORM_MAP = {"``": '"', "''": '"'}
def _normalize(tokens): def _normalize(tokens):
tokens = [w.replace(" ", "").lower() for w in tokens]
return [_NORM_MAP.get(word, word) for word in tokens] return [_NORM_MAP.get(word, word) for word in tokens]