explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-12-26 01:46:28 +03:00
Code Issues Packages Projects Releases Wiki Activity

Update base norm exceptions with more unicode characters

Browse Source 
e.g. unicode variations of punctuation used in Chinese
This commit is contained in:
ines 2017-10-14 14:58:52 +02:00
parent 3516aa0cea
commit 9d6c8eaa49
1 changed files with 10 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
spacy/lang/norm_exceptions.py
View File

@ -31,11 +31,21 @@ BASE_NORMS = {
"„": '"', "„": '"',
"»": '"', "»": '"',
"«": '"', "«": '"',
"‘‘": '"',
"’’": '"',
"？": "?",
"！": "!",
"，": ",",
"；": ";",
"：": ":",
"。": ".",
"।": ".",
"…": "...", "…": "...",
"—": "-", "—": "-",
"–": "-", "–": "-",
"--": "-", "--": "-",
"---": "-", "---": "-",
"——": "-",
"€": "$", "€": "$",
"£": "$", "£": "$",
"¥": "$", "¥": "$",