explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-12-25 09:26:27 +03:00
Code Issues Packages Projects Releases Wiki Activity

Change JA inflection separator to semicolon

Browse Source 
Hyphen is unsuitable because of interactions with the JA data fields,
but pipe is also unsuitable because it has a different meaning in UD
data, so it's better to use something that has no significance in either
case. So this uses semicolon.
This commit is contained in:
Paul O'Leary McCann 2021-10-07 17:28:15 +09:00
parent 227f98081b
commit c4e3b7a5db
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
spacy/lang/ja/__init__.py
View File

@ -94,7 +94,7 @@ class JapaneseTokenizer(DummyTokenizer):
DetailedToken(
token.surface(), # orth
"-".join([xx for xx in token.part_of_speech()[:4] if xx != "*"]), # tag
"|".join([xx for xx in token.part_of_speech()[4:] if xx != "*"]), # inf
";".join([xx for xx in token.part_of_speech()[4:] if xx != "*"]), # inf
token.dictionary_form(), # lemma
token.normalized_form(),
token.reading_form(),