explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-09-22 11:59:14 +03:00
Code Issues Packages Projects Releases Wiki Activity
71f5a5daa1
spaCy/spacy/tests/lang
History
adrianeboyd 0b9a5f4074 Rework Chinese language initialization and tokenization (#4619) 
* Rework Chinese language initialization

* Create a `ChineseTokenizer` class
  * Modify jieba post-processing to handle whitespace correctly
  * Modify non-jieba character tokenization to handle whitespace correctly

* Add a `create_tokenizer()` method to `ChineseDefaults`

* Load lexical attributes

* Update Chinese tag_map for UD v2

* Add very basic Chinese tests

* Test tokenization with and without jieba

* Test `like_num` attribute

* Fix try_jieba_import()

* Fix zh code formatting
2019-11-11 14:23:21 +01:00
..
ar Revert #4334 2019-09-29 17:32:12 +02:00
bn Revert #4334 2019-09-29 17:32:12 +02:00
ca Revert #4334 2019-09-29 17:32:12 +02:00
da Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
de Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
el Revert #4334 2019-09-29 17:32:12 +02:00
en Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
es Revert #4334 2019-09-29 17:32:12 +02:00
fi Revert #4334 2019-09-29 17:32:12 +02:00
fr Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
ga Revert #4334 2019-09-29 17:32:12 +02:00
he Revert #4334 2019-09-29 17:32:12 +02:00
hu Revert #4334 2019-09-29 17:32:12 +02:00
id Revert #4334 2019-09-29 17:32:12 +02:00
it Revert #4334 2019-09-29 17:32:12 +02:00
ja Revert #4334 2019-09-29 17:32:12 +02:00
ko Revert #4334 2019-09-29 17:32:12 +02:00
lb Tidy up and auto-format 2019-10-18 11:27:38 +02:00
lt Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
nb Revert #4334 2019-09-29 17:32:12 +02:00
nl Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
pl Revert #4334 2019-09-29 17:32:12 +02:00
pt Revert #4334 2019-09-29 17:32:12 +02:00
ro Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
ru Revert #4334 2019-09-29 17:32:12 +02:00
sr Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
sv Tidy up and auto-format [ci skip] 2019-10-24 16:20:48 +02:00
th Revert #4334 2019-09-29 17:32:12 +02:00
tr Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
tt Revert #4334 2019-09-29 17:32:12 +02:00
uk Revert #4334 2019-09-29 17:32:12 +02:00
ur Revert #4334 2019-09-29 17:32:12 +02:00
zh Rework Chinese language initialization and tokenization (#4619) 2019-11-11 14:23:21 +01:00
__init__.py Revert #4334 2019-09-29 17:32:12 +02:00
test_attrs.py Revert #4334 2019-09-29 17:32:12 +02:00
test_initialize.py Revert #4334 2019-09-29 17:32:12 +02:00