explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-09-21 19:39:13 +03:00
Code Issues Packages Projects Releases Wiki Activity
5d4eede1e4
spaCy/spacy/tests/lang/zh
History
adrianeboyd 0b9a5f4074 Rework Chinese language initialization and tokenization (#4619) 
* Rework Chinese language initialization

* Create a `ChineseTokenizer` class
  * Modify jieba post-processing to handle whitespace correctly
  * Modify non-jieba character tokenization to handle whitespace correctly

* Add a `create_tokenizer()` method to `ChineseDefaults`

* Load lexical attributes

* Update Chinese tag_map for UD v2

* Add very basic Chinese tests

* Test tokenization with and without jieba

* Test `like_num` attribute

* Fix try_jieba_import()

* Fix zh code formatting
2019-11-11 14:23:21 +01:00
..
__init__.py Rework Chinese language initialization and tokenization (#4619) 2019-11-11 14:23:21 +01:00
test_text.py Rework Chinese language initialization and tokenization (#4619) 2019-11-11 14:23:21 +01:00
test_tokenizer.py Rework Chinese language initialization and tokenization (#4619) 2019-11-11 14:23:21 +01:00