spaCy/spacy/zh/__init__.py

from ..language import Language
from ..tokens import Doc


class Chinese(Language):
    lang = u'zh'

    def make_doc(self, text):
        import jieba
        words = list(jieba.cut(text, cut_all=False))
        words=[x for x in words if x]
        return Doc(self.vocab, words=words, spaces=[False]*len(words))
Add draft Jieba tokenizer for Chinese 2016-11-02 21:57:38 +03:00			`from ..language import Language`
			`from ..tokens import Doc`
* Add initial stuff for Chinese parsing 2016-04-24 19:44:24 +03:00

			`class Chinese(Language):`
			`lang = u'zh'`
* Work on Chinese support 2016-05-05 12:39:12 +03:00
Add draft Jieba tokenizer for Chinese 2016-11-02 21:57:38 +03:00			`def make_doc(self, text):`
Import Jieba inside zh.make_doc 2016-11-03 01:49:19 +03:00			`import jieba`
Update __init__.py Set the "cut_all" parameter to False, or jieba will return ALL POSSIBLE word segmentations. 2017-09-12 11:23:47 +03:00			`words = list(jieba.cut(text, cut_all=False))`
Update __init__.py remove the empty string return from jieba.cut,this will cause the list of tokens cant be pushed assert error 2017-07-01 08:12:00 +03:00			`words=[x for x in words if x]`
Add draft Jieba tokenizer for Chinese 2016-11-02 21:57:38 +03:00			`return Doc(self.vocab, words=words, spaces=[False]*len(words))`