JapaneseTokenizer.pipe added (#6515)

* JapaneseTokenizer.pipe added

For [spacymoji](https://spacy.io/universe/project/spacymoji)  with `Japanese()`.

* DummyTokenizer.pipe added instead
This commit is contained in:
Koichi Yasuoka 2020-12-09 04:02:23 +09:00 committed by GitHub
parent df4891bed1
commit 0afb54ac93
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -838,6 +838,13 @@ class SimpleFrozenDict(dict):
class DummyTokenizer(object):
def __call__(self, text):
raise NotImplementedError
def pipe(self, texts, **kwargs):
for text in texts:
yield self(text)
# add dummy methods for to_bytes, from_bytes, to_disk and from_disk to
# allow serialization (see #1557)
def to_bytes(self, **kwargs):