spaCy/spacy/lang
adrianeboyd f7471abd82
Add pkuseg and serialization support for Chinese (#5308)
* Add pkuseg and serialization support for Chinese

Add support for pkuseg alongside jieba

* Specify model through `Language` meta:

  * split on characters (if no word segmentation packages are installed)

```
Chinese(meta={"tokenizer": {"config": {"use_jieba": False, "use_pkuseg": False}}})
```

  * jieba (remains the default tokenizer if installed)

```
Chinese()
Chinese(meta={"tokenizer": {"config": {"use_jieba": True}}}) # explicit
```

  * pkuseg

```
Chinese(meta={"tokenizer": {"config": {"pkuseg_model": "default", "use_jieba": False, "use_pkuseg": True}}})
```

* The new tokenizer setting `require_pkuseg` is used to override
`use_jieba` default, which is intended for models that provide a pkuseg
model:

```
nlp_pkuseg = Chinese(meta={"tokenizer": {"config": {"pkuseg_model": "default", "require_pkuseg": True}}})
nlp = Chinese() # has `use_jieba` as `True` by default
nlp.from_bytes(nlp_pkuseg.to_bytes()) # `require_pkuseg` overrides `use_jieba` when calling the tokenizer
```

Add support for serialization of tokenizer settings and pkuseg model, if
loaded

* Add sorting for `Language.to_bytes()` serialization of `Language.meta`
so that the (emptied, but still present) tokenizer metadata is in a
consistent position in the serialized data

Extend tests to cover all three tokenizer configurations and
serialization

* Fix from_disk and tests without jieba or pkuseg

* Load cfg first and only show error if `use_pkuseg`
* Fix blank/default initialization in serialization tests

* Explicitly initialize jieba's cache on init

* Add serialization for pkuseg pre/postprocessors

* Reformat pkuseg install message
2020-04-18 17:01:53 +02:00
..
af 💫 Add base Language classes for more languages (#3276) 2019-02-15 01:31:19 +11:00
ar Add writing_system to ArabicDefaults (experimental) 2019-03-11 14:22:23 +01:00
bg Update examples and languages.json [ci skip] 2019-09-15 17:56:40 +02:00
bn Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
ca Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
cs 💫 Add base Language classes for more languages (#3276) 2019-02-15 01:31:19 +11:00
da Extend and fix Danish examples (#5227) 2020-04-02 10:42:35 +02:00
de Tidy up and auto-format 2020-03-25 12:28:12 +01:00
el Standardize Greek tag map setup (#4997) 2020-02-11 17:44:56 -05:00
en Tidy up and auto-format [ci skip] 2019-10-24 16:20:48 +02:00
es Improve tokenization for UD Spanish AnCora (#5253) 2020-04-06 13:18:23 +02:00
et 💫 Add base Language classes for more languages (#3276) 2019-02-15 01:31:19 +11:00
eu Tidy up and auto-format 2020-03-25 12:28:12 +01:00
fa Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
fi add two abbreviations and some additional unit tests (#5040) 2020-02-22 14:12:32 +01:00
fr Use inline flags in token_match patterns (#5257) 2020-04-06 13:19:04 +02:00
ga 💫 Tidy up and auto-format .py files (#2983) 2018-11-30 17:03:03 +01:00
he Auto-format [ci skip] 2019-03-11 17:10:50 +01:00
hi Fix example sentences in Hindi for grammatical errors (#4343) 2019-09-30 23:32:49 +02:00
hr Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
hu Improve URL_PATTERN and handling in tokenizer (#4374) 2019-10-05 13:00:09 +02:00
hy Armenian language support (#5246) 2020-04-03 13:02:18 +02:00
id Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
is 💫 Add base Language classes for more languages (#3276) 2019-02-15 01:31:19 +11:00
it Tidy up and auto-format 2020-03-25 12:28:12 +01:00
ja Tidy up and auto-format 2019-12-21 19:04:17 +01:00
kn Add kannada examples (#5162) 2020-03-29 13:54:42 +02:00
ko Auto-format 2019-11-20 13:15:24 +01:00
lb Update tokenizer_exceptions.py 2020-02-14 12:02:15 +01:00
lij Tidy up and auto-format 2020-03-25 12:28:12 +01:00
lt Tidy up and auto-format 2020-03-25 12:28:12 +01:00
lv 💫 Add base Language classes for more languages (#3276) 2019-02-15 01:31:19 +11:00
mr Tidy up [ci skip] 2019-06-12 13:38:23 +02:00
nb Tidy up and auto-format 2020-03-25 12:28:12 +01:00
nl Improve tokenization for UD Dutch corpora (#5259) 2020-04-06 13:18:07 +02:00
pl Tidy up and auto-format 2019-08-20 17:36:34 +02:00
pt Tidy up and auto-format 2020-03-25 12:28:12 +01:00
ro Improved Romanian tokenization for UD RRT (#5036) 2020-02-19 16:15:59 +01:00
ru Refactor lemmatizer and data table integration (#4353) 2019-10-01 21:36:03 +02:00
si 💫 Tidy up and auto-format .py files (#2983) 2018-11-30 17:03:03 +01:00
sk Tidy up and auto-format 2020-03-25 12:28:12 +01:00
sl 💫 Add base Language classes for more languages (#3276) 2019-02-15 01:31:19 +11:00
sq Update languages and examples (see #1107) 2019-06-26 16:19:17 +02:00
sr Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
sv Adding noun_chunks to the Swedish language model (sv) (#4422) 2019-10-21 12:57:06 +02:00
ta Remove stray print statement (closes #3342) 2019-02-27 15:35:04 +01:00
te 💫 Tidy up and auto-format .py files (#2983) 2018-11-30 17:03:03 +01:00
th fix thai bug (#3693) 2019-05-10 14:21:34 +02:00
tl Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
tr Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
tt Tidy up and auto-format 2019-08-20 17:36:34 +02:00
uk Update Ukrainian lemmatizer with new lookups (#4359) 2019-10-02 12:04:06 +02:00
ur Move lookup tables out of the core library (#4346) 2019-10-01 00:01:27 +02:00
vi 💫 Tidy up and auto-format .py files (#2983) 2018-11-30 17:03:03 +01:00
xx Minor updates to language example sentences (#4608) 2019-11-07 22:34:58 +01:00
yo Adding support for Yoruba Language (#4614) 2019-12-21 14:11:50 +01:00
zh Add pkuseg and serialization support for Chinese (#5308) 2020-04-18 17:01:53 +02:00
__init__.py Remove imports in /lang/__init__.py 2017-05-08 23:58:07 +02:00
char_classes.py Add CJK to character classes (#4884) 2020-01-08 16:50:19 +01:00
lex_attrs.py Replacing regex library with re to increase tokenization speed (#3218) 2019-02-01 18:05:22 +11:00
norm_exceptions.py Update norm_exceptions.py (#3778) 2019-05-27 11:52:52 +02:00
punctuation.py Allow period as suffix following punctuation (#4248) 2019-09-09 19:19:22 +02:00
tag_map.py 💫 Tidy up and auto-format .py files (#2983) 2018-11-30 17:03:03 +01:00
tokenizer_exceptions.py Use inline flags in token_match patterns (#5257) 2020-04-06 13:19:04 +02:00