mirror of
https://github.com/explosion/spaCy.git
synced 2024-11-14 05:37:03 +03:00
a5cd203284
* Reduce stored lexemes data, move feats to lookups
* Move non-derivable lexemes features (`norm / cluster / prob`) to
`spacy-lookups-data` as lookups
* Get/set `norm` in both lookups and `LexemeC`, serialize in lookups
* Remove `cluster` and `prob` from `LexemesC`, get/set/serialize in
lookups only
* Remove serialization of lexemes data as `vocab/lexemes.bin`
* Remove `SerializedLexemeC`
* Remove `Lexeme.to_bytes/from_bytes`
* Modify normalization exception loading:
* Always create `Vocab.lookups` table `lexeme_norm` for
normalization exceptions
* Load base exceptions from `lang.norm_exceptions`, but load
language-specific exceptions from lookups
* Set `lex_attr_getter[NORM]` including new lookups table in
`BaseDefaults.create_vocab()` and when deserializing `Vocab`
* Remove all cached lexemes when deserializing vocab to override
existing normalizations with the new normalizations (as a replacement
for the previous step that replaced all lexemes data with the
deserialized data)
* Skip English normalization test
Skip English normalization test because the data is now in
`spacy-lookups-data`.
* Remove norm exceptions
Moved to spacy-lookups-data.
* Move norm exceptions test to spacy-lookups-data
* Load extra lookups from spacy-lookups-data lazily
Load extra lookups (currently for cluster and prob) lazily from the
entry point `lg_extra` as `Vocab.lookups_extra`.
* Skip creating lexeme cache on load
To improve model loading times, do not create the full lexeme cache when
loading. The lexemes will be created on demand when processing.
* Identify numeric values in Lexeme.set_attrs()
With the removal of a special case for `PROB`, also identify `float` to
avoid trying to convert it with the `StringStore`.
* Skip lexeme cache init in from_bytes
* Unskip and update lookups tests for python3.6+
* Update vocab pickle to include lookups_extra
* Update vocab serialization tests
Check strings rather than lexemes since lexemes aren't initialized
automatically, account for addition of "_SP".
* Re-skip lookups test because of python3.5
* Skip PROB/float values in Lexeme.set_attrs
* Convert is_oov from lexeme flag to lex in vectors
Instead of storing `is_oov` as a lexeme flag, `is_oov` reports whether
the lexeme has a vector.
Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
97 lines
1.1 KiB
Cython
97 lines
1.1 KiB
Cython
# Reserve 64 values for flag features
|
|
from . cimport symbols
|
|
|
|
cdef enum attr_id_t:
|
|
NULL_ATTR
|
|
IS_ALPHA
|
|
IS_ASCII
|
|
IS_DIGIT
|
|
IS_LOWER
|
|
IS_PUNCT
|
|
IS_SPACE
|
|
IS_TITLE
|
|
IS_UPPER
|
|
LIKE_URL
|
|
LIKE_NUM
|
|
LIKE_EMAIL
|
|
IS_STOP
|
|
IS_OOV_DEPRECATED
|
|
IS_BRACKET
|
|
IS_QUOTE
|
|
IS_LEFT_PUNCT
|
|
IS_RIGHT_PUNCT
|
|
IS_CURRENCY
|
|
|
|
FLAG19 = 19
|
|
FLAG20
|
|
FLAG21
|
|
FLAG22
|
|
FLAG23
|
|
FLAG24
|
|
FLAG25
|
|
FLAG26
|
|
FLAG27
|
|
FLAG28
|
|
FLAG29
|
|
FLAG30
|
|
FLAG31
|
|
FLAG32
|
|
FLAG33
|
|
FLAG34
|
|
FLAG35
|
|
FLAG36
|
|
FLAG37
|
|
FLAG38
|
|
FLAG39
|
|
FLAG40
|
|
FLAG41
|
|
FLAG42
|
|
FLAG43
|
|
FLAG44
|
|
FLAG45
|
|
FLAG46
|
|
FLAG47
|
|
FLAG48
|
|
FLAG49
|
|
FLAG50
|
|
FLAG51
|
|
FLAG52
|
|
FLAG53
|
|
FLAG54
|
|
FLAG55
|
|
FLAG56
|
|
FLAG57
|
|
FLAG58
|
|
FLAG59
|
|
FLAG60
|
|
FLAG61
|
|
FLAG62
|
|
FLAG63
|
|
|
|
ID
|
|
ORTH
|
|
LOWER
|
|
NORM
|
|
SHAPE
|
|
PREFIX
|
|
SUFFIX
|
|
|
|
LENGTH
|
|
CLUSTER
|
|
LEMMA
|
|
POS
|
|
TAG
|
|
DEP
|
|
ENT_IOB
|
|
ENT_TYPE
|
|
HEAD
|
|
SENT_START
|
|
SPACY
|
|
PROB
|
|
|
|
LANG
|
|
ENT_KB_ID = symbols.ENT_KB_ID
|
|
ENT_ID = symbols.ENT_ID
|
|
|
|
IDX
|
|
SENT_END |