explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-11-10 19:57:17 +03:00
Code Issues Packages Projects Releases Wiki Activity
56fc3bc0f3
spaCy/spacy/symbols.pxd

470 lines
6.8 KiB
Cython
Raw Normal View History

Fix code explosion from long enum in Python 3, Cython 0.24+
2017-09-16 13:20:04 +03:00
cdef enum symbol_t:
* Remove qualified naming in symbols
2015-10-10 14:11:38 +03:00
NIL
IS_ALPHA
IS_ASCII
IS_DIGIT
IS_LOWER
IS_PUNCT
IS_SPACE
IS_TITLE
IS_UPPER
LIKE_URL
LIKE_NUM
LIKE_EMAIL
IS_STOP
Reduce stored lexemes data, move feats to lookups (#5238) * Reduce stored lexemes data, move feats to lookups * Move non-derivable lexemes features (`norm / cluster / prob`) to `spacy-lookups-data` as lookups * Get/set `norm` in both lookups and `LexemeC`, serialize in lookups * Remove `cluster` and `prob` from `LexemesC`, get/set/serialize in lookups only * Remove serialization of lexemes data as `vocab/lexemes.bin` * Remove `SerializedLexemeC` * Remove `Lexeme.to_bytes/from_bytes` * Modify normalization exception loading: * Always create `Vocab.lookups` table `lexeme_norm` for normalization exceptions * Load base exceptions from `lang.norm_exceptions`, but load language-specific exceptions from lookups * Set `lex_attr_getter[NORM]` including new lookups table in `BaseDefaults.create_vocab()` and when deserializing `Vocab` * Remove all cached lexemes when deserializing vocab to override existing normalizations with the new normalizations (as a replacement for the previous step that replaced all lexemes data with the deserialized data) * Skip English normalization test Skip English normalization test because the data is now in `spacy-lookups-data`. * Remove norm exceptions Moved to spacy-lookups-data. * Move norm exceptions test to spacy-lookups-data * Load extra lookups from spacy-lookups-data lazily Load extra lookups (currently for cluster and prob) lazily from the entry point `lg_extra` as `Vocab.lookups_extra`. * Skip creating lexeme cache on load To improve model loading times, do not create the full lexeme cache when loading. The lexemes will be created on demand when processing. * Identify numeric values in Lexeme.set_attrs() With the removal of a special case for `PROB`, also identify `float` to avoid trying to convert it with the `StringStore`. * Skip lexeme cache init in from_bytes * Unskip and update lookups tests for python3.6+ * Update vocab pickle to include lookups_extra * Update vocab serialization tests Check strings rather than lexemes since lexemes aren't initialized automatically, account for addition of "_SP". * Re-skip lookups test because of python3.5 * Skip PROB/float values in Lexeme.set_attrs * Convert is_oov from lexeme flag to lex in vectors Instead of storing `is_oov` as a lexeme flag, `is_oov` reports whether the lexeme has a vector. Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
2020-05-19 16:59:14 +03:00
IS_OOV_DEPRECATED
Update symbols and document missing token attributes (see #1439)
2017-10-20 14:08:44 +03:00
IS_BRACKET
IS_QUOTE
IS_LEFT_PUNCT
IS_RIGHT_PUNCT
added new symbol
2018-02-11 20:55:32 +03:00
IS_CURRENCY
Add support for Universal Dependencies v2.0
2017-02-27 00:27:11 +03:00
added new symbol
2018-02-11 20:55:32 +03:00
FLAG19 = 19
* Remove qualified naming in symbols
2015-10-10 14:11:38 +03:00
FLAG20
FLAG21
FLAG22
FLAG23
FLAG24
FLAG25
FLAG26
FLAG27
FLAG28
FLAG29
FLAG30
FLAG31
FLAG32
FLAG33
FLAG34
FLAG35
FLAG36
FLAG37
FLAG38
FLAG39
FLAG40
FLAG41
FLAG42
FLAG43
FLAG44
FLAG45
FLAG46
FLAG47
FLAG48
FLAG49
FLAG50
FLAG51
FLAG52
FLAG53
FLAG54
FLAG55
FLAG56
FLAG57
FLAG58
FLAG59
FLAG60
FLAG61
FLAG62
FLAG63
* Enumerate all symbols in one file
2015-10-06 16:41:17 +03:00
* Remove qualified naming in symbols
2015-10-10 14:11:38 +03:00
ID
ORTH
LOWER
NORM
SHAPE
PREFIX
SUFFIX
* Enumerate all symbols in one file
2015-10-06 16:41:17 +03:00
* Remove qualified naming in symbols
2015-10-10 14:11:38 +03:00
LENGTH
CLUSTER
LEMMA
POS
TAG
DEP
ENT_IOB
ENT_TYPE
HEAD
Add SENT_START attribute, for custom sentence boundary detection
2016-05-05 13:11:57 +03:00
SENT_START
* Remove qualified naming in symbols
2015-10-10 14:11:38 +03:00
SPACY
PROB
Revert "Merge branch 'develop' of https://github.com/explosion/spaCy into develop" This reverts commit c9ba3d3c2dc7067cf8bd55f878cec45a8c6d73d4, reversing changes made to 92c26a35d425d4e8ca1b805ea776ea10f5ded3df.
2018-03-27 20:23:02 +03:00
LANG
* Enumerate all symbols in one file
2015-10-06 16:41:17 +03:00
* Remove qualified naming in symbols
2015-10-10 14:11:38 +03:00
ADJ
ADP
ADV
AUX
CONJ
ci: add cython linter (#12694) * chore: add cython-linter dev dependency * fix: lexeme.pyx * fix: morphology.pxd * fix: tokenizer.pxd * fix: vocab.pxd * fix: morphology.pxd (line length) * ci: add cython-lint * ci: fix cython-lint call * Fix kb/candidate.pyx. * Fix kb/kb.pyx. * Fix kb/kb_in_memory.pyx. * Fix kb. * Fix training/ partially. * Fix training/. Ignore trailing whitespaces and too long lines. * Fix ml/. * Fix matcher/. * Fix pipeline/. * Fix tokens/. * Fix build errors. Fix vocab.pyx. * Fix cython-lint install and run. * Fix lexeme.pyx, parts_of_speech.pxd, vectors.pyx. Temporarily disable cython-lint execution. * Fix attrs.pyx, lexeme.pyx, symbols.pxd, isort issues. * Make cython-lint install conditional. Fix tokenizer.pyx. * Fix remaining files. Reenable cython-lint check. * Readded parentheses. * Fix test_build_dependencies(). * Add explanatory comment to cython-lint execution. --------- Co-authored-by: Raphael Mitsch <r.mitsch@outlook.com>
2023-07-19 13:03:31 +03:00
CCONJ # U20
* Remove qualified naming in symbols
2015-10-10 14:11:38 +03:00
DET
INTJ
NOUN
NUM
PART
PRON
PROPN
PUNCT
SCONJ
SYM
VERB
X
EOL
SPACE
* Enumerate all symbols in one file
2015-10-06 16:41:17 +03:00
Modify morphology to support arbitrary features (#4932) * Restructure tag maps for MorphAnalysis changes Prepare tag maps for upcoming MorphAnalysis changes that allow arbritrary features. * Use default tag map rather than duplicating for ca / uk / vi * Import tag map into defaults for ga * Modify tag maps so all morphological fields and features are strings * Move features from `"Other"` to the top level * Rewrite tuples as strings separated by `","` * Rewrite morph symbols for fr lemmatizer as strings * Export MorphAnalysis under spacy.tokens * Modify morphology to support arbitrary features Modify `Morphology` and `MorphAnalysis` so that arbitrary features are supported. * Modify `MorphAnalysisC` so that it can support arbitrary features and multiple values per field. `MorphAnalysisC` is redesigned to contain: * key: hash of UD FEATS string of morphological features * array of `MorphFeatureC` structs that each contain a hash of `Field` and `Field=Value` for a given morphological feature, which makes it possible to: * find features by field * represent multiple values for a given field * `get_field()` is renamed to `get_by_field()` and is no longer `nogil`. Instead a new helper function `get_n_by_field()` is `nogil` and returns `n` features by field. * `MorphAnalysis.get()` returns all possible values for a field as a list of individual features such as `["Tense=Pres", "Tense=Past"]`. * `MorphAnalysis`'s `str()` and `repr()` are the UD FEATS string. * `Morphology.feats_to_dict()` converts a UD FEATS string to a dict where: * Each field has one entry in the dict * Multiple values remain separated by a separator in the value string * `Token.morph_` returns the UD FEATS string and you can set `Token.morph_` with a UD FEATS string or with a tag map dict. * Modify get_by_field to use np.ndarray Modify `get_by_field()` to use np.ndarray. Remove `max_results` from `get_n_by_field()` and always iterate over all the fields. * Rewrite without MorphFeatureC * Add shortcut for existing feats strings as keys Add shortcut for existing feats strings as keys in `Morphology.add()`. * Check for '_' as empty analysis when adding morphs * Extend helper converters in Morphology Add and extend helper converters that convert and normalize between: * UD FEATS strings (`"Case=dat,gen|Number=sing"`) * per-field dict of feats (`{"Case": "dat,gen", "Number": "sing"}`) * list of individual features (`["Case=dat", "Case=gen", "Number=sing"]`) All converters sort fields and values where applicable.
2020-01-24 00:01:54 +03:00
DEPRECATED001
DEPRECATED002
DEPRECATED003
DEPRECATED004
DEPRECATED005
DEPRECATED006
DEPRECATED007
DEPRECATED008
DEPRECATED009
DEPRECATED010
DEPRECATED011
DEPRECATED012
DEPRECATED013
DEPRECATED014
DEPRECATED015
DEPRECATED016
DEPRECATED017
DEPRECATED018
DEPRECATED019
DEPRECATED020
DEPRECATED021
DEPRECATED022
DEPRECATED023
DEPRECATED024
DEPRECATED025
DEPRECATED026
DEPRECATED027
DEPRECATED028
DEPRECATED029
DEPRECATED030
DEPRECATED031
DEPRECATED032
DEPRECATED033
DEPRECATED034
DEPRECATED035
DEPRECATED036
DEPRECATED037
DEPRECATED038
DEPRECATED039
DEPRECATED040
DEPRECATED041
DEPRECATED042
DEPRECATED043
DEPRECATED044
DEPRECATED045
DEPRECATED046
DEPRECATED047
DEPRECATED048
DEPRECATED049
DEPRECATED050
DEPRECATED051
DEPRECATED052
DEPRECATED053
DEPRECATED054
DEPRECATED055
DEPRECATED056
DEPRECATED057
DEPRECATED058
DEPRECATED059
DEPRECATED060
DEPRECATED061
DEPRECATED062
DEPRECATED063
DEPRECATED064
DEPRECATED065
DEPRECATED066
DEPRECATED067
DEPRECATED068
DEPRECATED069
DEPRECATED070
DEPRECATED071
DEPRECATED072
DEPRECATED073
DEPRECATED074
DEPRECATED075
DEPRECATED076
DEPRECATED077
DEPRECATED078
DEPRECATED079
DEPRECATED080
DEPRECATED081
DEPRECATED082
DEPRECATED083
DEPRECATED084
DEPRECATED085
DEPRECATED086
DEPRECATED087
DEPRECATED088
DEPRECATED089
DEPRECATED090
DEPRECATED091
DEPRECATED092
DEPRECATED093
DEPRECATED094
DEPRECATED095
DEPRECATED096
DEPRECATED097
DEPRECATED098
DEPRECATED099
DEPRECATED100
DEPRECATED101
DEPRECATED102
DEPRECATED103
DEPRECATED104
DEPRECATED105
DEPRECATED106
DEPRECATED107
DEPRECATED108
DEPRECATED109
DEPRECATED110
DEPRECATED111
DEPRECATED112
DEPRECATED113
DEPRECATED114
DEPRECATED115
DEPRECATED116
DEPRECATED117
DEPRECATED118
DEPRECATED119
DEPRECATED120
DEPRECATED121
DEPRECATED122
DEPRECATED123
DEPRECATED124
DEPRECATED125
DEPRECATED126
DEPRECATED127
DEPRECATED128
DEPRECATED129
DEPRECATED130
DEPRECATED131
DEPRECATED132
DEPRECATED133
DEPRECATED134
DEPRECATED135
DEPRECATED136
DEPRECATED137
DEPRECATED138
DEPRECATED139
DEPRECATED140
DEPRECATED141
DEPRECATED142
DEPRECATED143
DEPRECATED144
DEPRECATED145
DEPRECATED146
DEPRECATED147
DEPRECATED148
DEPRECATED149
DEPRECATED150
DEPRECATED151
DEPRECATED152
DEPRECATED153
DEPRECATED154
DEPRECATED155
DEPRECATED156
DEPRECATED157
DEPRECATED158
DEPRECATED159
DEPRECATED160
DEPRECATED161
DEPRECATED162
DEPRECATED163
DEPRECATED164
DEPRECATED165
DEPRECATED166
DEPRECATED167
DEPRECATED168
DEPRECATED169
DEPRECATED170
DEPRECATED171
DEPRECATED172
DEPRECATED173
DEPRECATED174
DEPRECATED175
DEPRECATED176
DEPRECATED177
DEPRECATED178
DEPRECATED179
DEPRECATED180
DEPRECATED181
DEPRECATED182
DEPRECATED183
DEPRECATED184
DEPRECATED185
DEPRECATED186
DEPRECATED187
DEPRECATED188
DEPRECATED189
DEPRECATED190
DEPRECATED191
DEPRECATED192
DEPRECATED193
DEPRECATED194
DEPRECATED195
DEPRECATED196
DEPRECATED197
DEPRECATED198
DEPRECATED199
DEPRECATED200
DEPRECATED201
DEPRECATED202
DEPRECATED203
DEPRECATED204
DEPRECATED205
DEPRECATED206
DEPRECATED207
DEPRECATED208
DEPRECATED209
DEPRECATED210
DEPRECATED211
DEPRECATED212
DEPRECATED213
DEPRECATED214
DEPRECATED215
DEPRECATED216
DEPRECATED217
DEPRECATED218
DEPRECATED219
DEPRECATED220
DEPRECATED221
DEPRECATED222
DEPRECATED223
DEPRECATED224
DEPRECATED225
DEPRECATED226
DEPRECATED227
DEPRECATED228
DEPRECATED229
DEPRECATED230
DEPRECATED231
DEPRECATED232
DEPRECATED233
DEPRECATED234
DEPRECATED235
DEPRECATED236
DEPRECATED237
DEPRECATED238
DEPRECATED239
DEPRECATED240
DEPRECATED241
DEPRECATED242
DEPRECATED243
DEPRECATED244
DEPRECATED245
DEPRECATED246
DEPRECATED247
DEPRECATED248
DEPRECATED249
DEPRECATED250
DEPRECATED251
DEPRECATED252
DEPRECATED253
DEPRECATED254
DEPRECATED255
DEPRECATED256
DEPRECATED257
DEPRECATED258
DEPRECATED259
DEPRECATED260
DEPRECATED261
DEPRECATED262
DEPRECATED263
DEPRECATED264
DEPRECATED265
DEPRECATED266
DEPRECATED267
DEPRECATED268
DEPRECATED269
DEPRECATED270
DEPRECATED271
DEPRECATED272
DEPRECATED273
DEPRECATED274
DEPRECATED275
DEPRECATED276
* Enumerate all symbols in one file
2015-10-06 16:41:17 +03:00
* Use lower case strings for dependency label names in symbols enum
2015-10-10 14:59:14 +03:00
PERSON
NORP
FACILITY
ORG
GPE
LOC
PRODUCT
EVENT
WORK_OF_ART
LANGUAGE
Revert "Merge branch 'develop' of https://github.com/explosion/spaCy into develop" This reverts commit c9ba3d3c2dc7067cf8bd55f878cec45a8c6d73d4, reversing changes made to 92c26a35d425d4e8ca1b805ea776ea10f5ded3df.
2018-03-27 20:23:02 +03:00
LAW
* Enumerate all symbols in one file
2015-10-06 16:41:17 +03:00
* Use lower case strings for dependency label names in symbols enum
2015-10-10 14:59:14 +03:00
DATE
TIME
PERCENT
MONEY
QUANTITY
ORDINAL
CARDINAL
* Enumerate all symbols in one file
2015-10-06 16:41:17 +03:00
* Use lower case strings for dependency label names in symbols enum
2015-10-10 14:59:14 +03:00
acomp
advcl
advmod
agent
amod
appos
attr
aux
auxpass
cc
ccomp
complm
conj
ci: add cython linter (#12694) * chore: add cython-linter dev dependency * fix: lexeme.pyx * fix: morphology.pxd * fix: tokenizer.pxd * fix: vocab.pxd * fix: morphology.pxd (line length) * ci: add cython-lint * ci: fix cython-lint call * Fix kb/candidate.pyx. * Fix kb/kb.pyx. * Fix kb/kb_in_memory.pyx. * Fix kb. * Fix training/ partially. * Fix training/. Ignore trailing whitespaces and too long lines. * Fix ml/. * Fix matcher/. * Fix pipeline/. * Fix tokens/. * Fix build errors. Fix vocab.pyx. * Fix cython-lint install and run. * Fix lexeme.pyx, parts_of_speech.pxd, vectors.pyx. Temporarily disable cython-lint execution. * Fix attrs.pyx, lexeme.pyx, symbols.pxd, isort issues. * Make cython-lint install conditional. Fix tokenizer.pyx. * Fix remaining files. Reenable cython-lint check. * Readded parentheses. * Fix test_build_dependencies(). * Add explanatory comment to cython-lint execution. --------- Co-authored-by: Raphael Mitsch <r.mitsch@outlook.com>
2023-07-19 13:03:31 +03:00
cop # U20
* Use lower case strings for dependency label names in symbols enum
2015-10-10 14:59:14 +03:00
csubj
csubjpass
dep
det
dobj
expl
hmod
hyph
infmod
intj
iobj
mark
meta
neg
nmod
nn
npadvmod
nsubj
nsubjpass
num
number
oprd
ci: add cython linter (#12694) * chore: add cython-linter dev dependency * fix: lexeme.pyx * fix: morphology.pxd * fix: tokenizer.pxd * fix: vocab.pxd * fix: morphology.pxd (line length) * ci: add cython-lint * ci: fix cython-lint call * Fix kb/candidate.pyx. * Fix kb/kb.pyx. * Fix kb/kb_in_memory.pyx. * Fix kb. * Fix training/ partially. * Fix training/. Ignore trailing whitespaces and too long lines. * Fix ml/. * Fix matcher/. * Fix pipeline/. * Fix tokens/. * Fix build errors. Fix vocab.pyx. * Fix cython-lint install and run. * Fix lexeme.pyx, parts_of_speech.pxd, vectors.pyx. Temporarily disable cython-lint execution. * Fix attrs.pyx, lexeme.pyx, symbols.pxd, isort issues. * Make cython-lint install conditional. Fix tokenizer.pyx. * Fix remaining files. Reenable cython-lint check. * Readded parentheses. * Fix test_build_dependencies(). * Add explanatory comment to cython-lint execution. --------- Co-authored-by: Raphael Mitsch <r.mitsch@outlook.com>
2023-07-19 13:03:31 +03:00
obj # U20
obl # U20
* Use lower case strings for dependency label names in symbols enum
2015-10-10 14:59:14 +03:00
parataxis
partmod
pcomp
pobj
poss
possessive
preconj
prep
prt
punct
quantmod
Revert "Merge branch 'develop' of https://github.com/explosion/spaCy into develop" This reverts commit c9ba3d3c2dc7067cf8bd55f878cec45a8c6d73d4, reversing changes made to 92c26a35d425d4e8ca1b805ea776ea10f5ded3df.
2018-03-27 20:23:02 +03:00
relcl
* Use lower case strings for dependency label names in symbols enum
2015-10-10 14:59:14 +03:00
rcmod
root
xcomp
* Add placeholders for the new flags in attrs and symbols
2016-02-04 17:49:45 +03:00
Update symbols and document missing token attributes (see #1439)
2017-10-20 14:08:44 +03:00
acl
Fix symbol alignment
2019-07-12 18:48:16 +03:00
ENT_KB_ID
Add MORPH attr, add support in retokenizer (#4947) * Add MORPH attr / symbol for token attrs * Update retokenizer for MORPH
2020-01-29 19:45:46 +03:00
MORPH
serialize ENT_ID (#4852) * expand serialization test for custom token attribute * add failing test for issue 4849 * define ENT_ID as attr and use in doc serialization * fix few typos
2020-01-06 16:57:34 +03:00
ENT_ID
make idx available via to_array (#5030)
2020-02-22 16:13:06 +03:00
Add is_sent_end token property (#5375) Reconstruction of the original PR #4697 by @MiniLau. Removes unused `SENT_END` symbol and `IS_SENT_END` from `Matcher` schema because the Matcher is only going to be able to support `IS_SENT_START`.
2020-04-29 13:53:16 +03:00
IDX
Add _ as a symbol (#6153) * Add _ to StringStore in Morphology * Add _ as a symbol Add `_` as a symbol instead of adding to the `StringStore`.
2020-09-27 23:20:14 +03:00
_
Reference in New Issue Copy Permalink