spaCy/spacy/structs.pxd

114 lines
2.4 KiB
Cython
Raw Normal View History

from libc.stdint cimport int32_t, int64_t, uint8_t, uint32_t, uint64_t
from libcpp.unordered_map cimport unordered_map
from libcpp.unordered_set cimport unordered_set
from libcpp.vector cimport vector
from .parts_of_speech cimport univ_pos_t
from .typedefs cimport attr_t, flags_t, hash_t
cdef struct LexemeC:
flags_t flags
2015-04-19 11:31:31 +03:00
attr_t lang
attr_t id
attr_t length
2015-01-22 18:08:25 +03:00
attr_t orth
attr_t lower
attr_t norm
attr_t shape
attr_t prefix
attr_t suffix
2015-04-19 11:31:31 +03:00
cdef struct SpanC:
hash_t id
2015-03-09 08:46:22 +03:00
int start
int end
int start_char
int end_char
2017-05-28 15:06:40 +03:00
attr_t label
attr_t kb_id
2015-03-09 08:46:22 +03:00
cdef struct TokenC:
const LexemeC* lex
uint64_t morph
univ_pos_t pos
2015-07-13 20:48:07 +03:00
bint spacy
2017-05-28 15:06:40 +03:00
attr_t tag
int idx
attr_t lemma
attr_t norm
int head
attr_t dep
uint32_t l_kids
uint32_t r_kids
uint32_t l_edge
uint32_t r_edge
int sent_start
int ent_iob
2017-05-28 15:06:40 +03:00
attr_t ent_type # TODO: Is there a better way to do this? Multiple sources of truth..
2019-03-14 17:48:40 +03:00
attr_t ent_kb_id
hash_t ent_id
2018-09-25 00:58:08 +03:00
2019-03-07 16:03:07 +03:00
cdef struct MorphAnalysisC:
Modify morphology to support arbitrary features (#4932) * Restructure tag maps for MorphAnalysis changes Prepare tag maps for upcoming MorphAnalysis changes that allow arbritrary features. * Use default tag map rather than duplicating for ca / uk / vi * Import tag map into defaults for ga * Modify tag maps so all morphological fields and features are strings * Move features from `"Other"` to the top level * Rewrite tuples as strings separated by `","` * Rewrite morph symbols for fr lemmatizer as strings * Export MorphAnalysis under spacy.tokens * Modify morphology to support arbitrary features Modify `Morphology` and `MorphAnalysis` so that arbitrary features are supported. * Modify `MorphAnalysisC` so that it can support arbitrary features and multiple values per field. `MorphAnalysisC` is redesigned to contain: * key: hash of UD FEATS string of morphological features * array of `MorphFeatureC` structs that each contain a hash of `Field` and `Field=Value` for a given morphological feature, which makes it possible to: * find features by field * represent multiple values for a given field * `get_field()` is renamed to `get_by_field()` and is no longer `nogil`. Instead a new helper function `get_n_by_field()` is `nogil` and returns `n` features by field. * `MorphAnalysis.get()` returns all possible values for a field as a list of individual features such as `["Tense=Pres", "Tense=Past"]`. * `MorphAnalysis`'s `str()` and `repr()` are the UD FEATS string. * `Morphology.feats_to_dict()` converts a UD FEATS string to a dict where: * Each field has one entry in the dict * Multiple values remain separated by a separator in the value string * `Token.morph_` returns the UD FEATS string and you can set `Token.morph_` with a UD FEATS string or with a tag map dict. * Modify get_by_field to use np.ndarray Modify `get_by_field()` to use np.ndarray. Remove `max_results` from `get_n_by_field()` and always iterate over all the fields. * Rewrite without MorphFeatureC * Add shortcut for existing feats strings as keys Add shortcut for existing feats strings as keys in `Morphology.add()`. * Check for '_' as empty analysis when adding morphs * Extend helper converters in Morphology Add and extend helper converters that convert and normalize between: * UD FEATS strings (`"Case=dat,gen|Number=sing"`) * per-field dict of feats (`{"Case": "dat,gen", "Number": "sing"}`) * list of individual features (`["Case=dat", "Case=gen", "Number=sing"]`) All converters sort fields and values where applicable.
2020-01-24 00:01:54 +03:00
hash_t key
2019-03-08 02:08:57 +03:00
int length
Modify morphology to support arbitrary features (#4932) * Restructure tag maps for MorphAnalysis changes Prepare tag maps for upcoming MorphAnalysis changes that allow arbritrary features. * Use default tag map rather than duplicating for ca / uk / vi * Import tag map into defaults for ga * Modify tag maps so all morphological fields and features are strings * Move features from `"Other"` to the top level * Rewrite tuples as strings separated by `","` * Rewrite morph symbols for fr lemmatizer as strings * Export MorphAnalysis under spacy.tokens * Modify morphology to support arbitrary features Modify `Morphology` and `MorphAnalysis` so that arbitrary features are supported. * Modify `MorphAnalysisC` so that it can support arbitrary features and multiple values per field. `MorphAnalysisC` is redesigned to contain: * key: hash of UD FEATS string of morphological features * array of `MorphFeatureC` structs that each contain a hash of `Field` and `Field=Value` for a given morphological feature, which makes it possible to: * find features by field * represent multiple values for a given field * `get_field()` is renamed to `get_by_field()` and is no longer `nogil`. Instead a new helper function `get_n_by_field()` is `nogil` and returns `n` features by field. * `MorphAnalysis.get()` returns all possible values for a field as a list of individual features such as `["Tense=Pres", "Tense=Past"]`. * `MorphAnalysis`'s `str()` and `repr()` are the UD FEATS string. * `Morphology.feats_to_dict()` converts a UD FEATS string to a dict where: * Each field has one entry in the dict * Multiple values remain separated by a separator in the value string * `Token.morph_` returns the UD FEATS string and you can set `Token.morph_` with a UD FEATS string or with a tag map dict. * Modify get_by_field to use np.ndarray Modify `get_by_field()` to use np.ndarray. Remove `max_results` from `get_n_by_field()` and always iterate over all the fields. * Rewrite without MorphFeatureC * Add shortcut for existing feats strings as keys Add shortcut for existing feats strings as keys in `Morphology.add()`. * Check for '_' as empty analysis when adding morphs * Extend helper converters in Morphology Add and extend helper converters that convert and normalize between: * UD FEATS strings (`"Case=dat,gen|Number=sing"`) * per-field dict of feats (`{"Case": "dat,gen", "Number": "sing"}`) * list of individual features (`["Case=dat", "Case=gen", "Number=sing"]`) All converters sort fields and values where applicable.
2020-01-24 00:01:54 +03:00
attr_t* fields
attr_t* features
2018-09-25 00:58:08 +03:00
# Internal struct, for storage and disambiguation of entities.
2019-06-26 16:55:26 +03:00
cdef struct KBEntryC:
# The hash of this entry's unique ID/name in the kB
hash_t entity_hash
# Allows retrieval of the entity vector, as an index into a vectors table of the KB.
# Can be expanded later to refer to multiple rows (compositional model to reduce storage footprint).
int32_t vector_index
# Allows retrieval of a struct of non-vector features.
# This is currently not implemented and set to -1 for the common case where there are no features.
int32_t feats_row
# log probability of entity, based on corpus frequency
2019-07-19 18:40:28 +03:00
float freq
# Each alias struct stores a list of Entry pointers with their prior probabilities
# for this specific mention/alias.
cdef struct AliasC:
# All entry candidates for this alias
vector[int64_t] entry_indices
# Prior probability P(entity|alias) - should sum up to (at most) 1.
vector[float] probs
cdef struct EdgeC:
hash_t label
int32_t head
int32_t tail
cdef struct GraphC:
vector[vector[int32_t]] nodes
vector[EdgeC] edges
vector[float] weights
vector[int] n_heads
vector[int] n_tails
vector[int] first_head
vector[int] first_tail
unordered_set[int]* roots
unordered_map[hash_t, int]* node_map
unordered_map[hash_t, int]* edge_map