spaCy/spacy/tests/doc
adrianeboyd adc9745718 Modify morphology to support arbitrary features (#4932)
* Restructure tag maps for MorphAnalysis changes

Prepare tag maps for upcoming MorphAnalysis changes that allow
arbritrary features.

* Use default tag map rather than duplicating for ca / uk / vi

* Import tag map into defaults for ga

* Modify tag maps so all morphological fields and features are strings
  * Move features from `"Other"` to the top level
  * Rewrite tuples as strings separated by `","`

* Rewrite morph symbols for fr lemmatizer as strings

* Export MorphAnalysis under spacy.tokens

* Modify morphology to support arbitrary features

Modify `Morphology` and `MorphAnalysis` so that arbitrary features are
supported.

* Modify `MorphAnalysisC` so that it can support arbitrary features and
multiple values per field. `MorphAnalysisC` is redesigned to contain:
  * key: hash of UD FEATS string of morphological features
  * array of `MorphFeatureC` structs that each contain a hash of `Field`
and `Field=Value` for a given morphological feature, which makes it
possible to:
    * find features by field
    * represent multiple values for a given field

* `get_field()` is renamed to `get_by_field()` and is no longer `nogil`.
Instead a new helper function `get_n_by_field()` is `nogil` and returns
`n` features by field.

* `MorphAnalysis.get()` returns all possible values for a field as a
list of individual features such as `["Tense=Pres", "Tense=Past"]`.

* `MorphAnalysis`'s `str()` and `repr()` are the UD FEATS string.

* `Morphology.feats_to_dict()` converts a UD FEATS string to a dict
where:
  * Each field has one entry in the dict
  * Multiple values remain separated by a separator in the value string

* `Token.morph_` returns the UD FEATS string and you can set
`Token.morph_` with a UD FEATS string or with a tag map dict.

* Modify get_by_field to use np.ndarray

Modify `get_by_field()` to use np.ndarray. Remove `max_results` from
`get_n_by_field()` and always iterate over all the fields.

* Rewrite without MorphFeatureC

* Add shortcut for existing feats strings as keys

Add shortcut for existing feats strings as keys in `Morphology.add()`.

* Check for '_' as empty analysis when adding morphs

* Extend helper converters in Morphology

Add and extend helper converters that convert and normalize between:

* UD FEATS strings (`"Case=dat,gen|Number=sing"`)
* per-field dict of feats (`{"Case": "dat,gen", "Number": "sing"}`)
* list of individual features (`["Case=dat", "Case=gen",
"Number=sing"]`)

All converters sort fields and values where applicable.
2020-01-23 22:01:54 +01:00
..
__init__.py Revert #4334 2019-09-29 17:32:12 +02:00
test_add_entities.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00
test_array.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00
test_creation.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00
test_doc_api.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00
test_morphanalysis.py Modify morphology to support arbitrary features (#4932) 2020-01-23 22:01:54 +01:00
test_pickle_doc.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00
test_retokenize_merge.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00
test_retokenize_split.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00
test_span.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00
test_to_json.py Add better schemas and validation using Pydantic (#4831) 2019-12-25 12:39:49 +01:00
test_token_api.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00
test_underscore.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00