spaCy/spacy/pipeline
Adriane Boyd e962784531
Add Lemmatizer and simplify related components (#5848)
* Add Lemmatizer and simplify related components

* Add `Lemmatizer` pipe with `lookup` and `rule` modes using the
`Lookups` tables.
* Reduce `Tagger` to a simple tagger that sets `Token.tag` (no pos or lemma)
* Reduce `Morphology` to only keep track of morph tags (no tag map, lemmatizer,
or morph rules)
* Remove lemmatizer from `Vocab`
* Adjust many many tests

Differences:

* No default lookup lemmas
* No special treatment of TAG in `from_array` and similar required
* Easier to modify labels in a `Tagger`
* No extra strings added from morphology / tag map

* Fix test

* Initial fix for Lemmatizer config/serialization

* Adjust init test to be more generic

* Adjust init test to force empty Lookups

* Add simple cache to rule-based lemmatizer

* Convert language-specific lemmatizers

Convert language-specific lemmatizers to component lemmatizers. Remove
previous lemmatizer class.

* Fix French and Polish lemmatizers

* Remove outdated UPOS conversions

* Update Russian lemmatizer init in tests

* Add minimal init/run tests for custom lemmatizers

* Add option to overwrite existing lemmas

* Update mode setting, lookup loading, and caching

* Make `mode` an immutable property
* Only enforce strict `load_lookups` for known supported modes
* Move caching into individual `_lemmatize` methods

* Implement strict when lang is not found in lookups

* Fix tables/lookups in make_lemmatizer

* Reallow provided lookups and allow for stricter checks

* Add lookups asset to all Lemmatizer pipe tests

* Rename lookups in lemmatizer init test

* Clean up merge

* Refactor lookup table loading

* Add helper from `load_lemmatizer_lookups` that loads required and
optional lookups tables based on settings provided by a config.

Additional slight refactor of lookups:

* Add `Lookups.set_table` to set a table from a provided `Table`
* Reorder class definitions to be able to specify type as `Table`

* Move registry assets into test methods

* Refactor lookups tables config

Use class methods within `Lemmatizer` to provide the config for
particular modes and to load the lookups from a config.

* Add pipe and score to lemmatizer

* Simplify Tagger.score

* Add missing import

* Clean up imports and auto-format

* Remove unused kwarg

* Tidy up and auto-format

* Update docstrings for Lemmatizer

Update docstrings for Lemmatizer.

Additionally modify `is_base_form` API to take `Token` instead of
individual features.

* Update docstrings

* Remove tag map values from Tagger.add_label

* Update API docs

* Fix relative link in Lemmatizer API docs
2020-08-07 15:27:13 +02:00
..
_parser_internals The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
__init__.py Add Lemmatizer and simplify related components (#5848) 2020-08-07 15:27:13 +02:00
attributeruler.py Add AttributeRuler API docs 2020-08-07 12:43:23 +02:00
dep_parser.pyx Fix default arguments in DependencyParser.score 2020-07-31 10:55:44 +02:00
entity_linker.py Use "raise ... from" in custom errors for better tracebacks 2020-08-05 23:53:21 +02:00
entityruler.py Update docs and consistency 2020-07-29 15:14:07 +02:00
functions.py Option for returning only greedy matches (#5771) 2020-07-29 11:04:43 +02:00
lemmatizer.py Add Lemmatizer and simplify related components (#5848) 2020-08-07 15:27:13 +02:00
morphologizer.pyx Use "raise ... from" in custom errors for better tracebacks 2020-08-05 23:53:21 +02:00
multitask.pyx Use "raise ... from" in custom errors for better tracebacks 2020-08-05 23:53:21 +02:00
ner.pyx The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
nn_parser.pyx The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
pipe.pxd The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
pipe.pyx Use "raise ... from" in custom errors for better tracebacks 2020-08-05 23:53:21 +02:00
sentencizer.pyx The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
senter.pyx Use "raise ... from" in custom errors for better tracebacks 2020-08-05 23:53:21 +02:00
simple_ner.py Update SimpleNER (#5878) 2020-08-05 14:43:29 +02:00
tagger.pyx Add Lemmatizer and simplify related components (#5848) 2020-08-07 15:27:13 +02:00
textcat.py Fix format 2020-08-06 19:30:53 +02:00
tok2vec.py The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
transition_parser.pxd The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
transition_parser.pyx Use "raise ... from" in custom errors for better tracebacks 2020-08-05 23:53:21 +02:00