spaCy/spacy/pipeline
Daniël de Kok efdbb722c5
Store activations in Docs when save_activations is enabled (#11002)
* Store activations in Doc when `store_activations` is enabled

This change adds the new `activations` attribute to `Doc`. This
attribute can be used by trainable pipes to store their activations,
probabilities, and guesses for downstream users.

As an example, this change modifies the `tagger` and `senter` pipes to
add an `store_activations` option. When this option is enabled, the
probabilities and guesses are stored in `set_annotations`.

* Change type of `store_activations` to `Union[bool, List[str]]`

When the value is:

- A bool: all activations are stored when set to `True`.
- A List[str]: the activations named in the list are stored

* Formatting fixes in Tagger

* Support store_activations in spancat and morphologizer

* Make Doc.activations type visible to MyPy

* textcat/textcat_multilabel: add store_activations option

* trainable_lemmatizer/entity_linker: add store_activations option

* parser/ner: do not currently support returning activations

* Extend tagger and senter tests

So that they, like the other tests, also check that we get no
activations if no activations were requested.

* Document `Doc.activations` and `store_activations` in the relevant pipes

* Start errors/warnings at higher numbers to avoid merge conflicts

Between the master and v4 branches.

* Add `store_activations` to docstrings.

* Replace store_activations setter by set_store_activations method

Setters that take a different type than what the getter returns are still
problematic for MyPy. Replace the setter by a method, so that type inference
works everywhere.

* Use dict comprehension suggested by @svlandeg

* Revert "Use dict comprehension suggested by @svlandeg"

This reverts commit 6e7b958f70.

* EntityLinker: add type annotations to _add_activations

* _store_activations: make kwarg-only, remove doc_scores_lens arg

* set_annotations: add type annotations

* Apply suggestions from code review

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* TextCat.predict: return dict

* Make the `TrainablePipe.store_activations` property a bool

This means that we can also bring back `store_activations` setter.

* Remove `TrainablePipe.activations`

We do not need to enumerate the activations anymore since `store_activations` is
`bool`.

* Add type annotations for activations in predict/set_annotations

* Rename `TrainablePipe.store_activations` to `save_activations`

* Error E1400 is not used anymore

This error was used when activations were still `Union[bool, List[str]]`.

* Change wording in API docs after store -> save change

* docs: tag (save_)activations as new in spaCy 4.0

* Fix copied line in morphologizer activations test

* Don't train in any test_save_activations test

* Rename activations

- "probs" -> "probabilities"
- "guesses" -> "label_ids", except in the edit tree lemmatizer, where
  "guesses" -> "tree_ids".

* Remove unused W400 warning.

This warning was used when we still allowed the user to specify
which activations to save.

* Formatting fixes

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Replace "kb_ids" by a constant

* spancat: replace a cast by an assertion

* Fix EOF spacing

* Fix comments in test_save_activations tests

* Do not set RNG seed in activation saving tests

* Revert "spancat: replace a cast by an assertion"

This reverts commit 0bd5730d16.

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
2022-09-13 09:51:12 +02:00
..
_edit_tree_internals Refactor error messages to remove hardcoded strings (#10729) 2022-05-02 13:38:46 +02:00
_parser_internals Merge remote-tracking branch 'upstream/master' into merge-master-v4-20220627-2 2022-06-27 17:45:45 +02:00
legacy NEL confidence threshold (#11016) 2022-07-04 17:05:21 +02:00
__init__.py Rename modules for consistency (#11286) 2022-08-10 11:44:05 +02:00
attribute_ruler.py Make stable private modules public and adjust names (#11353) 2022-08-30 13:56:35 +02:00
dep_parser.pyx account for NER labels with a hyphen in the name (#10960) 2022-06-17 20:02:37 +01:00
edit_tree_lemmatizer.py Store activations in Docs when save_activations is enabled (#11002) 2022-09-13 09:51:12 +02:00
entity_linker.py Store activations in Docs when save_activations is enabled (#11002) 2022-09-13 09:51:12 +02:00
entity_ruler.py Rename modules for consistency (#11286) 2022-08-10 11:44:05 +02:00
functions.py Add doc_cleaner component (#9659) 2021-11-23 15:33:33 +01:00
lemmatizer.py Merge remote-tracking branch 'upstream/master' into chore/update-develop-from-master-v3.2-1 2021-10-26 11:53:50 +02:00
morphologizer.pyx Store activations in Docs when save_activations is enabled (#11002) 2022-09-13 09:51:12 +02:00
multitask.pyx Replace negative rows with 0 in StaticVectors (#7674) 2021-04-22 18:04:15 +10:00
ner.pyx account for NER labels with a hyphen in the name (#10960) 2022-06-17 20:02:37 +01:00
pipe.pxd TrainablePipe (#6213) 2020-10-08 21:33:49 +02:00
pipe.pyi Add Pipe.hide_labels to omit labels from pipeline meta (#10175) 2022-02-05 17:59:24 +01:00
pipe.pyx fix typo + CI slow testing (#10835) 2022-06-02 00:10:16 +02:00
sentencizer.pyx Add overwrite settings for more components (#9050) 2021-09-30 15:35:55 +02:00
senter.pyx Store activations in Docs when save_activations is enabled (#11002) 2022-09-13 09:51:12 +02:00
span_ruler.py Add SpanRuler component (#9880) 2022-06-02 13:12:53 +02:00
spancat.py Store activations in Docs when save_activations is enabled (#11002) 2022-09-13 09:51:12 +02:00
tagger.pyx Store activations in Docs when save_activations is enabled (#11002) 2022-09-13 09:51:12 +02:00
textcat_multilabel.py Store activations in Docs when save_activations is enabled (#11002) 2022-09-13 09:51:12 +02:00
textcat.py Store activations in Docs when save_activations is enabled (#11002) 2022-09-13 09:51:12 +02:00
tok2vec.py Fix Tok2Vec for empty batches (#10324) 2022-02-21 10:22:36 +01:00
trainable_pipe.pxd Store activations in Docs when save_activations is enabled (#11002) 2022-09-13 09:51:12 +02:00
trainable_pipe.pyx Store activations in Docs when save_activations is enabled (#11002) 2022-09-13 09:51:12 +02:00
transition_parser.pxd precompute_hiddens/Parser: look up CPU ops once (v4) (#11068) 2022-07-29 15:12:19 +02:00
transition_parser.pyx precompute_hiddens/Parser: look up CPU ops once (v4) (#11068) 2022-07-29 15:12:19 +02:00