Daniël de Kok efdbb722c5 Store activations in Docs when save_activations is enabled (#11002) ... * Store activations in Doc when `store_activations` is enabled This change adds the new `activations` attribute to `Doc`. This attribute can be used by trainable pipes to store their activations, probabilities, and guesses for downstream users. As an example, this change modifies the `tagger` and `senter` pipes to add an `store_activations` option. When this option is enabled, the probabilities and guesses are stored in `set_annotations`. * Change type of `store_activations` to `Union[bool, List[str]]` When the value is: - A bool: all activations are stored when set to `True`. - A List[str]: the activations named in the list are stored * Formatting fixes in Tagger * Support store_activations in spancat and morphologizer * Make Doc.activations type visible to MyPy * textcat/textcat_multilabel: add store_activations option * trainable_lemmatizer/entity_linker: add store_activations option * parser/ner: do not currently support returning activations * Extend tagger and senter tests So that they, like the other tests, also check that we get no activations if no activations were requested. * Document `Doc.activations` and `store_activations` in the relevant pipes * Start errors/warnings at higher numbers to avoid merge conflicts Between the master and v4 branches. * Add `store_activations` to docstrings. * Replace store_activations setter by set_store_activations method Setters that take a different type than what the getter returns are still problematic for MyPy. Replace the setter by a method, so that type inference works everywhere. * Use dict comprehension suggested by @svlandeg * Revert "Use dict comprehension suggested by @svlandeg" This reverts commit 6e7b958f70. * EntityLinker: add type annotations to _add_activations * _store_activations: make kwarg-only, remove doc_scores_lens arg * set_annotations: add type annotations * Apply suggestions from code review Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> * TextCat.predict: return dict * Make the `TrainablePipe.store_activations` property a bool This means that we can also bring back `store_activations` setter. * Remove `TrainablePipe.activations` We do not need to enumerate the activations anymore since `store_activations` is `bool`. * Add type annotations for activations in predict/set_annotations * Rename `TrainablePipe.store_activations` to `save_activations` * Error E1400 is not used anymore This error was used when activations were still `Union[bool, List[str]]`. * Change wording in API docs after store -> save change * docs: tag (save_)activations as new in spaCy 4.0 * Fix copied line in morphologizer activations test * Don't train in any test_save_activations test * Rename activations - "probs" -> "probabilities" - "guesses" -> "label_ids", except in the edit tree lemmatizer, where "guesses" -> "tree_ids". * Remove unused W400 warning. This warning was used when we still allowed the user to specify which activations to save. * Formatting fixes Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> * Replace "kb_ids" by a constant * spancat: replace a cast by an assertion * Fix EOF spacing * Fix comments in test_save_activations tests * Do not set RNG seed in activation saving tests * Revert "spancat: replace a cast by an assertion" This reverts commit 0bd5730d16. Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>		2022-09-13 09:51:12 +02:00
..
__init__.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_analysis.py	Simplify pipe analysis	2020-08-01 13:40:06 +02:00
test_annotates_on_update.py	Tidy up and auto-format	2021-07-18 15:44:56 +10:00
test_attributeruler.py	Refactor scoring methods to use registered functions (#8766)	2021-08-10 15:13:39 +02:00
test_edit_tree_lemmatizer.py	Store activations in Docs when save_activations is enabled (#11002)	2022-09-13 09:51:12 +02:00
test_entity_linker.py	Store activations in Docs when save_activations is enabled (#11002)	2022-09-13 09:51:12 +02:00
test_entity_ruler.py	Auto-format code with black (#10908)	2022-06-03 11:01:55 +02:00
test_functions.py	Add doc_cleaner component (#9659)	2021-11-23 15:33:33 +01:00
test_initialize.py	Test with default value	2020-09-29 17:00:40 +02:00
test_lemmatizer.py	Tidy up and auto-format	2021-07-18 15:44:56 +10:00
test_models.py	Make stable private modules public and adjust names (#11353)	2022-08-30 13:56:35 +02:00
test_morphologizer.py	Store activations in Docs when save_activations is enabled (#11002)	2022-09-13 09:51:12 +02:00
test_pipe_factories.py	Auto-format code with black (#10795)	2022-05-13 19:02:08 +02:00
test_pipe_methods.py	Allow string argument for disable/enable/exclude (#11406)	2022-08-31 09:02:34 +02:00
test_sentencizer.py	Refactor Docs.is_ flags (#6044)	2020-09-17 00:14:01 +02:00
test_senter.py	Store activations in Docs when save_activations is enabled (#11002)	2022-09-13 09:51:12 +02:00
test_span_ruler.py	Add SpanRuler component (#9880)	2022-06-02 13:12:53 +02:00
test_spancat.py	Store activations in Docs when save_activations is enabled (#11002)	2022-09-13 09:51:12 +02:00
test_tagger.py	Store activations in Docs when save_activations is enabled (#11002)	2022-09-13 09:51:12 +02:00
test_textcat.py	Store activations in Docs when save_activations is enabled (#11002)	2022-09-13 09:51:12 +02:00
test_tok2vec.py	Add test for old architectures (#10751)	2022-05-10 08:24:42 +02:00