* Add `TrainablePipe.{distill,get_teacher_student_loss}`
This change adds two methods:
- `TrainablePipe::distill` which performs a training step of a
student pipe on a teacher pipe, giving a batch of `Doc`s.
- `TrainablePipe::get_teacher_student_loss` computes the loss
of a student relative to the teacher.
The `distill` or `get_teacher_student_loss` methods are also implemented
in the tagger, edit tree lemmatizer, and parser pipes, to enable
distillation in those pipes and as an example for other pipes.
* Fix stray `Beam` import
* Fix incorrect import
* Apply suggestions from code review
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
* TrainablePipe.distill: use `Iterable[Example]`
* Add Pipe.is_distillable method
* Add `validate_distillation_examples`
This first calls `validate_examples` and then checks that the
student/teacher tokens are the same.
* Update distill documentation
* Add distill documentation for all pipes that support distillation
* Fix incorrect identifier
* Apply suggestions from code review
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
* Add comment to explain `is_distillable`
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
* Enable Cython<->Python bindings for `Pipe` and `TrainablePipe` methods
* `pipes_with_nvtx_range`: Skip hooking methods whose signature cannot be ascertained
When loading pipelines from a config file, the arguments passed to individual pipeline components is validated by `pydantic` during init. For this, the validation model attempts to parse the function signature of the component's c'tor/entry point so that it can check if all mandatory parameters are present in the config file.
When using the `models_and_pipes_with_nvtx_range` as a `after_pipeline_creation` callback, the methods of all pipeline components get replaced by a NVTX range wrapper **before** the above-mentioned validation takes place. This can be problematic for components that are implemented as Cython extension types - if the extension type is not compiled with Python bindings for its methods, they will have no signatures at runtime. This resulted in `pydantic` matching the *wrapper's* parameters with the those in the config and raising errors.
To avoid this, we now skip applying the wrapper to any (Cython) methods that do not have signatures.
* `TrainablePipe`: Add NVTX range decorator
* Annotate `TrainablePipe` subclasses with NVTX ranges
* Export function signature to allow introspection of args in tests
* Revert "Annotate `TrainablePipe` subclasses with NVTX ranges"
This reverts commit d8684f7372.
* Revert "Export function signature to allow introspection of args in tests"
This reverts commit f4405ca3ad.
* Revert "`TrainablePipe`: Add NVTX range decorator"
This reverts commit 26536eb6b8.
* Add `spacy.pipes_with_nvtx_range` pipeline callback
* Show warnings for all missing user-defined pipe functions that need to be annotated
Fix imports, typos
* Rename `DEFAULT_ANNOTATABLE_PIPE_METHODS` to `DEFAULT_NVTX_ANNOTATABLE_PIPE_METHODS`
Reorder import
* Walk model nodes directly whilst applying NVTX ranges
Ignore pipe method wrapper when applying range
* Add the spacy.models_with_nvtx_range.v1 callback
This callback recursively adds NVTX ranges to the Models in each pipe in
a pipeline.
* Fix create_models_with_nvtx_range type signature
* NVTX range: wrap models of all trainable pipes jointly
This avoids that (sub-)models that are shared between pipes get wrapped
twice.
* NVTX range callback: make color configurable
Add forward_color and backprop_color options to set the color for the
NVTX range.
* Move create_models_with_nvtx_range to spacy.ml
* Update create_models_with_nvtx_range for thinc changes
with_nvtx_range now updates an existing node, rather than returning a
wrapper node. So, we can simply walk over the nodes and update them.
* NVTX: use after_pipeline_creation in example
