From 99d0412b6ebf1af8e5bd4755f2ec8d1947d8cef5 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Sun, 15 Nov 2020 18:35:56 +0100 Subject: [PATCH 1/7] add link to REL project --- website/docs/usage/layers-architectures.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index 641db02f5..481a60574 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -534,9 +534,14 @@ steps required: machine learning model that sets annotations on the [`Doc`](/api/doc) passing through the pipeline. - + +Run this example use-case by using our project template. It includes all the +code to create the ML model and the pipeline component from scratch. +It contains two config files to train the model: +one to run on CPU with a Tok2Vec layer, and one for the GPU using a transformer. +The project applies the relation extraction component to identify biomolecular +interactions, but you can easily swap in your own dataset for your experiments. + #### Step 1: Implementing the Model {#component-rel-model} @@ -924,6 +929,11 @@ def make_relation_extractor(nlp, name, model): return RelationExtractor(nlp.vocab, model, name) ``` - + +Run this example use-case by using our project template. It includes all the +code to create the ML model and the pipeline component from scratch. +It contains two config files to train the model: +one to run on CPU with a Tok2Vec layer, and one for the GPU using a transformer. +The project applies the relation extraction component to identify biomolecular +interactions, but you can easily swap in your own dataset for your experiments. + From 124f49feb6f44fdab65036f1413dba237ba14599 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Fri, 20 Nov 2020 15:25:20 +0100 Subject: [PATCH 2/7] update REL model code --- website/docs/usage/layers-architectures.md | 224 ++++++++++++++------- 1 file changed, 155 insertions(+), 69 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index 481a60574..60d6224dc 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -540,7 +540,8 @@ code to create the ML model and the pipeline component from scratch. It contains two config files to train the model: one to run on CPU with a Tok2Vec layer, and one for the GPU using a transformer. The project applies the relation extraction component to identify biomolecular -interactions, but you can easily swap in your own dataset for your experiments. +interactions, but you can easily swap in your own dataset for your experiments +in any other domain. #### Step 1: Implementing the Model {#component-rel-model} @@ -558,40 +559,17 @@ matrix** (~~Floats2d~~) of predictions: ```python ### Register the model architecture -@registry.architectures.register("rel_model.v1") +@spacy.registry.architectures.register("rel_model.v1") def create_relation_model(...) -> Model[List[Doc], Floats2d]: model = ... # 👈 model will go here return model ``` -The first layer in this model will typically be an -[embedding layer](/usage/embeddings-transformers) such as a -[`Tok2Vec`](/api/tok2vec) component or a [`Transformer`](/api/transformer). This -layer is assumed to be of type ~~Model[List[Doc], List[Floats2d]]~~ as it -transforms each **document into a list of tokens**, with each token being -represented by its embedding in the vector space. +We will adapt a **modular approach** to the definition of this relation model, and +define it as chaining to layers together: the first layer that generates an +instance tensor from a given set of documents, and the second layer that +transforms this tensor into a final tensor holding the predictions: -Next, we need a method that **generates pairs of entities** that we want to -classify as being related or not. As these candidate pairs are typically formed -within one document, this function takes a [`Doc`](/api/doc) as input and -outputs a `List` of `Span` tuples. For instance, a very straightforward -implementation would be to just take any two entities from the same document: - -```python -### Simple candiate generation -def get_candidates(doc: Doc) -> List[Tuple[Span, Span]]: - candidates = [] - for ent1 in doc.ents: - for ent2 in doc.ents: - candidates.append((ent1, ent2)) - return candidates -``` - -But we could also refine this further by **excluding relations** of an entity -with itself, and posing a **maximum distance** (in number of tokens) between two -entities. We register this function in the -[`@misc` registry](/api/top-level#registry) so we can refer to it from the -config, and easily swap it out for any other candidate generation function. > #### config.cfg (excerpt) > @@ -599,17 +577,151 @@ config, and easily swap it out for any other candidate generation function. > [model] > @architectures = "rel_model.v1" > -> [model.tok2vec] +> [model.create_instance_tensor] > # ... > -> [model.get_candidates] -> @misc = "rel_cand_generator.v1" -> max_length = 20 +> [model.classification_layer] +> ... > ``` ```python -### Extended candidate generation {highlight="1,2,7,8"} -@registry.misc.register("rel_cand_generator.v1") +### Implement the model architecture +@spacy.registry.architectures.register("rel_model.v1") +def create_relation_model( + create_instance_tensor: Model[List[Doc], Floats2d], + classification_layer: Model[Floats2d, Floats2d], +) -> Model[List[Doc], Floats2d]: + model = chain(create_instance_tensor, classification_layer) + return model +``` + +The `classification_layer` could be something simple like a Linear layer +followed by a logistic activation function: + + +> #### config.cfg (excerpt) +> +> ```ini +> [model.classification_layer] +> @architectures = "rel_classification_layer.v1" +> nI = null +> nO = null +> ``` + +```python +### Implement the classification layer +@spacy.registry.architectures.register("rel_classification_layer.v1") +def create_classification_layer( + nO: int = None, nI: int = None +) -> Model[Floats2d, Floats2d]: + return chain(Linear(nO=nO, nI=nI), Logistic()) +``` + +The first layer that **creates the instance tensor** can be defined +by implementing a +[custom forward function](https://thinc.ai/docs/usage-models#weights-layers-forward) +with an appropriate backpropagation callback. We also define an +[initialization method](https://thinc.ai/docs/usage-models#weights-layers-init) +that ensures that the layer is properly set up for training. + +```python +### Implement the custom forward function +def instance_forward( + model: Model[List[Doc], Floats2d], + docs: List[Doc], + is_train: bool +) -> Tuple[Floats2d, Callable]: + ... + tok2vec = model.get_ref("tok2vec") + tokvecs, bp_tokvecs = tok2vec(docs, is_train) + relations = ... + + def backprop(d_relations: Floats2d) -> List[Doc]: + d_tokvecs = ... + return bp_tokvecs(d_tokvecs) + + return relations, backprop + + +### Implement the custom initialization method +def instance_init( + model: Model, + X: List[Doc] = None, + Y: Floats2d = None +) -> Model: + tok2vec = model.get_ref("tok2vec") + tok2vec.initialize(X) + return model + + +### Implement the layer that creates the instance tensor +@spacy.registry.architectures.register("rel_instance_tensor.v1") +def create_tensors( + tok2vec: Model[List[Doc], List[Floats2d]], + pooling: Model[Ragged, Floats2d], + get_instances: Callable[[Doc], List[Tuple[Span, Span]]], +) -> Model[List[Doc], Floats2d]: + + return Model( + "instance_tensors", + instance_forward, + layers=[tok2vec, pooling], + refs={"tok2vec": tok2vec, "pooling": pooling}, + attrs={"get_instances": get_instances}, + init=instance_init, + ) +``` + +> #### config.cfg (excerpt) +> +> ```ini +> [model.create_instance_tensor] +> @architectures = "rel_instance_tensor.v1" +> +> [model.create_instance_tensor.tok2vec] +> @architectures = "spacy.HashEmbedCNN.v1" +> ... +> +> [model.create_instance_tensor.pooling] +> @layers = "reduce_mean.v1" +> +> [model.create_instance_tensor.get_instances] +> ... +> ` + +This custom layer uses an +**[embedding layer](/usage/embeddings-transformers)** such as a +[`Tok2Vec`](/api/tok2vec) component or a [`Transformer`](/api/transformer). This +layer is assumed to be of type ~~Model[List[Doc], List[Floats2d]]~~ as it +transforms each **document into a list of tokens**, with each token being +represented by its embedding in the vector space. + +The **`pooling`** layer will be applied to summarize the token vectors into entity +vectors, as named entities (represented by `Span` objects) can consist of one +or multiple tokens. For instance, the pooling layer could resort to calculating +the average of all token vectors in an entity. Thinc provides several +[built-in pooling operators](https://thinc.ai/docs/api-layers#reduction-ops) for +this purpose. + +> #### config.cfg (excerpt) +> +> ```ini +> +> [model.create_instance_tensor.get_instances] +> @misc = "rel_instance_generator.v1" +> max_length = 100 +> ``` + +Finally, we need a `get_instances` method that **generates pairs of entities** +that we want to classify as being related or not. As these candidate pairs are typically formed +within one document, this function takes a [`Doc`](/api/doc) as input and +outputs a `List` of `Span` tuples. For instance, this +implementation takes any two entities from the same document, as long as they +are within a **maximum distance** (in number of tokens) of eachother: + +```python +### Simple candiate generation +@spacy.registry.misc.register("rel_instance_generator.v1") def create_candidate_indices(max_length: int) -> Callable[[Doc], List[Tuple[Span, Span]]]: def get_candidates(doc: "Doc") -> List[Tuple[Span, Span]]: candidates = [] @@ -621,46 +733,19 @@ def create_candidate_indices(max_length: int) -> Callable[[Doc], List[Tuple[Span return candidates return get_candidates ``` +This function in added to the +[`@misc` registry](/api/top-level#registry) so we can refer to it from the +config, and easily swap it out for any other candidate generation function. -Finally, we require a method that transforms the candidate entity pairs into a -2D tensor using the specified [`Tok2Vec`](/api/tok2vec) or -[`Transformer`](/api/transformer). The resulting ~~Floats2~~ object will then be -processed by a final `output_layer` of the network. Putting all this together, -we can define our relation model in a config file as such: - -```ini -### config.cfg -[model] -@architectures = "rel_model.v1" -# ... - -[model.tok2vec] -# ... - -[model.get_candidates] -@misc = "rel_cand_generator.v1" -max_length = 20 - -[model.create_candidate_tensor] -@misc = "rel_cand_tensor.v1" - -[model.output_layer] -@architectures = "rel_output_layer.v1" -# ... -``` - - - When creating this model, we store the custom functions as [attributes](https://thinc.ai/docs/api-model#properties) and the sublayers as references, so we can access them easily: ```python -tok2vec_layer = model.get_ref("tok2vec") -output_layer = model.get_ref("output_layer") -create_candidate_tensor = model.attrs["create_candidate_tensor"] -get_candidates = model.attrs["get_candidates"] +pooling = model.get_ref("pooling") +tok2vec = model.get_ref("tok2vec") +get_instances = model.attrs["get_instances"] ``` #### Step 2: Implementing the pipeline component {#component-rel-pipe} @@ -935,5 +1020,6 @@ code to create the ML model and the pipeline component from scratch. It contains two config files to train the model: one to run on CPU with a Tok2Vec layer, and one for the GPU using a transformer. The project applies the relation extraction component to identify biomolecular -interactions, but you can easily swap in your own dataset for your experiments. +interactions, but you can easily swap in your own dataset for your experiments +in any other domain. From 4a3e611abc1d183da22c5a6c70852d2d21fee7b5 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Fri, 20 Nov 2020 15:55:05 +0100 Subject: [PATCH 3/7] small fixes and formatting --- website/docs/usage/layers-architectures.md | 214 +++++++++++---------- 1 file changed, 110 insertions(+), 104 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index 60d6224dc..2df4745db 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -502,7 +502,7 @@ with Model.define_operators({">>": chain}): ## Create new trainable components {#components} -In addition to [swapping out](#swap-architectures) default models in built-in +In addition to [swapping out](#swap-architectures) layers in existing components, you can also implement an entirely new, [trainable](/usage/processing-pipelines#trainable-components) pipeline component from scratch. This can be done by creating a new class inheriting from @@ -523,25 +523,27 @@ overview of the `TrainablePipe` methods used by This section outlines an example use-case of implementing a **novel relation extraction component** from scratch. We'll implement a binary relation extraction method that determines whether or not **two entities** in a document -are related, and if so, what type of relation. We'll allow multiple types of -relations between two such entities (multi-label setting). There are two major -steps required: +are related, and if so, what type of relation connects them. We allow multiple +types of relations between two such entities (a multi-label setting). There are +two major steps required: 1. Implement a [machine learning model](#component-rel-model) specific to this - task. It will have to extract candidates from a [`Doc`](/api/doc) and predict - a relation for the available candidate pairs. -2. Implement a custom [pipeline component](#component-rel-pipe) powered by the - machine learning model that sets annotations on the [`Doc`](/api/doc) passing - through the pipeline. + task. It will have to extract candidate relation instances from a + [`Doc`](/api/doc) and predict the corresponding scores for each relation + label. +2. Implement a custom [pipeline component](#component-rel-pipe) - powered by the + machine learning model from step 1 - that translates the predicted scores + into annotations that are stored on the [`Doc`](/api/doc) objects as they + pass through the `nlp` pipeline. Run this example use-case by using our project template. It includes all the code to create the ML model and the pipeline component from scratch. -It contains two config files to train the model: +It also contains two config files to train the model: one to run on CPU with a Tok2Vec layer, and one for the GPU using a transformer. The project applies the relation extraction component to identify biomolecular -interactions, but you can easily swap in your own dataset for your experiments -in any other domain. +interactions in a sample dataset, but you can easily swap in your own dataset +for your experiments in any other domain. #### Step 1: Implementing the Model {#component-rel-model} @@ -558,18 +560,17 @@ matrix** (~~Floats2d~~) of predictions: > for details. ```python -### Register the model architecture +### The model architecture @spacy.registry.architectures.register("rel_model.v1") def create_relation_model(...) -> Model[List[Doc], Floats2d]: model = ... # 👈 model will go here return model ``` -We will adapt a **modular approach** to the definition of this relation model, and -define it as chaining to layers together: the first layer that generates an -instance tensor from a given set of documents, and the second layer that -transforms this tensor into a final tensor holding the predictions: - +We will adapt a **modular approach** to the definition of this relation model, +and define it as chaining two layers together: the first layer that generates an +instance tensor from a given set of documents, and the second layer that +transforms the instance tensor into a final tensor holding the predictions. > #### config.cfg (excerpt) > @@ -581,11 +582,11 @@ transforms this tensor into a final tensor holding the predictions: > # ... > > [model.classification_layer] -> ... +> # ... > ``` ```python -### Implement the model architecture +### The model architecture @spacy.registry.architectures.register("rel_model.v1") def create_relation_model( create_instance_tensor: Model[List[Doc], Floats2d], @@ -595,9 +596,8 @@ def create_relation_model( return model ``` -The `classification_layer` could be something simple like a Linear layer -followed by a logistic activation function: - +The `classification_layer` could be something like a Linear layer followed by a +logistic activation function: > #### config.cfg (excerpt) > @@ -609,7 +609,7 @@ followed by a logistic activation function: > ``` ```python -### Implement the classification layer +### The classification layer @spacy.registry.architectures.register("rel_classification_layer.v1") def create_classification_layer( nO: int = None, nI: int = None @@ -617,60 +617,16 @@ def create_classification_layer( return chain(Linear(nO=nO, nI=nI), Logistic()) ``` -The first layer that **creates the instance tensor** can be defined -by implementing a -[custom forward function](https://thinc.ai/docs/usage-models#weights-layers-forward) -with an appropriate backpropagation callback. We also define an -[initialization method](https://thinc.ai/docs/usage-models#weights-layers-init) +The first layer that **creates the instance tensor** can be defined by +implementing a +[custom forward function](https://thinc.ai/docs/usage-models#weights-layers-forward) +with an appropriate backpropagation callback. We also define an +[initialization method](https://thinc.ai/docs/usage-models#weights-layers-init) that ensures that the layer is properly set up for training. -```python -### Implement the custom forward function -def instance_forward( - model: Model[List[Doc], Floats2d], - docs: List[Doc], - is_train: bool -) -> Tuple[Floats2d, Callable]: - ... - tok2vec = model.get_ref("tok2vec") - tokvecs, bp_tokvecs = tok2vec(docs, is_train) - relations = ... - - def backprop(d_relations: Floats2d) -> List[Doc]: - d_tokvecs = ... - return bp_tokvecs(d_tokvecs) - - return relations, backprop - - -### Implement the custom initialization method -def instance_init( - model: Model, - X: List[Doc] = None, - Y: Floats2d = None -) -> Model: - tok2vec = model.get_ref("tok2vec") - tok2vec.initialize(X) - return model - - -### Implement the layer that creates the instance tensor -@spacy.registry.architectures.register("rel_instance_tensor.v1") -def create_tensors( - tok2vec: Model[List[Doc], List[Floats2d]], - pooling: Model[Ragged, Floats2d], - get_instances: Callable[[Doc], List[Tuple[Span, Span]]], -) -> Model[List[Doc], Floats2d]: - - return Model( - "instance_tensors", - instance_forward, - layers=[tok2vec, pooling], - refs={"tok2vec": tok2vec, "pooling": pooling}, - attrs={"get_instances": get_instances}, - init=instance_init, - ) -``` +We omit some of the implementation details here, and refer to the spaCy project +that has the full implementation +[here](https://github.com/explosion/projects/tree/v3/tutorials/rel_component). > #### config.cfg (excerpt) > @@ -688,19 +644,69 @@ def create_tensors( > [model.create_instance_tensor.get_instances] > ... > ` +> ``` -This custom layer uses an -**[embedding layer](/usage/embeddings-transformers)** such as a -[`Tok2Vec`](/api/tok2vec) component or a [`Transformer`](/api/transformer). This -layer is assumed to be of type ~~Model[List[Doc], List[Floats2d]]~~ as it +```python +### The layer that creates the instance tensor +@spacy.registry.architectures.register("rel_instance_tensor.v1") +def create_tensors( + tok2vec: Model[List[Doc], List[Floats2d]], + pooling: Model[Ragged, Floats2d], + get_instances: Callable[[Doc], List[Tuple[Span, Span]]], +) -> Model[List[Doc], Floats2d]: + + return Model( + "instance_tensors", + instance_forward, + layers=[tok2vec, pooling], + refs={"tok2vec": tok2vec, "pooling": pooling}, + attrs={"get_instances": get_instances}, + init=instance_init, + ) + + +### The custom forward function +def instance_forward( + model: Model[List[Doc], Floats2d], + docs: List[Doc], + is_train: bool, +) -> Tuple[Floats2d, Callable]: + # ... + tok2vec = model.get_ref("tok2vec") + tokvecs, bp_tokvecs = tok2vec(docs, is_train) + relations = ... + + def backprop(d_relations: Floats2d) -> List[Doc]: + d_tokvecs = ... + return bp_tokvecs(d_tokvecs) + + return relations, backprop + + +### The custom initialization method +def instance_init( + model: Model, + X: List[Doc] = None, + Y: Floats2d = None, +) -> Model: + tok2vec = model.get_ref("tok2vec") + tok2vec.initialize(X) + return model + +``` + +This custom layer uses an [embedding layer](/usage/embeddings-transformers) such +as a [`Tok2Vec`](/api/tok2vec) component or a [`Transformer`](/api/transformer). +This layer is assumed to be of type ~~Model[List[Doc], List[Floats2d]]~~ as it transforms each **document into a list of tokens**, with each token being -represented by its embedding in the vector space. +represented by its embedding in the vector space. -The **`pooling`** layer will be applied to summarize the token vectors into entity -vectors, as named entities (represented by `Span` objects) can consist of one -or multiple tokens. For instance, the pooling layer could resort to calculating -the average of all token vectors in an entity. Thinc provides several -[built-in pooling operators](https://thinc.ai/docs/api-layers#reduction-ops) for +The `pooling` layer will be applied to summarize the token vectors into **entity +vectors**, as named entities (represented by ~~Span~~ objects) can consist of +one or multiple tokens. For instance, the pooling layer could resort to +calculating the average of all token vectors in an entity. Thinc provides +several +[built-in pooling operators](https://thinc.ai/docs/api-layers#reduction-ops) for this purpose. > #### config.cfg (excerpt) @@ -712,15 +718,15 @@ this purpose. > max_length = 100 > ``` -Finally, we need a `get_instances` method that **generates pairs of entities** -that we want to classify as being related or not. As these candidate pairs are typically formed -within one document, this function takes a [`Doc`](/api/doc) as input and -outputs a `List` of `Span` tuples. For instance, this +Finally, we need a `get_instances` method that **generates pairs of entities** +that we want to classify as being related or not. As these candidate pairs are +typically formed within one document, this function takes a [`Doc`](/api/doc) as +input and outputs a `List` of `Span` tuples. For instance, the following implementation takes any two entities from the same document, as long as they are within a **maximum distance** (in number of tokens) of eachother: ```python -### Simple candiate generation +### Candiate generation @spacy.registry.misc.register("rel_instance_generator.v1") def create_candidate_indices(max_length: int) -> Callable[[Doc], List[Tuple[Span, Span]]]: def get_candidates(doc: "Doc") -> List[Tuple[Span, Span]]: @@ -733,10 +739,10 @@ def create_candidate_indices(max_length: int) -> Callable[[Doc], List[Tuple[Span return candidates return get_candidates ``` -This function in added to the -[`@misc` registry](/api/top-level#registry) so we can refer to it from the -config, and easily swap it out for any other candidate generation function. +This function in added to the [`@misc` registry](/api/top-level#registry) so we +can refer to it from the config, and easily swap it out for any other candidate +generation function. When creating this model, we store the custom functions as [attributes](https://thinc.ai/docs/api-model#properties) and the sublayers as @@ -851,13 +857,13 @@ def update( sgd: Optional[Optimizer] = None, losses: Optional[Dict[str, float]] = None, ) -> Dict[str, float]: - ... + # ... docs = [ex.predicted for ex in examples] predictions, backprop = self.model.begin_update(docs) loss, gradient = self.get_loss(examples, predictions) backprop(gradient) losses[self.name] += loss - ... + # ... return losses ``` @@ -933,9 +939,9 @@ def __call__(self, Doc doc): return doc ``` -There is one more optional method to implement: [`score`](/api/pipe#score) -calculates the performance of your component on a set of examples, and -returns the results as a dictionary: +There is one more optional method to implement: [`score`](/api/pipe#score) +calculates the performance of your component on a set of examples, and returns +the results as a dictionary: ```python ### The score method @@ -951,8 +957,8 @@ def score(self, examples: Iterable[Example]) -> Dict[str, Any]: } ``` -This is particularly useful to see the scores on the development corpus -when training the component with [`spacy train`](/api/cli#training). +This is particularly useful to see the scores on the development corpus when +training the component with [`spacy train`](/api/cli#training). Once our `TrainablePipe` subclass is fully implemented, we can [register](/usage/processing-pipelines#custom-components-factories) the @@ -976,7 +982,7 @@ assigns it a name and lets you create the component with > [components.relation_extractor.model.get_candidates] > @misc = "rel_cand_generator.v1" > max_length = 20 -> +> > [training.score_weights] > rel_micro_p = 0.0 > rel_micro_r = 0.0 @@ -992,8 +998,8 @@ def make_relation_extractor(nlp, name, model): return RelationExtractor(nlp.vocab, model, name) ``` -You can extend the decorator to include information such as the type of -annotations that are required for this component to run, the type of annotations +You can extend the decorator to include information such as the type of +annotations that are required for this component to run, the type of annotations it produces, and the scores that can be calculated: ```python From 331ec83493a4158c5eda98e7e23bb8ae8d492de2 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Fri, 20 Nov 2020 21:41:52 +0100 Subject: [PATCH 4/7] edits and updates to implementing REL component docs --- website/docs/usage/layers-architectures.md | 139 ++++++++++++--------- 1 file changed, 80 insertions(+), 59 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index 2df4745db..01108e5c8 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -624,9 +624,9 @@ with an appropriate backpropagation callback. We also define an [initialization method](https://thinc.ai/docs/usage-models#weights-layers-init) that ensures that the layer is properly set up for training. -We omit some of the implementation details here, and refer to the spaCy project -that has the full implementation -[here](https://github.com/explosion/projects/tree/v3/tutorials/rel_component). +We omit some of the implementation details here, and refer to the +[spaCy project](https://github.com/explosion/projects/tree/v3/tutorials/rel_component) +that has the full implementation. > #### config.cfg (excerpt) > @@ -636,13 +636,13 @@ that has the full implementation > > [model.create_instance_tensor.tok2vec] > @architectures = "spacy.HashEmbedCNN.v1" -> ... +> # ... > > [model.create_instance_tensor.pooling] > @layers = "reduce_mean.v1" > > [model.create_instance_tensor.get_instances] -> ... +> # ... > ` > ``` @@ -658,10 +658,10 @@ def create_tensors( return Model( "instance_tensors", instance_forward, + init=instance_init, layers=[tok2vec, pooling], refs={"tok2vec": tok2vec, "pooling": pooling}, attrs={"get_instances": get_instances}, - init=instance_init, ) @@ -671,9 +671,11 @@ def instance_forward( docs: List[Doc], is_train: bool, ) -> Tuple[Floats2d, Callable]: - # ... tok2vec = model.get_ref("tok2vec") tokvecs, bp_tokvecs = tok2vec(docs, is_train) + get_instances = model.attrs["get_instances"] + all_instances = [get_instances(doc) for doc in docs] + pooling = model.get_ref("pooling") relations = ... def backprop(d_relations: Floats2d) -> List[Doc]: @@ -744,14 +746,35 @@ This function in added to the [`@misc` registry](/api/top-level#registry) so we can refer to it from the config, and easily swap it out for any other candidate generation function. -When creating this model, we store the custom functions as -[attributes](https://thinc.ai/docs/api-model#properties) and the sublayers as -references, so we can access them easily: +#### Intermezzo: define how to store the relations data {#component-rel-attribute} + +For our new relation extraction component, we will use a custom +[extension attribute](/usage/processing-pipelines#custom-components-attributes) +`doc._.rel` in which we store relation data. The attribute refers to a +dictionary, keyed by the **start offsets of each entity** involved in the +candidate relation. The values in the dictionary refer to another dictionary +where relation labels are mapped to values between 0 and 1. We assume anything +above 0.5 to be a `True` relation. The ~~Example~~ instances that we'll use as +training data, will include their gold-standard relation annotations in +`example.reference._.rel`. + +> #### Example output +> +> ```python +> doc = nlp("Amsterdam is the capital of the Netherlands.") +> print("spans", [(e.start, e.text, e.label_) for e in doc.ents]) +> for value, rel_dict in doc._.rel.items(): +> print(f"{value}: {rel_dict}") +> +> # spans [(0, 'Amsterdam', 'LOC'), (6, 'Netherlands', 'LOC')] +> # (0, 6): {'CAPITAL_OF': 0.89, 'LOCATED_IN': 0.75, 'UNRELATED': 0.002} +> # (6, 0): {'CAPITAL_OF': 0.01, 'LOCATED_IN': 0.13, 'UNRELATED': 0.017} +> ``` ```python -pooling = model.get_ref("pooling") -tok2vec = model.get_ref("tok2vec") -get_instances = model.attrs["get_instances"] +### Registering the extension attribute +from spacy.tokens import Doc +Doc.set_extension("rel", default={}) ``` #### Step 2: Implementing the pipeline component {#component-rel-pipe} @@ -794,19 +817,43 @@ class RelationExtractor(TrainablePipe): ... ``` -Before the model can be used, it needs to be -[initialized](/usage/training#initialization). This function receives a callback -to access the full **training data set**, or a representative sample. This data -set can be used to deduce all **relevant labels**. Alternatively, a list of -labels can be provided to `initialize`, or you can call -`RelationExtractor.add_label` directly. The number of labels defines the output -dimensionality of the network, and will be used to do +Typically, the constructor defines the vocab, the Machine Learning model, and +the name of this component. Additionally, this component, just like the +`textcat` and the `tagger`, stores an internal list of labels. The ML model will +predict scores for each label. We add convenience method to easily retrieve and +add to them. + +```python + def __init__(self, vocab, model, name="rel"): + """Create a component instance.""" + # ... + self.cfg = {"labels": []} + + @property + def labels(self) -> Tuple[str]: + """Returns the labels currently added to the component.""" + return tuple(self.cfg["labels"]) + + def add_label(self, label: str): + """Add a new label to the pipe.""" + self.cfg["labels"] = list(self.labels) + [label] +``` + +After creation, the component needs to be +[initialized](/usage/training#initialization). This method can define the +relevant labels in two ways: explicitely by setting the `labels` argument in the +[`initialize` block](/api/data-formats#config-initialize) of the config, or +implicately by deducing them from the `get_examples` callback that generates the +full **training data set**, or a representative sample. + +The final number of labels defines the output dimensionality of the network, and +will be used to do [shape inference](https://thinc.ai/docs/usage-models#validation) throughout the layers of the neural network. This is triggered by calling [`Model.initialize`](https://thinc.ai/api/model#initialize). ```python -### The initialize method {highlight="12,18,22"} +### The initialize method {highlight="12,15,18,22"} from itertools import islice def initialize( @@ -837,7 +884,7 @@ Typically, this happens when the pipeline is set up before training in [`spacy train`](/api/cli#training). After initialization, the pipeline component and its internal model can be trained and used to make predictions. -During training, the function [`update`](/api/pipe#update) is invoked which +During training, the method [`update`](/api/pipe#update) is invoked which delegates to [`Model.begin_update`](https://thinc.ai/docs/api-model#begin_update) and a [`get_loss`](/api/pipe#get_loss) function that **calculates the loss** for a @@ -858,7 +905,7 @@ def update( losses: Optional[Dict[str, float]] = None, ) -> Dict[str, float]: # ... - docs = [ex.predicted for ex in examples] + docs = [eg.predicted for eg in examples] predictions, backprop = self.model.begin_update(docs) loss, gradient = self.get_loss(examples, predictions) backprop(gradient) @@ -867,8 +914,8 @@ def update( return losses ``` -When the internal model is trained, the component can be used to make novel -**predictions**. The [`predict`](/api/pipe#predict) function needs to be +After training the model, the component can be used to make novel +**predictions**. The [`predict`](/api/pipe#predict) method needs to be implemented for each subclass of `TrainablePipe`. In our case, we can simply delegate to the internal model's [predict](https://thinc.ai/docs/api-model#predict) function that takes a batch @@ -884,42 +931,21 @@ def predict(self, docs: Iterable[Doc]) -> Floats2d: The final method that needs to be implemented, is [`set_annotations`](/api/pipe#set_annotations). This function takes the predictions, and modifies the given `Doc` object in place to store them. For our -relation extraction component, we store the data as a dictionary in a custom -[extension attribute](/usage/processing-pipelines#custom-components-attributes) -`doc._.rel`. As keys, we represent the candidate pair by the **start offsets of -each entity**, as this defines an entity pair uniquely within one document. +relation extraction component, we store the data in the +[custom attribute](#component-rel-attribute)`doc._.rel`. To interpret the scores predicted by the relation extraction model correctly, we -need to refer to the model's `get_candidates` function that defined which pairs +need to refer to the model's `get_instances` function that defined which pairs of entities were relevant candidates, so that the predictions can be linked to those exact entities: -> #### Example output -> -> ```python -> doc = nlp("Amsterdam is the capital of the Netherlands.") -> print("spans", [(e.start, e.text, e.label_) for e in doc.ents]) -> for value, rel_dict in doc._.rel.items(): -> print(f"{value}: {rel_dict}") -> -> # spans [(0, 'Amsterdam', 'LOC'), (6, 'Netherlands', 'LOC')] -> # (0, 6): {'CAPITAL_OF': 0.89, 'LOCATED_IN': 0.75, 'UNRELATED': 0.002} -> # (6, 0): {'CAPITAL_OF': 0.01, 'LOCATED_IN': 0.13, 'UNRELATED': 0.017} -> ``` - -```python -### Registering the extension attribute -from spacy.tokens import Doc -Doc.set_extension("rel", default={}) -``` - ```python ### The set_annotations method {highlight="5-6,10"} def set_annotations(self, docs: Iterable[Doc], predictions: Floats2d): c = 0 - get_candidates = self.model.attrs["get_candidates"] + get_instances = self.model.attrs["get_instances"] for doc in docs: - for (e1, e2) in get_candidates(doc): + for (e1, e2) in get_instances(doc): offset = (e1.start, e2.start) if offset not in doc._.rel: doc._.rel[offset] = {} @@ -933,7 +959,7 @@ Under the hood, when the pipe is applied to a document, it delegates to the ```python ### The __call__ method -def __call__(self, Doc doc): +def __call__(self, doc: Doc): predictions = self.predict([doc]) self.set_annotations([doc], predictions) return doc @@ -957,8 +983,8 @@ def score(self, examples: Iterable[Example]) -> Dict[str, Any]: } ``` -This is particularly useful to see the scores on the development corpus when -training the component with [`spacy train`](/api/cli#training). +This is particularly useful for calculating relevant scores on the development +corpus when training the component with [`spacy train`](/api/cli#training). Once our `TrainablePipe` subclass is fully implemented, we can [register](/usage/processing-pipelines#custom-components-factories) the @@ -975,13 +1001,8 @@ assigns it a name and lets you create the component with > > [components.relation_extractor.model] > @architectures = "rel_model.v1" -> -> [components.relation_extractor.model.tok2vec] > # ... > -> [components.relation_extractor.model.get_candidates] -> @misc = "rel_cand_generator.v1" -> max_length = 20 > > [training.score_weights] > rel_micro_p = 0.0 From 5ac0867427cc639ee3addd8af33560ae7a2b6ab6 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Fri, 20 Nov 2020 22:18:53 +0100 Subject: [PATCH 5/7] final fixes --- website/docs/usage/layers-architectures.md | 44 +++++++++++----------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index 01108e5c8..37d6afaf0 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -567,10 +567,10 @@ def create_relation_model(...) -> Model[List[Doc], Floats2d]: return model ``` -We will adapt a **modular approach** to the definition of this relation model, -and define it as chaining two layers together: the first layer that generates an +We adapt a **modular approach** to the definition of this relation model, and +define it as chaining two layers together: the first layer that generates an instance tensor from a given set of documents, and the second layer that -transforms the instance tensor into a final tensor holding the predictions. +transforms the instance tensor into a final tensor holding the predictions: > #### config.cfg (excerpt) > @@ -586,7 +586,7 @@ transforms the instance tensor into a final tensor holding the predictions. > ``` ```python -### The model architecture +### The model architecture {highlight="6"} @spacy.registry.architectures.register("rel_model.v1") def create_relation_model( create_instance_tensor: Model[List[Doc], Floats2d], @@ -596,8 +596,9 @@ def create_relation_model( return model ``` -The `classification_layer` could be something like a Linear layer followed by a -logistic activation function: +The `classification_layer` could be something like a +[Linear](https://thinc.ai/docs/api-layers#linear) layer followed by a +[logistic](https://thinc.ai/docs/api-layers#logistic) activation function: > #### config.cfg (excerpt) > @@ -748,16 +749,6 @@ generation function. #### Intermezzo: define how to store the relations data {#component-rel-attribute} -For our new relation extraction component, we will use a custom -[extension attribute](/usage/processing-pipelines#custom-components-attributes) -`doc._.rel` in which we store relation data. The attribute refers to a -dictionary, keyed by the **start offsets of each entity** involved in the -candidate relation. The values in the dictionary refer to another dictionary -where relation labels are mapped to values between 0 and 1. We assume anything -above 0.5 to be a `True` relation. The ~~Example~~ instances that we'll use as -training data, will include their gold-standard relation annotations in -`example.reference._.rel`. - > #### Example output > > ```python @@ -771,6 +762,16 @@ training data, will include their gold-standard relation annotations in > # (6, 0): {'CAPITAL_OF': 0.01, 'LOCATED_IN': 0.13, 'UNRELATED': 0.017} > ``` +For our new relation extraction component, we will use a custom +[extension attribute](/usage/processing-pipelines#custom-components-attributes) +`doc._.rel` in which we store relation data. The attribute refers to a +dictionary, keyed by the **start offsets of each entity** involved in the +candidate relation. The values in the dictionary refer to another dictionary +where relation labels are mapped to values between 0 and 1. We assume anything +above 0.5 to be a `True` relation. The ~~Example~~ instances that we'll use as +training data, will include their gold-standard relation annotations in +`example.reference._.rel`. + ```python ### Registering the extension attribute from spacy.tokens import Doc @@ -817,11 +818,11 @@ class RelationExtractor(TrainablePipe): ... ``` -Typically, the constructor defines the vocab, the Machine Learning model, and -the name of this component. Additionally, this component, just like the -`textcat` and the `tagger`, stores an internal list of labels. The ML model will -predict scores for each label. We add convenience method to easily retrieve and -add to them. +Typically, the **constructor** defines the vocab, the Machine Learning model, +and the name of this component. Additionally, this component, just like the +`textcat` and the `tagger`, stores an **internal list of labels**. The ML model +will predict scores for each label. We add convenience methods to easily +retrieve and add to them. ```python def __init__(self, vocab, model, name="rel"): @@ -1003,7 +1004,6 @@ assigns it a name and lets you create the component with > @architectures = "rel_model.v1" > # ... > -> > [training.score_weights] > rel_micro_p = 0.0 > rel_micro_r = 0.0 From e861e928df5307fcd1e836d85b6e11c0b076eb25 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Fri, 20 Nov 2020 22:29:58 +0100 Subject: [PATCH 6/7] more small corrections --- website/docs/usage/layers-architectures.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index 37d6afaf0..a2a6e3167 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -644,7 +644,6 @@ that has the full implementation. > > [model.create_instance_tensor.get_instances] > # ... -> ` > ``` ```python @@ -666,7 +665,7 @@ def create_tensors( ) -### The custom forward function +# The custom forward function def instance_forward( model: Model[List[Doc], Floats2d], docs: List[Doc], @@ -686,7 +685,7 @@ def instance_forward( return relations, backprop -### The custom initialization method +# The custom initialization method def instance_init( model: Model, X: List[Doc] = None, @@ -712,6 +711,13 @@ several [built-in pooling operators](https://thinc.ai/docs/api-layers#reduction-ops) for this purpose. +Finally, we need a `get_instances` method that **generates pairs of entities** +that we want to classify as being related or not. As these candidate pairs are +typically formed within one document, this function takes a [`Doc`](/api/doc) as +input and outputs a `List` of `Span` tuples. For instance, the following +implementation takes any two entities from the same document, as long as they +are within a **maximum distance** (in number of tokens) of eachother: + > #### config.cfg (excerpt) > > ```ini @@ -721,17 +727,10 @@ this purpose. > max_length = 100 > ``` -Finally, we need a `get_instances` method that **generates pairs of entities** -that we want to classify as being related or not. As these candidate pairs are -typically formed within one document, this function takes a [`Doc`](/api/doc) as -input and outputs a `List` of `Span` tuples. For instance, the following -implementation takes any two entities from the same document, as long as they -are within a **maximum distance** (in number of tokens) of eachother: - ```python ### Candiate generation @spacy.registry.misc.register("rel_instance_generator.v1") -def create_candidate_indices(max_length: int) -> Callable[[Doc], List[Tuple[Span, Span]]]: +def create_instances(max_length: int) -> Callable[[Doc], List[Tuple[Span, Span]]]: def get_candidates(doc: "Doc") -> List[Tuple[Span, Span]]: candidates = [] for ent1 in doc.ents: @@ -825,6 +824,7 @@ will predict scores for each label. We add convenience methods to easily retrieve and add to them. ```python +### The constructor (continued) def __init__(self, vocab, model, name="rel"): """Create a component instance.""" # ... From 218abaa69aed66c90fcd9d67659684832a96828e Mon Sep 17 00:00:00 2001 From: svlandeg Date: Fri, 20 Nov 2020 22:36:49 +0100 Subject: [PATCH 7/7] typo --- website/docs/usage/layers-architectures.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index a2a6e3167..eb6f8b288 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -728,7 +728,7 @@ are within a **maximum distance** (in number of tokens) of eachother: > ``` ```python -### Candiate generation +### Candidate generation @spacy.registry.misc.register("rel_instance_generator.v1") def create_instances(max_length: int) -> Callable[[Doc], List[Tuple[Span, Span]]]: def get_candidates(doc: "Doc") -> List[Tuple[Span, Span]]: