From 3d9ae9286ff15049eccd0f840d78daf47a25f045 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Wed, 2 Sep 2020 10:46:38 +0200 Subject: [PATCH 01/10] small fixes --- website/docs/usage/layers-architectures.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index aa398f752..aca9a76e5 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -62,7 +62,7 @@ are: ​ | ~~Ints2d~~ | A two-dimensional `numpy` or `cupy` array of integers. Common dtypes include uint64, int32 and int8. | | ~~List[Floats2d]~~ | A list of two-dimensional arrays, generally with one array per `Doc` and one row per token. | | ~~Ragged~~ | A container to handle variable-length sequence data in an unpadded contiguous array. | -| ~~Padded~~ | A container to handle variable-length sequence data in a passed contiguous array. | +| ~~Padded~~ | A container to handle variable-length sequence data in a padded contiguous array. | The model type signatures help you figure out which model architectures and components can **fit together**. For instance, the @@ -94,7 +94,7 @@ code. ## Defining sublayers {#sublayers} -​ Model architecture functions often accept **sublayers as arguments**, so that +Model architecture functions often accept **sublayers as arguments**, so that you can try **substituting a different layer** into the network. Depending on how the architecture function is structured, you might be able to define your network structure entirely through the [config system](/usage/training#config), @@ -112,7 +112,7 @@ you can control this important part of the network separately. This makes it easy to **switch between** transformer, CNN, BiLSTM or other feature extraction approaches. And if you want to define your own solution, all you need to do is register a ~~Model[List[Doc], List[Floats2d]]~~ architecture function, and -you'll be able to try it out in any of spaCy components. ​ +you'll be able to try it out in any of the spaCy components. ​ From 6fd7f140ecbf8604f39aa32dda74c27a157ff9d7 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Wed, 2 Sep 2020 11:14:06 +0200 Subject: [PATCH 02/10] custom-architectures section --- website/docs/usage/training.md | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/website/docs/usage/training.md b/website/docs/usage/training.md index 2d7905230..6d56f5767 100644 --- a/website/docs/usage/training.md +++ b/website/docs/usage/training.md @@ -669,7 +669,7 @@ def custom_logger(log_path): #### Example: Custom batch size schedule {#custom-code-schedule} -For example, let's say you've implemented your own batch size schedule to use +You can also implement your own batch size schedule to use during training. The `@spacy.registry.schedules` decorator lets you register that function in the `schedules` [registry](/api/top-level#registry) and assign it a string name: @@ -806,7 +806,37 @@ def filter_batch(size: int) -> Callable[[Iterable[Example]], Iterator[List[Examp ### Defining custom architectures {#custom-architectures} - +Built-in pipeline components such as the tagger or named entity recognizer are +constructed with default neural network [models](/api/architectures). +You can change the model architecture +entirely by implementing your own custom models and providing those in the config +when creating the pipeline component. See the +documentation on +[layers and model architectures](/usage/layers-architectures) for more details. + + +```python +### functions.py +from typing import List +from thinc.types import Floats2d +from thinc.api import Model +import spacy +from spacy.tokens import Doc + +@spacy.registry.architectures("custom_neural_network.v1") +def MyModel(output_width: int) -> Model[List[Doc], List[Floats2d]]: + # ... +``` + +```ini +### config.cfg (excerpt) +[components.tagger] +factory = "tagger" + +[components.tagger.model] +@architectures = "custom_neural_network.v1" +output_width = 512 +``` ## Internal training API {#api} From 474abb2e5969f7708ba93bae901fda3ad542dc78 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Wed, 2 Sep 2020 11:37:56 +0200 Subject: [PATCH 03/10] remove unused MORPH_RULES from test --- spacy/tests/pipeline/test_tagger.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/spacy/tests/pipeline/test_tagger.py b/spacy/tests/pipeline/test_tagger.py index b1b52b9fa..a1aa7e1e1 100644 --- a/spacy/tests/pipeline/test_tagger.py +++ b/spacy/tests/pipeline/test_tagger.py @@ -28,8 +28,6 @@ def test_tagger_begin_training_tag_map(): TAGS = ("N", "V", "J") -MORPH_RULES = {"V": {"like": {"lemma": "luck"}}} - TRAIN_DATA = [ ("I like green eggs", {"tags": ["N", "V", "J", "N"]}), ("Eat blue ham", {"tags": ["V", "J", "N"]}), From e29a33449dc66192868a8290c564f622ee2d79b5 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Wed, 2 Sep 2020 13:41:18 +0200 Subject: [PATCH 04/10] rewrite intro, simpel Model example --- website/docs/usage/layers-architectures.md | 38 +++++++++++++++------- 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index 3ef28acaf..ac91ca0ad 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -10,18 +10,32 @@ menu: next: /usage/projects --- -​A **model architecture** is a function that wires up a -[Thinc `Model`](https://thinc.ai/docs/api-model) instance, which you can then -use in a component or as a layer of a larger network. You can use Thinc as a -thin wrapper around frameworks such as PyTorch, TensorFlow or MXNet, or you can -implement your logic in Thinc directly. ​ spaCy's built-in components will never -construct their `Model` instances themselves, so you won't have to subclass the -component to change its model architecture. You can just **update the config** -so that it refers to a different registered function. Once the component has -been created, its model instance has already been assigned, so you cannot change -its model architecture. The architecture is like a recipe for the network, and -you can't change the recipe once the dish has already been prepared. You have to -make a new one. +> #### Example +> +> ````python +> from thinc.api import Model, chain +> +> def build_model(width: int, classes: int) -> Model: +> tok2vec = build_tok2vec(width) +> output_layer = build_output_layer(width, classes) +> model = chain(tok2vec, output_layer) +> return model +> ```` + +A **model architecture** is a function that wires up a +[Thinc `Model`](https://thinc.ai/docs/api-model) instance. It describes the +neural network that is run internally as part of a component in a spaCy +pipeline. To define the actual architecture, you can implement your logic in +Thinc directly, but you can also use Thinc as a thin wrapper around frameworks +such as PyTorch, TensorFlow or MXNet. + +spaCy's built-in components require a `Model` instance to be passed to them via +the config system. To change the model architecture of an existing component, +you just need to **update the config** so that it refers to a different +registered function. Once the component has been created from this config, you +won't be able to change it anymore. The architecture is like a recipe for the +network, and you can't change the recipe once the dish has already been +prepared. You have to make a new one. ## Type signatures {#type-sigs} From 821b2d4e630438a7fd5b93439a28d67b0823c451 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Wed, 2 Sep 2020 14:15:50 +0200 Subject: [PATCH 05/10] update examples --- website/docs/usage/layers-architectures.md | 46 ++++++++++++++-------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index ac91ca0ad..ea0427903 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -14,7 +14,8 @@ next: /usage/projects > > ````python > from thinc.api import Model, chain -> +> +> @spacy.registry.architectures.register("model.v1") > def build_model(width: int, classes: int) -> Model: > tok2vec = build_tok2vec(width) > output_layer = build_output_layer(width, classes) @@ -24,10 +25,12 @@ next: /usage/projects A **model architecture** is a function that wires up a [Thinc `Model`](https://thinc.ai/docs/api-model) instance. It describes the -neural network that is run internally as part of a component in a spaCy -pipeline. To define the actual architecture, you can implement your logic in -Thinc directly, but you can also use Thinc as a thin wrapper around frameworks -such as PyTorch, TensorFlow or MXNet. +neural network that is run internally as part of a component in a spaCy pipeline. +To define the actual architecture, you can implement your logic in +Thinc directly, or you can use Thinc as a thin wrapper around frameworks +such as PyTorch, TensorFlow and MXNet. Each Model can also be used as a sublayer +of a larger network, allowing you to freely combine implementations from different +frameworks into one `Thinc` Model. spaCy's built-in components require a `Model` instance to be passed to them via the config system. To change the model architecture of an existing component, @@ -37,6 +40,17 @@ won't be able to change it anymore. The architecture is like a recipe for the network, and you can't change the recipe once the dish has already been prepared. You have to make a new one. +```ini +### config.cfg (excerpt) +[components.tagger] +factory = "tagger" + +[components.tagger.model] +@architectures = "model.v1" +width = 512 +classes = 16 +``` + ## Type signatures {#type-sigs} @@ -44,17 +58,15 @@ prepared. You have to make a new one. > #### Example > > ```python -> @spacy.registry.architectures.register("spacy.Tagger.v1") -> def build_tagger_model( -> tok2vec: Model[List[Doc], List[Floats2d]], nO: Optional[int] = None -> ) -> Model[List[Doc], List[Floats2d]]: -> t2v_width = tok2vec.get_dim("nO") if tok2vec.has_dim("nO") else None -> output_layer = Softmax(nO, t2v_width, init_W=zero_init) -> softmax = with_array(output_layer) -> model = chain(tok2vec, softmax) -> model.set_ref("tok2vec", tok2vec) -> model.set_ref("softmax", output_layer) -> model.set_ref("output_layer", output_layer) +> from typing import List +> from thinc.api import Model, chain +> from thinc.types import Floats2d +> def chain_model( +> tok2vec: Model[List[Doc], List[Floats2d]], +> layer1: Model[List[Floats2d], Floats2d], +> layer2: Model[Floats2d, Floats2d] +> ) -> Model[List[Doc], Floats2d]: +> model = chain(tok2vec, layer1, layer2) > return model > ``` @@ -65,7 +77,7 @@ list, and the outputs will be a dictionary. Both `typing.List` and `typing.Dict` are also generics, allowing you to be more specific about the data. For instance, you can write ~~Model[List[Doc], Dict[str, float]]~~ to specify that the model expects a list of [`Doc`](/api/doc) objects as input, and returns a -dictionary mapping strings to floats. Some of the most common types you'll see +dictionary mapping of strings to floats. Some of the most common types you'll see are: ​ | Type | Description | From d19ec6c67b18db504172bc8bd1e2a89694e6b5c7 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Wed, 2 Sep 2020 14:25:18 +0200 Subject: [PATCH 06/10] small rewrites in types paragraph --- website/docs/usage/layers-architectures.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index ea0427903..32319ca07 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -70,12 +70,11 @@ classes = 16 > return model > ``` -​ The Thinc `Model` class is a **generic type** that can specify its input and +The Thinc `Model` class is a **generic type** that can specify its input and output types. Python uses a square-bracket notation for this, so the type ~~Model[List, Dict]~~ says that each batch of inputs to the model will be a -list, and the outputs will be a dictionary. Both `typing.List` and `typing.Dict` -are also generics, allowing you to be more specific about the data. For -instance, you can write ~~Model[List[Doc], Dict[str, float]]~~ to specify that +list, and the outputs will be a dictionary. You can be even more specific and +write for instance~~Model[List[Doc], Dict[str, float]]~~ to specify that the model expects a list of [`Doc`](/api/doc) objects as input, and returns a dictionary mapping of strings to floats. Some of the most common types you'll see are: ​ @@ -103,8 +102,8 @@ interchangeably. There are many other ways they could be incompatible. However, if the types don't match, they almost surely _won't_ be compatible. This little bit of validation goes a long way, especially if you [configure your editor](https://thinc.ai/docs/usage-type-checking) or other -tools to highlight these errors early. Thinc will also verify that your types -match correctly when your config file is processed at the beginning of training. +tools to highlight these errors early. The config file is also validated +at the beginning of training, to verify that all the types match correctly. From 57e432ba2aaa5258abf1b18e2e19995bd284e3a1 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Wed, 2 Sep 2020 14:26:57 +0200 Subject: [PATCH 07/10] editor tip as Accordion instead of Infobox --- website/docs/usage/layers-architectures.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index 32319ca07..f5cdb1ca1 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -105,7 +105,7 @@ bit of validation goes a long way, especially if you tools to highlight these errors early. The config file is also validated at the beginning of training, to verify that all the types match correctly. - + If you're using a modern editor like Visual Studio Code, you can [set up `mypy`](https://thinc.ai/docs/usage-type-checking#install) with the @@ -114,7 +114,7 @@ code. [![](../images/thinc_mypy.jpg)](https://thinc.ai/docs/usage-type-checking#linting) - + ## Swapping model architectures {#swap-architectures} From 1be7ff02a61879b002e4c1236f91b8b1329278af Mon Sep 17 00:00:00 2001 From: svlandeg Date: Wed, 2 Sep 2020 15:26:07 +0200 Subject: [PATCH 08/10] swapping section --- website/docs/usage/layers-architectures.md | 93 ++++++++++++++++------ 1 file changed, 68 insertions(+), 25 deletions(-) diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index f5cdb1ca1..8f10f4069 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -12,33 +12,33 @@ next: /usage/projects > #### Example > -> ````python +> ```python > from thinc.api import Model, chain -> +> > @spacy.registry.architectures.register("model.v1") > def build_model(width: int, classes: int) -> Model: > tok2vec = build_tok2vec(width) > output_layer = build_output_layer(width, classes) > model = chain(tok2vec, output_layer) > return model -> ```` +> ``` A **model architecture** is a function that wires up a [Thinc `Model`](https://thinc.ai/docs/api-model) instance. It describes the -neural network that is run internally as part of a component in a spaCy pipeline. -To define the actual architecture, you can implement your logic in -Thinc directly, or you can use Thinc as a thin wrapper around frameworks -such as PyTorch, TensorFlow and MXNet. Each Model can also be used as a sublayer -of a larger network, allowing you to freely combine implementations from different +neural network that is run internally as part of a component in a spaCy +pipeline. To define the actual architecture, you can implement your logic in +Thinc directly, or you can use Thinc as a thin wrapper around frameworks such as +PyTorch, TensorFlow and MXNet. Each Model can also be used as a sublayer of a +larger network, allowing you to freely combine implementations from different frameworks into one `Thinc` Model. spaCy's built-in components require a `Model` instance to be passed to them via the config system. To change the model architecture of an existing component, -you just need to **update the config** so that it refers to a different -registered function. Once the component has been created from this config, you -won't be able to change it anymore. The architecture is like a recipe for the -network, and you can't change the recipe once the dish has already been -prepared. You have to make a new one. +you just need to [**update the config**](#swap-architectures) so that it refers +to a different registered function. Once the component has been created from +this config, you won't be able to change it anymore. The architecture is like a +recipe for the network, and you can't change the recipe once the dish has +already been prepared. You have to make a new one. ```ini ### config.cfg (excerpt) @@ -53,8 +53,6 @@ classes = 16 ## Type signatures {#type-sigs} - - > #### Example > > ```python @@ -62,8 +60,8 @@ classes = 16 > from thinc.api import Model, chain > from thinc.types import Floats2d > def chain_model( -> tok2vec: Model[List[Doc], List[Floats2d]], -> layer1: Model[List[Floats2d], Floats2d], +> tok2vec: Model[List[Doc], List[Floats2d]], +> layer1: Model[List[Floats2d], Floats2d], > layer2: Model[Floats2d, Floats2d] > ) -> Model[List[Doc], Floats2d]: > model = chain(tok2vec, layer1, layer2) @@ -73,11 +71,11 @@ classes = 16 The Thinc `Model` class is a **generic type** that can specify its input and output types. Python uses a square-bracket notation for this, so the type ~~Model[List, Dict]~~ says that each batch of inputs to the model will be a -list, and the outputs will be a dictionary. You can be even more specific and -write for instance~~Model[List[Doc], Dict[str, float]]~~ to specify that -the model expects a list of [`Doc`](/api/doc) objects as input, and returns a -dictionary mapping of strings to floats. Some of the most common types you'll see -are: ​ +list, and the outputs will be a dictionary. You can be even more specific and +write for instance~~Model[List[Doc], Dict[str, float]]~~ to specify that the +model expects a list of [`Doc`](/api/doc) objects as input, and returns a +dictionary mapping of strings to floats. Some of the most common types you'll +see are: ​ | Type | Description | | ------------------ | ---------------------------------------------------------------------------------------------------- | @@ -102,8 +100,8 @@ interchangeably. There are many other ways they could be incompatible. However, if the types don't match, they almost surely _won't_ be compatible. This little bit of validation goes a long way, especially if you [configure your editor](https://thinc.ai/docs/usage-type-checking) or other -tools to highlight these errors early. The config file is also validated -at the beginning of training, to verify that all the types match correctly. +tools to highlight these errors early. The config file is also validated at the +beginning of training, to verify that all the types match correctly. @@ -118,7 +116,52 @@ code. ## Swapping model architectures {#swap-architectures} - +If no model is specified for the [`TextCategorizer`](/api/textcategorizer), the +[TextCatEnsemble](/api/architectures#TextCatEnsemble) architecture is used by +default. This architecture combines a simpel bag-of-words model with a neural +network, usually resulting in the most accurate results, but at the cost of +speed. The config file for this model would look something like this: + +```ini +### config.cfg (excerpt) +[components.textcat] +factory = "textcat" +labels = [] + +[components.textcat.model] +@architectures = "spacy.TextCatEnsemble.v1" +exclusive_classes = false +pretrained_vectors = null +width = 64 +conv_depth = 2 +embed_size = 2000 +window_size = 1 +ngram_size = 1 +dropout = 0 +nO = null +``` + +spaCy has two additional built-in `textcat` architectures, and you can easily +use those by swapping out the definition of the textcat's model. For instance, +to use the simpel and fast [bag-of-words model](/api/architectures#TextCatBOW), +you can change the config to: + +```ini +### config.cfg (excerpt) +[components.textcat] +factory = "textcat" +labels = [] + +[components.textcat.model] +@architectures = "spacy.TextCatBOW.v1" +exclusive_classes = false +ngram_size = 1 +no_output_layer = false +nO = null +``` + +The details of all prebuilt architectures and their parameters, can be consulted +on the [API page for model architectures](/api/architectures). ### Defining sublayers {#sublayers} From bbaea530f6ede494e708a10306f517e5b60c6ba2 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Wed, 2 Sep 2020 17:36:22 +0200 Subject: [PATCH 09/10] sublayers paragraph --- website/docs/api/architectures.md | 66 ++++++++++++---------- website/docs/usage/layers-architectures.md | 59 ++++++++++++++----- 2 files changed, 81 insertions(+), 44 deletions(-) diff --git a/website/docs/api/architectures.md b/website/docs/api/architectures.md index b55027356..93e50bfb3 100644 --- a/website/docs/api/architectures.md +++ b/website/docs/api/architectures.md @@ -25,36 +25,6 @@ usage documentation on ## Tok2Vec architectures {#tok2vec-arch source="spacy/ml/models/tok2vec.py"} -### spacy.HashEmbedCNN.v1 {#HashEmbedCNN} - -> #### Example Config -> -> ```ini -> [model] -> @architectures = "spacy.HashEmbedCNN.v1" -> pretrained_vectors = null -> width = 96 -> depth = 4 -> embed_size = 2000 -> window_size = 1 -> maxout_pieces = 3 -> subword_features = true -> ``` - -Build spaCy's "standard" embedding layer, which uses hash embedding with subword -features and a CNN with layer-normalized maxout. - -| Name | Description | -| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `width` | The width of the input and output. These are required to be the same, so that residual connections can be used. Recommended values are `96`, `128` or `300`. ~~int~~ | -| `depth` | The number of convolutional layers to use. Recommended values are between `2` and `8`. ~~int~~ | -| `embed_size` | The number of rows in the hash embedding tables. This can be surprisingly small, due to the use of the hash embeddings. Recommended values are between `2000` and `10000`. ~~int~~ | -| `window_size` | The number of tokens on either side to concatenate during the convolutions. The receptive field of the CNN will be `depth * (window_size * 2 + 1)`, so a 4-layer network with a window size of `2` will be sensitive to 17 words at a time. Recommended value is `1`. ~~int~~ | -| `maxout_pieces` | The number of pieces to use in the maxout non-linearity. If `1`, the [`Mish`](https://thinc.ai/docs/api-layers#mish) non-linearity is used instead. Recommended values are `1`-`3`. ~~int~~ | -| `subword_features` | Whether to also embed subword features, specifically the prefix, suffix and word shape. This is recommended for alphabetic languages like English, but not if single-character tokens are used for a language such as Chinese. ~~bool~~ | -| `pretrained_vectors` | Whether to also use static vectors. ~~bool~~ | -| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | - ### spacy.Tok2Vec.v1 {#Tok2Vec} > #### Example config @@ -72,7 +42,8 @@ features and a CNN with layer-normalized maxout. > # ... > ``` -Construct a tok2vec model out of embedding and encoding subnetworks. See the +Construct a tok2vec model out of two subnetworks: one for embedding and one for +encoding. See the ["Embed, Encode, Attend, Predict"](https://explosion.ai/blog/deep-learning-formula-nlp) blog post for background. @@ -82,6 +53,39 @@ blog post for background. | `encode` | Encode context into the embeddings, using an architecture such as a CNN, BiLSTM or transformer. For example, [MaxoutWindowEncoder](/api/architectures#MaxoutWindowEncoder). ~~Model[List[Floats2d], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | +### spacy.HashEmbedCNN.v1 {#HashEmbedCNN} + +> #### Example Config +> +> ```ini +> [model] +> @architectures = "spacy.HashEmbedCNN.v1" +> pretrained_vectors = null +> width = 96 +> depth = 4 +> embed_size = 2000 +> window_size = 1 +> maxout_pieces = 3 +> subword_features = true +> ``` + +Build spaCy's "standard" tok2vec layer. This layer is defined by a +[MultiHashEmbed](/api/architectures#MultiHashEmbed) embedding layer that uses +subword features, and a +[MaxoutWindowEncoder](/api/architectures#MaxoutWindowEncoder) encoding layer +consisting of a CNN and a layer-normalized maxout activation function. + +| Name | Description | +| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `width` | The width of the input and output. These are required to be the same, so that residual connections can be used. Recommended values are `96`, `128` or `300`. ~~int~~ | +| `depth` | The number of convolutional layers to use. Recommended values are between `2` and `8`. ~~int~~ | +| `embed_size` | The number of rows in the hash embedding tables. This can be surprisingly small, due to the use of the hash embeddings. Recommended values are between `2000` and `10000`. ~~int~~ | +| `window_size` | The number of tokens on either side to concatenate during the convolutions. The receptive field of the CNN will be `depth * (window_size * 2 + 1)`, so a 4-layer network with a window size of `2` will be sensitive to 17 words at a time. Recommended value is `1`. ~~int~~ | +| `maxout_pieces` | The number of pieces to use in the maxout non-linearity. If `1`, the [`Mish`](https://thinc.ai/docs/api-layers#mish) non-linearity is used instead. Recommended values are `1`-`3`. ~~int~~ | +| `subword_features` | Whether to also embed subword features, specifically the prefix, suffix and word shape. This is recommended for alphabetic languages like English, but not if single-character tokens are used for a language such as Chinese. ~~bool~~ | +| `pretrained_vectors` | Whether to also use static vectors. ~~bool~~ | +| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | + ### spacy.Tok2VecListener.v1 {#Tok2VecListener} > #### Example config diff --git a/website/docs/usage/layers-architectures.md b/website/docs/usage/layers-architectures.md index 8f10f4069..419048f65 100644 --- a/website/docs/usage/layers-architectures.md +++ b/website/docs/usage/layers-architectures.md @@ -165,27 +165,60 @@ on the [API page for model architectures](/api/architectures). ### Defining sublayers {#sublayers} -​Model architecture functions often accept **sublayers as arguments**, so that +Model architecture functions often accept **sublayers as arguments**, so that you can try **substituting a different layer** into the network. Depending on how the architecture function is structured, you might be able to define your network structure entirely through the [config system](/usage/training#config), -using layers that have already been defined. ​The -[transformers documentation](/usage/embeddings-transformers#transformers) -section shows a common example of swapping in a different sublayer. +using layers that have already been defined. ​ In most neural network models for NLP, the most important parts of the network are what we refer to as the -[embed and encode](https://explosion.ai/blog/embed-encode-attend-predict) steps. +[embed and encode](https://explosion.ai/blog/deep-learning-formula-nlp) steps. These steps together compute dense, context-sensitive representations of the -tokens. Most of spaCy's default architectures accept a -[`tok2vec` embedding layer](/api/architectures#tok2vec-arch) as an argument, so -you can control this important part of the network separately. This makes it -easy to **switch between** transformer, CNN, BiLSTM or other feature extraction -approaches. And if you want to define your own solution, all you need to do is -register a ~~Model[List[Doc], List[Floats2d]]~~ architecture function, and -you'll be able to try it out in any of the spaCy components. ​ +tokens, and their combination forms a typical +[`Tok2Vec`](/api/architectures#Tok2Vec) layer: - +```ini +### config.cfg (excerpt) +[components.tok2vec] +factory = "tok2vec" + +[components.tok2vec.model] +@architectures = "spacy.Tok2Vec.v1" + +[components.tok2vec.model.embed] +@architectures = "spacy.MultiHashEmbed.v1" +# ... + +[components.tok2vec.model.encode] +@architectures = "spacy.MaxoutWindowEncoder.v1" +# ... +``` + +By defining these sublayers specifically, it becomes straightforward to swap out +a sublayer for another one, for instance changing the first sublayer to a +character embedding with the [CharacterEmbed](/api/architectures#CharacterEmbed) +architecture: + +```ini +### config.cfg (excerpt) +[components.tok2vec.model.embed] +@architectures = "spacy.CharacterEmbed.v1" +# ... + +[components.tok2vec.model.encode] +@architectures = "spacy.MaxoutWindowEncoder.v1" +# ... +``` + +Most of spaCy's default architectures accept a `tok2vec` layer as a sublayer +within the larger task-specific neural network. This makes it easy to **switch +between** transformer, CNN, BiLSTM or other feature extraction approaches. The +[transformers documentation](/usage/embeddings-transformers#training-custom-model) +section shows an example of swapping out a model's standard `tok2vec` layer with +a transformer. And if you want to define your own solution, all you need to do +is register a ~~Model[List[Doc], List[Floats2d]]~~ architecture function, and +you'll be able to try it out in any of the spaCy components. ​ ## Wrapping PyTorch, TensorFlow and other frameworks {#frameworks} From 19298de3524d2aee05579c6b451d2306960a6591 Mon Sep 17 00:00:00 2001 From: svlandeg Date: Wed, 2 Sep 2020 17:43:11 +0200 Subject: [PATCH 10/10] small fix --- website/docs/usage/training.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/usage/training.md b/website/docs/usage/training.md index 6d56f5767..2967a0353 100644 --- a/website/docs/usage/training.md +++ b/website/docs/usage/training.md @@ -825,7 +825,7 @@ from spacy.tokens import Doc @spacy.registry.architectures("custom_neural_network.v1") def MyModel(output_width: int) -> Model[List[Doc], List[Floats2d]]: - # ... + return create_model(output_width) ``` ```ini