Website migration from Gatsby to Next (#12058)

* Rename all MDX file to `.mdx`

* Lock current node version (#11885)

* Apply Prettier (#11996)

* Minor website fixes (#11974) [ci skip]

* fix table

* Migrate to Next WEB-17 (#12005)

* Initial commit

* Run `npx create-next-app@13 next-blog`

* Install MDX packages

Following: 77b5f79a4d/packages/next-mdx/readme.md

* Add MDX to Next

* Allow Next to handle `.md` and `.mdx` files.

* Add VSCode extension recommendation

* Disabled TypeScript strict mode for now

* Add prettier

* Apply Prettier to all files

* Make sure to use correct Node version

* Add basic implementation for `MDXRemote`

* Add experimental Rust MDX parser

* Add `/public`

* Add SASS support

* Remove default pages and styling

* Convert to module

This allows to use `import/export` syntax

* Add import for custom components

* Add ability to load plugins

* Extract function

This will make the next commit easier to read

* Allow to handle directories for page creation

* Refactoring

* Allow to parse subfolders for pages

* Extract logic

* Redirect `index.mdx` to parent directory

* Disabled ESLint during builds

* Disabled typescript during build

* Remove Gatsby from `README.md`

* Rephrase Docker part of `README.md`

* Update project structure in `README.md`

* Move and rename plugins

* Update plugin for wrapping sections

* Add dependencies for  plugin

* Use  plugin

* Rename wrapper type

* Simplify unnessary adding of id to sections

The slugified section ids are useless, because they can not be referenced anywhere anyway. The navigation only works if the section has the same id as the heading.

* Add plugin for custom attributes on Markdown elements

* Add plugin to readd support for tables

* Add plugin to fix problem with wrapped images

For more details see this issue: https://github.com/mdx-js/mdx/issues/1798

* Add necessary meta data to pages

* Install necessary dependencies

* Remove outdated MDX handling

* Remove reliance on `InlineList`

* Use existing Remark components

* Remove unallowed heading

Before `h1` components where not overwritten and would never have worked and they aren't used anywhere either.

* Add missing components to MDX

* Add correct styling

* Fix broken list

* Fix broken CSS classes

* Implement layout

* Fix links

* Fix broken images

* Fix pattern image

* Fix heading attributes

* Rename heading attribute

`new` was causing some weird issue, so renaming it to `version`

* Update comment syntax in MDX

* Merge imports

* Fix markdown rendering inside components

* Add model pages

* Simplify anchors

* Fix default value for theme

* Add Universe index page

* Add Universe categories

* Add Universe projects

* Fix Next problem with copy

Next complains when the server renders something different then the client, therfor we move the differing logic to `useEffect`

* Fix improper component nesting

Next doesn't allow block elements inside a `<p>`

* Replace landing page MDX with page component

* Remove inlined iframe content

* Remove ability to inline HTML content in iFrames

* Remove MDX imports

* Fix problem with image inside link in MDX

* Escape character for MDX

* Fix unescaped characters in MDX

* Fix headings with logo

* Allow to export static HTML pages

* Add prebuild script

This command is automatically run by Next

* Replace `svg-loader` with `react-inlinesvg`

`svg-loader` is no longer maintained

* Fix ESLint `react-hooks/exhaustive-deps`

* Fix dropdowns

* Change code language from `cli` to `bash`

* Remove unnessary language `none`

* Fix invalid code language

`markdown_` with an underscore was used to basically turn of syntax highlighting, but using unknown languages know throws an error.

* Enable code blocks plugin

* Readd `InlineCode` component

MDX2 removed the `inlineCode` component

> The special component name `inlineCode` was removed, we recommend to use `pre` for the block version of code, and code for both the block and inline versions

Source: https://mdxjs.com/migrating/v2/#update-mdx-content

* Remove unused code

* Extract function to own file

* Fix code syntax highlighting

* Update syntax for code block meta data

* Remove unused prop

* Fix internal link recognition

There is a problem with regex between Node and browser, and since Next runs the component on both, this create an error.

`Prop `rel` did not match. Server: "null" Client: "noopener nofollow noreferrer"`

This simplifies the implementation and fixes the above error.

* Replace `react-helmet` with `next/head`

* Fix `className` problem for JSX component

* Fix broken bold markdown

* Convert file to `.mjs` to be used by Node process

* Add plugin to replace strings

* Fix custom table row styling

* Fix problem with `span` inside inline `code`

React doesn't allow a `span` inside an inline `code` element and throws an error in dev mode.

* Add `_document` to be able to customize `<html>` and `<body>`

* Add `lang="en"`

* Store Netlify settings in file

This way we don't need to update via Netlify UI, which can be tricky if changing build settings.

* Add sitemap

* Add Smartypants

* Add PWA support

* Add `manifest.webmanifest`

* Fix bug with anchor links after reloading

There was no need for the previous implementation, since the browser handles this nativly. Additional the manual scrolling into view was actually broken, because the heading would disappear behind the menu bar.

* Rename custom event

I was googeling for ages to find out what kind of event `inview` is, only to figure out it was a custom event with a name that sounds pretty much like a native one. 🫠

* Fix missing comment syntax highlighting

* Refactor Quickstart component

The previous implementation was hidding the irrelevant lines via data-props and dynamically generated CSS. This created problems with Next and was also hard to follow. CSS was used to do what React is supposed to handle.

The new implementation simplfy filters the list of children (React elements) via their props.

* Fix syntax highlighting for Training Quickstart

* Unify code rendering

* Improve error logging in Juniper

* Fix Juniper component

* Automatically generate "Read Next" link

* Add Plausible

* Use recent DocSearch component and adjust styling

* Fix images

* Turn of image optimization

> Image Optimization using Next.js' default loader is not compatible with `next export`.

We currently deploy to Netlify via `next export`

* Dont build pages starting with `_`

* Remove unused files

* Add Next plugin to Netlify

* Fix button layout

MDX automatically adds `p` tags around text on a new line and Prettier wants to put the text on a new line. Hacking with JSX string.

* Add 404 page

* Apply Prettier

* Update Prettier for `package.json`

Next sometimes wants to patch `package-lock.json`. The old Prettier setting indended with 4 spaces, but Next always indends with 2 spaces. Since `npm install` automatically uses the indendation from `package.json` for `package-lock.json` and to avoid the format switching back and forth, both files are now set to 2 spaces.

* Apply Next patch to `package-lock.json`

When starting the dev server Next would warn `warn  - Found lockfile missing swc dependencies, patching...` and update the `package-lock.json`. These are the patched changes.

* fix link

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* small backslash fixes

* adjust to new style

Co-authored-by: Marcus Blättermann <marcus@essenmitsosse.de>
This commit is contained in:
Sofie Van Landeghem 2023-01-11 17:30:07 +01:00 committed by GitHub
parent e0168ccce9
commit 554df9ef20
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
241 changed files with 26957 additions and 34416 deletions

3
website/.eslintrc.json Normal file
View File

@ -0,0 +1,3 @@
{
"extends": "next/core-web-vitals"
}

44
website/.gitignore vendored Normal file
View File

@ -0,0 +1,44 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
# dependencies
/node_modules
/.pnp
.pnp.js
# testing
/coverage
# next.js
/.next/
/out/
# production
/build
# misc
.DS_Store
*.pem
# debug
npm-debug.log*
yarn-debug.log*
yarn-error.log*
.pnpm-debug.log*
# local env files
.env*.local
# vercel
.vercel
# typescript
*.tsbuildinfo
next-env.d.ts
!.vscode/extensions.json
!public
public/robots.txt
public/sitemap*
public/sw.js*
public/workbox*

1
website/.nvmrc Normal file
View File

@ -0,0 +1 @@
18

1
website/.prettierignore Normal file
View File

@ -0,0 +1 @@
.next

View File

@ -20,12 +20,11 @@
} }
}, },
{ {
"files": "*.md", "files": ["package.json", "package-lock.json"],
"options": { "options": {
"tabWidth": 2, "tabWidth": 2,
"printWidth": 80, "printWidth": 80,
"proseWrap": "always", "proseWrap": "always"
"htmlWhitespaceSensitivity": "strict"
} }
}, },
{ {

8
website/.vscode/extensions.json vendored Normal file
View File

@ -0,0 +1,8 @@
{
"recommendations": [
"dbaeumer.vscode-eslint",
"unifiedjs.vscode-mdx",
"esbenp.prettier-vscode",
"syler.sass-indented"
]
}

View File

@ -7,17 +7,16 @@ The styleguide for the spaCy website is available at
## Setup and installation ## Setup and installation
Before running the setup, make sure your versions of
[Node](https://nodejs.org/en/) and [npm](https://www.npmjs.com/) are up to date.
Node v10.15 or later is required.
```bash ```bash
# Clone the repository # Clone the repository
git clone https://github.com/explosion/spaCy git clone https://github.com/explosion/spaCy
cd spaCy/website cd spaCy/website
# Install Gatsby's command-line tool # Switch to the correct Node version
npm install --global gatsby-cli #
# If you don't have NVM and don't want to use it, you can manually switch to the Node version
# stated in /.nvmrc and skip this step
nvm use
# Install the dependencies # Install the dependencies
npm install npm install
@ -36,8 +35,7 @@ file in the root defines the settings used in this codebase.
## Building & developing the site with Docker ## Building & developing the site with Docker
Sometimes it's hard to get a local environment working due to rapid updates to While it shouldn't be necessary and is not recommended you can run this site in a Docker container.
node dependencies, so it may be easier to use docker for building the docs.
If you'd like to do this, **be sure you do _not_ include your local If you'd like to do this, **be sure you do _not_ include your local
`node_modules` folder**, since there are some dependencies that need to be built `node_modules` folder**, since there are some dependencies that need to be built
@ -76,12 +74,14 @@ bit of time.
```yaml ```yaml
├── docs # the actual markdown content ├── docs # the actual markdown content
├── meta # JSON-formatted site metadata ├── meta # JSON-formatted site metadata
| ├── dynamicMeta.js # At build time generated meta data
| ├── languages.json # supported languages and statistical models | ├── languages.json # supported languages and statistical models
| ├── sidebars.json # sidebar navigations for different sections | ├── sidebars.json # sidebar navigations for different sections
| ├── site.json # general site metadata | ├── site.json # general site metadata
| ├── type-annotations.json # Type annotations | ├── type-annotations.json # Type annotations
| └── universe.json # data for the spaCy universe section | └── universe.json # data for the spaCy universe section
├── public # compiled site ├── pages # Next router pages
├── public # static images and other assets
├── setup # Jinja setup ├── setup # Jinja setup
├── src # source ├── src # source
| ├── components # React components | ├── components # React components
@ -96,9 +96,11 @@ bit of time.
| | └── universe.js # layout templates for universe | | └── universe.js # layout templates for universe
| └── widgets # non-reusable components with content, e.g. changelog | └── widgets # non-reusable components with content, e.g. changelog
├── .eslintrc.json # ESLint config file ├── .eslintrc.json # ESLint config file
├── .nvmrc # NVM config file
| # (to support "nvm use" to switch to correct Node version)
|
├── .prettierrc # Prettier config file ├── .prettierrc # Prettier config file
├── gatsby-browser.js # browser-specific hooks for Gatsby ├── next.config.mjs # Next config file
├── gatsby-config.js # Gatsby configuration ├── package.json # package settings and dependencies
├── gatsby-node.js # Node-specific hooks for Gatsby └── tsconfig.json # TypeScript config file
└── package.json # package settings and dependencies
``` ```

View File

@ -2,9 +2,10 @@
# spaCy Universe # spaCy Universe
The [spaCy Universe](https://spacy.io/universe) collects the many great resources developed with or for spaCy. It The [spaCy Universe](https://spacy.io/universe) collects the many great
includes standalone packages, plugins, extensions, educational materials, resources developed with or for spaCy. It includes standalone packages, plugins,
operational utilities and bindings for other languages. extensions, educational materials, operational utilities and bindings for other
languages.
If you have a project that you want the spaCy community to make use of, you can If you have a project that you want the spaCy community to make use of, you can
suggest it by submitting a pull request to this repository. The Universe suggest it by submitting a pull request to this repository. The Universe
@ -18,26 +19,35 @@ discussion forum.
### Projects ### Projects
✅ Libraries and packages should be **open-source** (with a user-friendly license) and at least somewhat **documented** (e.g. a simple `README` with usage instructions). ✅ Libraries and packages should be **open-source** (with a user-friendly
license) and at least somewhat **documented** (e.g. a simple `README` with usage
instructions).
✅ We're happy to include work in progress and prereleases, but we'd like to keep the emphasis on projects that should be useful to the community **right away**. ✅ We're happy to include work in progress and prereleases, but we'd like to
keep the emphasis on projects that should be useful to the community **right
away**.
✅ Demos and visualizers should be available via a **public URL**. ✅ Demos and visualizers should be available via a **public URL**.
### Educational Materials ### Educational Materials
✅ Books should be **available for purchase or download** (not just pre-order). Ebooks and self-published books are fine, too, if they include enough substantial content. ✅ Books should be **available for purchase or download** (not just pre-order).
Ebooks and self-published books are fine, too, if they include enough
substantial content.
✅ The `"url"` of book entries should either point to the publisher's website or a reseller of your choice (ideally one that ships worldwide or as close as possible). ✅ The `"url"` of book entries should either point to the publisher's website or
a reseller of your choice (ideally one that ships worldwide or as close as
possible).
✅ If an online course is only available behind a paywall, it should at least have a **free excerpt** or chapter available, so users know what to expect. ✅ If an online course is only available behind a paywall, it should at least
have a **free excerpt** or chapter available, so users know what to expect.
## JSON format ## JSON format
To add a project, fork this repository, edit the [`universe.json`](meta/universe.json) To add a project, fork this repository, edit the
and add an object of the following format to the list of `"resources"`. Before [`universe.json`](meta/universe.json) and add an object of the following format
you submit your pull request, make sure to use a linter to verify that your to the list of `"resources"`. Before you submit your pull request, make sure to
markup is correct. use a linter to verify that your markup is correct.
```json ```json
{ {
@ -70,7 +80,7 @@ markup is correct.
``` ```
| Field | Type | Description | | Field | Type | Description |
| --- | --- | --- | | --------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | string | Unique ID of the project. | | `id` | string | Unique ID of the project. |
| `title` | string | Project title. If not set, the `id` will be used as the display title. | | `title` | string | Project title. If not set, the `id` will be used as the display title. |
| `slogan` | string | A short description of the project. Displayed in the overview and under the title. | | `slogan` | string | A short description of the project. Displayed in the overview and under the title. |
@ -89,6 +99,6 @@ markup is correct.
| `tags` | list | Still experimental and not used for filtering: one or more tags to assign to project. | | `tags` | list | Still experimental and not used for filtering: one or more tags to assign to project. |
To separate them from the projects, educational materials also specify To separate them from the projects, educational materials also specify
`"type": "education`. Books can also set a `"cover"` field containing a URL `"type": "education`. Books can also set a `"cover"` field containing a URL to a
to a cover image. If available, it's used in the overview and displayed on cover image. If available, it's used in the overview and displayed on the
the individual book page. individual book page.

View File

@ -26,9 +26,9 @@ part of the [training config](/usage/training#custom-functions). Also see the
usage documentation on usage documentation on
[layers and model architectures](/usage/layers-architectures). [layers and model architectures](/usage/layers-architectures).
## Tok2Vec architectures {#tok2vec-arch source="spacy/ml/models/tok2vec.py"} ## Tok2Vec architectures {id="tok2vec-arch",source="spacy/ml/models/tok2vec.py"}
### spacy.Tok2Vec.v2 {#Tok2Vec} ### spacy.Tok2Vec.v2 {id="Tok2Vec"}
> #### Example config > #### Example config
> >
@ -56,7 +56,7 @@ blog post for background.
| `encode` | Encode context into the embeddings, using an architecture such as a CNN, BiLSTM or transformer. For example, [MaxoutWindowEncoder](/api/architectures#MaxoutWindowEncoder). ~~Model[List[Floats2d], List[Floats2d]]~~ | | `encode` | Encode context into the embeddings, using an architecture such as a CNN, BiLSTM or transformer. For example, [MaxoutWindowEncoder](/api/architectures#MaxoutWindowEncoder). ~~Model[List[Floats2d], List[Floats2d]]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.HashEmbedCNN.v2 {#HashEmbedCNN} ### spacy.HashEmbedCNN.v2 {id="HashEmbedCNN"}
> #### Example Config > #### Example Config
> >
@ -89,7 +89,7 @@ consisting of a CNN and a layer-normalized maxout activation function.
| `pretrained_vectors` | Whether to also use static vectors. ~~bool~~ | | `pretrained_vectors` | Whether to also use static vectors. ~~bool~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.Tok2VecListener.v1 {#Tok2VecListener} ### spacy.Tok2VecListener.v1 {id="Tok2VecListener"}
> #### Example config > #### Example config
> >
@ -139,7 +139,7 @@ the `Tok2Vec` component.
| `upstream` | A string to identify the "upstream" `Tok2Vec` component to communicate with. By default, the upstream name is the wildcard string `"*"`, but you could also specify the name of the `Tok2Vec` component. You'll almost never have multiple upstream `Tok2Vec` components, so the wildcard string will almost always be fine. ~~str~~ | | `upstream` | A string to identify the "upstream" `Tok2Vec` component to communicate with. By default, the upstream name is the wildcard string `"*"`, but you could also specify the name of the `Tok2Vec` component. You'll almost never have multiple upstream `Tok2Vec` components, so the wildcard string will almost always be fine. ~~str~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.MultiHashEmbed.v2 {#MultiHashEmbed} ### spacy.MultiHashEmbed.v2 {id="MultiHashEmbed"}
> #### Example config > #### Example config
> >
@ -170,7 +170,7 @@ updated).
| `include_static_vectors` | Whether to also use static word vectors. Requires a vectors table to be loaded in the [`Doc`](/api/doc) objects' vocab. ~~bool~~ | | `include_static_vectors` | Whether to also use static word vectors. Requires a vectors table to be loaded in the [`Doc`](/api/doc) objects' vocab. ~~bool~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.CharacterEmbed.v2 {#CharacterEmbed} ### spacy.CharacterEmbed.v2 {id="CharacterEmbed"}
> #### Example config > #### Example config
> >
@ -207,7 +207,7 @@ network to construct a single vector to represent the information.
| `nC` | The number of UTF-8 bytes to embed per word. Recommended values are between `3` and `8`, although it may depend on the length of words in the language. ~~int~~ | | `nC` | The number of UTF-8 bytes to embed per word. Recommended values are between `3` and `8`, although it may depend on the length of words in the language. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.MaxoutWindowEncoder.v2 {#MaxoutWindowEncoder} ### spacy.MaxoutWindowEncoder.v2 {id="MaxoutWindowEncoder"}
> #### Example config > #### Example config
> >
@ -231,7 +231,7 @@ and residual connections.
| `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ | | `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ |
### spacy.MishWindowEncoder.v2 {#MishWindowEncoder} ### spacy.MishWindowEncoder.v2 {id="MishWindowEncoder"}
> #### Example config > #### Example config
> >
@ -254,7 +254,7 @@ and residual connections.
| `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ | | `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ |
### spacy.TorchBiLSTMEncoder.v1 {#TorchBiLSTMEncoder} ### spacy.TorchBiLSTMEncoder.v1 {id="TorchBiLSTMEncoder"}
> #### Example config > #### Example config
> >
@ -276,7 +276,7 @@ Encode context using bidirectional LSTM layers. Requires
| `dropout` | Creates a Dropout layer on the outputs of each LSTM layer except the last layer. Set to 0.0 to disable this functionality. ~~float~~ | | `dropout` | Creates a Dropout layer on the outputs of each LSTM layer except the last layer. Set to 0.0 to disable this functionality. ~~float~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ |
### spacy.StaticVectors.v2 {#StaticVectors} ### spacy.StaticVectors.v2 {id="StaticVectors"}
> #### Example config > #### Example config
> >
@ -306,7 +306,7 @@ mapped to a zero vector. See the documentation on
| `key_attr` | Defaults to `"ORTH"`. ~~str~~ | | `key_attr` | Defaults to `"ORTH"`. ~~str~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Ragged]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Ragged]~~ |
### spacy.FeatureExtractor.v1 {#FeatureExtractor} ### spacy.FeatureExtractor.v1 {id="FeatureExtractor"}
> #### Example config > #### Example config
> >
@ -324,7 +324,7 @@ of feature names to extract, which should refer to token attributes.
| `columns` | The token attributes to extract. ~~List[Union[int, str]]~~ | | `columns` | The token attributes to extract. ~~List[Union[int, str]]~~ |
| **CREATES** | The created feature extraction layer. ~~Model[List[Doc], List[Ints2d]]~~ | | **CREATES** | The created feature extraction layer. ~~Model[List[Doc], List[Ints2d]]~~ |
## Transformer architectures {#transformers source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/architectures.py"} ## Transformer architectures {id="transformers",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/architectures.py"}
The following architectures are provided by the package The following architectures are provided by the package
[`spacy-transformers`](https://github.com/explosion/spacy-transformers). See the [`spacy-transformers`](https://github.com/explosion/spacy-transformers). See the
@ -341,7 +341,7 @@ for details and system requirements.
</Infobox> </Infobox>
### spacy-transformers.TransformerModel.v3 {#TransformerModel} ### spacy-transformers.TransformerModel.v3 {id="TransformerModel"}
> #### Example Config > #### Example Config
> >
@ -404,7 +404,7 @@ The other arguments are shared between all versions.
</Accordion> </Accordion>
### spacy-transformers.TransformerListener.v1 {#TransformerListener} ### spacy-transformers.TransformerListener.v1 {id="TransformerListener"}
> #### Example Config > #### Example Config
> >
@ -434,7 +434,7 @@ a single token vector given zero or more wordpiece vectors.
| `upstream` | A string to identify the "upstream" `Transformer` component to communicate with. By default, the upstream name is the wildcard string `"*"`, but you could also specify the name of the `Transformer` component. You'll almost never have multiple upstream `Transformer` components, so the wildcard string will almost always be fine. ~~str~~ | | `upstream` | A string to identify the "upstream" `Transformer` component to communicate with. By default, the upstream name is the wildcard string `"*"`, but you could also specify the name of the `Transformer` component. You'll almost never have multiple upstream `Transformer` components, so the wildcard string will almost always be fine. ~~str~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy-transformers.Tok2VecTransformer.v3 {#Tok2VecTransformer} ### spacy-transformers.Tok2VecTransformer.v3 {id="Tok2VecTransformer"}
> #### Example Config > #### Example Config
> >
@ -481,7 +481,7 @@ The other arguments are shared between all versions.
</Accordion> </Accordion>
## Pretraining architectures {#pretrain source="spacy/ml/models/multi_task.py"} ## Pretraining architectures {id="pretrain",source="spacy/ml/models/multi_task.py"}
The spacy `pretrain` command lets you initialize a `Tok2Vec` layer in your The spacy `pretrain` command lets you initialize a `Tok2Vec` layer in your
pipeline with information from raw text. To this end, additional layers are pipeline with information from raw text. To this end, additional layers are
@ -494,7 +494,7 @@ BERT.
For more information, see the section on For more information, see the section on
[pretraining](/usage/embeddings-transformers#pretraining). [pretraining](/usage/embeddings-transformers#pretraining).
### spacy.PretrainVectors.v1 {#pretrain_vectors} ### spacy.PretrainVectors.v1 {id="pretrain_vectors"}
> #### Example config > #### Example config
> >
@ -525,7 +525,7 @@ vectors.
| `loss` | The loss function can be either "cosine" or "L2". We typically recommend to use "cosine". ~~~str~~ | | `loss` | The loss function can be either "cosine" or "L2". We typically recommend to use "cosine". ~~~str~~ |
| **CREATES** | A callable function that can create the Model, given the `vocab` of the pipeline and the `tok2vec` layer to pretrain. ~~Callable[[Vocab, Model], Model]~~ | | **CREATES** | A callable function that can create the Model, given the `vocab` of the pipeline and the `tok2vec` layer to pretrain. ~~Callable[[Vocab, Model], Model]~~ |
### spacy.PretrainCharacters.v1 {#pretrain_chars} ### spacy.PretrainCharacters.v1 {id="pretrain_chars"}
> #### Example config > #### Example config
> >
@ -551,9 +551,9 @@ for a Tok2Vec layer.
| `n_characters` | The window of characters - e.g. if `n_characters = 2`, the model will try to predict the first two and last two characters of the word. ~~int~~ | | `n_characters` | The window of characters - e.g. if `n_characters = 2`, the model will try to predict the first two and last two characters of the word. ~~int~~ |
| **CREATES** | A callable function that can create the Model, given the `vocab` of the pipeline and the `tok2vec` layer to pretrain. ~~Callable[[Vocab, Model], Model]~~ | | **CREATES** | A callable function that can create the Model, given the `vocab` of the pipeline and the `tok2vec` layer to pretrain. ~~Callable[[Vocab, Model], Model]~~ |
## Parser & NER architectures {#parser} ## Parser & NER architectures {id="parser"}
### spacy.TransitionBasedParser.v2 {#TransitionBasedParser source="spacy/ml/models/parser.py"} ### spacy.TransitionBasedParser.v2 {id="TransitionBasedParser",source="spacy/ml/models/parser.py"}
> #### Example Config > #### Example Config
> >
@ -612,9 +612,9 @@ same signature, but the `use_upper` argument was `True` by default.
</Accordion> </Accordion>
## Tagging architectures {#tagger source="spacy/ml/models/tagger.py"} ## Tagging architectures {id="tagger",source="spacy/ml/models/tagger.py"}
### spacy.Tagger.v2 {#Tagger} ### spacy.Tagger.v2 {id="Tagger"}
> #### Example Config > #### Example Config
> >
@ -648,7 +648,7 @@ The other arguments are shared between all versions.
</Accordion> </Accordion>
## Text classification architectures {#textcat source="spacy/ml/models/textcat.py"} ## Text classification architectures {id="textcat",source="spacy/ml/models/textcat.py"}
A text classification architecture needs to take a [`Doc`](/api/doc) as input, A text classification architecture needs to take a [`Doc`](/api/doc) as input,
and produce a score for each potential label class. Textcat challenges can be and produce a score for each potential label class. Textcat challenges can be
@ -672,7 +672,7 @@ single-label use-cases where `exclusive_classes = true`, while the
</Infobox> </Infobox>
### spacy.TextCatEnsemble.v2 {#TextCatEnsemble} ### spacy.TextCatEnsemble.v2 {id="TextCatEnsemble"}
> #### Example Config > #### Example Config
> >
@ -737,7 +737,7 @@ but used an internal `tok2vec` instead of taking it as argument:
</Accordion> </Accordion>
### spacy.TextCatCNN.v2 {#TextCatCNN} ### spacy.TextCatCNN.v2 {id="TextCatCNN"}
> #### Example Config > #### Example Config
> >
@ -777,7 +777,7 @@ after training.
</Accordion> </Accordion>
### spacy.TextCatBOW.v2 {#TextCatBOW} ### spacy.TextCatBOW.v2 {id="TextCatBOW"}
> #### Example Config > #### Example Config
> >
@ -809,9 +809,9 @@ after training.
</Accordion> </Accordion>
## Span classification architectures {#spancat source="spacy/ml/models/spancat.py"} ## Span classification architectures {id="spancat",source="spacy/ml/models/spancat.py"}
### spacy.SpanCategorizer.v1 {#SpanCategorizer} ### spacy.SpanCategorizer.v1 {id="SpanCategorizer"}
> #### Example Config > #### Example Config
> >
@ -848,7 +848,7 @@ single vector, and a scorer model to map the vectors to probabilities.
| `scorer` | The scorer model. ~~Model[Floats2d, Floats2d]~~ | | `scorer` | The scorer model. ~~Model[Floats2d, Floats2d]~~ |
| **CREATES** | The model using the architecture. ~~Model[Tuple[List[Doc], Ragged], Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[Tuple[List[Doc], Ragged], Floats2d]~~ |
### spacy.mean_max_reducer.v1 {#mean_max_reducer} ### spacy.mean_max_reducer.v1 {id="mean_max_reducer"}
Reduce sequences by concatenating their mean and max pooled vectors, and then Reduce sequences by concatenating their mean and max pooled vectors, and then
combine the concatenated vectors with a hidden layer. combine the concatenated vectors with a hidden layer.
@ -857,7 +857,7 @@ combine the concatenated vectors with a hidden layer.
| ------------- | ------------------------------------- | | ------------- | ------------------------------------- |
| `hidden_size` | The size of the hidden layer. ~~int~~ | | `hidden_size` | The size of the hidden layer. ~~int~~ |
## Entity linking architectures {#entitylinker source="spacy/ml/models/entity_linker.py"} ## Entity linking architectures {id="entitylinker",source="spacy/ml/models/entity_linker.py"}
An [`EntityLinker`](/api/entitylinker) component disambiguates textual mentions An [`EntityLinker`](/api/entitylinker) component disambiguates textual mentions
(tagged as named entities) to unique identifiers, grounding the named entities (tagged as named entities) to unique identifiers, grounding the named entities
@ -870,7 +870,7 @@ into the "real world". This requires 3 main components:
- A machine learning [`Model`](https://thinc.ai/docs/api-model) that picks the - A machine learning [`Model`](https://thinc.ai/docs/api-model) that picks the
most plausible ID from the set of candidates. most plausible ID from the set of candidates.
### spacy.EntityLinker.v2 {#EntityLinker} ### spacy.EntityLinker.v2 {id="EntityLinker"}
> #### Example Config > #### Example Config
> >
@ -899,7 +899,7 @@ The `EntityLinker` model architecture is a Thinc `Model` with a
| `nO` | Output dimension, determined by the length of the vectors encoding each entity in the KB. If the `nO` dimension is not set, the entity linking component will set it when `initialize` is called. ~~Optional[int]~~ | | `nO` | Output dimension, determined by the length of the vectors encoding each entity in the KB. If the `nO` dimension is not set, the entity linking component will set it when `initialize` is called. ~~Optional[int]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
### spacy.EmptyKB.v1 {#EmptyKB} ### spacy.EmptyKB.v1 {id="EmptyKB"}
A function that creates an empty `KnowledgeBase` from a [`Vocab`](/api/vocab) A function that creates an empty `KnowledgeBase` from a [`Vocab`](/api/vocab)
instance. This is the default when a new entity linker component is created. instance. This is the default when a new entity linker component is created.
@ -908,7 +908,7 @@ instance. This is the default when a new entity linker component is created.
| ---------------------- | ----------------------------------------------------------------------------------- | | ---------------------- | ----------------------------------------------------------------------------------- |
| `entity_vector_length` | The length of the vectors encoding each entity in the KB. Defaults to `64`. ~~int~~ | | `entity_vector_length` | The length of the vectors encoding each entity in the KB. Defaults to `64`. ~~int~~ |
### spacy.KBFromFile.v1 {#KBFromFile} ### spacy.KBFromFile.v1 {id="KBFromFile"}
A function that reads an existing `KnowledgeBase` from file. A function that reads an existing `KnowledgeBase` from file.
@ -916,7 +916,7 @@ A function that reads an existing `KnowledgeBase` from file.
| --------- | -------------------------------------------------------- | | --------- | -------------------------------------------------------- |
| `kb_path` | The location of the KB that was stored to file. ~~Path~~ | | `kb_path` | The location of the KB that was stored to file. ~~Path~~ |
### spacy.CandidateGenerator.v1 {#CandidateGenerator} ### spacy.CandidateGenerator.v1 {id="CandidateGenerator"}
A function that takes as input a [`KnowledgeBase`](/api/kb) and a A function that takes as input a [`KnowledgeBase`](/api/kb) and a
[`Span`](/api/span) object denoting a named entity, and returns a list of [`Span`](/api/span) object denoting a named entity, and returns a list of
@ -924,7 +924,7 @@ plausible [`Candidate`](/api/kb/#candidate) objects. The default
`CandidateGenerator` uses the text of a mention to find its potential aliases in `CandidateGenerator` uses the text of a mention to find its potential aliases in
the `KnowledgeBase`. Note that this function is case-dependent. the `KnowledgeBase`. Note that this function is case-dependent.
## Coreference {#coref-architectures tag="experimental"} ## Coreference {id="coref-architectures",tag="experimental"}
A [`CoreferenceResolver`](/api/coref) component identifies tokens that refer to A [`CoreferenceResolver`](/api/coref) component identifies tokens that refer to
the same entity. A [`SpanResolver`](/api/span-resolver) component infers spans the same entity. A [`SpanResolver`](/api/span-resolver) component infers spans
@ -932,7 +932,7 @@ from single tokens. Together these components can be used to reproduce
traditional coreference models. You can also omit the `SpanResolver` if working traditional coreference models. You can also omit the `SpanResolver` if working
with only token-level clusters is acceptable. with only token-level clusters is acceptable.
### spacy-experimental.Coref.v1 {#Coref tag="experimental"} ### spacy-experimental.Coref.v1 {id="Coref",tag="experimental"}
> #### Example Config > #### Example Config
> >
@ -967,7 +967,7 @@ The `Coref` model architecture is a Thinc `Model`.
| `antecedent_batch_size` | Internal batch size. ~~int~~ | | `antecedent_batch_size` | Internal batch size. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
### spacy-experimental.SpanResolver.v1 {#SpanResolver tag="experimental"} ### spacy-experimental.SpanResolver.v1 {id="SpanResolver",tag="experimental"}
> #### Example Config > #### Example Config
> >

View File

@ -2,7 +2,7 @@
title: AttributeRuler title: AttributeRuler
tag: class tag: class
source: spacy/pipeline/attributeruler.py source: spacy/pipeline/attributeruler.py
new: 3 version: 3
teaser: 'Pipeline component for rule-based token attribute assignment' teaser: 'Pipeline component for rule-based token attribute assignment'
api_string_name: attribute_ruler api_string_name: attribute_ruler
api_trainable: false api_trainable: false
@ -15,7 +15,7 @@ between attributes such as mapping fine-grained POS tags to coarse-grained POS
tags. See the [usage guide](/usage/linguistic-features/#mappings-exceptions) for tags. See the [usage guide](/usage/linguistic-features/#mappings-exceptions) for
examples. examples.
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -37,7 +37,7 @@ how the component should be configured. You can override its settings via the
%%GITHUB_SPACY/spacy/pipeline/attributeruler.py %%GITHUB_SPACY/spacy/pipeline/attributeruler.py
``` ```
## AttributeRuler.\_\_init\_\_ {#init tag="method"} ## AttributeRuler.\_\_init\_\_ {id="init",tag="method"}
Initialize the attribute ruler. Initialize the attribute ruler.
@ -56,7 +56,7 @@ Initialize the attribute ruler.
| `validate` | Whether patterns should be validated (passed to the [`Matcher`](/api/matcher#init)). Defaults to `False`. ~~bool~~ | | `validate` | Whether patterns should be validated (passed to the [`Matcher`](/api/matcher#init)). Defaults to `False`. ~~bool~~ |
| `scorer` | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attributes `"tag`", `"pos"`, `"morph"` and `"lemma"` and [`Scorer.score_token_attr_per_feat`](/api/scorer#score_token_attr_per_feat) for the attribute `"morph"`. ~~Optional[Callable]~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attributes `"tag`", `"pos"`, `"morph"` and `"lemma"` and [`Scorer.score_token_attr_per_feat`](/api/scorer#score_token_attr_per_feat) for the attribute `"morph"`. ~~Optional[Callable]~~ |
## AttributeRuler.\_\_call\_\_ {#call tag="method"} ## AttributeRuler.\_\_call\_\_ {id="call",tag="method"}
Apply the attribute ruler to a `Doc`, setting token attributes for tokens Apply the attribute ruler to a `Doc`, setting token attributes for tokens
matched by the provided patterns. matched by the provided patterns.
@ -66,7 +66,7 @@ matched by the provided patterns.
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## AttributeRuler.add {#add tag="method"} ## AttributeRuler.add {id="add",tag="method"}
Add patterns to the attribute ruler. The patterns are a list of `Matcher` Add patterns to the attribute ruler. The patterns are a list of `Matcher`
patterns and the attributes are a dict of attributes to set on the matched patterns and the attributes are a dict of attributes to set on the matched
@ -89,7 +89,7 @@ may be negative to index from the end of the span.
| `attrs` | The attributes to assign to the target token in the matched span. ~~Dict[str, Any]~~ | | `attrs` | The attributes to assign to the target token in the matched span. ~~Dict[str, Any]~~ |
| `index` | The index of the token in the matched span to modify. May be negative to index from the end of the span. Defaults to `0`. ~~int~~ | | `index` | The index of the token in the matched span to modify. May be negative to index from the end of the span. Defaults to `0`. ~~int~~ |
## AttributeRuler.add_patterns {#add_patterns tag="method"} ## AttributeRuler.add_patterns {id="add_patterns",tag="method"}
> #### Example > #### Example
> >
@ -116,7 +116,7 @@ keys `"patterns"`, `"attrs"` and `"index"`, which match the arguments of
| ---------- | -------------------------------------------------------------------------- | | ---------- | -------------------------------------------------------------------------- |
| `patterns` | The patterns to add. ~~Iterable[Dict[str, Union[List[dict], dict, int]]]~~ | | `patterns` | The patterns to add. ~~Iterable[Dict[str, Union[List[dict], dict, int]]]~~ |
## AttributeRuler.patterns {#patterns tag="property"} ## AttributeRuler.patterns {id="patterns",tag="property"}
Get all patterns that have been added to the attribute ruler in the Get all patterns that have been added to the attribute ruler in the
`patterns_dict` format accepted by `patterns_dict` format accepted by
@ -126,7 +126,7 @@ Get all patterns that have been added to the attribute ruler in the
| ----------- | -------------------------------------------------------------------------------------------- | | ----------- | -------------------------------------------------------------------------------------------- |
| **RETURNS** | The patterns added to the attribute ruler. ~~List[Dict[str, Union[List[dict], dict, int]]]~~ | | **RETURNS** | The patterns added to the attribute ruler. ~~List[Dict[str, Union[List[dict], dict, int]]]~~ |
## AttributeRuler.initialize {#initialize tag="method"} ## AttributeRuler.initialize {id="initialize",tag="method"}
Initialize the component with data and used before training to load in rules Initialize the component with data and used before training to load in rules
from a file. This method is typically called by from a file. This method is typically called by
@ -160,7 +160,7 @@ config.
| `tag_map` | The tag map that maps fine-grained tags to coarse-grained tags and morphological features. Defaults to `None`. ~~Optional[Dict[str, Dict[Union[int, str], Union[int, str]]]]~~ | | `tag_map` | The tag map that maps fine-grained tags to coarse-grained tags and morphological features. Defaults to `None`. ~~Optional[Dict[str, Dict[Union[int, str], Union[int, str]]]]~~ |
| `morph_rules` | The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Dict[Union[int, str], Union[int, str]]]]]~~ | | `morph_rules` | The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Dict[Union[int, str], Union[int, str]]]]]~~ |
## AttributeRuler.load_from_tag_map {#load_from_tag_map tag="method"} ## AttributeRuler.load_from_tag_map {id="load_from_tag_map",tag="method"}
Load attribute ruler patterns from a tag map. Load attribute ruler patterns from a tag map.
@ -168,7 +168,7 @@ Load attribute ruler patterns from a tag map.
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `tag_map` | The tag map that maps fine-grained tags to coarse-grained tags and morphological features. ~~Dict[str, Dict[Union[int, str], Union[int, str]]]~~ | | `tag_map` | The tag map that maps fine-grained tags to coarse-grained tags and morphological features. ~~Dict[str, Dict[Union[int, str], Union[int, str]]]~~ |
## AttributeRuler.load_from_morph_rules {#load_from_morph_rules tag="method"} ## AttributeRuler.load_from_morph_rules {id="load_from_morph_rules",tag="method"}
Load attribute ruler patterns from morph rules. Load attribute ruler patterns from morph rules.
@ -176,7 +176,7 @@ Load attribute ruler patterns from morph rules.
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `morph_rules` | The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. ~~Dict[str, Dict[str, Dict[Union[int, str], Union[int, str]]]]~~ | | `morph_rules` | The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. ~~Dict[str, Dict[str, Dict[Union[int, str], Union[int, str]]]]~~ |
## AttributeRuler.to_disk {#to_disk tag="method"} ## AttributeRuler.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -193,7 +193,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## AttributeRuler.from_disk {#from_disk tag="method"} ## AttributeRuler.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -211,7 +211,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `AttributeRuler` object. ~~AttributeRuler~~ | | **RETURNS** | The modified `AttributeRuler` object. ~~AttributeRuler~~ |
## AttributeRuler.to_bytes {#to_bytes tag="method"} ## AttributeRuler.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -228,7 +228,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `AttributeRuler` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `AttributeRuler` object. ~~bytes~~ |
## AttributeRuler.from_bytes {#from_bytes tag="method"} ## AttributeRuler.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -247,7 +247,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `AttributeRuler` object. ~~AttributeRuler~~ | | **RETURNS** | The `AttributeRuler` object. ~~AttributeRuler~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -41,10 +41,9 @@ from string attribute names to internal attribute IDs is stored in
The corresponding [`Token` object attributes](/api/token#attributes) can be The corresponding [`Token` object attributes](/api/token#attributes) can be
accessed using the same names in lowercase, e.g. `token.orth` or `token.length`. accessed using the same names in lowercase, e.g. `token.orth` or `token.length`.
For attributes that represent string values, the internal integer ID is For attributes that represent string values, the internal integer ID is accessed
accessed as `Token.attr`, e.g. `token.dep`, while the string value can be as `Token.attr`, e.g. `token.dep`, while the string value can be retrieved by
retrieved by appending `_` as in `token.dep_`. appending `_` as in `token.dep_`.
| Attribute | Description | | Attribute | Description |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |

View File

@ -26,7 +26,7 @@ a list of available commands, you can type `python -m spacy --help`. You can
also add the `--help` flag to any command or subcommand to see the description, also add the `--help` flag to any command or subcommand to see the description,
available arguments and usage. available arguments and usage.
## download {#download tag="command"} ## download {id="download",tag="command"}
Download [trained pipelines](/usage/models) for spaCy. The downloader finds the Download [trained pipelines](/usage/models) for spaCy. The downloader finds the
best-matching compatible version and uses `pip install` to download the Python best-matching compatible version and uses `pip install` to download the Python
@ -44,7 +44,7 @@ pipeline name to be specified with its version (e.g. `en_core_web_sm-3.0.0`).
> will also allow you to add it as a versioned package dependency to your > will also allow you to add it as a versioned package dependency to your
> project. > project.
```cli ```bash
$ python -m spacy download [model] [--direct] [--sdist] [pip_args] $ python -m spacy download [model] [--direct] [--sdist] [pip_args]
``` ```
@ -57,24 +57,24 @@ $ python -m spacy download [model] [--direct] [--sdist] [pip_args]
| pip args | Additional installation options to be passed to `pip install` when installing the pipeline package. For example, `--user` to install to the user home directory or `--no-deps` to not install package dependencies. ~~Any (option/flag)~~ | | pip args | Additional installation options to be passed to `pip install` when installing the pipeline package. For example, `--user` to install to the user home directory or `--no-deps` to not install package dependencies. ~~Any (option/flag)~~ |
| **CREATES** | The installed pipeline package in your `site-packages` directory. | | **CREATES** | The installed pipeline package in your `site-packages` directory. |
## info {#info tag="command"} ## info {id="info",tag="command"}
Print information about your spaCy installation, trained pipelines and local Print information about your spaCy installation, trained pipelines and local
setup, and generate [Markdown](https://en.wikipedia.org/wiki/Markdown)-formatted setup, and generate [Markdown](https://en.wikipedia.org/wiki/Markdown)-formatted
markup to copy-paste into markup to copy-paste into
[GitHub issues](https://github.com/explosion/spaCy/issues). [GitHub issues](https://github.com/explosion/spaCy/issues).
```cli ```bash
$ python -m spacy info [--markdown] [--silent] [--exclude] $ python -m spacy info [--markdown] [--silent] [--exclude]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy info en_core_web_lg --markdown > $ python -m spacy info en_core_web_lg --markdown
> ``` > ```
```cli ```bash
$ python -m spacy info [model] [--markdown] [--silent] [--exclude] $ python -m spacy info [model] [--markdown] [--silent] [--exclude]
``` ```
@ -88,7 +88,7 @@ $ python -m spacy info [model] [--markdown] [--silent] [--exclude]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **PRINTS** | Information about your spaCy installation. | | **PRINTS** | Information about your spaCy installation. |
## validate {#validate new="2" tag="command"} ## validate {id="validate",version="2",tag="command"}
Find all trained pipeline packages installed in the current environment and Find all trained pipeline packages installed in the current environment and
check whether they are compatible with the currently installed version of spaCy. check whether they are compatible with the currently installed version of spaCy.
@ -103,7 +103,7 @@ compatible versions and command for updating are shown.
> suite, to ensure all packages are up to date before proceeding. If > suite, to ensure all packages are up to date before proceeding. If
> incompatible packages are found, it will return `1`. > incompatible packages are found, it will return `1`.
```cli ```bash
$ python -m spacy validate $ python -m spacy validate
``` ```
@ -111,12 +111,12 @@ $ python -m spacy validate
| ---------- | -------------------------------------------------------------------- | | ---------- | -------------------------------------------------------------------- |
| **PRINTS** | Details about the compatibility of your installed pipeline packages. | | **PRINTS** | Details about the compatibility of your installed pipeline packages. |
## init {#init new="3"} ## init {id="init",version="3"}
The `spacy init` CLI includes helpful commands for initializing training config The `spacy init` CLI includes helpful commands for initializing training config
files and pipeline directories. files and pipeline directories.
### init config {#init-config new="3" tag="command"} ### init config {id="init-config",version="3",tag="command"}
Initialize and save a [`config.cfg` file](/usage/training#config) using the Initialize and save a [`config.cfg` file](/usage/training#config) using the
**recommended settings** for your use case. It works just like the **recommended settings** for your use case. It works just like the
@ -128,11 +128,11 @@ customize those settings in your config file later.
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy init config config.cfg --lang en --pipeline ner,textcat --optimize accuracy > $ python -m spacy init config config.cfg --lang en --pipeline ner,textcat --optimize accuracy
> ``` > ```
```cli ```bash
$ python -m spacy init config [output_file] [--lang] [--pipeline] [--optimize] [--gpu] [--pretraining] [--force] $ python -m spacy init config [output_file] [--lang] [--pipeline] [--optimize] [--gpu] [--pretraining] [--force]
``` ```
@ -148,7 +148,7 @@ $ python -m spacy init config [output_file] [--lang] [--pipeline] [--optimize] [
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | The config file for training. | | **CREATES** | The config file for training. |
### init fill-config {#init-fill-config new="3"} ### init fill-config {id="init-fill-config",version="3"}
Auto-fill a partial [.cfg file](/usage/training#config) with **all default Auto-fill a partial [.cfg file](/usage/training#config) with **all default
values**, e.g. a config generated with the values**, e.g. a config generated with the
@ -162,15 +162,15 @@ validation error with more details.
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy init fill-config base.cfg config.cfg --diff > $ python -m spacy init fill-config base.cfg config.cfg --diff
> ``` > ```
> >
> #### Example diff > #### Example diff
> >
> ![Screenshot of visual diff in terminal](../images/cli_init_fill-config_diff.jpg) > ![Screenshot of visual diff in terminal](/images/cli_init_fill-config_diff.jpg)
```cli ```bash
$ python -m spacy init fill-config [base_path] [output_file] [--diff] $ python -m spacy init fill-config [base_path] [output_file] [--diff]
``` ```
@ -184,7 +184,7 @@ $ python -m spacy init fill-config [base_path] [output_file] [--diff]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | Complete and auto-filled config file for training. | | **CREATES** | Complete and auto-filled config file for training. |
### init vectors {#init-vectors new="3" tag="command"} ### init vectors {id="init-vectors",version="3",tag="command"}
Convert [word vectors](/usage/linguistic-features#vectors-similarity) for use Convert [word vectors](/usage/linguistic-features#vectors-similarity) for use
with spaCy. Will export an `nlp` object that you can use in the with spaCy. Will export an `nlp` object that you can use in the
@ -199,7 +199,7 @@ This functionality was previously available as part of the command `init-model`.
</Infobox> </Infobox>
```cli ```bash
$ python -m spacy init vectors [lang] [vectors_loc] [output_dir] [--prune] [--truncate] [--name] [--verbose] $ python -m spacy init vectors [lang] [vectors_loc] [output_dir] [--prune] [--truncate] [--name] [--verbose]
``` ```
@ -216,7 +216,7 @@ $ python -m spacy init vectors [lang] [vectors_loc] [output_dir] [--prune] [--tr
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | A spaCy pipeline directory containing the vocab and vectors. | | **CREATES** | A spaCy pipeline directory containing the vocab and vectors. |
### init labels {#init-labels new="3" tag="command"} ### init labels {id="init-labels",version="3",tag="command"}
Generate JSON files for the labels in the data. This helps speed up the training Generate JSON files for the labels in the data. This helps speed up the training
process, since spaCy won't have to preprocess the data to extract the labels. process, since spaCy won't have to preprocess the data to extract the labels.
@ -234,7 +234,7 @@ After generating the labels, you can provide them to components that accept a
> path = "corpus/labels/ner.json > path = "corpus/labels/ner.json
> ``` > ```
```cli ```bash
$ python -m spacy init labels [config_path] [output_path] [--code] [--verbose] [--gpu-id] [overrides] $ python -m spacy init labels [config_path] [output_path] [--code] [--verbose] [--gpu-id] [overrides]
``` ```
@ -249,7 +249,7 @@ $ python -m spacy init labels [config_path] [output_path] [--code] [--verbose] [
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ | | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ |
| **CREATES** | The label files. | | **CREATES** | The label files. |
## convert {#convert tag="command"} ## convert {id="convert",tag="command"}
Convert files into spaCy's Convert files into spaCy's
[binary training data format](/api/data-formats#binary-training), a serialized [binary training data format](/api/data-formats#binary-training), a serialized
@ -257,7 +257,7 @@ Convert files into spaCy's
management functions. The converter can be specified on the command line, or management functions. The converter can be specified on the command line, or
chosen based on the file extension of the input file. chosen based on the file extension of the input file.
```cli ```bash
$ python -m spacy convert [input_file] [output_dir] [--converter] [--file-type] [--n-sents] [--seg-sents] [--base] [--morphology] [--merge-subtokens] [--ner-map] [--lang] $ python -m spacy convert [input_file] [output_dir] [--converter] [--file-type] [--n-sents] [--seg-sents] [--base] [--morphology] [--merge-subtokens] [--ner-map] [--lang]
``` ```
@ -278,7 +278,7 @@ $ python -m spacy convert [input_file] [output_dir] [--converter] [--file-type]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | Binary [`DocBin`](/api/docbin) training data that can be used with [`spacy train`](/api/cli#train). | | **CREATES** | Binary [`DocBin`](/api/docbin) training data that can be used with [`spacy train`](/api/cli#train). |
### Converters {#converters} ### Converters {id="converters"}
| ID | Description | | ID | Description |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@ -288,12 +288,12 @@ $ python -m spacy convert [input_file] [output_dir] [--converter] [--file-type]
| `ner` / `conll` | NER with IOB/IOB2/BILUO tags, one token per line with columns separated by whitespace. The first column is the token and the final column is the NER tag. Sentences are separated by blank lines and documents are separated by the line `-DOCSTART- -X- O O`. Supports CoNLL 2003 NER format. See [sample data](%%GITHUB_SPACY/extra/example_data/ner_example_data). | | `ner` / `conll` | NER with IOB/IOB2/BILUO tags, one token per line with columns separated by whitespace. The first column is the token and the final column is the NER tag. Sentences are separated by blank lines and documents are separated by the line `-DOCSTART- -X- O O`. Supports CoNLL 2003 NER format. See [sample data](%%GITHUB_SPACY/extra/example_data/ner_example_data). |
| `iob` | NER with IOB/IOB2/BILUO tags, one sentence per line with tokens separated by whitespace and annotation separated by `\|`, either `word\|B-ENT`or`word\|POS\|B-ENT`. See [sample data](%%GITHUB_SPACY/extra/example_data/ner_example_data). | | `iob` | NER with IOB/IOB2/BILUO tags, one sentence per line with tokens separated by whitespace and annotation separated by `\|`, either `word\|B-ENT`or`word\|POS\|B-ENT`. See [sample data](%%GITHUB_SPACY/extra/example_data/ner_example_data). |
## debug {#debug new="3"} ## debug {id="debug",version="3"}
The `spacy debug` CLI includes helpful commands for debugging and profiling your The `spacy debug` CLI includes helpful commands for debugging and profiling your
configs, data and implementations. configs, data and implementations.
### debug config {#debug-config new="3" tag="command"} ### debug config {id="debug-config",version="3",tag="command"}
Debug a [`config.cfg` file](/usage/training#config) and show validation errors. Debug a [`config.cfg` file](/usage/training#config) and show validation errors.
The command will create all objects in the tree and validate them. Note that The command will create all objects in the tree and validate them. Note that
@ -303,13 +303,13 @@ errors at once and some issues are only shown once previous errors have been
fixed. To auto-fill a partial config and save the result, you can use the fixed. To auto-fill a partial config and save the result, you can use the
[`init fill-config`](/api/cli#init-fill-config) command. [`init fill-config`](/api/cli#init-fill-config) command.
```cli ```bash
$ python -m spacy debug config [config_path] [--code] [--show-functions] [--show-variables] [overrides] $ python -m spacy debug config [config_path] [--code] [--show-functions] [--show-variables] [overrides]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy debug config config.cfg > $ python -m spacy debug config config.cfg
> ``` > ```
@ -333,7 +333,7 @@ python -m spacy init fill-config tmp/starter-config_invalid.cfg tmp/starter-conf
<Accordion title="Example output (valid config and all options)" spaced> <Accordion title="Example output (valid config and all options)" spaced>
```cli ```bash
$ python -m spacy debug config ./config.cfg --show-functions --show-variables $ python -m spacy debug config ./config.cfg --show-functions --show-variables
``` ```
@ -453,7 +453,7 @@ File /path/to/thinc/thinc/schedules.py (line 91)
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ | | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ |
| **PRINTS** | Config validation errors, if available. | | **PRINTS** | Config validation errors, if available. |
### debug data {#debug-data tag="command"} ### debug data {id="debug-data",tag="command"}
Analyze, debug and validate your training and development data. Get useful Analyze, debug and validate your training and development data. Get useful
stats, and find problems like invalid entity annotations, cyclic dependencies, stats, and find problems like invalid entity annotations, cyclic dependencies,
@ -479,13 +479,13 @@ the token distributions. To learn more, you can check out Papay et al.'s work on
</Infobox> </Infobox>
```cli ```bash
$ python -m spacy debug data [config_path] [--code] [--ignore-warnings] [--verbose] [--no-format] [overrides] $ python -m spacy debug data [config_path] [--code] [--ignore-warnings] [--verbose] [--no-format] [overrides]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy debug data ./config.cfg > $ python -m spacy debug data ./config.cfg
> ``` > ```
@ -639,7 +639,7 @@ will not be available.
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ | | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ |
| **PRINTS** | Debugging information. | | **PRINTS** | Debugging information. |
### debug diff-config {#debug-diff tag="command"} ### debug diff-config {id="debug-diff",tag="command"}
Show a diff of a config file with respect to spaCy's defaults or another config Show a diff of a config file with respect to spaCy's defaults or another config
file. If additional settings were used in the creation of the config file, then file. If additional settings were used in the creation of the config file, then
@ -647,13 +647,13 @@ you must supply these as extra parameters to the command when comparing to the
default settings. The generated diff can also be used when posting to the default settings. The generated diff can also be used when posting to the
discussion forum to provide more information for the maintainers. discussion forum to provide more information for the maintainers.
```cli ```bash
$ python -m spacy debug diff-config [config_path] [--compare-to] [--optimize] [--gpu] [--pretraining] [--markdown] $ python -m spacy debug diff-config [config_path] [--compare-to] [--optimize] [--gpu] [--pretraining] [--markdown]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy debug diff-config ./config.cfg > $ python -m spacy debug diff-config ./config.cfg
> ``` > ```
@ -868,7 +868,7 @@ after_init = null
| `markdown`, `-md` | Generate Markdown for Github issues. Defaults to `False`. ~~bool (flag)~~ | | `markdown`, `-md` | Generate Markdown for Github issues. Defaults to `False`. ~~bool (flag)~~ |
| **PRINTS** | Diff between the two config files. | | **PRINTS** | Diff between the two config files. |
### debug profile {#debug-profile tag="command"} ### debug profile {id="debug-profile",tag="command"}
Profile which functions take the most time in a spaCy pipeline. Input should be Profile which functions take the most time in a spaCy pipeline. Input should be
formatted as one JSON object per line with a key `"text"`. It can either be formatted as one JSON object per line with a key `"text"`. It can either be
@ -882,7 +882,7 @@ The `profile` command is now available as a subcommand of `spacy debug`.
</Infobox> </Infobox>
```cli ```bash
$ python -m spacy debug profile [model] [inputs] [--n-texts] $ python -m spacy debug profile [model] [inputs] [--n-texts]
``` ```
@ -894,12 +894,12 @@ $ python -m spacy debug profile [model] [inputs] [--n-texts]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **PRINTS** | Profiling information for the pipeline. | | **PRINTS** | Profiling information for the pipeline. |
### debug model {#debug-model new="3" tag="command"} ### debug model {id="debug-model",version="3",tag="command"}
Debug a Thinc [`Model`](https://thinc.ai/docs/api-model) by running it on a Debug a Thinc [`Model`](https://thinc.ai/docs/api-model) by running it on a
sample text and checking how it updates its internal weights and parameters. sample text and checking how it updates its internal weights and parameters.
```cli ```bash
$ python -m spacy debug model [config_path] [component] [--layers] [--dimensions] [--parameters] [--gradients] [--attributes] [--print-step0] [--print-step1] [--print-step2] [--print-step3] [--gpu-id] $ python -m spacy debug model [config_path] [component] [--layers] [--dimensions] [--parameters] [--gradients] [--attributes] [--print-step0] [--print-step1] [--print-step2] [--print-step3] [--gpu-id]
``` ```
@ -910,7 +910,7 @@ model ("Step 0"), which helps us to understand the internal structure of the
Neural Network, and to focus on specific layers that we want to inspect further Neural Network, and to focus on specific layers that we want to inspect further
(see next example). (see next example).
```cli ```bash
$ python -m spacy debug model ./config.cfg tagger -P0 $ python -m spacy debug model ./config.cfg tagger -P0
``` ```
@ -956,7 +956,7 @@ an all-zero matrix determined by the `nO` and `nI` dimensions. After a first
training step (Step 2), this matrix has clearly updated its values through the training step (Step 2), this matrix has clearly updated its values through the
training feedback loop. training feedback loop.
```cli ```bash
$ python -m spacy debug model ./config.cfg tagger -l "5,15" -DIM -PAR -P0 -P1 -P2 $ python -m spacy debug model ./config.cfg tagger -l "5,15" -DIM -PAR -P0 -P1 -P2
``` ```
@ -1017,7 +1017,7 @@ $ python -m spacy debug model ./config.cfg tagger -l "5,15" -DIM -PAR -P0 -P1 -P
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ | | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ |
| **PRINTS** | Debugging information. | | **PRINTS** | Debugging information. |
## train {#train tag="command"} ## train {id="train",tag="command"}
Train a pipeline. Expects data in spaCy's Train a pipeline. Expects data in spaCy's
[binary format](/api/data-formats#training) and a [binary format](/api/data-formats#training) and a
@ -1043,11 +1043,11 @@ in the section `[paths]`.
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy train config.cfg --output ./output --paths.train ./train --paths.dev ./dev > $ python -m spacy train config.cfg --output ./output --paths.train ./train --paths.dev ./dev
> ``` > ```
```cli ```bash
$ python -m spacy train [config_path] [--output] [--code] [--verbose] [--gpu-id] [overrides] $ python -m spacy train [config_path] [--output] [--code] [--verbose] [--gpu-id] [overrides]
``` ```
@ -1062,7 +1062,7 @@ $ python -m spacy train [config_path] [--output] [--code] [--verbose] [--gpu-id]
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ | | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ |
| **CREATES** | The final trained pipeline and the best trained pipeline. | | **CREATES** | The final trained pipeline and the best trained pipeline. |
### Calling the training function from Python {#train-function new="3.2"} ### Calling the training function from Python {id="train-function",version="3.2"}
The training CLI exposes a `train` helper function that lets you run the The training CLI exposes a `train` helper function that lets you run the
training just like `spacy train`. Usually it's easier to use the command line training just like `spacy train`. Usually it's easier to use the command line
@ -1085,7 +1085,7 @@ directly, but if you need to kick off training from code this is how to do it.
| `use_gpu` | Which GPU to use. Defaults to -1 for no GPU. ~~int~~ | | `use_gpu` | Which GPU to use. Defaults to -1 for no GPU. ~~int~~ |
| `overrides` | Values to override config settings. ~~Dict[str, Any]~~ | | `overrides` | Values to override config settings. ~~Dict[str, Any]~~ |
## pretrain {#pretrain new="2.1" tag="command,experimental"} ## pretrain {id="pretrain",version="2.1",tag="command,experimental"}
Pretrain the "token to vector" ([`Tok2vec`](/api/tok2vec)) layer of pipeline Pretrain the "token to vector" ([`Tok2vec`](/api/tok2vec)) layer of pipeline
components on raw text, using an approximate language-modeling objective. components on raw text, using an approximate language-modeling objective.
@ -1113,11 +1113,11 @@ auto-generated by setting `--pretraining` on
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy pretrain config.cfg ./output_pretrain --paths.raw_text ./data.jsonl > $ python -m spacy pretrain config.cfg ./output_pretrain --paths.raw_text ./data.jsonl
> ``` > ```
```cli ```bash
$ python -m spacy pretrain [config_path] [output_dir] [--code] [--resume-path] [--epoch-resume] [--gpu-id] [overrides] $ python -m spacy pretrain [config_path] [output_dir] [--code] [--resume-path] [--epoch-resume] [--gpu-id] [overrides]
``` ```
@ -1133,7 +1133,7 @@ $ python -m spacy pretrain [config_path] [output_dir] [--code] [--resume-path] [
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--training.dropout 0.2`. ~~Any (option/flag)~~ | | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--training.dropout 0.2`. ~~Any (option/flag)~~ |
| **CREATES** | The pretrained weights that can be used to initialize `spacy train`. | | **CREATES** | The pretrained weights that can be used to initialize `spacy train`. |
## evaluate {#evaluate new="2" tag="command"} ## evaluate {id="evaluate",version="2",tag="command"}
Evaluate a trained pipeline. Expects a loadable spaCy pipeline (package name or Evaluate a trained pipeline. Expects a loadable spaCy pipeline (package name or
path) and evaluation data in the path) and evaluation data in the
@ -1146,7 +1146,7 @@ skew. To render a sample of dependency parses in a HTML file using the
[displaCy visualizations](/usage/visualizers), set as output directory as the [displaCy visualizations](/usage/visualizers), set as output directory as the
`--displacy-path` argument. `--displacy-path` argument.
```cli ```bash
$ python -m spacy evaluate [model] [data_path] [--output] [--code] [--gold-preproc] [--gpu-id] [--displacy-path] [--displacy-limit] $ python -m spacy evaluate [model] [data_path] [--output] [--code] [--gold-preproc] [--gpu-id] [--displacy-path] [--displacy-limit]
``` ```
@ -1163,7 +1163,7 @@ $ python -m spacy evaluate [model] [data_path] [--output] [--code] [--gold-prepr
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | Training results and optional metrics and visualizations. | | **CREATES** | Training results and optional metrics and visualizations. |
## apply {#apply new="3.5" tag="command"} ## apply {id="apply", version="3.5", tag="command"}
Applies a trained pipeline to data and stores the resulting annotated documents Applies a trained pipeline to data and stores the resulting annotated documents
in a `DocBin`. The input can be a single file or a directory. The recognized in a `DocBin`. The input can be a single file or a directory. The recognized
@ -1194,7 +1194,8 @@ $ python -m spacy apply [model] [data-path] [output-file] [--code] [--text-key]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | A `DocBin` with the annotations from the `model` for all the files found in `data-path`. | | **CREATES** | A `DocBin` with the annotations from the `model` for all the files found in `data-path`. |
## find-threshold {#find-threshold new="3.5" tag="command"}
## find-threshold {id="find-threshold",version="3.5",tag="command"}
Runs prediction trials for a trained model with varying tresholds to maximize Runs prediction trials for a trained model with varying tresholds to maximize
the specified metric. The search space for the threshold is traversed linearly the specified metric. The search space for the threshold is traversed linearly
@ -1209,12 +1210,12 @@ be provided.
> #### Examples > #### Examples
> >
> ```cli > ```bash
> # For textcat_multilabel: > # For textcat_multilabel:
> $ python -m spacy find-threshold my_nlp data.spacy textcat_multilabel threshold cats_macro_f > $ python -m spacy find-threshold my_nlp data.spacy textcat_multilabel threshold cats_macro_f
> ``` > ```
> >
> ```cli > ```bash
> # For spancat: > # For spancat:
> $ python -m spacy find-threshold my_nlp data.spacy spancat threshold spans_sc_f > $ python -m spacy find-threshold my_nlp data.spacy spancat threshold spans_sc_f
> ``` > ```
@ -1233,7 +1234,7 @@ be provided.
| `--silent`, `-V`, `-VV` | GPU to use, if any. Defaults to `-1` for CPU. ~~int (option)~~ | | `--silent`, `-V`, `-VV` | GPU to use, if any. Defaults to `-1` for CPU. ~~int (option)~~ |
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
## assemble {#assemble tag="command"} ## assemble {id="assemble",tag="command"}
Assemble a pipeline from a config file without additional training. Expects a Assemble a pipeline from a config file without additional training. Expects a
[config file](/api/data-formats#config) with all settings and hyperparameters. [config file](/api/data-formats#config) with all settings and hyperparameters.
@ -1243,11 +1244,11 @@ config.
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy assemble config.cfg ./output > $ python -m spacy assemble config.cfg ./output
> ``` > ```
```cli ```bash
$ python -m spacy assemble [config_path] [output_dir] [--code] [--verbose] [overrides] $ python -m spacy assemble [config_path] [output_dir] [--code] [--verbose] [overrides]
``` ```
@ -1261,7 +1262,7 @@ $ python -m spacy assemble [config_path] [output_dir] [--code] [--verbose] [over
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.data ./data`. ~~Any (option/flag)~~ | | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.data ./data`. ~~Any (option/flag)~~ |
| **CREATES** | The final assembled pipeline. | | **CREATES** | The final assembled pipeline. |
## package {#package tag="command"} ## package {id="package",tag="command"}
Generate an installable [Python package](/usage/training#models-generating) from Generate an installable [Python package](/usage/training#models-generating) from
an existing pipeline data directory. All data files are copied over. If an existing pipeline data directory. All data files are copied over. If
@ -1287,13 +1288,13 @@ the sdist and wheel by setting `--build sdist,wheel`.
</Infobox> </Infobox>
```cli ```bash
$ python -m spacy package [input_dir] [output_dir] [--code] [--meta-path] [--create-meta] [--build] [--name] [--version] [--force] $ python -m spacy package [input_dir] [output_dir] [--code] [--meta-path] [--create-meta] [--build] [--name] [--version] [--force]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy package /input /output > $ python -m spacy package /input /output
> $ cd /output/en_pipeline-0.0.0 > $ cd /output/en_pipeline-0.0.0
> $ pip install dist/en_pipeline-0.0.0.tar.gz > $ pip install dist/en_pipeline-0.0.0.tar.gz
@ -1313,13 +1314,13 @@ $ python -m spacy package [input_dir] [output_dir] [--code] [--meta-path] [--cre
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | A Python package containing the spaCy pipeline. | | **CREATES** | A Python package containing the spaCy pipeline. |
## project {#project new="3"} ## project {id="project",version="3"}
The `spacy project` CLI includes subcommands for working with The `spacy project` CLI includes subcommands for working with
[spaCy projects](/usage/projects), end-to-end workflows for building and [spaCy projects](/usage/projects), end-to-end workflows for building and
deploying custom spaCy pipelines. deploying custom spaCy pipelines.
### project clone {#project-clone tag="command"} ### project clone {id="project-clone",tag="command"}
Clone a project template from a Git repository. Calls into `git` under the hood Clone a project template from a Git repository. Calls into `git` under the hood
and can use the sparse checkout feature if available, so you're only downloading and can use the sparse checkout feature if available, so you're only downloading
@ -1328,19 +1329,19 @@ what you need. By default, spaCy's
can provide any other repo (public or private) that you have access to using the can provide any other repo (public or private) that you have access to using the
`--repo` option. `--repo` option.
```cli ```bash
$ python -m spacy project clone [name] [dest] [--repo] [--branch] [--sparse] $ python -m spacy project clone [name] [dest] [--repo] [--branch] [--sparse]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy project clone pipelines/ner_wikiner > $ python -m spacy project clone pipelines/ner_wikiner
> ``` > ```
> >
> Clone from custom repo: > Clone from custom repo:
> >
> ```cli > ```bash
> $ python -m spacy project clone template --repo https://github.com/your_org/your_repo > $ python -m spacy project clone template --repo https://github.com/your_org/your_repo
> ``` > ```
@ -1354,7 +1355,7 @@ $ python -m spacy project clone [name] [dest] [--repo] [--branch] [--sparse]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | The cloned [project directory](/usage/projects#project-files). | | **CREATES** | The cloned [project directory](/usage/projects#project-files). |
### project assets {#project-assets tag="command"} ### project assets {id="project-assets",tag="command"}
Fetch project assets like datasets and pretrained weights. Assets are defined in Fetch project assets like datasets and pretrained weights. Assets are defined in
the `assets` section of the [`project.yml`](/usage/projects#project-yml). If a the `assets` section of the [`project.yml`](/usage/projects#project-yml). If a
@ -1365,13 +1366,13 @@ considered "private" and you have to take care of putting them into the
destination directory yourself. If a local path is provided, the asset is copied destination directory yourself. If a local path is provided, the asset is copied
into the current project. into the current project.
```cli ```bash
$ python -m spacy project assets [project_dir] $ python -m spacy project assets [project_dir]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy project assets [--sparse] > $ python -m spacy project assets [--sparse]
> ``` > ```
@ -1382,7 +1383,7 @@ $ python -m spacy project assets [project_dir]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | Downloaded or copied assets defined in the `project.yml`. | | **CREATES** | Downloaded or copied assets defined in the `project.yml`. |
### project run {#project-run tag="command"} ### project run {id="project-run",tag="command"}
Run a named command or workflow defined in the Run a named command or workflow defined in the
[`project.yml`](/usage/projects#project-yml). If a workflow name is specified, [`project.yml`](/usage/projects#project-yml). If a workflow name is specified,
@ -1391,13 +1392,13 @@ all commands in the workflow are run, in order. If commands define
re-run if state has changed. For example, if the input dataset changes, a re-run if state has changed. For example, if the input dataset changes, a
preprocessing command that depends on those files will be re-run. preprocessing command that depends on those files will be re-run.
```cli ```bash
$ python -m spacy project run [subcommand] [project_dir] [--force] [--dry] $ python -m spacy project run [subcommand] [project_dir] [--force] [--dry]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy project run train > $ python -m spacy project run train
> ``` > ```
@ -1410,7 +1411,7 @@ $ python -m spacy project run [subcommand] [project_dir] [--force] [--dry]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **EXECUTES** | The command defined in the `project.yml`. | | **EXECUTES** | The command defined in the `project.yml`. |
### project push {#project-push tag="command"} ### project push {id="project-push",tag="command"}
Upload all available files or directories listed as in the `outputs` section of Upload all available files or directories listed as in the `outputs` section of
commands to a remote storage. Outputs are archived and compressed prior to commands to a remote storage. Outputs are archived and compressed prior to
@ -1430,13 +1431,13 @@ remote storages, so you can use any protocol that `Pathy` supports, including
filesystem, although you may need to install extra dependencies to use certain filesystem, although you may need to install extra dependencies to use certain
protocols. protocols.
```cli ```bash
$ python -m spacy project push [remote] [project_dir] $ python -m spacy project push [remote] [project_dir]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy project push my_bucket > $ python -m spacy project push my_bucket
> ``` > ```
> >
@ -1453,7 +1454,7 @@ $ python -m spacy project push [remote] [project_dir]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **UPLOADS** | All project outputs that exist and are not already stored in the remote. | | **UPLOADS** | All project outputs that exist and are not already stored in the remote. |
### project pull {#project-pull tag="command"} ### project pull {id="project-pull",tag="command"}
Download all files or directories listed as `outputs` for commands, unless they Download all files or directories listed as `outputs` for commands, unless they
are not already present locally. When searching for files in the remote, `pull` are not already present locally. When searching for files in the remote, `pull`
@ -1475,13 +1476,13 @@ remote storages, so you can use any protocol that `Pathy` supports, including
filesystem, although you may need to install extra dependencies to use certain filesystem, although you may need to install extra dependencies to use certain
protocols. protocols.
```cli ```bash
$ python -m spacy project pull [remote] [project_dir] $ python -m spacy project pull [remote] [project_dir]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy project pull my_bucket > $ python -m spacy project pull my_bucket
> ``` > ```
> >
@ -1498,7 +1499,7 @@ $ python -m spacy project pull [remote] [project_dir]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **DOWNLOADS** | All project outputs that do not exist locally and can be found in the remote. | | **DOWNLOADS** | All project outputs that do not exist locally and can be found in the remote. |
### project document {#project-document tag="command"} ### project document {id="project-document",tag="command"}
Auto-generate a pretty Markdown-formatted `README` for your project, based on Auto-generate a pretty Markdown-formatted `README` for your project, based on
its [`project.yml`](/usage/projects#project-yml). Will create sections that its [`project.yml`](/usage/projects#project-yml). Will create sections that
@ -1507,13 +1508,13 @@ content will be placed between two hidden markers, so you can add your own
custom content before or after the auto-generated documentation. When you re-run custom content before or after the auto-generated documentation. When you re-run
the `project document` command, only the auto-generated part is replaced. the `project document` command, only the auto-generated part is replaced.
```cli ```bash
$ python -m spacy project document [project_dir] [--output] [--no-emoji] $ python -m spacy project document [project_dir] [--output] [--no-emoji]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy project document --output README.md > $ python -m spacy project document --output README.md
> ``` > ```
@ -1522,7 +1523,7 @@ $ python -m spacy project document [project_dir] [--output] [--no-emoji]
For more examples, see the templates in our For more examples, see the templates in our
[`projects`](https://github.com/explosion/projects) repo. [`projects`](https://github.com/explosion/projects) repo.
![Screenshot of auto-generated Markdown Readme](../images/project_document.jpg) ![Screenshot of auto-generated Markdown Readme](/images/project_document.jpg)
</Accordion> </Accordion>
@ -1533,7 +1534,7 @@ For more examples, see the templates in our
| `--no-emoji`, `-NE` | Don't use emoji in the titles. ~~bool (flag)~~ | | `--no-emoji`, `-NE` | Don't use emoji in the titles. ~~bool (flag)~~ |
| **CREATES** | The Markdown-formatted project documentation. | | **CREATES** | The Markdown-formatted project documentation. |
### project dvc {#project-dvc tag="command"} ### project dvc {id="project-dvc",tag="command"}
Auto-generate [Data Version Control](https://dvc.org) (DVC) config file. Calls Auto-generate [Data Version Control](https://dvc.org) (DVC) config file. Calls
[`dvc run`](https://dvc.org/doc/command-reference/run) with `--no-exec` under [`dvc run`](https://dvc.org/doc/command-reference/run) with `--no-exec` under
@ -1553,13 +1554,13 @@ You'll also need to add the assets you want to track with
</Infobox> </Infobox>
```cli ```bash
$ python -m spacy project dvc [project_dir] [workflow] [--force] [--verbose] [--quiet] $ python -m spacy project dvc [project_dir] [workflow] [--force] [--verbose] [--quiet]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ git init > $ git init
> $ dvc init > $ dvc init
> $ python -m spacy project dvc all > $ python -m spacy project dvc all
@ -1575,14 +1576,14 @@ $ python -m spacy project dvc [project_dir] [workflow] [--force] [--verbose] [--
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | A `dvc.yaml` file in the project directory, based on the steps defined in the given workflow. | | **CREATES** | A `dvc.yaml` file in the project directory, based on the steps defined in the given workflow. |
## huggingface-hub {#huggingface-hub new="3.1"} ## huggingface-hub {id="huggingface-hub",version="3.1"}
The `spacy huggingface-cli` CLI includes commands for uploading your trained The `spacy huggingface-cli` CLI includes commands for uploading your trained
spaCy pipelines to the [Hugging Face Hub](https://huggingface.co/). spaCy pipelines to the [Hugging Face Hub](https://huggingface.co/).
> #### Installation > #### Installation
> >
> ```cli > ```bash
> $ pip install spacy-huggingface-hub > $ pip install spacy-huggingface-hub
> $ huggingface-cli login > $ huggingface-cli login
> ``` > ```
@ -1596,19 +1597,19 @@ package installed. Installing the package will automatically add the
</Infobox> </Infobox>
### huggingface-hub push {#huggingface-hub-push tag="command"} ### huggingface-hub push {id="huggingface-hub-push",tag="command"}
Push a spaCy pipeline to the Hugging Face Hub. Expects a `.whl` file packaged Push a spaCy pipeline to the Hugging Face Hub. Expects a `.whl` file packaged
with [`spacy package`](/api/cli#package) and `--build wheel`. For more details, with [`spacy package`](/api/cli#package) and `--build wheel`. For more details,
see the spaCy project [integration](/usage/projects#huggingface_hub). see the spaCy project [integration](/usage/projects#huggingface_hub).
```cli ```bash
$ python -m spacy huggingface-hub push [whl_path] [--org] [--msg] [--local-repo] [--verbose] $ python -m spacy huggingface-hub push [whl_path] [--org] [--msg] [--local-repo] [--verbose]
``` ```
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl > $ python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl
> ``` > ```

View File

@ -34,7 +34,7 @@ same thing. Clusters are represented as SpanGroups that start with a prefix
A `CoreferenceResolver` component can be paired with a A `CoreferenceResolver` component can be paired with a
[`SpanResolver`](/api/span-resolver) to expand single tokens to spans. [`SpanResolver`](/api/span-resolver) to expand single tokens to spans.
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Predictions will be saved to `Doc.spans` as a [`SpanGroup`](/api/spangroup). The Predictions will be saved to `Doc.spans` as a [`SpanGroup`](/api/spangroup). The
span key will be a prefix plus a serial number referring to the coreference span key will be a prefix plus a serial number referring to the coreference
@ -47,7 +47,7 @@ parameter.
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------- | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
| `Doc.spans[prefix + "_" + cluster_number]` | One coreference cluster, represented as single-token spans. Cluster numbers start from 1. ~~SpanGroup~~ | | `Doc.spans[prefix + "_" + cluster_number]` | One coreference cluster, represented as single-token spans. Cluster numbers start from 1. ~~SpanGroup~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -73,7 +73,7 @@ details on the architectures and their arguments and hyperparameters.
| `model` | The [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. Defaults to [Coref](/api/architectures#Coref). ~~Model~~ | | `model` | The [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. Defaults to [Coref](/api/architectures#Coref). ~~Model~~ |
| `span_cluster_prefix` | The prefix for the keys for clusters saved to `doc.spans`. Defaults to `coref_clusters`. ~~str~~ | | `span_cluster_prefix` | The prefix for the keys for clusters saved to `doc.spans`. Defaults to `coref_clusters`. ~~str~~ |
## CoreferenceResolver.\_\_init\_\_ {#init tag="method"} ## CoreferenceResolver.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -102,7 +102,7 @@ shortcut for this and instantiate the component using its string name and
| _keyword-only_ | | | _keyword-only_ | |
| `span_cluster_prefix` | The prefix for the key for saving clusters of spans. ~~bool~~ | | `span_cluster_prefix` | The prefix for the key for saving clusters of spans. ~~bool~~ |
## CoreferenceResolver.\_\_call\_\_ {#call tag="method"} ## CoreferenceResolver.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place and returned. Apply the pipe to one document. The document is modified in place and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -125,7 +125,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## CoreferenceResolver.pipe {#pipe tag="method"} ## CoreferenceResolver.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -148,7 +148,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/coref#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## CoreferenceResolver.initialize {#initialize tag="method"} ## CoreferenceResolver.initialize {id="initialize",tag="method"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -172,7 +172,7 @@ by [`Language.initialize`](/api/language#initialize).
| _keyword-only_ | | | _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## CoreferenceResolver.predict {#predict tag="method"} ## CoreferenceResolver.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. Clusters are returned as a list of `MentionClusters`, one for modifying them. Clusters are returned as a list of `MentionClusters`, one for
@ -192,7 +192,7 @@ to token indices.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The predicted coreference clusters for the `docs`. ~~List[MentionClusters]~~ | | **RETURNS** | The predicted coreference clusters for the `docs`. ~~List[MentionClusters]~~ |
## CoreferenceResolver.set_annotations {#set_annotations tag="method"} ## CoreferenceResolver.set_annotations {id="set_annotations",tag="method"}
Modify a batch of documents, saving coreference clusters in `Doc.spans`. Modify a batch of documents, saving coreference clusters in `Doc.spans`.
@ -209,7 +209,7 @@ Modify a batch of documents, saving coreference clusters in `Doc.spans`.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `clusters` | The predicted coreference clusters for the `docs`. ~~List[MentionClusters]~~ | | `clusters` | The predicted coreference clusters for the `docs`. ~~List[MentionClusters]~~ |
## CoreferenceResolver.update {#update tag="method"} ## CoreferenceResolver.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects. Delegates to Learn from a batch of [`Example`](/api/example) objects. Delegates to
[`predict`](/api/coref#predict). [`predict`](/api/coref#predict).
@ -231,7 +231,7 @@ Learn from a batch of [`Example`](/api/example) objects. Delegates to
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## CoreferenceResolver.create_optimizer {#create_optimizer tag="method"} ## CoreferenceResolver.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -246,7 +246,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## CoreferenceResolver.use_params {#use_params tag="method, contextmanager"} ## CoreferenceResolver.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -263,7 +263,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## CoreferenceResolver.to_disk {#to_disk tag="method"} ## CoreferenceResolver.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -280,7 +280,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## CoreferenceResolver.from_disk {#from_disk tag="method"} ## CoreferenceResolver.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -298,7 +298,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `CoreferenceResolver` object. ~~CoreferenceResolver~~ | | **RETURNS** | The modified `CoreferenceResolver` object. ~~CoreferenceResolver~~ |
## CoreferenceResolver.to_bytes {#to_bytes tag="method"} ## CoreferenceResolver.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -315,7 +315,7 @@ Serialize the pipe to a bytestring, including the `KnowledgeBase`.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `CoreferenceResolver` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `CoreferenceResolver` object. ~~bytes~~ |
## CoreferenceResolver.from_bytes {#from_bytes tag="method"} ## CoreferenceResolver.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -334,7 +334,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `CoreferenceResolver` object. ~~CoreferenceResolver~~ | | **RETURNS** | The `CoreferenceResolver` object. ~~CoreferenceResolver~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -3,7 +3,7 @@ title: Corpus
teaser: An annotated corpus teaser: An annotated corpus
tag: class tag: class
source: spacy/training/corpus.py source: spacy/training/corpus.py
new: 3 version: 3
--- ---
This class manages annotated corpora and can be used for training and This class manages annotated corpora and can be used for training and
@ -13,7 +13,7 @@ customize the data loading during training, you can register your own
see the usage guide on [data utilities](/usage/training#data) for more details see the usage guide on [data utilities](/usage/training#data) for more details
and examples. and examples.
## Config and implementation {#config} ## Config and implementation {id="config"}
`spacy.Corpus.v1` is a registered function that creates a `Corpus` of training `spacy.Corpus.v1` is a registered function that creates a `Corpus` of training
or evaluation data. It takes the same arguments as the `Corpus` class and or evaluation data. It takes the same arguments as the `Corpus` class and
@ -49,7 +49,7 @@ streaming.
%%GITHUB_SPACY/spacy/training/corpus.py %%GITHUB_SPACY/spacy/training/corpus.py
``` ```
## Corpus.\_\_init\_\_ {#init tag="method"} ## Corpus.\_\_init\_\_ {id="init",tag="method"}
Create a `Corpus` for iterating [Example](/api/example) objects from a file or Create a `Corpus` for iterating [Example](/api/example) objects from a file or
directory of [`.spacy` data files](/api/data-formats#binary-training). The directory of [`.spacy` data files](/api/data-formats#binary-training). The
@ -81,7 +81,7 @@ train/test skew.
| `augmenter` | Optional data augmentation callback. ~~Callable[[Language, Example], Iterable[Example]]~~ | | `augmenter` | Optional data augmentation callback. ~~Callable[[Language, Example], Iterable[Example]]~~ |
| `shuffle` | Whether to shuffle the examples. Defaults to `False`. ~~bool~~ | | `shuffle` | Whether to shuffle the examples. Defaults to `False`. ~~bool~~ |
## Corpus.\_\_call\_\_ {#call tag="method"} ## Corpus.\_\_call\_\_ {id="call",tag="method"}
Yield examples from the data. Yield examples from the data.
@ -101,7 +101,7 @@ Yield examples from the data.
| `nlp` | The current `nlp` object. ~~Language~~ | | `nlp` | The current `nlp` object. ~~Language~~ |
| **YIELDS** | The examples. ~~Example~~ | | **YIELDS** | The examples. ~~Example~~ |
## JsonlCorpus {#jsonlcorpus tag="class"} ## JsonlCorpus {id="jsonlcorpus",tag="class"}
Iterate Doc objects from a file or directory of JSONL (newline-delimited JSON) Iterate Doc objects from a file or directory of JSONL (newline-delimited JSON)
formatted raw text files. Can be used to read the raw text corpus for language formatted raw text files. Can be used to read the raw text corpus for language
@ -120,14 +120,13 @@ file.
> srsly.write_jsonl("/path/to/text.jsonl", data) > srsly.write_jsonl("/path/to/text.jsonl", data)
> ``` > ```
```json ```json {title="Example"}
### Example
{"text": "Can I ask where you work now and what you do, and if you enjoy it?"} {"text": "Can I ask where you work now and what you do, and if you enjoy it?"}
{"text": "They may just pull out of the Seattle market completely, at least until they have autonomous vehicles."} {"text": "They may just pull out of the Seattle market completely, at least until they have autonomous vehicles."}
{"text": "My cynical view on this is that it will never be free to the public. Reason: what would be the draw of joining the military? Right now their selling point is free Healthcare and Education. Ironically both are run horribly and most, that I've talked to, come out wishing they never went in."} {"text": "My cynical view on this is that it will never be free to the public. Reason: what would be the draw of joining the military? Right now their selling point is free Healthcare and Education. Ironically both are run horribly and most, that I've talked to, come out wishing they never went in."}
``` ```
### JsonlCorpus.\_\init\_\_ {#jsonlcorpus tag="method"} ### JsonlCorpus.\_\_init\_\_ {id="jsonlcorpus",tag="method"}
Initialize the reader. Initialize the reader.
@ -157,7 +156,7 @@ Initialize the reader.
| `max_length` | Maximum document length (in tokens). Longer documents will be skipped. Defaults to `0`, which indicates no limit. ~~int~~ | | `max_length` | Maximum document length (in tokens). Longer documents will be skipped. Defaults to `0`, which indicates no limit. ~~int~~ |
| `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ | | `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ |
### JsonlCorpus.\_\_call\_\_ {#jsonlcorpus-call tag="method"} ### JsonlCorpus.\_\_call\_\_ {id="jsonlcorpus-call",tag="method"}
Yield examples from the data. Yield examples from the data.

View File

@ -9,7 +9,7 @@ menu:
- ['StringStore', 'stringstore'] - ['StringStore', 'stringstore']
--- ---
## Doc {#doc tag="cdef class" source="spacy/tokens/doc.pxd"} ## Doc {id="doc",tag="cdef class",source="spacy/tokens/doc.pxd"}
The `Doc` object holds an array of [`TokenC`](/api/cython-structs#tokenc) The `Doc` object holds an array of [`TokenC`](/api/cython-structs#tokenc)
structs. structs.
@ -21,7 +21,7 @@ accessed from Python. For the Python documentation, see [`Doc`](/api/doc).
</Infobox> </Infobox>
### Attributes {#doc_attributes} ### Attributes {id="doc_attributes"}
| Name | Description | | Name | Description |
| ------------ | -------------------------------------------------------------------------------------------------------- | | ------------ | -------------------------------------------------------------------------------------------------------- |
@ -31,7 +31,7 @@ accessed from Python. For the Python documentation, see [`Doc`](/api/doc).
| `length` | The number of tokens in the document. ~~int~~ | | `length` | The number of tokens in the document. ~~int~~ |
| `max_length` | The underlying size of the `Doc.c` array. ~~int~~ | | `max_length` | The underlying size of the `Doc.c` array. ~~int~~ |
### Doc.push_back {#doc_push_back tag="method"} ### Doc.push_back {id="doc_push_back",tag="method"}
Append a token to the `Doc`. The token can be provided as a Append a token to the `Doc`. The token can be provided as a
[`LexemeC`](/api/cython-structs#lexemec) or [`LexemeC`](/api/cython-structs#lexemec) or
@ -55,7 +55,7 @@ Append a token to the `Doc`. The token can be provided as a
| `lex_or_tok` | The word to append to the `Doc`. ~~LexemeOrToken~~ | | `lex_or_tok` | The word to append to the `Doc`. ~~LexemeOrToken~~ |
| `has_space` | Whether the word has trailing whitespace. ~~bint~~ | | `has_space` | Whether the word has trailing whitespace. ~~bint~~ |
## Token {#token tag="cdef class" source="spacy/tokens/token.pxd"} ## Token {id="token",tag="cdef class",source="spacy/tokens/token.pxd"}
A Cython class providing access and methods for a A Cython class providing access and methods for a
[`TokenC`](/api/cython-structs#tokenc) struct. Note that the `Token` object does [`TokenC`](/api/cython-structs#tokenc) struct. Note that the `Token` object does
@ -68,7 +68,7 @@ accessed from Python. For the Python documentation, see [`Token`](/api/token).
</Infobox> </Infobox>
### Attributes {#token_attributes} ### Attributes {id="token_attributes"}
| Name | Description | | Name | Description |
| ------- | -------------------------------------------------------------------------- | | ------- | -------------------------------------------------------------------------- |
@ -77,7 +77,7 @@ accessed from Python. For the Python documentation, see [`Token`](/api/token).
| `i` | The offset of the token within the document. ~~int~~ | | `i` | The offset of the token within the document. ~~int~~ |
| `doc` | The parent document. ~~Doc~~ | | `doc` | The parent document. ~~Doc~~ |
### Token.cinit {#token_cinit tag="method"} ### Token.cinit {id="token_cinit",tag="method"}
Create a `Token` object from a `TokenC*` pointer. Create a `Token` object from a `TokenC*` pointer.
@ -94,7 +94,7 @@ Create a `Token` object from a `TokenC*` pointer.
| `offset` | The offset of the token within the document. ~~int~~ | | `offset` | The offset of the token within the document. ~~int~~ |
| `doc` | The parent document. ~~int~~ | | `doc` | The parent document. ~~int~~ |
## Span {#span tag="cdef class" source="spacy/tokens/span.pxd"} ## Span {id="span",tag="cdef class",source="spacy/tokens/span.pxd"}
A Cython class providing access and methods for a slice of a `Doc` object. A Cython class providing access and methods for a slice of a `Doc` object.
@ -105,7 +105,7 @@ accessed from Python. For the Python documentation, see [`Span`](/api/span).
</Infobox> </Infobox>
### Attributes {#span_attributes} ### Attributes {id="span_attributes"}
| Name | Description | | Name | Description |
| ------------ | ----------------------------------------------------------------------------- | | ------------ | ----------------------------------------------------------------------------- |
@ -116,7 +116,7 @@ accessed from Python. For the Python documentation, see [`Span`](/api/span).
| `end_char` | The index of the last character of the span. ~~int~~ | | `end_char` | The index of the last character of the span. ~~int~~ |
| `label` | A label to attach to the span, e.g. for named entities. ~~attr_t (uint64_t)~~ | | `label` | A label to attach to the span, e.g. for named entities. ~~attr_t (uint64_t)~~ |
## Lexeme {#lexeme tag="cdef class" source="spacy/lexeme.pxd"} ## Lexeme {id="lexeme",tag="cdef class",source="spacy/lexeme.pxd"}
A Cython class providing access and methods for an entry in the vocabulary. A Cython class providing access and methods for an entry in the vocabulary.
@ -127,7 +127,7 @@ accessed from Python. For the Python documentation, see [`Lexeme`](/api/lexeme).
</Infobox> </Infobox>
### Attributes {#lexeme_attributes} ### Attributes {id="lexeme_attributes"}
| Name | Description | | Name | Description |
| ------- | ----------------------------------------------------------------------------- | | ------- | ----------------------------------------------------------------------------- |
@ -135,7 +135,7 @@ accessed from Python. For the Python documentation, see [`Lexeme`](/api/lexeme).
| `vocab` | A reference to the shared `Vocab` object. ~~Vocab~~ | | `vocab` | A reference to the shared `Vocab` object. ~~Vocab~~ |
| `orth` | ID of the verbatim text content. ~~attr_t (uint64_t)~~ | | `orth` | ID of the verbatim text content. ~~attr_t (uint64_t)~~ |
## Vocab {#vocab tag="cdef class" source="spacy/vocab.pxd"} ## Vocab {id="vocab",tag="cdef class",source="spacy/vocab.pxd"}
A Cython class providing access and methods for a vocabulary and other data A Cython class providing access and methods for a vocabulary and other data
shared across a language. shared across a language.
@ -147,7 +147,7 @@ accessed from Python. For the Python documentation, see [`Vocab`](/api/vocab).
</Infobox> </Infobox>
### Attributes {#vocab_attributes} ### Attributes {id="vocab_attributes"}
| Name | Description | | Name | Description |
| --------- | ---------------------------------------------------------------------------------------------------------- | | --------- | ---------------------------------------------------------------------------------------------------------- |
@ -155,7 +155,7 @@ accessed from Python. For the Python documentation, see [`Vocab`](/api/vocab).
| `strings` | A `StringStore` that maps string to hash values and vice versa. ~~StringStore~~ | | `strings` | A `StringStore` that maps string to hash values and vice versa. ~~StringStore~~ |
| `length` | The number of entries in the vocabulary. ~~int~~ | | `length` | The number of entries in the vocabulary. ~~int~~ |
### Vocab.get {#vocab_get tag="method"} ### Vocab.get {id="vocab_get",tag="method"}
Retrieve a [`LexemeC*`](/api/cython-structs#lexemec) pointer from the Retrieve a [`LexemeC*`](/api/cython-structs#lexemec) pointer from the
vocabulary. vocabulary.
@ -172,7 +172,7 @@ vocabulary.
| `string` | The string of the word to look up. ~~str~~ | | `string` | The string of the word to look up. ~~str~~ |
| **RETURNS** | The lexeme in the vocabulary. ~~const LexemeC\*~~ | | **RETURNS** | The lexeme in the vocabulary. ~~const LexemeC\*~~ |
### Vocab.get_by_orth {#vocab_get_by_orth tag="method"} ### Vocab.get_by_orth {id="vocab_get_by_orth",tag="method"}
Retrieve a [`LexemeC*`](/api/cython-structs#lexemec) pointer from the Retrieve a [`LexemeC*`](/api/cython-structs#lexemec) pointer from the
vocabulary. vocabulary.
@ -189,7 +189,7 @@ vocabulary.
| `orth` | ID of the verbatim text content. ~~attr_t (uint64_t)~~ | | `orth` | ID of the verbatim text content. ~~attr_t (uint64_t)~~ |
| **RETURNS** | The lexeme in the vocabulary. ~~const LexemeC\*~~ | | **RETURNS** | The lexeme in the vocabulary. ~~const LexemeC\*~~ |
## StringStore {#stringstore tag="cdef class" source="spacy/strings.pxd"} ## StringStore {id="stringstore",tag="cdef class",source="spacy/strings.pxd"}
A lookup table to retrieve strings by 64-bit hashes. A lookup table to retrieve strings by 64-bit hashes.
@ -201,7 +201,7 @@ accessed from Python. For the Python documentation, see
</Infobox> </Infobox>
### Attributes {#stringstore_attributes} ### Attributes {id="stringstore_attributes"}
| Name | Description | | Name | Description |
| ------ | ---------------------------------------------------------------------------------------------------------------- | | ------ | ---------------------------------------------------------------------------------------------------------------- |

View File

@ -7,7 +7,7 @@ menu:
- ['LexemeC', 'lexemec'] - ['LexemeC', 'lexemec']
--- ---
## TokenC {#tokenc tag="C struct" source="spacy/structs.pxd"} ## TokenC {id="tokenc",tag="C struct",source="spacy/structs.pxd"}
Cython data container for the `Token` object. Cython data container for the `Token` object.
@ -39,7 +39,7 @@ Cython data container for the `Token` object.
| `ent_type` | Named entity type. ~~attr_t (uint64_t)~~ | | `ent_type` | Named entity type. ~~attr_t (uint64_t)~~ |
| `ent_id` | ID of the entity the token is an instance of, if any. Currently not used, but potentially for coreference resolution. ~~attr_t (uint64_t)~~ | | `ent_id` | ID of the entity the token is an instance of, if any. Currently not used, but potentially for coreference resolution. ~~attr_t (uint64_t)~~ |
### Token.get_struct_attr {#token_get_struct_attr tag="staticmethod, nogil" source="spacy/tokens/token.pxd"} ### Token.get_struct_attr {id="token_get_struct_attr",tag="staticmethod, nogil",source="spacy/tokens/token.pxd"}
Get the value of an attribute from the `TokenC` struct by attribute ID. Get the value of an attribute from the `TokenC` struct by attribute ID.
@ -58,7 +58,7 @@ Get the value of an attribute from the `TokenC` struct by attribute ID.
| `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ | | `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ |
| **RETURNS** | The value of the attribute. ~~attr_t (uint64_t)~~ | | **RETURNS** | The value of the attribute. ~~attr_t (uint64_t)~~ |
### Token.set_struct_attr {#token_set_struct_attr tag="staticmethod, nogil" source="spacy/tokens/token.pxd"} ### Token.set_struct_attr {id="token_set_struct_attr",tag="staticmethod, nogil",source="spacy/tokens/token.pxd"}
Set the value of an attribute of the `TokenC` struct by attribute ID. Set the value of an attribute of the `TokenC` struct by attribute ID.
@ -78,7 +78,7 @@ Set the value of an attribute of the `TokenC` struct by attribute ID.
| `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ | | `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ |
| `value` | The value to set. ~~attr_t (uint64_t)~~ | | `value` | The value to set. ~~attr_t (uint64_t)~~ |
### token_by_start {#token_by_start tag="function" source="spacy/tokens/doc.pxd"} ### token_by_start {id="token_by_start",tag="function",source="spacy/tokens/doc.pxd"}
Find a token in a `TokenC*` array by the offset of its first character. Find a token in a `TokenC*` array by the offset of its first character.
@ -100,7 +100,7 @@ Find a token in a `TokenC*` array by the offset of its first character.
| `start_char` | The start index to search for. ~~int~~ | | `start_char` | The start index to search for. ~~int~~ |
| **RETURNS** | The index of the token in the array or `-1` if not found. ~~int~~ | | **RETURNS** | The index of the token in the array or `-1` if not found. ~~int~~ |
### token_by_end {#token_by_end tag="function" source="spacy/tokens/doc.pxd"} ### token_by_end {id="token_by_end",tag="function",source="spacy/tokens/doc.pxd"}
Find a token in a `TokenC*` array by the offset of its final character. Find a token in a `TokenC*` array by the offset of its final character.
@ -122,7 +122,7 @@ Find a token in a `TokenC*` array by the offset of its final character.
| `end_char` | The end index to search for. ~~int~~ | | `end_char` | The end index to search for. ~~int~~ |
| **RETURNS** | The index of the token in the array or `-1` if not found. ~~int~~ | | **RETURNS** | The index of the token in the array or `-1` if not found. ~~int~~ |
### set_children_from_heads {#set_children_from_heads tag="function" source="spacy/tokens/doc.pxd"} ### set_children_from_heads {id="set_children_from_heads",tag="function",source="spacy/tokens/doc.pxd"}
Set attributes that allow lookup of syntactic children on a `TokenC*` array. Set attributes that allow lookup of syntactic children on a `TokenC*` array.
This function must be called after making changes to the `TokenC.head` This function must be called after making changes to the `TokenC.head`
@ -148,7 +148,7 @@ attribute, in order to make the parse tree navigation consistent.
| `tokens` | A `TokenC*` array. ~~const TokenC\*~~ | | `tokens` | A `TokenC*` array. ~~const TokenC\*~~ |
| `length` | The number of tokens in the array. ~~int~~ | | `length` | The number of tokens in the array. ~~int~~ |
## LexemeC {#lexemec tag="C struct" source="spacy/structs.pxd"} ## LexemeC {id="lexemec",tag="C struct",source="spacy/structs.pxd"}
Struct holding information about a lexical type. `LexemeC` structs are usually Struct holding information about a lexical type. `LexemeC` structs are usually
owned by the `Vocab`, and accessed through a read-only pointer on the `TokenC` owned by the `Vocab`, and accessed through a read-only pointer on the `TokenC`
@ -172,7 +172,7 @@ struct.
| `prefix` | Length-N substring from the start of the lexeme. Defaults to `N=1`. ~~attr_t (uint64_t)~~ | | `prefix` | Length-N substring from the start of the lexeme. Defaults to `N=1`. ~~attr_t (uint64_t)~~ |
| `suffix` | Length-N substring from the end of the lexeme. Defaults to `N=3`. ~~attr_t (uint64_t)~~ | | `suffix` | Length-N substring from the end of the lexeme. Defaults to `N=3`. ~~attr_t (uint64_t)~~ |
### Lexeme.get_struct_attr {#lexeme_get_struct_attr tag="staticmethod, nogil" source="spacy/lexeme.pxd"} ### Lexeme.get_struct_attr {id="lexeme_get_struct_attr",tag="staticmethod, nogil",source="spacy/lexeme.pxd"}
Get the value of an attribute from the `LexemeC` struct by attribute ID. Get the value of an attribute from the `LexemeC` struct by attribute ID.
@ -192,7 +192,7 @@ Get the value of an attribute from the `LexemeC` struct by attribute ID.
| `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ | | `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ |
| **RETURNS** | The value of the attribute. ~~attr_t (uint64_t)~~ | | **RETURNS** | The value of the attribute. ~~attr_t (uint64_t)~~ |
### Lexeme.set_struct_attr {#lexeme_set_struct_attr tag="staticmethod, nogil" source="spacy/lexeme.pxd"} ### Lexeme.set_struct_attr {id="lexeme_set_struct_attr",tag="staticmethod, nogil",source="spacy/lexeme.pxd"}
Set the value of an attribute of the `LexemeC` struct by attribute ID. Set the value of an attribute of the `LexemeC` struct by attribute ID.
@ -212,7 +212,7 @@ Set the value of an attribute of the `LexemeC` struct by attribute ID.
| `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ | | `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ |
| `value` | The value to set. ~~attr_t (uint64_t)~~ | | `value` | The value to set. ~~attr_t (uint64_t)~~ |
### Lexeme.c_check_flag {#lexeme_c_check_flag tag="staticmethod, nogil" source="spacy/lexeme.pxd"} ### Lexeme.c_check_flag {id="lexeme_c_check_flag",tag="staticmethod, nogil",source="spacy/lexeme.pxd"}
Check the value of a binary flag attribute. Check the value of a binary flag attribute.
@ -232,7 +232,7 @@ Check the value of a binary flag attribute.
| `flag_id` | The ID of the flag to look up. The flag IDs are enumerated in `spacy.typedefs`. ~~attr_id_t~~ | | `flag_id` | The ID of the flag to look up. The flag IDs are enumerated in `spacy.typedefs`. ~~attr_id_t~~ |
| **RETURNS** | The boolean value of the flag. ~~bint~~ | | **RETURNS** | The boolean value of the flag. ~~bint~~ |
### Lexeme.c_set_flag {#lexeme_c_set_flag tag="staticmethod, nogil" source="spacy/lexeme.pxd"} ### Lexeme.c_set_flag {id="lexeme_c_set_flag",tag="staticmethod, nogil",source="spacy/lexeme.pxd"}
Set the value of a binary flag attribute. Set the value of a binary flag attribute.

View File

@ -6,7 +6,7 @@ menu:
- ['Conventions', 'conventions'] - ['Conventions', 'conventions']
--- ---
## Overview {#overview hidden="true"} ## Overview {id="overview",hidden="true"}
> #### What's Cython? > #### What's Cython?
> >
@ -37,7 +37,7 @@ class holds a [`LexemeC`](/api/cython-structs#lexemec) struct, at `Lexeme.c`.
This lets you shed the Python container, and pass a pointer to the underlying This lets you shed the Python container, and pass a pointer to the underlying
data into C-level functions. data into C-level functions.
## Conventions {#conventions} ## Conventions {id="conventions"}
spaCy's core data structures are implemented as [Cython](http://cython.org/) spaCy's core data structures are implemented as [Cython](http://cython.org/)
`cdef` classes. Memory is managed through the `cdef` classes. Memory is managed through the

View File

@ -14,7 +14,7 @@ vocabulary data. For an overview of label schemes used by the models, see the
[models directory](/models). Each trained pipeline documents the label schemes [models directory](/models). Each trained pipeline documents the label schemes
used in its components, depending on the data it was trained on. used in its components, depending on the data it was trained on.
## Training config {#config new="3"} ## Training config {id="config",version="3"}
Config files define the training process and pipeline and can be passed to Config files define the training process and pipeline and can be passed to
[`spacy train`](/api/cli#train). They use [`spacy train`](/api/cli#train). They use
@ -52,7 +52,7 @@ your config and check that it's valid, you can run the
</Infobox> </Infobox>
### nlp {#config-nlp tag="section"} ### nlp {id="config-nlp",tag="section"}
> #### Example > #### Example
> >
@ -83,7 +83,7 @@ Defines the `nlp` object, its tokenizer and
| `tokenizer` | The tokenizer to use. Defaults to [`Tokenizer`](/api/tokenizer). ~~Callable[[str], Doc]~~ | | `tokenizer` | The tokenizer to use. Defaults to [`Tokenizer`](/api/tokenizer). ~~Callable[[str], Doc]~~ |
| `batch_size` | Default batch size for [`Language.pipe`](/api/language#pipe) and [`Language.evaluate`](/api/language#evaluate). ~~int~~ | | `batch_size` | Default batch size for [`Language.pipe`](/api/language#pipe) and [`Language.evaluate`](/api/language#evaluate). ~~int~~ |
### components {#config-components tag="section"} ### components {id="config-components",tag="section"}
> #### Example > #### Example
> >
@ -106,7 +106,7 @@ function to use to create component) or a `source` (name of path of trained
pipeline to copy components from). See the docs on pipeline to copy components from). See the docs on
[defining pipeline components](/usage/training#config-components) for details. [defining pipeline components](/usage/training#config-components) for details.
### paths, system {#config-variables tag="variables"} ### paths, system {id="config-variables",tag="variables"}
These sections define variables that can be referenced across the other sections These sections define variables that can be referenced across the other sections
as variables. For example `${paths.train}` uses the value of `train` defined in as variables. For example `${paths.train}` uses the value of `train` defined in
@ -116,11 +116,11 @@ need paths, you can define them here. All config values can also be
[`spacy train`](/api/cli#train), which is especially relevant for data paths [`spacy train`](/api/cli#train), which is especially relevant for data paths
that you don't want to hard-code in your config file. that you don't want to hard-code in your config file.
```cli ```bash
$ python -m spacy train config.cfg --paths.train ./corpus/train.spacy $ python -m spacy train config.cfg --paths.train ./corpus/train.spacy
``` ```
### corpora {#config-corpora tag="section"} ### corpora {id="config-corpora",tag="section"}
> #### Example > #### Example
> >
@ -176,7 +176,7 @@ single corpus once and then divide it up into `train` and `dev` partitions.
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `corpora` | A dictionary keyed by string names, mapped to corpus functions that receive the current `nlp` object and return an iterator of [`Example`](/api/example) objects. ~~Dict[str, Callable[[Language], Iterator[Example]]]~~ | | `corpora` | A dictionary keyed by string names, mapped to corpus functions that receive the current `nlp` object and return an iterator of [`Example`](/api/example) objects. ~~Dict[str, Callable[[Language], Iterator[Example]]]~~ |
### training {#config-training tag="section"} ### training {id="config-training",tag="section"}
This section defines settings and controls for the training and evaluation This section defines settings and controls for the training and evaluation
process that are used when you run [`spacy train`](/api/cli#train). process that are used when you run [`spacy train`](/api/cli#train).
@ -202,7 +202,7 @@ process that are used when you run [`spacy train`](/api/cli#train).
| `seed` | The random seed. Defaults to variable `${system.seed}`. ~~int~~ | | `seed` | The random seed. Defaults to variable `${system.seed}`. ~~int~~ |
| `train_corpus` | Dot notation of the config location defining the train corpus. Defaults to `corpora.train`. ~~str~~ | | `train_corpus` | Dot notation of the config location defining the train corpus. Defaults to `corpora.train`. ~~str~~ |
### pretraining {#config-pretraining tag="section,optional"} ### pretraining {id="config-pretraining",tag="section,optional"}
This section is optional and defines settings and controls for This section is optional and defines settings and controls for
[language model pretraining](/usage/embeddings-transformers#pretraining). It's [language model pretraining](/usage/embeddings-transformers#pretraining). It's
@ -220,7 +220,7 @@ used when you run [`spacy pretrain`](/api/cli#pretrain).
| `component` | Component name to identify the layer with the model to pretrain. Defaults to `"tok2vec"`. ~~str~~ | | `component` | Component name to identify the layer with the model to pretrain. Defaults to `"tok2vec"`. ~~str~~ |
| `layer` | The specific layer of the model to pretrain. If empty, the whole model will be used. ~~str~~ | | `layer` | The specific layer of the model to pretrain. If empty, the whole model will be used. ~~str~~ |
### initialize {#config-initialize tag="section"} ### initialize {id="config-initialize",tag="section"}
This config block lets you define resources for **initializing the pipeline**. This config block lets you define resources for **initializing the pipeline**.
It's used by [`Language.initialize`](/api/language#initialize) and typically It's used by [`Language.initialize`](/api/language#initialize) and typically
@ -255,9 +255,9 @@ Also see the usage guides on the
| `vectors` | Name or path of pipeline containing pretrained word vectors to use, e.g. created with [`init vectors`](/api/cli#init-vectors). Defaults to `null`. ~~Optional[str]~~ | | `vectors` | Name or path of pipeline containing pretrained word vectors to use, e.g. created with [`init vectors`](/api/cli#init-vectors). Defaults to `null`. ~~Optional[str]~~ |
| `vocab_data` | Path to JSONL-formatted [vocabulary file](/api/data-formats#vocab-jsonl) to initialize vocabulary. ~~Optional[str]~~ | | `vocab_data` | Path to JSONL-formatted [vocabulary file](/api/data-formats#vocab-jsonl) to initialize vocabulary. ~~Optional[str]~~ |
## Training data {#training} ## Training data {id="training"}
### Binary training format {#binary-training new="3"} ### Binary training format {id="binary-training",version="3"}
> #### Example > #### Example
> >
@ -288,7 +288,7 @@ Note that while this is the format used to save training data, you do not have
to understand the internal details to use it or create training data. See the to understand the internal details to use it or create training data. See the
section on [preparing training data](/usage/training#training-data). section on [preparing training data](/usage/training#training-data).
### JSON training format {#json-input tag="deprecated"} ### JSON training format {id="json-input",tag="deprecated"}
<Infobox variant="warning" title="Changed in v3.0"> <Infobox variant="warning" title="Changed in v3.0">
@ -300,7 +300,7 @@ objects to JSON, you can now serialize them directly using the
[`spacy convert`](/api/cli) lets you convert your JSON data to the new `.spacy` [`spacy convert`](/api/cli) lets you convert your JSON data to the new `.spacy`
format: format:
```cli ```bash
$ python -m spacy convert ./data.json . $ python -m spacy convert ./data.json .
``` ```
@ -317,8 +317,7 @@ $ python -m spacy convert ./data.json .
> [`offsets_to_biluo_tags`](/api/top-level#offsets_to_biluo_tags) function can > [`offsets_to_biluo_tags`](/api/top-level#offsets_to_biluo_tags) function can
> help you convert entity offsets to the right format. > help you convert entity offsets to the right format.
```python ```python {title="Example structure"}
### Example structure
[{ [{
"id": int, # ID of the document within the corpus "id": int, # ID of the document within the corpus
"paragraphs": [{ # list of paragraphs in the corpus "paragraphs": [{ # list of paragraphs in the corpus
@ -357,7 +356,7 @@ https://github.com/explosion/spaCy/blob/v2.3.x/examples/training/training-data.j
</Accordion> </Accordion>
### Annotation format for creating training examples {#dict-input} ### Annotation format for creating training examples {id="dict-input"}
An [`Example`](/api/example) object holds the information for one training An [`Example`](/api/example) object holds the information for one training
instance. It stores two [`Doc`](/api/doc) objects: one for holding the instance. It stores two [`Doc`](/api/doc) objects: one for holding the
@ -436,8 +435,7 @@ file to keep track of your settings and hyperparameters and your own
</Infobox> </Infobox>
```python ```python {title="Examples"}
### Examples
# Training data for a part-of-speech tagger # Training data for a part-of-speech tagger
doc = Doc(vocab, words=["I", "like", "stuff"]) doc = Doc(vocab, words=["I", "like", "stuff"])
gold_dict = {"tags": ["NOUN", "VERB", "NOUN"]} gold_dict = {"tags": ["NOUN", "VERB", "NOUN"]}
@ -466,7 +464,7 @@ gold_dict = {"entities": [(0, 12, "PERSON")],
example = Example.from_dict(doc, gold_dict) example = Example.from_dict(doc, gold_dict)
``` ```
## Lexical data for vocabulary {#vocab-jsonl new="2"} ## Lexical data for vocabulary {id="vocab-jsonl",version="2"}
This data file can be provided via the `vocab_data` setting in the This data file can be provided via the `vocab_data` setting in the
`[initialize]` block of the training config to pre-define the lexical data to `[initialize]` block of the training config to pre-define the lexical data to
@ -483,13 +481,11 @@ spaCy's [`Lexeme`](/api/lexeme#attributes) object.
> vocab_data = "/path/to/vocab-data.jsonl" > vocab_data = "/path/to/vocab-data.jsonl"
> ``` > ```
```python ```python {title="First line"}
### First line
{"lang": "en", "settings": {"oov_prob": -20.502029418945312}} {"lang": "en", "settings": {"oov_prob": -20.502029418945312}}
``` ```
```python ```python {title="Entry structure"}
### Entry structure
{ {
"orth": string, # the word text "orth": string, # the word text
"id": int, # can correspond to row in vectors table "id": int, # can correspond to row in vectors table
@ -526,7 +522,7 @@ Here's an example of the 20 most frequent lexemes in the English training data:
%%GITHUB_SPACY/extra/example_data/vocab-data.jsonl %%GITHUB_SPACY/extra/example_data/vocab-data.jsonl
``` ```
## Pipeline meta {#meta} ## Pipeline meta {id="meta"}
The pipeline meta is available as the file `meta.json` and exported The pipeline meta is available as the file `meta.json` and exported
automatically when you save an `nlp` object to disk. Its contents are available automatically when you save an `nlp` object to disk. Its contents are available

View File

@ -2,7 +2,7 @@
title: DependencyMatcher title: DependencyMatcher
teaser: Match subtrees within a dependency parse teaser: Match subtrees within a dependency parse
tag: class tag: class
new: 3 version: 3
source: spacy/matcher/dependencymatcher.pyx source: spacy/matcher/dependencymatcher.pyx
--- ---
@ -14,7 +14,7 @@ It requires a pretrained [`DependencyParser`](/api/parser) or other component
that sets the `Token.dep` and `Token.head` attributes. See the that sets the `Token.dep` and `Token.head` attributes. See the
[usage guide](/usage/rule-based-matching#dependencymatcher) for examples. [usage guide](/usage/rule-based-matching#dependencymatcher) for examples.
## Pattern format {#patterns} ## Pattern format {id="patterns"}
> ```python > ```python
> ### Example > ### Example
@ -62,7 +62,7 @@ of relations, see the usage guide on
</Infobox> </Infobox>
### Operators {#operators} ### Operators {id="operators"}
The following operators are supported by the `DependencyMatcher`, most of which The following operators are supported by the `DependencyMatcher`, most of which
come directly from come directly from
@ -87,8 +87,7 @@ come directly from
| `A <++ B` | `B` is a right parent of `A`, i.e. `A` is a child of `B` and `A.i < B.i` _(not in Semgrex)_. | | `A <++ B` | `B` is a right parent of `A`, i.e. `A` is a child of `B` and `A.i < B.i` _(not in Semgrex)_. |
| `A <-- B` | `B` is a left parent of `A`, i.e. `A` is a child of `B` and `A.i > B.i` _(not in Semgrex)_. | | `A <-- B` | `B` is a left parent of `A`, i.e. `A` is a child of `B` and `A.i > B.i` _(not in Semgrex)_. |
## DependencyMatcher.\_\_init\_\_ {id="init",tag="method"}
## DependencyMatcher.\_\_init\_\_ {#init tag="method"}
Create a `DependencyMatcher`. Create a `DependencyMatcher`.
@ -105,7 +104,7 @@ Create a `DependencyMatcher`.
| _keyword-only_ | | | _keyword-only_ | |
| `validate` | Validate all patterns added to this matcher. ~~bool~~ | | `validate` | Validate all patterns added to this matcher. ~~bool~~ |
## DependencyMatcher.\_\call\_\_ {#call tag="method"} ## DependencyMatcher.\_\_call\_\_ {id="call",tag="method"}
Find all tokens matching the supplied patterns on the `Doc` or `Span`. Find all tokens matching the supplied patterns on the `Doc` or `Span`.
@ -127,7 +126,7 @@ Find all tokens matching the supplied patterns on the `Doc` or `Span`.
| `doclike` | The `Doc` or `Span` to match over. ~~Union[Doc, Span]~~ | | `doclike` | The `Doc` or `Span` to match over. ~~Union[Doc, Span]~~ |
| **RETURNS** | A list of `(match_id, token_ids)` tuples, describing the matches. The `match_id` is the ID of the match pattern and `token_ids` is a list of token indices matched by the pattern, where the position of each token in the list corresponds to the position of the node specification in the pattern. ~~List[Tuple[int, List[int]]]~~ | | **RETURNS** | A list of `(match_id, token_ids)` tuples, describing the matches. The `match_id` is the ID of the match pattern and `token_ids` is a list of token indices matched by the pattern, where the position of each token in the list corresponds to the position of the node specification in the pattern. ~~List[Tuple[int, List[int]]]~~ |
## DependencyMatcher.\_\_len\_\_ {#len tag="method"} ## DependencyMatcher.\_\_len\_\_ {id="len",tag="method"}
Get the number of rules added to the dependency matcher. Note that this only Get the number of rules added to the dependency matcher. Note that this only
returns the number of rules (identical with the number of IDs), not the number returns the number of rules (identical with the number of IDs), not the number
@ -148,7 +147,7 @@ of individual patterns.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The number of rules. ~~int~~ | | **RETURNS** | The number of rules. ~~int~~ |
## DependencyMatcher.\_\_contains\_\_ {#contains tag="method"} ## DependencyMatcher.\_\_contains\_\_ {id="contains",tag="method"}
Check whether the matcher contains rules for a match ID. Check whether the matcher contains rules for a match ID.
@ -166,7 +165,7 @@ Check whether the matcher contains rules for a match ID.
| `key` | The match ID. ~~str~~ | | `key` | The match ID. ~~str~~ |
| **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ | | **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ |
## DependencyMatcher.add {#add tag="method"} ## DependencyMatcher.add {id="add",tag="method"}
Add a rule to the matcher, consisting of an ID key, one or more patterns, and an Add a rule to the matcher, consisting of an ID key, one or more patterns, and an
optional callback function to act on the matches. The callback function will optional callback function to act on the matches. The callback function will
@ -191,7 +190,7 @@ will be overwritten.
| _keyword-only_ | | | _keyword-only_ | |
| `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[DependencyMatcher, Doc, int, List[Tuple], Any]]~~ | | `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[DependencyMatcher, Doc, int, List[Tuple], Any]]~~ |
## DependencyMatcher.get {#get tag="method"} ## DependencyMatcher.get {id="get",tag="method"}
Retrieve the pattern stored for a key. Returns the rule as an Retrieve the pattern stored for a key. Returns the rule as an
`(on_match, patterns)` tuple containing the callback and available patterns. `(on_match, patterns)` tuple containing the callback and available patterns.
@ -208,7 +207,7 @@ Retrieve the pattern stored for a key. Returns the rule as an
| `key` | The ID of the match rule. ~~str~~ | | `key` | The ID of the match rule. ~~str~~ |
| **RETURNS** | The rule, as an `(on_match, patterns)` tuple. ~~Tuple[Optional[Callable], List[List[Union[Dict, Tuple]]]]~~ | | **RETURNS** | The rule, as an `(on_match, patterns)` tuple. ~~Tuple[Optional[Callable], List[List[Union[Dict, Tuple]]]]~~ |
## DependencyMatcher.remove {#remove tag="method"} ## DependencyMatcher.remove {id="remove",tag="method"}
Remove a rule from the dependency matcher. A `KeyError` is raised if the match Remove a rule from the dependency matcher. A `KeyError` is raised if the match
ID does not exist. ID does not exist.

View File

@ -25,7 +25,7 @@ current state. The weights are updated such that the scores assigned to the set
of optimal actions is increased, while scores assigned to other actions are of optimal actions is increased, while scores assigned to other actions are
decreased. Note that more than one action may be optimal for a given state. decreased. Note that more than one action may be optimal for a given state.
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Dependency predictions are assigned to the `Token.dep` and `Token.head` fields. Dependency predictions are assigned to the `Token.dep` and `Token.head` fields.
Beside the dependencies themselves, the parser decides sentence boundaries, Beside the dependencies themselves, the parser decides sentence boundaries,
@ -39,7 +39,7 @@ which are saved in `Token.is_sent_start` and accessible via `Doc.sents`.
| `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. After the parser runs this will be `True` or `False` for all tokens. ~~bool~~ | | `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. After the parser runs this will be `True` or `False` for all tokens. ~~bool~~ |
| `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ | | `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -74,7 +74,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/dep_parser.pyx %%GITHUB_SPACY/spacy/pipeline/dep_parser.pyx
``` ```
## DependencyParser.\_\_init\_\_ {#init tag="method"} ## DependencyParser.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -107,7 +107,7 @@ shortcut for this and instantiate the component using its string name and
| `min_action_freq` | The minimum frequency of labelled actions to retain. Rarer labelled actions have their label backed-off to "dep". While this primarily affects the label accuracy, it can also affect the attachment structure, as the labels are used to represent the pseudo-projectivity transformation. ~~int~~ | | `min_action_freq` | The minimum frequency of labelled actions to retain. Rarer labelled actions have their label backed-off to "dep". While this primarily affects the label accuracy, it can also affect the attachment structure, as the labels are used to represent the pseudo-projectivity transformation. ~~int~~ |
| `scorer` | The scoring method. Defaults to [`Scorer.score_deps`](/api/scorer#score_deps) for the attribute `"dep"` ignoring the labels `p` and `punct` and [`Scorer.score_spans`](/api/scorer/#score_spans) for the attribute `"sents"`. ~~Optional[Callable]~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_deps`](/api/scorer#score_deps) for the attribute `"dep"` ignoring the labels `p` and `punct` and [`Scorer.score_spans`](/api/scorer/#score_spans) for the attribute `"sents"`. ~~Optional[Callable]~~ |
## DependencyParser.\_\_call\_\_ {#call tag="method"} ## DependencyParser.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned. Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -131,7 +131,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## DependencyParser.pipe {#pipe tag="method"} ## DependencyParser.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -155,7 +155,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/dependencyparser#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## DependencyParser.initialize {#initialize tag="method" new="3"} ## DependencyParser.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -198,7 +198,7 @@ This method was previously called `begin_training`.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Dict[str, Dict[str, int]]]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Dict[str, Dict[str, int]]]~~ |
## DependencyParser.predict {#predict tag="method"} ## DependencyParser.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. modifying them.
@ -215,7 +215,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | A helper class for the parse state (internal). ~~StateClass~~ | | **RETURNS** | A helper class for the parse state (internal). ~~StateClass~~ |
## DependencyParser.set_annotations {#set_annotations tag="method"} ## DependencyParser.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -232,7 +232,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `DependencyParser.predict`. Returns an internal helper class for the parse state. ~~List[StateClass]~~ | | `scores` | The scores to set, produced by `DependencyParser.predict`. Returns an internal helper class for the parse state. ~~List[StateClass]~~ |
## DependencyParser.update {#update tag="method"} ## DependencyParser.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects, updating the pipe's Learn from a batch of [`Example`](/api/example) objects, updating the pipe's
model. Delegates to [`predict`](/api/dependencyparser#predict) and model. Delegates to [`predict`](/api/dependencyparser#predict) and
@ -255,7 +255,7 @@ model. Delegates to [`predict`](/api/dependencyparser#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## DependencyParser.get_loss {#get_loss tag="method"} ## DependencyParser.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their Find the loss and gradient of loss for the batch of documents and their
predicted scores. predicted scores.
@ -274,7 +274,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. ~~StateClass~~ | | `scores` | Scores representing the model's predictions. ~~StateClass~~ |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## DependencyParser.create_optimizer {#create_optimizer tag="method"} ## DependencyParser.create_optimizer {id="create_optimizer",tag="method"}
Create an [`Optimizer`](https://thinc.ai/docs/api-optimizers) for the pipeline Create an [`Optimizer`](https://thinc.ai/docs/api-optimizers) for the pipeline
component. component.
@ -290,7 +290,7 @@ component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## DependencyParser.use_params {#use_params tag="method, contextmanager"} ## DependencyParser.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -307,7 +307,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## DependencyParser.add_label {#add_label tag="method"} ## DependencyParser.add_label {id="add_label",tag="method"}
Add a new label to the pipe. Note that you don't have to call this method if you Add a new label to the pipe. Note that you don't have to call this method if you
provide a **representative data sample** to the [`initialize`](#initialize) provide a **representative data sample** to the [`initialize`](#initialize)
@ -327,7 +327,7 @@ to the model, and the output dimension will be
| `label` | The label to add. ~~str~~ | | `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## DependencyParser.set_output {#set_output tag="method"} ## DependencyParser.set_output {id="set_output",tag="method"}
Change the output dimension of the component's model by calling the model's Change the output dimension of the component's model by calling the model's
attribute `resize_output`. This is a function that takes the original model and attribute `resize_output`. This is a function that takes the original model and
@ -346,7 +346,7 @@ forgetting" problem.
| ---- | --------------------------------- | | ---- | --------------------------------- |
| `nO` | The new output dimension. ~~int~~ | | `nO` | The new output dimension. ~~int~~ |
## DependencyParser.to_disk {#to_disk tag="method"} ## DependencyParser.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -363,7 +363,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## DependencyParser.from_disk {#from_disk tag="method"} ## DependencyParser.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -381,7 +381,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `DependencyParser` object. ~~DependencyParser~~ | | **RETURNS** | The modified `DependencyParser` object. ~~DependencyParser~~ |
## DependencyParser.to_bytes {#to_bytes tag="method"} ## DependencyParser.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -398,7 +398,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `DependencyParser` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `DependencyParser` object. ~~bytes~~ |
## DependencyParser.from_bytes {#from_bytes tag="method"} ## DependencyParser.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -417,7 +417,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `DependencyParser` object. ~~DependencyParser~~ | | **RETURNS** | The `DependencyParser` object. ~~DependencyParser~~ |
## DependencyParser.labels {#labels tag="property"} ## DependencyParser.labels {id="labels",tag="property"}
The labels currently added to the component. The labels currently added to the component.
@ -432,7 +432,7 @@ The labels currently added to the component.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## DependencyParser.label_data {#label_data tag="property" new="3"} ## DependencyParser.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information. The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -450,7 +450,7 @@ the model with a pre-defined label set.
| ----------- | ------------------------------------------------------------------------------- | | ----------- | ------------------------------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Dict[str, Dict[str, Dict[str, int]]]~~ | | **RETURNS** | The label data added to the component. ~~Dict[str, Dict[str, Dict[str, int]]]~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -12,7 +12,7 @@ compressed binary strings. The `Doc` object holds an array of
[`Span`](/api/span) objects are views of this array, i.e. they don't own the [`Span`](/api/span) objects are views of this array, i.e. they don't own the
data themselves. data themselves.
## Doc.\_\_init\_\_ {#init tag="method"} ## Doc.\_\_init\_\_ {id="init",tag="method"}
Construct a `Doc` object. The most common way to get a `Doc` object is via the Construct a `Doc` object. The most common way to get a `Doc` object is via the
`nlp` object. `nlp` object.
@ -47,7 +47,7 @@ Construct a `Doc` object. The most common way to get a `Doc` object is via the
| `sent_starts` <Tag variant="new">3</Tag> | A list of values, of the same length as `words`, to assign as `token.is_sent_start`. Will be overridden by heads if `heads` is provided. Defaults to `None`. ~~Optional[List[Union[bool, int, None]]]~~ | | `sent_starts` <Tag variant="new">3</Tag> | A list of values, of the same length as `words`, to assign as `token.is_sent_start`. Will be overridden by heads if `heads` is provided. Defaults to `None`. ~~Optional[List[Union[bool, int, None]]]~~ |
| `ents` <Tag variant="new">3</Tag> | A list of strings, of the same length of `words`, to assign the token-based IOB tag. Defaults to `None`. ~~Optional[List[str]]~~ | | `ents` <Tag variant="new">3</Tag> | A list of strings, of the same length of `words`, to assign the token-based IOB tag. Defaults to `None`. ~~Optional[List[str]]~~ |
## Doc.\_\_getitem\_\_ {#getitem tag="method"} ## Doc.\_\_getitem\_\_ {id="getitem",tag="method"}
Get a [`Token`](/api/token) object at position `i`, where `i` is an integer. Get a [`Token`](/api/token) object at position `i`, where `i` is an integer.
Negative indexing is supported, and follows the usual Python semantics, i.e. Negative indexing is supported, and follows the usual Python semantics, i.e.
@ -80,7 +80,7 @@ semantics.
| `start_end` | The slice of the document to get. ~~Tuple[int, int]~~ | | `start_end` | The slice of the document to get. ~~Tuple[int, int]~~ |
| **RETURNS** | The span at `doc[start:end]`. ~~Span~~ | | **RETURNS** | The span at `doc[start:end]`. ~~Span~~ |
## Doc.\_\_iter\_\_ {#iter tag="method"} ## Doc.\_\_iter\_\_ {id="iter",tag="method"}
Iterate over `Token` objects, from which the annotations can be easily accessed. Iterate over `Token` objects, from which the annotations can be easily accessed.
@ -100,7 +100,7 @@ underlying C data directly from Cython.
| ---------- | --------------------------- | | ---------- | --------------------------- |
| **YIELDS** | A `Token` object. ~~Token~~ | | **YIELDS** | A `Token` object. ~~Token~~ |
## Doc.\_\_len\_\_ {#len tag="method"} ## Doc.\_\_len\_\_ {id="len",tag="method"}
Get the number of tokens in the document. Get the number of tokens in the document.
@ -115,7 +115,7 @@ Get the number of tokens in the document.
| ----------- | --------------------------------------------- | | ----------- | --------------------------------------------- |
| **RETURNS** | The number of tokens in the document. ~~int~~ | | **RETURNS** | The number of tokens in the document. ~~int~~ |
## Doc.set_extension {#set_extension tag="classmethod" new="2"} ## Doc.set_extension {id="set_extension",tag="classmethod",version="2"}
Define a custom attribute on the `Doc` which becomes available via `Doc._`. For Define a custom attribute on the `Doc` which becomes available via `Doc._`. For
details, see the documentation on details, see the documentation on
@ -140,7 +140,7 @@ details, see the documentation on
| `setter` | Setter function that takes the `Doc` and a value, and modifies the object. Is called when the user writes to the `Doc._` attribute. ~~Optional[Callable[[Doc, Any], None]]~~ | | `setter` | Setter function that takes the `Doc` and a value, and modifies the object. Is called when the user writes to the `Doc._` attribute. ~~Optional[Callable[[Doc, Any], None]]~~ |
| `force` | Force overwriting existing attribute. ~~bool~~ | | `force` | Force overwriting existing attribute. ~~bool~~ |
## Doc.get_extension {#get_extension tag="classmethod" new="2"} ## Doc.get_extension {id="get_extension",tag="classmethod",version="2"}
Look up a previously registered extension by name. Returns a 4-tuple Look up a previously registered extension by name. Returns a 4-tuple
`(default, method, getter, setter)` if the extension is registered. Raises a `(default, method, getter, setter)` if the extension is registered. Raises a
@ -160,7 +160,7 @@ Look up a previously registered extension by name. Returns a 4-tuple
| `name` | Name of the extension. ~~str~~ | | `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Doc.has_extension {#has_extension tag="classmethod" new="2"} ## Doc.has_extension {id="has_extension",tag="classmethod",version="2"}
Check whether an extension has been registered on the `Doc` class. Check whether an extension has been registered on the `Doc` class.
@ -177,7 +177,7 @@ Check whether an extension has been registered on the `Doc` class.
| `name` | Name of the extension to check. ~~str~~ | | `name` | Name of the extension to check. ~~str~~ |
| **RETURNS** | Whether the extension has been registered. ~~bool~~ | | **RETURNS** | Whether the extension has been registered. ~~bool~~ |
## Doc.remove_extension {#remove_extension tag="classmethod" new="2.0.12"} ## Doc.remove_extension {id="remove_extension",tag="classmethod",version="2.0.12"}
Remove a previously registered extension. Remove a previously registered extension.
@ -195,7 +195,7 @@ Remove a previously registered extension.
| `name` | Name of the extension. ~~str~~ | | `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Doc.char_span {#char_span tag="method" new="2"} ## Doc.char_span {id="char_span",tag="method",version="2"}
Create a `Span` object from the slice `doc.text[start_idx:end_idx]`. Returns Create a `Span` object from the slice `doc.text[start_idx:end_idx]`. Returns
`None` if the character indices don't map to a valid span using the default `None` if the character indices don't map to a valid span using the default
@ -219,7 +219,7 @@ alignment mode `"strict".
| `alignment_mode` | How character indices snap to token boundaries. Options: `"strict"` (no snapping), `"contract"` (span of all tokens completely within the character span), `"expand"` (span of all tokens at least partially covered by the character span). Defaults to `"strict"`. ~~str~~ | | `alignment_mode` | How character indices snap to token boundaries. Options: `"strict"` (no snapping), `"contract"` (span of all tokens completely within the character span), `"expand"` (span of all tokens at least partially covered by the character span). Defaults to `"strict"`. ~~str~~ |
| **RETURNS** | The newly constructed object or `None`. ~~Optional[Span]~~ | | **RETURNS** | The newly constructed object or `None`. ~~Optional[Span]~~ |
## Doc.set_ents {#set_ents tag="method" new="3"} ## Doc.set_ents {id="set_ents",tag="method",version="3"}
Set the named entities in the document. Set the named entities in the document.
@ -243,7 +243,7 @@ Set the named entities in the document.
| `outside` | Spans outside of entities (O in IOB). ~~Optional[List[Span]]~~ | | `outside` | Spans outside of entities (O in IOB). ~~Optional[List[Span]]~~ |
| `default` | How to set entity annotation for tokens outside of any provided spans. Options: `"blocked"`, `"missing"`, `"outside"` and `"unmodified"` (preserve current state). Defaults to `"outside"`. ~~str~~ | | `default` | How to set entity annotation for tokens outside of any provided spans. Options: `"blocked"`, `"missing"`, `"outside"` and `"unmodified"` (preserve current state). Defaults to `"outside"`. ~~str~~ |
## Doc.similarity {#similarity tag="method" model="vectors"} ## Doc.similarity {id="similarity",tag="method",model="vectors"}
Make a semantic similarity estimate. The default estimate is cosine similarity Make a semantic similarity estimate. The default estimate is cosine similarity
using an average of word vectors. using an average of word vectors.
@ -263,7 +263,7 @@ using an average of word vectors.
| `other` | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ | | `other` | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ |
| **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ | | **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ |
## Doc.count_by {#count_by tag="method"} ## Doc.count_by {id="count_by",tag="method"}
Count the frequencies of a given attribute. Produces a dict of Count the frequencies of a given attribute. Produces a dict of
`{attr (int): count (ints)}` frequencies, keyed by the values of the given `{attr (int): count (ints)}` frequencies, keyed by the values of the given
@ -284,7 +284,7 @@ attribute ID.
| `attr_id` | The attribute ID. ~~int~~ | | `attr_id` | The attribute ID. ~~int~~ |
| **RETURNS** | A dictionary mapping attributes to integer counts. ~~Dict[int, int]~~ | | **RETURNS** | A dictionary mapping attributes to integer counts. ~~Dict[int, int]~~ |
## Doc.get_lca_matrix {#get_lca_matrix tag="method"} ## Doc.get_lca_matrix {id="get_lca_matrix",tag="method"}
Calculates the lowest common ancestor matrix for a given `Doc`. Returns LCA Calculates the lowest common ancestor matrix for a given `Doc`. Returns LCA
matrix containing the integer index of the ancestor, or `-1` if no common matrix containing the integer index of the ancestor, or `-1` if no common
@ -302,7 +302,7 @@ ancestor is found, e.g. if span excludes a necessary ancestor.
| ----------- | -------------------------------------------------------------------------------------- | | ----------- | -------------------------------------------------------------------------------------- |
| **RETURNS** | The lowest common ancestor matrix of the `Doc`. ~~numpy.ndarray[ndim=2, dtype=int32]~~ | | **RETURNS** | The lowest common ancestor matrix of the `Doc`. ~~numpy.ndarray[ndim=2, dtype=int32]~~ |
## Doc.has_annotation {#has_annotation tag="method"} ## Doc.has_annotation {id="has_annotation",tag="method"}
Check whether the doc contains annotation on a Check whether the doc contains annotation on a
[`Token` attribute](/api/token#attributes). [`Token` attribute](/api/token#attributes).
@ -327,7 +327,7 @@ doc = nlp("This is a text")
| `require_complete` | Whether to check that the attribute is set on every token in the doc. Defaults to `False`. ~~bool~~ | | `require_complete` | Whether to check that the attribute is set on every token in the doc. Defaults to `False`. ~~bool~~ |
| **RETURNS** | Whether specified annotation is present in the doc. ~~bool~~ | | **RETURNS** | Whether specified annotation is present in the doc. ~~bool~~ |
## Doc.to_array {#to_array tag="method"} ## Doc.to_array {id="to_array",tag="method"}
Export given token attributes to a numpy `ndarray`. If `attr_ids` is a sequence Export given token attributes to a numpy `ndarray`. If `attr_ids` is a sequence
of `M` attributes, the output array will be of shape `(N, M)`, where `N` is the of `M` attributes, the output array will be of shape `(N, M)`, where `N` is the
@ -355,7 +355,7 @@ Returns a 2D array with one row per token and one column per attribute (when
| `attr_ids` | A list of attributes (int IDs or string names) or a single attribute (int ID or string name). ~~Union[int, str, List[Union[int, str]]]~~ | | `attr_ids` | A list of attributes (int IDs or string names) or a single attribute (int ID or string name). ~~Union[int, str, List[Union[int, str]]]~~ |
| **RETURNS** | The exported attributes as a numpy array. ~~Union[numpy.ndarray[ndim=2, dtype=uint64], numpy.ndarray[ndim=1, dtype=uint64]]~~ | | **RETURNS** | The exported attributes as a numpy array. ~~Union[numpy.ndarray[ndim=2, dtype=uint64], numpy.ndarray[ndim=1, dtype=uint64]]~~ |
## Doc.from_array {#from_array tag="method"} ## Doc.from_array {id="from_array",tag="method"}
Load attributes from a numpy array. Write to a `Doc` object, from an `(M, N)` Load attributes from a numpy array. Write to a `Doc` object, from an `(M, N)`
array of attributes. array of attributes.
@ -379,7 +379,7 @@ array of attributes.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Doc` itself. ~~Doc~~ | | **RETURNS** | The `Doc` itself. ~~Doc~~ |
## Doc.from_docs {#from_docs tag="staticmethod" new="3"} ## Doc.from_docs {id="from_docs",tag="staticmethod",version="3"}
Concatenate multiple `Doc` objects to form a new one. Raises an error if the Concatenate multiple `Doc` objects to form a new one. Raises an error if the
`Doc` objects do not all share the same `Vocab`. `Doc` objects do not all share the same `Vocab`.
@ -408,7 +408,7 @@ Concatenate multiple `Doc` objects to form a new one. Raises an error if the
| `exclude` <Tag variant="new">3.3</Tag> | String names of Doc attributes to exclude. Supported: `spans`, `tensor`, `user_data`. ~~Iterable[str]~~ | | `exclude` <Tag variant="new">3.3</Tag> | String names of Doc attributes to exclude. Supported: `spans`, `tensor`, `user_data`. ~~Iterable[str]~~ |
| **RETURNS** | The new `Doc` object that is containing the other docs or `None`, if `docs` is empty or `None`. ~~Optional[Doc]~~ | | **RETURNS** | The new `Doc` object that is containing the other docs or `None`, if `docs` is empty or `None`. ~~Optional[Doc]~~ |
## Doc.to_disk {#to_disk tag="method" new="2"} ## Doc.to_disk {id="to_disk",tag="method",version="2"}
Save the current state to a directory. Save the current state to a directory.
@ -424,7 +424,7 @@ Save the current state to a directory.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Doc.from_disk {#from_disk tag="method" new="2"} ## Doc.from_disk {id="from_disk",tag="method",version="2"}
Loads state from a directory. Modifies the object in place and returns it. Loads state from a directory. Modifies the object in place and returns it.
@ -443,7 +443,7 @@ Loads state from a directory. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Doc` object. ~~Doc~~ | | **RETURNS** | The modified `Doc` object. ~~Doc~~ |
## Doc.to_bytes {#to_bytes tag="method"} ## Doc.to_bytes {id="to_bytes",tag="method"}
Serialize, i.e. export the document contents to a binary string. Serialize, i.e. export the document contents to a binary string.
@ -460,7 +460,7 @@ Serialize, i.e. export the document contents to a binary string.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | A losslessly serialized copy of the `Doc`, including all annotations. ~~bytes~~ | | **RETURNS** | A losslessly serialized copy of the `Doc`, including all annotations. ~~bytes~~ |
## Doc.from_bytes {#from_bytes tag="method"} ## Doc.from_bytes {id="from_bytes",tag="method"}
Deserialize, i.e. import the document contents from a binary string. Deserialize, i.e. import the document contents from a binary string.
@ -481,7 +481,7 @@ Deserialize, i.e. import the document contents from a binary string.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Doc` object. ~~Doc~~ | | **RETURNS** | The `Doc` object. ~~Doc~~ |
## Doc.to_json {#to_json tag="method"} ## Doc.to_json {id="to_json",tag="method"}
Serializes a document to JSON. Note that this is format differs from the Serializes a document to JSON. Note that this is format differs from the
deprecated [`JSON training format`](/api/data-formats#json-input). deprecated [`JSON training format`](/api/data-formats#json-input).
@ -498,7 +498,7 @@ deprecated [`JSON training format`](/api/data-formats#json-input).
| `underscore` | Optional list of string names of custom `Doc` attributes. Attribute values need to be JSON-serializable. Values will be added to an `"_"` key in the data, e.g. `"_": {"foo": "bar"}`. ~~Optional[List[str]]~~ | | `underscore` | Optional list of string names of custom `Doc` attributes. Attribute values need to be JSON-serializable. Values will be added to an `"_"` key in the data, e.g. `"_": {"foo": "bar"}`. ~~Optional[List[str]]~~ |
| **RETURNS** | The data in JSON format. ~~Dict[str, Any]~~ | | **RETURNS** | The data in JSON format. ~~Dict[str, Any]~~ |
## Doc.from_json {#from_json tag="method" new="3.3.1"} ## Doc.from_json {id="from_json",tag="method",version="3.3.1"}
Deserializes a document from JSON, i.e. generates a document from the provided Deserializes a document from JSON, i.e. generates a document from the provided
JSON data as generated by [`Doc.to_json()`](/api/doc#to_json). JSON data as generated by [`Doc.to_json()`](/api/doc#to_json).
@ -520,7 +520,7 @@ JSON data as generated by [`Doc.to_json()`](/api/doc#to_json).
| `validate` | Whether to validate the JSON input against the expected schema for detailed debugging. Defaults to `False`. ~~bool~~ | | `validate` | Whether to validate the JSON input against the expected schema for detailed debugging. Defaults to `False`. ~~bool~~ |
| **RETURNS** | A `Doc` corresponding to the provided JSON. ~~Doc~~ | | **RETURNS** | A `Doc` corresponding to the provided JSON. ~~Doc~~ |
## Doc.retokenize {#retokenize tag="contextmanager" new="2.1"} ## Doc.retokenize {id="retokenize",tag="contextmanager",version="2.1"}
Context manager to handle retokenization of the `Doc`. Modifications to the Context manager to handle retokenization of the `Doc`. Modifications to the
`Doc`'s tokenization are stored, and then made all at once when the context `Doc`'s tokenization are stored, and then made all at once when the context
@ -540,7 +540,7 @@ invalidated, although they may accidentally continue to work.
| ----------- | -------------------------------- | | ----------- | -------------------------------- |
| **RETURNS** | The retokenizer. ~~Retokenizer~~ | | **RETURNS** | The retokenizer. ~~Retokenizer~~ |
### Retokenizer.merge {#retokenizer.merge tag="method"} ### Retokenizer.merge {id="retokenizer.merge",tag="method"}
Mark a span for merging. The `attrs` will be applied to the resulting token (if Mark a span for merging. The `attrs` will be applied to the resulting token (if
they're context-dependent token attributes like `LEMMA` or `DEP`) or to the they're context-dependent token attributes like `LEMMA` or `DEP`) or to the
@ -563,7 +563,7 @@ values.
| `span` | The span to merge. ~~Span~~ | | `span` | The span to merge. ~~Span~~ |
| `attrs` | Attributes to set on the merged token. ~~Dict[Union[str, int], Any]~~ | | `attrs` | Attributes to set on the merged token. ~~Dict[Union[str, int], Any]~~ |
### Retokenizer.split {#retokenizer.split tag="method"} ### Retokenizer.split {id="retokenizer.split",tag="method"}
Mark a token for splitting, into the specified `orths`. The `heads` are required Mark a token for splitting, into the specified `orths`. The `heads` are required
to specify how the new subtokens should be integrated into the dependency tree. to specify how the new subtokens should be integrated into the dependency tree.
@ -599,7 +599,7 @@ underlying lexeme (if they're context-independent lexical attributes like
| `heads` | List of `token` or `(token, subtoken)` tuples specifying the tokens to attach the newly split subtokens to. ~~List[Union[Token, Tuple[Token, int]]]~~ | | `heads` | List of `token` or `(token, subtoken)` tuples specifying the tokens to attach the newly split subtokens to. ~~List[Union[Token, Tuple[Token, int]]]~~ |
| `attrs` | Attributes to set on all split tokens. Attribute names mapped to list of per-token attribute values. ~~Dict[Union[str, int], List[Any]]~~ | | `attrs` | Attributes to set on all split tokens. Attribute names mapped to list of per-token attribute values. ~~Dict[Union[str, int], List[Any]]~~ |
## Doc.ents {#ents tag="property" model="NER"} ## Doc.ents {id="ents",tag="property",model="NER"}
The named entities in the document. Returns a tuple of named entity `Span` The named entities in the document. Returns a tuple of named entity `Span`
objects, if the entity recognizer has been applied. objects, if the entity recognizer has been applied.
@ -617,7 +617,7 @@ objects, if the entity recognizer has been applied.
| ----------- | ---------------------------------------------------------------- | | ----------- | ---------------------------------------------------------------- |
| **RETURNS** | Entities in the document, one `Span` per entity. ~~Tuple[Span]~~ | | **RETURNS** | Entities in the document, one `Span` per entity. ~~Tuple[Span]~~ |
## Doc.spans {#spans tag="property"} ## Doc.spans {id="spans",tag="property"}
A dictionary of named span groups, to store and access additional span A dictionary of named span groups, to store and access additional span
annotations. You can write to it by assigning a list of [`Span`](/api/span) annotations. You can write to it by assigning a list of [`Span`](/api/span)
@ -634,7 +634,7 @@ objects or a [`SpanGroup`](/api/spangroup) to a given key.
| ----------- | ------------------------------------------------------------------ | | ----------- | ------------------------------------------------------------------ |
| **RETURNS** | The span groups assigned to the document. ~~Dict[str, SpanGroup]~~ | | **RETURNS** | The span groups assigned to the document. ~~Dict[str, SpanGroup]~~ |
## Doc.cats {#cats tag="property" model="text classifier"} ## Doc.cats {id="cats",tag="property",model="text classifier"}
Maps a label to a score for categories applied to the document. Typically set by Maps a label to a score for categories applied to the document. Typically set by
the [`TextCategorizer`](/api/textcategorizer). the [`TextCategorizer`](/api/textcategorizer).
@ -650,7 +650,7 @@ the [`TextCategorizer`](/api/textcategorizer).
| ----------- | ---------------------------------------------------------- | | ----------- | ---------------------------------------------------------- |
| **RETURNS** | The text categories mapped to scores. ~~Dict[str, float]~~ | | **RETURNS** | The text categories mapped to scores. ~~Dict[str, float]~~ |
## Doc.noun_chunks {#noun_chunks tag="property" model="parser"} ## Doc.noun_chunks {id="noun_chunks",tag="property",model="parser"}
Iterate over the base noun phrases in the document. Yields base noun-phrase Iterate over the base noun phrases in the document. Yields base noun-phrase
`Span` objects, if the document has been syntactically parsed. A base noun `Span` objects, if the document has been syntactically parsed. A base noun
@ -677,7 +677,7 @@ implemented for the given language, a `NotImplementedError` is raised.
| ---------- | ------------------------------------- | | ---------- | ------------------------------------- |
| **YIELDS** | Noun chunks in the document. ~~Span~~ | | **YIELDS** | Noun chunks in the document. ~~Span~~ |
## Doc.sents {#sents tag="property" model="sentences"} ## Doc.sents {id="sents",tag="property",model="sentences"}
Iterate over the sentences in the document. Sentence spans have no label. Iterate over the sentences in the document. Sentence spans have no label.
@ -699,7 +699,7 @@ will raise an error otherwise.
| ---------- | ----------------------------------- | | ---------- | ----------------------------------- |
| **YIELDS** | Sentences in the document. ~~Span~~ | | **YIELDS** | Sentences in the document. ~~Span~~ |
## Doc.has_vector {#has_vector tag="property" model="vectors"} ## Doc.has_vector {id="has_vector",tag="property",model="vectors"}
A boolean value indicating whether a word vector is associated with the object. A boolean value indicating whether a word vector is associated with the object.
@ -714,7 +714,7 @@ A boolean value indicating whether a word vector is associated with the object.
| ----------- | --------------------------------------------------------- | | ----------- | --------------------------------------------------------- |
| **RETURNS** | Whether the document has a vector data attached. ~~bool~~ | | **RETURNS** | Whether the document has a vector data attached. ~~bool~~ |
## Doc.vector {#vector tag="property" model="vectors"} ## Doc.vector {id="vector",tag="property",model="vectors"}
A real-valued meaning representation. Defaults to an average of the token A real-valued meaning representation. Defaults to an average of the token
vectors. vectors.
@ -731,7 +731,7 @@ vectors.
| ----------- | -------------------------------------------------------------------------------------------------- | | ----------- | -------------------------------------------------------------------------------------------------- |
| **RETURNS** | A 1-dimensional array representing the document's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | | **RETURNS** | A 1-dimensional array representing the document's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Doc.vector_norm {#vector_norm tag="property" model="vectors"} ## Doc.vector_norm {id="vector_norm",tag="property",model="vectors"}
The L2 norm of the document's vector representation. The L2 norm of the document's vector representation.
@ -749,7 +749,7 @@ The L2 norm of the document's vector representation.
| ----------- | --------------------------------------------------- | | ----------- | --------------------------------------------------- |
| **RETURNS** | The L2 norm of the vector representation. ~~float~~ | | **RETURNS** | The L2 norm of the vector representation. ~~float~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
@ -768,7 +768,7 @@ The L2 norm of the document's vector representation.
| `has_unknown_spaces` | Whether the document was constructed without known spacing between tokens (typically when created from gold tokenization). ~~bool~~ | | `has_unknown_spaces` | Whether the document was constructed without known spacing between tokens (typically when created from gold tokenization). ~~bool~~ |
| `_` | User space for adding custom [attribute extensions](/usage/processing-pipelines#custom-components-attributes). ~~Underscore~~ | | `_` | User space for adding custom [attribute extensions](/usage/processing-pipelines#custom-components-attributes). ~~Underscore~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -1,7 +1,7 @@
--- ---
title: DocBin title: DocBin
tag: class tag: class
new: 2.2 version: 2.2
teaser: Pack Doc objects for binary serialization teaser: Pack Doc objects for binary serialization
source: spacy/tokens/_serialize.py source: spacy/tokens/_serialize.py
--- ---
@ -15,8 +15,7 @@ notable downside to this format is that you can't easily extract just one
document from the `DocBin`. The serialization format is gzipped msgpack, where document from the `DocBin`. The serialization format is gzipped msgpack, where
the msgpack object has the following structure: the msgpack object has the following structure:
```python ```python {title="msgpack object structure"}
### msgpack object structure
{ {
"version": str, # DocBin version number "version": str, # DocBin version number
"attrs": List[uint64], # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE] "attrs": List[uint64], # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE]
@ -33,7 +32,7 @@ object. This means the storage is more efficient if you pack more documents
together, because you have less duplication in the strings. For usage examples, together, because you have less duplication in the strings. For usage examples,
see the docs on [serializing `Doc` objects](/usage/saving-loading#docs). see the docs on [serializing `Doc` objects](/usage/saving-loading#docs).
## DocBin.\_\_init\_\_ {#init tag="method"} ## DocBin.\_\_init\_\_ {id="init",tag="method"}
Create a `DocBin` object to hold serialized annotations. Create a `DocBin` object to hold serialized annotations.
@ -50,7 +49,7 @@ Create a `DocBin` object to hold serialized annotations.
| `store_user_data` | Whether to write the `Doc.user_data` and the values of custom extension attributes to file/bytes. Defaults to `False`. ~~bool~~ | | `store_user_data` | Whether to write the `Doc.user_data` and the values of custom extension attributes to file/bytes. Defaults to `False`. ~~bool~~ |
| `docs` | `Doc` objects to add on initialization. ~~Iterable[Doc]~~ | | `docs` | `Doc` objects to add on initialization. ~~Iterable[Doc]~~ |
## DocBin.\_\len\_\_ {#len tag="method"} ## DocBin.\_\_len\_\_ {id="len",tag="method"}
Get the number of `Doc` objects that were added to the `DocBin`. Get the number of `Doc` objects that were added to the `DocBin`.
@ -67,7 +66,7 @@ Get the number of `Doc` objects that were added to the `DocBin`.
| ----------- | --------------------------------------------------- | | ----------- | --------------------------------------------------- |
| **RETURNS** | The number of `Doc`s added to the `DocBin`. ~~int~~ | | **RETURNS** | The number of `Doc`s added to the `DocBin`. ~~int~~ |
## DocBin.add {#add tag="method"} ## DocBin.add {id="add",tag="method"}
Add a `Doc`'s annotations to the `DocBin` for serialization. Add a `Doc`'s annotations to the `DocBin` for serialization.
@ -83,7 +82,7 @@ Add a `Doc`'s annotations to the `DocBin` for serialization.
| -------- | -------------------------------- | | -------- | -------------------------------- |
| `doc` | The `Doc` object to add. ~~Doc~~ | | `doc` | The `Doc` object to add. ~~Doc~~ |
## DocBin.get_docs {#get_docs tag="method"} ## DocBin.get_docs {id="get_docs",tag="method"}
Recover `Doc` objects from the annotations, using the given vocab. Recover `Doc` objects from the annotations, using the given vocab.
@ -98,7 +97,7 @@ Recover `Doc` objects from the annotations, using the given vocab.
| `vocab` | The shared vocab. ~~Vocab~~ | | `vocab` | The shared vocab. ~~Vocab~~ |
| **YIELDS** | The `Doc` objects. ~~Doc~~ | | **YIELDS** | The `Doc` objects. ~~Doc~~ |
## DocBin.merge {#merge tag="method"} ## DocBin.merge {id="merge",tag="method"}
Extend the annotations of this `DocBin` with the annotations from another. Will Extend the annotations of this `DocBin` with the annotations from another. Will
raise an error if the pre-defined `attrs` of the two `DocBin`s don't match. raise an error if the pre-defined `attrs` of the two `DocBin`s don't match.
@ -118,7 +117,7 @@ raise an error if the pre-defined `attrs` of the two `DocBin`s don't match.
| -------- | ------------------------------------------------------ | | -------- | ------------------------------------------------------ |
| `other` | The `DocBin` to merge into the current bin. ~~DocBin~~ | | `other` | The `DocBin` to merge into the current bin. ~~DocBin~~ |
## DocBin.to_bytes {#to_bytes tag="method"} ## DocBin.to_bytes {id="to_bytes",tag="method"}
Serialize the `DocBin`'s annotations to a bytestring. Serialize the `DocBin`'s annotations to a bytestring.
@ -134,7 +133,7 @@ Serialize the `DocBin`'s annotations to a bytestring.
| ----------- | ---------------------------------- | | ----------- | ---------------------------------- |
| **RETURNS** | The serialized `DocBin`. ~~bytes~~ | | **RETURNS** | The serialized `DocBin`. ~~bytes~~ |
## DocBin.from_bytes {#from_bytes tag="method"} ## DocBin.from_bytes {id="from_bytes",tag="method"}
Deserialize the `DocBin`'s annotations from a bytestring. Deserialize the `DocBin`'s annotations from a bytestring.
@ -150,7 +149,7 @@ Deserialize the `DocBin`'s annotations from a bytestring.
| `bytes_data` | The data to load from. ~~bytes~~ | | `bytes_data` | The data to load from. ~~bytes~~ |
| **RETURNS** | The loaded `DocBin`. ~~DocBin~~ | | **RETURNS** | The loaded `DocBin`. ~~DocBin~~ |
## DocBin.to_disk {#to_disk tag="method" new="3"} ## DocBin.to_disk {id="to_disk",tag="method",version="3"}
Save the serialized `DocBin` to a file. Typically uses the `.spacy` extension Save the serialized `DocBin` to a file. Typically uses the `.spacy` extension
and the result can be used as the input data for and the result can be used as the input data for
@ -168,7 +167,7 @@ and the result can be used as the input data for
| -------- | -------------------------------------------------------------------------- | | -------- | -------------------------------------------------------------------------- |
| `path` | The file path, typically with the `.spacy` extension. ~~Union[str, Path]~~ | | `path` | The file path, typically with the `.spacy` extension. ~~Union[str, Path]~~ |
## DocBin.from_disk {#from_disk tag="method" new="3"} ## DocBin.from_disk {id="from_disk",tag="method",version="3"}
Load a serialized `DocBin` from a file. Typically uses the `.spacy` extension. Load a serialized `DocBin` from a file. Typically uses the `.spacy` extension.

View File

@ -2,7 +2,7 @@
title: EditTreeLemmatizer title: EditTreeLemmatizer
tag: class tag: class
source: spacy/pipeline/edit_tree_lemmatizer.py source: spacy/pipeline/edit_tree_lemmatizer.py
new: 3.3 version: 3.3
teaser: 'Pipeline component for lemmatization' teaser: 'Pipeline component for lemmatization'
api_base_class: /api/pipe api_base_class: /api/pipe
api_string_name: trainable_lemmatizer api_string_name: trainable_lemmatizer
@ -18,7 +18,7 @@ and construction method used by this lemmatizer were proposed in
For a lookup and rule-based lemmatizer, see [`Lemmatizer`](/api/lemmatizer). For a lookup and rule-based lemmatizer, see [`Lemmatizer`](/api/lemmatizer).
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Predictions are assigned to `Token.lemma`. Predictions are assigned to `Token.lemma`.
@ -27,7 +27,7 @@ Predictions are assigned to `Token.lemma`.
| `Token.lemma` | The lemma (hash). ~~int~~ | | `Token.lemma` | The lemma (hash). ~~int~~ |
| `Token.lemma_` | The lemma. ~~str~~ | | `Token.lemma_` | The lemma. ~~str~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -57,7 +57,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/edit_tree_lemmatizer.py %%GITHUB_SPACY/spacy/pipeline/edit_tree_lemmatizer.py
``` ```
## EditTreeLemmatizer.\_\_init\_\_ {#init tag="method"} ## EditTreeLemmatizer.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -90,7 +90,7 @@ shortcut for this and instantiate the component using its string name and
| `top_k` | The number of most probable edit trees to try before resorting to `backoff`. Defaults to `1`. ~~int~~ | | `top_k` | The number of most probable edit trees to try before resorting to `backoff`. Defaults to `1`. ~~int~~ |
| `scorer` | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attribute `"lemma"`. ~~Optional[Callable]~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attribute `"lemma"`. ~~Optional[Callable]~~ |
## EditTreeLemmatizer.\_\_call\_\_ {#call tag="method"} ## EditTreeLemmatizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned. Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -114,7 +114,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## EditTreeLemmatizer.pipe {#pipe tag="method"} ## EditTreeLemmatizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -138,7 +138,7 @@ and [`pipe`](/api/edittreelemmatizer#pipe) delegate to the
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## EditTreeLemmatizer.initialize {#initialize tag="method" new="3"} ## EditTreeLemmatizer.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -175,7 +175,7 @@ config.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ |
## EditTreeLemmatizer.predict {#predict tag="method"} ## EditTreeLemmatizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. modifying them.
@ -192,7 +192,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. | | **RETURNS** | The model's prediction for each document. |
## EditTreeLemmatizer.set_annotations {#set_annotations tag="method"} ## EditTreeLemmatizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed tree Modify a batch of [`Doc`](/api/doc) objects, using pre-computed tree
identifiers. identifiers.
@ -210,7 +210,7 @@ identifiers.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `tree_ids` | The identifiers of the edit trees to apply, produced by `EditTreeLemmatizer.predict`. | | `tree_ids` | The identifiers of the edit trees to apply, produced by `EditTreeLemmatizer.predict`. |
## EditTreeLemmatizer.update {#update tag="method"} ## EditTreeLemmatizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model. predictions and gold-standard annotations, and update the component's model.
@ -234,7 +234,7 @@ Delegates to [`predict`](/api/edittreelemmatizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## EditTreeLemmatizer.get_loss {#get_loss tag="method"} ## EditTreeLemmatizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their Find the loss and gradient of loss for the batch of documents and their
predicted scores. predicted scores.
@ -253,7 +253,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. | | `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## EditTreeLemmatizer.create_optimizer {#create_optimizer tag="method"} ## EditTreeLemmatizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -268,7 +268,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## EditTreeLemmatizer.use_params {#use_params tag="method, contextmanager"} ## EditTreeLemmatizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -285,7 +285,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## EditTreeLemmatizer.to_disk {#to_disk tag="method"} ## EditTreeLemmatizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -302,7 +302,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## EditTreeLemmatizer.from_disk {#from_disk tag="method"} ## EditTreeLemmatizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -320,7 +320,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `EditTreeLemmatizer` object. ~~EditTreeLemmatizer~~ | | **RETURNS** | The modified `EditTreeLemmatizer` object. ~~EditTreeLemmatizer~~ |
## EditTreeLemmatizer.to_bytes {#to_bytes tag="method"} ## EditTreeLemmatizer.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -337,7 +337,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `EditTreeLemmatizer` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `EditTreeLemmatizer` object. ~~bytes~~ |
## EditTreeLemmatizer.from_bytes {#from_bytes tag="method"} ## EditTreeLemmatizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -356,7 +356,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `EditTreeLemmatizer` object. ~~EditTreeLemmatizer~~ | | **RETURNS** | The `EditTreeLemmatizer` object. ~~EditTreeLemmatizer~~ |
## EditTreeLemmatizer.labels {#labels tag="property"} ## EditTreeLemmatizer.labels {id="labels",tag="property"}
The labels currently added to the component. The labels currently added to the component.
@ -371,7 +371,7 @@ identifiers of edit trees.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## EditTreeLemmatizer.label_data {#label_data tag="property" new="3"} ## EditTreeLemmatizer.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information. The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -389,7 +389,7 @@ initialize the model with a pre-defined label set.
| ----------- | ---------------------------------------------------------- | | ----------- | ---------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -2,7 +2,7 @@
title: EntityLinker title: EntityLinker
tag: class tag: class
source: spacy/pipeline/entity_linker.py source: spacy/pipeline/entity_linker.py
new: 2.2 version: 2.2
teaser: 'Pipeline component for named entity linking and disambiguation' teaser: 'Pipeline component for named entity linking and disambiguation'
api_base_class: /api/pipe api_base_class: /api/pipe
api_string_name: entity_linker api_string_name: entity_linker
@ -17,7 +17,7 @@ and a machine learning model to pick the right candidate, given the local
context of the mention. `EntityLinker` defaults to using the context of the mention. `EntityLinker` defaults to using the
[`InMemoryLookupKB`](/api/kb_in_memory) implementation. [`InMemoryLookupKB`](/api/kb_in_memory) implementation.
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Predictions, in the form of knowledge base IDs, will be assigned to Predictions, in the form of knowledge base IDs, will be assigned to
`Token.ent_kb_id_`. `Token.ent_kb_id_`.
@ -27,7 +27,7 @@ Predictions, in the form of knowledge base IDs, will be assigned to
| `Token.ent_kb_id` | Knowledge base ID (hash). ~~int~~ | | `Token.ent_kb_id` | Knowledge base ID (hash). ~~int~~ |
| `Token.ent_kb_id_` | Knowledge base ID. ~~str~~ | | `Token.ent_kb_id_` | Knowledge base ID. ~~str~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -71,7 +71,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/entity_linker.py %%GITHUB_SPACY/spacy/pipeline/entity_linker.py
``` ```
## EntityLinker.\_\_init\_\_ {#init tag="method"} ## EntityLinker.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -114,7 +114,7 @@ custom knowledge base, you should either call
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_links`](/api/scorer#score_links). ~~Optional[Callable]~~ | | `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_links`](/api/scorer#score_links). ~~Optional[Callable]~~ |
| `threshold` <Tag variant="new">3.4</Tag> | Confidence threshold for entity predictions. The default of `None` implies that all predictions are accepted, otherwise those with a score beneath the treshold are discarded. If there are no predictions with scores above the threshold, the linked entity is `NIL`. ~~Optional[float]~~ | | `threshold` <Tag variant="new">3.4</Tag> | Confidence threshold for entity predictions. The default of `None` implies that all predictions are accepted, otherwise those with a score beneath the treshold are discarded. If there are no predictions with scores above the threshold, the linked entity is `NIL`. ~~Optional[float]~~ |
## EntityLinker.\_\_call\_\_ {#call tag="method"} ## EntityLinker.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place and returned. Apply the pipe to one document. The document is modified in place and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -137,7 +137,7 @@ delegate to the [`predict`](/api/entitylinker#predict) and
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## EntityLinker.pipe {#pipe tag="method"} ## EntityLinker.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -161,7 +161,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/entitylinker#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## EntityLinker.set_kb {#set_kb tag="method" new="3"} ## EntityLinker.set_kb {id="set_kb",tag="method",version="3"}
The `kb_loader` should be a function that takes a `Vocab` instance and creates The `kb_loader` should be a function that takes a `Vocab` instance and creates
the `KnowledgeBase`, ensuring that the strings of the knowledge base are synced the `KnowledgeBase`, ensuring that the strings of the knowledge base are synced
@ -183,7 +183,7 @@ with the current vocab.
| ----------- | ---------------------------------------------------------------------------------------------------------------- | | ----------- | ---------------------------------------------------------------------------------------------------------------- |
| `kb_loader` | Function that creates a [`KnowledgeBase`](/api/kb) from a `Vocab` instance. ~~Callable[[Vocab], KnowledgeBase]~~ | | `kb_loader` | Function that creates a [`KnowledgeBase`](/api/kb) from a `Vocab` instance. ~~Callable[[Vocab], KnowledgeBase]~~ |
## EntityLinker.initialize {#initialize tag="method" new="3"} ## EntityLinker.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -219,7 +219,7 @@ This method was previously called `begin_training`.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `kb_loader` | Function that creates a [`KnowledgeBase`](/api/kb) from a `Vocab` instance. ~~Callable[[Vocab], KnowledgeBase]~~ | | `kb_loader` | Function that creates a [`KnowledgeBase`](/api/kb) from a `Vocab` instance. ~~Callable[[Vocab], KnowledgeBase]~~ |
## EntityLinker.predict {#predict tag="method"} ## EntityLinker.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. Returns the KB IDs for each entity in each doc, including `NIL` modifying them. Returns the KB IDs for each entity in each doc, including `NIL`
@ -237,7 +237,7 @@ if there is no prediction.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The predicted KB identifiers for the entities in the `docs`. ~~List[str]~~ | | **RETURNS** | The predicted KB identifiers for the entities in the `docs`. ~~List[str]~~ |
## EntityLinker.set_annotations {#set_annotations tag="method"} ## EntityLinker.set_annotations {id="set_annotations",tag="method"}
Modify a batch of documents, using pre-computed entity IDs for a list of named Modify a batch of documents, using pre-computed entity IDs for a list of named
entities. entities.
@ -255,7 +255,7 @@ entities.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `kb_ids` | The knowledge base identifiers for the entities in the docs, predicted by `EntityLinker.predict`. ~~List[str]~~ | | `kb_ids` | The knowledge base identifiers for the entities in the docs, predicted by `EntityLinker.predict`. ~~List[str]~~ |
## EntityLinker.update {#update tag="method"} ## EntityLinker.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects, updating both the Learn from a batch of [`Example`](/api/example) objects, updating both the
pipe's entity linking model and context encoder. Delegates to pipe's entity linking model and context encoder. Delegates to
@ -278,7 +278,7 @@ pipe's entity linking model and context encoder. Delegates to
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## EntityLinker.create_optimizer {#create_optimizer tag="method"} ## EntityLinker.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -293,7 +293,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## EntityLinker.use_params {#use_params tag="method, contextmanager"} ## EntityLinker.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -310,7 +310,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## EntityLinker.to_disk {#to_disk tag="method"} ## EntityLinker.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -327,7 +327,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## EntityLinker.from_disk {#from_disk tag="method"} ## EntityLinker.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -345,7 +345,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `EntityLinker` object. ~~EntityLinker~~ | | **RETURNS** | The modified `EntityLinker` object. ~~EntityLinker~~ |
## EntityLinker.to_bytes {#to_bytes tag="method"} ## EntityLinker.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -362,7 +362,7 @@ Serialize the pipe to a bytestring, including the `KnowledgeBase`.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `EntityLinker` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `EntityLinker` object. ~~bytes~~ |
## EntityLinker.from_bytes {#from_bytes tag="method"} ## EntityLinker.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -381,7 +381,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `EntityLinker` object. ~~EntityLinker~~ | | **RETURNS** | The `EntityLinker` object. ~~EntityLinker~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -20,7 +20,7 @@ your entities will be close to their initial tokens. If your entities are long
and characterized by tokens in their middle, the component will likely not be a and characterized by tokens in their middle, the component will likely not be a
good fit for your task. good fit for your task.
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Predictions will be saved to `Doc.ents` as a tuple. Each label will also be Predictions will be saved to `Doc.ents` as a tuple. Each label will also be
reflected to each underlying token, where it is saved in the `Token.ent_type` reflected to each underlying token, where it is saved in the `Token.ent_type`
@ -38,7 +38,7 @@ non-overlapping, or an error will be thrown.
| `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ | | `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ |
| `Token.ent_type_` | The label part of the named entity tag. ~~str~~ | | `Token.ent_type_` | The label part of the named entity tag. ~~str~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -72,7 +72,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/ner.pyx %%GITHUB_SPACY/spacy/pipeline/ner.pyx
``` ```
## EntityRecognizer.\_\_init\_\_ {#init tag="method"} ## EntityRecognizer.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -103,7 +103,7 @@ shortcut for this and instantiate the component using its string name and
| `update_with_oracle_cut_size` | During training, cut long sequences into shorter segments by creating intermediate states based on the gold-standard history. The model is not very sensitive to this parameter, so you usually won't need to change it. Defaults to `100`. ~~int~~ | | `update_with_oracle_cut_size` | During training, cut long sequences into shorter segments by creating intermediate states based on the gold-standard history. The model is not very sensitive to this parameter, so you usually won't need to change it. Defaults to `100`. ~~int~~ |
| `incorrect_spans_key` | Identifies spans that are known to be incorrect entity annotations. The incorrect entity annotations can be stored in the span group in [`Doc.spans`](/api/doc#spans), under this key. Defaults to `None`. ~~Optional[str]~~ | | `incorrect_spans_key` | Identifies spans that are known to be incorrect entity annotations. The incorrect entity annotations can be stored in the span group in [`Doc.spans`](/api/doc#spans), under this key. Defaults to `None`. ~~Optional[str]~~ |
## EntityRecognizer.\_\_call\_\_ {#call tag="method"} ## EntityRecognizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place and returned. Apply the pipe to one document. The document is modified in place and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -127,7 +127,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## EntityRecognizer.pipe {#pipe tag="method"} ## EntityRecognizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -151,7 +151,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/entityrecognizer#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## EntityRecognizer.initialize {#initialize tag="method" new="3"} ## EntityRecognizer.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -194,7 +194,7 @@ This method was previously called `begin_training`.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Dict[str, Dict[str, int]]]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Dict[str, Dict[str, int]]]~~ |
## EntityRecognizer.predict {#predict tag="method"} ## EntityRecognizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. modifying them.
@ -211,7 +211,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | A helper class for the parse state (internal). ~~StateClass~~ | | **RETURNS** | A helper class for the parse state (internal). ~~StateClass~~ |
## EntityRecognizer.set_annotations {#set_annotations tag="method"} ## EntityRecognizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -228,7 +228,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `EntityRecognizer.predict`. Returns an internal helper class for the parse state. ~~List[StateClass]~~ | | `scores` | The scores to set, produced by `EntityRecognizer.predict`. Returns an internal helper class for the parse state. ~~List[StateClass]~~ |
## EntityRecognizer.update {#update tag="method"} ## EntityRecognizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects, updating the pipe's Learn from a batch of [`Example`](/api/example) objects, updating the pipe's
model. Delegates to [`predict`](/api/entityrecognizer#predict) and model. Delegates to [`predict`](/api/entityrecognizer#predict) and
@ -251,7 +251,7 @@ model. Delegates to [`predict`](/api/entityrecognizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## EntityRecognizer.get_loss {#get_loss tag="method"} ## EntityRecognizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their Find the loss and gradient of loss for the batch of documents and their
predicted scores. predicted scores.
@ -270,7 +270,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. ~~StateClass~~ | | `scores` | Scores representing the model's predictions. ~~StateClass~~ |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## EntityRecognizer.create_optimizer {#create_optimizer tag="method"} ## EntityRecognizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -285,7 +285,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## EntityRecognizer.use_params {#use_params tag="method, contextmanager"} ## EntityRecognizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -302,7 +302,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## EntityRecognizer.add_label {#add_label tag="method"} ## EntityRecognizer.add_label {id="add_label",tag="method"}
Add a new label to the pipe. Note that you don't have to call this method if you Add a new label to the pipe. Note that you don't have to call this method if you
provide a **representative data sample** to the [`initialize`](#initialize) provide a **representative data sample** to the [`initialize`](#initialize)
@ -322,7 +322,7 @@ to the model, and the output dimension will be
| `label` | The label to add. ~~str~~ | | `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## EntityRecognizer.set_output {#set_output tag="method"} ## EntityRecognizer.set_output {id="set_output",tag="method"}
Change the output dimension of the component's model by calling the model's Change the output dimension of the component's model by calling the model's
attribute `resize_output`. This is a function that takes the original model and attribute `resize_output`. This is a function that takes the original model and
@ -341,7 +341,7 @@ forgetting" problem.
| ---- | --------------------------------- | | ---- | --------------------------------- |
| `nO` | The new output dimension. ~~int~~ | | `nO` | The new output dimension. ~~int~~ |
## EntityRecognizer.to_disk {#to_disk tag="method"} ## EntityRecognizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -358,7 +358,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## EntityRecognizer.from_disk {#from_disk tag="method"} ## EntityRecognizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -376,7 +376,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `EntityRecognizer` object. ~~EntityRecognizer~~ | | **RETURNS** | The modified `EntityRecognizer` object. ~~EntityRecognizer~~ |
## EntityRecognizer.to_bytes {#to_bytes tag="method"} ## EntityRecognizer.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -393,7 +393,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `EntityRecognizer` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `EntityRecognizer` object. ~~bytes~~ |
## EntityRecognizer.from_bytes {#from_bytes tag="method"} ## EntityRecognizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -412,7 +412,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `EntityRecognizer` object. ~~EntityRecognizer~~ | | **RETURNS** | The `EntityRecognizer` object. ~~EntityRecognizer~~ |
## EntityRecognizer.labels {#labels tag="property"} ## EntityRecognizer.labels {id="labels",tag="property"}
The labels currently added to the component. The labels currently added to the component.
@ -427,7 +427,7 @@ The labels currently added to the component.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## EntityRecognizer.label_data {#label_data tag="property" new="3"} ## EntityRecognizer.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information. The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -445,7 +445,7 @@ the model with a pre-defined label set.
| ----------- | ------------------------------------------------------------------------------- | | ----------- | ------------------------------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Dict[str, Dict[str, Dict[str, int]]]~~ | | **RETURNS** | The label data added to the component. ~~Dict[str, Dict[str, Dict[str, int]]]~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -2,7 +2,7 @@
title: EntityRuler title: EntityRuler
tag: class tag: class
source: spacy/pipeline/entityruler.py source: spacy/pipeline/entityruler.py
new: 2.1 version: 2.1
teaser: 'Pipeline component for rule-based named entity recognition' teaser: 'Pipeline component for rule-based named entity recognition'
api_string_name: entity_ruler api_string_name: entity_ruler
api_trainable: false api_trainable: false
@ -15,7 +15,7 @@ used on its own to implement a purely rule-based entity recognition system. For
usage examples, see the docs on usage examples, see the docs on
[rule-based entity recognition](/usage/rule-based-matching#entityruler). [rule-based entity recognition](/usage/rule-based-matching#entityruler).
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
This component assigns predictions basically the same way as the This component assigns predictions basically the same way as the
[`EntityRecognizer`](/api/entityrecognizer). [`EntityRecognizer`](/api/entityrecognizer).
@ -36,7 +36,7 @@ non-overlapping, or an error will be thrown.
| `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ | | `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ |
| `Token.ent_type_` | The label part of the named entity tag. ~~str~~ | | `Token.ent_type_` | The label part of the named entity tag. ~~str~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -68,7 +68,7 @@ how the component should be configured. You can override its settings via the
%%GITHUB_SPACY/spacy/pipeline/entityruler.py %%GITHUB_SPACY/spacy/pipeline/entityruler.py
``` ```
## EntityRuler.\_\_init\_\_ {#init tag="method"} ## EntityRuler.\_\_init\_\_ {id="init",tag="method"}
Initialize the entity ruler. If patterns are supplied here, they need to be a Initialize the entity ruler. If patterns are supplied here, they need to be a
list of dictionaries with a `"label"` and `"pattern"` key. A pattern can either list of dictionaries with a `"label"` and `"pattern"` key. A pattern can either
@ -99,7 +99,7 @@ be a token pattern (list) or a phrase pattern (string). For example:
| `patterns` | Optional patterns to load in on initialization. ~~Optional[List[Dict[str, Union[str, List[dict]]]]]~~ | | `patterns` | Optional patterns to load in on initialization. ~~Optional[List[Dict[str, Union[str, List[dict]]]]]~~ |
| `scorer` | The scoring method. Defaults to [`spacy.scorer.get_ner_prf`](/api/scorer#get_ner_prf). ~~Optional[Callable]~~ | | `scorer` | The scoring method. Defaults to [`spacy.scorer.get_ner_prf`](/api/scorer#get_ner_prf). ~~Optional[Callable]~~ |
## EntityRuler.initialize {#initialize tag="method" new="3"} ## EntityRuler.initialize {id="initialize",tag="method",version="3"}
Initialize the component with data and used before training to load in rules Initialize the component with data and used before training to load in rules
from a [pattern file](/usage/rule-based-matching/#entityruler-files). This from a [pattern file](/usage/rule-based-matching/#entityruler-files). This
@ -131,7 +131,7 @@ config.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `patterns` | The list of patterns. Defaults to `None`. ~~Optional[Sequence[Dict[str, Union[str, List[Dict[str, Any]]]]]]~~ | | `patterns` | The list of patterns. Defaults to `None`. ~~Optional[Sequence[Dict[str, Union[str, List[Dict[str, Any]]]]]]~~ |
## EntityRuler.\_\len\_\_ {#len tag="method"} ## EntityRuler.\_\_len\_\_ {id="len",tag="method"}
The number of all patterns added to the entity ruler. The number of all patterns added to the entity ruler.
@ -148,7 +148,7 @@ The number of all patterns added to the entity ruler.
| ----------- | ------------------------------- | | ----------- | ------------------------------- |
| **RETURNS** | The number of patterns. ~~int~~ | | **RETURNS** | The number of patterns. ~~int~~ |
## EntityRuler.\_\_contains\_\_ {#contains tag="method"} ## EntityRuler.\_\_contains\_\_ {id="contains",tag="method"}
Whether a label is present in the patterns. Whether a label is present in the patterns.
@ -166,7 +166,7 @@ Whether a label is present in the patterns.
| `label` | The label to check. ~~str~~ | | `label` | The label to check. ~~str~~ |
| **RETURNS** | Whether the entity ruler contains the label. ~~bool~~ | | **RETURNS** | Whether the entity ruler contains the label. ~~bool~~ |
## EntityRuler.\_\_call\_\_ {#call tag="method"} ## EntityRuler.\_\_call\_\_ {id="call",tag="method"}
Find matches in the `Doc` and add them to the `doc.ents`. Typically, this Find matches in the `Doc` and add them to the `doc.ents`. Typically, this
happens automatically after the component has been added to the pipeline using happens automatically after the component has been added to the pipeline using
@ -192,7 +192,7 @@ is chosen.
| `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ | | `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ |
| **RETURNS** | The modified `Doc` with added entities, if available. ~~Doc~~ | | **RETURNS** | The modified `Doc` with added entities, if available. ~~Doc~~ |
## EntityRuler.add_patterns {#add_patterns tag="method"} ## EntityRuler.add_patterns {id="add_patterns",tag="method"}
Add patterns to the entity ruler. A pattern can either be a token pattern (list Add patterns to the entity ruler. A pattern can either be a token pattern (list
of dicts) or a phrase pattern (string). For more details, see the usage guide on of dicts) or a phrase pattern (string). For more details, see the usage guide on
@ -213,7 +213,7 @@ of dicts) or a phrase pattern (string). For more details, see the usage guide on
| ---------- | ---------------------------------------------------------------- | | ---------- | ---------------------------------------------------------------- |
| `patterns` | The patterns to add. ~~List[Dict[str, Union[str, List[dict]]]]~~ | | `patterns` | The patterns to add. ~~List[Dict[str, Union[str, List[dict]]]]~~ |
## EntityRuler.remove {#remove tag="method" new="3.2.1"} ## EntityRuler.remove {id="remove",tag="method",version="3.2.1"}
Remove a pattern by its ID from the entity ruler. A `ValueError` is raised if Remove a pattern by its ID from the entity ruler. A `ValueError` is raised if
the ID does not exist. the ID does not exist.
@ -231,7 +231,7 @@ the ID does not exist.
| ---- | ----------------------------------- | | ---- | ----------------------------------- |
| `id` | The ID of the pattern rule. ~~str~~ | | `id` | The ID of the pattern rule. ~~str~~ |
## EntityRuler.to_disk {#to_disk tag="method"} ## EntityRuler.to_disk {id="to_disk",tag="method"}
Save the entity ruler patterns to a directory. The patterns will be saved as Save the entity ruler patterns to a directory. The patterns will be saved as
newline-delimited JSON (JSONL). If a file with the suffix `.jsonl` is provided, newline-delimited JSON (JSONL). If a file with the suffix `.jsonl` is provided,
@ -250,7 +250,7 @@ only the patterns are saved as JSONL. If a directory name is provided, a
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `path` | A path to a JSONL file or directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a JSONL file or directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## EntityRuler.from_disk {#from_disk tag="method"} ## EntityRuler.from_disk {id="from_disk",tag="method"}
Load the entity ruler from a path. Expects either a file containing Load the entity ruler from a path. Expects either a file containing
newline-delimited JSON (JSONL) with one entry per line, or a directory newline-delimited JSON (JSONL) with one entry per line, or a directory
@ -270,7 +270,7 @@ configuration.
| `path` | A path to a JSONL file or directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a JSONL file or directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The modified `EntityRuler` object. ~~EntityRuler~~ | | **RETURNS** | The modified `EntityRuler` object. ~~EntityRuler~~ |
## EntityRuler.to_bytes {#to_bytes tag="method"} ## EntityRuler.to_bytes {id="to_bytes",tag="method"}
Serialize the entity ruler patterns to a bytestring. Serialize the entity ruler patterns to a bytestring.
@ -285,7 +285,7 @@ Serialize the entity ruler patterns to a bytestring.
| ----------- | ---------------------------------- | | ----------- | ---------------------------------- |
| **RETURNS** | The serialized patterns. ~~bytes~~ | | **RETURNS** | The serialized patterns. ~~bytes~~ |
## EntityRuler.from_bytes {#from_bytes tag="method"} ## EntityRuler.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -302,7 +302,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `bytes_data` | The bytestring to load. ~~bytes~~ | | `bytes_data` | The bytestring to load. ~~bytes~~ |
| **RETURNS** | The modified `EntityRuler` object. ~~EntityRuler~~ | | **RETURNS** | The modified `EntityRuler` object. ~~EntityRuler~~ |
## EntityRuler.labels {#labels tag="property"} ## EntityRuler.labels {id="labels",tag="property"}
All labels present in the match patterns. All labels present in the match patterns.
@ -310,7 +310,7 @@ All labels present in the match patterns.
| ----------- | -------------------------------------- | | ----------- | -------------------------------------- |
| **RETURNS** | The string labels. ~~Tuple[str, ...]~~ | | **RETURNS** | The string labels. ~~Tuple[str, ...]~~ |
## EntityRuler.ent_ids {#ent_ids tag="property" new="2.2.2"} ## EntityRuler.ent_ids {id="ent_ids",tag="property",version="2.2.2"}
All entity IDs present in the `id` properties of the match patterns. All entity IDs present in the `id` properties of the match patterns.
@ -318,7 +318,7 @@ All entity IDs present in the `id` properties of the match patterns.
| ----------- | ----------------------------------- | | ----------- | ----------------------------------- |
| **RETURNS** | The string IDs. ~~Tuple[str, ...]~~ | | **RETURNS** | The string IDs. ~~Tuple[str, ...]~~ |
## EntityRuler.patterns {#patterns tag="property"} ## EntityRuler.patterns {id="patterns",tag="property"}
Get all patterns that were added to the entity ruler. Get all patterns that were added to the entity ruler.
@ -326,7 +326,7 @@ Get all patterns that were added to the entity ruler.
| ----------- | ---------------------------------------------------------------------------------------- | | ----------- | ---------------------------------------------------------------------------------------- |
| **RETURNS** | The original patterns, one dictionary per pattern. ~~List[Dict[str, Union[str, dict]]]~~ | | **RETURNS** | The original patterns, one dictionary per pattern. ~~List[Dict[str, Union[str, dict]]]~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| ----------------- | --------------------------------------------------------------------------------------------------------------------- | | ----------------- | --------------------------------------------------------------------------------------------------------------------- |

View File

@ -3,7 +3,7 @@ title: Example
teaser: A training instance teaser: A training instance
tag: class tag: class
source: spacy/training/example.pyx source: spacy/training/example.pyx
new: 3.0 version: 3.0
--- ---
An `Example` holds the information for one training instance. It stores two An `Example` holds the information for one training instance. It stores two
@ -12,7 +12,7 @@ holding the predictions of the pipeline. An
[`Alignment`](/api/example#alignment-object) object stores the alignment between [`Alignment`](/api/example#alignment-object) object stores the alignment between
these two documents, as they can differ in tokenization. these two documents, as they can differ in tokenization.
## Example.\_\_init\_\_ {#init tag="method"} ## Example.\_\_init\_\_ {id="init",tag="method"}
Construct an `Example` object from the `predicted` document and the `reference` Construct an `Example` object from the `predicted` document and the `reference`
document. If `alignment` is `None`, it will be initialized from the words in document. If `alignment` is `None`, it will be initialized from the words in
@ -40,7 +40,7 @@ both documents.
| _keyword-only_ | | | _keyword-only_ | |
| `alignment` | An object holding the alignment between the tokens of the `predicted` and `reference` documents. ~~Optional[Alignment]~~ | | `alignment` | An object holding the alignment between the tokens of the `predicted` and `reference` documents. ~~Optional[Alignment]~~ |
## Example.from_dict {#from_dict tag="classmethod"} ## Example.from_dict {id="from_dict",tag="classmethod"}
Construct an `Example` object from the `predicted` document and the reference Construct an `Example` object from the `predicted` document and the reference
annotations provided as a dictionary. For more details on the required format, annotations provided as a dictionary. For more details on the required format,
@ -64,7 +64,7 @@ see the [training format documentation](/api/data-formats#dict-input).
| `example_dict` | The gold-standard annotations as a dictionary. Cannot be `None`. ~~Dict[str, Any]~~ | | `example_dict` | The gold-standard annotations as a dictionary. Cannot be `None`. ~~Dict[str, Any]~~ |
| **RETURNS** | The newly constructed object. ~~Example~~ | | **RETURNS** | The newly constructed object. ~~Example~~ |
## Example.text {#text tag="property"} ## Example.text {id="text",tag="property"}
The text of the `predicted` document in this `Example`. The text of the `predicted` document in this `Example`.
@ -78,7 +78,7 @@ The text of the `predicted` document in this `Example`.
| ----------- | --------------------------------------------- | | ----------- | --------------------------------------------- |
| **RETURNS** | The text of the `predicted` document. ~~str~~ | | **RETURNS** | The text of the `predicted` document. ~~str~~ |
## Example.predicted {#predicted tag="property"} ## Example.predicted {id="predicted",tag="property"}
The `Doc` holding the predictions. Occasionally also referred to as `example.x`. The `Doc` holding the predictions. Occasionally also referred to as `example.x`.
@ -94,7 +94,7 @@ The `Doc` holding the predictions. Occasionally also referred to as `example.x`.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | The document containing (partial) predictions. ~~Doc~~ | | **RETURNS** | The document containing (partial) predictions. ~~Doc~~ |
## Example.reference {#reference tag="property"} ## Example.reference {id="reference",tag="property"}
The `Doc` holding the gold-standard annotations. Occasionally also referred to The `Doc` holding the gold-standard annotations. Occasionally also referred to
as `example.y`. as `example.y`.
@ -111,7 +111,7 @@ as `example.y`.
| ----------- | ---------------------------------------------------------- | | ----------- | ---------------------------------------------------------- |
| **RETURNS** | The document containing gold-standard annotations. ~~Doc~~ | | **RETURNS** | The document containing gold-standard annotations. ~~Doc~~ |
## Example.alignment {#alignment tag="property"} ## Example.alignment {id="alignment",tag="property"}
The [`Alignment`](/api/example#alignment-object) object mapping the tokens of The [`Alignment`](/api/example#alignment-object) object mapping the tokens of
the `predicted` document to those of the `reference` document. the `predicted` document to those of the `reference` document.
@ -131,7 +131,7 @@ the `predicted` document to those of the `reference` document.
| ----------- | ---------------------------------------------------------------- | | ----------- | ---------------------------------------------------------------- |
| **RETURNS** | The document containing gold-standard annotations. ~~Alignment~~ | | **RETURNS** | The document containing gold-standard annotations. ~~Alignment~~ |
## Example.get_aligned {#get_aligned tag="method"} ## Example.get_aligned {id="get_aligned",tag="method"}
Get the aligned view of a certain token attribute, denoted by its int ID or Get the aligned view of a certain token attribute, denoted by its int ID or
string name. string name.
@ -152,7 +152,7 @@ string name.
| `as_string` | Whether or not to return the list of values as strings. Defaults to `False`. ~~bool~~ | | `as_string` | Whether or not to return the list of values as strings. Defaults to `False`. ~~bool~~ |
| **RETURNS** | List of integer values, or string values if `as_string` is `True`. ~~Union[List[int], List[str]]~~ | | **RETURNS** | List of integer values, or string values if `as_string` is `True`. ~~Union[List[int], List[str]]~~ |
## Example.get_aligned_parse {#get_aligned_parse tag="method"} ## Example.get_aligned_parse {id="get_aligned_parse",tag="method"}
Get the aligned view of the dependency parse. If `projectivize` is set to Get the aligned view of the dependency parse. If `projectivize` is set to
`True`, non-projective dependency trees are made projective through the `True`, non-projective dependency trees are made projective through the
@ -172,7 +172,7 @@ Pseudo-Projective Dependency Parsing algorithm by Nivre and Nilsson (2005).
| `projectivize` | Whether or not to projectivize the dependency trees. Defaults to `True`. ~~bool~~ | | `projectivize` | Whether or not to projectivize the dependency trees. Defaults to `True`. ~~bool~~ |
| **RETURNS** | List of integer values, or string values if `as_string` is `True`. ~~Union[List[int], List[str]]~~ | | **RETURNS** | List of integer values, or string values if `as_string` is `True`. ~~Union[List[int], List[str]]~~ |
## Example.get_aligned_ner {#get_aligned_ner tag="method"} ## Example.get_aligned_ner {id="get_aligned_ner",tag="method"}
Get the aligned view of the NER Get the aligned view of the NER
[BILUO](/usage/linguistic-features#accessing-ner) tags. [BILUO](/usage/linguistic-features#accessing-ner) tags.
@ -193,7 +193,7 @@ Get the aligned view of the NER
| ----------- | ------------------------------------------------------------------------------------------------- | | ----------- | ------------------------------------------------------------------------------------------------- |
| **RETURNS** | List of BILUO values, denoting whether tokens are part of an NER annotation or not. ~~List[str]~~ | | **RETURNS** | List of BILUO values, denoting whether tokens are part of an NER annotation or not. ~~List[str]~~ |
## Example.get_aligned_spans_y2x {#get_aligned_spans_y2x tag="method"} ## Example.get_aligned_spans_y2x {id="get_aligned_spans_y2x",tag="method"}
Get the aligned view of any set of [`Span`](/api/span) objects defined over Get the aligned view of any set of [`Span`](/api/span) objects defined over
[`Example.reference`](/api/example#reference). The resulting span indices will [`Example.reference`](/api/example#reference). The resulting span indices will
@ -219,7 +219,7 @@ align to the tokenization in [`Example.predicted`](/api/example#predicted).
| `allow_overlap` | Whether the resulting `Span` objects may overlap or not. Set to `False` by default. ~~bool~~ | | `allow_overlap` | Whether the resulting `Span` objects may overlap or not. Set to `False` by default. ~~bool~~ |
| **RETURNS** | `Span` objects aligned to the tokenization of `predicted`. ~~List[Span]~~ | | **RETURNS** | `Span` objects aligned to the tokenization of `predicted`. ~~List[Span]~~ |
## Example.get_aligned_spans_x2y {#get_aligned_spans_x2y tag="method"} ## Example.get_aligned_spans_x2y {id="get_aligned_spans_x2y",tag="method"}
Get the aligned view of any set of [`Span`](/api/span) objects defined over Get the aligned view of any set of [`Span`](/api/span) objects defined over
[`Example.predicted`](/api/example#predicted). The resulting span indices will [`Example.predicted`](/api/example#predicted). The resulting span indices will
@ -247,7 +247,7 @@ against the original gold-standard annotation.
| `allow_overlap` | Whether the resulting `Span` objects may overlap or not. Set to `False` by default. ~~bool~~ | | `allow_overlap` | Whether the resulting `Span` objects may overlap or not. Set to `False` by default. ~~bool~~ |
| **RETURNS** | `Span` objects aligned to the tokenization of `reference`. ~~List[Span]~~ | | **RETURNS** | `Span` objects aligned to the tokenization of `reference`. ~~List[Span]~~ |
## Example.to_dict {#to_dict tag="method"} ## Example.to_dict {id="to_dict",tag="method"}
Return a [dictionary representation](/api/data-formats#dict-input) of the Return a [dictionary representation](/api/data-formats#dict-input) of the
reference annotation contained in this `Example`. reference annotation contained in this `Example`.
@ -262,7 +262,7 @@ reference annotation contained in this `Example`.
| ----------- | ------------------------------------------------------------------------- | | ----------- | ------------------------------------------------------------------------- |
| **RETURNS** | Dictionary representation of the reference annotation. ~~Dict[str, Any]~~ | | **RETURNS** | Dictionary representation of the reference annotation. ~~Dict[str, Any]~~ |
## Example.split_sents {#split_sents tag="method"} ## Example.split_sents {id="split_sents",tag="method"}
Split one `Example` into multiple `Example` objects, one for each sentence. Split one `Example` into multiple `Example` objects, one for each sentence.
@ -282,15 +282,15 @@ Split one `Example` into multiple `Example` objects, one for each sentence.
| ----------- | ---------------------------------------------------------------------------- | | ----------- | ---------------------------------------------------------------------------- |
| **RETURNS** | List of `Example` objects, one for each original sentence. ~~List[Example]~~ | | **RETURNS** | List of `Example` objects, one for each original sentence. ~~List[Example]~~ |
## Alignment {#alignment-object new="3"} ## Alignment {id="alignment-object",version="3"}
Calculate alignment tables between two tokenizations. Calculate alignment tables between two tokenizations.
### Alignment attributes {#alignment-attributes"} ### Alignment attributes {id="alignment-attributes"}
Alignment attributes are managed using `AlignmentArray`, which is a Alignment attributes are managed using `AlignmentArray`, which is a simplified
simplified version of Thinc's [Ragged](https://thinc.ai/docs/api-types#ragged) version of Thinc's [Ragged](https://thinc.ai/docs/api-types#ragged) type that
type that only supports the `data` and `length` attributes. only supports the `data` and `length` attributes.
| Name | Description | | Name | Description |
| ----- | ------------------------------------------------------------------------------------- | | ----- | ------------------------------------------------------------------------------------- |
@ -321,7 +321,7 @@ tokenizations add up to the same string. For example, you'll be able to align
> If `a2b.data[1] == a2b.data[2] == 1`, that means that `A[1]` (`"'"`) and > If `a2b.data[1] == a2b.data[2] == 1`, that means that `A[1]` (`"'"`) and
> `A[2]` (`"s"`) both align to `B[1]` (`"'s"`). > `A[2]` (`"s"`) both align to `B[1]` (`"'s"`).
### Alignment.from_strings {#classmethod tag="function"} ### Alignment.from_strings {id="classmethod",tag="function"}
| Name | Description | | Name | Description |
| ----------- | ------------------------------------------------------------- | | ----------- | ------------------------------------------------------------- |

View File

@ -3,6 +3,4 @@ title: Library Architecture
next: /api/architectures next: /api/architectures
--- ---
import Architecture101 from 'usage/101/\_architecture.md'
<Architecture101 /> <Architecture101 />

View File

@ -5,7 +5,7 @@ teaser:
(ontology) (ontology)
tag: class tag: class
source: spacy/kb/kb.pyx source: spacy/kb/kb.pyx
new: 2.2 version: 2.2
--- ---
The `KnowledgeBase` object is an abstract class providing a method to generate The `KnowledgeBase` object is an abstract class providing a method to generate
@ -26,7 +26,7 @@ onwards.
</Infobox> </Infobox>
## KnowledgeBase.\_\_init\_\_ {#init tag="method"} ## KnowledgeBase.\_\_init\_\_ {id="init",tag="method"}
`KnowledgeBase` is an abstract class and cannot be instantiated. Its child `KnowledgeBase` is an abstract class and cannot be instantiated. Its child
classes should call `__init__()` to set up some necessary attributes. classes should call `__init__()` to set up some necessary attributes.
@ -50,7 +50,7 @@ classes should call `__init__()` to set up some necessary attributes.
| `vocab` | The shared vocabulary. ~~Vocab~~ | | `vocab` | The shared vocabulary. ~~Vocab~~ |
| `entity_vector_length` | Length of the fixed-size entity vectors. ~~int~~ | | `entity_vector_length` | Length of the fixed-size entity vectors. ~~int~~ |
## KnowledgeBase.entity_vector_length {#entity_vector_length tag="property"} ## KnowledgeBase.entity_vector_length {id="entity_vector_length",tag="property"}
The length of the fixed-size entity vectors in the knowledge base. The length of the fixed-size entity vectors in the knowledge base.
@ -58,7 +58,7 @@ The length of the fixed-size entity vectors in the knowledge base.
| ----------- | ------------------------------------------------ | | ----------- | ------------------------------------------------ |
| **RETURNS** | Length of the fixed-size entity vectors. ~~int~~ | | **RETURNS** | Length of the fixed-size entity vectors. ~~int~~ |
## KnowledgeBase.get_candidates {#get_candidates tag="method"} ## KnowledgeBase.get_candidates {id="get_candidates",tag="method"}
Given a certain textual mention as input, retrieve a list of candidate entities Given a certain textual mention as input, retrieve a list of candidate entities
of type [`Candidate`](/api/kb#candidate). of type [`Candidate`](/api/kb#candidate).
@ -77,7 +77,7 @@ of type [`Candidate`](/api/kb#candidate).
| `mention` | The textual mention or alias. ~~Span~~ | | `mention` | The textual mention or alias. ~~Span~~ |
| **RETURNS** | An iterable of relevant `Candidate` objects. ~~Iterable[Candidate]~~ | | **RETURNS** | An iterable of relevant `Candidate` objects. ~~Iterable[Candidate]~~ |
## KnowledgeBase.get_candidates_batch {#get_candidates_batch tag="method"} ## KnowledgeBase.get_candidates_batch {id="get_candidates_batch",tag="method"}
Same as [`get_candidates()`](/api/kb#get_candidates), but for an arbitrary Same as [`get_candidates()`](/api/kb#get_candidates), but for an arbitrary
number of mentions. The [`EntityLinker`](/api/entitylinker) component will call number of mentions. The [`EntityLinker`](/api/entitylinker) component will call
@ -103,7 +103,7 @@ to you.
| `mentions` | The textual mention or alias. ~~Iterable[Span]~~ | | `mentions` | The textual mention or alias. ~~Iterable[Span]~~ |
| **RETURNS** | An iterable of iterable with relevant `Candidate` objects. ~~Iterable[Iterable[Candidate]]~~ | | **RETURNS** | An iterable of iterable with relevant `Candidate` objects. ~~Iterable[Iterable[Candidate]]~~ |
## KnowledgeBase.get_alias_candidates {#get_alias_candidates tag="method"} ## KnowledgeBase.get_alias_candidates {id="get_alias_candidates",tag="method"}
<Infobox variant="warning"> <Infobox variant="warning">
This method is _not_ available from spaCy 3.5 onwards. This method is _not_ available from spaCy 3.5 onwards.
@ -119,7 +119,7 @@ Note: [`InMemoryLookupKB.get_candidates()`](/api/kb_in_memory#get_candidates)
defaults to defaults to
[`InMemoryLookupKB.get_alias_candidates()`](/api/kb_in_memory#get_alias_candidates). [`InMemoryLookupKB.get_alias_candidates()`](/api/kb_in_memory#get_alias_candidates).
## KnowledgeBase.get_vector {#get_vector tag="method"} ## KnowledgeBase.get_vector {id="get_vector",tag="method"}
Given a certain entity ID, retrieve its pretrained entity vector. Given a certain entity ID, retrieve its pretrained entity vector.
@ -134,7 +134,7 @@ Given a certain entity ID, retrieve its pretrained entity vector.
| `entity` | The entity ID. ~~str~~ | | `entity` | The entity ID. ~~str~~ |
| **RETURNS** | The entity vector. ~~Iterable[float]~~ | | **RETURNS** | The entity vector. ~~Iterable[float]~~ |
## KnowledgeBase.get_vectors {#get_vectors tag="method"} ## KnowledgeBase.get_vectors {id="get_vectors",tag="method"}
Same as [`get_vector()`](/api/kb#get_vector), but for an arbitrary number of Same as [`get_vector()`](/api/kb#get_vector), but for an arbitrary number of
entity IDs. entity IDs.
@ -154,7 +154,7 @@ entities at once, if performance is of concern to you.
| `entities` | The entity IDs. ~~Iterable[str]~~ | | `entities` | The entity IDs. ~~Iterable[str]~~ |
| **RETURNS** | The entity vectors. ~~Iterable[Iterable[numpy.ndarray]]~~ | | **RETURNS** | The entity vectors. ~~Iterable[Iterable[numpy.ndarray]]~~ |
## KnowledgeBase.to_disk {#to_disk tag="method"} ## KnowledgeBase.to_disk {id="to_disk",tag="method"}
Save the current state of the knowledge base to a directory. Save the current state of the knowledge base to a directory.
@ -169,7 +169,7 @@ Save the current state of the knowledge base to a directory.
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| `exclude` | List of components to exclude. ~~Iterable[str]~~ | | `exclude` | List of components to exclude. ~~Iterable[str]~~ |
## KnowledgeBase.from_disk {#from_disk tag="method"} ## KnowledgeBase.from_disk {id="from_disk",tag="method"}
Restore the state of the knowledge base from a given directory. Note that the Restore the state of the knowledge base from a given directory. Note that the
[`Vocab`](/api/vocab) should also be the same as the one used to create the KB. [`Vocab`](/api/vocab) should also be the same as the one used to create the KB.
@ -189,7 +189,7 @@ Restore the state of the knowledge base from a given directory. Note that the
| `exclude` | List of components to exclude. ~~Iterable[str]~~ | | `exclude` | List of components to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `KnowledgeBase` object. ~~KnowledgeBase~~ | | **RETURNS** | The modified `KnowledgeBase` object. ~~KnowledgeBase~~ |
## Candidate {#candidate tag="class"} ## Candidate {id="candidate",tag="class"}
A `Candidate` object refers to a textual mention (alias) that may or may not be A `Candidate` object refers to a textual mention (alias) that may or may not be
resolved to a specific entity from a `KnowledgeBase`. This will be used as input resolved to a specific entity from a `KnowledgeBase`. This will be used as input
@ -197,7 +197,7 @@ for the entity linking algorithm which will disambiguate the various candidates
to the correct one. Each candidate `(alias, entity)` pair is assigned to a to the correct one. Each candidate `(alias, entity)` pair is assigned to a
certain prior probability. certain prior probability.
### Candidate.\_\_init\_\_ {#candidate-init tag="method"} ### Candidate.\_\_init\_\_ {id="candidate-init",tag="method"}
Construct a `Candidate` object. Usually this constructor is not called directly, Construct a `Candidate` object. Usually this constructor is not called directly,
but instead these objects are returned by the `get_candidates` method of the but instead these objects are returned by the `get_candidates` method of the
@ -218,7 +218,7 @@ but instead these objects are returned by the `get_candidates` method of the
| `alias_hash` | The hash of the textual mention or alias. ~~int~~ | | `alias_hash` | The hash of the textual mention or alias. ~~int~~ |
| `prior_prob` | The prior probability of the `alias` referring to the `entity`. ~~float~~ | | `prior_prob` | The prior probability of the `alias` referring to the `entity`. ~~float~~ |
## Candidate attributes {#candidate-attributes} ## Candidate attributes {id="candidate-attributes"}
| Name | Description | | Name | Description |
| --------------- | ------------------------------------------------------------------------ | | --------------- | ------------------------------------------------------------------------ |

View File

@ -5,7 +5,7 @@ teaser:
information in-memory. information in-memory.
tag: class tag: class
source: spacy/kb/kb_in_memory.pyx source: spacy/kb/kb_in_memory.pyx
new: 3.5 version: 3.5
--- ---
The `InMemoryLookupKB` class inherits from [`KnowledgeBase`](/api/kb) and The `InMemoryLookupKB` class inherits from [`KnowledgeBase`](/api/kb) and
@ -14,7 +14,7 @@ implements all of its methods. It stores all KB data in-memory and generates
entity names. It's highly optimized for both a low memory footprint and speed of entity names. It's highly optimized for both a low memory footprint and speed of
retrieval. retrieval.
## InMemoryLookupKB.\_\_init\_\_ {#init tag="method"} ## InMemoryLookupKB.\_\_init\_\_ {id="init",tag="method"}
Create the knowledge base. Create the knowledge base.
@ -31,7 +31,7 @@ Create the knowledge base.
| `vocab` | The shared vocabulary. ~~Vocab~~ | | `vocab` | The shared vocabulary. ~~Vocab~~ |
| `entity_vector_length` | Length of the fixed-size entity vectors. ~~int~~ | | `entity_vector_length` | Length of the fixed-size entity vectors. ~~int~~ |
## InMemoryLookupKB.entity_vector_length {#entity_vector_length tag="property"} ## InMemoryLookupKB.entity_vector_length {id="entity_vector_length",tag="property"}
The length of the fixed-size entity vectors in the knowledge base. The length of the fixed-size entity vectors in the knowledge base.
@ -39,7 +39,7 @@ The length of the fixed-size entity vectors in the knowledge base.
| ----------- | ------------------------------------------------ | | ----------- | ------------------------------------------------ |
| **RETURNS** | Length of the fixed-size entity vectors. ~~int~~ | | **RETURNS** | Length of the fixed-size entity vectors. ~~int~~ |
## InMemoryLookupKB.add_entity {#add_entity tag="method"} ## InMemoryLookupKB.add_entity {id="add_entity",tag="method"}
Add an entity to the knowledge base, specifying its corpus frequency and entity Add an entity to the knowledge base, specifying its corpus frequency and entity
vector, which should be of length vector, which should be of length
@ -58,7 +58,7 @@ vector, which should be of length
| `freq` | The frequency of the entity in a typical corpus. ~~float~~ | | `freq` | The frequency of the entity in a typical corpus. ~~float~~ |
| `entity_vector` | The pretrained vector of the entity. ~~numpy.ndarray~~ | | `entity_vector` | The pretrained vector of the entity. ~~numpy.ndarray~~ |
## InMemoryLookupKB.set_entities {#set_entities tag="method"} ## InMemoryLookupKB.set_entities {id="set_entities",tag="method"}
Define the full list of entities in the knowledge base, specifying the corpus Define the full list of entities in the knowledge base, specifying the corpus
frequency and entity vector for each entity. frequency and entity vector for each entity.
@ -75,7 +75,7 @@ frequency and entity vector for each entity.
| `freq_list` | List of entity frequencies. ~~Iterable[int]~~ | | `freq_list` | List of entity frequencies. ~~Iterable[int]~~ |
| `vector_list` | List of entity vectors. ~~Iterable[numpy.ndarray]~~ | | `vector_list` | List of entity vectors. ~~Iterable[numpy.ndarray]~~ |
## InMemoryLookupKB.add_alias {#add_alias tag="method"} ## InMemoryLookupKB.add_alias {id="add_alias",tag="method"}
Add an alias or mention to the knowledge base, specifying its potential KB Add an alias or mention to the knowledge base, specifying its potential KB
identifiers and their prior probabilities. The entity identifiers should refer identifiers and their prior probabilities. The entity identifiers should refer
@ -96,7 +96,7 @@ alias.
| `entities` | The potential entities that the alias may refer to. ~~Iterable[Union[str, int]]~~ | | `entities` | The potential entities that the alias may refer to. ~~Iterable[Union[str, int]]~~ |
| `probabilities` | The prior probabilities of each entity. ~~Iterable[float]~~ | | `probabilities` | The prior probabilities of each entity. ~~Iterable[float]~~ |
## InMemoryLookupKB.\_\_len\_\_ {#len tag="method"} ## InMemoryLookupKB.\_\_len\_\_ {id="len",tag="method"}
Get the total number of entities in the knowledge base. Get the total number of entities in the knowledge base.
@ -110,7 +110,7 @@ Get the total number of entities in the knowledge base.
| ----------- | ----------------------------------------------------- | | ----------- | ----------------------------------------------------- |
| **RETURNS** | The number of entities in the knowledge base. ~~int~~ | | **RETURNS** | The number of entities in the knowledge base. ~~int~~ |
## InMemoryLookupKB.get_entity_strings {#get_entity_strings tag="method"} ## InMemoryLookupKB.get_entity_strings {id="get_entity_strings",tag="method"}
Get a list of all entity IDs in the knowledge base. Get a list of all entity IDs in the knowledge base.
@ -124,7 +124,7 @@ Get a list of all entity IDs in the knowledge base.
| ----------- | --------------------------------------------------------- | | ----------- | --------------------------------------------------------- |
| **RETURNS** | The list of entities in the knowledge base. ~~List[str]~~ | | **RETURNS** | The list of entities in the knowledge base. ~~List[str]~~ |
## InMemoryLookupKB.get_size_aliases {#get_size_aliases tag="method"} ## InMemoryLookupKB.get_size_aliases {id="get_size_aliases",tag="method"}
Get the total number of aliases in the knowledge base. Get the total number of aliases in the knowledge base.
@ -138,7 +138,7 @@ Get the total number of aliases in the knowledge base.
| ----------- | ---------------------------------------------------- | | ----------- | ---------------------------------------------------- |
| **RETURNS** | The number of aliases in the knowledge base. ~~int~~ | | **RETURNS** | The number of aliases in the knowledge base. ~~int~~ |
## InMemoryLookupKB.get_alias_strings {#get_alias_strings tag="method"} ## InMemoryLookupKB.get_alias_strings {id="get_alias_strings",tag="method"}
Get a list of all aliases in the knowledge base. Get a list of all aliases in the knowledge base.
@ -152,7 +152,7 @@ Get a list of all aliases in the knowledge base.
| ----------- | -------------------------------------------------------- | | ----------- | -------------------------------------------------------- |
| **RETURNS** | The list of aliases in the knowledge base. ~~List[str]~~ | | **RETURNS** | The list of aliases in the knowledge base. ~~List[str]~~ |
## InMemoryLookupKB.get_candidates {#get_candidates tag="method"} ## InMemoryLookupKB.get_candidates {id="get_candidates",tag="method"}
Given a certain textual mention as input, retrieve a list of candidate entities Given a certain textual mention as input, retrieve a list of candidate entities
of type [`Candidate`](/api/kb#candidate). Wraps of type [`Candidate`](/api/kb#candidate). Wraps
@ -172,7 +172,7 @@ of type [`Candidate`](/api/kb#candidate). Wraps
| `mention` | The textual mention or alias. ~~Span~~ | | `mention` | The textual mention or alias. ~~Span~~ |
| **RETURNS** | An iterable of relevant `Candidate` objects. ~~Iterable[Candidate]~~ | | **RETURNS** | An iterable of relevant `Candidate` objects. ~~Iterable[Candidate]~~ |
## InMemoryLookupKB.get_candidates_batch {#get_candidates_batch tag="method"} ## InMemoryLookupKB.get_candidates_batch {id="get_candidates_batch",tag="method"}
Same as [`get_candidates()`](/api/kb_in_memory#get_candidates), but for an Same as [`get_candidates()`](/api/kb_in_memory#get_candidates), but for an
arbitrary number of mentions. The [`EntityLinker`](/api/entitylinker) component arbitrary number of mentions. The [`EntityLinker`](/api/entitylinker) component
@ -198,7 +198,7 @@ to you.
| `mentions` | The textual mention or alias. ~~Iterable[Span]~~ | | `mentions` | The textual mention or alias. ~~Iterable[Span]~~ |
| **RETURNS** | An iterable of iterable with relevant `Candidate` objects. ~~Iterable[Iterable[Candidate]]~~ | | **RETURNS** | An iterable of iterable with relevant `Candidate` objects. ~~Iterable[Iterable[Candidate]]~~ |
## InMemoryLookupKB.get_alias_candidates {#get_alias_candidates tag="method"} ## InMemoryLookupKB.get_alias_candidates {id="get_alias_candidates",tag="method"}
Given a certain textual mention as input, retrieve a list of candidate entities Given a certain textual mention as input, retrieve a list of candidate entities
of type [`Candidate`](/api/kb#candidate). of type [`Candidate`](/api/kb#candidate).
@ -214,7 +214,7 @@ of type [`Candidate`](/api/kb#candidate).
| `alias` | The textual mention or alias. ~~str~~ | | `alias` | The textual mention or alias. ~~str~~ |
| **RETURNS** | The list of relevant `Candidate` objects. ~~List[Candidate]~~ | | **RETURNS** | The list of relevant `Candidate` objects. ~~List[Candidate]~~ |
## InMemoryLookupKB.get_vector {#get_vector tag="method"} ## InMemoryLookupKB.get_vector {id="get_vector",tag="method"}
Given a certain entity ID, retrieve its pretrained entity vector. Given a certain entity ID, retrieve its pretrained entity vector.
@ -229,7 +229,7 @@ Given a certain entity ID, retrieve its pretrained entity vector.
| `entity` | The entity ID. ~~str~~ | | `entity` | The entity ID. ~~str~~ |
| **RETURNS** | The entity vector. ~~numpy.ndarray~~ | | **RETURNS** | The entity vector. ~~numpy.ndarray~~ |
## InMemoryLookupKB.get_vectors {#get_vectors tag="method"} ## InMemoryLookupKB.get_vectors {id="get_vectors",tag="method"}
Same as [`get_vector()`](/api/kb_in_memory#get_vector), but for an arbitrary Same as [`get_vector()`](/api/kb_in_memory#get_vector), but for an arbitrary
number of entity IDs. number of entity IDs.
@ -249,7 +249,7 @@ entities at once, if performance is of concern to you.
| `entities` | The entity IDs. ~~Iterable[str]~~ | | `entities` | The entity IDs. ~~Iterable[str]~~ |
| **RETURNS** | The entity vectors. ~~Iterable[Iterable[numpy.ndarray]]~~ | | **RETURNS** | The entity vectors. ~~Iterable[Iterable[numpy.ndarray]]~~ |
## InMemoryLookupKB.get_prior_prob {#get_prior_prob tag="method"} ## InMemoryLookupKB.get_prior_prob {id="get_prior_prob",tag="method"}
Given a certain entity ID and a certain textual mention, retrieve the prior Given a certain entity ID and a certain textual mention, retrieve the prior
probability of the fact that the mention links to the entity ID. probability of the fact that the mention links to the entity ID.
@ -266,7 +266,7 @@ probability of the fact that the mention links to the entity ID.
| `alias` | The textual mention or alias. ~~str~~ | | `alias` | The textual mention or alias. ~~str~~ |
| **RETURNS** | The prior probability of the `alias` referring to the `entity`. ~~float~~ | | **RETURNS** | The prior probability of the `alias` referring to the `entity`. ~~float~~ |
## InMemoryLookupKB.to_disk {#to_disk tag="method"} ## InMemoryLookupKB.to_disk {id="to_disk",tag="method"}
Save the current state of the knowledge base to a directory. Save the current state of the knowledge base to a directory.
@ -281,7 +281,7 @@ Save the current state of the knowledge base to a directory.
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| `exclude` | List of components to exclude. ~~Iterable[str]~~ | | `exclude` | List of components to exclude. ~~Iterable[str]~~ |
## InMemoryLookupKB.from_disk {#from_disk tag="method"} ## InMemoryLookupKB.from_disk {id="from_disk",tag="method"}
Restore the state of the knowledge base from a given directory. Note that the Restore the state of the knowledge base from a given directory. Note that the
[`Vocab`](/api/vocab) should also be the same as the one used to create the KB. [`Vocab`](/api/vocab) should also be the same as the one used to create the KB.

View File

@ -15,7 +15,7 @@ the tagger or parser that are called on a document in order. You can also add
your own processing pipeline components that take a `Doc` object, modify it and your own processing pipeline components that take a `Doc` object, modify it and
return it. return it.
## Language.\_\_init\_\_ {#init tag="method"} ## Language.\_\_init\_\_ {id="init",tag="method"}
Initialize a `Language` object. Note that the `meta` is only used for meta Initialize a `Language` object. Note that the `meta` is only used for meta
information in [`Language.meta`](/api/language#meta) and not to configure the information in [`Language.meta`](/api/language#meta) and not to configure the
@ -44,7 +44,7 @@ information in [`Language.meta`](/api/language#meta) and not to configure the
| `create_tokenizer` | Optional function that receives the `nlp` object and returns a tokenizer. ~~Callable[[Language], Callable[[str], Doc]]~~ | | `create_tokenizer` | Optional function that receives the `nlp` object and returns a tokenizer. ~~Callable[[Language], Callable[[str], Doc]]~~ |
| `batch_size` | Default batch size for [`pipe`](#pipe) and [`evaluate`](#evaluate). Defaults to `1000`. ~~int~~ | | `batch_size` | Default batch size for [`pipe`](#pipe) and [`evaluate`](#evaluate). Defaults to `1000`. ~~int~~ |
## Language.from_config {#from_config tag="classmethod" new="3"} ## Language.from_config {id="from_config",tag="classmethod",version="3"}
Create a `Language` object from a loaded config. Will set up the tokenizer and Create a `Language` object from a loaded config. Will set up the tokenizer and
language data, add pipeline components based on the pipeline and add pipeline language data, add pipeline components based on the pipeline and add pipeline
@ -76,7 +76,7 @@ spaCy loads a model under the hood based on its
| `validate` | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ | | `validate` | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The initialized object. ~~Language~~ | | **RETURNS** | The initialized object. ~~Language~~ |
## Language.component {#component tag="classmethod" new="3"} ## Language.component {id="component",tag="classmethod",version="3"}
Register a custom pipeline component under a given name. This allows Register a custom pipeline component under a given name. This allows
initializing the component by name using initializing the component by name using
@ -112,7 +112,7 @@ decorator. For more details and examples, see the
| `retokenizes` | Whether the component changes tokenization. Used for [pipe analysis](/usage/processing-pipelines#analysis). ~~bool~~ | | `retokenizes` | Whether the component changes tokenization. Used for [pipe analysis](/usage/processing-pipelines#analysis). ~~bool~~ |
| `func` | Optional function if not used as a decorator. ~~Optional[Callable[[Doc], Doc]]~~ | | `func` | Optional function if not used as a decorator. ~~Optional[Callable[[Doc], Doc]]~~ |
## Language.factory {#factory tag="classmethod"} ## Language.factory {id="factory",tag="classmethod"}
Register a custom pipeline component factory under a given name. This allows Register a custom pipeline component factory under a given name. This allows
initializing the component by name using initializing the component by name using
@ -159,7 +159,7 @@ examples, see the
| `default_score_weights` | The scores to report during training, and their default weight towards the final score used to select the best model. Weights should sum to `1.0` per component and will be combined and normalized for the whole pipeline. If a weight is set to `None`, the score will not be logged or weighted. ~~Dict[str, Optional[float]]~~ | | `default_score_weights` | The scores to report during training, and their default weight towards the final score used to select the best model. Weights should sum to `1.0` per component and will be combined and normalized for the whole pipeline. If a weight is set to `None`, the score will not be logged or weighted. ~~Dict[str, Optional[float]]~~ |
| `func` | Optional function if not used as a decorator. ~~Optional[Callable[[...], Callable[[Doc], Doc]]]~~ | | `func` | Optional function if not used as a decorator. ~~Optional[Callable[[...], Callable[[Doc], Doc]]]~~ |
## Language.\_\_call\_\_ {#call tag="method"} ## Language.\_\_call\_\_ {id="call",tag="method"}
Apply the pipeline to some text. The text can span multiple sentences, and can Apply the pipeline to some text. The text can span multiple sentences, and can
contain arbitrary whitespace. Alignment into the original string is preserved. contain arbitrary whitespace. Alignment into the original string is preserved.
@ -182,7 +182,7 @@ skipped, but the rest of the pipeline is run.
| `component_cfg` | Optional dictionary of keyword arguments for components, keyed by component names. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Any]]]~~ | | `component_cfg` | Optional dictionary of keyword arguments for components, keyed by component names. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Any]]]~~ |
| **RETURNS** | A container for accessing the annotations. ~~Doc~~ | | **RETURNS** | A container for accessing the annotations. ~~Doc~~ |
## Language.pipe {#pipe tag="method"} ## Language.pipe {id="pipe",tag="method"}
Process texts as a stream, and yield `Doc` objects in order. This is usually Process texts as a stream, and yield `Doc` objects in order. This is usually
more efficient than processing texts one-by-one. more efficient than processing texts one-by-one.
@ -209,7 +209,7 @@ tokenization is skipped but the rest of the pipeline is run.
| `n_process` | Number of processors to use. Defaults to `1`. ~~int~~ | | `n_process` | Number of processors to use. Defaults to `1`. ~~int~~ |
| **YIELDS** | Documents in the order of the original text. ~~Doc~~ | | **YIELDS** | Documents in the order of the original text. ~~Doc~~ |
## Language.set_error_handler {#set_error_handler tag="method" new="3"} ## Language.set_error_handler {id="set_error_handler",tag="method",version="3"}
Define a callback that will be invoked when an error is thrown during processing Define a callback that will be invoked when an error is thrown during processing
of one or more documents. Specifically, this function will call of one or more documents. Specifically, this function will call
@ -231,7 +231,7 @@ being processed, and the original error.
| --------------- | -------------------------------------------------------------------------------------------------------------- | | --------------- | -------------------------------------------------------------------------------------------------------------- |
| `error_handler` | A function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ | | `error_handler` | A function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ |
## Language.initialize {#initialize tag="method" new="3"} ## Language.initialize {id="initialize",tag="method",version="3"}
Initialize the pipeline for training and return an Initialize the pipeline for training and return an
[`Optimizer`](https://thinc.ai/docs/api-optimizers). Under the hood, it uses the [`Optimizer`](https://thinc.ai/docs/api-optimizers). Under the hood, it uses the
@ -282,7 +282,7 @@ objects.
| `sgd` | An optimizer. Will be created via [`create_optimizer`](#create_optimizer) if not set. ~~Optional[Optimizer]~~ | | `sgd` | An optimizer. Will be created via [`create_optimizer`](#create_optimizer) if not set. ~~Optional[Optimizer]~~ |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## Language.resume_training {#resume_training tag="method,experimental" new="3"} ## Language.resume_training {id="resume_training",tag="method,experimental",version="3"}
Continue training a trained pipeline. Create and return an optimizer, and Continue training a trained pipeline. Create and return an optimizer, and
initialize "rehearsal" for any pipeline component that has a `rehearse` method. initialize "rehearsal" for any pipeline component that has a `rehearse` method.
@ -304,7 +304,7 @@ a batch of [Example](/api/example) objects.
| `sgd` | An optimizer. Will be created via [`create_optimizer`](#create_optimizer) if not set. ~~Optional[Optimizer]~~ | | `sgd` | An optimizer. Will be created via [`create_optimizer`](#create_optimizer) if not set. ~~Optional[Optimizer]~~ |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## Language.update {#update tag="method"} ## Language.update {id="update",tag="method"}
Update the models in the pipeline. Update the models in the pipeline.
@ -342,7 +342,7 @@ and custom registered functions if needed. See the
| `component_cfg` | Optional dictionary of keyword arguments for components, keyed by component names. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Any]]]~~ | | `component_cfg` | Optional dictionary of keyword arguments for components, keyed by component names. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Any]]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Language.rehearse {#rehearse tag="method,experimental" new="3"} ## Language.rehearse {id="rehearse",tag="method,experimental",version="3"}
Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the
current model to make predictions similar to an initial model, to try to address current model to make predictions similar to an initial model, to try to address
@ -364,7 +364,7 @@ the "catastrophic forgetting" problem. This feature is experimental.
| `losses` | Dictionary to update with the loss, keyed by pipeline component. ~~Optional[Dict[str, float]]~~ | | `losses` | Dictionary to update with the loss, keyed by pipeline component. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Language.evaluate {#evaluate tag="method"} ## Language.evaluate {id="evaluate",tag="method"}
Evaluate a pipeline's components. Evaluate a pipeline's components.
@ -392,7 +392,7 @@ objects instead of tuples of `Doc` and `GoldParse` objects.
| `scorer_cfg` | Optional dictionary of keyword arguments for the `Scorer`. Defaults to `None`. ~~Optional[Dict[str, Any]]~~ | | `scorer_cfg` | Optional dictionary of keyword arguments for the `Scorer`. Defaults to `None`. ~~Optional[Dict[str, Any]]~~ |
| **RETURNS** | A dictionary of evaluation scores. ~~Dict[str, Union[float, Dict[str, float]]]~~ | | **RETURNS** | A dictionary of evaluation scores. ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## Language.use_params {#use_params tag="contextmanager, method"} ## Language.use_params {id="use_params",tag="contextmanager, method"}
Replace weights of models in the pipeline with those provided in the params Replace weights of models in the pipeline with those provided in the params
dictionary. Can be used as a context manager, in which case, models go back to dictionary. Can be used as a context manager, in which case, models go back to
@ -409,7 +409,7 @@ their original weights after the block.
| -------- | ------------------------------------------------------ | | -------- | ------------------------------------------------------ |
| `params` | A dictionary of parameters keyed by model ID. ~~dict~~ | | `params` | A dictionary of parameters keyed by model ID. ~~dict~~ |
## Language.add_pipe {#add_pipe tag="method" new="2"} ## Language.add_pipe {id="add_pipe",tag="method",version="2"}
Add a component to the processing pipeline. Expects a name that maps to a Add a component to the processing pipeline. Expects a name that maps to a
component factory registered using component factory registered using
@ -458,7 +458,7 @@ component, adds it to the pipeline and returns it.
| `validate` <Tag variant="new">3</Tag> | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ | | `validate` <Tag variant="new">3</Tag> | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ | | **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ |
## Language.create_pipe {#create_pipe tag="method" new="2"} ## Language.create_pipe {id="create_pipe",tag="method",version="2"}
Create a pipeline component from a factory. Create a pipeline component from a factory.
@ -487,7 +487,7 @@ To create a component and add it to the pipeline, you should always use
| `validate` <Tag variant="new">3</Tag> | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ | | `validate` <Tag variant="new">3</Tag> | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ | | **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ |
## Language.has_factory {#has_factory tag="classmethod" new="3"} ## Language.has_factory {id="has_factory",tag="classmethod",version="3"}
Check whether a factory name is registered on the `Language` class or subclass. Check whether a factory name is registered on the `Language` class or subclass.
Will check for Will check for
@ -514,7 +514,7 @@ the `Language` base class, available to all subclasses.
| `name` | Name of the pipeline factory to check. ~~str~~ | | `name` | Name of the pipeline factory to check. ~~str~~ |
| **RETURNS** | Whether a factory of that name is registered on the class. ~~bool~~ | | **RETURNS** | Whether a factory of that name is registered on the class. ~~bool~~ |
## Language.has_pipe {#has_pipe tag="method" new="2"} ## Language.has_pipe {id="has_pipe",tag="method",version="2"}
Check whether a component is present in the pipeline. Equivalent to Check whether a component is present in the pipeline. Equivalent to
`name in nlp.pipe_names`. `name in nlp.pipe_names`.
@ -536,7 +536,7 @@ Check whether a component is present in the pipeline. Equivalent to
| `name` | Name of the pipeline component to check. ~~str~~ | | `name` | Name of the pipeline component to check. ~~str~~ |
| **RETURNS** | Whether a component of that name exists in the pipeline. ~~bool~~ | | **RETURNS** | Whether a component of that name exists in the pipeline. ~~bool~~ |
## Language.get_pipe {#get_pipe tag="method" new="2"} ## Language.get_pipe {id="get_pipe",tag="method",version="2"}
Get a pipeline component for a given component name. Get a pipeline component for a given component name.
@ -552,7 +552,7 @@ Get a pipeline component for a given component name.
| `name` | Name of the pipeline component to get. ~~str~~ | | `name` | Name of the pipeline component to get. ~~str~~ |
| **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ | | **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ |
## Language.replace_pipe {#replace_pipe tag="method" new="2"} ## Language.replace_pipe {id="replace_pipe",tag="method",version="2"}
Replace a component in the pipeline and return the new component. Replace a component in the pipeline and return the new component.
@ -580,7 +580,7 @@ and instead expects the **name of a component factory** registered using
| `validate` <Tag variant="new">3</Tag> | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ | | `validate` <Tag variant="new">3</Tag> | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The new pipeline component. ~~Callable[[Doc], Doc]~~ | | **RETURNS** | The new pipeline component. ~~Callable[[Doc], Doc]~~ |
## Language.rename_pipe {#rename_pipe tag="method" new="2"} ## Language.rename_pipe {id="rename_pipe",tag="method",version="2"}
Rename a component in the pipeline. Useful to create custom names for Rename a component in the pipeline. Useful to create custom names for
pre-defined and pre-loaded components. To change the default name of a component pre-defined and pre-loaded components. To change the default name of a component
@ -598,7 +598,7 @@ added to the pipeline, you can also use the `name` argument on
| `old_name` | Name of the component to rename. ~~str~~ | | `old_name` | Name of the component to rename. ~~str~~ |
| `new_name` | New name of the component. ~~str~~ | | `new_name` | New name of the component. ~~str~~ |
## Language.remove_pipe {#remove_pipe tag="method" new="2"} ## Language.remove_pipe {id="remove_pipe",tag="method",version="2"}
Remove a component from the pipeline. Returns the removed component name and Remove a component from the pipeline. Returns the removed component name and
component function. component function.
@ -615,7 +615,7 @@ component function.
| `name` | Name of the component to remove. ~~str~~ | | `name` | Name of the component to remove. ~~str~~ |
| **RETURNS** | A `(name, component)` tuple of the removed component. ~~Tuple[str, Callable[[Doc], Doc]]~~ | | **RETURNS** | A `(name, component)` tuple of the removed component. ~~Tuple[str, Callable[[Doc], Doc]]~~ |
## Language.disable_pipe {#disable_pipe tag="method" new="3"} ## Language.disable_pipe {id="disable_pipe",tag="method",version="3"}
Temporarily disable a pipeline component so it's not run as part of the Temporarily disable a pipeline component so it's not run as part of the
pipeline. Disabled components are listed in pipeline. Disabled components are listed in
@ -641,7 +641,7 @@ does nothing.
| ------ | ----------------------------------------- | | ------ | ----------------------------------------- |
| `name` | Name of the component to disable. ~~str~~ | | `name` | Name of the component to disable. ~~str~~ |
## Language.enable_pipe {#enable_pipe tag="method" new="3"} ## Language.enable_pipe {id="enable_pipe",tag="method",version="3"}
Enable a previously disabled component (e.g. via Enable a previously disabled component (e.g. via
[`Language.disable_pipes`](/api/language#disable_pipes)) so it's run as part of [`Language.disable_pipes`](/api/language#disable_pipes)) so it's run as part of
@ -663,7 +663,7 @@ already enabled, this method does nothing.
| ------ | ---------------------------------------- | | ------ | ---------------------------------------- |
| `name` | Name of the component to enable. ~~str~~ | | `name` | Name of the component to enable. ~~str~~ |
## Language.select_pipes {#select_pipes tag="contextmanager, method" new="3"} ## Language.select_pipes {id="select_pipes",tag="contextmanager, method",version="3"}
Disable one or more pipeline components. If used as a context manager, the Disable one or more pipeline components. If used as a context manager, the
pipeline will be restored to the initial state at the end of the block. pipeline will be restored to the initial state at the end of the block.
@ -706,7 +706,7 @@ As of spaCy v3.0, the `disable_pipes` method has been renamed to `select_pipes`:
| `enable` | Name(s) of pipeline component(s) that will not be disabled. ~~Optional[Union[str, Iterable[str]]]~~ | | `enable` | Name(s) of pipeline component(s) that will not be disabled. ~~Optional[Union[str, Iterable[str]]]~~ |
| **RETURNS** | The disabled pipes that can be restored by calling the object's `.restore()` method. ~~DisabledPipes~~ | | **RETURNS** | The disabled pipes that can be restored by calling the object's `.restore()` method. ~~DisabledPipes~~ |
## Language.get_factory_meta {#get_factory_meta tag="classmethod" new="3"} ## Language.get_factory_meta {id="get_factory_meta",tag="classmethod",version="3"}
Get the factory meta information for a given pipeline component name. Expects Get the factory meta information for a given pipeline component name. Expects
the name of the component **factory**. The factory meta is an instance of the the name of the component **factory**. The factory meta is an instance of the
@ -728,7 +728,7 @@ information about the component and its default provided by the
| `name` | The factory name. ~~str~~ | | `name` | The factory name. ~~str~~ |
| **RETURNS** | The factory meta. ~~FactoryMeta~~ | | **RETURNS** | The factory meta. ~~FactoryMeta~~ |
## Language.get_pipe_meta {#get_pipe_meta tag="method" new="3"} ## Language.get_pipe_meta {id="get_pipe_meta",tag="method",version="3"}
Get the factory meta information for a given pipeline component name. Expects Get the factory meta information for a given pipeline component name. Expects
the name of the component **instance** in the pipeline. The factory meta is an the name of the component **instance** in the pipeline. The factory meta is an
@ -751,7 +751,7 @@ contains the information about the component and its default provided by the
| `name` | The pipeline component name. ~~str~~ | | `name` | The pipeline component name. ~~str~~ |
| **RETURNS** | The factory meta. ~~FactoryMeta~~ | | **RETURNS** | The factory meta. ~~FactoryMeta~~ |
## Language.analyze_pipes {#analyze_pipes tag="method" new="3"} ## Language.analyze_pipes {id="analyze_pipes",tag="method",version="3"}
Analyze the current pipeline components and show a summary of the attributes Analyze the current pipeline components and show a summary of the attributes
they assign and require, and the scores they set. The data is based on the they assign and require, and the scores they set. The data is based on the
@ -780,8 +780,7 @@ doesn't, the pipeline analysis won't catch that.
<Accordion title="Example output" spaced> <Accordion title="Example output" spaced>
```json ```json {title="Structured"}
### Structured
{ {
"summary": { "summary": {
"tagger": { "tagger": {
@ -799,7 +798,12 @@ doesn't, the pipeline analysis won't catch that.
}, },
"problems": { "problems": {
"tagger": [], "tagger": [],
"entity_linker": ["doc.ents", "doc.sents", "token.ent_iob", "token.ent_type"] "entity_linker": [
"doc.ents",
"doc.sents",
"token.ent_iob",
"token.ent_type"
]
}, },
"attrs": { "attrs": {
"token.ent_iob": { "assigns": [], "requires": ["entity_linker"] }, "token.ent_iob": { "assigns": [], "requires": ["entity_linker"] },
@ -840,7 +844,7 @@ token.ent_iob, token.ent_type
| `pretty` | Pretty-print the results as a table. Defaults to `False`. ~~bool~~ | | `pretty` | Pretty-print the results as a table. Defaults to `False`. ~~bool~~ |
| **RETURNS** | Dictionary containing the pipe analysis, keyed by `"summary"` (component meta by pipe), `"problems"` (attribute names by pipe) and `"attrs"` (pipes that assign and require an attribute, keyed by attribute). ~~Optional[Dict[str, Any]]~~ | | **RETURNS** | Dictionary containing the pipe analysis, keyed by `"summary"` (component meta by pipe), `"problems"` (attribute names by pipe) and `"attrs"` (pipes that assign and require an attribute, keyed by attribute). ~~Optional[Dict[str, Any]]~~ |
## Language.replace_listeners {#replace_listeners tag="method" new="3"} ## Language.replace_listeners {id="replace_listeners",tag="method",version="3"}
Find [listener layers](/usage/embeddings-transformers#embedding-layers) Find [listener layers](/usage/embeddings-transformers#embedding-layers)
(connecting to a shared token-to-vector embedding component) of a given pipeline (connecting to a shared token-to-vector embedding component) of a given pipeline
@ -885,7 +889,7 @@ when loading a config with
| `pipe_name` | Name of pipeline component to replace listeners for. ~~str~~ | | `pipe_name` | Name of pipeline component to replace listeners for. ~~str~~ |
| `listeners` | The paths to the listeners, relative to the component config, e.g. `["model.tok2vec"]`. Typically, implementations will only connect to one tok2vec component, `model.tok2vec`, but in theory, custom models can use multiple listeners. The value here can either be an empty list to not replace any listeners, or a _complete_ list of the paths to all listener layers used by the model that should be replaced.~~Iterable[str]~~ | | `listeners` | The paths to the listeners, relative to the component config, e.g. `["model.tok2vec"]`. Typically, implementations will only connect to one tok2vec component, `model.tok2vec`, but in theory, custom models can use multiple listeners. The value here can either be an empty list to not replace any listeners, or a _complete_ list of the paths to all listener layers used by the model that should be replaced.~~Iterable[str]~~ |
## Language.meta {#meta tag="property"} ## Language.meta {id="meta",tag="property"}
Meta data for the `Language` class, including name, version, data sources, Meta data for the `Language` class, including name, version, data sources,
license, author information and more. If a trained pipeline is loaded, this license, author information and more. If a trained pipeline is loaded, this
@ -911,7 +915,7 @@ information is expressed in the [`config.cfg`](/api/data-formats#config).
| ----------- | --------------------------------- | | ----------- | --------------------------------- |
| **RETURNS** | The meta data. ~~Dict[str, Any]~~ | | **RETURNS** | The meta data. ~~Dict[str, Any]~~ |
## Language.config {#config tag="property" new="3"} ## Language.config {id="config",tag="property",version="3"}
Export a trainable [`config.cfg`](/api/data-formats#config) for the current Export a trainable [`config.cfg`](/api/data-formats#config) for the current
`nlp` object. Includes the current pipeline, all configs used to create the `nlp` object. Includes the current pipeline, all configs used to create the
@ -932,7 +936,7 @@ subclass of the built-in `dict`. It supports the additional methods `to_disk`
| ----------- | ---------------------- | | ----------- | ---------------------- |
| **RETURNS** | The config. ~~Config~~ | | **RETURNS** | The config. ~~Config~~ |
## Language.to_disk {#to_disk tag="method" new="2"} ## Language.to_disk {id="to_disk",tag="method",version="2"}
Save the current state to a directory. Under the hood, this method delegates to Save the current state to a directory. Under the hood, this method delegates to
the `to_disk` methods of the individual pipeline components, if available. This the `to_disk` methods of the individual pipeline components, if available. This
@ -951,7 +955,7 @@ will be saved to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Language.from_disk {#from_disk tag="method" new="2"} ## Language.from_disk {id="from_disk",tag="method",version="2"}
Loads state from a directory, including all data that was saved with the Loads state from a directory, including all data that was saved with the
`Language` object. Modifies the object in place and returns it. `Language` object. Modifies the object in place and returns it.
@ -984,7 +988,7 @@ you want to load a serialized pipeline from a directory, you should use
| `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Language` object. ~~Language~~ | | **RETURNS** | The modified `Language` object. ~~Language~~ |
## Language.to_bytes {#to_bytes tag="method"} ## Language.to_bytes {id="to_bytes",tag="method"}
Serialize the current state to a binary string. Serialize the current state to a binary string.
@ -1000,7 +1004,7 @@ Serialize the current state to a binary string.
| `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~iterable~~ | | `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~iterable~~ |
| **RETURNS** | The serialized form of the `Language` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `Language` object. ~~bytes~~ |
## Language.from_bytes {#from_bytes tag="method"} ## Language.from_bytes {id="from_bytes",tag="method"}
Load state from a binary string. Note that this method is commonly used via the Load state from a binary string. Note that this method is commonly used via the
subclasses like `English` or `German` to make language-specific functionality subclasses like `English` or `German` to make language-specific functionality
@ -1028,7 +1032,7 @@ details.
| `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Language` object. ~~Language~~ | | **RETURNS** | The `Language` object. ~~Language~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
@ -1046,7 +1050,7 @@ details.
| `disabled` <Tag variant="new">3</Tag> | Names of components that are currently disabled and don't run as part of the pipeline. ~~List[str]~~ | | `disabled` <Tag variant="new">3</Tag> | Names of components that are currently disabled and don't run as part of the pipeline. ~~List[str]~~ |
| `path` | Path to the pipeline data directory, if a pipeline is loaded from a path or package. Otherwise `None`. ~~Optional[Path]~~ | | `path` | Path to the pipeline data directory, if a pipeline is loaded from a path or package. Otherwise `None`. ~~Optional[Path]~~ |
## Class attributes {#class-attributes} ## Class attributes {id="class-attributes"}
| Name | Description | | Name | Description |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@ -1054,7 +1058,7 @@ details.
| `lang` | [IETF language tag](https://www.w3.org/International/articles/language-tags/), such as 'en' for English. ~~str~~ | | `lang` | [IETF language tag](https://www.w3.org/International/articles/language-tags/), such as 'en' for English. ~~str~~ |
| `default_config` | Base [config](/usage/training#config) to use for [Language.config](/api/language#config). Defaults to [`default_config.cfg`](%%GITHUB_SPACY/spacy/default_config.cfg). ~~Config~~ | | `default_config` | Base [config](/usage/training#config) to use for [Language.config](/api/language#config). Defaults to [`default_config.cfg`](%%GITHUB_SPACY/spacy/default_config.cfg). ~~Config~~ |
## Defaults {#defaults} ## Defaults {id="defaults"}
The following attributes can be set on the `Language.Defaults` class to The following attributes can be set on the `Language.Defaults` class to
customize the default language data: customize the default language data:
@ -1097,7 +1101,7 @@ customize the default language data:
| `writing_system` | Information about the language's writing system, available via `Vocab.writing_system`. Defaults to: `{"direction": "ltr", "has_case": True, "has_letters": True}.`.<br />**Example:** [`zh/__init__.py`](%%GITHUB_SPACY/spacy/lang/zh/__init__.py) ~~Dict[str, Any]~~ | | `writing_system` | Information about the language's writing system, available via `Vocab.writing_system`. Defaults to: `{"direction": "ltr", "has_case": True, "has_letters": True}.`.<br />**Example:** [`zh/__init__.py`](%%GITHUB_SPACY/spacy/lang/zh/__init__.py) ~~Dict[str, Any]~~ |
| `config` | Default [config](/usage/training#config) added to `nlp.config`. This can include references to custom tokenizers or lemmatizers.<br />**Example:** [`zh/__init__.py`](%%GITHUB_SPACY/spacy/lang/zh/__init__.py) ~~Config~~ | | `config` | Default [config](/usage/training#config) added to `nlp.config`. This can include references to custom tokenizers or lemmatizers.<br />**Example:** [`zh/__init__.py`](%%GITHUB_SPACY/spacy/lang/zh/__init__.py) ~~Config~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from
@ -1117,7 +1121,7 @@ serialization by passing in the string names via the `exclude` argument.
| `meta` | The meta data, available as [`Language.meta`](/api/language#meta). | | `meta` | The meta data, available as [`Language.meta`](/api/language#meta). |
| ... | String names of pipeline components, e.g. `"ner"`. | | ... | String names of pipeline components, e.g. `"ner"`. |
## FactoryMeta {#factorymeta new="3" tag="dataclass"} ## FactoryMeta {id="factorymeta",version="3",tag="dataclass"}
The `FactoryMeta` contains the information about the component and its default The `FactoryMeta` contains the information about the component and its default
provided by the [`@Language.component`](/api/language#component) or provided by the [`@Language.component`](/api/language#component) or

View File

@ -12,11 +12,11 @@ functions that may still be used in projects.
You can find the detailed documentation of each such legacy function on this You can find the detailed documentation of each such legacy function on this
page. page.
## Architectures {#architectures} ## Architectures {id="architectures"}
These functions are available from `@spacy.registry.architectures`. These functions are available from `@spacy.registry.architectures`.
### spacy.Tok2Vec.v1 {#Tok2Vec_v1} ### spacy.Tok2Vec.v1 {id="Tok2Vec_v1"}
The `spacy.Tok2Vec.v1` architecture was expecting an `encode` model of type The `spacy.Tok2Vec.v1` architecture was expecting an `encode` model of type
`Model[Floats2D, Floats2D]` such as `spacy.MaxoutWindowEncoder.v1` or `Model[Floats2D, Floats2D]` such as `spacy.MaxoutWindowEncoder.v1` or
@ -48,7 +48,7 @@ blog post for background.
| `encode` | Encode context into the embeddings, using an architecture such as a CNN, BiLSTM or transformer. For example, [MaxoutWindowEncoder.v1](/api/legacy#MaxoutWindowEncoder_v1). ~~Model[Floats2d, Floats2d]~~ | | `encode` | Encode context into the embeddings, using an architecture such as a CNN, BiLSTM or transformer. For example, [MaxoutWindowEncoder.v1](/api/legacy#MaxoutWindowEncoder_v1). ~~Model[Floats2d, Floats2d]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.MaxoutWindowEncoder.v1 {#MaxoutWindowEncoder_v1} ### spacy.MaxoutWindowEncoder.v1 {id="MaxoutWindowEncoder_v1"}
The `spacy.MaxoutWindowEncoder.v1` architecture was producing a model of type The `spacy.MaxoutWindowEncoder.v1` architecture was producing a model of type
`Model[Floats2D, Floats2D]`. Since `spacy.MaxoutWindowEncoder.v2`, this has been `Model[Floats2D, Floats2D]`. Since `spacy.MaxoutWindowEncoder.v2`, this has been
@ -76,7 +76,7 @@ and residual connections.
| `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ | | `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[Floats2d, Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[Floats2d, Floats2d]~~ |
### spacy.MishWindowEncoder.v1 {#MishWindowEncoder_v1} ### spacy.MishWindowEncoder.v1 {id="MishWindowEncoder_v1"}
The `spacy.MishWindowEncoder.v1` architecture was producing a model of type The `spacy.MishWindowEncoder.v1` architecture was producing a model of type
`Model[Floats2D, Floats2D]`. Since `spacy.MishWindowEncoder.v2`, this has been `Model[Floats2D, Floats2D]`. Since `spacy.MishWindowEncoder.v2`, this has been
@ -103,24 +103,24 @@ and residual connections.
| `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ | | `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[Floats2d, Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[Floats2d, Floats2d]~~ |
### spacy.HashEmbedCNN.v1 {#HashEmbedCNN_v1} ### spacy.HashEmbedCNN.v1 {id="HashEmbedCNN_v1"}
Identical to [`spacy.HashEmbedCNN.v2`](/api/architectures#HashEmbedCNN) except Identical to [`spacy.HashEmbedCNN.v2`](/api/architectures#HashEmbedCNN) except
using [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are included. using [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are included.
### spacy.MultiHashEmbed.v1 {#MultiHashEmbed_v1} ### spacy.MultiHashEmbed.v1 {id="MultiHashEmbed_v1"}
Identical to [`spacy.MultiHashEmbed.v2`](/api/architectures#MultiHashEmbed) Identical to [`spacy.MultiHashEmbed.v2`](/api/architectures#MultiHashEmbed)
except with [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are except with [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are
included. included.
### spacy.CharacterEmbed.v1 {#CharacterEmbed_v1} ### spacy.CharacterEmbed.v1 {id="CharacterEmbed_v1"}
Identical to [`spacy.CharacterEmbed.v2`](/api/architectures#CharacterEmbed) Identical to [`spacy.CharacterEmbed.v2`](/api/architectures#CharacterEmbed)
except using [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are except using [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are
included. included.
### spacy.TextCatEnsemble.v1 {#TextCatEnsemble_v1} ### spacy.TextCatEnsemble.v1 {id="TextCatEnsemble_v1"}
The `spacy.TextCatEnsemble.v1` architecture built an internal `tok2vec` and The `spacy.TextCatEnsemble.v1` architecture built an internal `tok2vec` and
`linear_model`. Since `spacy.TextCatEnsemble.v2`, this has been refactored so `linear_model`. Since `spacy.TextCatEnsemble.v2`, this has been refactored so
@ -158,7 +158,7 @@ network has an internal CNN Tok2Vec layer and uses attention.
| `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
### spacy.TextCatCNN.v1 {#TextCatCNN_v1} ### spacy.TextCatCNN.v1 {id="TextCatCNN_v1"}
Since `spacy.TextCatCNN.v2`, this architecture has become resizable, which means Since `spacy.TextCatCNN.v2`, this architecture has become resizable, which means
that you can add labels to a previously trained textcat. `TextCatCNN` v1 did not that you can add labels to a previously trained textcat. `TextCatCNN` v1 did not
@ -194,7 +194,7 @@ architecture is usually less accurate than the ensemble, but runs faster.
| `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
### spacy.TextCatBOW.v1 {#TextCatBOW_v1} ### spacy.TextCatBOW.v1 {id="TextCatBOW_v1"}
Since `spacy.TextCatBOW.v2`, this architecture has become resizable, which means Since `spacy.TextCatBOW.v2`, this architecture has become resizable, which means
that you can add labels to a previously trained textcat. `TextCatBOW` v1 did not that you can add labels to a previously trained textcat. `TextCatBOW` v1 did not
@ -222,17 +222,17 @@ the others, but may not be as accurate, especially if texts are short.
| `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
### spacy.TransitionBasedParser.v1 {#TransitionBasedParser_v1} ### spacy.TransitionBasedParser.v1 {id="TransitionBasedParser_v1"}
Identical to Identical to
[`spacy.TransitionBasedParser.v2`](/api/architectures#TransitionBasedParser) [`spacy.TransitionBasedParser.v2`](/api/architectures#TransitionBasedParser)
except the `use_upper` was set to `True` by default. except the `use_upper` was set to `True` by default.
## Layers {#layers} ## Layers {id="layers"}
These functions are available from `@spacy.registry.layers`. These functions are available from `@spacy.registry.layers`.
### spacy.StaticVectors.v1 {#StaticVectors_v1} ### spacy.StaticVectors.v1 {id="StaticVectors_v1"}
Identical to [`spacy.StaticVectors.v2`](/api/architectures#StaticVectors) except Identical to [`spacy.StaticVectors.v2`](/api/architectures#StaticVectors) except
for the handling of tokens without vectors. for the handling of tokens without vectors.
@ -246,11 +246,11 @@ added to an existing vectors table. See more details in
</Infobox> </Infobox>
## Loggers {#loggers} ## Loggers {id="loggers"}
These functions are available from `@spacy.registry.loggers`. These functions are available from `@spacy.registry.loggers`.
### spacy.ConsoleLogger.v1 {#ConsoleLogger_v1} ### spacy.ConsoleLogger.v1 {id="ConsoleLogger_v1"}
> #### Example config > #### Example config
> >
@ -264,7 +264,7 @@ Writes the results of a training step to the console in a tabular format.
<Accordion title="Example console output" spaced> <Accordion title="Example console output" spaced>
```cli ```bash
$ python -m spacy train config.cfg $ python -m spacy train config.cfg
``` ```

View File

@ -2,7 +2,7 @@
title: Lemmatizer title: Lemmatizer
tag: class tag: class
source: spacy/pipeline/lemmatizer.py source: spacy/pipeline/lemmatizer.py
new: 3 version: 3
teaser: 'Pipeline component for lemmatization' teaser: 'Pipeline component for lemmatization'
api_string_name: lemmatizer api_string_name: lemmatizer
api_trainable: false api_trainable: false
@ -32,7 +32,7 @@ available in the pipeline and runs _before_ the lemmatizer.
</Infobox> </Infobox>
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Lemmas generated by rules or predicted will be saved to `Token.lemma`. Lemmas generated by rules or predicted will be saved to `Token.lemma`.
@ -94,7 +94,7 @@ libraries (`pymorphy3`).
%%GITHUB_SPACY/spacy/pipeline/lemmatizer.py %%GITHUB_SPACY/spacy/pipeline/lemmatizer.py
``` ```
## Lemmatizer.\_\_init\_\_ {#init tag="method"} ## Lemmatizer.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -120,7 +120,7 @@ shortcut for this and instantiate the component using its string name and
| mode | The lemmatizer mode, e.g. `"lookup"` or `"rule"`. Defaults to `"lookup"`. ~~str~~ | | mode | The lemmatizer mode, e.g. `"lookup"` or `"rule"`. Defaults to `"lookup"`. ~~str~~ |
| overwrite | Whether to overwrite existing lemmas. ~~bool~~ | | overwrite | Whether to overwrite existing lemmas. ~~bool~~ |
## Lemmatizer.\_\_call\_\_ {#call tag="method"} ## Lemmatizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned. Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -140,7 +140,7 @@ and all pipeline components are applied to the `Doc` in order.
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## Lemmatizer.pipe {#pipe tag="method"} ## Lemmatizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -161,7 +161,7 @@ applied to the `Doc` in order.
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## Lemmatizer.initialize {#initialize tag="method"} ## Lemmatizer.initialize {id="initialize",tag="method"}
Initialize the lemmatizer and load any data resources. This method is typically Initialize the lemmatizer and load any data resources. This method is typically
called by [`Language.initialize`](/api/language#initialize) and lets you called by [`Language.initialize`](/api/language#initialize) and lets you
@ -192,7 +192,7 @@ training. At runtime, all data is loaded from disk.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `lookups` | The lookups object containing the tables such as `"lemma_rules"`, `"lemma_index"`, `"lemma_exc"` and `"lemma_lookup"`. If `None`, default tables are loaded from [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data). Defaults to `None`. ~~Optional[Lookups]~~ | | `lookups` | The lookups object containing the tables such as `"lemma_rules"`, `"lemma_index"`, `"lemma_exc"` and `"lemma_lookup"`. If `None`, default tables are loaded from [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data). Defaults to `None`. ~~Optional[Lookups]~~ |
## Lemmatizer.lookup_lemmatize {#lookup_lemmatize tag="method"} ## Lemmatizer.lookup_lemmatize {id="lookup_lemmatize",tag="method"}
Lemmatize a token using a lookup-based approach. If no lemma is found, the Lemmatize a token using a lookup-based approach. If no lemma is found, the
original string is returned. original string is returned.
@ -202,7 +202,7 @@ original string is returned.
| `token` | The token to lemmatize. ~~Token~~ | | `token` | The token to lemmatize. ~~Token~~ |
| **RETURNS** | A list containing one or more lemmas. ~~List[str]~~ | | **RETURNS** | A list containing one or more lemmas. ~~List[str]~~ |
## Lemmatizer.rule_lemmatize {#rule_lemmatize tag="method"} ## Lemmatizer.rule_lemmatize {id="rule_lemmatize",tag="method"}
Lemmatize a token using a rule-based approach. Typically relies on POS tags. Lemmatize a token using a rule-based approach. Typically relies on POS tags.
@ -211,7 +211,7 @@ Lemmatize a token using a rule-based approach. Typically relies on POS tags.
| `token` | The token to lemmatize. ~~Token~~ | | `token` | The token to lemmatize. ~~Token~~ |
| **RETURNS** | A list containing one or more lemmas. ~~List[str]~~ | | **RETURNS** | A list containing one or more lemmas. ~~List[str]~~ |
## Lemmatizer.is_base_form {#is_base_form tag="method"} ## Lemmatizer.is_base_form {id="is_base_form",tag="method"}
Check whether we're dealing with an uninflected paradigm, so we can avoid Check whether we're dealing with an uninflected paradigm, so we can avoid
lemmatization entirely. lemmatization entirely.
@ -221,7 +221,7 @@ lemmatization entirely.
| `token` | The token to analyze. ~~Token~~ | | `token` | The token to analyze. ~~Token~~ |
| **RETURNS** | Whether the token's attributes (e.g., part-of-speech tag, morphological features) describe a base form. ~~bool~~ | | **RETURNS** | Whether the token's attributes (e.g., part-of-speech tag, morphological features) describe a base form. ~~bool~~ |
## Lemmatizer.get_lookups_config {#get_lookups_config tag="classmethod"} ## Lemmatizer.get_lookups_config {id="get_lookups_config",tag="classmethod"}
Returns the lookups configuration settings for a given mode for use in Returns the lookups configuration settings for a given mode for use in
[`Lemmatizer.load_lookups`](/api/lemmatizer#load_lookups). [`Lemmatizer.load_lookups`](/api/lemmatizer#load_lookups).
@ -231,7 +231,7 @@ Returns the lookups configuration settings for a given mode for use in
| `mode` | The lemmatizer mode. ~~str~~ | | `mode` | The lemmatizer mode. ~~str~~ |
| **RETURNS** | The required table names and the optional table names. ~~Tuple[List[str], List[str]]~~ | | **RETURNS** | The required table names and the optional table names. ~~Tuple[List[str], List[str]]~~ |
## Lemmatizer.to_disk {#to_disk tag="method"} ## Lemmatizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -248,7 +248,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Lemmatizer.from_disk {#from_disk tag="method"} ## Lemmatizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -266,7 +266,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Lemmatizer` object. ~~Lemmatizer~~ | | **RETURNS** | The modified `Lemmatizer` object. ~~Lemmatizer~~ |
## Lemmatizer.to_bytes {#to_bytes tag="method"} ## Lemmatizer.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -283,7 +283,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Lemmatizer` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `Lemmatizer` object. ~~bytes~~ |
## Lemmatizer.from_bytes {#from_bytes tag="method"} ## Lemmatizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -302,7 +302,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Lemmatizer` object. ~~Lemmatizer~~ | | **RETURNS** | The `Lemmatizer` object. ~~Lemmatizer~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| --------- | ------------------------------------------- | | --------- | ------------------------------------------- |
@ -310,7 +310,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `lookups` | The lookups object. ~~Lookups~~ | | `lookups` | The lookups object. ~~Lookups~~ |
| `mode` | The lemmatizer mode. ~~str~~ | | `mode` | The lemmatizer mode. ~~str~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -9,7 +9,7 @@ A `Lexeme` has no string context it's a word type, as opposed to a word toke
It therefore has no part-of-speech tag, dependency parse, or lemma (if It therefore has no part-of-speech tag, dependency parse, or lemma (if
lemmatization depends on the part-of-speech tag). lemmatization depends on the part-of-speech tag).
## Lexeme.\_\_init\_\_ {#init tag="method"} ## Lexeme.\_\_init\_\_ {id="init",tag="method"}
Create a `Lexeme` object. Create a `Lexeme` object.
@ -18,7 +18,7 @@ Create a `Lexeme` object.
| `vocab` | The parent vocabulary. ~~Vocab~~ | | `vocab` | The parent vocabulary. ~~Vocab~~ |
| `orth` | The orth id of the lexeme. ~~int~~ | | `orth` | The orth id of the lexeme. ~~int~~ |
## Lexeme.set_flag {#set_flag tag="method"} ## Lexeme.set_flag {id="set_flag",tag="method"}
Change the value of a boolean flag. Change the value of a boolean flag.
@ -34,7 +34,7 @@ Change the value of a boolean flag.
| `flag_id` | The attribute ID of the flag to set. ~~int~~ | | `flag_id` | The attribute ID of the flag to set. ~~int~~ |
| `value` | The new value of the flag. ~~bool~~ | | `value` | The new value of the flag. ~~bool~~ |
## Lexeme.check_flag {#check_flag tag="method"} ## Lexeme.check_flag {id="check_flag",tag="method"}
Check the value of a boolean flag. Check the value of a boolean flag.
@ -51,7 +51,7 @@ Check the value of a boolean flag.
| `flag_id` | The attribute ID of the flag to query. ~~int~~ | | `flag_id` | The attribute ID of the flag to query. ~~int~~ |
| **RETURNS** | The value of the flag. ~~bool~~ | | **RETURNS** | The value of the flag. ~~bool~~ |
## Lexeme.similarity {#similarity tag="method" model="vectors"} ## Lexeme.similarity {id="similarity",tag="method",model="vectors"}
Compute a semantic similarity estimate. Defaults to cosine over vectors. Compute a semantic similarity estimate. Defaults to cosine over vectors.
@ -70,7 +70,7 @@ Compute a semantic similarity estimate. Defaults to cosine over vectors.
| other | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ | | other | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ |
| **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ | | **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ |
## Lexeme.has_vector {#has_vector tag="property" model="vectors"} ## Lexeme.has_vector {id="has_vector",tag="property",model="vectors"}
A boolean value indicating whether a word vector is associated with the lexeme. A boolean value indicating whether a word vector is associated with the lexeme.
@ -85,7 +85,7 @@ A boolean value indicating whether a word vector is associated with the lexeme.
| ----------- | ------------------------------------------------------- | | ----------- | ------------------------------------------------------- |
| **RETURNS** | Whether the lexeme has a vector data attached. ~~bool~~ | | **RETURNS** | Whether the lexeme has a vector data attached. ~~bool~~ |
## Lexeme.vector {#vector tag="property" model="vectors"} ## Lexeme.vector {id="vector",tag="property",model="vectors"}
A real-valued meaning representation. A real-valued meaning representation.
@ -101,7 +101,7 @@ A real-valued meaning representation.
| ----------- | ------------------------------------------------------------------------------------------------ | | ----------- | ------------------------------------------------------------------------------------------------ |
| **RETURNS** | A 1-dimensional array representing the lexeme's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | | **RETURNS** | A 1-dimensional array representing the lexeme's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Lexeme.vector_norm {#vector_norm tag="property" model="vectors"} ## Lexeme.vector_norm {id="vector_norm",tag="property",model="vectors"}
The L2 norm of the lexeme's vector representation. The L2 norm of the lexeme's vector representation.
@ -119,7 +119,7 @@ The L2 norm of the lexeme's vector representation.
| ----------- | --------------------------------------------------- | | ----------- | --------------------------------------------------- |
| **RETURNS** | The L2 norm of the vector representation. ~~float~~ | | **RETURNS** | The L2 norm of the vector representation. ~~float~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

View File

@ -3,7 +3,7 @@ title: Lookups
teaser: A container for large lookup tables and dictionaries teaser: A container for large lookup tables and dictionaries
tag: class tag: class
source: spacy/lookups.py source: spacy/lookups.py
new: 2.2 version: 2.2
--- ---
This class allows convenient access to large lookup tables and dictionaries, This class allows convenient access to large lookup tables and dictionaries,
@ -13,7 +13,7 @@ can be accessed before the pipeline components are applied (e.g. in the
tokenizer and lemmatizer), as well as within the pipeline components via tokenizer and lemmatizer), as well as within the pipeline components via
`doc.vocab.lookups`. `doc.vocab.lookups`.
## Lookups.\_\_init\_\_ {#init tag="method"} ## Lookups.\_\_init\_\_ {id="init",tag="method"}
Create a `Lookups` object. Create a `Lookups` object.
@ -24,7 +24,7 @@ Create a `Lookups` object.
> lookups = Lookups() > lookups = Lookups()
> ``` > ```
## Lookups.\_\_len\_\_ {#len tag="method"} ## Lookups.\_\_len\_\_ {id="len",tag="method"}
Get the current number of tables in the lookups. Get the current number of tables in the lookups.
@ -39,7 +39,7 @@ Get the current number of tables in the lookups.
| ----------- | -------------------------------------------- | | ----------- | -------------------------------------------- |
| **RETURNS** | The number of tables in the lookups. ~~int~~ | | **RETURNS** | The number of tables in the lookups. ~~int~~ |
## Lookups.\_\contains\_\_ {#contains tag="method"} ## Lookups.\_\_contains\_\_ {id="contains",tag="method"}
Check if the lookups contain a table of a given name. Delegates to Check if the lookups contain a table of a given name. Delegates to
[`Lookups.has_table`](/api/lookups#has_table). [`Lookups.has_table`](/api/lookups#has_table).
@ -57,7 +57,7 @@ Check if the lookups contain a table of a given name. Delegates to
| `name` | Name of the table. ~~str~~ | | `name` | Name of the table. ~~str~~ |
| **RETURNS** | Whether a table of that name is in the lookups. ~~bool~~ | | **RETURNS** | Whether a table of that name is in the lookups. ~~bool~~ |
## Lookups.tables {#tables tag="property"} ## Lookups.tables {id="tables",tag="property"}
Get the names of all tables in the lookups. Get the names of all tables in the lookups.
@ -73,7 +73,7 @@ Get the names of all tables in the lookups.
| ----------- | ------------------------------------------------- | | ----------- | ------------------------------------------------- |
| **RETURNS** | Names of the tables in the lookups. ~~List[str]~~ | | **RETURNS** | Names of the tables in the lookups. ~~List[str]~~ |
## Lookups.add_table {#add_table tag="method"} ## Lookups.add_table {id="add_table",tag="method"}
Add a new table with optional data to the lookups. Raises an error if the table Add a new table with optional data to the lookups. Raises an error if the table
exists. exists.
@ -91,7 +91,7 @@ exists.
| `data` | Optional data to add to the table. ~~dict~~ | | `data` | Optional data to add to the table. ~~dict~~ |
| **RETURNS** | The newly added table. ~~Table~~ | | **RETURNS** | The newly added table. ~~Table~~ |
## Lookups.get_table {#get_table tag="method"} ## Lookups.get_table {id="get_table",tag="method"}
Get a table from the lookups. Raises an error if the table doesn't exist. Get a table from the lookups. Raises an error if the table doesn't exist.
@ -109,7 +109,7 @@ Get a table from the lookups. Raises an error if the table doesn't exist.
| `name` | Name of the table. ~~str~~ | | `name` | Name of the table. ~~str~~ |
| **RETURNS** | The table. ~~Table~~ | | **RETURNS** | The table. ~~Table~~ |
## Lookups.remove_table {#remove_table tag="method"} ## Lookups.remove_table {id="remove_table",tag="method"}
Remove a table from the lookups. Raises an error if the table doesn't exist. Remove a table from the lookups. Raises an error if the table doesn't exist.
@ -127,7 +127,7 @@ Remove a table from the lookups. Raises an error if the table doesn't exist.
| `name` | Name of the table to remove. ~~str~~ | | `name` | Name of the table to remove. ~~str~~ |
| **RETURNS** | The removed table. ~~Table~~ | | **RETURNS** | The removed table. ~~Table~~ |
## Lookups.has_table {#has_table tag="method"} ## Lookups.has_table {id="has_table",tag="method"}
Check if the lookups contain a table of a given name. Equivalent to Check if the lookups contain a table of a given name. Equivalent to
[`Lookups.__contains__`](/api/lookups#contains). [`Lookups.__contains__`](/api/lookups#contains).
@ -145,7 +145,7 @@ Check if the lookups contain a table of a given name. Equivalent to
| `name` | Name of the table. ~~str~~ | | `name` | Name of the table. ~~str~~ |
| **RETURNS** | Whether a table of that name is in the lookups. ~~bool~~ | | **RETURNS** | Whether a table of that name is in the lookups. ~~bool~~ |
## Lookups.to_bytes {#to_bytes tag="method"} ## Lookups.to_bytes {id="to_bytes",tag="method"}
Serialize the lookups to a bytestring. Serialize the lookups to a bytestring.
@ -159,7 +159,7 @@ Serialize the lookups to a bytestring.
| ----------- | --------------------------------- | | ----------- | --------------------------------- |
| **RETURNS** | The serialized lookups. ~~bytes~~ | | **RETURNS** | The serialized lookups. ~~bytes~~ |
## Lookups.from_bytes {#from_bytes tag="method"} ## Lookups.from_bytes {id="from_bytes",tag="method"}
Load the lookups from a bytestring. Load the lookups from a bytestring.
@ -176,7 +176,7 @@ Load the lookups from a bytestring.
| `bytes_data` | The data to load from. ~~bytes~~ | | `bytes_data` | The data to load from. ~~bytes~~ |
| **RETURNS** | The loaded lookups. ~~Lookups~~ | | **RETURNS** | The loaded lookups. ~~Lookups~~ |
## Lookups.to_disk {#to_disk tag="method"} ## Lookups.to_disk {id="to_disk",tag="method"}
Save the lookups to a directory as `lookups.bin`. Expects a path to a directory, Save the lookups to a directory as `lookups.bin`. Expects a path to a directory,
which will be created if it doesn't exist. which will be created if it doesn't exist.
@ -191,7 +191,7 @@ which will be created if it doesn't exist.
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## Lookups.from_disk {#from_disk tag="method"} ## Lookups.from_disk {id="from_disk",tag="method"}
Load lookups from a directory containing a `lookups.bin`. Will skip loading if Load lookups from a directory containing a `lookups.bin`. Will skip loading if
the file doesn't exist. the file doesn't exist.
@ -209,7 +209,7 @@ the file doesn't exist.
| `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The loaded lookups. ~~Lookups~~ | | **RETURNS** | The loaded lookups. ~~Lookups~~ |
## Table {#table tag="class, ordererddict"} ## Table {id="table",tag="class, ordererddict"}
A table in the lookups. Subclass of `OrderedDict` that implements a slightly A table in the lookups. Subclass of `OrderedDict` that implements a slightly
more consistent and unified API and includes a Bloom filter to speed up missed more consistent and unified API and includes a Bloom filter to speed up missed
@ -218,7 +218,7 @@ lookups. Supports **all other methods and attributes** of `OrderedDict` /
accept both integers and strings (which will be hashed before being added to the accept both integers and strings (which will be hashed before being added to the
table). table).
### Table.\_\_init\_\_ {#table.init tag="method"} ### Table.\_\_init\_\_ {id="table.init",tag="method"}
Initialize a new table. Initialize a new table.
@ -236,7 +236,7 @@ Initialize a new table.
| ------ | ------------------------------------------ | | ------ | ------------------------------------------ |
| `name` | Optional table name for reference. ~~str~~ | | `name` | Optional table name for reference. ~~str~~ |
### Table.from_dict {#table.from_dict tag="classmethod"} ### Table.from_dict {id="table.from_dict",tag="classmethod"}
Initialize a new table from a dict. Initialize a new table from a dict.
@ -254,7 +254,7 @@ Initialize a new table from a dict.
| `name` | Optional table name for reference. ~~str~~ | | `name` | Optional table name for reference. ~~str~~ |
| **RETURNS** | The newly constructed object. ~~Table~~ | | **RETURNS** | The newly constructed object. ~~Table~~ |
### Table.set {#table.set tag="method"} ### Table.set {id="table.set",tag="method"}
Set a new key / value pair. String keys will be hashed. Same as Set a new key / value pair. String keys will be hashed. Same as
`table[key] = value`. `table[key] = value`.
@ -273,7 +273,7 @@ Set a new key / value pair. String keys will be hashed. Same as
| `key` | The key. ~~Union[str, int]~~ | | `key` | The key. ~~Union[str, int]~~ |
| `value` | The value. | | `value` | The value. |
### Table.to_bytes {#table.to_bytes tag="method"} ### Table.to_bytes {id="table.to_bytes",tag="method"}
Serialize the table to a bytestring. Serialize the table to a bytestring.
@ -287,7 +287,7 @@ Serialize the table to a bytestring.
| ----------- | ------------------------------- | | ----------- | ------------------------------- |
| **RETURNS** | The serialized table. ~~bytes~~ | | **RETURNS** | The serialized table. ~~bytes~~ |
### Table.from_bytes {#table.from_bytes tag="method"} ### Table.from_bytes {id="table.from_bytes",tag="method"}
Load a table from a bytestring. Load a table from a bytestring.
@ -304,7 +304,7 @@ Load a table from a bytestring.
| `bytes_data` | The data to load. ~~bytes~~ | | `bytes_data` | The data to load. ~~bytes~~ |
| **RETURNS** | The loaded table. ~~Table~~ | | **RETURNS** | The loaded table. ~~Table~~ |
### Attributes {#table-attributes} ### Attributes {id="table-attributes"}
| Name | Description | | Name | Description |
| -------------- | ------------------------------------------------------------- | | -------------- | ------------------------------------------------------------- |

View File

@ -13,7 +13,7 @@ tokens in context. For in-depth examples and workflows for combining rules and
statistical models, see the [usage guide](/usage/rule-based-matching) on statistical models, see the [usage guide](/usage/rule-based-matching) on
rule-based matching. rule-based matching.
## Pattern format {#patterns} ## Pattern format {id="patterns"}
> ```json > ```json
> ### Example > ### Example
@ -101,7 +101,7 @@ it compares to another value.
As of spaCy v3.5, `REGEX` and `FUZZY` can be used in combination with `IN` and As of spaCy v3.5, `REGEX` and `FUZZY` can be used in combination with `IN` and
`NOT_IN`. `NOT_IN`.
## Matcher.\_\_init\_\_ {#init tag="method"} ## Matcher.\_\_init\_\_ {id="init",tag="method"}
Create the rule-based `Matcher`. If `validate=True` is set, all patterns added Create the rule-based `Matcher`. If `validate=True` is set, all patterns added
to the matcher will be validated against a JSON schema and a `MatchPatternError` to the matcher will be validated against a JSON schema and a `MatchPatternError`
@ -121,7 +121,7 @@ string where an integer is expected) or unexpected property names.
| `validate` | Validate all patterns added to this matcher. ~~bool~~ | | `validate` | Validate all patterns added to this matcher. ~~bool~~ |
| `fuzzy_compare` | The comparison method used for the `FUZZY` operators. ~~Callable[[str, str, int], bool]~~ | | `fuzzy_compare` | The comparison method used for the `FUZZY` operators. ~~Callable[[str, str, int], bool]~~ |
## Matcher.\_\_call\_\_ {#call tag="method"} ## Matcher.\_\_call\_\_ {id="call",tag="method"}
Find all token sequences matching the supplied patterns on the `Doc` or `Span`. Find all token sequences matching the supplied patterns on the `Doc` or `Span`.
@ -150,7 +150,7 @@ the match.
| `with_alignments` <Tag variant="new">3.0.6</Tag> | Return match alignment information as part of the match tuple as `List[int]` with the same length as the matched span. Each entry denotes the corresponding index of the token in the pattern. If `as_spans` is set to `True`, this setting is ignored. Defaults to `False`. ~~bool~~ | | `with_alignments` <Tag variant="new">3.0.6</Tag> | Return match alignment information as part of the match tuple as `List[int]` with the same length as the matched span. Each entry denotes the corresponding index of the token in the pattern. If `as_spans` is set to `True`, this setting is ignored. Defaults to `False`. ~~bool~~ |
| **RETURNS** | A list of `(match_id, start, end)` tuples, describing the matches. A match tuple describes a span `doc[start:end`]. The `match_id` is the ID of the added match pattern. If `as_spans` is set to `True`, a list of `Span` objects is returned instead. ~~Union[List[Tuple[int, int, int]], List[Span]]~~ | | **RETURNS** | A list of `(match_id, start, end)` tuples, describing the matches. A match tuple describes a span `doc[start:end`]. The `match_id` is the ID of the added match pattern. If `as_spans` is set to `True`, a list of `Span` objects is returned instead. ~~Union[List[Tuple[int, int, int]], List[Span]]~~ |
## Matcher.\_\_len\_\_ {#len tag="method" new="2"} ## Matcher.\_\_len\_\_ {id="len",tag="method",version="2"}
Get the number of rules added to the matcher. Note that this only returns the Get the number of rules added to the matcher. Note that this only returns the
number of rules (identical with the number of IDs), not the number of individual number of rules (identical with the number of IDs), not the number of individual
@ -169,7 +169,7 @@ patterns.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The number of rules. ~~int~~ | | **RETURNS** | The number of rules. ~~int~~ |
## Matcher.\_\_contains\_\_ {#contains tag="method" new="2"} ## Matcher.\_\_contains\_\_ {id="contains",tag="method",version="2"}
Check whether the matcher contains rules for a match ID. Check whether the matcher contains rules for a match ID.
@ -187,7 +187,7 @@ Check whether the matcher contains rules for a match ID.
| `key` | The match ID. ~~str~~ | | `key` | The match ID. ~~str~~ |
| **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ | | **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ |
## Matcher.add {#add tag="method" new="2"} ## Matcher.add {id="add",tag="method",version="2"}
Add a rule to the matcher, consisting of an ID key, one or more patterns, and an Add a rule to the matcher, consisting of an ID key, one or more patterns, and an
optional callback function to act on the matches. The callback function will optional callback function to act on the matches. The callback function will
@ -233,7 +233,7 @@ patterns = [[{"TEXT": "Google"}, {"TEXT": "Now"}], [{"TEXT": "GoogleNow"}]]
| `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[Matcher, Doc, int, List[tuple], Any]]~~ | | `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[Matcher, Doc, int, List[tuple], Any]]~~ |
| `greedy` <Tag variant="new">3</Tag> | Optional filter for greedy matches. Can either be `"FIRST"` or `"LONGEST"`. ~~Optional[str]~~ | | `greedy` <Tag variant="new">3</Tag> | Optional filter for greedy matches. Can either be `"FIRST"` or `"LONGEST"`. ~~Optional[str]~~ |
## Matcher.remove {#remove tag="method" new="2"} ## Matcher.remove {id="remove",tag="method",version="2"}
Remove a rule from the matcher. A `KeyError` is raised if the match ID does not Remove a rule from the matcher. A `KeyError` is raised if the match ID does not
exist. exist.
@ -251,7 +251,7 @@ exist.
| ----- | --------------------------------- | | ----- | --------------------------------- |
| `key` | The ID of the match rule. ~~str~~ | | `key` | The ID of the match rule. ~~str~~ |
## Matcher.get {#get tag="method" new="2"} ## Matcher.get {id="get",tag="method",version="2"}
Retrieve the pattern stored for a key. Returns the rule as an Retrieve the pattern stored for a key. Returns the rule as an
`(on_match, patterns)` tuple containing the callback and available patterns. `(on_match, patterns)` tuple containing the callback and available patterns.

View File

@ -2,7 +2,7 @@
title: Morphologizer title: Morphologizer
tag: class tag: class
source: spacy/pipeline/morphologizer.pyx source: spacy/pipeline/morphologizer.pyx
new: 3 version: 3
teaser: 'Pipeline component for predicting morphological features' teaser: 'Pipeline component for predicting morphological features'
api_base_class: /api/tagger api_base_class: /api/tagger
api_string_name: morphologizer api_string_name: morphologizer
@ -15,7 +15,7 @@ coarse-grained POS tags following the Universal Dependencies
[FEATS](https://universaldependencies.org/format.html#morphological-annotation) [FEATS](https://universaldependencies.org/format.html#morphological-annotation)
annotation guidelines. annotation guidelines.
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Predictions are saved to `Token.morph` and `Token.pos`. Predictions are saved to `Token.morph` and `Token.pos`.
@ -25,7 +25,7 @@ Predictions are saved to `Token.morph` and `Token.pos`.
| `Token.pos_` | The UPOS part of speech. ~~str~~ | | `Token.pos_` | The UPOS part of speech. ~~str~~ |
| `Token.morph` | Morphological features. ~~MorphAnalysis~~ | | `Token.morph` | Morphological features. ~~MorphAnalysis~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -53,7 +53,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/morphologizer.pyx %%GITHUB_SPACY/spacy/pipeline/morphologizer.pyx
``` ```
## Morphologizer.\_\_init\_\_ {#init tag="method"} ## Morphologizer.\_\_init\_\_ {id="init",tag="method"}
Create a new pipeline instance. In your application, you would normally use a Create a new pipeline instance. In your application, you would normally use a
shortcut for this and instantiate the component using its string name and shortcut for this and instantiate the component using its string name and
@ -97,7 +97,7 @@ annotation `C=E|X=Y`):
| `extend` <Tag variant="new">3.2</Tag> | Whether existing feature types (whose values may or may not be overwritten depending on `overwrite`) are preserved. Defaults to `False`. ~~bool~~ | | `extend` <Tag variant="new">3.2</Tag> | Whether existing feature types (whose values may or may not be overwritten depending on `overwrite`) are preserved. Defaults to `False`. ~~bool~~ |
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attributes `"pos"` and `"morph"` and [`Scorer.score_token_attr_per_feat`](/api/scorer#score_token_attr_per_feat) for the attribute `"morph"`. ~~Optional[Callable]~~ | | `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attributes `"pos"` and `"morph"` and [`Scorer.score_token_attr_per_feat`](/api/scorer#score_token_attr_per_feat) for the attribute `"morph"`. ~~Optional[Callable]~~ |
## Morphologizer.\_\_call\_\_ {#call tag="method"} ## Morphologizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned. Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -120,7 +120,7 @@ delegate to the [`predict`](/api/morphologizer#predict) and
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## Morphologizer.pipe {#pipe tag="method"} ## Morphologizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -144,7 +144,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/morphologizer#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## Morphologizer.initialize {#initialize tag="method"} ## Morphologizer.initialize {id="initialize",tag="method"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -181,7 +181,7 @@ config.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[dict]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[dict]~~ |
## Morphologizer.predict {#predict tag="method"} ## Morphologizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. modifying them.
@ -198,7 +198,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. | | **RETURNS** | The model's prediction for each document. |
## Morphologizer.set_annotations {#set_annotations tag="method"} ## Morphologizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -215,7 +215,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `Morphologizer.predict`. | | `scores` | The scores to set, produced by `Morphologizer.predict`. |
## Morphologizer.update {#update tag="method"} ## Morphologizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model. predictions and gold-standard annotations, and update the component's model.
@ -239,7 +239,7 @@ Delegates to [`predict`](/api/morphologizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Morphologizer.get_loss {#get_loss tag="method"} ## Morphologizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their Find the loss and gradient of loss for the batch of documents and their
predicted scores. predicted scores.
@ -258,7 +258,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. | | `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## Morphologizer.create_optimizer {#create_optimizer tag="method"} ## Morphologizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -273,7 +273,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## Morphologizer.use_params {#use_params tag="method, contextmanager"} ## Morphologizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -290,7 +290,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## Morphologizer.add_label {#add_label tag="method"} ## Morphologizer.add_label {id="add_label",tag="method"}
Add a new label to the pipe. If the `Morphologizer` should set annotations for Add a new label to the pipe. If the `Morphologizer` should set annotations for
both `pos` and `morph`, the label should include the UPOS as the feature `POS`. both `pos` and `morph`, the label should include the UPOS as the feature `POS`.
@ -313,7 +313,7 @@ will be automatically added to the model, and the output dimension will be
| `label` | The label to add. ~~str~~ | | `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## Morphologizer.to_disk {#to_disk tag="method"} ## Morphologizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -330,7 +330,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Morphologizer.from_disk {#from_disk tag="method"} ## Morphologizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -348,7 +348,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Morphologizer` object. ~~Morphologizer~~ | | **RETURNS** | The modified `Morphologizer` object. ~~Morphologizer~~ |
## Morphologizer.to_bytes {#to_bytes tag="method"} ## Morphologizer.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -365,7 +365,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Morphologizer` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `Morphologizer` object. ~~bytes~~ |
## Morphologizer.from_bytes {#from_bytes tag="method"} ## Morphologizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -384,7 +384,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Morphologizer` object. ~~Morphologizer~~ | | **RETURNS** | The `Morphologizer` object. ~~Morphologizer~~ |
## Morphologizer.labels {#labels tag="property"} ## Morphologizer.labels {id="labels",tag="property"}
The labels currently added to the component in the Universal Dependencies The labels currently added to the component in the Universal Dependencies
[FEATS](https://universaldependencies.org/format.html#morphological-annotation) [FEATS](https://universaldependencies.org/format.html#morphological-annotation)
@ -403,7 +403,7 @@ coarse-grained POS as the feature `POS`.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## Morphologizer.label_data {#label_data tag="property" new="3"} ## Morphologizer.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information. The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -421,7 +421,7 @@ model with a pre-defined label set.
| ----------- | ----------------------------------------------- | | ----------- | ----------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~dict~~ | | **RETURNS** | The label data added to the component. ~~dict~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -10,7 +10,7 @@ morphological analysis, so queries of morphological attributes are delegated to
this class. See [`MorphAnalysis`](/api/morphology#morphanalysis) for the this class. See [`MorphAnalysis`](/api/morphology#morphanalysis) for the
container storing a single morphological analysis. container storing a single morphological analysis.
## Morphology.\_\_init\_\_ {#init tag="method"} ## Morphology.\_\_init\_\_ {id="init",tag="method"}
Create a `Morphology` object. Create a `Morphology` object.
@ -26,7 +26,7 @@ Create a `Morphology` object.
| --------- | --------------------------------- | | --------- | --------------------------------- |
| `strings` | The string store. ~~StringStore~~ | | `strings` | The string store. ~~StringStore~~ |
## Morphology.add {#add tag="method"} ## Morphology.add {id="add",tag="method"}
Insert a morphological analysis in the morphology table, if not already present. Insert a morphological analysis in the morphology table, if not already present.
The morphological analysis may be provided in the Universal Dependencies The morphological analysis may be provided in the Universal Dependencies
@ -46,7 +46,7 @@ new analysis.
| ---------- | ------------------------------------------------ | | ---------- | ------------------------------------------------ |
| `features` | The morphological features. ~~Union[Dict, str]~~ | | `features` | The morphological features. ~~Union[Dict, str]~~ |
## Morphology.get {#get tag="method"} ## Morphology.get {id="get",tag="method"}
> #### Example > #### Example
> >
@ -64,7 +64,7 @@ string for the hash of the morphological analysis.
| ------- | ----------------------------------------------- | | ------- | ----------------------------------------------- |
| `morph` | The hash of the morphological analysis. ~~int~~ | | `morph` | The hash of the morphological analysis. ~~int~~ |
## Morphology.feats_to_dict {#feats_to_dict tag="staticmethod"} ## Morphology.feats_to_dict {id="feats_to_dict",tag="staticmethod"}
Convert a string Convert a string
[FEATS](https://universaldependencies.org/format.html#morphological-annotation) [FEATS](https://universaldependencies.org/format.html#morphological-annotation)
@ -84,7 +84,7 @@ tag map.
| `feats` | The morphological features in Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ | | `feats` | The morphological features in Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ |
| **RETURNS** | The morphological features as a dictionary. ~~Dict[str, str]~~ | | **RETURNS** | The morphological features as a dictionary. ~~Dict[str, str]~~ |
## Morphology.dict_to_feats {#dict_to_feats tag="staticmethod"} ## Morphology.dict_to_feats {id="dict_to_feats",tag="staticmethod"}
Convert a dictionary of features and values to a string Convert a dictionary of features and values to a string
[FEATS](https://universaldependencies.org/format.html#morphological-annotation) [FEATS](https://universaldependencies.org/format.html#morphological-annotation)
@ -103,19 +103,19 @@ representation.
| `feats_dict` | The morphological features as a dictionary. ~~Dict[str, str]~~ | | `feats_dict` | The morphological features as a dictionary. ~~Dict[str, str]~~ |
| **RETURNS** | The morphological features in Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ | | **RETURNS** | The morphological features in Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------ | | ------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `FEATURE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) feature separator. Default is `|`. ~~str~~ | | `FEATURE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) feature separator. Default is `\|`. ~~str~~ |
| `FIELD_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) field separator. Default is `=`. ~~str~~ | | `FIELD_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) field separator. Default is `=`. ~~str~~ |
| `VALUE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) value separator. Default is `,`. ~~str~~ | | `VALUE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) value separator. Default is `,`. ~~str~~ |
## MorphAnalysis {#morphanalysis tag="class" source="spacy/tokens/morphanalysis.pyx"} ## MorphAnalysis {id="morphanalysis",tag="class",source="spacy/tokens/morphanalysis.pyx"}
Stores a single morphological analysis. Stores a single morphological analysis.
### MorphAnalysis.\_\_init\_\_ {#morphanalysis-init tag="method"} ### MorphAnalysis.\_\_init\_\_ {id="morphanalysis-init",tag="method"}
Initialize a MorphAnalysis object from a Universal Dependencies Initialize a MorphAnalysis object from a Universal Dependencies
[FEATS](https://universaldependencies.org/format.html#morphological-annotation) [FEATS](https://universaldependencies.org/format.html#morphological-annotation)
@ -135,7 +135,7 @@ string or a dictionary of morphological features.
| `vocab` | The vocab. ~~Vocab~~ | | `vocab` | The vocab. ~~Vocab~~ |
| `features` | The morphological features. ~~Union[Dict[str, str], str]~~ | | `features` | The morphological features. ~~Union[Dict[str, str], str]~~ |
### MorphAnalysis.\_\_contains\_\_ {#morphanalysis-contains tag="method"} ### MorphAnalysis.\_\_contains\_\_ {id="morphanalysis-contains",tag="method"}
Whether a feature/value pair is in the analysis. Whether a feature/value pair is in the analysis.
@ -151,7 +151,7 @@ Whether a feature/value pair is in the analysis.
| ----------- | --------------------------------------------- | | ----------- | --------------------------------------------- |
| **RETURNS** | A feature/value pair in the analysis. ~~str~~ | | **RETURNS** | A feature/value pair in the analysis. ~~str~~ |
### MorphAnalysis.\_\_iter\_\_ {#morphanalysis-iter tag="method"} ### MorphAnalysis.\_\_iter\_\_ {id="morphanalysis-iter",tag="method"}
Iterate over the feature/value pairs in the analysis. Iterate over the feature/value pairs in the analysis.
@ -167,7 +167,7 @@ Iterate over the feature/value pairs in the analysis.
| ---------- | --------------------------------------------- | | ---------- | --------------------------------------------- |
| **YIELDS** | A feature/value pair in the analysis. ~~str~~ | | **YIELDS** | A feature/value pair in the analysis. ~~str~~ |
### MorphAnalysis.\_\_len\_\_ {#morphanalysis-len tag="method"} ### MorphAnalysis.\_\_len\_\_ {id="morphanalysis-len",tag="method"}
Returns the number of features in the analysis. Returns the number of features in the analysis.
@ -183,7 +183,7 @@ Returns the number of features in the analysis.
| ----------- | ----------------------------------------------- | | ----------- | ----------------------------------------------- |
| **RETURNS** | The number of features in the analysis. ~~int~~ | | **RETURNS** | The number of features in the analysis. ~~int~~ |
### MorphAnalysis.\_\_str\_\_ {#morphanalysis-str tag="method"} ### MorphAnalysis.\_\_str\_\_ {id="morphanalysis-str",tag="method"}
Returns the morphological analysis in the Universal Dependencies Returns the morphological analysis in the Universal Dependencies
[FEATS](https://universaldependencies.org/format.html#morphological-annotation) [FEATS](https://universaldependencies.org/format.html#morphological-annotation)
@ -201,7 +201,7 @@ string format.
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **RETURNS** | The analysis in the Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ | | **RETURNS** | The analysis in the Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ |
### MorphAnalysis.get {#morphanalysis-get tag="method"} ### MorphAnalysis.get {id="morphanalysis-get",tag="method"}
Retrieve values for a feature by field. Retrieve values for a feature by field.
@ -218,7 +218,7 @@ Retrieve values for a feature by field.
| `field` | The field to retrieve. ~~str~~ | | `field` | The field to retrieve. ~~str~~ |
| **RETURNS** | A list of the individual features. ~~List[str]~~ | | **RETURNS** | A list of the individual features. ~~List[str]~~ |
### MorphAnalysis.to_dict {#morphanalysis-to_dict tag="method"} ### MorphAnalysis.to_dict {id="morphanalysis-to_dict",tag="method"}
Produce a dict representation of the analysis, in the same format as the tag Produce a dict representation of the analysis, in the same format as the tag
map. map.
@ -235,7 +235,7 @@ map.
| ----------- | ----------------------------------------------------------- | | ----------- | ----------------------------------------------------------- |
| **RETURNS** | The dict representation of the analysis. ~~Dict[str, str]~~ | | **RETURNS** | The dict representation of the analysis. ~~Dict[str, str]~~ |
### MorphAnalysis.from_id {#morphanalysis-from_id tag="classmethod"} ### MorphAnalysis.from_id {id="morphanalysis-from_id",tag="classmethod"}
Create a morphological analysis from a given hash ID. Create a morphological analysis from a given hash ID.

View File

@ -3,7 +3,7 @@ title: PhraseMatcher
teaser: Match sequences of tokens, based on documents teaser: Match sequences of tokens, based on documents
tag: class tag: class
source: spacy/matcher/phrasematcher.pyx source: spacy/matcher/phrasematcher.pyx
new: 2 version: 2
--- ---
The `PhraseMatcher` lets you efficiently match large terminology lists. While The `PhraseMatcher` lets you efficiently match large terminology lists. While
@ -12,7 +12,7 @@ descriptions, the `PhraseMatcher` accepts match patterns in the form of `Doc`
objects. See the [usage guide](/usage/rule-based-matching#phrasematcher) for objects. See the [usage guide](/usage/rule-based-matching#phrasematcher) for
examples. examples.
## PhraseMatcher.\_\_init\_\_ {#init tag="method"} ## PhraseMatcher.\_\_init\_\_ {id="init",tag="method"}
Create the rule-based `PhraseMatcher`. Setting a different `attr` to match on Create the rule-based `PhraseMatcher`. Setting a different `attr` to match on
will change the token attributes that will be compared to determine a match. By will change the token attributes that will be compared to determine a match. By
@ -42,7 +42,7 @@ be shown.
| `attr` | The token attribute to match on. Defaults to `ORTH`, i.e. the verbatim token text. ~~Union[int, str]~~ | | `attr` | The token attribute to match on. Defaults to `ORTH`, i.e. the verbatim token text. ~~Union[int, str]~~ |
| `validate` | Validate patterns added to the matcher. ~~bool~~ | | `validate` | Validate patterns added to the matcher. ~~bool~~ |
## PhraseMatcher.\_\_call\_\_ {#call tag="method"} ## PhraseMatcher.\_\_call\_\_ {id="call",tag="method"}
Find all token sequences matching the supplied patterns on the `Doc` or `Span`. Find all token sequences matching the supplied patterns on the `Doc` or `Span`.
@ -76,7 +76,7 @@ match_id_string = nlp.vocab.strings[match_id]
</Infobox> </Infobox>
## PhraseMatcher.\_\_len\_\_ {#len tag="method"} ## PhraseMatcher.\_\_len\_\_ {id="len",tag="method"}
Get the number of rules added to the matcher. Note that this only returns the Get the number of rules added to the matcher. Note that this only returns the
number of rules (identical with the number of IDs), not the number of individual number of rules (identical with the number of IDs), not the number of individual
@ -95,7 +95,7 @@ patterns.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The number of rules. ~~int~~ | | **RETURNS** | The number of rules. ~~int~~ |
## PhraseMatcher.\_\_contains\_\_ {#contains tag="method"} ## PhraseMatcher.\_\_contains\_\_ {id="contains",tag="method"}
Check whether the matcher contains rules for a match ID. Check whether the matcher contains rules for a match ID.
@ -113,7 +113,7 @@ Check whether the matcher contains rules for a match ID.
| `key` | The match ID. ~~str~~ | | `key` | The match ID. ~~str~~ |
| **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ | | **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ |
## PhraseMatcher.add {#add tag="method"} ## PhraseMatcher.add {id="add",tag="method"}
Add a rule to the matcher, consisting of an ID key, one or more patterns, and a Add a rule to the matcher, consisting of an ID key, one or more patterns, and a
callback function to act on the matches. The callback function will receive the callback function to act on the matches. The callback function will receive the
@ -155,7 +155,7 @@ patterns = [nlp("health care reform"), nlp("healthcare reform")]
| _keyword-only_ | | | _keyword-only_ | |
| `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[Matcher, Doc, int, List[tuple], Any]]~~ | | `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[Matcher, Doc, int, List[tuple], Any]]~~ |
## PhraseMatcher.remove {#remove tag="method" new="2.2"} ## PhraseMatcher.remove {id="remove",tag="method",version="2.2"}
Remove a rule from the matcher by match ID. A `KeyError` is raised if the key Remove a rule from the matcher by match ID. A `KeyError` is raised if the key
does not exist. does not exist.

View File

@ -12,7 +12,7 @@ spaCy pipeline. See the docs on
[writing trainable components](/usage/processing-pipelines#trainable-components) [writing trainable components](/usage/processing-pipelines#trainable-components)
for how to use the `TrainablePipe` base class to implement custom components. for how to use the `TrainablePipe` base class to implement custom components.
<!-- TODO: Pipe vs TrainablePipe, check methods below (all renamed to TrainablePipe for now) --> {/* TODO: Pipe vs TrainablePipe, check methods below (all renamed to TrainablePipe for now) */}
> #### Why is it implemented in Cython? > #### Why is it implemented in Cython?
> >
@ -27,7 +27,7 @@ for how to use the `TrainablePipe` base class to implement custom components.
%%GITHUB_SPACY/spacy/pipeline/trainable_pipe.pyx %%GITHUB_SPACY/spacy/pipeline/trainable_pipe.pyx
``` ```
## TrainablePipe.\_\_init\_\_ {#init tag="method"} ## TrainablePipe.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -54,7 +54,7 @@ shortcut for this and instantiate the component using its string name and
| `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ | | `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ |
| `**cfg` | Additional config parameters and settings. Will be available as the dictionary `cfg` and is serialized with the component. | | `**cfg` | Additional config parameters and settings. Will be available as the dictionary `cfg` and is serialized with the component. |
## TrainablePipe.\_\_call\_\_ {#call tag="method"} ## TrainablePipe.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned. Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -77,7 +77,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## TrainablePipe.pipe {#pipe tag="method"} ## TrainablePipe.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -100,7 +100,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/pipe#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## TrainablePipe.set_error_handler {#set_error_handler tag="method" new="3"} ## TrainablePipe.set_error_handler {id="set_error_handler",tag="method",version="3"}
Define a callback that will be invoked when an error is thrown during processing Define a callback that will be invoked when an error is thrown during processing
of one or more documents with either [`__call__`](/api/pipe#call) or of one or more documents with either [`__call__`](/api/pipe#call) or
@ -122,7 +122,7 @@ processed, and the original error.
| --------------- | -------------------------------------------------------------------------------------------------------------- | | --------------- | -------------------------------------------------------------------------------------------------------------- |
| `error_handler` | A function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ | | `error_handler` | A function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ |
## TrainablePipe.get_error_handler {#get_error_handler tag="method" new="3"} ## TrainablePipe.get_error_handler {id="get_error_handler",tag="method",version="3"}
Retrieve the callback that performs error handling for this component's Retrieve the callback that performs error handling for this component's
[`__call__`](/api/pipe#call) and [`pipe`](/api/pipe#pipe) methods. If no custom [`__call__`](/api/pipe#call) and [`pipe`](/api/pipe#pipe) methods. If no custom
@ -141,7 +141,7 @@ returned that simply reraises the exception.
| ----------- | ---------------------------------------------------------------------------------------------------------------- | | ----------- | ---------------------------------------------------------------------------------------------------------------- |
| **RETURNS** | The function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ | | **RETURNS** | The function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ |
## TrainablePipe.initialize {#initialize tag="method" new="3"} ## TrainablePipe.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. The data examples are returns an iterable of [`Example`](/api/example) objects. The data examples are
@ -171,7 +171,7 @@ This method was previously called `begin_training`.
| _keyword-only_ | | | _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## TrainablePipe.predict {#predict tag="method"} ## TrainablePipe.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. modifying them.
@ -194,7 +194,7 @@ This method needs to be overwritten with your own custom `predict` method.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. | | **RETURNS** | The model's prediction for each document. |
## TrainablePipe.set_annotations {#set_annotations tag="method"} ## TrainablePipe.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -218,7 +218,7 @@ method.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `Tagger.predict`. | | `scores` | The scores to set, produced by `Tagger.predict`. |
## TrainablePipe.update {#update tag="method"} ## TrainablePipe.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model. predictions and gold-standard annotations, and update the component's model.
@ -240,7 +240,7 @@ predictions and gold-standard annotations, and update the component's model.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## TrainablePipe.rehearse {#rehearse tag="method,experimental" new="3"} ## TrainablePipe.rehearse {id="rehearse",tag="method,experimental",version="3"}
Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the
current model to make predictions similar to an initial model, to try to address current model to make predictions similar to an initial model, to try to address
@ -262,7 +262,7 @@ the "catastrophic forgetting" problem. This feature is experimental.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## TrainablePipe.get_loss {#get_loss tag="method"} ## TrainablePipe.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their Find the loss and gradient of loss for the batch of documents and their
predicted scores. predicted scores.
@ -287,7 +287,7 @@ This method needs to be overwritten with your own custom `get_loss` method.
| `scores` | Scores representing the model's predictions. | | `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## TrainablePipe.score {#score tag="method" new="3"} ## TrainablePipe.score {id="score",tag="method",version="3"}
Score a batch of examples. Score a batch of examples.
@ -304,7 +304,7 @@ Score a batch of examples.
| `\*\*kwargs` | Any additional settings to pass on to the scorer. ~~Any~~ | | `\*\*kwargs` | Any additional settings to pass on to the scorer. ~~Any~~ |
| **RETURNS** | The scores, e.g. produced by the [`Scorer`](/api/scorer). ~~Dict[str, Union[float, Dict[str, float]]]~~ | | **RETURNS** | The scores, e.g. produced by the [`Scorer`](/api/scorer). ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## TrainablePipe.create_optimizer {#create_optimizer tag="method"} ## TrainablePipe.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Defaults to Create an optimizer for the pipeline component. Defaults to
[`Adam`](https://thinc.ai/docs/api-optimizers#adam) with default settings. [`Adam`](https://thinc.ai/docs/api-optimizers#adam) with default settings.
@ -320,7 +320,7 @@ Create an optimizer for the pipeline component. Defaults to
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## TrainablePipe.use_params {#use_params tag="method, contextmanager"} ## TrainablePipe.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -337,7 +337,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## TrainablePipe.finish_update {#finish_update tag="method"} ## TrainablePipe.finish_update {id="finish_update",tag="method"}
Update parameters using the current parameter gradients. Defaults to calling Update parameters using the current parameter gradients. Defaults to calling
[`self.model.finish_update`](https://thinc.ai/docs/api-model#finish_update). [`self.model.finish_update`](https://thinc.ai/docs/api-model#finish_update).
@ -355,7 +355,7 @@ Update parameters using the current parameter gradients. Defaults to calling
| ----- | ------------------------------------- | | ----- | ------------------------------------- |
| `sgd` | An optimizer. ~~Optional[Optimizer]~~ | | `sgd` | An optimizer. ~~Optional[Optimizer]~~ |
## TrainablePipe.add_label {#add_label tag="method"} ## TrainablePipe.add_label {id="add_label",tag="method"}
> #### Example > #### Example
> >
@ -390,7 +390,7 @@ case, all labels found in the sample will be automatically added to the model,
and the output dimension will be and the output dimension will be
[inferred](/usage/layers-architectures#thinc-shape-inference) automatically. [inferred](/usage/layers-architectures#thinc-shape-inference) automatically.
## TrainablePipe.is_resizable {#is_resizable tag="property"} ## TrainablePipe.is_resizable {id="is_resizable",tag="property"}
> #### Example > #### Example
> >
@ -421,7 +421,7 @@ as an attribute to the component's model.
| ----------- | ---------------------------------------------------------------------------------------------- | | ----------- | ---------------------------------------------------------------------------------------------- |
| **RETURNS** | Whether or not the output dimension of the model can be changed after initialization. ~~bool~~ | | **RETURNS** | Whether or not the output dimension of the model can be changed after initialization. ~~bool~~ |
## TrainablePipe.set_output {#set_output tag="method"} ## TrainablePipe.set_output {id="set_output",tag="method"}
Change the output dimension of the component's model. If the component is not Change the output dimension of the component's model. If the component is not
[resizable](#is_resizable), this method will raise a `NotImplementedError`. If a [resizable](#is_resizable), this method will raise a `NotImplementedError`. If a
@ -441,7 +441,7 @@ care should be taken to avoid the "catastrophic forgetting" problem.
| ---- | --------------------------------- | | ---- | --------------------------------- |
| `nO` | The new output dimension. ~~int~~ | | `nO` | The new output dimension. ~~int~~ |
## TrainablePipe.to_disk {#to_disk tag="method"} ## TrainablePipe.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -458,7 +458,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## TrainablePipe.from_disk {#from_disk tag="method"} ## TrainablePipe.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -476,7 +476,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified pipe. ~~TrainablePipe~~ | | **RETURNS** | The modified pipe. ~~TrainablePipe~~ |
## TrainablePipe.to_bytes {#to_bytes tag="method"} ## TrainablePipe.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -493,7 +493,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the pipe. ~~bytes~~ | | **RETURNS** | The serialized form of the pipe. ~~bytes~~ |
## TrainablePipe.from_bytes {#from_bytes tag="method"} ## TrainablePipe.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -512,7 +512,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The pipe. ~~TrainablePipe~~ | | **RETURNS** | The pipe. ~~TrainablePipe~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| ------- | --------------------------------------------------------------------------------------------------------------------------------- | | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
@ -521,7 +521,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `name` | The name of the component instance in the pipeline. Can be used in the losses. ~~str~~ | | `name` | The name of the component instance in the pipeline. Can be used in the losses. ~~str~~ |
| `cfg` | Keyword arguments passed to [`TrainablePipe.__init__`](/api/pipe#init). Will be serialized with the component. ~~Dict[str, Any]~~ | | `cfg` | Keyword arguments passed to [`TrainablePipe.__init__`](/api/pipe#init). Will be serialized with the component. ~~Dict[str, Any]~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -10,7 +10,7 @@ menu:
- ['doc_cleaner', 'doc_cleaner'] - ['doc_cleaner', 'doc_cleaner']
--- ---
## merge_noun_chunks {#merge_noun_chunks tag="function"} ## merge_noun_chunks {id="merge_noun_chunks",tag="function"}
Merge noun chunks into a single token. Also available via the string name Merge noun chunks into a single token. Also available via the string name
`"merge_noun_chunks"`. `"merge_noun_chunks"`.
@ -40,7 +40,7 @@ all other components.
| `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ | | `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ |
| **RETURNS** | The modified `Doc` with merged noun chunks. ~~Doc~~ | | **RETURNS** | The modified `Doc` with merged noun chunks. ~~Doc~~ |
## merge_entities {#merge_entities tag="function"} ## merge_entities {id="merge_entities",tag="function"}
Merge named entities into a single token. Also available via the string name Merge named entities into a single token. Also available via the string name
`"merge_entities"`. `"merge_entities"`.
@ -70,7 +70,7 @@ components to the end of the pipeline and after all other components.
| `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ | | `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ |
| **RETURNS** | The modified `Doc` with merged entities. ~~Doc~~ | | **RETURNS** | The modified `Doc` with merged entities. ~~Doc~~ |
## merge_subtokens {#merge_subtokens tag="function" new="2.1"} ## merge_subtokens {id="merge_subtokens",tag="function",version="2.1"}
Merge subtokens into a single token. Also available via the string name Merge subtokens into a single token. Also available via the string name
`"merge_subtokens"`. As of v2.1, the parser is able to predict "subtokens" that `"merge_subtokens"`. As of v2.1, the parser is able to predict "subtokens" that
@ -110,7 +110,7 @@ end of the pipeline and after all other components.
| `label` | The subtoken dependency label. Defaults to `"subtok"`. ~~str~~ | | `label` | The subtoken dependency label. Defaults to `"subtok"`. ~~str~~ |
| **RETURNS** | The modified `Doc` with merged subtokens. ~~Doc~~ | | **RETURNS** | The modified `Doc` with merged subtokens. ~~Doc~~ |
## token_splitter {#token_splitter tag="function" new="3.0"} ## token_splitter {id="token_splitter",tag="function",version="3.0"}
Split tokens longer than a minimum length into shorter tokens. Intended for use Split tokens longer than a minimum length into shorter tokens. Intended for use
with transformer pipelines where long spaCy tokens lead to input text that with transformer pipelines where long spaCy tokens lead to input text that
@ -132,7 +132,7 @@ exceed the transformer model max length.
| `split_length` | The length of the split tokens. Defaults to `5`. ~~int~~ | | `split_length` | The length of the split tokens. Defaults to `5`. ~~int~~ |
| **RETURNS** | The modified `Doc` with the split tokens. ~~Doc~~ | | **RETURNS** | The modified `Doc` with the split tokens. ~~Doc~~ |
## doc_cleaner {#doc_cleaner tag="function" new="3.2.1"} ## doc_cleaner {id="doc_cleaner",tag="function",version="3.2.1"}
Clean up `Doc` attributes. Intended for use at the end of pipelines with Clean up `Doc` attributes. Intended for use at the end of pipelines with
`tok2vec` or `transformer` pipeline components that store tensors and other `tok2vec` or `transformer` pipeline components that store tensors and other
@ -154,7 +154,7 @@ whole pipeline has run.
| `silent` | If `False`, show warnings if attributes aren't found or can't be set. Defaults to `True`. ~~bool~~ | | `silent` | If `False`, show warnings if attributes aren't found or can't be set. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The modified `Doc` with the modified attributes. ~~Doc~~ | | **RETURNS** | The modified `Doc` with the modified attributes. ~~Doc~~ |
## span_cleaner {#span_cleaner tag="function,experimental"} ## span_cleaner {id="span_cleaner",tag="function,experimental"}
Remove `SpanGroup`s from `doc.spans` based on a key prefix. This is used to Remove `SpanGroup`s from `doc.spans` based on a key prefix. This is used to
clean up after the [`CoreferenceResolver`](/api/coref) when it's paired with a clean up after the [`CoreferenceResolver`](/api/coref) when it's paired with a

View File

@ -10,7 +10,7 @@ The `Scorer` computes evaluation scores. It's typically created by
provides a number of evaluation methods for evaluating [`Token`](/api/token) and provides a number of evaluation methods for evaluating [`Token`](/api/token) and
[`Doc`](/api/doc) attributes. [`Doc`](/api/doc) attributes.
## Scorer.\_\_init\_\_ {#init tag="method"} ## Scorer.\_\_init\_\_ {id="init",tag="method"}
Create a new `Scorer`. Create a new `Scorer`.
@ -35,7 +35,7 @@ Create a new `Scorer`.
| _keyword-only_ | | | _keyword-only_ | |
| `\*\*kwargs` | Any additional settings to pass on to the individual scoring methods. ~~Any~~ | | `\*\*kwargs` | Any additional settings to pass on to the individual scoring methods. ~~Any~~ |
## Scorer.score {#score tag="method"} ## Scorer.score {id="score",tag="method"}
Calculate the scores for a list of [`Example`](/api/example) objects using the Calculate the scores for a list of [`Example`](/api/example) objects using the
scoring methods provided by the components in the pipeline. scoring methods provided by the components in the pipeline.
@ -72,7 +72,7 @@ core pipeline components, the individual score names start with the `Token` or
| `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ | | `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ |
| **RETURNS** | A dictionary of scores. ~~Dict[str, Union[float, Dict[str, float]]]~~ | | **RETURNS** | A dictionary of scores. ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## Scorer.score_tokenization {#score_tokenization tag="staticmethod" new="3"} ## Scorer.score_tokenization {id="score_tokenization",tag="staticmethod",version="3"}
Scores the tokenization: Scores the tokenization:
@ -93,7 +93,7 @@ Docs with `has_unknown_spaces` are skipped during scoring.
| `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ | | `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ |
| **RETURNS** | `Dict` | A dictionary containing the scores `token_acc`, `token_p`, `token_r`, `token_f`. ~~Dict[str, float]]~~ | | **RETURNS** | `Dict` | A dictionary containing the scores `token_acc`, `token_p`, `token_r`, `token_f`. ~~Dict[str, float]]~~ |
## Scorer.score_token_attr {#score_token_attr tag="staticmethod" new="3"} ## Scorer.score_token_attr {id="score_token_attr",tag="staticmethod",version="3"}
Scores a single token attribute. Tokens with missing values in the reference doc Scores a single token attribute. Tokens with missing values in the reference doc
are skipped during scoring. are skipped during scoring.
@ -114,7 +114,7 @@ are skipped during scoring.
| `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ | | `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ |
| **RETURNS** | A dictionary containing the score `{attr}_acc`. ~~Dict[str, float]~~ | | **RETURNS** | A dictionary containing the score `{attr}_acc`. ~~Dict[str, float]~~ |
## Scorer.score_token_attr_per_feat {#score_token_attr_per_feat tag="staticmethod" new="3"} ## Scorer.score_token_attr_per_feat {id="score_token_attr_per_feat",tag="staticmethod",version="3"}
Scores a single token attribute per feature for a token attribute in the Scores a single token attribute per feature for a token attribute in the
Universal Dependencies Universal Dependencies
@ -138,7 +138,7 @@ scoring.
| `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ | | `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ |
| **RETURNS** | A dictionary containing the micro PRF scores under the key `{attr}_micro_p/r/f` and the per-feature PRF scores under `{attr}_per_feat`. ~~Dict[str, Dict[str, float]]~~ | | **RETURNS** | A dictionary containing the micro PRF scores under the key `{attr}_micro_p/r/f` and the per-feature PRF scores under `{attr}_per_feat`. ~~Dict[str, Dict[str, float]]~~ |
## Scorer.score_spans {#score_spans tag="staticmethod" new="3"} ## Scorer.score_spans {id="score_spans",tag="staticmethod",version="3"}
Returns PRF scores for labeled or unlabeled spans. Returns PRF scores for labeled or unlabeled spans.
@ -160,7 +160,7 @@ Returns PRF scores for labeled or unlabeled spans.
| `allow_overlap` | Defaults to `False`. Whether or not to allow overlapping spans. If set to `False`, the alignment will automatically resolve conflicts. ~~bool~~ | | `allow_overlap` | Defaults to `False`. Whether or not to allow overlapping spans. If set to `False`, the alignment will automatically resolve conflicts. ~~bool~~ |
| **RETURNS** | A dictionary containing the PRF scores under the keys `{attr}_p`, `{attr}_r`, `{attr}_f` and the per-type PRF scores under `{attr}_per_type`. ~~Dict[str, Union[float, Dict[str, float]]]~~ | | **RETURNS** | A dictionary containing the PRF scores under the keys `{attr}_p`, `{attr}_r`, `{attr}_f` and the per-type PRF scores under `{attr}_per_type`. ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## Scorer.score_deps {#score_deps tag="staticmethod" new="3"} ## Scorer.score_deps {id="score_deps",tag="staticmethod",version="3"}
Calculate the UAS, LAS, and LAS per type scores for dependency parses. Tokens Calculate the UAS, LAS, and LAS per type scores for dependency parses. Tokens
with missing values for the `attr` (typically `dep`) are skipped during scoring. with missing values for the `attr` (typically `dep`) are skipped during scoring.
@ -194,7 +194,7 @@ with missing values for the `attr` (typically `dep`) are skipped during scoring.
| `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ | | `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ |
| **RETURNS** | A dictionary containing the scores: `{attr}_uas`, `{attr}_las`, and `{attr}_las_per_type`. ~~Dict[str, Union[float, Dict[str, float]]]~~ | | **RETURNS** | A dictionary containing the scores: `{attr}_uas`, `{attr}_las`, and `{attr}_las_per_type`. ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## Scorer.score_cats {#score_cats tag="staticmethod" new="3"} ## Scorer.score_cats {id="score_cats",tag="staticmethod",version="3"}
Calculate PRF and ROC AUC scores for a doc-level attribute that is a dict Calculate PRF and ROC AUC scores for a doc-level attribute that is a dict
containing scores for each label like `Doc.cats`. The returned dictionary containing scores for each label like `Doc.cats`. The returned dictionary
@ -241,7 +241,7 @@ The reported `{attr}_score` depends on the classification properties:
| `threshold` | Cutoff to consider a prediction "positive". Defaults to `0.5` for multi-label, and `0.0` (i.e. whatever's highest scoring) otherwise. ~~float~~ | | `threshold` | Cutoff to consider a prediction "positive". Defaults to `0.5` for multi-label, and `0.0` (i.e. whatever's highest scoring) otherwise. ~~float~~ |
| **RETURNS** | A dictionary containing the scores, with inapplicable scores as `None`. ~~Dict[str, Optional[float]]~~ | | **RETURNS** | A dictionary containing the scores, with inapplicable scores as `None`. ~~Dict[str, Optional[float]]~~ |
## Scorer.score_links {#score_links tag="staticmethod" new="3"} ## Scorer.score_links {id="score_links",tag="staticmethod",version="3"}
Returns PRF for predicted links on the entity level. To disentangle the Returns PRF for predicted links on the entity level. To disentangle the
performance of the NEL from the NER, this method only evaluates NEL links for performance of the NEL from the NER, this method only evaluates NEL links for
@ -264,7 +264,7 @@ entities that overlap between the gold reference and the predictions.
| `negative_labels` | The string values that refer to no annotation (e.g. "NIL"). ~~Iterable[str]~~ | | `negative_labels` | The string values that refer to no annotation (e.g. "NIL"). ~~Iterable[str]~~ |
| **RETURNS** | A dictionary containing the scores. ~~Dict[str, Optional[float]]~~ | | **RETURNS** | A dictionary containing the scores. ~~Dict[str, Optional[float]]~~ |
## get_ner_prf {#get_ner_prf new="3"} ## get_ner_prf {id="get_ner_prf",version="3"}
Compute micro-PRF and per-entity PRF scores. Compute micro-PRF and per-entity PRF scores.
@ -272,7 +272,7 @@ Compute micro-PRF and per-entity PRF scores.
| ---------- | ------------------------------------------------------------------------------------------------------------------- | | ---------- | ------------------------------------------------------------------------------------------------------------------- |
| `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ | | `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ |
## score_coref_clusters {#score_coref_clusters tag="experimental"} ## score_coref_clusters {id="score_coref_clusters",tag="experimental"}
Returns LEA ([Moosavi and Strube, 2016](https://aclanthology.org/P16-1060/)) PRF Returns LEA ([Moosavi and Strube, 2016](https://aclanthology.org/P16-1060/)) PRF
scores for coreference clusters. scores for coreference clusters.
@ -301,7 +301,7 @@ the [CoreferenceResolver](/api/coref) docs.
| `span_cluster_prefix` | The prefix used for spans representing coreference clusters. ~~str~~ | | `span_cluster_prefix` | The prefix used for spans representing coreference clusters. ~~str~~ |
| **RETURNS** | A dictionary containing the scores. ~~Dict[str, Optional[float]]~~ | | **RETURNS** | A dictionary containing the scores. ~~Dict[str, Optional[float]]~~ |
## score_span_predictions {#score_span_predictions tag="experimental"} ## score_span_predictions {id="score_span_predictions",tag="experimental"}
Return accuracy for reconstructions of spans from single tokens. Only exactly Return accuracy for reconstructions of spans from single tokens. Only exactly
correct predictions are counted as correct, there is no partial credit for near correct predictions are counted as correct, there is no partial credit for near

View File

@ -2,7 +2,7 @@
title: SentenceRecognizer title: SentenceRecognizer
tag: class tag: class
source: spacy/pipeline/senter.pyx source: spacy/pipeline/senter.pyx
new: 3 version: 3
teaser: 'Pipeline component for sentence segmentation' teaser: 'Pipeline component for sentence segmentation'
api_base_class: /api/tagger api_base_class: /api/tagger
api_string_name: senter api_string_name: senter
@ -12,7 +12,7 @@ api_trainable: true
A trainable pipeline component for sentence segmentation. For a simpler, A trainable pipeline component for sentence segmentation. For a simpler,
rule-based strategy, see the [`Sentencizer`](/api/sentencizer). rule-based strategy, see the [`Sentencizer`](/api/sentencizer).
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Predicted values will be assigned to `Token.is_sent_start`. The resulting Predicted values will be assigned to `Token.is_sent_start`. The resulting
sentences can be accessed using `Doc.sents`. sentences can be accessed using `Doc.sents`.
@ -22,7 +22,7 @@ sentences can be accessed using `Doc.sents`.
| `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. This will be either `True` or `False` for all tokens. ~~bool~~ | | `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. This will be either `True` or `False` for all tokens. ~~bool~~ |
| `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ | | `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -49,7 +49,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/senter.pyx %%GITHUB_SPACY/spacy/pipeline/senter.pyx
``` ```
## SentenceRecognizer.\_\_init\_\_ {#init tag="method"} ## SentenceRecognizer.\_\_init\_\_ {id="init",tag="method"}
Initialize the sentence recognizer. Initialize the sentence recognizer.
@ -81,7 +81,7 @@ shortcut for this and instantiate the component using its string name and
| `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ | | `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ |
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"`. ~~Optional[Callable]~~ | | `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"`. ~~Optional[Callable]~~ |
## SentenceRecognizer.\_\_call\_\_ {#call tag="method"} ## SentenceRecognizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned. Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -105,7 +105,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## SentenceRecognizer.pipe {#pipe tag="method"} ## SentenceRecognizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -129,7 +129,7 @@ and [`pipe`](/api/sentencerecognizer#pipe) delegate to the
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## SentenceRecognizer.initialize {#initialize tag="method"} ## SentenceRecognizer.initialize {id="initialize",tag="method"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -153,7 +153,7 @@ by [`Language.initialize`](/api/language#initialize).
| _keyword-only_ | | | _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## SentenceRecognizer.predict {#predict tag="method"} ## SentenceRecognizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. modifying them.
@ -170,7 +170,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. | | **RETURNS** | The model's prediction for each document. |
## SentenceRecognizer.set_annotations {#set_annotations tag="method"} ## SentenceRecognizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -187,7 +187,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `SentenceRecognizer.predict`. | | `scores` | The scores to set, produced by `SentenceRecognizer.predict`. |
## SentenceRecognizer.update {#update tag="method"} ## SentenceRecognizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model. predictions and gold-standard annotations, and update the component's model.
@ -211,7 +211,7 @@ Delegates to [`predict`](/api/sentencerecognizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## SentenceRecognizer.rehearse {#rehearse tag="method,experimental" new="3"} ## SentenceRecognizer.rehearse {id="rehearse",tag="method,experimental",version="3"}
Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the
current model to make predictions similar to an initial model to try to address current model to make predictions similar to an initial model to try to address
@ -234,7 +234,7 @@ the "catastrophic forgetting" problem. This feature is experimental.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## SentenceRecognizer.get_loss {#get_loss tag="method"} ## SentenceRecognizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their Find the loss and gradient of loss for the batch of documents and their
predicted scores. predicted scores.
@ -253,7 +253,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. | | `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## SentenceRecognizer.create_optimizer {#create_optimizer tag="method"} ## SentenceRecognizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -268,7 +268,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## SentenceRecognizer.use_params {#use_params tag="method, contextmanager"} ## SentenceRecognizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -285,7 +285,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## SentenceRecognizer.to_disk {#to_disk tag="method"} ## SentenceRecognizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -302,7 +302,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## SentenceRecognizer.from_disk {#from_disk tag="method"} ## SentenceRecognizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -320,7 +320,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `SentenceRecognizer` object. ~~SentenceRecognizer~~ | | **RETURNS** | The modified `SentenceRecognizer` object. ~~SentenceRecognizer~~ |
## SentenceRecognizer.to_bytes {#to_bytes tag="method"} ## SentenceRecognizer.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -337,7 +337,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `SentenceRecognizer` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `SentenceRecognizer` object. ~~bytes~~ |
## SentenceRecognizer.from_bytes {#from_bytes tag="method"} ## SentenceRecognizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -356,7 +356,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `SentenceRecognizer` object. ~~SentenceRecognizer~~ | | **RETURNS** | The `SentenceRecognizer` object. ~~SentenceRecognizer~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -13,7 +13,7 @@ performed by the [`DependencyParser`](/api/dependencyparser), so the
`Sentencizer` lets you implement a simpler, rule-based strategy that doesn't `Sentencizer` lets you implement a simpler, rule-based strategy that doesn't
require a statistical model to be loaded. require a statistical model to be loaded.
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Calculated values will be assigned to `Token.is_sent_start`. The resulting Calculated values will be assigned to `Token.is_sent_start`. The resulting
sentences can be accessed using `Doc.sents`. sentences can be accessed using `Doc.sents`.
@ -23,7 +23,7 @@ sentences can be accessed using `Doc.sents`.
| `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. This will be either `True` or `False` for all tokens. ~~bool~~ | | `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. This will be either `True` or `False` for all tokens. ~~bool~~ |
| `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ | | `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -39,7 +39,7 @@ how the component should be configured. You can override its settings via the
| Setting | Description | | Setting | Description |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `punct_chars` | Optional custom list of punctuation characters that mark sentence ends. See below for defaults if not set. Defaults to `None`. ~~Optional[List[str]]~~ | `None` | | `punct_chars` | Optional custom list of punctuation characters that mark sentence ends. See below for defaults if not set. Defaults to `None`. ~~Optional[List[str]]~~ |
| `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ | | `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ |
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"` ~~Optional[Callable]~~ | | `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"` ~~Optional[Callable]~~ |
@ -47,7 +47,7 @@ how the component should be configured. You can override its settings via the
%%GITHUB_SPACY/spacy/pipeline/sentencizer.pyx %%GITHUB_SPACY/spacy/pipeline/sentencizer.pyx
``` ```
## Sentencizer.\_\_init\_\_ {#init tag="method"} ## Sentencizer.\_\_init\_\_ {id="init",tag="method"}
Initialize the sentencizer. Initialize the sentencizer.
@ -69,8 +69,7 @@ Initialize the sentencizer.
| `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ | | `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ |
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"` ~~Optional[Callable]~~ | | `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"` ~~Optional[Callable]~~ |
```python ```python {title="punct_chars defaults"}
### punct_chars defaults
['!', '.', '?', '։', '؟', '۔', '܀', '܁', '܂', '߹', '।', '॥', '၊', '။', '።', ['!', '.', '?', '։', '؟', '۔', '܀', '܁', '܂', '߹', '।', '॥', '၊', '။', '።',
'፧', '፨', '', '', '᜶', '', '', '᥄', '᥅', '᪨', '᪩', '᪪', '᪫', '፧', '፨', '', '', '᜶', '', '', '᥄', '᥅', '᪨', '᪩', '᪪', '᪫',
'᭚', '᭛', '᭞', '᭟', '᰻', '᰼', '᱾', '᱿', '‼', '‽', '⁇', '⁈', '⁉', '᭚', '᭛', '᭞', '᭟', '᰻', '᰼', '᱾', '᱿', '‼', '‽', '⁇', '⁈', '⁉',
@ -83,7 +82,7 @@ Initialize the sentencizer.
'𑪜', '𑱁', '𑱂', '𖩮', '𖩯', '𖫵', '𖬷', '𖬸', '𖭄', '𛲟', '𝪈', '。', '。'] '𑪜', '𑱁', '𑱂', '𖩮', '𖩯', '𖫵', '𖬷', '𖬸', '𖭄', '𛲟', '𝪈', '。', '。']
``` ```
## Sentencizer.\_\_call\_\_ {#call tag="method"} ## Sentencizer.\_\_call\_\_ {id="call",tag="method"}
Apply the sentencizer on a `Doc`. Typically, this happens automatically after Apply the sentencizer on a `Doc`. Typically, this happens automatically after
the component has been added to the pipeline using the component has been added to the pipeline using
@ -105,7 +104,7 @@ the component has been added to the pipeline using
| `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ | | `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ |
| **RETURNS** | The modified `Doc` with added sentence boundaries. ~~Doc~~ | | **RETURNS** | The modified `Doc` with added sentence boundaries. ~~Doc~~ |
## Sentencizer.pipe {#pipe tag="method"} ## Sentencizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -126,7 +125,7 @@ applied to the `Doc` in order.
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## Sentencizer.to_disk {#to_disk tag="method"} ## Sentencizer.to_disk {id="to_disk",tag="method"}
Save the sentencizer settings (punctuation characters) to a directory. Will Save the sentencizer settings (punctuation characters) to a directory. Will
create a file `sentencizer.json`. This also happens automatically when you save create a file `sentencizer.json`. This also happens automatically when you save
@ -144,7 +143,7 @@ an `nlp` object with a sentencizer added to its pipeline.
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path` | A path to a JSON file, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a JSON file, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## Sentencizer.from_disk {#from_disk tag="method"} ## Sentencizer.from_disk {id="from_disk",tag="method"}
Load the sentencizer settings from a file. Expects a JSON file. This also Load the sentencizer settings from a file. Expects a JSON file. This also
happens automatically when you load an `nlp` object or model with a sentencizer happens automatically when you load an `nlp` object or model with a sentencizer
@ -162,7 +161,7 @@ added to its pipeline.
| `path` | A path to a JSON file. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a JSON file. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The modified `Sentencizer` object. ~~Sentencizer~~ | | **RETURNS** | The modified `Sentencizer` object. ~~Sentencizer~~ |
## Sentencizer.to_bytes {#to_bytes tag="method"} ## Sentencizer.to_bytes {id="to_bytes",tag="method"}
Serialize the sentencizer settings to a bytestring. Serialize the sentencizer settings to a bytestring.
@ -178,7 +177,7 @@ Serialize the sentencizer settings to a bytestring.
| ----------- | ------------------------------ | | ----------- | ------------------------------ |
| **RETURNS** | The serialized data. ~~bytes~~ | | **RETURNS** | The serialized data. ~~bytes~~ |
## Sentencizer.from_bytes {#from_bytes tag="method"} ## Sentencizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.

View File

@ -33,7 +33,7 @@ use case is as a post-processing step on word-level
[coreference resolution](/api/coref). The input and output keys used to store [coreference resolution](/api/coref). The input and output keys used to store
`Span` objects are configurable. `Span` objects are configurable.
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Predictions will be saved to `Doc.spans` as [`SpanGroup`s](/api/spangroup). Predictions will be saved to `Doc.spans` as [`SpanGroup`s](/api/spangroup).
@ -46,7 +46,7 @@ prefixes are configurable.
| ------------------------------------------------- | ------------------------------------------------------------------------- | | ------------------------------------------------- | ------------------------------------------------------------------------- |
| `Doc.spans[output_prefix + "_" + cluster_number]` | One group of predicted spans. Cluster number starts from 1. ~~SpanGroup~~ | | `Doc.spans[output_prefix + "_" + cluster_number]` | One group of predicted spans. Cluster number starts from 1. ~~SpanGroup~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -74,7 +74,7 @@ details on the architectures and their arguments and hyperparameters.
| `input_prefix` | The prefix to use for input `SpanGroup`s. Defaults to `coref_head_clusters`. ~~str~~ | | `input_prefix` | The prefix to use for input `SpanGroup`s. Defaults to `coref_head_clusters`. ~~str~~ |
| `output_prefix` | The prefix for predicted `SpanGroup`s. Defaults to `coref_clusters`. ~~str~~ | | `output_prefix` | The prefix for predicted `SpanGroup`s. Defaults to `coref_clusters`. ~~str~~ |
## SpanResolver.\_\_init\_\_ {#init tag="method"} ## SpanResolver.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -104,7 +104,7 @@ shortcut for this and instantiate the component using its string name and
| `input_prefix` | The prefix to use for input `SpanGroup`s. Defaults to `coref_head_clusters`. ~~str~~ | | `input_prefix` | The prefix to use for input `SpanGroup`s. Defaults to `coref_head_clusters`. ~~str~~ |
| `output_prefix` | The prefix for predicted `SpanGroup`s. Defaults to `coref_clusters`. ~~str~~ | | `output_prefix` | The prefix for predicted `SpanGroup`s. Defaults to `coref_clusters`. ~~str~~ |
## SpanResolver.\_\_call\_\_ {#call tag="method"} ## SpanResolver.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place and returned. Apply the pipe to one document. The document is modified in place and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -126,7 +126,7 @@ and [`set_annotations`](#set_annotations) methods.
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## SpanResolver.pipe {#pipe tag="method"} ## SpanResolver.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -150,7 +150,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/span-resolver#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## SpanResolver.initialize {#initialize tag="method"} ## SpanResolver.initialize {id="initialize",tag="method"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -174,7 +174,7 @@ by [`Language.initialize`](/api/language#initialize).
| _keyword-only_ | | | _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## SpanResolver.predict {#predict tag="method"} ## SpanResolver.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. Predictions are returned as a list of `MentionClusters`, one for modifying them. Predictions are returned as a list of `MentionClusters`, one for
@ -194,7 +194,7 @@ correspond to token indices.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The predicted spans for the `Doc`s. ~~List[MentionClusters]~~ | | **RETURNS** | The predicted spans for the `Doc`s. ~~List[MentionClusters]~~ |
## SpanResolver.set_annotations {#set_annotations tag="method"} ## SpanResolver.set_annotations {id="set_annotations",tag="method"}
Modify a batch of documents, saving predictions using the output prefix in Modify a batch of documents, saving predictions using the output prefix in
`Doc.spans`. `Doc.spans`.
@ -212,7 +212,7 @@ Modify a batch of documents, saving predictions using the output prefix in
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `spans` | The predicted spans for the `docs`. ~~List[MentionClusters]~~ | | `spans` | The predicted spans for the `docs`. ~~List[MentionClusters]~~ |
## SpanResolver.update {#update tag="method"} ## SpanResolver.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects. Delegates to Learn from a batch of [`Example`](/api/example) objects. Delegates to
[`predict`](/api/span-resolver#predict). [`predict`](/api/span-resolver#predict).
@ -234,7 +234,7 @@ Learn from a batch of [`Example`](/api/example) objects. Delegates to
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## SpanResolver.create_optimizer {#create_optimizer tag="method"} ## SpanResolver.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -249,7 +249,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## SpanResolver.use_params {#use_params tag="method, contextmanager"} ## SpanResolver.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -266,7 +266,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## SpanResolver.to_disk {#to_disk tag="method"} ## SpanResolver.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -283,7 +283,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## SpanResolver.from_disk {#from_disk tag="method"} ## SpanResolver.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -301,7 +301,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `SpanResolver` object. ~~SpanResolver~~ | | **RETURNS** | The modified `SpanResolver` object. ~~SpanResolver~~ |
## SpanResolver.to_bytes {#to_bytes tag="method"} ## SpanResolver.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -318,7 +318,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `SpanResolver` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `SpanResolver` object. ~~bytes~~ |
## SpanResolver.from_bytes {#from_bytes tag="method"} ## SpanResolver.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -337,7 +337,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `SpanResolver` object. ~~SpanResolver~~ | | **RETURNS** | The `SpanResolver` object. ~~SpanResolver~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -6,7 +6,7 @@ source: spacy/tokens/span.pyx
A slice from a [`Doc`](/api/doc) object. A slice from a [`Doc`](/api/doc) object.
## Span.\_\_init\_\_ {#init tag="method"} ## Span.\_\_init\_\_ {id="init",tag="method"}
Create a `Span` object from the slice `doc[start : end]`. Create a `Span` object from the slice `doc[start : end]`.
@ -29,7 +29,7 @@ Create a `Span` object from the slice `doc[start : end]`.
| `kb_id` | A knowledge base ID to attach to the span, e.g. for named entities. ~~Union[str, int]~~ | | `kb_id` | A knowledge base ID to attach to the span, e.g. for named entities. ~~Union[str, int]~~ |
| `span_id` | An ID to associate with the span. ~~Union[str, int]~~ | | `span_id` | An ID to associate with the span. ~~Union[str, int]~~ |
## Span.\_\_getitem\_\_ {#getitem tag="method"} ## Span.\_\_getitem\_\_ {id="getitem",tag="method"}
Get a `Token` object. Get a `Token` object.
@ -61,7 +61,7 @@ Get a `Span` object.
| `start_end` | The slice of the span to get. ~~Tuple[int, int]~~ | | `start_end` | The slice of the span to get. ~~Tuple[int, int]~~ |
| **RETURNS** | The span at `span[start : end]`. ~~Span~~ | | **RETURNS** | The span at `span[start : end]`. ~~Span~~ |
## Span.\_\_iter\_\_ {#iter tag="method"} ## Span.\_\_iter\_\_ {id="iter",tag="method"}
Iterate over `Token` objects. Iterate over `Token` objects.
@ -77,7 +77,7 @@ Iterate over `Token` objects.
| ---------- | --------------------------- | | ---------- | --------------------------- |
| **YIELDS** | A `Token` object. ~~Token~~ | | **YIELDS** | A `Token` object. ~~Token~~ |
## Span.\_\_len\_\_ {#len tag="method"} ## Span.\_\_len\_\_ {id="len",tag="method"}
Get the number of tokens in the span. Get the number of tokens in the span.
@ -93,7 +93,7 @@ Get the number of tokens in the span.
| ----------- | ----------------------------------------- | | ----------- | ----------------------------------------- |
| **RETURNS** | The number of tokens in the span. ~~int~~ | | **RETURNS** | The number of tokens in the span. ~~int~~ |
## Span.set_extension {#set_extension tag="classmethod" new="2"} ## Span.set_extension {id="set_extension",tag="classmethod",version="2"}
Define a custom attribute on the `Span` which becomes available via `Span._`. Define a custom attribute on the `Span` which becomes available via `Span._`.
For details, see the documentation on For details, see the documentation on
@ -118,7 +118,7 @@ For details, see the documentation on
| `setter` | Setter function that takes the `Span` and a value, and modifies the object. Is called when the user writes to the `Span._` attribute. ~~Optional[Callable[[Span, Any], None]]~~ | | `setter` | Setter function that takes the `Span` and a value, and modifies the object. Is called when the user writes to the `Span._` attribute. ~~Optional[Callable[[Span, Any], None]]~~ |
| `force` | Force overwriting existing attribute. ~~bool~~ | | `force` | Force overwriting existing attribute. ~~bool~~ |
## Span.get_extension {#get_extension tag="classmethod" new="2"} ## Span.get_extension {id="get_extension",tag="classmethod",version="2"}
Look up a previously registered extension by name. Returns a 4-tuple Look up a previously registered extension by name. Returns a 4-tuple
`(default, method, getter, setter)` if the extension is registered. Raises a `(default, method, getter, setter)` if the extension is registered. Raises a
@ -138,7 +138,7 @@ Look up a previously registered extension by name. Returns a 4-tuple
| `name` | Name of the extension. ~~str~~ | | `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Span.has_extension {#has_extension tag="classmethod" new="2"} ## Span.has_extension {id="has_extension",tag="classmethod",version="2"}
Check whether an extension has been registered on the `Span` class. Check whether an extension has been registered on the `Span` class.
@ -155,7 +155,7 @@ Check whether an extension has been registered on the `Span` class.
| `name` | Name of the extension to check. ~~str~~ | | `name` | Name of the extension to check. ~~str~~ |
| **RETURNS** | Whether the extension has been registered. ~~bool~~ | | **RETURNS** | Whether the extension has been registered. ~~bool~~ |
## Span.remove_extension {#remove_extension tag="classmethod" new="2.0.12"} ## Span.remove_extension {id="remove_extension",tag="classmethod",version="2.0.12"}
Remove a previously registered extension. Remove a previously registered extension.
@ -173,7 +173,7 @@ Remove a previously registered extension.
| `name` | Name of the extension. ~~str~~ | | `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Span.char_span {#char_span tag="method" new="2.2.4"} ## Span.char_span {id="char_span",tag="method",version="2.2.4"}
Create a `Span` object from the slice `span.text[start:end]`. Returns `None` if Create a `Span` object from the slice `span.text[start:end]`. Returns `None` if
the character indices don't map to a valid span. the character indices don't map to a valid span.
@ -195,7 +195,7 @@ the character indices don't map to a valid span.
| `vector` | A meaning representation of the span. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | | `vector` | A meaning representation of the span. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
| **RETURNS** | The newly constructed object or `None`. ~~Optional[Span]~~ | | **RETURNS** | The newly constructed object or `None`. ~~Optional[Span]~~ |
## Span.similarity {#similarity tag="method" model="vectors"} ## Span.similarity {id="similarity",tag="method",model="vectors"}
Make a semantic similarity estimate. The default estimate is cosine similarity Make a semantic similarity estimate. The default estimate is cosine similarity
using an average of word vectors. using an average of word vectors.
@ -216,7 +216,7 @@ using an average of word vectors.
| `other` | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ | | `other` | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ |
| **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ | | **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ |
## Span.get_lca_matrix {#get_lca_matrix tag="method"} ## Span.get_lca_matrix {id="get_lca_matrix",tag="method"}
Calculates the lowest common ancestor matrix for a given `Span`. Returns LCA Calculates the lowest common ancestor matrix for a given `Span`. Returns LCA
matrix containing the integer index of the ancestor, or `-1` if no common matrix containing the integer index of the ancestor, or `-1` if no common
@ -235,7 +235,7 @@ ancestor is found, e.g. if span excludes a necessary ancestor.
| ----------- | --------------------------------------------------------------------------------------- | | ----------- | --------------------------------------------------------------------------------------- |
| **RETURNS** | The lowest common ancestor matrix of the `Span`. ~~numpy.ndarray[ndim=2, dtype=int32]~~ | | **RETURNS** | The lowest common ancestor matrix of the `Span`. ~~numpy.ndarray[ndim=2, dtype=int32]~~ |
## Span.to_array {#to_array tag="method" new="2"} ## Span.to_array {id="to_array",tag="method",version="2"}
Given a list of `M` attribute IDs, export the tokens to a numpy `ndarray` of Given a list of `M` attribute IDs, export the tokens to a numpy `ndarray` of
shape `(N, M)`, where `N` is the length of the document. The values will be shape `(N, M)`, where `N` is the length of the document. The values will be
@ -256,7 +256,7 @@ shape `(N, M)`, where `N` is the length of the document. The values will be
| `attr_ids` | A list of attributes (int IDs or string names) or a single attribute (int ID or string name). ~~Union[int, str, List[Union[int, str]]]~~ | | `attr_ids` | A list of attributes (int IDs or string names) or a single attribute (int ID or string name). ~~Union[int, str, List[Union[int, str]]]~~ |
| **RETURNS** | The exported attributes as a numpy array. ~~Union[numpy.ndarray[ndim=2, dtype=uint64], numpy.ndarray[ndim=1, dtype=uint64]]~~ | | **RETURNS** | The exported attributes as a numpy array. ~~Union[numpy.ndarray[ndim=2, dtype=uint64], numpy.ndarray[ndim=1, dtype=uint64]]~~ |
## Span.ents {#ents tag="property" new="2.0.13" model="ner"} ## Span.ents {id="ents",tag="property",version="2.0.13",model="ner"}
The named entities that fall completely within the span. Returns a tuple of The named entities that fall completely within the span. Returns a tuple of
`Span` objects. `Span` objects.
@ -276,7 +276,7 @@ The named entities that fall completely within the span. Returns a tuple of
| ----------- | ----------------------------------------------------------------- | | ----------- | ----------------------------------------------------------------- |
| **RETURNS** | Entities in the span, one `Span` per entity. ~~Tuple[Span, ...]~~ | | **RETURNS** | Entities in the span, one `Span` per entity. ~~Tuple[Span, ...]~~ |
## Span.noun_chunks {#noun_chunks tag="property" model="parser"} ## Span.noun_chunks {id="noun_chunks",tag="property",model="parser"}
Iterate over the base noun phrases in the span. Yields base noun-phrase `Span` Iterate over the base noun phrases in the span. Yields base noun-phrase `Span`
objects, if the document has been syntactically parsed. A base noun phrase, or objects, if the document has been syntactically parsed. A base noun phrase, or
@ -302,7 +302,7 @@ raised.
| ---------- | --------------------------------- | | ---------- | --------------------------------- |
| **YIELDS** | Noun chunks in the span. ~~Span~~ | | **YIELDS** | Noun chunks in the span. ~~Span~~ |
## Span.as_doc {#as_doc tag="method"} ## Span.as_doc {id="as_doc",tag="method"}
Create a new `Doc` object corresponding to the `Span`, with a copy of the data. Create a new `Doc` object corresponding to the `Span`, with a copy of the data.
@ -326,7 +326,7 @@ time.
| `array` | Precomputed array version of the original doc as generated by [`Doc.to_array`](/api/doc#to_array). ~~numpy.ndarray~~ | | `array` | Precomputed array version of the original doc as generated by [`Doc.to_array`](/api/doc#to_array). ~~numpy.ndarray~~ |
| **RETURNS** | A `Doc` object of the `Span`'s content. ~~Doc~~ | | **RETURNS** | A `Doc` object of the `Span`'s content. ~~Doc~~ |
## Span.root {#root tag="property" model="parser"} ## Span.root {id="root",tag="property",model="parser"}
The token with the shortest path to the root of the sentence (or the root The token with the shortest path to the root of the sentence (or the root
itself). If multiple tokens are equally high in the tree, the first token is itself). If multiple tokens are equally high in the tree, the first token is
@ -347,7 +347,7 @@ taken.
| ----------- | ------------------------- | | ----------- | ------------------------- |
| **RETURNS** | The root token. ~~Token~~ | | **RETURNS** | The root token. ~~Token~~ |
## Span.conjuncts {#conjuncts tag="property" model="parser"} ## Span.conjuncts {id="conjuncts",tag="property",model="parser"}
A tuple of tokens coordinated to `span.root`. A tuple of tokens coordinated to `span.root`.
@ -363,7 +363,7 @@ A tuple of tokens coordinated to `span.root`.
| ----------- | --------------------------------------------- | | ----------- | --------------------------------------------- |
| **RETURNS** | The coordinated tokens. ~~Tuple[Token, ...]~~ | | **RETURNS** | The coordinated tokens. ~~Tuple[Token, ...]~~ |
## Span.lefts {#lefts tag="property" model="parser"} ## Span.lefts {id="lefts",tag="property",model="parser"}
Tokens that are to the left of the span, whose heads are within the span. Tokens that are to the left of the span, whose heads are within the span.
@ -379,7 +379,7 @@ Tokens that are to the left of the span, whose heads are within the span.
| ---------- | ---------------------------------------------- | | ---------- | ---------------------------------------------- |
| **YIELDS** | A left-child of a token of the span. ~~Token~~ | | **YIELDS** | A left-child of a token of the span. ~~Token~~ |
## Span.rights {#rights tag="property" model="parser"} ## Span.rights {id="rights",tag="property",model="parser"}
Tokens that are to the right of the span, whose heads are within the span. Tokens that are to the right of the span, whose heads are within the span.
@ -395,7 +395,7 @@ Tokens that are to the right of the span, whose heads are within the span.
| ---------- | ----------------------------------------------- | | ---------- | ----------------------------------------------- |
| **YIELDS** | A right-child of a token of the span. ~~Token~~ | | **YIELDS** | A right-child of a token of the span. ~~Token~~ |
## Span.n_lefts {#n_lefts tag="property" model="parser"} ## Span.n_lefts {id="n_lefts",tag="property",model="parser"}
The number of tokens that are to the left of the span, whose heads are within The number of tokens that are to the left of the span, whose heads are within
the span. the span.
@ -411,7 +411,7 @@ the span.
| ----------- | ---------------------------------------- | | ----------- | ---------------------------------------- |
| **RETURNS** | The number of left-child tokens. ~~int~~ | | **RETURNS** | The number of left-child tokens. ~~int~~ |
## Span.n_rights {#n_rights tag="property" model="parser"} ## Span.n_rights {id="n_rights",tag="property",model="parser"}
The number of tokens that are to the right of the span, whose heads are within The number of tokens that are to the right of the span, whose heads are within
the span. the span.
@ -427,7 +427,7 @@ the span.
| ----------- | ----------------------------------------- | | ----------- | ----------------------------------------- |
| **RETURNS** | The number of right-child tokens. ~~int~~ | | **RETURNS** | The number of right-child tokens. ~~int~~ |
## Span.subtree {#subtree tag="property" model="parser"} ## Span.subtree {id="subtree",tag="property",model="parser"}
Tokens within the span and tokens which descend from them. Tokens within the span and tokens which descend from them.
@ -443,7 +443,7 @@ Tokens within the span and tokens which descend from them.
| ---------- | ----------------------------------------------------------- | | ---------- | ----------------------------------------------------------- |
| **YIELDS** | A token within the span, or a descendant from it. ~~Token~~ | | **YIELDS** | A token within the span, or a descendant from it. ~~Token~~ |
## Span.has_vector {#has_vector tag="property" model="vectors"} ## Span.has_vector {id="has_vector",tag="property",model="vectors"}
A boolean value indicating whether a word vector is associated with the object. A boolean value indicating whether a word vector is associated with the object.
@ -458,7 +458,7 @@ A boolean value indicating whether a word vector is associated with the object.
| ----------- | ----------------------------------------------------- | | ----------- | ----------------------------------------------------- |
| **RETURNS** | Whether the span has a vector data attached. ~~bool~~ | | **RETURNS** | Whether the span has a vector data attached. ~~bool~~ |
## Span.vector {#vector tag="property" model="vectors"} ## Span.vector {id="vector",tag="property",model="vectors"}
A real-valued meaning representation. Defaults to an average of the token A real-valued meaning representation. Defaults to an average of the token
vectors. vectors.
@ -475,7 +475,7 @@ vectors.
| ----------- | ----------------------------------------------------------------------------------------------- | | ----------- | ----------------------------------------------------------------------------------------------- |
| **RETURNS** | A 1-dimensional array representing the span's vector. ~~`numpy.ndarray[ndim=1, dtype=float32]~~ | | **RETURNS** | A 1-dimensional array representing the span's vector. ~~`numpy.ndarray[ndim=1, dtype=float32]~~ |
## Span.vector_norm {#vector_norm tag="property" model="vectors"} ## Span.vector_norm {id="vector_norm",tag="property",model="vectors"}
The L2 norm of the span's vector representation. The L2 norm of the span's vector representation.
@ -492,7 +492,7 @@ The L2 norm of the span's vector representation.
| ----------- | --------------------------------------------------- | | ----------- | --------------------------------------------------- |
| **RETURNS** | The L2 norm of the vector representation. ~~float~~ | | **RETURNS** | The L2 norm of the vector representation. ~~float~~ |
## Span.sent {#sent tag="property" model="sentences"} ## Span.sent {id="sent",tag="property",model="sentences"}
The sentence span that this span is a part of. This property is only available The sentence span that this span is a part of. This property is only available
when [sentence boundaries](/usage/linguistic-features#sbd) have been set on the when [sentence boundaries](/usage/linguistic-features#sbd) have been set on the
@ -520,7 +520,7 @@ sent = doc[sent.start : max(sent.end, span.end)]
| ----------- | ------------------------------------------------------- | | ----------- | ------------------------------------------------------- |
| **RETURNS** | The sentence span that this span is a part of. ~~Span~~ | | **RETURNS** | The sentence span that this span is a part of. ~~Span~~ |
## Span.sents {#sents tag="property" model="sentences" new="3.2.1"} ## Span.sents {id="sents",tag="property",model="sentences",version="3.2.1"}
Returns a generator over the sentences the span belongs to. This property is Returns a generator over the sentences the span belongs to. This property is
only available when [sentence boundaries](/usage/linguistic-features#sbd) have only available when [sentence boundaries](/usage/linguistic-features#sbd) have
@ -542,7 +542,7 @@ overlaps with will be returned.
| ----------- | -------------------------------------------------------------------------- | | ----------- | -------------------------------------------------------------------------- |
| **RETURNS** | A generator yielding sentences this `Span` is a part of ~~Iterable[Span]~~ | | **RETURNS** | A generator yielding sentences this `Span` is a part of ~~Iterable[Span]~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------- | | -------------- | ----------------------------------------------------------------------------------------------------------------------------- |

View File

@ -2,7 +2,7 @@
title: SpanCategorizer title: SpanCategorizer
tag: class,experimental tag: class,experimental
source: spacy/pipeline/spancat.py source: spacy/pipeline/spancat.py
new: 3.1 version: 3.1
teaser: 'Pipeline component for labeling potentially overlapping spans of text' teaser: 'Pipeline component for labeling potentially overlapping spans of text'
api_base_class: /api/pipe api_base_class: /api/pipe
api_string_name: spancat api_string_name: spancat
@ -16,7 +16,7 @@ that predicts zero or more labels for each candidate.
Predicted spans will be saved in a [`SpanGroup`](/api/spangroup) on the doc. Predicted spans will be saved in a [`SpanGroup`](/api/spangroup) on the doc.
Individual span scores can be found in `spangroup.attrs["scores"]`. Individual span scores can be found in `spangroup.attrs["scores"]`.
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Predictions will be saved to `Doc.spans[spans_key]` as a Predictions will be saved to `Doc.spans[spans_key]` as a
[`SpanGroup`](/api/spangroup). The scores for the spans in the `SpanGroup` will [`SpanGroup`](/api/spangroup). The scores for the spans in the `SpanGroup` will
@ -29,7 +29,7 @@ be saved in `SpanGroup.attrs["scores"]`.
| `Doc.spans[spans_key]` | The annotated spans. ~~SpanGroup~~ | | `Doc.spans[spans_key]` | The annotated spans. ~~SpanGroup~~ |
| `Doc.spans[spans_key].attrs["scores"]` | The score for each span in the `SpanGroup`. ~~Floats1d~~ | | `Doc.spans[spans_key].attrs["scores"]` | The score for each span in the `SpanGroup`. ~~Floats1d~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -65,7 +65,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/spancat.py %%GITHUB_SPACY/spacy/pipeline/spancat.py
``` ```
## SpanCategorizer.\_\_init\_\_ {#init tag="method"} ## SpanCategorizer.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -97,7 +97,7 @@ shortcut for this and instantiate the component using its string name and
| `threshold` | Minimum probability to consider a prediction positive. Spans with a positive prediction will be saved on the Doc. Defaults to `0.5`. ~~float~~ | | `threshold` | Minimum probability to consider a prediction positive. Spans with a positive prediction will be saved on the Doc. Defaults to `0.5`. ~~float~~ |
| `max_positive` | Maximum number of labels to consider positive per span. Defaults to `None`, indicating no limit. ~~Optional[int]~~ | | `max_positive` | Maximum number of labels to consider positive per span. Defaults to `None`, indicating no limit. ~~Optional[int]~~ |
## SpanCategorizer.\_\_call\_\_ {#call tag="method"} ## SpanCategorizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned. Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -120,7 +120,7 @@ delegate to the [`predict`](/api/spancategorizer#predict) and
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## SpanCategorizer.pipe {#pipe tag="method"} ## SpanCategorizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -144,7 +144,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/spancategorizer#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## SpanCategorizer.initialize {#initialize tag="method"} ## SpanCategorizer.initialize {id="initialize",tag="method"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -181,7 +181,7 @@ config.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ |
## SpanCategorizer.predict {#predict tag="method"} ## SpanCategorizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects without Apply the component's model to a batch of [`Doc`](/api/doc) objects without
modifying them. modifying them.
@ -198,7 +198,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. | | **RETURNS** | The model's prediction for each document. |
## SpanCategorizer.set_annotations {#set_annotations tag="method"} ## SpanCategorizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores. Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores.
@ -215,7 +215,7 @@ Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `SpanCategorizer.predict`. | | `scores` | The scores to set, produced by `SpanCategorizer.predict`. |
## SpanCategorizer.update {#update tag="method"} ## SpanCategorizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model. predictions and gold-standard annotations, and update the component's model.
@ -239,7 +239,7 @@ Delegates to [`predict`](/api/spancategorizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## SpanCategorizer.set_candidates {#set_candidates tag="method", new="3.3"} ## SpanCategorizer.set_candidates {id="set_candidates",tag="method", version="3.3"}
Use the suggester to add a list of [`Span`](/api/span) candidates to a list of Use the suggester to add a list of [`Span`](/api/span) candidates to a list of
[`Doc`](/api/doc) objects. This method is intended to be used for debugging [`Doc`](/api/doc) objects. This method is intended to be used for debugging
@ -257,7 +257,7 @@ purposes.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `candidates_key` | Key of the Doc.spans dict to save the candidate spans under. ~~str~~ | | `candidates_key` | Key of the Doc.spans dict to save the candidate spans under. ~~str~~ |
## SpanCategorizer.get_loss {#get_loss tag="method"} ## SpanCategorizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their Find the loss and gradient of loss for the batch of documents and their
predicted scores. predicted scores.
@ -276,7 +276,7 @@ predicted scores.
| `spans_scores` | Scores representing the model's predictions. ~~Tuple[Ragged, Floats2d]~~ | | `spans_scores` | Scores representing the model's predictions. ~~Tuple[Ragged, Floats2d]~~ |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## SpanCategorizer.create_optimizer {#create_optimizer tag="method"} ## SpanCategorizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -291,7 +291,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## SpanCategorizer.use_params {#use_params tag="method, contextmanager"} ## SpanCategorizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model to use the given parameter values. Modify the pipe's model to use the given parameter values.
@ -307,7 +307,7 @@ Modify the pipe's model to use the given parameter values.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## SpanCategorizer.add_label {#add_label tag="method"} ## SpanCategorizer.add_label {id="add_label",tag="method"}
Add a new label to the pipe. Raises an error if the output dimension is already Add a new label to the pipe. Raises an error if the output dimension is already
set, or if the model has already been fully [initialized](#initialize). Note set, or if the model has already been fully [initialized](#initialize). Note
@ -329,7 +329,7 @@ automatically.
| `label` | The label to add. ~~str~~ | | `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## SpanCategorizer.to_disk {#to_disk tag="method"} ## SpanCategorizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -346,7 +346,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## SpanCategorizer.from_disk {#from_disk tag="method"} ## SpanCategorizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -364,7 +364,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `SpanCategorizer` object. ~~SpanCategorizer~~ | | **RETURNS** | The modified `SpanCategorizer` object. ~~SpanCategorizer~~ |
## SpanCategorizer.to_bytes {#to_bytes tag="method"} ## SpanCategorizer.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -381,7 +381,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `SpanCategorizer` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `SpanCategorizer` object. ~~bytes~~ |
## SpanCategorizer.from_bytes {#from_bytes tag="method"} ## SpanCategorizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -400,7 +400,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `SpanCategorizer` object. ~~SpanCategorizer~~ | | **RETURNS** | The `SpanCategorizer` object. ~~SpanCategorizer~~ |
## SpanCategorizer.labels {#labels tag="property"} ## SpanCategorizer.labels {id="labels",tag="property"}
The labels currently added to the component. The labels currently added to the component.
@ -415,7 +415,7 @@ The labels currently added to the component.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## SpanCategorizer.label_data {#label_data tag="property"} ## SpanCategorizer.label_data {id="label_data",tag="property"}
The labels currently added to the component and their internal meta information. The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -433,7 +433,7 @@ the model with a pre-defined label set.
| ----------- | ---------------------------------------------------------- | | ----------- | ---------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from
@ -451,9 +451,9 @@ serialization by passing in the string names via the `exclude` argument.
| `cfg` | The config file. You usually don't want to exclude this. | | `cfg` | The config file. You usually don't want to exclude this. |
| `model` | The binary model data. You usually don't want to exclude this. | | `model` | The binary model data. You usually don't want to exclude this. |
## Suggesters {#suggesters tag="registered functions" source="spacy/pipeline/spancat.py"} ## Suggesters {id="suggesters",tag="registered functions",source="spacy/pipeline/spancat.py"}
### spacy.ngram_suggester.v1 {#ngram_suggester} ### spacy.ngram_suggester.v1 {id="ngram_suggester"}
> #### Example Config > #### Example Config
> >
@ -471,7 +471,7 @@ integers. The array has two columns, indicating the start and end position.
| `sizes` | The phrase lengths to suggest. For example, `[1, 2]` will suggest phrases consisting of 1 or 2 tokens. ~~List[int]~~ | | `sizes` | The phrase lengths to suggest. For example, `[1, 2]` will suggest phrases consisting of 1 or 2 tokens. ~~List[int]~~ |
| **CREATES** | The suggester function. ~~Callable[[Iterable[Doc], Optional[Ops]], Ragged]~~ | | **CREATES** | The suggester function. ~~Callable[[Iterable[Doc], Optional[Ops]], Ragged]~~ |
### spacy.ngram_range_suggester.v1 {#ngram_range_suggester} ### spacy.ngram_range_suggester.v1 {id="ngram_range_suggester"}
> #### Example Config > #### Example Config
> >

View File

@ -2,7 +2,7 @@
title: SpanGroup title: SpanGroup
tag: class tag: class
source: spacy/tokens/span_group.pyx source: spacy/tokens/span_group.pyx
new: 3 version: 3
--- ---
A group of arbitrary, potentially overlapping [`Span`](/api/span) objects that A group of arbitrary, potentially overlapping [`Span`](/api/span) objects that
@ -13,7 +13,7 @@ into a `SpanGroup` object for you automatically on assignment. `SpanGroup`
objects behave similar to `list`s, so you can append `Span` objects to them or objects behave similar to `list`s, so you can append `Span` objects to them or
access a member at a given index. access a member at a given index.
## SpanGroup.\_\_init\_\_ {#init tag="method"} ## SpanGroup.\_\_init\_\_ {id="init",tag="method"}
Create a `SpanGroup`. Create a `SpanGroup`.
@ -42,7 +42,7 @@ Create a `SpanGroup`.
| `attrs` | Optional JSON-serializable attributes to attach to the span group. ~~Dict[str, Any]~~ | | `attrs` | Optional JSON-serializable attributes to attach to the span group. ~~Dict[str, Any]~~ |
| `spans` | The spans to add to the span group. ~~Iterable[Span]~~ | | `spans` | The spans to add to the span group. ~~Iterable[Span]~~ |
## SpanGroup.doc {#doc tag="property"} ## SpanGroup.doc {id="doc",tag="property"}
The [`Doc`](/api/doc) object the span group is referring to. The [`Doc`](/api/doc) object the span group is referring to.
@ -68,7 +68,7 @@ the scope of your function.
| ----------- | ------------------------------- | | ----------- | ------------------------------- |
| **RETURNS** | The reference document. ~~Doc~~ | | **RETURNS** | The reference document. ~~Doc~~ |
## SpanGroup.has_overlap {#has_overlap tag="property"} ## SpanGroup.has_overlap {id="has_overlap",tag="property"}
Check whether the span group contains overlapping spans. Check whether the span group contains overlapping spans.
@ -86,7 +86,7 @@ Check whether the span group contains overlapping spans.
| ----------- | -------------------------------------------------- | | ----------- | -------------------------------------------------- |
| **RETURNS** | Whether the span group contains overlaps. ~~bool~~ | | **RETURNS** | Whether the span group contains overlaps. ~~bool~~ |
## SpanGroup.\_\_len\_\_ {#len tag="method"} ## SpanGroup.\_\_len\_\_ {id="len",tag="method"}
Get the number of spans in the group. Get the number of spans in the group.
@ -102,7 +102,7 @@ Get the number of spans in the group.
| ----------- | ----------------------------------------- | | ----------- | ----------------------------------------- |
| **RETURNS** | The number of spans in the group. ~~int~~ | | **RETURNS** | The number of spans in the group. ~~int~~ |
## SpanGroup.\_\_getitem\_\_ {#getitem tag="method"} ## SpanGroup.\_\_getitem\_\_ {id="getitem",tag="method"}
Get a span from the group. Note that a copy of the span is returned, so if any Get a span from the group. Note that a copy of the span is returned, so if any
changes are made to this span, they are not reflected in the corresponding changes are made to this span, they are not reflected in the corresponding
@ -125,7 +125,7 @@ changes to be reflected in the span group.
| `i` | The item index. ~~int~~ | | `i` | The item index. ~~int~~ |
| **RETURNS** | The span at the given index. ~~Span~~ | | **RETURNS** | The span at the given index. ~~Span~~ |
## SpanGroup.\_\_setitem\_\_ {#setitem tag="method", new="3.3"} ## SpanGroup.\_\_setitem\_\_ {id="setitem",tag="method", version="3.3"}
Set a span in the span group. Set a span in the span group.
@ -144,7 +144,7 @@ Set a span in the span group.
| `i` | The item index. ~~int~~ | | `i` | The item index. ~~int~~ |
| `span` | The new value. ~~Span~~ | | `span` | The new value. ~~Span~~ |
## SpanGroup.\_\_delitem\_\_ {#delitem tag="method", new="3.3"} ## SpanGroup.\_\_delitem\_\_ {id="delitem",tag="method", version="3.3"}
Delete a span from the span group. Delete a span from the span group.
@ -161,7 +161,7 @@ Delete a span from the span group.
| ---- | ----------------------- | | ---- | ----------------------- |
| `i` | The item index. ~~int~~ | | `i` | The item index. ~~int~~ |
## SpanGroup.\_\_add\_\_ {#add tag="method", new="3.3"} ## SpanGroup.\_\_add\_\_ {id="add",tag="method", version="3.3"}
Concatenate the current span group with another span group and return the result Concatenate the current span group with another span group and return the result
in a new span group. Any `attrs` from the first span group will have precedence in a new span group. Any `attrs` from the first span group will have precedence
@ -182,7 +182,7 @@ over `attrs` in the second.
| `other` | The span group or spans to concatenate. ~~Union[SpanGroup, Iterable[Span]]~~ | | `other` | The span group or spans to concatenate. ~~Union[SpanGroup, Iterable[Span]]~~ |
| **RETURNS** | The new span group. ~~SpanGroup~~ | | **RETURNS** | The new span group. ~~SpanGroup~~ |
## SpanGroup.\_\_iadd\_\_ {#iadd tag="method", new="3.3"} ## SpanGroup.\_\_iadd\_\_ {id="iadd",tag="method", version="3.3"}
Append an iterable of spans or the content of a span group to the current span Append an iterable of spans or the content of a span group to the current span
group. Any `attrs` in the other span group will be added for keys that are not group. Any `attrs` in the other span group will be added for keys that are not
@ -202,7 +202,7 @@ already present in the current span group.
| `other` | The span group or spans to append. ~~Union[SpanGroup, Iterable[Span]]~~ | | `other` | The span group or spans to append. ~~Union[SpanGroup, Iterable[Span]]~~ |
| **RETURNS** | The span group. ~~SpanGroup~~ | | **RETURNS** | The span group. ~~SpanGroup~~ |
## SpanGroup.\_\_iter\_\_ {#iter tag="method" new="3.5"} ## SpanGroup.\_\_iter\_\_ {id="iter",tag="method",version="3.5"}
Iterate over the spans in this span group. Iterate over the spans in this span group.
@ -219,7 +219,8 @@ Iterate over the spans in this span group.
| ---------- | ----------------------------------- | | ---------- | ----------------------------------- |
| **YIELDS** | A span in this span group. ~~Span~~ | | **YIELDS** | A span in this span group. ~~Span~~ |
## SpanGroup.append {#append tag="method"}
## SpanGroup.append {id="append",tag="method"}
Add a [`Span`](/api/span) object to the group. The span must refer to the same Add a [`Span`](/api/span) object to the group. The span must refer to the same
[`Doc`](/api/doc) object as the span group. [`Doc`](/api/doc) object as the span group.
@ -237,7 +238,7 @@ Add a [`Span`](/api/span) object to the group. The span must refer to the same
| ------ | ---------------------------- | | ------ | ---------------------------- |
| `span` | The span to append. ~~Span~~ | | `span` | The span to append. ~~Span~~ |
## SpanGroup.extend {#extend tag="method"} ## SpanGroup.extend {id="extend",tag="method"}
Add multiple [`Span`](/api/span) objects or contents of another `SpanGroup` to Add multiple [`Span`](/api/span) objects or contents of another `SpanGroup` to
the group. All spans must refer to the same [`Doc`](/api/doc) object as the span the group. All spans must refer to the same [`Doc`](/api/doc) object as the span
@ -258,7 +259,7 @@ group.
| ------- | -------------------------------------------------------- | | ------- | -------------------------------------------------------- |
| `spans` | The spans to add. ~~Union[SpanGroup, Iterable["Span"]]~~ | | `spans` | The spans to add. ~~Union[SpanGroup, Iterable["Span"]]~~ |
## SpanGroup.copy {#copy tag="method", new="3.3"} ## SpanGroup.copy {id="copy",tag="method", version="3.3"}
Return a copy of the span group. Return a copy of the span group.
@ -277,7 +278,7 @@ Return a copy of the span group.
| `doc` | The document to which the copy is bound. Defaults to `None` for the current doc. ~~Optional[Doc]~~ | | `doc` | The document to which the copy is bound. Defaults to `None` for the current doc. ~~Optional[Doc]~~ |
| **RETURNS** | A copy of the `SpanGroup` object. ~~SpanGroup~~ | | **RETURNS** | A copy of the `SpanGroup` object. ~~SpanGroup~~ |
## SpanGroup.to_bytes {#to_bytes tag="method"} ## SpanGroup.to_bytes {id="to_bytes",tag="method"}
Serialize the span group to a bytestring. Serialize the span group to a bytestring.
@ -293,7 +294,7 @@ Serialize the span group to a bytestring.
| ----------- | ------------------------------------- | | ----------- | ------------------------------------- |
| **RETURNS** | The serialized `SpanGroup`. ~~bytes~~ | | **RETURNS** | The serialized `SpanGroup`. ~~bytes~~ |
## SpanGroup.from_bytes {#from_bytes tag="method"} ## SpanGroup.from_bytes {id="from_bytes",tag="method"}
Load the span group from a bytestring. Modifies the object in place and returns Load the span group from a bytestring. Modifies the object in place and returns
it. it.

View File

@ -2,7 +2,7 @@
title: SpanRuler title: SpanRuler
tag: class tag: class
source: spacy/pipeline/span_ruler.py source: spacy/pipeline/span_ruler.py
new: 3.3 version: 3.3
teaser: 'Pipeline component for rule-based span and named entity recognition' teaser: 'Pipeline component for rule-based span and named entity recognition'
api_string_name: span_ruler api_string_name: span_ruler
api_trainable: false api_trainable: false
@ -13,7 +13,7 @@ The span ruler lets you add spans to [`Doc.spans`](/api/doc#spans) and/or
usage examples, see the docs on usage examples, see the docs on
[rule-based span matching](/usage/rule-based-matching#spanruler). [rule-based span matching](/usage/rule-based-matching#spanruler).
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Matches will be saved to `Doc.spans[spans_key]` as a Matches will be saved to `Doc.spans[spans_key]` as a
[`SpanGroup`](/api/spangroup) and/or to `Doc.ents`, where the annotation is [`SpanGroup`](/api/spangroup) and/or to `Doc.ents`, where the annotation is
@ -28,7 +28,7 @@ saved in the `Token.ent_type` and `Token.ent_iob` fields.
| `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ | | `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ |
| `Token.ent_type_` | The label part of the named entity tag. ~~str~~ | | `Token.ent_type_` | The label part of the named entity tag. ~~str~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -62,7 +62,7 @@ how the component should be configured. You can override its settings via the
%%GITHUB_SPACY/spacy/pipeline/span_ruler.py %%GITHUB_SPACY/spacy/pipeline/span_ruler.py
``` ```
## SpanRuler.\_\_init\_\_ {#init tag="method"} ## SpanRuler.\_\_init\_\_ {id="init",tag="method"}
Initialize the span ruler. If patterns are supplied here, they need to be a list Initialize the span ruler. If patterns are supplied here, they need to be a list
of dictionaries with a `"label"` and `"pattern"` key. A pattern can either be a of dictionaries with a `"label"` and `"pattern"` key. A pattern can either be a
@ -95,7 +95,7 @@ token pattern (list) or a phrase pattern (string). For example:
| `overwrite` | Whether to remove any existing spans under `Doc.spans[spans key]` if `spans_key` is set, or to remove any ents under `Doc.ents` if `annotate_ents` is set. Defaults to `True`. ~~bool~~ | | `overwrite` | Whether to remove any existing spans under `Doc.spans[spans key]` if `spans_key` is set, or to remove any ents under `Doc.ents` if `annotate_ents` is set. Defaults to `True`. ~~bool~~ |
| `scorer` | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for `Doc.spans[spans_key]` with overlapping spans allowed. ~~Optional[Callable]~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for `Doc.spans[spans_key]` with overlapping spans allowed. ~~Optional[Callable]~~ |
## SpanRuler.initialize {#initialize tag="method"} ## SpanRuler.initialize {id="initialize",tag="method"}
Initialize the component with data and used before training to load in rules Initialize the component with data and used before training to load in rules
from a [pattern file](/usage/rule-based-matching/#spanruler-files). This method from a [pattern file](/usage/rule-based-matching/#spanruler-files). This method
@ -127,7 +127,7 @@ config. Any existing patterns are removed on initialization.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `patterns` | The list of patterns. Defaults to `None`. ~~Optional[Sequence[Dict[str, Union[str, List[Dict[str, Any]]]]]]~~ | | `patterns` | The list of patterns. Defaults to `None`. ~~Optional[Sequence[Dict[str, Union[str, List[Dict[str, Any]]]]]]~~ |
## SpanRuler.\_\len\_\_ {#len tag="method"} ## SpanRuler.\_\_len\_\_ {id="len",tag="method"}
The number of all patterns added to the span ruler. The number of all patterns added to the span ruler.
@ -144,7 +144,7 @@ The number of all patterns added to the span ruler.
| ----------- | ------------------------------- | | ----------- | ------------------------------- |
| **RETURNS** | The number of patterns. ~~int~~ | | **RETURNS** | The number of patterns. ~~int~~ |
## SpanRuler.\_\_contains\_\_ {#contains tag="method"} ## SpanRuler.\_\_contains\_\_ {id="contains",tag="method"}
Whether a label is present in the patterns. Whether a label is present in the patterns.
@ -162,7 +162,7 @@ Whether a label is present in the patterns.
| `label` | The label to check. ~~str~~ | | `label` | The label to check. ~~str~~ |
| **RETURNS** | Whether the span ruler contains the label. ~~bool~~ | | **RETURNS** | Whether the span ruler contains the label. ~~bool~~ |
## SpanRuler.\_\_call\_\_ {#call tag="method"} ## SpanRuler.\_\_call\_\_ {id="call",tag="method"}
Find matches in the `Doc` and add them to `doc.spans[span_key]` and/or Find matches in the `Doc` and add them to `doc.spans[span_key]` and/or
`doc.ents`. Typically, this happens automatically after the component has been `doc.ents`. Typically, this happens automatically after the component has been
@ -186,7 +186,7 @@ will be removed.
| `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ | | `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ |
| **RETURNS** | The modified `Doc` with added spans/entities. ~~Doc~~ | | **RETURNS** | The modified `Doc` with added spans/entities. ~~Doc~~ |
## SpanRuler.add_patterns {#add_patterns tag="method"} ## SpanRuler.add_patterns {id="add_patterns",tag="method"}
Add patterns to the span ruler. A pattern can either be a token pattern (list of Add patterns to the span ruler. A pattern can either be a token pattern (list of
dicts) or a phrase pattern (string). For more details, see the usage guide on dicts) or a phrase pattern (string). For more details, see the usage guide on
@ -207,7 +207,7 @@ dicts) or a phrase pattern (string). For more details, see the usage guide on
| ---------- | ---------------------------------------------------------------- | | ---------- | ---------------------------------------------------------------- |
| `patterns` | The patterns to add. ~~List[Dict[str, Union[str, List[dict]]]]~~ | | `patterns` | The patterns to add. ~~List[Dict[str, Union[str, List[dict]]]]~~ |
## SpanRuler.remove {#remove tag="method"} ## SpanRuler.remove {id="remove",tag="method"}
Remove patterns by label from the span ruler. A `ValueError` is raised if the Remove patterns by label from the span ruler. A `ValueError` is raised if the
label does not exist in any patterns. label does not exist in any patterns.
@ -225,7 +225,7 @@ label does not exist in any patterns.
| ------- | -------------------------------------- | | ------- | -------------------------------------- |
| `label` | The label of the pattern rule. ~~str~~ | | `label` | The label of the pattern rule. ~~str~~ |
## SpanRuler.remove_by_id {#remove_by_id tag="method"} ## SpanRuler.remove_by_id {id="remove_by_id",tag="method"}
Remove patterns by ID from the span ruler. A `ValueError` is raised if the ID Remove patterns by ID from the span ruler. A `ValueError` is raised if the ID
does not exist in any patterns. does not exist in any patterns.
@ -243,7 +243,7 @@ does not exist in any patterns.
| ------------ | ----------------------------------- | | ------------ | ----------------------------------- |
| `pattern_id` | The ID of the pattern rule. ~~str~~ | | `pattern_id` | The ID of the pattern rule. ~~str~~ |
## SpanRuler.clear {#clear tag="method"} ## SpanRuler.clear {id="clear",tag="method"}
Remove all patterns the span ruler. Remove all patterns the span ruler.
@ -256,7 +256,7 @@ Remove all patterns the span ruler.
> ruler.clear() > ruler.clear()
> ``` > ```
## SpanRuler.to_disk {#to_disk tag="method"} ## SpanRuler.to_disk {id="to_disk",tag="method"}
Save the span ruler patterns to a directory. The patterns will be saved as Save the span ruler patterns to a directory. The patterns will be saved as
newline-delimited JSON (JSONL). newline-delimited JSON (JSONL).
@ -272,7 +272,7 @@ newline-delimited JSON (JSONL).
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## SpanRuler.from_disk {#from_disk tag="method"} ## SpanRuler.from_disk {id="from_disk",tag="method"}
Load the span ruler from a path. Load the span ruler from a path.
@ -288,7 +288,7 @@ Load the span ruler from a path.
| `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The modified `SpanRuler` object. ~~SpanRuler~~ | | **RETURNS** | The modified `SpanRuler` object. ~~SpanRuler~~ |
## SpanRuler.to_bytes {#to_bytes tag="method"} ## SpanRuler.to_bytes {id="to_bytes",tag="method"}
Serialize the span ruler to a bytestring. Serialize the span ruler to a bytestring.
@ -303,7 +303,7 @@ Serialize the span ruler to a bytestring.
| ----------- | ---------------------------------- | | ----------- | ---------------------------------- |
| **RETURNS** | The serialized patterns. ~~bytes~~ | | **RETURNS** | The serialized patterns. ~~bytes~~ |
## SpanRuler.from_bytes {#from_bytes tag="method"} ## SpanRuler.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -320,7 +320,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `bytes_data` | The bytestring to load. ~~bytes~~ | | `bytes_data` | The bytestring to load. ~~bytes~~ |
| **RETURNS** | The modified `SpanRuler` object. ~~SpanRuler~~ | | **RETURNS** | The modified `SpanRuler` object. ~~SpanRuler~~ |
## SpanRuler.labels {#labels tag="property"} ## SpanRuler.labels {id="labels",tag="property"}
All labels present in the match patterns. All labels present in the match patterns.
@ -328,7 +328,7 @@ All labels present in the match patterns.
| ----------- | -------------------------------------- | | ----------- | -------------------------------------- |
| **RETURNS** | The string labels. ~~Tuple[str, ...]~~ | | **RETURNS** | The string labels. ~~Tuple[str, ...]~~ |
## SpanRuler.ids {#ids tag="property"} ## SpanRuler.ids {id="ids",tag="property"}
All IDs present in the `id` property of the match patterns. All IDs present in the `id` property of the match patterns.
@ -336,7 +336,7 @@ All IDs present in the `id` property of the match patterns.
| ----------- | ----------------------------------- | | ----------- | ----------------------------------- |
| **RETURNS** | The string IDs. ~~Tuple[str, ...]~~ | | **RETURNS** | The string IDs. ~~Tuple[str, ...]~~ |
## SpanRuler.patterns {#patterns tag="property"} ## SpanRuler.patterns {id="patterns",tag="property"}
All patterns that were added to the span ruler. All patterns that were added to the span ruler.
@ -344,7 +344,7 @@ All patterns that were added to the span ruler.
| ----------- | ---------------------------------------------------------------------------------------- | | ----------- | ---------------------------------------------------------------------------------------- |
| **RETURNS** | The original patterns, one dictionary per pattern. ~~List[Dict[str, Union[str, dict]]]~~ | | **RETURNS** | The original patterns, one dictionary per pattern. ~~List[Dict[str, Union[str, dict]]]~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| ---------------- | -------------------------------------------------------------------------------- | | ---------------- | -------------------------------------------------------------------------------- |

View File

@ -8,7 +8,7 @@ Look up strings by 64-bit hashes. As of v2.0, spaCy uses hash values instead of
integer IDs. This ensures that strings always map to the same ID, even from integer IDs. This ensures that strings always map to the same ID, even from
different `StringStores`. different `StringStores`.
## StringStore.\_\_init\_\_ {#init tag="method"} ## StringStore.\_\_init\_\_ {id="init",tag="method"}
Create the `StringStore`. Create the `StringStore`.
@ -23,7 +23,7 @@ Create the `StringStore`.
| --------- | ---------------------------------------------------------------------- | | --------- | ---------------------------------------------------------------------- |
| `strings` | A sequence of strings to add to the store. ~~Optional[Iterable[str]]~~ | | `strings` | A sequence of strings to add to the store. ~~Optional[Iterable[str]]~~ |
## StringStore.\_\_len\_\_ {#len tag="method"} ## StringStore.\_\_len\_\_ {id="len",tag="method"}
Get the number of strings in the store. Get the number of strings in the store.
@ -38,7 +38,7 @@ Get the number of strings in the store.
| ----------- | ------------------------------------------- | | ----------- | ------------------------------------------- |
| **RETURNS** | The number of strings in the store. ~~int~~ | | **RETURNS** | The number of strings in the store. ~~int~~ |
## StringStore.\_\_getitem\_\_ {#getitem tag="method"} ## StringStore.\_\_getitem\_\_ {id="getitem",tag="method"}
Retrieve a string from a given hash, or vice versa. Retrieve a string from a given hash, or vice versa.
@ -56,7 +56,7 @@ Retrieve a string from a given hash, or vice versa.
| `string_or_id` | The value to encode. ~~Union[bytes, str, int]~~ | | `string_or_id` | The value to encode. ~~Union[bytes, str, int]~~ |
| **RETURNS** | The value to be retrieved. ~~Union[str, int]~~ | | **RETURNS** | The value to be retrieved. ~~Union[str, int]~~ |
## StringStore.\_\_contains\_\_ {#contains tag="method"} ## StringStore.\_\_contains\_\_ {id="contains",tag="method"}
Check whether a string is in the store. Check whether a string is in the store.
@ -73,7 +73,7 @@ Check whether a string is in the store.
| `string` | The string to check. ~~str~~ | | `string` | The string to check. ~~str~~ |
| **RETURNS** | Whether the store contains the string. ~~bool~~ | | **RETURNS** | Whether the store contains the string. ~~bool~~ |
## StringStore.\_\_iter\_\_ {#iter tag="method"} ## StringStore.\_\_iter\_\_ {id="iter",tag="method"}
Iterate over the strings in the store, in order. Note that a newly initialized Iterate over the strings in the store, in order. Note that a newly initialized
store will always include an empty string `""` at position `0`. store will always include an empty string `""` at position `0`.
@ -90,7 +90,7 @@ store will always include an empty string `""` at position `0`.
| ---------- | ------------------------------ | | ---------- | ------------------------------ |
| **YIELDS** | A string in the store. ~~str~~ | | **YIELDS** | A string in the store. ~~str~~ |
## StringStore.add {#add tag="method" new="2"} ## StringStore.add {id="add",tag="method",version="2"}
Add a string to the `StringStore`. Add a string to the `StringStore`.
@ -110,7 +110,7 @@ Add a string to the `StringStore`.
| `string` | The string to add. ~~str~~ | | `string` | The string to add. ~~str~~ |
| **RETURNS** | The string's hash value. ~~int~~ | | **RETURNS** | The string's hash value. ~~int~~ |
## StringStore.to_disk {#to_disk tag="method" new="2"} ## StringStore.to_disk {id="to_disk",tag="method",version="2"}
Save the current state to a directory. Save the current state to a directory.
@ -124,7 +124,7 @@ Save the current state to a directory.
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## StringStore.from_disk {#from_disk tag="method" new="2"} ## StringStore.from_disk {id="from_disk",tag="method",version="2"}
Loads state from a directory. Modifies the object in place and returns it. Loads state from a directory. Modifies the object in place and returns it.
@ -140,7 +140,7 @@ Loads state from a directory. Modifies the object in place and returns it.
| `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The modified `StringStore` object. ~~StringStore~~ | | **RETURNS** | The modified `StringStore` object. ~~StringStore~~ |
## StringStore.to_bytes {#to_bytes tag="method"} ## StringStore.to_bytes {id="to_bytes",tag="method"}
Serialize the current state to a binary string. Serialize the current state to a binary string.
@ -154,7 +154,7 @@ Serialize the current state to a binary string.
| ----------- | ---------------------------------------------------------- | | ----------- | ---------------------------------------------------------- |
| **RETURNS** | The serialized form of the `StringStore` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `StringStore` object. ~~bytes~~ |
## StringStore.from_bytes {#from_bytes tag="method"} ## StringStore.from_bytes {id="from_bytes",tag="method"}
Load state from a binary string. Load state from a binary string.
@ -171,9 +171,9 @@ Load state from a binary string.
| `bytes_data` | The data to load from. ~~bytes~~ | | `bytes_data` | The data to load from. ~~bytes~~ |
| **RETURNS** | The `StringStore` object. ~~StringStore~~ | | **RETURNS** | The `StringStore` object. ~~StringStore~~ |
## Utilities {#util} ## Utilities {id="util"}
### strings.hash_string {#hash_string tag="function"} ### strings.hash_string {id="hash_string",tag="function"}
Get a 64-bit hash for a given string. Get a 64-bit hash for a given string.

View File

@ -14,7 +14,7 @@ part-of-speech tag set.
In the pre-trained pipelines, the tag schemas vary by language; see the In the pre-trained pipelines, the tag schemas vary by language; see the
[individual model pages](/models) for details. [individual model pages](/models) for details.
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Predictions are assigned to `Token.tag`. Predictions are assigned to `Token.tag`.
@ -23,7 +23,7 @@ Predictions are assigned to `Token.tag`.
| `Token.tag` | The part of speech (hash). ~~int~~ | | `Token.tag` | The part of speech (hash). ~~int~~ |
| `Token.tag_` | The part of speech. ~~str~~ | | `Token.tag_` | The part of speech. ~~str~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -51,7 +51,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/tagger.pyx %%GITHUB_SPACY/spacy/pipeline/tagger.pyx
``` ```
## Tagger.\_\_init\_\_ {#init tag="method"} ## Tagger.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -81,7 +81,7 @@ shortcut for this and instantiate the component using its string name and
| `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ | | `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ |
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attribute `"tag"`. ~~Optional[Callable]~~ | | `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attribute `"tag"`. ~~Optional[Callable]~~ |
## Tagger.\_\_call\_\_ {#call tag="method"} ## Tagger.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned. Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -104,7 +104,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## Tagger.pipe {#pipe tag="method"} ## Tagger.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -127,7 +127,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/tagger#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## Tagger.initialize {#initialize tag="method" new="3"} ## Tagger.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -170,7 +170,7 @@ This method was previously called `begin_training`.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ |
## Tagger.predict {#predict tag="method"} ## Tagger.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. modifying them.
@ -187,7 +187,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. | | **RETURNS** | The model's prediction for each document. |
## Tagger.set_annotations {#set_annotations tag="method"} ## Tagger.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -204,7 +204,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `Tagger.predict`. | | `scores` | The scores to set, produced by `Tagger.predict`. |
## Tagger.update {#update tag="method"} ## Tagger.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model. predictions and gold-standard annotations, and update the component's model.
@ -228,7 +228,7 @@ Delegates to [`predict`](/api/tagger#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Tagger.rehearse {#rehearse tag="method,experimental" new="3"} ## Tagger.rehearse {id="rehearse",tag="method,experimental",version="3"}
Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the
current model to make predictions similar to an initial model, to try to address current model to make predictions similar to an initial model, to try to address
@ -251,7 +251,7 @@ the "catastrophic forgetting" problem. This feature is experimental.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Tagger.get_loss {#get_loss tag="method"} ## Tagger.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their Find the loss and gradient of loss for the batch of documents and their
predicted scores. predicted scores.
@ -270,7 +270,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. | | `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## Tagger.create_optimizer {#create_optimizer tag="method"} ## Tagger.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -285,7 +285,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## Tagger.use_params {#use_params tag="method, contextmanager"} ## Tagger.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -302,7 +302,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## Tagger.add_label {#add_label tag="method"} ## Tagger.add_label {id="add_label",tag="method"}
Add a new label to the pipe. Raises an error if the output dimension is already Add a new label to the pipe. Raises an error if the output dimension is already
set, or if the model has already been fully [initialized](#initialize). Note set, or if the model has already been fully [initialized](#initialize). Note
@ -324,7 +324,7 @@ automatically.
| `label` | The label to add. ~~str~~ | | `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## Tagger.to_disk {#to_disk tag="method"} ## Tagger.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -341,7 +341,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Tagger.from_disk {#from_disk tag="method"} ## Tagger.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -359,7 +359,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Tagger` object. ~~Tagger~~ | | **RETURNS** | The modified `Tagger` object. ~~Tagger~~ |
## Tagger.to_bytes {#to_bytes tag="method"} ## Tagger.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -376,7 +376,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Tagger` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `Tagger` object. ~~bytes~~ |
## Tagger.from_bytes {#from_bytes tag="method"} ## Tagger.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -395,7 +395,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Tagger` object. ~~Tagger~~ | | **RETURNS** | The `Tagger` object. ~~Tagger~~ |
## Tagger.labels {#labels tag="property"} ## Tagger.labels {id="labels",tag="property"}
The labels currently added to the component. The labels currently added to the component.
@ -410,7 +410,7 @@ The labels currently added to the component.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## Tagger.label_data {#label_data tag="property" new="3"} ## Tagger.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information. The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -428,7 +428,7 @@ pre-defined label set.
| ----------- | ---------------------------------------------------------- | | ----------- | ---------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -2,7 +2,7 @@
title: TextCategorizer title: TextCategorizer
tag: class tag: class
source: spacy/pipeline/textcat.py source: spacy/pipeline/textcat.py
new: 2 version: 2
teaser: 'Pipeline component for text classification' teaser: 'Pipeline component for text classification'
api_base_class: /api/pipe api_base_class: /api/pipe
api_string_name: textcat api_string_name: textcat
@ -29,7 +29,7 @@ only.
</Infobox> </Infobox>
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
Predictions will be saved to `doc.cats` as a dictionary, where the key is the Predictions will be saved to `doc.cats` as a dictionary, where the key is the
name of the category and the value is a score between 0 and 1 (inclusive). For name of the category and the value is a score between 0 and 1 (inclusive). For
@ -49,7 +49,7 @@ supported.
| ---------- | ------------------------------------- | | ---------- | ------------------------------------- |
| `Doc.cats` | Category scores. ~~Dict[str, float]~~ | | `Doc.cats` | Category scores. ~~Dict[str, float]~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -93,7 +93,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/textcat_multilabel.py %%GITHUB_SPACY/spacy/pipeline/textcat_multilabel.py
``` ```
## TextCategorizer.\_\_init\_\_ {#init tag="method"} ## TextCategorizer.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -125,7 +125,7 @@ shortcut for this and instantiate the component using its string name and
| `threshold` | Cutoff to consider a prediction "positive", relevant for `textcat_multilabel` when calculating accuracy scores. ~~float~~ | | `threshold` | Cutoff to consider a prediction "positive", relevant for `textcat_multilabel` when calculating accuracy scores. ~~float~~ |
| `scorer` | The scoring method. Defaults to [`Scorer.score_cats`](/api/scorer#score_cats) for the attribute `"cats"`. ~~Optional[Callable]~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_cats`](/api/scorer#score_cats) for the attribute `"cats"`. ~~Optional[Callable]~~ |
## TextCategorizer.\_\_call\_\_ {#call tag="method"} ## TextCategorizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned. Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -148,7 +148,7 @@ delegate to the [`predict`](/api/textcategorizer#predict) and
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## TextCategorizer.pipe {#pipe tag="method"} ## TextCategorizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -172,7 +172,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/textcategorizer#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## TextCategorizer.initialize {#initialize tag="method" new="3"} ## TextCategorizer.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -217,7 +217,7 @@ This method was previously called `begin_training`.
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ |
| `positive_label` | The positive label for a binary task with exclusive classes, `None` otherwise and by default. This parameter is only used during scoring. It is not available when using the `textcat_multilabel` component. ~~Optional[str]~~ | | `positive_label` | The positive label for a binary task with exclusive classes, `None` otherwise and by default. This parameter is only used during scoring. It is not available when using the `textcat_multilabel` component. ~~Optional[str]~~ |
## TextCategorizer.predict {#predict tag="method"} ## TextCategorizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects without Apply the component's model to a batch of [`Doc`](/api/doc) objects without
modifying them. modifying them.
@ -234,7 +234,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. | | **RETURNS** | The model's prediction for each document. |
## TextCategorizer.set_annotations {#set_annotations tag="method"} ## TextCategorizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores. Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores.
@ -251,7 +251,7 @@ Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `TextCategorizer.predict`. | | `scores` | The scores to set, produced by `TextCategorizer.predict`. |
## TextCategorizer.update {#update tag="method"} ## TextCategorizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model. predictions and gold-standard annotations, and update the component's model.
@ -275,7 +275,7 @@ Delegates to [`predict`](/api/textcategorizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## TextCategorizer.rehearse {#rehearse tag="method,experimental" new="3"} ## TextCategorizer.rehearse {id="rehearse",tag="method,experimental",version="3"}
Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the
current model to make predictions similar to an initial model to try to address current model to make predictions similar to an initial model to try to address
@ -298,7 +298,7 @@ the "catastrophic forgetting" problem. This feature is experimental.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## TextCategorizer.get_loss {#get_loss tag="method"} ## TextCategorizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their Find the loss and gradient of loss for the batch of documents and their
predicted scores. predicted scores.
@ -317,7 +317,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. | | `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## TextCategorizer.score {#score tag="method" new="3"} ## TextCategorizer.score {id="score",tag="method",version="3"}
Score a batch of examples. Score a batch of examples.
@ -333,7 +333,7 @@ Score a batch of examples.
| _keyword-only_ | | | _keyword-only_ | |
| **RETURNS** | The scores, produced by [`Scorer.score_cats`](/api/scorer#score_cats). ~~Dict[str, Union[float, Dict[str, float]]]~~ | | **RETURNS** | The scores, produced by [`Scorer.score_cats`](/api/scorer#score_cats). ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## TextCategorizer.create_optimizer {#create_optimizer tag="method"} ## TextCategorizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -348,7 +348,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## TextCategorizer.use_params {#use_params tag="method, contextmanager"} ## TextCategorizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model to use the given parameter values. Modify the pipe's model to use the given parameter values.
@ -364,7 +364,7 @@ Modify the pipe's model to use the given parameter values.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## TextCategorizer.add_label {#add_label tag="method"} ## TextCategorizer.add_label {id="add_label",tag="method"}
Add a new label to the pipe. Raises an error if the output dimension is already Add a new label to the pipe. Raises an error if the output dimension is already
set, or if the model has already been fully [initialized](#initialize). Note set, or if the model has already been fully [initialized](#initialize). Note
@ -386,7 +386,7 @@ automatically.
| `label` | The label to add. ~~str~~ | | `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## TextCategorizer.to_disk {#to_disk tag="method"} ## TextCategorizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -403,7 +403,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## TextCategorizer.from_disk {#from_disk tag="method"} ## TextCategorizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -421,7 +421,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `TextCategorizer` object. ~~TextCategorizer~~ | | **RETURNS** | The modified `TextCategorizer` object. ~~TextCategorizer~~ |
## TextCategorizer.to_bytes {#to_bytes tag="method"} ## TextCategorizer.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -438,7 +438,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `TextCategorizer` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `TextCategorizer` object. ~~bytes~~ |
## TextCategorizer.from_bytes {#from_bytes tag="method"} ## TextCategorizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -457,7 +457,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `TextCategorizer` object. ~~TextCategorizer~~ | | **RETURNS** | The `TextCategorizer` object. ~~TextCategorizer~~ |
## TextCategorizer.labels {#labels tag="property"} ## TextCategorizer.labels {id="labels",tag="property"}
The labels currently added to the component. The labels currently added to the component.
@ -472,7 +472,7 @@ The labels currently added to the component.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## TextCategorizer.label_data {#label_data tag="property" new="3"} ## TextCategorizer.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information. The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -490,7 +490,7 @@ the model with a pre-defined label set.
| ----------- | ---------------------------------------------------------- | | ----------- | ---------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ | | **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -1,7 +1,7 @@
--- ---
title: Tok2Vec title: Tok2Vec
source: spacy/pipeline/tok2vec.py source: spacy/pipeline/tok2vec.py
new: 3 version: 3
teaser: null teaser: null
api_base_class: /api/pipe api_base_class: /api/pipe
api_string_name: tok2vec api_string_name: tok2vec
@ -23,7 +23,7 @@ components can backpropagate to the shared weights. This implementation is used
because it allows us to avoid relying on object identity within the models to because it allows us to avoid relying on object identity within the models to
achieve the parameter sharing. achieve the parameter sharing.
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -48,7 +48,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/tok2vec.py %%GITHUB_SPACY/spacy/pipeline/tok2vec.py
``` ```
## Tok2Vec.\_\_init\_\_ {#init tag="method"} ## Tok2Vec.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -75,7 +75,7 @@ shortcut for this and instantiate the component using its string name and
| `model` | The Thinc [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. ~~Model[List[Doc], List[Floats2d]~~ | | `model` | The Thinc [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. ~~Model[List[Doc], List[Floats2d]~~ |
| `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ | | `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ |
## Tok2Vec.\_\_call\_\_ {#call tag="method"} ## Tok2Vec.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document and add context-sensitive embeddings to the Apply the pipe to one document and add context-sensitive embeddings to the
`Doc.tensor` attribute, allowing them to be used as features by downstream `Doc.tensor` attribute, allowing them to be used as features by downstream
@ -100,7 +100,7 @@ pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## Tok2Vec.pipe {#pipe tag="method"} ## Tok2Vec.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -123,7 +123,7 @@ and [`set_annotations`](/api/tok2vec#set_annotations) methods.
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## Tok2Vec.initialize {#initialize tag="method"} ## Tok2Vec.initialize {id="initialize",tag="method"}
Initialize the component for training and return an Initialize the component for training and return an
[`Optimizer`](https://thinc.ai/docs/api-optimizers). `get_examples` should be a [`Optimizer`](https://thinc.ai/docs/api-optimizers). `get_examples` should be a
@ -148,7 +148,7 @@ by [`Language.initialize`](/api/language#initialize).
| _keyword-only_ | | | _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## Tok2Vec.predict {#predict tag="method"} ## Tok2Vec.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects without Apply the component's model to a batch of [`Doc`](/api/doc) objects without
modifying them. modifying them.
@ -165,7 +165,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. | | **RETURNS** | The model's prediction for each document. |
## Tok2Vec.set_annotations {#set_annotations tag="method"} ## Tok2Vec.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -182,7 +182,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `Tok2Vec.predict`. | | `scores` | The scores to set, produced by `Tok2Vec.predict`. |
## Tok2Vec.update {#update tag="method"} ## Tok2Vec.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model. predictions and gold-standard annotations, and update the component's model.
@ -205,7 +205,7 @@ Delegates to [`predict`](/api/tok2vec#predict).
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Tok2Vec.create_optimizer {#create_optimizer tag="method"} ## Tok2Vec.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -220,7 +220,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## Tok2Vec.use_params {#use_params tag="method, contextmanager"} ## Tok2Vec.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model to use the given parameter values. At the end of the Modify the pipe's model to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -237,7 +237,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## Tok2Vec.to_disk {#to_disk tag="method"} ## Tok2Vec.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -254,7 +254,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Tok2Vec.from_disk {#from_disk tag="method"} ## Tok2Vec.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -272,7 +272,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Tok2Vec` object. ~~Tok2Vec~~ | | **RETURNS** | The modified `Tok2Vec` object. ~~Tok2Vec~~ |
## Tok2Vec.to_bytes {#to_bytes tag="method"} ## Tok2Vec.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -289,7 +289,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Tok2Vec` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `Tok2Vec` object. ~~bytes~~ |
## Tok2Vec.from_bytes {#from_bytes tag="method"} ## Tok2Vec.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -308,7 +308,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Tok2Vec` object. ~~Tok2Vec~~ | | **RETURNS** | The `Tok2Vec` object. ~~Tok2Vec~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -5,7 +5,7 @@ tag: class
source: spacy/tokens/token.pyx source: spacy/tokens/token.pyx
--- ---
## Token.\_\_init\_\_ {#init tag="method"} ## Token.\_\_init\_\_ {id="init",tag="method"}
Construct a `Token` object. Construct a `Token` object.
@ -23,7 +23,7 @@ Construct a `Token` object.
| `doc` | The parent document. ~~Doc~~ | | `doc` | The parent document. ~~Doc~~ |
| `offset` | The index of the token within the document. ~~int~~ | | `offset` | The index of the token within the document. ~~int~~ |
## Token.\_\_len\_\_ {#len tag="method"} ## Token.\_\_len\_\_ {id="len",tag="method"}
The number of unicode characters in the token, i.e. `token.text`. The number of unicode characters in the token, i.e. `token.text`.
@ -39,7 +39,7 @@ The number of unicode characters in the token, i.e. `token.text`.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | The number of unicode characters in the token. ~~int~~ | | **RETURNS** | The number of unicode characters in the token. ~~int~~ |
## Token.set_extension {#set_extension tag="classmethod" new="2"} ## Token.set_extension {id="set_extension",tag="classmethod",version="2"}
Define a custom attribute on the `Token` which becomes available via `Token._`. Define a custom attribute on the `Token` which becomes available via `Token._`.
For details, see the documentation on For details, see the documentation on
@ -64,7 +64,7 @@ For details, see the documentation on
| `setter` | Setter function that takes the `Token` and a value, and modifies the object. Is called when the user writes to the `Token._` attribute. ~~Optional[Callable[[Token, Any], None]]~~ | | `setter` | Setter function that takes the `Token` and a value, and modifies the object. Is called when the user writes to the `Token._` attribute. ~~Optional[Callable[[Token, Any], None]]~~ |
| `force` | Force overwriting existing attribute. ~~bool~~ | | `force` | Force overwriting existing attribute. ~~bool~~ |
## Token.get_extension {#get_extension tag="classmethod" new="2"} ## Token.get_extension {id="get_extension",tag="classmethod",version="2"}
Look up a previously registered extension by name. Returns a 4-tuple Look up a previously registered extension by name. Returns a 4-tuple
`(default, method, getter, setter)` if the extension is registered. Raises a `(default, method, getter, setter)` if the extension is registered. Raises a
@ -84,7 +84,7 @@ Look up a previously registered extension by name. Returns a 4-tuple
| `name` | Name of the extension. ~~str~~ | | `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Token.has_extension {#has_extension tag="classmethod" new="2"} ## Token.has_extension {id="has_extension",tag="classmethod",version="2"}
Check whether an extension has been registered on the `Token` class. Check whether an extension has been registered on the `Token` class.
@ -101,7 +101,7 @@ Check whether an extension has been registered on the `Token` class.
| `name` | Name of the extension to check. ~~str~~ | | `name` | Name of the extension to check. ~~str~~ |
| **RETURNS** | Whether the extension has been registered. ~~bool~~ | | **RETURNS** | Whether the extension has been registered. ~~bool~~ |
## Token.remove_extension {#remove_extension tag="classmethod" new=""2.0.11""} ## Token.remove_extension {id="remove_extension",tag="classmethod",version="2.0.11"}
Remove a previously registered extension. Remove a previously registered extension.
@ -119,7 +119,7 @@ Remove a previously registered extension.
| `name` | Name of the extension. ~~str~~ | | `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Token.check_flag {#check_flag tag="method"} ## Token.check_flag {id="check_flag",tag="method"}
Check the value of a boolean flag. Check the value of a boolean flag.
@ -137,7 +137,7 @@ Check the value of a boolean flag.
| `flag_id` | The attribute ID of the flag to check. ~~int~~ | | `flag_id` | The attribute ID of the flag to check. ~~int~~ |
| **RETURNS** | Whether the flag is set. ~~bool~~ | | **RETURNS** | Whether the flag is set. ~~bool~~ |
## Token.similarity {#similarity tag="method" model="vectors"} ## Token.similarity {id="similarity",tag="method",model="vectors"}
Compute a semantic similarity estimate. Defaults to cosine over vectors. Compute a semantic similarity estimate. Defaults to cosine over vectors.
@ -155,7 +155,7 @@ Compute a semantic similarity estimate. Defaults to cosine over vectors.
| other | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ | | other | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ |
| **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ | | **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ |
## Token.nbor {#nbor tag="method"} ## Token.nbor {id="nbor",tag="method"}
Get a neighboring token. Get a neighboring token.
@ -172,7 +172,7 @@ Get a neighboring token.
| `i` | The relative position of the token to get. Defaults to `1`. ~~int~~ | | `i` | The relative position of the token to get. Defaults to `1`. ~~int~~ |
| **RETURNS** | The token at position `self.doc[self.i+i]`. ~~Token~~ | | **RETURNS** | The token at position `self.doc[self.i+i]`. ~~Token~~ |
## Token.set_morph {#set_morph tag="method"} ## Token.set_morph {id="set_morph",tag="method"}
Set the morphological analysis from a UD FEATS string, hash value of a UD FEATS Set the morphological analysis from a UD FEATS string, hash value of a UD FEATS
string, features dict or `MorphAnalysis`. The value `None` can be used to reset string, features dict or `MorphAnalysis`. The value `None` can be used to reset
@ -191,7 +191,7 @@ the morph to an unset state.
| -------- | --------------------------------------------------------------------------------- | | -------- | --------------------------------------------------------------------------------- |
| features | The morphological features to set. ~~Union[int, dict, str, MorphAnalysis, None]~~ | | features | The morphological features to set. ~~Union[int, dict, str, MorphAnalysis, None]~~ |
## Token.has_morph {#has_morph tag="method"} ## Token.has_morph {id="has_morph",tag="method"}
Check whether the token has annotated morph information. Return `False` when the Check whether the token has annotated morph information. Return `False` when the
morph annotation is unset/missing. morph annotation is unset/missing.
@ -200,7 +200,7 @@ morph annotation is unset/missing.
| ----------- | --------------------------------------------- | | ----------- | --------------------------------------------- |
| **RETURNS** | Whether the morph annotation is set. ~~bool~~ | | **RETURNS** | Whether the morph annotation is set. ~~bool~~ |
## Token.is_ancestor {#is_ancestor tag="method" model="parser"} ## Token.is_ancestor {id="is_ancestor",tag="method",model="parser"}
Check whether this token is a parent, grandparent, etc. of another in the Check whether this token is a parent, grandparent, etc. of another in the
dependency tree. dependency tree.
@ -219,7 +219,7 @@ dependency tree.
| descendant | Another token. ~~Token~~ | | descendant | Another token. ~~Token~~ |
| **RETURNS** | Whether this token is the ancestor of the descendant. ~~bool~~ | | **RETURNS** | Whether this token is the ancestor of the descendant. ~~bool~~ |
## Token.ancestors {#ancestors tag="property" model="parser"} ## Token.ancestors {id="ancestors",tag="property",model="parser"}
A sequence of the token's syntactic ancestors (parents, grandparents, etc). A sequence of the token's syntactic ancestors (parents, grandparents, etc).
@ -237,7 +237,7 @@ A sequence of the token's syntactic ancestors (parents, grandparents, etc).
| ---------- | ------------------------------------------------------------------------------- | | ---------- | ------------------------------------------------------------------------------- |
| **YIELDS** | A sequence of ancestor tokens such that `ancestor.is_ancestor(self)`. ~~Token~~ | | **YIELDS** | A sequence of ancestor tokens such that `ancestor.is_ancestor(self)`. ~~Token~~ |
## Token.conjuncts {#conjuncts tag="property" model="parser"} ## Token.conjuncts {id="conjuncts",tag="property",model="parser"}
A tuple of coordinated tokens, not including the token itself. A tuple of coordinated tokens, not including the token itself.
@ -253,7 +253,7 @@ A tuple of coordinated tokens, not including the token itself.
| ----------- | --------------------------------------------- | | ----------- | --------------------------------------------- |
| **RETURNS** | The coordinated tokens. ~~Tuple[Token, ...]~~ | | **RETURNS** | The coordinated tokens. ~~Tuple[Token, ...]~~ |
## Token.children {#children tag="property" model="parser"} ## Token.children {id="children",tag="property",model="parser"}
A sequence of the token's immediate syntactic children. A sequence of the token's immediate syntactic children.
@ -269,7 +269,7 @@ A sequence of the token's immediate syntactic children.
| ---------- | ------------------------------------------------------- | | ---------- | ------------------------------------------------------- |
| **YIELDS** | A child token such that `child.head == self`. ~~Token~~ | | **YIELDS** | A child token such that `child.head == self`. ~~Token~~ |
## Token.lefts {#lefts tag="property" model="parser"} ## Token.lefts {id="lefts",tag="property",model="parser"}
The leftward immediate children of the word in the syntactic dependency parse. The leftward immediate children of the word in the syntactic dependency parse.
@ -285,7 +285,7 @@ The leftward immediate children of the word in the syntactic dependency parse.
| ---------- | ------------------------------------ | | ---------- | ------------------------------------ |
| **YIELDS** | A left-child of the token. ~~Token~~ | | **YIELDS** | A left-child of the token. ~~Token~~ |
## Token.rights {#rights tag="property" model="parser"} ## Token.rights {id="rights",tag="property",model="parser"}
The rightward immediate children of the word in the syntactic dependency parse. The rightward immediate children of the word in the syntactic dependency parse.
@ -301,7 +301,7 @@ The rightward immediate children of the word in the syntactic dependency parse.
| ---------- | ------------------------------------- | | ---------- | ------------------------------------- |
| **YIELDS** | A right-child of the token. ~~Token~~ | | **YIELDS** | A right-child of the token. ~~Token~~ |
## Token.n_lefts {#n_lefts tag="property" model="parser"} ## Token.n_lefts {id="n_lefts",tag="property",model="parser"}
The number of leftward immediate children of the word in the syntactic The number of leftward immediate children of the word in the syntactic
dependency parse. dependency parse.
@ -317,7 +317,7 @@ dependency parse.
| ----------- | ---------------------------------------- | | ----------- | ---------------------------------------- |
| **RETURNS** | The number of left-child tokens. ~~int~~ | | **RETURNS** | The number of left-child tokens. ~~int~~ |
## Token.n_rights {#n_rights tag="property" model="parser"} ## Token.n_rights {id="n_rights",tag="property",model="parser"}
The number of rightward immediate children of the word in the syntactic The number of rightward immediate children of the word in the syntactic
dependency parse. dependency parse.
@ -333,7 +333,7 @@ dependency parse.
| ----------- | ----------------------------------------- | | ----------- | ----------------------------------------- |
| **RETURNS** | The number of right-child tokens. ~~int~~ | | **RETURNS** | The number of right-child tokens. ~~int~~ |
## Token.subtree {#subtree tag="property" model="parser"} ## Token.subtree {id="subtree",tag="property",model="parser"}
A sequence containing the token and all the token's syntactic descendants. A sequence containing the token and all the token's syntactic descendants.
@ -349,7 +349,7 @@ A sequence containing the token and all the token's syntactic descendants.
| ---------- | ------------------------------------------------------------------------------------ | | ---------- | ------------------------------------------------------------------------------------ |
| **YIELDS** | A descendant token such that `self.is_ancestor(token)` or `token == self`. ~~Token~~ | | **YIELDS** | A descendant token such that `self.is_ancestor(token)` or `token == self`. ~~Token~~ |
## Token.has_vector {#has_vector tag="property" model="vectors"} ## Token.has_vector {id="has_vector",tag="property",model="vectors"}
A boolean value indicating whether a word vector is associated with the token. A boolean value indicating whether a word vector is associated with the token.
@ -365,7 +365,7 @@ A boolean value indicating whether a word vector is associated with the token.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | Whether the token has a vector data attached. ~~bool~~ | | **RETURNS** | Whether the token has a vector data attached. ~~bool~~ |
## Token.vector {#vector tag="property" model="vectors"} ## Token.vector {id="vector",tag="property",model="vectors"}
A real-valued meaning representation. A real-valued meaning representation.
@ -382,7 +382,7 @@ A real-valued meaning representation.
| ----------- | ----------------------------------------------------------------------------------------------- | | ----------- | ----------------------------------------------------------------------------------------------- |
| **RETURNS** | A 1-dimensional array representing the token's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | | **RETURNS** | A 1-dimensional array representing the token's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Token.vector_norm {#vector_norm tag="property" model="vectors"} ## Token.vector_norm {id="vector_norm",tag="property",model="vectors"}
The L2 norm of the token's vector representation. The L2 norm of the token's vector representation.
@ -401,7 +401,7 @@ The L2 norm of the token's vector representation.
| ----------- | --------------------------------------------------- | | ----------- | --------------------------------------------------- |
| **RETURNS** | The L2 norm of the vector representation. ~~float~~ | | **RETURNS** | The L2 norm of the vector representation. ~~float~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

View File

@ -20,7 +20,7 @@ The tokenizer is typically created automatically when a
like punctuation and special case rules from the like punctuation and special case rules from the
[`Language.Defaults`](/api/language#defaults) provided by the language subclass. [`Language.Defaults`](/api/language#defaults) provided by the language subclass.
## Tokenizer.\_\_init\_\_ {#init tag="method"} ## Tokenizer.\_\_init\_\_ {id="init",tag="method"}
Create a `Tokenizer` to create `Doc` objects given unicode text. For examples of Create a `Tokenizer` to create `Doc` objects given unicode text. For examples of
how to construct a custom tokenizer with different tokenization rules, see the how to construct a custom tokenizer with different tokenization rules, see the
@ -55,7 +55,7 @@ how to construct a custom tokenizer with different tokenization rules, see the
| `url_match` | A function matching the signature of `re.compile(string).match` to find token matches after considering prefixes and suffixes. ~~Optional[Callable[[str], Optional[Match]]]~~ | | `url_match` | A function matching the signature of `re.compile(string).match` to find token matches after considering prefixes and suffixes. ~~Optional[Callable[[str], Optional[Match]]]~~ |
| `faster_heuristics` <Tag variant="new">3.3.0</Tag> | Whether to restrict the final `Matcher`-based pass for rules to those containing affixes or space. Defaults to `True`. ~~bool~~ | | `faster_heuristics` <Tag variant="new">3.3.0</Tag> | Whether to restrict the final `Matcher`-based pass for rules to those containing affixes or space. Defaults to `True`. ~~bool~~ |
## Tokenizer.\_\_call\_\_ {#call tag="method"} ## Tokenizer.\_\_call\_\_ {id="call",tag="method"}
Tokenize a string. Tokenize a string.
@ -71,7 +71,7 @@ Tokenize a string.
| `string` | The string to tokenize. ~~str~~ | | `string` | The string to tokenize. ~~str~~ |
| **RETURNS** | A container for linguistic annotations. ~~Doc~~ | | **RETURNS** | A container for linguistic annotations. ~~Doc~~ |
## Tokenizer.pipe {#pipe tag="method"} ## Tokenizer.pipe {id="pipe",tag="method"}
Tokenize a stream of texts. Tokenize a stream of texts.
@ -89,7 +89,7 @@ Tokenize a stream of texts.
| `batch_size` | The number of texts to accumulate in an internal buffer. Defaults to `1000`. ~~int~~ | | `batch_size` | The number of texts to accumulate in an internal buffer. Defaults to `1000`. ~~int~~ |
| **YIELDS** | The tokenized `Doc` objects, in order. ~~Doc~~ | | **YIELDS** | The tokenized `Doc` objects, in order. ~~Doc~~ |
## Tokenizer.find_infix {#find_infix tag="method"} ## Tokenizer.find_infix {id="find_infix",tag="method"}
Find internal split points of the string. Find internal split points of the string.
@ -98,7 +98,7 @@ Find internal split points of the string.
| `string` | The string to split. ~~str~~ | | `string` | The string to split. ~~str~~ |
| **RETURNS** | A list of `re.MatchObject` objects that have `.start()` and `.end()` methods, denoting the placement of internal segment separators, e.g. hyphens. ~~List[Match]~~ | | **RETURNS** | A list of `re.MatchObject` objects that have `.start()` and `.end()` methods, denoting the placement of internal segment separators, e.g. hyphens. ~~List[Match]~~ |
## Tokenizer.find_prefix {#find_prefix tag="method"} ## Tokenizer.find_prefix {id="find_prefix",tag="method"}
Find the length of a prefix that should be segmented from the string, or `None` Find the length of a prefix that should be segmented from the string, or `None`
if no prefix rules match. if no prefix rules match.
@ -108,7 +108,7 @@ if no prefix rules match.
| `string` | The string to segment. ~~str~~ | | `string` | The string to segment. ~~str~~ |
| **RETURNS** | The length of the prefix if present, otherwise `None`. ~~Optional[int]~~ | | **RETURNS** | The length of the prefix if present, otherwise `None`. ~~Optional[int]~~ |
## Tokenizer.find_suffix {#find_suffix tag="method"} ## Tokenizer.find_suffix {id="find_suffix",tag="method"}
Find the length of a suffix that should be segmented from the string, or `None` Find the length of a suffix that should be segmented from the string, or `None`
if no suffix rules match. if no suffix rules match.
@ -118,7 +118,7 @@ if no suffix rules match.
| `string` | The string to segment. ~~str~~ | | `string` | The string to segment. ~~str~~ |
| **RETURNS** | The length of the suffix if present, otherwise `None`. ~~Optional[int]~~ | | **RETURNS** | The length of the suffix if present, otherwise `None`. ~~Optional[int]~~ |
## Tokenizer.add_special_case {#add_special_case tag="method"} ## Tokenizer.add_special_case {id="add_special_case",tag="method"}
Add a special-case tokenization rule. This mechanism is also used to add custom Add a special-case tokenization rule. This mechanism is also used to add custom
tokenizer exceptions to the language data. See the usage guide on the tokenizer exceptions to the language data. See the usage guide on the
@ -139,7 +139,7 @@ details and examples.
| `string` | The string to specially tokenize. ~~str~~ | | `string` | The string to specially tokenize. ~~str~~ |
| `token_attrs` | A sequence of dicts, where each dict describes a token and its attributes. The `ORTH` fields of the attributes must exactly match the string when they are concatenated. ~~Iterable[Dict[int, str]]~~ | | `token_attrs` | A sequence of dicts, where each dict describes a token and its attributes. The `ORTH` fields of the attributes must exactly match the string when they are concatenated. ~~Iterable[Dict[int, str]]~~ |
## Tokenizer.explain {#explain tag="method"} ## Tokenizer.explain {id="explain",tag="method"}
Tokenize a string with a slow debugging tokenizer that provides information Tokenize a string with a slow debugging tokenizer that provides information
about which tokenizer rule or pattern was matched for each token. The tokens about which tokenizer rule or pattern was matched for each token. The tokens
@ -158,7 +158,7 @@ produced are identical to `Tokenizer.__call__` except for whitespace tokens.
| `string` | The string to tokenize with the debugging tokenizer. ~~str~~ | | `string` | The string to tokenize with the debugging tokenizer. ~~str~~ |
| **RETURNS** | A list of `(pattern_string, token_string)` tuples. ~~List[Tuple[str, str]]~~ | | **RETURNS** | A list of `(pattern_string, token_string)` tuples. ~~List[Tuple[str, str]]~~ |
## Tokenizer.to_disk {#to_disk tag="method"} ## Tokenizer.to_disk {id="to_disk",tag="method"}
Serialize the tokenizer to disk. Serialize the tokenizer to disk.
@ -175,7 +175,7 @@ Serialize the tokenizer to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Tokenizer.from_disk {#from_disk tag="method"} ## Tokenizer.from_disk {id="from_disk",tag="method"}
Load the tokenizer from disk. Modifies the object in place and returns it. Load the tokenizer from disk. Modifies the object in place and returns it.
@ -193,7 +193,7 @@ Load the tokenizer from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Tokenizer` object. ~~Tokenizer~~ | | **RETURNS** | The modified `Tokenizer` object. ~~Tokenizer~~ |
## Tokenizer.to_bytes {#to_bytes tag="method"} ## Tokenizer.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -210,7 +210,7 @@ Serialize the tokenizer to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Tokenizer` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `Tokenizer` object. ~~bytes~~ |
## Tokenizer.from_bytes {#from_bytes tag="method"} ## Tokenizer.from_bytes {id="from_bytes",tag="method"}
Load the tokenizer from a bytestring. Modifies the object in place and returns Load the tokenizer from a bytestring. Modifies the object in place and returns
it. it.
@ -230,7 +230,7 @@ it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Tokenizer` object. ~~Tokenizer~~ | | **RETURNS** | The `Tokenizer` object. ~~Tokenizer~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@ -241,7 +241,7 @@ it.
| `token_match` | A function matching the signature of `re.compile(string).match` to find token matches. Returns an `re.MatchObject` or `None`. ~~Optional[Callable[[str], Optional[Match]]]~~ | | `token_match` | A function matching the signature of `re.compile(string).match` to find token matches. Returns an `re.MatchObject` or `None`. ~~Optional[Callable[[str], Optional[Match]]]~~ |
| `rules` | A dictionary of tokenizer exceptions and special cases. ~~Optional[Dict[str, List[Dict[int, str]]]]~~ | | `rules` | A dictionary of tokenizer exceptions and special cases. ~~Optional[Dict[str, List[Dict[int, str]]]]~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -13,9 +13,9 @@ menu:
- ['Utility Functions', 'util'] - ['Utility Functions', 'util']
--- ---
## spaCy {#spacy hidden="true"} ## spaCy {id="spacy",hidden="true"}
### spacy.load {#spacy.load tag="function"} ### spacy.load {id="spacy.load",tag="function"}
Load a pipeline using the name of an installed Load a pipeline using the name of an installed
[package](/usage/saving-loading#models), a string path or a `Path`-like object. [package](/usage/saving-loading#models), a string path or a `Path`-like object.
@ -61,8 +61,7 @@ Essentially, `spacy.load()` is a convenience wrapper that reads the pipeline's
information to construct a `Language` object, loads in the model data and information to construct a `Language` object, loads in the model data and
weights, and returns it. weights, and returns it.
```python ```python {title="Abstract example"}
### Abstract example
cls = spacy.util.get_lang_class(lang) # 1. Get Language class, e.g. English cls = spacy.util.get_lang_class(lang) # 1. Get Language class, e.g. English
nlp = cls() # 2. Initialize it nlp = cls() # 2. Initialize it
for name in pipeline: for name in pipeline:
@ -70,7 +69,7 @@ for name in pipeline:
nlp.from_disk(data_path) # 4. Load in the binary data nlp.from_disk(data_path) # 4. Load in the binary data
``` ```
### spacy.blank {#spacy.blank tag="function" new="2"} ### spacy.blank {id="spacy.blank",tag="function",version="2"}
Create a blank pipeline of a given language class. This function is the twin of Create a blank pipeline of a given language class. This function is the twin of
`spacy.load()`. `spacy.load()`.
@ -91,7 +90,7 @@ Create a blank pipeline of a given language class. This function is the twin of
| `meta` | Optional meta overrides for [`nlp.meta`](/api/language#meta). ~~Dict[str, Any]~~ | | `meta` | Optional meta overrides for [`nlp.meta`](/api/language#meta). ~~Dict[str, Any]~~ |
| **RETURNS** | An empty `Language` object of the appropriate subclass. ~~Language~~ | | **RETURNS** | An empty `Language` object of the appropriate subclass. ~~Language~~ |
### spacy.info {#spacy.info tag="function"} ### spacy.info {id="spacy.info",tag="function"}
The same as the [`info` command](/api/cli#info). Pretty-print information about The same as the [`info` command](/api/cli#info). Pretty-print information about
your installation, installed pipelines and local setup from within spaCy. your installation, installed pipelines and local setup from within spaCy.
@ -111,7 +110,7 @@ your installation, installed pipelines and local setup from within spaCy.
| `markdown` | Print information as Markdown. ~~bool~~ | | `markdown` | Print information as Markdown. ~~bool~~ |
| `silent` | Don't print anything, just return. ~~bool~~ | | `silent` | Don't print anything, just return. ~~bool~~ |
### spacy.explain {#spacy.explain tag="function"} ### spacy.explain {id="spacy.explain",tag="function"}
Get a description for a given POS tag, dependency label or entity type. For a Get a description for a given POS tag, dependency label or entity type. For a
list of available terms, see [`glossary.py`](%%GITHUB_SPACY/spacy/glossary.py). list of available terms, see [`glossary.py`](%%GITHUB_SPACY/spacy/glossary.py).
@ -134,7 +133,7 @@ list of available terms, see [`glossary.py`](%%GITHUB_SPACY/spacy/glossary.py).
| `term` | Term to explain. ~~str~~ | | `term` | Term to explain. ~~str~~ |
| **RETURNS** | The explanation, or `None` if not found in the glossary. ~~Optional[str]~~ | | **RETURNS** | The explanation, or `None` if not found in the glossary. ~~Optional[str]~~ |
### spacy.prefer_gpu {#spacy.prefer_gpu tag="function" new="2.0.14"} ### spacy.prefer_gpu {id="spacy.prefer_gpu",tag="function",version="2.0.14"}
Allocate data and perform operations on [GPU](/usage/#gpu), if available. If Allocate data and perform operations on [GPU](/usage/#gpu), if available. If
data has already been allocated on CPU, it will not be moved. Ideally, this data has already been allocated on CPU, it will not be moved. Ideally, this
@ -162,7 +161,7 @@ ensure that the model is loaded on the correct device. See
| `gpu_id` | Device index to select. Defaults to `0`. ~~int~~ | | `gpu_id` | Device index to select. Defaults to `0`. ~~int~~ |
| **RETURNS** | Whether the GPU was activated. ~~bool~~ | | **RETURNS** | Whether the GPU was activated. ~~bool~~ |
### spacy.require_gpu {#spacy.require_gpu tag="function" new="2.0.14"} ### spacy.require_gpu {id="spacy.require_gpu",tag="function",version="2.0.14"}
Allocate data and perform operations on [GPU](/usage/#gpu). Will raise an error Allocate data and perform operations on [GPU](/usage/#gpu). Will raise an error
if no GPU is available. If data has already been allocated on CPU, it will not if no GPU is available. If data has already been allocated on CPU, it will not
@ -190,7 +189,7 @@ ensure that the model is loaded on the correct device. See
| `gpu_id` | Device index to select. Defaults to `0`. ~~int~~ | | `gpu_id` | Device index to select. Defaults to `0`. ~~int~~ |
| **RETURNS** | `True` ~~bool~~ | | **RETURNS** | `True` ~~bool~~ |
### spacy.require_cpu {#spacy.require_cpu tag="function" new="3.0.0"} ### spacy.require_cpu {id="spacy.require_cpu",tag="function",version="3.0.0"}
Allocate data and perform operations on CPU. If data has already been allocated Allocate data and perform operations on CPU. If data has already been allocated
on GPU, it will not be moved. Ideally, this function should be called right on GPU, it will not be moved. Ideally, this function should be called right
@ -216,12 +215,12 @@ ensure that the model is loaded on the correct device. See
| ----------- | --------------- | | ----------- | --------------- |
| **RETURNS** | `True` ~~bool~~ | | **RETURNS** | `True` ~~bool~~ |
## displaCy {#displacy source="spacy/displacy"} ## displaCy {id="displacy",source="spacy/displacy"}
As of v2.0, spaCy comes with a built-in visualization suite. For more info and As of v2.0, spaCy comes with a built-in visualization suite. For more info and
examples, see the usage guide on [visualizing spaCy](/usage/visualizers). examples, see the usage guide on [visualizing spaCy](/usage/visualizers).
### displacy.serve {#displacy.serve tag="method" new="2"} ### displacy.serve {id="displacy.serve",tag="method",version="2"}
Serve a dependency parse tree or named entity visualization to view it in your Serve a dependency parse tree or named entity visualization to view it in your
browser. Will run a simple web server. browser. Will run a simple web server.
@ -249,7 +248,7 @@ browser. Will run a simple web server.
| `host` | Host to serve visualization. Defaults to `"0.0.0.0"`. ~~str~~ | | `host` | Host to serve visualization. Defaults to `"0.0.0.0"`. ~~str~~ |
| `auto_select_port` | If `True`, automatically switch to a different port if the specified port is already in use. Defaults to `False`. ~~bool~~ | | `auto_select_port` | If `True`, automatically switch to a different port if the specified port is already in use. Defaults to `False`. ~~bool~~ |
### displacy.render {#displacy.render tag="method" new="2"} ### displacy.render {id="displacy.render",tag="method",version="2"}
Render a dependency parse tree or named entity visualization. Render a dependency parse tree or named entity visualization.
@ -274,7 +273,7 @@ Render a dependency parse tree or named entity visualization.
| `jupyter` | Explicitly enable or disable "[Jupyter](http://jupyter.org/) mode" to return markup ready to be rendered in a notebook. Detected automatically if `None` (default). ~~Optional[bool]~~ | | `jupyter` | Explicitly enable or disable "[Jupyter](http://jupyter.org/) mode" to return markup ready to be rendered in a notebook. Detected automatically if `None` (default). ~~Optional[bool]~~ |
| **RETURNS** | The rendered HTML markup. ~~str~~ | | **RETURNS** | The rendered HTML markup. ~~str~~ |
### displacy.parse_deps {#displacy.parse_deps tag="method" new="2"} ### displacy.parse_deps {id="displacy.parse_deps",tag="method",version="2"}
Generate dependency parse in `{'words': [], 'arcs': []}` format. For use with Generate dependency parse in `{'words': [], 'arcs': []}` format. For use with
the `manual=True` argument in `displacy.render`. the `manual=True` argument in `displacy.render`.
@ -296,7 +295,7 @@ the `manual=True` argument in `displacy.render`.
| `options` | Dependency parse specific visualisation options. ~~Dict[str, Any]~~ | | `options` | Dependency parse specific visualisation options. ~~Dict[str, Any]~~ |
| **RETURNS** | Generated dependency parse keyed by words and arcs. ~~dict~~ | | **RETURNS** | Generated dependency parse keyed by words and arcs. ~~dict~~ |
### displacy.parse_ents {#displacy.parse_ents tag="method" new="2"} ### displacy.parse_ents {id="displacy.parse_ents",tag="method",version="2"}
Generate named entities in `[{start: i, end: i, label: 'label'}]` format. For Generate named entities in `[{start: i, end: i, label: 'label'}]` format. For
use with the `manual=True` argument in `displacy.render`. use with the `manual=True` argument in `displacy.render`.
@ -318,7 +317,7 @@ use with the `manual=True` argument in `displacy.render`.
| `options` | NER-specific visualisation options. ~~Dict[str, Any]~~ | | `options` | NER-specific visualisation options. ~~Dict[str, Any]~~ |
| **RETURNS** | Generated entities keyed by text (original text) and ents. ~~dict~~ | | **RETURNS** | Generated entities keyed by text (original text) and ents. ~~dict~~ |
### displacy.parse_spans {#displacy.parse_spans tag="method" new="2"} ### displacy.parse_spans {id="displacy.parse_spans",tag="method",version="2"}
Generate spans in `[{start_token: i, end_token: i, label: 'label'}]` format. For Generate spans in `[{start_token: i, end_token: i, label: 'label'}]` format. For
use with the `manual=True` argument in `displacy.render`. use with the `manual=True` argument in `displacy.render`.
@ -341,12 +340,12 @@ use with the `manual=True` argument in `displacy.render`.
| `options` | Span-specific visualisation options. ~~Dict[str, Any]~~ | | `options` | Span-specific visualisation options. ~~Dict[str, Any]~~ |
| **RETURNS** | Generated entities keyed by text (original text) and ents. ~~dict~~ | | **RETURNS** | Generated entities keyed by text (original text) and ents. ~~dict~~ |
### Visualizer options {#displacy_options} ### Visualizer options {id="displacy_options"}
The `options` argument lets you specify additional settings for each visualizer. The `options` argument lets you specify additional settings for each visualizer.
If a setting is not present in the options, the default value will be used. If a setting is not present in the options, the default value will be used.
#### Dependency Visualizer options {#options-dep} #### Dependency Visualizer options {id="options-dep"}
> #### Example > #### Example
> >
@ -372,7 +371,7 @@ If a setting is not present in the options, the default value will be used.
| `word_spacing` | Vertical spacing between words and arcs in px. Defaults to `45`. ~~int~~ | | `word_spacing` | Vertical spacing between words and arcs in px. Defaults to `45`. ~~int~~ |
| `distance` | Distance between words in px. Defaults to `175` in regular mode and `150` in compact mode. ~~int~~ | | `distance` | Distance between words in px. Defaults to `175` in regular mode and `150` in compact mode. ~~int~~ |
#### Named Entity Visualizer options {#displacy_options-ent} #### Named Entity Visualizer options {id="displacy_options-ent"}
> #### Example > #### Example
> >
@ -389,7 +388,7 @@ If a setting is not present in the options, the default value will be used.
| `template` | Optional template to overwrite the HTML used to render entity spans. Should be a format string and can use `{bg}`, `{text}` and `{label}`. See [`templates.py`](%%GITHUB_SPACY/spacy/displacy/templates.py) for examples. ~~Optional[str]~~ | | `template` | Optional template to overwrite the HTML used to render entity spans. Should be a format string and can use `{bg}`, `{text}` and `{label}`. See [`templates.py`](%%GITHUB_SPACY/spacy/displacy/templates.py) for examples. ~~Optional[str]~~ |
| `kb_url_template` <Tag variant="new">3.2.1</Tag> | Optional template to construct the KB url for the entity to link to. Expects a python f-string format with single field to fill in. ~~Optional[str]~~ | | `kb_url_template` <Tag variant="new">3.2.1</Tag> | Optional template to construct the KB url for the entity to link to. Expects a python f-string format with single field to fill in. ~~Optional[str]~~ |
#### Span Visualizer options {#displacy_options-span} #### Span Visualizer options {id="displacy_options-span"}
> #### Example > #### Example
> >
@ -420,7 +419,7 @@ span. If you wish to link an entity to their URL then consider using the
should redirect you to their Wikidata page, in this case should redirect you to their Wikidata page, in this case
`https://www.wikidata.org/wiki/Q95`. `https://www.wikidata.org/wiki/Q95`.
## registry {#registry source="spacy/util.py" new="3"} ## registry {id="registry",source="spacy/util.py",version="3"}
spaCy's function registry extends spaCy's function registry extends
[Thinc's `registry`](https://thinc.ai/docs/api-config#registry) and allows you [Thinc's `registry`](https://thinc.ai/docs/api-config#registry) and allows you
@ -470,7 +469,7 @@ factories.
| `scorers` | Registry for functions that create scoring methods for user with the [`Scorer`](/api/scorer). Scoring methods are called with `Iterable[Example]` and arbitrary `\*\*kwargs` and return scores as `Dict[str, Any]`. | | `scorers` | Registry for functions that create scoring methods for user with the [`Scorer`](/api/scorer). Scoring methods are called with `Iterable[Example]` and arbitrary `\*\*kwargs` and return scores as `Dict[str, Any]`. |
| `tokenizers` | Registry for tokenizer factories. Registered functions should return a callback that receives the `nlp` object and returns a [`Tokenizer`](/api/tokenizer) or a custom callable. | | `tokenizers` | Registry for tokenizer factories. Registered functions should return a callback that receives the `nlp` object and returns a [`Tokenizer`](/api/tokenizer) or a custom callable. |
### spacy-transformers registry {#registry-transformers} ### spacy-transformers registry {id="registry-transformers"}
The following registries are added by the The following registries are added by the
[`spacy-transformers`](https://github.com/explosion/spacy-transformers) package. [`spacy-transformers`](https://github.com/explosion/spacy-transformers) package.
@ -495,7 +494,7 @@ See the [`Transformer`](/api/transformer) API reference and
| [`span_getters`](/api/transformer#span_getters) | Registry for functions that take a batch of `Doc` objects and return a list of `Span` objects to process by the transformer, e.g. sentences. | | [`span_getters`](/api/transformer#span_getters) | Registry for functions that take a batch of `Doc` objects and return a list of `Span` objects to process by the transformer, e.g. sentences. |
| [`annotation_setters`](/api/transformer#annotation_setters) | Registry for functions that create annotation setters. Annotation setters are functions that take a batch of `Doc` objects and a [`FullTransformerBatch`](/api/transformer#fulltransformerbatch) and can set additional annotations on the `Doc`. | | [`annotation_setters`](/api/transformer#annotation_setters) | Registry for functions that create annotation setters. Annotation setters are functions that take a batch of `Doc` objects and a [`FullTransformerBatch`](/api/transformer#fulltransformerbatch) and can set additional annotations on the `Doc`. |
## Loggers {#loggers source="spacy/training/loggers.py" new="3"} ## Loggers {id="loggers",source="spacy/training/loggers.py",version="3"}
A logger records the training results. When a logger is created, two functions A logger records the training results. When a logger is created, two functions
are returned: one for logging the information for each training step, and a are returned: one for logging the information for each training step, and a
@ -531,7 +530,7 @@ saves them to a `jsonl` file.
<Accordion title="Example console output" spaced> <Accordion title="Example console output" spaced>
```cli ```bash
$ python -m spacy train config.cfg $ python -m spacy train config.cfg
``` ```
@ -571,7 +570,7 @@ start decreasing across epochs.
| `console_output` | Whether the logger should print the logs in the console (default: `True`). ~~bool~~ | | `console_output` | Whether the logger should print the logs in the console (default: `True`). ~~bool~~ |
| `output_file` | The file to save the training logs to (default: `None`). ~~Optional[Union[str, Path]]~~ | | `output_file` | The file to save the training logs to (default: `None`). ~~Optional[Union[str, Path]]~~ |
#### spacy.ConsoleLogger.v3 {#ConsoleLogger tag="registered function"} #### spacy.ConsoleLogger.v3 {id="ConsoleLogger",tag="registered function"}
> #### Example config > #### Example config
> >
@ -593,9 +592,9 @@ optionally saves them to a `jsonl` file.
| `console_output` | Whether the logger should print the logs in the console (default: `True`). ~~bool~~ | | `console_output` | Whether the logger should print the logs in the console (default: `True`). ~~bool~~ |
| `output_file` | The file to save the training logs to (default: `None`). ~~Optional[Union[str, Path]]~~ | | `output_file` | The file to save the training logs to (default: `None`). ~~Optional[Union[str, Path]]~~ |
## Readers {#readers} ## Readers {id="readers"}
### File readers {#file-readers source="github.com/explosion/srsly" new="3"} ### File readers {id="file-readers",source="github.com/explosion/srsly",version="3"}
The following file readers are provided by our serialization library The following file readers are provided by our serialization library
[`srsly`](https://github.com/explosion/srsly). All registered functions take one [`srsly`](https://github.com/explosion/srsly). All registered functions take one
@ -625,7 +624,7 @@ blocks that are **not executed at runtime** for example, in `[training]` and
</Infobox> </Infobox>
#### spacy.read_labels.v1 {#read_labels tag="registered function"} #### spacy.read_labels.v1 {id="read_labels",tag="registered function"}
Read a JSON-formatted labels file generated with Read a JSON-formatted labels file generated with
[`init labels`](/api/cli#init-labels). Typically used in the [`init labels`](/api/cli#init-labels). Typically used in the
@ -651,7 +650,7 @@ label sets.
| `require` | Whether to require the file to exist. If set to `False` and the labels file doesn't exist, the loader will return `None` and the `initialize` method will extract the labels from the data. Defaults to `False`. ~~bool~~ | | `require` | Whether to require the file to exist. If set to `False` and the labels file doesn't exist, the loader will return `None` and the `initialize` method will extract the labels from the data. Defaults to `False`. ~~bool~~ |
| **CREATES** | The list of labels. ~~List[str]~~ | | **CREATES** | The list of labels. ~~List[str]~~ |
### Corpus readers {#corpus-readers source="spacy/training/corpus.py" new="3"} ### Corpus readers {id="corpus-readers",source="spacy/training/corpus.py",version="3"}
Corpus readers are registered functions that load data and return a function Corpus readers are registered functions that load data and return a function
that takes the current `nlp` object and yields [`Example`](/api/example) objects that takes the current `nlp` object and yields [`Example`](/api/example) objects
@ -661,7 +660,7 @@ with your own registered function in the
[`@readers` registry](/api/top-level#registry) to customize the data loading and [`@readers` registry](/api/top-level#registry) to customize the data loading and
streaming. streaming.
#### spacy.Corpus.v1 {#corpus tag="registered function"} #### spacy.Corpus.v1 {id="corpus",tag="registered function"}
The `Corpus` reader manages annotated corpora and can be used for training and The `Corpus` reader manages annotated corpora and can be used for training and
development datasets in the [DocBin](/api/docbin) (`.spacy`) format. Also see development datasets in the [DocBin](/api/docbin) (`.spacy`) format. Also see
@ -690,7 +689,7 @@ the [`Corpus`](/api/corpus) class.
| `augmenter` | Apply some simply data augmentation, where we replace tokens with variations. This is especially useful for punctuation and case replacement, to help generalize beyond corpora that don't have smart-quotes, or only have smart quotes, etc. Defaults to `None`. ~~Optional[Callable]~~ | | `augmenter` | Apply some simply data augmentation, where we replace tokens with variations. This is especially useful for punctuation and case replacement, to help generalize beyond corpora that don't have smart-quotes, or only have smart quotes, etc. Defaults to `None`. ~~Optional[Callable]~~ |
| **CREATES** | The corpus reader. ~~Corpus~~ | | **CREATES** | The corpus reader. ~~Corpus~~ |
#### spacy.JsonlCorpus.v1 {#jsonlcorpus tag="registered function"} #### spacy.JsonlCorpus.v1 {id="jsonlcorpus",tag="registered function"}
Create [`Example`](/api/example) objects from a JSONL (newline-delimited JSON) Create [`Example`](/api/example) objects from a JSONL (newline-delimited JSON)
file of texts keyed by `"text"`. Can be used to read the raw text corpus for file of texts keyed by `"text"`. Can be used to read the raw text corpus for
@ -719,7 +718,7 @@ JSONL file. Also see the [`JsonlCorpus`](/api/corpus#jsonlcorpus) class.
| `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ | | `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ |
| **CREATES** | The corpus reader. ~~JsonlCorpus~~ | | **CREATES** | The corpus reader. ~~JsonlCorpus~~ |
## Batchers {#batchers source="spacy/training/batchers.py" new="3"} ## Batchers {id="batchers",source="spacy/training/batchers.py",version="3"}
A data batcher implements a batching strategy that essentially turns a stream of A data batcher implements a batching strategy that essentially turns a stream of
items into a stream of batches, with each batch consisting of one item or a list items into a stream of batches, with each batch consisting of one item or a list
@ -733,7 +732,7 @@ Instead of using one of the built-in batchers listed here, you can also
[implement your own](/usage/training#custom-code-readers-batchers), which may or [implement your own](/usage/training#custom-code-readers-batchers), which may or
may not use a custom schedule. may not use a custom schedule.
### spacy.batch_by_words.v1 {#batch_by_words tag="registered function"} ### spacy.batch_by_words.v1 {id="batch_by_words",tag="registered function"}
Create minibatches of roughly a given number of words. If any examples are Create minibatches of roughly a given number of words. If any examples are
longer than the specified batch length, they will appear in a batch by longer than the specified batch length, they will appear in a batch by
@ -761,7 +760,7 @@ themselves, or be discarded if `discard_oversize` is set to `True`. The argument
| `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ | | `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ |
| **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ | | **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ |
### spacy.batch_by_sequence.v1 {#batch_by_sequence tag="registered function"} ### spacy.batch_by_sequence.v1 {id="batch_by_sequence",tag="registered function"}
> #### Example config > #### Example config
> >
@ -780,7 +779,7 @@ Create a batcher that creates batches of the specified size.
| `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ | | `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ |
| **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ | | **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ |
### spacy.batch_by_padded.v1 {#batch_by_padded tag="registered function"} ### spacy.batch_by_padded.v1 {id="batch_by_padded",tag="registered function"}
> #### Example config > #### Example config
> >
@ -806,7 +805,7 @@ sequences in the batch.
| `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ | | `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ |
| **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ | | **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ |
## Augmenters {#augmenters source="spacy/training/augment.py" new="3"} ## Augmenters {id="augmenters",source="spacy/training/augment.py",version="3"}
Data augmentation is the process of applying small modifications to the training Data augmentation is the process of applying small modifications to the training
data. It can be especially useful for punctuation and case replacement for data. It can be especially useful for punctuation and case replacement for
@ -815,7 +814,7 @@ variations using regular quotes, or to make the model less sensitive to
capitalization by including a mix of capitalized and lowercase examples. See the capitalization by including a mix of capitalized and lowercase examples. See the
[usage guide](/usage/training#data-augmentation) for details and examples. [usage guide](/usage/training#data-augmentation) for details and examples.
### spacy.orth_variants.v1 {#orth_variants tag="registered function"} ### spacy.orth_variants.v1 {id="orth_variants",tag="registered function"}
> #### Example config > #### Example config
> >
@ -842,7 +841,7 @@ beyond corpora that don't have smart quotes, or only have smart quotes etc.
| `orth_variants` | A dictionary containing the single and paired orth variants. Typically loaded from a JSON file. See [`en_orth_variants.json`](https://github.com/explosion/spacy-lookups-data/blob/master/spacy_lookups_data/data/en_orth_variants.json) for an example. ~~Dict[str, Dict[List[Union[str, List[str]]]]]~~ | | `orth_variants` | A dictionary containing the single and paired orth variants. Typically loaded from a JSON file. See [`en_orth_variants.json`](https://github.com/explosion/spacy-lookups-data/blob/master/spacy_lookups_data/data/en_orth_variants.json) for an example. ~~Dict[str, Dict[List[Union[str, List[str]]]]]~~ |
| **CREATES** | A function that takes the current `nlp` object and an [`Example`](/api/example) and yields augmented `Example` objects. ~~Callable[[Language, Example], Iterator[Example]]~~ | | **CREATES** | A function that takes the current `nlp` object and an [`Example`](/api/example) and yields augmented `Example` objects. ~~Callable[[Language, Example], Iterator[Example]]~~ |
### spacy.lower_case.v1 {#lower_case tag="registered function"} ### spacy.lower_case.v1 {id="lower_case",tag="registered function"}
> #### Example config > #### Example config
> >
@ -861,12 +860,12 @@ useful for making the model less sensitive to capitalization.
| `level` | The percentage of texts that will be augmented. ~~float~~ | | `level` | The percentage of texts that will be augmented. ~~float~~ |
| **CREATES** | A function that takes the current `nlp` object and an [`Example`](/api/example) and yields augmented `Example` objects. ~~Callable[[Language, Example], Iterator[Example]]~~ | | **CREATES** | A function that takes the current `nlp` object and an [`Example`](/api/example) and yields augmented `Example` objects. ~~Callable[[Language, Example], Iterator[Example]]~~ |
## Callbacks {#callbacks source="spacy/training/callbacks.py" new="3"} ## Callbacks {id="callbacks",source="spacy/training/callbacks.py",version="3"}
The config supports [callbacks](/usage/training#custom-code-nlp-callbacks) at The config supports [callbacks](/usage/training#custom-code-nlp-callbacks) at
several points in the lifecycle that can be used modify the `nlp` object. several points in the lifecycle that can be used modify the `nlp` object.
### spacy.copy_from_base_model.v1 {#copy_from_base_model tag="registered function"} ### spacy.copy_from_base_model.v1 {id="copy_from_base_model",tag="registered function"}
> #### Example config > #### Example config
> >
@ -890,7 +889,7 @@ from the specified model. Intended for use in `[initialize.before_init]`.
| `vocab` | The pipeline to copy the vocab from. The vocab includes the lookups and vectors. Defaults to `None`. ~~Optional[str]~~ | | `vocab` | The pipeline to copy the vocab from. The vocab includes the lookups and vectors. Defaults to `None`. ~~Optional[str]~~ |
| **CREATES** | A function that takes the current `nlp` object and modifies its `tokenizer` and `vocab`. ~~Callable[[Language], None]~~ | | **CREATES** | A function that takes the current `nlp` object and modifies its `tokenizer` and `vocab`. ~~Callable[[Language], None]~~ |
### spacy.models_with_nvtx_range.v1 {#models_with_nvtx_range tag="registered function"} ### spacy.models_with_nvtx_range.v1 {id="models_with_nvtx_range",tag="registered function"}
> #### Example config > #### Example config
> >
@ -910,7 +909,7 @@ backprop passes.
| `backprop_color` | Color identifier for backpropagation passes. Defaults to `-1`. ~~int~~ | | `backprop_color` | Color identifier for backpropagation passes. Defaults to `-1`. ~~int~~ |
| **CREATES** | A function that takes the current `nlp` and wraps forward/backprop passes in NVTX ranges. ~~Callable[[Language], Language]~~ | | **CREATES** | A function that takes the current `nlp` and wraps forward/backprop passes in NVTX ranges. ~~Callable[[Language], Language]~~ |
### spacy.models_and_pipes_with_nvtx_range.v1 {#models_and_pipes_with_nvtx_range tag="registered function" new="3.4"} ### spacy.models_and_pipes_with_nvtx_range.v1 {id="models_and_pipes_with_nvtx_range",tag="registered function",version="3.4"}
> #### Example config > #### Example config
> >
@ -931,9 +930,9 @@ methods are wrapped: `pipe`, `predict`, `set_annotations`, `update`, `rehearse`,
| `additional_pipe_functions` | Additional pipeline methods to wrap. Keys are pipeline names and values are lists of method identifiers. Defaults to `None`. ~~Optional[Dict[str, List[str]]]~~ | | `additional_pipe_functions` | Additional pipeline methods to wrap. Keys are pipeline names and values are lists of method identifiers. Defaults to `None`. ~~Optional[Dict[str, List[str]]]~~ |
| **CREATES** | A function that takes the current `nlp` and wraps pipe models and methods in NVTX ranges. ~~Callable[[Language], Language]~~ | | **CREATES** | A function that takes the current `nlp` and wraps pipe models and methods in NVTX ranges. ~~Callable[[Language], Language]~~ |
## Training data and alignment {#gold source="spacy/training"} ## Training data and alignment {id="gold",source="spacy/training"}
### training.offsets_to_biluo_tags {#offsets_to_biluo_tags tag="function"} ### training.offsets_to_biluo_tags {id="offsets_to_biluo_tags",tag="function"}
Encode labelled spans into per-token tags, using the Encode labelled spans into per-token tags, using the
[BILUO scheme](/usage/linguistic-features#accessing-ner) (Begin, In, Last, Unit, [BILUO scheme](/usage/linguistic-features#accessing-ner) (Begin, In, Last, Unit,
@ -970,7 +969,7 @@ This method was previously available as `spacy.gold.biluo_tags_from_offsets`.
| `missing` | The label used for missing values, e.g. if tokenization doesn't align with the entity offsets. Defaults to `"O"`. ~~str~~ | | `missing` | The label used for missing values, e.g. if tokenization doesn't align with the entity offsets. Defaults to `"O"`. ~~str~~ |
| **RETURNS** | A list of strings, describing the [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ | | **RETURNS** | A list of strings, describing the [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ |
### training.biluo_tags_to_offsets {#biluo_tags_to_offsets tag="function"} ### training.biluo_tags_to_offsets {id="biluo_tags_to_offsets",tag="function"}
Encode per-token tags following the Encode per-token tags following the
[BILUO scheme](/usage/linguistic-features#accessing-ner) into entity offsets. [BILUO scheme](/usage/linguistic-features#accessing-ner) into entity offsets.
@ -998,7 +997,7 @@ This method was previously available as `spacy.gold.offsets_from_biluo_tags`.
| `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags with each tag describing one token. Each tag string will be of the form of either `""`, `"O"` or `"{action}-{label}"`, where action is one of `"B"`, `"I"`, `"L"`, `"U"`. ~~List[str]~~ | | `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags with each tag describing one token. Each tag string will be of the form of either `""`, `"O"` or `"{action}-{label}"`, where action is one of `"B"`, `"I"`, `"L"`, `"U"`. ~~List[str]~~ |
| **RETURNS** | A sequence of `(start, end, label)` triples. `start` and `end` will be character-offset integers denoting the slice into the original string. ~~List[Tuple[int, int, str]]~~ | | **RETURNS** | A sequence of `(start, end, label)` triples. `start` and `end` will be character-offset integers denoting the slice into the original string. ~~List[Tuple[int, int, str]]~~ |
### training.biluo_tags_to_spans {#biluo_tags_to_spans tag="function" new="2.1"} ### training.biluo_tags_to_spans {id="biluo_tags_to_spans",tag="function",version="2.1"}
Encode per-token tags following the Encode per-token tags following the
[BILUO scheme](/usage/linguistic-features#accessing-ner) into [BILUO scheme](/usage/linguistic-features#accessing-ner) into
@ -1027,7 +1026,7 @@ This method was previously available as `spacy.gold.spans_from_biluo_tags`.
| `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags with each tag describing one token. Each tag string will be of the form of either `""`, `"O"` or `"{action}-{label}"`, where action is one of `"B"`, `"I"`, `"L"`, `"U"`. ~~List[str]~~ | | `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags with each tag describing one token. Each tag string will be of the form of either `""`, `"O"` or `"{action}-{label}"`, where action is one of `"B"`, `"I"`, `"L"`, `"U"`. ~~List[str]~~ |
| **RETURNS** | A sequence of `Span` objects with added entity labels. ~~List[Span]~~ | | **RETURNS** | A sequence of `Span` objects with added entity labels. ~~List[Span]~~ |
### training.biluo_to_iob {#biluo_to_iob tag="function"} ### training.biluo_to_iob {id="biluo_to_iob",tag="function"}
Convert a sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags to Convert a sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags to
[IOB](/usage/linguistic-features#accessing-ner) tags. This is useful if you want [IOB](/usage/linguistic-features#accessing-ner) tags. This is useful if you want
@ -1048,7 +1047,7 @@ use the BILUO tags with a model that only supports IOB tags.
| `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ | | `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ |
| **RETURNS** | A list of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ | | **RETURNS** | A list of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ |
### training.iob_to_biluo {#iob_to_biluo tag="function"} ### training.iob_to_biluo {id="iob_to_biluo",tag="function"}
Convert a sequence of [IOB](/usage/linguistic-features#accessing-ner) tags to Convert a sequence of [IOB](/usage/linguistic-features#accessing-ner) tags to
[BILUO](/usage/linguistic-features#accessing-ner) tags. This is useful if you [BILUO](/usage/linguistic-features#accessing-ner) tags. This is useful if you
@ -1075,7 +1074,55 @@ This method was previously available as `spacy.gold.iob_to_biluo`.
| `tags` | A sequence of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ | | `tags` | A sequence of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ |
| **RETURNS** | A list of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ | | **RETURNS** | A list of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ |
## Utility functions {#util source="spacy/util.py"} ### training.biluo_to_iob {id="biluo_to_iob",tag="function"}
Convert a sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags to
[IOB](/usage/linguistic-features#accessing-ner) tags. This is useful if you want
use the BILUO tags with a model that only supports IOB tags.
> #### Example
>
> ```python
> from spacy.training import biluo_to_iob
>
> tags = ["O", "O", "B-LOC", "I-LOC", "L-LOC", "O"]
> iob_tags = biluo_to_iob(tags)
> assert iob_tags == ["O", "O", "B-LOC", "I-LOC", "I-LOC", "O"]
> ```
| Name | Description |
| ----------- | --------------------------------------------------------------------------------------- |
| `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ |
| **RETURNS** | A list of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ |
### training.iob_to_biluo {id="iob_to_biluo",tag="function"}
Convert a sequence of [IOB](/usage/linguistic-features#accessing-ner) tags to
[BILUO](/usage/linguistic-features#accessing-ner) tags. This is useful if you
want use the IOB tags with a model that only supports BILUO tags.
<Infobox title="Changed in v3.0" variant="warning" id="iob_to_biluo">
This method was previously available as `spacy.gold.iob_to_biluo`.
</Infobox>
> #### Example
>
> ```python
> from spacy.training import iob_to_biluo
>
> tags = ["O", "O", "B-LOC", "I-LOC", "O"]
> biluo_tags = iob_to_biluo(tags)
> assert biluo_tags == ["O", "O", "B-LOC", "L-LOC", "O"]
> ```
| Name | Description |
| ----------- | ------------------------------------------------------------------------------------- |
| `tags` | A sequence of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ |
| **RETURNS** | A list of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ |
## Utility functions {id="util",source="spacy/util.py"}
spaCy comes with a small collection of utility functions located in spaCy comes with a small collection of utility functions located in
[`spacy/util.py`](%%GITHUB_SPACY/spacy/util.py). Because utility functions are [`spacy/util.py`](%%GITHUB_SPACY/spacy/util.py). Because utility functions are
@ -1085,7 +1132,7 @@ use and we'll try to ensure backwards compatibility. However, we recommend
having additional tests in place if your application depends on any of spaCy's having additional tests in place if your application depends on any of spaCy's
utilities. utilities.
### util.get_lang_class {#util.get_lang_class tag="function"} ### util.get_lang_class {id="util.get_lang_class",tag="function"}
Import and load a `Language` class. Allows lazy-loading Import and load a `Language` class. Allows lazy-loading
[language data](/usage/linguistic-features#language-data) and importing [language data](/usage/linguistic-features#language-data) and importing
@ -1106,7 +1153,7 @@ custom language class, you can register it using the
| `lang` | Two-letter language code, e.g. `"en"`. ~~str~~ | | `lang` | Two-letter language code, e.g. `"en"`. ~~str~~ |
| **RETURNS** | The respective subclass. ~~Language~~ | | **RETURNS** | The respective subclass. ~~Language~~ |
### util.lang_class_is_loaded {#util.lang_class_is_loaded tag="function" new="2.1"} ### util.lang_class_is_loaded {id="util.lang_class_is_loaded",tag="function",version="2.1"}
Check whether a `Language` subclass is already loaded. `Language` subclasses are Check whether a `Language` subclass is already loaded. `Language` subclasses are
loaded lazily to avoid expensive setup code associated with the language data. loaded lazily to avoid expensive setup code associated with the language data.
@ -1124,7 +1171,7 @@ loaded lazily to avoid expensive setup code associated with the language data.
| `name` | Two-letter language code, e.g. `"en"`. ~~str~~ | | `name` | Two-letter language code, e.g. `"en"`. ~~str~~ |
| **RETURNS** | Whether the class has been loaded. ~~bool~~ | | **RETURNS** | Whether the class has been loaded. ~~bool~~ |
### util.load_model {#util.load_model tag="function" new="2"} ### util.load_model {id="util.load_model",tag="function",version="2"}
Load a pipeline from a package or data path. If called with a string name, spaCy Load a pipeline from a package or data path. If called with a string name, spaCy
will assume the pipeline is a Python package and import and call its `load()` will assume the pipeline is a Python package and import and call its `load()`
@ -1152,7 +1199,7 @@ and create a `Language` object. The model data will then be loaded in via
| `config` <Tag variant="new">3</Tag> | Config overrides as nested dict or flat dict keyed by section values in dot notation, e.g. `"nlp.pipeline"`. ~~Union[Dict[str, Any], Config]~~ | | `config` <Tag variant="new">3</Tag> | Config overrides as nested dict or flat dict keyed by section values in dot notation, e.g. `"nlp.pipeline"`. ~~Union[Dict[str, Any], Config]~~ |
| **RETURNS** | `Language` class with the loaded pipeline. ~~Language~~ | | **RETURNS** | `Language` class with the loaded pipeline. ~~Language~~ |
### util.load_model_from_init_py {#util.load_model_from_init_py tag="function" new="2"} ### util.load_model_from_init_py {id="util.load_model_from_init_py",tag="function",version="2"}
A helper function to use in the `load()` method of a pipeline package's A helper function to use in the `load()` method of a pipeline package's
[`__init__.py`](https://github.com/explosion/spacy-models/tree/master/template/model/xx_model_name/__init__.py). [`__init__.py`](https://github.com/explosion/spacy-models/tree/master/template/model/xx_model_name/__init__.py).
@ -1177,7 +1224,7 @@ A helper function to use in the `load()` method of a pipeline package's
| `config` <Tag variant="new">3</Tag> | Config overrides as nested dict or flat dict keyed by section values in dot notation, e.g. `"nlp.pipeline"`. ~~Union[Dict[str, Any], Config]~~ | | `config` <Tag variant="new">3</Tag> | Config overrides as nested dict or flat dict keyed by section values in dot notation, e.g. `"nlp.pipeline"`. ~~Union[Dict[str, Any], Config]~~ |
| **RETURNS** | `Language` class with the loaded pipeline. ~~Language~~ | | **RETURNS** | `Language` class with the loaded pipeline. ~~Language~~ |
### util.load_config {#util.load_config tag="function" new="3"} ### util.load_config {id="util.load_config",tag="function",version="3"}
Load a pipeline's [`config.cfg`](/api/data-formats#config) from a file path. The Load a pipeline's [`config.cfg`](/api/data-formats#config) from a file path. The
config typically includes details about the components and how they're created, config typically includes details about the components and how they're created,
@ -1197,7 +1244,7 @@ as well as all training settings and hyperparameters.
| `interpolate` | Whether to interpolate the config and replace variables like `${paths.train}` with their values. Defaults to `False`. ~~bool~~ | | `interpolate` | Whether to interpolate the config and replace variables like `${paths.train}` with their values. Defaults to `False`. ~~bool~~ |
| **RETURNS** | The pipeline's config. ~~Config~~ | | **RETURNS** | The pipeline's config. ~~Config~~ |
### util.load_meta {#util.load_meta tag="function" new="3"} ### util.load_meta {id="util.load_meta",tag="function",version="3"}
Get a pipeline's [`meta.json`](/api/data-formats#meta) from a file path and Get a pipeline's [`meta.json`](/api/data-formats#meta) from a file path and
validate its contents. The meta typically includes details about author, validate its contents. The meta typically includes details about author,
@ -1214,7 +1261,7 @@ licensing, data sources and version.
| `path` | Path to the pipeline's `meta.json`. ~~Union[str, Path]~~ | | `path` | Path to the pipeline's `meta.json`. ~~Union[str, Path]~~ |
| **RETURNS** | The pipeline's meta data. ~~Dict[str, Any]~~ | | **RETURNS** | The pipeline's meta data. ~~Dict[str, Any]~~ |
### util.get_installed_models {#util.get_installed_models tag="function" new="3"} ### util.get_installed_models {id="util.get_installed_models",tag="function",version="3"}
List all pipeline packages installed in the current environment. This will List all pipeline packages installed in the current environment. This will
include any spaCy pipeline that was packaged with include any spaCy pipeline that was packaged with
@ -1232,7 +1279,7 @@ object.
| ----------- | ------------------------------------------------------------------------------------- | | ----------- | ------------------------------------------------------------------------------------- |
| **RETURNS** | The string names of the pipelines installed in the current environment. ~~List[str]~~ | | **RETURNS** | The string names of the pipelines installed in the current environment. ~~List[str]~~ |
### util.is_package {#util.is_package tag="function"} ### util.is_package {id="util.is_package",tag="function"}
Check if string maps to a package installed via pip. Mainly used to validate Check if string maps to a package installed via pip. Mainly used to validate
[pipeline packages](/usage/models). [pipeline packages](/usage/models).
@ -1249,7 +1296,7 @@ Check if string maps to a package installed via pip. Mainly used to validate
| `name` | Name of package. ~~str~~ | | `name` | Name of package. ~~str~~ |
| **RETURNS** | `True` if installed package, `False` if not. ~~bool~~ | | **RETURNS** | `True` if installed package, `False` if not. ~~bool~~ |
### util.get_package_path {#util.get_package_path tag="function" new="2"} ### util.get_package_path {id="util.get_package_path",tag="function",version="2"}
Get path to an installed package. Mainly used to resolve the location of Get path to an installed package. Mainly used to resolve the location of
[pipeline packages](/usage/models). Currently imports the package to find its [pipeline packages](/usage/models). Currently imports the package to find its
@ -1267,7 +1314,7 @@ path.
| `package_name` | Name of installed package. ~~str~~ | | `package_name` | Name of installed package. ~~str~~ |
| **RETURNS** | Path to pipeline package directory. ~~Path~~ | | **RETURNS** | Path to pipeline package directory. ~~Path~~ |
### util.is_in_jupyter {#util.is_in_jupyter tag="function" new="2"} ### util.is_in_jupyter {id="util.is_in_jupyter",tag="function",version="2"}
Check if user is running spaCy from a [Jupyter](https://jupyter.org) notebook by Check if user is running spaCy from a [Jupyter](https://jupyter.org) notebook by
detecting the IPython kernel. Mainly used for the detecting the IPython kernel. Mainly used for the
@ -1286,7 +1333,7 @@ detecting the IPython kernel. Mainly used for the
| ----------- | ---------------------------------------------- | | ----------- | ---------------------------------------------- |
| **RETURNS** | `True` if in Jupyter, `False` if not. ~~bool~~ | | **RETURNS** | `True` if in Jupyter, `False` if not. ~~bool~~ |
### util.compile_prefix_regex {#util.compile_prefix_regex tag="function"} ### util.compile_prefix_regex {id="util.compile_prefix_regex",tag="function"}
Compile a sequence of prefix rules into a regex object. Compile a sequence of prefix rules into a regex object.
@ -1303,7 +1350,7 @@ Compile a sequence of prefix rules into a regex object.
| `entries` | The prefix rules, e.g. [`lang.punctuation.TOKENIZER_PREFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ | | `entries` | The prefix rules, e.g. [`lang.punctuation.TOKENIZER_PREFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ |
| **RETURNS** | The regex object to be used for [`Tokenizer.prefix_search`](/api/tokenizer#attributes). ~~Pattern~~ | | **RETURNS** | The regex object to be used for [`Tokenizer.prefix_search`](/api/tokenizer#attributes). ~~Pattern~~ |
### util.compile_suffix_regex {#util.compile_suffix_regex tag="function"} ### util.compile_suffix_regex {id="util.compile_suffix_regex",tag="function"}
Compile a sequence of suffix rules into a regex object. Compile a sequence of suffix rules into a regex object.
@ -1320,7 +1367,7 @@ Compile a sequence of suffix rules into a regex object.
| `entries` | The suffix rules, e.g. [`lang.punctuation.TOKENIZER_SUFFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ | | `entries` | The suffix rules, e.g. [`lang.punctuation.TOKENIZER_SUFFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ |
| **RETURNS** | The regex object to be used for [`Tokenizer.suffix_search`](/api/tokenizer#attributes). ~~Pattern~~ | | **RETURNS** | The regex object to be used for [`Tokenizer.suffix_search`](/api/tokenizer#attributes). ~~Pattern~~ |
### util.compile_infix_regex {#util.compile_infix_regex tag="function"} ### util.compile_infix_regex {id="util.compile_infix_regex",tag="function"}
Compile a sequence of infix rules into a regex object. Compile a sequence of infix rules into a regex object.
@ -1337,7 +1384,7 @@ Compile a sequence of infix rules into a regex object.
| `entries` | The infix rules, e.g. [`lang.punctuation.TOKENIZER_INFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ | | `entries` | The infix rules, e.g. [`lang.punctuation.TOKENIZER_INFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ |
| **RETURNS** | The regex object to be used for [`Tokenizer.infix_finditer`](/api/tokenizer#attributes). ~~Pattern~~ | | **RETURNS** | The regex object to be used for [`Tokenizer.infix_finditer`](/api/tokenizer#attributes). ~~Pattern~~ |
### util.minibatch {#util.minibatch tag="function" new="2"} ### util.minibatch {id="util.minibatch",tag="function",version="2"}
Iterate over batches of items. `size` may be an iterator, so that batch-size can Iterate over batches of items. `size` may be an iterator, so that batch-size can
vary on each step. vary on each step.
@ -1356,7 +1403,7 @@ vary on each step.
| `size` | The batch size(s). ~~Union[int, Sequence[int]]~~ | | `size` | The batch size(s). ~~Union[int, Sequence[int]]~~ |
| **YIELDS** | The batches. | | **YIELDS** | The batches. |
### util.filter_spans {#util.filter_spans tag="function" new="2.1.4"} ### util.filter_spans {id="util.filter_spans",tag="function",version="2.1.4"}
Filter a sequence of [`Span`](/api/span) objects and remove duplicates or Filter a sequence of [`Span`](/api/span) objects and remove duplicates or
overlaps. Useful for creating named entities (where one token can only be part overlaps. Useful for creating named entities (where one token can only be part
@ -1377,7 +1424,7 @@ of one entity) or when merging spans with
| `spans` | The spans to filter. ~~Iterable[Span]~~ | | `spans` | The spans to filter. ~~Iterable[Span]~~ |
| **RETURNS** | The filtered spans. ~~List[Span]~~ | | **RETURNS** | The filtered spans. ~~List[Span]~~ |
### util.get_words_and_spaces {#get_words_and_spaces tag="function" new="3"} ### util.get_words_and_spaces {id="get_words_and_spaces",tag="function",version="3"}
Given a list of words and a text, reconstruct the original tokens and return a Given a list of words and a text, reconstruct the original tokens and return a
list of words and spaces that can be used to create a [`Doc`](/api/doc#init). list of words and spaces that can be used to create a [`Doc`](/api/doc#init).

View File

@ -3,7 +3,7 @@ title: Transformer
teaser: Pipeline component for multi-task learning with transformer models teaser: Pipeline component for multi-task learning with transformer models
tag: class tag: class
source: github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py source: github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py
new: 3 version: 3
api_base_class: /api/pipe api_base_class: /api/pipe
api_string_name: transformer api_string_name: transformer
--- ---
@ -44,7 +44,7 @@ package also adds the function registries [`@span_getters`](#span_getters) and
functions. For more details, see the functions. For more details, see the
[usage documentation](/usage/embeddings-transformers). [usage documentation](/usage/embeddings-transformers).
## Assigned Attributes {#assigned-attributes} ## Assigned Attributes {id="assigned-attributes"}
The component sets the following The component sets the following
[custom extension attribute](/usage/processing-pipeline#custom-components-attributes): [custom extension attribute](/usage/processing-pipeline#custom-components-attributes):
@ -53,7 +53,7 @@ The component sets the following
| ---------------- | ------------------------------------------------------------------------ | | ---------------- | ------------------------------------------------------------------------ |
| `Doc._.trf_data` | Transformer tokens and outputs for the `Doc` object. ~~TransformerData~~ | | `Doc._.trf_data` | Transformer tokens and outputs for the `Doc` object. ~~TransformerData~~ |
## Config and implementation {#config} ## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the how the component should be configured. You can override its settings via the
@ -81,7 +81,7 @@ on the transformer architectures and their arguments and hyperparameters.
https://github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py https://github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py
``` ```
## Transformer.\_\_init\_\_ {#init tag="method"} ## Transformer.\_\_init\_\_ {id="init",tag="method"}
> #### Example > #### Example
> >
@ -124,7 +124,7 @@ component using its string name and [`nlp.add_pipe`](/api/language#create_pipe).
| `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ | | `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ |
| `max_batch_items` | Maximum size of a padded batch. Defaults to `128*32`. ~~int~~ | | `max_batch_items` | Maximum size of a padded batch. Defaults to `128*32`. ~~int~~ |
## Transformer.\_\_call\_\_ {#call tag="method"} ## Transformer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned. Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text This usually happens under the hood when the `nlp` object is called on a text
@ -147,7 +147,7 @@ to the [`predict`](/api/transformer#predict) and
| `doc` | The document to process. ~~Doc~~ | | `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ |
## Transformer.pipe {#pipe tag="method"} ## Transformer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are when the `nlp` object is called on a text and all pipeline components are
@ -171,7 +171,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/transformer#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ |
## Transformer.initialize {#initialize tag="method"} ## Transformer.initialize {id="initialize",tag="method"}
Initialize the component for training and return an Initialize the component for training and return an
[`Optimizer`](https://thinc.ai/docs/api-optimizers). `get_examples` should be a [`Optimizer`](https://thinc.ai/docs/api-optimizers). `get_examples` should be a
@ -196,7 +196,7 @@ by [`Language.initialize`](/api/language#initialize).
| _keyword-only_ | | | _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## Transformer.predict {#predict tag="method"} ## Transformer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects without Apply the component's model to a batch of [`Doc`](/api/doc) objects without
modifying them. modifying them.
@ -213,7 +213,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ | | `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. | | **RETURNS** | The model's prediction for each document. |
## Transformer.set_annotations {#set_annotations tag="method"} ## Transformer.set_annotations {id="set_annotations",tag="method"}
Assign the extracted features to the `Doc` objects. By default, the Assign the extracted features to the `Doc` objects. By default, the
[`TransformerData`](/api/transformer#transformerdata) object is written to the [`TransformerData`](/api/transformer#transformerdata) object is written to the
@ -233,7 +233,7 @@ callback is then called, if provided.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `Transformer.predict`. | | `scores` | The scores to set, produced by `Transformer.predict`. |
## Transformer.update {#update tag="method"} ## Transformer.update {id="update",tag="method"}
Prepare for an update to the transformer. Like the [`Tok2Vec`](/api/tok2vec) Prepare for an update to the transformer. Like the [`Tok2Vec`](/api/tok2vec)
component, the `Transformer` component is unusual in that it does not receive component, the `Transformer` component is unusual in that it does not receive
@ -266,7 +266,7 @@ and call the optimizer, while the others simply increment the gradients.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Transformer.create_optimizer {#create_optimizer tag="method"} ## Transformer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Create an optimizer for the pipeline component.
@ -281,7 +281,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- | | ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ |
## Transformer.use_params {#use_params tag="method, contextmanager"} ## Transformer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model to use the given parameter values. At the end of the Modify the pipe's model to use the given parameter values. At the end of the
context, the original parameters are restored. context, the original parameters are restored.
@ -298,7 +298,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- | | -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ | | `params` | The parameter values to use in the model. ~~dict~~ |
## Transformer.to_disk {#to_disk tag="method"} ## Transformer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk. Serialize the pipe to disk.
@ -315,7 +315,7 @@ Serialize the pipe to disk.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Transformer.from_disk {#from_disk tag="method"} ## Transformer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it. Load the pipe from disk. Modifies the object in place and returns it.
@ -333,7 +333,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Transformer` object. ~~Transformer~~ | | **RETURNS** | The modified `Transformer` object. ~~Transformer~~ |
## Transformer.to_bytes {#to_bytes tag="method"} ## Transformer.to_bytes {id="to_bytes",tag="method"}
> #### Example > #### Example
> >
@ -350,7 +350,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Transformer` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `Transformer` object. ~~bytes~~ |
## Transformer.from_bytes {#from_bytes tag="method"} ## Transformer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it. Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -369,7 +369,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Transformer` object. ~~Transformer~~ | | **RETURNS** | The `Transformer` object. ~~Transformer~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from
@ -387,7 +387,7 @@ serialization by passing in the string names via the `exclude` argument.
| `cfg` | The config file. You usually don't want to exclude this. | | `cfg` | The config file. You usually don't want to exclude this. |
| `model` | The binary model data. You usually don't want to exclude this. | | `model` | The binary model data. You usually don't want to exclude this. |
## TransformerData {#transformerdata tag="dataclass"} ## TransformerData {id="transformerdata",tag="dataclass"}
Transformer tokens and outputs for one `Doc` object. The transformer models Transformer tokens and outputs for one `Doc` object. The transformer models
return tensors that refer to a whole padded batch of documents. These tensors return tensors that refer to a whole padded batch of documents. These tensors
@ -405,7 +405,7 @@ by this class. Instances of this class are typically assigned to the
| `align` | Alignment from the `Doc`'s tokenization to the wordpieces. This is a ragged array, where `align.lengths[i]` indicates the number of wordpiece tokens that token `i` aligns against. The actual indices are provided at `align[i].dataXd`. ~~Ragged~~ | | `align` | Alignment from the `Doc`'s tokenization to the wordpieces. This is a ragged array, where `align.lengths[i]` indicates the number of wordpiece tokens that token `i` aligns against. The actual indices are provided at `align[i].dataXd`. ~~Ragged~~ |
| `width` | The width of the last hidden layer. ~~int~~ | | `width` | The width of the last hidden layer. ~~int~~ |
### TransformerData.empty {#transformerdata-emoty tag="classmethod"} ### TransformerData.empty {id="transformerdata-emoty",tag="classmethod"}
Create an empty `TransformerData` container. Create an empty `TransformerData` container.
@ -425,7 +425,7 @@ model.
</Accordion> </Accordion>
## FullTransformerBatch {#fulltransformerbatch tag="dataclass"} ## FullTransformerBatch {id="fulltransformerbatch",tag="dataclass"}
Holds a batch of input and output objects for a transformer model. The data can Holds a batch of input and output objects for a transformer model. The data can
then be split to a list of [`TransformerData`](/api/transformer#transformerdata) then be split to a list of [`TransformerData`](/api/transformer#transformerdata)
@ -440,7 +440,7 @@ objects to associate the outputs to each [`Doc`](/api/doc) in the batch.
| `align` | Alignment from the spaCy tokenization to the wordpieces. This is a ragged array, where `align.lengths[i]` indicates the number of wordpiece tokens that token `i` aligns against. The actual indices are provided at `align[i].dataXd`. ~~Ragged~~ | | `align` | Alignment from the spaCy tokenization to the wordpieces. This is a ragged array, where `align.lengths[i]` indicates the number of wordpiece tokens that token `i` aligns against. The actual indices are provided at `align[i].dataXd`. ~~Ragged~~ |
| `doc_data` | The outputs, split per `Doc` object. ~~List[TransformerData]~~ | | `doc_data` | The outputs, split per `Doc` object. ~~List[TransformerData]~~ |
### FullTransformerBatch.unsplit_by_doc {#fulltransformerbatch-unsplit_by_doc tag="method"} ### FullTransformerBatch.unsplit_by_doc {id="fulltransformerbatch-unsplit_by_doc",tag="method"}
Return a new `FullTransformerBatch` from a split batch of activations, using the Return a new `FullTransformerBatch` from a split batch of activations, using the
current object's spans, tokens and alignment. This is used during the backward current object's spans, tokens and alignment. This is used during the backward
@ -452,7 +452,7 @@ model.
| `arrays` | The split batch of activations. ~~List[List[Floats3d]]~~ | | `arrays` | The split batch of activations. ~~List[List[Floats3d]]~~ |
| **RETURNS** | The transformer batch. ~~FullTransformerBatch~~ | | **RETURNS** | The transformer batch. ~~FullTransformerBatch~~ |
### FullTransformerBatch.split_by_doc {#fulltransformerbatch-split_by_doc tag="method"} ### FullTransformerBatch.split_by_doc {id="fulltransformerbatch-split_by_doc",tag="method"}
Split a `TransformerData` object that represents a batch into a list with one Split a `TransformerData` object that represents a batch into a list with one
`TransformerData` per `Doc`. `TransformerData` per `Doc`.
@ -468,7 +468,7 @@ In `spacy-transformers` v1.0, the model output is stored in
</Accordion> </Accordion>
## Span getters {#span_getters source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/span_getters.py"} ## Span getters {id="span_getters",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/span_getters.py"}
Span getters are functions that take a batch of [`Doc`](/api/doc) objects and Span getters are functions that take a batch of [`Doc`](/api/doc) objects and
return a lists of [`Span`](/api/span) objects for each doc to be processed by return a lists of [`Span`](/api/span) objects for each doc to be processed by
@ -498,7 +498,7 @@ using the `@spacy.registry.span_getters` decorator.
| `docs` | A batch of `Doc` objects. ~~Iterable[Doc]~~ | | `docs` | A batch of `Doc` objects. ~~Iterable[Doc]~~ |
| **RETURNS** | The spans to process by the transformer. ~~List[List[Span]]~~ | | **RETURNS** | The spans to process by the transformer. ~~List[List[Span]]~~ |
### doc_spans.v1 {#doc_spans tag="registered function"} ### doc_spans.v1 {id="doc_spans",tag="registered function"}
> #### Example config > #### Example config
> >
@ -511,7 +511,7 @@ Create a span getter that uses the whole document as its spans. This is the best
approach if your [`Doc`](/api/doc) objects already refer to relatively short approach if your [`Doc`](/api/doc) objects already refer to relatively short
texts. texts.
### sent_spans.v1 {#sent_spans tag="registered function"} ### sent_spans.v1 {id="sent_spans",tag="registered function"}
> #### Example config > #### Example config
> >
@ -531,7 +531,7 @@ To set sentence boundaries with the `sentencizer` during training, add a
[`[training.annotating_components]`](/usage/training#annotating-components) to [`[training.annotating_components]`](/usage/training#annotating-components) to
have it set the sentence boundaries before the `transformer` component runs. have it set the sentence boundaries before the `transformer` component runs.
### strided_spans.v1 {#strided_spans tag="registered function"} ### strided_spans.v1 {id="strided_spans",tag="registered function"}
> #### Example config > #### Example config
> >
@ -553,7 +553,7 @@ right context.
| `window` | The window size. ~~int~~ | | `window` | The window size. ~~int~~ |
| `stride` | The stride size. ~~int~~ | | `stride` | The stride size. ~~int~~ |
## Annotation setters {#annotation_setters tag="registered functions" source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/annotation_setters.py"} ## Annotation setters {id="annotation_setters",tag="registered functions",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/annotation_setters.py"}
Annotation setters are functions that take a batch of `Doc` objects and a Annotation setters are functions that take a batch of `Doc` objects and a
[`FullTransformerBatch`](/api/transformer#fulltransformerbatch) and can set [`FullTransformerBatch`](/api/transformer#fulltransformerbatch) and can set

View File

@ -3,7 +3,7 @@ title: Vectors
teaser: Store, save and load word vectors teaser: Store, save and load word vectors
tag: class tag: class
source: spacy/vectors.pyx source: spacy/vectors.pyx
new: 2 version: 2
--- ---
Vectors data is kept in the `Vectors.data` attribute, which should be an Vectors data is kept in the `Vectors.data` attribute, which should be an
@ -25,7 +25,7 @@ As of spaCy v3.2, `Vectors` supports two types of vector tables:
the sum of one or more rows as determined by the settings related to character the sum of one or more rows as determined by the settings related to character
ngrams and the hash table. ngrams and the hash table.
## Vectors.\_\_init\_\_ {#init tag="method"} ## Vectors.\_\_init\_\_ {id="init",tag="method"}
Create a new vector store. With the default mode, you can set the vector values Create a new vector store. With the default mode, you can set the vector values
and keys directly on initialization, or supply a `shape` keyword argument to and keys directly on initialization, or supply a `shape` keyword argument to
@ -61,7 +61,7 @@ modified later.
| `bow` <Tag variant="new">3.2</Tag> | The floret BOW string (default: `"<"`). ~~str~~ | | `bow` <Tag variant="new">3.2</Tag> | The floret BOW string (default: `"<"`). ~~str~~ |
| `eow` <Tag variant="new">3.2</Tag> | The floret EOW string (default: `">"`). ~~str~~ | | `eow` <Tag variant="new">3.2</Tag> | The floret EOW string (default: `">"`). ~~str~~ |
## Vectors.\_\_getitem\_\_ {#getitem tag="method"} ## Vectors.\_\_getitem\_\_ {id="getitem",tag="method"}
Get a vector by key. If the key is not found in the table, a `KeyError` is Get a vector by key. If the key is not found in the table, a `KeyError` is
raised. raised.
@ -79,7 +79,7 @@ raised.
| `key` | The key to get the vector for. ~~Union[int, str]~~ | | `key` | The key to get the vector for. ~~Union[int, str]~~ |
| **RETURNS** | The vector for the key. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | | **RETURNS** | The vector for the key. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Vectors.\_\_setitem\_\_ {#setitem tag="method"} ## Vectors.\_\_setitem\_\_ {id="setitem",tag="method"}
Set a vector for the given key. Not supported for `floret` mode. Set a vector for the given key. Not supported for `floret` mode.
@ -96,7 +96,7 @@ Set a vector for the given key. Not supported for `floret` mode.
| `key` | The key to set the vector for. ~~int~~ | | `key` | The key to set the vector for. ~~int~~ |
| `vector` | The vector to set. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | | `vector` | The vector to set. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Vectors.\_\_iter\_\_ {#iter tag="method"} ## Vectors.\_\_iter\_\_ {id="iter",tag="method"}
Iterate over the keys in the table. In `floret` mode, the keys table is not Iterate over the keys in the table. In `floret` mode, the keys table is not
used. used.
@ -112,7 +112,7 @@ used.
| ---------- | --------------------------- | | ---------- | --------------------------- |
| **YIELDS** | A key in the table. ~~int~~ | | **YIELDS** | A key in the table. ~~int~~ |
## Vectors.\_\_len\_\_ {#len tag="method"} ## Vectors.\_\_len\_\_ {id="len",tag="method"}
Return the number of vectors in the table. Return the number of vectors in the table.
@ -127,7 +127,7 @@ Return the number of vectors in the table.
| ----------- | ------------------------------------------- | | ----------- | ------------------------------------------- |
| **RETURNS** | The number of vectors in the table. ~~int~~ | | **RETURNS** | The number of vectors in the table. ~~int~~ |
## Vectors.\_\_contains\_\_ {#contains tag="method"} ## Vectors.\_\_contains\_\_ {id="contains",tag="method"}
Check whether a key has been mapped to a vector entry in the table. In `floret` Check whether a key has been mapped to a vector entry in the table. In `floret`
mode, returns `True` for all keys. mode, returns `True` for all keys.
@ -145,7 +145,7 @@ mode, returns `True` for all keys.
| `key` | The key to check. ~~int~~ | | `key` | The key to check. ~~int~~ |
| **RETURNS** | Whether the key has a vector entry. ~~bool~~ | | **RETURNS** | Whether the key has a vector entry. ~~bool~~ |
## Vectors.add {#add tag="method"} ## Vectors.add {id="add",tag="method"}
Add a key to the table, optionally setting a vector value as well. Keys can be Add a key to the table, optionally setting a vector value as well. Keys can be
mapped to an existing vector by setting `row`, or a new vector can be added. Not mapped to an existing vector by setting `row`, or a new vector can be added. Not
@ -168,7 +168,7 @@ supported for `floret` mode.
| `row` | An optional row number of a vector to map the key to. ~~int~~ | | `row` | An optional row number of a vector to map the key to. ~~int~~ |
| **RETURNS** | The row the vector was added to. ~~int~~ | | **RETURNS** | The row the vector was added to. ~~int~~ |
## Vectors.resize {#resize tag="method"} ## Vectors.resize {id="resize",tag="method"}
Resize the underlying vectors array. If `inplace=True`, the memory is Resize the underlying vectors array. If `inplace=True`, the memory is
reallocated. This may cause other references to the data to become invalid, so reallocated. This may cause other references to the data to become invalid, so
@ -189,7 +189,7 @@ for `floret` mode.
| `inplace` | Reallocate the memory. ~~bool~~ | | `inplace` | Reallocate the memory. ~~bool~~ |
| **RETURNS** | The removed items as a list of `(key, row)` tuples. ~~List[Tuple[int, int]]~~ | | **RETURNS** | The removed items as a list of `(key, row)` tuples. ~~List[Tuple[int, int]]~~ |
## Vectors.keys {#keys tag="method"} ## Vectors.keys {id="keys",tag="method"}
A sequence of the keys in the table. In `floret` mode, the keys table is not A sequence of the keys in the table. In `floret` mode, the keys table is not
used. used.
@ -205,7 +205,7 @@ used.
| ----------- | --------------------------- | | ----------- | --------------------------- |
| **RETURNS** | The keys. ~~Iterable[int]~~ | | **RETURNS** | The keys. ~~Iterable[int]~~ |
## Vectors.values {#values tag="method"} ## Vectors.values {id="values",tag="method"}
Iterate over vectors that have been assigned to at least one key. Note that some Iterate over vectors that have been assigned to at least one key. Note that some
vectors may be unassigned, so the number of vectors returned may be less than vectors may be unassigned, so the number of vectors returned may be less than
@ -222,7 +222,7 @@ the length of the vectors table. In `floret` mode, the keys table is not used.
| ---------- | --------------------------------------------------------------- | | ---------- | --------------------------------------------------------------- |
| **YIELDS** | A vector in the table. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | | **YIELDS** | A vector in the table. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Vectors.items {#items tag="method"} ## Vectors.items {id="items",tag="method"}
Iterate over `(key, vector)` pairs, in order. In `floret` mode, the keys table Iterate over `(key, vector)` pairs, in order. In `floret` mode, the keys table
is empty. is empty.
@ -238,7 +238,7 @@ is empty.
| ---------- | ------------------------------------------------------------------------------------- | | ---------- | ------------------------------------------------------------------------------------- |
| **YIELDS** | `(key, vector)` pairs, in order. ~~Tuple[int, numpy.ndarray[ndim=1, dtype=float32]]~~ | | **YIELDS** | `(key, vector)` pairs, in order. ~~Tuple[int, numpy.ndarray[ndim=1, dtype=float32]]~~ |
## Vectors.find {#find tag="method"} ## Vectors.find {id="find",tag="method"}
Look up one or more keys by row, or vice versa. Not supported for `floret` mode. Look up one or more keys by row, or vice versa. Not supported for `floret` mode.
@ -260,7 +260,7 @@ Look up one or more keys by row, or vice versa. Not supported for `floret` mode.
| `rows` | Find the keys that point to the rows. Returns `numpy.ndarray`. ~~Iterable[int]~~ | | `rows` | Find the keys that point to the rows. Returns `numpy.ndarray`. ~~Iterable[int]~~ |
| **RETURNS** | The requested key, keys, row or rows. ~~Union[int, numpy.ndarray[ndim=1, dtype=float32]]~~ | | **RETURNS** | The requested key, keys, row or rows. ~~Union[int, numpy.ndarray[ndim=1, dtype=float32]]~~ |
## Vectors.shape {#shape tag="property"} ## Vectors.shape {id="shape",tag="property"}
Get `(rows, dims)` tuples of number of rows and number of dimensions in the Get `(rows, dims)` tuples of number of rows and number of dimensions in the
vector table. vector table.
@ -279,7 +279,7 @@ vector table.
| ----------- | ------------------------------------------ | | ----------- | ------------------------------------------ |
| **RETURNS** | A `(rows, dims)` pair. ~~Tuple[int, int]~~ | | **RETURNS** | A `(rows, dims)` pair. ~~Tuple[int, int]~~ |
## Vectors.size {#size tag="property"} ## Vectors.size {id="size",tag="property"}
The vector size, i.e. `rows * dims`. The vector size, i.e. `rows * dims`.
@ -294,7 +294,7 @@ The vector size, i.e. `rows * dims`.
| ----------- | ------------------------ | | ----------- | ------------------------ |
| **RETURNS** | The vector size. ~~int~~ | | **RETURNS** | The vector size. ~~int~~ |
## Vectors.is_full {#is_full tag="property"} ## Vectors.is_full {id="is_full",tag="property"}
Whether the vectors table is full and has no slots are available for new keys. Whether the vectors table is full and has no slots are available for new keys.
If a table is full, it can be resized using If a table is full, it can be resized using
@ -313,7 +313,7 @@ full and cannot be resized.
| ----------- | ------------------------------------------- | | ----------- | ------------------------------------------- |
| **RETURNS** | Whether the vectors table is full. ~~bool~~ | | **RETURNS** | Whether the vectors table is full. ~~bool~~ |
## Vectors.n_keys {#n_keys tag="property"} ## Vectors.n_keys {id="n_keys",tag="property"}
Get the number of keys in the table. Note that this is the number of _all_ keys, Get the number of keys in the table. Note that this is the number of _all_ keys,
not just unique vectors. If several keys are mapped to the same vectors, they not just unique vectors. If several keys are mapped to the same vectors, they
@ -331,7 +331,7 @@ will be counted individually. In `floret` mode, the keys table is not used.
| ----------- | ----------------------------------------------------------------------------- | | ----------- | ----------------------------------------------------------------------------- |
| **RETURNS** | The number of all keys in the table. Returns `-1` for floret vectors. ~~int~~ | | **RETURNS** | The number of all keys in the table. Returns `-1` for floret vectors. ~~int~~ |
## Vectors.most_similar {#most_similar tag="method"} ## Vectors.most_similar {id="most_similar",tag="method"}
For each of the given vectors, find the `n` most similar entries to it by For each of the given vectors, find the `n` most similar entries to it by
cosine. Queries are by vector. Results are returned as a cosine. Queries are by vector. Results are returned as a
@ -356,7 +356,7 @@ supported for `floret` mode.
| `sort` | Whether to sort the entries returned by score. Defaults to `True`. ~~bool~~ | | `sort` | Whether to sort the entries returned by score. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The most similar entries as a `(keys, best_rows, scores)` tuple. ~~Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]~~ | | **RETURNS** | The most similar entries as a `(keys, best_rows, scores)` tuple. ~~Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]~~ |
## Vectors.get_batch {#get_batch tag="method" new="3.2"} ## Vectors.get_batch {id="get_batch",tag="method",version="3.2"}
Get the vectors for the provided keys efficiently as a batch. Get the vectors for the provided keys efficiently as a batch.
@ -371,7 +371,7 @@ Get the vectors for the provided keys efficiently as a batch.
| ------ | --------------------------------------- | | ------ | --------------------------------------- |
| `keys` | The keys. ~~Iterable[Union[int, str]]~~ | | `keys` | The keys. ~~Iterable[Union[int, str]]~~ |
## Vectors.to_ops {#to_ops tag="method"} ## Vectors.to_ops {id="to_ops",tag="method"}
Change the embedding matrix to use different Thinc ops. Change the embedding matrix to use different Thinc ops.
@ -388,7 +388,7 @@ Change the embedding matrix to use different Thinc ops.
| ----- | -------------------------------------------------------- | | ----- | -------------------------------------------------------- |
| `ops` | The Thinc ops to switch the embedding matrix to. ~~Ops~~ | | `ops` | The Thinc ops to switch the embedding matrix to. ~~Ops~~ |
## Vectors.to_disk {#to_disk tag="method"} ## Vectors.to_disk {id="to_disk",tag="method"}
Save the current state to a directory. Save the current state to a directory.
@ -403,7 +403,7 @@ Save the current state to a directory.
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## Vectors.from_disk {#from_disk tag="method"} ## Vectors.from_disk {id="from_disk",tag="method"}
Loads state from a directory. Modifies the object in place and returns it. Loads state from a directory. Modifies the object in place and returns it.
@ -419,7 +419,7 @@ Loads state from a directory. Modifies the object in place and returns it.
| `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The modified `Vectors` object. ~~Vectors~~ | | **RETURNS** | The modified `Vectors` object. ~~Vectors~~ |
## Vectors.to_bytes {#to_bytes tag="method"} ## Vectors.to_bytes {id="to_bytes",tag="method"}
Serialize the current state to a binary string. Serialize the current state to a binary string.
@ -433,7 +433,7 @@ Serialize the current state to a binary string.
| ----------- | ------------------------------------------------------ | | ----------- | ------------------------------------------------------ |
| **RETURNS** | The serialized form of the `Vectors` object. ~~bytes~~ | | **RETURNS** | The serialized form of the `Vectors` object. ~~bytes~~ |
## Vectors.from_bytes {#from_bytes tag="method"} ## Vectors.from_bytes {id="from_bytes",tag="method"}
Load state from a binary string. Load state from a binary string.
@ -451,7 +451,7 @@ Load state from a binary string.
| `data` | The data to load from. ~~bytes~~ | | `data` | The data to load from. ~~bytes~~ |
| **RETURNS** | The `Vectors` object. ~~Vectors~~ | | **RETURNS** | The `Vectors` object. ~~Vectors~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
| Name | Description | | Name | Description |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

View File

@ -10,7 +10,7 @@ The `Vocab` object provides a lookup table that allows you to access
[`StringStore`](/api/stringstore). It also owns underlying C-data that is shared [`StringStore`](/api/stringstore). It also owns underlying C-data that is shared
between `Doc` objects. between `Doc` objects.
## Vocab.\_\_init\_\_ {#init tag="method"} ## Vocab.\_\_init\_\_ {id="init",tag="method"}
Create the vocabulary. Create the vocabulary.
@ -31,7 +31,7 @@ Create the vocabulary.
| `writing_system` | A dictionary describing the language's writing system. Typically provided by [`Language.Defaults`](/api/language#defaults). ~~Dict[str, Any]~~ | | `writing_system` | A dictionary describing the language's writing system. Typically provided by [`Language.Defaults`](/api/language#defaults). ~~Dict[str, Any]~~ |
| `get_noun_chunks` | A function that yields base noun phrases used for [`Doc.noun_chunks`](/api/doc#noun_chunks). ~~Optional[Callable[[Union[Doc, Span], Iterator[Tuple[int, int, int]]]]]~~ | | `get_noun_chunks` | A function that yields base noun phrases used for [`Doc.noun_chunks`](/api/doc#noun_chunks). ~~Optional[Callable[[Union[Doc, Span], Iterator[Tuple[int, int, int]]]]]~~ |
## Vocab.\_\_len\_\_ {#len tag="method"} ## Vocab.\_\_len\_\_ {id="len",tag="method"}
Get the current number of lexemes in the vocabulary. Get the current number of lexemes in the vocabulary.
@ -46,7 +46,7 @@ Get the current number of lexemes in the vocabulary.
| ----------- | ------------------------------------------------ | | ----------- | ------------------------------------------------ |
| **RETURNS** | The number of lexemes in the vocabulary. ~~int~~ | | **RETURNS** | The number of lexemes in the vocabulary. ~~int~~ |
## Vocab.\_\_getitem\_\_ {#getitem tag="method"} ## Vocab.\_\_getitem\_\_ {id="getitem",tag="method"}
Retrieve a lexeme, given an int ID or a string. If a previously unseen string is Retrieve a lexeme, given an int ID or a string. If a previously unseen string is
given, a new lexeme is created and stored. given, a new lexeme is created and stored.
@ -63,7 +63,7 @@ given, a new lexeme is created and stored.
| `id_or_string` | The hash value of a word, or its string. ~~Union[int, str]~~ | | `id_or_string` | The hash value of a word, or its string. ~~Union[int, str]~~ |
| **RETURNS** | The lexeme indicated by the given ID. ~~Lexeme~~ | | **RETURNS** | The lexeme indicated by the given ID. ~~Lexeme~~ |
## Vocab.\_\_iter\_\_ {#iter tag="method"} ## Vocab.\_\_iter\_\_ {id="iter",tag="method"}
Iterate over the lexemes in the vocabulary. Iterate over the lexemes in the vocabulary.
@ -77,7 +77,7 @@ Iterate over the lexemes in the vocabulary.
| ---------- | -------------------------------------- | | ---------- | -------------------------------------- |
| **YIELDS** | An entry in the vocabulary. ~~Lexeme~~ | | **YIELDS** | An entry in the vocabulary. ~~Lexeme~~ |
## Vocab.\_\_contains\_\_ {#contains tag="method"} ## Vocab.\_\_contains\_\_ {id="contains",tag="method"}
Check whether the string has an entry in the vocabulary. To get the ID for a Check whether the string has an entry in the vocabulary. To get the ID for a
given string, you need to look it up in given string, you need to look it up in
@ -97,7 +97,7 @@ given string, you need to look it up in
| `string` | The ID string. ~~str~~ | | `string` | The ID string. ~~str~~ |
| **RETURNS** | Whether the string has an entry in the vocabulary. ~~bool~~ | | **RETURNS** | Whether the string has an entry in the vocabulary. ~~bool~~ |
## Vocab.add_flag {#add_flag tag="method"} ## Vocab.add_flag {id="add_flag",tag="method"}
Set a new boolean flag to words in the vocabulary. The `flag_getter` function Set a new boolean flag to words in the vocabulary. The `flag_getter` function
will be called over the words currently in the vocab, and then applied to new will be called over the words currently in the vocab, and then applied to new
@ -122,7 +122,7 @@ using `token.check_flag(flag_id)`.
| `flag_id` | An integer between `1` and `63` (inclusive), specifying the bit at which the flag will be stored. If `-1`, the lowest available bit will be chosen. ~~int~~ | | `flag_id` | An integer between `1` and `63` (inclusive), specifying the bit at which the flag will be stored. If `-1`, the lowest available bit will be chosen. ~~int~~ |
| **RETURNS** | The integer ID by which the flag value can be checked. ~~int~~ | | **RETURNS** | The integer ID by which the flag value can be checked. ~~int~~ |
## Vocab.reset_vectors {#reset_vectors tag="method" new="2"} ## Vocab.reset_vectors {id="reset_vectors",tag="method",version="2"}
Drop the current vector table. Because all vectors must be the same width, you Drop the current vector table. Because all vectors must be the same width, you
have to call this to change the size of the vectors. Only one of the `width` and have to call this to change the size of the vectors. Only one of the `width` and
@ -140,7 +140,7 @@ have to call this to change the size of the vectors. Only one of the `width` and
| `width` | The new width. ~~int~~ | | `width` | The new width. ~~int~~ |
| `shape` | The new shape. ~~int~~ | | `shape` | The new shape. ~~int~~ |
## Vocab.prune_vectors {#prune_vectors tag="method" new="2"} ## Vocab.prune_vectors {id="prune_vectors",tag="method",version="2"}
Reduce the current vector table to `nr_row` unique entries. Words mapped to the Reduce the current vector table to `nr_row` unique entries. Words mapped to the
discarded vectors will be remapped to the closest vector among those remaining. discarded vectors will be remapped to the closest vector among those remaining.
@ -165,7 +165,7 @@ cosines are calculated in minibatches to reduce memory usage.
| `batch_size` | Batch of vectors for calculating the similarities. Larger batch sizes might be faster, while temporarily requiring more memory. ~~int~~ | | `batch_size` | Batch of vectors for calculating the similarities. Larger batch sizes might be faster, while temporarily requiring more memory. ~~int~~ |
| **RETURNS** | A dictionary keyed by removed words mapped to `(string, score)` tuples, where `string` is the entry the removed word was mapped to, and `score` the similarity score between the two words. ~~Dict[str, Tuple[str, float]]~~ | | **RETURNS** | A dictionary keyed by removed words mapped to `(string, score)` tuples, where `string` is the entry the removed word was mapped to, and `score` the similarity score between the two words. ~~Dict[str, Tuple[str, float]]~~ |
## Vocab.deduplicate_vectors {#deduplicate_vectors tag="method" new="3.3"} ## Vocab.deduplicate_vectors {id="deduplicate_vectors",tag="method",version="3.3"}
> #### Example > #### Example
> >
@ -176,7 +176,7 @@ cosines are calculated in minibatches to reduce memory usage.
Remove any duplicate rows from the current vector table, maintaining the Remove any duplicate rows from the current vector table, maintaining the
mappings for all words in the vectors. mappings for all words in the vectors.
## Vocab.get_vector {#get_vector tag="method" new="2"} ## Vocab.get_vector {id="get_vector",tag="method",version="2"}
Retrieve a vector for a word in the vocabulary. Words can be looked up by string Retrieve a vector for a word in the vocabulary. Words can be looked up by string
or hash value. If the current vectors do not contain an entry for the word, a or hash value. If the current vectors do not contain an entry for the word, a
@ -194,7 +194,7 @@ or hash value. If the current vectors do not contain an entry for the word, a
| `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ | | `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ |
| **RETURNS** | A word vector. Size and shape are determined by the `Vocab.vectors` instance. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | | **RETURNS** | A word vector. Size and shape are determined by the `Vocab.vectors` instance. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Vocab.set_vector {#set_vector tag="method" new="2"} ## Vocab.set_vector {id="set_vector",tag="method",version="2"}
Set a vector for a word in the vocabulary. Words can be referenced by string or Set a vector for a word in the vocabulary. Words can be referenced by string or
hash value. hash value.
@ -210,7 +210,7 @@ hash value.
| `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ | | `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ |
| `vector` | The vector to set. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | | `vector` | The vector to set. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Vocab.has_vector {#has_vector tag="method" new="2"} ## Vocab.has_vector {id="has_vector",tag="method",version="2"}
Check whether a word has a vector. Returns `False` if no vectors are loaded. Check whether a word has a vector. Returns `False` if no vectors are loaded.
Words can be looked up by string or hash value. Words can be looked up by string or hash value.
@ -227,7 +227,7 @@ Words can be looked up by string or hash value.
| `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ | | `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ |
| **RETURNS** | Whether the word has a vector. ~~bool~~ | | **RETURNS** | Whether the word has a vector. ~~bool~~ |
## Vocab.to_disk {#to_disk tag="method" new="2"} ## Vocab.to_disk {id="to_disk",tag="method",version="2"}
Save the current state to a directory. Save the current state to a directory.
@ -243,7 +243,7 @@ Save the current state to a directory.
| _keyword-only_ | | | _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Vocab.from_disk {#from_disk tag="method" new="2"} ## Vocab.from_disk {id="from_disk",tag="method",version="2"}
Loads state from a directory. Modifies the object in place and returns it. Loads state from a directory. Modifies the object in place and returns it.
@ -261,7 +261,7 @@ Loads state from a directory. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Vocab` object. ~~Vocab~~ | | **RETURNS** | The modified `Vocab` object. ~~Vocab~~ |
## Vocab.to_bytes {#to_bytes tag="method"} ## Vocab.to_bytes {id="to_bytes",tag="method"}
Serialize the current state to a binary string. Serialize the current state to a binary string.
@ -277,7 +277,7 @@ Serialize the current state to a binary string.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Vocab` object. ~~Vocab~~ | | **RETURNS** | The serialized form of the `Vocab` object. ~~Vocab~~ |
## Vocab.from_bytes {#from_bytes tag="method"} ## Vocab.from_bytes {id="from_bytes",tag="method"}
Load state from a binary string. Load state from a binary string.
@ -297,7 +297,7 @@ Load state from a binary string.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Vocab` object. ~~Vocab~~ | | **RETURNS** | The `Vocab` object. ~~Vocab~~ |
## Attributes {#attributes} ## Attributes {id="attributes"}
> #### Example > #### Example
> >
@ -317,7 +317,7 @@ Load state from a binary string.
| `writing_system` | A dict with information about the language's writing system. ~~Dict[str, Any]~~ | | `writing_system` | A dict with information about the language's writing system. ~~Dict[str, Any]~~ |
| `get_noun_chunks` <Tag variant="new">3.0</Tag> | A function that yields base noun phrases used for [`Doc.noun_chunks`](/api/doc#noun_chunks). ~~Optional[Callable[[Union[Doc, Span], Iterator[Tuple[int, int, int]]]]]~~ | | `get_noun_chunks` <Tag variant="new">3.0</Tag> | A function that yields base noun phrases used for [`Doc.noun_chunks`](/api/doc#noun_chunks). ~~Optional[Callable[[Union[Doc, Span], Iterator[Tuple[int, int, int]]]]]~~ |
## Serialization fields {#serialization-fields} ## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from different aspects of the object. If needed, you can exclude them from

View File

@ -1,58 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" xlink="http://www.w3.org/1999/xlink" xml:lang="en" id="c3124cc3e661444cb9d4175a5b7c09d1-0" class="displacy" width="925" height="399.5" direction="ltr" style="max-width: none; height: 399.5px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr">
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="50">Smith</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="50"></tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="225">founded</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="225"></tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="400">a</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="400"></tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="575">healthcare</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="575"></tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="750">company</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="750"></tspan>
</text>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-0" stroke-width="2px" d="M70,264.5 C70,177.0 215.0,177.0 215.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textPath xlink:href="#arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-0" class="displacy-label" startOffset="50%" side="left" fill="currentColor" text-anchor="middle">nsubj</textPath>
</text>
<path class="displacy-arrowhead" d="M70,266.5 L62,254.5 78,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-1" stroke-width="2px" d="M420,264.5 C420,89.5 745.0,89.5 745.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textPath xlink:href="#arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-1" class="displacy-label" startOffset="50%" side="left" fill="currentColor" text-anchor="middle">det</textPath>
</text>
<path class="displacy-arrowhead" d="M420,266.5 L412,254.5 428,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-2" stroke-width="2px" d="M595,264.5 C595,177.0 740.0,177.0 740.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textPath xlink:href="#arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-2" class="displacy-label" startOffset="50%" side="left" fill="currentColor" text-anchor="middle">compound</textPath>
</text>
<path class="displacy-arrowhead" d="M595,266.5 L587,254.5 603,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-3" stroke-width="2px" d="M245,264.5 C245,2.0 750.0,2.0 750.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textPath xlink:href="#arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-3" class="displacy-label" startOffset="50%" side="left" fill="currentColor" text-anchor="middle">dobj</textPath>
</text>
<path class="displacy-arrowhead" d="M750.0,266.5 L758.0,254.5 742.0,254.5" fill="currentColor"></path>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 3.8 KiB

View File

@ -1,33 +0,0 @@
<div
class="entities"
style="line-height: 2.5; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 18px"
>But
<mark
class="entity"
style="background: linear-gradient(90deg, #AA9CFC, #FC9CE7); padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>Google
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>ORG</span
></mark
>is starting from behind. The company made a late push into hardware, and
<mark
class="entity"
style="background: linear-gradient(90deg, #AA9CFC, #FC9CE7); padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>Apple
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>ORG</span
></mark
>s Siri, available on iPhones, and
<mark
class="entity"
style="background: linear-gradient(90deg, #AA9CFC, #FC9CE7); padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>Amazon
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>ORG</span
></mark
>s Alexa software, which runs on its Echo and Dot devices, have clear leads in consumer
adoption.</div
>

View File

@ -1,26 +0,0 @@
<div
class="entities"
style="line-height: 2.5; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px"
>
🌱🌿
<mark
class="entity"
style="background: #3dff74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>🐍
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>SNEK</span
></mark
>
____ 🌳🌲 ____
<mark
class="entity"
style="background: #cfc5ff; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>👨‍🌾
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>HUMAN</span
></mark
>
🏘️
</div>

View File

@ -1,37 +0,0 @@
<div
class="entities"
style="line-height: 2.5; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px"
>
<mark
class="entity"
style="background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
Apple
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>ORG</span
>
</mark>
is looking at buying
<mark
class="entity"
style="background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
U.K.
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>GPE</span
>
</mark>
startup for
<mark
class="entity"
style="background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
$1 billion
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>MONEY</span
>
</mark>
</div>

View File

@ -1,39 +0,0 @@
<div
class="entities"
style="line-height: 2.5; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 18px"
>
When
<mark
class="entity"
style="background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
Sebastian Thrun
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>PERSON</span
>
</mark>
started working on self-driving cars at
<mark
class="entity"
style="background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
Google
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>ORG</span
>
</mark>
in
<mark
class="entity"
style="background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
2007
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>DATE</span
>
</mark>
, few people outside of the company took him seriously.
</div>

View File

@ -1,84 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" id="0" class="displacy" width="1275" height="399.5" style="max-width: none; height: 399.5px; color: #000000; background: #ffffff; font-family: Arial">
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="50">Autonomous</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="50">ADJ</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="225">cars</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="225">NOUN</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="400">shift</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="400">VERB</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="575">insurance</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="575">NOUN</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="750">liability</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="750">NOUN</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="925">toward</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="925">ADP</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="1100">manufacturers</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="1100">NOUN</tspan>
</text>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-0" stroke-width="2px" d="M70,264.5 C70,177.0 215.0,177.0 215.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-0" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">amod</textpath>
</text>
<path class="displacy-arrowhead" d="M70,266.5 L62,254.5 78,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-1" stroke-width="2px" d="M245,264.5 C245,177.0 390.0,177.0 390.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-1" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">nsubj</textpath>
</text>
<path class="displacy-arrowhead" d="M245,266.5 L237,254.5 253,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-2" stroke-width="2px" d="M595,264.5 C595,177.0 740.0,177.0 740.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-2" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">compound</textpath>
</text>
<path class="displacy-arrowhead" d="M595,266.5 L587,254.5 603,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-3" stroke-width="2px" d="M420,264.5 C420,89.5 745.0,89.5 745.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-3" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">dobj</textpath>
</text>
<path class="displacy-arrowhead" d="M745.0,266.5 L753.0,254.5 737.0,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-4" stroke-width="2px" d="M420,264.5 C420,2.0 925.0,2.0 925.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-4" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">prep</textpath>
</text>
<path class="displacy-arrowhead" d="M925.0,266.5 L933.0,254.5 917.0,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-5" stroke-width="2px" d="M945,264.5 C945,177.0 1090.0,177.0 1090.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-5" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">pobj</textpath>
</text>
<path class="displacy-arrowhead" d="M1090.0,266.5 L1098.0,254.5 1082.0,254.5" fill="currentColor"></path>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 5.0 KiB

View File

@ -1,31 +0,0 @@
<div class="spans"
style="line-height: 2.5; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 18px; direction: ltr">
Welcome to the
<span style="font-weight: bold; display: inline-block; position: relative;">
Bank
<span
style="background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
<span
style="background: #ddd; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;">
<span
style="background: #ddd; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px">
BANK
</span>
</span>
</span>
<span style="font-weight: bold; display: inline-block; position: relative;">
of
<span
style="background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
</span>
<span style="font-weight: bold; display: inline-block; position: relative;">
China
<span
style="background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
</span>
.
</div>

View File

@ -1,41 +0,0 @@
<div class="spans"
style="line-height: 2.5; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 18px">
Welcome to the
<span style="font-weight: bold; display: inline-block; position: relative;">
Bank
<span
style="background: #7aecec; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
<span
style="background: #7aecec; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;">
<span
style="background: #7aecec; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px">
ORG
</span>
</span>
</span>
<span style="font-weight: bold; display: inline-block; position: relative;">
of
<span
style="background: #7aecec; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
</span>
<span style="font-weight: bold; display: inline-block; position: relative;">
China
<span
style="background: #7aecec; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
<span
style="background: #feca74; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
<span
style="background: #feca74; top: 57px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;">
<span
style="background: #feca74; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px">
GPE
</span>
</span>
</span>
.
</div>

View File

@ -1,6 +0,0 @@
---
---
import Landing from 'widgets/landing.js'
<Landing />

View File

@ -7,7 +7,7 @@ menu:
- ['Pipeline Design', 'design'] - ['Pipeline Design', 'design']
--- ---
<!-- TODO: include interactive demo --> {/* TODO: include interactive demo */}
### Quickstart {hidden="true"} ### Quickstart {hidden="true"}
@ -16,11 +16,9 @@ menu:
> For more details on how to use trained pipelines with spaCy, see the > For more details on how to use trained pipelines with spaCy, see the
> [usage guide](/usage/models). > [usage guide](/usage/models).
import QuickstartModels from 'widgets/quickstart-models.js'
<QuickstartModels id="quickstart" /> <QuickstartModels id="quickstart" />
## Package naming conventions {#conventions} ## Package naming conventions {id="conventions"}
In general, spaCy expects all pipeline packages to follow the naming convention In general, spaCy expects all pipeline packages to follow the naming convention
of `[lang]\_[name]`. For spaCy's pipelines, we also chose to divide the name of `[lang]\_[name]`. For spaCy's pipelines, we also chose to divide the name
@ -45,7 +43,7 @@ For example, [`en_core_web_sm`](/models/en#en_core_web_sm) is a small English
pipeline trained on written web text (blogs, news, comments), that includes pipeline trained on written web text (blogs, news, comments), that includes
vocabulary, syntax and entities. vocabulary, syntax and entities.
### Package versioning {#model-versioning} ### Package versioning {id="model-versioning"}
Additionally, the pipeline package versioning reflects both the compatibility Additionally, the pipeline package versioning reflects both the compatibility
with spaCy, as well as the model version. A package version `a.b.c` translates with spaCy, as well as the model version. A package version `a.b.c` translates
@ -62,7 +60,7 @@ For a detailed compatibility overview, see the
This is also the source of spaCy's internal compatibility check, performed when This is also the source of spaCy's internal compatibility check, performed when
you run the [`download`](/api/cli#download) command. you run the [`download`](/api/cli#download) command.
## Trained pipeline design {#design} ## Trained pipeline design {id="design"}
The spaCy v3 trained pipelines are designed to be efficient and configurable. The spaCy v3 trained pipelines are designed to be efficient and configurable.
For example, multiple components can share a common "token-to-vector" model and For example, multiple components can share a common "token-to-vector" model and
@ -89,9 +87,9 @@ Main changes from spaCy v2 models:
- The lemmatizer tables and processing move from the vocab and tagger to a - The lemmatizer tables and processing move from the vocab and tagger to a
separate `lemmatizer` component. separate `lemmatizer` component.
### CNN/CPU pipeline design {#design-cnn} ### CNN/CPU pipeline design {id="design-cnn"}
![Components and their dependencies in the CNN pipelines](../images/pipeline-design.svg) ![Components and their dependencies in the CNN pipelines](/images/pipeline-design.svg)
In the `sm`/`md`/`lg` models: In the `sm`/`md`/`lg` models:
@ -132,13 +130,13 @@ vector keys for default vectors.
- [`Vectors.most_similar`](/api/vectors#most_similar) is not supported because - [`Vectors.most_similar`](/api/vectors#most_similar) is not supported because
there's no fixed list of vectors to compare your vectors to. there's no fixed list of vectors to compare your vectors to.
### Transformer pipeline design {#design-trf} ### Transformer pipeline design {id="design-trf"}
In the transformer (`trf`) models, the `tagger`, `parser` and `ner` (if present) In the transformer (`trf`) models, the `tagger`, `parser` and `ner` (if present)
all listen to the `transformer` component. The `attribute_ruler` and all listen to the `transformer` component. The `attribute_ruler` and
`lemmatizer` have the same configuration as in the CNN models. `lemmatizer` have the same configuration as in the CNN models.
### Modifying the default pipeline {#design-modify} ### Modifying the default pipeline {id="design-modify"}
For faster processing, you may only want to run a subset of the components in a For faster processing, you may only want to run a subset of the components in a
trained pipeline. The `disable` and `exclude` arguments to trained pipeline. The `disable` and `exclude` arguments to
@ -189,8 +187,8 @@ than the rule-based `sentencizer`.
#### Switch from trainable lemmatizer to default lemmatizer #### Switch from trainable lemmatizer to default lemmatizer
Since v3.3, a number of pipelines use a trainable lemmatizer. You can check whether Since v3.3, a number of pipelines use a trainable lemmatizer. You can check
the lemmatizer is trainable: whether the lemmatizer is trainable:
```python ```python
nlp = spacy.load("de_core_web_sm") nlp = spacy.load("de_core_web_sm")

View File

@ -42,9 +42,7 @@ enough, JSX components can be used.
> For more details on editing the site locally, see the installation > For more details on editing the site locally, see the installation
> instructions and markdown reference below. > instructions and markdown reference below.
## Logo {#logo source="website/src/images/logo.svg"} ## Logo {id="logo",source="website/src/images/logo.svg"}
import { Logos } from 'widgets/styleguide'
If you would like to use the spaCy logo on your site, please get in touch and If you would like to use the spaCy logo on your site, please get in touch and
ask us first. However, if you want to show support and tell others that your ask us first. However, if you want to show support and tell others that your
@ -53,9 +51,7 @@ project is using spaCy, you can grab one of our
<Logos /> <Logos />
## Colors {#colors} ## Colors {id="colors"}
import { Colors, Patterns } from 'widgets/styleguide'
<Colors /> <Colors />
@ -63,17 +59,16 @@ import { Colors, Patterns } from 'widgets/styleguide'
<Patterns /> <Patterns />
## Typography {#typography} ## Typography {id="typography"}
import { H1, H2, H3, H4, H5, Label, InlineList, Comment } from
'components/typography'
> #### Markdown > #### Markdown
> >
> ```markdown_ > ```markdown
> ## Headline 2 > ## Headline 2
> ## Headline 2 {#some_id} >
> ## Headline 2 {#some_id tag="method"} > ## Headline 2 {id="some_id"}
>
> ## Headline 2 {id="some_id" tag="method"}
> ``` > ```
> >
> #### JSX > #### JSX
@ -101,7 +96,6 @@ in the sidebar menu.
</Infobox> </Infobox>
<div> <div>
<H1>Headline 1</H1>
<H2>Headline 2</H2> <H2>Headline 2</H2>
<H3>Headline 3</H3> <H3>Headline 3</H3>
<H4>Headline 4</H4> <H4>Headline 4</H4>
@ -117,15 +111,15 @@ introduced in a specific version or require statistical models to be loaded.
Tags are also available as standalone `<Tag />` components. Tags are also available as standalone `<Tag />` components.
| Argument | Example | Result | | Argument | Example | Result |
| -------- | -------------------------- | ----------------------------------------- | | --------- | -------------------------- | ----------------------------------------- |
| `tag` | `{tag="method"}` | <Tag>method</Tag> | | `tag` | `{tag="method"}` | <Tag>method</Tag> |
| `new` | `{new="3"}` | <Tag variant="new">3</Tag> | | `version` | `{version="3"}` | <Tag variant="new">3</Tag> |
| `model` | `{model="tagger, parser"}` | <Tag variant="model">tagger, parser</Tag> | | `model` | `{model="tagger, parser"}` | <Tag variant="model">tagger, parser</Tag> |
| `hidden` | `{hidden="true"}` | | | `hidden` | `{hidden="true"}` | |
## Elements {#elements} ## Elements {id="elements"}
### Links {#links} ### Links {id="links"}
> #### Markdown > #### Markdown
> >
@ -147,9 +141,7 @@ Special link styles are used depending on the link URL.
- [I am a link to a model](/models/en#en_core_web_sm) - [I am a link to a model](/models/en#en_core_web_sm)
- [I am a link to GitHub](https://github.com/explosion/spaCy) - [I am a link to GitHub](https://github.com/explosion/spaCy)
### Abbreviations {#abbr} ### Abbreviations {id="abbr"}
import { Abbr } from 'components/typography'
> #### JSX > #### JSX
> >
@ -161,13 +153,11 @@ Some text with <Abbr title="Explanation here">an abbreviation</Abbr>. On small
screens, I collapse and the explanation text is displayed next to the screens, I collapse and the explanation text is displayed next to the
abbreviation. abbreviation.
### Tags {#tags} ### Tags {id="tags"}
import Tag from 'components/tag'
> ```jsx > ```jsx
> <Tag>method</Tag> > <Tag>method</Tag>
> <Tag variant="new">4</Tag> > <Tag variant="version">4</Tag>
> <Tag variant="model">tagger, parser</Tag> > <Tag variant="model">tagger, parser</Tag>
> ``` > ```
@ -180,16 +170,13 @@ new anymore. Setting `variant="model"` takes a description of model capabilities
and can be used to mark features that require a respective model to be and can be used to mark features that require a respective model to be
installed. installed.
<InlineList> <p>
<Tag>method</Tag>
<Tag variant="new">4</Tag>
<Tag variant="model">tagger, parser</Tag>
</p>
<Tag>method</Tag> <Tag variant="new">4</Tag> <Tag variant="model">tagger, ### Buttons {id="buttons"}
parser</Tag>
</InlineList>
### Buttons {#buttons}
import Button from 'components/button'
> ```jsx > ```jsx
> <Button to="#" variant="primary">Primary small</Button> > <Button to="#" variant="primary">Primary small</Button>
@ -200,21 +187,29 @@ Link buttons come in two variants, `primary` and `secondary` and two sizes, with
an optional `large` size modifier. Since they're mostly used as enhanced links, an optional `large` size modifier. Since they're mostly used as enhanced links,
the buttons are implemented as styled links instead of native button elements. the buttons are implemented as styled links instead of native button elements.
<InlineList><Button to="#" variant="primary">Primary small</Button> <p>
<Button to="#" variant="secondary">Secondary small</Button></InlineList> <Button to="#" variant="primary">Primary small</Button>
<br /> {' '}
<InlineList><Button to="#" variant="primary" large>Primary large</Button> <Button to="#" variant="secondary">Secondary small</Button>
<Button to="#" variant="secondary" large>Secondary large</Button></InlineList> </p>
<p>
<Button to="#" variant="primary">Primary small</Button>
{' '}
<Button to="#" variant="secondary">Secondary small</Button>
</p>
## Components ## Components
### Table {#table} ### Table {id="table"}
> #### Markdown > #### Markdown
> >
> ```markdown_ > ```markdown
> | Header 1 | Header 2 | > | Header 1 | Header 2 |
> | -------- | -------- | > | -------- | -------- |
> | Column 1 | Column 2 | > | Column 1 | Column 2 |
@ -248,7 +243,7 @@ be italicized:
> #### Markdown > #### Markdown
> >
> ```markdown_ > ```markdown
> | Header 1 | Header 2 | Header 3 | > | Header 1 | Header 2 | Header 3 |
> | -------- | -------- | -------- | > | -------- | -------- | -------- |
> | Column 1 | Column 2 | Column 3 | > | Column 1 | Column 2 | Column 3 |
@ -262,11 +257,11 @@ be italicized:
| _Hello_ | | | | _Hello_ | | |
| Column 1 | Column 2 | Column 3 | | Column 1 | Column 2 | Column 3 |
### Type Annotations {#type-annotations} ### Type Annotations {id="type-annotations"}
> #### Markdown > #### Markdown
> >
> ```markdown_ > ```markdown
> ~~Model[List[Doc], Floats2d]~~ > ~~Model[List[Doc], Floats2d]~~
> ``` > ```
> >
@ -295,9 +290,9 @@ always be the **last element** in the row.
> #### Markdown > #### Markdown
> >
> ```markdown_ > ```markdown
> | Header 1 | Header 2 | > | Header 1 | Header 2 |
> | -------- | ----------------------- | > | -------- | ---------------------- |
> | Column 1 | Column 2 ~~List[Doc]~~ | > | Column 1 | Column 2 ~~List[Doc]~~ |
> ``` > ```
@ -307,11 +302,11 @@ always be the **last element** in the row.
| `model` | The Thinc [`Model`](https://thinc.ai/docs/api-model) wrapping the transformer. ~~Model[List[Doc], FullTransformerBatch]~~ | | `model` | The Thinc [`Model`](https://thinc.ai/docs/api-model) wrapping the transformer. ~~Model[List[Doc], FullTransformerBatch]~~ |
| `set_extra_annotations` | Function that takes a batch of `Doc` objects and transformer outputs and can set additional annotations on the `Doc`. ~~Callable[[List[Doc], FullTransformerBatch], None]~~ | | `set_extra_annotations` | Function that takes a batch of `Doc` objects and transformer outputs and can set additional annotations on the `Doc`. ~~Callable[[List[Doc], FullTransformerBatch], None]~~ |
### List {#list} ### List {id="list"}
> #### Markdown > #### Markdown
> >
> ```markdown_ > ```markdown
> 1. One > 1. One
> 2. Two > 2. Two
> ``` > ```
@ -338,12 +333,13 @@ automatically.
3. Lorem ipsum dolor 3. Lorem ipsum dolor
4. consectetur adipiscing elit 4. consectetur adipiscing elit
### Aside {#aside} ### Aside {id="aside"}
> #### Markdown > #### Markdown
> >
> ```markdown_ > ```markdown
> > #### Aside title > > #### Aside title
> >
> > This is aside text. > > This is aside text.
> ``` > ```
> >
@ -363,11 +359,11 @@ To make them easier to use in Markdown, paragraphs formatted as blockquotes will
turn into asides by default. Level 4 headlines (with a leading `####`) will turn into asides by default. Level 4 headlines (with a leading `####`) will
become aside titles. become aside titles.
### Code Block {#code-block} ### Code Block {id="code-block"}
> #### Markdown > #### Markdown
> >
> ````markdown_ > ````markdown
> ```python > ```python
> ### This is a title > ### This is a title
> import spacy > import spacy
@ -388,8 +384,7 @@ to raw text with no highlighting. An optional label can be added as the first
line with the prefix `####` (Python-like) and `///` (JavaScript-like). the line with the prefix `####` (Python-like) and `///` (JavaScript-like). the
indented block as plain text and preserve whitespace. indented block as plain text and preserve whitespace.
```python ```python {title="Using spaCy"}
### Using spaCy
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a sentence.") doc = nlp("This is a sentence.")
@ -403,7 +398,7 @@ adding `{highlight="..."}` to the headline. Acceptable ranges are spans like
> #### Markdown > #### Markdown
> >
> ````markdown_ > ````markdown
> ```python > ```python
> ### This is a title {highlight="1-2"} > ### This is a title {highlight="1-2"}
> import spacy > import spacy
@ -411,8 +406,7 @@ adding `{highlight="..."}` to the headline. Acceptable ranges are spans like
> ``` > ```
> ```` > ````
```python ```python {title="Using the matcher",highlight="5-7"}
### Using the matcher {highlight="5-7"}
import spacy import spacy
from spacy.matcher import Matcher from spacy.matcher import Matcher
@ -431,7 +425,7 @@ interactive widget defaults to a regular code block.
> #### Markdown > #### Markdown
> >
> ````markdown_ > ````markdown
> ```python > ```python
> ### {executable="true"} > ### {executable="true"}
> import spacy > import spacy
@ -439,8 +433,7 @@ interactive widget defaults to a regular code block.
> ``` > ```
> ```` > ````
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a sentence.") doc = nlp("This is a sentence.")
@ -454,7 +447,7 @@ original file is shown at the top of the widget.
> #### Markdown > #### Markdown
> >
> ````markdown_ > ````markdown
> ```python > ```python
> https://github.com/... > https://github.com/...
> ``` > ```
@ -470,9 +463,7 @@ original file is shown at the top of the widget.
https://github.com/explosion/spaCy/tree/master/spacy/language.py https://github.com/explosion/spaCy/tree/master/spacy/language.py
``` ```
### Infobox {#infobox} ### Infobox {id="infobox"}
import Infobox from 'components/infobox'
> #### JSX > #### JSX
> >
@ -508,9 +499,7 @@ blocks.
</Infobox> </Infobox>
### Accordion {#accordion} ### Accordion {id="accordion"}
import Accordion from 'components/accordion'
> #### JSX > #### JSX
> >
@ -537,9 +526,9 @@ sit amet dignissim justo congue.
</Accordion> </Accordion>
## Markdown reference {#markdown} ## Markdown reference {id="markdown"}
All page content and page meta lives in the `.md` files in the `/docs` All page content and page meta lives in the `.mdx` files in the `/docs`
directory. The frontmatter block at the top of each file defines the page title directory. The frontmatter block at the top of each file defines the page title
and other settings like the sidebar menu. and other settings like the sidebar menu.
@ -548,7 +537,7 @@ and other settings like the sidebar menu.
title: Page title title: Page title
--- ---
## Headline starting a section {#some_id} ## Headline starting a section {id="some_id"}
This is a regular paragraph with a [link](https://spacy.io) and **bold text**. This is a regular paragraph with a [link](https://spacy.io) and **bold text**.
@ -562,8 +551,7 @@ This is a regular paragraph with a [link](https://spacy.io) and **bold text**.
| -------- | -------- | | -------- | -------- |
| Column 1 | Column 2 | | Column 1 | Column 2 |
```python ```python {title="Code block title",highlight="2-3"}
### Code block title {highlight="2-3"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
doc = nlp("Hello world") doc = nlp("Hello world")
@ -585,7 +573,7 @@ In addition to the native markdown elements, you can use the components
[abbr]: https://spacy.io/styleguide#abbr [abbr]: https://spacy.io/styleguide#abbr
[tag]: https://spacy.io/styleguide#tag [tag]: https://spacy.io/styleguide#tag
## Editorial {#editorial} ## Editorial {id="editorial"}
- "spaCy" should always be spelled with a lowercase "s" and a capital "C", - "spaCy" should always be spelled with a lowercase "s" and a capital "C",
unless it specifically refers to the Python package or Python import `spacy` unless it specifically refers to the Python package or Python import `spacy`
@ -609,21 +597,16 @@ In addition to the native markdown elements, you can use the components
- ❌ The [`Span`](/api/span) and [`Token`](/api/token) objects are views of a - ❌ The [`Span`](/api/span) and [`Token`](/api/token) objects are views of a
[`Doc`](/api/doc). [`Span.as_doc`](/api/span#as_doc) creates a [`Doc`](/api/doc). [`Span.as_doc`](/api/span#as_doc) creates a
[`Doc`](/api/doc) object from a [`Span`](/api/span). [`Doc`](/api/doc) object from a [`Span`](/api/span).
- Other things we format as code are: references to trained pipeline packages
* Other things we format as code are: references to trained pipeline packages
like `en_core_web_sm` or file names like `code.py` or `meta.json`. like `en_core_web_sm` or file names like `code.py` or `meta.json`.
- ✅ After training, the `config.cfg` is saved to disk. - ✅ After training, the `config.cfg` is saved to disk.
- [Type annotations](#type-annotations) are a special type of code formatting,
* [Type annotations](#type-annotations) are a special type of code formatting,
expressed by wrapping the text in `~~` instead of backticks. The result looks expressed by wrapping the text in `~~` instead of backticks. The result looks
like this: ~~List[Doc]~~. All references to known types will be linked like this: ~~List[Doc]~~. All references to known types will be linked
automatically. automatically.
- ✅ The model has the input type ~~List[Doc]~~ and it outputs a - ✅ The model has the input type ~~List[Doc]~~ and it outputs a
~~List[Array2d]~~. ~~List[Array2d]~~.
- We try to keep links meaningful but short.
* We try to keep links meaningful but short.
- ✅ For details, see the usage guide on - ✅ For details, see the usage guide on
[training with custom code](/usage/training#custom-code). [training with custom code](/usage/training#custom-code).
- ❌ For details, see - ❌ For details, see

View File

@ -14,9 +14,9 @@ of the pipeline. The `Language` object coordinates these components. It takes
raw text and sends it through the pipeline, returning an **annotated document**. raw text and sends it through the pipeline, returning an **annotated document**.
It also orchestrates training and serialization. It also orchestrates training and serialization.
![Library architecture](../../images/architecture.svg) ![Library architecture {{w:1080, h:1254}}](/images/architecture.svg)
### Container objects {#architecture-containers} ### Container objects {id="architecture-containers"}
| Name | Description | | Name | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
@ -29,7 +29,7 @@ It also orchestrates training and serialization.
| [`SpanGroup`](/api/spangroup) | A named collection of spans belonging to a `Doc`. | | [`SpanGroup`](/api/spangroup) | A named collection of spans belonging to a `Doc`. |
| [`Token`](/api/token) | An individual token — i.e. a word, punctuation symbol, whitespace, etc. | | [`Token`](/api/token) | An individual token — i.e. a word, punctuation symbol, whitespace, etc. |
### Processing pipeline {#architecture-pipeline} ### Processing pipeline {id="architecture-pipeline"}
The processing pipeline consists of one or more **pipeline components** that are The processing pipeline consists of one or more **pipeline components** that are
called on the `Doc` in order. The tokenizer runs before the components. Pipeline called on the `Doc` in order. The tokenizer runs before the components. Pipeline
@ -39,7 +39,7 @@ rule-based modifications to the `Doc`. spaCy provides a range of built-in
components for different language processing tasks and also allows adding components for different language processing tasks and also allows adding
[custom components](/usage/processing-pipelines#custom-components). [custom components](/usage/processing-pipelines#custom-components).
![The processing pipeline](../../images/pipeline.svg) ![The processing pipeline](/images/pipeline.svg)
| Name | Description | | Name | Description |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------- | | ----------------------------------------------- | ------------------------------------------------------------------------------------------- |
@ -61,7 +61,7 @@ components for different language processing tasks and also allows adding
| [`Transformer`](/api/transformer) | Use a transformer model and set its outputs. | | [`Transformer`](/api/transformer) | Use a transformer model and set its outputs. |
| [Other functions](/api/pipeline-functions) | Automatically apply something to the `Doc`, e.g. to merge spans of tokens. | | [Other functions](/api/pipeline-functions) | Automatically apply something to the `Doc`, e.g. to merge spans of tokens. |
### Matchers {#architecture-matchers} ### Matchers {id="architecture-matchers"}
Matchers help you find and extract information from [`Doc`](/api/doc) objects Matchers help you find and extract information from [`Doc`](/api/doc) objects
based on match patterns describing the sequences you're looking for. A matcher based on match patterns describing the sequences you're looking for. A matcher
@ -73,7 +73,7 @@ operates on a `Doc` and gives you access to the matched tokens **in context**.
| [`Matcher`](/api/matcher) | Match sequences of tokens, based on pattern rules, similar to regular expressions. | | [`Matcher`](/api/matcher) | Match sequences of tokens, based on pattern rules, similar to regular expressions. |
| [`PhraseMatcher`](/api/phrasematcher) | Match sequences of tokens based on phrases. | | [`PhraseMatcher`](/api/phrasematcher) | Match sequences of tokens based on phrases. |
### Other classes {#architecture-other} ### Other classes {id="architecture-other"}
| Name | Description | | Name | Description |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------- | | ------------------------------------------------ | -------------------------------------------------------------------------------------------------- |

View File

@ -1,14 +1,13 @@
A named entity is a "real-world object" that's assigned a name for example, a A named entity is a "real-world object" that's assigned a name for example, a
person, a country, a product or a book title. spaCy can **recognize various person, a country, a product or a book title. spaCy can **recognize various
types of named entities in a document, by asking the model for a types of named entities in a document, by asking the model for a prediction**.
prediction**. Because models are statistical and strongly depend on the Because models are statistical and strongly depend on the examples they were
examples they were trained on, this doesn't always work _perfectly_ and might trained on, this doesn't always work _perfectly_ and might need some tuning
need some tuning later, depending on your use case. later, depending on your use case.
Named entities are available as the `ents` property of a `Doc`: Named entities are available as the `ents` property of a `Doc`:
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -32,7 +31,8 @@ for ent in doc.ents:
Using spaCy's built-in [displaCy visualizer](/usage/visualizers), here's what Using spaCy's built-in [displaCy visualizer](/usage/visualizers), here's what
our example sentence and its named entities look like: our example sentence and its named entities look like:
import DisplaCyEntHtml from 'images/displacy-ent1.html'; import { Iframe } from <Iframe
'components/embed' title="displaCy visualization of entities"
src="/images/displacy-ent1.html"
<Iframe title="displaCy visualization of entities" html={DisplaCyEntHtml} height={100} /> height={100}
/>

View File

@ -5,7 +5,7 @@ referred to as the **processing pipeline**. The pipeline used by the
and an entity recognizer. Each pipeline component returns the processed `Doc`, and an entity recognizer. Each pipeline component returns the processed `Doc`,
which is then passed on to the next component. which is then passed on to the next component.
![The processing pipeline](../../images/pipeline.svg) ![The processing pipeline](/images/pipeline.svg)
> - **Name**: ID of the pipeline component. > - **Name**: ID of the pipeline component.
> - **Component:** spaCy's implementation of the component. > - **Component:** spaCy's implementation of the component.
@ -35,8 +35,6 @@ the [config](/usage/training#config):
pipeline = ["tok2vec", "tagger", "parser", "ner"] pipeline = ["tok2vec", "tagger", "parser", "ner"]
``` ```
import Accordion from 'components/accordion.js'
<Accordion title="Does the order of pipeline components matter?" id="pipeline-components-order"> <Accordion title="Does the order of pipeline components matter?" id="pipeline-components-order">
The statistical components like the tagger or parser are typically independent The statistical components like the tagger or parser are typically independent

View File

@ -11,8 +11,7 @@ Linguistic annotations are available as
efficiency. So to get the readable string representation of an attribute, we efficiency. So to get the readable string representation of an attribute, we
need to add an underscore `_` to its name: need to add an underscore `_` to its name:
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -57,7 +56,8 @@ for token in doc:
Using spaCy's built-in [displaCy visualizer](/usage/visualizers), here's what Using spaCy's built-in [displaCy visualizer](/usage/visualizers), here's what
our example sentence and its dependencies look like: our example sentence and its dependencies look like:
import DisplaCyLongHtml from 'images/displacy-long.html'; import { Iframe } from <Iframe
'components/embed' title="displaCy visualization of dependencies and entities"
src="/images/displacy-long.html"
<Iframe title="displaCy visualization of dependencies and entities" html={DisplaCyLongHtml} height={450} /> height={450}
/>

View File

@ -4,8 +4,7 @@ language. For example, punctuation at the end of a sentence should be split off
whereas "U.K." should remain one token. Each `Doc` consists of individual whereas "U.K." should remain one token. Each `Doc` consists of individual
tokens, and we can iterate over them: tokens, and we can iterate over them:
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -41,7 +40,7 @@ marks.
> - **Suffix:** Character(s) at the end, e.g. `km`, `)`, `”`, `!`. > - **Suffix:** Character(s) at the end, e.g. `km`, `)`, `”`, `!`.
> - **Infix:** Character(s) in between, e.g. `-`, `--`, `/`, `…`. > - **Infix:** Character(s) in between, e.g. `-`, `--`, `/`, `…`.
![Example of the tokenization process](../../images/tokenization.svg) ![Example of the tokenization process](/images/tokenization.svg)
While punctuation rules are usually pretty general, tokenizer exceptions While punctuation rules are usually pretty general, tokenizer exceptions
strongly depend on the specifics of the individual language. This is why each strongly depend on the specifics of the individual language. This is why each

View File

@ -10,9 +10,9 @@ any other information.
Training is an iterative process in which the model's predictions are compared Training is an iterative process in which the model's predictions are compared
against the reference annotations in order to estimate the **gradient of the against the reference annotations in order to estimate the **gradient of the
loss**. The gradient of the loss is then used to calculate the gradient of the loss**. The gradient of the loss is then used to calculate the gradient of the
weights through [backpropagation](https://thinc.ai/docs/backprop101). The gradients weights through [backpropagation](https://thinc.ai/docs/backprop101). The
indicate how the weight values should be changed so that the model's predictions gradients indicate how the weight values should be changed so that the model's
become more similar to the reference labels over time. predictions become more similar to the reference labels over time.
> - **Training data:** Examples and their annotations. > - **Training data:** Examples and their annotations.
> - **Text:** The input text the model should predict a label for. > - **Text:** The input text the model should predict a label for.
@ -21,7 +21,7 @@ become more similar to the reference labels over time.
> Minimising the gradient of the weights should result in predictions that are > Minimising the gradient of the weights should result in predictions that are
> closer to the reference labels on the training data. > closer to the reference labels on the training data.
![The training process](../../images/training.svg) ![The training process](/images/training.svg)
When training a model, we don't just want it to memorize our examples we want When training a model, we don't just want it to memorize our examples we want
it to come up with a theory that can be **generalized across unseen data**. it to come up with a theory that can be **generalized across unseen data**.

View File

@ -1,12 +1,9 @@
import Infobox from 'components/infobox'
Similarity is determined by comparing **word vectors** or "word embeddings", Similarity is determined by comparing **word vectors** or "word embeddings",
multi-dimensional meaning representations of a word. Word vectors can be multi-dimensional meaning representations of a word. Word vectors can be
generated using an algorithm like generated using an algorithm like
[word2vec](https://en.wikipedia.org/wiki/Word2vec) and usually look like this: [word2vec](https://en.wikipedia.org/wiki/Word2vec) and usually look like this:
```python ```python {title="banana.vector"}
### banana.vector
array([2.02280000e-01, -7.66180009e-02, 3.70319992e-01, array([2.02280000e-01, -7.66180009e-02, 3.70319992e-01,
3.28450017e-02, -4.19569999e-01, 7.20689967e-02, 3.28450017e-02, -4.19569999e-01, 7.20689967e-02,
-3.74760002e-01, 5.74599989e-02, -1.24009997e-02, -3.74760002e-01, 5.74599989e-02, -1.24009997e-02,
@ -44,8 +41,7 @@ the [`Token.vector`](/api/token#vector) attribute.
default to an average of their token vectors. You can also check if a token has default to an average of their token vectors. You can also check if a token has
a vector assigned, and get the L2 norm, which can be used to normalize vectors. a vector assigned, and get the L2 norm, which can be used to normalize vectors.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_md") nlp = spacy.load("en_core_web_md")
@ -95,8 +91,7 @@ similarity.
> You should see that the similarity results are identical to the token > You should see that the similarity results are identical to the token
> similarity. > similarity.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_md") # make sure to use larger package! nlp = spacy.load("en_core_web_md") # make sure to use larger package!
@ -111,7 +106,7 @@ burgers = doc1[5]
print(french_fries, "<->", burgers, french_fries.similarity(burgers)) print(french_fries, "<->", burgers, french_fries.similarity(burgers))
``` ```
### What to expect from similarity results {#similarity-expectations} ### What to expect from similarity results {id="similarity-expectations"}
Computing similarity scores can be helpful in many situations, but it's also Computing similarity scores can be helpful in many situations, but it's also
important to maintain **realistic expectations** about what information it can important to maintain **realistic expectations** about what information it can
@ -136,7 +131,10 @@ useful for your purpose. Here are some important considerations to keep in mind:
<Infobox title="Tip: Check out sense2vec" emoji="💡"> <Infobox title="Tip: Check out sense2vec" emoji="💡">
[![](../../images/sense2vec.jpg)](https://github.com/explosion/sense2vec) <Image
src="/images/sense2vec.jpg"
href="https://github.com/explosion/sense2vec"
/>
[`sense2vec`](https://github.com/explosion/sense2vec) is a library developed by [`sense2vec`](https://github.com/explosion/sense2vec) is a library developed by
us that builds on top of spaCy and lets you train and query more interesting and us that builds on top of spaCy and lets you train and query more interesting and

View File

@ -1,5 +1,3 @@
import { Help } from 'components/typography'; import Link from 'components/link'
<figure> <figure>
| Pipeline | Parser | Tagger | NER | | Pipeline | Parser | Tagger | NER |
@ -8,7 +6,7 @@ import { Help } from 'components/typography'; import Link from 'components/link'
| [`en_core_web_lg`](/models/en#en_core_web_lg) (spaCy v3) | 92.0 | 97.4 | 85.5 | | [`en_core_web_lg`](/models/en#en_core_web_lg) (spaCy v3) | 92.0 | 97.4 | 85.5 |
| `en_core_web_lg` (spaCy v2) | 91.9 | 97.2 | 85.5 | | `en_core_web_lg` (spaCy v2) | 91.9 | 97.2 | 85.5 |
<figcaption class="caption"> <figcaption className="caption">
**Full pipeline accuracy** on the **Full pipeline accuracy** on the
[OntoNotes 5.0](https://catalog.ldc.upenn.edu/LDC2013T19) corpus (reported on [OntoNotes 5.0](https://catalog.ldc.upenn.edu/LDC2013T19) corpus (reported on
@ -26,7 +24,7 @@ the development set).
| Stanza (StanfordNLP)<sup>1</sup> | 88.8 | 92.1 | | Stanza (StanfordNLP)<sup>1</sup> | 88.8 | 92.1 |
| Flair<sup>2</sup> | 89.7 | 93.1 | | Flair<sup>2</sup> | 89.7 | 93.1 |
<figcaption class="caption"> <figcaption className="caption">
**Named entity recognition accuracy** on the **Named entity recognition accuracy** on the
[OntoNotes 5.0](https://catalog.ldc.upenn.edu/LDC2013T19) and [OntoNotes 5.0](https://catalog.ldc.upenn.edu/LDC2013T19) and

View File

@ -74,7 +74,7 @@ of performance.
</Accordion> </Accordion>
## Shared embedding layers {#embedding-layers} ## Shared embedding layers {id="embedding-layers"}
spaCy lets you share a single transformer or other token-to-vector ("tok2vec") spaCy lets you share a single transformer or other token-to-vector ("tok2vec")
embedding layer between multiple components. You can even update the shared embedding layer between multiple components. You can even update the shared
@ -85,7 +85,7 @@ difficult to swap components or retrain parts of the pipeline. Multi-task
learning can affect your accuracy (either positively or negatively), and may learning can affect your accuracy (either positively or negatively), and may
require some retuning of your hyper-parameters. require some retuning of your hyper-parameters.
![Pipeline components using a shared embedding component vs. independent embedding layers](../images/tok2vec.svg) ![Pipeline components using a shared embedding component vs. independent embedding layers](/images/tok2vec.svg)
| Shared | Independent | | Shared | Independent |
| ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
@ -99,7 +99,7 @@ components by adding a [`Transformer`](/api/transformer) or
later in the pipeline can "connect" to it by including a **listener layer** like later in the pipeline can "connect" to it by including a **listener layer** like
[Tok2VecListener](/api/architectures#Tok2VecListener) within their model. [Tok2VecListener](/api/architectures#Tok2VecListener) within their model.
![Pipeline components listening to shared embedding component](../images/tok2vec-listener.svg) ![Pipeline components listening to shared embedding component](/images/tok2vec-listener.svg)
At the beginning of training, the [`Tok2Vec`](/api/tok2vec) component will grab At the beginning of training, the [`Tok2Vec`](/api/tok2vec) component will grab
a reference to the relevant listener layers in the rest of your pipeline. When a reference to the relevant listener layers in the rest of your pipeline. When
@ -113,7 +113,7 @@ transformer outputs to the
[`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute, [`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute,
giving you access to them after the pipeline has finished running. giving you access to them after the pipeline has finished running.
### Example: Shared vs. independent config {#embedding-layers-config} ### Example: Shared vs. independent config {id="embedding-layers-config"}
The [config system](/usage/training#config) lets you express model configuration The [config system](/usage/training#config) lets you express model configuration
for both shared and independent embedding layers. The shared setup uses a single for both shared and independent embedding layers. The shared setup uses a single
@ -123,8 +123,7 @@ the entity recognizer, use a
[Tok2VecListener](/api/architectures#Tok2VecListener) layer as their model's [Tok2VecListener](/api/architectures#Tok2VecListener) layer as their model's
`tok2vec` argument, which connects to the `tok2vec` component model. `tok2vec` argument, which connects to the `tok2vec` component model.
```ini ```ini {title="Shared",highlight="1-2,4-5,19-20"}
### Shared {highlight="1-2,4-5,19-20"}
[components.tok2vec] [components.tok2vec]
factory = "tok2vec" factory = "tok2vec"
@ -152,8 +151,7 @@ In the independent setup, the entity recognizer component defines its own
same. This makes them fully independent and doesn't require an upstream same. This makes them fully independent and doesn't require an upstream
[`Tok2Vec`](/api/tok2vec) component to be present in the pipeline. [`Tok2Vec`](/api/tok2vec) component to be present in the pipeline.
```ini ```ini {title="Independent", highlight="7-8"}
### Independent {highlight="7-8"}
[components.ner] [components.ner]
factory = "ner" factory = "ner"
@ -170,9 +168,9 @@ factory = "ner"
@architectures = "spacy.MaxoutWindowEncoder.v2" @architectures = "spacy.MaxoutWindowEncoder.v2"
``` ```
<!-- TODO: Once rehearsal is tested, mention it here. --> {/* TODO: Once rehearsal is tested, mention it here. */}
## Using transformer models {#transformers} ## Using transformer models {id="transformers"}
Transformers are a family of neural network architectures that compute **dense, Transformers are a family of neural network architectures that compute **dense,
context-sensitive representations** for the tokens in your documents. Downstream context-sensitive representations** for the tokens in your documents. Downstream
@ -188,7 +186,7 @@ transformer models, but for practical purposes, you can simply think of them as
drop-in replacements that let you achieve **higher accuracy** in exchange for drop-in replacements that let you achieve **higher accuracy** in exchange for
**higher training and runtime costs**. **higher training and runtime costs**.
### Setup and installation {#transformers-installation} ### Setup and installation {id="transformers-installation"}
> #### System requirements > #### System requirements
> >
@ -210,22 +208,20 @@ your package manager and CUDA version. If you skip this step, pip will install
PyTorch as a dependency below, but it may not find the best version for your PyTorch as a dependency below, but it may not find the best version for your
setup. setup.
```bash ```bash {title="Example: Install PyTorch 1.11.0 for CUDA 11.3 with pip"}
### Example: Install PyTorch 1.11.0 for CUDA 11.3 with pip
# See: https://pytorch.org/get-started/locally/ # See: https://pytorch.org/get-started/locally/
$ pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html $ pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
``` ```
Next, install spaCy with the extras for your CUDA version and transformers. The Next, install spaCy with the extras for your CUDA version and transformers. The
CUDA extra (e.g., `cuda102`, `cuda113`) installs the correct version of CUDA extra (e.g., `cuda102`, `cuda113`) installs the correct version of
[`cupy`](https://docs.cupy.dev/en/stable/install.html#installing-cupy), which [`cupy`](https://docs.cupy.dev/en/stable/install.html#installing-cupy), which is
is just like `numpy`, but for GPU. You may also need to set the `CUDA_PATH` just like `numpy`, but for GPU. You may also need to set the `CUDA_PATH`
environment variable if your CUDA runtime is installed in a non-standard environment variable if your CUDA runtime is installed in a non-standard
location. Putting it all together, if you had installed CUDA 11.3 in location. Putting it all together, if you had installed CUDA 11.3 in
`/opt/nvidia/cuda`, you would run: `/opt/nvidia/cuda`, you would run:
```bash ```bash {title="Installation with CUDA"}
### Installation with CUDA
$ export CUDA_PATH="/opt/nvidia/cuda" $ export CUDA_PATH="/opt/nvidia/cuda"
$ pip install -U %%SPACY_PKG_NAME[cuda113,transformers]%%SPACY_PKG_FLAGS $ pip install -U %%SPACY_PKG_NAME[cuda113,transformers]%%SPACY_PKG_FLAGS
``` ```
@ -235,12 +231,11 @@ that require [`SentencePiece`](https://github.com/google/sentencepiece) (e.g.,
ALBERT, CamemBERT, XLNet, Marian, and T5), install the additional dependencies ALBERT, CamemBERT, XLNet, Marian, and T5), install the additional dependencies
with: with:
```bash ```bash {title="Install sentencepiece"}
### Install sentencepiece
$ pip install transformers[sentencepiece] $ pip install transformers[sentencepiece]
``` ```
### Runtime usage {#transformers-runtime} ### Runtime usage {id="transformers-runtime"}
Transformer models can be used as **drop-in replacements** for other types of Transformer models can be used as **drop-in replacements** for other types of
neural networks, so your spaCy pipeline can include them in a way that's neural networks, so your spaCy pipeline can include them in a way that's
@ -249,7 +244,7 @@ the standard way, like any other spaCy pipeline. Instead of using the
transformers as subnetworks directly, you can also use them via the transformers as subnetworks directly, you can also use them via the
[`Transformer`](/api/transformer) pipeline component. [`Transformer`](/api/transformer) pipeline component.
![The processing pipeline with the transformer component](../images/pipeline_transformer.svg) ![The processing pipeline with the transformer component](/images/pipeline_transformer.svg)
The `Transformer` component sets the The `Transformer` component sets the
[`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute, [`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute,
@ -257,12 +252,11 @@ which lets you access the transformers outputs at runtime. The trained
transformer-based [pipelines](/models) provided by spaCy end on `_trf`, e.g. transformer-based [pipelines](/models) provided by spaCy end on `_trf`, e.g.
[`en_core_web_trf`](/models/en#en_core_web_trf). [`en_core_web_trf`](/models/en#en_core_web_trf).
```cli ```bash
$ python -m spacy download en_core_web_trf $ python -m spacy download en_core_web_trf
``` ```
```python ```python {title="Example"}
### Example
import spacy import spacy
from thinc.api import set_gpu_allocator, require_gpu from thinc.api import set_gpu_allocator, require_gpu
@ -299,7 +293,7 @@ assert isinstance(doc._.custom_attr, TransformerData)
print(doc._.custom_attr.tensors) print(doc._.custom_attr.tensors)
``` ```
### Training usage {#transformers-training} ### Training usage {id="transformers-training"}
The recommended workflow for training is to use spaCy's The recommended workflow for training is to use spaCy's
[config system](/usage/training#config), usually via the [config system](/usage/training#config), usually via the
@ -309,14 +303,13 @@ of objects by referring to creation functions, including functions you register
yourself. For details on how to get started with training your own model, check yourself. For details on how to get started with training your own model, check
out the [training quickstart](/usage/training#quickstart). out the [training quickstart](/usage/training#quickstart).
<!-- TODO: <Project id="pipelines/transformers"> {/* TODO: <Project id="pipelines/transformers"> */}
The easiest way to get started is to clone a transformers-based project {/* The easiest way to get started is to clone a transformers-based project */}
template. Swap in your data, edit the settings and hyperparameters and train, {/* template. Swap in your data, edit the settings and hyperparameters and train, */}
evaluate, package and visualize your model. {/* evaluate, package and visualize your model. */}
</Project> {/* </Project> */}
-->
The `[components]` section in the [`config.cfg`](/api/data-formats#config) The `[components]` section in the [`config.cfg`](/api/data-formats#config)
describes the pipeline components and the settings used to construct them, describes the pipeline components and the settings used to construct them,
@ -344,8 +337,7 @@ component:
> ) > )
> ``` > ```
```ini ```ini {title="config.cfg",excerpt="true"}
### config.cfg (excerpt)
[components.transformer] [components.transformer]
factory = "transformer" factory = "transformer"
max_batch_items = 4096 max_batch_items = 4096
@ -405,7 +397,7 @@ all defaults.
</Infobox> </Infobox>
### Customizing the settings {#transformers-training-custom-settings} ### Customizing the settings {id="transformers-training-custom-settings"}
To change any of the settings, you can edit the `config.cfg` and re-run the To change any of the settings, you can edit the `config.cfg` and re-run the
training. To change any of the functions, like the span getter, you can replace training. To change any of the functions, like the span getter, you can replace
@ -425,8 +417,7 @@ subsentences of at most `max_length` tokens are returned.
> max_length = 25 > max_length = 25
> ``` > ```
```python ```python {title="code.py"}
### code.py
import spacy_transformers import spacy_transformers
@spacy_transformers.registry.span_getters("custom_sent_spans") @spacy_transformers.registry.span_getters("custom_sent_spans")
@ -454,11 +445,11 @@ function. You can make it available via the `--code` argument that can point to
a Python file. For more details on training with custom code, see the a Python file. For more details on training with custom code, see the
[training documentation](/usage/training#custom-functions). [training documentation](/usage/training#custom-functions).
```cli ```bash
python -m spacy train ./config.cfg --code ./code.py python -m spacy train ./config.cfg --code ./code.py
``` ```
### Customizing the model implementations {#training-custom-model} ### Customizing the model implementations {id="training-custom-model"}
The [`Transformer`](/api/transformer) component expects a Thinc The [`Transformer`](/api/transformer) component expects a Thinc
[`Model`](https://thinc.ai/docs/api-model) object to be passed in as its `model` [`Model`](https://thinc.ai/docs/api-model) object to be passed in as its `model`
@ -476,8 +467,7 @@ is where we'll plug in our transformer model, using the
[TransformerListener](/api/architectures#TransformerListener) layer, which [TransformerListener](/api/architectures#TransformerListener) layer, which
sneakily delegates to the `Transformer` pipeline component. sneakily delegates to the `Transformer` pipeline component.
```ini ```ini {title="config.cfg (excerpt)",highlight="12"}
### config.cfg (excerpt) {highlight="12"}
[components.ner] [components.ner]
factory = "ner" factory = "ner"
@ -517,7 +507,7 @@ custom learning rate for each component. Instead of a constant, you can also
provide a schedule, allowing you to freeze the shared parameters at the start of provide a schedule, allowing you to freeze the shared parameters at the start of
training. training.
## Static vectors {#static-vectors} ## Static vectors {id="static-vectors"}
If your pipeline includes a **word vectors table**, you'll be able to use the If your pipeline includes a **word vectors table**, you'll be able to use the
`.similarity()` method on the [`Doc`](/api/doc), [`Span`](/api/span), `.similarity()` method on the [`Doc`](/api/doc), [`Span`](/api/span),
@ -531,8 +521,7 @@ Word vectors in spaCy are "static" in the sense that they are not learned
parameters of the statistical models, and spaCy itself does not feature any parameters of the statistical models, and spaCy itself does not feature any
algorithms for learning word vector tables. You can train a word vectors table algorithms for learning word vector tables. You can train a word vectors table
using tools such as [floret](https://github.com/explosion/floret), using tools such as [floret](https://github.com/explosion/floret),
[Gensim](https://radimrehurek.com/gensim/), [Gensim](https://radimrehurek.com/gensim/), [FastText](https://fasttext.cc/) or
[FastText](https://fasttext.cc/) or
[GloVe](https://nlp.stanford.edu/projects/glove/), or download existing [GloVe](https://nlp.stanford.edu/projects/glove/), or download existing
pretrained vectors. The [`init vectors`](/api/cli#init-vectors) command lets you pretrained vectors. The [`init vectors`](/api/cli#init-vectors) command lets you
convert vectors for use with spaCy and will give you a directory you can load or convert vectors for use with spaCy and will give you a directory you can load or
@ -547,7 +536,7 @@ the usage guide on
</Infobox> </Infobox>
### Using word vectors in your models {#word-vectors-models} ### Using word vectors in your models {id="word-vectors-models"}
Many neural network models are able to use word vector tables as additional Many neural network models are able to use word vector tables as additional
features, which sometimes results in significant improvements in accuracy. features, which sometimes results in significant improvements in accuracy.
@ -580,7 +569,7 @@ handled by the [StaticVectors](/api/architectures#StaticVectors) layer.
</Infobox> </Infobox>
#### Creating a custom embedding layer {#custom-embedding-layer} #### Creating a custom embedding layer {id="custom-embedding-layer"}
The [MultiHashEmbed](/api/architectures#StaticVectors) layer is spaCy's The [MultiHashEmbed](/api/architectures#StaticVectors) layer is spaCy's
recommended strategy for constructing initial word representations for your recommended strategy for constructing initial word representations for your
@ -643,7 +632,7 @@ def MyCustomVectors(
) )
``` ```
## Pretraining {#pretraining} ## Pretraining {id="pretraining"}
The [`spacy pretrain`](/api/cli#pretrain) command lets you initialize your The [`spacy pretrain`](/api/cli#pretrain) command lets you initialize your
models with **information from raw text**. Without pretraining, the models for models with **information from raw text**. Without pretraining, the models for
@ -679,14 +668,14 @@ You can add a `[pretraining]` block to your config by setting the
`--pretraining` flag on [`init config`](/api/cli#init-config) or `--pretraining` flag on [`init config`](/api/cli#init-config) or
[`init fill-config`](/api/cli#init-fill-config): [`init fill-config`](/api/cli#init-fill-config):
```cli ```bash
$ python -m spacy init fill-config config.cfg config_pretrain.cfg --pretraining $ python -m spacy init fill-config config.cfg config_pretrain.cfg --pretraining
``` ```
You can then run [`spacy pretrain`](/api/cli#pretrain) with the updated config You can then run [`spacy pretrain`](/api/cli#pretrain) with the updated config
and pass in optional config overrides, like the path to the raw text file: and pass in optional config overrides, like the path to the raw text file:
```cli ```bash
$ python -m spacy pretrain config_pretrain.cfg ./output --paths.raw_text text.jsonl $ python -m spacy pretrain config_pretrain.cfg ./output --paths.raw_text text.jsonl
``` ```
@ -700,7 +689,7 @@ change the [objective](#pretraining-objectives).
%%GITHUB_SPACY/spacy/default_config_pretraining.cfg %%GITHUB_SPACY/spacy/default_config_pretraining.cfg
``` ```
### How pretraining works {#pretraining-details} ### How pretraining works {id="pretraining-details"}
The impact of [`spacy pretrain`](/api/cli#pretrain) varies, but it will usually The impact of [`spacy pretrain`](/api/cli#pretrain) varies, but it will usually
be worth trying if you're **not using a transformer** model and you have be worth trying if you're **not using a transformer** model and you have
@ -726,7 +715,7 @@ a "tok2vec" layer). The most common workflow is to use the
[`Tok2Vec`](/api/tok2vec) component to create a shared token-to-vector layer for [`Tok2Vec`](/api/tok2vec) component to create a shared token-to-vector layer for
several components of your pipeline, and apply pretraining to its whole model. several components of your pipeline, and apply pretraining to its whole model.
#### Configuring the pretraining {#pretraining-configure} #### Configuring the pretraining {id="pretraining-configure"}
The [`spacy pretrain`](/api/cli#pretrain) command is configured using the The [`spacy pretrain`](/api/cli#pretrain) command is configured using the
`[pretraining]` section of your [config file](/usage/training#config). The `[pretraining]` section of your [config file](/usage/training#config). The
@ -737,8 +726,7 @@ whole model), or a
spaCy's built-in model architectures have a reference named `"tok2vec"` that spaCy's built-in model architectures have a reference named `"tok2vec"` that
will refer to the right layer. will refer to the right layer.
```ini ```ini {title="config.cfg"}
### config.cfg
# 1. Use the whole model of the "tok2vec" component # 1. Use the whole model of the "tok2vec" component
[pretraining] [pretraining]
component = "tok2vec" component = "tok2vec"
@ -750,7 +738,7 @@ component = "textcat"
layer = "tok2vec" layer = "tok2vec"
``` ```
#### Connecting pretraining to training {#pretraining-training} #### Connecting pretraining to training {id="pretraining-training"}
To benefit from pretraining, your training step needs to know to initialize its To benefit from pretraining, your training step needs to know to initialize its
`tok2vec` component with the weights learned from the pretraining step. You do `tok2vec` component with the weights learned from the pretraining step. You do
@ -761,8 +749,7 @@ A pretraining step that runs for 5 epochs with an output path of `pretrain/`, as
an example, produces `pretrain/model0.bin` through `pretrain/model4.bin`. To an example, produces `pretrain/model0.bin` through `pretrain/model4.bin`. To
make use of the final output, you could fill in this value in your config file: make use of the final output, you could fill in this value in your config file:
```ini ```ini {title="config.cfg"}
### config.cfg
[paths] [paths]
init_tok2vec = "pretrain/model4.bin" init_tok2vec = "pretrain/model4.bin"
@ -781,7 +768,7 @@ an existing pipeline, so it goes in `initialize.init_tok2vec`.
</Infobox> </Infobox>
#### Pretraining objectives {#pretraining-objectives} #### Pretraining objectives {id="pretraining-objectives"}
> ```ini > ```ini
> ### Characters objective > ### Characters objective

View File

@ -8,7 +8,7 @@ menu:
# TODO: - ['Citing spaCy', 'citation'] # TODO: - ['Citing spaCy', 'citation']
--- ---
## Comparison {#comparison hidden="true"} ## Comparison {id="comparison",hidden="true"}
spaCy is a **free, open-source library** for advanced **Natural Language spaCy is a **free, open-source library** for advanced **Natural Language
Processing** (NLP) in Python. It's designed specifically for **production use** Processing** (NLP) in Python. It's designed specifically for **production use**
@ -16,13 +16,11 @@ and helps you build applications that process and "understand" large volumes of
text. It can be used to build information extraction or natural language text. It can be used to build information extraction or natural language
understanding systems. understanding systems.
### Feature overview {#comparison-features} ### Feature overview {id="comparison-features"}
import Features from 'widgets/features.js'
<Features /> <Features />
### When should I use spaCy? {#comparison-usage} ### When should I use spaCy? {id="comparison-usage"}
- ✅ **I'm a beginner and just getting started with NLP.** spaCy makes it easy - ✅ **I'm a beginner and just getting started with NLP.** spaCy makes it easy
to get started and comes with extensive documentation, including a to get started and comes with extensive documentation, including a
@ -51,13 +49,13 @@ import Features from 'widgets/features.js'
can use it to make the results of your research easily available for others to can use it to make the results of your research easily available for others to
use, e.g. via a custom spaCy component. use, e.g. via a custom spaCy component.
## Benchmarks {#benchmarks} ## Benchmarks {id="benchmarks"}
spaCy v3.0 introduces transformer-based pipelines that bring spaCy's accuracy spaCy v3.0 introduces transformer-based pipelines that bring spaCy's accuracy
right up to **current state-of-the-art**. You can also use a CPU-optimized right up to **current state-of-the-art**. You can also use a CPU-optimized
pipeline, which is less accurate but much cheaper to run. pipeline, which is less accurate but much cheaper to run.
<!-- TODO: update benchmarks and intro --> {/* TODO: update benchmarks and intro */}
> #### Evaluation details > #### Evaluation details
> >
@ -69,8 +67,6 @@ pipeline, which is less accurate but much cheaper to run.
> gold-standard segmentation and tokenization, from a pretty specific type of > gold-standard segmentation and tokenization, from a pretty specific type of
> text (articles from a single newspaper, 1984-1989). > text (articles from a single newspaper, 1984-1989).
import Benchmarks from 'usage/\_benchmarks-models.md'
<Benchmarks /> <Benchmarks />
<figure> <figure>
@ -81,7 +77,7 @@ import Benchmarks from 'usage/\_benchmarks-models.md'
| [Mrini et al.](https://khalilmrini.github.io/Label_Attention_Layer.pdf) (2019) | 97.4 | 96.3 | | [Mrini et al.](https://khalilmrini.github.io/Label_Attention_Layer.pdf) (2019) | 97.4 | 96.3 |
| [Zhou and Zhao](https://www.aclweb.org/anthology/P19-1230/) (2019) | 97.2 | 95.7 | | [Zhou and Zhao](https://www.aclweb.org/anthology/P19-1230/) (2019) | 97.2 | 95.7 |
<figcaption class="caption"> <figcaption className="caption">
**Dependency parsing accuracy** on the Penn Treebank. See **Dependency parsing accuracy** on the Penn Treebank. See
[NLP-progress](http://nlpprogress.com/english/dependency_parsing.html) for more [NLP-progress](http://nlpprogress.com/english/dependency_parsing.html) for more
@ -92,7 +88,7 @@ results. Project template:
</figure> </figure>
### Speed comparison {#benchmarks-speed} ### Speed comparison {id="benchmarks-speed"}
We compare the speed of different NLP libraries, measured in words per second We compare the speed of different NLP libraries, measured in words per second
(WPS) - higher is better. The evaluation was performed on 10,000 Reddit (WPS) - higher is better. The evaluation was performed on 10,000 Reddit
@ -108,7 +104,7 @@ comments.
| Flair | `pos`(`-fast`) & `ner`(`-fast`) | 323 | 1,184 | | Flair | `pos`(`-fast`) & `ner`(`-fast`) | 323 | 1,184 |
| UDPipe | `english-ewt-ud-2.5` | 1,101 | _n/a_ | | UDPipe | `english-ewt-ud-2.5` | 1,101 | _n/a_ |
<figcaption class="caption"> <figcaption className="caption">
**End-to-end processing speed** on raw unannotated text. Project template: **End-to-end processing speed** on raw unannotated text. Project template:
[`benchmarks/speed`](%%GITHUB_PROJECTS/benchmarks/speed). [`benchmarks/speed`](%%GITHUB_PROJECTS/benchmarks/speed).
@ -117,6 +113,4 @@ comments.
</figure> </figure>
<!-- TODO: ## Citing spaCy {#citation} {/* TODO: ## Citing spaCy {id="citation"} */}
-->

View File

@ -16,18 +16,16 @@ menu:
> website to [**v2.spacy.io**](https://v2.spacy.io/docs). To see what's changed > website to [**v2.spacy.io**](https://v2.spacy.io/docs). To see what's changed
> and how to migrate, see the [v3.0 guide](/usage/v3). > and how to migrate, see the [v3.0 guide](/usage/v3).
import QuickstartInstall from 'widgets/quickstart-install.js'
<QuickstartInstall id="quickstart" /> <QuickstartInstall id="quickstart" />
## Installation instructions {#installation} ## Installation instructions {id="installation"}
spaCy is compatible with **64-bit CPython 3.6+** and runs on **Unix/Linux**, spaCy is compatible with **64-bit CPython 3.6+** and runs on **Unix/Linux**,
**macOS/OS X** and **Windows**. The latest spaCy releases are available over **macOS/OS X** and **Windows**. The latest spaCy releases are available over
[pip](https://pypi.python.org/pypi/spacy) and [pip](https://pypi.python.org/pypi/spacy) and
[conda](https://anaconda.org/conda-forge/spacy). [conda](https://anaconda.org/conda-forge/spacy).
### pip {#pip} ### pip {id="pip"}
Using pip, spaCy releases are available as source packages and binary wheels. Using pip, spaCy releases are available as source packages and binary wheels.
Before you install spaCy and its dependencies, make sure that your `pip`, Before you install spaCy and its dependencies, make sure that your `pip`,
@ -38,7 +36,7 @@ Before you install spaCy and its dependencies, make sure that your `pip`,
> After installation you typically want to download a trained pipeline. For more > After installation you typically want to download a trained pipeline. For more
> info and available packages, see the [models directory](/models). > info and available packages, see the [models directory](/models).
> >
> ```cli > ```bash
> $ python -m spacy download en_core_web_sm > $ python -m spacy download en_core_web_sm
> >
> >>> import spacy > >>> import spacy
@ -79,7 +77,7 @@ spaCy's [`setup.cfg`](%%GITHUB_SPACY/setup.cfg) for details on what's included.
| `apple` | Install [`thinc-apple-ops`](https://github.com/explosion/thinc-apple-ops) to improve performance on an Apple M1. | | `apple` | Install [`thinc-apple-ops`](https://github.com/explosion/thinc-apple-ops) to improve performance on an Apple M1. |
| `ja`, `ko`, `th` | Install additional dependencies required for tokenization for the [languages](/usage/models#languages). | | `ja`, `ko`, `th` | Install additional dependencies required for tokenization for the [languages](/usage/models#languages). |
### conda {#conda} ### conda {id="conda"}
Thanks to our great community, we've been able to re-add conda support. You can Thanks to our great community, we've been able to re-add conda support. You can
also install spaCy via `conda-forge`: also install spaCy via `conda-forge`:
@ -92,7 +90,7 @@ For the feedstock including the build recipe and configuration, check out
[this repository](https://github.com/conda-forge/spacy-feedstock). Note that we [this repository](https://github.com/conda-forge/spacy-feedstock). Note that we
currently don't publish any [pre-releases](#changelog-pre) on conda. currently don't publish any [pre-releases](#changelog-pre) on conda.
### Upgrading spaCy {#upgrading} ### Upgrading spaCy {id="upgrading"}
> #### Upgrading from v2 to v3 > #### Upgrading from v2 to v3
> >
@ -116,12 +114,12 @@ version. If incompatible packages are found, tips and installation instructions
are printed. It's recommended to run the command with `python -m` to make sure are printed. It's recommended to run the command with `python -m` to make sure
you're executing the correct version of spaCy. you're executing the correct version of spaCy.
```cli ```bash
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS $ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
$ python -m spacy validate $ python -m spacy validate
``` ```
### Run spaCy with GPU {#gpu new="2.0.14"} ### Run spaCy with GPU {id="gpu",version="2.0.14"}
As of v2.0, spaCy comes with neural network models that are implemented in our As of v2.0, spaCy comes with neural network models that are implemented in our
machine learning library, [Thinc](https://thinc.ai). For GPU support, we've been machine learning library, [Thinc](https://thinc.ai). For GPU support, we've been
@ -151,7 +149,7 @@ spacy.prefer_gpu()
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
``` ```
### Compile from source {#source} ### Compile from source {id="source"}
The other way to install spaCy is to clone its The other way to install spaCy is to clone its
[GitHub repository](https://github.com/explosion/spaCy) and build it from [GitHub repository](https://github.com/explosion/spaCy) and build it from
@ -181,16 +179,13 @@ $ pip install --no-build-isolation --editable .[lookups,cuda102]
How to install compilers and related build tools: How to install compilers and related build tools:
<a id="source-ubuntu"></a><a id="source-osx"></a><a id="source-windows"></a> - <strong id="source-ubuntu">Ubuntu:</strong> Install system-level dependencies via
`apt-get`: `sudo apt-get install build-essential python-dev git`
- **Ubuntu:** Install system-level dependencies via `apt-get`: - <strong id="source-osx">macOS / OS X:</strong> Install a recent version of [XCode](https://developer.apple.com/xcode/),
`sudo apt-get install build-essential python-dev git` including the so-called "Command Line Tools". macOS and OS X ship with Python and
- **macOS / OS X:** Install a recent version of Git preinstalled.
[XCode](https://developer.apple.com/xcode/), including the so-called "Command - <strong id="source-windows">Windows:</strong> Install a version of the [Visual
Line Tools". macOS and OS X ship with Python and Git preinstalled. C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) or
- **Windows:** Install a version of the
[Visual C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)
or
[Visual Studio Express](https://www.visualstudio.com/vs/visual-studio-express/) [Visual Studio Express](https://www.visualstudio.com/vs/visual-studio-express/)
that matches the version that was used to compile your Python interpreter. that matches the version that was used to compile your Python interpreter.
@ -203,7 +198,7 @@ your environment requires an older version of `numpy`.
If `numpy` gets downgraded from the most recent release at any point after If `numpy` gets downgraded from the most recent release at any point after
you've compiled `spacy`, you might see an error that looks like this: you've compiled `spacy`, you might see an error that looks like this:
```none ```
numpy.ndarray size changed, may indicate binary incompatibility. numpy.ndarray size changed, may indicate binary incompatibility.
``` ```
@ -230,15 +225,15 @@ package to see what the oldest recommended versions of `numpy` are.
(_Warning_: don't use `pip install -c constraints.txt` instead of (_Warning_: don't use `pip install -c constraints.txt` instead of
`PIP_CONSTRAINT`, since this isn't applied to the isolated build environments.) `PIP_CONSTRAINT`, since this isn't applied to the isolated build environments.)
#### Additional options for developers {#source-developers} #### Additional options for developers {id="source-developers"}
Some additional options may be useful for spaCy developers who are editing the Some additional options may be useful for spaCy developers who are editing the
source code and recompiling frequently. source code and recompiling frequently.
- Install in editable mode. Changes to `.py` files will be reflected as soon - Install in editable mode. Changes to `.py` files will be reflected as soon as
as the files are saved, but edits to Cython files (`.pxd`, `.pyx`) will the files are saved, but edits to Cython files (`.pxd`, `.pyx`) will require
require the `pip install` command below to be run again. Before installing in the `pip install` command below to be run again. Before installing in editable
editable mode, be sure you have removed any previous installs with mode, be sure you have removed any previous installs with
`pip uninstall spacy`, which you may need to run multiple times to remove all `pip uninstall spacy`, which you may need to run multiple times to remove all
traces of earlier installs. traces of earlier installs.
@ -247,8 +242,8 @@ source code and recompiling frequently.
$ pip install --no-build-isolation --editable . $ pip install --no-build-isolation --editable .
``` ```
- Build in parallel. Starting in v3.4.0, you can specify the number of - Build in parallel. Starting in v3.4.0, you can specify the number of build
build jobs with the environment variable `SPACY_NUM_BUILD_JOBS`: jobs with the environment variable `SPACY_NUM_BUILD_JOBS`:
```bash ```bash
$ pip install -r requirements.txt $ pip install -r requirements.txt
@ -264,7 +259,7 @@ source code and recompiling frequently.
$ python setup.py develop $ python setup.py develop
``` ```
### Building an executable {#executable} ### Building an executable {id="executable"}
The spaCy repository includes a [`Makefile`](%%GITHUB_SPACY/Makefile) that The spaCy repository includes a [`Makefile`](%%GITHUB_SPACY/Makefile) that
builds an executable zip file using [`pex`](https://github.com/pantsbuild/pex) builds an executable zip file using [`pex`](https://github.com/pantsbuild/pex)
@ -298,7 +293,7 @@ You can configure the build process with the following environment variables:
| `PYVER` | The Python version to build against. This version needs to be available on your build and runtime machines. Defaults to `3.6`. | | `PYVER` | The Python version to build against. This version needs to be available on your build and runtime machines. Defaults to `3.6`. |
| `WHEELHOUSE` | Directory to store the wheel files during compilation. Defaults to `./wheelhouse`. | | `WHEELHOUSE` | Directory to store the wheel files during compilation. Defaults to `./wheelhouse`. |
### Run tests {#run-tests} ### Run tests {id="run-tests"}
spaCy comes with an [extensive test suite](%%GITHUB_SPACY/spacy/tests). In order spaCy comes with an [extensive test suite](%%GITHUB_SPACY/spacy/tests). In order
to run the tests, you'll usually want to clone the [repository](%%GITHUB_SPACY) to run the tests, you'll usually want to clone the [repository](%%GITHUB_SPACY)
@ -324,7 +319,7 @@ $ python -m pytest --pyargs %%SPACY_PKG_NAME # basic tests
$ python -m pytest --pyargs %%SPACY_PKG_NAME --slow # basic and slow tests $ python -m pytest --pyargs %%SPACY_PKG_NAME --slow # basic and slow tests
``` ```
## Troubleshooting guide {#troubleshooting} ## Troubleshooting guide {id="troubleshooting"}
This section collects some of the most common errors you may come across when This section collects some of the most common errors you may come across when
installing, loading and using spaCy, as well as their solutions. Also see the installing, loading and using spaCy, as well as their solutions. Also see the
@ -450,8 +445,6 @@ either of these, clone your repository again.
</Accordion> </Accordion>
## Changelog {#changelog} ## Changelog {id="changelog"}
import Changelog from 'widgets/changelog.js'
<Changelog /> <Changelog />

View File

@ -40,8 +40,7 @@ this config, you won't be able to change it anymore. The architecture is like a
recipe for the network, and you can't change the recipe once the dish has recipe for the network, and you can't change the recipe once the dish has
already been prepared. You have to make a new one. already been prepared. You have to make a new one.
```ini ```ini {title="config.cfg (excerpt)"}
### config.cfg (excerpt)
[components.tagger] [components.tagger]
factory = "tagger" factory = "tagger"
@ -51,7 +50,7 @@ width = 512
classes = 16 classes = 16
``` ```
## Type signatures {#type-sigs} ## Type signatures {id="type-sigs"}
> #### Example > #### Example
> >
@ -111,11 +110,14 @@ If you're using a modern editor like Visual Studio Code, you can
custom Thinc plugin and get live feedback about mismatched types as you write custom Thinc plugin and get live feedback about mismatched types as you write
code. code.
[![](../images/thinc_mypy.jpg)](https://thinc.ai/docs/usage-type-checking#linting) <Image
src="/images/thinc_mypy.jpg"
href="https://thinc.ai/docs/usage-type-checking#linting"
/>
</Accordion> </Accordion>
## Swapping model architectures {#swap-architectures} ## Swapping model architectures {id="swap-architectures"}
If no model is specified for the [`TextCategorizer`](/api/textcategorizer), the If no model is specified for the [`TextCategorizer`](/api/textcategorizer), the
[TextCatEnsemble](/api/architectures#TextCatEnsemble) architecture is used by [TextCatEnsemble](/api/architectures#TextCatEnsemble) architecture is used by
@ -123,8 +125,7 @@ default. This architecture combines a simple bag-of-words model with a neural
network, usually resulting in the most accurate results, but at the cost of network, usually resulting in the most accurate results, but at the cost of
speed. The config file for this model would look something like this: speed. The config file for this model would look something like this:
```ini ```ini {title="config.cfg (excerpt)"}
### config.cfg (excerpt)
[components.textcat] [components.textcat]
factory = "textcat" factory = "textcat"
labels = [] labels = []
@ -162,8 +163,7 @@ use those by swapping out the definition of the textcat's model. For instance,
to use the simple and fast bag-of-words model to use the simple and fast bag-of-words model
[TextCatBOW](/api/architectures#TextCatBOW), you can change the config to: [TextCatBOW](/api/architectures#TextCatBOW), you can change the config to:
```ini ```ini {title="config.cfg (excerpt)",highlight="6-10"}
### config.cfg (excerpt) {highlight="6-10"}
[components.textcat] [components.textcat]
factory = "textcat" factory = "textcat"
labels = [] labels = []
@ -180,7 +180,7 @@ For details on all pre-defined architectures shipped with spaCy and how to
configure them, check out the [model architectures](/api/architectures) configure them, check out the [model architectures](/api/architectures)
documentation. documentation.
### Defining sublayers {#sublayers} ### Defining sublayers {id="sublayers"}
Model architecture functions often accept **sublayers as arguments**, so that Model architecture functions often accept **sublayers as arguments**, so that
you can try **substituting a different layer** into the network. Depending on you can try **substituting a different layer** into the network. Depending on
@ -195,8 +195,7 @@ These steps together compute dense, context-sensitive representations of the
tokens, and their combination forms a typical tokens, and their combination forms a typical
[`Tok2Vec`](/api/architectures#Tok2Vec) layer: [`Tok2Vec`](/api/architectures#Tok2Vec) layer:
```ini ```ini {title="config.cfg (excerpt)"}
### config.cfg (excerpt)
[components.tok2vec] [components.tok2vec]
factory = "tok2vec" factory = "tok2vec"
@ -217,8 +216,7 @@ a sublayer for another one, for instance changing the first sublayer to a
character embedding with the [CharacterEmbed](/api/architectures#CharacterEmbed) character embedding with the [CharacterEmbed](/api/architectures#CharacterEmbed)
architecture: architecture:
```ini ```ini {title="config.cfg (excerpt)"}
### config.cfg (excerpt)
[components.tok2vec.model.embed] [components.tok2vec.model.embed]
@architectures = "spacy.CharacterEmbed.v2" @architectures = "spacy.CharacterEmbed.v2"
# ... # ...
@ -237,7 +235,7 @@ a transformer. And if you want to define your own solution, all you need to do
is register a ~~Model[List[Doc], List[Floats2d]]~~ architecture function, and is register a ~~Model[List[Doc], List[Floats2d]]~~ architecture function, and
you'll be able to try it out in any of the spaCy components. you'll be able to try it out in any of the spaCy components.
## Wrapping PyTorch, TensorFlow and other frameworks {#frameworks} ## Wrapping PyTorch, TensorFlow and other frameworks {id="frameworks"}
Thinc allows you to [wrap models](https://thinc.ai/docs/usage-frameworks) Thinc allows you to [wrap models](https://thinc.ai/docs/usage-frameworks)
written in other machine learning frameworks like PyTorch, TensorFlow and MXNet written in other machine learning frameworks like PyTorch, TensorFlow and MXNet
@ -257,8 +255,7 @@ Let's use PyTorch to define a very simple neural network consisting of two
hidden `Linear` layers with `ReLU` activation and dropout, and a hidden `Linear` layers with `ReLU` activation and dropout, and a
softmax-activated output layer: softmax-activated output layer:
```python ```python {title="PyTorch model"}
### PyTorch model
from torch import nn from torch import nn
torch_model = nn.Sequential( torch_model = nn.Sequential(
@ -300,7 +297,7 @@ layers, and "native" Thinc layers to do fiddly input and output transformations
and add on task-specific "heads", as efficiency is less of a consideration for and add on task-specific "heads", as efficiency is less of a consideration for
those parts of the network. those parts of the network.
### Using wrapped models {#frameworks-usage} ### Using wrapped models {id="frameworks-usage"}
To use our custom model including the PyTorch subnetwork, all we need to do is To use our custom model including the PyTorch subnetwork, all we need to do is
register the architecture using the register the architecture using the
@ -309,8 +306,7 @@ architecture a name so spaCy knows how to find it, and allows passing in
arguments like hyperparameters via the [config](/usage/training#config). The arguments like hyperparameters via the [config](/usage/training#config). The
full example then becomes: full example then becomes:
```python ```python {title="Registering the architecture",highlight="9"}
### Registering the architecture {highlight="9"}
from typing import List from typing import List
from thinc.types import Floats2d from thinc.types import Floats2d
from thinc.api import Model, PyTorchWrapper, chain, with_array from thinc.api import Model, PyTorchWrapper, chain, with_array
@ -349,8 +345,7 @@ by specifying it in the config file. In this configuration, all required
parameters for the various subcomponents of the custom architecture are passed parameters for the various subcomponents of the custom architecture are passed
in as settings via the config. in as settings via the config.
```ini ```ini {title="config.cfg (excerpt)",highlight="5-5"}
### config.cfg (excerpt) {highlight="5-5"}
[components.tagger] [components.tagger]
factory = "tagger" factory = "tagger"
@ -378,13 +373,12 @@ GPU memory allocator accordingly. When `gpu_allocator` is set to "pytorch" or
respective libraries, preventing OOM errors when there's available memory respective libraries, preventing OOM errors when there's available memory
sitting in the other library's pool. sitting in the other library's pool.
```ini ```ini {title="config.cfg (excerpt)"}
### config.cfg (excerpt)
[training] [training]
gpu_allocator = "pytorch" gpu_allocator = "pytorch"
``` ```
## Custom models with Thinc {#thinc} ## Custom models with Thinc {id="thinc"}
Of course it's also possible to define the `Model` from the previous section Of course it's also possible to define the `Model` from the previous section
entirely in Thinc. The Thinc documentation provides details on the entirely in Thinc. The Thinc documentation provides details on the
@ -418,7 +412,7 @@ the PyTorch layers are defined, where `in_features` precedes `out_features`.
</Infobox> </Infobox>
### Shape inference in Thinc {#thinc-shape-inference} ### Shape inference in Thinc {id="thinc-shape-inference"}
It is **not** strictly necessary to define all the input and output dimensions It is **not** strictly necessary to define all the input and output dimensions
for each layer, as Thinc can perform for each layer, as Thinc can perform
@ -458,8 +452,7 @@ you have to call
[`Model.initialize`](https://thinc.ai/docs/api-model#initialize) with an **input [`Model.initialize`](https://thinc.ai/docs/api-model#initialize) with an **input
sample** `X` and an **output sample** `Y` with the correct dimensions: sample** `X` and an **output sample** `Y` with the correct dimensions:
```python ```python {title="Shape inference with initialization",highlight="3,7,10"}
### Shape inference with initialization {highlight="3,7,10"}
with Model.define_operators({">>": chain}): with Model.define_operators({">>": chain}):
layers = ( layers = (
Relu(hidden_width) Relu(hidden_width)
@ -479,7 +472,7 @@ data. In this case, `X` is typically a ~~List[Doc]~~, while `Y` is typically a
functionality is triggered when [`nlp.initialize`](/api/language#initialize) is functionality is triggered when [`nlp.initialize`](/api/language#initialize) is
called. called.
### Dropout and normalization in Thinc {#thinc-dropout-norm} ### Dropout and normalization in Thinc {id="thinc-dropout-norm"}
Many of the available Thinc [layers](https://thinc.ai/docs/api-layers) allow you Many of the available Thinc [layers](https://thinc.ai/docs/api-layers) allow you
to define a `dropout` argument that will result in "chaining" an additional to define a `dropout` argument that will result in "chaining" an additional
@ -500,7 +493,7 @@ with Model.define_operators({">>": chain}):
model.initialize(X=input_sample, Y=output_sample) model.initialize(X=input_sample, Y=output_sample)
``` ```
## Create new trainable components {#components} ## Create new trainable components {id="components"}
In addition to [swapping out](#swap-architectures) layers in existing In addition to [swapping out](#swap-architectures) layers in existing
components, you can also implement an entirely new, components, you can also implement an entirely new,
@ -518,7 +511,7 @@ overview of the `TrainablePipe` methods used by
</Infobox> </Infobox>
### Example: Entity relation extraction component {#component-rel} ### Example: Entity relation extraction component {id="component-rel"}
This section outlines an example use-case of implementing a **novel relation This section outlines an example use-case of implementing a **novel relation
extraction component** from scratch. We'll implement a binary relation extraction component** from scratch. We'll implement a binary relation
@ -538,17 +531,17 @@ two major steps required:
<Project id="tutorials/rel_component"> <Project id="tutorials/rel_component">
Run this example use-case by using our project template. It includes all the Run this example use-case by using our project template. It includes all the
code to create the ML model and the pipeline component from scratch. code to create the ML model and the pipeline component from scratch. It also
It also contains two config files to train the model: contains two config files to train the model: one to run on CPU with a Tok2Vec
one to run on CPU with a Tok2Vec layer, and one for the GPU using a transformer. layer, and one for the GPU using a transformer. The project applies the
The project applies the relation extraction component to identify biomolecular relation extraction component to identify biomolecular interactions in a
interactions in a sample dataset, but you can easily swap in your own dataset sample dataset, but you can easily swap in your own dataset for your
for your experiments in any other domain. experiments in any other domain.
</Project> </Project>
<YouTube id="8HL-Ap5_Axo"></YouTube> <YouTube id="8HL-Ap5_Axo"></YouTube>
#### Step 1: Implementing the Model {#component-rel-model} #### Step 1: Implementing the Model {id="component-rel-model"}
We need to implement a [`Model`](https://thinc.ai/docs/api-model) that takes a We need to implement a [`Model`](https://thinc.ai/docs/api-model) that takes a
**list of documents** (~~List[Doc]~~) as input, and outputs a **two-dimensional **list of documents** (~~List[Doc]~~) as input, and outputs a **two-dimensional
@ -561,8 +554,7 @@ matrix** (~~Floats2d~~) of predictions:
> type checks and validation. See the section on [type signatures](#type-sigs) > type checks and validation. See the section on [type signatures](#type-sigs)
> for details. > for details.
```python ```python {title="The model architecture"}
### The model architecture
@spacy.registry.architectures("rel_model.v1") @spacy.registry.architectures("rel_model.v1")
def create_relation_model(...) -> Model[List[Doc], Floats2d]: def create_relation_model(...) -> Model[List[Doc], Floats2d]:
model = ... # 👈 model will go here model = ... # 👈 model will go here
@ -587,8 +579,7 @@ transforms the instance tensor into a final tensor holding the predictions:
> # ... > # ...
> ``` > ```
```python ```python {title="The model architecture",highlight="6"}
### The model architecture {highlight="6"}
@spacy.registry.architectures("rel_model.v1") @spacy.registry.architectures("rel_model.v1")
def create_relation_model( def create_relation_model(
create_instance_tensor: Model[List[Doc], Floats2d], create_instance_tensor: Model[List[Doc], Floats2d],
@ -611,8 +602,7 @@ The `classification_layer` could be something like a
> nO = null > nO = null
> ``` > ```
```python ```python {title="The classification layer"}
### The classification layer
@spacy.registry.architectures("rel_classification_layer.v1") @spacy.registry.architectures("rel_classification_layer.v1")
def create_classification_layer( def create_classification_layer(
nO: int = None, nI: int = None nO: int = None, nI: int = None
@ -648,8 +638,7 @@ that has the full implementation.
> # ... > # ...
> ``` > ```
```python ```python {title="The layer that creates the instance tensor"}
### The layer that creates the instance tensor
@spacy.registry.architectures("rel_instance_tensor.v1") @spacy.registry.architectures("rel_instance_tensor.v1")
def create_tensors( def create_tensors(
tok2vec: Model[List[Doc], List[Floats2d]], tok2vec: Model[List[Doc], List[Floats2d]],
@ -729,8 +718,7 @@ are within a **maximum distance** (in number of tokens) of each other:
> max_length = 100 > max_length = 100
> ``` > ```
```python ```python {title="Candidate generation"}
### Candidate generation
@spacy.registry.misc("rel_instance_generator.v1") @spacy.registry.misc("rel_instance_generator.v1")
def create_instances(max_length: int) -> Callable[[Doc], List[Tuple[Span, Span]]]: def create_instances(max_length: int) -> Callable[[Doc], List[Tuple[Span, Span]]]:
def get_candidates(doc: "Doc") -> List[Tuple[Span, Span]]: def get_candidates(doc: "Doc") -> List[Tuple[Span, Span]]:
@ -748,7 +736,7 @@ This function is added to the [`@misc` registry](/api/top-level#registry) so we
can refer to it from the config, and easily swap it out for any other candidate can refer to it from the config, and easily swap it out for any other candidate
generation function. generation function.
#### Intermezzo: define how to store the relations data {#component-rel-attribute} #### Intermezzo: define how to store the relations data {id="component-rel-attribute"}
> #### Example output > #### Example output
> >
@ -773,22 +761,20 @@ above 0.5 to be a `True` relation. The ~~Example~~ instances that we'll use as
training data, will include their gold-standard relation annotations in training data, will include their gold-standard relation annotations in
`example.reference._.rel`. `example.reference._.rel`.
```python ```python {title="Registering the extension attribute"}
### Registering the extension attribute
from spacy.tokens import Doc from spacy.tokens import Doc
Doc.set_extension("rel", default={}) Doc.set_extension("rel", default={})
``` ```
#### Step 2: Implementing the pipeline component {#component-rel-pipe} #### Step 2: Implementing the pipeline component {id="component-rel-pipe"}
To use our new relation extraction model as part of a custom To use our new relation extraction model as part of a custom
[trainable component](/usage/processing-pipelines#trainable-components), we [trainable component](/usage/processing-pipelines#trainable-components), we
create a subclass of [`TrainablePipe`](/api/pipe) that holds the model. create a subclass of [`TrainablePipe`](/api/pipe) that holds the model.
![Illustration of Pipe methods](../images/trainable_component.svg) ![Illustration of Pipe methods](/images/trainable_component.svg)
```python ```python {title="Pipeline component skeleton"}
### Pipeline component skeleton
from spacy.pipeline import TrainablePipe from spacy.pipeline import TrainablePipe
class RelationExtractor(TrainablePipe): class RelationExtractor(TrainablePipe):
@ -825,8 +811,7 @@ and the name of this component. Additionally, this component, just like the
will predict scores for each label. We add convenience methods to easily will predict scores for each label. We add convenience methods to easily
retrieve and add to them. retrieve and add to them.
```python ```python {title="The constructor (continued)"}
### The constructor (continued)
def __init__(self, vocab, model, name="rel"): def __init__(self, vocab, model, name="rel"):
"""Create a component instance.""" """Create a component instance."""
# ... # ...
@ -855,8 +840,7 @@ will be used to do
layers of the neural network. This is triggered by calling layers of the neural network. This is triggered by calling
[`Model.initialize`](https://thinc.ai/api/model#initialize). [`Model.initialize`](https://thinc.ai/api/model#initialize).
```python ```python {title="The initialize method",highlight="12,15,18,22"}
### The initialize method {highlight="12,15,18,22"}
from itertools import islice from itertools import islice
def initialize( def initialize(
@ -896,8 +880,7 @@ update the weights of the model layers. Thinc provides several
[loss functions](https://thinc.ai/docs/api-loss) that can be used for the [loss functions](https://thinc.ai/docs/api-loss) that can be used for the
implementation of the `get_loss` function. implementation of the `get_loss` function.
```python ```python {title="The update method",highlight="12-14"}
### The update method {highlight="12-14"}
def update( def update(
self, self,
examples: Iterable[Example], examples: Iterable[Example],
@ -923,8 +906,7 @@ delegate to the internal model's
[predict](https://thinc.ai/docs/api-model#predict) function that takes a batch [predict](https://thinc.ai/docs/api-model#predict) function that takes a batch
of `Doc` objects and returns a ~~Floats2d~~ array: of `Doc` objects and returns a ~~Floats2d~~ array:
```python ```python {title="The predict method"}
### The predict method
def predict(self, docs: Iterable[Doc]) -> Floats2d: def predict(self, docs: Iterable[Doc]) -> Floats2d:
predictions = self.model.predict(docs) predictions = self.model.predict(docs)
return self.model.ops.asarray(predictions) return self.model.ops.asarray(predictions)
@ -941,8 +923,7 @@ need to refer to the model's `get_instances` function that defined which pairs
of entities were relevant candidates, so that the predictions can be linked to of entities were relevant candidates, so that the predictions can be linked to
those exact entities: those exact entities:
```python ```python {title="The set_annotations method",highlight="5-6,10"}
### The set_annotations method {highlight="5-6,10"}
def set_annotations(self, docs: Iterable[Doc], predictions: Floats2d): def set_annotations(self, docs: Iterable[Doc], predictions: Floats2d):
c = 0 c = 0
get_instances = self.model.attrs["get_instances"] get_instances = self.model.attrs["get_instances"]
@ -959,8 +940,7 @@ def set_annotations(self, docs: Iterable[Doc], predictions: Floats2d):
Under the hood, when the pipe is applied to a document, it delegates to the Under the hood, when the pipe is applied to a document, it delegates to the
`predict` and `set_annotations` methods: `predict` and `set_annotations` methods:
```python ```python {title="The __call__ method"}
### The __call__ method
def __call__(self, doc: Doc): def __call__(self, doc: Doc):
predictions = self.predict([doc]) predictions = self.predict([doc])
self.set_annotations([doc], predictions) self.set_annotations([doc], predictions)
@ -971,8 +951,7 @@ There is one more optional method to implement: [`score`](/api/pipe#score)
calculates the performance of your component on a set of examples, and returns calculates the performance of your component on a set of examples, and returns
the results as a dictionary: the results as a dictionary:
```python ```python {title="The score method"}
### The score method
def score(self, examples: Iterable[Example]) -> Dict[str, Any]: def score(self, examples: Iterable[Example]) -> Dict[str, Any]:
prf = PRFScore() prf = PRFScore()
for example in examples: for example in examples:
@ -1011,8 +990,7 @@ assigns it a name and lets you create the component with
> rel_micro_f = 1.0 > rel_micro_f = 1.0
> ``` > ```
```python ```python {title="Registering the pipeline component"}
### Registering the pipeline component
from spacy.language import Language from spacy.language import Language
@Language.factory("relation_extractor") @Language.factory("relation_extractor")
@ -1024,8 +1002,7 @@ You can extend the decorator to include information such as the type of
annotations that are required for this component to run, the type of annotations annotations that are required for this component to run, the type of annotations
it produces, and the scores that can be calculated: it produces, and the scores that can be calculated:
```python ```python {title="Factory annotations",highlight="5-11"}
### Factory annotations {highlight="5-11"}
from spacy.language import Language from spacy.language import Language
@Language.factory( @Language.factory(
@ -1044,10 +1021,9 @@ def make_relation_extractor(nlp, name, model):
<Project id="tutorials/rel_component"> <Project id="tutorials/rel_component">
Run this example use-case by using our project template. It includes all the Run this example use-case by using our project template. It includes all the
code to create the ML model and the pipeline component from scratch. code to create the ML model and the pipeline component from scratch. It
It contains two config files to train the model: contains two config files to train the model: one to run on CPU with a Tok2Vec
one to run on CPU with a Tok2Vec layer, and one for the GPU using a transformer. layer, and one for the GPU using a transformer. The project applies the
The project applies the relation extraction component to identify biomolecular relation extraction component to identify biomolecular interactions, but you
interactions, but you can easily swap in your own dataset for your experiments can easily swap in your own dataset for your experiments in any other domain.
in any other domain.
</Project> </Project>

View File

@ -26,9 +26,7 @@ information. That's exactly what spaCy is designed to do: you put in raw text,
and get back a [`Doc`](/api/doc) object, that comes with a variety of and get back a [`Doc`](/api/doc) object, that comes with a variety of
annotations. annotations.
## Part-of-speech tagging {#pos-tagging model="tagger, parser"} ## Part-of-speech tagging {id="pos-tagging",model="tagger, parser"}
import PosDeps101 from 'usage/101/\_pos-deps.md'
<PosDeps101 /> <PosDeps101 />
@ -40,7 +38,7 @@ in the [models directory](/models).
</Infobox> </Infobox>
## Morphology {#morphology} ## Morphology {id="morphology"}
Inflectional morphology is the process by which a root form of a word is Inflectional morphology is the process by which a root form of a word is
modified by adding prefixes or suffixes that specify its grammatical function modified by adding prefixes or suffixes that specify its grammatical function
@ -64,8 +62,7 @@ allows you to access individual morphological features.
> and express that it's a pronoun in the third person. > and express that it's a pronoun in the third person.
> 2. Inspect `token.morph` for the other tokens. > 2. Inspect `token.morph` for the other tokens.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -76,14 +73,13 @@ print(token.morph) # 'Case=Nom|Number=Sing|Person=1|PronType=Prs'
print(token.morph.get("PronType")) # ['Prs'] print(token.morph.get("PronType")) # ['Prs']
``` ```
### Statistical morphology {#morphologizer new="3" model="morphologizer"} ### Statistical morphology {id="morphologizer",version="3",model="morphologizer"}
spaCy's statistical [`Morphologizer`](/api/morphologizer) component assigns the spaCy's statistical [`Morphologizer`](/api/morphologizer) component assigns the
morphological features and coarse-grained part-of-speech tags as `Token.morph` morphological features and coarse-grained part-of-speech tags as `Token.morph`
and `Token.pos`. and `Token.pos`.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("de_core_news_sm") nlp = spacy.load("de_core_news_sm")
@ -92,7 +88,7 @@ print(doc[2].morph) # 'Case=Nom|Number=Sing|Person=2|PronType=Prs'
print(doc[2].pos_) # 'PRON' print(doc[2].pos_) # 'PRON'
``` ```
### Rule-based morphology {#rule-based-morphology} ### Rule-based morphology {id="rule-based-morphology"}
For languages with relatively simple morphological systems like English, spaCy For languages with relatively simple morphological systems like English, spaCy
can assign morphological features through a rule-based approach, which uses the can assign morphological features through a rule-based approach, which uses the
@ -108,8 +104,7 @@ coarse-grained part-of-speech tags and morphological features.
[mapping table](#mappings-exceptions) maps the fine-grained tags to a [mapping table](#mappings-exceptions) maps the fine-grained tags to a
coarse-grained POS tags and morphological features. coarse-grained POS tags and morphological features.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -118,7 +113,7 @@ print(doc[2].morph) # 'Case=Nom|Person=2|PronType=Prs'
print(doc[2].pos_) # 'PRON' print(doc[2].pos_) # 'PRON'
``` ```
## Lemmatization {#lemmatization model="lemmatizer" new="3"} ## Lemmatization {id="lemmatization",model="lemmatizer",version="3"}
spaCy provides two pipeline components for lemmatization: spaCy provides two pipeline components for lemmatization:
@ -128,8 +123,7 @@ spaCy provides two pipeline components for lemmatization:
2. The [`EditTreeLemmatizer`](/api/edittreelemmatizer) 2. The [`EditTreeLemmatizer`](/api/edittreelemmatizer)
<Tag variant="new">3.3</Tag> component provides a trainable lemmatizer. <Tag variant="new">3.3</Tag> component provides a trainable lemmatizer.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
# English pipelines include a rule-based lemmatizer # English pipelines include a rule-based lemmatizer
@ -160,7 +154,7 @@ provided trained pipelines already include all the required tables, but if you
are creating new pipelines, you'll probably want to install `spacy-lookups-data` are creating new pipelines, you'll probably want to install `spacy-lookups-data`
to provide the data when the lemmatizer is initialized. to provide the data when the lemmatizer is initialized.
### Lookup lemmatizer {#lemmatizer-lookup} ### Lookup lemmatizer {id="lemmatizer-lookup"}
For pipelines without a tagger or morphologizer, a lookup lemmatizer can be For pipelines without a tagger or morphologizer, a lookup lemmatizer can be
added to the pipeline as long as a lookup table is provided, typically through added to the pipeline as long as a lookup table is provided, typically through
@ -176,7 +170,7 @@ nlp = spacy.blank("sv")
nlp.add_pipe("lemmatizer", config={"mode": "lookup"}) nlp.add_pipe("lemmatizer", config={"mode": "lookup"})
``` ```
### Rule-based lemmatizer {#lemmatizer-rule} ### Rule-based lemmatizer {id="lemmatizer-rule"}
When training pipelines that include a component that assigns part-of-speech When training pipelines that include a component that assigns part-of-speech
tags (a morphologizer or a tagger with a [POS mapping](#mappings-exceptions)), a tags (a morphologizer or a tagger with a [POS mapping](#mappings-exceptions)), a
@ -214,7 +208,7 @@ nlp = spacy.blank("de")
nlp.add_pipe("trainable_lemmatizer", name="lemmatizer") nlp.add_pipe("trainable_lemmatizer", name="lemmatizer")
``` ```
## Dependency Parsing {#dependency-parse model="parser"} ## Dependency Parsing {id="dependency-parse",model="parser"}
spaCy features a fast and accurate syntactic dependency parser, and has a rich spaCy features a fast and accurate syntactic dependency parser, and has a rich
API for navigating the tree. The parser also powers the sentence boundary API for navigating the tree. The parser also powers the sentence boundary
@ -232,7 +226,7 @@ different languages, see the label schemes documented in the
</Infobox> </Infobox>
### Noun chunks {#noun-chunks} ### Noun chunks {id="noun-chunks"}
Noun chunks are "base noun phrases" flat phrases that have a noun as their Noun chunks are "base noun phrases" flat phrases that have a noun as their
head. You can think of noun chunks as a noun plus the words describing the noun head. You can think of noun chunks as a noun plus the words describing the noun
@ -240,8 +234,7 @@ head. You can think of noun chunks as a noun plus the words describing the noun
get the noun chunks in a document, simply iterate over get the noun chunks in a document, simply iterate over
[`Doc.noun_chunks`](/api/doc#noun_chunks). [`Doc.noun_chunks`](/api/doc#noun_chunks).
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -263,7 +256,7 @@ for chunk in doc.noun_chunks:
| insurance liability | liability | `dobj` | shift | | insurance liability | liability | `dobj` | shift |
| manufacturers | manufacturers | `pobj` | toward | | manufacturers | manufacturers | `pobj` | toward |
### Navigating the parse tree {#navigating} ### Navigating the parse tree {id="navigating"}
spaCy uses the terms **head** and **child** to describe the words **connected by spaCy uses the terms **head** and **child** to describe the words **connected by
a single arc** in the dependency tree. The term **dep** is used for the arc a single arc** in the dependency tree. The term **dep** is used for the arc
@ -271,8 +264,7 @@ label, which describes the type of syntactic relation that connects the child to
the head. As with other attributes, the value of `.dep` is a hash value. You can the head. As with other attributes, the value of `.dep` is a hash value. You can
get the string value with `.dep_`. get the string value with `.dep_`.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -298,17 +290,18 @@ for token in doc:
| toward | `prep` | shift | `NOUN` | manufacturers | | toward | `prep` | shift | `NOUN` | manufacturers |
| manufacturers | `pobj` | toward | `ADP` | | | manufacturers | `pobj` | toward | `ADP` | |
import DisplaCyLong2Html from 'images/displacy-long2.html' <Iframe
title="displaCy visualization of dependencies and entities 2"
<Iframe title="displaCy visualization of dependencies and entities 2" html={DisplaCyLong2Html} height={450} /> src="/images/displacy-long2.html"
height={450}
/>
Because the syntactic relations form a tree, every word has **exactly one Because the syntactic relations form a tree, every word has **exactly one
head**. You can therefore iterate over the arcs in the tree by iterating over head**. You can therefore iterate over the arcs in the tree by iterating over
the words in the sentence. This is usually the best way to match an arc of the words in the sentence. This is usually the best way to match an arc of
interest from below: interest from below:
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.symbols import nsubj, VERB from spacy.symbols import nsubj, VERB
@ -340,7 +333,7 @@ for possible_verb in doc:
To iterate through the children, use the `token.children` attribute, which To iterate through the children, use the `token.children` attribute, which
provides a sequence of [`Token`](/api/token) objects. provides a sequence of [`Token`](/api/token) objects.
#### Iterating around the local tree {#navigating-around} #### Iterating around the local tree {id="navigating-around"}
A few more convenience attributes are provided for iterating around the local A few more convenience attributes are provided for iterating around the local
tree from the token. [`Token.lefts`](/api/token#lefts) and tree from the token. [`Token.lefts`](/api/token#lefts) and
@ -351,8 +344,7 @@ order. There are also two integer-typed attributes,
[`Token.n_rights`](/api/token#n_rights) that give the number of left and right [`Token.n_rights`](/api/token#n_rights) that give the number of left and right
children. children.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -363,8 +355,7 @@ print(doc[2].n_lefts) # 2
print(doc[2].n_rights) # 1 print(doc[2].n_rights) # 1
``` ```
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("de_core_news_sm") nlp = spacy.load("de_core_news_sm")
@ -387,8 +378,7 @@ sequence of tokens. You can walk up the tree with the
> true for the German pipelines, which have many > true for the German pipelines, which have many
> [non-projective dependencies](https://explosion.ai/blog/german-model#word-order). > [non-projective dependencies](https://explosion.ai/blog/german-model#word-order).
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -417,8 +407,7 @@ easiest way to create a `Span` object for a syntactic phrase. Note that
`.right_edge` gives a token **within** the subtree so if you use it as the `.right_edge` gives a token **within** the subtree so if you use it as the
end-point of a range, don't forget to `+1`! end-point of a range, don't forget to `+1`!
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -445,8 +434,7 @@ currency values, i.e. entities labeled as `MONEY`, and then uses the dependency
parse to find the noun phrase they are referring to for example `"Net income"` parse to find the noun phrase they are referring to for example `"Net income"`
&rarr; `"$9.4 million"`. &rarr; `"$9.4 million"`.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -480,7 +468,7 @@ see the usage guide on
</Infobox> </Infobox>
### Visualizing dependencies {#displacy} ### Visualizing dependencies {id="displacy"}
The best way to understand spaCy's dependency parser is interactively. To make The best way to understand spaCy's dependency parser is interactively. To make
this easier, spaCy comes with a visualization module. You can pass a `Doc` or a this easier, spaCy comes with a visualization module. You can pass a `Doc` or a
@ -491,8 +479,7 @@ If you want to know how to write rules that hook into some type of syntactic
construction, just plug the sentence into the visualizer and see how spaCy construction, just plug the sentence into the visualizer and see how spaCy
annotates it. annotates it.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy import displacy from spacy import displacy
@ -510,7 +497,7 @@ displaCy in our [online demo](https://explosion.ai/demos/displacy)..
</Infobox> </Infobox>
### Disabling the parser {#disabling} ### Disabling the parser {id="disabling"}
In the [trained pipelines](/models) provided by spaCy, the parser is loaded and In the [trained pipelines](/models) provided by spaCy, the parser is loaded and
enabled by default as part of the enabled by default as part of the
@ -525,7 +512,7 @@ the `nlp` object. For more details, see the usage guide on
nlp = spacy.load("en_core_web_sm", disable=["parser"]) nlp = spacy.load("en_core_web_sm", disable=["parser"])
``` ```
## Named Entity Recognition {#named-entities} ## Named Entity Recognition {id="named-entities"}
spaCy features an extremely fast statistical entity recognition system, that spaCy features an extremely fast statistical entity recognition system, that
assigns labels to contiguous spans of tokens. The default assigns labels to contiguous spans of tokens. The default
@ -534,13 +521,11 @@ entities, including companies, locations, organizations and products. You can
add arbitrary classes to the entity recognition system, and update the model add arbitrary classes to the entity recognition system, and update the model
with new examples. with new examples.
### Named Entity Recognition 101 {#named-entities-101} ### Named Entity Recognition 101 {id="named-entities-101"}
import NER101 from 'usage/101/\_named-entities.md'
<NER101 /> <NER101 />
### Accessing entity annotations and labels {#accessing-ner} ### Accessing entity annotations and labels {id="accessing-ner"}
The standard way to access entity annotations is the [`doc.ents`](/api/doc#ents) The standard way to access entity annotations is the [`doc.ents`](/api/doc#ents)
property, which produces a sequence of [`Span`](/api/span) objects. The entity property, which produces a sequence of [`Span`](/api/span) objects. The entity
@ -569,8 +554,7 @@ on a token, it will return an empty string.
> - `U` Token is a single-token **unit** entity. > - `U` Token is a single-token **unit** entity.
> - `O` Token is **outside** an entity. > - `O` Token is **outside** an entity.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -597,7 +581,7 @@ print(ent_francisco) # ['Francisco', 'I', 'GPE']
| delivery | `2` | `O` | `""` | outside an entity | | delivery | `2` | `O` | `""` | outside an entity |
| robots | `2` | `O` | `""` | outside an entity | | robots | `2` | `O` | `""` | outside an entity |
### Setting entity annotations {#setting-entities} ### Setting entity annotations {id="setting-entities"}
To ensure that the sequence of token annotations remains consistent, you have to To ensure that the sequence of token annotations remains consistent, you have to
set entity annotations **at the document level**. However, you can't write set entity annotations **at the document level**. However, you can't write
@ -605,8 +589,7 @@ directly to the `token.ent_iob` or `token.ent_type` attributes, so the easiest
way to set entities is to use the [`doc.set_ents`](/api/doc#set_ents) function way to set entities is to use the [`doc.set_ents`](/api/doc#set_ents) function
and create the new entity as a [`Span`](/api/span). and create the new entity as a [`Span`](/api/span).
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.tokens import Span from spacy.tokens import Span
@ -639,15 +622,14 @@ indices, not the character offsets. To create a span from character offsets, use
fb_ent = doc.char_span(0, 2, label="ORG") fb_ent = doc.char_span(0, 2, label="ORG")
``` ```
#### Setting entity annotations from array {#setting-from-array} #### Setting entity annotations from array {id="setting-from-array"}
You can also assign entity annotations using the You can also assign entity annotations using the
[`doc.from_array`](/api/doc#from_array) method. To do this, you should include [`doc.from_array`](/api/doc#from_array) method. To do this, you should include
both the `ENT_TYPE` and the `ENT_IOB` attributes in the array you're importing both the `ENT_TYPE` and the `ENT_IOB` attributes in the array you're importing
from. from.
```python ```python {executable="true"}
### {executable="true"}
import numpy import numpy
import spacy import spacy
from spacy.attrs import ENT_IOB, ENT_TYPE from spacy.attrs import ENT_IOB, ENT_TYPE
@ -664,7 +646,7 @@ doc.from_array(header, attr_array)
print("After", doc.ents) # [London] print("After", doc.ents) # [London]
``` ```
#### Setting entity annotations in Cython {#setting-cython} #### Setting entity annotations in Cython {id="setting-cython"}
Finally, you can always write to the underlying struct if you compile a Finally, you can always write to the underlying struct if you compile a
[Cython](http://cython.org/) function. This is easy to do, and allows you to [Cython](http://cython.org/) function. This is easy to do, and allows you to
@ -686,7 +668,7 @@ cpdef set_entity(Doc doc, int start, int end, attr_t ent_type):
Obviously, if you write directly to the array of `TokenC*` structs, you'll have Obviously, if you write directly to the array of `TokenC*` structs, you'll have
responsibility for ensuring that the data is left in a consistent state. responsibility for ensuring that the data is left in a consistent state.
### Built-in entity types {#entity-types} ### Built-in entity types {id="entity-types"}
> #### Tip: Understanding entity types > #### Tip: Understanding entity types
> >
@ -702,7 +684,7 @@ For details on the entity types available in spaCy's trained pipelines, see the
</Infobox> </Infobox>
### Visualizing named entities {#displacy} ### Visualizing named entities {id="displacy"}
The The
[displaCy <sup>ENT</sup> visualizer](https://explosion.ai/demos/displacy-ent) [displaCy <sup>ENT</sup> visualizer](https://explosion.ai/demos/displacy-ent)
@ -716,8 +698,7 @@ list of `Doc` objects to displaCy and run
For more details and examples, see the For more details and examples, see the
[usage guide on visualizing spaCy](/usage/visualizers). [usage guide on visualizing spaCy](/usage/visualizers).
```python ```python {title="Named Entity example"}
### Named Entity example
import spacy import spacy
from spacy import displacy from spacy import displacy
@ -728,11 +709,13 @@ doc = nlp(text)
displacy.serve(doc, style="ent") displacy.serve(doc, style="ent")
``` ```
import DisplacyEntHtml from 'images/displacy-ent2.html' <Iframe
title="displaCy visualizer for entities"
src="/images/displacy-ent2.html"
height={180}
/>
<Iframe title="displaCy visualizer for entities" html={DisplacyEntHtml} height={180} /> ## Entity Linking {id="entity-linking"}
## Entity Linking {#entity-linking}
To ground the named entities into the "real world", spaCy provides functionality To ground the named entities into the "real world", spaCy provides functionality
to perform entity linking, which resolves a textual entity to a unique to perform entity linking, which resolves a textual entity to a unique
@ -740,7 +723,7 @@ identifier from a knowledge base (KB). You can create your own
[`KnowledgeBase`](/api/kb) and [train](/usage/training) a new [`KnowledgeBase`](/api/kb) and [train](/usage/training) a new
[`EntityLinker`](/api/entitylinker) using that custom knowledge base. [`EntityLinker`](/api/entitylinker) using that custom knowledge base.
### Accessing entity identifiers {#entity-linking-accessing model="entity linking"} ### Accessing entity identifiers {id="entity-linking-accessing",model="entity linking"}
The annotated KB identifier is accessible as either a hash value or as a string, The annotated KB identifier is accessible as either a hash value or as a string,
using the attributes `ent.kb_id` and `ent.kb_id_` of a [`Span`](/api/span) using the attributes `ent.kb_id` and `ent.kb_id_` of a [`Span`](/api/span)
@ -766,7 +749,7 @@ print(ent_ada_1) # ['Lovelace', 'PERSON', 'Q7259']
print(ent_london_5) # ['London', 'GPE', 'Q84'] print(ent_london_5) # ['London', 'GPE', 'Q84']
``` ```
## Tokenization {#tokenization} ## Tokenization {id="tokenization"}
Tokenization is the task of splitting a text into meaningful segments, called Tokenization is the task of splitting a text into meaningful segments, called
_tokens_. The input to the tokenizer is a unicode text, and the output is a _tokens_. The input to the tokenizer is a unicode text, and the output is a
@ -785,8 +768,6 @@ during tokenization. This is kind of a core principle of spaCy's `Doc` object:
</Infobox> </Infobox>
import Tokenization101 from 'usage/101/\_tokenization.md'
<Tokenization101 /> <Tokenization101 />
<Accordion title="Algorithm details: How spaCy's tokenizer works" id="how-tokenizer-works" spaced> <Accordion title="Algorithm details: How spaCy's tokenizer works" id="how-tokenizer-works" spaced>
@ -914,15 +895,14 @@ might make sense to create an entirely custom subclass.
--- ---
### Adding special case tokenization rules {#special-cases} ### Adding special case tokenization rules {id="special-cases"}
Most domains have at least some idiosyncrasies that require custom tokenization Most domains have at least some idiosyncrasies that require custom tokenization
rules. This could be very certain expressions, or abbreviations only used in rules. This could be very certain expressions, or abbreviations only used in
this specific field. Here's how to add a special case rule to an existing this specific field. Here's how to add a special case rule to an existing
[`Tokenizer`](/api/tokenizer) instance: [`Tokenizer`](/api/tokenizer) instance:
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.symbols import ORTH from spacy.symbols import ORTH
@ -951,7 +931,7 @@ nlp.tokenizer.add_special_case("...gimme...?", [{"ORTH": "...gimme...?"}])
assert len(nlp("...gimme...?")) == 1 assert len(nlp("...gimme...?")) == 1
``` ```
#### Debugging the tokenizer {#tokenizer-debug new="2.2.3"} #### Debugging the tokenizer {id="tokenizer-debug",version="2.2.3"}
A working implementation of the pseudo-code above is available for debugging as A working implementation of the pseudo-code above is available for debugging as
[`nlp.tokenizer.explain(text)`](/api/tokenizer#explain). It returns a list of [`nlp.tokenizer.explain(text)`](/api/tokenizer#explain). It returns a list of
@ -969,8 +949,7 @@ tokens produced are identical to `nlp.tokenizer()` except for whitespace tokens:
> " SUFFIX > " SUFFIX
> ``` > ```
```python ```python {executable="true"}
### {executable="true"}
from spacy.lang.en import English from spacy.lang.en import English
nlp = English() nlp = English()
@ -982,7 +961,7 @@ for t in tok_exp:
print(t[1], "\\t", t[0]) print(t[1], "\\t", t[0])
``` ```
### Customizing spaCy's Tokenizer class {#native-tokenizers} ### Customizing spaCy's Tokenizer class {id="native-tokenizers"}
Let's imagine you wanted to create a tokenizer for a new language or specific Let's imagine you wanted to create a tokenizer for a new language or specific
domain. There are six things you may need to define: domain. There are six things you may need to define:
@ -1004,8 +983,7 @@ You shouldn't usually need to create a `Tokenizer` subclass. Standard usage is
to use `re.compile()` to build a regular expression object, and pass its to use `re.compile()` to build a regular expression object, and pass its
`.search()` and `.finditer()` methods: `.search()` and `.finditer()` methods:
```python ```python {executable="true"}
### {executable="true"}
import re import re
import spacy import spacy
from spacy.tokenizer import Tokenizer from spacy.tokenizer import Tokenizer
@ -1045,7 +1023,7 @@ only be applied at the **end of a token**, so your expression should end with a
</Infobox> </Infobox>
#### Modifying existing rule sets {#native-tokenizer-additions} #### Modifying existing rule sets {id="native-tokenizer-additions"}
In many situations, you don't necessarily need entirely custom rules. Sometimes In many situations, you don't necessarily need entirely custom rules. Sometimes
you just want to add another character to the prefixes, suffixes or infixes. The you just want to add another character to the prefixes, suffixes or infixes. The
@ -1098,8 +1076,7 @@ letters as an infix. If you do not want the tokenizer to split on hyphens
between letters, you can modify the existing infix definition from between letters, you can modify the existing infix definition from
[`lang/punctuation.py`](%%GITHUB_SPACY/spacy/lang/punctuation.py): [`lang/punctuation.py`](%%GITHUB_SPACY/spacy/lang/punctuation.py):
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.lang.char_classes import ALPHA, ALPHA_LOWER, ALPHA_UPPER from spacy.lang.char_classes import ALPHA, ALPHA_LOWER, ALPHA_UPPER
from spacy.lang.char_classes import CONCAT_QUOTES, LIST_ELLIPSES, LIST_ICONS from spacy.lang.char_classes import CONCAT_QUOTES, LIST_ELLIPSES, LIST_ICONS
@ -1138,7 +1115,7 @@ language-specific definitions such as
[`lang/de/punctuation.py`](%%GITHUB_SPACY/spacy/lang/de/punctuation.py) for [`lang/de/punctuation.py`](%%GITHUB_SPACY/spacy/lang/de/punctuation.py) for
German. German.
### Hooking a custom tokenizer into the pipeline {#custom-tokenizer} ### Hooking a custom tokenizer into the pipeline {id="custom-tokenizer"}
The tokenizer is the first component of the processing pipeline and the only one The tokenizer is the first component of the processing pipeline and the only one
that can't be replaced by writing to `nlp.pipeline`. This is because it has a that can't be replaced by writing to `nlp.pipeline`. This is because it has a
@ -1146,7 +1123,7 @@ different signature from all the other components: it takes a text and returns a
[`Doc`](/api/doc), whereas all other components expect to already receive a [`Doc`](/api/doc), whereas all other components expect to already receive a
tokenized `Doc`. tokenized `Doc`.
![The processing pipeline](../images/pipeline.svg) ![The processing pipeline](/images/pipeline.svg)
To overwrite the existing tokenizer, you need to replace `nlp.tokenizer` with a To overwrite the existing tokenizer, you need to replace `nlp.tokenizer` with a
custom function that takes a text and returns a [`Doc`](/api/doc). custom function that takes a text and returns a [`Doc`](/api/doc).
@ -1175,7 +1152,7 @@ nlp.tokenizer = my_tokenizer
| `text` | `str` | The raw text to tokenize. | | `text` | `str` | The raw text to tokenize. |
| **RETURNS** | [`Doc`](/api/doc) | The tokenized document. | | **RETURNS** | [`Doc`](/api/doc) | The tokenized document. |
#### Example 1: Basic whitespace tokenizer {#custom-tokenizer-example} #### Example 1: Basic whitespace tokenizer {id="custom-tokenizer-example"}
Here's an example of the most basic whitespace tokenizer. It takes the shared Here's an example of the most basic whitespace tokenizer. It takes the shared
vocab, so it can construct `Doc` objects. When it's called on a text, it returns vocab, so it can construct `Doc` objects. When it's called on a text, it returns
@ -1183,8 +1160,7 @@ a `Doc` object consisting of the text split on single space characters. We can
then overwrite the `nlp.tokenizer` attribute with an instance of our custom then overwrite the `nlp.tokenizer` attribute with an instance of our custom
tokenizer. tokenizer.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.tokens import Doc from spacy.tokens import Doc
@ -1215,7 +1191,7 @@ doc = nlp("What's happened to me? he thought. It wasn't a dream.")
print([token.text for token in doc]) print([token.text for token in doc])
``` ```
#### Example 2: Third-party tokenizers (BERT word pieces) {#custom-tokenizer-example2} #### Example 2: Third-party tokenizers (BERT word pieces) {id="custom-tokenizer-example2"}
You can use the same approach to plug in any other third-party tokenizers. Your You can use the same approach to plug in any other third-party tokenizers. Your
custom callable just needs to return a `Doc` object with the tokens produced by custom callable just needs to return a `Doc` object with the tokens produced by
@ -1234,8 +1210,7 @@ produced by the tokenizer.
> **training transformer models** in spaCy, as well as helpful utilities for > **training transformer models** in spaCy, as well as helpful utilities for
> aligning word pieces to linguistic tokenization. > aligning word pieces to linguistic tokenization.
```python ```python {title="Custom BERT word piece tokenizer"}
### Custom BERT word piece tokenizer
from tokenizers import BertWordPieceTokenizer from tokenizers import BertWordPieceTokenizer
from spacy.tokens import Doc from spacy.tokens import Doc
import spacy import spacy
@ -1279,7 +1254,7 @@ tokenizer** it will be using at runtime. See the docs on
</Infobox> </Infobox>
#### Training with custom tokenization {#custom-tokenizer-training new="3"} #### Training with custom tokenization {id="custom-tokenizer-training",version="3"}
spaCy's [training config](/usage/training#config) describes the settings, spaCy's [training config](/usage/training#config) describes the settings,
hyperparameters, pipeline and tokenizer used for constructing and training the hyperparameters, pipeline and tokenizer used for constructing and training the
@ -1297,8 +1272,7 @@ setting `--code functions.py` when you run [`spacy train`](/api/cli#train).
> @tokenizers = "whitespace_tokenizer" > @tokenizers = "whitespace_tokenizer"
> ``` > ```
```python ```python {title="functions.py",highlight="1"}
### functions.py {highlight="1"}
@spacy.registry.tokenizers("whitespace_tokenizer") @spacy.registry.tokenizers("whitespace_tokenizer")
def create_whitespace_tokenizer(): def create_whitespace_tokenizer():
def create_tokenizer(nlp): def create_tokenizer(nlp):
@ -1323,8 +1297,7 @@ correct type.
> lowercase = true > lowercase = true
> ``` > ```
```python ```python {title="functions.py",highlight="1"}
### functions.py {highlight="1"}
@spacy.registry.tokenizers("bert_word_piece_tokenizer") @spacy.registry.tokenizers("bert_word_piece_tokenizer")
def create_whitespace_tokenizer(vocab_file: str, lowercase: bool): def create_whitespace_tokenizer(vocab_file: str, lowercase: bool):
def create_tokenizer(nlp): def create_tokenizer(nlp):
@ -1348,7 +1321,7 @@ takes a text and returns a `Doc`.
</Infobox> </Infobox>
#### Using pre-tokenized text {#own-annotations} #### Using pre-tokenized text {id="own-annotations"}
spaCy generally assumes by default that your data is **raw text**. However, spaCy generally assumes by default that your data is **raw text**. However,
sometimes your data is partially annotated, e.g. with pre-existing tokenization, sometimes your data is partially annotated, e.g. with pre-existing tokenization,
@ -1367,8 +1340,7 @@ boolean values, indicating whether each word is followed by a space.
> `Doc` with `words` and `spaces` so that the `doc.text` matches the original > `Doc` with `words` and `spaces` so that the `doc.text` matches the original
> input text. > input text.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.tokens import Doc from spacy.tokens import Doc
@ -1388,7 +1360,7 @@ will assume that all words are followed by a space. Once you have a
part-of-speech tags, syntactic dependencies, named entities and other part-of-speech tags, syntactic dependencies, named entities and other
attributes. attributes.
#### Aligning tokenization {#aligning-tokenization} #### Aligning tokenization {id="aligning-tokenization"}
spaCy's tokenization is non-destructive and uses language-specific rules spaCy's tokenization is non-destructive and uses language-specific rules
optimized for compatibility with treebank annotations. Other tools and resources optimized for compatibility with treebank annotations. Other tools and resources
@ -1414,8 +1386,7 @@ token.
> 3. Make `other_tokens` and `spacy_tokens` identical. You'll see that all > 3. Make `other_tokens` and `spacy_tokens` identical. You'll see that all
> tokens now correspond 1-to-1. > tokens now correspond 1-to-1.
```python ```python {executable="true"}
### {executable="true"}
from spacy.training import Alignment from spacy.training import Alignment
other_tokens = ["i", "listened", "to", "obama", "'", "s", "podcasts", "."] other_tokens = ["i", "listened", "to", "obama", "'", "s", "podcasts", "."]
@ -1448,7 +1419,7 @@ tokenizations add up to the same string. For example, you'll be able to align
</Infobox> </Infobox>
## Merging and splitting {#retokenization new="2.1"} ## Merging and splitting {id="retokenization",version="2.1"}
The [`Doc.retokenize`](/api/doc#retokenize) context manager lets you merge and The [`Doc.retokenize`](/api/doc#retokenize) context manager lets you merge and
split tokens. Modifications to the tokenization are stored and performed all at split tokens. Modifications to the tokenization are stored and performed all at
@ -1467,8 +1438,7 @@ root.
> recognized as a named entity, this change will also be reflected in the > recognized as a named entity, this change will also be reflected in the
> `doc.ents`. > `doc.ents`.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -1537,8 +1507,7 @@ second split subtoken) and "York" should be attached to "in".
> 3. Split the token into three tokens instead of two for example, > 3. Split the token into three tokens instead of two for example,
> `["New", "Yo", "rk"]`. > `["New", "Yo", "rk"]`.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy import displacy from spacy import displacy
@ -1567,8 +1536,7 @@ the token indices after splitting.
If you don't care about the heads (for example, if you're only running the If you don't care about the heads (for example, if you're only running the
tokenizer and not the parser), you can attach each subtoken to itself: tokenizer and not the parser), you can attach each subtoken to itself:
```python ```python {highlight="3"}
### {highlight="3"}
doc = nlp("I live in NewYorkCity") doc = nlp("I live in NewYorkCity")
with doc.retokenize() as retokenizer: with doc.retokenize() as retokenizer:
heads = [(doc[3], 0), (doc[3], 1), (doc[3], 2)] heads = [(doc[3], 0), (doc[3], 1), (doc[3], 2)]
@ -1592,7 +1560,7 @@ with doc.retokenize() as retokenizer:
</Infobox> </Infobox>
### Overwriting custom extension attributes {#retokenization-extensions} ### Overwriting custom extension attributes {id="retokenization-extensions"}
If you've registered custom If you've registered custom
[extension attributes](/usage/processing-pipelines#custom-components-attributes), [extension attributes](/usage/processing-pipelines#custom-components-attributes),
@ -1624,8 +1592,7 @@ values can't be overwritten. For more details, see the
> you need to provide a list of extension attribute values as the `"_"` > you need to provide a list of extension attribute values as the `"_"`
> property, one for each split subtoken. > property, one for each split subtoken.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.tokens import Token from spacy.tokens import Token
@ -1641,7 +1608,7 @@ with doc.retokenize() as retokenizer:
print("After:", [(token.text, token._.is_musician) for token in doc]) print("After:", [(token.text, token._.is_musician) for token in doc])
``` ```
## Sentence Segmentation {#sbd} ## Sentence Segmentation {id="sbd"}
A [`Doc`](/api/doc) object's sentences are available via the `Doc.sents` A [`Doc`](/api/doc) object's sentences are available via the `Doc.sents`
property. To view a `Doc`'s sentences, you can iterate over the `Doc.sents`, a property. To view a `Doc`'s sentences, you can iterate over the `Doc.sents`, a
@ -1650,8 +1617,7 @@ has sentence boundaries by calling
[`Doc.has_annotation`](/api/doc#has_annotation) with the attribute name [`Doc.has_annotation`](/api/doc#has_annotation) with the attribute name
`"SENT_START"`. `"SENT_START"`.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -1676,7 +1642,7 @@ spaCy provides four alternatives for sentence segmentation:
processing pipeline can set sentence boundaries by writing to processing pipeline can set sentence boundaries by writing to
`Token.is_sent_start`. `Token.is_sent_start`.
### Default: Using the dependency parse {#sbd-parser model="parser"} ### Default: Using the dependency parse {id="sbd-parser",model="parser"}
Unlike other libraries, spaCy uses the dependency parse to determine sentence Unlike other libraries, spaCy uses the dependency parse to determine sentence
boundaries. This is usually the most accurate approach, but it requires a boundaries. This is usually the most accurate approach, but it requires a
@ -1686,8 +1652,7 @@ with spaCy's provided trained pipelines. For social media or conversational text
that doesn't follow the same rules, your application may benefit from a custom that doesn't follow the same rules, your application may benefit from a custom
trained or rule-based component. trained or rule-based component.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -1701,7 +1666,7 @@ your `Doc` using custom components _before_ it's parsed. Depending on your text,
this may also improve parse accuracy, since the parser is constrained to predict this may also improve parse accuracy, since the parser is constrained to predict
parses consistent with the sentence boundaries. parses consistent with the sentence boundaries.
### Statistical sentence segmenter {#sbd-senter model="senter" new="3"} ### Statistical sentence segmenter {id="sbd-senter",model="senter",version="3"}
The [`SentenceRecognizer`](/api/sentencerecognizer) is a simple statistical The [`SentenceRecognizer`](/api/sentencerecognizer) is a simple statistical
component that only provides sentence boundaries. Along with being faster and component that only provides sentence boundaries. Along with being faster and
@ -1721,8 +1686,7 @@ without the parser and then enable the sentence recognizer explicitly with
> which is better at predicting sentence boundaries when punctuation is not > which is better at predicting sentence boundaries when punctuation is not
> present. > present.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm", exclude=["parser"]) nlp = spacy.load("en_core_web_sm", exclude=["parser"])
@ -1732,15 +1696,14 @@ for sent in doc.sents:
print(sent.text) print(sent.text)
``` ```
### Rule-based pipeline component {#sbd-component} ### Rule-based pipeline component {id="sbd-component"}
The [`Sentencizer`](/api/sentencizer) component is a The [`Sentencizer`](/api/sentencizer) component is a
[pipeline component](/usage/processing-pipelines) that splits sentences on [pipeline component](/usage/processing-pipelines) that splits sentences on
punctuation like `.`, `!` or `?`. You can plug it into your pipeline if you only punctuation like `.`, `!` or `?`. You can plug it into your pipeline if you only
need sentence boundaries without dependency parses. need sentence boundaries without dependency parses.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.lang.en import English from spacy.lang.en import English
@ -1779,8 +1742,7 @@ for unset sentence boundaries. This approach can be useful if you want to
implement **additional** rules specific to your data, while still being able to implement **additional** rules specific to your data, while still being able to
take advantage of dependency-based sentence segmentation. take advantage of dependency-based sentence segmentation.
```python ```python {executable="true"}
### {executable="true"}
from spacy.language import Language from spacy.language import Language
import spacy import spacy
@ -1802,7 +1764,7 @@ doc = nlp(text)
print("After:", [sent.text for sent in doc.sents]) print("After:", [sent.text for sent in doc.sents])
``` ```
## Mappings & Exceptions {#mappings-exceptions new="3"} ## Mappings & Exceptions {id="mappings-exceptions",version="3"}
The [`AttributeRuler`](/api/attributeruler) manages **rule-based mappings and The [`AttributeRuler`](/api/attributeruler) manages **rule-based mappings and
exceptions** for all token-level attributes. As the number of exceptions** for all token-level attributes. As the number of
@ -1830,8 +1792,7 @@ The following example shows how the tag and POS `NNP`/`PROPN` can be specified
for the phrase `"The Who"`, overriding the tags provided by the statistical for the phrase `"The Who"`, overriding the tags provided by the statistical
tagger and the POS tag map. tagger and the POS tag map.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -1866,13 +1827,11 @@ initialized before training. See the
</Infobox> </Infobox>
## Word vectors and semantic similarity {#vectors-similarity} ## Word vectors and semantic similarity {id="vectors-similarity"}
import Vectors101 from 'usage/101/\_vectors-similarity.md'
<Vectors101 /> <Vectors101 />
### Adding word vectors {#adding-vectors} ### Adding word vectors {id="adding-vectors"}
Custom word vectors can be trained using a number of open-source libraries, such Custom word vectors can be trained using a number of open-source libraries, such
as [Gensim](https://radimrehurek.com/gensim), [FastText](https://fasttext.cc), as [Gensim](https://radimrehurek.com/gensim), [FastText](https://fasttext.cc),
@ -1898,7 +1857,7 @@ access to some nice Latin vectors. You can then pass the directory path to
> doc1.similarity(doc2) > doc1.similarity(doc2)
> ``` > ```
```cli ```bash
$ wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.la.300.vec.gz $ wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.la.300.vec.gz
$ python -m spacy init vectors en cc.la.300.vec.gz /tmp/la_vectors_wiki_lg $ python -m spacy init vectors en cc.la.300.vec.gz /tmp/la_vectors_wiki_lg
``` ```
@ -1936,8 +1895,7 @@ the removed words, mapped to `(string, score)` tuples, where `string` is the
entry the removed word was mapped to and `score` the similarity score between entry the removed word was mapped to and `score` the similarity score between
the two words. the two words.
```python ```python {title="Removed words"}
### Removed words
{ {
"Shore": ("coast", 0.732257), "Shore": ("coast", 0.732257),
"Precautionary": ("caution", 0.490973), "Precautionary": ("caution", 0.490973),
@ -1958,7 +1916,7 @@ the vector of "leaving", which is identical. If you're using the
option to easily reduce the size of the vectors as you add them to a spaCy option to easily reduce the size of the vectors as you add them to a spaCy
pipeline: pipeline:
```cli ```bash
$ python -m spacy init vectors en la.300d.vec.tgz /tmp/la_vectors_web_md --prune 10000 $ python -m spacy init vectors en la.300d.vec.tgz /tmp/la_vectors_web_md --prune 10000
``` ```
@ -1968,7 +1926,7 @@ among those retained.
</Accordion> </Accordion>
### Adding vectors individually {#adding-individual-vectors} ### Adding vectors individually {id="adding-individual-vectors"}
The `vector` attribute is a **read-only** numpy or cupy array (depending on The `vector` attribute is a **read-only** numpy or cupy array (depending on
whether you've configured spaCy to use GPU memory), with dtype `float32`. The whether you've configured spaCy to use GPU memory), with dtype `float32`. The
@ -1982,8 +1940,7 @@ be slower than approaches that work with the whole vectors table at once, but
it's a great approach for once-off conversions before you save out your `nlp` it's a great approach for once-off conversions before you save out your `nlp`
object to disk. object to disk.
```python ```python {title="Adding vectors"}
### Adding vectors
from spacy.vocab import Vocab from spacy.vocab import Vocab
vector_data = { vector_data = {
@ -1996,13 +1953,11 @@ for word, vector in vector_data.items():
vocab.set_vector(word, vector) vocab.set_vector(word, vector)
``` ```
## Language Data {#language-data} ## Language Data {id="language-data"}
import LanguageData101 from 'usage/101/\_language-data.md'
<LanguageData101 /> <LanguageData101 />
### Creating a custom language subclass {#language-subclass} ### Creating a custom language subclass {id="language-subclass"}
If you want to customize multiple components of the language data or add support If you want to customize multiple components of the language data or add support
for a custom language or domain-specific "dialect", you can also implement your for a custom language or domain-specific "dialect", you can also implement your
@ -2011,8 +1966,7 @@ own language subclass. The subclass should define two attributes: the `lang`
overview of the available attributes that can be overwritten, see the overview of the available attributes that can be overwritten, see the
[`Language.Defaults`](/api/language#defaults) documentation. [`Language.Defaults`](/api/language#defaults) documentation.
```python ```python {executable="true"}
### {executable="true"}
from spacy.lang.en import English from spacy.lang.en import English
class CustomEnglishDefaults(English.Defaults): class CustomEnglishDefaults(English.Defaults):
@ -2050,12 +2004,11 @@ language name, and even train pipelines with it and refer to it in your
> needs to be available during training. You can load a Python file containing > needs to be available during training. You can load a Python file containing
> the code using the `--code` argument: > the code using the `--code` argument:
> >
> ```cli > ```bash
> python -m spacy train config.cfg --code code.py > python -m spacy train config.cfg --code code.py
> ``` > ```
```python ```python {title="Registering a custom language",highlight="7,12-13"}
### Registering a custom language {highlight="7,12-13"}
import spacy import spacy
from spacy.lang.en import English from spacy.lang.en import English

View File

@ -23,9 +23,11 @@ located anywhere on your file system.
## Quickstart {hidden="true"} ## Quickstart {hidden="true"}
import QuickstartModels from 'widgets/quickstart-models.js' <QuickstartModels
title="Quickstart"
<QuickstartModels title="Quickstart" id="quickstart" description="Install a default trained pipeline package, get the code to load it from within spaCy and an example to test it. For more options, see the section on available packages below." /> id="quickstart"
description="Install a default trained pipeline package, get the code to load it from within spaCy and an example to test it. For more options, see the section on available packages below."
/>
### Usage note ### Usage note
@ -56,7 +58,7 @@ yields the same result as generating it using `spacy.blank()`. In both cases the
default configuration for the chosen language is loaded, and no pretrained default configuration for the chosen language is loaded, and no pretrained
components will be available. components will be available.
## Language support {#languages} ## Language support {id="languages"}
spaCy currently provides support for the following languages. You can help by spaCy currently provides support for the following languages. You can help by
improving the existing [language data](/usage/linguistic-features#language-data) improving the existing [language data](/usage/linguistic-features#language-data)
@ -66,11 +68,9 @@ contribute to development. Also see the
[training documentation](/usage/training) for how to train your own pipelines on [training documentation](/usage/training) for how to train your own pipelines on
your data. your data.
import Languages from 'widgets/languages.js'
<Languages /> <Languages />
### Multi-language support {#multi-language new="2"} ### Multi-language support {id="multi-language",version="2"}
> ```python > ```python
> # Standard import > # Standard import
@ -89,10 +89,10 @@ generic subclass containing only the base language data, can be found in
To train a pipeline using the neutral multi-language class, you can set To train a pipeline using the neutral multi-language class, you can set
`lang = "xx"` in your [training config](/usage/training#config). You can also `lang = "xx"` in your [training config](/usage/training#config). You can also
import the `MultiLanguage` class directly, or call \import the `MultiLanguage` class directly, or call
[`spacy.blank("xx")`](/api/top-level#spacy.blank) for lazy-loading. [`spacy.blank("xx")`](/api/top-level#spacy.blank) for lazy-loading.
### Chinese language support {#chinese new="2.3"} ### Chinese language support {id="chinese",version="2.3"}
The Chinese language class supports three word segmentation options, `char`, The Chinese language class supports three word segmentation options, `char`,
`jieba` and `pkuseg`. `jieba` and `pkuseg`.
@ -113,8 +113,7 @@ The Chinese language class supports three word segmentation options, `char`,
> nlp.tokenizer.initialize(pkuseg_model="mixed") > nlp.tokenizer.initialize(pkuseg_model="mixed")
> ``` > ```
```ini ```ini {title="config.cfg"}
### config.cfg
[nlp.tokenizer] [nlp.tokenizer]
@tokenizers = "spacy.zh.ChineseTokenizer" @tokenizers = "spacy.zh.ChineseTokenizer"
segmenter = "char" segmenter = "char"
@ -155,8 +154,7 @@ local path at runtime. See the usage guide on the
[config lifecycle](/usage/training#config-lifecycle) for more background on [config lifecycle](/usage/training#config-lifecycle) for more background on
this. this.
```ini ```ini {title="config.cfg"}
### config.cfg
[initialize] [initialize]
[initialize.tokenizer] [initialize.tokenizer]
@ -167,8 +165,7 @@ pkuseg_user_dict = "default"
You can also initialize the tokenizer for a blank language class by calling its You can also initialize the tokenizer for a blank language class by calling its
`initialize` method: `initialize` method:
```python ```python {title="Examples"}
### Examples
# Initialize the pkuseg tokenizer # Initialize the pkuseg tokenizer
cfg = {"segmenter": "pkuseg"} cfg = {"segmenter": "pkuseg"}
nlp = Chinese.from_config({"nlp": {"tokenizer": cfg}}) nlp = Chinese.from_config({"nlp": {"tokenizer": cfg}})
@ -227,7 +224,7 @@ nlp.tokenizer.initialize(pkuseg_model="/path/to/pkuseg_model")
</Accordion> </Accordion>
### Japanese language support {#japanese new=2.3} ### Japanese language support {id="japanese",version="2.3"}
> #### Manual setup > #### Manual setup
> >
@ -247,8 +244,7 @@ segmentation and part-of-speech tagging. The default Japanese language class and
the provided Japanese pipelines use SudachiPy split mode `A`. The tokenizer the provided Japanese pipelines use SudachiPy split mode `A`. The tokenizer
config can be used to configure the split mode to `A`, `B` or `C`. config can be used to configure the split mode to `A`, `B` or `C`.
```ini ```ini {title="config.cfg"}
### config.cfg
[nlp.tokenizer] [nlp.tokenizer]
@tokenizers = "spacy.ja.JapaneseTokenizer" @tokenizers = "spacy.ja.JapaneseTokenizer"
split_mode = "A" split_mode = "A"
@ -266,7 +262,7 @@ used for training the current [Japanese pipelines](/models/ja).
</Infobox> </Infobox>
### Korean language support {#korean} ### Korean language support {id="korean"}
> #### mecab-ko tokenizer > #### mecab-ko tokenizer
> >
@ -291,8 +287,7 @@ than MeCab. To configure a Korean pipeline with the rule-based tokenizer:
> nlp = spacy.blank("ko", config=config) > nlp = spacy.blank("ko", config=config)
> ``` > ```
```ini ```ini {title="config.cfg"}
### config.cfg
[nlp] [nlp]
lang = "ko" lang = "ko"
tokenizer = {"@tokenizers" = "spacy.Tokenizer.v1"} tokenizer = {"@tokenizers" = "spacy.Tokenizer.v1"}
@ -305,7 +300,7 @@ additional dependencies are required.
</Infobox> </Infobox>
## Installing and using trained pipelines {#download} ## Installing and using trained pipelines {id="download"}
The easiest way to download a trained pipeline is via spaCy's The easiest way to download a trained pipeline is via spaCy's
[`download`](/api/cli#download) command. It takes care of finding the [`download`](/api/cli#download) command. It takes care of finding the
@ -327,7 +322,7 @@ best-matching package compatible with your spaCy installation.
> + nlp = spacy.load("en_core_web_sm") > + nlp = spacy.load("en_core_web_sm")
> ``` > ```
```cli ```bash
# Download best-matching version of a package for your spaCy installation # Download best-matching version of a package for your spaCy installation
$ python -m spacy download en_core_web_sm $ python -m spacy download en_core_web_sm
@ -338,7 +333,7 @@ $ python -m spacy download en_core_web_sm-3.0.0 --direct
The download command will [install the package](/usage/models#download-pip) via The download command will [install the package](/usage/models#download-pip) via
pip and place the package in your `site-packages` directory. pip and place the package in your `site-packages` directory.
```cli ```bash
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS $ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
$ python -m spacy download en_core_web_sm $ python -m spacy download en_core_web_sm
``` ```
@ -356,18 +351,18 @@ Make sure to **restart your kernel** or runtime after installation (just like
you would when installing other Python packages) to make sure that the installed you would when installing other Python packages) to make sure that the installed
pipeline package can be found. pipeline package can be found.
```cli ```bash
!python -m spacy download en_core_web_sm !python -m spacy download en_core_web_sm
``` ```
### Installation via pip {#download-pip} ### Installation via pip {id="download-pip"}
To download a trained pipeline directly using To download a trained pipeline directly using
[pip](https://pypi.python.org/pypi/pip), point `pip install` to the URL or local [pip](https://pypi.python.org/pypi/pip), point `pip install` to the URL or local
path of the wheel file or archive. Installing the wheel is usually more path of the wheel file or archive. Installing the wheel is usually more
efficient. efficient.
> #### Pipeline Package URLs {#pipeline-urls} > #### Pipeline Package URLs {id="pipeline-urls"}
> >
> Pretrained pipeline distributions are hosted on > Pretrained pipeline distributions are hosted on
> [Github Releases](https://github.com/explosion/spacy-models/releases), and you > [Github Releases](https://github.com/explosion/spacy-models/releases), and you
@ -407,7 +402,7 @@ You can also add the direct download link to your application's
`requirements.txt`. For more details, see the section on `requirements.txt`. For more details, see the section on
[working with pipeline packages in production](#production). [working with pipeline packages in production](#production).
### Manual download and installation {#download-manual} ### Manual download and installation {id="download-manual"}
In some cases, you might prefer downloading the data manually, for example to In some cases, you might prefer downloading the data manually, for example to
place it into a custom directory. You can download the package via your browser place it into a custom directory. You can download the package via your browser
@ -416,8 +411,7 @@ or configure your own download script using the URL of the archive file. The
archive consists of a package directory that contains another directory with the archive consists of a package directory that contains another directory with the
pipeline data. pipeline data.
```yaml ```yaml {title="Directory structure",highlight="6"}
### Directory structure {highlight="6"}
└── en_core_web_md-3.0.0.tar.gz # downloaded archive └── en_core_web_md-3.0.0.tar.gz # downloaded archive
├── setup.py # setup file for pip installation ├── setup.py # setup file for pip installation
├── meta.json # copy of pipeline meta ├── meta.json # copy of pipeline meta
@ -432,7 +426,7 @@ pipeline data.
You can place the **pipeline package directory** anywhere on your local file You can place the **pipeline package directory** anywhere on your local file
system. system.
### Installation from Python {#download-python} ### Installation from Python {id="download-python"}
Since the [`spacy download`](/api/cli#download) command installs the pipeline as Since the [`spacy download`](/api/cli#download) command installs the pipeline as
a **Python package**, we always recommend running it from the command line, just a **Python package**, we always recommend running it from the command line, just
@ -453,7 +447,7 @@ import spacy
spacy.cli.download("en_core_web_sm") spacy.cli.download("en_core_web_sm")
``` ```
### Using trained pipelines with spaCy {#usage} ### Using trained pipelines with spaCy {id="usage"}
To load a pipeline package, use [`spacy.load`](/api/top-level#spacy.load) with To load a pipeline package, use [`spacy.load`](/api/top-level#spacy.load) with
the package name or a path to the data directory: the package name or a path to the data directory:
@ -487,14 +481,13 @@ exposes the pipeline's meta data as the attribute `meta`. For example,
</Infobox> </Infobox>
### Importing pipeline packages as modules {#usage-import} ### Importing pipeline packages as modules {id="usage-import"}
If you've installed a trained pipeline via [`spacy download`](/api/cli#download) If you've installed a trained pipeline via [`spacy download`](/api/cli#download)
or directly via pip, you can also `import` it and then call its `load()` method or directly via pip, you can also `import` it and then call its `load()` method
with no arguments: with no arguments:
```python ```python {executable="true"}
### {executable="true"}
import en_core_web_sm import en_core_web_sm
nlp = en_core_web_sm.load() nlp = en_core_web_sm.load()
@ -510,7 +503,7 @@ as your code will raise an `ImportError` immediately, instead of failing
somewhere down the line when calling `spacy.load()`. For more details, see the somewhere down the line when calling `spacy.load()`. For more details, see the
section on [working with pipeline packages in production](#production). section on [working with pipeline packages in production](#production).
## Using trained pipelines in production {#production} ## Using trained pipelines in production {id="production"}
If your application depends on one or more trained pipeline packages, you'll If your application depends on one or more trained pipeline packages, you'll
usually want to integrate them into your continuous integration workflow and usually want to integrate them into your continuous integration workflow and
@ -519,7 +512,7 @@ and loading pipeline packages, the underlying functionality is entirely based on
native Python packaging. This allows your application to handle a spaCy pipeline native Python packaging. This allows your application to handle a spaCy pipeline
like any other package dependency. like any other package dependency.
### Downloading and requiring package dependencies {#models-download} ### Downloading and requiring package dependencies {id="models-download"}
spaCy's built-in [`download`](/api/cli#download) command is mostly intended as a spaCy's built-in [`download`](/api/cli#download) command is mostly intended as a
convenient, interactive wrapper. It performs compatibility checks and prints convenient, interactive wrapper. It performs compatibility checks and prints
@ -535,8 +528,7 @@ installation, you can upload the pipeline packages there. pip's
supports both package names to download via a PyPi server, as well as supports both package names to download via a PyPi server, as well as
[direct URLs](#pipeline-urls). [direct URLs](#pipeline-urls).
```text ```text {title="requirements.txt"}
### requirements.txt
spacy>=3.0.0,<4.0.0 spacy>=3.0.0,<4.0.0
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.0/en_core_web_sm-3.4.0-py3-none-any.whl en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.0/en_core_web_sm-3.4.0-py3-none-any.whl
``` ```
@ -547,7 +539,7 @@ each pipeline. If you've [trained](/usage/training) your own pipeline, you can
use the [`spacy package`](/api/cli#package) command to generate the required use the [`spacy package`](/api/cli#package) command to generate the required
meta data and turn it into a loadable package. meta data and turn it into a loadable package.
### Loading and testing pipeline packages {#models-loading} ### Loading and testing pipeline packages {id="models-loading"}
Pipeline packages are regular Python packages, so you can also import them as a Pipeline packages are regular Python packages, so you can also import them as a
package using Python's native `import` syntax, and then call the `load` method package using Python's native `import` syntax, and then call the `load` method

View File

@ -12,11 +12,9 @@ menu:
- ['Plugins & Wrappers', 'plugins'] - ['Plugins & Wrappers', 'plugins']
--- ---
import Pipelines101 from 'usage/101/\_pipelines.md'
<Pipelines101 /> <Pipelines101 />
## Processing text {#processing} ## Processing text {id="processing"}
When you call `nlp` on a text, spaCy will **tokenize** it and then **call each When you call `nlp` on a text, spaCy will **tokenize** it and then **call each
component** on the `Doc`, in order. It then returns the processed `Doc` that you component** on the `Doc`, in order. It then returns the processed `Doc` that you
@ -62,8 +60,7 @@ so we can iterate over them and access the named entity predictions:
> 1. Also disable the `"ner"` component. You'll see that the `doc.ents` are now > 1. Also disable the `"ner"` component. You'll see that the `doc.ents` are now
> empty, because the entity recognizer didn't run. > empty, because the entity recognizer didn't run.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
texts = [ texts = [
@ -97,8 +94,7 @@ the input should be a sequence of `(text, context)` tuples and the output will
be a sequence of `(doc, context)` tuples. For example, you can pass metadata in be a sequence of `(doc, context)` tuples. For example, you can pass metadata in
the context and save it in a [custom attribute](#custom-components-attributes): the context and save it in a [custom attribute](#custom-components-attributes):
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.tokens import Doc from spacy.tokens import Doc
@ -122,7 +118,7 @@ for doc in docs:
print(f"{doc._.text_id}: {doc.text}") print(f"{doc._.text_id}: {doc.text}")
``` ```
### Multiprocessing {#multiprocessing} ### Multiprocessing {id="multiprocessing"}
spaCy includes built-in support for multiprocessing with spaCy includes built-in support for multiprocessing with
[`nlp.pipe`](/api/language#pipe) using the `n_process` option: [`nlp.pipe`](/api/language#pipe) using the `n_process` option:
@ -170,7 +166,7 @@ the number of threads before loading any models using
</Infobox> </Infobox>
## Pipelines and built-in components {#pipelines} ## Pipelines and built-in components {id="pipelines"}
spaCy makes it very easy to create your own pipelines consisting of reusable spaCy makes it very easy to create your own pipelines consisting of reusable
components this includes spaCy's default tagger, parser and entity recognizer, components this includes spaCy's default tagger, parser and entity recognizer,
@ -248,8 +244,7 @@ tagging pipeline. This is also why the pipeline state is always held by the
together and returns an instance of `Language` with a pipeline set and access to together and returns an instance of `Language` with a pipeline set and access to
the binary data: the binary data:
```python ```python {title="spacy.load under the hood"}
### spacy.load under the hood
lang = "en" lang = "en"
pipeline = ["tok2vec", "tagger", "parser", "ner", "attribute_ruler", "lemmatizer"] pipeline = ["tok2vec", "tagger", "parser", "ner", "attribute_ruler", "lemmatizer"]
data_path = "path/to/en_core_web_sm/en_core_web_sm-3.0.0" data_path = "path/to/en_core_web_sm/en_core_web_sm-3.0.0"
@ -268,8 +263,7 @@ subsequently to the `Token` and `Span` which are only views of the `Doc`, and
don't own any data themselves. All components return the modified document, don't own any data themselves. All components return the modified document,
which is then processed by the next component in the pipeline. which is then processed by the next component in the pipeline.
```python ```python {title="The pipeline under the hood"}
### The pipeline under the hood
doc = nlp.make_doc("This is a sentence") # Create a Doc from raw text doc = nlp.make_doc("This is a sentence") # Create a Doc from raw text
for name, proc in nlp.pipeline: # Iterate over components in order for name, proc in nlp.pipeline: # Iterate over components in order
doc = proc(doc) # Apply each component doc = proc(doc) # Apply each component
@ -286,7 +280,7 @@ print(nlp.pipe_names)
# ['tok2vec', 'tagger', 'parser', 'ner', 'attribute_ruler', 'lemmatizer'] # ['tok2vec', 'tagger', 'parser', 'ner', 'attribute_ruler', 'lemmatizer']
``` ```
### Built-in pipeline components {#built-in} ### Built-in pipeline components {id="built-in"}
spaCy ships with several built-in pipeline components that are registered with spaCy ships with several built-in pipeline components that are registered with
string names. This means that you can initialize them by calling string names. This means that you can initialize them by calling
@ -321,7 +315,7 @@ available pipeline components and component functions.
| `tok2vec` | [`Tok2Vec`](/api/tok2vec) | Assign token-to-vector embeddings. | | `tok2vec` | [`Tok2Vec`](/api/tok2vec) | Assign token-to-vector embeddings. |
| `transformer` | [`Transformer`](/api/transformer) | Assign the tokens and outputs of a transformer model. | | `transformer` | [`Transformer`](/api/transformer) | Assign the tokens and outputs of a transformer model. |
### Disabling, excluding and modifying components {#disabling} ### Disabling, excluding and modifying components {id="disabling"}
If you don't need a particular component of the pipeline for example, the If you don't need a particular component of the pipeline for example, the
tagger or the parser, you can **disable or exclude** it. This can sometimes make tagger or the parser, you can **disable or exclude** it. This can sometimes make
@ -364,7 +358,8 @@ nlp.enable_pipe("tagger")
In addition to `disable`, `spacy.load()` also accepts `enable`. If `enable` is In addition to `disable`, `spacy.load()` also accepts `enable`. If `enable` is
set, all components except for those in `enable` are disabled. If `enable` and set, all components except for those in `enable` are disabled. If `enable` and
`disable` conflict (i.e. the same component is included in both), an error is raised. `disable` conflict (i.e. the same component is included in both), an error is
raised.
```python ```python
# Load the complete pipeline, but disable all components except for tok2vec and tagger # Load the complete pipeline, but disable all components except for tok2vec and tagger
@ -391,8 +386,7 @@ call its `restore()` method to restore the disabled components when needed. This
can be useful if you want to prevent unnecessary code indentation of large can be useful if you want to prevent unnecessary code indentation of large
blocks. blocks.
```python ```python {title="Disable for block"}
### Disable for block
# 1. Use as a context manager # 1. Use as a context manager
with nlp.select_pipes(disable=["tagger", "parser", "lemmatizer"]): with nlp.select_pipes(disable=["tagger", "parser", "lemmatizer"]):
doc = nlp("I won't be tagged and parsed") doc = nlp("I won't be tagged and parsed")
@ -460,7 +454,7 @@ run as part of the pipeline.
| `nlp.component_names` | All component names, including disabled components. | | `nlp.component_names` | All component names, including disabled components. |
| `nlp.disabled` | Names of components that are currently disabled. | | `nlp.disabled` | Names of components that are currently disabled. |
### Sourcing components from existing pipelines {#sourced-components new="3"} ### Sourcing components from existing pipelines {id="sourced-components",version="3"}
Pipeline components that are independent can also be reused across pipelines. Pipeline components that are independent can also be reused across pipelines.
Instead of adding a new blank component, you can also copy an existing component Instead of adding a new blank component, you can also copy an existing component
@ -503,8 +497,7 @@ vectors available otherwise, it won't be able to make the same predictions.
> frozen_components = ["ner"] > frozen_components = ["ner"]
> ``` > ```
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
# The source pipeline with different components # The source pipeline with different components
@ -517,7 +510,7 @@ nlp.add_pipe("ner", source=source_nlp)
print(nlp.pipe_names) print(nlp.pipe_names)
``` ```
### Analyzing pipeline components {#analysis new="3"} ### Analyzing pipeline components {id="analysis",version="3"}
The [`nlp.analyze_pipes`](/api/language#analyze_pipes) method analyzes the The [`nlp.analyze_pipes`](/api/language#analyze_pipes) method analyzes the
components in the current pipeline and outputs information about them like the components in the current pipeline and outputs information about them like the
@ -534,8 +527,7 @@ table instead of only returning the structured data.
> `"entity_linker"`. The analysis should now show no problems, because > `"entity_linker"`. The analysis should now show no problems, because
> requirements are met. > requirements are met.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.blank("en") nlp = spacy.blank("en")
@ -547,8 +539,7 @@ analysis = nlp.analyze_pipes(pretty=True)
<Accordion title="Example output"> <Accordion title="Example output">
```json ```json {title="Structured"}
### Structured
{ {
"summary": { "summary": {
"tagger": { "tagger": {
@ -566,7 +557,12 @@ analysis = nlp.analyze_pipes(pretty=True)
}, },
"problems": { "problems": {
"tagger": [], "tagger": [],
"entity_linker": ["doc.ents", "doc.sents", "token.ent_iob", "token.ent_type"] "entity_linker": [
"doc.ents",
"doc.sents",
"token.ent_iob",
"token.ent_type"
]
}, },
"attrs": { "attrs": {
"token.ent_iob": { "assigns": [], "requires": ["entity_linker"] }, "token.ent_iob": { "assigns": [], "requires": ["entity_linker"] },
@ -609,7 +605,7 @@ doesn't, the pipeline analysis won't catch that.
</Infobox> </Infobox>
## Creating custom pipeline components {#custom-components} ## Creating custom pipeline components {id="custom-components"}
A pipeline component is a function that receives a `Doc` object, modifies it and A pipeline component is a function that receives a `Doc` object, modifies it and
returns it for example, by using the current weights to make a prediction and returns it for example, by using the current weights to make a prediction and
@ -678,7 +674,7 @@ your custom components, and make sure they can be saved and loaded.
</Infobox> </Infobox>
### Examples: Simple stateless pipeline components {#custom-components-simple} ### Examples: Simple stateless pipeline components {id="custom-components-simple"}
The following component receives the `Doc` in the pipeline and prints some The following component receives the `Doc` in the pipeline and prints some
information about it: the number of tokens, the part-of-speech tags of the information about it: the number of tokens, the part-of-speech tags of the
@ -699,8 +695,7 @@ component under the name `"info_component"`.
> else. spaCy should now complain that it doesn't know a component of the > else. spaCy should now complain that it doesn't know a component of the
> name `"info_component"`. > name `"info_component"`.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.language import Language from spacy.language import Language
@ -733,8 +728,7 @@ boundaries.
> to `None` (missing value), the parser will assign sentence boundaries in > to `None` (missing value), the parser will assign sentence boundaries in
> between. > between.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.language import Language from spacy.language import Language
@ -757,7 +751,7 @@ for sent in doc.sents:
print(sent.text) print(sent.text)
``` ```
### Component factories and stateful components {#custom-components-factories} ### Component factories and stateful components {id="custom-components-factories"}
Component factories are callables that take settings and return a **pipeline Component factories are callables that take settings and return a **pipeline
component function**. This is useful if your component is stateful and if you component function**. This is useful if your component is stateful and if you
@ -787,8 +781,7 @@ All other settings can be passed in by the user via the `config` argument on
[`@Language.factory`](/api/language#factory) decorator also lets you define a [`@Language.factory`](/api/language#factory) decorator also lets you define a
`default_config` that's used as a fallback. `default_config` that's used as a fallback.
```python ```python {title="With config",highlight="4,9"}
### With config {highlight="4,9"}
import spacy import spacy
from spacy.language import Language from spacy.language import Language
@ -837,7 +830,7 @@ make your factory a separate function. That's also how spaCy does it internally.
</Accordion> </Accordion>
### Language-specific factories {#factories-language new="3"} ### Language-specific factories {id="factories-language",version="3"}
There are many use cases where you might want your pipeline components to be There are many use cases where you might want your pipeline components to be
language-specific. Sometimes this requires entirely different implementation per language-specific. Sometimes this requires entirely different implementation per
@ -852,8 +845,7 @@ a token, the `Token.norm_` with an entry from a language-specific lookup table.
It's registered twice under the name `"token_normalizer"` once using It's registered twice under the name `"token_normalizer"` once using
`@English.factory` and once using `@German.factory`: `@English.factory` and once using `@German.factory`:
```python ```python {executable="true"}
### {executable="true"}
from spacy.lang.en import English from spacy.lang.en import English
from spacy.lang.de import German from spacy.lang.de import German
@ -895,7 +887,7 @@ available, falls back to looking up the regular factory name.
</Infobox> </Infobox>
### Example: Stateful component with settings {#example-stateful-components} ### Example: Stateful component with settings {id="example-stateful-components"}
This example shows a **stateful** pipeline component for handling acronyms: This example shows a **stateful** pipeline component for handling acronyms:
based on a dictionary, it will detect acronyms and their expanded forms in both based on a dictionary, it will detect acronyms and their expanded forms in both
@ -922,8 +914,7 @@ case-sensitive.
> should see an entry for the acronyms component, referencing the factory > should see an entry for the acronyms component, referencing the factory
> `acronyms` and the config settings. > `acronyms` and the config settings.
```python ```python {executable="true"}
### {executable="true"}
from spacy.language import Language from spacy.language import Language
from spacy.tokens import Doc from spacy.tokens import Doc
from spacy.matcher import PhraseMatcher from spacy.matcher import PhraseMatcher
@ -964,7 +955,7 @@ doc = nlp("LOL, be right back")
print(doc._.acronyms) print(doc._.acronyms)
``` ```
## Initializing and serializing component data {#component-data} ## Initializing and serializing component data {id="component-data"}
Many stateful components depend on **data resources** like dictionaries and Many stateful components depend on **data resources** like dictionaries and
lookup tables that should ideally be **configurable**. For example, it makes lookup tables that should ideally be **configurable**. For example, it makes
@ -995,7 +986,7 @@ that if a component saves out its config and settings, the
since that's the config the component was created with. It will also fail if the since that's the config the component was created with. It will also fail if the
data is not JSON-serializable. data is not JSON-serializable.
### Option 1: Using a registered function {#component-data-function} ### Option 1: Using a registered function {id="component-data-function"}
<Infobox> <Infobox>
@ -1025,8 +1016,7 @@ argument, the name:
> batchers. `misc` is intended for miscellaneous functions that don't fit > batchers. `misc` is intended for miscellaneous functions that don't fit
> anywhere else. > anywhere else.
```python ```python {title="Registered function for assets",highlight="1"}
### Registered function for assets {highlight="1"}
@spacy.registry.misc("acronyms.slang_dict.v1") @spacy.registry.misc("acronyms.slang_dict.v1")
def create_acronyms_slang_dict(): def create_acronyms_slang_dict():
dictionary = {"lol": "laughing out loud", "brb": "be right back"} dictionary = {"lol": "laughing out loud", "brb": "be right back"}
@ -1064,7 +1054,7 @@ the name. Registered functions can also take **arguments**, by the way, that can
be defined in the config as well you can read more about this in the docs on be defined in the config as well you can read more about this in the docs on
[training with custom code](/usage/training#custom-code). [training with custom code](/usage/training#custom-code).
### Option 2: Save data with the pipeline and load it in once on initialization {#component-data-initialization} ### Option 2: Save data with the pipeline and load it in once on initialization {id="component-data-initialization"}
<Infobox> <Infobox>
@ -1094,8 +1084,7 @@ on [serialization methods](/usage/saving-loading/#serialization-methods).
> receive the directory path `/path/acronyms` and can then create files in this > receive the directory path `/path/acronyms` and can then create files in this
> directory. > directory.
```python ```python {title="Custom serialization methods",highlight="7-11,13-15"}
### Custom serialization methods {highlight="7-11,13-15"}
import srsly import srsly
from spacy.util import ensure_path from spacy.util import ensure_path
@ -1157,7 +1146,7 @@ pipeline is loaded. For more background on this, see the usage guides on the
[config lifecycle](/usage/training#config-lifecycle) and [config lifecycle](/usage/training#config-lifecycle) and
[custom initialization](/usage/training#initialization). [custom initialization](/usage/training#initialization).
![Illustration of pipeline lifecycle](../images/lifecycle.svg) ![Illustration of pipeline lifecycle](/images/lifecycle.svg)
A component's `initialize` method needs to take at least **two named A component's `initialize` method needs to take at least **two named
arguments**: a `get_examples` callback that gives it access to the training arguments**: a `get_examples` callback that gives it access to the training
@ -1177,8 +1166,7 @@ be defined via the config in this case a dictionary `data`.
> path = "/path/to/slang_dict.json" > path = "/path/to/slang_dict.json"
> ``` > ```
```python ```python {title="Custom initialize method",highlight="5-6"}
### Custom initialize method {highlight="5-6"}
class AcronymComponent: class AcronymComponent:
def __init__(self): def __init__(self):
self.data = {} self.data = {}
@ -1196,7 +1184,7 @@ object is saved to disk, which will run the component's `to_disk` method. When
the pipeline is loaded back into spaCy later to use it, the `from_disk` method the pipeline is loaded back into spaCy later to use it, the `from_disk` method
will load the data back in. will load the data back in.
## Python type hints and validation {#type-hints new="3"} ## Python type hints and validation {id="type-hints",version="3"}
spaCy's configs are powered by our machine learning library Thinc's spaCy's configs are powered by our machine learning library Thinc's
[configuration system](https://thinc.ai/docs/usage-config), which supports [configuration system](https://thinc.ai/docs/usage-config), which supports
@ -1241,8 +1229,7 @@ string value.
> and write a type hint for `log_level` that only accepts the exact string > and write a type hint for `log_level` that only accepts the exact string
> values `"DEBUG"`, `"INFO"` or `"CRITICAL"`. > values `"DEBUG"`, `"INFO"` or `"CRITICAL"`.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.language import Language from spacy.language import Language
from spacy.tokens import Doc from spacy.tokens import Doc
@ -1266,14 +1253,14 @@ nlp.add_pipe("debug", config={"log_level": "DEBUG"})
doc = nlp("This is a text...") doc = nlp("This is a text...")
``` ```
## Trainable components {#trainable-components new="3"} ## Trainable components {id="trainable-components",version="3"}
spaCy's [`TrainablePipe`](/api/pipe) class helps you implement your own spaCy's [`TrainablePipe`](/api/pipe) class helps you implement your own
trainable components that have their own model instance, make predictions over trainable components that have their own model instance, make predictions over
`Doc` objects and can be updated using [`spacy train`](/api/cli#train). This `Doc` objects and can be updated using [`spacy train`](/api/cli#train). This
lets you plug fully custom machine learning components into your pipeline. lets you plug fully custom machine learning components into your pipeline.
![Illustration of Pipe methods](../images/trainable_component.svg) ![Illustration of Pipe methods](/images/trainable_component.svg)
You'll need the following: You'll need the following:
@ -1331,8 +1318,7 @@ components. It also makes the components more **modular** and lets you
[swap](/usage/layers-architectures#swap-architectures) different architectures [swap](/usage/layers-architectures#swap-architectures) different architectures
in your config, and re-use model definitions. in your config, and re-use model definitions.
```ini ```ini {title="config.cfg (excerpt)"}
### config.cfg (excerpt)
[components] [components]
[components.textcat] [components.textcat]
@ -1383,7 +1369,7 @@ into your spaCy pipeline, see the usage guide on
</Infobox> </Infobox>
## Extension attributes {#custom-components-attributes new="2"} ## Extension attributes {id="custom-components-attributes",version="2"}
spaCy allows you to set any custom attributes and methods on the `Doc`, `Span` spaCy allows you to set any custom attributes and methods on the `Doc`, `Span`
and `Token`, which become available as `Doc._`, `Span._` and `Token._` for and `Token`, which become available as `Doc._`, `Span._` and `Token._` for
@ -1466,8 +1452,7 @@ particular instance. If an attribute of the same name already exists, or if
you're trying to access an attribute that hasn't been registered, spaCy will you're trying to access an attribute that hasn't been registered, spaCy will
raise an `AttributeError`. raise an `AttributeError`.
```python ```python {title="Example"}
### Example
from spacy.tokens import Doc, Span, Token from spacy.tokens import Doc, Span, Token
fruits = ["apple", "pear", "banana", "orange", "strawberry"] fruits = ["apple", "pear", "banana", "orange", "strawberry"]
@ -1494,7 +1479,7 @@ Once you've registered your custom attribute, you can also use the built-in
especially useful it you want to pass in a string instead of calling especially useful it you want to pass in a string instead of calling
`doc._.my_attr`. `doc._.my_attr`.
### Example: Pipeline component for GPE entities and country meta data via a REST API {#component-example3} ### Example: Pipeline component for GPE entities and country meta data via a REST API {id="component-example3"}
This example shows the implementation of a pipeline component that fetches This example shows the implementation of a pipeline component that fetches
country meta data via the [REST Countries API](https://restcountries.com), sets country meta data via the [REST Countries API](https://restcountries.com), sets
@ -1502,8 +1487,7 @@ entity annotations for countries and sets custom attributes on the `Doc` and
`Span` for example, the capital, latitude/longitude coordinates and even the `Span` for example, the capital, latitude/longitude coordinates and even the
country flag. country flag.
```python ```python {executable="true"}
### {executable="true"}
import requests import requests
from spacy.lang.en import English from spacy.lang.en import English
from spacy.language import Language from spacy.language import Language
@ -1570,7 +1554,7 @@ mistakes or foreign-language versions, you could also implement a
`like_country`-style getter function that makes a request to the search API `like_country`-style getter function that makes a request to the search API
endpoint and returns the best-matching result. endpoint and returns the best-matching result.
### User hooks {#custom-components-user-hooks} ### User hooks {id="custom-components-user-hooks"}
While it's generally recommended to use the `Doc._`, `Span._` and `Token._` While it's generally recommended to use the `Doc._`, `Span._` and `Token._`
proxies to add your own custom attributes, spaCy offers a few exceptions to proxies to add your own custom attributes, spaCy offers a few exceptions to
@ -1601,8 +1585,7 @@ to `Doc.user_span_hooks` and `Doc.user_token_hooks`.
| `user_token_hooks` | [`Token.similarity`](/api/token#similarity), [`Token.vector`](/api/token#vector), [`Token.has_vector`](/api/token#has_vector), [`Token.vector_norm`](/api/token#vector_norm), [`Token.conjuncts`](/api/token#conjuncts) | | `user_token_hooks` | [`Token.similarity`](/api/token#similarity), [`Token.vector`](/api/token#vector), [`Token.has_vector`](/api/token#has_vector), [`Token.vector_norm`](/api/token#vector_norm), [`Token.conjuncts`](/api/token#conjuncts) |
| `user_span_hooks` | [`Span.similarity`](/api/span#similarity), [`Span.vector`](/api/span#vector), [`Span.has_vector`](/api/span#has_vector), [`Span.vector_norm`](/api/span#vector_norm), [`Span.root`](/api/span#root) | | `user_span_hooks` | [`Span.similarity`](/api/span#similarity), [`Span.vector`](/api/span#vector), [`Span.has_vector`](/api/span#has_vector), [`Span.vector_norm`](/api/span#vector_norm), [`Span.root`](/api/span#root) |
```python ```python {title="Add custom similarity hooks"}
### Add custom similarity hooks
from spacy.language import Language from spacy.language import Language
@ -1626,7 +1609,7 @@ def create_similarity_component(nlp, name, index: int):
return SimilarityModel(name, index) return SimilarityModel(name, index)
``` ```
## Developing plugins and wrappers {#plugins} ## Developing plugins and wrappers {id="plugins"}
We're very excited about all the new possibilities for community extensions and We're very excited about all the new possibilities for community extensions and
plugins in spaCy, and we can't wait to see what you build with it! To get you plugins in spaCy, and we can't wait to see what you build with it! To get you
@ -1634,7 +1617,7 @@ started, here are a few tips, tricks and best
practices. [See here](/universe/?category=pipeline) for examples of other spaCy practices. [See here](/universe/?category=pipeline) for examples of other spaCy
extensions. extensions.
### Usage ideas {#custom-components-usage-ideas} ### Usage ideas {id="custom-components-usage-ideas"}
- **Adding new features and hooking in models.** For example, a sentiment - **Adding new features and hooking in models.** For example, a sentiment
analysis model, or your preferred solution for lemmatization or sentiment analysis model, or your preferred solution for lemmatization or sentiment
@ -1650,7 +1633,7 @@ extensions.
exports relevant information about the current state of the processed exports relevant information about the current state of the processed
document, and insert it at any point of your pipeline. document, and insert it at any point of your pipeline.
### Best practices {#custom-components-best-practices} ### Best practices {id="custom-components-best-practices"}
Extensions can claim their own `._` namespace and exist as standalone packages. Extensions can claim their own `._` namespace and exist as standalone packages.
If you're developing a tool or library and want to make it easy for others to If you're developing a tool or library and want to make it easy for others to
@ -1738,7 +1721,7 @@ function that takes a `Doc`, modifies it and returns it.
to help people find it. If you post it on Twitter, feel free to tag to help people find it. If you post it on Twitter, feel free to tag
[@spacy_io](https://twitter.com/spacy_io) so we can check it out. [@spacy_io](https://twitter.com/spacy_io) so we can check it out.
### Wrapping other models and libraries {#wrapping-models-libraries} ### Wrapping other models and libraries {id="wrapping-models-libraries"}
Let's say you have a custom entity recognizer that takes a list of strings and Let's say you have a custom entity recognizer that takes a list of strings and
returns their [BILUO tags](/usage/linguistic-features#accessing-ner). Given an returns their [BILUO tags](/usage/linguistic-features#accessing-ner). Given an
@ -1760,8 +1743,7 @@ wrapper has to do is compute the entity spans and overwrite the `doc.ents`.
> attributes. By definition, each token can only be part of one entity, so > attributes. By definition, each token can only be part of one entity, so
> overlapping entity spans are not allowed. > overlapping entity spans are not allowed.
```python ```python {highlight="1,8-9"}
### {highlight="1,8-9"}
import your_custom_entity_recognizer import your_custom_entity_recognizer
from spacy.training import biluo_tags_to_spans from spacy.training import biluo_tags_to_spans
from spacy.language import Language from spacy.language import Language
@ -1799,8 +1781,7 @@ label scheme than spaCy's default models.
> it fully replaces the `nlp` object instead of providing a pipeline component, > it fully replaces the `nlp` object instead of providing a pipeline component,
> since it also needs to handle tokenization. > since it also needs to handle tokenization.
```python ```python {highlight="1,11,17-19"}
### {highlight="1,11,17-19"}
import your_custom_model import your_custom_model
from spacy.language import Language from spacy.language import Language
from spacy.symbols import POS, TAG, DEP, HEAD from spacy.symbols import POS, TAG, DEP, HEAD

View File

@ -1,6 +1,6 @@
--- ---
title: Projects title: Projects
new: 3 version: 3
menu: menu:
- ['Intro & Workflow', 'intro'] - ['Intro & Workflow', 'intro']
- ['Directory & Assets', 'directory'] - ['Directory & Assets', 'directory']
@ -9,7 +9,7 @@ menu:
- ['Integrations', 'integrations'] - ['Integrations', 'integrations']
--- ---
## Introduction and workflow {#intro hidden="true"} ## Introduction and workflow {id="intro",hidden="true"}
> #### 🪐 Project templates > #### 🪐 Project templates
> >
@ -27,7 +27,7 @@ and share your results with your team. spaCy projects can be used via the new
[`spacy project`](/api/cli#project) command and we provide templates in our [`spacy project`](/api/cli#project) command and we provide templates in our
[`projects`](https://github.com/explosion/projects) repo. [`projects`](https://github.com/explosion/projects) repo.
![Illustration of project workflow and commands](../images/projects.svg) ![Illustration of project workflow and commands](/images/projects.svg)
<Project id="pipelines/tagger_parser_ud"> <Project id="pipelines/tagger_parser_ud">
@ -43,16 +43,34 @@ and experiments, iterate on demos and prototypes and ship your models into
production. production.
<Grid narrow cols={3}> <Grid narrow cols={3}>
<Integration title="DVC" logo="dvc" url="#dvc">Manage and version your data</Integration> <Integration title="DVC" logo="dvc" url="#dvc">
<Integration title="Prodigy" logo="prodigy" url="#prodigy">Create labelled training data</Integration> Manage and version your data
<Integration title="Streamlit" logo="streamlit" url="#streamlit">Visualize and demo your pipelines</Integration> </Integration>
<Integration title="FastAPI" logo="fastapi" url="#fastapi">Serve your models and host APIs</Integration> <Integration title="Prodigy" logo="prodigy" url="#prodigy">
<Integration title="Ray" logo="ray" url="#ray">Distributed and parallel training</Integration> Create labelled training data
<Integration title="Weights &amp; Biases" logo="wandb" url="#wandb">Track your experiments and results</Integration> </Integration>
<Integration title="Hugging Face Hub" logo="huggingface_hub" url="#huggingface_hub">Upload your pipelines to the Hugging Face Hub</Integration> <Integration title="Streamlit" logo="streamlit" url="#streamlit">
Visualize and demo your pipelines
</Integration>
<Integration title="FastAPI" logo="fastapi" url="#fastapi">
Serve your models and host APIs
</Integration>
<Integration title="Ray" logo="ray" url="#ray">
Distributed and parallel training
</Integration>
<Integration title="Weights &amp; Biases" logo="wandb" url="#wandb">
Track your experiments and results
</Integration>
<Integration
title="Hugging Face Hub"
logo="huggingface_hub"
url="#huggingface_hub"
>
Upload your pipelines to the Hugging Face Hub
</Integration>
</Grid> </Grid>
### 1. Clone a project template {#clone} ### 1. Clone a project template {id="clone"}
> #### Cloning under the hood > #### Cloning under the hood
> >
@ -64,7 +82,7 @@ project template and copies the files to a local directory. You can then run the
project, e.g. to train a pipeline and edit the commands and scripts to build project, e.g. to train a pipeline and edit the commands and scripts to build
fully custom workflows. fully custom workflows.
```cli ```bash
python -m spacy project clone pipelines/tagger_parser_ud python -m spacy project clone pipelines/tagger_parser_ud
``` ```
@ -74,7 +92,7 @@ can specify an optional second argument to define the output directory. The
use the spaCy [`projects`](https://github.com/explosion/projects) repo. You can use the spaCy [`projects`](https://github.com/explosion/projects) repo. You can
also use any private repo you have access to with Git. also use any private repo you have access to with Git.
### 2. Fetch the project assets {#assets} ### 2. Fetch the project assets {id="assets"}
> #### project.yml > #### project.yml
> >
@ -97,7 +115,7 @@ with. Each project template comes with a `project.yml` that defines the assets
to download and where to put them. The [`spacy project assets`](/api/cli#run) to download and where to put them. The [`spacy project assets`](/api/cli#run)
will fetch the project assets for you: will fetch the project assets for you:
```cli ```bash
$ cd some_example_project $ cd some_example_project
$ python -m spacy project assets $ python -m spacy project assets
``` ```
@ -112,7 +130,7 @@ necessarily want to download when you run `spacy project assets`. That's why
assets can be marked as [`extra`](#data-assets-url) - by default, these assets assets can be marked as [`extra`](#data-assets-url) - by default, these assets
are not downloaded. If they should be, run `spacy project assets --extra`. are not downloaded. If they should be, run `spacy project assets --extra`.
### 3. Run a command {#run} ### 3. Run a command {id="run"}
> #### project.yml > #### project.yml
> >
@ -135,7 +153,7 @@ Commands consist of one or more steps and can be run with
[`spacy project run`](/api/cli#project-run). The following will run the command [`spacy project run`](/api/cli#project-run). The following will run the command
`preprocess` defined in the `project.yml`: `preprocess` defined in the `project.yml`:
```cli ```bash
$ python -m spacy project run preprocess $ python -m spacy project run preprocess
``` ```
@ -155,7 +173,7 @@ detected, a corresponding warning is displayed. If you'd like to disable the
dependency check, set `check_requirements: false` in your project's dependency check, set `check_requirements: false` in your project's
`project.yml`. `project.yml`.
### 4. Run a workflow {#run-workfow} ### 4. Run a workflow {id="run-workfow"}
> #### project.yml > #### project.yml
> >
@ -175,7 +193,7 @@ pipeline on the converted data and if that's successful, run
installable Python package. The following command runs the workflow named `all` installable Python package. The following command runs the workflow named `all`
defined in the `project.yml`, and executes the commands it specifies, in order: defined in the `project.yml`, and executes the commands it specifies, in order:
```cli ```bash
$ python -m spacy project run all $ python -m spacy project run all
``` ```
@ -188,7 +206,7 @@ advanced data pipelines and track your changes in Git, check out the
from a workflow defined in your `project.yml` so you can manage your spaCy from a workflow defined in your `project.yml` so you can manage your spaCy
project as a DVC repo. project as a DVC repo.
### 5. Optional: Push to remote storage {#push} ### 5. Optional: Push to remote storage {id="push"}
> ```yaml > ```yaml
> ### project.yml > ### project.yml
@ -204,7 +222,7 @@ a remote storage, using protocols like [S3](https://aws.amazon.com/s3/),
you **export** your pipeline packages, **share** work with your team, or **cache you **export** your pipeline packages, **share** work with your team, or **cache
results** to avoid repeating work. results** to avoid repeating work.
```cli ```bash
$ python -m spacy project push $ python -m spacy project push
``` ```
@ -213,9 +231,9 @@ different storages. To download state from a remote storage, you can use the
[`spacy project pull`](/api/cli#project-pull) command. For more details, see the [`spacy project pull`](/api/cli#project-pull) command. For more details, see the
docs on [remote storage](#remote). docs on [remote storage](#remote).
## Project directory and assets {#directory} ## Project directory and assets {id="directory"}
### project.yml {#project-yml} ### project.yml {id="project-yml"}
The `project.yml` defines the assets a project depends on, like datasets and The `project.yml` defines the assets a project depends on, like datasets and
pretrained weights, as well as a series of commands that can be run separately pretrained weights, as well as a series of commands that can be run separately
@ -277,7 +295,7 @@ pipelines.
| `spacy_version` | Optional spaCy version range like `>=3.0.0,<3.1.0` that the project is compatible with. If it's loaded with an incompatible version, an error is raised when the project is loaded. | | `spacy_version` | Optional spaCy version range like `>=3.0.0,<3.1.0` that the project is compatible with. If it's loaded with an incompatible version, an error is raised when the project is loaded. |
| `check_requirements` <Tag variant="new">3.4.2</Tag> | A flag determining whether to verify that the installed dependencies align with the project's `requirements.txt`. Defaults to `true`. | | `check_requirements` <Tag variant="new">3.4.2</Tag> | A flag determining whether to verify that the installed dependencies align with the project's `requirements.txt`. Defaults to `true`. |
### Data assets {#data-assets} ### Data assets {id="data-assets"}
Assets are any files that your project might need, like training and development Assets are any files that your project might need, like training and development
corpora or pretrained weights for initializing your model. Assets are defined in corpora or pretrained weights for initializing your model. Assets are defined in
@ -288,7 +306,7 @@ Asset URLs can be a number of different **protocols**: HTTP, HTTPS, FTP, SSH,
and even **cloud storage** such as GCS and S3. You can also download assets from and even **cloud storage** such as GCS and S3. You can also download assets from
a **Git repo** instead. a **Git repo** instead.
#### Downloading from a URL or cloud storage {#data-assets-url} #### Downloading from a URL or cloud storage {id="data-assets-url"}
Under the hood, spaCy uses the Under the hood, spaCy uses the
[`smart-open`](https://github.com/RaRe-Technologies/smart_open) library so you [`smart-open`](https://github.com/RaRe-Technologies/smart_open) library so you
@ -318,7 +336,7 @@ dependencies to use certain protocols.
| `checksum` | Optional checksum of the file. If provided, it will be used to verify that the file matches and downloads will be skipped if a local file with the same checksum already exists. | | `checksum` | Optional checksum of the file. If provided, it will be used to verify that the file matches and downloads will be skipped if a local file with the same checksum already exists. |
| `description` | Optional asset description, used in [auto-generated docs](#custom-docs). | | `description` | Optional asset description, used in [auto-generated docs](#custom-docs). |
#### Downloading from a Git repo {#data-assets-git} #### Downloading from a Git repo {id="data-assets-git"}
If a `git` block is provided, the asset is downloaded from the given Git If a `git` block is provided, the asset is downloaded from the given Git
repository. You can download from any repo that you have access to. Under the repository. You can download from any repo that you have access to. Under the
@ -345,7 +363,7 @@ files you need and not the whole repo.
| `checksum` | Optional checksum of the file. If provided, it will be used to verify that the file matches and downloads will be skipped if a local file with the same checksum already exists. | | `checksum` | Optional checksum of the file. If provided, it will be used to verify that the file matches and downloads will be skipped if a local file with the same checksum already exists. |
| `description` | Optional asset description, used in [auto-generated docs](#custom-docs). | | `description` | Optional asset description, used in [auto-generated docs](#custom-docs). |
#### Working with private assets {#data-asets-private} #### Working with private assets {id="data-asets-private"}
> #### project.yml > #### project.yml
> >
@ -365,7 +383,7 @@ directory themselves. The [`project assets`](/api/cli#project-assets) command
will alert you about missing files and mismatched checksums, so you can ensure will alert you about missing files and mismatched checksums, so you can ensure
that others are running your project with the same data. that others are running your project with the same data.
### Dependencies and outputs {#deps-outputs} ### Dependencies and outputs {id="deps-outputs"}
Each command defined in the `project.yml` can optionally define a list of Each command defined in the `project.yml` can optionally define a list of
dependencies and outputs. These are the files the command requires and creates. dependencies and outputs. These are the files the command requires and creates.
@ -374,9 +392,8 @@ For example, a command for training a pipeline may depend on a
it will export a directory `model-best`, which you can then re-use in other it will export a directory `model-best`, which you can then re-use in other
commands. commands.
<!-- prettier-ignore --> {/* prettier-ignore */}
```yaml ```yaml {title="project.yml"}
### project.yml
commands: commands:
- name: train - name: train
help: 'Train a spaCy pipeline using the specified corpus and config' help: 'Train a spaCy pipeline using the specified corpus and config'
@ -415,7 +432,7 @@ If you're planning on integrating your spaCy project with DVC, you can also use
`outputs_no_cache` instead of `outputs` to define outputs that won't be cached `outputs_no_cache` instead of `outputs` to define outputs that won't be cached
or tracked. or tracked.
### Files and directory structure {#project-files} ### Files and directory structure {id="project-files"}
The `project.yml` can define a list of `directories` that should be created The `project.yml` can define a list of `directories` that should be created
within a project for instance, `assets`, `training`, `corpus` and so on. spaCy within a project for instance, `assets`, `training`, `corpus` and so on. spaCy
@ -427,13 +444,12 @@ directory:
> #### project.yml > #### project.yml
> >
> <!-- prettier-ignore --> > {/* prettier-ignore */}
> ```yaml > ```yaml
> directories: ['assets', 'configs', 'corpus', 'metas', 'metrics', 'notebooks', 'packages', 'scripts', 'training'] > directories: ['assets', 'configs', 'corpus', 'metas', 'metrics', 'notebooks', 'packages', 'scripts', 'training']
> ``` > ```
```yaml ```yaml {title="Example project directory"}
### Example project directory
├── project.yml # the project settings ├── project.yml # the project settings
├── project.lock # lockfile that tracks inputs/outputs ├── project.lock # lockfile that tracks inputs/outputs
├── assets/ # downloaded data assets ├── assets/ # downloaded data assets
@ -455,7 +471,7 @@ the only file that's required for a project is the `project.yml`.
--- ---
## Custom scripts and projects {#custom} ## Custom scripts and projects {id="custom"}
The `project.yml` lets you define any custom commands and run them as part of The `project.yml` lets you define any custom commands and run them as part of
your training, evaluation or deployment workflows. The `script` section defines your training, evaluation or deployment workflows. The `script` section defines
@ -467,8 +483,7 @@ calls into [`pytest`](https://docs.pytest.org/en/latest/), runs your tests and
uses [`pytest-html`](https://github.com/pytest-dev/pytest-html) to export a test uses [`pytest-html`](https://github.com/pytest-dev/pytest-html) to export a test
report: report:
```yaml ```yaml {title="project.yml"}
### project.yml
commands: commands:
- name: test - name: test
help: 'Test the trained pipeline' help: 'Test the trained pipeline'
@ -488,7 +503,7 @@ Setting `no_skip: true` means that the command will always run, even if the
dependencies (the trained pipeline) haven't changed. This makes sense here, dependencies (the trained pipeline) haven't changed. This makes sense here,
because you typically don't want to skip your tests. because you typically don't want to skip your tests.
### Writing custom scripts {#custom-scripts} ### Writing custom scripts {id="custom-scripts"}
Your project commands can include any custom scripts essentially, anything you Your project commands can include any custom scripts essentially, anything you
can run from the command line. Here's an example of a custom script that uses can run from the command line. Here's an example of a custom script that uses
@ -504,8 +519,7 @@ that you can define via your `project.yml`:
> types. For instance, `batch_size: int` means that the value provided via the > types. For instance, `batch_size: int` means that the value provided via the
> command line is converted to an integer. > command line is converted to an integer.
```python ```python {title="scripts/custom_evaluation.py"}
### scripts/custom_evaluation.py
import typer import typer
def custom_evaluation(batch_size: int = 128, model_path: str, data_path: str): def custom_evaluation(batch_size: int = 128, model_path: str, data_path: str):
@ -531,9 +545,8 @@ override settings on the command line for example using `--vars.batch_size`.
> everything with the same Python (not some other Python installed on your > everything with the same Python (not some other Python installed on your
> system). It also normalizes references to `python3`, `pip3` and `pip`. > system). It also normalizes references to `python3`, `pip3` and `pip`.
<!-- prettier-ignore --> {/* prettier-ignore */}
```yaml ```yaml {title="project.yml"}
### project.yml
vars: vars:
batch_size: 128 batch_size: 128
@ -557,8 +570,7 @@ settings on the command line and passing through system-level settings.
> BATCH_SIZE=128 python -m spacy project run evaluate > BATCH_SIZE=128 python -m spacy project run evaluate
> ``` > ```
```yaml ```yaml {title="project.yml"}
### project.yml
env: env:
batch_size: BATCH_SIZE batch_size: BATCH_SIZE
gpu_id: GPU_ID gpu_id: GPU_ID
@ -569,14 +581,14 @@ commands:
- 'python scripts/custom_evaluation.py ${env.batch_size}' - 'python scripts/custom_evaluation.py ${env.batch_size}'
``` ```
### Documenting your project {#custom-docs} ### Documenting your project {id="custom-docs"}
> #### Readme Example > #### Readme Example
> >
> For more examples, see the [`projects`](https://github.com/explosion/projects) > For more examples, see the [`projects`](https://github.com/explosion/projects)
> repo. > repo.
> >
> ![Screenshot of auto-generated Markdown Readme](../images/project_document.jpg) > ![Screenshot of auto-generated Markdown Readme](/images/project_document.jpg)
When your custom project is ready and you want to share it with others, you can When your custom project is ready and you want to share it with others, you can
use the [`spacy project document`](/api/cli#project-document) command to use the [`spacy project document`](/api/cli#project-document) command to
@ -586,7 +598,7 @@ in the project and include details on how to run the project, as well as links
to the relevant spaCy documentation to make it easy for others to get started to the relevant spaCy documentation to make it easy for others to get started
using your project. using your project.
```cli ```bash
$ python -m spacy project document --output README.md $ python -m spacy project document --output README.md
``` ```
@ -600,18 +612,18 @@ up to date.
Note that the contents of an existing file will be **replaced** if no existing Note that the contents of an existing file will be **replaced** if no existing
auto-generated docs are found. If you want spaCy to ignore a file and not update auto-generated docs are found. If you want spaCy to ignore a file and not update
it, you can add the comment marker `<!-- SPACY PROJECT: IGNORE -->` anywhere in it, you can add the comment marker `{/* SPACY PROJECT: IGNORE */}` anywhere in
your markup. your markup.
</Infobox> </Infobox>
### Cloning from your own repo {#custom-repo} ### Cloning from your own repo {id="custom-repo"}
The [`spacy project clone`](/api/cli#project-clone) command lets you customize The [`spacy project clone`](/api/cli#project-clone) command lets you customize
the repo to clone from using the `--repo` option. It calls into `git`, so you'll the repo to clone from using the `--repo` option. It calls into `git`, so you'll
be able to clone from any repo that you have access to, including private repos. be able to clone from any repo that you have access to, including private repos.
```cli ```bash
python -m spacy project clone your_project --repo https://github.com/you/repo python -m spacy project clone your_project --repo https://github.com/you/repo
``` ```
@ -632,7 +644,7 @@ projects.
</Infobox> </Infobox>
## Remote Storage {#remote} ## Remote Storage {id="remote"}
You can persist your project outputs to a remote storage using the You can persist your project outputs to a remote storage using the
[`project push`](/api/cli#project-push) command. This can help you **export** [`project push`](/api/cli#project-push) command. This can help you **export**
@ -653,12 +665,11 @@ protocols.
> #### Example > #### Example
> >
> ```cli > ```bash
> $ python -m spacy project pull local > $ python -m spacy project pull local
> ``` > ```
```yaml ```yaml {title="project.yml"}
### project.yml
remotes: remotes:
default: 's3://my-spacy-bucket' default: 's3://my-spacy-bucket'
local: '/mnt/scratch/cache' local: '/mnt/scratch/cache'
@ -673,9 +684,9 @@ according to a hash of the command string and the command's dependencies.
Finally, within those directories are files, named according to an MD5 hash of Finally, within those directories are files, named according to an MD5 hash of
their contents. their contents.
<!-- TODO: update with actual real example? --> {/* TODO: update with actual real example? */}
<!-- prettier-ignore --> {/* prettier-ignore */}
```yaml ```yaml
└── urlencoded_file_path # Path of original file └── urlencoded_file_path # Path of original file
├── some_command_hash # Hash of command you ran ├── some_command_hash # Hash of command you ran
@ -689,8 +700,7 @@ their contents.
For instance, let's say you had the following command in your `project.yml`: For instance, let's say you had the following command in your `project.yml`:
```yaml ```yaml {title="project.yml"}
### project.yml
- name: train - name: train
help: 'Train a spaCy pipeline using the specified corpus and config' help: 'Train a spaCy pipeline using the specified corpus and config'
script: script:
@ -719,7 +729,7 @@ execution context of your output. It would then compute an MD5 hash of the
`training/model-best` directory, and use those three pieces of information to `training/model-best` directory, and use those three pieces of information to
construct the storage URL. construct the storage URL.
```cli ```bash
$ python -m spacy project run train $ python -m spacy project run train
$ python -m spacy project push $ python -m spacy project push
``` ```
@ -740,9 +750,12 @@ state, and you don't have to come up with names or version numbers. However,
it's up to you to manage the size of your remote storage, and to remove files it's up to you to manage the size of your remote storage, and to remove files
that are no longer relevant to you. that are no longer relevant to you.
## Integrations {#integrations} ## Integrations {id="integrations"}
### Data Version Control (DVC) {#dvc} <IntegrationLogo name="dvc" title="DVC" width={70} height="auto" align="right" /> {<H3 id="dvc"> Data Version Control (DVC)
<IntegrationLogo name="dvc" title="DVC" width={70} height="auto" align="right"/>
</H3>}
Data assets like training corpora or pretrained weights are at the core of any Data assets like training corpora or pretrained weights are at the core of any
NLP project, but they're often difficult to manage: you can't just check them NLP project, but they're often difficult to manage: you can't just check them
@ -787,7 +800,7 @@ can then manage your spaCy project like any other DVC project, run
and [`dvc repro`](https://dvc.org/doc/command-reference/repro) to reproduce the and [`dvc repro`](https://dvc.org/doc/command-reference/repro) to reproduce the
workflow or individual commands. workflow or individual commands.
```cli ```bash
$ python -m spacy project dvc [project_dir] [workflow_name] $ python -m spacy project dvc [project_dir] [workflow_name]
``` ```
@ -800,13 +813,14 @@ workflows, but only one can be tracked by DVC.
</Infobox> </Infobox>
<!-- TODO: <Project id="integrations/dvc"> {/* { TODO: <Project id="integrations/dvc"></Project>} */}
</Project> -->
--- ---
### Prodigy {#prodigy} <IntegrationLogo name="prodigy" width={100} height="auto" align="right" /> {<H3 id="prodigy">Prodigy
<IntegrationLogo name="prodigy" width={100} height="auto" align="right" />
</H3>}
[Prodigy](https://prodi.gy) is a modern annotation tool for creating training [Prodigy](https://prodi.gy) is a modern annotation tool for creating training
data for machine learning models, developed by us. It integrates with spaCy data for machine learning models, developed by us. It integrates with spaCy
@ -831,13 +845,12 @@ collected with Prodigy and training a spaCy pipeline:
> #### Example usage > #### Example usage
> >
> ```cli > ```bash
> $ python -m spacy project run all > $ python -m spacy project run all
> ``` > ```
<!-- prettier-ignore --> {/* prettier-ignore */}
```yaml ```yaml {title="project.yml"}
### project.yml
vars: vars:
prodigy: prodigy:
train_dataset: "fashion_brands_training" train_dataset: "fashion_brands_training"
@ -869,7 +882,11 @@ commands:
> #### Example train curve output > #### Example train curve output
> >
> [![Screenshot of train curve terminal output](../images/prodigy_train_curve.jpg)](https://prodi.gy/docs/recipes#train-curve) > <Image
> src="/images/prodigy_train_curve.jpg"
> href="https://prodi.gy/docs/recipes#train-curve"
> alt="Screenshot of train curve terminal output"
> />
The [`train-curve`](https://prodi.gy/docs/recipes#train-curve) recipe is another The [`train-curve`](https://prodi.gy/docs/recipes#train-curve) recipe is another
cool workflow you can include in your project. It will run the training with cool workflow you can include in your project. It will run the training with
@ -877,9 +894,8 @@ different portions of the data, e.g. 25%, 50%, 75% and 100%. As a rule of thumb,
if accuracy increases in the last segment, this could indicate that collecting if accuracy increases in the last segment, this could indicate that collecting
more annotations of the same type might improve the model further. more annotations of the same type might improve the model further.
<!-- prettier-ignore --> {/* prettier-ignore */}
```yaml ```yaml {title="project.yml (excerpt)"}
### project.yml (excerpt)
- name: "train_curve" - name: "train_curve"
help: "Train the model with Prodigy by using different portions of training examples to evaluate if more annotations can potentially improve the performance" help: "Train the model with Prodigy by using different portions of training examples to evaluate if more annotations can potentially improve the performance"
script: script:
@ -908,7 +924,10 @@ improve performance.
--- ---
### Streamlit {#streamlit} <IntegrationLogo name="streamlit" width={150} height="auto" align="right" /> {<H3 id="streamlit">Streamlit
<IntegrationLogo name="streamlit" width={150} height="auto" align="right" />
</H3>}
[Streamlit](https://streamlit.io) is a Python framework for building interactive [Streamlit](https://streamlit.io) is a Python framework for building interactive
data apps. The [`spacy-streamlit`](https://github.com/explosion/spacy-streamlit) data apps. The [`spacy-streamlit`](https://github.com/explosion/spacy-streamlit)
@ -916,7 +935,7 @@ package helps you integrate spaCy visualizations into your Streamlit apps and
quickly spin up demos to explore your pipelines interactively. It includes a quickly spin up demos to explore your pipelines interactively. It includes a
full embedded visualizer, as well as individual components. full embedded visualizer, as well as individual components.
<!-- TODO: update once version is stable --> {/* TODO: update once version is stable */}
> #### Installation > #### Installation
> >
@ -924,7 +943,7 @@ full embedded visualizer, as well as individual components.
> $ pip install spacy-streamlit --pre > $ pip install spacy-streamlit --pre
> ``` > ```
![](../images/spacy-streamlit.png) ![](/images/spacy-streamlit.png)
Using [`spacy-streamlit`](https://github.com/explosion/spacy-streamlit), your Using [`spacy-streamlit`](https://github.com/explosion/spacy-streamlit), your
projects can easily define their own scripts that spin up an interactive projects can easily define their own scripts that spin up an interactive
@ -941,13 +960,12 @@ and explore your own custom trained pipelines.
> #### Example usage > #### Example usage
> >
> ```cli > ```bash
> $ python -m spacy project run visualize > $ python -m spacy project run visualize
> ``` > ```
<!-- prettier-ignore --> {/* prettier-ignore */}
```yaml ```yaml {title="project.yml"}
### project.yml
commands: commands:
- name: visualize - name: visualize
help: "Visualize the pipeline's output interactively using Streamlit" help: "Visualize the pipeline's output interactively using Streamlit"
@ -967,7 +985,10 @@ https://github.com/explosion/projects/blob/v3/integrations/streamlit/scripts/vis
--- ---
### FastAPI {#fastapi} <IntegrationLogo name="fastapi" width={100} height="auto" align="right" /> {<H3 id="fastapi">FastAPI
<IntegrationLogo name="fastapi" width={100} height="auto" align="right" />
</H3>}
[FastAPI](https://fastapi.tiangolo.com/) is a modern high-performance framework [FastAPI](https://fastapi.tiangolo.com/) is a modern high-performance framework
for building REST APIs with Python, based on Python for building REST APIs with Python, based on Python
@ -986,13 +1007,12 @@ query your API from Python and JavaScript (Vanilla JS and React).
> #### Example usage > #### Example usage
> >
> ```cli > ```bash
> $ python -m spacy project run serve > $ python -m spacy project run serve
> ``` > ```
<!-- prettier-ignore --> {/* prettier-ignore */}
```yaml ```yaml {title="project.yml"}
### project.yml
- name: "serve" - name: "serve"
help: "Serve the models via a FastAPI REST API using the given host and port" help: "Serve the models via a FastAPI REST API using the given host and port"
script: script:
@ -1014,7 +1034,10 @@ https://github.com/explosion/projects/blob/v3/integrations/fastapi/scripts/main.
--- ---
### Weights & Biases {#wandb} <IntegrationLogo name="wandb" width={175} height="auto" align="right" /> {<H3 id="wandb">Weights & Biases
<IntegrationLogo name="wandb" width={175} height="auto" align="right" />
</H3>}
[Weights & Biases](https://www.wandb.com/) is a popular platform for experiment [Weights & Biases](https://www.wandb.com/) is a popular platform for experiment
tracking. spaCy integrates with it out-of-the-box via the tracking. spaCy integrates with it out-of-the-box via the
@ -1036,9 +1059,9 @@ and you'll be able to see the impact it has on your results.
> model_log_interval = 1000 > model_log_interval = 1000
> ``` > ```
![Screenshot: Visualized training results](../images/wandb1.jpg) ![Screenshot: Visualized training results](/images/wandb1.jpg)
![Screenshot: Parameter importance using config values](../images/wandb2.jpg 'Parameter importance using config values') ![Screenshot: Parameter importance using config values](/images/wandb2.jpg 'Parameter importance using config values')
<Project id="integrations/wandb"> <Project id="integrations/wandb">
@ -1052,7 +1075,10 @@ logging the results.
--- ---
### Hugging Face Hub {#huggingface_hub} <IntegrationLogo name="huggingface_hub" width={175} height="auto" align="right" /> {<H3 id="huggingface_hub">Hugging Face Hub
<IntegrationLogo name="huggingface_hub" width={175} height="auto" align="right" />
</H3>}
The [Hugging Face Hub](https://huggingface.co/) lets you upload models and share The [Hugging Face Hub](https://huggingface.co/) lets you upload models and share
them with others. It hosts models as Git-based repositories which are storage them with others. It hosts models as Git-based repositories which are storage
@ -1065,7 +1091,7 @@ it's installed.
> #### Installation > #### Installation
> >
> ```cli > ```bash
> $ pip install spacy-huggingface-hub > $ pip install spacy-huggingface-hub
> # Check that the CLI is registered > # Check that the CLI is registered
> $ python -m spacy huggingface-hub --help > $ python -m spacy huggingface-hub --help
@ -1078,7 +1104,7 @@ package, including the auto-generated pretty `README.md` and the model details
available in the `meta.json`. For examples, check out the available in the `meta.json`. For examples, check out the
[spaCy pipelines](https://huggingface.co/spacy) we've uploaded. [spaCy pipelines](https://huggingface.co/spacy) we've uploaded.
```cli ```bash
$ huggingface-cli login $ huggingface-cli login
$ python -m spacy package ./en_ner_fashion ./output --build wheel $ python -m spacy package ./en_ner_fashion ./output --build wheel
$ cd ./output/en_ner_fashion-0.0.0/dist $ cd ./output/en_ner_fashion-0.0.0/dist
@ -1089,16 +1115,15 @@ After uploading, you will see the live URL of your pipeline packages, as well as
the direct URL to the model wheel you can install via `pip install`. You'll also the direct URL to the model wheel you can install via `pip install`. You'll also
be able to test your pipeline interactively from your browser: be able to test your pipeline interactively from your browser:
![Screenshot: interactive NER visualizer](../images/huggingface_hub.jpg) ![Screenshot: interactive NER visualizer](/images/huggingface_hub.jpg)
In your `project.yml`, you can add a command that uploads your trained and In your `project.yml`, you can add a command that uploads your trained and
packaged pipeline to the hub. You can either run this as a manual step, or packaged pipeline to the hub. You can either run this as a manual step, or
automatically as part of a workflow. Make sure to set `--build wheel` when automatically as part of a workflow. Make sure to set `--build wheel` when
running `spacy package` to build a wheel file for your pipeline package. running `spacy package` to build a wheel file for your pipeline package.
<!-- prettier-ignore --> {/* prettier-ignore */}
```yaml ```yaml {title="project.yml"}
### project.yml
- name: "push_to_hub" - name: "push_to_hub"
help: "Upload the trained model to the Hugging Face Hub" help: "Upload the trained model to the Hugging Face Hub"
script: script:

View File

@ -60,7 +60,7 @@ another token that's at least 10 characters long.
</Accordion> </Accordion>
## Token-based matching {#matcher} ## Token-based matching {id="matcher"}
spaCy features a rule-matching engine, the [`Matcher`](/api/matcher), that spaCy features a rule-matching engine, the [`Matcher`](/api/matcher), that
operates over tokens, similar to regular expressions. The rules can refer to operates over tokens, similar to regular expressions. The rules can refer to
@ -72,7 +72,7 @@ To match large terminology lists, you can use the
[`PhraseMatcher`](/api/phrasematcher), which accepts `Doc` objects as match [`PhraseMatcher`](/api/phrasematcher), which accepts `Doc` objects as match
patterns. patterns.
### Adding patterns {#adding-patterns} ### Adding patterns {id="adding-patterns"}
Let's say we want to enable spaCy to find a combination of three tokens: Let's say we want to enable spaCy to find a combination of three tokens:
@ -102,8 +102,7 @@ First, we initialize the `Matcher` with a vocab. The matcher must always share
the same vocab with the documents it will operate on. We can now call the same vocab with the documents it will operate on. We can now call
[`matcher.add()`](/api/matcher#add) with an ID and a list of patterns. [`matcher.add()`](/api/matcher#add) with an ID and a list of patterns.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.matcher import Matcher from spacy.matcher import Matcher
@ -153,7 +152,7 @@ merge _some_ patterns into one token, while adding entity labels for other
pattern types. You shouldn't have to create different matchers for each of those pattern types. You shouldn't have to create different matchers for each of those
processes. processes.
#### Available token attributes {#adding-patterns-attributes} #### Available token attributes {id="adding-patterns-attributes"}
The available token pattern keys correspond to a number of The available token pattern keys correspond to a number of
[`Token` attributes](/api/token#attributes). The supported attributes for [`Token` attributes](/api/token#attributes). The supported attributes for
@ -208,7 +207,11 @@ you need to describe fields like this.
<Infobox title="Tip: Try the interactive matcher explorer"> <Infobox title="Tip: Try the interactive matcher explorer">
[![Matcher demo](../images/matcher-demo.jpg)](https://explosion.ai/demos/matcher) <Image
src="/images/matcher-demo.jpg"
href="https://explosion.ai/demos/matcher"
alt="Matcher demo"
/>
The [Matcher Explorer](https://explosion.ai/demos/matcher) lets you test the The [Matcher Explorer](https://explosion.ai/demos/matcher) lets you test the
rule-based `Matcher` by creating token patterns interactively and running them rule-based `Matcher` by creating token patterns interactively and running them
@ -218,7 +221,7 @@ spaCy processes your text and why your pattern matches, or why it doesn't.
</Infobox> </Infobox>
#### Extended pattern syntax and attributes {#adding-patterns-attributes-extended new="2.1"} #### Extended pattern syntax and attributes {id="adding-patterns-attributes-extended",version="2.1"}
Instead of mapping to a single value, token patterns can also map to a Instead of mapping to a single value, token patterns can also map to a
**dictionary of properties**. For example, to specify that the value of a lemma **dictionary of properties**. For example, to specify that the value of a lemma
@ -251,7 +254,7 @@ following rich comparison attributes are available:
| `INTERSECTS` | Attribute value (for `MORPH` or custom list attributes) has a non-empty intersection with a list. ~~Any~~ | | `INTERSECTS` | Attribute value (for `MORPH` or custom list attributes) has a non-empty intersection with a list. ~~Any~~ |
| `==`, `>=`, `<=`, `>`, `<` | Attribute value is equal, greater or equal, smaller or equal, greater or smaller. ~~Union[int, float]~~ | | `==`, `>=`, `<=`, `>`, `<` | Attribute value is equal, greater or equal, smaller or equal, greater or smaller. ~~Union[int, float]~~ |
#### Regular expressions {#regex new="2.1"} #### Regular expressions {id="regex",version="2.1"}
In some cases, only matching tokens and token attributes isn't enough for In some cases, only matching tokens and token attributes isn't enough for
example, you might want to match different spellings of a word, without having example, you might want to match different spellings of a word, without having
@ -287,7 +290,7 @@ token. If you need to match on the whole text instead, see the details on
</Infobox> </Infobox>
##### Matching regular expressions on the full text {#regex-text} ##### Matching regular expressions on the full text {id="regex-text"}
If your expressions apply to multiple tokens, a simple solution is to match on If your expressions apply to multiple tokens, a simple solution is to match on
the `doc.text` with `re.finditer` and use the the `doc.text` with `re.finditer` and use the
@ -301,8 +304,7 @@ more valid tokens, `Doc.char_span` returns `None`.
> `"USA"` is a single token and `Span` objects are **sequences of tokens**. So > `"USA"` is a single token and `Span` objects are **sequences of tokens**. So
> `"US"` cannot be its own span, because it does not end on a token boundary. > `"US"` cannot be its own span, because it does not end on a token boundary.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
import re import re
@ -349,8 +351,7 @@ the (white)space tokens are split on. That hopefully shouldn't happen, though,
because it'd mean your regex is producing matches with leading or trailing because it'd mean your regex is producing matches with leading or trailing
whitespace. whitespace.
```python ```python {highlight="5-8"}
### {highlight="5-8"}
span = doc.char_span(start, end) span = doc.char_span(start, end)
if span is not None: if span is not None:
print("Found match:", span.text) print("Found match:", span.text)
@ -364,7 +365,7 @@ else:
</Accordion> </Accordion>
#### Fuzzy matching {#fuzzy new="3.5"} #### Fuzzy matching {id="fuzzy", version="3.5"}
Fuzzy matching allows you to match tokens with alternate spellings, typos, etc. Fuzzy matching allows you to match tokens with alternate spellings, typos, etc.
without specifying every possible variant. without specifying every possible variant.
@ -393,7 +394,7 @@ pattern = [{"_": {"country": {"FUZZY": "Kyrgyzstan"}}}]
pattern = [{"_": {"country": {"FUZZY3": "Kyrgyzstan"}}}] pattern = [{"_": {"country": {"FUZZY3": "Kyrgyzstan"}}}]
``` ```
#### Regex and fuzzy matching with lists {#regex-fuzzy-lists new="3.5"} #### Regex and fuzzy matching with lists {id="regex-fuzzy-lists", version="3.5"}
Starting in spaCy v3.5, both `REGEX` and `FUZZY` can be combined with the Starting in spaCy v3.5, both `REGEX` and `FUZZY` can be combined with the
attributes `IN` and `NOT_IN`: attributes `IN` and `NOT_IN`:
@ -406,7 +407,7 @@ pattern = [{"TEXT": {"REGEX": {"NOT_IN": ["^awe(some)?$", "^wonder(ful)?"]}}}]
--- ---
#### Operators and quantifiers {#quantifiers} #### Operators and quantifiers {id="quantifiers"}
The matcher also lets you use quantifiers, specified as the `'OP'` key. The matcher also lets you use quantifiers, specified as the `'OP'` key.
Quantifiers let you define sequences of tokens to be matched, e.g. one or more Quantifiers let you define sequences of tokens to be matched, e.g. one or more
@ -442,28 +443,28 @@ This quirk in the semantics is corrected in spaCy v2.1.0.
</Infobox> </Infobox>
#### Using wildcard token patterns {#adding-patterns-wildcard new="2"} #### Using wildcard token patterns {id="adding-patterns-wildcard",version="2"}
While the token attributes offer many options to write highly specific patterns, While the token attributes offer many options to write highly specific patterns,
you can also use an empty dictionary, `{}` as a wildcard representing **any you can also use an empty dictionary, `{}` as a wildcard representing **any
token**. This is useful if you know the context of what you're trying to match, token**. This is useful if you know the context of what you're trying to match,
but very little about the specific token and its characters. For example, let's but very little about the specific token and its characters. For example, let's
say you're trying to extract people's user names from your data. All you know is say you're trying to extract people's user names from your data. All you know is
that they are listed as "User name: {username}". The name itself may contain any that they are listed as "User name: \{username\}". The name itself may contain
character, but no whitespace so you'll know it will be handled as one token. any character, but no whitespace so you'll know it will be handled as one
token.
```python ```python
[{"ORTH": "User"}, {"ORTH": "name"}, {"ORTH": ":"}, {}] [{"ORTH": "User"}, {"ORTH": "name"}, {"ORTH": ":"}, {}]
``` ```
#### Validating and debugging patterns {#pattern-validation new="2.1"} #### Validating and debugging patterns {id="pattern-validation",version="2.1"}
The `Matcher` can validate patterns against a JSON schema with the option The `Matcher` can validate patterns against a JSON schema with the option
`validate=True`. This is useful for debugging patterns during development, in `validate=True`. This is useful for debugging patterns during development, in
particular for catching unsupported attributes. particular for catching unsupported attributes.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.matcher import Matcher from spacy.matcher import Matcher
@ -479,7 +480,7 @@ matcher.add("HelloWorld", [pattern])
``` ```
### Adding on_match rules {#on_match} ### Adding on_match rules {id="on_match"}
To move on to a more realistic example, let's say you're working with a large To move on to a more realistic example, let's say you're working with a large
corpus of blog articles, and you want to match all mentions of "Google I/O" corpus of blog articles, and you want to match all mentions of "Google I/O"
@ -487,8 +488,7 @@ corpus of blog articles, and you want to match all mentions of "Google I/O"
match on the uppercase versions, avoiding matches with phrases such as "Google match on the uppercase versions, avoiding matches with phrases such as "Google
i/o". i/o".
```python ```python {executable="true"}
### {executable="true"}
from spacy.lang.en import English from spacy.lang.en import English
from spacy.matcher import Matcher from spacy.matcher import Matcher
from spacy.tokens import Span from spacy.tokens import Span
@ -551,7 +551,7 @@ you prefer.
| `i` | Index of the current match (`matches[i`]). ~~int~~ | | `i` | Index of the current match (`matches[i`]). ~~int~~ |
| `matches` | A list of `(match_id, start, end)` tuples, describing the matches. A match tuple describes a span `doc[start:end`]. ~~List[Tuple[int, int int]]~~ | | `matches` | A list of `(match_id, start, end)` tuples, describing the matches. A match tuple describes a span `doc[start:end`]. ~~List[Tuple[int, int int]]~~ |
### Creating spans from matches {#matcher-spans} ### Creating spans from matches {id="matcher-spans"}
Creating [`Span`](/api/span) objects from the returned matches is a very common Creating [`Span`](/api/span) objects from the returned matches is a very common
use case. spaCy makes this easy by giving you access to the `start` and `end` use case. spaCy makes this easy by giving you access to the `start` and `end`
@ -560,8 +560,7 @@ label. As of spaCy v3.0, you can also set `as_spans=True` when calling the
matcher on a `Doc`, which will return a list of [`Span`](/api/span) objects matcher on a `Doc`, which will return a list of [`Span`](/api/span) objects
using the `match_id` as the span label. using the `match_id` as the span label.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.matcher import Matcher from spacy.matcher import Matcher
from spacy.tokens import Span from spacy.tokens import Span
@ -584,7 +583,7 @@ for span in matches:
print(span.text, span.label_) print(span.text, span.label_)
``` ```
### Using custom pipeline components {#matcher-pipeline} ### Using custom pipeline components {id="matcher-pipeline"}
Let's say your data also contains some annoying pre-processing artifacts, like Let's say your data also contains some annoying pre-processing artifacts, like
leftover HTML line breaks (e.g. `<br>` or `<BR/>`). To make your text easier to leftover HTML line breaks (e.g. `<br>` or `<BR/>`). To make your text easier to
@ -595,8 +594,7 @@ process the text. You can achieve this by adding a
that's called on each `Doc` object, merges the leftover HTML spans and sets an that's called on each `Doc` object, merges the leftover HTML spans and sets an
attribute `bad_html` on the token. attribute `bad_html` on the token.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.language import Language from spacy.language import Language
from spacy.matcher import Matcher from spacy.matcher import Matcher
@ -662,7 +660,7 @@ and **extension attributes**, see the
</Infobox> </Infobox>
### Example: Using linguistic annotations {#example1} ### Example: Using linguistic annotations {id="example1"}
Let's say you're analyzing user comments and you want to find out what people Let's say you're analyzing user comments and you want to find out what people
are saying about Facebook. You want to start off by finding adjectives following are saying about Facebook. You want to start off by finding adjectives following
@ -691,8 +689,7 @@ calculate the start and end of the matched span within the sentence. Using
displaCy in ["manual" mode](/usage/visualizers#manual-usage) lets you pass in a displaCy in ["manual" mode](/usage/visualizers#manual-usage) lets you pass in a
list of dictionaries containing the text and entities to render. list of dictionaries containing the text and entities to render.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy import displacy from spacy import displacy
from spacy.matcher import Matcher from spacy.matcher import Matcher
@ -728,7 +725,7 @@ matches = matcher(doc)
displacy.render(matched_sents, style="ent", manual=True) displacy.render(matched_sents, style="ent", manual=True)
``` ```
### Example: Phone numbers {#example2} ### Example: Phone numbers {id="example2"}
Phone numbers can have many different formats and matching them is often tricky. Phone numbers can have many different formats and matching them is often tricky.
During tokenization, spaCy will leave sequences of numbers intact and only split During tokenization, spaCy will leave sequences of numbers intact and only split
@ -763,8 +760,7 @@ set of rules like this is often better than training a model. It'll produce more
predictable results, is much easier to modify and extend, and doesn't require predictable results, is much easier to modify and extend, and doesn't require
any training data only a set of test cases. any training data only a set of test cases.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.matcher import Matcher from spacy.matcher import Matcher
@ -782,7 +778,7 @@ for match_id, start, end in matches:
print(span.text) print(span.text)
``` ```
### Example: Hashtags and emoji on social media {#example3} ### Example: Hashtags and emoji on social media {id="example3"}
Social media posts, especially tweets, can be difficult to work with. They're Social media posts, especially tweets, can be difficult to work with. They're
very short and often contain various emoji and hashtags. By only looking at the very short and often contain various emoji and hashtags. By only looking at the
@ -812,8 +808,7 @@ that you can create a pattern for one or more emoji tokens. Valid hashtags
usually consist of a `#`, plus a sequence of ASCII characters with no usually consist of a `#`, plus a sequence of ASCII characters with no
whitespace, making them easy to match as well. whitespace, making them easy to match as well.
```python ```python {executable="true"}
### {executable="true"}
from spacy.lang.en import English from spacy.lang.en import English
from spacy.matcher import Matcher from spacy.matcher import Matcher
@ -883,8 +878,7 @@ To label the hashtags, we can use a
[custom attribute](/usage/processing-pipelines#custom-components-attributes) set [custom attribute](/usage/processing-pipelines#custom-components-attributes) set
on the respective token: on the respective token:
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.matcher import Matcher from spacy.matcher import Matcher
from spacy.tokens import Token from spacy.tokens import Token
@ -914,17 +908,16 @@ for token in doc:
print(token.text, token._.is_hashtag) print(token.text, token._.is_hashtag)
``` ```
## Efficient phrase matching {#phrasematcher} ## Efficient phrase matching {id="phrasematcher"}
If you need to match large terminology lists, you can also use the If you need to match large terminology lists, you can also use the
[`PhraseMatcher`](/api/phrasematcher) and create [`Doc`](/api/doc) objects [`PhraseMatcher`](/api/phrasematcher) and create [`Doc`](/api/doc) objects
instead of token patterns, which is much more efficient overall. The `Doc` instead of token patterns, which is much more efficient overall. The `Doc`
patterns can contain single or multiple tokens. patterns can contain single or multiple tokens.
### Adding phrase patterns {#adding-phrase-patterns} ### Adding phrase patterns {id="adding-phrase-patterns"}
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.matcher import PhraseMatcher from spacy.matcher import PhraseMatcher
@ -967,7 +960,7 @@ as a stream.
</Infobox> </Infobox>
### Matching on other token attributes {#phrasematcher-attrs new="2.1"} ### Matching on other token attributes {id="phrasematcher-attrs",version="2.1"}
By default, the `PhraseMatcher` will match on the verbatim token text, e.g. By default, the `PhraseMatcher` will match on the verbatim token text, e.g.
`Token.text`. By setting the `attr` argument on initialization, you can change `Token.text`. By setting the `attr` argument on initialization, you can change
@ -975,8 +968,7 @@ By default, the `PhraseMatcher` will match on the verbatim token text, e.g.
pattern to the matched `Doc`. For example, using the attribute `LOWER` lets you pattern to the matched `Doc`. For example, using the attribute `LOWER` lets you
match on `Token.lower` and create case-insensitive match patterns: match on `Token.lower` and create case-insensitive match patterns:
```python ```python {executable="true"}
### {executable="true"}
from spacy.lang.en import English from spacy.lang.en import English
from spacy.matcher import PhraseMatcher from spacy.matcher import PhraseMatcher
@ -1010,8 +1002,7 @@ will be tokenized and you'll be able to find tokens and combinations of tokens
based on a few examples. Here, we're matching on the shapes `ddd.d.d.d` and based on a few examples. Here, we're matching on the shapes `ddd.d.d.d` and
`ddd.ddd.d.d`: `ddd.ddd.d.d`:
```python ```python {executable="true"}
### {executable="true"}
from spacy.lang.en import English from spacy.lang.en import English
from spacy.matcher import PhraseMatcher from spacy.matcher import PhraseMatcher
@ -1031,7 +1022,7 @@ to match phrases with the same sequence of punctuation and non-punctuation
tokens as the pattern. But this can easily get confusing and doesn't have much tokens as the pattern. But this can easily get confusing and doesn't have much
of an advantage over writing one or two token patterns. of an advantage over writing one or two token patterns.
## Dependency Matcher {#dependencymatcher new="3" model="parser"} ## Dependency Matcher {id="dependencymatcher",version="3",model="parser"}
The [`DependencyMatcher`](/api/dependencymatcher) lets you match patterns within The [`DependencyMatcher`](/api/dependencymatcher) lets you match patterns within
the dependency parse using the dependency parse using
@ -1099,7 +1090,7 @@ can be used as `LEFT_ID` in another dict.
</Infobox> </Infobox>
### Dependency matcher operators {#dependencymatcher-operators} ### Dependency matcher operators {id="dependencymatcher-operators"}
The following operators are supported by the `DependencyMatcher`, most of which The following operators are supported by the `DependencyMatcher`, most of which
come directly from come directly from
@ -1120,7 +1111,7 @@ come directly from
| `A $++ B` | `B` is a right sibling of `A`, i.e. `A` and `B` have the same parent and `A.i < B.i`. | | `A $++ B` | `B` is a right sibling of `A`, i.e. `A` and `B` have the same parent and `A.i < B.i`. |
| `A $-- B` | `B` is a left sibling of `A`, i.e. `A` and `B` have the same parent and `A.i > B.i`. | | `A $-- B` | `B` is a left sibling of `A`, i.e. `A` and `B` have the same parent and `A.i > B.i`. |
### Designing dependency matcher patterns {#dependencymatcher-patterns} ### Designing dependency matcher patterns {id="dependencymatcher-patterns"}
Let's say we want to find sentences describing who founded what kind of company: Let's say we want to find sentences describing who founded what kind of company:
@ -1145,9 +1136,11 @@ relations and tokens we want to match:
> displacy.serve(doc) > displacy.serve(doc)
> ``` > ```
import DisplaCyDepFoundedHtml from 'images/displacy-dep-founded.html' <Iframe
title="displaCy visualization of dependencies"
<Iframe title="displaCy visualization of dependencies" html={DisplaCyDepFoundedHtml} height={450} /> src="/images/displacy-dep-founded.html"
height={450}
/>
The relations we're interested in are: The relations we're interested in are:
@ -1165,8 +1158,7 @@ head to an immediate dependent as `head > child`.
The simplest dependency matcher pattern will identify and name a single token in The simplest dependency matcher pattern will identify and name a single token in
the tree: the tree:
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.matcher import DependencyMatcher from spacy.matcher import DependencyMatcher
@ -1187,8 +1179,7 @@ print(matches) # [(4851363122962674176, [1])]
Now that we have a named anchor token (`anchor_founded`), we can add the founder Now that we have a named anchor token (`anchor_founded`), we can add the founder
as the immediate dependent (`>`) of `founded` with the dependency label `nsubj`: as the immediate dependent (`>`) of `founded` with the dependency label `nsubj`:
```python ```python {title="Step 1",highlight="8,10"}
### Step 1 {highlight="8,10"}
pattern = [ pattern = [
{ {
"RIGHT_ID": "anchor_founded", "RIGHT_ID": "anchor_founded",
@ -1206,8 +1197,7 @@ pattern = [
The direct object (`dobj`) is added in the same way: The direct object (`dobj`) is added in the same way:
```python ```python {title="Step 2"}
### Step 2 {highlight=""}
pattern = [ pattern = [
#... #...
{ {
@ -1227,8 +1217,7 @@ tokens into the pattern**. For the final part of our pattern, we'll specify that
the token `founded_object` should have a modifier with the dependency relation the token `founded_object` should have a modifier with the dependency relation
`amod` or `compound`: `amod` or `compound`:
```python ```python {title="Step 3",highlight="7"}
### Step 3 {highlight="7"}
pattern = [ pattern = [
# ... # ...
{ {
@ -1247,12 +1236,11 @@ each new token needs to be linked to an existing token on its left. As for
`founded` in this example, a token may be linked to more than one token on its `founded` in this example, a token may be linked to more than one token on its
right: right:
![Dependency matcher pattern](../images/dep-match-diagram.svg) ![Dependency matcher pattern](/images/dep-match-diagram.svg)
The full pattern comes together as shown in the example below: The full pattern comes together as shown in the example below:
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.matcher import DependencyMatcher from spacy.matcher import DependencyMatcher
@ -1308,14 +1296,14 @@ of patterns such as `{}` that match any token in the sentence.
</Infobox> </Infobox>
## Rule-based entity recognition {#entityruler new="2.1"} ## Rule-based entity recognition {id="entityruler",version="2.1"}
The [`EntityRuler`](/api/entityruler) is a component that lets you add named The [`EntityRuler`](/api/entityruler) is a component that lets you add named
entities based on pattern dictionaries, which makes it easy to combine entities based on pattern dictionaries, which makes it easy to combine
rule-based and statistical named entity recognition for even more powerful rule-based and statistical named entity recognition for even more powerful
pipelines. pipelines.
### Entity Patterns {#entityruler-patterns} ### Entity Patterns {id="entityruler-patterns"}
Entity patterns are dictionaries with two keys: `"label"`, specifying the label Entity patterns are dictionaries with two keys: `"label"`, specifying the label
to assign to the entity if the pattern is matched, and `"pattern"`, the match to assign to the entity if the pattern is matched, and `"pattern"`, the match
@ -1333,7 +1321,7 @@ pattern. The entity ruler accepts two types of patterns:
{"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]} {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}
``` ```
### Using the entity ruler {#entityruler-usage} ### Using the entity ruler {id="entityruler-usage"}
The [`EntityRuler`](/api/entityruler) is a pipeline component that's typically The [`EntityRuler`](/api/entityruler) is a pipeline component that's typically
added via [`nlp.add_pipe`](/api/language#add_pipe). When the `nlp` object is added via [`nlp.add_pipe`](/api/language#add_pipe). When the `nlp` object is
@ -1343,8 +1331,7 @@ matches were to overlap, the pattern matching most tokens takes priority. If
they also happen to be equally long, then the match occurring first in the `Doc` they also happen to be equally long, then the match occurring first in the `Doc`
is chosen. is chosen.
```python ```python {executable="true"}
### {executable="true"}
from spacy.lang.en import English from spacy.lang.en import English
nlp = English() nlp = English()
@ -1366,8 +1353,7 @@ entity ruler will only add spans to the `doc.ents` if they don't overlap with
existing entities predicted by the model. To overwrite overlapping entities, you existing entities predicted by the model. To overwrite overlapping entities, you
can set `overwrite_ents=True` on initialization. can set `overwrite_ents=True` on initialization.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -1379,7 +1365,7 @@ doc = nlp("MyCorp Inc. is a company in the U.S.")
print([(ent.text, ent.label_) for ent in doc.ents]) print([(ent.text, ent.label_) for ent in doc.ents])
``` ```
#### Validating and debugging EntityRuler patterns {#entityruler-pattern-validation new="2.1.8"} #### Validating and debugging EntityRuler patterns {id="entityruler-pattern-validation",version="2.1.8"}
The entity ruler can validate patterns against a JSON schema with the config The entity ruler can validate patterns against a JSON schema with the config
setting `"validate"`. See details under setting `"validate"`. See details under
@ -1389,14 +1375,13 @@ setting `"validate"`. See details under
ruler = nlp.add_pipe("entity_ruler", config={"validate": True}) ruler = nlp.add_pipe("entity_ruler", config={"validate": True})
``` ```
### Adding IDs to patterns {#entityruler-ent-ids new="2.2.2"} ### Adding IDs to patterns {id="entityruler-ent-ids",version="2.2.2"}
The [`EntityRuler`](/api/entityruler) can also accept an `id` attribute for each The [`EntityRuler`](/api/entityruler) can also accept an `id` attribute for each
pattern. Using the `id` attribute allows multiple patterns to be associated with pattern. Using the `id` attribute allows multiple patterns to be associated with
the same entity. the same entity.
```python ```python {executable="true"}
### {executable="true"}
from spacy.lang.en import English from spacy.lang.en import English
nlp = English() nlp = English()
@ -1418,15 +1403,14 @@ patterns, the `ent_id_` property of the matched entity is set to the `id` given
in the patterns. So in the example above it's easy to identify that "San in the patterns. So in the example above it's easy to identify that "San
Francisco" and "San Fran" are both the same entity. Francisco" and "San Fran" are both the same entity.
### Using pattern files {#entityruler-files} ### Using pattern files {id="entityruler-files"}
The [`to_disk`](/api/entityruler#to_disk) and The [`to_disk`](/api/entityruler#to_disk) and
[`from_disk`](/api/entityruler#from_disk) let you save and load patterns to and [`from_disk`](/api/entityruler#from_disk) let you save and load patterns to and
from JSONL (newline-delimited JSON) files, containing one pattern object per from JSONL (newline-delimited JSON) files, containing one pattern object per
line. line.
```json ```json {title="patterns.jsonl"}
### patterns.jsonl
{"label": "ORG", "pattern": "Apple"} {"label": "ORG", "pattern": "Apple"}
{"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]} {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}
``` ```
@ -1463,9 +1447,9 @@ all pipeline components will be restored and deserialized including the enti
ruler. This lets you ship powerful pipeline packages with binary weights _and_ ruler. This lets you ship powerful pipeline packages with binary weights _and_
rules included! rules included!
### Using a large number of phrase patterns {#entityruler-large-phrase-patterns new="2.2.4"} ### Using a large number of phrase patterns {id="entityruler-large-phrase-patterns",version="2.2.4"}
<!-- TODO: double-check that this still works if the ruler is added to the pipeline on creation, and include suggestion if needed --> {/* TODO: double-check that this still works if the ruler is added to the pipeline on creation, and include suggestion if needed */}
When using a large amount of **phrase patterns** (roughly > 10000) it's useful When using a large amount of **phrase patterns** (roughly > 10000) it's useful
to understand how the `add_patterns` function of the entity ruler works. For to understand how the `add_patterns` function of the entity ruler works. For
@ -1491,14 +1475,14 @@ with nlp.select_pipes(enable="tagger"):
ruler.add_patterns(patterns) ruler.add_patterns(patterns)
``` ```
## Rule-based span matching {#spanruler new="3.3.1"} ## Rule-based span matching {id="spanruler",version="3.3.1"}
The [`SpanRuler`](/api/spanruler) is a generalized version of the entity ruler The [`SpanRuler`](/api/spanruler) is a generalized version of the entity ruler
that lets you add spans to `doc.spans` or `doc.ents` based on pattern that lets you add spans to `doc.spans` or `doc.ents` based on pattern
dictionaries, which makes it easy to combine rule-based and statistical pipeline dictionaries, which makes it easy to combine rule-based and statistical pipeline
components. components.
### Span patterns {#spanruler-patterns} ### Span patterns {id="spanruler-patterns"}
The [pattern format](#entityruler-patterns) is the same as for the entity ruler: The [pattern format](#entityruler-patterns) is the same as for the entity ruler:
@ -1514,7 +1498,7 @@ The [pattern format](#entityruler-patterns) is the same as for the entity ruler:
{"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]} {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}
``` ```
### Using the span ruler {#spanruler-usage} ### Using the span ruler {id="spanruler-usage"}
The [`SpanRuler`](/api/spanruler) is a pipeline component that's typically added The [`SpanRuler`](/api/spanruler) is a pipeline component that's typically added
via [`nlp.add_pipe`](/api/language#add_pipe). When the `nlp` object is called on via [`nlp.add_pipe`](/api/language#add_pipe). When the `nlp` object is called on
@ -1524,8 +1508,7 @@ Unlike in `doc.ents`, overlapping matches are allowed in `doc.spans`, so no
filtering is required, but optional filtering and sorting can be applied to the filtering is required, but optional filtering and sorting can be applied to the
spans before they're saved. spans before they're saved.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.blank("en") nlp = spacy.blank("en")
@ -1547,8 +1530,7 @@ always filtered, using [`util.filter_spans`](/api/top-level#util.filter_spans)
by default. See the [`SpanRuler` API docs](/api/spanruler) for more information by default. See the [`SpanRuler` API docs](/api/spanruler) for more information
about how to customize the sorting and filtering of matched spans. about how to customize the sorting and filtering of matched spans.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -1562,14 +1544,13 @@ doc = nlp("MyCorp Inc. is a company in the U.S.")
print([(ent.text, ent.label_) for ent in doc.ents]) print([(ent.text, ent.label_) for ent in doc.ents])
``` ```
### Using pattern files {#spanruler-files} ### Using pattern files {id="spanruler-files"}
You can save patterns in a JSONL file (newline-delimited JSON) to load with You can save patterns in a JSONL file (newline-delimited JSON) to load with
[`SpanRuler.initialize`](/api/spanruler#initialize) or [`SpanRuler.initialize`](/api/spanruler#initialize) or
[`SpanRuler.add_patterns`](/api/spanruler#add_patterns). [`SpanRuler.add_patterns`](/api/spanruler#add_patterns).
```json ```json {title="patterns.jsonl"}
### patterns.jsonl
{"label": "ORG", "pattern": "Apple"} {"label": "ORG", "pattern": "Apple"}
{"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]} {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}
``` ```
@ -1593,7 +1574,7 @@ separately and then added through
</Infobox> </Infobox>
## Combining models and rules {#models-rules} ## Combining models and rules {id="models-rules"}
You can combine statistical and rule-based components in a variety of ways. You can combine statistical and rule-based components in a variety of ways.
Rule-based components can be used to improve the accuracy of statistical models, Rule-based components can be used to improve the accuracy of statistical models,
@ -1604,7 +1585,7 @@ components after a statistical model to correct common errors. Finally,
rule-based components can reference the attributes set by statistical models, in rule-based components can reference the attributes set by statistical models, in
order to implement more abstract logic. order to implement more abstract logic.
### Example: Expanding named entities {#models-rules-ner} ### Example: Expanding named entities {id="models-rules-ner"}
When using a trained When using a trained
[named entity recognition](/usage/linguistic-features/#named-entities) model to [named entity recognition](/usage/linguistic-features/#named-entities) model to
@ -1630,8 +1611,7 @@ or "Dr.". This makes sense, because it makes it easier to resolve the entity
type back to a knowledge base. But what if your application needs the full type back to a knowledge base. But what if your application needs the full
names, _including_ the titles? names, _including_ the titles?
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -1650,8 +1630,7 @@ expands the entity span by one token. After all, what all titles in this example
have in common is that _if_ they occur, they occur in the **previous token** have in common is that _if_ they occur, they occur in the **previous token**
right before the person entity. right before the person entity.
```python ```python {highlight="9-13"}
### {highlight="9-13"}
from spacy.language import Language from spacy.language import Language
from spacy.tokens import Span from spacy.tokens import Span
@ -1679,8 +1658,7 @@ register it as a [pipeline component](/usage/processing-pipelines) so it can run
automatically when processing a text. We can use automatically when processing a text. We can use
[`nlp.add_pipe`](/api/language#add_pipe) to add it to the current pipeline. [`nlp.add_pipe`](/api/language#add_pipe) to add it to the current pipeline.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.language import Language from spacy.language import Language
from spacy.tokens import Span from spacy.tokens import Span
@ -1729,8 +1707,7 @@ We can now use the [`Span.set_extension`](/api/span#set_extension) method to add
the custom extension attribute `"person_title"`, using `get_person_title` as the the custom extension attribute `"person_title"`, using `get_person_title` as the
getter function. getter function.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.tokens import Span from spacy.tokens import Span
@ -1749,7 +1726,7 @@ doc = nlp("Dr Alex Smith chaired first board meeting of Acme Corp Inc.")
print([(ent.text, ent.label_, ent._.person_title) for ent in doc.ents]) print([(ent.text, ent.label_, ent._.person_title) for ent in doc.ents])
``` ```
### Example: Using entities, part-of-speech tags and the dependency parse {#models-rules-pos-dep} ### Example: Using entities, part-of-speech tags and the dependency parse {id="models-rules-pos-dep"}
> #### Linguistic features > #### Linguistic features
> >
@ -1772,8 +1749,7 @@ tense**, whether company names are attached to it and whether the person is the
subject. All of this information is available in the part-of-speech tags and the subject. All of this information is available in the part-of-speech tags and the
dependency parse. dependency parse.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -1788,7 +1764,7 @@ print([(ent.text, ent.label_) for ent in doc.ents])
> - `VBD`: Verb, past tense. > - `VBD`: Verb, past tense.
> - `IN`: Conjunction, subordinating or preposition. > - `IN`: Conjunction, subordinating or preposition.
![Visualization of dependency parse](../images/displacy-model-rules.svg "[`spacy.displacy`](/api/top-level#displacy) visualization with `options={'fine_grained': True}` to output the fine-grained part-of-speech tags, i.e. `Token.tag_`") ![Visualization of dependency parse](/images/displacy-model-rules.svg "[`spacy.displacy`](/api/top-level#displacy) visualization with `options={'fine_grained': True}` to output the fine-grained part-of-speech tags, i.e. `Token.tag_`")
In this example, "worked" is the root of the sentence and is a past tense verb. In this example, "worked" is the root of the sentence and is a past tense verb.
Its subject is "Alex Smith", the person who worked. "at Acme Corp Inc." is a Its subject is "Alex Smith", the person who worked. "at Acme Corp Inc." is a
@ -1837,8 +1813,7 @@ the entity `Span` for example `._.orgs` or `._.prev_orgs` and
> retokenizer.merge(ent) > retokenizer.merge(ent)
> ``` > ```
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.language import Language from spacy.language import Language
from spacy import displacy from spacy import displacy
@ -1871,12 +1846,11 @@ notice that our current logic fails and doesn't correctly detect the company as
a past organization. That's because the root is a participle and the tense a past organization. That's because the root is a participle and the tense
information is in the attached auxiliary "was": information is in the attached auxiliary "was":
![Visualization of dependency parse](../images/displacy-model-rules2.svg) ![Visualization of dependency parse](/images/displacy-model-rules2.svg)
To solve this, we can adjust the rules to also check for the above construction: To solve this, we can adjust the rules to also check for the above construction:
```python ```python {highlight="10-12"}
### {highlight="10-12"}
@Language.component("extract_person_orgs") @Language.component("extract_person_orgs")
def extract_person_orgs(doc): def extract_person_orgs(doc):
person_entities = [ent for ent in doc.ents if ent.label_ == "PERSON"] person_entities = [ent for ent in doc.ents if ent.label_ == "PERSON"]

View File

@ -8,13 +8,11 @@ menu:
- ['Trained Pipelines', 'models'] - ['Trained Pipelines', 'models']
--- ---
## Basics {#basics hidden="true"} ## Basics {id="basics",hidden="true"}
import Serialization101 from 'usage/101/\_serialization.md'
<Serialization101 /> <Serialization101 />
### Serializing the pipeline {#pipeline} ### Serializing the pipeline {id="pipeline"}
When serializing the pipeline, keep in mind that this will only save out the When serializing the pipeline, keep in mind that this will only save out the
**binary data for the individual components** to allow spaCy to restore them **binary data for the individual components** to allow spaCy to restore them
@ -30,14 +28,12 @@ contains the pipeline configuration and all the relevant settings.
> dictionary containing the training configuration, pipeline component factories > dictionary containing the training configuration, pipeline component factories
> and other settings. It is saved out with a pipeline as the `config.cfg`. > and other settings. It is saved out with a pipeline as the `config.cfg`.
```python ```python {title="Serialize"}
### Serialize
config = nlp.config config = nlp.config
bytes_data = nlp.to_bytes() bytes_data = nlp.to_bytes()
``` ```
```python ```python {title="Deserialize"}
### Deserialize
lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"]) lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
nlp = lang_cls.from_config(config) nlp = lang_cls.from_config(config)
nlp.from_bytes(bytes_data) nlp.from_bytes(bytes_data)
@ -49,7 +45,7 @@ the language class, creates and adds the pipeline components based on the config
and _then_ loads in the binary data. You can read more about this process and _then_ loads in the binary data. You can read more about this process
[here](/usage/processing-pipelines#pipelines). [here](/usage/processing-pipelines#pipelines).
## Serializing Doc objects efficiently {#docs new="2.2"} ## Serializing Doc objects efficiently {id="docs",version="2.2"}
If you're working with lots of data, you'll probably need to pass analyses If you're working with lots of data, you'll probably need to pass analyses
between machines, either to use something like [Dask](https://dask.org) or between machines, either to use something like [Dask](https://dask.org) or
@ -64,8 +60,7 @@ collection of `Doc` objects together, and is much more efficient than calling
also control what data gets saved, and you can merge pallets together for easy also control what data gets saved, and you can merge pallets together for easy
map/reduce-style processing. map/reduce-style processing.
```python ```python {highlight="4,8,9,13,14"}
### {highlight="4,8,9,13,14"}
import spacy import spacy
from spacy.tokens import DocBin from spacy.tokens import DocBin
@ -101,7 +96,7 @@ print([doc._.my_custom_attr for doc in docs])
</Infobox> </Infobox>
### Using Pickle {#pickle} ### Using Pickle {id="pickle"}
> #### Example > #### Example
> >
@ -124,8 +119,7 @@ the entire pipeline once. And instead of pickling several `Doc` objects
separately, pickle a list of `Doc` objects. Since they all share a reference to separately, pickle a list of `Doc` objects. Since they all share a reference to
the _same_ `Vocab` object, it will only be included once. the _same_ `Vocab` object, it will only be included once.
```python ```python {title="Pickling objects with shared data",highlight="8-9"}
### Pickling objects with shared data {highlight="8-9"}
doc1 = nlp("Hello world") doc1 = nlp("Hello world")
doc2 = nlp("This is a test") doc2 = nlp("This is a test")
@ -161,7 +155,7 @@ data = pickle.dumps(span_doc)
</Infobox> </Infobox>
## Implementing serialization methods {#serialization-methods} ## Implementing serialization methods {id="serialization-methods"}
When you call [`nlp.to_disk`](/api/language#to_disk), When you call [`nlp.to_disk`](/api/language#to_disk),
[`nlp.from_disk`](/api/language#from_disk) or load a pipeline package, spaCy [`nlp.from_disk`](/api/language#from_disk) or load a pipeline package, spaCy
@ -201,8 +195,7 @@ the data to and from a JSON file.
> saving out a pipeline with a rule-based entity recognizer and including all > saving out a pipeline with a rule-based entity recognizer and including all
> rules _with_ the component data. > rules _with_ the component data.
```python ```python {highlight="16-23,25-30"}
### {highlight="16-23,25-30"}
import json import json
from spacy import Language from spacy import Language
from spacy.util import ensure_path from spacy.util import ensure_path
@ -242,8 +235,7 @@ After adding the component to the pipeline and adding some data to it, we can
serialize the `nlp` object to a directory, which will call the custom serialize the `nlp` object to a directory, which will call the custom
component's `to_disk` method. component's `to_disk` method.
```python ```python {highlight="2-4"}
### {highlight="2-4"}
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
my_component = nlp.add_pipe("my_component") my_component = nlp.add_pipe("my_component")
my_component.add({"hello": "world"}) my_component.add({"hello": "world"})
@ -254,8 +246,7 @@ The contents of the directory would then look like this.
`CustomComponent.to_disk` converted the data to a JSON string and saved it to a `CustomComponent.to_disk` converted the data to a JSON string and saved it to a
file `data.json` in its subdirectory: file `data.json` in its subdirectory:
```yaml ```yaml {title="Directory structure",highlight="2-3"}
### Directory structure {highlight="2-3"}
└── /path/to/pipeline └── /path/to/pipeline
├── my_component # data serialized by "my_component" ├── my_component # data serialized by "my_component"
│ └── data.json │ └── data.json
@ -292,9 +283,9 @@ custom components to spaCy automatically.
</Infobox> </Infobox>
<!-- ## Initializing components with data {#initialization new="3"} --> {/* ## Initializing components with data {id="initialization",version="3"} */}
## Using entry points {#entry-points new="2.1"} ## Using entry points {id="entry-points",version="2.1"}
Entry points let you expose parts of a Python package you write to other Python Entry points let you expose parts of a Python package you write to other Python
packages. This lets one application easily customize the behavior of another, by packages. This lets one application easily customize the behavior of another, by
@ -313,7 +304,7 @@ installed in the same environment that's it.
| `spacy_lookups` | Group of entry points for custom [`Lookups`](/api/lookups), including lemmatizer data. Used by spaCy's [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) package. | | `spacy_lookups` | Group of entry points for custom [`Lookups`](/api/lookups), including lemmatizer data. Used by spaCy's [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) package. |
| [`spacy_displacy_colors`](#entry-points-displacy) | Group of entry points of custom label colors for the [displaCy visualizer](/usage/visualizers#ent). The key name doesn't matter, but it should point to a dict of labels and color values. Useful for custom models that predict different entity types. | | [`spacy_displacy_colors`](#entry-points-displacy) | Group of entry points of custom label colors for the [displaCy visualizer](/usage/visualizers#ent). The key name doesn't matter, but it should point to a dict of labels and color values. Useful for custom models that predict different entity types. |
### Custom components via entry points {#entry-points-components} ### Custom components via entry points {id="entry-points-components"}
When you load a pipeline, spaCy will generally use its `config.cfg` to set up When you load a pipeline, spaCy will generally use its `config.cfg` to set up
the language class and construct the pipeline. The pipeline is specified as a the language class and construct the pipeline. The pipeline is specified as a
@ -343,8 +334,7 @@ snake when it's called:
> └── setup.py # setup file for pip installation > └── setup.py # setup file for pip installation
> ``` > ```
```python ```python {title="snek.py"}
### snek.py
from spacy.language import Language from spacy.language import Language
snek = """ snek = """
@ -377,8 +367,7 @@ entry to the factories, you can now expose it in your `setup.py` via the
> the created entry point is named `snek` and points to the function > the created entry point is named `snek` and points to the function
> `snek_component` in the module `snek`, i.e. `snek.py`. > `snek_component` in the module `snek`, i.e. `snek.py`.
```python ```python {title="setup.py",highlight="5-7"}
### setup.py {highlight="5-7"}
from setuptools import setup from setuptools import setup
setup( setup(
@ -459,8 +448,7 @@ class SnekFactory:
return doc return doc
``` ```
```diff ```diff {title="setup.py"}
### setup.py
entry_points={ entry_points={
- "spacy_factories": ["snek = snek:snek_component"] - "spacy_factories": ["snek = snek:snek_component"]
+ "spacy_factories": ["snek = snek:SnekFactory"] + "spacy_factories": ["snek = snek:SnekFactory"]
@ -496,7 +484,7 @@ The above example will serialize the current snake in a `snek.txt` in the data
directory. When a pipeline using the `snek` component is loaded, it will open directory. When a pipeline using the `snek` component is loaded, it will open
the `snek.txt` and make it available to the component. the `snek.txt` and make it available to the component.
### Custom language classes via entry points {#entry-points-languages} ### Custom language classes via entry points {id="entry-points-languages"}
To stay with the theme of the previous example and To stay with the theme of the previous example and
[this blog post on entry points](https://amir.rachum.com/blog/2017/07/28/python-entry-points/), [this blog post on entry points](https://amir.rachum.com/blog/2017/07/28/python-entry-points/),
@ -505,8 +493,7 @@ custom pipeline  but you don't necessarily want to modify spaCy's code to ad
language. In your package, you could then implement the following language. In your package, you could then implement the following
[custom language subclass](/usage/linguistic-features#language-subclass): [custom language subclass](/usage/linguistic-features#language-subclass):
```python ```python {title="snek.py"}
### snek.py
from spacy.language import Language from spacy.language import Language
class SnekDefaults(Language.Defaults): class SnekDefaults(Language.Defaults):
@ -521,8 +508,7 @@ Alongside the `spacy_factories`, there's also an entry point option for
`spacy_languages`, which maps language codes to language-specific `Language` `spacy_languages`, which maps language codes to language-specific `Language`
subclasses: subclasses:
```diff ```diff {title="setup.py"}
### setup.py
from setuptools import setup from setuptools import setup
setup( setup(
@ -540,7 +526,7 @@ pipeline packages you [train](/usage/training), which could then specify
`lang = snk` in their `config.cfg` without spaCy raising an error because the `lang = snk` in their `config.cfg` without spaCy raising an error because the
language is not available in the core library. language is not available in the core library.
### Custom displaCy colors via entry points {#entry-points-displacy new="2.2"} ### Custom displaCy colors via entry points {id="entry-points-displacy",version="2.2"}
If you're training a named entity recognition model for a custom domain, you may If you're training a named entity recognition model for a custom domain, you may
end up training different labels that don't have pre-defined colors in the end up training different labels that don't have pre-defined colors in the
@ -555,8 +541,7 @@ values.
> [scispaCy](/universe/project/scispacy) and > [scispaCy](/universe/project/scispacy) and
> [Blackstone](/universe/project/blackstone). > [Blackstone](/universe/project/blackstone).
```python ```python {title="snek.py"}
### snek.py
displacy_colors = {"SNEK": "#3dff74", "HUMAN": "#cfc5ff"} displacy_colors = {"SNEK": "#3dff74", "HUMAN": "#cfc5ff"}
``` ```
@ -564,8 +549,7 @@ Given the above colors, the entry point can be defined as follows. Entry points
need to have a name, so we use the key `colors`. However, the name doesn't need to have a name, so we use the key `colors`. However, the name doesn't
matter and whatever is defined in the entry point group will be used. matter and whatever is defined in the entry point group will be used.
```diff ```diff {title="setup.py"}
### setup.py
from setuptools import setup from setuptools import setup
setup( setup(
@ -580,11 +564,13 @@ After installing the package, the custom colors will be used when visualizing
text with `displacy`. Whenever the label `SNEK` is assigned, it will be text with `displacy`. Whenever the label `SNEK` is assigned, it will be
displayed in `#3dff74`. displayed in `#3dff74`.
import DisplaCyEntSnekHtml from 'images/displacy-ent-snek.html' <Iframe
title="displaCy visualization of entities"
src="/images/displacy-ent-snek.html"
height={100}
/>
<Iframe title="displaCy visualization of entities" html={DisplaCyEntSnekHtml} height={100} /> ## Saving, loading and distributing trained pipelines {id="models"}
## Saving, loading and distributing trained pipelines {#models}
After training your pipeline, you'll usually want to save its state, and load it After training your pipeline, you'll usually want to save its state, and load it
back later. You can do this with the [`Language.to_disk`](/api/language#to_disk) back later. You can do this with the [`Language.to_disk`](/api/language#to_disk)
@ -629,7 +615,7 @@ Universal Dependencies treebank and generates an installable Python package.
</Project> </Project>
### Generating a pipeline package {#models-generating} ### Generating a pipeline package {id="models-generating"}
<Infobox title="Important note" variant="warning"> <Infobox title="Important note" variant="warning">
@ -662,7 +648,7 @@ this, see the [`package`](/api/cli#package) docs.
> } > }
> ``` > ```
```cli ```bash
$ python -m spacy package ./en_example_pipeline ./packages $ python -m spacy package ./en_example_pipeline ./packages
``` ```
@ -671,8 +657,7 @@ This command will create a pipeline package directory and will run
`.tar.gz` archive of your package that can be installed using `pip install`. `.tar.gz` archive of your package that can be installed using `pip install`.
Installing the binary wheel is usually more efficient. Installing the binary wheel is usually more efficient.
```yaml ```yaml {title="Directory structure"}
### Directory structure
└── / └── /
├── MANIFEST.in # to include meta.json ├── MANIFEST.in # to include meta.json
├── meta.json # pipeline meta data ├── meta.json # pipeline meta data
@ -693,7 +678,7 @@ If you're creating the package manually, keep in mind that the directories need
to be named according to the naming conventions of `lang_name` and to be named according to the naming conventions of `lang_name` and
`lang_name-version`. `lang_name-version`.
### Including custom functions and components {#models-custom} ### Including custom functions and components {id="models-custom"}
If your pipeline includes If your pipeline includes
[custom components](/usage/processing-pipelines#custom-components), model [custom components](/usage/processing-pipelines#custom-components), model
@ -713,7 +698,7 @@ argument.
> ... > ...
> ``` > ```
```cli ```bash
$ python -m spacy package ./en_example_pipeline ./packages --code functions.py $ python -m spacy package ./en_example_pipeline ./packages --code functions.py
``` ```
@ -744,7 +729,7 @@ spaCy to export the current state of its `nlp` objects via
</Infobox> </Infobox>
### Loading a custom pipeline package {#loading} ### Loading a custom pipeline package {id="loading"}
To load a pipeline from a data directory, you can use To load a pipeline from a data directory, you can use
[`spacy.load()`](/api/top-level#spacy.load) with the local path. This will look [`spacy.load()`](/api/top-level#spacy.load) with the local path. This will look

View File

@ -30,18 +30,24 @@ quick introduction.
<Infobox title="Take the free interactive course"> <Infobox title="Take the free interactive course">
[![Advanced NLP with spaCy](../images/course.jpg)](https://course.spacy.io) <Image
src="/images/course.jpg"
href="https://course.spacy.io"
alt="Advanced NLP with spaCy"
/>
In this course you'll learn how to use spaCy to build advanced natural language In this course you'll learn how to use spaCy to build advanced natural language
understanding systems, using both rule-based and machine learning approaches. It understanding systems, using both rule-based and machine learning approaches. It
includes 55 exercises featuring interactive coding practice, multiple-choice includes 55 exercises featuring interactive coding practice, multiple-choice
questions and slide decks. questions and slide decks.
<p><Button to="https://course.spacy.io" variant="primary">Start the course</Button></p> <Button to="https://course.spacy.io" variant="primary">
{'Start the course'}
</Button>
</Infobox> </Infobox>
## What's spaCy? {#whats-spacy} ## What's spaCy? {id="whats-spacy"}
<Grid cols={2}> <Grid cols={2}>
@ -82,7 +88,7 @@ systems, or to pre-process text for **deep learning**.
</Grid> </Grid>
### What spaCy isn't {#what-spacy-isnt} ### What spaCy isn't {id="what-spacy-isnt"}
- ❌ **spaCy is not a platform or "an API"**. Unlike a platform, spaCy does not - ❌ **spaCy is not a platform or "an API"**. Unlike a platform, spaCy does not
provide a software as a service, or a web application. It's an open-source provide a software as a service, or a web application. It's an open-source
@ -103,7 +109,7 @@ systems, or to pre-process text for **deep learning**.
publishing spaCy and other software is called publishing spaCy and other software is called
[Explosion](https://explosion.ai). [Explosion](https://explosion.ai).
## Features {#features} ## Features {id="features"}
In the documentation, you'll come across mentions of spaCy's features and In the documentation, you'll come across mentions of spaCy's features and
capabilities. Some of them refer to linguistic concepts, while others are capabilities. Some of them refer to linguistic concepts, while others are
@ -124,7 +130,7 @@ related to more general machine learning functionality.
| **Training** | Updating and improving a statistical model's predictions. | | **Training** | Updating and improving a statistical model's predictions. |
| **Serialization** | Saving objects to files or byte strings. | | **Serialization** | Saving objects to files or byte strings. |
### Statistical models {#statistical-models} ### Statistical models {id="statistical-models"}
While some of spaCy's features work independently, others require While some of spaCy's features work independently, others require
[trained pipelines](/models) to be loaded, which enable spaCy to **predict** [trained pipelines](/models) to be loaded, which enable spaCy to **predict**
@ -148,7 +154,7 @@ are always a good start. They typically include the following components:
and model implementations to use, to put spaCy in the correct state when you and model implementations to use, to put spaCy in the correct state when you
load the pipeline. load the pipeline.
## Linguistic annotations {#annotations} ## Linguistic annotations {id="annotations"}
spaCy provides a variety of linguistic annotations to give you **insights into a spaCy provides a variety of linguistic annotations to give you **insights into a
text's grammatical structure**. This includes the word types, like the parts of text's grammatical structure**. This includes the word types, like the parts of
@ -159,7 +165,7 @@ the website or company in a specific context.
> #### Loading pipelines > #### Loading pipelines
> >
> ```cli > ```bash
> $ python -m spacy download en_core_web_sm > $ python -m spacy download en_core_web_sm
> >
> >>> import spacy > >>> import spacy
@ -172,8 +178,7 @@ can load it via [`spacy.load`](/api/top-level#spacy.load). This will return a
usually call it `nlp`. Calling the `nlp` object on a string of text will return usually call it `nlp`. Calling the `nlp` object on a string of text will return
a processed `Doc`: a processed `Doc`:
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -189,9 +194,7 @@ original string, or reconstruct the original by joining the tokens and their
trailing whitespace. This way, you'll never lose any information when processing trailing whitespace. This way, you'll never lose any information when processing
text with spaCy. text with spaCy.
### Tokenization {#annotations-token} ### Tokenization {id="annotations-token"}
import Tokenization101 from 'usage/101/\_tokenization.md'
<Tokenization101 /> <Tokenization101 />
@ -205,9 +208,7 @@ language-specific data**, see the usage guides on
</Infobox> </Infobox>
### Part-of-speech tags and dependencies {#annotations-pos-deps model="parser"} ### Part-of-speech tags and dependencies {id="annotations-pos-deps",model="parser"}
import PosDeps101 from 'usage/101/\_pos-deps.md'
<PosDeps101 /> <PosDeps101 />
@ -220,9 +221,7 @@ how to **navigate and use the parse tree** effectively, see the usage guides on
</Infobox> </Infobox>
### Named Entities {#annotations-ner model="ner"} ### Named Entities {id="annotations-ner",model="ner"}
import NER101 from 'usage/101/\_named-entities.md'
<NER101 /> <NER101 />
@ -236,9 +235,7 @@ of a model, see the usage guides on
</Infobox> </Infobox>
### Word vectors and similarity {#vectors-similarity model="vectors"} ### Word vectors and similarity {id="vectors-similarity",model="vectors"}
import Vectors101 from 'usage/101/\_vectors-similarity.md'
<Vectors101 /> <Vectors101 />
@ -250,9 +247,7 @@ To learn more about word vectors, how to **customize them** and how to load
</Infobox> </Infobox>
## Pipelines {#pipelines} ## Pipelines {id="pipelines"}
import Pipelines101 from 'usage/101/\_pipelines.md'
<Pipelines101 /> <Pipelines101 />
@ -264,13 +259,11 @@ guide on [language processing pipelines](/usage/processing-pipelines).
</Infobox> </Infobox>
## Architecture {#architecture} ## Architecture {id="architecture"}
import Architecture101 from 'usage/101/\_architecture.md'
<Architecture101 /> <Architecture101 />
## Vocab, hashes and lexemes {#vocab} ## Vocab, hashes and lexemes {id="vocab"}
Whenever possible, spaCy tries to store data in a vocabulary, the Whenever possible, spaCy tries to store data in a vocabulary, the
[`Vocab`](/api/vocab), that will be **shared by multiple documents**. To save [`Vocab`](/api/vocab), that will be **shared by multiple documents**. To save
@ -288,7 +281,7 @@ and part-of-speech tags like "VERB" are also encoded. Internally, spaCy only
> - **StringStore**: The dictionary mapping hash values to strings, for example > - **StringStore**: The dictionary mapping hash values to strings, for example
> `3197928453018144401` → "coffee". > `3197928453018144401` → "coffee".
![Doc, Vocab, Lexeme and StringStore](../images/vocab_stringstore.svg) ![Doc, Vocab, Lexeme and StringStore](/images/vocab_stringstore.svg)
If you process lots of documents containing the word "coffee" in all kinds of If you process lots of documents containing the word "coffee" in all kinds of
different contexts, storing the exact string "coffee" every time would take up different contexts, storing the exact string "coffee" every time would take up
@ -297,8 +290,7 @@ way too much space. So instead, spaCy hashes the string and stores it in the
**lookup table that works in both directions** you can look up a string to get **lookup table that works in both directions** you can look up a string to get
its hash, or a hash to get its string: its hash, or a hash to get its string:
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -315,8 +307,7 @@ a word. For example, no matter if "love" is used as a verb or a noun in some
context, its spelling and whether it consists of alphabetic characters won't context, its spelling and whether it consists of alphabetic characters won't
ever change. Its hash value will also always be the same. ever change. Its hash value will also always be the same.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
nlp = spacy.load("en_core_web_sm") nlp = spacy.load("en_core_web_sm")
@ -354,8 +345,7 @@ vocabulary. That's why you always need to make sure all objects you create have
access to the same vocabulary. If they don't, spaCy might not be able to find access to the same vocabulary. If they don't, spaCy might not be able to find
the strings it needs. the strings it needs.
```python ```python {executable="true"}
### {executable="true"}
import spacy import spacy
from spacy.tokens import Doc from spacy.tokens import Doc
from spacy.vocab import Vocab from spacy.vocab import Vocab
@ -382,9 +372,7 @@ spaCy will also export the `Vocab` when you save a `Doc` or `nlp` object. This
will give you the object and its encoded annotations, plus the "key" to decode will give you the object and its encoded annotations, plus the "key" to decode
it. it.
## Serialization {#serialization} ## Serialization {id="serialization"}
import Serialization101 from 'usage/101/\_serialization.md'
<Serialization101 /> <Serialization101 />
@ -395,9 +383,7 @@ guide on [saving and loading](/usage/saving-loading#models).
</Infobox> </Infobox>
## Training {#training} ## Training {id="training"}
import Training101 from 'usage/101/\_training.md'
<Training101 /> <Training101 />
@ -409,7 +395,7 @@ data and how to improve spaCy's named models, see the usage guides on
</Infobox> </Infobox>
### Training config and lifecycle {#training-config} ### Training config and lifecycle {id="training-config"}
Training config files include all **settings and hyperparameters** for training Training config files include all **settings and hyperparameters** for training
your pipeline. Instead of providing lots of arguments on the command line, you your pipeline. Instead of providing lots of arguments on the command line, you
@ -433,7 +419,7 @@ source of truth", both at **training** and **runtime**.
> initial_rate = 0.01 > initial_rate = 0.01
> ``` > ```
![Illustration of pipeline lifecycle](../images/lifecycle.svg) ![Illustration of pipeline lifecycle](/images/lifecycle.svg)
<Infobox title="Training configuration system" emoji="📖"> <Infobox title="Training configuration system" emoji="📖">
@ -443,7 +429,7 @@ hyperparameters, see the [training config](/usage/training#config) usage guide.
</Infobox> </Infobox>
### Trainable components {#training-components} ### Trainable components {id="training-components"}
spaCy's [`Pipe`](/api/pipe) class helps you implement your own trainable spaCy's [`Pipe`](/api/pipe) class helps you implement your own trainable
components that have their own model instance, make predictions over `Doc` components that have their own model instance, make predictions over `Doc`
@ -462,7 +448,7 @@ configured via a single training config.
> width = 128 > width = 128
> ``` > ```
![Illustration of Pipe methods](../images/trainable_component.svg) ![Illustration of Pipe methods](/images/trainable_component.svg)
<Infobox title="Custom trainable components" emoji="📖"> <Infobox title="Custom trainable components" emoji="📖">
@ -474,13 +460,11 @@ for trainable components.
</Infobox> </Infobox>
## Language data {#language-data} ## Language data {id="language-data"}
import LanguageData101 from 'usage/101/\_language-data.md'
<LanguageData101 /> <LanguageData101 />
## Community & FAQ {#community-faq} ## Community & FAQ {id="community-faq"}
We're very happy to see the spaCy community grow and include a mix of people We're very happy to see the spaCy community grow and include a mix of people
from all kinds of different backgrounds computational linguistics, data from all kinds of different backgrounds computational linguistics, data
@ -488,7 +472,7 @@ science, deep learning, research and more. If you'd like to get involved, below
are some answers to the most important questions and resources for further are some answers to the most important questions and resources for further
reading. reading.
### Help, my code isn't working! {#faq-help-code} ### Help, my code isn't working! {id="faq-help-code"}
Bugs suck, and we're doing our best to continuously improve the tests and fix Bugs suck, and we're doing our best to continuously improve the tests and fix
bugs as soon as possible. Before you submit an issue, do a quick search and bugs as soon as possible. Before you submit an issue, do a quick search and
@ -513,10 +497,11 @@ via the following platforms:
questions** and everything related to problems with your specific code. The questions** and everything related to problems with your specific code. The
Stack Overflow community is much larger than ours, so if your problem can be Stack Overflow community is much larger than ours, so if your problem can be
solved by others, you'll receive help much quicker. solved by others, you'll receive help much quicker.
- [GitHub discussions](https://github.com/explosion/spaCy/discussions): **General - [GitHub discussions](https://github.com/explosion/spaCy/discussions):
discussion**, **project ideas** and **usage questions**. Meet other community **General discussion**, **project ideas** and **usage questions**. Meet other
members to get help with a specific code implementation, discuss ideas for new community members to get help with a specific code implementation, discuss
projects/plugins, support more languages, and share best practices. ideas for new projects/plugins, support more languages, and share best
practices.
- [GitHub issue tracker](https://github.com/explosion/spaCy/issues): **Bug - [GitHub issue tracker](https://github.com/explosion/spaCy/issues): **Bug
reports** and **improvement suggestions**, i.e. everything that's likely reports** and **improvement suggestions**, i.e. everything that's likely
spaCy's fault. This also includes problems with the trained pipelines beyond spaCy's fault. This also includes problems with the trained pipelines beyond
@ -533,7 +518,7 @@ and the next time you need help, they might repay the favor.
</Infobox> </Infobox>
### How can I contribute to spaCy? {#faq-contributing} ### How can I contribute to spaCy? {id="faq-contributing"}
You don't have to be an NLP expert or Python pro to contribute, and we're happy You don't have to be an NLP expert or Python pro to contribute, and we're happy
to help you get started. If you're new to spaCy, a good place to start is the to help you get started. If you're new to spaCy, a good place to start is the
@ -567,7 +552,7 @@ By participating, you are expected to uphold this code.
</Infobox> </Infobox>
### I've built something cool with spaCy how can I get the word out? {#faq-project-with-spacy} ### I've built something cool with spaCy how can I get the word out? {id="faq-project-with-spacy"}
First, congrats we'd love to check it out! When you share your project on First, congrats we'd love to check it out! When you share your project on
Twitter, don't forget to tag [@spacy_io](https://twitter.com/spacy_io) so we Twitter, don't forget to tag [@spacy_io](https://twitter.com/spacy_io) so we
@ -588,7 +573,8 @@ project is using spaCy, you can grab one of our **spaCy badges** here:
[![Built with spaCy](https://img.shields.io/badge/built%20with-spaCy-09a3d5.svg)](https://spacy.io) [![Built with spaCy](https://img.shields.io/badge/built%20with-spaCy-09a3d5.svg)](https://spacy.io)
``` ```
<img src={`https://img.shields.io/badge/made%20with%20❤%20and-spaCy-09a3d5.svg`} <img
src={`https://img.shields.io/badge/made%20with%20❤%20and-spaCy-09a3d5.svg`}
/> />
```markdown ```markdown

View File

@ -15,15 +15,17 @@ menu:
- ['Internal API', 'api'] - ['Internal API', 'api']
--- ---
## Introduction to training {#basics hidden="true"} ## Introduction to training {id="basics",hidden="true"}
import Training101 from 'usage/101/\_training.md'
<Training101 /> <Training101 />
<Infobox title="Tip: Try the Prodigy annotation tool"> <Infobox title="Tip: Try the Prodigy annotation tool">
[![Prodigy: Radically efficient machine teaching](../images/prodigy.jpg)](https://prodi.gy) <Image
src="/images/prodigy.jpg"
href="https://prodi.gy"
alt="Prodigy: Radically efficient machine teaching"
/>
If you need to label a lot of data, check out [Prodigy](https://prodi.gy), a If you need to label a lot of data, check out [Prodigy](https://prodi.gy), a
new, active learning-powered annotation tool we've developed. Prodigy is fast new, active learning-powered annotation tool we've developed. Prodigy is fast
@ -34,7 +36,7 @@ ready-to-use spaCy pipelines.
</Infobox> </Infobox>
## Quickstart {#quickstart tag="new"} ## Quickstart {id="quickstart",tag="new"}
The recommended way to train your spaCy pipelines is via the The recommended way to train your spaCy pipelines is via the
[`spacy train`](/api/cli#train) command on the command line. It only needs a [`spacy train`](/api/cli#train) command on the command line. It only needs a
@ -69,8 +71,6 @@ config.
> requirements and settings as CLI arguments. > requirements and settings as CLI arguments.
> 2. Run [`train`](/api/cli#train) with the exported config and data. > 2. Run [`train`](/api/cli#train) with the exported config and data.
import QuickstartTraining from 'widgets/quickstart-training.js'
<QuickstartTraining /> <QuickstartTraining />
After you've saved the starter config to a file `base_config.cfg`, you can use After you've saved the starter config to a file `base_config.cfg`, you can use
@ -78,7 +78,7 @@ the [`init fill-config`](/api/cli#init-fill-config) command to fill in the
remaining defaults. Training configs should always be **complete and without remaining defaults. Training configs should always be **complete and without
hidden defaults**, to keep your experiments reproducible. hidden defaults**, to keep your experiments reproducible.
```cli ```bash
$ python -m spacy init fill-config base_config.cfg config.cfg $ python -m spacy init fill-config base_config.cfg config.cfg
``` ```
@ -88,7 +88,7 @@ $ python -m spacy init fill-config base_config.cfg config.cfg
> your training and development data, get useful stats, and find problems like > your training and development data, get useful stats, and find problems like
> invalid entity annotations, cyclic dependencies, low data labels and more. > invalid entity annotations, cyclic dependencies, low data labels and more.
> >
> ```cli > ```bash
> $ python -m spacy debug data config.cfg > $ python -m spacy debug data config.cfg
> ``` > ```
@ -100,7 +100,7 @@ add your data and run [`train`](/api/cli#train) with your config. See the
spaCy's binary `.spacy` format. You can either include the data paths in the spaCy's binary `.spacy` format. You can either include the data paths in the
`[paths]` section of your config, or pass them in via the command line. `[paths]` section of your config, or pass them in via the command line.
```cli ```bash
$ python -m spacy train config.cfg --output ./output --paths.train ./train.spacy --paths.dev ./dev.spacy $ python -m spacy train config.cfg --output ./output --paths.train ./train.spacy --paths.dev ./dev.spacy
``` ```
@ -108,7 +108,7 @@ $ python -m spacy train config.cfg --output ./output --paths.train ./train.spacy
> >
> Use the `--gpu-id` option to select the GPU: > Use the `--gpu-id` option to select the GPU:
> >
> ```cli > ```bash
> $ python -m spacy train config.cfg --gpu-id 0 > $ python -m spacy train config.cfg --gpu-id 0
> ``` > ```
@ -141,7 +141,7 @@ treebank.
</Project> </Project>
## Training config system {#config} ## Training config system {id="config"}
Training config files include all **settings and hyperparameters** for training Training config files include all **settings and hyperparameters** for training
your pipeline. Instead of providing lots of arguments on the command line, you your pipeline. Instead of providing lots of arguments on the command line, you
@ -212,7 +212,7 @@ available for the different architectures are documented with the
<YouTube id="BWhh3r6W-qE"></YouTube> <YouTube id="BWhh3r6W-qE"></YouTube>
### Config lifecycle at runtime and training {#config-lifecycle} ### Config lifecycle at runtime and training {id="config-lifecycle"}
A pipeline's `config.cfg` is considered the "single source of truth", both at A pipeline's `config.cfg` is considered the "single source of truth", both at
**training** and **runtime**. Under the hood, **training** and **runtime**. Under the hood,
@ -222,7 +222,7 @@ config is available as [`nlp.config`](/api/language#config) and it includes all
information about the pipeline, as well as the settings used to train and information about the pipeline, as well as the settings used to train and
initialize it. initialize it.
![Illustration of pipeline lifecycle](../images/lifecycle.svg) ![Illustration of pipeline lifecycle](/images/lifecycle.svg)
At runtime spaCy will only use the `[nlp]` and `[components]` blocks of the At runtime spaCy will only use the `[nlp]` and `[components]` blocks of the
config and load all data, including tokenization rules, model weights and other config and load all data, including tokenization rules, model weights and other
@ -240,7 +240,7 @@ requiring it to be available at runtime. You can also use this mechanism to
provide data paths to custom pipeline components and custom tokenizers see the provide data paths to custom pipeline components and custom tokenizers see the
section on [custom initialization](#initialization) for details. section on [custom initialization](#initialization) for details.
### Overwriting config settings on the command line {#config-overrides} ### Overwriting config settings on the command line {id="config-overrides"}
The config system means that you can define all settings **in one place** and in The config system means that you can define all settings **in one place** and in
a consistent format. There are no command-line arguments that need to be set, a consistent format. There are no command-line arguments that need to be set,
@ -254,7 +254,7 @@ For cases like this, you can set additional command-line options starting with
`--paths.train ./corpus/train.spacy` sets the `train` value in the `[paths]` `--paths.train ./corpus/train.spacy` sets the `train` value in the `[paths]`
block. block.
```cli ```bash
$ python -m spacy train config.cfg --paths.train ./corpus/train.spacy --paths.dev ./corpus/dev.spacy --training.batch_size 128 $ python -m spacy train config.cfg --paths.train ./corpus/train.spacy --paths.dev ./corpus/dev.spacy --training.batch_size 128
``` ```
@ -271,7 +271,7 @@ reference it across your config and override it on the CLI once.
> [`spacy train`](/api/cli#train) to make spaCy log more info, including which > [`spacy train`](/api/cli#train) to make spaCy log more info, including which
> overrides were set via the CLI and environment variables. > overrides were set via the CLI and environment variables.
#### Adding overrides via environment variables {#config-overrides-env} #### Adding overrides via environment variables {id="config-overrides-env"}
Instead of defining the overrides as CLI arguments, you can also use the Instead of defining the overrides as CLI arguments, you can also use the
`SPACY_CONFIG_OVERRIDES` environment variable using the same argument syntax. `SPACY_CONFIG_OVERRIDES` environment variable using the same argument syntax.
@ -279,11 +279,11 @@ This is especially useful if you're training models as part of an automated
process. Environment variables **take precedence** over CLI overrides and values process. Environment variables **take precedence** over CLI overrides and values
defined in the config file. defined in the config file.
```cli ```bash
$ SPACY_CONFIG_OVERRIDES="--system.gpu_allocator pytorch --training.batch_size 128" ./your_script.sh $ SPACY_CONFIG_OVERRIDES="--system.gpu_allocator pytorch --training.batch_size 128" ./your_script.sh
``` ```
### Reading from standard input {#config-stdin} ### Reading from standard input {id="config-stdin"}
Setting the config path to `-` on the command line lets you read the config from Setting the config path to `-` on the command line lets you read the config from
standard input and pipe it forward from a different process, like standard input and pipe it forward from a different process, like
@ -297,11 +297,11 @@ fly without having to save to and load from disk.
> stdout. In a custom script, you can print the string config, e.g. > stdout. In a custom script, you can print the string config, e.g.
> `print(nlp.config.to_str())`. > `print(nlp.config.to_str())`.
```cli ```bash
$ python -m spacy init config - --lang en --pipeline ner,textcat --optimize accuracy | python -m spacy train - --paths.train ./corpus/train.spacy --paths.dev ./corpus/dev.spacy $ python -m spacy init config - --lang en --pipeline ner,textcat --optimize accuracy | python -m spacy train - --paths.train ./corpus/train.spacy --paths.dev ./corpus/dev.spacy
``` ```
### Using variable interpolation {#config-interpolation} ### Using variable interpolation {id="config-interpolation"}
Another very useful feature of the config system is that it supports variable Another very useful feature of the config system is that it supports variable
interpolation for both **values and sections**. This means that you only need to interpolation for both **values and sections**. This means that you only need to
@ -310,8 +310,7 @@ define a setting once and can reference it across your config using the
the `[training]` block, and the whole block of `[training.optimizer]` is reused the `[training]` block, and the whole block of `[training.optimizer]` is reused
in `[pretraining]` and will become `pretraining.optimizer`. in `[pretraining]` and will become `pretraining.optimizer`.
```ini ```ini {title="config.cfg (excerpt)",highlight="5,18"}
### config.cfg (excerpt) {highlight="5,18"}
[system] [system]
seed = 0 seed = 0
@ -354,7 +353,7 @@ that reference this variable.
</Infobox> </Infobox>
## Preparing Training Data {#training-data} ## Preparing Training Data {id="training-data"}
Training data for NLP projects comes in many different formats. For some common Training data for NLP projects comes in many different formats. For some common
formats such as CoNLL, spaCy provides [converters](/api/cli#convert) you can use formats such as CoNLL, spaCy provides [converters](/api/cli#convert) you can use
@ -372,8 +371,7 @@ are handled automatically.
Here's an example of creating a `.spacy` file from some NER annotations. Here's an example of creating a `.spacy` file from some NER annotations.
```python ```python {title="preprocess.py"}
### preprocess.py
import spacy import spacy
from spacy.tokens import DocBin from spacy.tokens import DocBin
@ -407,9 +405,9 @@ convert your data to JSON before creating a `.spacy` file.
</Accordion> </Accordion>
## Customizing the pipeline and training {#config-custom} ## Customizing the pipeline and training {id="config-custom"}
### Defining pipeline components {#config-components} ### Defining pipeline components {id="config-components"}
You typically train a [pipeline](/usage/processing-pipelines) of **one or more You typically train a [pipeline](/usage/processing-pipelines) of **one or more
components**. The `[components]` block in the config defines the available components**. The `[components]` block in the config defines the available
@ -438,8 +436,7 @@ existing weights. This lets you include an already trained component in your
pipeline, or update a trained component with more data specific to your use pipeline, or update a trained component with more data specific to your use
case. case.
```ini ```ini {title="config.cfg (excerpt)"}
### config.cfg (excerpt)
[components] [components]
# "parser" and "ner" are sourced from a trained pipeline # "parser" and "ner" are sourced from a trained pipeline
@ -518,7 +515,7 @@ replace_listeners = ["model.tok2vec"]
</Infobox> </Infobox>
### Using predictions from preceding components {#annotating-components new="3.1"} ### Using predictions from preceding components {id="annotating-components",version="3.1"}
By default, components are updated in isolation during training, which means By default, components are updated in isolation during training, which means
that they don't see the predictions of any earlier components in the pipeline. A that they don't see the predictions of any earlier components in the pipeline. A
@ -532,8 +529,7 @@ list of components. For example, the feature `DEP` from the parser could be used
as a tagger feature by including `DEP` in the tok2vec `attrs` and including as a tagger feature by including `DEP` in the tok2vec `attrs` and including
`parser` in `annotating_components`: `parser` in `annotating_components`:
```ini ```ini {title="config.cfg (excerpt)",highlight="7,12"}
### config.cfg (excerpt) {highlight="7,12"}
[nlp] [nlp]
pipeline = ["parser", "tagger"] pipeline = ["parser", "tagger"]
@ -555,8 +551,7 @@ pipeline is run. The config excerpt below shows how a frozen `ner` component and
a `sentencizer` can provide the required `doc.sents` and `doc.ents` for the a `sentencizer` can provide the required `doc.sents` and `doc.ents` for the
entity linker during training: entity linker during training:
```ini ```ini {title="config.cfg (excerpt)"}
### config.cfg (excerpt)
[nlp] [nlp]
pipeline = ["sentencizer", "ner", "entity_linker"] pipeline = ["sentencizer", "ner", "entity_linker"]
@ -580,7 +575,7 @@ now-updated model to the predicted docs.
</Infobox> </Infobox>
### Using registered functions {#config-functions} ### Using registered functions {id="config-functions"}
The training configuration defined in the config file doesn't have to only The training configuration defined in the config file doesn't have to only
consist of static values. Some settings can also be **functions**. For instance, consist of static values. Some settings can also be **functions**. For instance,
@ -588,8 +583,7 @@ the `batch_size` can be a number that doesn't change, or a schedule, like a
sequence of compounding values, which has shown to be an effective trick (see sequence of compounding values, which has shown to be an effective trick (see
[Smith et al., 2017](https://arxiv.org/abs/1711.00489)). [Smith et al., 2017](https://arxiv.org/abs/1711.00489)).
```ini ```ini {title="With static value"}
### With static value
[training] [training]
batch_size = 128 batch_size = 128
``` ```
@ -612,8 +606,7 @@ from your configs.
> instance, a learning rate schedule can be provided as the an argument of an > instance, a learning rate schedule can be provided as the an argument of an
> optimizer. > optimizer.
```ini ```ini {title="With registered function"}
### With registered function
[training.batch_size] [training.batch_size]
@schedules = "compounding.v1" @schedules = "compounding.v1"
start = 100 start = 100
@ -621,7 +614,7 @@ stop = 1000
compound = 1.001 compound = 1.001
``` ```
### Model architectures {#model-architectures} ### Model architectures {id="model-architectures"}
> #### 💡 Model type annotations > #### 💡 Model type annotations
> >
@ -659,7 +652,7 @@ different tasks. For example:
| [TransitionBasedParser](/api/architectures#TransitionBasedParser) | Build a [transition-based parser](https://explosion.ai/blog/parsing-english-in-python) model used in the default [`EntityRecognizer`](/api/entityrecognizer) and [`DependencyParser`](/api/dependencyparser). ~~Model[List[Docs], List[List[Floats2d]]]~~ | | [TransitionBasedParser](/api/architectures#TransitionBasedParser) | Build a [transition-based parser](https://explosion.ai/blog/parsing-english-in-python) model used in the default [`EntityRecognizer`](/api/entityrecognizer) and [`DependencyParser`](/api/dependencyparser). ~~Model[List[Docs], List[List[Floats2d]]]~~ |
| [TextCatEnsemble](/api/architectures#TextCatEnsemble) | Stacked ensemble of a bag-of-words model and a neural network model with an internal CNN embedding layer. Used in the default [`TextCategorizer`](/api/textcategorizer). ~~Model[List[Doc], Floats2d]~~ | | [TextCatEnsemble](/api/architectures#TextCatEnsemble) | Stacked ensemble of a bag-of-words model and a neural network model with an internal CNN embedding layer. Used in the default [`TextCategorizer`](/api/textcategorizer). ~~Model[List[Doc], Floats2d]~~ |
### Metrics, training output and weighted scores {#metrics} ### Metrics, training output and weighted scores {id="metrics"}
When you train a pipeline using the [`spacy train`](/api/cli#train) command, When you train a pipeline using the [`spacy train`](/api/cli#train) command,
you'll see a table showing the metrics after each pass over the data. The you'll see a table showing the metrics after each pass over the data. The
@ -721,7 +714,7 @@ still look good.
</Accordion> </Accordion>
## Custom functions {#custom-functions} ## Custom functions {id="custom-functions"}
Registered functions in the training config files can refer to built-in Registered functions in the training config files can refer to built-in
implementations, but you can also plug in fully **custom implementations**. All implementations, but you can also plug in fully **custom implementations**. All
@ -740,14 +733,14 @@ Each custom function can have any number of arguments that are passed in via the
given parameter is always explicitly set in the config, avoid setting a default given parameter is always explicitly set in the config, avoid setting a default
value for it. value for it.
### Training with custom code {#custom-code} ### Training with custom code {id="custom-code"}
> ```cli > ```bash
> ### Training > ### Training
> $ python -m spacy train config.cfg --code functions.py > $ python -m spacy train config.cfg --code functions.py
> ``` > ```
> >
> ```cli > ```bash
> ### Packaging > ### Packaging
> $ python -m spacy package ./model-best ./packages --code functions.py > $ python -m spacy package ./model-best ./packages --code functions.py
> ``` > ```
@ -765,7 +758,7 @@ any custom architectures, functions or
your pipeline and registered when it's loaded. See the documentation on your pipeline and registered when it's loaded. See the documentation on
[saving and loading pipelines](/usage/saving-loading#models-custom) for details. [saving and loading pipelines](/usage/saving-loading#models-custom) for details.
#### Example: Modifying the nlp object {#custom-code-nlp-callbacks} #### Example: Modifying the nlp object {id="custom-code-nlp-callbacks"}
For many use cases, you don't necessarily want to implement the whole `Language` For many use cases, you don't necessarily want to implement the whole `Language`
subclass and language data from scratch it's often enough to make a few small subclass and language data from scratch it's often enough to make a few small
@ -799,8 +792,7 @@ stop word to the defaults:
> @callbacks = "customize_language_data" > @callbacks = "customize_language_data"
> ``` > ```
```python ```python {title="functions.py",highlight="3,6"}
### functions.py {highlight="3,6"}
import spacy import spacy
@spacy.registry.callbacks("customize_language_data") @spacy.registry.callbacks("customize_language_data")
@ -836,8 +828,7 @@ we're adding the arguments `extra_stop_words` (a list of strings) and `debug`
> debug = true > debug = true
> ``` > ```
```python ```python {title="functions.py",highlight="5,7-9"}
### functions.py {highlight="5,7-9"}
from typing import List from typing import List
import spacy import spacy
@ -873,11 +864,11 @@ you can now run [`spacy train`](/api/cli#train) and point the argument `--code`
to your Python file. Before loading the config, spaCy will import the to your Python file. Before loading the config, spaCy will import the
`functions.py` module and your custom functions will be registered. `functions.py` module and your custom functions will be registered.
```cli ```bash
$ python -m spacy train config.cfg --output ./output --code ./functions.py $ python -m spacy train config.cfg --output ./output --code ./functions.py
``` ```
#### Example: Modifying tokenizer settings {#custom-tokenizer} #### Example: Modifying tokenizer settings {id="custom-tokenizer"}
Use the `initialize.before_init` callback to modify the tokenizer settings when Use the `initialize.before_init` callback to modify the tokenizer settings when
training a new pipeline. Write a registered callback that modifies the tokenizer training a new pipeline. Write a registered callback that modifies the tokenizer
@ -892,8 +883,7 @@ settings and specify this callback in your config:
> @callbacks = "customize_tokenizer" > @callbacks = "customize_tokenizer"
> ``` > ```
```python ```python {title="functions.py"}
### functions.py
from spacy.util import registry, compile_suffix_regex from spacy.util import registry, compile_suffix_regex
@registry.callbacks("customize_tokenizer") @registry.callbacks("customize_tokenizer")
@ -912,7 +902,7 @@ def make_customize_tokenizer():
When training, provide the function above with the `--code` option: When training, provide the function above with the `--code` option:
```cli ```bash
$ python -m spacy train config.cfg --code ./functions.py $ python -m spacy train config.cfg --code ./functions.py
``` ```
@ -936,7 +926,7 @@ disk.
</Infobox> </Infobox>
#### Example: Custom logging function {#custom-logging} #### Example: Custom logging function {id="custom-logging"}
During training, the results of each step are passed to a logger function. By During training, the results of each step are passed to a logger function. By
default, these results are written to the console with the default, these results are written to the console with the
@ -966,8 +956,7 @@ tabular results to a file:
> log_path = "my_file.tab" > log_path = "my_file.tab"
> ``` > ```
```python ```python {title="functions.py"}
### functions.py
import sys import sys
from typing import IO, Tuple, Callable, Dict, Any, Optional from typing import IO, Tuple, Callable, Dict, Any, Optional
import spacy import spacy
@ -1005,7 +994,7 @@ def custom_logger(log_path):
return setup_logger return setup_logger
``` ```
#### Example: Custom batch size schedule {#custom-code-schedule} #### Example: Custom batch size schedule {id="custom-code-schedule"}
You can also implement your own batch size schedule to use during training. The You can also implement your own batch size schedule to use during training. The
`@spacy.registry.schedules` decorator lets you register that function in the `@spacy.registry.schedules` decorator lets you register that function in the
@ -1019,8 +1008,7 @@ You can also implement your own batch size schedule to use during training. The
> know that a config referencing `v1` means a different function than a config > know that a config referencing `v1` means a different function than a config
> referencing `v2`. > referencing `v2`.
```python ```python {title="functions.py"}
### functions.py
import spacy import spacy
@spacy.registry.schedules("my_custom_schedule.v1") @spacy.registry.schedules("my_custom_schedule.v1")
@ -1037,15 +1025,14 @@ settings in the block will be passed to the function as keyword arguments. Keep
in mind that the config shouldn't have any hidden defaults and all arguments on in mind that the config shouldn't have any hidden defaults and all arguments on
the functions need to be represented in the config. the functions need to be represented in the config.
```ini ```ini {title="config.cfg (excerpt)"}
### config.cfg (excerpt)
[training.batch_size] [training.batch_size]
@schedules = "my_custom_schedule.v1" @schedules = "my_custom_schedule.v1"
start = 2 start = 2
factor = 1.005 factor = 1.005
``` ```
### Defining custom architectures {#custom-architectures} ### Defining custom architectures {id="custom-architectures"}
Built-in pipeline components such as the tagger or named entity recognizer are Built-in pipeline components such as the tagger or named entity recognizer are
constructed with default neural network [models](/api/architectures). You can constructed with default neural network [models](/api/architectures). You can
@ -1064,8 +1051,7 @@ for more details.
> output_width = 512 > output_width = 512
> ``` > ```
```python ```python {title="functions.py"}
### functions.py
from typing import List from typing import List
from thinc.types import Floats2d from thinc.types import Floats2d
from thinc.api import Model from thinc.api import Model
@ -1077,7 +1063,7 @@ def custom_neural_network(output_width: int) -> Model[List[Doc], List[Floats2d]]
return create_model(output_width) return create_model(output_width)
``` ```
## Customizing the initialization {#initialization} ## Customizing the initialization {id="initialization"}
When you start training a new model from scratch, When you start training a new model from scratch,
[`spacy train`](/api/cli#train) will call [`spacy train`](/api/cli#train) will call
@ -1120,7 +1106,7 @@ because the component settings required for training (load data from an external
file) wouldn't match the component settings required at runtime (load what's file) wouldn't match the component settings required at runtime (load what's
included with the saved `nlp` object and don't depend on external file). included with the saved `nlp` object and don't depend on external file).
![Illustration of pipeline lifecycle](../images/lifecycle.svg) ![Illustration of pipeline lifecycle](/images/lifecycle.svg)
<Infobox title="How components save and load data" emoji="📖"> <Infobox title="How components save and load data" emoji="📖">
@ -1131,7 +1117,7 @@ initialization is implemented under the hood, see the usage guide on
</Infobox> </Infobox>
#### Initializing labels {#initialization-labels} #### Initializing labels {id="initialization-labels"}
Built-in pipeline components like the Built-in pipeline components like the
[`EntityRecognizer`](/api/entityrecognizer) or [`EntityRecognizer`](/api/entityrecognizer) or
@ -1157,7 +1143,7 @@ allow them to initialize faster.
> path = "corpus/labels/ner.json > path = "corpus/labels/ner.json
> ``` > ```
```cli ```bash
$ python -m spacy init labels config.cfg ./corpus --paths.train ./corpus/train.spacy $ python -m spacy init labels config.cfg ./corpus --paths.train ./corpus/train.spacy
``` ```
@ -1174,20 +1160,20 @@ should always let spaCy **auto-generate the labels** for you.
</Infobox> </Infobox>
## Data utilities {#data} ## Data utilities {id="data"}
spaCy includes various features and utilities to make it easy to train models spaCy includes various features and utilities to make it easy to train models
using your own data, manage training and evaluation corpora, convert existing using your own data, manage training and evaluation corpora, convert existing
annotations and configure data augmentation strategies for more robust models. annotations and configure data augmentation strategies for more robust models.
### Converting existing corpora and annotations {#data-convert} ### Converting existing corpora and annotations {id="data-convert"}
If you have training data in a standard format like `.conll` or `.conllu`, the If you have training data in a standard format like `.conll` or `.conllu`, the
easiest way to convert it for use with spaCy is to run easiest way to convert it for use with spaCy is to run
[`spacy convert`](/api/cli#convert) and pass it a file and an output directory. [`spacy convert`](/api/cli#convert) and pass it a file and an output directory.
By default, the command will pick the converter based on the file extension. By default, the command will pick the converter based on the file extension.
```cli ```bash
$ python -m spacy convert ./train.gold.conll ./corpus $ python -m spacy convert ./train.gold.conll ./corpus
``` ```
@ -1215,8 +1201,7 @@ especially when packing multiple documents together. You can also create `Doc`
objects manually, so you can write your own custom logic to convert and store objects manually, so you can write your own custom logic to convert and store
existing annotations for use in spaCy. existing annotations for use in spaCy.
```python ```python {title="Training data from Doc objects",highlight="6-9"}
### Training data from Doc objects {highlight="6-9"}
import spacy import spacy
from spacy.tokens import Doc, DocBin from spacy.tokens import Doc, DocBin
@ -1230,7 +1215,7 @@ docbin.add(doc)
docbin.to_disk("./train.spacy") docbin.to_disk("./train.spacy")
``` ```
### Working with corpora {#data-corpora} ### Working with corpora {id="data-corpora"}
> #### Example > #### Example
> >
@ -1271,7 +1256,7 @@ By default, the training data is loaded into memory and shuffled before each
epoch. If the corpus is **too large to fit into memory** during training, stream epoch. If the corpus is **too large to fit into memory** during training, stream
the corpus using a custom reader as described in the next section. the corpus using a custom reader as described in the next section.
### Custom data reading and batching {#custom-code-readers-batchers} ### Custom data reading and batching {id="custom-code-readers-batchers"}
Some use-cases require **streaming in data** or manipulating datasets on the Some use-cases require **streaming in data** or manipulating datasets on the
fly, rather than generating all data beforehand and storing it to disk. Instead fly, rather than generating all data beforehand and storing it to disk. Instead
@ -1296,8 +1281,7 @@ as **config settings** in this case, `source`.
> source = "s3://your_bucket/path/data.csv" > source = "s3://your_bucket/path/data.csv"
> ``` > ```
```python ```python {title="functions.py",highlight="7-8"}
### functions.py {highlight="7-8"}
from typing import Callable, Iterator, List from typing import Callable, Iterator, List
import spacy import spacy
from spacy.training import Example from spacy.training import Example
@ -1354,8 +1338,7 @@ training should stop.
> max_steps = 2000 > max_steps = 2000
> ``` > ```
```python ```python {title="functions.py"}
### functions.py
from typing import Callable, Iterable, Iterator from typing import Callable, Iterable, Iterator
from spacy import util from spacy import util
import random import random
@ -1418,8 +1401,7 @@ annotations are the same.
> size = 150 > size = 150
> ``` > ```
```python ```python {title="functions.py"}
### functions.py
from typing import Callable, Iterable, Iterator, List from typing import Callable, Iterable, Iterator, List
import spacy import spacy
from spacy.training import Example from spacy.training import Example
@ -1439,12 +1421,9 @@ def filter_batch(size: int) -> Callable[[Iterable[Example]], Iterator[List[Examp
return create_filtered_batches return create_filtered_batches
``` ```
<!-- TODO: {/* TODO: Custom corpus class, Minibatching */}
* Custom corpus class
* Minibatching
-->
### Data augmentation {#data-augmentation} ### Data augmentation {id="data-augmentation"}
Data augmentation is the process of applying small **modifications** to the Data augmentation is the process of applying small **modifications** to the
training data. It can be especially useful for punctuation and case replacement training data. It can be especially useful for punctuation and case replacement
@ -1458,8 +1437,7 @@ your config. The built-in [`orth_variants`](/api/top-level#orth_variants)
augmenter creates a data augmentation callback that uses orth-variant augmenter creates a data augmentation callback that uses orth-variant
replacement. replacement.
```ini ```ini {title="config.cfg (excerpt)",highlight="8,14"}
### config.cfg (excerpt) {highlight="8,14"}
[corpora.train] [corpora.train]
@readers = "spacy.Corpus.v1" @readers = "spacy.Corpus.v1"
path = ${paths.train} path = ${paths.train}
@ -1483,12 +1461,18 @@ typically loaded from a JSON file. There are two types of orth variant rules:
`"single"` for single tokens that should be replaced (e.g. hyphens) and `"single"` for single tokens that should be replaced (e.g. hyphens) and
`"paired"` for pairs of tokens (e.g. quotes). `"paired"` for pairs of tokens (e.g. quotes).
<!-- prettier-ignore --> ```json {title="orth_variants.json"}
```json
### orth_variants.json
{ {
"single": [{ "tags": ["NFP"], "variants": ["…", "..."] }], "single": [{ "tags": ["NFP"], "variants": ["…", "..."] }],
"paired": [{ "tags": ["``", "''"], "variants": [["'", "'"], ["", ""]] }] "paired": [
{
"tags": ["``", "''"],
"variants": [
["'", "'"],
["", ""]
]
}
]
} }
``` ```
@ -1511,7 +1495,7 @@ to apply it to the **training corpus**, not the development data.
</Infobox> </Infobox>
#### Writing custom data augmenters {#data-augmentation-custom} #### Writing custom data augmenters {id="data-augmentation-custom"}
Using the [`@spacy.augmenters`](/api/top-level#registry) registry, you can also Using the [`@spacy.augmenters`](/api/top-level#registry) registry, you can also
register your own data augmentation callbacks. The callback should be a function register your own data augmentation callbacks. The callback should be a function
@ -1572,7 +1556,78 @@ token-based annotations like the dependency parse or entity labels, you'll need
to take care to adjust the `Example` object so its annotations match and remain to take care to adjust the `Example` object so its annotations match and remain
valid. valid.
## Internal training API {#api} ## Parallel & distributed training with Ray {id="parallel-training"}
> #### Installation
>
> ```bash
> $ pip install -U %%SPACY_PKG_NAME[ray]%%SPACY_PKG_FLAGS
> # Check that the CLI is registered
> $ python -m spacy ray --help
> ```
[Ray](https://ray.io/) is a fast and simple framework for building and running
**distributed applications**. You can use Ray to train spaCy on one or more
remote machines, potentially speeding up your training process. Parallel
training won't always be faster though it depends on your batch size, models,
and hardware.
<Infobox variant="warning">
To use Ray with spaCy, you need the
[`spacy-ray`](https://github.com/explosion/spacy-ray) package installed.
Installing the package will automatically add the `ray` command to the spaCy
CLI.
</Infobox>
The [`spacy ray train`](/api/cli#ray-train) command follows the same API as
[`spacy train`](/api/cli#train), with a few extra options to configure the Ray
setup. You can optionally set the `--address` option to point to your Ray
cluster. If it's not set, Ray will run locally.
```bash
python -m spacy ray train config.cfg --n-workers 2
```
<Project id="integrations/ray">
Get started with parallel training using our project template. It trains a
simple model on a Universal Dependencies Treebank and lets you parallelize the
training with Ray.
</Project>
### How parallel training works {id="parallel-training-details"}
Each worker receives a shard of the **data** and builds a copy of the **model
and optimizer** from the [`config.cfg`](#config). It also has a communication
channel to **pass gradients and parameters** to the other workers. Additionally,
each worker is given ownership of a subset of the parameter arrays. Every
parameter array is owned by exactly one worker, and the workers are given a
mapping so they know which worker owns which parameter.
![Illustration of setup](/images/spacy-ray.svg)
As training proceeds, every worker will be computing gradients for **all** of
the model parameters. When they compute gradients for parameters they don't own,
they'll **send them to the worker** that does own that parameter, along with a
version identifier so that the owner can decide whether to discard the gradient.
Workers use the gradients they receive and the ones they compute locally to
update the parameters they own, and then broadcast the updated array and a new
version ID to the other workers.
This training procedure is **asynchronous** and **non-blocking**. Workers always
push their gradient increments and parameter updates, they do not have to pull
them and block on the result, so the transfers can happen in the background,
overlapped with the actual training work. The workers also do not have to stop
and wait for each other ("synchronize") at the start of each batch. This is very
useful for spaCy, because spaCy is often trained on long documents, which means
**batches can vary in size** significantly. Uneven workloads make synchronous
gradient descent inefficient, because if one batch is slow, all of the other
workers are stuck waiting for it to complete before they can continue.
## Internal training API {id="api"}
<Infobox variant="danger"> <Infobox variant="danger">
@ -1586,7 +1641,7 @@ typically give you everything you need to train fully custom pipelines with
</Infobox> </Infobox>
### Training from a Python script {#api-train new="3.2"} ### Training from a Python script {id="api-train",version="3.2"}
If you want to run the training from a Python script instead of using the If you want to run the training from a Python script instead of using the
[`spacy train`](/api/cli#train) CLI command, you can call into the [`spacy train`](/api/cli#train) CLI command, you can call into the
@ -1600,7 +1655,7 @@ from spacy.cli.train import train
train("./config.cfg", overrides={"paths.train": "./train.spacy", "paths.dev": "./dev.spacy"}) train("./config.cfg", overrides={"paths.train": "./train.spacy", "paths.dev": "./dev.spacy"})
``` ```
### Internal training loop API {#api-loop} ### Internal training loop API {id="api-loop"}
<Infobox variant="warning"> <Infobox variant="warning">
@ -1691,8 +1746,7 @@ of being dropped.
> - [`nlp.to_disk`](/api/language#to_disk): Save the updated pipeline to a > - [`nlp.to_disk`](/api/language#to_disk): Save the updated pipeline to a
> directory. > directory.
```python ```python {title="Example training loop"}
### Example training loop
optimizer = nlp.initialize() optimizer = nlp.initialize()
for itn in range(100): for itn in range(100):
random.shuffle(train_data) random.shuffle(train_data)

View File

@ -6,7 +6,7 @@ menu:
- ['Backwards Incompatibilities', 'incompat'] - ['Backwards Incompatibilities', 'incompat']
--- ---
## New Features {#features hidden="true"} ## New Features {id="features",hidden="true"}
spaCy v2.1 has focussed primarily on stability and performance, solidifying the spaCy v2.1 has focussed primarily on stability and performance, solidifying the
design changes introduced in [v2.0](/usage/v2). As well as smaller models, design changes introduced in [v2.0](/usage/v2). As well as smaller models,
@ -16,7 +16,7 @@ for some exciting new NLP innovations. For the full changelog, see the
For more details and a behind-the-scenes look at the new release, For more details and a behind-the-scenes look at the new release,
[see our blog post](https://explosion.ai/blog/spacy-v2-1). [see our blog post](https://explosion.ai/blog/spacy-v2-1).
### BERT/ULMFit/Elmo-style pre-training {#pretraining tag="experimental"} ### BERT/ULMFit/Elmo-style pre-training {id="pretraining",tag="experimental"}
> #### Example > #### Example
> >
@ -41,7 +41,7 @@ it.
</Infobox> </Infobox>
### Extended match pattern API {#matcher-api} ### Extended match pattern API {id="matcher-api"}
> #### Example > #### Example
> >
@ -69,7 +69,7 @@ values.
</Infobox> </Infobox>
### Easy rule-based entity recognition {#entity-ruler} ### Easy rule-based entity recognition {id="entity-ruler"}
> #### Example > #### Example
> >
@ -93,7 +93,7 @@ flexibility.
</Infobox> </Infobox>
### Phrase matching with other attributes {#phrasematcher} ### Phrase matching with other attributes {id="phrasematcher"}
> #### Example > #### Example
> >
@ -117,7 +117,7 @@ or `POS` for finding sequences of the same part-of-speech tags.
</Infobox> </Infobox>
### Retokenizer for merging and splitting {#retokenizer} ### Retokenizer for merging and splitting {id="retokenizer"}
> #### Example > #### Example
> >
@ -144,7 +144,7 @@ deprecated.
</Infobox> </Infobox>
### Components and languages via entry points {#entry-points} ### Components and languages via entry points {id="entry-points"}
> #### Example > #### Example
> >
@ -171,7 +171,7 @@ is required.
</Infobox> </Infobox>
### Improved documentation {#docs} ### Improved documentation {id="docs"}
Although it looks pretty much the same, we've rebuilt the entire documentation Although it looks pretty much the same, we've rebuilt the entire documentation
using [Gatsby](https://www.gatsbyjs.org/) and [MDX](https://mdxjs.com/). It's using [Gatsby](https://www.gatsbyjs.org/) and [MDX](https://mdxjs.com/). It's
@ -180,9 +180,9 @@ entirely **in Markdown**, without having to compromise on easy-to-use custom UI
components. We're hoping that the Markdown source will make it even easier to components. We're hoping that the Markdown source will make it even easier to
contribute to the documentation. For more details, check out the contribute to the documentation. For more details, check out the
[styleguide](/styleguide) and [styleguide](/styleguide) and
[source](https://github.com/explosion/spacy/tree/v2.x/website). While [source](https://github.com/explosion/spacy/tree/v2.x/website). While converting
converting the pages to Markdown, we've also fixed a bunch of typos, improved the pages to Markdown, we've also fixed a bunch of typos, improved the existing
the existing pages and added some new content: pages and added some new content:
- **Usage Guide:** [Rule-based Matching](/usage/rule-based-matching)<br/>How to - **Usage Guide:** [Rule-based Matching](/usage/rule-based-matching)<br/>How to
use the `Matcher`, `PhraseMatcher` and the new `EntityRuler`, and write use the `Matcher`, `PhraseMatcher` and the new `EntityRuler`, and write
@ -200,7 +200,7 @@ the existing pages and added some new content:
- **API:** [`Sentencizer`](/api/sentencizer) - **API:** [`Sentencizer`](/api/sentencizer)
- **API:** [Pipeline functions](/api/pipeline-functions) - **API:** [Pipeline functions](/api/pipeline-functions)
## Backwards incompatibilities {#incompat} ## Backwards incompatibilities {id="incompat"}
<Infobox title="Important note on models" variant="warning"> <Infobox title="Important note on models" variant="warning">

View File

@ -7,7 +7,7 @@ menu:
- ['Migrating from v2.1', 'migrating'] - ['Migrating from v2.1', 'migrating']
--- ---
## New Features {#features hidden="true"} ## New Features {id="features",hidden="true"}
spaCy v2.2 features improved statistical models, new pretrained models for spaCy v2.2 features improved statistical models, new pretrained models for
Norwegian and Lithuanian, better Dutch NER, as well as a new mechanism for Norwegian and Lithuanian, better Dutch NER, as well as a new mechanism for
@ -22,7 +22,7 @@ full changelog, see the
For more details and a behind-the-scenes look at the new release, For more details and a behind-the-scenes look at the new release,
[see our blog post](https://explosion.ai/blog/spacy-v2-2). [see our blog post](https://explosion.ai/blog/spacy-v2-2).
### Better pretrained models and more languages {#models} ### Better pretrained models and more languages {id="models"}
> #### Example > #### Example
> >
@ -46,7 +46,7 @@ overall. We've also added new core models for [Norwegian](/models/nb) (MIT) and
</Infobox> </Infobox>
### Text classification scores and CLI training {#train-textcat-cli} ### Text classification scores and CLI training {id="train-textcat-cli"}
> #### Example > #### Example
> >
@ -106,7 +106,7 @@ processing.
</Infobox> </Infobox>
### Serializable lookup tables and smaller installation {#lookups} ### Serializable lookup tables and smaller installation {id="lookups"}
> #### Example > #### Example
> >
@ -140,7 +140,7 @@ languages that don't yet ship with pretrained models.
</Infobox> </Infobox>
### CLI command to debug and validate training data {#debug-data} ### CLI command to debug and validate training data {id="debug-data"}
> #### Example > #### Example
> >
@ -300,7 +300,7 @@ will not be available.
</Infobox> </Infobox>
## Backwards incompatibilities {#incompat} ## Backwards incompatibilities {id="incompat"}
<Infobox title="Important note on models" variant="warning"> <Infobox title="Important note on models" variant="warning">
@ -379,7 +379,7 @@ check if all of your models are up to date, you can run the
list of dicts. This is mostly internals, but if your code used list of dicts. This is mostly internals, but if your code used
`nlp.meta["sources"]`, you might have to update it. `nlp.meta["sources"]`, you might have to update it.
### Migrating from spaCy 2.1 {#migrating} ### Migrating from spaCy 2.1 {id="migrating"}
#### Lemmatization data and lookup tables #### Lemmatization data and lookup tables
@ -389,8 +389,7 @@ with only tokenizers, you now need to install that data explicitly via
setup is required the package just needs to be installed in the same setup is required the package just needs to be installed in the same
environment as spaCy. environment as spaCy.
```python ```python {highlight="3-4"}
### {highlight="3-4"}
nlp = Turkish() nlp = Turkish()
doc = nlp("Bu bir cümledir.") doc = nlp("Bu bir cümledir.")
# 🚨 This now requires the lookups data to be installed explicitly # 🚨 This now requires the lookups data to be installed explicitly

View File

@ -7,7 +7,7 @@ menu:
- ['Migrating from v2.2', 'migrating'] - ['Migrating from v2.2', 'migrating']
--- ---
## New Features {#features hidden="true"} ## New Features {id="features",hidden="true"}
spaCy v2.3 features new pretrained models for five languages, word vectors for spaCy v2.3 features new pretrained models for five languages, word vectors for
all language models, and decreased model size and loading times for models with all language models, and decreased model size and loading times for models with
@ -19,7 +19,7 @@ Model packages with vectors are about **2&times** smaller on disk and load
For more details and a behind-the-scenes look at the new release, For more details and a behind-the-scenes look at the new release,
[see our blog post](https://explosion.ai/blog/spacy-v2-3). [see our blog post](https://explosion.ai/blog/spacy-v2-3).
### Expanded model families with vectors {#models} ### Expanded model families with vectors {id="models"}
> #### Example > #### Example
> >
@ -45,7 +45,7 @@ and Dutch has been extended to include both UD Dutch Alpino and LassySmall.
</Infobox> </Infobox>
### Chinese {#chinese} ### Chinese {id="chinese"}
> #### Example > #### Example
> >
@ -77,7 +77,7 @@ Python 3.8.
</Infobox> </Infobox>
### Japanese {#japanese} ### Japanese {id="japanese"}
The updated Japanese language class switches to The updated Japanese language class switches to
[`SudachiPy`](https://github.com/WorksApplications/SudachiPy) for word [`SudachiPy`](https://github.com/WorksApplications/SudachiPy) for word
@ -102,7 +102,7 @@ installing spaCy for Japanese, which is now possible with a single command:
- [`spacy train`](/api/cli#train) on GPU restricts the CPU timing evaluation to - [`spacy train`](/api/cli#train) on GPU restricts the CPU timing evaluation to
the first iteration the first iteration
## Backwards incompatibilities {#incompat} ## Backwards incompatibilities {id="incompat"}
<Infobox title="Important note on models" variant="warning"> <Infobox title="Important note on models" variant="warning">
@ -140,7 +140,7 @@ the [`spacy validate`](/api/cli#validate) command.
spaCy's tokenization and Universal Dependencies multi-word tokens used for spaCy's tokenization and Universal Dependencies multi-word tokens used for
contractions. contractions.
### Migrating from spaCy 2.2 {#migrating} ### Migrating from spaCy 2.2 {id="migrating"}
#### Tokenizer settings #### Tokenizer settings

View File

@ -17,7 +17,7 @@ spaCy, or just want to brush up on some NLP basics and the details of the
library, check out the [spaCy 101 guide](/usage/spacy-101) that explains the library, check out the [spaCy 101 guide](/usage/spacy-101) that explains the
most important concepts with examples and illustrations. most important concepts with examples and illustrations.
## Summary {#summary} ## Summary {id="summary"}
<Grid cols={2}> <Grid cols={2}>
@ -73,12 +73,12 @@ differentiable. Even if you don't have explicitly annotated data, you can update
spaCy using all the **latest deep learning tricks** like adversarial training, spaCy using all the **latest deep learning tricks** like adversarial training,
noise contrastive estimation or reinforcement learning. noise contrastive estimation or reinforcement learning.
## New features {#features} ## New features {id="features"}
This section contains an overview of the most important **new features and This section contains an overview of the most important **new features and
improvements**. The [API docs](/api) include additional deprecation notes. improvements**. The [API docs](/api) include additional deprecation notes.
### Convolutional neural network models {#features-models} ### Convolutional neural network models {id="features-models"}
> #### Example > #### Example
> >
@ -109,7 +109,7 @@ process.
</Infobox> </Infobox>
### Improved processing pipelines {#features-pipelines} ### Improved processing pipelines {id="features-pipelines"}
> #### Example > #### Example
> >
@ -130,7 +130,7 @@ write any **attributes, properties and methods** to the `Doc`, `Token` and
`Span`. You can add data, implement new features, integrate other libraries with `Span`. You can add data, implement new features, integrate other libraries with
spaCy or plug in your own machine learning models. spaCy or plug in your own machine learning models.
![The processing pipeline](../images/pipeline.svg) ![The processing pipeline](/images/pipeline.svg)
<Infobox> <Infobox>
@ -143,7 +143,7 @@ spaCy or plug in your own machine learning models.
</Infobox> </Infobox>
### Text classification {#features-text-classification} ### Text classification {id="features-text-classification"}
> #### Example > #### Example
> >
@ -172,7 +172,7 @@ network to assign position-sensitive vectors to each word in the document.
</Infobox> </Infobox>
### Hash values instead of integer IDs {#features-hash-ids} ### Hash values instead of integer IDs {id="features-hash-ids"}
> #### Example > #### Example
> >
@ -201,7 +201,7 @@ available via `token.orth`.
</Infobox> </Infobox>
### Improved word vectors support {#features-vectors} ### Improved word vectors support {id="features-vectors"}
> #### Example > #### Example
> >
@ -231,7 +231,7 @@ you set the `--prune-vectors` flag. Otherwise, you can use the new
</Infobox> </Infobox>
### Saving, loading and serialization {#features-serializer} ### Saving, loading and serialization {id="features-serializer"}
> #### Example > #### Example
> >
@ -266,7 +266,7 @@ the class directly, e.g. `from spacy.lang.en import English` or use
</Infobox> </Infobox>
### displaCy visualizer with Jupyter support {#features-displacy} ### displaCy visualizer with Jupyter support {id="features-displacy"}
> #### Example > #### Example
> >
@ -291,7 +291,7 @@ notebook.
</Infobox> </Infobox>
### Improved language data and lazy loading {#features-language} ### Improved language data and lazy loading {id="features-language"}
Language-specific data now lives in its own submodule, `spacy.lang`. Languages Language-specific data now lives in its own submodule, `spacy.lang`. Languages
are lazy-loaded, i.e. only loaded when you import a `Language` class, or load a are lazy-loaded, i.e. only loaded when you import a `Language` class, or load a
@ -308,7 +308,7 @@ lookup-based lemmatization and **many new languages**!
</Infobox> </Infobox>
### Revised matcher API and phrase matcher {#features-matcher} ### Revised matcher API and phrase matcher {id="features-matcher"}
> #### Example > #### Example
> >
@ -338,7 +338,7 @@ patterns.
</Infobox> </Infobox>
## Backwards incompatibilities {#incompat} ## Backwards incompatibilities {id="incompat"}
The following modules, classes and methods have changed between v1.x and v2.0. The following modules, classes and methods have changed between v1.x and v2.0.
@ -373,7 +373,7 @@ The following modules, classes and methods have changed between v1.x and v2.0.
| `Doc.read_bytes` | [`Doc.to_bytes`](/api/doc#to_bytes) [`Doc.from_bytes`](/api/doc#from_bytes) [`Doc.to_disk`](/api/doc#to_disk) [`Doc.from_disk`](/api/doc#from_disk) | | `Doc.read_bytes` | [`Doc.to_bytes`](/api/doc#to_bytes) [`Doc.from_bytes`](/api/doc#from_bytes) [`Doc.to_disk`](/api/doc#to_disk) [`Doc.from_disk`](/api/doc#from_disk) |
| `Token.is_ancestor_of` | [`Token.is_ancestor`](/api/token#is_ancestor) | | `Token.is_ancestor_of` | [`Token.is_ancestor`](/api/token#is_ancestor) |
### Deprecated {#deprecated} ### Deprecated {id="deprecated"}
The following methods are deprecated. They can still be used, but should be The following methods are deprecated. They can still be used, but should be
replaced. replaced.
@ -383,7 +383,7 @@ replaced.
| `Tokenizer.tokens_from_list` | [`Doc`](/api/doc) | | `Tokenizer.tokens_from_list` | [`Doc`](/api/doc) |
| `Span.sent_start` | [`Span.is_sent_start`](/api/span#is_sent_start) | | `Span.sent_start` | [`Span.is_sent_start`](/api/span#is_sent_start) |
## Migrating from spaCy 1.x {#migrating} ## Migrating from spaCy 1.x {id="migrating"}
Because we'e made so many architectural changes to the library, we've tried to Because we'e made so many architectural changes to the library, we've tried to
**keep breaking changes to a minimum**. A lot of projects follow the philosophy **keep breaking changes to a minimum**. A lot of projects follow the philosophy
@ -403,7 +403,7 @@ v2.0.
</Infobox> </Infobox>
### Document processing {#migrating-document-processing} ### Document processing {id="migrating-document-processing"}
The [`Language.pipe`](/api/language#pipe) method allows spaCy to batch The [`Language.pipe`](/api/language#pipe) method allows spaCy to batch
documents, which brings a **significant performance advantage** in v2.0. The new documents, which brings a **significant performance advantage** in v2.0. The new
@ -421,7 +421,7 @@ To make usage easier, there's now a boolean `as_tuples` keyword argument, that
lets you pass in an iterator of `(text, context)` pairs, so you can get back an lets you pass in an iterator of `(text, context)` pairs, so you can get back an
iterator of `(doc, context)` tuples. iterator of `(doc, context)` tuples.
### Saving, loading and serialization {#migrating-saving-loading} ### Saving, loading and serialization {id="migrating-saving-loading"}
Double-check all calls to `spacy.load()` and make sure they don't use the `path` Double-check all calls to `spacy.load()` and make sure they don't use the `path`
keyword argument. If you're only loading in binary data and not a model package keyword argument. If you're only loading in binary data and not a model package
@ -451,7 +451,7 @@ If you've trained models with input from v1.x, you'll need to **retrain them**
with spaCy v2.0. All previous models will not be compatible with the new with spaCy v2.0. All previous models will not be compatible with the new
version. version.
### Processing pipelines and language data {#migrating-languages} ### Processing pipelines and language data {id="migrating-languages"}
If you're importing language data or `Language` classes, make sure to change If you're importing language data or `Language` classes, make sure to change
your import statements to import from `spacy.lang`. If you've added your own your import statements to import from `spacy.lang`. If you've added your own
@ -496,7 +496,7 @@ import and instantiate them directly but it's more convenient to use the new
+ nlp.add_pipe(tagger, first=True) + nlp.add_pipe(tagger, first=True)
``` ```
### Training {#migrating-training} ### Training {id="migrating-training"}
All built-in pipeline components are now subclasses of [`Pipe`](/api/pipe), All built-in pipeline components are now subclasses of [`Pipe`](/api/pipe),
fully trainable and serializable, and follow the same API. Instead of updating fully trainable and serializable, and follow the same API. Instead of updating
@ -525,7 +525,7 @@ training.
+ nlp.to_disk("/model") + nlp.to_disk("/model")
``` ```
### Attaching custom data to the Doc {#migrating-doc} ### Attaching custom data to the Doc {id="migrating-doc"}
Previously, you had to create a new container in order to attach custom data to Previously, you had to create a new container in order to attach custom data to
a `Doc` object. This often required converting the `Doc` objects to and from a `Doc` object. This often required converting the `Doc` objects to and from
@ -561,7 +561,7 @@ own extension module.
+ meta = doc._.meta + meta = doc._.meta
``` ```
### Strings and hash values {#migrating-strings} ### Strings and hash values {id="migrating-strings"}
The change from integer IDs to hash values may not actually affect your code The change from integer IDs to hash values may not actually affect your code
very much. However, if you're adding strings to the vocab manually, you now need very much. However, if you're adding strings to the vocab manually, you now need
@ -577,7 +577,7 @@ be sure that the string-to-hash mapping will always match across vocabularies.
+ other_nlp.vocab.strings["coffee"] # 3197928453018144401 + other_nlp.vocab.strings["coffee"] # 3197928453018144401
``` ```
### Adding patterns and callbacks to the matcher {#migrating-matcher} ### Adding patterns and callbacks to the matcher {id="migrating-matcher"}
If you're using the matcher, you can now add patterns in one step. This should If you're using the matcher, you can now add patterns in one step. This should
be easy to update simply merge the ID, callback and patterns into one call to be easy to update simply merge the ID, callback and patterns into one call to

View File

@ -6,7 +6,7 @@ menu:
- ['Upgrading Notes', 'upgrading'] - ['Upgrading Notes', 'upgrading']
--- ---
## New Features {#features hidden="true"} ## New Features {id="features",hidden="true"}
It's been great to see the adoption of the new spaCy v3, which introduced It's been great to see the adoption of the new spaCy v3, which introduced
[transformer-based](/usage/embeddings-transformers) pipelines, a new [transformer-based](/usage/embeddings-transformers) pipelines, a new
@ -18,7 +18,7 @@ component for predicting arbitrary and potentially overlapping spans, support
for partial incorrect annotations in the entity recognizer, new trained for partial incorrect annotations in the entity recognizer, new trained
pipelines for Catalan and Danish, as well as many bug fixes and improvements. pipelines for Catalan and Danish, as well as many bug fixes and improvements.
### Using predicted annotations during training {#predicted-annotations-training} ### Using predicted annotations during training {id="predicted-annotations-training"}
By default, components are updated in isolation during training, which means By default, components are updated in isolation during training, which means
that they don't see the predictions of any earlier components in the pipeline. that they don't see the predictions of any earlier components in the pipeline.
@ -29,8 +29,7 @@ on the predicted docs during training. This makes it easy to use the predictions
of a previous component in the pipeline as features for a subsequent component, of a previous component in the pipeline as features for a subsequent component,
e.g. the dependency labels in the tagger: e.g. the dependency labels in the tagger:
```ini ```ini {title="config.cfg (excerpt)",highlight="7,12"}
### config.cfg (excerpt) {highlight="7,12"}
[nlp] [nlp]
pipeline = ["parser", "tagger"] pipeline = ["parser", "tagger"]
@ -52,7 +51,7 @@ as a feature for a subsequent tagger component in the pipeline.
</Project> </Project>
### SpanCategorizer for predicting arbitrary and overlapping spans {#spancategorizer tag="experimental"} ### SpanCategorizer for predicting arbitrary and overlapping spans {id="spancategorizer",tag="experimental"}
A common task in applied NLP is extracting spans of texts from documents, A common task in applied NLP is extracting spans of texts from documents,
including longer phrases or nested expressions. Named entity recognition isn't including longer phrases or nested expressions. Named entity recognition isn't
@ -76,7 +75,11 @@ This project trains a span categorizer for Indonesian NER.
<Infobox title="Tip: Create data with Prodigy's new span annotation UI"> <Infobox title="Tip: Create data with Prodigy's new span annotation UI">
[![Prodigy: example of the new manual spans UI](../images/prodigy_spans-manual.jpg)](https://support.prodi.gy/t/3861) <Image
src="/images/prodigy_spans-manual.jpg"
href="https://support.prodi.gy/t/3861"
alt="Prodigy: example of the new manual spans UI"
/>
The upcoming version of our annotation tool [Prodigy](https://prodi.gy) The upcoming version of our annotation tool [Prodigy](https://prodi.gy)
(currently available as a [pre-release](https://support.prodi.gy/t/3861) for all (currently available as a [pre-release](https://support.prodi.gy/t/3861) for all
@ -86,7 +89,7 @@ for spaCy's `SpanCategorizer` component.
</Infobox> </Infobox>
### Update the entity recognizer with partial incorrect annotations {#negative-samples} ### Update the entity recognizer with partial incorrect annotations {id="negative-samples"}
> #### config.cfg (excerpt) > #### config.cfg (excerpt)
> >
@ -116,9 +119,9 @@ train_doc.spans["incorrect_spans"] = [
] ]
``` ```
<!-- TODO: more details and/or example project? --> {/* TODO: more details and/or example project? */}
### New pipeline packages for Catalan and Danish {#pipeline-packages} ### New pipeline packages for Catalan and Danish {id="pipeline-packages"}
spaCy v3.1 adds 5 new pipeline packages, including a new core family for Catalan spaCy v3.1 adds 5 new pipeline packages, including a new core family for Catalan
and a new transformer-based pipeline for Danish using the and a new transformer-based pipeline for Danish using the
@ -140,7 +143,7 @@ your own.
| [`ca_core_news_trf`](/models/ca#ca_core_news_trf) | Catalan | 98.9 | 93.0 | 91.2 | | [`ca_core_news_trf`](/models/ca#ca_core_news_trf) | Catalan | 98.9 | 93.0 | 91.2 |
| [`da_core_news_trf`](/models/da#da_core_news_trf) | Danish | 98.0 | 85.0 | 82.9 | | [`da_core_news_trf`](/models/da#da_core_news_trf) | Danish | 98.0 | 85.0 | 82.9 |
### Resizable text classification architectures {#resizable-textcat} ### Resizable text classification architectures {id="resizable-textcat"}
Previously, the [`TextCategorizer`](/api/textcategorizer) architectures could Previously, the [`TextCategorizer`](/api/textcategorizer) architectures could
not be resized, meaning that you couldn't add new labels to an already trained not be resized, meaning that you couldn't add new labels to an already trained
@ -148,18 +151,18 @@ model. In spaCy v3.1, the [TextCatCNN](/api/architectures#TextCatCNN) and
[TextCatBOW](/api/architectures#TextCatBOW) architectures are now resizable, [TextCatBOW](/api/architectures#TextCatBOW) architectures are now resizable,
while ensuring that the predictions for the old labels remain the same. while ensuring that the predictions for the old labels remain the same.
### CLI command to assemble pipeline from config {#assemble} ### CLI command to assemble pipeline from config {id="assemble"}
The [`spacy assemble`](/api/cli#assemble) command lets you assemble a pipeline The [`spacy assemble`](/api/cli#assemble) command lets you assemble a pipeline
from a config file without additional training. It can be especially useful for from a config file without additional training. It can be especially useful for
creating a blank pipeline with a custom tokenizer, rule-based components or word creating a blank pipeline with a custom tokenizer, rule-based components or word
vectors. vectors.
```cli ```bash
$ python -m spacy assemble config.cfg ./output $ python -m spacy assemble config.cfg ./output
``` ```
### Pretty pipeline package READMEs {#package-readme} ### Pretty pipeline package READMEs {id="package-readme"}
The [`spacy package`](/api/cli#package) command now auto-generates a pretty The [`spacy package`](/api/cli#package) command now auto-generates a pretty
`README.md` based on the pipeline information defined in the `meta.json`. This `README.md` based on the pipeline information defined in the `meta.json`. This
@ -167,7 +170,7 @@ includes a table with a general overview, as well as the label scheme and
accuracy figures, if available. For an example, see the accuracy figures, if available. For an example, see the
[model releases](https://github.com/explosion/spacy-models/releases). [model releases](https://github.com/explosion/spacy-models/releases).
### Support for streaming large or infinite corpora {#streaming-corpora} ### Support for streaming large or infinite corpora {id="streaming-corpora"}
> #### config.cfg (excerpt) > #### config.cfg (excerpt)
> >
@ -193,7 +196,7 @@ available labels in every example. If necessary, you can use the
your components using a representative sample so the model can be initialized your components using a representative sample so the model can be initialized
correctly before training. correctly before training.
### New lemmatizers for Catalan and Italian {#pos-lemmatizers} ### New lemmatizers for Catalan and Italian {id="pos-lemmatizers"}
The trained pipelines for [Catalan](/models/ca) and [Italian](/models/it) now The trained pipelines for [Catalan](/models/ca) and [Italian](/models/it) now
include lemmatizers that use the predicted part-of-speech tags as part of the include lemmatizers that use the predicted part-of-speech tags as part of the
@ -203,7 +206,7 @@ sure you have the
[`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) package [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) package
installed, which provides the relevant tables. installed, which provides the relevant tables.
### Upload your pipelines to the Hugging Face Hub {#huggingface-hub} ### Upload your pipelines to the Hugging Face Hub {id="huggingface-hub"}
The [Hugging Face Hub](https://huggingface.co/) lets you upload models and share The [Hugging Face Hub](https://huggingface.co/) lets you upload models and share
them with others, and it now supports spaCy pipelines out-of-the-box. The new them with others, and it now supports spaCy pipelines out-of-the-box. The new
@ -218,7 +221,7 @@ details, files and interactive visualizers, as well as a direct URL to the wheel
file that you can install via `pip install`. For examples, check out the file that you can install via `pip install`. For examples, check out the
[spaCy pipelines](https://huggingface.co/spacy) we've uploaded. [spaCy pipelines](https://huggingface.co/spacy) we've uploaded.
```cli ```bash
$ pip install spacy-huggingface-hub $ pip install spacy-huggingface-hub
$ huggingface-cli login $ huggingface-cli login
$ python -m spacy package ./en_ner_fashion ./output --build wheel $ python -m spacy package ./en_ner_fashion ./output --build wheel
@ -238,9 +241,9 @@ packaged model has changed. This makes it easy to deploy your models end-to-end.
</Project> </Project>
## Notes about upgrading from v3.0 {#upgrading} ## Notes about upgrading from v3.0 {id="upgrading"}
### Pipeline package version compatibility {#version-compat} ### Pipeline package version compatibility {id="version-compat"}
> #### Using legacy implementations > #### Using legacy implementations
> >
@ -280,7 +283,7 @@ In many cases (`spacy train`, `spacy.load()`), the new defaults will be filled
in automatically, but you'll need to fill in the new settings to run in automatically, but you'll need to fill in the new settings to run
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data). [`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
### Sourcing pipeline components with vectors {#source-vectors} ### Sourcing pipeline components with vectors {id="source-vectors"}
If you're sourcing a pipeline component that requires static vectors (for If you're sourcing a pipeline component that requires static vectors (for
example, a tagger or parser from an `md` or `lg` pretrained pipeline), be sure example, a tagger or parser from an `md` or `lg` pretrained pipeline), be sure
@ -289,8 +292,7 @@ spaCy v3.0, a bug allowed vectors to be loaded implicitly through `source`,
however in v3.1 this setting must be provided explicitly as however in v3.1 this setting must be provided explicitly as
`[initialize.vectors]`: `[initialize.vectors]`:
```ini ```ini {title="config.cfg (excerpt)"}
### config.cfg (excerpt)
[components.ner] [components.ner]
source = "en_core_web_md" source = "en_core_web_md"
@ -312,7 +314,7 @@ vectors. If you are sourcing a rule-based component like an entity ruler or
lemmatizer that does not use the vectors as a model feature, then this warning lemmatizer that does not use the vectors as a model feature, then this warning
can be safely ignored. can be safely ignored.
### Warnings {#warnings} ### Warnings {id="warnings"}
Logger warnings have been converted to Python warnings. Use Logger warnings have been converted to Python warnings. Use
[`warnings.filterwarnings`](https://docs.python.org/3/library/warnings.html#warnings.filterwarnings) [`warnings.filterwarnings`](https://docs.python.org/3/library/warnings.html#warnings.filterwarnings)

View File

@ -6,7 +6,7 @@ menu:
- ['Upgrading Notes', 'upgrading'] - ['Upgrading Notes', 'upgrading']
--- ---
## New Features {#features hidden="true"} ## New Features {id="features",hidden="true"}
spaCy v3.2 adds support for [`floret`](https://github.com/explosion/floret) spaCy v3.2 adds support for [`floret`](https://github.com/explosion/floret)
vectors, makes custom `Doc` creation and scoring easier, and includes many bug vectors, makes custom `Doc` creation and scoring easier, and includes many bug
@ -26,31 +26,29 @@ $ pip install spacy[apple]
</Infobox> </Infobox>
### Registered scoring functions {#registered-scoring-functions} ### Registered scoring functions {id="registered-scoring-functions"}
To customize the scoring, you can specify a scoring function for each component To customize the scoring, you can specify a scoring function for each component
in your config from the new [`scorers` registry](/api/top-level#registry): in your config from the new [`scorers` registry](/api/top-level#registry):
```ini ```ini {title="config.cfg (excerpt)",highlight="3"}
### config.cfg (excerpt) {highlight="3"}
[components.tagger] [components.tagger]
factory = "tagger" factory = "tagger"
scorer = {"@scorers":"spacy.tagger_scorer.v1"} scorer = {"@scorers":"spacy.tagger_scorer.v1"}
``` ```
### Overwrite settings {#overwrite} ### Overwrite settings {id="overwrite"}
Most pipeline components now include an `overwrite` setting in the config that Most pipeline components now include an `overwrite` setting in the config that
determines whether existing annotation in the `Doc` is preserved or overwritten: determines whether existing annotation in the `Doc` is preserved or overwritten:
```ini ```ini {title="config.cfg (excerpt)",highlight="3"}
### config.cfg (excerpt) {highlight="3"}
[components.tagger] [components.tagger]
factory = "tagger" factory = "tagger"
overwrite = false overwrite = false
``` ```
### Doc input for pipelines {#doc-input} ### Doc input for pipelines {id="doc-input"}
[`nlp`](/api/language#call) and [`nlp.pipe`](/api/language#pipe) accept [`nlp`](/api/language#call) and [`nlp.pipe`](/api/language#pipe) accept
[`Doc`](/api/doc) input, skipping the tokenizer if a `Doc` is provided instead [`Doc`](/api/doc) input, skipping the tokenizer if a `Doc` is provided instead
@ -63,13 +61,13 @@ doc._.text_id = 500
doc = nlp(doc) doc = nlp(doc)
``` ```
### Support for floret vectors {#vectors} ### Support for floret vectors {id="vectors"}
We recently published [`floret`](https://github.com/explosion/floret), an We recently published [`floret`](https://github.com/explosion/floret), an
extended version of [fastText](https://fasttext.cc) that combines fastText's extended version of [fastText](https://fasttext.cc) that combines fastText's
subwords with Bloom embeddings for compact, full-coverage vectors. The use of subwords with Bloom embeddings for compact, full-coverage vectors. The use of
subwords means that there are no OOV words and due to Bloom embeddings, the subwords means that there are no OOV words and due to Bloom embeddings, the
vector table can be kept very small at <100K entries. Bloom embeddings are vector table can be kept very small at \<100K entries. Bloom embeddings are
already used by [HashEmbed](https://thinc.ai/docs/api-layers#hashembed) in already used by [HashEmbed](https://thinc.ai/docs/api-layers#hashembed) in
[tok2vec](/api/architectures#tok2vec-arch) for compact spaCy models. [tok2vec](/api/architectures#tok2vec-arch) for compact spaCy models.
@ -127,7 +125,7 @@ For the default project settings with 1M (3.3G) tokenized training texts and 50K
</Project> </Project>
### Updates for spacy-transformers v1.1 {#spacy-transformers} ### Updates for spacy-transformers v1.1 {id="spacy-transformers"}
[`spacy-transformers`](https://github.com/explosion/spacy-transformers) v1.1 has [`spacy-transformers`](https://github.com/explosion/spacy-transformers) v1.1 has
been refactored to improve serialization and support of inline transformer been refactored to improve serialization and support of inline transformer
@ -147,7 +145,7 @@ such as `output_attentions`. Additional output is stored under
has been improved by streamlining allocations for tokenizer output and there is has been improved by streamlining allocations for tokenizer output and there is
new support for [mixed-precision training](/api/architectures#TransformerModel). new support for [mixed-precision training](/api/architectures#TransformerModel).
### New transformer package for Japanese {#pipeline-packages} ### New transformer package for Japanese {id="pipeline-packages"}
spaCy v3.2 adds a new transformer pipeline package for Japanese spaCy v3.2 adds a new transformer pipeline package for Japanese
[`ja_core_news_trf`](/models/ja#ja_core_news_trf), which uses the `basic` [`ja_core_news_trf`](/models/ja#ja_core_news_trf), which uses the `basic`
@ -155,7 +153,7 @@ pretokenizer instead of `mecab` to limit the number of dependencies required for
the pipeline. Thanks to Hiroshi Matsuda and the spaCy Japanese community for the pipeline. Thanks to Hiroshi Matsuda and the spaCy Japanese community for
their contributions! their contributions!
### Pipeline and language updates {#pipeline-updates} ### Pipeline and language updates {id="pipeline-updates"}
- All Universal Dependencies training data has been updated to v2.8. - All Universal Dependencies training data has been updated to v2.8.
- The Catalan data, tokenizer and lemmatizer have been updated, thanks to Carlos - The Catalan data, tokenizer and lemmatizer have been updated, thanks to Carlos
@ -173,9 +171,9 @@ spaCy v3.2 also features a new Irish lemmatizer, support for `noun_chunks` in
Portuguese, improved `noun_chunks` for Spanish and additional updates for Portuguese, improved `noun_chunks` for Spanish and additional updates for
Bulgarian, Catalan, Sinhala, Tagalog, Tigrinya and Vietnamese. Bulgarian, Catalan, Sinhala, Tagalog, Tigrinya and Vietnamese.
## Notes about upgrading from v3.1 {#upgrading} ## Notes about upgrading from v3.1 {id="upgrading"}
### Pipeline package version compatibility {#version-compat} ### Pipeline package version compatibility {id="version-compat"}
> #### Using legacy implementations > #### Using legacy implementations
> >
@ -207,7 +205,7 @@ working as expected, you can update the spaCy version requirements in the
To update a config from spaCy v3.1 with the new v3.2 settings, run To update a config from spaCy v3.1 with the new v3.2 settings, run
[`init fill-config`](/api/cli#init-fill-config): [`init fill-config`](/api/cli#init-fill-config):
```cli ```bash
$ python -m spacy init fill-config config-v3.1.cfg config-v3.2.cfg $ python -m spacy init fill-config config-v3.1.cfg config-v3.2.cfg
``` ```
@ -216,7 +214,7 @@ In many cases ([`spacy train`](/api/cli#train),
automatically, but you'll need to fill in the new settings to run automatically, but you'll need to fill in the new settings to run
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data). [`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
## Notes about upgrading from spacy-transformers v1.0 {#upgrading-transformers} ## Notes about upgrading from spacy-transformers v1.0 {id="upgrading-transformers"}
When you're loading a transformer pipeline package trained with When you're loading a transformer pipeline package trained with
[`spacy-transformers`](https://github.com/explosion/spacy-transformers) v1.0 [`spacy-transformers`](https://github.com/explosion/spacy-transformers) v1.0

View File

@ -6,12 +6,12 @@ menu:
- ['Upgrading Notes', 'upgrading'] - ['Upgrading Notes', 'upgrading']
--- ---
## New features {#features hidden="true"} ## New features {id="features",hidden="true"}
spaCy v3.3 improves the speed of core pipeline components, adds a new trainable spaCy v3.3 improves the speed of core pipeline components, adds a new trainable
lemmatizer, and introduces trained pipelines for Finnish, Korean and Swedish. lemmatizer, and introduces trained pipelines for Finnish, Korean and Swedish.
### Speed improvements {#speed} ### Speed improvements {id="speed"}
v3.3 includes a slew of speed improvements: v3.3 includes a slew of speed improvements:
@ -50,14 +50,14 @@ sizes on Linux (Intel Xeon W-2265) and OS X (M1) to compare spaCy v3.2 vs. v3.3:
| (+v3.3 trainable lemmatizer) | 1000 | 20047 | 20628 | 2.90% | | (+v3.3 trainable lemmatizer) | 1000 | 20047 | 20628 | 2.90% |
| | 10000 | 15921 | 18546 | 16.49% | | | 10000 | 15921 | 18546 | 16.49% |
### Trainable lemmatizer {#trainable-lemmatizer} ### Trainable lemmatizer {id="trainable-lemmatizer"}
The new [trainable lemmatizer](/api/edittreelemmatizer) component uses The new [trainable lemmatizer](/api/edittreelemmatizer) component uses
[edit trees](https://explosion.ai/blog/edit-tree-lemmatizer) to transform tokens [edit trees](https://explosion.ai/blog/edit-tree-lemmatizer) to transform tokens
into lemmas. Try out the trainable lemmatizer with the into lemmas. Try out the trainable lemmatizer with the
[training quickstart](/usage/training#quickstart)! [training quickstart](/usage/training#quickstart)!
### displaCy support for overlapping spans and arcs {#displacy} ### displaCy support for overlapping spans and arcs {id="displacy"}
displaCy now supports overlapping spans with a new displaCy now supports overlapping spans with a new
[`span`](/usage/visualizers#span) style and multiple arcs with different labels [`span`](/usage/visualizers#span) style and multiple arcs with different labels
@ -77,9 +77,11 @@ doc.spans["custom"] = [Span(doc, 3, 6, "ORG"), Span(doc, 5, 6, "GPE")]
displacy.serve(doc, style="span", options={"spans_key": "custom"}) displacy.serve(doc, style="span", options={"spans_key": "custom"})
``` ```
import DisplacySpanHtml from 'images/displacy-span.html' <Iframe
title="displaCy visualizer for overlapping spans"
<Iframe title="displaCy visualizer for overlapping spans" html={DisplacySpanHtml} height={180} /> src="/images/displacy-span.html"
height={180}
/>
## Additional features and improvements ## Additional features and improvements
@ -94,9 +96,9 @@ import DisplacySpanHtml from 'images/displacy-span.html'
Russian, Slovenian, Spanish, Turkish, Ukrainian and Vietnamese. Russian, Slovenian, Spanish, Turkish, Ukrainian and Vietnamese.
- New noun chunks for Finnish. - New noun chunks for Finnish.
## Trained pipelines {#pipelines} ## Trained pipelines {id="pipelines"}
### New trained pipelines {#new-pipelines} ### New trained pipelines {id="new-pipelines"}
v3.3 introduces new CPU/CNN pipelines for Finnish, Korean and Swedish, which use v3.3 introduces new CPU/CNN pipelines for Finnish, Korean and Swedish, which use
the new trainable lemmatizer and the new trainable lemmatizer and
@ -116,7 +118,7 @@ pipelines have compact vectors with no out-of-vocabulary words.
| [`sv_core_news_md`](/models/sv#sv_core_news_md) | Swedish | 96.3 | 78.5 | 79.3 | | [`sv_core_news_md`](/models/sv#sv_core_news_md) | Swedish | 96.3 | 78.5 | 79.3 |
| [`sv_core_news_lg`](/models/sv#sv_core_news_lg) | Swedish | 96.3 | 79.1 | 81.1 | | [`sv_core_news_lg`](/models/sv#sv_core_news_lg) | Swedish | 96.3 | 79.1 | 81.1 |
### Pipeline updates {#pipeline-updates} ### Pipeline updates {id="pipeline-updates"}
The following languages switch from lookup or rule-based lemmatizers to the new The following languages switch from lookup or rule-based lemmatizers to the new
trainable lemmatizer: Danish, Dutch, German, Greek, Italian, Lithuanian, trainable lemmatizer: Danish, Dutch, German, Greek, Italian, Lithuanian,
@ -149,7 +151,7 @@ continue using the previous lemmatizer, you can
In addition, the vectors in the English pipelines are deduplicated to improve In addition, the vectors in the English pipelines are deduplicated to improve
the pruned vectors in the `md` models and reduce the `lg` model size. the pruned vectors in the `md` models and reduce the `lg` model size.
## Notes about upgrading from v3.2 {#upgrading} ## Notes about upgrading from v3.2 {id="upgrading"}
### Span comparisons ### Span comparisons
@ -199,7 +201,7 @@ word list similar to the list of in-vocabulary vector keys with default vectors.
- `Vectors.most_similar` is not supported because there's no fixed list of - `Vectors.most_similar` is not supported because there's no fixed list of
vectors to compare your vectors to. vectors to compare your vectors to.
### Pipeline package version compatibility {#version-compat} ### Pipeline package version compatibility {id="version-compat"}
> #### Using legacy implementations > #### Using legacy implementations
> >
@ -233,7 +235,7 @@ working as expected, you can update the spaCy version requirements in the
To update a config from spaCy v3.2 with the new v3.3 settings, run To update a config from spaCy v3.2 with the new v3.3 settings, run
[`init fill-config`](/api/cli#init-fill-config): [`init fill-config`](/api/cli#init-fill-config):
```cli ```bash
$ python -m spacy init fill-config config-v3.2.cfg config-v3.3.cfg $ python -m spacy init fill-config config-v3.2.cfg config-v3.3.cfg
``` ```

View File

@ -6,19 +6,19 @@ menu:
- ['Upgrading Notes', 'upgrading'] - ['Upgrading Notes', 'upgrading']
--- ---
## New features {#features hidden="true"} ## New features {id="features",hidden="true"}
spaCy v3.4 brings typing and speed improvements along with new vectors for spaCy v3.4 brings typing and speed improvements along with new vectors for
English CNN pipelines and new trained pipelines for Croatian. This release also English CNN pipelines and new trained pipelines for Croatian. This release also
includes prebuilt linux aarch64 wheels for all spaCy dependencies distributed by includes prebuilt linux aarch64 wheels for all spaCy dependencies distributed by
Explosion. Explosion.
### Typing improvements {#typing} ### Typing improvements {id="typing"}
spaCy v3.4 supports pydantic v1.9 and mypy 0.950+ through extensive updates to spaCy v3.4 supports pydantic v1.9 and mypy 0.950+ through extensive updates to
types in Thinc v8.1. types in Thinc v8.1.
### Speed improvements {#speed} ### Speed improvements {id="speed"}
- For the parser, use C `saxpy`/`sgemm` provided by the `Ops` implementation in - For the parser, use C `saxpy`/`sgemm` provided by the `Ops` implementation in
order to use Accelerate through `thinc-apple-ops`. order to use Accelerate through `thinc-apple-ops`.
@ -42,9 +42,9 @@ types in Thinc v8.1.
- Support env variable `SPACY_NUM_BUILD_JOBS` to specify the number of build - Support env variable `SPACY_NUM_BUILD_JOBS` to specify the number of build
jobs to run in parallel with `pip`. jobs to run in parallel with `pip`.
## Trained pipelines {#pipelines} ## Trained pipelines {id="pipelines"}
### New trained pipelines {#new-pipelines} ### New trained pipelines {id="new-pipelines"}
v3.4 introduces new CPU/CNN pipelines for Croatian, which use the trainable v3.4 introduces new CPU/CNN pipelines for Croatian, which use the trainable
lemmatizer and [floret vectors](https://github.com/explosion/floret). Due to the lemmatizer and [floret vectors](https://github.com/explosion/floret). Due to the
@ -57,20 +57,20 @@ subwords, the pipelines have compact vectors with no out-of-vocabulary words.
| [`hr_core_news_md`](/models/hr#hr_core_news_md) | 97.3 | 80.1 | 81.8 | | [`hr_core_news_md`](/models/hr#hr_core_news_md) | 97.3 | 80.1 | 81.8 |
| [`hr_core_news_lg`](/models/hr#hr_core_news_lg) | 97.5 | 80.4 | 83.0 | | [`hr_core_news_lg`](/models/hr#hr_core_news_lg) | 97.5 | 80.4 | 83.0 |
### Pipeline updates {#pipeline-updates} ### Pipeline updates {id="pipeline-updates"}
All CNN pipelines have been extended with whitespace augmentation. All CNN pipelines have been extended with whitespace augmentation.
The English CNN pipelines have new word vectors: The English CNN pipelines have new word vectors:
| Package | Model Version | TAG | Parser LAS | NER F | | Package | Model Version | TAG | Parser LAS | NER F |
| ----------------------------------------------- | ------------- | ---: | ---------: | ----: | | --------------------------------------------- | ------------- | ---: | ---------: | ----: |
| [`en_core_web_md`](/models/en#en_core_web_md) | v3.3.0 | 97.3 | 90.1 | 84.6 | | [`en_core_web_md`](/models/en#en_core_web_md) | v3.3.0 | 97.3 | 90.1 | 84.6 |
| [`en_core_web_md`](/models/en#en_core_web_md) | v3.4.0 | 97.2 | 90.3 | 85.5 | | [`en_core_web_md`](/models/en#en_core_web_md) | v3.4.0 | 97.2 | 90.3 | 85.5 |
| [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.3.0 | 97.4 | 90.1 | 85.3 | | [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.3.0 | 97.4 | 90.1 | 85.3 |
| [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.4.0 | 97.3 | 90.2 | 85.6 | | [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.4.0 | 97.3 | 90.2 | 85.6 |
## Notes about upgrading from v3.3 {#upgrading} ## Notes about upgrading from v3.3 {id="upgrading"}
### Doc.has_vector ### Doc.has_vector
@ -99,7 +99,7 @@ word list similar to the list of in-vocabulary vector keys with default vectors.
- `Vectors.most_similar` is not supported because there's no fixed list of - `Vectors.most_similar` is not supported because there's no fixed list of
vectors to compare your vectors to. vectors to compare your vectors to.
### Pipeline package version compatibility {#version-compat} ### Pipeline package version compatibility {id="version-compat"}
> #### Using legacy implementations > #### Using legacy implementations
> >
@ -133,7 +133,7 @@ working as expected, you can update the spaCy version requirements in the
To update a config from spaCy v3.3 with the new v3.4 settings, run To update a config from spaCy v3.3 with the new v3.4 settings, run
[`init fill-config`](/api/cli#init-fill-config): [`init fill-config`](/api/cli#init-fill-config):
```cli ```bash
$ python -m spacy init fill-config config-v3.3.cfg config-v3.4.cfg $ python -m spacy init fill-config config-v3.3.cfg config-v3.4.cfg
``` ```

Some files were not shown because too many files have changed in this diff Show More