explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-12-24 00:46:28 +03:00
Code Issues Packages Projects Releases Wiki Activity

Website migration from Gatsby to Next (#12058)

Browse Source 
* Rename all MDX file to `.mdx`

* Lock current node version (#11885)

* Apply Prettier (#11996)

* Minor website fixes (#11974) [ci skip]

* fix table

* Migrate to Next WEB-17 (#12005)

* Initial commit

* Run `npx create-next-app@13 next-blog`

* Install MDX packages

Following: 77b5f79a4d/packages/next-mdx/readme.md

* Add MDX to Next

* Allow Next to handle `.md` and `.mdx` files.

* Add VSCode extension recommendation

* Disabled TypeScript strict mode for now

* Add prettier

* Apply Prettier to all files

* Make sure to use correct Node version

* Add basic implementation for `MDXRemote`

* Add experimental Rust MDX parser

* Add `/public`

* Add SASS support

* Remove default pages and styling

* Convert to module

This allows to use `import/export` syntax

* Add import for custom components

* Add ability to load plugins

* Extract function

This will make the next commit easier to read

* Allow to handle directories for page creation

* Refactoring

* Allow to parse subfolders for pages

* Extract logic

* Redirect `index.mdx` to parent directory

* Disabled ESLint during builds

* Disabled typescript during build

* Remove Gatsby from `README.md`

* Rephrase Docker part of `README.md`

* Update project structure in `README.md`

* Move and rename plugins

* Update plugin for wrapping sections

* Add dependencies for  plugin

* Use  plugin

* Rename wrapper type

* Simplify unnessary adding of id to sections

The slugified section ids are useless, because they can not be referenced anywhere anyway. The navigation only works if the section has the same id as the heading.

* Add plugin for custom attributes on Markdown elements

* Add plugin to readd support for tables

* Add plugin to fix problem with wrapped images

For more details see this issue: https://github.com/mdx-js/mdx/issues/1798

* Add necessary meta data to pages

* Install necessary dependencies

* Remove outdated MDX handling

* Remove reliance on `InlineList`

* Use existing Remark components

* Remove unallowed heading

Before `h1` components where not overwritten and would never have worked and they aren't used anywhere either.

* Add missing components to MDX

* Add correct styling

* Fix broken list

* Fix broken CSS classes

* Implement layout

* Fix links

* Fix broken images

* Fix pattern image

* Fix heading attributes

* Rename heading attribute

`new` was causing some weird issue, so renaming it to `version`

* Update comment syntax in MDX

* Merge imports

* Fix markdown rendering inside components

* Add model pages

* Simplify anchors

* Fix default value for theme

* Add Universe index page

* Add Universe categories

* Add Universe projects

* Fix Next problem with copy

Next complains when the server renders something different then the client, therfor we move the differing logic to `useEffect`

* Fix improper component nesting

Next doesn't allow block elements inside a `<p>`

* Replace landing page MDX with page component

* Remove inlined iframe content

* Remove ability to inline HTML content in iFrames

* Remove MDX imports

* Fix problem with image inside link in MDX

* Escape character for MDX

* Fix unescaped characters in MDX

* Fix headings with logo

* Allow to export static HTML pages

* Add prebuild script

This command is automatically run by Next

* Replace `svg-loader` with `react-inlinesvg`

`svg-loader` is no longer maintained

* Fix ESLint `react-hooks/exhaustive-deps`

* Fix dropdowns

* Change code language from `cli` to `bash`

* Remove unnessary language `none`

* Fix invalid code language

`markdown_` with an underscore was used to basically turn of syntax highlighting, but using unknown languages know throws an error.

* Enable code blocks plugin

* Readd `InlineCode` component

MDX2 removed the `inlineCode` component

> The special component name `inlineCode` was removed, we recommend to use `pre` for the block version of code, and code for both the block and inline versions

Source: https://mdxjs.com/migrating/v2/#update-mdx-content

* Remove unused code

* Extract function to own file

* Fix code syntax highlighting

* Update syntax for code block meta data

* Remove unused prop

* Fix internal link recognition

There is a problem with regex between Node and browser, and since Next runs the component on both, this create an error.

`Prop `rel` did not match. Server: "null" Client: "noopener nofollow noreferrer"`

This simplifies the implementation and fixes the above error.

* Replace `react-helmet` with `next/head`

* Fix `className` problem for JSX component

* Fix broken bold markdown

* Convert file to `.mjs` to be used by Node process

* Add plugin to replace strings

* Fix custom table row styling

* Fix problem with `span` inside inline `code`

React doesn't allow a `span` inside an inline `code` element and throws an error in dev mode.

* Add `_document` to be able to customize `<html>` and `<body>`

* Add `lang="en"`

* Store Netlify settings in file

This way we don't need to update via Netlify UI, which can be tricky if changing build settings.

* Add sitemap

* Add Smartypants

* Add PWA support

* Add `manifest.webmanifest`

* Fix bug with anchor links after reloading

There was no need for the previous implementation, since the browser handles this nativly. Additional the manual scrolling into view was actually broken, because the heading would disappear behind the menu bar.

* Rename custom event

I was googeling for ages to find out what kind of event `inview` is, only to figure out it was a custom event with a name that sounds pretty much like a native one. 🫠

* Fix missing comment syntax highlighting

* Refactor Quickstart component

The previous implementation was hidding the irrelevant lines via data-props and dynamically generated CSS. This created problems with Next and was also hard to follow. CSS was used to do what React is supposed to handle.

The new implementation simplfy filters the list of children (React elements) via their props.

* Fix syntax highlighting for Training Quickstart

* Unify code rendering

* Improve error logging in Juniper

* Fix Juniper component

* Automatically generate "Read Next" link

* Add Plausible

* Use recent DocSearch component and adjust styling

* Fix images

* Turn of image optimization

> Image Optimization using Next.js' default loader is not compatible with `next export`.

We currently deploy to Netlify via `next export`

* Dont build pages starting with `_`

* Remove unused files

* Add Next plugin to Netlify

* Fix button layout

MDX automatically adds `p` tags around text on a new line and Prettier wants to put the text on a new line. Hacking with JSX string.

* Add 404 page

* Apply Prettier

* Update Prettier for `package.json`

Next sometimes wants to patch `package-lock.json`. The old Prettier setting indended with 4 spaces, but Next always indends with 2 spaces. Since `npm install` automatically uses the indendation from `package.json` for `package-lock.json` and to avoid the format switching back and forth, both files are now set to 2 spaces.

* Apply Next patch to `package-lock.json`

When starting the dev server Next would warn `warn  - Found lockfile missing swc dependencies, patching...` and update the `package-lock.json`. These are the patched changes.

* fix link

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* small backslash fixes

* adjust to new style

Co-authored-by: Marcus Blättermann <marcus@essenmitsosse.de>
This commit is contained in:
Sofie Van Landeghem 2023-01-11 17:30:07 +01:00 committed by GitHub
parent e0168ccce9
commit 554df9ef20
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
241 changed files with 26957 additions and 34416 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
website/.eslintrc.json Normal file
View File

@ -0,0 +1,3 @@
{
"extends": "next/core-web-vitals"
}

44
website/.gitignore vendored Normal file
View File

@ -0,0 +1,44 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
# dependencies
/node_modules
/.pnp
.pnp.js
# testing
/coverage
# next.js
/.next/
/out/
# production
/build
# misc
.DS_Store
*.pem
# debug
npm-debug.log*
yarn-debug.log*
yarn-error.log*
.pnpm-debug.log*
# local env files
.env*.local
# vercel
.vercel
# typescript
*.tsbuildinfo
next-env.d.ts
!.vscode/extensions.json
!public
public/robots.txt
public/sitemap*
public/sw.js*
public/workbox*

1
website/.nvmrc Normal file
View File

@ -0,0 +1 @@
18

1
website/.prettierignore Normal file
View File

@ -0,0 +1 @@
.next

5
website/.prettierrc
View File

@ -20,12 +20,11 @@
}
},
{
"files": "*.md",
"files": ["package.json", "package-lock.json"],
"options": {
"tabWidth": 2,
"printWidth": 80,
"proseWrap": "always",
"htmlWhitespaceSensitivity": "strict"
"proseWrap": "always"
}
},
{

8
website/.vscode/extensions.json vendored Normal file
View File

@ -0,0 +1,8 @@
{
"recommendations": [
"dbaeumer.vscode-eslint",
"unifiedjs.vscode-mdx",
"esbenp.prettier-vscode",
"syler.sass-indented"
]
}

28
website/README.md
View File

@ -7,17 +7,16 @@ The styleguide for the spaCy website is available at
## Setup and installation
Before running the setup, make sure your versions of
[Node](https://nodejs.org/en/) and [npm](https://www.npmjs.com/) are up to date.
Node v10.15 or later is required.
```bash
# Clone the repository
git clone https://github.com/explosion/spaCy
cd spaCy/website
# Install Gatsby's command-line tool
npm install --global gatsby-cli
# Switch to the correct Node version
#
# If you don't have NVM and don't want to use it, you can manually switch to the Node version
# stated in /.nvmrc and skip this step
nvm use
# Install the dependencies
npm install
@ -36,8 +35,7 @@ file in the root defines the settings used in this codebase.
## Building & developing the site with Docker
Sometimes it's hard to get a local environment working due to rapid updates to
node dependencies, so it may be easier to use docker for building the docs.
While it shouldn't be necessary and is not recommended you can run this site in a Docker container.
If you'd like to do this, **be sure you do _not_ include your local
`node_modules` folder**, since there are some dependencies that need to be built
@ -76,12 +74,14 @@ bit of time.
```yaml
├── docs # the actual markdown content
├── meta # JSON-formatted site metadata
| ├── dynamicMeta.js # At build time generated meta data
| ├── languages.json # supported languages and statistical models
| ├── sidebars.json # sidebar navigations for different sections
| ├── site.json # general site metadata
| ├── type-annotations.json # Type annotations
| └── universe.json # data for the spaCy universe section
├── public # compiled site
├── pages # Next router pages
├── public # static images and other assets
├── setup # Jinja setup
├── src # source
| ├── components # React components
@ -96,9 +96,11 @@ bit of time.
| | └── universe.js # layout templates for universe
| └── widgets # non-reusable components with content, e.g. changelog
├── .eslintrc.json # ESLint config file
├── .nvmrc # NVM config file
| # (to support "nvm use" to switch to correct Node version)
|
├── .prettierrc # Prettier config file
├── gatsby-browser.js # browser-specific hooks for Gatsby
├── gatsby-config.js # Gatsby configuration
├── gatsby-node.js # Node-specific hooks for Gatsby
└── package.json # package settings and dependencies
├── next.config.mjs # Next config file
├── package.json # package settings and dependencies
└── tsconfig.json # TypeScript config file
```

78
website/UNIVERSE.md
View File

@ -2,42 +2,52 @@
# spaCy Universe
The [spaCy Universe](https://spacy.io/universe) collects the many great resources developed with or for spaCy. It
includes standalone packages, plugins, extensions, educational materials,
operational utilities and bindings for other languages.
The [spaCy Universe](https://spacy.io/universe) collects the many great
resources developed with or for spaCy. It includes standalone packages, plugins,
extensions, educational materials, operational utilities and bindings for other
languages.
If you have a project that you want the spaCy community to make use of, you can
suggest it by submitting a pull request to this repository. The Universe
database is open-source and collected in a simple JSON file.
Looking for inspiration for your own spaCy plugin or extension? Check out the
[`project ideas`](https://github.com/explosion/spaCy/discussions?discussions_q=category%3A%22New+Features+%26+Project+Ideas%22)
[`project ideas`](https://github.com/explosion/spaCy/discussions?discussions_q=category%3A%22New+Features+%26+Project+Ideas%22)
discussion forum.
## Checklist
### Projects
✅ Libraries and packages should be **open-source** (with a user-friendly license) and at least somewhat **documented** (e.g. a simple `README` with usage instructions).
✅ Libraries and packages should be **open-source** (with a user-friendly
license) and at least somewhat **documented** (e.g. a simple `README` with usage
instructions).
✅ We're happy to include work in progress and prereleases, but we'd like to keep the emphasis on projects that should be useful to the community **right away**.
✅ We're happy to include work in progress and prereleases, but we'd like to
keep the emphasis on projects that should be useful to the community **right
away**.
✅ Demos and visualizers should be available via a **public URL**.
### Educational Materials
✅ Books should be **available for purchase or download** (not just pre-order). Ebooks and self-published books are fine, too, if they include enough substantial content.
✅ Books should be **available for purchase or download** (not just pre-order).
Ebooks and self-published books are fine, too, if they include enough
substantial content.
✅ The `"url"` of book entries should either point to the publisher's website or a reseller of your choice (ideally one that ships worldwide or as close as possible).
✅ The `"url"` of book entries should either point to the publisher's website or
a reseller of your choice (ideally one that ships worldwide or as close as
possible).
✅ If an online course is only available behind a paywall, it should at least have a **free excerpt** or chapter available, so users know what to expect.
✅ If an online course is only available behind a paywall, it should at least
have a **free excerpt** or chapter available, so users know what to expect.
## JSON format
To add a project, fork this repository, edit the [`universe.json`](meta/universe.json)
and add an object of the following format to the list of `"resources"`. Before
you submit your pull request, make sure to use a linter to verify that your
markup is correct.
To add a project, fork this repository, edit the
[`universe.json`](meta/universe.json) and add an object of the following format
to the list of `"resources"`. Before you submit your pull request, make sure to
use a linter to verify that your markup is correct.
```json
{
@ -69,26 +79,26 @@ markup is correct.
}
```
| Field | Type | Description |
| --- | --- | --- |
| `id` | string | Unique ID of the project. |
| `title` | string | Project title. If not set, the `id` will be used as the display title. |
| `slogan` | string | A short description of the project. Displayed in the overview and under the title. |
| `description` | string | A longer description of the project. Markdown is allowed, but should be limited to basic formatting like bold, italics, code or links. |
| `github` | string | Associated GitHub repo in the format `user/repo`. Will be displayed as a link and used for release, license and star badges. |
| `pip` | string | Package name on pip. If available, the installation command will be displayed. |
| `cran` | string | For R packages: package name on CRAN. If available, the installation command will be displayed. |
| `code_example` | array | Short example that shows how to use the project. Formatted as an array with one string per line. |
| `code_language` | string | Defaults to `'python'`. Optional code language used for syntax highlighting with [Prism](http://prismjs.com/). |
| `url` | string | Optional project link to display as button. |
| `thumb` | string | Optional URL to project thumbnail to display in overview and project header. Recommended size is 100x100px. |
| `image` | string | Optional URL to project image to display with description. |
| `author` | string | Name(s) of project author(s). |
| `author_links` | object | Usernames and links to display as icons to author info. Currently supports `twitter` and `github` usernames, as well as `website` link. |
| `category` | list | One or more categories to assign to project. Must be one of the available options. |
| `tags` | list | Still experimental and not used for filtering: one or more tags to assign to project. |
| Field | Type | Description |
| --------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | string | Unique ID of the project. |
| `title` | string | Project title. If not set, the `id` will be used as the display title. |
| `slogan` | string | A short description of the project. Displayed in the overview and under the title. |
| `description` | string | A longer description of the project. Markdown is allowed, but should be limited to basic formatting like bold, italics, code or links. |
| `github` | string | Associated GitHub repo in the format `user/repo`. Will be displayed as a link and used for release, license and star badges. |
| `pip` | string | Package name on pip. If available, the installation command will be displayed. |
| `cran` | string | For R packages: package name on CRAN. If available, the installation command will be displayed. |
| `code_example` | array | Short example that shows how to use the project. Formatted as an array with one string per line. |
| `code_language` | string | Defaults to `'python'`. Optional code language used for syntax highlighting with [Prism](http://prismjs.com/). |
| `url` | string | Optional project link to display as button. |
| `thumb` | string | Optional URL to project thumbnail to display in overview and project header. Recommended size is 100x100px. |
| `image` | string | Optional URL to project image to display with description. |
| `author` | string | Name(s) of project author(s). |
| `author_links` | object | Usernames and links to display as icons to author info. Currently supports `twitter` and `github` usernames, as well as `website` link. |
| `category` | list | One or more categories to assign to project. Must be one of the available options. |
| `tags` | list | Still experimental and not used for filtering: one or more tags to assign to project. |
To separate them from the projects, educational materials also specify
`"type": "education`. Books can also set a `"cover"` field containing a URL
to a cover image. If available, it's used in the overview and displayed on
the individual book page.
`"type": "education`. Books can also set a `"cover"` field containing a URL to a
cover image. If available, it's used in the overview and displayed on the
individual book page.

78
website/docs/api/architectures.md → website/docs/api/architectures.mdx
View File

@ -26,9 +26,9 @@ part of the [training config](/usage/training#custom-functions). Also see the
usage documentation on
[layers and model architectures](/usage/layers-architectures).
## Tok2Vec architectures {#tok2vec-arch source="spacy/ml/models/tok2vec.py"}
## Tok2Vec architectures {id="tok2vec-arch",source="spacy/ml/models/tok2vec.py"}
### spacy.Tok2Vec.v2 {#Tok2Vec}
### spacy.Tok2Vec.v2 {id="Tok2Vec"}
> #### Example config
>
@ -56,7 +56,7 @@ blog post for background.
| `encode` | Encode context into the embeddings, using an architecture such as a CNN, BiLSTM or transformer. For example, [MaxoutWindowEncoder](/api/architectures#MaxoutWindowEncoder). ~~Model[List[Floats2d], List[Floats2d]]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.HashEmbedCNN.v2 {#HashEmbedCNN}
### spacy.HashEmbedCNN.v2 {id="HashEmbedCNN"}
> #### Example Config
>
@ -89,7 +89,7 @@ consisting of a CNN and a layer-normalized maxout activation function.
| `pretrained_vectors` | Whether to also use static vectors. ~~bool~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.Tok2VecListener.v1 {#Tok2VecListener}
### spacy.Tok2VecListener.v1 {id="Tok2VecListener"}
> #### Example config
>
@ -139,7 +139,7 @@ the `Tok2Vec` component.
| `upstream` | A string to identify the "upstream" `Tok2Vec` component to communicate with. By default, the upstream name is the wildcard string `"*"`, but you could also specify the name of the `Tok2Vec` component. You'll almost never have multiple upstream `Tok2Vec` components, so the wildcard string will almost always be fine. ~~str~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.MultiHashEmbed.v2 {#MultiHashEmbed}
### spacy.MultiHashEmbed.v2 {id="MultiHashEmbed"}
> #### Example config
>
@ -170,7 +170,7 @@ updated).
| `include_static_vectors` | Whether to also use static word vectors. Requires a vectors table to be loaded in the [`Doc`](/api/doc) objects' vocab. ~~bool~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.CharacterEmbed.v2 {#CharacterEmbed}
### spacy.CharacterEmbed.v2 {id="CharacterEmbed"}
> #### Example config
>
@ -207,7 +207,7 @@ network to construct a single vector to represent the information.
| `nC` | The number of UTF-8 bytes to embed per word. Recommended values are between `3` and `8`, although it may depend on the length of words in the language. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.MaxoutWindowEncoder.v2 {#MaxoutWindowEncoder}
### spacy.MaxoutWindowEncoder.v2 {id="MaxoutWindowEncoder"}
> #### Example config
>
@ -231,7 +231,7 @@ and residual connections.
| `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ |
### spacy.MishWindowEncoder.v2 {#MishWindowEncoder}
### spacy.MishWindowEncoder.v2 {id="MishWindowEncoder"}
> #### Example config
>
@ -254,7 +254,7 @@ and residual connections.
| `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ |
### spacy.TorchBiLSTMEncoder.v1 {#TorchBiLSTMEncoder}
### spacy.TorchBiLSTMEncoder.v1 {id="TorchBiLSTMEncoder"}
> #### Example config
>
@ -276,7 +276,7 @@ Encode context using bidirectional LSTM layers. Requires
| `dropout` | Creates a Dropout layer on the outputs of each LSTM layer except the last layer. Set to 0.0 to disable this functionality. ~~float~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ |
### spacy.StaticVectors.v2 {#StaticVectors}
### spacy.StaticVectors.v2 {id="StaticVectors"}
> #### Example config
>
@ -306,7 +306,7 @@ mapped to a zero vector. See the documentation on
| `key_attr` | Defaults to `"ORTH"`. ~~str~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Ragged]~~ |
### spacy.FeatureExtractor.v1 {#FeatureExtractor}
### spacy.FeatureExtractor.v1 {id="FeatureExtractor"}
> #### Example config
>
@ -324,7 +324,7 @@ of feature names to extract, which should refer to token attributes.
| `columns` | The token attributes to extract. ~~List[Union[int, str]]~~ |
| **CREATES** | The created feature extraction layer. ~~Model[List[Doc], List[Ints2d]]~~ |
## Transformer architectures {#transformers source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/architectures.py"}
## Transformer architectures {id="transformers",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/architectures.py"}
The following architectures are provided by the package
[`spacy-transformers`](https://github.com/explosion/spacy-transformers). See the
@ -341,7 +341,7 @@ for details and system requirements.
</Infobox>
### spacy-transformers.TransformerModel.v3 {#TransformerModel}
### spacy-transformers.TransformerModel.v3 {id="TransformerModel"}
> #### Example Config
>
@ -390,7 +390,7 @@ in other components, see
| | |
<Infobox title="Mixed precision support" variant="warning">
Mixed-precision support is currently an experimental feature.
Mixed-precision support is currently an experimental feature.
</Infobox>
<Accordion title="Previous versions of spacy-transformers.TransformerModel" spaced>
@ -404,7 +404,7 @@ The other arguments are shared between all versions.
</Accordion>
### spacy-transformers.TransformerListener.v1 {#TransformerListener}
### spacy-transformers.TransformerListener.v1 {id="TransformerListener"}
> #### Example Config
>
@ -434,7 +434,7 @@ a single token vector given zero or more wordpiece vectors.
| `upstream` | A string to identify the "upstream" `Transformer` component to communicate with. By default, the upstream name is the wildcard string `"*"`, but you could also specify the name of the `Transformer` component. You'll almost never have multiple upstream `Transformer` components, so the wildcard string will almost always be fine. ~~str~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy-transformers.Tok2VecTransformer.v3 {#Tok2VecTransformer}
### spacy-transformers.Tok2VecTransformer.v3 {id="Tok2VecTransformer"}
> #### Example Config
>
@ -467,7 +467,7 @@ one component.
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
<Infobox title="Mixed precision support" variant="warning">
Mixed-precision support is currently an experimental feature.
Mixed-precision support is currently an experimental feature.
</Infobox>
<Accordion title="Previous versions of spacy-transformers.Tok2VecTransformer" spaced>
@ -481,7 +481,7 @@ The other arguments are shared between all versions.
</Accordion>
## Pretraining architectures {#pretrain source="spacy/ml/models/multi_task.py"}
## Pretraining architectures {id="pretrain",source="spacy/ml/models/multi_task.py"}
The spacy `pretrain` command lets you initialize a `Tok2Vec` layer in your
pipeline with information from raw text. To this end, additional layers are
@ -494,7 +494,7 @@ BERT.
For more information, see the section on
[pretraining](/usage/embeddings-transformers#pretraining).
### spacy.PretrainVectors.v1 {#pretrain_vectors}
### spacy.PretrainVectors.v1 {id="pretrain_vectors"}
> #### Example config
>
@ -525,7 +525,7 @@ vectors.
| `loss` | The loss function can be either "cosine" or "L2". We typically recommend to use "cosine". ~~~str~~ |
| **CREATES** | A callable function that can create the Model, given the `vocab` of the pipeline and the `tok2vec` layer to pretrain. ~~Callable[[Vocab, Model], Model]~~ |
### spacy.PretrainCharacters.v1 {#pretrain_chars}
### spacy.PretrainCharacters.v1 {id="pretrain_chars"}
> #### Example config
>
@ -551,9 +551,9 @@ for a Tok2Vec layer.
| `n_characters` | The window of characters - e.g. if `n_characters = 2`, the model will try to predict the first two and last two characters of the word. ~~int~~ |
| **CREATES** | A callable function that can create the Model, given the `vocab` of the pipeline and the `tok2vec` layer to pretrain. ~~Callable[[Vocab, Model], Model]~~ |
## Parser & NER architectures {#parser}
## Parser & NER architectures {id="parser"}
### spacy.TransitionBasedParser.v2 {#TransitionBasedParser source="spacy/ml/models/parser.py"}
### spacy.TransitionBasedParser.v2 {id="TransitionBasedParser",source="spacy/ml/models/parser.py"}
> #### Example Config
>
@ -612,9 +612,9 @@ same signature, but the `use_upper` argument was `True` by default.
</Accordion>
## Tagging architectures {#tagger source="spacy/ml/models/tagger.py"}
## Tagging architectures {id="tagger",source="spacy/ml/models/tagger.py"}
### spacy.Tagger.v2 {#Tagger}
### spacy.Tagger.v2 {id="Tagger"}
> #### Example Config
>
@ -648,7 +648,7 @@ The other arguments are shared between all versions.
</Accordion>
## Text classification architectures {#textcat source="spacy/ml/models/textcat.py"}
## Text classification architectures {id="textcat",source="spacy/ml/models/textcat.py"}
A text classification architecture needs to take a [`Doc`](/api/doc) as input,
and produce a score for each potential label class. Textcat challenges can be
@ -672,7 +672,7 @@ single-label use-cases where `exclusive_classes = true`, while the
</Infobox>
### spacy.TextCatEnsemble.v2 {#TextCatEnsemble}
### spacy.TextCatEnsemble.v2 {id="TextCatEnsemble"}
> #### Example Config
>
@ -737,7 +737,7 @@ but used an internal `tok2vec` instead of taking it as argument:
</Accordion>
### spacy.TextCatCNN.v2 {#TextCatCNN}
### spacy.TextCatCNN.v2 {id="TextCatCNN"}
> #### Example Config
>
@ -777,7 +777,7 @@ after training.
</Accordion>
### spacy.TextCatBOW.v2 {#TextCatBOW}
### spacy.TextCatBOW.v2 {id="TextCatBOW"}
> #### Example Config
>
@ -809,9 +809,9 @@ after training.
</Accordion>
## Span classification architectures {#spancat source="spacy/ml/models/spancat.py"}
## Span classification architectures {id="spancat",source="spacy/ml/models/spancat.py"}
### spacy.SpanCategorizer.v1 {#SpanCategorizer}
### spacy.SpanCategorizer.v1 {id="SpanCategorizer"}
> #### Example Config
>
@ -848,7 +848,7 @@ single vector, and a scorer model to map the vectors to probabilities.
| `scorer` | The scorer model. ~~Model[Floats2d, Floats2d]~~ |
| **CREATES** | The model using the architecture. ~~Model[Tuple[List[Doc], Ragged], Floats2d]~~ |
### spacy.mean_max_reducer.v1 {#mean_max_reducer}
### spacy.mean_max_reducer.v1 {id="mean_max_reducer"}
Reduce sequences by concatenating their mean and max pooled vectors, and then
combine the concatenated vectors with a hidden layer.
@ -857,7 +857,7 @@ combine the concatenated vectors with a hidden layer.
| ------------- | ------------------------------------- |
| `hidden_size` | The size of the hidden layer. ~~int~~ |
## Entity linking architectures {#entitylinker source="spacy/ml/models/entity_linker.py"}
## Entity linking architectures {id="entitylinker",source="spacy/ml/models/entity_linker.py"}
An [`EntityLinker`](/api/entitylinker) component disambiguates textual mentions
(tagged as named entities) to unique identifiers, grounding the named entities
@ -870,7 +870,7 @@ into the "real world". This requires 3 main components:
- A machine learning [`Model`](https://thinc.ai/docs/api-model) that picks the
most plausible ID from the set of candidates.
### spacy.EntityLinker.v2 {#EntityLinker}
### spacy.EntityLinker.v2 {id="EntityLinker"}
> #### Example Config
>
@ -899,7 +899,7 @@ The `EntityLinker` model architecture is a Thinc `Model` with a
| `nO` | Output dimension, determined by the length of the vectors encoding each entity in the KB. If the `nO` dimension is not set, the entity linking component will set it when `initialize` is called. ~~Optional[int]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
### spacy.EmptyKB.v1 {#EmptyKB}
### spacy.EmptyKB.v1 {id="EmptyKB"}
A function that creates an empty `KnowledgeBase` from a [`Vocab`](/api/vocab)
instance. This is the default when a new entity linker component is created.
@ -908,7 +908,7 @@ instance. This is the default when a new entity linker component is created.
| ---------------------- | ----------------------------------------------------------------------------------- |
| `entity_vector_length` | The length of the vectors encoding each entity in the KB. Defaults to `64`. ~~int~~ |
### spacy.KBFromFile.v1 {#KBFromFile}
### spacy.KBFromFile.v1 {id="KBFromFile"}
A function that reads an existing `KnowledgeBase` from file.
@ -916,7 +916,7 @@ A function that reads an existing `KnowledgeBase` from file.
| --------- | -------------------------------------------------------- |
| `kb_path` | The location of the KB that was stored to file. ~~Path~~ |
### spacy.CandidateGenerator.v1 {#CandidateGenerator}
### spacy.CandidateGenerator.v1 {id="CandidateGenerator"}
A function that takes as input a [`KnowledgeBase`](/api/kb) and a
[`Span`](/api/span) object denoting a named entity, and returns a list of
@ -924,7 +924,7 @@ plausible [`Candidate`](/api/kb/#candidate) objects. The default
`CandidateGenerator` uses the text of a mention to find its potential aliases in
the `KnowledgeBase`. Note that this function is case-dependent.
## Coreference {#coref-architectures tag="experimental"}
## Coreference {id="coref-architectures",tag="experimental"}
A [`CoreferenceResolver`](/api/coref) component identifies tokens that refer to
the same entity. A [`SpanResolver`](/api/span-resolver) component infers spans
@ -932,7 +932,7 @@ from single tokens. Together these components can be used to reproduce
traditional coreference models. You can also omit the `SpanResolver` if working
with only token-level clusters is acceptable.
### spacy-experimental.Coref.v1 {#Coref tag="experimental"}
### spacy-experimental.Coref.v1 {id="Coref",tag="experimental"}
> #### Example Config
>
@ -967,7 +967,7 @@ The `Coref` model architecture is a Thinc `Model`.
| `antecedent_batch_size` | Internal batch size. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
### spacy-experimental.SpanResolver.v1 {#SpanResolver tag="experimental"}
### spacy-experimental.SpanResolver.v1 {id="SpanResolver",tag="experimental"}
> #### Example Config
>

30
website/docs/api/attributeruler.md → website/docs/api/attributeruler.mdx
View File

@ -2,7 +2,7 @@
title: AttributeRuler
tag: class
source: spacy/pipeline/attributeruler.py
new: 3
version: 3
teaser: 'Pipeline component for rule-based token attribute assignment'
api_string_name: attribute_ruler
api_trainable: false
@ -15,7 +15,7 @@ between attributes such as mapping fine-grained POS tags to coarse-grained POS
tags. See the [usage guide](/usage/linguistic-features/#mappings-exceptions) for
examples.
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -37,7 +37,7 @@ how the component should be configured. You can override its settings via the
%%GITHUB_SPACY/spacy/pipeline/attributeruler.py
```
## AttributeRuler.\_\_init\_\_ {#init tag="method"}
## AttributeRuler.\_\_init\_\_ {id="init",tag="method"}
Initialize the attribute ruler.
@ -56,7 +56,7 @@ Initialize the attribute ruler.
| `validate` | Whether patterns should be validated (passed to the [`Matcher`](/api/matcher#init)). Defaults to `False`. ~~bool~~ |
| `scorer` | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attributes `"tag`", `"pos"`, `"morph"` and `"lemma"` and [`Scorer.score_token_attr_per_feat`](/api/scorer#score_token_attr_per_feat) for the attribute `"morph"`. ~~Optional[Callable]~~ |
## AttributeRuler.\_\_call\_\_ {#call tag="method"}
## AttributeRuler.\_\_call\_\_ {id="call",tag="method"}
Apply the attribute ruler to a `Doc`, setting token attributes for tokens
matched by the provided patterns.
@ -66,7 +66,7 @@ matched by the provided patterns.
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## AttributeRuler.add {#add tag="method"}
## AttributeRuler.add {id="add",tag="method"}
Add patterns to the attribute ruler. The patterns are a list of `Matcher`
patterns and the attributes are a dict of attributes to set on the matched
@ -89,7 +89,7 @@ may be negative to index from the end of the span.
| `attrs` | The attributes to assign to the target token in the matched span. ~~Dict[str, Any]~~ |
| `index` | The index of the token in the matched span to modify. May be negative to index from the end of the span. Defaults to `0`. ~~int~~ |
## AttributeRuler.add_patterns {#add_patterns tag="method"}
## AttributeRuler.add_patterns {id="add_patterns",tag="method"}
> #### Example
>
@ -116,7 +116,7 @@ keys `"patterns"`, `"attrs"` and `"index"`, which match the arguments of
| ---------- | -------------------------------------------------------------------------- |
| `patterns` | The patterns to add. ~~Iterable[Dict[str, Union[List[dict], dict, int]]]~~ |
## AttributeRuler.patterns {#patterns tag="property"}
## AttributeRuler.patterns {id="patterns",tag="property"}
Get all patterns that have been added to the attribute ruler in the
`patterns_dict` format accepted by
@ -126,7 +126,7 @@ Get all patterns that have been added to the attribute ruler in the
| ----------- | -------------------------------------------------------------------------------------------- |
| **RETURNS** | The patterns added to the attribute ruler. ~~List[Dict[str, Union[List[dict], dict, int]]]~~ |
## AttributeRuler.initialize {#initialize tag="method"}
## AttributeRuler.initialize {id="initialize",tag="method"}
Initialize the component with data and used before training to load in rules
from a file. This method is typically called by
@ -160,7 +160,7 @@ config.
| `tag_map` | The tag map that maps fine-grained tags to coarse-grained tags and morphological features. Defaults to `None`. ~~Optional[Dict[str, Dict[Union[int, str], Union[int, str]]]]~~ |
| `morph_rules` | The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Dict[Union[int, str], Union[int, str]]]]]~~ |
## AttributeRuler.load_from_tag_map {#load_from_tag_map tag="method"}
## AttributeRuler.load_from_tag_map {id="load_from_tag_map",tag="method"}
Load attribute ruler patterns from a tag map.
@ -168,7 +168,7 @@ Load attribute ruler patterns from a tag map.
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `tag_map` | The tag map that maps fine-grained tags to coarse-grained tags and morphological features. ~~Dict[str, Dict[Union[int, str], Union[int, str]]]~~ |
## AttributeRuler.load_from_morph_rules {#load_from_morph_rules tag="method"}
## AttributeRuler.load_from_morph_rules {id="load_from_morph_rules",tag="method"}
Load attribute ruler patterns from morph rules.
@ -176,7 +176,7 @@ Load attribute ruler patterns from morph rules.
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `morph_rules` | The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. ~~Dict[str, Dict[str, Dict[Union[int, str], Union[int, str]]]]~~ |
## AttributeRuler.to_disk {#to_disk tag="method"}
## AttributeRuler.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -193,7 +193,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## AttributeRuler.from_disk {#from_disk tag="method"}
## AttributeRuler.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -211,7 +211,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `AttributeRuler` object. ~~AttributeRuler~~ |
## AttributeRuler.to_bytes {#to_bytes tag="method"}
## AttributeRuler.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -228,7 +228,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `AttributeRuler` object. ~~bytes~~ |
## AttributeRuler.from_bytes {#from_bytes tag="method"}
## AttributeRuler.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -247,7 +247,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `AttributeRuler` object. ~~AttributeRuler~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

7
website/docs/api/attributes.md → website/docs/api/attributes.mdx
View File

@ -41,10 +41,9 @@ from string attribute names to internal attribute IDs is stored in
The corresponding [`Token` object attributes](/api/token#attributes) can be
accessed using the same names in lowercase, e.g. `token.orth` or `token.length`.
For attributes that represent string values, the internal integer ID is
accessed as `Token.attr`, e.g. `token.dep`, while the string value can be
retrieved by appending `_` as in `token.dep_`.
For attributes that represent string values, the internal integer ID is accessed
as `Token.attr`, e.g. `token.dep`, while the string value can be retrieved by
appending `_` as in `token.dep_`.
| Attribute | Description |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |

177
website/docs/api/cli.md → website/docs/api/cli.mdx
View File

@ -26,7 +26,7 @@ a list of available commands, you can type `python -m spacy --help`. You can
also add the `--help` flag to any command or subcommand to see the description,
available arguments and usage.
## download {#download tag="command"}
## download {id="download",tag="command"}
Download [trained pipelines](/usage/models) for spaCy. The downloader finds the
best-matching compatible version and uses `pip install` to download the Python
@ -44,7 +44,7 @@ pipeline name to be specified with its version (e.g. `en_core_web_sm-3.0.0`).
> will also allow you to add it as a versioned package dependency to your
> project.
```cli
```bash
$ python -m spacy download [model] [--direct] [--sdist] [pip_args]
```
@ -57,24 +57,24 @@ $ python -m spacy download [model] [--direct] [--sdist] [pip_args]
| pip args | Additional installation options to be passed to `pip install` when installing the pipeline package. For example, `--user` to install to the user home directory or `--no-deps` to not install package dependencies. ~~Any (option/flag)~~ |
| **CREATES** | The installed pipeline package in your `site-packages` directory. |
## info {#info tag="command"}
## info {id="info",tag="command"}
Print information about your spaCy installation, trained pipelines and local
setup, and generate [Markdown](https://en.wikipedia.org/wiki/Markdown)-formatted
markup to copy-paste into
[GitHub issues](https://github.com/explosion/spaCy/issues).
```cli
```bash
$ python -m spacy info [--markdown] [--silent] [--exclude]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy info en_core_web_lg --markdown
> ```
```cli
```bash
$ python -m spacy info [model] [--markdown] [--silent] [--exclude]
```
@ -88,7 +88,7 @@ $ python -m spacy info [model] [--markdown] [--silent] [--exclude]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **PRINTS** | Information about your spaCy installation. |
## validate {#validate new="2" tag="command"}
## validate {id="validate",version="2",tag="command"}
Find all trained pipeline packages installed in the current environment and
check whether they are compatible with the currently installed version of spaCy.
@ -103,7 +103,7 @@ compatible versions and command for updating are shown.
> suite, to ensure all packages are up to date before proceeding. If
> incompatible packages are found, it will return `1`.
```cli
```bash
$ python -m spacy validate
```
@ -111,12 +111,12 @@ $ python -m spacy validate
| ---------- | -------------------------------------------------------------------- |
| **PRINTS** | Details about the compatibility of your installed pipeline packages. |
## init {#init new="3"}
## init {id="init",version="3"}
The `spacy init` CLI includes helpful commands for initializing training config
files and pipeline directories.
### init config {#init-config new="3" tag="command"}
### init config {id="init-config",version="3",tag="command"}
Initialize and save a [`config.cfg` file](/usage/training#config) using the
**recommended settings** for your use case. It works just like the
@ -128,11 +128,11 @@ customize those settings in your config file later.
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy init config config.cfg --lang en --pipeline ner,textcat --optimize accuracy
> ```
```cli
```bash
$ python -m spacy init config [output_file] [--lang] [--pipeline] [--optimize] [--gpu] [--pretraining] [--force]
```
@ -148,7 +148,7 @@ $ python -m spacy init config [output_file] [--lang] [--pipeline] [--optimize] [
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | The config file for training. |
### init fill-config {#init-fill-config new="3"}
### init fill-config {id="init-fill-config",version="3"}
Auto-fill a partial [.cfg file](/usage/training#config) with **all default
values**, e.g. a config generated with the
@ -162,15 +162,15 @@ validation error with more details.
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy init fill-config base.cfg config.cfg --diff
> ```
>
> #### Example diff
>
> ![Screenshot of visual diff in terminal](../images/cli_init_fill-config_diff.jpg)
> ![Screenshot of visual diff in terminal](/images/cli_init_fill-config_diff.jpg)
```cli
```bash
$ python -m spacy init fill-config [base_path] [output_file] [--diff]
```
@ -184,7 +184,7 @@ $ python -m spacy init fill-config [base_path] [output_file] [--diff]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | Complete and auto-filled config file for training. |
### init vectors {#init-vectors new="3" tag="command"}
### init vectors {id="init-vectors",version="3",tag="command"}
Convert [word vectors](/usage/linguistic-features#vectors-similarity) for use
with spaCy. Will export an `nlp` object that you can use in the
@ -199,7 +199,7 @@ This functionality was previously available as part of the command `init-model`.
</Infobox>
```cli
```bash
$ python -m spacy init vectors [lang] [vectors_loc] [output_dir] [--prune] [--truncate] [--name] [--verbose]
```
@ -216,7 +216,7 @@ $ python -m spacy init vectors [lang] [vectors_loc] [output_dir] [--prune] [--tr
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | A spaCy pipeline directory containing the vocab and vectors. |
### init labels {#init-labels new="3" tag="command"}
### init labels {id="init-labels",version="3",tag="command"}
Generate JSON files for the labels in the data. This helps speed up the training
process, since spaCy won't have to preprocess the data to extract the labels.
@ -234,7 +234,7 @@ After generating the labels, you can provide them to components that accept a
> path = "corpus/labels/ner.json
> ```
```cli
```bash
$ python -m spacy init labels [config_path] [output_path] [--code] [--verbose] [--gpu-id] [overrides]
```
@ -249,7 +249,7 @@ $ python -m spacy init labels [config_path] [output_path] [--code] [--verbose] [
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ |
| **CREATES** | The label files. |
## convert {#convert tag="command"}
## convert {id="convert",tag="command"}
Convert files into spaCy's
[binary training data format](/api/data-formats#binary-training), a serialized
@ -257,7 +257,7 @@ Convert files into spaCy's
management functions. The converter can be specified on the command line, or
chosen based on the file extension of the input file.
```cli
```bash
$ python -m spacy convert [input_file] [output_dir] [--converter] [--file-type] [--n-sents] [--seg-sents] [--base] [--morphology] [--merge-subtokens] [--ner-map] [--lang]
```
@ -278,7 +278,7 @@ $ python -m spacy convert [input_file] [output_dir] [--converter] [--file-type]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | Binary [`DocBin`](/api/docbin) training data that can be used with [`spacy train`](/api/cli#train). |
### Converters {#converters}
### Converters {id="converters"}
| ID | Description |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@ -288,12 +288,12 @@ $ python -m spacy convert [input_file] [output_dir] [--converter] [--file-type]
| `ner` / `conll` | NER with IOB/IOB2/BILUO tags, one token per line with columns separated by whitespace. The first column is the token and the final column is the NER tag. Sentences are separated by blank lines and documents are separated by the line `-DOCSTART- -X- O O`. Supports CoNLL 2003 NER format. See [sample data](%%GITHUB_SPACY/extra/example_data/ner_example_data). |
| `iob` | NER with IOB/IOB2/BILUO tags, one sentence per line with tokens separated by whitespace and annotation separated by `\|`, either `word\|B-ENT`or`word\|POS\|B-ENT`. See [sample data](%%GITHUB_SPACY/extra/example_data/ner_example_data). |
## debug {#debug new="3"}
## debug {id="debug",version="3"}
The `spacy debug` CLI includes helpful commands for debugging and profiling your
configs, data and implementations.
### debug config {#debug-config new="3" tag="command"}
### debug config {id="debug-config",version="3",tag="command"}
Debug a [`config.cfg` file](/usage/training#config) and show validation errors.
The command will create all objects in the tree and validate them. Note that
@ -303,13 +303,13 @@ errors at once and some issues are only shown once previous errors have been
fixed. To auto-fill a partial config and save the result, you can use the
[`init fill-config`](/api/cli#init-fill-config) command.
```cli
```bash
$ python -m spacy debug config [config_path] [--code] [--show-functions] [--show-variables] [overrides]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy debug config config.cfg
> ```
@ -333,7 +333,7 @@ python -m spacy init fill-config tmp/starter-config_invalid.cfg tmp/starter-conf
<Accordion title="Example output (valid config and all options)" spaced>
```cli
```bash
$ python -m spacy debug config ./config.cfg --show-functions --show-variables
```
@ -453,7 +453,7 @@ File /path/to/thinc/thinc/schedules.py (line 91)
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ |
| **PRINTS** | Config validation errors, if available. |
### debug data {#debug-data tag="command"}
### debug data {id="debug-data",tag="command"}
Analyze, debug and validate your training and development data. Get useful
stats, and find problems like invalid entity annotations, cyclic dependencies,
@ -479,13 +479,13 @@ the token distributions. To learn more, you can check out Papay et al.'s work on
</Infobox>
```cli
```bash
$ python -m spacy debug data [config_path] [--code] [--ignore-warnings] [--verbose] [--no-format] [overrides]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy debug data ./config.cfg
> ```
@ -639,7 +639,7 @@ will not be available.
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ |
| **PRINTS** | Debugging information. |
### debug diff-config {#debug-diff tag="command"}
### debug diff-config {id="debug-diff",tag="command"}
Show a diff of a config file with respect to spaCy's defaults or another config
file. If additional settings were used in the creation of the config file, then
@ -647,13 +647,13 @@ you must supply these as extra parameters to the command when comparing to the
default settings. The generated diff can also be used when posting to the
discussion forum to provide more information for the maintainers.
```cli
```bash
$ python -m spacy debug diff-config [config_path] [--compare-to] [--optimize] [--gpu] [--pretraining] [--markdown]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy debug diff-config ./config.cfg
> ```
@ -868,7 +868,7 @@ after_init = null
| `markdown`, `-md` | Generate Markdown for Github issues. Defaults to `False`. ~~bool (flag)~~ |
| **PRINTS** | Diff between the two config files. |
### debug profile {#debug-profile tag="command"}
### debug profile {id="debug-profile",tag="command"}
Profile which functions take the most time in a spaCy pipeline. Input should be
formatted as one JSON object per line with a key `"text"`. It can either be
@ -882,7 +882,7 @@ The `profile` command is now available as a subcommand of `spacy debug`.
</Infobox>
```cli
```bash
$ python -m spacy debug profile [model] [inputs] [--n-texts]
```
@ -894,12 +894,12 @@ $ python -m spacy debug profile [model] [inputs] [--n-texts]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **PRINTS** | Profiling information for the pipeline. |
### debug model {#debug-model new="3" tag="command"}
### debug model {id="debug-model",version="3",tag="command"}
Debug a Thinc [`Model`](https://thinc.ai/docs/api-model) by running it on a
sample text and checking how it updates its internal weights and parameters.
```cli
```bash
$ python -m spacy debug model [config_path] [component] [--layers] [--dimensions] [--parameters] [--gradients] [--attributes] [--print-step0] [--print-step1] [--print-step2] [--print-step3] [--gpu-id]
```
@ -910,7 +910,7 @@ model ("Step 0"), which helps us to understand the internal structure of the
Neural Network, and to focus on specific layers that we want to inspect further
(see next example).
```cli
```bash
$ python -m spacy debug model ./config.cfg tagger -P0
```
@ -956,7 +956,7 @@ an all-zero matrix determined by the `nO` and `nI` dimensions. After a first
training step (Step 2), this matrix has clearly updated its values through the
training feedback loop.
```cli
```bash
$ python -m spacy debug model ./config.cfg tagger -l "5,15" -DIM -PAR -P0 -P1 -P2
```
@ -1017,7 +1017,7 @@ $ python -m spacy debug model ./config.cfg tagger -l "5,15" -DIM -PAR -P0 -P1 -P
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ |
| **PRINTS** | Debugging information. |
## train {#train tag="command"}
## train {id="train",tag="command"}
Train a pipeline. Expects data in spaCy's
[binary format](/api/data-formats#training) and a
@ -1043,11 +1043,11 @@ in the section `[paths]`.
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy train config.cfg --output ./output --paths.train ./train --paths.dev ./dev
> ```
```cli
```bash
$ python -m spacy train [config_path] [--output] [--code] [--verbose] [--gpu-id] [overrides]
```
@ -1062,7 +1062,7 @@ $ python -m spacy train [config_path] [--output] [--code] [--verbose] [--gpu-id]
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ |
| **CREATES** | The final trained pipeline and the best trained pipeline. |
### Calling the training function from Python {#train-function new="3.2"}
### Calling the training function from Python {id="train-function",version="3.2"}
The training CLI exposes a `train` helper function that lets you run the
training just like `spacy train`. Usually it's easier to use the command line
@ -1085,7 +1085,7 @@ directly, but if you need to kick off training from code this is how to do it.
| `use_gpu` | Which GPU to use. Defaults to -1 for no GPU. ~~int~~ |
| `overrides` | Values to override config settings. ~~Dict[str, Any]~~ |
## pretrain {#pretrain new="2.1" tag="command,experimental"}
## pretrain {id="pretrain",version="2.1",tag="command,experimental"}
Pretrain the "token to vector" ([`Tok2vec`](/api/tok2vec)) layer of pipeline
components on raw text, using an approximate language-modeling objective.
@ -1113,11 +1113,11 @@ auto-generated by setting `--pretraining` on
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy pretrain config.cfg ./output_pretrain --paths.raw_text ./data.jsonl
> ```
```cli
```bash
$ python -m spacy pretrain [config_path] [output_dir] [--code] [--resume-path] [--epoch-resume] [--gpu-id] [overrides]
```
@ -1133,7 +1133,7 @@ $ python -m spacy pretrain [config_path] [output_dir] [--code] [--resume-path] [
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--training.dropout 0.2`. ~~Any (option/flag)~~ |
| **CREATES** | The pretrained weights that can be used to initialize `spacy train`. |
## evaluate {#evaluate new="2" tag="command"}
## evaluate {id="evaluate",version="2",tag="command"}
Evaluate a trained pipeline. Expects a loadable spaCy pipeline (package name or
path) and evaluation data in the
@ -1146,7 +1146,7 @@ skew. To render a sample of dependency parses in a HTML file using the
[displaCy visualizations](/usage/visualizers), set as output directory as the
`--displacy-path` argument.
```cli
```bash
$ python -m spacy evaluate [model] [data_path] [--output] [--code] [--gold-preproc] [--gpu-id] [--displacy-path] [--displacy-limit]
```
@ -1163,7 +1163,7 @@ $ python -m spacy evaluate [model] [data_path] [--output] [--code] [--gold-prepr
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | Training results and optional metrics and visualizations. |
## apply {#apply new="3.5" tag="command"}
## apply {id="apply", version="3.5", tag="command"}
Applies a trained pipeline to data and stores the resulting annotated documents
in a `DocBin`. The input can be a single file or a directory. The recognized
@ -1194,7 +1194,8 @@ $ python -m spacy apply [model] [data-path] [output-file] [--code] [--text-key]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | A `DocBin` with the annotations from the `model` for all the files found in `data-path`. |
## find-threshold {#find-threshold new="3.5" tag="command"}
## find-threshold {id="find-threshold",version="3.5",tag="command"}
Runs prediction trials for a trained model with varying tresholds to maximize
the specified metric. The search space for the threshold is traversed linearly
@ -1209,12 +1210,12 @@ be provided.
> #### Examples
>
> ```cli
> ```bash
> # For textcat_multilabel:
> $ python -m spacy find-threshold my_nlp data.spacy textcat_multilabel threshold cats_macro_f
> ```
>
> ```cli
> ```bash
> # For spancat:
> $ python -m spacy find-threshold my_nlp data.spacy spancat threshold spans_sc_f
> ```
@ -1233,7 +1234,7 @@ be provided.
| `--silent`, `-V`, `-VV` | GPU to use, if any. Defaults to `-1` for CPU. ~~int (option)~~ |
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
## assemble {#assemble tag="command"}
## assemble {id="assemble",tag="command"}
Assemble a pipeline from a config file without additional training. Expects a
[config file](/api/data-formats#config) with all settings and hyperparameters.
@ -1243,11 +1244,11 @@ config.
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy assemble config.cfg ./output
> ```
```cli
```bash
$ python -m spacy assemble [config_path] [output_dir] [--code] [--verbose] [overrides]
```
@ -1261,7 +1262,7 @@ $ python -m spacy assemble [config_path] [output_dir] [--code] [--verbose] [over
| overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.data ./data`. ~~Any (option/flag)~~ |
| **CREATES** | The final assembled pipeline. |
## package {#package tag="command"}
## package {id="package",tag="command"}
Generate an installable [Python package](/usage/training#models-generating) from
an existing pipeline data directory. All data files are copied over. If
@ -1287,13 +1288,13 @@ the sdist and wheel by setting `--build sdist,wheel`.
</Infobox>
```cli
```bash
$ python -m spacy package [input_dir] [output_dir] [--code] [--meta-path] [--create-meta] [--build] [--name] [--version] [--force]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy package /input /output
> $ cd /output/en_pipeline-0.0.0
> $ pip install dist/en_pipeline-0.0.0.tar.gz
@ -1313,13 +1314,13 @@ $ python -m spacy package [input_dir] [output_dir] [--code] [--meta-path] [--cre
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | A Python package containing the spaCy pipeline. |
## project {#project new="3"}
## project {id="project",version="3"}
The `spacy project` CLI includes subcommands for working with
[spaCy projects](/usage/projects), end-to-end workflows for building and
deploying custom spaCy pipelines.
### project clone {#project-clone tag="command"}
### project clone {id="project-clone",tag="command"}
Clone a project template from a Git repository. Calls into `git` under the hood
and can use the sparse checkout feature if available, so you're only downloading
@ -1328,19 +1329,19 @@ what you need. By default, spaCy's
can provide any other repo (public or private) that you have access to using the
`--repo` option.
```cli
```bash
$ python -m spacy project clone [name] [dest] [--repo] [--branch] [--sparse]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy project clone pipelines/ner_wikiner
> ```
>
> Clone from custom repo:
>
> ```cli
> ```bash
> $ python -m spacy project clone template --repo https://github.com/your_org/your_repo
> ```
@ -1354,7 +1355,7 @@ $ python -m spacy project clone [name] [dest] [--repo] [--branch] [--sparse]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | The cloned [project directory](/usage/projects#project-files). |
### project assets {#project-assets tag="command"}
### project assets {id="project-assets",tag="command"}
Fetch project assets like datasets and pretrained weights. Assets are defined in
the `assets` section of the [`project.yml`](/usage/projects#project-yml). If a
@ -1365,13 +1366,13 @@ considered "private" and you have to take care of putting them into the
destination directory yourself. If a local path is provided, the asset is copied
into the current project.
```cli
```bash
$ python -m spacy project assets [project_dir]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy project assets [--sparse]
> ```
@ -1382,7 +1383,7 @@ $ python -m spacy project assets [project_dir]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | Downloaded or copied assets defined in the `project.yml`. |
### project run {#project-run tag="command"}
### project run {id="project-run",tag="command"}
Run a named command or workflow defined in the
[`project.yml`](/usage/projects#project-yml). If a workflow name is specified,
@ -1391,13 +1392,13 @@ all commands in the workflow are run, in order. If commands define
re-run if state has changed. For example, if the input dataset changes, a
preprocessing command that depends on those files will be re-run.
```cli
```bash
$ python -m spacy project run [subcommand] [project_dir] [--force] [--dry]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy project run train
> ```
@ -1410,7 +1411,7 @@ $ python -m spacy project run [subcommand] [project_dir] [--force] [--dry]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **EXECUTES** | The command defined in the `project.yml`. |
### project push {#project-push tag="command"}
### project push {id="project-push",tag="command"}
Upload all available files or directories listed as in the `outputs` section of
commands to a remote storage. Outputs are archived and compressed prior to
@ -1430,13 +1431,13 @@ remote storages, so you can use any protocol that `Pathy` supports, including
filesystem, although you may need to install extra dependencies to use certain
protocols.
```cli
```bash
$ python -m spacy project push [remote] [project_dir]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy project push my_bucket
> ```
>
@ -1453,7 +1454,7 @@ $ python -m spacy project push [remote] [project_dir]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **UPLOADS** | All project outputs that exist and are not already stored in the remote. |
### project pull {#project-pull tag="command"}
### project pull {id="project-pull",tag="command"}
Download all files or directories listed as `outputs` for commands, unless they
are not already present locally. When searching for files in the remote, `pull`
@ -1475,13 +1476,13 @@ remote storages, so you can use any protocol that `Pathy` supports, including
filesystem, although you may need to install extra dependencies to use certain
protocols.
```cli
```bash
$ python -m spacy project pull [remote] [project_dir]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy project pull my_bucket
> ```
>
@ -1498,7 +1499,7 @@ $ python -m spacy project pull [remote] [project_dir]
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **DOWNLOADS** | All project outputs that do not exist locally and can be found in the remote. |
### project document {#project-document tag="command"}
### project document {id="project-document",tag="command"}
Auto-generate a pretty Markdown-formatted `README` for your project, based on
its [`project.yml`](/usage/projects#project-yml). Will create sections that
@ -1507,13 +1508,13 @@ content will be placed between two hidden markers, so you can add your own
custom content before or after the auto-generated documentation. When you re-run
the `project document` command, only the auto-generated part is replaced.
```cli
```bash
$ python -m spacy project document [project_dir] [--output] [--no-emoji]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy project document --output README.md
> ```
@ -1522,7 +1523,7 @@ $ python -m spacy project document [project_dir] [--output] [--no-emoji]
For more examples, see the templates in our
[`projects`](https://github.com/explosion/projects) repo.
![Screenshot of auto-generated Markdown Readme](../images/project_document.jpg)
![Screenshot of auto-generated Markdown Readme](/images/project_document.jpg)
</Accordion>
@ -1533,7 +1534,7 @@ For more examples, see the templates in our
| `--no-emoji`, `-NE` | Don't use emoji in the titles. ~~bool (flag)~~ |
| **CREATES** | The Markdown-formatted project documentation. |
### project dvc {#project-dvc tag="command"}
### project dvc {id="project-dvc",tag="command"}
Auto-generate [Data Version Control](https://dvc.org) (DVC) config file. Calls
[`dvc run`](https://dvc.org/doc/command-reference/run) with `--no-exec` under
@ -1553,13 +1554,13 @@ You'll also need to add the assets you want to track with
</Infobox>
```cli
```bash
$ python -m spacy project dvc [project_dir] [workflow] [--force] [--verbose] [--quiet]
```
> #### Example
>
> ```cli
> ```bash
> $ git init
> $ dvc init
> $ python -m spacy project dvc all
@ -1575,14 +1576,14 @@ $ python -m spacy project dvc [project_dir] [workflow] [--force] [--verbose] [--
| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ |
| **CREATES** | A `dvc.yaml` file in the project directory, based on the steps defined in the given workflow. |
## huggingface-hub {#huggingface-hub new="3.1"}
## huggingface-hub {id="huggingface-hub",version="3.1"}
The `spacy huggingface-cli` CLI includes commands for uploading your trained
spaCy pipelines to the [Hugging Face Hub](https://huggingface.co/).
> #### Installation
>
> ```cli
> ```bash
> $ pip install spacy-huggingface-hub
> $ huggingface-cli login
> ```
@ -1596,19 +1597,19 @@ package installed. Installing the package will automatically add the
</Infobox>
### huggingface-hub push {#huggingface-hub-push tag="command"}
### huggingface-hub push {id="huggingface-hub-push",tag="command"}
Push a spaCy pipeline to the Hugging Face Hub. Expects a `.whl` file packaged
with [`spacy package`](/api/cli#package) and `--build wheel`. For more details,
see the spaCy project [integration](/usage/projects#huggingface_hub).
```cli
```bash
$ python -m spacy huggingface-hub push [whl_path] [--org] [--msg] [--local-repo] [--verbose]
```
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl
> ```

32
website/docs/api/coref.md → website/docs/api/coref.mdx
View File

@ -34,7 +34,7 @@ same thing. Clusters are represented as SpanGroups that start with a prefix
A `CoreferenceResolver` component can be paired with a
[`SpanResolver`](/api/span-resolver) to expand single tokens to spans.
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Predictions will be saved to `Doc.spans` as a [`SpanGroup`](/api/spangroup). The
span key will be a prefix plus a serial number referring to the coreference
@ -47,7 +47,7 @@ parameter.
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
| `Doc.spans[prefix + "_" + cluster_number]` | One coreference cluster, represented as single-token spans. Cluster numbers start from 1. ~~SpanGroup~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -73,7 +73,7 @@ details on the architectures and their arguments and hyperparameters.
| `model` | The [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. Defaults to [Coref](/api/architectures#Coref). ~~Model~~ |
| `span_cluster_prefix` | The prefix for the keys for clusters saved to `doc.spans`. Defaults to `coref_clusters`. ~~str~~ |
## CoreferenceResolver.\_\_init\_\_ {#init tag="method"}
## CoreferenceResolver.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -102,7 +102,7 @@ shortcut for this and instantiate the component using its string name and
| _keyword-only_ | |
| `span_cluster_prefix` | The prefix for the key for saving clusters of spans. ~~bool~~ |
## CoreferenceResolver.\_\_call\_\_ {#call tag="method"}
## CoreferenceResolver.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -125,7 +125,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## CoreferenceResolver.pipe {#pipe tag="method"}
## CoreferenceResolver.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -148,7 +148,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/coref#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## CoreferenceResolver.initialize {#initialize tag="method"}
## CoreferenceResolver.initialize {id="initialize",tag="method"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -172,7 +172,7 @@ by [`Language.initialize`](/api/language#initialize).
| _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## CoreferenceResolver.predict {#predict tag="method"}
## CoreferenceResolver.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. Clusters are returned as a list of `MentionClusters`, one for
@ -192,7 +192,7 @@ to token indices.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The predicted coreference clusters for the `docs`. ~~List[MentionClusters]~~ |
## CoreferenceResolver.set_annotations {#set_annotations tag="method"}
## CoreferenceResolver.set_annotations {id="set_annotations",tag="method"}
Modify a batch of documents, saving coreference clusters in `Doc.spans`.
@ -209,7 +209,7 @@ Modify a batch of documents, saving coreference clusters in `Doc.spans`.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `clusters` | The predicted coreference clusters for the `docs`. ~~List[MentionClusters]~~ |
## CoreferenceResolver.update {#update tag="method"}
## CoreferenceResolver.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects. Delegates to
[`predict`](/api/coref#predict).
@ -231,7 +231,7 @@ Learn from a batch of [`Example`](/api/example) objects. Delegates to
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## CoreferenceResolver.create_optimizer {#create_optimizer tag="method"}
## CoreferenceResolver.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -246,7 +246,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## CoreferenceResolver.use_params {#use_params tag="method, contextmanager"}
## CoreferenceResolver.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -263,7 +263,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## CoreferenceResolver.to_disk {#to_disk tag="method"}
## CoreferenceResolver.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -280,7 +280,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## CoreferenceResolver.from_disk {#from_disk tag="method"}
## CoreferenceResolver.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -298,7 +298,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `CoreferenceResolver` object. ~~CoreferenceResolver~~ |
## CoreferenceResolver.to_bytes {#to_bytes tag="method"}
## CoreferenceResolver.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -315,7 +315,7 @@ Serialize the pipe to a bytestring, including the `KnowledgeBase`.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `CoreferenceResolver` object. ~~bytes~~ |
## CoreferenceResolver.from_bytes {#from_bytes tag="method"}
## CoreferenceResolver.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -334,7 +334,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `CoreferenceResolver` object. ~~CoreferenceResolver~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

17
website/docs/api/corpus.md → website/docs/api/corpus.mdx
View File

@ -3,7 +3,7 @@ title: Corpus
teaser: An annotated corpus
tag: class
source: spacy/training/corpus.py
new: 3
version: 3
---
This class manages annotated corpora and can be used for training and
@ -13,7 +13,7 @@ customize the data loading during training, you can register your own
see the usage guide on [data utilities](/usage/training#data) for more details
and examples.
## Config and implementation {#config}
## Config and implementation {id="config"}
`spacy.Corpus.v1` is a registered function that creates a `Corpus` of training
or evaluation data. It takes the same arguments as the `Corpus` class and
@ -49,7 +49,7 @@ streaming.
%%GITHUB_SPACY/spacy/training/corpus.py
```
## Corpus.\_\_init\_\_ {#init tag="method"}
## Corpus.\_\_init\_\_ {id="init",tag="method"}
Create a `Corpus` for iterating [Example](/api/example) objects from a file or
directory of [`.spacy` data files](/api/data-formats#binary-training). The
@ -81,7 +81,7 @@ train/test skew.
| `augmenter` | Optional data augmentation callback. ~~Callable[[Language, Example], Iterable[Example]]~~ |
| `shuffle` | Whether to shuffle the examples. Defaults to `False`. ~~bool~~ |
## Corpus.\_\_call\_\_ {#call tag="method"}
## Corpus.\_\_call\_\_ {id="call",tag="method"}
Yield examples from the data.
@ -101,7 +101,7 @@ Yield examples from the data.
| `nlp` | The current `nlp` object. ~~Language~~ |
| **YIELDS** | The examples. ~~Example~~ |
## JsonlCorpus {#jsonlcorpus tag="class"}
## JsonlCorpus {id="jsonlcorpus",tag="class"}
Iterate Doc objects from a file or directory of JSONL (newline-delimited JSON)
formatted raw text files. Can be used to read the raw text corpus for language
@ -120,14 +120,13 @@ file.
> srsly.write_jsonl("/path/to/text.jsonl", data)
> ```
```json
### Example
```json {title="Example"}
{"text": "Can I ask where you work now and what you do, and if you enjoy it?"}
{"text": "They may just pull out of the Seattle market completely, at least until they have autonomous vehicles."}
{"text": "My cynical view on this is that it will never be free to the public. Reason: what would be the draw of joining the military? Right now their selling point is free Healthcare and Education. Ironically both are run horribly and most, that I've talked to, come out wishing they never went in."}
```
### JsonlCorpus.\_\init\_\_ {#jsonlcorpus tag="method"}
### JsonlCorpus.\_\_init\_\_ {id="jsonlcorpus",tag="method"}
Initialize the reader.
@ -157,7 +156,7 @@ Initialize the reader.
| `max_length` | Maximum document length (in tokens). Longer documents will be skipped. Defaults to `0`, which indicates no limit. ~~int~~ |
| `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ |
### JsonlCorpus.\_\_call\_\_ {#jsonlcorpus-call tag="method"}
### JsonlCorpus.\_\_call\_\_ {id="jsonlcorpus-call",tag="method"}
Yield examples from the data.

32
website/docs/api/cython-classes.md → website/docs/api/cython-classes.mdx
View File

@ -9,7 +9,7 @@ menu:
- ['StringStore', 'stringstore']
---
## Doc {#doc tag="cdef class" source="spacy/tokens/doc.pxd"}
## Doc {id="doc",tag="cdef class",source="spacy/tokens/doc.pxd"}
The `Doc` object holds an array of [`TokenC`](/api/cython-structs#tokenc)
structs.
@ -21,7 +21,7 @@ accessed from Python. For the Python documentation, see [`Doc`](/api/doc).
</Infobox>
### Attributes {#doc_attributes}
### Attributes {id="doc_attributes"}
| Name | Description |
| ------------ | -------------------------------------------------------------------------------------------------------- |
@ -31,7 +31,7 @@ accessed from Python. For the Python documentation, see [`Doc`](/api/doc).
| `length` | The number of tokens in the document. ~~int~~ |
| `max_length` | The underlying size of the `Doc.c` array. ~~int~~ |
### Doc.push_back {#doc_push_back tag="method"}
### Doc.push_back {id="doc_push_back",tag="method"}
Append a token to the `Doc`. The token can be provided as a
[`LexemeC`](/api/cython-structs#lexemec) or
@ -55,7 +55,7 @@ Append a token to the `Doc`. The token can be provided as a
| `lex_or_tok` | The word to append to the `Doc`. ~~LexemeOrToken~~ |
| `has_space` | Whether the word has trailing whitespace. ~~bint~~ |
## Token {#token tag="cdef class" source="spacy/tokens/token.pxd"}
## Token {id="token",tag="cdef class",source="spacy/tokens/token.pxd"}
A Cython class providing access and methods for a
[`TokenC`](/api/cython-structs#tokenc) struct. Note that the `Token` object does
@ -68,7 +68,7 @@ accessed from Python. For the Python documentation, see [`Token`](/api/token).
</Infobox>
### Attributes {#token_attributes}
### Attributes {id="token_attributes"}
| Name | Description |
| ------- | -------------------------------------------------------------------------- |
@ -77,7 +77,7 @@ accessed from Python. For the Python documentation, see [`Token`](/api/token).
| `i` | The offset of the token within the document. ~~int~~ |
| `doc` | The parent document. ~~Doc~~ |
### Token.cinit {#token_cinit tag="method"}
### Token.cinit {id="token_cinit",tag="method"}
Create a `Token` object from a `TokenC*` pointer.
@ -94,7 +94,7 @@ Create a `Token` object from a `TokenC*` pointer.
| `offset` | The offset of the token within the document. ~~int~~ |
| `doc` | The parent document. ~~int~~ |
## Span {#span tag="cdef class" source="spacy/tokens/span.pxd"}
## Span {id="span",tag="cdef class",source="spacy/tokens/span.pxd"}
A Cython class providing access and methods for a slice of a `Doc` object.
@ -105,7 +105,7 @@ accessed from Python. For the Python documentation, see [`Span`](/api/span).
</Infobox>
### Attributes {#span_attributes}
### Attributes {id="span_attributes"}
| Name | Description |
| ------------ | ----------------------------------------------------------------------------- |
@ -116,7 +116,7 @@ accessed from Python. For the Python documentation, see [`Span`](/api/span).
| `end_char` | The index of the last character of the span. ~~int~~ |
| `label` | A label to attach to the span, e.g. for named entities. ~~attr_t (uint64_t)~~ |
## Lexeme {#lexeme tag="cdef class" source="spacy/lexeme.pxd"}
## Lexeme {id="lexeme",tag="cdef class",source="spacy/lexeme.pxd"}
A Cython class providing access and methods for an entry in the vocabulary.
@ -127,7 +127,7 @@ accessed from Python. For the Python documentation, see [`Lexeme`](/api/lexeme).
</Infobox>
### Attributes {#lexeme_attributes}
### Attributes {id="lexeme_attributes"}
| Name | Description |
| ------- | ----------------------------------------------------------------------------- |
@ -135,7 +135,7 @@ accessed from Python. For the Python documentation, see [`Lexeme`](/api/lexeme).
| `vocab` | A reference to the shared `Vocab` object. ~~Vocab~~ |
| `orth` | ID of the verbatim text content. ~~attr_t (uint64_t)~~ |
## Vocab {#vocab tag="cdef class" source="spacy/vocab.pxd"}
## Vocab {id="vocab",tag="cdef class",source="spacy/vocab.pxd"}
A Cython class providing access and methods for a vocabulary and other data
shared across a language.
@ -147,7 +147,7 @@ accessed from Python. For the Python documentation, see [`Vocab`](/api/vocab).
</Infobox>
### Attributes {#vocab_attributes}
### Attributes {id="vocab_attributes"}
| Name | Description |
| --------- | ---------------------------------------------------------------------------------------------------------- |
@ -155,7 +155,7 @@ accessed from Python. For the Python documentation, see [`Vocab`](/api/vocab).
| `strings` | A `StringStore` that maps string to hash values and vice versa. ~~StringStore~~ |
| `length` | The number of entries in the vocabulary. ~~int~~ |
### Vocab.get {#vocab_get tag="method"}
### Vocab.get {id="vocab_get",tag="method"}
Retrieve a [`LexemeC*`](/api/cython-structs#lexemec) pointer from the
vocabulary.
@ -172,7 +172,7 @@ vocabulary.
| `string` | The string of the word to look up. ~~str~~ |
| **RETURNS** | The lexeme in the vocabulary. ~~const LexemeC\*~~ |
### Vocab.get_by_orth {#vocab_get_by_orth tag="method"}
### Vocab.get_by_orth {id="vocab_get_by_orth",tag="method"}
Retrieve a [`LexemeC*`](/api/cython-structs#lexemec) pointer from the
vocabulary.
@ -189,7 +189,7 @@ vocabulary.
| `orth` | ID of the verbatim text content. ~~attr_t (uint64_t)~~ |
| **RETURNS** | The lexeme in the vocabulary. ~~const LexemeC\*~~ |
## StringStore {#stringstore tag="cdef class" source="spacy/strings.pxd"}
## StringStore {id="stringstore",tag="cdef class",source="spacy/strings.pxd"}
A lookup table to retrieve strings by 64-bit hashes.
@ -201,7 +201,7 @@ accessed from Python. For the Python documentation, see
</Infobox>
### Attributes {#stringstore_attributes}
### Attributes {id="stringstore_attributes"}
| Name | Description |
| ------ | ---------------------------------------------------------------------------------------------------------------- |

22
website/docs/api/cython-structs.md → website/docs/api/cython-structs.mdx
View File

@ -7,7 +7,7 @@ menu:
- ['LexemeC', 'lexemec']
---
## TokenC {#tokenc tag="C struct" source="spacy/structs.pxd"}
## TokenC {id="tokenc",tag="C struct",source="spacy/structs.pxd"}
Cython data container for the `Token` object.
@ -39,7 +39,7 @@ Cython data container for the `Token` object.
| `ent_type` | Named entity type. ~~attr_t (uint64_t)~~ |
| `ent_id` | ID of the entity the token is an instance of, if any. Currently not used, but potentially for coreference resolution. ~~attr_t (uint64_t)~~ |
### Token.get_struct_attr {#token_get_struct_attr tag="staticmethod, nogil" source="spacy/tokens/token.pxd"}
### Token.get_struct_attr {id="token_get_struct_attr",tag="staticmethod, nogil",source="spacy/tokens/token.pxd"}
Get the value of an attribute from the `TokenC` struct by attribute ID.
@ -58,7 +58,7 @@ Get the value of an attribute from the `TokenC` struct by attribute ID.
| `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ |
| **RETURNS** | The value of the attribute. ~~attr_t (uint64_t)~~ |
### Token.set_struct_attr {#token_set_struct_attr tag="staticmethod, nogil" source="spacy/tokens/token.pxd"}
### Token.set_struct_attr {id="token_set_struct_attr",tag="staticmethod, nogil",source="spacy/tokens/token.pxd"}
Set the value of an attribute of the `TokenC` struct by attribute ID.
@ -78,7 +78,7 @@ Set the value of an attribute of the `TokenC` struct by attribute ID.
| `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ |
| `value` | The value to set. ~~attr_t (uint64_t)~~ |
### token_by_start {#token_by_start tag="function" source="spacy/tokens/doc.pxd"}
### token_by_start {id="token_by_start",tag="function",source="spacy/tokens/doc.pxd"}
Find a token in a `TokenC*` array by the offset of its first character.
@ -100,7 +100,7 @@ Find a token in a `TokenC*` array by the offset of its first character.
| `start_char` | The start index to search for. ~~int~~ |
| **RETURNS** | The index of the token in the array or `-1` if not found. ~~int~~ |
### token_by_end {#token_by_end tag="function" source="spacy/tokens/doc.pxd"}
### token_by_end {id="token_by_end",tag="function",source="spacy/tokens/doc.pxd"}
Find a token in a `TokenC*` array by the offset of its final character.
@ -122,7 +122,7 @@ Find a token in a `TokenC*` array by the offset of its final character.
| `end_char` | The end index to search for. ~~int~~ |
| **RETURNS** | The index of the token in the array or `-1` if not found. ~~int~~ |
### set_children_from_heads {#set_children_from_heads tag="function" source="spacy/tokens/doc.pxd"}
### set_children_from_heads {id="set_children_from_heads",tag="function",source="spacy/tokens/doc.pxd"}
Set attributes that allow lookup of syntactic children on a `TokenC*` array.
This function must be called after making changes to the `TokenC.head`
@ -148,7 +148,7 @@ attribute, in order to make the parse tree navigation consistent.
| `tokens` | A `TokenC*` array. ~~const TokenC\*~~ |
| `length` | The number of tokens in the array. ~~int~~ |
## LexemeC {#lexemec tag="C struct" source="spacy/structs.pxd"}
## LexemeC {id="lexemec",tag="C struct",source="spacy/structs.pxd"}
Struct holding information about a lexical type. `LexemeC` structs are usually
owned by the `Vocab`, and accessed through a read-only pointer on the `TokenC`
@ -172,7 +172,7 @@ struct.
| `prefix` | Length-N substring from the start of the lexeme. Defaults to `N=1`. ~~attr_t (uint64_t)~~ |
| `suffix` | Length-N substring from the end of the lexeme. Defaults to `N=3`. ~~attr_t (uint64_t)~~ |
### Lexeme.get_struct_attr {#lexeme_get_struct_attr tag="staticmethod, nogil" source="spacy/lexeme.pxd"}
### Lexeme.get_struct_attr {id="lexeme_get_struct_attr",tag="staticmethod, nogil",source="spacy/lexeme.pxd"}
Get the value of an attribute from the `LexemeC` struct by attribute ID.
@ -192,7 +192,7 @@ Get the value of an attribute from the `LexemeC` struct by attribute ID.
| `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ |
| **RETURNS** | The value of the attribute. ~~attr_t (uint64_t)~~ |
### Lexeme.set_struct_attr {#lexeme_set_struct_attr tag="staticmethod, nogil" source="spacy/lexeme.pxd"}
### Lexeme.set_struct_attr {id="lexeme_set_struct_attr",tag="staticmethod, nogil",source="spacy/lexeme.pxd"}
Set the value of an attribute of the `LexemeC` struct by attribute ID.
@ -212,7 +212,7 @@ Set the value of an attribute of the `LexemeC` struct by attribute ID.
| `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ |
| `value` | The value to set. ~~attr_t (uint64_t)~~ |
### Lexeme.c_check_flag {#lexeme_c_check_flag tag="staticmethod, nogil" source="spacy/lexeme.pxd"}
### Lexeme.c_check_flag {id="lexeme_c_check_flag",tag="staticmethod, nogil",source="spacy/lexeme.pxd"}
Check the value of a binary flag attribute.
@ -232,7 +232,7 @@ Check the value of a binary flag attribute.
| `flag_id` | The ID of the flag to look up. The flag IDs are enumerated in `spacy.typedefs`. ~~attr_id_t~~ |
| **RETURNS** | The boolean value of the flag. ~~bint~~ |
### Lexeme.c_set_flag {#lexeme_c_set_flag tag="staticmethod, nogil" source="spacy/lexeme.pxd"}
### Lexeme.c_set_flag {id="lexeme_c_set_flag",tag="staticmethod, nogil",source="spacy/lexeme.pxd"}
Set the value of a binary flag attribute.

4
website/docs/api/cython.md → website/docs/api/cython.mdx
View File

@ -6,7 +6,7 @@ menu:
- ['Conventions', 'conventions']
---
## Overview {#overview hidden="true"}
## Overview {id="overview",hidden="true"}
> #### What's Cython?
>
@ -37,7 +37,7 @@ class holds a [`LexemeC`](/api/cython-structs#lexemec) struct, at `Lexeme.c`.
This lets you shed the Python container, and pass a pointer to the underlying
data into C-level functions.
## Conventions {#conventions}
## Conventions {id="conventions"}
spaCy's core data structures are implemented as [Cython](http://cython.org/)
`cdef` classes. Memory is managed through the

44
website/docs/api/data-formats.md → website/docs/api/data-formats.mdx
View File

@ -14,7 +14,7 @@ vocabulary data. For an overview of label schemes used by the models, see the
[models directory](/models). Each trained pipeline documents the label schemes
used in its components, depending on the data it was trained on.
## Training config {#config new="3"}
## Training config {id="config",version="3"}
Config files define the training process and pipeline and can be passed to
[`spacy train`](/api/cli#train). They use
@ -52,7 +52,7 @@ your config and check that it's valid, you can run the
</Infobox>
### nlp {#config-nlp tag="section"}
### nlp {id="config-nlp",tag="section"}
> #### Example
>
@ -83,7 +83,7 @@ Defines the `nlp` object, its tokenizer and
| `tokenizer` | The tokenizer to use. Defaults to [`Tokenizer`](/api/tokenizer). ~~Callable[[str], Doc]~~ |
| `batch_size` | Default batch size for [`Language.pipe`](/api/language#pipe) and [`Language.evaluate`](/api/language#evaluate). ~~int~~ |
### components {#config-components tag="section"}
### components {id="config-components",tag="section"}
> #### Example
>
@ -106,7 +106,7 @@ function to use to create component) or a `source` (name of path of trained
pipeline to copy components from). See the docs on
[defining pipeline components](/usage/training#config-components) for details.
### paths, system {#config-variables tag="variables"}
### paths, system {id="config-variables",tag="variables"}
These sections define variables that can be referenced across the other sections
as variables. For example `${paths.train}` uses the value of `train` defined in
@ -116,11 +116,11 @@ need paths, you can define them here. All config values can also be
[`spacy train`](/api/cli#train), which is especially relevant for data paths
that you don't want to hard-code in your config file.
```cli
```bash
$ python -m spacy train config.cfg --paths.train ./corpus/train.spacy
```
### corpora {#config-corpora tag="section"}
### corpora {id="config-corpora",tag="section"}
> #### Example
>
@ -176,7 +176,7 @@ single corpus once and then divide it up into `train` and `dev` partitions.
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `corpora` | A dictionary keyed by string names, mapped to corpus functions that receive the current `nlp` object and return an iterator of [`Example`](/api/example) objects. ~~Dict[str, Callable[[Language], Iterator[Example]]]~~ |
### training {#config-training tag="section"}
### training {id="config-training",tag="section"}
This section defines settings and controls for the training and evaluation
process that are used when you run [`spacy train`](/api/cli#train).
@ -202,7 +202,7 @@ process that are used when you run [`spacy train`](/api/cli#train).
| `seed` | The random seed. Defaults to variable `${system.seed}`. ~~int~~ |
| `train_corpus` | Dot notation of the config location defining the train corpus. Defaults to `corpora.train`. ~~str~~ |
### pretraining {#config-pretraining tag="section,optional"}
### pretraining {id="config-pretraining",tag="section,optional"}
This section is optional and defines settings and controls for
[language model pretraining](/usage/embeddings-transformers#pretraining). It's
@ -220,7 +220,7 @@ used when you run [`spacy pretrain`](/api/cli#pretrain).
| `component` | Component name to identify the layer with the model to pretrain. Defaults to `"tok2vec"`. ~~str~~ |
| `layer` | The specific layer of the model to pretrain. If empty, the whole model will be used. ~~str~~ |
### initialize {#config-initialize tag="section"}
### initialize {id="config-initialize",tag="section"}
This config block lets you define resources for **initializing the pipeline**.
It's used by [`Language.initialize`](/api/language#initialize) and typically
@ -255,9 +255,9 @@ Also see the usage guides on the
| `vectors` | Name or path of pipeline containing pretrained word vectors to use, e.g. created with [`init vectors`](/api/cli#init-vectors). Defaults to `null`. ~~Optional[str]~~ |
| `vocab_data` | Path to JSONL-formatted [vocabulary file](/api/data-formats#vocab-jsonl) to initialize vocabulary. ~~Optional[str]~~ |
## Training data {#training}
## Training data {id="training"}
### Binary training format {#binary-training new="3"}
### Binary training format {id="binary-training",version="3"}
> #### Example
>
@ -288,7 +288,7 @@ Note that while this is the format used to save training data, you do not have
to understand the internal details to use it or create training data. See the
section on [preparing training data](/usage/training#training-data).
### JSON training format {#json-input tag="deprecated"}
### JSON training format {id="json-input",tag="deprecated"}
<Infobox variant="warning" title="Changed in v3.0">
@ -300,7 +300,7 @@ objects to JSON, you can now serialize them directly using the
[`spacy convert`](/api/cli) lets you convert your JSON data to the new `.spacy`
format:
```cli
```bash
$ python -m spacy convert ./data.json .
```
@ -317,8 +317,7 @@ $ python -m spacy convert ./data.json .
> [`offsets_to_biluo_tags`](/api/top-level#offsets_to_biluo_tags) function can
> help you convert entity offsets to the right format.
```python
### Example structure
```python {title="Example structure"}
[{
"id": int, # ID of the document within the corpus
"paragraphs": [{ # list of paragraphs in the corpus
@ -357,7 +356,7 @@ https://github.com/explosion/spaCy/blob/v2.3.x/examples/training/training-data.j
</Accordion>
### Annotation format for creating training examples {#dict-input}
### Annotation format for creating training examples {id="dict-input"}
An [`Example`](/api/example) object holds the information for one training
instance. It stores two [`Doc`](/api/doc) objects: one for holding the
@ -436,8 +435,7 @@ file to keep track of your settings and hyperparameters and your own
</Infobox>
```python
### Examples
```python {title="Examples"}
# Training data for a part-of-speech tagger
doc = Doc(vocab, words=["I", "like", "stuff"])
gold_dict = {"tags": ["NOUN", "VERB", "NOUN"]}
@ -466,7 +464,7 @@ gold_dict = {"entities": [(0, 12, "PERSON")],
example = Example.from_dict(doc, gold_dict)
```
## Lexical data for vocabulary {#vocab-jsonl new="2"}
## Lexical data for vocabulary {id="vocab-jsonl",version="2"}
This data file can be provided via the `vocab_data` setting in the
`[initialize]` block of the training config to pre-define the lexical data to
@ -483,13 +481,11 @@ spaCy's [`Lexeme`](/api/lexeme#attributes) object.
> vocab_data = "/path/to/vocab-data.jsonl"
> ```
```python
### First line
```python {title="First line"}
{"lang": "en", "settings": {"oov_prob": -20.502029418945312}}
```
```python
### Entry structure
```python {title="Entry structure"}
{
"orth": string, # the word text
"id": int, # can correspond to row in vectors table
@ -526,7 +522,7 @@ Here's an example of the 20 most frequent lexemes in the English training data:
%%GITHUB_SPACY/extra/example_data/vocab-data.jsonl
```
## Pipeline meta {#meta}
## Pipeline meta {id="meta"}
The pipeline meta is available as the file `meta.json` and exported
automatically when you save an `nlp` object to disk. Its contents are available

21
website/docs/api/dependencymatcher.md → website/docs/api/dependencymatcher.mdx
View File

@ -2,7 +2,7 @@
title: DependencyMatcher
teaser: Match subtrees within a dependency parse
tag: class
new: 3
version: 3
source: spacy/matcher/dependencymatcher.pyx
---
@ -14,7 +14,7 @@ It requires a pretrained [`DependencyParser`](/api/parser) or other component
that sets the `Token.dep` and `Token.head` attributes. See the
[usage guide](/usage/rule-based-matching#dependencymatcher) for examples.
## Pattern format {#patterns}
## Pattern format {id="patterns"}
> ```python
> ### Example
@ -62,7 +62,7 @@ of relations, see the usage guide on
</Infobox>
### Operators {#operators}
### Operators {id="operators"}
The following operators are supported by the `DependencyMatcher`, most of which
come directly from
@ -87,8 +87,7 @@ come directly from
| `A <++ B` | `B` is a right parent of `A`, i.e. `A` is a child of `B` and `A.i < B.i` _(not in Semgrex)_. |
| `A <-- B` | `B` is a left parent of `A`, i.e. `A` is a child of `B` and `A.i > B.i` _(not in Semgrex)_. |
## DependencyMatcher.\_\_init\_\_ {#init tag="method"}
## DependencyMatcher.\_\_init\_\_ {id="init",tag="method"}
Create a `DependencyMatcher`.
@ -105,7 +104,7 @@ Create a `DependencyMatcher`.
| _keyword-only_ | |
| `validate` | Validate all patterns added to this matcher. ~~bool~~ |
## DependencyMatcher.\_\call\_\_ {#call tag="method"}
## DependencyMatcher.\_\_call\_\_ {id="call",tag="method"}
Find all tokens matching the supplied patterns on the `Doc` or `Span`.
@ -127,7 +126,7 @@ Find all tokens matching the supplied patterns on the `Doc` or `Span`.
| `doclike` | The `Doc` or `Span` to match over. ~~Union[Doc, Span]~~ |
| **RETURNS** | A list of `(match_id, token_ids)` tuples, describing the matches. The `match_id` is the ID of the match pattern and `token_ids` is a list of token indices matched by the pattern, where the position of each token in the list corresponds to the position of the node specification in the pattern. ~~List[Tuple[int, List[int]]]~~ |
## DependencyMatcher.\_\_len\_\_ {#len tag="method"}
## DependencyMatcher.\_\_len\_\_ {id="len",tag="method"}
Get the number of rules added to the dependency matcher. Note that this only
returns the number of rules (identical with the number of IDs), not the number
@ -148,7 +147,7 @@ of individual patterns.
| ----------- | ---------------------------- |
| **RETURNS** | The number of rules. ~~int~~ |
## DependencyMatcher.\_\_contains\_\_ {#contains tag="method"}
## DependencyMatcher.\_\_contains\_\_ {id="contains",tag="method"}
Check whether the matcher contains rules for a match ID.
@ -166,7 +165,7 @@ Check whether the matcher contains rules for a match ID.
| `key` | The match ID. ~~str~~ |
| **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ |
## DependencyMatcher.add {#add tag="method"}
## DependencyMatcher.add {id="add",tag="method"}
Add a rule to the matcher, consisting of an ID key, one or more patterns, and an
optional callback function to act on the matches. The callback function will
@ -191,7 +190,7 @@ will be overwritten.
| _keyword-only_ | |
| `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[DependencyMatcher, Doc, int, List[Tuple], Any]]~~ |
## DependencyMatcher.get {#get tag="method"}
## DependencyMatcher.get {id="get",tag="method"}
Retrieve the pattern stored for a key. Returns the rule as an
`(on_match, patterns)` tuple containing the callback and available patterns.
@ -208,7 +207,7 @@ Retrieve the pattern stored for a key. Returns the rule as an
| `key` | The ID of the match rule. ~~str~~ |
| **RETURNS** | The rule, as an `(on_match, patterns)` tuple. ~~Tuple[Optional[Callable], List[List[Union[Dict, Tuple]]]]~~ |
## DependencyMatcher.remove {#remove tag="method"}
## DependencyMatcher.remove {id="remove",tag="method"}
Remove a rule from the dependency matcher. A `KeyError` is raised if the match
ID does not exist.

42
website/docs/api/dependencyparser.md → website/docs/api/dependencyparser.mdx
View File

@ -25,7 +25,7 @@ current state. The weights are updated such that the scores assigned to the set
of optimal actions is increased, while scores assigned to other actions are
decreased. Note that more than one action may be optimal for a given state.
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Dependency predictions are assigned to the `Token.dep` and `Token.head` fields.
Beside the dependencies themselves, the parser decides sentence boundaries,
@ -39,7 +39,7 @@ which are saved in `Token.is_sent_start` and accessible via `Doc.sents`.
| `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. After the parser runs this will be `True` or `False` for all tokens. ~~bool~~ |
| `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -74,7 +74,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/dep_parser.pyx
```
## DependencyParser.\_\_init\_\_ {#init tag="method"}
## DependencyParser.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -107,7 +107,7 @@ shortcut for this and instantiate the component using its string name and
| `min_action_freq` | The minimum frequency of labelled actions to retain. Rarer labelled actions have their label backed-off to "dep". While this primarily affects the label accuracy, it can also affect the attachment structure, as the labels are used to represent the pseudo-projectivity transformation. ~~int~~ |
| `scorer` | The scoring method. Defaults to [`Scorer.score_deps`](/api/scorer#score_deps) for the attribute `"dep"` ignoring the labels `p` and `punct` and [`Scorer.score_spans`](/api/scorer/#score_spans) for the attribute `"sents"`. ~~Optional[Callable]~~ |
## DependencyParser.\_\_call\_\_ {#call tag="method"}
## DependencyParser.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -131,7 +131,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## DependencyParser.pipe {#pipe tag="method"}
## DependencyParser.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -155,7 +155,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/dependencyparser#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## DependencyParser.initialize {#initialize tag="method" new="3"}
## DependencyParser.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -198,7 +198,7 @@ This method was previously called `begin_training`.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Dict[str, Dict[str, int]]]~~ |
## DependencyParser.predict {#predict tag="method"}
## DependencyParser.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them.
@ -215,7 +215,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | A helper class for the parse state (internal). ~~StateClass~~ |
## DependencyParser.set_annotations {#set_annotations tag="method"}
## DependencyParser.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -232,7 +232,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `DependencyParser.predict`. Returns an internal helper class for the parse state. ~~List[StateClass]~~ |
## DependencyParser.update {#update tag="method"}
## DependencyParser.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects, updating the pipe's
model. Delegates to [`predict`](/api/dependencyparser#predict) and
@ -255,7 +255,7 @@ model. Delegates to [`predict`](/api/dependencyparser#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## DependencyParser.get_loss {#get_loss tag="method"}
## DependencyParser.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their
predicted scores.
@ -274,7 +274,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. ~~StateClass~~ |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## DependencyParser.create_optimizer {#create_optimizer tag="method"}
## DependencyParser.create_optimizer {id="create_optimizer",tag="method"}
Create an [`Optimizer`](https://thinc.ai/docs/api-optimizers) for the pipeline
component.
@ -290,7 +290,7 @@ component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## DependencyParser.use_params {#use_params tag="method, contextmanager"}
## DependencyParser.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -307,7 +307,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## DependencyParser.add_label {#add_label tag="method"}
## DependencyParser.add_label {id="add_label",tag="method"}
Add a new label to the pipe. Note that you don't have to call this method if you
provide a **representative data sample** to the [`initialize`](#initialize)
@ -327,7 +327,7 @@ to the model, and the output dimension will be
| `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## DependencyParser.set_output {#set_output tag="method"}
## DependencyParser.set_output {id="set_output",tag="method"}
Change the output dimension of the component's model by calling the model's
attribute `resize_output`. This is a function that takes the original model and
@ -346,7 +346,7 @@ forgetting" problem.
| ---- | --------------------------------- |
| `nO` | The new output dimension. ~~int~~ |
## DependencyParser.to_disk {#to_disk tag="method"}
## DependencyParser.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -363,7 +363,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## DependencyParser.from_disk {#from_disk tag="method"}
## DependencyParser.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -381,7 +381,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `DependencyParser` object. ~~DependencyParser~~ |
## DependencyParser.to_bytes {#to_bytes tag="method"}
## DependencyParser.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -398,7 +398,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `DependencyParser` object. ~~bytes~~ |
## DependencyParser.from_bytes {#from_bytes tag="method"}
## DependencyParser.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -417,7 +417,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `DependencyParser` object. ~~DependencyParser~~ |
## DependencyParser.labels {#labels tag="property"}
## DependencyParser.labels {id="labels",tag="property"}
The labels currently added to the component.
@ -432,7 +432,7 @@ The labels currently added to the component.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## DependencyParser.label_data {#label_data tag="property" new="3"}
## DependencyParser.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -450,7 +450,7 @@ the model with a pre-defined label set.
| ----------- | ------------------------------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Dict[str, Dict[str, Dict[str, int]]]~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

72
website/docs/api/doc.md → website/docs/api/doc.mdx
View File

@ -12,7 +12,7 @@ compressed binary strings. The `Doc` object holds an array of
[`Span`](/api/span) objects are views of this array, i.e. they don't own the
data themselves.
## Doc.\_\_init\_\_ {#init tag="method"}
## Doc.\_\_init\_\_ {id="init",tag="method"}
Construct a `Doc` object. The most common way to get a `Doc` object is via the
`nlp` object.
@ -47,7 +47,7 @@ Construct a `Doc` object. The most common way to get a `Doc` object is via the
| `sent_starts` <Tag variant="new">3</Tag> | A list of values, of the same length as `words`, to assign as `token.is_sent_start`. Will be overridden by heads if `heads` is provided. Defaults to `None`. ~~Optional[List[Union[bool, int, None]]]~~ |
| `ents` <Tag variant="new">3</Tag> | A list of strings, of the same length of `words`, to assign the token-based IOB tag. Defaults to `None`. ~~Optional[List[str]]~~ |
## Doc.\_\_getitem\_\_ {#getitem tag="method"}
## Doc.\_\_getitem\_\_ {id="getitem",tag="method"}
Get a [`Token`](/api/token) object at position `i`, where `i` is an integer.
Negative indexing is supported, and follows the usual Python semantics, i.e.
@ -80,7 +80,7 @@ semantics.
| `start_end` | The slice of the document to get. ~~Tuple[int, int]~~ |
| **RETURNS** | The span at `doc[start:end]`. ~~Span~~ |
## Doc.\_\_iter\_\_ {#iter tag="method"}
## Doc.\_\_iter\_\_ {id="iter",tag="method"}
Iterate over `Token` objects, from which the annotations can be easily accessed.
@ -100,7 +100,7 @@ underlying C data directly from Cython.
| ---------- | --------------------------- |
| **YIELDS** | A `Token` object. ~~Token~~ |
## Doc.\_\_len\_\_ {#len tag="method"}
## Doc.\_\_len\_\_ {id="len",tag="method"}
Get the number of tokens in the document.
@ -115,7 +115,7 @@ Get the number of tokens in the document.
| ----------- | --------------------------------------------- |
| **RETURNS** | The number of tokens in the document. ~~int~~ |
## Doc.set_extension {#set_extension tag="classmethod" new="2"}
## Doc.set_extension {id="set_extension",tag="classmethod",version="2"}
Define a custom attribute on the `Doc` which becomes available via `Doc._`. For
details, see the documentation on
@ -140,7 +140,7 @@ details, see the documentation on
| `setter` | Setter function that takes the `Doc` and a value, and modifies the object. Is called when the user writes to the `Doc._` attribute. ~~Optional[Callable[[Doc, Any], None]]~~ |
| `force` | Force overwriting existing attribute. ~~bool~~ |
## Doc.get_extension {#get_extension tag="classmethod" new="2"}
## Doc.get_extension {id="get_extension",tag="classmethod",version="2"}
Look up a previously registered extension by name. Returns a 4-tuple
`(default, method, getter, setter)` if the extension is registered. Raises a
@ -160,7 +160,7 @@ Look up a previously registered extension by name. Returns a 4-tuple
| `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Doc.has_extension {#has_extension tag="classmethod" new="2"}
## Doc.has_extension {id="has_extension",tag="classmethod",version="2"}
Check whether an extension has been registered on the `Doc` class.
@ -177,7 +177,7 @@ Check whether an extension has been registered on the `Doc` class.
| `name` | Name of the extension to check. ~~str~~ |
| **RETURNS** | Whether the extension has been registered. ~~bool~~ |
## Doc.remove_extension {#remove_extension tag="classmethod" new="2.0.12"}
## Doc.remove_extension {id="remove_extension",tag="classmethod",version="2.0.12"}
Remove a previously registered extension.
@ -195,7 +195,7 @@ Remove a previously registered extension.
| `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Doc.char_span {#char_span tag="method" new="2"}
## Doc.char_span {id="char_span",tag="method",version="2"}
Create a `Span` object from the slice `doc.text[start_idx:end_idx]`. Returns
`None` if the character indices don't map to a valid span using the default
@ -219,7 +219,7 @@ alignment mode `"strict".
| `alignment_mode` | How character indices snap to token boundaries. Options: `"strict"` (no snapping), `"contract"` (span of all tokens completely within the character span), `"expand"` (span of all tokens at least partially covered by the character span). Defaults to `"strict"`. ~~str~~ |
| **RETURNS** | The newly constructed object or `None`. ~~Optional[Span]~~ |
## Doc.set_ents {#set_ents tag="method" new="3"}
## Doc.set_ents {id="set_ents",tag="method",version="3"}
Set the named entities in the document.
@ -243,7 +243,7 @@ Set the named entities in the document.
| `outside` | Spans outside of entities (O in IOB). ~~Optional[List[Span]]~~ |
| `default` | How to set entity annotation for tokens outside of any provided spans. Options: `"blocked"`, `"missing"`, `"outside"` and `"unmodified"` (preserve current state). Defaults to `"outside"`. ~~str~~ |
## Doc.similarity {#similarity tag="method" model="vectors"}
## Doc.similarity {id="similarity",tag="method",model="vectors"}
Make a semantic similarity estimate. The default estimate is cosine similarity
using an average of word vectors.
@ -263,7 +263,7 @@ using an average of word vectors.
| `other` | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ |
| **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ |
## Doc.count_by {#count_by tag="method"}
## Doc.count_by {id="count_by",tag="method"}
Count the frequencies of a given attribute. Produces a dict of
`{attr (int): count (ints)}` frequencies, keyed by the values of the given
@ -284,7 +284,7 @@ attribute ID.
| `attr_id` | The attribute ID. ~~int~~ |
| **RETURNS** | A dictionary mapping attributes to integer counts. ~~Dict[int, int]~~ |
## Doc.get_lca_matrix {#get_lca_matrix tag="method"}
## Doc.get_lca_matrix {id="get_lca_matrix",tag="method"}
Calculates the lowest common ancestor matrix for a given `Doc`. Returns LCA
matrix containing the integer index of the ancestor, or `-1` if no common
@ -302,7 +302,7 @@ ancestor is found, e.g. if span excludes a necessary ancestor.
| ----------- | -------------------------------------------------------------------------------------- |
| **RETURNS** | The lowest common ancestor matrix of the `Doc`. ~~numpy.ndarray[ndim=2, dtype=int32]~~ |
## Doc.has_annotation {#has_annotation tag="method"}
## Doc.has_annotation {id="has_annotation",tag="method"}
Check whether the doc contains annotation on a
[`Token` attribute](/api/token#attributes).
@ -327,7 +327,7 @@ doc = nlp("This is a text")
| `require_complete` | Whether to check that the attribute is set on every token in the doc. Defaults to `False`. ~~bool~~ |
| **RETURNS** | Whether specified annotation is present in the doc. ~~bool~~ |
## Doc.to_array {#to_array tag="method"}
## Doc.to_array {id="to_array",tag="method"}
Export given token attributes to a numpy `ndarray`. If `attr_ids` is a sequence
of `M` attributes, the output array will be of shape `(N, M)`, where `N` is the
@ -355,7 +355,7 @@ Returns a 2D array with one row per token and one column per attribute (when
| `attr_ids` | A list of attributes (int IDs or string names) or a single attribute (int ID or string name). ~~Union[int, str, List[Union[int, str]]]~~ |
| **RETURNS** | The exported attributes as a numpy array. ~~Union[numpy.ndarray[ndim=2, dtype=uint64], numpy.ndarray[ndim=1, dtype=uint64]]~~ |
## Doc.from_array {#from_array tag="method"}
## Doc.from_array {id="from_array",tag="method"}
Load attributes from a numpy array. Write to a `Doc` object, from an `(M, N)`
array of attributes.
@ -379,7 +379,7 @@ array of attributes.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Doc` itself. ~~Doc~~ |
## Doc.from_docs {#from_docs tag="staticmethod" new="3"}
## Doc.from_docs {id="from_docs",tag="staticmethod",version="3"}
Concatenate multiple `Doc` objects to form a new one. Raises an error if the
`Doc` objects do not all share the same `Vocab`.
@ -408,7 +408,7 @@ Concatenate multiple `Doc` objects to form a new one. Raises an error if the
| `exclude` <Tag variant="new">3.3</Tag> | String names of Doc attributes to exclude. Supported: `spans`, `tensor`, `user_data`. ~~Iterable[str]~~ |
| **RETURNS** | The new `Doc` object that is containing the other docs or `None`, if `docs` is empty or `None`. ~~Optional[Doc]~~ |
## Doc.to_disk {#to_disk tag="method" new="2"}
## Doc.to_disk {id="to_disk",tag="method",version="2"}
Save the current state to a directory.
@ -424,7 +424,7 @@ Save the current state to a directory.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Doc.from_disk {#from_disk tag="method" new="2"}
## Doc.from_disk {id="from_disk",tag="method",version="2"}
Loads state from a directory. Modifies the object in place and returns it.
@ -443,7 +443,7 @@ Loads state from a directory. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Doc` object. ~~Doc~~ |
## Doc.to_bytes {#to_bytes tag="method"}
## Doc.to_bytes {id="to_bytes",tag="method"}
Serialize, i.e. export the document contents to a binary string.
@ -460,7 +460,7 @@ Serialize, i.e. export the document contents to a binary string.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | A losslessly serialized copy of the `Doc`, including all annotations. ~~bytes~~ |
## Doc.from_bytes {#from_bytes tag="method"}
## Doc.from_bytes {id="from_bytes",tag="method"}
Deserialize, i.e. import the document contents from a binary string.
@ -481,7 +481,7 @@ Deserialize, i.e. import the document contents from a binary string.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Doc` object. ~~Doc~~ |
## Doc.to_json {#to_json tag="method"}
## Doc.to_json {id="to_json",tag="method"}
Serializes a document to JSON. Note that this is format differs from the
deprecated [`JSON training format`](/api/data-formats#json-input).
@ -498,7 +498,7 @@ deprecated [`JSON training format`](/api/data-formats#json-input).
| `underscore` | Optional list of string names of custom `Doc` attributes. Attribute values need to be JSON-serializable. Values will be added to an `"_"` key in the data, e.g. `"_": {"foo": "bar"}`. ~~Optional[List[str]]~~ |
| **RETURNS** | The data in JSON format. ~~Dict[str, Any]~~ |
## Doc.from_json {#from_json tag="method" new="3.3.1"}
## Doc.from_json {id="from_json",tag="method",version="3.3.1"}
Deserializes a document from JSON, i.e. generates a document from the provided
JSON data as generated by [`Doc.to_json()`](/api/doc#to_json).
@ -520,7 +520,7 @@ JSON data as generated by [`Doc.to_json()`](/api/doc#to_json).
| `validate` | Whether to validate the JSON input against the expected schema for detailed debugging. Defaults to `False`. ~~bool~~ |
| **RETURNS** | A `Doc` corresponding to the provided JSON. ~~Doc~~ |
## Doc.retokenize {#retokenize tag="contextmanager" new="2.1"}
## Doc.retokenize {id="retokenize",tag="contextmanager",version="2.1"}
Context manager to handle retokenization of the `Doc`. Modifications to the
`Doc`'s tokenization are stored, and then made all at once when the context
@ -540,7 +540,7 @@ invalidated, although they may accidentally continue to work.
| ----------- | -------------------------------- |
| **RETURNS** | The retokenizer. ~~Retokenizer~~ |
### Retokenizer.merge {#retokenizer.merge tag="method"}
### Retokenizer.merge {id="retokenizer.merge",tag="method"}
Mark a span for merging. The `attrs` will be applied to the resulting token (if
they're context-dependent token attributes like `LEMMA` or `DEP`) or to the
@ -563,7 +563,7 @@ values.
| `span` | The span to merge. ~~Span~~ |
| `attrs` | Attributes to set on the merged token. ~~Dict[Union[str, int], Any]~~ |
### Retokenizer.split {#retokenizer.split tag="method"}
### Retokenizer.split {id="retokenizer.split",tag="method"}
Mark a token for splitting, into the specified `orths`. The `heads` are required
to specify how the new subtokens should be integrated into the dependency tree.
@ -599,7 +599,7 @@ underlying lexeme (if they're context-independent lexical attributes like
| `heads` | List of `token` or `(token, subtoken)` tuples specifying the tokens to attach the newly split subtokens to. ~~List[Union[Token, Tuple[Token, int]]]~~ |
| `attrs` | Attributes to set on all split tokens. Attribute names mapped to list of per-token attribute values. ~~Dict[Union[str, int], List[Any]]~~ |
## Doc.ents {#ents tag="property" model="NER"}
## Doc.ents {id="ents",tag="property",model="NER"}
The named entities in the document. Returns a tuple of named entity `Span`
objects, if the entity recognizer has been applied.
@ -617,7 +617,7 @@ objects, if the entity recognizer has been applied.
| ----------- | ---------------------------------------------------------------- |
| **RETURNS** | Entities in the document, one `Span` per entity. ~~Tuple[Span]~~ |
## Doc.spans {#spans tag="property"}
## Doc.spans {id="spans",tag="property"}
A dictionary of named span groups, to store and access additional span
annotations. You can write to it by assigning a list of [`Span`](/api/span)
@ -634,7 +634,7 @@ objects or a [`SpanGroup`](/api/spangroup) to a given key.
| ----------- | ------------------------------------------------------------------ |
| **RETURNS** | The span groups assigned to the document. ~~Dict[str, SpanGroup]~~ |
## Doc.cats {#cats tag="property" model="text classifier"}
## Doc.cats {id="cats",tag="property",model="text classifier"}
Maps a label to a score for categories applied to the document. Typically set by
the [`TextCategorizer`](/api/textcategorizer).
@ -650,7 +650,7 @@ the [`TextCategorizer`](/api/textcategorizer).
| ----------- | ---------------------------------------------------------- |
| **RETURNS** | The text categories mapped to scores. ~~Dict[str, float]~~ |
## Doc.noun_chunks {#noun_chunks tag="property" model="parser"}
## Doc.noun_chunks {id="noun_chunks",tag="property",model="parser"}
Iterate over the base noun phrases in the document. Yields base noun-phrase
`Span` objects, if the document has been syntactically parsed. A base noun
@ -677,7 +677,7 @@ implemented for the given language, a `NotImplementedError` is raised.
| ---------- | ------------------------------------- |
| **YIELDS** | Noun chunks in the document. ~~Span~~ |
## Doc.sents {#sents tag="property" model="sentences"}
## Doc.sents {id="sents",tag="property",model="sentences"}
Iterate over the sentences in the document. Sentence spans have no label.
@ -699,7 +699,7 @@ will raise an error otherwise.
| ---------- | ----------------------------------- |
| **YIELDS** | Sentences in the document. ~~Span~~ |
## Doc.has_vector {#has_vector tag="property" model="vectors"}
## Doc.has_vector {id="has_vector",tag="property",model="vectors"}
A boolean value indicating whether a word vector is associated with the object.
@ -714,7 +714,7 @@ A boolean value indicating whether a word vector is associated with the object.
| ----------- | --------------------------------------------------------- |
| **RETURNS** | Whether the document has a vector data attached. ~~bool~~ |
## Doc.vector {#vector tag="property" model="vectors"}
## Doc.vector {id="vector",tag="property",model="vectors"}
A real-valued meaning representation. Defaults to an average of the token
vectors.
@ -731,7 +731,7 @@ vectors.
| ----------- | -------------------------------------------------------------------------------------------------- |
| **RETURNS** | A 1-dimensional array representing the document's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Doc.vector_norm {#vector_norm tag="property" model="vectors"}
## Doc.vector_norm {id="vector_norm",tag="property",model="vectors"}
The L2 norm of the document's vector representation.
@ -749,7 +749,7 @@ The L2 norm of the document's vector representation.
| ----------- | --------------------------------------------------- |
| **RETURNS** | The L2 norm of the vector representation. ~~float~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
@ -768,7 +768,7 @@ The L2 norm of the document's vector representation.
| `has_unknown_spaces` | Whether the document was constructed without known spacing between tokens (typically when created from gold tokenization). ~~bool~~ |
| `_` | User space for adding custom [attribute extensions](/usage/processing-pipelines#custom-components-attributes). ~~Underscore~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

23
website/docs/api/docbin.md → website/docs/api/docbin.mdx
View File

@ -1,7 +1,7 @@
---
title: DocBin
tag: class
new: 2.2
version: 2.2
teaser: Pack Doc objects for binary serialization
source: spacy/tokens/_serialize.py
---
@ -15,8 +15,7 @@ notable downside to this format is that you can't easily extract just one
document from the `DocBin`. The serialization format is gzipped msgpack, where
the msgpack object has the following structure:
```python
### msgpack object structure
```python {title="msgpack object structure"}
{
"version": str, # DocBin version number
"attrs": List[uint64], # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE]
@ -33,7 +32,7 @@ object. This means the storage is more efficient if you pack more documents
together, because you have less duplication in the strings. For usage examples,
see the docs on [serializing `Doc` objects](/usage/saving-loading#docs).
## DocBin.\_\_init\_\_ {#init tag="method"}
## DocBin.\_\_init\_\_ {id="init",tag="method"}
Create a `DocBin` object to hold serialized annotations.
@ -50,7 +49,7 @@ Create a `DocBin` object to hold serialized annotations.
| `store_user_data` | Whether to write the `Doc.user_data` and the values of custom extension attributes to file/bytes. Defaults to `False`. ~~bool~~ |
| `docs` | `Doc` objects to add on initialization. ~~Iterable[Doc]~~ |
## DocBin.\_\len\_\_ {#len tag="method"}
## DocBin.\_\_len\_\_ {id="len",tag="method"}
Get the number of `Doc` objects that were added to the `DocBin`.
@ -67,7 +66,7 @@ Get the number of `Doc` objects that were added to the `DocBin`.
| ----------- | --------------------------------------------------- |
| **RETURNS** | The number of `Doc`s added to the `DocBin`. ~~int~~ |
## DocBin.add {#add tag="method"}
## DocBin.add {id="add",tag="method"}
Add a `Doc`'s annotations to the `DocBin` for serialization.
@ -83,7 +82,7 @@ Add a `Doc`'s annotations to the `DocBin` for serialization.
| -------- | -------------------------------- |
| `doc` | The `Doc` object to add. ~~Doc~~ |
## DocBin.get_docs {#get_docs tag="method"}
## DocBin.get_docs {id="get_docs",tag="method"}
Recover `Doc` objects from the annotations, using the given vocab.
@ -98,7 +97,7 @@ Recover `Doc` objects from the annotations, using the given vocab.
| `vocab` | The shared vocab. ~~Vocab~~ |
| **YIELDS** | The `Doc` objects. ~~Doc~~ |
## DocBin.merge {#merge tag="method"}
## DocBin.merge {id="merge",tag="method"}
Extend the annotations of this `DocBin` with the annotations from another. Will
raise an error if the pre-defined `attrs` of the two `DocBin`s don't match.
@ -118,7 +117,7 @@ raise an error if the pre-defined `attrs` of the two `DocBin`s don't match.
| -------- | ------------------------------------------------------ |
| `other` | The `DocBin` to merge into the current bin. ~~DocBin~~ |
## DocBin.to_bytes {#to_bytes tag="method"}
## DocBin.to_bytes {id="to_bytes",tag="method"}
Serialize the `DocBin`'s annotations to a bytestring.
@ -134,7 +133,7 @@ Serialize the `DocBin`'s annotations to a bytestring.
| ----------- | ---------------------------------- |
| **RETURNS** | The serialized `DocBin`. ~~bytes~~ |
## DocBin.from_bytes {#from_bytes tag="method"}
## DocBin.from_bytes {id="from_bytes",tag="method"}
Deserialize the `DocBin`'s annotations from a bytestring.
@ -150,7 +149,7 @@ Deserialize the `DocBin`'s annotations from a bytestring.
| `bytes_data` | The data to load from. ~~bytes~~ |
| **RETURNS** | The loaded `DocBin`. ~~DocBin~~ |
## DocBin.to_disk {#to_disk tag="method" new="3"}
## DocBin.to_disk {id="to_disk",tag="method",version="3"}
Save the serialized `DocBin` to a file. Typically uses the `.spacy` extension
and the result can be used as the input data for
@ -168,7 +167,7 @@ and the result can be used as the input data for
| -------- | -------------------------------------------------------------------------- |
| `path` | The file path, typically with the `.spacy` extension. ~~Union[str, Path]~~ |
## DocBin.from_disk {#from_disk tag="method" new="3"}
## DocBin.from_disk {id="from_disk",tag="method",version="3"}
Load a serialized `DocBin` from a file. Typically uses the `.spacy` extension.

40
website/docs/api/edittreelemmatizer.md → website/docs/api/edittreelemmatizer.mdx
View File

@ -2,7 +2,7 @@
title: EditTreeLemmatizer
tag: class
source: spacy/pipeline/edit_tree_lemmatizer.py
new: 3.3
version: 3.3
teaser: 'Pipeline component for lemmatization'
api_base_class: /api/pipe
api_string_name: trainable_lemmatizer
@ -18,7 +18,7 @@ and construction method used by this lemmatizer were proposed in
For a lookup and rule-based lemmatizer, see [`Lemmatizer`](/api/lemmatizer).
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Predictions are assigned to `Token.lemma`.
@ -27,7 +27,7 @@ Predictions are assigned to `Token.lemma`.
| `Token.lemma` | The lemma (hash). ~~int~~ |
| `Token.lemma_` | The lemma. ~~str~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -57,7 +57,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/edit_tree_lemmatizer.py
```
## EditTreeLemmatizer.\_\_init\_\_ {#init tag="method"}
## EditTreeLemmatizer.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -90,7 +90,7 @@ shortcut for this and instantiate the component using its string name and
| `top_k` | The number of most probable edit trees to try before resorting to `backoff`. Defaults to `1`. ~~int~~ |
| `scorer` | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attribute `"lemma"`. ~~Optional[Callable]~~ |
## EditTreeLemmatizer.\_\_call\_\_ {#call tag="method"}
## EditTreeLemmatizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -114,7 +114,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## EditTreeLemmatizer.pipe {#pipe tag="method"}
## EditTreeLemmatizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -138,7 +138,7 @@ and [`pipe`](/api/edittreelemmatizer#pipe) delegate to the
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## EditTreeLemmatizer.initialize {#initialize tag="method" new="3"}
## EditTreeLemmatizer.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -175,7 +175,7 @@ config.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ |
## EditTreeLemmatizer.predict {#predict tag="method"}
## EditTreeLemmatizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them.
@ -192,7 +192,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. |
## EditTreeLemmatizer.set_annotations {#set_annotations tag="method"}
## EditTreeLemmatizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed tree
identifiers.
@ -210,7 +210,7 @@ identifiers.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `tree_ids` | The identifiers of the edit trees to apply, produced by `EditTreeLemmatizer.predict`. |
## EditTreeLemmatizer.update {#update tag="method"}
## EditTreeLemmatizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model.
@ -234,7 +234,7 @@ Delegates to [`predict`](/api/edittreelemmatizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## EditTreeLemmatizer.get_loss {#get_loss tag="method"}
## EditTreeLemmatizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their
predicted scores.
@ -253,7 +253,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## EditTreeLemmatizer.create_optimizer {#create_optimizer tag="method"}
## EditTreeLemmatizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -268,7 +268,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## EditTreeLemmatizer.use_params {#use_params tag="method, contextmanager"}
## EditTreeLemmatizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -285,7 +285,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## EditTreeLemmatizer.to_disk {#to_disk tag="method"}
## EditTreeLemmatizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -302,7 +302,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## EditTreeLemmatizer.from_disk {#from_disk tag="method"}
## EditTreeLemmatizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -320,7 +320,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `EditTreeLemmatizer` object. ~~EditTreeLemmatizer~~ |
## EditTreeLemmatizer.to_bytes {#to_bytes tag="method"}
## EditTreeLemmatizer.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -337,7 +337,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `EditTreeLemmatizer` object. ~~bytes~~ |
## EditTreeLemmatizer.from_bytes {#from_bytes tag="method"}
## EditTreeLemmatizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -356,7 +356,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `EditTreeLemmatizer` object. ~~EditTreeLemmatizer~~ |
## EditTreeLemmatizer.labels {#labels tag="property"}
## EditTreeLemmatizer.labels {id="labels",tag="property"}
The labels currently added to the component.
@ -371,7 +371,7 @@ identifiers of edit trees.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## EditTreeLemmatizer.label_data {#label_data tag="property" new="3"}
## EditTreeLemmatizer.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -389,7 +389,7 @@ initialize the model with a pre-defined label set.
| ----------- | ---------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

36
website/docs/api/entitylinker.md → website/docs/api/entitylinker.mdx
View File

@ -2,7 +2,7 @@
title: EntityLinker
tag: class
source: spacy/pipeline/entity_linker.py
new: 2.2
version: 2.2
teaser: 'Pipeline component for named entity linking and disambiguation'
api_base_class: /api/pipe
api_string_name: entity_linker
@ -17,7 +17,7 @@ and a machine learning model to pick the right candidate, given the local
context of the mention. `EntityLinker` defaults to using the
[`InMemoryLookupKB`](/api/kb_in_memory) implementation.
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Predictions, in the form of knowledge base IDs, will be assigned to
`Token.ent_kb_id_`.
@ -27,7 +27,7 @@ Predictions, in the form of knowledge base IDs, will be assigned to
| `Token.ent_kb_id` | Knowledge base ID (hash). ~~int~~ |
| `Token.ent_kb_id_` | Knowledge base ID. ~~str~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -71,7 +71,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/entity_linker.py
```
## EntityLinker.\_\_init\_\_ {#init tag="method"}
## EntityLinker.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -114,7 +114,7 @@ custom knowledge base, you should either call
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_links`](/api/scorer#score_links). ~~Optional[Callable]~~ |
| `threshold` <Tag variant="new">3.4</Tag> | Confidence threshold for entity predictions. The default of `None` implies that all predictions are accepted, otherwise those with a score beneath the treshold are discarded. If there are no predictions with scores above the threshold, the linked entity is `NIL`. ~~Optional[float]~~ |
## EntityLinker.\_\_call\_\_ {#call tag="method"}
## EntityLinker.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -137,7 +137,7 @@ delegate to the [`predict`](/api/entitylinker#predict) and
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## EntityLinker.pipe {#pipe tag="method"}
## EntityLinker.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -161,7 +161,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/entitylinker#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## EntityLinker.set_kb {#set_kb tag="method" new="3"}
## EntityLinker.set_kb {id="set_kb",tag="method",version="3"}
The `kb_loader` should be a function that takes a `Vocab` instance and creates
the `KnowledgeBase`, ensuring that the strings of the knowledge base are synced
@ -183,7 +183,7 @@ with the current vocab.
| ----------- | ---------------------------------------------------------------------------------------------------------------- |
| `kb_loader` | Function that creates a [`KnowledgeBase`](/api/kb) from a `Vocab` instance. ~~Callable[[Vocab], KnowledgeBase]~~ |
## EntityLinker.initialize {#initialize tag="method" new="3"}
## EntityLinker.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -219,7 +219,7 @@ This method was previously called `begin_training`.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `kb_loader` | Function that creates a [`KnowledgeBase`](/api/kb) from a `Vocab` instance. ~~Callable[[Vocab], KnowledgeBase]~~ |
## EntityLinker.predict {#predict tag="method"}
## EntityLinker.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. Returns the KB IDs for each entity in each doc, including `NIL`
@ -237,7 +237,7 @@ if there is no prediction.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The predicted KB identifiers for the entities in the `docs`. ~~List[str]~~ |
## EntityLinker.set_annotations {#set_annotations tag="method"}
## EntityLinker.set_annotations {id="set_annotations",tag="method"}
Modify a batch of documents, using pre-computed entity IDs for a list of named
entities.
@ -255,7 +255,7 @@ entities.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `kb_ids` | The knowledge base identifiers for the entities in the docs, predicted by `EntityLinker.predict`. ~~List[str]~~ |
## EntityLinker.update {#update tag="method"}
## EntityLinker.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects, updating both the
pipe's entity linking model and context encoder. Delegates to
@ -278,7 +278,7 @@ pipe's entity linking model and context encoder. Delegates to
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## EntityLinker.create_optimizer {#create_optimizer tag="method"}
## EntityLinker.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -293,7 +293,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## EntityLinker.use_params {#use_params tag="method, contextmanager"}
## EntityLinker.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -310,7 +310,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## EntityLinker.to_disk {#to_disk tag="method"}
## EntityLinker.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -327,7 +327,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## EntityLinker.from_disk {#from_disk tag="method"}
## EntityLinker.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -345,7 +345,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `EntityLinker` object. ~~EntityLinker~~ |
## EntityLinker.to_bytes {#to_bytes tag="method"}
## EntityLinker.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -362,7 +362,7 @@ Serialize the pipe to a bytestring, including the `KnowledgeBase`.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `EntityLinker` object. ~~bytes~~ |
## EntityLinker.from_bytes {#from_bytes tag="method"}
## EntityLinker.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -381,7 +381,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `EntityLinker` object. ~~EntityLinker~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

42
website/docs/api/entityrecognizer.md → website/docs/api/entityrecognizer.mdx
View File

@ -20,7 +20,7 @@ your entities will be close to their initial tokens. If your entities are long
and characterized by tokens in their middle, the component will likely not be a
good fit for your task.
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Predictions will be saved to `Doc.ents` as a tuple. Each label will also be
reflected to each underlying token, where it is saved in the `Token.ent_type`
@ -38,7 +38,7 @@ non-overlapping, or an error will be thrown.
| `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ |
| `Token.ent_type_` | The label part of the named entity tag. ~~str~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -72,7 +72,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/ner.pyx
```
## EntityRecognizer.\_\_init\_\_ {#init tag="method"}
## EntityRecognizer.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -103,7 +103,7 @@ shortcut for this and instantiate the component using its string name and
| `update_with_oracle_cut_size` | During training, cut long sequences into shorter segments by creating intermediate states based on the gold-standard history. The model is not very sensitive to this parameter, so you usually won't need to change it. Defaults to `100`. ~~int~~ |
| `incorrect_spans_key` | Identifies spans that are known to be incorrect entity annotations. The incorrect entity annotations can be stored in the span group in [`Doc.spans`](/api/doc#spans), under this key. Defaults to `None`. ~~Optional[str]~~ |
## EntityRecognizer.\_\_call\_\_ {#call tag="method"}
## EntityRecognizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -127,7 +127,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## EntityRecognizer.pipe {#pipe tag="method"}
## EntityRecognizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -151,7 +151,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/entityrecognizer#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## EntityRecognizer.initialize {#initialize tag="method" new="3"}
## EntityRecognizer.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -194,7 +194,7 @@ This method was previously called `begin_training`.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Dict[str, Dict[str, int]]]~~ |
## EntityRecognizer.predict {#predict tag="method"}
## EntityRecognizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them.
@ -211,7 +211,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | A helper class for the parse state (internal). ~~StateClass~~ |
## EntityRecognizer.set_annotations {#set_annotations tag="method"}
## EntityRecognizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -228,7 +228,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `EntityRecognizer.predict`. Returns an internal helper class for the parse state. ~~List[StateClass]~~ |
## EntityRecognizer.update {#update tag="method"}
## EntityRecognizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects, updating the pipe's
model. Delegates to [`predict`](/api/entityrecognizer#predict) and
@ -251,7 +251,7 @@ model. Delegates to [`predict`](/api/entityrecognizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## EntityRecognizer.get_loss {#get_loss tag="method"}
## EntityRecognizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their
predicted scores.
@ -270,7 +270,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. ~~StateClass~~ |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## EntityRecognizer.create_optimizer {#create_optimizer tag="method"}
## EntityRecognizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -285,7 +285,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## EntityRecognizer.use_params {#use_params tag="method, contextmanager"}
## EntityRecognizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -302,7 +302,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## EntityRecognizer.add_label {#add_label tag="method"}
## EntityRecognizer.add_label {id="add_label",tag="method"}
Add a new label to the pipe. Note that you don't have to call this method if you
provide a **representative data sample** to the [`initialize`](#initialize)
@ -322,7 +322,7 @@ to the model, and the output dimension will be
| `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## EntityRecognizer.set_output {#set_output tag="method"}
## EntityRecognizer.set_output {id="set_output",tag="method"}
Change the output dimension of the component's model by calling the model's
attribute `resize_output`. This is a function that takes the original model and
@ -341,7 +341,7 @@ forgetting" problem.
| ---- | --------------------------------- |
| `nO` | The new output dimension. ~~int~~ |
## EntityRecognizer.to_disk {#to_disk tag="method"}
## EntityRecognizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -358,7 +358,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## EntityRecognizer.from_disk {#from_disk tag="method"}
## EntityRecognizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -376,7 +376,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `EntityRecognizer` object. ~~EntityRecognizer~~ |
## EntityRecognizer.to_bytes {#to_bytes tag="method"}
## EntityRecognizer.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -393,7 +393,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `EntityRecognizer` object. ~~bytes~~ |
## EntityRecognizer.from_bytes {#from_bytes tag="method"}
## EntityRecognizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -412,7 +412,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `EntityRecognizer` object. ~~EntityRecognizer~~ |
## EntityRecognizer.labels {#labels tag="property"}
## EntityRecognizer.labels {id="labels",tag="property"}
The labels currently added to the component.
@ -427,7 +427,7 @@ The labels currently added to the component.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## EntityRecognizer.label_data {#label_data tag="property" new="3"}
## EntityRecognizer.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -445,7 +445,7 @@ the model with a pre-defined label set.
| ----------- | ------------------------------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Dict[str, Dict[str, Dict[str, int]]]~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

36
website/docs/api/entityruler.md → website/docs/api/entityruler.mdx
View File

@ -2,7 +2,7 @@
title: EntityRuler
tag: class
source: spacy/pipeline/entityruler.py
new: 2.1
version: 2.1
teaser: 'Pipeline component for rule-based named entity recognition'
api_string_name: entity_ruler
api_trainable: false
@ -15,7 +15,7 @@ used on its own to implement a purely rule-based entity recognition system. For
usage examples, see the docs on
[rule-based entity recognition](/usage/rule-based-matching#entityruler).
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
This component assigns predictions basically the same way as the
[`EntityRecognizer`](/api/entityrecognizer).
@ -36,7 +36,7 @@ non-overlapping, or an error will be thrown.
| `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ |
| `Token.ent_type_` | The label part of the named entity tag. ~~str~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -68,7 +68,7 @@ how the component should be configured. You can override its settings via the
%%GITHUB_SPACY/spacy/pipeline/entityruler.py
```
## EntityRuler.\_\_init\_\_ {#init tag="method"}
## EntityRuler.\_\_init\_\_ {id="init",tag="method"}
Initialize the entity ruler. If patterns are supplied here, they need to be a
list of dictionaries with a `"label"` and `"pattern"` key. A pattern can either
@ -99,7 +99,7 @@ be a token pattern (list) or a phrase pattern (string). For example:
| `patterns` | Optional patterns to load in on initialization. ~~Optional[List[Dict[str, Union[str, List[dict]]]]]~~ |
| `scorer` | The scoring method. Defaults to [`spacy.scorer.get_ner_prf`](/api/scorer#get_ner_prf). ~~Optional[Callable]~~ |
## EntityRuler.initialize {#initialize tag="method" new="3"}
## EntityRuler.initialize {id="initialize",tag="method",version="3"}
Initialize the component with data and used before training to load in rules
from a [pattern file](/usage/rule-based-matching/#entityruler-files). This
@ -131,7 +131,7 @@ config.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `patterns` | The list of patterns. Defaults to `None`. ~~Optional[Sequence[Dict[str, Union[str, List[Dict[str, Any]]]]]]~~ |
## EntityRuler.\_\len\_\_ {#len tag="method"}
## EntityRuler.\_\_len\_\_ {id="len",tag="method"}
The number of all patterns added to the entity ruler.
@ -148,7 +148,7 @@ The number of all patterns added to the entity ruler.
| ----------- | ------------------------------- |
| **RETURNS** | The number of patterns. ~~int~~ |
## EntityRuler.\_\_contains\_\_ {#contains tag="method"}
## EntityRuler.\_\_contains\_\_ {id="contains",tag="method"}
Whether a label is present in the patterns.
@ -166,7 +166,7 @@ Whether a label is present in the patterns.
| `label` | The label to check. ~~str~~ |
| **RETURNS** | Whether the entity ruler contains the label. ~~bool~~ |
## EntityRuler.\_\_call\_\_ {#call tag="method"}
## EntityRuler.\_\_call\_\_ {id="call",tag="method"}
Find matches in the `Doc` and add them to the `doc.ents`. Typically, this
happens automatically after the component has been added to the pipeline using
@ -192,7 +192,7 @@ is chosen.
| `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ |
| **RETURNS** | The modified `Doc` with added entities, if available. ~~Doc~~ |
## EntityRuler.add_patterns {#add_patterns tag="method"}
## EntityRuler.add_patterns {id="add_patterns",tag="method"}
Add patterns to the entity ruler. A pattern can either be a token pattern (list
of dicts) or a phrase pattern (string). For more details, see the usage guide on
@ -213,7 +213,7 @@ of dicts) or a phrase pattern (string). For more details, see the usage guide on
| ---------- | ---------------------------------------------------------------- |
| `patterns` | The patterns to add. ~~List[Dict[str, Union[str, List[dict]]]]~~ |
## EntityRuler.remove {#remove tag="method" new="3.2.1"}
## EntityRuler.remove {id="remove",tag="method",version="3.2.1"}
Remove a pattern by its ID from the entity ruler. A `ValueError` is raised if
the ID does not exist.
@ -231,7 +231,7 @@ the ID does not exist.
| ---- | ----------------------------------- |
| `id` | The ID of the pattern rule. ~~str~~ |
## EntityRuler.to_disk {#to_disk tag="method"}
## EntityRuler.to_disk {id="to_disk",tag="method"}
Save the entity ruler patterns to a directory. The patterns will be saved as
newline-delimited JSON (JSONL). If a file with the suffix `.jsonl` is provided,
@ -250,7 +250,7 @@ only the patterns are saved as JSONL. If a directory name is provided, a
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `path` | A path to a JSONL file or directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## EntityRuler.from_disk {#from_disk tag="method"}
## EntityRuler.from_disk {id="from_disk",tag="method"}
Load the entity ruler from a path. Expects either a file containing
newline-delimited JSON (JSONL) with one entry per line, or a directory
@ -270,7 +270,7 @@ configuration.
| `path` | A path to a JSONL file or directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The modified `EntityRuler` object. ~~EntityRuler~~ |
## EntityRuler.to_bytes {#to_bytes tag="method"}
## EntityRuler.to_bytes {id="to_bytes",tag="method"}
Serialize the entity ruler patterns to a bytestring.
@ -285,7 +285,7 @@ Serialize the entity ruler patterns to a bytestring.
| ----------- | ---------------------------------- |
| **RETURNS** | The serialized patterns. ~~bytes~~ |
## EntityRuler.from_bytes {#from_bytes tag="method"}
## EntityRuler.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -302,7 +302,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `bytes_data` | The bytestring to load. ~~bytes~~ |
| **RETURNS** | The modified `EntityRuler` object. ~~EntityRuler~~ |
## EntityRuler.labels {#labels tag="property"}
## EntityRuler.labels {id="labels",tag="property"}
All labels present in the match patterns.
@ -310,7 +310,7 @@ All labels present in the match patterns.
| ----------- | -------------------------------------- |
| **RETURNS** | The string labels. ~~Tuple[str, ...]~~ |
## EntityRuler.ent_ids {#ent_ids tag="property" new="2.2.2"}
## EntityRuler.ent_ids {id="ent_ids",tag="property",version="2.2.2"}
All entity IDs present in the `id` properties of the match patterns.
@ -318,7 +318,7 @@ All entity IDs present in the `id` properties of the match patterns.
| ----------- | ----------------------------------- |
| **RETURNS** | The string IDs. ~~Tuple[str, ...]~~ |
## EntityRuler.patterns {#patterns tag="property"}
## EntityRuler.patterns {id="patterns",tag="property"}
Get all patterns that were added to the entity ruler.
@ -326,7 +326,7 @@ Get all patterns that were added to the entity ruler.
| ----------- | ---------------------------------------------------------------------------------------- |
| **RETURNS** | The original patterns, one dictionary per pattern. ~~List[Dict[str, Union[str, dict]]]~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| ----------------- | --------------------------------------------------------------------------------------------------------------------- |

40
website/docs/api/example.md → website/docs/api/example.mdx
View File

@ -3,7 +3,7 @@ title: Example
teaser: A training instance
tag: class
source: spacy/training/example.pyx
new: 3.0
version: 3.0
---
An `Example` holds the information for one training instance. It stores two
@ -12,7 +12,7 @@ holding the predictions of the pipeline. An
[`Alignment`](/api/example#alignment-object) object stores the alignment between
these two documents, as they can differ in tokenization.
## Example.\_\_init\_\_ {#init tag="method"}
## Example.\_\_init\_\_ {id="init",tag="method"}
Construct an `Example` object from the `predicted` document and the `reference`
document. If `alignment` is `None`, it will be initialized from the words in
@ -40,7 +40,7 @@ both documents.
| _keyword-only_ | |
| `alignment` | An object holding the alignment between the tokens of the `predicted` and `reference` documents. ~~Optional[Alignment]~~ |
## Example.from_dict {#from_dict tag="classmethod"}
## Example.from_dict {id="from_dict",tag="classmethod"}
Construct an `Example` object from the `predicted` document and the reference
annotations provided as a dictionary. For more details on the required format,
@ -64,7 +64,7 @@ see the [training format documentation](/api/data-formats#dict-input).
| `example_dict` | The gold-standard annotations as a dictionary. Cannot be `None`. ~~Dict[str, Any]~~ |
| **RETURNS** | The newly constructed object. ~~Example~~ |
## Example.text {#text tag="property"}
## Example.text {id="text",tag="property"}
The text of the `predicted` document in this `Example`.
@ -78,7 +78,7 @@ The text of the `predicted` document in this `Example`.
| ----------- | --------------------------------------------- |
| **RETURNS** | The text of the `predicted` document. ~~str~~ |
## Example.predicted {#predicted tag="property"}
## Example.predicted {id="predicted",tag="property"}
The `Doc` holding the predictions. Occasionally also referred to as `example.x`.
@ -94,7 +94,7 @@ The `Doc` holding the predictions. Occasionally also referred to as `example.x`.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The document containing (partial) predictions. ~~Doc~~ |
## Example.reference {#reference tag="property"}
## Example.reference {id="reference",tag="property"}
The `Doc` holding the gold-standard annotations. Occasionally also referred to
as `example.y`.
@ -111,7 +111,7 @@ as `example.y`.
| ----------- | ---------------------------------------------------------- |
| **RETURNS** | The document containing gold-standard annotations. ~~Doc~~ |
## Example.alignment {#alignment tag="property"}
## Example.alignment {id="alignment",tag="property"}
The [`Alignment`](/api/example#alignment-object) object mapping the tokens of
the `predicted` document to those of the `reference` document.
@ -131,7 +131,7 @@ the `predicted` document to those of the `reference` document.
| ----------- | ---------------------------------------------------------------- |
| **RETURNS** | The document containing gold-standard annotations. ~~Alignment~~ |
## Example.get_aligned {#get_aligned tag="method"}
## Example.get_aligned {id="get_aligned",tag="method"}
Get the aligned view of a certain token attribute, denoted by its int ID or
string name.
@ -152,7 +152,7 @@ string name.
| `as_string` | Whether or not to return the list of values as strings. Defaults to `False`. ~~bool~~ |
| **RETURNS** | List of integer values, or string values if `as_string` is `True`. ~~Union[List[int], List[str]]~~ |
## Example.get_aligned_parse {#get_aligned_parse tag="method"}
## Example.get_aligned_parse {id="get_aligned_parse",tag="method"}
Get the aligned view of the dependency parse. If `projectivize` is set to
`True`, non-projective dependency trees are made projective through the
@ -172,7 +172,7 @@ Pseudo-Projective Dependency Parsing algorithm by Nivre and Nilsson (2005).
| `projectivize` | Whether or not to projectivize the dependency trees. Defaults to `True`. ~~bool~~ |
| **RETURNS** | List of integer values, or string values if `as_string` is `True`. ~~Union[List[int], List[str]]~~ |
## Example.get_aligned_ner {#get_aligned_ner tag="method"}
## Example.get_aligned_ner {id="get_aligned_ner",tag="method"}
Get the aligned view of the NER
[BILUO](/usage/linguistic-features#accessing-ner) tags.
@ -193,7 +193,7 @@ Get the aligned view of the NER
| ----------- | ------------------------------------------------------------------------------------------------- |
| **RETURNS** | List of BILUO values, denoting whether tokens are part of an NER annotation or not. ~~List[str]~~ |
## Example.get_aligned_spans_y2x {#get_aligned_spans_y2x tag="method"}
## Example.get_aligned_spans_y2x {id="get_aligned_spans_y2x",tag="method"}
Get the aligned view of any set of [`Span`](/api/span) objects defined over
[`Example.reference`](/api/example#reference). The resulting span indices will
@ -219,7 +219,7 @@ align to the tokenization in [`Example.predicted`](/api/example#predicted).
| `allow_overlap` | Whether the resulting `Span` objects may overlap or not. Set to `False` by default. ~~bool~~ |
| **RETURNS** | `Span` objects aligned to the tokenization of `predicted`. ~~List[Span]~~ |
## Example.get_aligned_spans_x2y {#get_aligned_spans_x2y tag="method"}
## Example.get_aligned_spans_x2y {id="get_aligned_spans_x2y",tag="method"}
Get the aligned view of any set of [`Span`](/api/span) objects defined over
[`Example.predicted`](/api/example#predicted). The resulting span indices will
@ -247,7 +247,7 @@ against the original gold-standard annotation.
| `allow_overlap` | Whether the resulting `Span` objects may overlap or not. Set to `False` by default. ~~bool~~ |
| **RETURNS** | `Span` objects aligned to the tokenization of `reference`. ~~List[Span]~~ |
## Example.to_dict {#to_dict tag="method"}
## Example.to_dict {id="to_dict",tag="method"}
Return a [dictionary representation](/api/data-formats#dict-input) of the
reference annotation contained in this `Example`.
@ -262,7 +262,7 @@ reference annotation contained in this `Example`.
| ----------- | ------------------------------------------------------------------------- |
| **RETURNS** | Dictionary representation of the reference annotation. ~~Dict[str, Any]~~ |
## Example.split_sents {#split_sents tag="method"}
## Example.split_sents {id="split_sents",tag="method"}
Split one `Example` into multiple `Example` objects, one for each sentence.
@ -282,15 +282,15 @@ Split one `Example` into multiple `Example` objects, one for each sentence.
| ----------- | ---------------------------------------------------------------------------- |
| **RETURNS** | List of `Example` objects, one for each original sentence. ~~List[Example]~~ |
## Alignment {#alignment-object new="3"}
## Alignment {id="alignment-object",version="3"}
Calculate alignment tables between two tokenizations.
### Alignment attributes {#alignment-attributes"}
### Alignment attributes {id="alignment-attributes"}
Alignment attributes are managed using `AlignmentArray`, which is a
simplified version of Thinc's [Ragged](https://thinc.ai/docs/api-types#ragged)
type that only supports the `data` and `length` attributes.
Alignment attributes are managed using `AlignmentArray`, which is a simplified
version of Thinc's [Ragged](https://thinc.ai/docs/api-types#ragged) type that
only supports the `data` and `length` attributes.
| Name | Description |
| ----- | ------------------------------------------------------------------------------------- |
@ -321,7 +321,7 @@ tokenizations add up to the same string. For example, you'll be able to align
> If `a2b.data[1] == a2b.data[2] == 1`, that means that `A[1]` (`"'"`) and
> `A[2]` (`"s"`) both align to `B[1]` (`"'s"`).
### Alignment.from_strings {#classmethod tag="function"}
### Alignment.from_strings {id="classmethod",tag="function"}
| Name | Description |
| ----------- | ------------------------------------------------------------- |

2
website/docs/api/index.md → website/docs/api/index.mdx
View File

@ -3,6 +3,4 @@ title: Library Architecture
next: /api/architectures
---
import Architecture101 from 'usage/101/\_architecture.md'
<Architecture101 />

28
website/docs/api/kb.md → website/docs/api/kb.mdx
View File

@ -5,7 +5,7 @@ teaser:
(ontology)
tag: class
source: spacy/kb/kb.pyx
new: 2.2
version: 2.2
---
The `KnowledgeBase` object is an abstract class providing a method to generate
@ -26,7 +26,7 @@ onwards.
</Infobox>
## KnowledgeBase.\_\_init\_\_ {#init tag="method"}
## KnowledgeBase.\_\_init\_\_ {id="init",tag="method"}
`KnowledgeBase` is an abstract class and cannot be instantiated. Its child
classes should call `__init__()` to set up some necessary attributes.
@ -50,7 +50,7 @@ classes should call `__init__()` to set up some necessary attributes.
| `vocab` | The shared vocabulary. ~~Vocab~~ |
| `entity_vector_length` | Length of the fixed-size entity vectors. ~~int~~ |
## KnowledgeBase.entity_vector_length {#entity_vector_length tag="property"}
## KnowledgeBase.entity_vector_length {id="entity_vector_length",tag="property"}
The length of the fixed-size entity vectors in the knowledge base.
@ -58,7 +58,7 @@ The length of the fixed-size entity vectors in the knowledge base.
| ----------- | ------------------------------------------------ |
| **RETURNS** | Length of the fixed-size entity vectors. ~~int~~ |
## KnowledgeBase.get_candidates {#get_candidates tag="method"}
## KnowledgeBase.get_candidates {id="get_candidates",tag="method"}
Given a certain textual mention as input, retrieve a list of candidate entities
of type [`Candidate`](/api/kb#candidate).
@ -77,7 +77,7 @@ of type [`Candidate`](/api/kb#candidate).
| `mention` | The textual mention or alias. ~~Span~~ |
| **RETURNS** | An iterable of relevant `Candidate` objects. ~~Iterable[Candidate]~~ |
## KnowledgeBase.get_candidates_batch {#get_candidates_batch tag="method"}
## KnowledgeBase.get_candidates_batch {id="get_candidates_batch",tag="method"}
Same as [`get_candidates()`](/api/kb#get_candidates), but for an arbitrary
number of mentions. The [`EntityLinker`](/api/entitylinker) component will call
@ -103,10 +103,10 @@ to you.
| `mentions` | The textual mention or alias. ~~Iterable[Span]~~ |
| **RETURNS** | An iterable of iterable with relevant `Candidate` objects. ~~Iterable[Iterable[Candidate]]~~ |
## KnowledgeBase.get_alias_candidates {#get_alias_candidates tag="method"}
## KnowledgeBase.get_alias_candidates {id="get_alias_candidates",tag="method"}
<Infobox variant="warning">
This method is _not_ available from spaCy 3.5 onwards.
This method is _not_ available from spaCy 3.5 onwards.
</Infobox>
From spaCy 3.5 on `KnowledgeBase` is an abstract class (with
@ -119,7 +119,7 @@ Note: [`InMemoryLookupKB.get_candidates()`](/api/kb_in_memory#get_candidates)
defaults to
[`InMemoryLookupKB.get_alias_candidates()`](/api/kb_in_memory#get_alias_candidates).
## KnowledgeBase.get_vector {#get_vector tag="method"}
## KnowledgeBase.get_vector {id="get_vector",tag="method"}
Given a certain entity ID, retrieve its pretrained entity vector.
@ -134,7 +134,7 @@ Given a certain entity ID, retrieve its pretrained entity vector.
| `entity` | The entity ID. ~~str~~ |
| **RETURNS** | The entity vector. ~~Iterable[float]~~ |
## KnowledgeBase.get_vectors {#get_vectors tag="method"}
## KnowledgeBase.get_vectors {id="get_vectors",tag="method"}
Same as [`get_vector()`](/api/kb#get_vector), but for an arbitrary number of
entity IDs.
@ -154,7 +154,7 @@ entities at once, if performance is of concern to you.
| `entities` | The entity IDs. ~~Iterable[str]~~ |
| **RETURNS** | The entity vectors. ~~Iterable[Iterable[numpy.ndarray]]~~ |
## KnowledgeBase.to_disk {#to_disk tag="method"}
## KnowledgeBase.to_disk {id="to_disk",tag="method"}
Save the current state of the knowledge base to a directory.
@ -169,7 +169,7 @@ Save the current state of the knowledge base to a directory.
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| `exclude` | List of components to exclude. ~~Iterable[str]~~ |
## KnowledgeBase.from_disk {#from_disk tag="method"}
## KnowledgeBase.from_disk {id="from_disk",tag="method"}
Restore the state of the knowledge base from a given directory. Note that the
[`Vocab`](/api/vocab) should also be the same as the one used to create the KB.
@ -189,7 +189,7 @@ Restore the state of the knowledge base from a given directory. Note that the
| `exclude` | List of components to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `KnowledgeBase` object. ~~KnowledgeBase~~ |
## Candidate {#candidate tag="class"}
## Candidate {id="candidate",tag="class"}
A `Candidate` object refers to a textual mention (alias) that may or may not be
resolved to a specific entity from a `KnowledgeBase`. This will be used as input
@ -197,7 +197,7 @@ for the entity linking algorithm which will disambiguate the various candidates
to the correct one. Each candidate `(alias, entity)` pair is assigned to a
certain prior probability.
### Candidate.\_\_init\_\_ {#candidate-init tag="method"}
### Candidate.\_\_init\_\_ {id="candidate-init",tag="method"}
Construct a `Candidate` object. Usually this constructor is not called directly,
but instead these objects are returned by the `get_candidates` method of the
@ -218,7 +218,7 @@ but instead these objects are returned by the `get_candidates` method of the
| `alias_hash` | The hash of the textual mention or alias. ~~int~~ |
| `prior_prob` | The prior probability of the `alias` referring to the `entity`. ~~float~~ |
## Candidate attributes {#candidate-attributes}
## Candidate attributes {id="candidate-attributes"}
| Name | Description |
| --------------- | ------------------------------------------------------------------------ |

36
website/docs/api/kb_in_memory.md → website/docs/api/kb_in_memory.mdx
View File

@ -5,7 +5,7 @@ teaser:
information in-memory.
tag: class
source: spacy/kb/kb_in_memory.pyx
new: 3.5
version: 3.5
---
The `InMemoryLookupKB` class inherits from [`KnowledgeBase`](/api/kb) and
@ -14,7 +14,7 @@ implements all of its methods. It stores all KB data in-memory and generates
entity names. It's highly optimized for both a low memory footprint and speed of
retrieval.
## InMemoryLookupKB.\_\_init\_\_ {#init tag="method"}
## InMemoryLookupKB.\_\_init\_\_ {id="init",tag="method"}
Create the knowledge base.
@ -31,7 +31,7 @@ Create the knowledge base.
| `vocab` | The shared vocabulary. ~~Vocab~~ |
| `entity_vector_length` | Length of the fixed-size entity vectors. ~~int~~ |
## InMemoryLookupKB.entity_vector_length {#entity_vector_length tag="property"}
## InMemoryLookupKB.entity_vector_length {id="entity_vector_length",tag="property"}
The length of the fixed-size entity vectors in the knowledge base.
@ -39,7 +39,7 @@ The length of the fixed-size entity vectors in the knowledge base.
| ----------- | ------------------------------------------------ |
| **RETURNS** | Length of the fixed-size entity vectors. ~~int~~ |
## InMemoryLookupKB.add_entity {#add_entity tag="method"}
## InMemoryLookupKB.add_entity {id="add_entity",tag="method"}
Add an entity to the knowledge base, specifying its corpus frequency and entity
vector, which should be of length
@ -58,7 +58,7 @@ vector, which should be of length
| `freq` | The frequency of the entity in a typical corpus. ~~float~~ |
| `entity_vector` | The pretrained vector of the entity. ~~numpy.ndarray~~ |
## InMemoryLookupKB.set_entities {#set_entities tag="method"}
## InMemoryLookupKB.set_entities {id="set_entities",tag="method"}
Define the full list of entities in the knowledge base, specifying the corpus
frequency and entity vector for each entity.
@ -75,7 +75,7 @@ frequency and entity vector for each entity.
| `freq_list` | List of entity frequencies. ~~Iterable[int]~~ |
| `vector_list` | List of entity vectors. ~~Iterable[numpy.ndarray]~~ |
## InMemoryLookupKB.add_alias {#add_alias tag="method"}
## InMemoryLookupKB.add_alias {id="add_alias",tag="method"}
Add an alias or mention to the knowledge base, specifying its potential KB
identifiers and their prior probabilities. The entity identifiers should refer
@ -96,7 +96,7 @@ alias.
| `entities` | The potential entities that the alias may refer to. ~~Iterable[Union[str, int]]~~ |
| `probabilities` | The prior probabilities of each entity. ~~Iterable[float]~~ |
## InMemoryLookupKB.\_\_len\_\_ {#len tag="method"}
## InMemoryLookupKB.\_\_len\_\_ {id="len",tag="method"}
Get the total number of entities in the knowledge base.
@ -110,7 +110,7 @@ Get the total number of entities in the knowledge base.
| ----------- | ----------------------------------------------------- |
| **RETURNS** | The number of entities in the knowledge base. ~~int~~ |
## InMemoryLookupKB.get_entity_strings {#get_entity_strings tag="method"}
## InMemoryLookupKB.get_entity_strings {id="get_entity_strings",tag="method"}
Get a list of all entity IDs in the knowledge base.
@ -124,7 +124,7 @@ Get a list of all entity IDs in the knowledge base.
| ----------- | --------------------------------------------------------- |
| **RETURNS** | The list of entities in the knowledge base. ~~List[str]~~ |
## InMemoryLookupKB.get_size_aliases {#get_size_aliases tag="method"}
## InMemoryLookupKB.get_size_aliases {id="get_size_aliases",tag="method"}
Get the total number of aliases in the knowledge base.
@ -138,7 +138,7 @@ Get the total number of aliases in the knowledge base.
| ----------- | ---------------------------------------------------- |
| **RETURNS** | The number of aliases in the knowledge base. ~~int~~ |
## InMemoryLookupKB.get_alias_strings {#get_alias_strings tag="method"}
## InMemoryLookupKB.get_alias_strings {id="get_alias_strings",tag="method"}
Get a list of all aliases in the knowledge base.
@ -152,7 +152,7 @@ Get a list of all aliases in the knowledge base.
| ----------- | -------------------------------------------------------- |
| **RETURNS** | The list of aliases in the knowledge base. ~~List[str]~~ |
## InMemoryLookupKB.get_candidates {#get_candidates tag="method"}
## InMemoryLookupKB.get_candidates {id="get_candidates",tag="method"}
Given a certain textual mention as input, retrieve a list of candidate entities
of type [`Candidate`](/api/kb#candidate). Wraps
@ -172,7 +172,7 @@ of type [`Candidate`](/api/kb#candidate). Wraps
| `mention` | The textual mention or alias. ~~Span~~ |
| **RETURNS** | An iterable of relevant `Candidate` objects. ~~Iterable[Candidate]~~ |
## InMemoryLookupKB.get_candidates_batch {#get_candidates_batch tag="method"}
## InMemoryLookupKB.get_candidates_batch {id="get_candidates_batch",tag="method"}
Same as [`get_candidates()`](/api/kb_in_memory#get_candidates), but for an
arbitrary number of mentions. The [`EntityLinker`](/api/entitylinker) component
@ -198,7 +198,7 @@ to you.
| `mentions` | The textual mention or alias. ~~Iterable[Span]~~ |
| **RETURNS** | An iterable of iterable with relevant `Candidate` objects. ~~Iterable[Iterable[Candidate]]~~ |
## InMemoryLookupKB.get_alias_candidates {#get_alias_candidates tag="method"}
## InMemoryLookupKB.get_alias_candidates {id="get_alias_candidates",tag="method"}
Given a certain textual mention as input, retrieve a list of candidate entities
of type [`Candidate`](/api/kb#candidate).
@ -214,7 +214,7 @@ of type [`Candidate`](/api/kb#candidate).
| `alias` | The textual mention or alias. ~~str~~ |
| **RETURNS** | The list of relevant `Candidate` objects. ~~List[Candidate]~~ |
## InMemoryLookupKB.get_vector {#get_vector tag="method"}
## InMemoryLookupKB.get_vector {id="get_vector",tag="method"}
Given a certain entity ID, retrieve its pretrained entity vector.
@ -229,7 +229,7 @@ Given a certain entity ID, retrieve its pretrained entity vector.
| `entity` | The entity ID. ~~str~~ |
| **RETURNS** | The entity vector. ~~numpy.ndarray~~ |
## InMemoryLookupKB.get_vectors {#get_vectors tag="method"}
## InMemoryLookupKB.get_vectors {id="get_vectors",tag="method"}
Same as [`get_vector()`](/api/kb_in_memory#get_vector), but for an arbitrary
number of entity IDs.
@ -249,7 +249,7 @@ entities at once, if performance is of concern to you.
| `entities` | The entity IDs. ~~Iterable[str]~~ |
| **RETURNS** | The entity vectors. ~~Iterable[Iterable[numpy.ndarray]]~~ |
## InMemoryLookupKB.get_prior_prob {#get_prior_prob tag="method"}
## InMemoryLookupKB.get_prior_prob {id="get_prior_prob",tag="method"}
Given a certain entity ID and a certain textual mention, retrieve the prior
probability of the fact that the mention links to the entity ID.
@ -266,7 +266,7 @@ probability of the fact that the mention links to the entity ID.
| `alias` | The textual mention or alias. ~~str~~ |
| **RETURNS** | The prior probability of the `alias` referring to the `entity`. ~~float~~ |
## InMemoryLookupKB.to_disk {#to_disk tag="method"}
## InMemoryLookupKB.to_disk {id="to_disk",tag="method"}
Save the current state of the knowledge base to a directory.
@ -281,7 +281,7 @@ Save the current state of the knowledge base to a directory.
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| `exclude` | List of components to exclude. ~~Iterable[str]~~ |
## InMemoryLookupKB.from_disk {#from_disk tag="method"}
## InMemoryLookupKB.from_disk {id="from_disk",tag="method"}
Restore the state of the knowledge base from a given directory. Note that the
[`Vocab`](/api/vocab) should also be the same as the one used to create the KB.

88
website/docs/api/language.md → website/docs/api/language.mdx
View File

@ -15,7 +15,7 @@ the tagger or parser that are called on a document in order. You can also add
your own processing pipeline components that take a `Doc` object, modify it and
return it.
## Language.\_\_init\_\_ {#init tag="method"}
## Language.\_\_init\_\_ {id="init",tag="method"}
Initialize a `Language` object. Note that the `meta` is only used for meta
information in [`Language.meta`](/api/language#meta) and not to configure the
@ -44,7 +44,7 @@ information in [`Language.meta`](/api/language#meta) and not to configure the
| `create_tokenizer` | Optional function that receives the `nlp` object and returns a tokenizer. ~~Callable[[Language], Callable[[str], Doc]]~~ |
| `batch_size` | Default batch size for [`pipe`](#pipe) and [`evaluate`](#evaluate). Defaults to `1000`. ~~int~~ |
## Language.from_config {#from_config tag="classmethod" new="3"}
## Language.from_config {id="from_config",tag="classmethod",version="3"}
Create a `Language` object from a loaded config. Will set up the tokenizer and
language data, add pipeline components based on the pipeline and add pipeline
@ -76,7 +76,7 @@ spaCy loads a model under the hood based on its
| `validate` | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The initialized object. ~~Language~~ |
## Language.component {#component tag="classmethod" new="3"}
## Language.component {id="component",tag="classmethod",version="3"}
Register a custom pipeline component under a given name. This allows
initializing the component by name using
@ -112,7 +112,7 @@ decorator. For more details and examples, see the
| `retokenizes` | Whether the component changes tokenization. Used for [pipe analysis](/usage/processing-pipelines#analysis). ~~bool~~ |
| `func` | Optional function if not used as a decorator. ~~Optional[Callable[[Doc], Doc]]~~ |
## Language.factory {#factory tag="classmethod"}
## Language.factory {id="factory",tag="classmethod"}
Register a custom pipeline component factory under a given name. This allows
initializing the component by name using
@ -159,7 +159,7 @@ examples, see the
| `default_score_weights` | The scores to report during training, and their default weight towards the final score used to select the best model. Weights should sum to `1.0` per component and will be combined and normalized for the whole pipeline. If a weight is set to `None`, the score will not be logged or weighted. ~~Dict[str, Optional[float]]~~ |
| `func` | Optional function if not used as a decorator. ~~Optional[Callable[[...], Callable[[Doc], Doc]]]~~ |
## Language.\_\_call\_\_ {#call tag="method"}
## Language.\_\_call\_\_ {id="call",tag="method"}
Apply the pipeline to some text. The text can span multiple sentences, and can
contain arbitrary whitespace. Alignment into the original string is preserved.
@ -182,7 +182,7 @@ skipped, but the rest of the pipeline is run.
| `component_cfg` | Optional dictionary of keyword arguments for components, keyed by component names. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Any]]]~~ |
| **RETURNS** | A container for accessing the annotations. ~~Doc~~ |
## Language.pipe {#pipe tag="method"}
## Language.pipe {id="pipe",tag="method"}
Process texts as a stream, and yield `Doc` objects in order. This is usually
more efficient than processing texts one-by-one.
@ -209,7 +209,7 @@ tokenization is skipped but the rest of the pipeline is run.
| `n_process` | Number of processors to use. Defaults to `1`. ~~int~~ |
| **YIELDS** | Documents in the order of the original text. ~~Doc~~ |
## Language.set_error_handler {#set_error_handler tag="method" new="3"}
## Language.set_error_handler {id="set_error_handler",tag="method",version="3"}
Define a callback that will be invoked when an error is thrown during processing
of one or more documents. Specifically, this function will call
@ -231,7 +231,7 @@ being processed, and the original error.
| --------------- | -------------------------------------------------------------------------------------------------------------- |
| `error_handler` | A function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ |
## Language.initialize {#initialize tag="method" new="3"}
## Language.initialize {id="initialize",tag="method",version="3"}
Initialize the pipeline for training and return an
[`Optimizer`](https://thinc.ai/docs/api-optimizers). Under the hood, it uses the
@ -282,7 +282,7 @@ objects.
| `sgd` | An optimizer. Will be created via [`create_optimizer`](#create_optimizer) if not set. ~~Optional[Optimizer]~~ |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## Language.resume_training {#resume_training tag="method,experimental" new="3"}
## Language.resume_training {id="resume_training",tag="method,experimental",version="3"}
Continue training a trained pipeline. Create and return an optimizer, and
initialize "rehearsal" for any pipeline component that has a `rehearse` method.
@ -304,7 +304,7 @@ a batch of [Example](/api/example) objects.
| `sgd` | An optimizer. Will be created via [`create_optimizer`](#create_optimizer) if not set. ~~Optional[Optimizer]~~ |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## Language.update {#update tag="method"}
## Language.update {id="update",tag="method"}
Update the models in the pipeline.
@ -342,7 +342,7 @@ and custom registered functions if needed. See the
| `component_cfg` | Optional dictionary of keyword arguments for components, keyed by component names. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Any]]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Language.rehearse {#rehearse tag="method,experimental" new="3"}
## Language.rehearse {id="rehearse",tag="method,experimental",version="3"}
Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the
current model to make predictions similar to an initial model, to try to address
@ -364,7 +364,7 @@ the "catastrophic forgetting" problem. This feature is experimental.
| `losses` | Dictionary to update with the loss, keyed by pipeline component. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Language.evaluate {#evaluate tag="method"}
## Language.evaluate {id="evaluate",tag="method"}
Evaluate a pipeline's components.
@ -392,7 +392,7 @@ objects instead of tuples of `Doc` and `GoldParse` objects.
| `scorer_cfg` | Optional dictionary of keyword arguments for the `Scorer`. Defaults to `None`. ~~Optional[Dict[str, Any]]~~ |
| **RETURNS** | A dictionary of evaluation scores. ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## Language.use_params {#use_params tag="contextmanager, method"}
## Language.use_params {id="use_params",tag="contextmanager, method"}
Replace weights of models in the pipeline with those provided in the params
dictionary. Can be used as a context manager, in which case, models go back to
@ -409,7 +409,7 @@ their original weights after the block.
| -------- | ------------------------------------------------------ |
| `params` | A dictionary of parameters keyed by model ID. ~~dict~~ |
## Language.add_pipe {#add_pipe tag="method" new="2"}
## Language.add_pipe {id="add_pipe",tag="method",version="2"}
Add a component to the processing pipeline. Expects a name that maps to a
component factory registered using
@ -458,7 +458,7 @@ component, adds it to the pipeline and returns it.
| `validate` <Tag variant="new">3</Tag> | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ |
## Language.create_pipe {#create_pipe tag="method" new="2"}
## Language.create_pipe {id="create_pipe",tag="method",version="2"}
Create a pipeline component from a factory.
@ -487,7 +487,7 @@ To create a component and add it to the pipeline, you should always use
| `validate` <Tag variant="new">3</Tag> | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ |
## Language.has_factory {#has_factory tag="classmethod" new="3"}
## Language.has_factory {id="has_factory",tag="classmethod",version="3"}
Check whether a factory name is registered on the `Language` class or subclass.
Will check for
@ -514,7 +514,7 @@ the `Language` base class, available to all subclasses.
| `name` | Name of the pipeline factory to check. ~~str~~ |
| **RETURNS** | Whether a factory of that name is registered on the class. ~~bool~~ |
## Language.has_pipe {#has_pipe tag="method" new="2"}
## Language.has_pipe {id="has_pipe",tag="method",version="2"}
Check whether a component is present in the pipeline. Equivalent to
`name in nlp.pipe_names`.
@ -536,7 +536,7 @@ Check whether a component is present in the pipeline. Equivalent to
| `name` | Name of the pipeline component to check. ~~str~~ |
| **RETURNS** | Whether a component of that name exists in the pipeline. ~~bool~~ |
## Language.get_pipe {#get_pipe tag="method" new="2"}
## Language.get_pipe {id="get_pipe",tag="method",version="2"}
Get a pipeline component for a given component name.
@ -552,7 +552,7 @@ Get a pipeline component for a given component name.
| `name` | Name of the pipeline component to get. ~~str~~ |
| **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ |
## Language.replace_pipe {#replace_pipe tag="method" new="2"}
## Language.replace_pipe {id="replace_pipe",tag="method",version="2"}
Replace a component in the pipeline and return the new component.
@ -580,7 +580,7 @@ and instead expects the **name of a component factory** registered using
| `validate` <Tag variant="new">3</Tag> | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The new pipeline component. ~~Callable[[Doc], Doc]~~ |
## Language.rename_pipe {#rename_pipe tag="method" new="2"}
## Language.rename_pipe {id="rename_pipe",tag="method",version="2"}
Rename a component in the pipeline. Useful to create custom names for
pre-defined and pre-loaded components. To change the default name of a component
@ -598,7 +598,7 @@ added to the pipeline, you can also use the `name` argument on
| `old_name` | Name of the component to rename. ~~str~~ |
| `new_name` | New name of the component. ~~str~~ |
## Language.remove_pipe {#remove_pipe tag="method" new="2"}
## Language.remove_pipe {id="remove_pipe",tag="method",version="2"}
Remove a component from the pipeline. Returns the removed component name and
component function.
@ -615,7 +615,7 @@ component function.
| `name` | Name of the component to remove. ~~str~~ |
| **RETURNS** | A `(name, component)` tuple of the removed component. ~~Tuple[str, Callable[[Doc], Doc]]~~ |
## Language.disable_pipe {#disable_pipe tag="method" new="3"}
## Language.disable_pipe {id="disable_pipe",tag="method",version="3"}
Temporarily disable a pipeline component so it's not run as part of the
pipeline. Disabled components are listed in
@ -641,7 +641,7 @@ does nothing.
| ------ | ----------------------------------------- |
| `name` | Name of the component to disable. ~~str~~ |
## Language.enable_pipe {#enable_pipe tag="method" new="3"}
## Language.enable_pipe {id="enable_pipe",tag="method",version="3"}
Enable a previously disabled component (e.g. via
[`Language.disable_pipes`](/api/language#disable_pipes)) so it's run as part of
@ -663,7 +663,7 @@ already enabled, this method does nothing.
| ------ | ---------------------------------------- |
| `name` | Name of the component to enable. ~~str~~ |
## Language.select_pipes {#select_pipes tag="contextmanager, method" new="3"}
## Language.select_pipes {id="select_pipes",tag="contextmanager, method",version="3"}
Disable one or more pipeline components. If used as a context manager, the
pipeline will be restored to the initial state at the end of the block.
@ -706,7 +706,7 @@ As of spaCy v3.0, the `disable_pipes` method has been renamed to `select_pipes`:
| `enable` | Name(s) of pipeline component(s) that will not be disabled. ~~Optional[Union[str, Iterable[str]]]~~ |
| **RETURNS** | The disabled pipes that can be restored by calling the object's `.restore()` method. ~~DisabledPipes~~ |
## Language.get_factory_meta {#get_factory_meta tag="classmethod" new="3"}
## Language.get_factory_meta {id="get_factory_meta",tag="classmethod",version="3"}
Get the factory meta information for a given pipeline component name. Expects
the name of the component **factory**. The factory meta is an instance of the
@ -728,7 +728,7 @@ information about the component and its default provided by the
| `name` | The factory name. ~~str~~ |
| **RETURNS** | The factory meta. ~~FactoryMeta~~ |
## Language.get_pipe_meta {#get_pipe_meta tag="method" new="3"}
## Language.get_pipe_meta {id="get_pipe_meta",tag="method",version="3"}
Get the factory meta information for a given pipeline component name. Expects
the name of the component **instance** in the pipeline. The factory meta is an
@ -751,7 +751,7 @@ contains the information about the component and its default provided by the
| `name` | The pipeline component name. ~~str~~ |
| **RETURNS** | The factory meta. ~~FactoryMeta~~ |
## Language.analyze_pipes {#analyze_pipes tag="method" new="3"}
## Language.analyze_pipes {id="analyze_pipes",tag="method",version="3"}
Analyze the current pipeline components and show a summary of the attributes
they assign and require, and the scores they set. The data is based on the
@ -780,8 +780,7 @@ doesn't, the pipeline analysis won't catch that.
<Accordion title="Example output" spaced>
```json
### Structured
```json {title="Structured"}
{
"summary": {
"tagger": {
@ -799,7 +798,12 @@ doesn't, the pipeline analysis won't catch that.
},
"problems": {
"tagger": [],
"entity_linker": ["doc.ents", "doc.sents", "token.ent_iob", "token.ent_type"]
"entity_linker": [
"doc.ents",
"doc.sents",
"token.ent_iob",
"token.ent_type"
]
},
"attrs": {
"token.ent_iob": { "assigns": [], "requires": ["entity_linker"] },
@ -840,7 +844,7 @@ token.ent_iob, token.ent_type
| `pretty` | Pretty-print the results as a table. Defaults to `False`. ~~bool~~ |
| **RETURNS** | Dictionary containing the pipe analysis, keyed by `"summary"` (component meta by pipe), `"problems"` (attribute names by pipe) and `"attrs"` (pipes that assign and require an attribute, keyed by attribute). ~~Optional[Dict[str, Any]]~~ |
## Language.replace_listeners {#replace_listeners tag="method" new="3"}
## Language.replace_listeners {id="replace_listeners",tag="method",version="3"}
Find [listener layers](/usage/embeddings-transformers#embedding-layers)
(connecting to a shared token-to-vector embedding component) of a given pipeline
@ -885,7 +889,7 @@ when loading a config with
| `pipe_name` | Name of pipeline component to replace listeners for. ~~str~~ |
| `listeners` | The paths to the listeners, relative to the component config, e.g. `["model.tok2vec"]`. Typically, implementations will only connect to one tok2vec component, `model.tok2vec`, but in theory, custom models can use multiple listeners. The value here can either be an empty list to not replace any listeners, or a _complete_ list of the paths to all listener layers used by the model that should be replaced.~~Iterable[str]~~ |
## Language.meta {#meta tag="property"}
## Language.meta {id="meta",tag="property"}
Meta data for the `Language` class, including name, version, data sources,
license, author information and more. If a trained pipeline is loaded, this
@ -911,7 +915,7 @@ information is expressed in the [`config.cfg`](/api/data-formats#config).
| ----------- | --------------------------------- |
| **RETURNS** | The meta data. ~~Dict[str, Any]~~ |
## Language.config {#config tag="property" new="3"}
## Language.config {id="config",tag="property",version="3"}
Export a trainable [`config.cfg`](/api/data-formats#config) for the current
`nlp` object. Includes the current pipeline, all configs used to create the
@ -932,7 +936,7 @@ subclass of the built-in `dict`. It supports the additional methods `to_disk`
| ----------- | ---------------------- |
| **RETURNS** | The config. ~~Config~~ |
## Language.to_disk {#to_disk tag="method" new="2"}
## Language.to_disk {id="to_disk",tag="method",version="2"}
Save the current state to a directory. Under the hood, this method delegates to
the `to_disk` methods of the individual pipeline components, if available. This
@ -951,7 +955,7 @@ will be saved to disk.
| _keyword-only_ | |
| `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Language.from_disk {#from_disk tag="method" new="2"}
## Language.from_disk {id="from_disk",tag="method",version="2"}
Loads state from a directory, including all data that was saved with the
`Language` object. Modifies the object in place and returns it.
@ -984,7 +988,7 @@ you want to load a serialized pipeline from a directory, you should use
| `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Language` object. ~~Language~~ |
## Language.to_bytes {#to_bytes tag="method"}
## Language.to_bytes {id="to_bytes",tag="method"}
Serialize the current state to a binary string.
@ -1000,7 +1004,7 @@ Serialize the current state to a binary string.
| `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~iterable~~ |
| **RETURNS** | The serialized form of the `Language` object. ~~bytes~~ |
## Language.from_bytes {#from_bytes tag="method"}
## Language.from_bytes {id="from_bytes",tag="method"}
Load state from a binary string. Note that this method is commonly used via the
subclasses like `English` or `German` to make language-specific functionality
@ -1028,7 +1032,7 @@ details.
| `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Language` object. ~~Language~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
@ -1046,7 +1050,7 @@ details.
| `disabled` <Tag variant="new">3</Tag> | Names of components that are currently disabled and don't run as part of the pipeline. ~~List[str]~~ |
| `path` | Path to the pipeline data directory, if a pipeline is loaded from a path or package. Otherwise `None`. ~~Optional[Path]~~ |
## Class attributes {#class-attributes}
## Class attributes {id="class-attributes"}
| Name | Description |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@ -1054,7 +1058,7 @@ details.
| `lang` | [IETF language tag](https://www.w3.org/International/articles/language-tags/), such as 'en' for English. ~~str~~ |
| `default_config` | Base [config](/usage/training#config) to use for [Language.config](/api/language#config). Defaults to [`default_config.cfg`](%%GITHUB_SPACY/spacy/default_config.cfg). ~~Config~~ |
## Defaults {#defaults}
## Defaults {id="defaults"}
The following attributes can be set on the `Language.Defaults` class to
customize the default language data:
@ -1097,7 +1101,7 @@ customize the default language data:
| `writing_system` | Information about the language's writing system, available via `Vocab.writing_system`. Defaults to: `{"direction": "ltr", "has_case": True, "has_letters": True}.`.<br />**Example:** [`zh/__init__.py`](%%GITHUB_SPACY/spacy/lang/zh/__init__.py) ~~Dict[str, Any]~~ |
| `config` | Default [config](/usage/training#config) added to `nlp.config`. This can include references to custom tokenizers or lemmatizers.<br />**Example:** [`zh/__init__.py`](%%GITHUB_SPACY/spacy/lang/zh/__init__.py) ~~Config~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from
@ -1117,7 +1121,7 @@ serialization by passing in the string names via the `exclude` argument.
| `meta` | The meta data, available as [`Language.meta`](/api/language#meta). |
| ... | String names of pipeline components, e.g. `"ner"`. |
## FactoryMeta {#factorymeta new="3" tag="dataclass"}
## FactoryMeta {id="factorymeta",version="3",tag="dataclass"}
The `FactoryMeta` contains the information about the component and its default
provided by the [`@Language.component`](/api/language#component) or

32
website/docs/api/legacy.md → website/docs/api/legacy.mdx
View File

@ -12,11 +12,11 @@ functions that may still be used in projects.
You can find the detailed documentation of each such legacy function on this
page.
## Architectures {#architectures}
## Architectures {id="architectures"}
These functions are available from `@spacy.registry.architectures`.
### spacy.Tok2Vec.v1 {#Tok2Vec_v1}
### spacy.Tok2Vec.v1 {id="Tok2Vec_v1"}
The `spacy.Tok2Vec.v1` architecture was expecting an `encode` model of type
`Model[Floats2D, Floats2D]` such as `spacy.MaxoutWindowEncoder.v1` or
@ -48,7 +48,7 @@ blog post for background.
| `encode` | Encode context into the embeddings, using an architecture such as a CNN, BiLSTM or transformer. For example, [MaxoutWindowEncoder.v1](/api/legacy#MaxoutWindowEncoder_v1). ~~Model[Floats2d, Floats2d]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ |
### spacy.MaxoutWindowEncoder.v1 {#MaxoutWindowEncoder_v1}
### spacy.MaxoutWindowEncoder.v1 {id="MaxoutWindowEncoder_v1"}
The `spacy.MaxoutWindowEncoder.v1` architecture was producing a model of type
`Model[Floats2D, Floats2D]`. Since `spacy.MaxoutWindowEncoder.v2`, this has been
@ -76,7 +76,7 @@ and residual connections.
| `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[Floats2d, Floats2d]~~ |
### spacy.MishWindowEncoder.v1 {#MishWindowEncoder_v1}
### spacy.MishWindowEncoder.v1 {id="MishWindowEncoder_v1"}
The `spacy.MishWindowEncoder.v1` architecture was producing a model of type
`Model[Floats2D, Floats2D]`. Since `spacy.MishWindowEncoder.v2`, this has been
@ -103,24 +103,24 @@ and residual connections.
| `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ |
| **CREATES** | The model using the architecture. ~~Model[Floats2d, Floats2d]~~ |
### spacy.HashEmbedCNN.v1 {#HashEmbedCNN_v1}
### spacy.HashEmbedCNN.v1 {id="HashEmbedCNN_v1"}
Identical to [`spacy.HashEmbedCNN.v2`](/api/architectures#HashEmbedCNN) except
using [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are included.
### spacy.MultiHashEmbed.v1 {#MultiHashEmbed_v1}
### spacy.MultiHashEmbed.v1 {id="MultiHashEmbed_v1"}
Identical to [`spacy.MultiHashEmbed.v2`](/api/architectures#MultiHashEmbed)
except with [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are
included.
### spacy.CharacterEmbed.v1 {#CharacterEmbed_v1}
### spacy.CharacterEmbed.v1 {id="CharacterEmbed_v1"}
Identical to [`spacy.CharacterEmbed.v2`](/api/architectures#CharacterEmbed)
except using [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are
included.
### spacy.TextCatEnsemble.v1 {#TextCatEnsemble_v1}
### spacy.TextCatEnsemble.v1 {id="TextCatEnsemble_v1"}
The `spacy.TextCatEnsemble.v1` architecture built an internal `tok2vec` and
`linear_model`. Since `spacy.TextCatEnsemble.v2`, this has been refactored so
@ -158,7 +158,7 @@ network has an internal CNN Tok2Vec layer and uses attention.
| `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
### spacy.TextCatCNN.v1 {#TextCatCNN_v1}
### spacy.TextCatCNN.v1 {id="TextCatCNN_v1"}
Since `spacy.TextCatCNN.v2`, this architecture has become resizable, which means
that you can add labels to a previously trained textcat. `TextCatCNN` v1 did not
@ -194,7 +194,7 @@ architecture is usually less accurate than the ensemble, but runs faster.
| `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
### spacy.TextCatBOW.v1 {#TextCatBOW_v1}
### spacy.TextCatBOW.v1 {id="TextCatBOW_v1"}
Since `spacy.TextCatBOW.v2`, this architecture has become resizable, which means
that you can add labels to a previously trained textcat. `TextCatBOW` v1 did not
@ -222,17 +222,17 @@ the others, but may not be as accurate, especially if texts are short.
| `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ |
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
### spacy.TransitionBasedParser.v1 {#TransitionBasedParser_v1}
### spacy.TransitionBasedParser.v1 {id="TransitionBasedParser_v1"}
Identical to
[`spacy.TransitionBasedParser.v2`](/api/architectures#TransitionBasedParser)
except the `use_upper` was set to `True` by default.
## Layers {#layers}
## Layers {id="layers"}
These functions are available from `@spacy.registry.layers`.
### spacy.StaticVectors.v1 {#StaticVectors_v1}
### spacy.StaticVectors.v1 {id="StaticVectors_v1"}
Identical to [`spacy.StaticVectors.v2`](/api/architectures#StaticVectors) except
for the handling of tokens without vectors.
@ -246,11 +246,11 @@ added to an existing vectors table. See more details in
</Infobox>
## Loggers {#loggers}
## Loggers {id="loggers"}
These functions are available from `@spacy.registry.loggers`.
### spacy.ConsoleLogger.v1 {#ConsoleLogger_v1}
### spacy.ConsoleLogger.v1 {id="ConsoleLogger_v1"}
> #### Example config
>
@ -264,7 +264,7 @@ Writes the results of a training step to the console in a tabular format.
<Accordion title="Example console output" spaced>
```cli
```bash
$ python -m spacy train config.cfg
```

32
website/docs/api/lemmatizer.md → website/docs/api/lemmatizer.mdx
View File

@ -2,7 +2,7 @@
title: Lemmatizer
tag: class
source: spacy/pipeline/lemmatizer.py
new: 3
version: 3
teaser: 'Pipeline component for lemmatization'
api_string_name: lemmatizer
api_trainable: false
@ -32,7 +32,7 @@ available in the pipeline and runs _before_ the lemmatizer.
</Infobox>
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Lemmas generated by rules or predicted will be saved to `Token.lemma`.
@ -94,7 +94,7 @@ libraries (`pymorphy3`).
%%GITHUB_SPACY/spacy/pipeline/lemmatizer.py
```
## Lemmatizer.\_\_init\_\_ {#init tag="method"}
## Lemmatizer.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -120,7 +120,7 @@ shortcut for this and instantiate the component using its string name and
| mode | The lemmatizer mode, e.g. `"lookup"` or `"rule"`. Defaults to `"lookup"`. ~~str~~ |
| overwrite | Whether to overwrite existing lemmas. ~~bool~~ |
## Lemmatizer.\_\_call\_\_ {#call tag="method"}
## Lemmatizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -140,7 +140,7 @@ and all pipeline components are applied to the `Doc` in order.
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## Lemmatizer.pipe {#pipe tag="method"}
## Lemmatizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -161,7 +161,7 @@ applied to the `Doc` in order.
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## Lemmatizer.initialize {#initialize tag="method"}
## Lemmatizer.initialize {id="initialize",tag="method"}
Initialize the lemmatizer and load any data resources. This method is typically
called by [`Language.initialize`](/api/language#initialize) and lets you
@ -192,7 +192,7 @@ training. At runtime, all data is loaded from disk.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `lookups` | The lookups object containing the tables such as `"lemma_rules"`, `"lemma_index"`, `"lemma_exc"` and `"lemma_lookup"`. If `None`, default tables are loaded from [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data). Defaults to `None`. ~~Optional[Lookups]~~ |
## Lemmatizer.lookup_lemmatize {#lookup_lemmatize tag="method"}
## Lemmatizer.lookup_lemmatize {id="lookup_lemmatize",tag="method"}
Lemmatize a token using a lookup-based approach. If no lemma is found, the
original string is returned.
@ -202,7 +202,7 @@ original string is returned.
| `token` | The token to lemmatize. ~~Token~~ |
| **RETURNS** | A list containing one or more lemmas. ~~List[str]~~ |
## Lemmatizer.rule_lemmatize {#rule_lemmatize tag="method"}
## Lemmatizer.rule_lemmatize {id="rule_lemmatize",tag="method"}
Lemmatize a token using a rule-based approach. Typically relies on POS tags.
@ -211,7 +211,7 @@ Lemmatize a token using a rule-based approach. Typically relies on POS tags.
| `token` | The token to lemmatize. ~~Token~~ |
| **RETURNS** | A list containing one or more lemmas. ~~List[str]~~ |
## Lemmatizer.is_base_form {#is_base_form tag="method"}
## Lemmatizer.is_base_form {id="is_base_form",tag="method"}
Check whether we're dealing with an uninflected paradigm, so we can avoid
lemmatization entirely.
@ -221,7 +221,7 @@ lemmatization entirely.
| `token` | The token to analyze. ~~Token~~ |
| **RETURNS** | Whether the token's attributes (e.g., part-of-speech tag, morphological features) describe a base form. ~~bool~~ |
## Lemmatizer.get_lookups_config {#get_lookups_config tag="classmethod"}
## Lemmatizer.get_lookups_config {id="get_lookups_config",tag="classmethod"}
Returns the lookups configuration settings for a given mode for use in
[`Lemmatizer.load_lookups`](/api/lemmatizer#load_lookups).
@ -231,7 +231,7 @@ Returns the lookups configuration settings for a given mode for use in
| `mode` | The lemmatizer mode. ~~str~~ |
| **RETURNS** | The required table names and the optional table names. ~~Tuple[List[str], List[str]]~~ |
## Lemmatizer.to_disk {#to_disk tag="method"}
## Lemmatizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -248,7 +248,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Lemmatizer.from_disk {#from_disk tag="method"}
## Lemmatizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -266,7 +266,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Lemmatizer` object. ~~Lemmatizer~~ |
## Lemmatizer.to_bytes {#to_bytes tag="method"}
## Lemmatizer.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -283,7 +283,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Lemmatizer` object. ~~bytes~~ |
## Lemmatizer.from_bytes {#from_bytes tag="method"}
## Lemmatizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -302,7 +302,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Lemmatizer` object. ~~Lemmatizer~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| --------- | ------------------------------------------- |
@ -310,7 +310,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `lookups` | The lookups object. ~~Lookups~~ |
| `mode` | The lemmatizer mode. ~~str~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

16
website/docs/api/lexeme.md → website/docs/api/lexeme.mdx
View File

@ -9,7 +9,7 @@ A `Lexeme` has no string context – it's a word type, as opposed to a word toke
It therefore has no part-of-speech tag, dependency parse, or lemma (if
lemmatization depends on the part-of-speech tag).
## Lexeme.\_\_init\_\_ {#init tag="method"}
## Lexeme.\_\_init\_\_ {id="init",tag="method"}
Create a `Lexeme` object.
@ -18,7 +18,7 @@ Create a `Lexeme` object.
| `vocab` | The parent vocabulary. ~~Vocab~~ |
| `orth` | The orth id of the lexeme. ~~int~~ |
## Lexeme.set_flag {#set_flag tag="method"}
## Lexeme.set_flag {id="set_flag",tag="method"}
Change the value of a boolean flag.
@ -34,7 +34,7 @@ Change the value of a boolean flag.
| `flag_id` | The attribute ID of the flag to set. ~~int~~ |
| `value` | The new value of the flag. ~~bool~~ |
## Lexeme.check_flag {#check_flag tag="method"}
## Lexeme.check_flag {id="check_flag",tag="method"}
Check the value of a boolean flag.
@ -51,7 +51,7 @@ Check the value of a boolean flag.
| `flag_id` | The attribute ID of the flag to query. ~~int~~ |
| **RETURNS** | The value of the flag. ~~bool~~ |
## Lexeme.similarity {#similarity tag="method" model="vectors"}
## Lexeme.similarity {id="similarity",tag="method",model="vectors"}
Compute a semantic similarity estimate. Defaults to cosine over vectors.
@ -70,7 +70,7 @@ Compute a semantic similarity estimate. Defaults to cosine over vectors.
| other | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ |
| **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ |
## Lexeme.has_vector {#has_vector tag="property" model="vectors"}
## Lexeme.has_vector {id="has_vector",tag="property",model="vectors"}
A boolean value indicating whether a word vector is associated with the lexeme.
@ -85,7 +85,7 @@ A boolean value indicating whether a word vector is associated with the lexeme.
| ----------- | ------------------------------------------------------- |
| **RETURNS** | Whether the lexeme has a vector data attached. ~~bool~~ |
## Lexeme.vector {#vector tag="property" model="vectors"}
## Lexeme.vector {id="vector",tag="property",model="vectors"}
A real-valued meaning representation.
@ -101,7 +101,7 @@ A real-valued meaning representation.
| ----------- | ------------------------------------------------------------------------------------------------ |
| **RETURNS** | A 1-dimensional array representing the lexeme's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Lexeme.vector_norm {#vector_norm tag="property" model="vectors"}
## Lexeme.vector_norm {id="vector_norm",tag="property",model="vectors"}
The L2 norm of the lexeme's vector representation.
@ -119,7 +119,7 @@ The L2 norm of the lexeme's vector representation.
| ----------- | --------------------------------------------------- |
| **RETURNS** | The L2 norm of the vector representation. ~~float~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

40
website/docs/api/lookups.md → website/docs/api/lookups.mdx
View File

@ -3,7 +3,7 @@ title: Lookups
teaser: A container for large lookup tables and dictionaries
tag: class
source: spacy/lookups.py
new: 2.2
version: 2.2
---
This class allows convenient access to large lookup tables and dictionaries,
@ -13,7 +13,7 @@ can be accessed before the pipeline components are applied (e.g. in the
tokenizer and lemmatizer), as well as within the pipeline components via
`doc.vocab.lookups`.
## Lookups.\_\_init\_\_ {#init tag="method"}
## Lookups.\_\_init\_\_ {id="init",tag="method"}
Create a `Lookups` object.
@ -24,7 +24,7 @@ Create a `Lookups` object.
> lookups = Lookups()
> ```
## Lookups.\_\_len\_\_ {#len tag="method"}
## Lookups.\_\_len\_\_ {id="len",tag="method"}
Get the current number of tables in the lookups.
@ -39,7 +39,7 @@ Get the current number of tables in the lookups.
| ----------- | -------------------------------------------- |
| **RETURNS** | The number of tables in the lookups. ~~int~~ |
## Lookups.\_\contains\_\_ {#contains tag="method"}
## Lookups.\_\_contains\_\_ {id="contains",tag="method"}
Check if the lookups contain a table of a given name. Delegates to
[`Lookups.has_table`](/api/lookups#has_table).
@ -57,7 +57,7 @@ Check if the lookups contain a table of a given name. Delegates to
| `name` | Name of the table. ~~str~~ |
| **RETURNS** | Whether a table of that name is in the lookups. ~~bool~~ |
## Lookups.tables {#tables tag="property"}
## Lookups.tables {id="tables",tag="property"}
Get the names of all tables in the lookups.
@ -73,7 +73,7 @@ Get the names of all tables in the lookups.
| ----------- | ------------------------------------------------- |
| **RETURNS** | Names of the tables in the lookups. ~~List[str]~~ |
## Lookups.add_table {#add_table tag="method"}
## Lookups.add_table {id="add_table",tag="method"}
Add a new table with optional data to the lookups. Raises an error if the table
exists.
@ -91,7 +91,7 @@ exists.
| `data` | Optional data to add to the table. ~~dict~~ |
| **RETURNS** | The newly added table. ~~Table~~ |
## Lookups.get_table {#get_table tag="method"}
## Lookups.get_table {id="get_table",tag="method"}
Get a table from the lookups. Raises an error if the table doesn't exist.
@ -109,7 +109,7 @@ Get a table from the lookups. Raises an error if the table doesn't exist.
| `name` | Name of the table. ~~str~~ |
| **RETURNS** | The table. ~~Table~~ |
## Lookups.remove_table {#remove_table tag="method"}
## Lookups.remove_table {id="remove_table",tag="method"}
Remove a table from the lookups. Raises an error if the table doesn't exist.
@ -127,7 +127,7 @@ Remove a table from the lookups. Raises an error if the table doesn't exist.
| `name` | Name of the table to remove. ~~str~~ |
| **RETURNS** | The removed table. ~~Table~~ |
## Lookups.has_table {#has_table tag="method"}
## Lookups.has_table {id="has_table",tag="method"}
Check if the lookups contain a table of a given name. Equivalent to
[`Lookups.__contains__`](/api/lookups#contains).
@ -145,7 +145,7 @@ Check if the lookups contain a table of a given name. Equivalent to
| `name` | Name of the table. ~~str~~ |
| **RETURNS** | Whether a table of that name is in the lookups. ~~bool~~ |
## Lookups.to_bytes {#to_bytes tag="method"}
## Lookups.to_bytes {id="to_bytes",tag="method"}
Serialize the lookups to a bytestring.
@ -159,7 +159,7 @@ Serialize the lookups to a bytestring.
| ----------- | --------------------------------- |
| **RETURNS** | The serialized lookups. ~~bytes~~ |
## Lookups.from_bytes {#from_bytes tag="method"}
## Lookups.from_bytes {id="from_bytes",tag="method"}
Load the lookups from a bytestring.
@ -176,7 +176,7 @@ Load the lookups from a bytestring.
| `bytes_data` | The data to load from. ~~bytes~~ |
| **RETURNS** | The loaded lookups. ~~Lookups~~ |
## Lookups.to_disk {#to_disk tag="method"}
## Lookups.to_disk {id="to_disk",tag="method"}
Save the lookups to a directory as `lookups.bin`. Expects a path to a directory,
which will be created if it doesn't exist.
@ -191,7 +191,7 @@ which will be created if it doesn't exist.
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## Lookups.from_disk {#from_disk tag="method"}
## Lookups.from_disk {id="from_disk",tag="method"}
Load lookups from a directory containing a `lookups.bin`. Will skip loading if
the file doesn't exist.
@ -209,7 +209,7 @@ the file doesn't exist.
| `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The loaded lookups. ~~Lookups~~ |
## Table {#table tag="class, ordererddict"}
## Table {id="table",tag="class, ordererddict"}
A table in the lookups. Subclass of `OrderedDict` that implements a slightly
more consistent and unified API and includes a Bloom filter to speed up missed
@ -218,7 +218,7 @@ lookups. Supports **all other methods and attributes** of `OrderedDict` /
accept both integers and strings (which will be hashed before being added to the
table).
### Table.\_\_init\_\_ {#table.init tag="method"}
### Table.\_\_init\_\_ {id="table.init",tag="method"}
Initialize a new table.
@ -236,7 +236,7 @@ Initialize a new table.
| ------ | ------------------------------------------ |
| `name` | Optional table name for reference. ~~str~~ |
### Table.from_dict {#table.from_dict tag="classmethod"}
### Table.from_dict {id="table.from_dict",tag="classmethod"}
Initialize a new table from a dict.
@ -254,7 +254,7 @@ Initialize a new table from a dict.
| `name` | Optional table name for reference. ~~str~~ |
| **RETURNS** | The newly constructed object. ~~Table~~ |
### Table.set {#table.set tag="method"}
### Table.set {id="table.set",tag="method"}
Set a new key / value pair. String keys will be hashed. Same as
`table[key] = value`.
@ -273,7 +273,7 @@ Set a new key / value pair. String keys will be hashed. Same as
| `key` | The key. ~~Union[str, int]~~ |
| `value` | The value. |
### Table.to_bytes {#table.to_bytes tag="method"}
### Table.to_bytes {id="table.to_bytes",tag="method"}
Serialize the table to a bytestring.
@ -287,7 +287,7 @@ Serialize the table to a bytestring.
| ----------- | ------------------------------- |
| **RETURNS** | The serialized table. ~~bytes~~ |
### Table.from_bytes {#table.from_bytes tag="method"}
### Table.from_bytes {id="table.from_bytes",tag="method"}
Load a table from a bytestring.
@ -304,7 +304,7 @@ Load a table from a bytestring.
| `bytes_data` | The data to load. ~~bytes~~ |
| **RETURNS** | The loaded table. ~~Table~~ |
### Attributes {#table-attributes}
### Attributes {id="table-attributes"}
| Name | Description |
| -------------- | ------------------------------------------------------------- |

16
website/docs/api/matcher.md → website/docs/api/matcher.mdx
View File

@ -13,7 +13,7 @@ tokens in context. For in-depth examples and workflows for combining rules and
statistical models, see the [usage guide](/usage/rule-based-matching) on
rule-based matching.
## Pattern format {#patterns}
## Pattern format {id="patterns"}
> ```json
> ### Example
@ -101,7 +101,7 @@ it compares to another value.
As of spaCy v3.5, `REGEX` and `FUZZY` can be used in combination with `IN` and
`NOT_IN`.
## Matcher.\_\_init\_\_ {#init tag="method"}
## Matcher.\_\_init\_\_ {id="init",tag="method"}
Create the rule-based `Matcher`. If `validate=True` is set, all patterns added
to the matcher will be validated against a JSON schema and a `MatchPatternError`
@ -121,7 +121,7 @@ string where an integer is expected) or unexpected property names.
| `validate` | Validate all patterns added to this matcher. ~~bool~~ |
| `fuzzy_compare` | The comparison method used for the `FUZZY` operators. ~~Callable[[str, str, int], bool]~~ |
## Matcher.\_\_call\_\_ {#call tag="method"}
## Matcher.\_\_call\_\_ {id="call",tag="method"}
Find all token sequences matching the supplied patterns on the `Doc` or `Span`.
@ -150,7 +150,7 @@ the match.
| `with_alignments` <Tag variant="new">3.0.6</Tag> | Return match alignment information as part of the match tuple as `List[int]` with the same length as the matched span. Each entry denotes the corresponding index of the token in the pattern. If `as_spans` is set to `True`, this setting is ignored. Defaults to `False`. ~~bool~~ |
| **RETURNS** | A list of `(match_id, start, end)` tuples, describing the matches. A match tuple describes a span `doc[start:end`]. The `match_id` is the ID of the added match pattern. If `as_spans` is set to `True`, a list of `Span` objects is returned instead. ~~Union[List[Tuple[int, int, int]], List[Span]]~~ |
## Matcher.\_\_len\_\_ {#len tag="method" new="2"}
## Matcher.\_\_len\_\_ {id="len",tag="method",version="2"}
Get the number of rules added to the matcher. Note that this only returns the
number of rules (identical with the number of IDs), not the number of individual
@ -169,7 +169,7 @@ patterns.
| ----------- | ---------------------------- |
| **RETURNS** | The number of rules. ~~int~~ |
## Matcher.\_\_contains\_\_ {#contains tag="method" new="2"}
## Matcher.\_\_contains\_\_ {id="contains",tag="method",version="2"}
Check whether the matcher contains rules for a match ID.
@ -187,7 +187,7 @@ Check whether the matcher contains rules for a match ID.
| `key` | The match ID. ~~str~~ |
| **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ |
## Matcher.add {#add tag="method" new="2"}
## Matcher.add {id="add",tag="method",version="2"}
Add a rule to the matcher, consisting of an ID key, one or more patterns, and an
optional callback function to act on the matches. The callback function will
@ -233,7 +233,7 @@ patterns = [[{"TEXT": "Google"}, {"TEXT": "Now"}], [{"TEXT": "GoogleNow"}]]
| `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[Matcher, Doc, int, List[tuple], Any]]~~ |
| `greedy` <Tag variant="new">3</Tag> | Optional filter for greedy matches. Can either be `"FIRST"` or `"LONGEST"`. ~~Optional[str]~~ |
## Matcher.remove {#remove tag="method" new="2"}
## Matcher.remove {id="remove",tag="method",version="2"}
Remove a rule from the matcher. A `KeyError` is raised if the match ID does not
exist.
@ -251,7 +251,7 @@ exist.
| ----- | --------------------------------- |
| `key` | The ID of the match rule. ~~str~~ |
## Matcher.get {#get tag="method" new="2"}
## Matcher.get {id="get",tag="method",version="2"}
Retrieve the pattern stored for a key. Returns the rule as an
`(on_match, patterns)` tuple containing the callback and available patterns.

42
website/docs/api/morphologizer.md → website/docs/api/morphologizer.mdx
View File

@ -2,7 +2,7 @@
title: Morphologizer
tag: class
source: spacy/pipeline/morphologizer.pyx
new: 3
version: 3
teaser: 'Pipeline component for predicting morphological features'
api_base_class: /api/tagger
api_string_name: morphologizer
@ -15,7 +15,7 @@ coarse-grained POS tags following the Universal Dependencies
[FEATS](https://universaldependencies.org/format.html#morphological-annotation)
annotation guidelines.
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Predictions are saved to `Token.morph` and `Token.pos`.
@ -25,7 +25,7 @@ Predictions are saved to `Token.morph` and `Token.pos`.
| `Token.pos_` | The UPOS part of speech. ~~str~~ |
| `Token.morph` | Morphological features. ~~MorphAnalysis~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -53,7 +53,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/morphologizer.pyx
```
## Morphologizer.\_\_init\_\_ {#init tag="method"}
## Morphologizer.\_\_init\_\_ {id="init",tag="method"}
Create a new pipeline instance. In your application, you would normally use a
shortcut for this and instantiate the component using its string name and
@ -97,7 +97,7 @@ annotation `C=E|X=Y`):
| `extend` <Tag variant="new">3.2</Tag> | Whether existing feature types (whose values may or may not be overwritten depending on `overwrite`) are preserved. Defaults to `False`. ~~bool~~ |
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attributes `"pos"` and `"morph"` and [`Scorer.score_token_attr_per_feat`](/api/scorer#score_token_attr_per_feat) for the attribute `"morph"`. ~~Optional[Callable]~~ |
## Morphologizer.\_\_call\_\_ {#call tag="method"}
## Morphologizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -120,7 +120,7 @@ delegate to the [`predict`](/api/morphologizer#predict) and
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## Morphologizer.pipe {#pipe tag="method"}
## Morphologizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -144,7 +144,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/morphologizer#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## Morphologizer.initialize {#initialize tag="method"}
## Morphologizer.initialize {id="initialize",tag="method"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -181,7 +181,7 @@ config.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[dict]~~ |
## Morphologizer.predict {#predict tag="method"}
## Morphologizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them.
@ -198,7 +198,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. |
## Morphologizer.set_annotations {#set_annotations tag="method"}
## Morphologizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -215,7 +215,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `Morphologizer.predict`. |
## Morphologizer.update {#update tag="method"}
## Morphologizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model.
@ -239,7 +239,7 @@ Delegates to [`predict`](/api/morphologizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Morphologizer.get_loss {#get_loss tag="method"}
## Morphologizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their
predicted scores.
@ -258,7 +258,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## Morphologizer.create_optimizer {#create_optimizer tag="method"}
## Morphologizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -273,7 +273,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## Morphologizer.use_params {#use_params tag="method, contextmanager"}
## Morphologizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -290,7 +290,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## Morphologizer.add_label {#add_label tag="method"}
## Morphologizer.add_label {id="add_label",tag="method"}
Add a new label to the pipe. If the `Morphologizer` should set annotations for
both `pos` and `morph`, the label should include the UPOS as the feature `POS`.
@ -313,7 +313,7 @@ will be automatically added to the model, and the output dimension will be
| `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## Morphologizer.to_disk {#to_disk tag="method"}
## Morphologizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -330,7 +330,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Morphologizer.from_disk {#from_disk tag="method"}
## Morphologizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -348,7 +348,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Morphologizer` object. ~~Morphologizer~~ |
## Morphologizer.to_bytes {#to_bytes tag="method"}
## Morphologizer.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -365,7 +365,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Morphologizer` object. ~~bytes~~ |
## Morphologizer.from_bytes {#from_bytes tag="method"}
## Morphologizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -384,7 +384,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Morphologizer` object. ~~Morphologizer~~ |
## Morphologizer.labels {#labels tag="property"}
## Morphologizer.labels {id="labels",tag="property"}
The labels currently added to the component in the Universal Dependencies
[FEATS](https://universaldependencies.org/format.html#morphological-annotation)
@ -403,7 +403,7 @@ coarse-grained POS as the feature `POS`.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## Morphologizer.label_data {#label_data tag="property" new="3"}
## Morphologizer.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -421,7 +421,7 @@ model with a pre-defined label set.
| ----------- | ----------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~dict~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

40
website/docs/api/morphology.md → website/docs/api/morphology.mdx
View File

@ -10,7 +10,7 @@ morphological analysis, so queries of morphological attributes are delegated to
this class. See [`MorphAnalysis`](/api/morphology#morphanalysis) for the
container storing a single morphological analysis.
## Morphology.\_\_init\_\_ {#init tag="method"}
## Morphology.\_\_init\_\_ {id="init",tag="method"}
Create a `Morphology` object.
@ -26,7 +26,7 @@ Create a `Morphology` object.
| --------- | --------------------------------- |
| `strings` | The string store. ~~StringStore~~ |
## Morphology.add {#add tag="method"}
## Morphology.add {id="add",tag="method"}
Insert a morphological analysis in the morphology table, if not already present.
The morphological analysis may be provided in the Universal Dependencies
@ -46,7 +46,7 @@ new analysis.
| ---------- | ------------------------------------------------ |
| `features` | The morphological features. ~~Union[Dict, str]~~ |
## Morphology.get {#get tag="method"}
## Morphology.get {id="get",tag="method"}
> #### Example
>
@ -64,7 +64,7 @@ string for the hash of the morphological analysis.
| ------- | ----------------------------------------------- |
| `morph` | The hash of the morphological analysis. ~~int~~ |
## Morphology.feats_to_dict {#feats_to_dict tag="staticmethod"}
## Morphology.feats_to_dict {id="feats_to_dict",tag="staticmethod"}
Convert a string
[FEATS](https://universaldependencies.org/format.html#morphological-annotation)
@ -84,7 +84,7 @@ tag map.
| `feats` | The morphological features in Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ |
| **RETURNS** | The morphological features as a dictionary. ~~Dict[str, str]~~ |
## Morphology.dict_to_feats {#dict_to_feats tag="staticmethod"}
## Morphology.dict_to_feats {id="dict_to_feats",tag="staticmethod"}
Convert a dictionary of features and values to a string
[FEATS](https://universaldependencies.org/format.html#morphological-annotation)
@ -103,19 +103,19 @@ representation.
| `feats_dict` | The morphological features as a dictionary. ~~Dict[str, str]~~ |
| **RETURNS** | The morphological features in Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `FEATURE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) feature separator. Default is `|`. ~~str~~ |
| `FIELD_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) field separator. Default is `=`. ~~str~~ |
| `VALUE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) value separator. Default is `,`. ~~str~~ |
| Name | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `FEATURE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) feature separator. Default is `\|`. ~~str~~ |
| `FIELD_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) field separator. Default is `=`. ~~str~~ |
| `VALUE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) value separator. Default is `,`. ~~str~~ |
## MorphAnalysis {#morphanalysis tag="class" source="spacy/tokens/morphanalysis.pyx"}
## MorphAnalysis {id="morphanalysis",tag="class",source="spacy/tokens/morphanalysis.pyx"}
Stores a single morphological analysis.
### MorphAnalysis.\_\_init\_\_ {#morphanalysis-init tag="method"}
### MorphAnalysis.\_\_init\_\_ {id="morphanalysis-init",tag="method"}
Initialize a MorphAnalysis object from a Universal Dependencies
[FEATS](https://universaldependencies.org/format.html#morphological-annotation)
@ -135,7 +135,7 @@ string or a dictionary of morphological features.
| `vocab` | The vocab. ~~Vocab~~ |
| `features` | The morphological features. ~~Union[Dict[str, str], str]~~ |
### MorphAnalysis.\_\_contains\_\_ {#morphanalysis-contains tag="method"}
### MorphAnalysis.\_\_contains\_\_ {id="morphanalysis-contains",tag="method"}
Whether a feature/value pair is in the analysis.
@ -151,7 +151,7 @@ Whether a feature/value pair is in the analysis.
| ----------- | --------------------------------------------- |
| **RETURNS** | A feature/value pair in the analysis. ~~str~~ |
### MorphAnalysis.\_\_iter\_\_ {#morphanalysis-iter tag="method"}
### MorphAnalysis.\_\_iter\_\_ {id="morphanalysis-iter",tag="method"}
Iterate over the feature/value pairs in the analysis.
@ -167,7 +167,7 @@ Iterate over the feature/value pairs in the analysis.
| ---------- | --------------------------------------------- |
| **YIELDS** | A feature/value pair in the analysis. ~~str~~ |
### MorphAnalysis.\_\_len\_\_ {#morphanalysis-len tag="method"}
### MorphAnalysis.\_\_len\_\_ {id="morphanalysis-len",tag="method"}
Returns the number of features in the analysis.
@ -183,7 +183,7 @@ Returns the number of features in the analysis.
| ----------- | ----------------------------------------------- |
| **RETURNS** | The number of features in the analysis. ~~int~~ |
### MorphAnalysis.\_\_str\_\_ {#morphanalysis-str tag="method"}
### MorphAnalysis.\_\_str\_\_ {id="morphanalysis-str",tag="method"}
Returns the morphological analysis in the Universal Dependencies
[FEATS](https://universaldependencies.org/format.html#morphological-annotation)
@ -201,7 +201,7 @@ string format.
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **RETURNS** | The analysis in the Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ |
### MorphAnalysis.get {#morphanalysis-get tag="method"}
### MorphAnalysis.get {id="morphanalysis-get",tag="method"}
Retrieve values for a feature by field.
@ -218,7 +218,7 @@ Retrieve values for a feature by field.
| `field` | The field to retrieve. ~~str~~ |
| **RETURNS** | A list of the individual features. ~~List[str]~~ |
### MorphAnalysis.to_dict {#morphanalysis-to_dict tag="method"}
### MorphAnalysis.to_dict {id="morphanalysis-to_dict",tag="method"}
Produce a dict representation of the analysis, in the same format as the tag
map.
@ -235,7 +235,7 @@ map.
| ----------- | ----------------------------------------------------------- |
| **RETURNS** | The dict representation of the analysis. ~~Dict[str, str]~~ |
### MorphAnalysis.from_id {#morphanalysis-from_id tag="classmethod"}
### MorphAnalysis.from_id {id="morphanalysis-from_id",tag="classmethod"}
Create a morphological analysis from a given hash ID.

14
website/docs/api/phrasematcher.md → website/docs/api/phrasematcher.mdx
View File

@ -3,7 +3,7 @@ title: PhraseMatcher
teaser: Match sequences of tokens, based on documents
tag: class
source: spacy/matcher/phrasematcher.pyx
new: 2
version: 2
---
The `PhraseMatcher` lets you efficiently match large terminology lists. While
@ -12,7 +12,7 @@ descriptions, the `PhraseMatcher` accepts match patterns in the form of `Doc`
objects. See the [usage guide](/usage/rule-based-matching#phrasematcher) for
examples.
## PhraseMatcher.\_\_init\_\_ {#init tag="method"}
## PhraseMatcher.\_\_init\_\_ {id="init",tag="method"}
Create the rule-based `PhraseMatcher`. Setting a different `attr` to match on
will change the token attributes that will be compared to determine a match. By
@ -42,7 +42,7 @@ be shown.
| `attr` | The token attribute to match on. Defaults to `ORTH`, i.e. the verbatim token text. ~~Union[int, str]~~ |
| `validate` | Validate patterns added to the matcher. ~~bool~~ |
## PhraseMatcher.\_\_call\_\_ {#call tag="method"}
## PhraseMatcher.\_\_call\_\_ {id="call",tag="method"}
Find all token sequences matching the supplied patterns on the `Doc` or `Span`.
@ -76,7 +76,7 @@ match_id_string = nlp.vocab.strings[match_id]
</Infobox>
## PhraseMatcher.\_\_len\_\_ {#len tag="method"}
## PhraseMatcher.\_\_len\_\_ {id="len",tag="method"}
Get the number of rules added to the matcher. Note that this only returns the
number of rules (identical with the number of IDs), not the number of individual
@ -95,7 +95,7 @@ patterns.
| ----------- | ---------------------------- |
| **RETURNS** | The number of rules. ~~int~~ |
## PhraseMatcher.\_\_contains\_\_ {#contains tag="method"}
## PhraseMatcher.\_\_contains\_\_ {id="contains",tag="method"}
Check whether the matcher contains rules for a match ID.
@ -113,7 +113,7 @@ Check whether the matcher contains rules for a match ID.
| `key` | The match ID. ~~str~~ |
| **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ |
## PhraseMatcher.add {#add tag="method"}
## PhraseMatcher.add {id="add",tag="method"}
Add a rule to the matcher, consisting of an ID key, one or more patterns, and a
callback function to act on the matches. The callback function will receive the
@ -155,7 +155,7 @@ patterns = [nlp("health care reform"), nlp("healthcare reform")]
| _keyword-only_ | |
| `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[Matcher, Doc, int, List[tuple], Any]]~~ |
## PhraseMatcher.remove {#remove tag="method" new="2.2"}
## PhraseMatcher.remove {id="remove",tag="method",version="2.2"}
Remove a rule from the matcher by match ID. A `KeyError` is raised if the key
does not exist.

50
website/docs/api/pipe.md → website/docs/api/pipe.mdx
View File

@ -12,7 +12,7 @@ spaCy pipeline. See the docs on
[writing trainable components](/usage/processing-pipelines#trainable-components)
for how to use the `TrainablePipe` base class to implement custom components.
<!-- TODO: Pipe vs TrainablePipe, check methods below (all renamed to TrainablePipe for now) -->
{/* TODO: Pipe vs TrainablePipe, check methods below (all renamed to TrainablePipe for now) */}
> #### Why is it implemented in Cython?
>
@ -27,7 +27,7 @@ for how to use the `TrainablePipe` base class to implement custom components.
%%GITHUB_SPACY/spacy/pipeline/trainable_pipe.pyx
```
## TrainablePipe.\_\_init\_\_ {#init tag="method"}
## TrainablePipe.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -54,7 +54,7 @@ shortcut for this and instantiate the component using its string name and
| `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ |
| `**cfg` | Additional config parameters and settings. Will be available as the dictionary `cfg` and is serialized with the component. |
## TrainablePipe.\_\_call\_\_ {#call tag="method"}
## TrainablePipe.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -77,7 +77,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## TrainablePipe.pipe {#pipe tag="method"}
## TrainablePipe.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -100,7 +100,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/pipe#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## TrainablePipe.set_error_handler {#set_error_handler tag="method" new="3"}
## TrainablePipe.set_error_handler {id="set_error_handler",tag="method",version="3"}
Define a callback that will be invoked when an error is thrown during processing
of one or more documents with either [`__call__`](/api/pipe#call) or
@ -122,7 +122,7 @@ processed, and the original error.
| --------------- | -------------------------------------------------------------------------------------------------------------- |
| `error_handler` | A function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ |
## TrainablePipe.get_error_handler {#get_error_handler tag="method" new="3"}
## TrainablePipe.get_error_handler {id="get_error_handler",tag="method",version="3"}
Retrieve the callback that performs error handling for this component's
[`__call__`](/api/pipe#call) and [`pipe`](/api/pipe#pipe) methods. If no custom
@ -141,7 +141,7 @@ returned that simply reraises the exception.
| ----------- | ---------------------------------------------------------------------------------------------------------------- |
| **RETURNS** | The function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ |
## TrainablePipe.initialize {#initialize tag="method" new="3"}
## TrainablePipe.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. The data examples are
@ -171,7 +171,7 @@ This method was previously called `begin_training`.
| _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## TrainablePipe.predict {#predict tag="method"}
## TrainablePipe.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them.
@ -194,7 +194,7 @@ This method needs to be overwritten with your own custom `predict` method.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. |
## TrainablePipe.set_annotations {#set_annotations tag="method"}
## TrainablePipe.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -218,7 +218,7 @@ method.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `Tagger.predict`. |
## TrainablePipe.update {#update tag="method"}
## TrainablePipe.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model.
@ -240,7 +240,7 @@ predictions and gold-standard annotations, and update the component's model.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## TrainablePipe.rehearse {#rehearse tag="method,experimental" new="3"}
## TrainablePipe.rehearse {id="rehearse",tag="method,experimental",version="3"}
Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the
current model to make predictions similar to an initial model, to try to address
@ -262,7 +262,7 @@ the "catastrophic forgetting" problem. This feature is experimental.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## TrainablePipe.get_loss {#get_loss tag="method"}
## TrainablePipe.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their
predicted scores.
@ -287,7 +287,7 @@ This method needs to be overwritten with your own custom `get_loss` method.
| `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## TrainablePipe.score {#score tag="method" new="3"}
## TrainablePipe.score {id="score",tag="method",version="3"}
Score a batch of examples.
@ -304,7 +304,7 @@ Score a batch of examples.
| `\*\*kwargs` | Any additional settings to pass on to the scorer. ~~Any~~ |
| **RETURNS** | The scores, e.g. produced by the [`Scorer`](/api/scorer). ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## TrainablePipe.create_optimizer {#create_optimizer tag="method"}
## TrainablePipe.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component. Defaults to
[`Adam`](https://thinc.ai/docs/api-optimizers#adam) with default settings.
@ -320,7 +320,7 @@ Create an optimizer for the pipeline component. Defaults to
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## TrainablePipe.use_params {#use_params tag="method, contextmanager"}
## TrainablePipe.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -337,7 +337,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## TrainablePipe.finish_update {#finish_update tag="method"}
## TrainablePipe.finish_update {id="finish_update",tag="method"}
Update parameters using the current parameter gradients. Defaults to calling
[`self.model.finish_update`](https://thinc.ai/docs/api-model#finish_update).
@ -355,7 +355,7 @@ Update parameters using the current parameter gradients. Defaults to calling
| ----- | ------------------------------------- |
| `sgd` | An optimizer. ~~Optional[Optimizer]~~ |
## TrainablePipe.add_label {#add_label tag="method"}
## TrainablePipe.add_label {id="add_label",tag="method"}
> #### Example
>
@ -390,7 +390,7 @@ case, all labels found in the sample will be automatically added to the model,
and the output dimension will be
[inferred](/usage/layers-architectures#thinc-shape-inference) automatically.
## TrainablePipe.is_resizable {#is_resizable tag="property"}
## TrainablePipe.is_resizable {id="is_resizable",tag="property"}
> #### Example
>
@ -421,7 +421,7 @@ as an attribute to the component's model.
| ----------- | ---------------------------------------------------------------------------------------------- |
| **RETURNS** | Whether or not the output dimension of the model can be changed after initialization. ~~bool~~ |
## TrainablePipe.set_output {#set_output tag="method"}
## TrainablePipe.set_output {id="set_output",tag="method"}
Change the output dimension of the component's model. If the component is not
[resizable](#is_resizable), this method will raise a `NotImplementedError`. If a
@ -441,7 +441,7 @@ care should be taken to avoid the "catastrophic forgetting" problem.
| ---- | --------------------------------- |
| `nO` | The new output dimension. ~~int~~ |
## TrainablePipe.to_disk {#to_disk tag="method"}
## TrainablePipe.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -458,7 +458,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## TrainablePipe.from_disk {#from_disk tag="method"}
## TrainablePipe.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -476,7 +476,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified pipe. ~~TrainablePipe~~ |
## TrainablePipe.to_bytes {#to_bytes tag="method"}
## TrainablePipe.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -493,7 +493,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the pipe. ~~bytes~~ |
## TrainablePipe.from_bytes {#from_bytes tag="method"}
## TrainablePipe.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -512,7 +512,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The pipe. ~~TrainablePipe~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| ------- | --------------------------------------------------------------------------------------------------------------------------------- |
@ -521,7 +521,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `name` | The name of the component instance in the pipeline. Can be used in the losses. ~~str~~ |
| `cfg` | Keyword arguments passed to [`TrainablePipe.__init__`](/api/pipe#init). Will be serialized with the component. ~~Dict[str, Any]~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

12
website/docs/api/pipeline-functions.md → website/docs/api/pipeline-functions.mdx
View File

@ -10,7 +10,7 @@ menu:
- ['doc_cleaner', 'doc_cleaner']
---
## merge_noun_chunks {#merge_noun_chunks tag="function"}
## merge_noun_chunks {id="merge_noun_chunks",tag="function"}
Merge noun chunks into a single token. Also available via the string name
`"merge_noun_chunks"`.
@ -40,7 +40,7 @@ all other components.
| `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ |
| **RETURNS** | The modified `Doc` with merged noun chunks. ~~Doc~~ |
## merge_entities {#merge_entities tag="function"}
## merge_entities {id="merge_entities",tag="function"}
Merge named entities into a single token. Also available via the string name
`"merge_entities"`.
@ -70,7 +70,7 @@ components to the end of the pipeline and after all other components.
| `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ |
| **RETURNS** | The modified `Doc` with merged entities. ~~Doc~~ |
## merge_subtokens {#merge_subtokens tag="function" new="2.1"}
## merge_subtokens {id="merge_subtokens",tag="function",version="2.1"}
Merge subtokens into a single token. Also available via the string name
`"merge_subtokens"`. As of v2.1, the parser is able to predict "subtokens" that
@ -110,7 +110,7 @@ end of the pipeline and after all other components.
| `label` | The subtoken dependency label. Defaults to `"subtok"`. ~~str~~ |
| **RETURNS** | The modified `Doc` with merged subtokens. ~~Doc~~ |
## token_splitter {#token_splitter tag="function" new="3.0"}
## token_splitter {id="token_splitter",tag="function",version="3.0"}
Split tokens longer than a minimum length into shorter tokens. Intended for use
with transformer pipelines where long spaCy tokens lead to input text that
@ -132,7 +132,7 @@ exceed the transformer model max length.
| `split_length` | The length of the split tokens. Defaults to `5`. ~~int~~ |
| **RETURNS** | The modified `Doc` with the split tokens. ~~Doc~~ |
## doc_cleaner {#doc_cleaner tag="function" new="3.2.1"}
## doc_cleaner {id="doc_cleaner",tag="function",version="3.2.1"}
Clean up `Doc` attributes. Intended for use at the end of pipelines with
`tok2vec` or `transformer` pipeline components that store tensors and other
@ -154,7 +154,7 @@ whole pipeline has run.
| `silent` | If `False`, show warnings if attributes aren't found or can't be set. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The modified `Doc` with the modified attributes. ~~Doc~~ |
## span_cleaner {#span_cleaner tag="function,experimental"}
## span_cleaner {id="span_cleaner",tag="function,experimental"}
Remove `SpanGroup`s from `doc.spans` based on a key prefix. This is used to
clean up after the [`CoreferenceResolver`](/api/coref) when it's paired with a

24
website/docs/api/scorer.md → website/docs/api/scorer.mdx
View File

@ -10,7 +10,7 @@ The `Scorer` computes evaluation scores. It's typically created by
provides a number of evaluation methods for evaluating [`Token`](/api/token) and
[`Doc`](/api/doc) attributes.
## Scorer.\_\_init\_\_ {#init tag="method"}
## Scorer.\_\_init\_\_ {id="init",tag="method"}
Create a new `Scorer`.
@ -35,7 +35,7 @@ Create a new `Scorer`.
| _keyword-only_ | |
| `\*\*kwargs` | Any additional settings to pass on to the individual scoring methods. ~~Any~~ |
## Scorer.score {#score tag="method"}
## Scorer.score {id="score",tag="method"}
Calculate the scores for a list of [`Example`](/api/example) objects using the
scoring methods provided by the components in the pipeline.
@ -72,7 +72,7 @@ core pipeline components, the individual score names start with the `Token` or
| `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ |
| **RETURNS** | A dictionary of scores. ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## Scorer.score_tokenization {#score_tokenization tag="staticmethod" new="3"}
## Scorer.score_tokenization {id="score_tokenization",tag="staticmethod",version="3"}
Scores the tokenization:
@ -93,7 +93,7 @@ Docs with `has_unknown_spaces` are skipped during scoring.
| `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ |
| **RETURNS** | `Dict` | A dictionary containing the scores `token_acc`, `token_p`, `token_r`, `token_f`. ~~Dict[str, float]]~~ |
## Scorer.score_token_attr {#score_token_attr tag="staticmethod" new="3"}
## Scorer.score_token_attr {id="score_token_attr",tag="staticmethod",version="3"}
Scores a single token attribute. Tokens with missing values in the reference doc
are skipped during scoring.
@ -114,7 +114,7 @@ are skipped during scoring.
| `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ |
| **RETURNS** | A dictionary containing the score `{attr}_acc`. ~~Dict[str, float]~~ |
## Scorer.score_token_attr_per_feat {#score_token_attr_per_feat tag="staticmethod" new="3"}
## Scorer.score_token_attr_per_feat {id="score_token_attr_per_feat",tag="staticmethod",version="3"}
Scores a single token attribute per feature for a token attribute in the
Universal Dependencies
@ -138,7 +138,7 @@ scoring.
| `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ |
| **RETURNS** | A dictionary containing the micro PRF scores under the key `{attr}_micro_p/r/f` and the per-feature PRF scores under `{attr}_per_feat`. ~~Dict[str, Dict[str, float]]~~ |
## Scorer.score_spans {#score_spans tag="staticmethod" new="3"}
## Scorer.score_spans {id="score_spans",tag="staticmethod",version="3"}
Returns PRF scores for labeled or unlabeled spans.
@ -160,7 +160,7 @@ Returns PRF scores for labeled or unlabeled spans.
| `allow_overlap` | Defaults to `False`. Whether or not to allow overlapping spans. If set to `False`, the alignment will automatically resolve conflicts. ~~bool~~ |
| **RETURNS** | A dictionary containing the PRF scores under the keys `{attr}_p`, `{attr}_r`, `{attr}_f` and the per-type PRF scores under `{attr}_per_type`. ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## Scorer.score_deps {#score_deps tag="staticmethod" new="3"}
## Scorer.score_deps {id="score_deps",tag="staticmethod",version="3"}
Calculate the UAS, LAS, and LAS per type scores for dependency parses. Tokens
with missing values for the `attr` (typically `dep`) are skipped during scoring.
@ -194,7 +194,7 @@ with missing values for the `attr` (typically `dep`) are skipped during scoring.
| `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ |
| **RETURNS** | A dictionary containing the scores: `{attr}_uas`, `{attr}_las`, and `{attr}_las_per_type`. ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## Scorer.score_cats {#score_cats tag="staticmethod" new="3"}
## Scorer.score_cats {id="score_cats",tag="staticmethod",version="3"}
Calculate PRF and ROC AUC scores for a doc-level attribute that is a dict
containing scores for each label like `Doc.cats`. The returned dictionary
@ -241,7 +241,7 @@ The reported `{attr}_score` depends on the classification properties:
| `threshold` | Cutoff to consider a prediction "positive". Defaults to `0.5` for multi-label, and `0.0` (i.e. whatever's highest scoring) otherwise. ~~float~~ |
| **RETURNS** | A dictionary containing the scores, with inapplicable scores as `None`. ~~Dict[str, Optional[float]]~~ |
## Scorer.score_links {#score_links tag="staticmethod" new="3"}
## Scorer.score_links {id="score_links",tag="staticmethod",version="3"}
Returns PRF for predicted links on the entity level. To disentangle the
performance of the NEL from the NER, this method only evaluates NEL links for
@ -264,7 +264,7 @@ entities that overlap between the gold reference and the predictions.
| `negative_labels` | The string values that refer to no annotation (e.g. "NIL"). ~~Iterable[str]~~ |
| **RETURNS** | A dictionary containing the scores. ~~Dict[str, Optional[float]]~~ |
## get_ner_prf {#get_ner_prf new="3"}
## get_ner_prf {id="get_ner_prf",version="3"}
Compute micro-PRF and per-entity PRF scores.
@ -272,7 +272,7 @@ Compute micro-PRF and per-entity PRF scores.
| ---------- | ------------------------------------------------------------------------------------------------------------------- |
| `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ |
## score_coref_clusters {#score_coref_clusters tag="experimental"}
## score_coref_clusters {id="score_coref_clusters",tag="experimental"}
Returns LEA ([Moosavi and Strube, 2016](https://aclanthology.org/P16-1060/)) PRF
scores for coreference clusters.
@ -301,7 +301,7 @@ the [CoreferenceResolver](/api/coref) docs.
| `span_cluster_prefix` | The prefix used for spans representing coreference clusters. ~~str~~ |
| **RETURNS** | A dictionary containing the scores. ~~Dict[str, Optional[float]]~~ |
## score_span_predictions {#score_span_predictions tag="experimental"}
## score_span_predictions {id="score_span_predictions",tag="experimental"}
Return accuracy for reconstructions of spans from single tokens. Only exactly
correct predictions are counted as correct, there is no partial credit for near

38
website/docs/api/sentencerecognizer.md → website/docs/api/sentencerecognizer.mdx
View File

@ -2,7 +2,7 @@
title: SentenceRecognizer
tag: class
source: spacy/pipeline/senter.pyx
new: 3
version: 3
teaser: 'Pipeline component for sentence segmentation'
api_base_class: /api/tagger
api_string_name: senter
@ -12,7 +12,7 @@ api_trainable: true
A trainable pipeline component for sentence segmentation. For a simpler,
rule-based strategy, see the [`Sentencizer`](/api/sentencizer).
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Predicted values will be assigned to `Token.is_sent_start`. The resulting
sentences can be accessed using `Doc.sents`.
@ -22,7 +22,7 @@ sentences can be accessed using `Doc.sents`.
| `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. This will be either `True` or `False` for all tokens. ~~bool~~ |
| `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -49,7 +49,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/senter.pyx
```
## SentenceRecognizer.\_\_init\_\_ {#init tag="method"}
## SentenceRecognizer.\_\_init\_\_ {id="init",tag="method"}
Initialize the sentence recognizer.
@ -81,7 +81,7 @@ shortcut for this and instantiate the component using its string name and
| `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ |
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"`. ~~Optional[Callable]~~ |
## SentenceRecognizer.\_\_call\_\_ {#call tag="method"}
## SentenceRecognizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -105,7 +105,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## SentenceRecognizer.pipe {#pipe tag="method"}
## SentenceRecognizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -129,7 +129,7 @@ and [`pipe`](/api/sentencerecognizer#pipe) delegate to the
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## SentenceRecognizer.initialize {#initialize tag="method"}
## SentenceRecognizer.initialize {id="initialize",tag="method"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -153,7 +153,7 @@ by [`Language.initialize`](/api/language#initialize).
| _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## SentenceRecognizer.predict {#predict tag="method"}
## SentenceRecognizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them.
@ -170,7 +170,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. |
## SentenceRecognizer.set_annotations {#set_annotations tag="method"}
## SentenceRecognizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -187,7 +187,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `SentenceRecognizer.predict`. |
## SentenceRecognizer.update {#update tag="method"}
## SentenceRecognizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model.
@ -211,7 +211,7 @@ Delegates to [`predict`](/api/sentencerecognizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## SentenceRecognizer.rehearse {#rehearse tag="method,experimental" new="3"}
## SentenceRecognizer.rehearse {id="rehearse",tag="method,experimental",version="3"}
Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the
current model to make predictions similar to an initial model to try to address
@ -234,7 +234,7 @@ the "catastrophic forgetting" problem. This feature is experimental.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## SentenceRecognizer.get_loss {#get_loss tag="method"}
## SentenceRecognizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their
predicted scores.
@ -253,7 +253,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## SentenceRecognizer.create_optimizer {#create_optimizer tag="method"}
## SentenceRecognizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -268,7 +268,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## SentenceRecognizer.use_params {#use_params tag="method, contextmanager"}
## SentenceRecognizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -285,7 +285,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## SentenceRecognizer.to_disk {#to_disk tag="method"}
## SentenceRecognizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -302,7 +302,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## SentenceRecognizer.from_disk {#from_disk tag="method"}
## SentenceRecognizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -320,7 +320,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `SentenceRecognizer` object. ~~SentenceRecognizer~~ |
## SentenceRecognizer.to_bytes {#to_bytes tag="method"}
## SentenceRecognizer.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -337,7 +337,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `SentenceRecognizer` object. ~~bytes~~ |
## SentenceRecognizer.from_bytes {#from_bytes tag="method"}
## SentenceRecognizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -356,7 +356,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `SentenceRecognizer` object. ~~SentenceRecognizer~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

23
website/docs/api/sentencizer.md → website/docs/api/sentencizer.mdx
View File

@ -13,7 +13,7 @@ performed by the [`DependencyParser`](/api/dependencyparser), so the
`Sentencizer` lets you implement a simpler, rule-based strategy that doesn't
require a statistical model to be loaded.
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Calculated values will be assigned to `Token.is_sent_start`. The resulting
sentences can be accessed using `Doc.sents`.
@ -23,7 +23,7 @@ sentences can be accessed using `Doc.sents`.
| `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. This will be either `True` or `False` for all tokens. ~~bool~~ |
| `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -39,7 +39,7 @@ how the component should be configured. You can override its settings via the
| Setting | Description |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `punct_chars` | Optional custom list of punctuation characters that mark sentence ends. See below for defaults if not set. Defaults to `None`. ~~Optional[List[str]]~~ | `None` |
| `punct_chars` | Optional custom list of punctuation characters that mark sentence ends. See below for defaults if not set. Defaults to `None`. ~~Optional[List[str]]~~ |
| `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ |
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"` ~~Optional[Callable]~~ |
@ -47,7 +47,7 @@ how the component should be configured. You can override its settings via the
%%GITHUB_SPACY/spacy/pipeline/sentencizer.pyx
```
## Sentencizer.\_\_init\_\_ {#init tag="method"}
## Sentencizer.\_\_init\_\_ {id="init",tag="method"}
Initialize the sentencizer.
@ -69,8 +69,7 @@ Initialize the sentencizer.
| `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ |
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"` ~~Optional[Callable]~~ |
```python
### punct_chars defaults
```python {title="punct_chars defaults"}
['!', '.', '?', '։', '؟', '۔', '܀', '܁', '܂', '߹', '।', '॥', '၊', '။', '።',
'፧', '፨', '᙮', '᜵', '᜶', '᠃', '᠉', '᥄', '᥅', '᪨', '᪩', '᪪', '᪫',
'᭚', '᭛', '᭞', '᭟', '᰻', '᰼', '᱾', '᱿', '‼', '‽', '⁇', '⁈', '⁉',
@ -83,7 +82,7 @@ Initialize the sentencizer.
'𑪜', '𑱁', '𑱂', '𖩮', '𖩯', '𖫵', '𖬷', '𖬸', '𖭄', '𛲟', '𝪈', '｡', '。']
```
## Sentencizer.\_\_call\_\_ {#call tag="method"}
## Sentencizer.\_\_call\_\_ {id="call",tag="method"}
Apply the sentencizer on a `Doc`. Typically, this happens automatically after
the component has been added to the pipeline using
@ -105,7 +104,7 @@ the component has been added to the pipeline using
| `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ |
| **RETURNS** | The modified `Doc` with added sentence boundaries. ~~Doc~~ |
## Sentencizer.pipe {#pipe tag="method"}
## Sentencizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -126,7 +125,7 @@ applied to the `Doc` in order.
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## Sentencizer.to_disk {#to_disk tag="method"}
## Sentencizer.to_disk {id="to_disk",tag="method"}
Save the sentencizer settings (punctuation characters) to a directory. Will
create a file `sentencizer.json`. This also happens automatically when you save
@ -144,7 +143,7 @@ an `nlp` object with a sentencizer added to its pipeline.
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path` | A path to a JSON file, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## Sentencizer.from_disk {#from_disk tag="method"}
## Sentencizer.from_disk {id="from_disk",tag="method"}
Load the sentencizer settings from a file. Expects a JSON file. This also
happens automatically when you load an `nlp` object or model with a sentencizer
@ -162,7 +161,7 @@ added to its pipeline.
| `path` | A path to a JSON file. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The modified `Sentencizer` object. ~~Sentencizer~~ |
## Sentencizer.to_bytes {#to_bytes tag="method"}
## Sentencizer.to_bytes {id="to_bytes",tag="method"}
Serialize the sentencizer settings to a bytestring.
@ -178,7 +177,7 @@ Serialize the sentencizer settings to a bytestring.
| ----------- | ------------------------------ |
| **RETURNS** | The serialized data. ~~bytes~~ |
## Sentencizer.from_bytes {#from_bytes tag="method"}
## Sentencizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.

32
website/docs/api/span-resolver.md → website/docs/api/span-resolver.mdx
View File

@ -33,7 +33,7 @@ use case is as a post-processing step on word-level
[coreference resolution](/api/coref). The input and output keys used to store
`Span` objects are configurable.
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Predictions will be saved to `Doc.spans` as [`SpanGroup`s](/api/spangroup).
@ -46,7 +46,7 @@ prefixes are configurable.
| ------------------------------------------------- | ------------------------------------------------------------------------- |
| `Doc.spans[output_prefix + "_" + cluster_number]` | One group of predicted spans. Cluster number starts from 1. ~~SpanGroup~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -74,7 +74,7 @@ details on the architectures and their arguments and hyperparameters.
| `input_prefix` | The prefix to use for input `SpanGroup`s. Defaults to `coref_head_clusters`. ~~str~~ |
| `output_prefix` | The prefix for predicted `SpanGroup`s. Defaults to `coref_clusters`. ~~str~~ |
## SpanResolver.\_\_init\_\_ {#init tag="method"}
## SpanResolver.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -104,7 +104,7 @@ shortcut for this and instantiate the component using its string name and
| `input_prefix` | The prefix to use for input `SpanGroup`s. Defaults to `coref_head_clusters`. ~~str~~ |
| `output_prefix` | The prefix for predicted `SpanGroup`s. Defaults to `coref_clusters`. ~~str~~ |
## SpanResolver.\_\_call\_\_ {#call tag="method"}
## SpanResolver.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -126,7 +126,7 @@ and [`set_annotations`](#set_annotations) methods.
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## SpanResolver.pipe {#pipe tag="method"}
## SpanResolver.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -150,7 +150,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/span-resolver#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## SpanResolver.initialize {#initialize tag="method"}
## SpanResolver.initialize {id="initialize",tag="method"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -174,7 +174,7 @@ by [`Language.initialize`](/api/language#initialize).
| _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## SpanResolver.predict {#predict tag="method"}
## SpanResolver.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them. Predictions are returned as a list of `MentionClusters`, one for
@ -194,7 +194,7 @@ correspond to token indices.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The predicted spans for the `Doc`s. ~~List[MentionClusters]~~ |
## SpanResolver.set_annotations {#set_annotations tag="method"}
## SpanResolver.set_annotations {id="set_annotations",tag="method"}
Modify a batch of documents, saving predictions using the output prefix in
`Doc.spans`.
@ -212,7 +212,7 @@ Modify a batch of documents, saving predictions using the output prefix in
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `spans` | The predicted spans for the `docs`. ~~List[MentionClusters]~~ |
## SpanResolver.update {#update tag="method"}
## SpanResolver.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects. Delegates to
[`predict`](/api/span-resolver#predict).
@ -234,7 +234,7 @@ Learn from a batch of [`Example`](/api/example) objects. Delegates to
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## SpanResolver.create_optimizer {#create_optimizer tag="method"}
## SpanResolver.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -249,7 +249,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## SpanResolver.use_params {#use_params tag="method, contextmanager"}
## SpanResolver.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -266,7 +266,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## SpanResolver.to_disk {#to_disk tag="method"}
## SpanResolver.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -283,7 +283,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## SpanResolver.from_disk {#from_disk tag="method"}
## SpanResolver.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -301,7 +301,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `SpanResolver` object. ~~SpanResolver~~ |
## SpanResolver.to_bytes {#to_bytes tag="method"}
## SpanResolver.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -318,7 +318,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `SpanResolver` object. ~~bytes~~ |
## SpanResolver.from_bytes {#from_bytes tag="method"}
## SpanResolver.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -337,7 +337,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `SpanResolver` object. ~~SpanResolver~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

56
website/docs/api/span.md → website/docs/api/span.mdx
View File

@ -6,7 +6,7 @@ source: spacy/tokens/span.pyx
A slice from a [`Doc`](/api/doc) object.
## Span.\_\_init\_\_ {#init tag="method"}
## Span.\_\_init\_\_ {id="init",tag="method"}
Create a `Span` object from the slice `doc[start : end]`.
@ -29,7 +29,7 @@ Create a `Span` object from the slice `doc[start : end]`.
| `kb_id` | A knowledge base ID to attach to the span, e.g. for named entities. ~~Union[str, int]~~ |
| `span_id` | An ID to associate with the span. ~~Union[str, int]~~ |
## Span.\_\_getitem\_\_ {#getitem tag="method"}
## Span.\_\_getitem\_\_ {id="getitem",tag="method"}
Get a `Token` object.
@ -61,7 +61,7 @@ Get a `Span` object.
| `start_end` | The slice of the span to get. ~~Tuple[int, int]~~ |
| **RETURNS** | The span at `span[start : end]`. ~~Span~~ |
## Span.\_\_iter\_\_ {#iter tag="method"}
## Span.\_\_iter\_\_ {id="iter",tag="method"}
Iterate over `Token` objects.
@ -77,7 +77,7 @@ Iterate over `Token` objects.
| ---------- | --------------------------- |
| **YIELDS** | A `Token` object. ~~Token~~ |
## Span.\_\_len\_\_ {#len tag="method"}
## Span.\_\_len\_\_ {id="len",tag="method"}
Get the number of tokens in the span.
@ -93,7 +93,7 @@ Get the number of tokens in the span.
| ----------- | ----------------------------------------- |
| **RETURNS** | The number of tokens in the span. ~~int~~ |
## Span.set_extension {#set_extension tag="classmethod" new="2"}
## Span.set_extension {id="set_extension",tag="classmethod",version="2"}
Define a custom attribute on the `Span` which becomes available via `Span._`.
For details, see the documentation on
@ -118,7 +118,7 @@ For details, see the documentation on
| `setter` | Setter function that takes the `Span` and a value, and modifies the object. Is called when the user writes to the `Span._` attribute. ~~Optional[Callable[[Span, Any], None]]~~ |
| `force` | Force overwriting existing attribute. ~~bool~~ |
## Span.get_extension {#get_extension tag="classmethod" new="2"}
## Span.get_extension {id="get_extension",tag="classmethod",version="2"}
Look up a previously registered extension by name. Returns a 4-tuple
`(default, method, getter, setter)` if the extension is registered. Raises a
@ -138,7 +138,7 @@ Look up a previously registered extension by name. Returns a 4-tuple
| `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Span.has_extension {#has_extension tag="classmethod" new="2"}
## Span.has_extension {id="has_extension",tag="classmethod",version="2"}
Check whether an extension has been registered on the `Span` class.
@ -155,7 +155,7 @@ Check whether an extension has been registered on the `Span` class.
| `name` | Name of the extension to check. ~~str~~ |
| **RETURNS** | Whether the extension has been registered. ~~bool~~ |
## Span.remove_extension {#remove_extension tag="classmethod" new="2.0.12"}
## Span.remove_extension {id="remove_extension",tag="classmethod",version="2.0.12"}
Remove a previously registered extension.
@ -173,7 +173,7 @@ Remove a previously registered extension.
| `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Span.char_span {#char_span tag="method" new="2.2.4"}
## Span.char_span {id="char_span",tag="method",version="2.2.4"}
Create a `Span` object from the slice `span.text[start:end]`. Returns `None` if
the character indices don't map to a valid span.
@ -195,7 +195,7 @@ the character indices don't map to a valid span.
| `vector` | A meaning representation of the span. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
| **RETURNS** | The newly constructed object or `None`. ~~Optional[Span]~~ |
## Span.similarity {#similarity tag="method" model="vectors"}
## Span.similarity {id="similarity",tag="method",model="vectors"}
Make a semantic similarity estimate. The default estimate is cosine similarity
using an average of word vectors.
@ -216,7 +216,7 @@ using an average of word vectors.
| `other` | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ |
| **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ |
## Span.get_lca_matrix {#get_lca_matrix tag="method"}
## Span.get_lca_matrix {id="get_lca_matrix",tag="method"}
Calculates the lowest common ancestor matrix for a given `Span`. Returns LCA
matrix containing the integer index of the ancestor, or `-1` if no common
@ -235,7 +235,7 @@ ancestor is found, e.g. if span excludes a necessary ancestor.
| ----------- | --------------------------------------------------------------------------------------- |
| **RETURNS** | The lowest common ancestor matrix of the `Span`. ~~numpy.ndarray[ndim=2, dtype=int32]~~ |
## Span.to_array {#to_array tag="method" new="2"}
## Span.to_array {id="to_array",tag="method",version="2"}
Given a list of `M` attribute IDs, export the tokens to a numpy `ndarray` of
shape `(N, M)`, where `N` is the length of the document. The values will be
@ -256,7 +256,7 @@ shape `(N, M)`, where `N` is the length of the document. The values will be
| `attr_ids` | A list of attributes (int IDs or string names) or a single attribute (int ID or string name). ~~Union[int, str, List[Union[int, str]]]~~ |
| **RETURNS** | The exported attributes as a numpy array. ~~Union[numpy.ndarray[ndim=2, dtype=uint64], numpy.ndarray[ndim=1, dtype=uint64]]~~ |
## Span.ents {#ents tag="property" new="2.0.13" model="ner"}
## Span.ents {id="ents",tag="property",version="2.0.13",model="ner"}
The named entities that fall completely within the span. Returns a tuple of
`Span` objects.
@ -276,7 +276,7 @@ The named entities that fall completely within the span. Returns a tuple of
| ----------- | ----------------------------------------------------------------- |
| **RETURNS** | Entities in the span, one `Span` per entity. ~~Tuple[Span, ...]~~ |
## Span.noun_chunks {#noun_chunks tag="property" model="parser"}
## Span.noun_chunks {id="noun_chunks",tag="property",model="parser"}
Iterate over the base noun phrases in the span. Yields base noun-phrase `Span`
objects, if the document has been syntactically parsed. A base noun phrase, or
@ -302,7 +302,7 @@ raised.
| ---------- | --------------------------------- |
| **YIELDS** | Noun chunks in the span. ~~Span~~ |
## Span.as_doc {#as_doc tag="method"}
## Span.as_doc {id="as_doc",tag="method"}
Create a new `Doc` object corresponding to the `Span`, with a copy of the data.
@ -326,7 +326,7 @@ time.
| `array` | Precomputed array version of the original doc as generated by [`Doc.to_array`](/api/doc#to_array). ~~numpy.ndarray~~ |
| **RETURNS** | A `Doc` object of the `Span`'s content. ~~Doc~~ |
## Span.root {#root tag="property" model="parser"}
## Span.root {id="root",tag="property",model="parser"}
The token with the shortest path to the root of the sentence (or the root
itself). If multiple tokens are equally high in the tree, the first token is
@ -347,7 +347,7 @@ taken.
| ----------- | ------------------------- |
| **RETURNS** | The root token. ~~Token~~ |
## Span.conjuncts {#conjuncts tag="property" model="parser"}
## Span.conjuncts {id="conjuncts",tag="property",model="parser"}
A tuple of tokens coordinated to `span.root`.
@ -363,7 +363,7 @@ A tuple of tokens coordinated to `span.root`.
| ----------- | --------------------------------------------- |
| **RETURNS** | The coordinated tokens. ~~Tuple[Token, ...]~~ |
## Span.lefts {#lefts tag="property" model="parser"}
## Span.lefts {id="lefts",tag="property",model="parser"}
Tokens that are to the left of the span, whose heads are within the span.
@ -379,7 +379,7 @@ Tokens that are to the left of the span, whose heads are within the span.
| ---------- | ---------------------------------------------- |
| **YIELDS** | A left-child of a token of the span. ~~Token~~ |
## Span.rights {#rights tag="property" model="parser"}
## Span.rights {id="rights",tag="property",model="parser"}
Tokens that are to the right of the span, whose heads are within the span.
@ -395,7 +395,7 @@ Tokens that are to the right of the span, whose heads are within the span.
| ---------- | ----------------------------------------------- |
| **YIELDS** | A right-child of a token of the span. ~~Token~~ |
## Span.n_lefts {#n_lefts tag="property" model="parser"}
## Span.n_lefts {id="n_lefts",tag="property",model="parser"}
The number of tokens that are to the left of the span, whose heads are within
the span.
@ -411,7 +411,7 @@ the span.
| ----------- | ---------------------------------------- |
| **RETURNS** | The number of left-child tokens. ~~int~~ |
## Span.n_rights {#n_rights tag="property" model="parser"}
## Span.n_rights {id="n_rights",tag="property",model="parser"}
The number of tokens that are to the right of the span, whose heads are within
the span.
@ -427,7 +427,7 @@ the span.
| ----------- | ----------------------------------------- |
| **RETURNS** | The number of right-child tokens. ~~int~~ |
## Span.subtree {#subtree tag="property" model="parser"}
## Span.subtree {id="subtree",tag="property",model="parser"}
Tokens within the span and tokens which descend from them.
@ -443,7 +443,7 @@ Tokens within the span and tokens which descend from them.
| ---------- | ----------------------------------------------------------- |
| **YIELDS** | A token within the span, or a descendant from it. ~~Token~~ |
## Span.has_vector {#has_vector tag="property" model="vectors"}
## Span.has_vector {id="has_vector",tag="property",model="vectors"}
A boolean value indicating whether a word vector is associated with the object.
@ -458,7 +458,7 @@ A boolean value indicating whether a word vector is associated with the object.
| ----------- | ----------------------------------------------------- |
| **RETURNS** | Whether the span has a vector data attached. ~~bool~~ |
## Span.vector {#vector tag="property" model="vectors"}
## Span.vector {id="vector",tag="property",model="vectors"}
A real-valued meaning representation. Defaults to an average of the token
vectors.
@ -475,7 +475,7 @@ vectors.
| ----------- | ----------------------------------------------------------------------------------------------- |
| **RETURNS** | A 1-dimensional array representing the span's vector. ~~`numpy.ndarray[ndim=1, dtype=float32]~~ |
## Span.vector_norm {#vector_norm tag="property" model="vectors"}
## Span.vector_norm {id="vector_norm",tag="property",model="vectors"}
The L2 norm of the span's vector representation.
@ -492,7 +492,7 @@ The L2 norm of the span's vector representation.
| ----------- | --------------------------------------------------- |
| **RETURNS** | The L2 norm of the vector representation. ~~float~~ |
## Span.sent {#sent tag="property" model="sentences"}
## Span.sent {id="sent",tag="property",model="sentences"}
The sentence span that this span is a part of. This property is only available
when [sentence boundaries](/usage/linguistic-features#sbd) have been set on the
@ -520,7 +520,7 @@ sent = doc[sent.start : max(sent.end, span.end)]
| ----------- | ------------------------------------------------------- |
| **RETURNS** | The sentence span that this span is a part of. ~~Span~~ |
## Span.sents {#sents tag="property" model="sentences" new="3.2.1"}
## Span.sents {id="sents",tag="property",model="sentences",version="3.2.1"}
Returns a generator over the sentences the span belongs to. This property is
only available when [sentence boundaries](/usage/linguistic-features#sbd) have
@ -542,7 +542,7 @@ overlaps with will be returned.
| ----------- | -------------------------------------------------------------------------- |
| **RETURNS** | A generator yielding sentences this `Span` is a part of ~~Iterable[Span]~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------- |

50
website/docs/api/spancategorizer.md → website/docs/api/spancategorizer.mdx
View File

@ -2,7 +2,7 @@
title: SpanCategorizer
tag: class,experimental
source: spacy/pipeline/spancat.py
new: 3.1
version: 3.1
teaser: 'Pipeline component for labeling potentially overlapping spans of text'
api_base_class: /api/pipe
api_string_name: spancat
@ -16,7 +16,7 @@ that predicts zero or more labels for each candidate.
Predicted spans will be saved in a [`SpanGroup`](/api/spangroup) on the doc.
Individual span scores can be found in `spangroup.attrs["scores"]`.
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Predictions will be saved to `Doc.spans[spans_key]` as a
[`SpanGroup`](/api/spangroup). The scores for the spans in the `SpanGroup` will
@ -29,7 +29,7 @@ be saved in `SpanGroup.attrs["scores"]`.
| `Doc.spans[spans_key]` | The annotated spans. ~~SpanGroup~~ |
| `Doc.spans[spans_key].attrs["scores"]` | The score for each span in the `SpanGroup`. ~~Floats1d~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -65,7 +65,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/spancat.py
```
## SpanCategorizer.\_\_init\_\_ {#init tag="method"}
## SpanCategorizer.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -97,7 +97,7 @@ shortcut for this and instantiate the component using its string name and
| `threshold` | Minimum probability to consider a prediction positive. Spans with a positive prediction will be saved on the Doc. Defaults to `0.5`. ~~float~~ |
| `max_positive` | Maximum number of labels to consider positive per span. Defaults to `None`, indicating no limit. ~~Optional[int]~~ |
## SpanCategorizer.\_\_call\_\_ {#call tag="method"}
## SpanCategorizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -120,7 +120,7 @@ delegate to the [`predict`](/api/spancategorizer#predict) and
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## SpanCategorizer.pipe {#pipe tag="method"}
## SpanCategorizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -144,7 +144,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/spancategorizer#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## SpanCategorizer.initialize {#initialize tag="method"}
## SpanCategorizer.initialize {id="initialize",tag="method"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -181,7 +181,7 @@ config.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ |
## SpanCategorizer.predict {#predict tag="method"}
## SpanCategorizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects without
modifying them.
@ -198,7 +198,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. |
## SpanCategorizer.set_annotations {#set_annotations tag="method"}
## SpanCategorizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores.
@ -215,7 +215,7 @@ Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `SpanCategorizer.predict`. |
## SpanCategorizer.update {#update tag="method"}
## SpanCategorizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model.
@ -239,7 +239,7 @@ Delegates to [`predict`](/api/spancategorizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## SpanCategorizer.set_candidates {#set_candidates tag="method", new="3.3"}
## SpanCategorizer.set_candidates {id="set_candidates",tag="method", version="3.3"}
Use the suggester to add a list of [`Span`](/api/span) candidates to a list of
[`Doc`](/api/doc) objects. This method is intended to be used for debugging
@ -257,7 +257,7 @@ purposes.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `candidates_key` | Key of the Doc.spans dict to save the candidate spans under. ~~str~~ |
## SpanCategorizer.get_loss {#get_loss tag="method"}
## SpanCategorizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their
predicted scores.
@ -276,7 +276,7 @@ predicted scores.
| `spans_scores` | Scores representing the model's predictions. ~~Tuple[Ragged, Floats2d]~~ |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## SpanCategorizer.create_optimizer {#create_optimizer tag="method"}
## SpanCategorizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -291,7 +291,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## SpanCategorizer.use_params {#use_params tag="method, contextmanager"}
## SpanCategorizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model to use the given parameter values.
@ -307,7 +307,7 @@ Modify the pipe's model to use the given parameter values.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## SpanCategorizer.add_label {#add_label tag="method"}
## SpanCategorizer.add_label {id="add_label",tag="method"}
Add a new label to the pipe. Raises an error if the output dimension is already
set, or if the model has already been fully [initialized](#initialize). Note
@ -329,7 +329,7 @@ automatically.
| `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## SpanCategorizer.to_disk {#to_disk tag="method"}
## SpanCategorizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -346,7 +346,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## SpanCategorizer.from_disk {#from_disk tag="method"}
## SpanCategorizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -364,7 +364,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `SpanCategorizer` object. ~~SpanCategorizer~~ |
## SpanCategorizer.to_bytes {#to_bytes tag="method"}
## SpanCategorizer.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -381,7 +381,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `SpanCategorizer` object. ~~bytes~~ |
## SpanCategorizer.from_bytes {#from_bytes tag="method"}
## SpanCategorizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -400,7 +400,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `SpanCategorizer` object. ~~SpanCategorizer~~ |
## SpanCategorizer.labels {#labels tag="property"}
## SpanCategorizer.labels {id="labels",tag="property"}
The labels currently added to the component.
@ -415,7 +415,7 @@ The labels currently added to the component.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## SpanCategorizer.label_data {#label_data tag="property"}
## SpanCategorizer.label_data {id="label_data",tag="property"}
The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -433,7 +433,7 @@ the model with a pre-defined label set.
| ----------- | ---------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from
@ -451,9 +451,9 @@ serialization by passing in the string names via the `exclude` argument.
| `cfg` | The config file. You usually don't want to exclude this. |
| `model` | The binary model data. You usually don't want to exclude this. |
## Suggesters {#suggesters tag="registered functions" source="spacy/pipeline/spancat.py"}
## Suggesters {id="suggesters",tag="registered functions",source="spacy/pipeline/spancat.py"}
### spacy.ngram_suggester.v1 {#ngram_suggester}
### spacy.ngram_suggester.v1 {id="ngram_suggester"}
> #### Example Config
>
@ -471,7 +471,7 @@ integers. The array has two columns, indicating the start and end position.
| `sizes` | The phrase lengths to suggest. For example, `[1, 2]` will suggest phrases consisting of 1 or 2 tokens. ~~List[int]~~ |
| **CREATES** | The suggester function. ~~Callable[[Iterable[Doc], Optional[Ops]], Ragged]~~ |
### spacy.ngram_range_suggester.v1 {#ngram_range_suggester}
### spacy.ngram_range_suggester.v1 {id="ngram_range_suggester"}
> #### Example Config
>

33
website/docs/api/spangroup.md → website/docs/api/spangroup.mdx
View File

@ -2,7 +2,7 @@
title: SpanGroup
tag: class
source: spacy/tokens/span_group.pyx
new: 3
version: 3
---
A group of arbitrary, potentially overlapping [`Span`](/api/span) objects that
@ -13,7 +13,7 @@ into a `SpanGroup` object for you automatically on assignment. `SpanGroup`
objects behave similar to `list`s, so you can append `Span` objects to them or
access a member at a given index.
## SpanGroup.\_\_init\_\_ {#init tag="method"}
## SpanGroup.\_\_init\_\_ {id="init",tag="method"}
Create a `SpanGroup`.
@ -42,7 +42,7 @@ Create a `SpanGroup`.
| `attrs` | Optional JSON-serializable attributes to attach to the span group. ~~Dict[str, Any]~~ |
| `spans` | The spans to add to the span group. ~~Iterable[Span]~~ |
## SpanGroup.doc {#doc tag="property"}
## SpanGroup.doc {id="doc",tag="property"}
The [`Doc`](/api/doc) object the span group is referring to.
@ -68,7 +68,7 @@ the scope of your function.
| ----------- | ------------------------------- |
| **RETURNS** | The reference document. ~~Doc~~ |
## SpanGroup.has_overlap {#has_overlap tag="property"}
## SpanGroup.has_overlap {id="has_overlap",tag="property"}
Check whether the span group contains overlapping spans.
@ -86,7 +86,7 @@ Check whether the span group contains overlapping spans.
| ----------- | -------------------------------------------------- |
| **RETURNS** | Whether the span group contains overlaps. ~~bool~~ |
## SpanGroup.\_\_len\_\_ {#len tag="method"}
## SpanGroup.\_\_len\_\_ {id="len",tag="method"}
Get the number of spans in the group.
@ -102,7 +102,7 @@ Get the number of spans in the group.
| ----------- | ----------------------------------------- |
| **RETURNS** | The number of spans in the group. ~~int~~ |
## SpanGroup.\_\_getitem\_\_ {#getitem tag="method"}
## SpanGroup.\_\_getitem\_\_ {id="getitem",tag="method"}
Get a span from the group. Note that a copy of the span is returned, so if any
changes are made to this span, they are not reflected in the corresponding
@ -125,7 +125,7 @@ changes to be reflected in the span group.
| `i` | The item index. ~~int~~ |
| **RETURNS** | The span at the given index. ~~Span~~ |
## SpanGroup.\_\_setitem\_\_ {#setitem tag="method", new="3.3"}
## SpanGroup.\_\_setitem\_\_ {id="setitem",tag="method", version="3.3"}
Set a span in the span group.
@ -144,7 +144,7 @@ Set a span in the span group.
| `i` | The item index. ~~int~~ |
| `span` | The new value. ~~Span~~ |
## SpanGroup.\_\_delitem\_\_ {#delitem tag="method", new="3.3"}
## SpanGroup.\_\_delitem\_\_ {id="delitem",tag="method", version="3.3"}
Delete a span from the span group.
@ -161,7 +161,7 @@ Delete a span from the span group.
| ---- | ----------------------- |
| `i` | The item index. ~~int~~ |
## SpanGroup.\_\_add\_\_ {#add tag="method", new="3.3"}
## SpanGroup.\_\_add\_\_ {id="add",tag="method", version="3.3"}
Concatenate the current span group with another span group and return the result
in a new span group. Any `attrs` from the first span group will have precedence
@ -182,7 +182,7 @@ over `attrs` in the second.
| `other` | The span group or spans to concatenate. ~~Union[SpanGroup, Iterable[Span]]~~ |
| **RETURNS** | The new span group. ~~SpanGroup~~ |
## SpanGroup.\_\_iadd\_\_ {#iadd tag="method", new="3.3"}
## SpanGroup.\_\_iadd\_\_ {id="iadd",tag="method", version="3.3"}
Append an iterable of spans or the content of a span group to the current span
group. Any `attrs` in the other span group will be added for keys that are not
@ -202,7 +202,7 @@ already present in the current span group.
| `other` | The span group or spans to append. ~~Union[SpanGroup, Iterable[Span]]~~ |
| **RETURNS** | The span group. ~~SpanGroup~~ |
## SpanGroup.\_\_iter\_\_ {#iter tag="method" new="3.5"}
## SpanGroup.\_\_iter\_\_ {id="iter",tag="method",version="3.5"}
Iterate over the spans in this span group.
@ -219,7 +219,8 @@ Iterate over the spans in this span group.
| ---------- | ----------------------------------- |
| **YIELDS** | A span in this span group. ~~Span~~ |
## SpanGroup.append {#append tag="method"}
## SpanGroup.append {id="append",tag="method"}
Add a [`Span`](/api/span) object to the group. The span must refer to the same
[`Doc`](/api/doc) object as the span group.
@ -237,7 +238,7 @@ Add a [`Span`](/api/span) object to the group. The span must refer to the same
| ------ | ---------------------------- |
| `span` | The span to append. ~~Span~~ |
## SpanGroup.extend {#extend tag="method"}
## SpanGroup.extend {id="extend",tag="method"}
Add multiple [`Span`](/api/span) objects or contents of another `SpanGroup` to
the group. All spans must refer to the same [`Doc`](/api/doc) object as the span
@ -258,7 +259,7 @@ group.
| ------- | -------------------------------------------------------- |
| `spans` | The spans to add. ~~Union[SpanGroup, Iterable["Span"]]~~ |
## SpanGroup.copy {#copy tag="method", new="3.3"}
## SpanGroup.copy {id="copy",tag="method", version="3.3"}
Return a copy of the span group.
@ -277,7 +278,7 @@ Return a copy of the span group.
| `doc` | The document to which the copy is bound. Defaults to `None` for the current doc. ~~Optional[Doc]~~ |
| **RETURNS** | A copy of the `SpanGroup` object. ~~SpanGroup~~ |
## SpanGroup.to_bytes {#to_bytes tag="method"}
## SpanGroup.to_bytes {id="to_bytes",tag="method"}
Serialize the span group to a bytestring.
@ -293,7 +294,7 @@ Serialize the span group to a bytestring.
| ----------- | ------------------------------------- |
| **RETURNS** | The serialized `SpanGroup`. ~~bytes~~ |
## SpanGroup.from_bytes {#from_bytes tag="method"}
## SpanGroup.from_bytes {id="from_bytes",tag="method"}
Load the span group from a bytestring. Modifies the object in place and returns
it.

40
website/docs/api/spanruler.md → website/docs/api/spanruler.mdx
View File

@ -2,7 +2,7 @@
title: SpanRuler
tag: class
source: spacy/pipeline/span_ruler.py
new: 3.3
version: 3.3
teaser: 'Pipeline component for rule-based span and named entity recognition'
api_string_name: span_ruler
api_trainable: false
@ -13,7 +13,7 @@ The span ruler lets you add spans to [`Doc.spans`](/api/doc#spans) and/or
usage examples, see the docs on
[rule-based span matching](/usage/rule-based-matching#spanruler).
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Matches will be saved to `Doc.spans[spans_key]` as a
[`SpanGroup`](/api/spangroup) and/or to `Doc.ents`, where the annotation is
@ -28,7 +28,7 @@ saved in the `Token.ent_type` and `Token.ent_iob` fields.
| `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ |
| `Token.ent_type_` | The label part of the named entity tag. ~~str~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -62,7 +62,7 @@ how the component should be configured. You can override its settings via the
%%GITHUB_SPACY/spacy/pipeline/span_ruler.py
```
## SpanRuler.\_\_init\_\_ {#init tag="method"}
## SpanRuler.\_\_init\_\_ {id="init",tag="method"}
Initialize the span ruler. If patterns are supplied here, they need to be a list
of dictionaries with a `"label"` and `"pattern"` key. A pattern can either be a
@ -95,7 +95,7 @@ token pattern (list) or a phrase pattern (string). For example:
| `overwrite` | Whether to remove any existing spans under `Doc.spans[spans key]` if `spans_key` is set, or to remove any ents under `Doc.ents` if `annotate_ents` is set. Defaults to `True`. ~~bool~~ |
| `scorer` | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for `Doc.spans[spans_key]` with overlapping spans allowed. ~~Optional[Callable]~~ |
## SpanRuler.initialize {#initialize tag="method"}
## SpanRuler.initialize {id="initialize",tag="method"}
Initialize the component with data and used before training to load in rules
from a [pattern file](/usage/rule-based-matching/#spanruler-files). This method
@ -127,7 +127,7 @@ config. Any existing patterns are removed on initialization.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `patterns` | The list of patterns. Defaults to `None`. ~~Optional[Sequence[Dict[str, Union[str, List[Dict[str, Any]]]]]]~~ |
## SpanRuler.\_\len\_\_ {#len tag="method"}
## SpanRuler.\_\_len\_\_ {id="len",tag="method"}
The number of all patterns added to the span ruler.
@ -144,7 +144,7 @@ The number of all patterns added to the span ruler.
| ----------- | ------------------------------- |
| **RETURNS** | The number of patterns. ~~int~~ |
## SpanRuler.\_\_contains\_\_ {#contains tag="method"}
## SpanRuler.\_\_contains\_\_ {id="contains",tag="method"}
Whether a label is present in the patterns.
@ -162,7 +162,7 @@ Whether a label is present in the patterns.
| `label` | The label to check. ~~str~~ |
| **RETURNS** | Whether the span ruler contains the label. ~~bool~~ |
## SpanRuler.\_\_call\_\_ {#call tag="method"}
## SpanRuler.\_\_call\_\_ {id="call",tag="method"}
Find matches in the `Doc` and add them to `doc.spans[span_key]` and/or
`doc.ents`. Typically, this happens automatically after the component has been
@ -186,7 +186,7 @@ will be removed.
| `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ |
| **RETURNS** | The modified `Doc` with added spans/entities. ~~Doc~~ |
## SpanRuler.add_patterns {#add_patterns tag="method"}
## SpanRuler.add_patterns {id="add_patterns",tag="method"}
Add patterns to the span ruler. A pattern can either be a token pattern (list of
dicts) or a phrase pattern (string). For more details, see the usage guide on
@ -207,7 +207,7 @@ dicts) or a phrase pattern (string). For more details, see the usage guide on
| ---------- | ---------------------------------------------------------------- |
| `patterns` | The patterns to add. ~~List[Dict[str, Union[str, List[dict]]]]~~ |
## SpanRuler.remove {#remove tag="method"}
## SpanRuler.remove {id="remove",tag="method"}
Remove patterns by label from the span ruler. A `ValueError` is raised if the
label does not exist in any patterns.
@ -225,7 +225,7 @@ label does not exist in any patterns.
| ------- | -------------------------------------- |
| `label` | The label of the pattern rule. ~~str~~ |
## SpanRuler.remove_by_id {#remove_by_id tag="method"}
## SpanRuler.remove_by_id {id="remove_by_id",tag="method"}
Remove patterns by ID from the span ruler. A `ValueError` is raised if the ID
does not exist in any patterns.
@ -243,7 +243,7 @@ does not exist in any patterns.
| ------------ | ----------------------------------- |
| `pattern_id` | The ID of the pattern rule. ~~str~~ |
## SpanRuler.clear {#clear tag="method"}
## SpanRuler.clear {id="clear",tag="method"}
Remove all patterns the span ruler.
@ -256,7 +256,7 @@ Remove all patterns the span ruler.
> ruler.clear()
> ```
## SpanRuler.to_disk {#to_disk tag="method"}
## SpanRuler.to_disk {id="to_disk",tag="method"}
Save the span ruler patterns to a directory. The patterns will be saved as
newline-delimited JSON (JSONL).
@ -272,7 +272,7 @@ newline-delimited JSON (JSONL).
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## SpanRuler.from_disk {#from_disk tag="method"}
## SpanRuler.from_disk {id="from_disk",tag="method"}
Load the span ruler from a path.
@ -288,7 +288,7 @@ Load the span ruler from a path.
| `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The modified `SpanRuler` object. ~~SpanRuler~~ |
## SpanRuler.to_bytes {#to_bytes tag="method"}
## SpanRuler.to_bytes {id="to_bytes",tag="method"}
Serialize the span ruler to a bytestring.
@ -303,7 +303,7 @@ Serialize the span ruler to a bytestring.
| ----------- | ---------------------------------- |
| **RETURNS** | The serialized patterns. ~~bytes~~ |
## SpanRuler.from_bytes {#from_bytes tag="method"}
## SpanRuler.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -320,7 +320,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `bytes_data` | The bytestring to load. ~~bytes~~ |
| **RETURNS** | The modified `SpanRuler` object. ~~SpanRuler~~ |
## SpanRuler.labels {#labels tag="property"}
## SpanRuler.labels {id="labels",tag="property"}
All labels present in the match patterns.
@ -328,7 +328,7 @@ All labels present in the match patterns.
| ----------- | -------------------------------------- |
| **RETURNS** | The string labels. ~~Tuple[str, ...]~~ |
## SpanRuler.ids {#ids tag="property"}
## SpanRuler.ids {id="ids",tag="property"}
All IDs present in the `id` property of the match patterns.
@ -336,7 +336,7 @@ All IDs present in the `id` property of the match patterns.
| ----------- | ----------------------------------- |
| **RETURNS** | The string IDs. ~~Tuple[str, ...]~~ |
## SpanRuler.patterns {#patterns tag="property"}
## SpanRuler.patterns {id="patterns",tag="property"}
All patterns that were added to the span ruler.
@ -344,7 +344,7 @@ All patterns that were added to the span ruler.
| ----------- | ---------------------------------------------------------------------------------------- |
| **RETURNS** | The original patterns, one dictionary per pattern. ~~List[Dict[str, Union[str, dict]]]~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| ---------------- | -------------------------------------------------------------------------------- |

24
website/docs/api/stringstore.md → website/docs/api/stringstore.mdx
View File

@ -8,7 +8,7 @@ Look up strings by 64-bit hashes. As of v2.0, spaCy uses hash values instead of
integer IDs. This ensures that strings always map to the same ID, even from
different `StringStores`.
## StringStore.\_\_init\_\_ {#init tag="method"}
## StringStore.\_\_init\_\_ {id="init",tag="method"}
Create the `StringStore`.
@ -23,7 +23,7 @@ Create the `StringStore`.
| --------- | ---------------------------------------------------------------------- |
| `strings` | A sequence of strings to add to the store. ~~Optional[Iterable[str]]~~ |
## StringStore.\_\_len\_\_ {#len tag="method"}
## StringStore.\_\_len\_\_ {id="len",tag="method"}
Get the number of strings in the store.
@ -38,7 +38,7 @@ Get the number of strings in the store.
| ----------- | ------------------------------------------- |
| **RETURNS** | The number of strings in the store. ~~int~~ |
## StringStore.\_\_getitem\_\_ {#getitem tag="method"}
## StringStore.\_\_getitem\_\_ {id="getitem",tag="method"}
Retrieve a string from a given hash, or vice versa.
@ -56,7 +56,7 @@ Retrieve a string from a given hash, or vice versa.
| `string_or_id` | The value to encode. ~~Union[bytes, str, int]~~ |
| **RETURNS** | The value to be retrieved. ~~Union[str, int]~~ |
## StringStore.\_\_contains\_\_ {#contains tag="method"}
## StringStore.\_\_contains\_\_ {id="contains",tag="method"}
Check whether a string is in the store.
@ -73,7 +73,7 @@ Check whether a string is in the store.
| `string` | The string to check. ~~str~~ |
| **RETURNS** | Whether the store contains the string. ~~bool~~ |
## StringStore.\_\_iter\_\_ {#iter tag="method"}
## StringStore.\_\_iter\_\_ {id="iter",tag="method"}
Iterate over the strings in the store, in order. Note that a newly initialized
store will always include an empty string `""` at position `0`.
@ -90,7 +90,7 @@ store will always include an empty string `""` at position `0`.
| ---------- | ------------------------------ |
| **YIELDS** | A string in the store. ~~str~~ |
## StringStore.add {#add tag="method" new="2"}
## StringStore.add {id="add",tag="method",version="2"}
Add a string to the `StringStore`.
@ -110,7 +110,7 @@ Add a string to the `StringStore`.
| `string` | The string to add. ~~str~~ |
| **RETURNS** | The string's hash value. ~~int~~ |
## StringStore.to_disk {#to_disk tag="method" new="2"}
## StringStore.to_disk {id="to_disk",tag="method",version="2"}
Save the current state to a directory.
@ -124,7 +124,7 @@ Save the current state to a directory.
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## StringStore.from_disk {#from_disk tag="method" new="2"}
## StringStore.from_disk {id="from_disk",tag="method",version="2"}
Loads state from a directory. Modifies the object in place and returns it.
@ -140,7 +140,7 @@ Loads state from a directory. Modifies the object in place and returns it.
| `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The modified `StringStore` object. ~~StringStore~~ |
## StringStore.to_bytes {#to_bytes tag="method"}
## StringStore.to_bytes {id="to_bytes",tag="method"}
Serialize the current state to a binary string.
@ -154,7 +154,7 @@ Serialize the current state to a binary string.
| ----------- | ---------------------------------------------------------- |
| **RETURNS** | The serialized form of the `StringStore` object. ~~bytes~~ |
## StringStore.from_bytes {#from_bytes tag="method"}
## StringStore.from_bytes {id="from_bytes",tag="method"}
Load state from a binary string.
@ -171,9 +171,9 @@ Load state from a binary string.
| `bytes_data` | The data to load from. ~~bytes~~ |
| **RETURNS** | The `StringStore` object. ~~StringStore~~ |
## Utilities {#util}
## Utilities {id="util"}
### strings.hash_string {#hash_string tag="function"}
### strings.hash_string {id="hash_string",tag="function"}
Get a 64-bit hash for a given string.

42
website/docs/api/tagger.md → website/docs/api/tagger.mdx
View File

@ -14,7 +14,7 @@ part-of-speech tag set.
In the pre-trained pipelines, the tag schemas vary by language; see the
[individual model pages](/models) for details.
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Predictions are assigned to `Token.tag`.
@ -23,7 +23,7 @@ Predictions are assigned to `Token.tag`.
| `Token.tag` | The part of speech (hash). ~~int~~ |
| `Token.tag_` | The part of speech. ~~str~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -51,7 +51,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/tagger.pyx
```
## Tagger.\_\_init\_\_ {#init tag="method"}
## Tagger.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -81,7 +81,7 @@ shortcut for this and instantiate the component using its string name and
| `overwrite` <Tag variant="new">3.2</Tag> | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ |
| `scorer` <Tag variant="new">3.2</Tag> | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attribute `"tag"`. ~~Optional[Callable]~~ |
## Tagger.\_\_call\_\_ {#call tag="method"}
## Tagger.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -104,7 +104,7 @@ and all pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## Tagger.pipe {#pipe tag="method"}
## Tagger.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -127,7 +127,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/tagger#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## Tagger.initialize {#initialize tag="method" new="3"}
## Tagger.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -170,7 +170,7 @@ This method was previously called `begin_training`.
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ |
## Tagger.predict {#predict tag="method"}
## Tagger.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
modifying them.
@ -187,7 +187,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. |
## Tagger.set_annotations {#set_annotations tag="method"}
## Tagger.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -204,7 +204,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `Tagger.predict`. |
## Tagger.update {#update tag="method"}
## Tagger.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model.
@ -228,7 +228,7 @@ Delegates to [`predict`](/api/tagger#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Tagger.rehearse {#rehearse tag="method,experimental" new="3"}
## Tagger.rehearse {id="rehearse",tag="method,experimental",version="3"}
Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the
current model to make predictions similar to an initial model, to try to address
@ -251,7 +251,7 @@ the "catastrophic forgetting" problem. This feature is experimental.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Tagger.get_loss {#get_loss tag="method"}
## Tagger.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their
predicted scores.
@ -270,7 +270,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## Tagger.create_optimizer {#create_optimizer tag="method"}
## Tagger.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -285,7 +285,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## Tagger.use_params {#use_params tag="method, contextmanager"}
## Tagger.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model, to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -302,7 +302,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## Tagger.add_label {#add_label tag="method"}
## Tagger.add_label {id="add_label",tag="method"}
Add a new label to the pipe. Raises an error if the output dimension is already
set, or if the model has already been fully [initialized](#initialize). Note
@ -324,7 +324,7 @@ automatically.
| `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## Tagger.to_disk {#to_disk tag="method"}
## Tagger.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -341,7 +341,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Tagger.from_disk {#from_disk tag="method"}
## Tagger.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -359,7 +359,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Tagger` object. ~~Tagger~~ |
## Tagger.to_bytes {#to_bytes tag="method"}
## Tagger.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -376,7 +376,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Tagger` object. ~~bytes~~ |
## Tagger.from_bytes {#from_bytes tag="method"}
## Tagger.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -395,7 +395,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Tagger` object. ~~Tagger~~ |
## Tagger.labels {#labels tag="property"}
## Tagger.labels {id="labels",tag="property"}
The labels currently added to the component.
@ -410,7 +410,7 @@ The labels currently added to the component.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## Tagger.label_data {#label_data tag="property" new="3"}
## Tagger.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -428,7 +428,7 @@ pre-defined label set.
| ----------- | ---------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

46
website/docs/api/textcategorizer.md → website/docs/api/textcategorizer.mdx
View File

@ -2,7 +2,7 @@
title: TextCategorizer
tag: class
source: spacy/pipeline/textcat.py
new: 2
version: 2
teaser: 'Pipeline component for text classification'
api_base_class: /api/pipe
api_string_name: textcat
@ -29,7 +29,7 @@ only.
</Infobox>
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
Predictions will be saved to `doc.cats` as a dictionary, where the key is the
name of the category and the value is a score between 0 and 1 (inclusive). For
@ -49,7 +49,7 @@ supported.
| ---------- | ------------------------------------- |
| `Doc.cats` | Category scores. ~~Dict[str, float]~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -93,7 +93,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/textcat_multilabel.py
```
## TextCategorizer.\_\_init\_\_ {#init tag="method"}
## TextCategorizer.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -125,7 +125,7 @@ shortcut for this and instantiate the component using its string name and
| `threshold` | Cutoff to consider a prediction "positive", relevant for `textcat_multilabel` when calculating accuracy scores. ~~float~~ |
| `scorer` | The scoring method. Defaults to [`Scorer.score_cats`](/api/scorer#score_cats) for the attribute `"cats"`. ~~Optional[Callable]~~ |
## TextCategorizer.\_\_call\_\_ {#call tag="method"}
## TextCategorizer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -148,7 +148,7 @@ delegate to the [`predict`](/api/textcategorizer#predict) and
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## TextCategorizer.pipe {#pipe tag="method"}
## TextCategorizer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -172,7 +172,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/textcategorizer#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## TextCategorizer.initialize {#initialize tag="method" new="3"}
## TextCategorizer.initialize {id="initialize",tag="method",version="3"}
Initialize the component for training. `get_examples` should be a function that
returns an iterable of [`Example`](/api/example) objects. **At least one example
@ -217,7 +217,7 @@ This method was previously called `begin_training`.
| `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ |
| `positive_label` | The positive label for a binary task with exclusive classes, `None` otherwise and by default. This parameter is only used during scoring. It is not available when using the `textcat_multilabel` component. ~~Optional[str]~~ |
## TextCategorizer.predict {#predict tag="method"}
## TextCategorizer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects without
modifying them.
@ -234,7 +234,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. |
## TextCategorizer.set_annotations {#set_annotations tag="method"}
## TextCategorizer.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores.
@ -251,7 +251,7 @@ Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `TextCategorizer.predict`. |
## TextCategorizer.update {#update tag="method"}
## TextCategorizer.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model.
@ -275,7 +275,7 @@ Delegates to [`predict`](/api/textcategorizer#predict) and
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## TextCategorizer.rehearse {#rehearse tag="method,experimental" new="3"}
## TextCategorizer.rehearse {id="rehearse",tag="method,experimental",version="3"}
Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the
current model to make predictions similar to an initial model to try to address
@ -298,7 +298,7 @@ the "catastrophic forgetting" problem. This feature is experimental.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## TextCategorizer.get_loss {#get_loss tag="method"}
## TextCategorizer.get_loss {id="get_loss",tag="method"}
Find the loss and gradient of loss for the batch of documents and their
predicted scores.
@ -317,7 +317,7 @@ predicted scores.
| `scores` | Scores representing the model's predictions. |
| **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ |
## TextCategorizer.score {#score tag="method" new="3"}
## TextCategorizer.score {id="score",tag="method",version="3"}
Score a batch of examples.
@ -333,7 +333,7 @@ Score a batch of examples.
| _keyword-only_ | |
| **RETURNS** | The scores, produced by [`Scorer.score_cats`](/api/scorer#score_cats). ~~Dict[str, Union[float, Dict[str, float]]]~~ |
## TextCategorizer.create_optimizer {#create_optimizer tag="method"}
## TextCategorizer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -348,7 +348,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## TextCategorizer.use_params {#use_params tag="method, contextmanager"}
## TextCategorizer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model to use the given parameter values.
@ -364,7 +364,7 @@ Modify the pipe's model to use the given parameter values.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## TextCategorizer.add_label {#add_label tag="method"}
## TextCategorizer.add_label {id="add_label",tag="method"}
Add a new label to the pipe. Raises an error if the output dimension is already
set, or if the model has already been fully [initialized](#initialize). Note
@ -386,7 +386,7 @@ automatically.
| `label` | The label to add. ~~str~~ |
| **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ |
## TextCategorizer.to_disk {#to_disk tag="method"}
## TextCategorizer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -403,7 +403,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## TextCategorizer.from_disk {#from_disk tag="method"}
## TextCategorizer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -421,7 +421,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `TextCategorizer` object. ~~TextCategorizer~~ |
## TextCategorizer.to_bytes {#to_bytes tag="method"}
## TextCategorizer.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -438,7 +438,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `TextCategorizer` object. ~~bytes~~ |
## TextCategorizer.from_bytes {#from_bytes tag="method"}
## TextCategorizer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -457,7 +457,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `TextCategorizer` object. ~~TextCategorizer~~ |
## TextCategorizer.labels {#labels tag="property"}
## TextCategorizer.labels {id="labels",tag="property"}
The labels currently added to the component.
@ -472,7 +472,7 @@ The labels currently added to the component.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ |
## TextCategorizer.label_data {#label_data tag="property" new="3"}
## TextCategorizer.label_data {id="label_data",tag="property",version="3"}
The labels currently added to the component and their internal meta information.
This is the data generated by [`init labels`](/api/cli#init-labels) and used by
@ -490,7 +490,7 @@ the model with a pre-defined label set.
| ----------- | ---------------------------------------------------------- |
| **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

32
website/docs/api/tok2vec.md → website/docs/api/tok2vec.mdx
View File

@ -1,7 +1,7 @@
---
title: Tok2Vec
source: spacy/pipeline/tok2vec.py
new: 3
version: 3
teaser: null
api_base_class: /api/pipe
api_string_name: tok2vec
@ -23,7 +23,7 @@ components can backpropagate to the shared weights. This implementation is used
because it allows us to avoid relying on object identity within the models to
achieve the parameter sharing.
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -48,7 +48,7 @@ architectures and their arguments and hyperparameters.
%%GITHUB_SPACY/spacy/pipeline/tok2vec.py
```
## Tok2Vec.\_\_init\_\_ {#init tag="method"}
## Tok2Vec.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -75,7 +75,7 @@ shortcut for this and instantiate the component using its string name and
| `model` | The Thinc [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. ~~Model[List[Doc], List[Floats2d]~~ |
| `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ |
## Tok2Vec.\_\_call\_\_ {#call tag="method"}
## Tok2Vec.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document and add context-sensitive embeddings to the
`Doc.tensor` attribute, allowing them to be used as features by downstream
@ -100,7 +100,7 @@ pipeline components are applied to the `Doc` in order. Both
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## Tok2Vec.pipe {#pipe tag="method"}
## Tok2Vec.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -123,7 +123,7 @@ and [`set_annotations`](/api/tok2vec#set_annotations) methods.
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## Tok2Vec.initialize {#initialize tag="method"}
## Tok2Vec.initialize {id="initialize",tag="method"}
Initialize the component for training and return an
[`Optimizer`](https://thinc.ai/docs/api-optimizers). `get_examples` should be a
@ -148,7 +148,7 @@ by [`Language.initialize`](/api/language#initialize).
| _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## Tok2Vec.predict {#predict tag="method"}
## Tok2Vec.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects without
modifying them.
@ -165,7 +165,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. |
## Tok2Vec.set_annotations {#set_annotations tag="method"}
## Tok2Vec.set_annotations {id="set_annotations",tag="method"}
Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
@ -182,7 +182,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `Tok2Vec.predict`. |
## Tok2Vec.update {#update tag="method"}
## Tok2Vec.update {id="update",tag="method"}
Learn from a batch of [`Example`](/api/example) objects containing the
predictions and gold-standard annotations, and update the component's model.
@ -205,7 +205,7 @@ Delegates to [`predict`](/api/tok2vec#predict).
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Tok2Vec.create_optimizer {#create_optimizer tag="method"}
## Tok2Vec.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -220,7 +220,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## Tok2Vec.use_params {#use_params tag="method, contextmanager"}
## Tok2Vec.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -237,7 +237,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## Tok2Vec.to_disk {#to_disk tag="method"}
## Tok2Vec.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -254,7 +254,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Tok2Vec.from_disk {#from_disk tag="method"}
## Tok2Vec.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -272,7 +272,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Tok2Vec` object. ~~Tok2Vec~~ |
## Tok2Vec.to_bytes {#to_bytes tag="method"}
## Tok2Vec.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -289,7 +289,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Tok2Vec` object. ~~bytes~~ |
## Tok2Vec.from_bytes {#from_bytes tag="method"}
## Tok2Vec.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -308,7 +308,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Tok2Vec` object. ~~Tok2Vec~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

48
website/docs/api/token.md → website/docs/api/token.mdx
View File

@ -5,7 +5,7 @@ tag: class
source: spacy/tokens/token.pyx
---
## Token.\_\_init\_\_ {#init tag="method"}
## Token.\_\_init\_\_ {id="init",tag="method"}
Construct a `Token` object.
@ -23,7 +23,7 @@ Construct a `Token` object.
| `doc` | The parent document. ~~Doc~~ |
| `offset` | The index of the token within the document. ~~int~~ |
## Token.\_\_len\_\_ {#len tag="method"}
## Token.\_\_len\_\_ {id="len",tag="method"}
The number of unicode characters in the token, i.e. `token.text`.
@ -39,7 +39,7 @@ The number of unicode characters in the token, i.e. `token.text`.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The number of unicode characters in the token. ~~int~~ |
## Token.set_extension {#set_extension tag="classmethod" new="2"}
## Token.set_extension {id="set_extension",tag="classmethod",version="2"}
Define a custom attribute on the `Token` which becomes available via `Token._`.
For details, see the documentation on
@ -64,7 +64,7 @@ For details, see the documentation on
| `setter` | Setter function that takes the `Token` and a value, and modifies the object. Is called when the user writes to the `Token._` attribute. ~~Optional[Callable[[Token, Any], None]]~~ |
| `force` | Force overwriting existing attribute. ~~bool~~ |
## Token.get_extension {#get_extension tag="classmethod" new="2"}
## Token.get_extension {id="get_extension",tag="classmethod",version="2"}
Look up a previously registered extension by name. Returns a 4-tuple
`(default, method, getter, setter)` if the extension is registered. Raises a
@ -84,7 +84,7 @@ Look up a previously registered extension by name. Returns a 4-tuple
| `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Token.has_extension {#has_extension tag="classmethod" new="2"}
## Token.has_extension {id="has_extension",tag="classmethod",version="2"}
Check whether an extension has been registered on the `Token` class.
@ -101,7 +101,7 @@ Check whether an extension has been registered on the `Token` class.
| `name` | Name of the extension to check. ~~str~~ |
| **RETURNS** | Whether the extension has been registered. ~~bool~~ |
## Token.remove_extension {#remove_extension tag="classmethod" new=""2.0.11""}
## Token.remove_extension {id="remove_extension",tag="classmethod",version="2.0.11"}
Remove a previously registered extension.
@ -119,7 +119,7 @@ Remove a previously registered extension.
| `name` | Name of the extension. ~~str~~ |
| **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ |
## Token.check_flag {#check_flag tag="method"}
## Token.check_flag {id="check_flag",tag="method"}
Check the value of a boolean flag.
@ -137,7 +137,7 @@ Check the value of a boolean flag.
| `flag_id` | The attribute ID of the flag to check. ~~int~~ |
| **RETURNS** | Whether the flag is set. ~~bool~~ |
## Token.similarity {#similarity tag="method" model="vectors"}
## Token.similarity {id="similarity",tag="method",model="vectors"}
Compute a semantic similarity estimate. Defaults to cosine over vectors.
@ -155,7 +155,7 @@ Compute a semantic similarity estimate. Defaults to cosine over vectors.
| other | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ |
| **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ |
## Token.nbor {#nbor tag="method"}
## Token.nbor {id="nbor",tag="method"}
Get a neighboring token.
@ -172,7 +172,7 @@ Get a neighboring token.
| `i` | The relative position of the token to get. Defaults to `1`. ~~int~~ |
| **RETURNS** | The token at position `self.doc[self.i+i]`. ~~Token~~ |
## Token.set_morph {#set_morph tag="method"}
## Token.set_morph {id="set_morph",tag="method"}
Set the morphological analysis from a UD FEATS string, hash value of a UD FEATS
string, features dict or `MorphAnalysis`. The value `None` can be used to reset
@ -191,7 +191,7 @@ the morph to an unset state.
| -------- | --------------------------------------------------------------------------------- |
| features | The morphological features to set. ~~Union[int, dict, str, MorphAnalysis, None]~~ |
## Token.has_morph {#has_morph tag="method"}
## Token.has_morph {id="has_morph",tag="method"}
Check whether the token has annotated morph information. Return `False` when the
morph annotation is unset/missing.
@ -200,7 +200,7 @@ morph annotation is unset/missing.
| ----------- | --------------------------------------------- |
| **RETURNS** | Whether the morph annotation is set. ~~bool~~ |
## Token.is_ancestor {#is_ancestor tag="method" model="parser"}
## Token.is_ancestor {id="is_ancestor",tag="method",model="parser"}
Check whether this token is a parent, grandparent, etc. of another in the
dependency tree.
@ -219,7 +219,7 @@ dependency tree.
| descendant | Another token. ~~Token~~ |
| **RETURNS** | Whether this token is the ancestor of the descendant. ~~bool~~ |
## Token.ancestors {#ancestors tag="property" model="parser"}
## Token.ancestors {id="ancestors",tag="property",model="parser"}
A sequence of the token's syntactic ancestors (parents, grandparents, etc).
@ -237,7 +237,7 @@ A sequence of the token's syntactic ancestors (parents, grandparents, etc).
| ---------- | ------------------------------------------------------------------------------- |
| **YIELDS** | A sequence of ancestor tokens such that `ancestor.is_ancestor(self)`. ~~Token~~ |
## Token.conjuncts {#conjuncts tag="property" model="parser"}
## Token.conjuncts {id="conjuncts",tag="property",model="parser"}
A tuple of coordinated tokens, not including the token itself.
@ -253,7 +253,7 @@ A tuple of coordinated tokens, not including the token itself.
| ----------- | --------------------------------------------- |
| **RETURNS** | The coordinated tokens. ~~Tuple[Token, ...]~~ |
## Token.children {#children tag="property" model="parser"}
## Token.children {id="children",tag="property",model="parser"}
A sequence of the token's immediate syntactic children.
@ -269,7 +269,7 @@ A sequence of the token's immediate syntactic children.
| ---------- | ------------------------------------------------------- |
| **YIELDS** | A child token such that `child.head == self`. ~~Token~~ |
## Token.lefts {#lefts tag="property" model="parser"}
## Token.lefts {id="lefts",tag="property",model="parser"}
The leftward immediate children of the word in the syntactic dependency parse.
@ -285,7 +285,7 @@ The leftward immediate children of the word in the syntactic dependency parse.
| ---------- | ------------------------------------ |
| **YIELDS** | A left-child of the token. ~~Token~~ |
## Token.rights {#rights tag="property" model="parser"}
## Token.rights {id="rights",tag="property",model="parser"}
The rightward immediate children of the word in the syntactic dependency parse.
@ -301,7 +301,7 @@ The rightward immediate children of the word in the syntactic dependency parse.
| ---------- | ------------------------------------- |
| **YIELDS** | A right-child of the token. ~~Token~~ |
## Token.n_lefts {#n_lefts tag="property" model="parser"}
## Token.n_lefts {id="n_lefts",tag="property",model="parser"}
The number of leftward immediate children of the word in the syntactic
dependency parse.
@ -317,7 +317,7 @@ dependency parse.
| ----------- | ---------------------------------------- |
| **RETURNS** | The number of left-child tokens. ~~int~~ |
## Token.n_rights {#n_rights tag="property" model="parser"}
## Token.n_rights {id="n_rights",tag="property",model="parser"}
The number of rightward immediate children of the word in the syntactic
dependency parse.
@ -333,7 +333,7 @@ dependency parse.
| ----------- | ----------------------------------------- |
| **RETURNS** | The number of right-child tokens. ~~int~~ |
## Token.subtree {#subtree tag="property" model="parser"}
## Token.subtree {id="subtree",tag="property",model="parser"}
A sequence containing the token and all the token's syntactic descendants.
@ -349,7 +349,7 @@ A sequence containing the token and all the token's syntactic descendants.
| ---------- | ------------------------------------------------------------------------------------ |
| **YIELDS** | A descendant token such that `self.is_ancestor(token)` or `token == self`. ~~Token~~ |
## Token.has_vector {#has_vector tag="property" model="vectors"}
## Token.has_vector {id="has_vector",tag="property",model="vectors"}
A boolean value indicating whether a word vector is associated with the token.
@ -365,7 +365,7 @@ A boolean value indicating whether a word vector is associated with the token.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | Whether the token has a vector data attached. ~~bool~~ |
## Token.vector {#vector tag="property" model="vectors"}
## Token.vector {id="vector",tag="property",model="vectors"}
A real-valued meaning representation.
@ -382,7 +382,7 @@ A real-valued meaning representation.
| ----------- | ----------------------------------------------------------------------------------------------- |
| **RETURNS** | A 1-dimensional array representing the token's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Token.vector_norm {#vector_norm tag="property" model="vectors"}
## Token.vector_norm {id="vector_norm",tag="property",model="vectors"}
The L2 norm of the token's vector representation.
@ -401,7 +401,7 @@ The L2 norm of the token's vector representation.
| ----------- | --------------------------------------------------- |
| **RETURNS** | The L2 norm of the vector representation. ~~float~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

28
website/docs/api/tokenizer.md → website/docs/api/tokenizer.mdx
View File

@ -20,7 +20,7 @@ The tokenizer is typically created automatically when a
like punctuation and special case rules from the
[`Language.Defaults`](/api/language#defaults) provided by the language subclass.
## Tokenizer.\_\_init\_\_ {#init tag="method"}
## Tokenizer.\_\_init\_\_ {id="init",tag="method"}
Create a `Tokenizer` to create `Doc` objects given unicode text. For examples of
how to construct a custom tokenizer with different tokenization rules, see the
@ -55,7 +55,7 @@ how to construct a custom tokenizer with different tokenization rules, see the
| `url_match` | A function matching the signature of `re.compile(string).match` to find token matches after considering prefixes and suffixes. ~~Optional[Callable[[str], Optional[Match]]]~~ |
| `faster_heuristics` <Tag variant="new">3.3.0</Tag> | Whether to restrict the final `Matcher`-based pass for rules to those containing affixes or space. Defaults to `True`. ~~bool~~ |
## Tokenizer.\_\_call\_\_ {#call tag="method"}
## Tokenizer.\_\_call\_\_ {id="call",tag="method"}
Tokenize a string.
@ -71,7 +71,7 @@ Tokenize a string.
| `string` | The string to tokenize. ~~str~~ |
| **RETURNS** | A container for linguistic annotations. ~~Doc~~ |
## Tokenizer.pipe {#pipe tag="method"}
## Tokenizer.pipe {id="pipe",tag="method"}
Tokenize a stream of texts.
@ -89,7 +89,7 @@ Tokenize a stream of texts.
| `batch_size` | The number of texts to accumulate in an internal buffer. Defaults to `1000`. ~~int~~ |
| **YIELDS** | The tokenized `Doc` objects, in order. ~~Doc~~ |
## Tokenizer.find_infix {#find_infix tag="method"}
## Tokenizer.find_infix {id="find_infix",tag="method"}
Find internal split points of the string.
@ -98,7 +98,7 @@ Find internal split points of the string.
| `string` | The string to split. ~~str~~ |
| **RETURNS** | A list of `re.MatchObject` objects that have `.start()` and `.end()` methods, denoting the placement of internal segment separators, e.g. hyphens. ~~List[Match]~~ |
## Tokenizer.find_prefix {#find_prefix tag="method"}
## Tokenizer.find_prefix {id="find_prefix",tag="method"}
Find the length of a prefix that should be segmented from the string, or `None`
if no prefix rules match.
@ -108,7 +108,7 @@ if no prefix rules match.
| `string` | The string to segment. ~~str~~ |
| **RETURNS** | The length of the prefix if present, otherwise `None`. ~~Optional[int]~~ |
## Tokenizer.find_suffix {#find_suffix tag="method"}
## Tokenizer.find_suffix {id="find_suffix",tag="method"}
Find the length of a suffix that should be segmented from the string, or `None`
if no suffix rules match.
@ -118,7 +118,7 @@ if no suffix rules match.
| `string` | The string to segment. ~~str~~ |
| **RETURNS** | The length of the suffix if present, otherwise `None`. ~~Optional[int]~~ |
## Tokenizer.add_special_case {#add_special_case tag="method"}
## Tokenizer.add_special_case {id="add_special_case",tag="method"}
Add a special-case tokenization rule. This mechanism is also used to add custom
tokenizer exceptions to the language data. See the usage guide on the
@ -139,7 +139,7 @@ details and examples.
| `string` | The string to specially tokenize. ~~str~~ |
| `token_attrs` | A sequence of dicts, where each dict describes a token and its attributes. The `ORTH` fields of the attributes must exactly match the string when they are concatenated. ~~Iterable[Dict[int, str]]~~ |
## Tokenizer.explain {#explain tag="method"}
## Tokenizer.explain {id="explain",tag="method"}
Tokenize a string with a slow debugging tokenizer that provides information
about which tokenizer rule or pattern was matched for each token. The tokens
@ -158,7 +158,7 @@ produced are identical to `Tokenizer.__call__` except for whitespace tokens.
| `string` | The string to tokenize with the debugging tokenizer. ~~str~~ |
| **RETURNS** | A list of `(pattern_string, token_string)` tuples. ~~List[Tuple[str, str]]~~ |
## Tokenizer.to_disk {#to_disk tag="method"}
## Tokenizer.to_disk {id="to_disk",tag="method"}
Serialize the tokenizer to disk.
@ -175,7 +175,7 @@ Serialize the tokenizer to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Tokenizer.from_disk {#from_disk tag="method"}
## Tokenizer.from_disk {id="from_disk",tag="method"}
Load the tokenizer from disk. Modifies the object in place and returns it.
@ -193,7 +193,7 @@ Load the tokenizer from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Tokenizer` object. ~~Tokenizer~~ |
## Tokenizer.to_bytes {#to_bytes tag="method"}
## Tokenizer.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -210,7 +210,7 @@ Serialize the tokenizer to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Tokenizer` object. ~~bytes~~ |
## Tokenizer.from_bytes {#from_bytes tag="method"}
## Tokenizer.from_bytes {id="from_bytes",tag="method"}
Load the tokenizer from a bytestring. Modifies the object in place and returns
it.
@ -230,7 +230,7 @@ it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Tokenizer` object. ~~Tokenizer~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@ -241,7 +241,7 @@ it.
| `token_match` | A function matching the signature of `re.compile(string).match` to find token matches. Returns an `re.MatchObject` or `None`. ~~Optional[Callable[[str], Optional[Match]]]~~ |
| `rules` | A dictionary of tokenizer exceptions and special cases. ~~Optional[Dict[str, List[Dict[int, str]]]]~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

177
website/docs/api/top-level.md → website/docs/api/top-level.mdx
View File

@ -13,9 +13,9 @@ menu:
- ['Utility Functions', 'util']
---
## spaCy {#spacy hidden="true"}
## spaCy {id="spacy",hidden="true"}
### spacy.load {#spacy.load tag="function"}
### spacy.load {id="spacy.load",tag="function"}
Load a pipeline using the name of an installed
[package](/usage/saving-loading#models), a string path or a `Path`-like object.
@ -61,8 +61,7 @@ Essentially, `spacy.load()` is a convenience wrapper that reads the pipeline's
information to construct a `Language` object, loads in the model data and
weights, and returns it.
```python
### Abstract example
```python {title="Abstract example"}
cls = spacy.util.get_lang_class(lang) # 1. Get Language class, e.g. English
nlp = cls() # 2. Initialize it
for name in pipeline:
@ -70,7 +69,7 @@ for name in pipeline:
nlp.from_disk(data_path) # 4. Load in the binary data
```
### spacy.blank {#spacy.blank tag="function" new="2"}
### spacy.blank {id="spacy.blank",tag="function",version="2"}
Create a blank pipeline of a given language class. This function is the twin of
`spacy.load()`.
@ -91,7 +90,7 @@ Create a blank pipeline of a given language class. This function is the twin of
| `meta` | Optional meta overrides for [`nlp.meta`](/api/language#meta). ~~Dict[str, Any]~~ |
| **RETURNS** | An empty `Language` object of the appropriate subclass. ~~Language~~ |
### spacy.info {#spacy.info tag="function"}
### spacy.info {id="spacy.info",tag="function"}
The same as the [`info` command](/api/cli#info). Pretty-print information about
your installation, installed pipelines and local setup from within spaCy.
@ -111,7 +110,7 @@ your installation, installed pipelines and local setup from within spaCy.
| `markdown` | Print information as Markdown. ~~bool~~ |
| `silent` | Don't print anything, just return. ~~bool~~ |
### spacy.explain {#spacy.explain tag="function"}
### spacy.explain {id="spacy.explain",tag="function"}
Get a description for a given POS tag, dependency label or entity type. For a
list of available terms, see [`glossary.py`](%%GITHUB_SPACY/spacy/glossary.py).
@ -134,7 +133,7 @@ list of available terms, see [`glossary.py`](%%GITHUB_SPACY/spacy/glossary.py).
| `term` | Term to explain. ~~str~~ |
| **RETURNS** | The explanation, or `None` if not found in the glossary. ~~Optional[str]~~ |
### spacy.prefer_gpu {#spacy.prefer_gpu tag="function" new="2.0.14"}
### spacy.prefer_gpu {id="spacy.prefer_gpu",tag="function",version="2.0.14"}
Allocate data and perform operations on [GPU](/usage/#gpu), if available. If
data has already been allocated on CPU, it will not be moved. Ideally, this
@ -162,7 +161,7 @@ ensure that the model is loaded on the correct device. See
| `gpu_id` | Device index to select. Defaults to `0`. ~~int~~ |
| **RETURNS** | Whether the GPU was activated. ~~bool~~ |
### spacy.require_gpu {#spacy.require_gpu tag="function" new="2.0.14"}
### spacy.require_gpu {id="spacy.require_gpu",tag="function",version="2.0.14"}
Allocate data and perform operations on [GPU](/usage/#gpu). Will raise an error
if no GPU is available. If data has already been allocated on CPU, it will not
@ -190,7 +189,7 @@ ensure that the model is loaded on the correct device. See
| `gpu_id` | Device index to select. Defaults to `0`. ~~int~~ |
| **RETURNS** | `True` ~~bool~~ |
### spacy.require_cpu {#spacy.require_cpu tag="function" new="3.0.0"}
### spacy.require_cpu {id="spacy.require_cpu",tag="function",version="3.0.0"}
Allocate data and perform operations on CPU. If data has already been allocated
on GPU, it will not be moved. Ideally, this function should be called right
@ -216,12 +215,12 @@ ensure that the model is loaded on the correct device. See
| ----------- | --------------- |
| **RETURNS** | `True` ~~bool~~ |
## displaCy {#displacy source="spacy/displacy"}
## displaCy {id="displacy",source="spacy/displacy"}
As of v2.0, spaCy comes with a built-in visualization suite. For more info and
examples, see the usage guide on [visualizing spaCy](/usage/visualizers).
### displacy.serve {#displacy.serve tag="method" new="2"}
### displacy.serve {id="displacy.serve",tag="method",version="2"}
Serve a dependency parse tree or named entity visualization to view it in your
browser. Will run a simple web server.
@ -249,7 +248,7 @@ browser. Will run a simple web server.
| `host` | Host to serve visualization. Defaults to `"0.0.0.0"`. ~~str~~ |
| `auto_select_port` | If `True`, automatically switch to a different port if the specified port is already in use. Defaults to `False`. ~~bool~~ |
### displacy.render {#displacy.render tag="method" new="2"}
### displacy.render {id="displacy.render",tag="method",version="2"}
Render a dependency parse tree or named entity visualization.
@ -274,7 +273,7 @@ Render a dependency parse tree or named entity visualization.
| `jupyter` | Explicitly enable or disable "[Jupyter](http://jupyter.org/) mode" to return markup ready to be rendered in a notebook. Detected automatically if `None` (default). ~~Optional[bool]~~ |
| **RETURNS** | The rendered HTML markup. ~~str~~ |
### displacy.parse_deps {#displacy.parse_deps tag="method" new="2"}
### displacy.parse_deps {id="displacy.parse_deps",tag="method",version="2"}
Generate dependency parse in `{'words': [], 'arcs': []}` format. For use with
the `manual=True` argument in `displacy.render`.
@ -296,7 +295,7 @@ the `manual=True` argument in `displacy.render`.
| `options` | Dependency parse specific visualisation options. ~~Dict[str, Any]~~ |
| **RETURNS** | Generated dependency parse keyed by words and arcs. ~~dict~~ |
### displacy.parse_ents {#displacy.parse_ents tag="method" new="2"}
### displacy.parse_ents {id="displacy.parse_ents",tag="method",version="2"}
Generate named entities in `[{start: i, end: i, label: 'label'}]` format. For
use with the `manual=True` argument in `displacy.render`.
@ -318,7 +317,7 @@ use with the `manual=True` argument in `displacy.render`.
| `options` | NER-specific visualisation options. ~~Dict[str, Any]~~ |
| **RETURNS** | Generated entities keyed by text (original text) and ents. ~~dict~~ |
### displacy.parse_spans {#displacy.parse_spans tag="method" new="2"}
### displacy.parse_spans {id="displacy.parse_spans",tag="method",version="2"}
Generate spans in `[{start_token: i, end_token: i, label: 'label'}]` format. For
use with the `manual=True` argument in `displacy.render`.
@ -341,12 +340,12 @@ use with the `manual=True` argument in `displacy.render`.
| `options` | Span-specific visualisation options. ~~Dict[str, Any]~~ |
| **RETURNS** | Generated entities keyed by text (original text) and ents. ~~dict~~ |
### Visualizer options {#displacy_options}
### Visualizer options {id="displacy_options"}
The `options` argument lets you specify additional settings for each visualizer.
If a setting is not present in the options, the default value will be used.
#### Dependency Visualizer options {#options-dep}
#### Dependency Visualizer options {id="options-dep"}
> #### Example
>
@ -372,7 +371,7 @@ If a setting is not present in the options, the default value will be used.
| `word_spacing` | Vertical spacing between words and arcs in px. Defaults to `45`. ~~int~~ |
| `distance` | Distance between words in px. Defaults to `175` in regular mode and `150` in compact mode. ~~int~~ |
#### Named Entity Visualizer options {#displacy_options-ent}
#### Named Entity Visualizer options {id="displacy_options-ent"}
> #### Example
>
@ -389,7 +388,7 @@ If a setting is not present in the options, the default value will be used.
| `template` | Optional template to overwrite the HTML used to render entity spans. Should be a format string and can use `{bg}`, `{text}` and `{label}`. See [`templates.py`](%%GITHUB_SPACY/spacy/displacy/templates.py) for examples. ~~Optional[str]~~ |
| `kb_url_template` <Tag variant="new">3.2.1</Tag> | Optional template to construct the KB url for the entity to link to. Expects a python f-string format with single field to fill in. ~~Optional[str]~~ |
#### Span Visualizer options {#displacy_options-span}
#### Span Visualizer options {id="displacy_options-span"}
> #### Example
>
@ -420,7 +419,7 @@ span. If you wish to link an entity to their URL then consider using the
should redirect you to their Wikidata page, in this case
`https://www.wikidata.org/wiki/Q95`.
## registry {#registry source="spacy/util.py" new="3"}
## registry {id="registry",source="spacy/util.py",version="3"}
spaCy's function registry extends
[Thinc's `registry`](https://thinc.ai/docs/api-config#registry) and allows you
@ -470,7 +469,7 @@ factories.
| `scorers` | Registry for functions that create scoring methods for user with the [`Scorer`](/api/scorer). Scoring methods are called with `Iterable[Example]` and arbitrary `\*\*kwargs` and return scores as `Dict[str, Any]`. |
| `tokenizers` | Registry for tokenizer factories. Registered functions should return a callback that receives the `nlp` object and returns a [`Tokenizer`](/api/tokenizer) or a custom callable. |
### spacy-transformers registry {#registry-transformers}
### spacy-transformers registry {id="registry-transformers"}
The following registries are added by the
[`spacy-transformers`](https://github.com/explosion/spacy-transformers) package.
@ -495,7 +494,7 @@ See the [`Transformer`](/api/transformer) API reference and
| [`span_getters`](/api/transformer#span_getters) | Registry for functions that take a batch of `Doc` objects and return a list of `Span` objects to process by the transformer, e.g. sentences. |
| [`annotation_setters`](/api/transformer#annotation_setters) | Registry for functions that create annotation setters. Annotation setters are functions that take a batch of `Doc` objects and a [`FullTransformerBatch`](/api/transformer#fulltransformerbatch) and can set additional annotations on the `Doc`. |
## Loggers {#loggers source="spacy/training/loggers.py" new="3"}
## Loggers {id="loggers",source="spacy/training/loggers.py",version="3"}
A logger records the training results. When a logger is created, two functions
are returned: one for logging the information for each training step, and a
@ -531,7 +530,7 @@ saves them to a `jsonl` file.
<Accordion title="Example console output" spaced>
```cli
```bash
$ python -m spacy train config.cfg
```
@ -571,7 +570,7 @@ start decreasing across epochs.
| `console_output` | Whether the logger should print the logs in the console (default: `True`). ~~bool~~ |
| `output_file` | The file to save the training logs to (default: `None`). ~~Optional[Union[str, Path]]~~ |
#### spacy.ConsoleLogger.v3 {#ConsoleLogger tag="registered function"}
#### spacy.ConsoleLogger.v3 {id="ConsoleLogger",tag="registered function"}
> #### Example config
>
@ -593,9 +592,9 @@ optionally saves them to a `jsonl` file.
| `console_output` | Whether the logger should print the logs in the console (default: `True`). ~~bool~~ |
| `output_file` | The file to save the training logs to (default: `None`). ~~Optional[Union[str, Path]]~~ |
## Readers {#readers}
## Readers {id="readers"}
### File readers {#file-readers source="github.com/explosion/srsly" new="3"}
### File readers {id="file-readers",source="github.com/explosion/srsly",version="3"}
The following file readers are provided by our serialization library
[`srsly`](https://github.com/explosion/srsly). All registered functions take one
@ -625,7 +624,7 @@ blocks that are **not executed at runtime** – for example, in `[training]` and
</Infobox>
#### spacy.read_labels.v1 {#read_labels tag="registered function"}
#### spacy.read_labels.v1 {id="read_labels",tag="registered function"}
Read a JSON-formatted labels file generated with
[`init labels`](/api/cli#init-labels). Typically used in the
@ -651,7 +650,7 @@ label sets.
| `require` | Whether to require the file to exist. If set to `False` and the labels file doesn't exist, the loader will return `None` and the `initialize` method will extract the labels from the data. Defaults to `False`. ~~bool~~ |
| **CREATES** | The list of labels. ~~List[str]~~ |
### Corpus readers {#corpus-readers source="spacy/training/corpus.py" new="3"}
### Corpus readers {id="corpus-readers",source="spacy/training/corpus.py",version="3"}
Corpus readers are registered functions that load data and return a function
that takes the current `nlp` object and yields [`Example`](/api/example) objects
@ -661,7 +660,7 @@ with your own registered function in the
[`@readers` registry](/api/top-level#registry) to customize the data loading and
streaming.
#### spacy.Corpus.v1 {#corpus tag="registered function"}
#### spacy.Corpus.v1 {id="corpus",tag="registered function"}
The `Corpus` reader manages annotated corpora and can be used for training and
development datasets in the [DocBin](/api/docbin) (`.spacy`) format. Also see
@ -690,7 +689,7 @@ the [`Corpus`](/api/corpus) class.
| `augmenter` | Apply some simply data augmentation, where we replace tokens with variations. This is especially useful for punctuation and case replacement, to help generalize beyond corpora that don't have smart-quotes, or only have smart quotes, etc. Defaults to `None`. ~~Optional[Callable]~~ |
| **CREATES** | The corpus reader. ~~Corpus~~ |
#### spacy.JsonlCorpus.v1 {#jsonlcorpus tag="registered function"}
#### spacy.JsonlCorpus.v1 {id="jsonlcorpus",tag="registered function"}
Create [`Example`](/api/example) objects from a JSONL (newline-delimited JSON)
file of texts keyed by `"text"`. Can be used to read the raw text corpus for
@ -719,7 +718,7 @@ JSONL file. Also see the [`JsonlCorpus`](/api/corpus#jsonlcorpus) class.
| `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ |
| **CREATES** | The corpus reader. ~~JsonlCorpus~~ |
## Batchers {#batchers source="spacy/training/batchers.py" new="3"}
## Batchers {id="batchers",source="spacy/training/batchers.py",version="3"}
A data batcher implements a batching strategy that essentially turns a stream of
items into a stream of batches, with each batch consisting of one item or a list
@ -733,7 +732,7 @@ Instead of using one of the built-in batchers listed here, you can also
[implement your own](/usage/training#custom-code-readers-batchers), which may or
may not use a custom schedule.
### spacy.batch_by_words.v1 {#batch_by_words tag="registered function"}
### spacy.batch_by_words.v1 {id="batch_by_words",tag="registered function"}
Create minibatches of roughly a given number of words. If any examples are
longer than the specified batch length, they will appear in a batch by
@ -761,7 +760,7 @@ themselves, or be discarded if `discard_oversize` is set to `True`. The argument
| `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ |
| **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ |
### spacy.batch_by_sequence.v1 {#batch_by_sequence tag="registered function"}
### spacy.batch_by_sequence.v1 {id="batch_by_sequence",tag="registered function"}
> #### Example config
>
@ -780,7 +779,7 @@ Create a batcher that creates batches of the specified size.
| `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ |
| **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ |
### spacy.batch_by_padded.v1 {#batch_by_padded tag="registered function"}
### spacy.batch_by_padded.v1 {id="batch_by_padded",tag="registered function"}
> #### Example config
>
@ -806,7 +805,7 @@ sequences in the batch.
| `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ |
| **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ |
## Augmenters {#augmenters source="spacy/training/augment.py" new="3"}
## Augmenters {id="augmenters",source="spacy/training/augment.py",version="3"}
Data augmentation is the process of applying small modifications to the training
data. It can be especially useful for punctuation and case replacement – for
@ -815,7 +814,7 @@ variations using regular quotes, or to make the model less sensitive to
capitalization by including a mix of capitalized and lowercase examples. See the
[usage guide](/usage/training#data-augmentation) for details and examples.
### spacy.orth_variants.v1 {#orth_variants tag="registered function"}
### spacy.orth_variants.v1 {id="orth_variants",tag="registered function"}
> #### Example config
>
@ -842,7 +841,7 @@ beyond corpora that don't have smart quotes, or only have smart quotes etc.
| `orth_variants` | A dictionary containing the single and paired orth variants. Typically loaded from a JSON file. See [`en_orth_variants.json`](https://github.com/explosion/spacy-lookups-data/blob/master/spacy_lookups_data/data/en_orth_variants.json) for an example. ~~Dict[str, Dict[List[Union[str, List[str]]]]]~~ |
| **CREATES** | A function that takes the current `nlp` object and an [`Example`](/api/example) and yields augmented `Example` objects. ~~Callable[[Language, Example], Iterator[Example]]~~ |
### spacy.lower_case.v1 {#lower_case tag="registered function"}
### spacy.lower_case.v1 {id="lower_case",tag="registered function"}
> #### Example config
>
@ -861,12 +860,12 @@ useful for making the model less sensitive to capitalization.
| `level` | The percentage of texts that will be augmented. ~~float~~ |
| **CREATES** | A function that takes the current `nlp` object and an [`Example`](/api/example) and yields augmented `Example` objects. ~~Callable[[Language, Example], Iterator[Example]]~~ |
## Callbacks {#callbacks source="spacy/training/callbacks.py" new="3"}
## Callbacks {id="callbacks",source="spacy/training/callbacks.py",version="3"}
The config supports [callbacks](/usage/training#custom-code-nlp-callbacks) at
several points in the lifecycle that can be used modify the `nlp` object.
### spacy.copy_from_base_model.v1 {#copy_from_base_model tag="registered function"}
### spacy.copy_from_base_model.v1 {id="copy_from_base_model",tag="registered function"}
> #### Example config
>
@ -890,7 +889,7 @@ from the specified model. Intended for use in `[initialize.before_init]`.
| `vocab` | The pipeline to copy the vocab from. The vocab includes the lookups and vectors. Defaults to `None`. ~~Optional[str]~~ |
| **CREATES** | A function that takes the current `nlp` object and modifies its `tokenizer` and `vocab`. ~~Callable[[Language], None]~~ |
### spacy.models_with_nvtx_range.v1 {#models_with_nvtx_range tag="registered function"}
### spacy.models_with_nvtx_range.v1 {id="models_with_nvtx_range",tag="registered function"}
> #### Example config
>
@ -910,7 +909,7 @@ backprop passes.
| `backprop_color` | Color identifier for backpropagation passes. Defaults to `-1`. ~~int~~ |
| **CREATES** | A function that takes the current `nlp` and wraps forward/backprop passes in NVTX ranges. ~~Callable[[Language], Language]~~ |
### spacy.models_and_pipes_with_nvtx_range.v1 {#models_and_pipes_with_nvtx_range tag="registered function" new="3.4"}
### spacy.models_and_pipes_with_nvtx_range.v1 {id="models_and_pipes_with_nvtx_range",tag="registered function",version="3.4"}
> #### Example config
>
@ -931,9 +930,9 @@ methods are wrapped: `pipe`, `predict`, `set_annotations`, `update`, `rehearse`,
| `additional_pipe_functions` | Additional pipeline methods to wrap. Keys are pipeline names and values are lists of method identifiers. Defaults to `None`. ~~Optional[Dict[str, List[str]]]~~ |
| **CREATES** | A function that takes the current `nlp` and wraps pipe models and methods in NVTX ranges. ~~Callable[[Language], Language]~~ |
## Training data and alignment {#gold source="spacy/training"}
## Training data and alignment {id="gold",source="spacy/training"}
### training.offsets_to_biluo_tags {#offsets_to_biluo_tags tag="function"}
### training.offsets_to_biluo_tags {id="offsets_to_biluo_tags",tag="function"}
Encode labelled spans into per-token tags, using the
[BILUO scheme](/usage/linguistic-features#accessing-ner) (Begin, In, Last, Unit,
@ -970,7 +969,7 @@ This method was previously available as `spacy.gold.biluo_tags_from_offsets`.
| `missing` | The label used for missing values, e.g. if tokenization doesn't align with the entity offsets. Defaults to `"O"`. ~~str~~ |
| **RETURNS** | A list of strings, describing the [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ |
### training.biluo_tags_to_offsets {#biluo_tags_to_offsets tag="function"}
### training.biluo_tags_to_offsets {id="biluo_tags_to_offsets",tag="function"}
Encode per-token tags following the
[BILUO scheme](/usage/linguistic-features#accessing-ner) into entity offsets.
@ -998,7 +997,7 @@ This method was previously available as `spacy.gold.offsets_from_biluo_tags`.
| `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags with each tag describing one token. Each tag string will be of the form of either `""`, `"O"` or `"{action}-{label}"`, where action is one of `"B"`, `"I"`, `"L"`, `"U"`. ~~List[str]~~ |
| **RETURNS** | A sequence of `(start, end, label)` triples. `start` and `end` will be character-offset integers denoting the slice into the original string. ~~List[Tuple[int, int, str]]~~ |
### training.biluo_tags_to_spans {#biluo_tags_to_spans tag="function" new="2.1"}
### training.biluo_tags_to_spans {id="biluo_tags_to_spans",tag="function",version="2.1"}
Encode per-token tags following the
[BILUO scheme](/usage/linguistic-features#accessing-ner) into
@ -1027,7 +1026,7 @@ This method was previously available as `spacy.gold.spans_from_biluo_tags`.
| `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags with each tag describing one token. Each tag string will be of the form of either `""`, `"O"` or `"{action}-{label}"`, where action is one of `"B"`, `"I"`, `"L"`, `"U"`. ~~List[str]~~ |
| **RETURNS** | A sequence of `Span` objects with added entity labels. ~~List[Span]~~ |
### training.biluo_to_iob {#biluo_to_iob tag="function"}
### training.biluo_to_iob {id="biluo_to_iob",tag="function"}
Convert a sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags to
[IOB](/usage/linguistic-features#accessing-ner) tags. This is useful if you want
@ -1048,7 +1047,7 @@ use the BILUO tags with a model that only supports IOB tags.
| `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ |
| **RETURNS** | A list of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ |
### training.iob_to_biluo {#iob_to_biluo tag="function"}
### training.iob_to_biluo {id="iob_to_biluo",tag="function"}
Convert a sequence of [IOB](/usage/linguistic-features#accessing-ner) tags to
[BILUO](/usage/linguistic-features#accessing-ner) tags. This is useful if you
@ -1075,7 +1074,55 @@ This method was previously available as `spacy.gold.iob_to_biluo`.
| `tags` | A sequence of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ |
| **RETURNS** | A list of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ |
## Utility functions {#util source="spacy/util.py"}
### training.biluo_to_iob {id="biluo_to_iob",tag="function"}
Convert a sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags to
[IOB](/usage/linguistic-features#accessing-ner) tags. This is useful if you want
use the BILUO tags with a model that only supports IOB tags.
> #### Example
>
> ```python
> from spacy.training import biluo_to_iob
>
> tags = ["O", "O", "B-LOC", "I-LOC", "L-LOC", "O"]
> iob_tags = biluo_to_iob(tags)
> assert iob_tags == ["O", "O", "B-LOC", "I-LOC", "I-LOC", "O"]
> ```
| Name | Description |
| ----------- | --------------------------------------------------------------------------------------- |
| `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ |
| **RETURNS** | A list of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ |
### training.iob_to_biluo {id="iob_to_biluo",tag="function"}
Convert a sequence of [IOB](/usage/linguistic-features#accessing-ner) tags to
[BILUO](/usage/linguistic-features#accessing-ner) tags. This is useful if you
want use the IOB tags with a model that only supports BILUO tags.
<Infobox title="Changed in v3.0" variant="warning" id="iob_to_biluo">
This method was previously available as `spacy.gold.iob_to_biluo`.
</Infobox>
> #### Example
>
> ```python
> from spacy.training import iob_to_biluo
>
> tags = ["O", "O", "B-LOC", "I-LOC", "O"]
> biluo_tags = iob_to_biluo(tags)
> assert biluo_tags == ["O", "O", "B-LOC", "L-LOC", "O"]
> ```
| Name | Description |
| ----------- | ------------------------------------------------------------------------------------- |
| `tags` | A sequence of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ |
| **RETURNS** | A list of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ |
## Utility functions {id="util",source="spacy/util.py"}
spaCy comes with a small collection of utility functions located in
[`spacy/util.py`](%%GITHUB_SPACY/spacy/util.py). Because utility functions are
@ -1085,7 +1132,7 @@ use and we'll try to ensure backwards compatibility. However, we recommend
having additional tests in place if your application depends on any of spaCy's
utilities.
### util.get_lang_class {#util.get_lang_class tag="function"}
### util.get_lang_class {id="util.get_lang_class",tag="function"}
Import and load a `Language` class. Allows lazy-loading
[language data](/usage/linguistic-features#language-data) and importing
@ -1106,7 +1153,7 @@ custom language class, you can register it using the
| `lang` | Two-letter language code, e.g. `"en"`. ~~str~~ |
| **RETURNS** | The respective subclass. ~~Language~~ |
### util.lang_class_is_loaded {#util.lang_class_is_loaded tag="function" new="2.1"}
### util.lang_class_is_loaded {id="util.lang_class_is_loaded",tag="function",version="2.1"}
Check whether a `Language` subclass is already loaded. `Language` subclasses are
loaded lazily to avoid expensive setup code associated with the language data.
@ -1124,7 +1171,7 @@ loaded lazily to avoid expensive setup code associated with the language data.
| `name` | Two-letter language code, e.g. `"en"`. ~~str~~ |
| **RETURNS** | Whether the class has been loaded. ~~bool~~ |
### util.load_model {#util.load_model tag="function" new="2"}
### util.load_model {id="util.load_model",tag="function",version="2"}
Load a pipeline from a package or data path. If called with a string name, spaCy
will assume the pipeline is a Python package and import and call its `load()`
@ -1152,7 +1199,7 @@ and create a `Language` object. The model data will then be loaded in via
| `config` <Tag variant="new">3</Tag> | Config overrides as nested dict or flat dict keyed by section values in dot notation, e.g. `"nlp.pipeline"`. ~~Union[Dict[str, Any], Config]~~ |
| **RETURNS** | `Language` class with the loaded pipeline. ~~Language~~ |
### util.load_model_from_init_py {#util.load_model_from_init_py tag="function" new="2"}
### util.load_model_from_init_py {id="util.load_model_from_init_py",tag="function",version="2"}
A helper function to use in the `load()` method of a pipeline package's
[`__init__.py`](https://github.com/explosion/spacy-models/tree/master/template/model/xx_model_name/__init__.py).
@ -1177,7 +1224,7 @@ A helper function to use in the `load()` method of a pipeline package's
| `config` <Tag variant="new">3</Tag> | Config overrides as nested dict or flat dict keyed by section values in dot notation, e.g. `"nlp.pipeline"`. ~~Union[Dict[str, Any], Config]~~ |
| **RETURNS** | `Language` class with the loaded pipeline. ~~Language~~ |
### util.load_config {#util.load_config tag="function" new="3"}
### util.load_config {id="util.load_config",tag="function",version="3"}
Load a pipeline's [`config.cfg`](/api/data-formats#config) from a file path. The
config typically includes details about the components and how they're created,
@ -1197,7 +1244,7 @@ as well as all training settings and hyperparameters.
| `interpolate` | Whether to interpolate the config and replace variables like `${paths.train}` with their values. Defaults to `False`. ~~bool~~ |
| **RETURNS** | The pipeline's config. ~~Config~~ |
### util.load_meta {#util.load_meta tag="function" new="3"}
### util.load_meta {id="util.load_meta",tag="function",version="3"}
Get a pipeline's [`meta.json`](/api/data-formats#meta) from a file path and
validate its contents. The meta typically includes details about author,
@ -1214,7 +1261,7 @@ licensing, data sources and version.
| `path` | Path to the pipeline's `meta.json`. ~~Union[str, Path]~~ |
| **RETURNS** | The pipeline's meta data. ~~Dict[str, Any]~~ |
### util.get_installed_models {#util.get_installed_models tag="function" new="3"}
### util.get_installed_models {id="util.get_installed_models",tag="function",version="3"}
List all pipeline packages installed in the current environment. This will
include any spaCy pipeline that was packaged with
@ -1232,7 +1279,7 @@ object.
| ----------- | ------------------------------------------------------------------------------------- |
| **RETURNS** | The string names of the pipelines installed in the current environment. ~~List[str]~~ |
### util.is_package {#util.is_package tag="function"}
### util.is_package {id="util.is_package",tag="function"}
Check if string maps to a package installed via pip. Mainly used to validate
[pipeline packages](/usage/models).
@ -1249,7 +1296,7 @@ Check if string maps to a package installed via pip. Mainly used to validate
| `name` | Name of package. ~~str~~ |
| **RETURNS** | `True` if installed package, `False` if not. ~~bool~~ |
### util.get_package_path {#util.get_package_path tag="function" new="2"}
### util.get_package_path {id="util.get_package_path",tag="function",version="2"}
Get path to an installed package. Mainly used to resolve the location of
[pipeline packages](/usage/models). Currently imports the package to find its
@ -1267,7 +1314,7 @@ path.
| `package_name` | Name of installed package. ~~str~~ |
| **RETURNS** | Path to pipeline package directory. ~~Path~~ |
### util.is_in_jupyter {#util.is_in_jupyter tag="function" new="2"}
### util.is_in_jupyter {id="util.is_in_jupyter",tag="function",version="2"}
Check if user is running spaCy from a [Jupyter](https://jupyter.org) notebook by
detecting the IPython kernel. Mainly used for the
@ -1286,7 +1333,7 @@ detecting the IPython kernel. Mainly used for the
| ----------- | ---------------------------------------------- |
| **RETURNS** | `True` if in Jupyter, `False` if not. ~~bool~~ |
### util.compile_prefix_regex {#util.compile_prefix_regex tag="function"}
### util.compile_prefix_regex {id="util.compile_prefix_regex",tag="function"}
Compile a sequence of prefix rules into a regex object.
@ -1303,7 +1350,7 @@ Compile a sequence of prefix rules into a regex object.
| `entries` | The prefix rules, e.g. [`lang.punctuation.TOKENIZER_PREFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ |
| **RETURNS** | The regex object to be used for [`Tokenizer.prefix_search`](/api/tokenizer#attributes). ~~Pattern~~ |
### util.compile_suffix_regex {#util.compile_suffix_regex tag="function"}
### util.compile_suffix_regex {id="util.compile_suffix_regex",tag="function"}
Compile a sequence of suffix rules into a regex object.
@ -1320,7 +1367,7 @@ Compile a sequence of suffix rules into a regex object.
| `entries` | The suffix rules, e.g. [`lang.punctuation.TOKENIZER_SUFFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ |
| **RETURNS** | The regex object to be used for [`Tokenizer.suffix_search`](/api/tokenizer#attributes). ~~Pattern~~ |
### util.compile_infix_regex {#util.compile_infix_regex tag="function"}
### util.compile_infix_regex {id="util.compile_infix_regex",tag="function"}
Compile a sequence of infix rules into a regex object.
@ -1337,7 +1384,7 @@ Compile a sequence of infix rules into a regex object.
| `entries` | The infix rules, e.g. [`lang.punctuation.TOKENIZER_INFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ |
| **RETURNS** | The regex object to be used for [`Tokenizer.infix_finditer`](/api/tokenizer#attributes). ~~Pattern~~ |
### util.minibatch {#util.minibatch tag="function" new="2"}
### util.minibatch {id="util.minibatch",tag="function",version="2"}
Iterate over batches of items. `size` may be an iterator, so that batch-size can
vary on each step.
@ -1356,7 +1403,7 @@ vary on each step.
| `size` | The batch size(s). ~~Union[int, Sequence[int]]~~ |
| **YIELDS** | The batches. |
### util.filter_spans {#util.filter_spans tag="function" new="2.1.4"}
### util.filter_spans {id="util.filter_spans",tag="function",version="2.1.4"}
Filter a sequence of [`Span`](/api/span) objects and remove duplicates or
overlaps. Useful for creating named entities (where one token can only be part
@ -1377,7 +1424,7 @@ of one entity) or when merging spans with
| `spans` | The spans to filter. ~~Iterable[Span]~~ |
| **RETURNS** | The filtered spans. ~~List[Span]~~ |
### util.get_words_and_spaces {#get_words_and_spaces tag="function" new="3"}
### util.get_words_and_spaces {id="get_words_and_spaces",tag="function",version="3"}
Given a list of words and a text, reconstruct the original tokens and return a
list of words and spaces that can be used to create a [`Doc`](/api/doc#init).

54
website/docs/api/transformer.md → website/docs/api/transformer.mdx
View File

@ -3,7 +3,7 @@ title: Transformer
teaser: Pipeline component for multi-task learning with transformer models
tag: class
source: github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py
new: 3
version: 3
api_base_class: /api/pipe
api_string_name: transformer
---
@ -44,7 +44,7 @@ package also adds the function registries [`@span_getters`](#span_getters) and
functions. For more details, see the
[usage documentation](/usage/embeddings-transformers).
## Assigned Attributes {#assigned-attributes}
## Assigned Attributes {id="assigned-attributes"}
The component sets the following
[custom extension attribute](/usage/processing-pipeline#custom-components-attributes):
@ -53,7 +53,7 @@ The component sets the following
| ---------------- | ------------------------------------------------------------------------ |
| `Doc._.trf_data` | Transformer tokens and outputs for the `Doc` object. ~~TransformerData~~ |
## Config and implementation {#config}
## Config and implementation {id="config"}
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
@ -81,7 +81,7 @@ on the transformer architectures and their arguments and hyperparameters.
https://github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py
```
## Transformer.\_\_init\_\_ {#init tag="method"}
## Transformer.\_\_init\_\_ {id="init",tag="method"}
> #### Example
>
@ -124,7 +124,7 @@ component using its string name and [`nlp.add_pipe`](/api/language#create_pipe).
| `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ |
| `max_batch_items` | Maximum size of a padded batch. Defaults to `128*32`. ~~int~~ |
## Transformer.\_\_call\_\_ {#call tag="method"}
## Transformer.\_\_call\_\_ {id="call",tag="method"}
Apply the pipe to one document. The document is modified in place, and returned.
This usually happens under the hood when the `nlp` object is called on a text
@ -147,7 +147,7 @@ to the [`predict`](/api/transformer#predict) and
| `doc` | The document to process. ~~Doc~~ |
| **RETURNS** | The processed document. ~~Doc~~ |
## Transformer.pipe {#pipe tag="method"}
## Transformer.pipe {id="pipe",tag="method"}
Apply the pipe to a stream of documents. This usually happens under the hood
when the `nlp` object is called on a text and all pipeline components are
@ -171,7 +171,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/transformer#call) and
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
| **YIELDS** | The processed documents in order. ~~Doc~~ |
## Transformer.initialize {#initialize tag="method"}
## Transformer.initialize {id="initialize",tag="method"}
Initialize the component for training and return an
[`Optimizer`](https://thinc.ai/docs/api-optimizers). `get_examples` should be a
@ -196,7 +196,7 @@ by [`Language.initialize`](/api/language#initialize).
| _keyword-only_ | |
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
## Transformer.predict {#predict tag="method"}
## Transformer.predict {id="predict",tag="method"}
Apply the component's model to a batch of [`Doc`](/api/doc) objects without
modifying them.
@ -213,7 +213,7 @@ modifying them.
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
| **RETURNS** | The model's prediction for each document. |
## Transformer.set_annotations {#set_annotations tag="method"}
## Transformer.set_annotations {id="set_annotations",tag="method"}
Assign the extracted features to the `Doc` objects. By default, the
[`TransformerData`](/api/transformer#transformerdata) object is written to the
@ -233,7 +233,7 @@ callback is then called, if provided.
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
| `scores` | The scores to set, produced by `Transformer.predict`. |
## Transformer.update {#update tag="method"}
## Transformer.update {id="update",tag="method"}
Prepare for an update to the transformer. Like the [`Tok2Vec`](/api/tok2vec)
component, the `Transformer` component is unusual in that it does not receive
@ -266,7 +266,7 @@ and call the optimizer, while the others simply increment the gradients.
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
## Transformer.create_optimizer {#create_optimizer tag="method"}
## Transformer.create_optimizer {id="create_optimizer",tag="method"}
Create an optimizer for the pipeline component.
@ -281,7 +281,7 @@ Create an optimizer for the pipeline component.
| ----------- | ---------------------------- |
| **RETURNS** | The optimizer. ~~Optimizer~~ |
## Transformer.use_params {#use_params tag="method, contextmanager"}
## Transformer.use_params {id="use_params",tag="method, contextmanager"}
Modify the pipe's model to use the given parameter values. At the end of the
context, the original parameters are restored.
@ -298,7 +298,7 @@ context, the original parameters are restored.
| -------- | -------------------------------------------------- |
| `params` | The parameter values to use in the model. ~~dict~~ |
## Transformer.to_disk {#to_disk tag="method"}
## Transformer.to_disk {id="to_disk",tag="method"}
Serialize the pipe to disk.
@ -315,7 +315,7 @@ Serialize the pipe to disk.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Transformer.from_disk {#from_disk tag="method"}
## Transformer.from_disk {id="from_disk",tag="method"}
Load the pipe from disk. Modifies the object in place and returns it.
@ -333,7 +333,7 @@ Load the pipe from disk. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Transformer` object. ~~Transformer~~ |
## Transformer.to_bytes {#to_bytes tag="method"}
## Transformer.to_bytes {id="to_bytes",tag="method"}
> #### Example
>
@ -350,7 +350,7 @@ Serialize the pipe to a bytestring.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Transformer` object. ~~bytes~~ |
## Transformer.from_bytes {#from_bytes tag="method"}
## Transformer.from_bytes {id="from_bytes",tag="method"}
Load the pipe from a bytestring. Modifies the object in place and returns it.
@ -369,7 +369,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Transformer` object. ~~Transformer~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from
@ -387,7 +387,7 @@ serialization by passing in the string names via the `exclude` argument.
| `cfg` | The config file. You usually don't want to exclude this. |
| `model` | The binary model data. You usually don't want to exclude this. |
## TransformerData {#transformerdata tag="dataclass"}
## TransformerData {id="transformerdata",tag="dataclass"}
Transformer tokens and outputs for one `Doc` object. The transformer models
return tensors that refer to a whole padded batch of documents. These tensors
@ -405,7 +405,7 @@ by this class. Instances of this class are typically assigned to the
| `align` | Alignment from the `Doc`'s tokenization to the wordpieces. This is a ragged array, where `align.lengths[i]` indicates the number of wordpiece tokens that token `i` aligns against. The actual indices are provided at `align[i].dataXd`. ~~Ragged~~ |
| `width` | The width of the last hidden layer. ~~int~~ |
### TransformerData.empty {#transformerdata-emoty tag="classmethod"}
### TransformerData.empty {id="transformerdata-emoty",tag="classmethod"}
Create an empty `TransformerData` container.
@ -425,7 +425,7 @@ model.
</Accordion>
## FullTransformerBatch {#fulltransformerbatch tag="dataclass"}
## FullTransformerBatch {id="fulltransformerbatch",tag="dataclass"}
Holds a batch of input and output objects for a transformer model. The data can
then be split to a list of [`TransformerData`](/api/transformer#transformerdata)
@ -440,7 +440,7 @@ objects to associate the outputs to each [`Doc`](/api/doc) in the batch.
| `align` | Alignment from the spaCy tokenization to the wordpieces. This is a ragged array, where `align.lengths[i]` indicates the number of wordpiece tokens that token `i` aligns against. The actual indices are provided at `align[i].dataXd`. ~~Ragged~~ |
| `doc_data` | The outputs, split per `Doc` object. ~~List[TransformerData]~~ |
### FullTransformerBatch.unsplit_by_doc {#fulltransformerbatch-unsplit_by_doc tag="method"}
### FullTransformerBatch.unsplit_by_doc {id="fulltransformerbatch-unsplit_by_doc",tag="method"}
Return a new `FullTransformerBatch` from a split batch of activations, using the
current object's spans, tokens and alignment. This is used during the backward
@ -452,7 +452,7 @@ model.
| `arrays` | The split batch of activations. ~~List[List[Floats3d]]~~ |
| **RETURNS** | The transformer batch. ~~FullTransformerBatch~~ |
### FullTransformerBatch.split_by_doc {#fulltransformerbatch-split_by_doc tag="method"}
### FullTransformerBatch.split_by_doc {id="fulltransformerbatch-split_by_doc",tag="method"}
Split a `TransformerData` object that represents a batch into a list with one
`TransformerData` per `Doc`.
@ -468,7 +468,7 @@ In `spacy-transformers` v1.0, the model output is stored in
</Accordion>
## Span getters {#span_getters source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/span_getters.py"}
## Span getters {id="span_getters",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/span_getters.py"}
Span getters are functions that take a batch of [`Doc`](/api/doc) objects and
return a lists of [`Span`](/api/span) objects for each doc to be processed by
@ -498,7 +498,7 @@ using the `@spacy.registry.span_getters` decorator.
| `docs` | A batch of `Doc` objects. ~~Iterable[Doc]~~ |
| **RETURNS** | The spans to process by the transformer. ~~List[List[Span]]~~ |
### doc_spans.v1 {#doc_spans tag="registered function"}
### doc_spans.v1 {id="doc_spans",tag="registered function"}
> #### Example config
>
@ -511,7 +511,7 @@ Create a span getter that uses the whole document as its spans. This is the best
approach if your [`Doc`](/api/doc) objects already refer to relatively short
texts.
### sent_spans.v1 {#sent_spans tag="registered function"}
### sent_spans.v1 {id="sent_spans",tag="registered function"}
> #### Example config
>
@ -531,7 +531,7 @@ To set sentence boundaries with the `sentencizer` during training, add a
[`[training.annotating_components]`](/usage/training#annotating-components) to
have it set the sentence boundaries before the `transformer` component runs.
### strided_spans.v1 {#strided_spans tag="registered function"}
### strided_spans.v1 {id="strided_spans",tag="registered function"}
> #### Example config
>
@ -553,7 +553,7 @@ right context.
| `window` | The window size. ~~int~~ |
| `stride` | The stride size. ~~int~~ |
## Annotation setters {#annotation_setters tag="registered functions" source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/annotation_setters.py"}
## Annotation setters {id="annotation_setters",tag="registered functions",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/annotation_setters.py"}
Annotation setters are functions that take a batch of `Doc` objects and a
[`FullTransformerBatch`](/api/transformer#fulltransformerbatch) and can set

50
website/docs/api/vectors.md → website/docs/api/vectors.mdx
View File

@ -3,7 +3,7 @@ title: Vectors
teaser: Store, save and load word vectors
tag: class
source: spacy/vectors.pyx
new: 2
version: 2
---
Vectors data is kept in the `Vectors.data` attribute, which should be an
@ -25,7 +25,7 @@ As of spaCy v3.2, `Vectors` supports two types of vector tables:
the sum of one or more rows as determined by the settings related to character
ngrams and the hash table.
## Vectors.\_\_init\_\_ {#init tag="method"}
## Vectors.\_\_init\_\_ {id="init",tag="method"}
Create a new vector store. With the default mode, you can set the vector values
and keys directly on initialization, or supply a `shape` keyword argument to
@ -61,7 +61,7 @@ modified later.
| `bow` <Tag variant="new">3.2</Tag> | The floret BOW string (default: `"<"`). ~~str~~ |
| `eow` <Tag variant="new">3.2</Tag> | The floret EOW string (default: `">"`). ~~str~~ |
## Vectors.\_\_getitem\_\_ {#getitem tag="method"}
## Vectors.\_\_getitem\_\_ {id="getitem",tag="method"}
Get a vector by key. If the key is not found in the table, a `KeyError` is
raised.
@ -79,7 +79,7 @@ raised.
| `key` | The key to get the vector for. ~~Union[int, str]~~ |
| **RETURNS** | The vector for the key. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Vectors.\_\_setitem\_\_ {#setitem tag="method"}
## Vectors.\_\_setitem\_\_ {id="setitem",tag="method"}
Set a vector for the given key. Not supported for `floret` mode.
@ -96,7 +96,7 @@ Set a vector for the given key. Not supported for `floret` mode.
| `key` | The key to set the vector for. ~~int~~ |
| `vector` | The vector to set. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Vectors.\_\_iter\_\_ {#iter tag="method"}
## Vectors.\_\_iter\_\_ {id="iter",tag="method"}
Iterate over the keys in the table. In `floret` mode, the keys table is not
used.
@ -112,7 +112,7 @@ used.
| ---------- | --------------------------- |
| **YIELDS** | A key in the table. ~~int~~ |
## Vectors.\_\_len\_\_ {#len tag="method"}
## Vectors.\_\_len\_\_ {id="len",tag="method"}
Return the number of vectors in the table.
@ -127,7 +127,7 @@ Return the number of vectors in the table.
| ----------- | ------------------------------------------- |
| **RETURNS** | The number of vectors in the table. ~~int~~ |
## Vectors.\_\_contains\_\_ {#contains tag="method"}
## Vectors.\_\_contains\_\_ {id="contains",tag="method"}
Check whether a key has been mapped to a vector entry in the table. In `floret`
mode, returns `True` for all keys.
@ -145,7 +145,7 @@ mode, returns `True` for all keys.
| `key` | The key to check. ~~int~~ |
| **RETURNS** | Whether the key has a vector entry. ~~bool~~ |
## Vectors.add {#add tag="method"}
## Vectors.add {id="add",tag="method"}
Add a key to the table, optionally setting a vector value as well. Keys can be
mapped to an existing vector by setting `row`, or a new vector can be added. Not
@ -168,7 +168,7 @@ supported for `floret` mode.
| `row` | An optional row number of a vector to map the key to. ~~int~~ |
| **RETURNS** | The row the vector was added to. ~~int~~ |
## Vectors.resize {#resize tag="method"}
## Vectors.resize {id="resize",tag="method"}
Resize the underlying vectors array. If `inplace=True`, the memory is
reallocated. This may cause other references to the data to become invalid, so
@ -189,7 +189,7 @@ for `floret` mode.
| `inplace` | Reallocate the memory. ~~bool~~ |
| **RETURNS** | The removed items as a list of `(key, row)` tuples. ~~List[Tuple[int, int]]~~ |
## Vectors.keys {#keys tag="method"}
## Vectors.keys {id="keys",tag="method"}
A sequence of the keys in the table. In `floret` mode, the keys table is not
used.
@ -205,7 +205,7 @@ used.
| ----------- | --------------------------- |
| **RETURNS** | The keys. ~~Iterable[int]~~ |
## Vectors.values {#values tag="method"}
## Vectors.values {id="values",tag="method"}
Iterate over vectors that have been assigned to at least one key. Note that some
vectors may be unassigned, so the number of vectors returned may be less than
@ -222,7 +222,7 @@ the length of the vectors table. In `floret` mode, the keys table is not used.
| ---------- | --------------------------------------------------------------- |
| **YIELDS** | A vector in the table. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Vectors.items {#items tag="method"}
## Vectors.items {id="items",tag="method"}
Iterate over `(key, vector)` pairs, in order. In `floret` mode, the keys table
is empty.
@ -238,7 +238,7 @@ is empty.
| ---------- | ------------------------------------------------------------------------------------- |
| **YIELDS** | `(key, vector)` pairs, in order. ~~Tuple[int, numpy.ndarray[ndim=1, dtype=float32]]~~ |
## Vectors.find {#find tag="method"}
## Vectors.find {id="find",tag="method"}
Look up one or more keys by row, or vice versa. Not supported for `floret` mode.
@ -260,7 +260,7 @@ Look up one or more keys by row, or vice versa. Not supported for `floret` mode.
| `rows` | Find the keys that point to the rows. Returns `numpy.ndarray`. ~~Iterable[int]~~ |
| **RETURNS** | The requested key, keys, row or rows. ~~Union[int, numpy.ndarray[ndim=1, dtype=float32]]~~ |
## Vectors.shape {#shape tag="property"}
## Vectors.shape {id="shape",tag="property"}
Get `(rows, dims)` tuples of number of rows and number of dimensions in the
vector table.
@ -279,7 +279,7 @@ vector table.
| ----------- | ------------------------------------------ |
| **RETURNS** | A `(rows, dims)` pair. ~~Tuple[int, int]~~ |
## Vectors.size {#size tag="property"}
## Vectors.size {id="size",tag="property"}
The vector size, i.e. `rows * dims`.
@ -294,7 +294,7 @@ The vector size, i.e. `rows * dims`.
| ----------- | ------------------------ |
| **RETURNS** | The vector size. ~~int~~ |
## Vectors.is_full {#is_full tag="property"}
## Vectors.is_full {id="is_full",tag="property"}
Whether the vectors table is full and has no slots are available for new keys.
If a table is full, it can be resized using
@ -313,7 +313,7 @@ full and cannot be resized.
| ----------- | ------------------------------------------- |
| **RETURNS** | Whether the vectors table is full. ~~bool~~ |
## Vectors.n_keys {#n_keys tag="property"}
## Vectors.n_keys {id="n_keys",tag="property"}
Get the number of keys in the table. Note that this is the number of _all_ keys,
not just unique vectors. If several keys are mapped to the same vectors, they
@ -331,7 +331,7 @@ will be counted individually. In `floret` mode, the keys table is not used.
| ----------- | ----------------------------------------------------------------------------- |
| **RETURNS** | The number of all keys in the table. Returns `-1` for floret vectors. ~~int~~ |
## Vectors.most_similar {#most_similar tag="method"}
## Vectors.most_similar {id="most_similar",tag="method"}
For each of the given vectors, find the `n` most similar entries to it by
cosine. Queries are by vector. Results are returned as a
@ -356,7 +356,7 @@ supported for `floret` mode.
| `sort` | Whether to sort the entries returned by score. Defaults to `True`. ~~bool~~ |
| **RETURNS** | The most similar entries as a `(keys, best_rows, scores)` tuple. ~~Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]~~ |
## Vectors.get_batch {#get_batch tag="method" new="3.2"}
## Vectors.get_batch {id="get_batch",tag="method",version="3.2"}
Get the vectors for the provided keys efficiently as a batch.
@ -371,7 +371,7 @@ Get the vectors for the provided keys efficiently as a batch.
| ------ | --------------------------------------- |
| `keys` | The keys. ~~Iterable[Union[int, str]]~~ |
## Vectors.to_ops {#to_ops tag="method"}
## Vectors.to_ops {id="to_ops",tag="method"}
Change the embedding matrix to use different Thinc ops.
@ -388,7 +388,7 @@ Change the embedding matrix to use different Thinc ops.
| ----- | -------------------------------------------------------- |
| `ops` | The Thinc ops to switch the embedding matrix to. ~~Ops~~ |
## Vectors.to_disk {#to_disk tag="method"}
## Vectors.to_disk {id="to_disk",tag="method"}
Save the current state to a directory.
@ -403,7 +403,7 @@ Save the current state to a directory.
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
## Vectors.from_disk {#from_disk tag="method"}
## Vectors.from_disk {id="from_disk",tag="method"}
Loads state from a directory. Modifies the object in place and returns it.
@ -419,7 +419,7 @@ Loads state from a directory. Modifies the object in place and returns it.
| `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
| **RETURNS** | The modified `Vectors` object. ~~Vectors~~ |
## Vectors.to_bytes {#to_bytes tag="method"}
## Vectors.to_bytes {id="to_bytes",tag="method"}
Serialize the current state to a binary string.
@ -433,7 +433,7 @@ Serialize the current state to a binary string.
| ----------- | ------------------------------------------------------ |
| **RETURNS** | The serialized form of the `Vectors` object. ~~bytes~~ |
## Vectors.from_bytes {#from_bytes tag="method"}
## Vectors.from_bytes {id="from_bytes",tag="method"}
Load state from a binary string.
@ -451,7 +451,7 @@ Load state from a binary string.
| `data` | The data to load from. ~~bytes~~ |
| **RETURNS** | The `Vectors` object. ~~Vectors~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
| Name | Description |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

36
website/docs/api/vocab.md → website/docs/api/vocab.mdx
View File

@ -10,7 +10,7 @@ The `Vocab` object provides a lookup table that allows you to access
[`StringStore`](/api/stringstore). It also owns underlying C-data that is shared
between `Doc` objects.
## Vocab.\_\_init\_\_ {#init tag="method"}
## Vocab.\_\_init\_\_ {id="init",tag="method"}
Create the vocabulary.
@ -31,7 +31,7 @@ Create the vocabulary.
| `writing_system` | A dictionary describing the language's writing system. Typically provided by [`Language.Defaults`](/api/language#defaults). ~~Dict[str, Any]~~ |
| `get_noun_chunks` | A function that yields base noun phrases used for [`Doc.noun_chunks`](/api/doc#noun_chunks). ~~Optional[Callable[[Union[Doc, Span], Iterator[Tuple[int, int, int]]]]]~~ |
## Vocab.\_\_len\_\_ {#len tag="method"}
## Vocab.\_\_len\_\_ {id="len",tag="method"}
Get the current number of lexemes in the vocabulary.
@ -46,7 +46,7 @@ Get the current number of lexemes in the vocabulary.
| ----------- | ------------------------------------------------ |
| **RETURNS** | The number of lexemes in the vocabulary. ~~int~~ |
## Vocab.\_\_getitem\_\_ {#getitem tag="method"}
## Vocab.\_\_getitem\_\_ {id="getitem",tag="method"}
Retrieve a lexeme, given an int ID or a string. If a previously unseen string is
given, a new lexeme is created and stored.
@ -63,7 +63,7 @@ given, a new lexeme is created and stored.
| `id_or_string` | The hash value of a word, or its string. ~~Union[int, str]~~ |
| **RETURNS** | The lexeme indicated by the given ID. ~~Lexeme~~ |
## Vocab.\_\_iter\_\_ {#iter tag="method"}
## Vocab.\_\_iter\_\_ {id="iter",tag="method"}
Iterate over the lexemes in the vocabulary.
@ -77,7 +77,7 @@ Iterate over the lexemes in the vocabulary.
| ---------- | -------------------------------------- |
| **YIELDS** | An entry in the vocabulary. ~~Lexeme~~ |
## Vocab.\_\_contains\_\_ {#contains tag="method"}
## Vocab.\_\_contains\_\_ {id="contains",tag="method"}
Check whether the string has an entry in the vocabulary. To get the ID for a
given string, you need to look it up in
@ -97,7 +97,7 @@ given string, you need to look it up in
| `string` | The ID string. ~~str~~ |
| **RETURNS** | Whether the string has an entry in the vocabulary. ~~bool~~ |
## Vocab.add_flag {#add_flag tag="method"}
## Vocab.add_flag {id="add_flag",tag="method"}
Set a new boolean flag to words in the vocabulary. The `flag_getter` function
will be called over the words currently in the vocab, and then applied to new
@ -122,7 +122,7 @@ using `token.check_flag(flag_id)`.
| `flag_id` | An integer between `1` and `63` (inclusive), specifying the bit at which the flag will be stored. If `-1`, the lowest available bit will be chosen. ~~int~~ |
| **RETURNS** | The integer ID by which the flag value can be checked. ~~int~~ |
## Vocab.reset_vectors {#reset_vectors tag="method" new="2"}
## Vocab.reset_vectors {id="reset_vectors",tag="method",version="2"}
Drop the current vector table. Because all vectors must be the same width, you
have to call this to change the size of the vectors. Only one of the `width` and
@ -140,7 +140,7 @@ have to call this to change the size of the vectors. Only one of the `width` and
| `width` | The new width. ~~int~~ |
| `shape` | The new shape. ~~int~~ |
## Vocab.prune_vectors {#prune_vectors tag="method" new="2"}
## Vocab.prune_vectors {id="prune_vectors",tag="method",version="2"}
Reduce the current vector table to `nr_row` unique entries. Words mapped to the
discarded vectors will be remapped to the closest vector among those remaining.
@ -165,7 +165,7 @@ cosines are calculated in minibatches to reduce memory usage.
| `batch_size` | Batch of vectors for calculating the similarities. Larger batch sizes might be faster, while temporarily requiring more memory. ~~int~~ |
| **RETURNS** | A dictionary keyed by removed words mapped to `(string, score)` tuples, where `string` is the entry the removed word was mapped to, and `score` the similarity score between the two words. ~~Dict[str, Tuple[str, float]]~~ |
## Vocab.deduplicate_vectors {#deduplicate_vectors tag="method" new="3.3"}
## Vocab.deduplicate_vectors {id="deduplicate_vectors",tag="method",version="3.3"}
> #### Example
>
@ -176,7 +176,7 @@ cosines are calculated in minibatches to reduce memory usage.
Remove any duplicate rows from the current vector table, maintaining the
mappings for all words in the vectors.
## Vocab.get_vector {#get_vector tag="method" new="2"}
## Vocab.get_vector {id="get_vector",tag="method",version="2"}
Retrieve a vector for a word in the vocabulary. Words can be looked up by string
or hash value. If the current vectors do not contain an entry for the word, a
@ -194,7 +194,7 @@ or hash value. If the current vectors do not contain an entry for the word, a
| `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ |
| **RETURNS** | A word vector. Size and shape are determined by the `Vocab.vectors` instance. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Vocab.set_vector {#set_vector tag="method" new="2"}
## Vocab.set_vector {id="set_vector",tag="method",version="2"}
Set a vector for a word in the vocabulary. Words can be referenced by string or
hash value.
@ -210,7 +210,7 @@ hash value.
| `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ |
| `vector` | The vector to set. ~~numpy.ndarray[ndim=1, dtype=float32]~~ |
## Vocab.has_vector {#has_vector tag="method" new="2"}
## Vocab.has_vector {id="has_vector",tag="method",version="2"}
Check whether a word has a vector. Returns `False` if no vectors are loaded.
Words can be looked up by string or hash value.
@ -227,7 +227,7 @@ Words can be looked up by string or hash value.
| `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ |
| **RETURNS** | Whether the word has a vector. ~~bool~~ |
## Vocab.to_disk {#to_disk tag="method" new="2"}
## Vocab.to_disk {id="to_disk",tag="method",version="2"}
Save the current state to a directory.
@ -243,7 +243,7 @@ Save the current state to a directory.
| _keyword-only_ | |
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
## Vocab.from_disk {#from_disk tag="method" new="2"}
## Vocab.from_disk {id="from_disk",tag="method",version="2"}
Loads state from a directory. Modifies the object in place and returns it.
@ -261,7 +261,7 @@ Loads state from a directory. Modifies the object in place and returns it.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The modified `Vocab` object. ~~Vocab~~ |
## Vocab.to_bytes {#to_bytes tag="method"}
## Vocab.to_bytes {id="to_bytes",tag="method"}
Serialize the current state to a binary string.
@ -277,7 +277,7 @@ Serialize the current state to a binary string.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The serialized form of the `Vocab` object. ~~Vocab~~ |
## Vocab.from_bytes {#from_bytes tag="method"}
## Vocab.from_bytes {id="from_bytes",tag="method"}
Load state from a binary string.
@ -297,7 +297,7 @@ Load state from a binary string.
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
| **RETURNS** | The `Vocab` object. ~~Vocab~~ |
## Attributes {#attributes}
## Attributes {id="attributes"}
> #### Example
>
@ -317,7 +317,7 @@ Load state from a binary string.
| `writing_system` | A dict with information about the language's writing system. ~~Dict[str, Any]~~ |
| `get_noun_chunks` <Tag variant="new">3.0</Tag> | A function that yields base noun phrases used for [`Doc.noun_chunks`](/api/doc#noun_chunks). ~~Optional[Callable[[Union[Doc, Span], Iterator[Tuple[int, int, int]]]]]~~ |
## Serialization fields {#serialization-fields}
## Serialization fields {id="serialization-fields"}
During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from

58
website/docs/images/displacy-dep-founded.html
View File

@ -1,58 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" xlink="http://www.w3.org/1999/xlink" xml:lang="en" id="c3124cc3e661444cb9d4175a5b7c09d1-0" class="displacy" width="925" height="399.5" direction="ltr" style="max-width: none; height: 399.5px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr">
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="50">Smith</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="50"></tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="225">founded</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="225"></tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="400">a</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="400"></tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="575">healthcare</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="575"></tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="750">company</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="750"></tspan>
</text>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-0" stroke-width="2px" d="M70,264.5 C70,177.0 215.0,177.0 215.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textPath xlink:href="#arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-0" class="displacy-label" startOffset="50%" side="left" fill="currentColor" text-anchor="middle">nsubj</textPath>
</text>
<path class="displacy-arrowhead" d="M70,266.5 L62,254.5 78,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-1" stroke-width="2px" d="M420,264.5 C420,89.5 745.0,89.5 745.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textPath xlink:href="#arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-1" class="displacy-label" startOffset="50%" side="left" fill="currentColor" text-anchor="middle">det</textPath>
</text>
<path class="displacy-arrowhead" d="M420,266.5 L412,254.5 428,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-2" stroke-width="2px" d="M595,264.5 C595,177.0 740.0,177.0 740.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textPath xlink:href="#arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-2" class="displacy-label" startOffset="50%" side="left" fill="currentColor" text-anchor="middle">compound</textPath>
</text>
<path class="displacy-arrowhead" d="M595,266.5 L587,254.5 603,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-3" stroke-width="2px" d="M245,264.5 C245,2.0 750.0,2.0 750.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textPath xlink:href="#arrow-c3124cc3e661444cb9d4175a5b7c09d1-0-3" class="displacy-label" startOffset="50%" side="left" fill="currentColor" text-anchor="middle">dobj</textPath>
</text>
<path class="displacy-arrowhead" d="M750.0,266.5 L758.0,254.5 742.0,254.5" fill="currentColor"></path>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 3.8 KiB

33
website/docs/images/displacy-ent-custom.html
View File

@ -1,33 +0,0 @@
<div
class="entities"
style="line-height: 2.5; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 18px"
>But
<mark
class="entity"
style="background: linear-gradient(90deg, #AA9CFC, #FC9CE7); padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>Google
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>ORG</span
></mark
>is starting from behind. The company made a late push into hardware, and
<mark
class="entity"
style="background: linear-gradient(90deg, #AA9CFC, #FC9CE7); padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>Apple
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>ORG</span
></mark
>’s Siri, available on iPhones, and
<mark
class="entity"
style="background: linear-gradient(90deg, #AA9CFC, #FC9CE7); padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>Amazon
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>ORG</span
></mark
>’s Alexa software, which runs on its Echo and Dot devices, have clear leads in consumer
adoption.</div
>

26
website/docs/images/displacy-ent-snek.html
View File

@ -1,26 +0,0 @@
<div
class="entities"
style="line-height: 2.5; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px"
>
🌱🌿
<mark
class="entity"
style="background: #3dff74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>🐍
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>SNEK</span
></mark
>
____ 🌳🌲 ____
<mark
class="entity"
style="background: #cfc5ff; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>👨‍🌾
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>HUMAN</span
></mark
>
🏘️
</div>

37
website/docs/images/displacy-ent1.html
View File

@ -1,37 +0,0 @@
<div
class="entities"
style="line-height: 2.5; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px"
>
<mark
class="entity"
style="background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
Apple
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>ORG</span
>
</mark>
is looking at buying
<mark
class="entity"
style="background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
U.K.
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>GPE</span
>
</mark>
startup for
<mark
class="entity"
style="background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
$1 billion
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>MONEY</span
>
</mark>
</div>

39
website/docs/images/displacy-ent2.html
View File

@ -1,39 +0,0 @@
<div
class="entities"
style="line-height: 2.5; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 18px"
>
When
<mark
class="entity"
style="background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
Sebastian Thrun
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>PERSON</span
>
</mark>
started working on self-driving cars at
<mark
class="entity"
style="background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
Google
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>ORG</span
>
</mark>
in
<mark
class="entity"
style="background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em"
>
2007
<span
style="font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem"
>DATE</span
>
</mark>
, few people outside of the company took him seriously.
</div>

84
website/docs/images/displacy-long2.html
View File

@ -1,84 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" id="0" class="displacy" width="1275" height="399.5" style="max-width: none; height: 399.5px; color: #000000; background: #ffffff; font-family: Arial">
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="50">Autonomous</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="50">ADJ</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="225">cars</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="225">NOUN</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="400">shift</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="400">VERB</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="575">insurance</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="575">NOUN</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="750">liability</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="750">NOUN</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="925">toward</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="925">ADP</tspan>
</text>
<text class="displacy-token" fill="currentColor" text-anchor="middle" y="309.5">
<tspan class="displacy-word" fill="currentColor" x="1100">manufacturers</tspan>
<tspan class="displacy-tag" dy="2em" fill="currentColor" x="1100">NOUN</tspan>
</text>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-0" stroke-width="2px" d="M70,264.5 C70,177.0 215.0,177.0 215.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-0" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">amod</textpath>
</text>
<path class="displacy-arrowhead" d="M70,266.5 L62,254.5 78,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-1" stroke-width="2px" d="M245,264.5 C245,177.0 390.0,177.0 390.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-1" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">nsubj</textpath>
</text>
<path class="displacy-arrowhead" d="M245,266.5 L237,254.5 253,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-2" stroke-width="2px" d="M595,264.5 C595,177.0 740.0,177.0 740.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-2" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">compound</textpath>
</text>
<path class="displacy-arrowhead" d="M595,266.5 L587,254.5 603,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-3" stroke-width="2px" d="M420,264.5 C420,89.5 745.0,89.5 745.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-3" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">dobj</textpath>
</text>
<path class="displacy-arrowhead" d="M745.0,266.5 L753.0,254.5 737.0,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-4" stroke-width="2px" d="M420,264.5 C420,2.0 925.0,2.0 925.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-4" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">prep</textpath>
</text>
<path class="displacy-arrowhead" d="M925.0,266.5 L933.0,254.5 917.0,254.5" fill="currentColor"></path>
</g>
<g class="displacy-arrow">
<path class="displacy-arc" id="arrow-0-5" stroke-width="2px" d="M945,264.5 C945,177.0 1090.0,177.0 1090.0,264.5" fill="none" stroke="currentColor"></path>
<text dy="1.25em" style="font-size: 0.8em; letter-spacing: 1px">
<textpath xlink:href="#arrow-0-5" class="displacy-label" startOffset="50%" fill="currentColor" text-anchor="middle">pobj</textpath>
</text>
<path class="displacy-arrowhead" d="M1090.0,266.5 L1098.0,254.5 1082.0,254.5" fill="currentColor"></path>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 5.0 KiB

31
website/docs/images/displacy-span-custom.html
View File

@ -1,31 +0,0 @@
<div class="spans"
style="line-height: 2.5; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 18px; direction: ltr">
Welcome to the
<span style="font-weight: bold; display: inline-block; position: relative;">
Bank
<span
style="background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
<span
style="background: #ddd; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;">
<span
style="background: #ddd; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px">
BANK
</span>
</span>
</span>
<span style="font-weight: bold; display: inline-block; position: relative;">
of
<span
style="background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
</span>
<span style="font-weight: bold; display: inline-block; position: relative;">
China
<span
style="background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
</span>
.
</div>

41
website/docs/images/displacy-span.html
View File

@ -1,41 +0,0 @@
<div class="spans"
style="line-height: 2.5; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 18px">
Welcome to the
<span style="font-weight: bold; display: inline-block; position: relative;">
Bank
<span
style="background: #7aecec; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
<span
style="background: #7aecec; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;">
<span
style="background: #7aecec; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px">
ORG
</span>
</span>
</span>
<span style="font-weight: bold; display: inline-block; position: relative;">
of
<span
style="background: #7aecec; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
</span>
<span style="font-weight: bold; display: inline-block; position: relative;">
China
<span
style="background: #7aecec; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
<span
style="background: #feca74; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;">
</span>
<span
style="background: #feca74; top: 57px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;">
<span
style="background: #feca74; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px">
GPE
</span>
</span>
</span>
.
</div>

6
website/docs/index.md
View File

@ -1,6 +0,0 @@
---
---
import Landing from 'widgets/landing.js'
<Landing />

22
website/docs/models/index.md → website/docs/models/index.mdx
View File

@ -7,7 +7,7 @@ menu:
- ['Pipeline Design', 'design']
---
<!-- TODO: include interactive demo -->
{/* TODO: include interactive demo */}
### Quickstart {hidden="true"}
@ -16,11 +16,9 @@ menu:
> For more details on how to use trained pipelines with spaCy, see the
> [usage guide](/usage/models).
import QuickstartModels from 'widgets/quickstart-models.js'
<QuickstartModels id="quickstart" />
## Package naming conventions {#conventions}
## Package naming conventions {id="conventions"}
In general, spaCy expects all pipeline packages to follow the naming convention
of `[lang]\_[name]`. For spaCy's pipelines, we also chose to divide the name
@ -45,7 +43,7 @@ For example, [`en_core_web_sm`](/models/en#en_core_web_sm) is a small English
pipeline trained on written web text (blogs, news, comments), that includes
vocabulary, syntax and entities.
### Package versioning {#model-versioning}
### Package versioning {id="model-versioning"}
Additionally, the pipeline package versioning reflects both the compatibility
with spaCy, as well as the model version. A package version `a.b.c` translates
@ -62,7 +60,7 @@ For a detailed compatibility overview, see the
This is also the source of spaCy's internal compatibility check, performed when
you run the [`download`](/api/cli#download) command.
## Trained pipeline design {#design}
## Trained pipeline design {id="design"}
The spaCy v3 trained pipelines are designed to be efficient and configurable.
For example, multiple components can share a common "token-to-vector" model and
@ -89,9 +87,9 @@ Main changes from spaCy v2 models:
- The lemmatizer tables and processing move from the vocab and tagger to a
separate `lemmatizer` component.
### CNN/CPU pipeline design {#design-cnn}
### CNN/CPU pipeline design {id="design-cnn"}
![Components and their dependencies in the CNN pipelines](../images/pipeline-design.svg)
![Components and their dependencies in the CNN pipelines](/images/pipeline-design.svg)
In the `sm`/`md`/`lg` models:
@ -132,13 +130,13 @@ vector keys for default vectors.
- [`Vectors.most_similar`](/api/vectors#most_similar) is not supported because
there's no fixed list of vectors to compare your vectors to.
### Transformer pipeline design {#design-trf}
### Transformer pipeline design {id="design-trf"}
In the transformer (`trf`) models, the `tagger`, `parser` and `ner` (if present)
all listen to the `transformer` component. The `attribute_ruler` and
`lemmatizer` have the same configuration as in the CNN models.
### Modifying the default pipeline {#design-modify}
### Modifying the default pipeline {id="design-modify"}
For faster processing, you may only want to run a subset of the components in a
trained pipeline. The `disable` and `exclude` arguments to
@ -189,8 +187,8 @@ than the rule-based `sentencizer`.
#### Switch from trainable lemmatizer to default lemmatizer
Since v3.3, a number of pipelines use a trainable lemmatizer. You can check whether
the lemmatizer is trainable:
Since v3.3, a number of pipelines use a trainable lemmatizer. You can check
whether the lemmatizer is trainable:
```python
nlp = spacy.load("de_core_web_sm")

165
website/docs/styleguide.md → website/docs/styleguide.mdx
View File

@ -42,9 +42,7 @@ enough, JSX components can be used.
> For more details on editing the site locally, see the installation
> instructions and markdown reference below.
## Logo {#logo source="website/src/images/logo.svg"}
import { Logos } from 'widgets/styleguide'
## Logo {id="logo",source="website/src/images/logo.svg"}
If you would like to use the spaCy logo on your site, please get in touch and
ask us first. However, if you want to show support and tell others that your
@ -53,9 +51,7 @@ project is using spaCy, you can grab one of our
<Logos />
## Colors {#colors}
import { Colors, Patterns } from 'widgets/styleguide'
## Colors {id="colors"}
<Colors />
@ -63,17 +59,16 @@ import { Colors, Patterns } from 'widgets/styleguide'
<Patterns />
## Typography {#typography}
import { H1, H2, H3, H4, H5, Label, InlineList, Comment } from
'components/typography'
## Typography {id="typography"}
> #### Markdown
>
> ```markdown_
> ```markdown
> ## Headline 2
> ## Headline 2 {#some_id}
> ## Headline 2 {#some_id tag="method"}
>
> ## Headline 2 {id="some_id"}
>
> ## Headline 2 {id="some_id" tag="method"}
> ```
>
> #### JSX
@ -101,12 +96,11 @@ in the sidebar menu.
</Infobox>
<div>
<H1>Headline 1</H1>
<H2>Headline 2</H2>
<H3>Headline 3</H3>
<H4>Headline 4</H4>
<H5>Headline 5</H5>
<Label>Label</Label>
<H2>Headline 2</H2>
<H3>Headline 3</H3>
<H4>Headline 4</H4>
<H5>Headline 5</H5>
<Label>Label</Label>
</div>
---
@ -116,16 +110,16 @@ example, to add a tag for the documented type or mark features that have been
introduced in a specific version or require statistical models to be loaded.
Tags are also available as standalone `<Tag />` components.
| Argument | Example | Result |
| -------- | -------------------------- | ----------------------------------------- |
| `tag` | `{tag="method"}` | <Tag>method</Tag> |
| `new` | `{new="3"}` | <Tag variant="new">3</Tag> |
| `model` | `{model="tagger, parser"}` | <Tag variant="model">tagger, parser</Tag> |
| `hidden` | `{hidden="true"}` | |
| Argument | Example | Result |
| --------- | -------------------------- | ----------------------------------------- |
| `tag` | `{tag="method"}` | <Tag>method</Tag> |
| `version` | `{version="3"}` | <Tag variant="new">3</Tag> |
| `model` | `{model="tagger, parser"}` | <Tag variant="model">tagger, parser</Tag> |
| `hidden` | `{hidden="true"}` | |
## Elements {#elements}
## Elements {id="elements"}
### Links {#links}
### Links {id="links"}
> #### Markdown
>
@ -147,9 +141,7 @@ Special link styles are used depending on the link URL.
- [I am a link to a model](/models/en#en_core_web_sm)
- [I am a link to GitHub](https://github.com/explosion/spaCy)
### Abbreviations {#abbr}
import { Abbr } from 'components/typography'
### Abbreviations {id="abbr"}
> #### JSX
>
@ -161,13 +153,11 @@ Some text with <Abbr title="Explanation here">an abbreviation</Abbr>. On small
screens, I collapse and the explanation text is displayed next to the
abbreviation.
### Tags {#tags}
import Tag from 'components/tag'
### Tags {id="tags"}
> ```jsx
> <Tag>method</Tag>
> <Tag variant="new">4</Tag>
> <Tag variant="version">4</Tag>
> <Tag variant="model">tagger, parser</Tag>
> ```
@ -180,16 +170,13 @@ new anymore. Setting `variant="model"` takes a description of model capabilities
and can be used to mark features that require a respective model to be
installed.
<InlineList>
<p>
<Tag>method</Tag>
<Tag variant="new">4</Tag>
<Tag variant="model">tagger, parser</Tag>
</p>
<Tag>method</Tag> <Tag variant="new">4</Tag> <Tag variant="model">tagger,
parser</Tag>
</InlineList>
### Buttons {#buttons}
import Button from 'components/button'
### Buttons {id="buttons"}
> ```jsx
> <Button to="#" variant="primary">Primary small</Button>
@ -200,21 +187,29 @@ Link buttons come in two variants, `primary` and `secondary` and two sizes, with
an optional `large` size modifier. Since they're mostly used as enhanced links,
the buttons are implemented as styled links instead of native button elements.
<InlineList><Button to="#" variant="primary">Primary small</Button>
<Button to="#" variant="secondary">Secondary small</Button></InlineList>
<p>
<Button to="#" variant="primary">Primary small</Button>
<br />
{' '}
<InlineList><Button to="#" variant="primary" large>Primary large</Button>
<Button to="#" variant="secondary" large>Secondary large</Button></InlineList>
<Button to="#" variant="secondary">Secondary small</Button>
</p>
<p>
<Button to="#" variant="primary">Primary small</Button>
{' '}
<Button to="#" variant="secondary">Secondary small</Button>
</p>
## Components
### Table {#table}
### Table {id="table"}
> #### Markdown
>
> ```markdown_
> ```markdown
> | Header 1 | Header 2 |
> | -------- | -------- |
> | Column 1 | Column 2 |
@ -248,7 +243,7 @@ be italicized:
> #### Markdown
>
> ```markdown_
> ```markdown
> | Header 1 | Header 2 | Header 3 |
> | -------- | -------- | -------- |
> | Column 1 | Column 2 | Column 3 |
@ -262,11 +257,11 @@ be italicized:
| _Hello_ | | |
| Column 1 | Column 2 | Column 3 |
### Type Annotations {#type-annotations}
### Type Annotations {id="type-annotations"}
> #### Markdown
>
> ```markdown_
> ```markdown
> ~~Model[List[Doc], Floats2d]~~
> ```
>
@ -295,9 +290,9 @@ always be the **last element** in the row.
> #### Markdown
>
> ```markdown_
> | Header 1 | Header 2 |
> | -------- | ----------------------- |
> ```markdown
> | Header 1 | Header 2 |
> | -------- | ---------------------- |
> | Column 1 | Column 2 ~~List[Doc]~~ |
> ```
@ -307,11 +302,11 @@ always be the **last element** in the row.
| `model` | The Thinc [`Model`](https://thinc.ai/docs/api-model) wrapping the transformer. ~~Model[List[Doc], FullTransformerBatch]~~ |
| `set_extra_annotations` | Function that takes a batch of `Doc` objects and transformer outputs and can set additional annotations on the `Doc`. ~~Callable[[List[Doc], FullTransformerBatch], None]~~ |
### List {#list}
### List {id="list"}
> #### Markdown
>
> ```markdown_
> ```markdown
> 1. One
> 2. Two
> ```
@ -338,12 +333,13 @@ automatically.
3. Lorem ipsum dolor
4. consectetur adipiscing elit
### Aside {#aside}
### Aside {id="aside"}
> #### Markdown
>
> ```markdown_
> ```markdown
> > #### Aside title
> >
> > This is aside text.
> ```
>
@ -363,11 +359,11 @@ To make them easier to use in Markdown, paragraphs formatted as blockquotes will
turn into asides by default. Level 4 headlines (with a leading `####`) will
become aside titles.
### Code Block {#code-block}
### Code Block {id="code-block"}
> #### Markdown
>
> ````markdown_
> ````markdown
> ```python
> ### This is a title
> import spacy
@ -388,8 +384,7 @@ to raw text with no highlighting. An optional label can be added as the first
line with the prefix `####` (Python-like) and `///` (JavaScript-like). the
indented block as plain text and preserve whitespace.
```python
### Using spaCy
```python {title="Using spaCy"}
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a sentence.")
@ -403,7 +398,7 @@ adding `{highlight="..."}` to the headline. Acceptable ranges are spans like
> #### Markdown
>
> ````markdown_
> ````markdown
> ```python
> ### This is a title {highlight="1-2"}
> import spacy
@ -411,8 +406,7 @@ adding `{highlight="..."}` to the headline. Acceptable ranges are spans like
> ```
> ````
```python
### Using the matcher {highlight="5-7"}
```python {title="Using the matcher",highlight="5-7"}
import spacy
from spacy.matcher import Matcher
@ -431,7 +425,7 @@ interactive widget defaults to a regular code block.
> #### Markdown
>
> ````markdown_
> ````markdown
> ```python
> ### {executable="true"}
> import spacy
@ -439,8 +433,7 @@ interactive widget defaults to a regular code block.
> ```
> ````
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a sentence.")
@ -454,7 +447,7 @@ original file is shown at the top of the widget.
> #### Markdown
>
> ````markdown_
> ````markdown
> ```python
> https://github.com/...
> ```
@ -470,9 +463,7 @@ original file is shown at the top of the widget.
https://github.com/explosion/spaCy/tree/master/spacy/language.py
```
### Infobox {#infobox}
import Infobox from 'components/infobox'
### Infobox {id="infobox"}
> #### JSX
>
@ -508,9 +499,7 @@ blocks.
</Infobox>
### Accordion {#accordion}
import Accordion from 'components/accordion'
### Accordion {id="accordion"}
> #### JSX
>
@ -537,9 +526,9 @@ sit amet dignissim justo congue.
</Accordion>
## Markdown reference {#markdown}
## Markdown reference {id="markdown"}
All page content and page meta lives in the `.md` files in the `/docs`
All page content and page meta lives in the `.mdx` files in the `/docs`
directory. The frontmatter block at the top of each file defines the page title
and other settings like the sidebar menu.
@ -548,7 +537,7 @@ and other settings like the sidebar menu.
title: Page title
---
## Headline starting a section {#some_id}
## Headline starting a section {id="some_id"}
This is a regular paragraph with a [link](https://spacy.io) and **bold text**.
@ -562,8 +551,7 @@ This is a regular paragraph with a [link](https://spacy.io) and **bold text**.
| -------- | -------- |
| Column 1 | Column 2 |
```python
### Code block title {highlight="2-3"}
```python {title="Code block title",highlight="2-3"}
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Hello world")
@ -585,7 +573,7 @@ In addition to the native markdown elements, you can use the components
[abbr]: https://spacy.io/styleguide#abbr
[tag]: https://spacy.io/styleguide#tag
## Editorial {#editorial}
## Editorial {id="editorial"}
- "spaCy" should always be spelled with a lowercase "s" and a capital "C",
unless it specifically refers to the Python package or Python import `spacy`
@ -609,21 +597,16 @@ In addition to the native markdown elements, you can use the components
- ❌ The [`Span`](/api/span) and [`Token`](/api/token) objects are views of a
[`Doc`](/api/doc). [`Span.as_doc`](/api/span#as_doc) creates a
[`Doc`](/api/doc) object from a [`Span`](/api/span).
* Other things we format as code are: references to trained pipeline packages
- Other things we format as code are: references to trained pipeline packages
like `en_core_web_sm` or file names like `code.py` or `meta.json`.
- ✅ After training, the `config.cfg` is saved to disk.
* [Type annotations](#type-annotations) are a special type of code formatting,
- [Type annotations](#type-annotations) are a special type of code formatting,
expressed by wrapping the text in `~~` instead of backticks. The result looks
like this: ~~List[Doc]~~. All references to known types will be linked
automatically.
- ✅ The model has the input type ~~List[Doc]~~ and it outputs a
~~List[Array2d]~~.
* We try to keep links meaningful but short.
- We try to keep links meaningful but short.
- ✅ For details, see the usage guide on
[training with custom code](/usage/training#custom-code).
- ❌ For details, see

12
website/docs/usage/101/_architecture.md → website/docs/usage/101/_architecture.mdx
View File

@ -14,9 +14,9 @@ of the pipeline. The `Language` object coordinates these components. It takes
raw text and sends it through the pipeline, returning an **annotated document**.
It also orchestrates training and serialization.
![Library architecture](../../images/architecture.svg)
![Library architecture {{w:1080, h:1254}}](/images/architecture.svg)
### Container objects {#architecture-containers}
### Container objects {id="architecture-containers"}
| Name | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
@ -29,7 +29,7 @@ It also orchestrates training and serialization.
| [`SpanGroup`](/api/spangroup) | A named collection of spans belonging to a `Doc`. |
| [`Token`](/api/token) | An individual token — i.e. a word, punctuation symbol, whitespace, etc. |
### Processing pipeline {#architecture-pipeline}
### Processing pipeline {id="architecture-pipeline"}
The processing pipeline consists of one or more **pipeline components** that are
called on the `Doc` in order. The tokenizer runs before the components. Pipeline
@ -39,7 +39,7 @@ rule-based modifications to the `Doc`. spaCy provides a range of built-in
components for different language processing tasks and also allows adding
[custom components](/usage/processing-pipelines#custom-components).
![The processing pipeline](../../images/pipeline.svg)
![The processing pipeline](/images/pipeline.svg)
| Name | Description |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------- |
@ -61,7 +61,7 @@ components for different language processing tasks and also allows adding
| [`Transformer`](/api/transformer) | Use a transformer model and set its outputs. |
| [Other functions](/api/pipeline-functions) | Automatically apply something to the `Doc`, e.g. to merge spans of tokens. |
### Matchers {#architecture-matchers}
### Matchers {id="architecture-matchers"}
Matchers help you find and extract information from [`Doc`](/api/doc) objects
based on match patterns describing the sequences you're looking for. A matcher
@ -73,7 +73,7 @@ operates on a `Doc` and gives you access to the matched tokens **in context**.
| [`Matcher`](/api/matcher) | Match sequences of tokens, based on pattern rules, similar to regular expressions. |
| [`PhraseMatcher`](/api/phrasematcher) | Match sequences of tokens based on phrases. |
### Other classes {#architecture-other}
### Other classes {id="architecture-other"}
| Name | Description |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------- |

0
website/docs/usage/101/_language-data.md → website/docs/usage/101/_language-data.mdx
View File

20
website/docs/usage/101/_named-entities.md → website/docs/usage/101/_named-entities.mdx
View File

@ -1,14 +1,13 @@
A named entity is a "real-world object" that's assigned a name – for example, a
person, a country, a product or a book title. spaCy can **recognize various
types of named entities in a document, by asking the model for a
prediction**. Because models are statistical and strongly depend on the
examples they were trained on, this doesn't always work _perfectly_ and might
need some tuning later, depending on your use case.
types of named entities in a document, by asking the model for a prediction**.
Because models are statistical and strongly depend on the examples they were
trained on, this doesn't always work _perfectly_ and might need some tuning
later, depending on your use case.
Named entities are available as the `ents` property of a `Doc`:
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -32,7 +31,8 @@ for ent in doc.ents:
Using spaCy's built-in [displaCy visualizer](/usage/visualizers), here's what
our example sentence and its named entities look like:
import DisplaCyEntHtml from 'images/displacy-ent1.html'; import { Iframe } from
'components/embed'
<Iframe title="displaCy visualization of entities" html={DisplaCyEntHtml} height={100} />
<Iframe
title="displaCy visualization of entities"
src="/images/displacy-ent1.html"
height={100}
/>

4
website/docs/usage/101/_pipelines.md → website/docs/usage/101/_pipelines.mdx
View File

@ -5,7 +5,7 @@ referred to as the **processing pipeline**. The pipeline used by the
and an entity recognizer. Each pipeline component returns the processed `Doc`,
which is then passed on to the next component.
![The processing pipeline](../../images/pipeline.svg)
![The processing pipeline](/images/pipeline.svg)
> - **Name**: ID of the pipeline component.
> - **Component:** spaCy's implementation of the component.
@ -35,8 +35,6 @@ the [config](/usage/training#config):
pipeline = ["tok2vec", "tagger", "parser", "ner"]
```
import Accordion from 'components/accordion.js'
<Accordion title="Does the order of pipeline components matter?" id="pipeline-components-order">
The statistical components like the tagger or parser are typically independent

12
website/docs/usage/101/_pos-deps.md → website/docs/usage/101/_pos-deps.mdx
View File

@ -11,8 +11,7 @@ Linguistic annotations are available as
efficiency. So to get the readable string representation of an attribute, we
need to add an underscore `_` to its name:
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -57,7 +56,8 @@ for token in doc:
Using spaCy's built-in [displaCy visualizer](/usage/visualizers), here's what
our example sentence and its dependencies look like:
import DisplaCyLongHtml from 'images/displacy-long.html'; import { Iframe } from
'components/embed'
<Iframe title="displaCy visualization of dependencies and entities" html={DisplaCyLongHtml} height={450} />
<Iframe
title="displaCy visualization of dependencies and entities"
src="/images/displacy-long.html"
height={450}
/>

0
website/docs/usage/101/_serialization.md → website/docs/usage/101/_serialization.mdx
View File

5
website/docs/usage/101/_tokenization.md → website/docs/usage/101/_tokenization.mdx
View File

@ -4,8 +4,7 @@ language. For example, punctuation at the end of a sentence should be split off
– whereas "U.K." should remain one token. Each `Doc` consists of individual
tokens, and we can iterate over them:
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -41,7 +40,7 @@ marks.
> - **Suffix:** Character(s) at the end, e.g. `km`, `)`, `”`, `!`.
> - **Infix:** Character(s) in between, e.g. `-`, `--`, `/`, `…`.
![Example of the tokenization process](../../images/tokenization.svg)
![Example of the tokenization process](/images/tokenization.svg)
While punctuation rules are usually pretty general, tokenizer exceptions
strongly depend on the specifics of the individual language. This is why each

8
website/docs/usage/101/_training.md → website/docs/usage/101/_training.mdx
View File

@ -10,9 +10,9 @@ any other information.
Training is an iterative process in which the model's predictions are compared
against the reference annotations in order to estimate the **gradient of the
loss**. The gradient of the loss is then used to calculate the gradient of the
weights through [backpropagation](https://thinc.ai/docs/backprop101). The gradients
indicate how the weight values should be changed so that the model's predictions
become more similar to the reference labels over time.
weights through [backpropagation](https://thinc.ai/docs/backprop101). The
gradients indicate how the weight values should be changed so that the model's
predictions become more similar to the reference labels over time.
> - **Training data:** Examples and their annotations.
> - **Text:** The input text the model should predict a label for.
@ -21,7 +21,7 @@ become more similar to the reference labels over time.
> Minimising the gradient of the weights should result in predictions that are
> closer to the reference labels on the training data.
![The training process](../../images/training.svg)
![The training process](/images/training.svg)
When training a model, we don't just want it to memorize our examples – we want
it to come up with a theory that can be **generalized across unseen data**.

18
website/docs/usage/101/_vectors-similarity.md → website/docs/usage/101/_vectors-similarity.mdx
View File

@ -1,12 +1,9 @@
import Infobox from 'components/infobox'
Similarity is determined by comparing **word vectors** or "word embeddings",
multi-dimensional meaning representations of a word. Word vectors can be
generated using an algorithm like
[word2vec](https://en.wikipedia.org/wiki/Word2vec) and usually look like this:
```python
### banana.vector
```python {title="banana.vector"}
array([2.02280000e-01, -7.66180009e-02, 3.70319992e-01,
3.28450017e-02, -4.19569999e-01, 7.20689967e-02,
-3.74760002e-01, 5.74599989e-02, -1.24009997e-02,
@ -44,8 +41,7 @@ the [`Token.vector`](/api/token#vector) attribute.
default to an average of their token vectors. You can also check if a token has
a vector assigned, and get the L2 norm, which can be used to normalize vectors.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_md")
@ -95,8 +91,7 @@ similarity.
> You should see that the similarity results are identical to the token
> similarity.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_md") # make sure to use larger package!
@ -111,7 +106,7 @@ burgers = doc1[5]
print(french_fries, "<->", burgers, french_fries.similarity(burgers))
```
### What to expect from similarity results {#similarity-expectations}
### What to expect from similarity results {id="similarity-expectations"}
Computing similarity scores can be helpful in many situations, but it's also
important to maintain **realistic expectations** about what information it can
@ -136,7 +131,10 @@ useful for your purpose. Here are some important considerations to keep in mind:
<Infobox title="Tip: Check out sense2vec" emoji="💡">
[![](../../images/sense2vec.jpg)](https://github.com/explosion/sense2vec)
<Image
src="/images/sense2vec.jpg"
href="https://github.com/explosion/sense2vec"
/>
[`sense2vec`](https://github.com/explosion/sense2vec) is a library developed by
us that builds on top of spaCy and lets you train and query more interesting and

10
website/docs/usage/_benchmarks-models.md → website/docs/usage/_benchmarks-models.mdx
View File

@ -1,5 +1,3 @@
import { Help } from 'components/typography'; import Link from 'components/link'
<figure>
| Pipeline | Parser | Tagger | NER |
@ -8,7 +6,7 @@ import { Help } from 'components/typography'; import Link from 'components/link'
| [`en_core_web_lg`](/models/en#en_core_web_lg) (spaCy v3) | 92.0 | 97.4 | 85.5 |
| `en_core_web_lg` (spaCy v2) | 91.9 | 97.2 | 85.5 |
<figcaption class="caption">
<figcaption className="caption">
**Full pipeline accuracy** on the
[OntoNotes 5.0](https://catalog.ldc.upenn.edu/LDC2013T19) corpus (reported on
@ -26,15 +24,15 @@ the development set).
| Stanza (StanfordNLP)<sup>1</sup> | 88.8 | 92.1 |
| Flair<sup>2</sup> | 89.7 | 93.1 |
<figcaption class="caption">
<figcaption className="caption">
**Named entity recognition accuracy** on the
[OntoNotes 5.0](https://catalog.ldc.upenn.edu/LDC2013T19) and
[CoNLL-2003](https://www.aclweb.org/anthology/W03-0419.pdf) corpora. See
[NLP-progress](http://nlpprogress.com/english/named_entity_recognition.html) for
more results. Project template:
[`benchmarks/ner_conll03`](%%GITHUB_PROJECTS/benchmarks/ner_conll03). **1. **
[Qi et al. (2020)](https://arxiv.org/pdf/2003.07082.pdf). **2. **
[`benchmarks/ner_conll03`](%%GITHUB_PROJECTS/benchmarks/ner_conll03). **1.**
[Qi et al. (2020)](https://arxiv.org/pdf/2003.07082.pdf). **2.**
[Akbik et al. (2018)](https://www.aclweb.org/anthology/C18-1139/).
</figcaption>

99
website/docs/usage/embeddings-transformers.md → website/docs/usage/embeddings-transformers.mdx
View File

@ -74,7 +74,7 @@ of performance.
</Accordion>
## Shared embedding layers {#embedding-layers}
## Shared embedding layers {id="embedding-layers"}
spaCy lets you share a single transformer or other token-to-vector ("tok2vec")
embedding layer between multiple components. You can even update the shared
@ -85,7 +85,7 @@ difficult to swap components or retrain parts of the pipeline. Multi-task
learning can affect your accuracy (either positively or negatively), and may
require some retuning of your hyper-parameters.
![Pipeline components using a shared embedding component vs. independent embedding layers](../images/tok2vec.svg)
![Pipeline components using a shared embedding component vs. independent embedding layers](/images/tok2vec.svg)
| Shared | Independent |
| ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
@ -99,7 +99,7 @@ components by adding a [`Transformer`](/api/transformer) or
later in the pipeline can "connect" to it by including a **listener layer** like
[Tok2VecListener](/api/architectures#Tok2VecListener) within their model.
![Pipeline components listening to shared embedding component](../images/tok2vec-listener.svg)
![Pipeline components listening to shared embedding component](/images/tok2vec-listener.svg)
At the beginning of training, the [`Tok2Vec`](/api/tok2vec) component will grab
a reference to the relevant listener layers in the rest of your pipeline. When
@ -113,7 +113,7 @@ transformer outputs to the
[`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute,
giving you access to them after the pipeline has finished running.
### Example: Shared vs. independent config {#embedding-layers-config}
### Example: Shared vs. independent config {id="embedding-layers-config"}
The [config system](/usage/training#config) lets you express model configuration
for both shared and independent embedding layers. The shared setup uses a single
@ -123,8 +123,7 @@ the entity recognizer, use a
[Tok2VecListener](/api/architectures#Tok2VecListener) layer as their model's
`tok2vec` argument, which connects to the `tok2vec` component model.
```ini
### Shared {highlight="1-2,4-5,19-20"}
```ini {title="Shared",highlight="1-2,4-5,19-20"}
[components.tok2vec]
factory = "tok2vec"
@ -152,8 +151,7 @@ In the independent setup, the entity recognizer component defines its own
same. This makes them fully independent and doesn't require an upstream
[`Tok2Vec`](/api/tok2vec) component to be present in the pipeline.
```ini
### Independent {highlight="7-8"}
```ini {title="Independent", highlight="7-8"}
[components.ner]
factory = "ner"
@ -170,9 +168,9 @@ factory = "ner"
@architectures = "spacy.MaxoutWindowEncoder.v2"
```
<!-- TODO: Once rehearsal is tested, mention it here. -->
{/* TODO: Once rehearsal is tested, mention it here. */}
## Using transformer models {#transformers}
## Using transformer models {id="transformers"}
Transformers are a family of neural network architectures that compute **dense,
context-sensitive representations** for the tokens in your documents. Downstream
@ -188,7 +186,7 @@ transformer models, but for practical purposes, you can simply think of them as
drop-in replacements that let you achieve **higher accuracy** in exchange for
**higher training and runtime costs**.
### Setup and installation {#transformers-installation}
### Setup and installation {id="transformers-installation"}
> #### System requirements
>
@ -210,22 +208,20 @@ your package manager and CUDA version. If you skip this step, pip will install
PyTorch as a dependency below, but it may not find the best version for your
setup.
```bash
### Example: Install PyTorch 1.11.0 for CUDA 11.3 with pip
```bash {title="Example: Install PyTorch 1.11.0 for CUDA 11.3 with pip"}
# See: https://pytorch.org/get-started/locally/
$ pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
```
Next, install spaCy with the extras for your CUDA version and transformers. The
CUDA extra (e.g., `cuda102`, `cuda113`) installs the correct version of
[`cupy`](https://docs.cupy.dev/en/stable/install.html#installing-cupy), which
is just like `numpy`, but for GPU. You may also need to set the `CUDA_PATH`
[`cupy`](https://docs.cupy.dev/en/stable/install.html#installing-cupy), which is
just like `numpy`, but for GPU. You may also need to set the `CUDA_PATH`
environment variable if your CUDA runtime is installed in a non-standard
location. Putting it all together, if you had installed CUDA 11.3 in
`/opt/nvidia/cuda`, you would run:
```bash
### Installation with CUDA
```bash {title="Installation with CUDA"}
$ export CUDA_PATH="/opt/nvidia/cuda"
$ pip install -U %%SPACY_PKG_NAME[cuda113,transformers]%%SPACY_PKG_FLAGS
```
@ -235,12 +231,11 @@ that require [`SentencePiece`](https://github.com/google/sentencepiece) (e.g.,
ALBERT, CamemBERT, XLNet, Marian, and T5), install the additional dependencies
with:
```bash
### Install sentencepiece
```bash {title="Install sentencepiece"}
$ pip install transformers[sentencepiece]
```
### Runtime usage {#transformers-runtime}
### Runtime usage {id="transformers-runtime"}
Transformer models can be used as **drop-in replacements** for other types of
neural networks, so your spaCy pipeline can include them in a way that's
@ -249,7 +244,7 @@ the standard way, like any other spaCy pipeline. Instead of using the
transformers as subnetworks directly, you can also use them via the
[`Transformer`](/api/transformer) pipeline component.
![The processing pipeline with the transformer component](../images/pipeline_transformer.svg)
![The processing pipeline with the transformer component](/images/pipeline_transformer.svg)
The `Transformer` component sets the
[`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute,
@ -257,12 +252,11 @@ which lets you access the transformers outputs at runtime. The trained
transformer-based [pipelines](/models) provided by spaCy end on `_trf`, e.g.
[`en_core_web_trf`](/models/en#en_core_web_trf).
```cli
```bash
$ python -m spacy download en_core_web_trf
```
```python
### Example
```python {title="Example"}
import spacy
from thinc.api import set_gpu_allocator, require_gpu
@ -299,7 +293,7 @@ assert isinstance(doc._.custom_attr, TransformerData)
print(doc._.custom_attr.tensors)
```
### Training usage {#transformers-training}
### Training usage {id="transformers-training"}
The recommended workflow for training is to use spaCy's
[config system](/usage/training#config), usually via the
@ -309,14 +303,13 @@ of objects by referring to creation functions, including functions you register
yourself. For details on how to get started with training your own model, check
out the [training quickstart](/usage/training#quickstart).
<!-- TODO: <Project id="pipelines/transformers">
{/* TODO: <Project id="pipelines/transformers"> */}
The easiest way to get started is to clone a transformers-based project
template. Swap in your data, edit the settings and hyperparameters and train,
evaluate, package and visualize your model.
{/* The easiest way to get started is to clone a transformers-based project */}
{/* template. Swap in your data, edit the settings and hyperparameters and train, */}
{/* evaluate, package and visualize your model. */}
</Project>
-->
{/* </Project> */}
The `[components]` section in the [`config.cfg`](/api/data-formats#config)
describes the pipeline components and the settings used to construct them,
@ -344,8 +337,7 @@ component:
> )
> ```
```ini
### config.cfg (excerpt)
```ini {title="config.cfg",excerpt="true"}
[components.transformer]
factory = "transformer"
max_batch_items = 4096
@ -405,7 +397,7 @@ all defaults.
</Infobox>
### Customizing the settings {#transformers-training-custom-settings}
### Customizing the settings {id="transformers-training-custom-settings"}
To change any of the settings, you can edit the `config.cfg` and re-run the
training. To change any of the functions, like the span getter, you can replace
@ -425,8 +417,7 @@ subsentences of at most `max_length` tokens are returned.
> max_length = 25
> ```
```python
### code.py
```python {title="code.py"}
import spacy_transformers
@spacy_transformers.registry.span_getters("custom_sent_spans")
@ -454,11 +445,11 @@ function. You can make it available via the `--code` argument that can point to
a Python file. For more details on training with custom code, see the
[training documentation](/usage/training#custom-functions).
```cli
```bash
python -m spacy train ./config.cfg --code ./code.py
```
### Customizing the model implementations {#training-custom-model}
### Customizing the model implementations {id="training-custom-model"}
The [`Transformer`](/api/transformer) component expects a Thinc
[`Model`](https://thinc.ai/docs/api-model) object to be passed in as its `model`
@ -476,8 +467,7 @@ is where we'll plug in our transformer model, using the
[TransformerListener](/api/architectures#TransformerListener) layer, which
sneakily delegates to the `Transformer` pipeline component.
```ini
### config.cfg (excerpt) {highlight="12"}
```ini {title="config.cfg (excerpt)",highlight="12"}
[components.ner]
factory = "ner"
@ -517,7 +507,7 @@ custom learning rate for each component. Instead of a constant, you can also
provide a schedule, allowing you to freeze the shared parameters at the start of
training.
## Static vectors {#static-vectors}
## Static vectors {id="static-vectors"}
If your pipeline includes a **word vectors table**, you'll be able to use the
`.similarity()` method on the [`Doc`](/api/doc), [`Span`](/api/span),
@ -531,8 +521,7 @@ Word vectors in spaCy are "static" in the sense that they are not learned
parameters of the statistical models, and spaCy itself does not feature any
algorithms for learning word vector tables. You can train a word vectors table
using tools such as [floret](https://github.com/explosion/floret),
[Gensim](https://radimrehurek.com/gensim/),
[FastText](https://fasttext.cc/) or
[Gensim](https://radimrehurek.com/gensim/), [FastText](https://fasttext.cc/) or
[GloVe](https://nlp.stanford.edu/projects/glove/), or download existing
pretrained vectors. The [`init vectors`](/api/cli#init-vectors) command lets you
convert vectors for use with spaCy and will give you a directory you can load or
@ -547,7 +536,7 @@ the usage guide on
</Infobox>
### Using word vectors in your models {#word-vectors-models}
### Using word vectors in your models {id="word-vectors-models"}
Many neural network models are able to use word vector tables as additional
features, which sometimes results in significant improvements in accuracy.
@ -580,7 +569,7 @@ handled by the [StaticVectors](/api/architectures#StaticVectors) layer.
</Infobox>
#### Creating a custom embedding layer {#custom-embedding-layer}
#### Creating a custom embedding layer {id="custom-embedding-layer"}
The [MultiHashEmbed](/api/architectures#StaticVectors) layer is spaCy's
recommended strategy for constructing initial word representations for your
@ -643,7 +632,7 @@ def MyCustomVectors(
)
```
## Pretraining {#pretraining}
## Pretraining {id="pretraining"}
The [`spacy pretrain`](/api/cli#pretrain) command lets you initialize your
models with **information from raw text**. Without pretraining, the models for
@ -679,14 +668,14 @@ You can add a `[pretraining]` block to your config by setting the
`--pretraining` flag on [`init config`](/api/cli#init-config) or
[`init fill-config`](/api/cli#init-fill-config):
```cli
```bash
$ python -m spacy init fill-config config.cfg config_pretrain.cfg --pretraining
```
You can then run [`spacy pretrain`](/api/cli#pretrain) with the updated config
and pass in optional config overrides, like the path to the raw text file:
```cli
```bash
$ python -m spacy pretrain config_pretrain.cfg ./output --paths.raw_text text.jsonl
```
@ -700,7 +689,7 @@ change the [objective](#pretraining-objectives).
%%GITHUB_SPACY/spacy/default_config_pretraining.cfg
```
### How pretraining works {#pretraining-details}
### How pretraining works {id="pretraining-details"}
The impact of [`spacy pretrain`](/api/cli#pretrain) varies, but it will usually
be worth trying if you're **not using a transformer** model and you have
@ -726,7 +715,7 @@ a "tok2vec" layer). The most common workflow is to use the
[`Tok2Vec`](/api/tok2vec) component to create a shared token-to-vector layer for
several components of your pipeline, and apply pretraining to its whole model.
#### Configuring the pretraining {#pretraining-configure}
#### Configuring the pretraining {id="pretraining-configure"}
The [`spacy pretrain`](/api/cli#pretrain) command is configured using the
`[pretraining]` section of your [config file](/usage/training#config). The
@ -737,8 +726,7 @@ whole model), or a
spaCy's built-in model architectures have a reference named `"tok2vec"` that
will refer to the right layer.
```ini
### config.cfg
```ini {title="config.cfg"}
# 1. Use the whole model of the "tok2vec" component
[pretraining]
component = "tok2vec"
@ -750,7 +738,7 @@ component = "textcat"
layer = "tok2vec"
```
#### Connecting pretraining to training {#pretraining-training}
#### Connecting pretraining to training {id="pretraining-training"}
To benefit from pretraining, your training step needs to know to initialize its
`tok2vec` component with the weights learned from the pretraining step. You do
@ -761,8 +749,7 @@ A pretraining step that runs for 5 epochs with an output path of `pretrain/`, as
an example, produces `pretrain/model0.bin` through `pretrain/model4.bin`. To
make use of the final output, you could fill in this value in your config file:
```ini
### config.cfg
```ini {title="config.cfg"}
[paths]
init_tok2vec = "pretrain/model4.bin"
@ -781,7 +768,7 @@ an existing pipeline, so it goes in `initialize.init_tok2vec`.
</Infobox>
#### Pretraining objectives {#pretraining-objectives}
#### Pretraining objectives {id="pretraining-objectives"}
> ```ini
> ### Characters objective

24
website/docs/usage/facts-figures.md → website/docs/usage/facts-figures.mdx
View File

@ -8,7 +8,7 @@ menu:
# TODO: - ['Citing spaCy', 'citation']
---
## Comparison {#comparison hidden="true"}
## Comparison {id="comparison",hidden="true"}
spaCy is a **free, open-source library** for advanced **Natural Language
Processing** (NLP) in Python. It's designed specifically for **production use**
@ -16,13 +16,11 @@ and helps you build applications that process and "understand" large volumes of
text. It can be used to build information extraction or natural language
understanding systems.
### Feature overview {#comparison-features}
import Features from 'widgets/features.js'
### Feature overview {id="comparison-features"}
<Features />
### When should I use spaCy? {#comparison-usage}
### When should I use spaCy? {id="comparison-usage"}
- ✅ **I'm a beginner and just getting started with NLP.** – spaCy makes it easy
to get started and comes with extensive documentation, including a
@ -51,13 +49,13 @@ import Features from 'widgets/features.js'
can use it to make the results of your research easily available for others to
use, e.g. via a custom spaCy component.
## Benchmarks {#benchmarks}
## Benchmarks {id="benchmarks"}
spaCy v3.0 introduces transformer-based pipelines that bring spaCy's accuracy
right up to **current state-of-the-art**. You can also use a CPU-optimized
pipeline, which is less accurate but much cheaper to run.
<!-- TODO: update benchmarks and intro -->
{/* TODO: update benchmarks and intro */}
> #### Evaluation details
>
@ -69,8 +67,6 @@ pipeline, which is less accurate but much cheaper to run.
> gold-standard segmentation and tokenization, from a pretty specific type of
> text (articles from a single newspaper, 1984-1989).
import Benchmarks from 'usage/\_benchmarks-models.md'
<Benchmarks />
<figure>
@ -81,7 +77,7 @@ import Benchmarks from 'usage/\_benchmarks-models.md'
| [Mrini et al.](https://khalilmrini.github.io/Label_Attention_Layer.pdf) (2019) | 97.4 | 96.3 |
| [Zhou and Zhao](https://www.aclweb.org/anthology/P19-1230/) (2019) | 97.2 | 95.7 |
<figcaption class="caption">
<figcaption className="caption">
**Dependency parsing accuracy** on the Penn Treebank. See
[NLP-progress](http://nlpprogress.com/english/dependency_parsing.html) for more
@ -92,7 +88,7 @@ results. Project template:
</figure>
### Speed comparison {#benchmarks-speed}
### Speed comparison {id="benchmarks-speed"}
We compare the speed of different NLP libraries, measured in words per second
(WPS) - higher is better. The evaluation was performed on 10,000 Reddit
@ -108,7 +104,7 @@ comments.
| Flair | `pos`(`-fast`) & `ner`(`-fast`) | 323 | 1,184 |
| UDPipe | `english-ewt-ud-2.5` | 1,101 | _n/a_ |
<figcaption class="caption">
<figcaption className="caption">
**End-to-end processing speed** on raw unannotated text. Project template:
[`benchmarks/speed`](%%GITHUB_PROJECTS/benchmarks/speed).
@ -117,6 +113,4 @@ comments.
</figure>
<!-- TODO: ## Citing spaCy {#citation}
-->
{/* TODO: ## Citing spaCy {id="citation"} */}

61
website/docs/usage/index.md → website/docs/usage/index.mdx
View File

@ -16,18 +16,16 @@ menu:
> website to [**v2.spacy.io**](https://v2.spacy.io/docs). To see what's changed
> and how to migrate, see the [v3.0 guide](/usage/v3).
import QuickstartInstall from 'widgets/quickstart-install.js'
<QuickstartInstall id="quickstart" />
## Installation instructions {#installation}
## Installation instructions {id="installation"}
spaCy is compatible with **64-bit CPython 3.6+** and runs on **Unix/Linux**,
**macOS/OS X** and **Windows**. The latest spaCy releases are available over
[pip](https://pypi.python.org/pypi/spacy) and
[conda](https://anaconda.org/conda-forge/spacy).
### pip {#pip}
### pip {id="pip"}
Using pip, spaCy releases are available as source packages and binary wheels.
Before you install spaCy and its dependencies, make sure that your `pip`,
@ -38,7 +36,7 @@ Before you install spaCy and its dependencies, make sure that your `pip`,
> After installation you typically want to download a trained pipeline. For more
> info and available packages, see the [models directory](/models).
>
> ```cli
> ```bash
> $ python -m spacy download en_core_web_sm
>
> >>> import spacy
@ -79,7 +77,7 @@ spaCy's [`setup.cfg`](%%GITHUB_SPACY/setup.cfg) for details on what's included.
| `apple` | Install [`thinc-apple-ops`](https://github.com/explosion/thinc-apple-ops) to improve performance on an Apple M1. |
| `ja`, `ko`, `th` | Install additional dependencies required for tokenization for the [languages](/usage/models#languages). |
### conda {#conda}
### conda {id="conda"}
Thanks to our great community, we've been able to re-add conda support. You can
also install spaCy via `conda-forge`:
@ -92,7 +90,7 @@ For the feedstock including the build recipe and configuration, check out
[this repository](https://github.com/conda-forge/spacy-feedstock). Note that we
currently don't publish any [pre-releases](#changelog-pre) on conda.
### Upgrading spaCy {#upgrading}
### Upgrading spaCy {id="upgrading"}
> #### Upgrading from v2 to v3
>
@ -116,12 +114,12 @@ version. If incompatible packages are found, tips and installation instructions
are printed. It's recommended to run the command with `python -m` to make sure
you're executing the correct version of spaCy.
```cli
```bash
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
$ python -m spacy validate
```
### Run spaCy with GPU {#gpu new="2.0.14"}
### Run spaCy with GPU {id="gpu",version="2.0.14"}
As of v2.0, spaCy comes with neural network models that are implemented in our
machine learning library, [Thinc](https://thinc.ai). For GPU support, we've been
@ -151,7 +149,7 @@ spacy.prefer_gpu()
nlp = spacy.load("en_core_web_sm")
```
### Compile from source {#source}
### Compile from source {id="source"}
The other way to install spaCy is to clone its
[GitHub repository](https://github.com/explosion/spaCy) and build it from
@ -181,16 +179,13 @@ $ pip install --no-build-isolation --editable .[lookups,cuda102]
How to install compilers and related build tools:
<a id="source-ubuntu"></a><a id="source-osx"></a><a id="source-windows"></a>
- **Ubuntu:** Install system-level dependencies via `apt-get`:
`sudo apt-get install build-essential python-dev git`
- **macOS / OS X:** Install a recent version of
[XCode](https://developer.apple.com/xcode/), including the so-called "Command
Line Tools". macOS and OS X ship with Python and Git preinstalled.
- **Windows:** Install a version of the
[Visual C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)
or
- <strong id="source-ubuntu">Ubuntu:</strong> Install system-level dependencies via
`apt-get`: `sudo apt-get install build-essential python-dev git`
- <strong id="source-osx">macOS / OS X:</strong> Install a recent version of [XCode](https://developer.apple.com/xcode/),
including the so-called "Command Line Tools". macOS and OS X ship with Python and
Git preinstalled.
- <strong id="source-windows">Windows:</strong> Install a version of the [Visual
C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) or
[Visual Studio Express](https://www.visualstudio.com/vs/visual-studio-express/)
that matches the version that was used to compile your Python interpreter.
@ -203,7 +198,7 @@ your environment requires an older version of `numpy`.
If `numpy` gets downgraded from the most recent release at any point after
you've compiled `spacy`, you might see an error that looks like this:
```none
```
numpy.ndarray size changed, may indicate binary incompatibility.
```
@ -230,15 +225,15 @@ package to see what the oldest recommended versions of `numpy` are.
(_Warning_: don't use `pip install -c constraints.txt` instead of
`PIP_CONSTRAINT`, since this isn't applied to the isolated build environments.)
#### Additional options for developers {#source-developers}
#### Additional options for developers {id="source-developers"}
Some additional options may be useful for spaCy developers who are editing the
source code and recompiling frequently.
- Install in editable mode. Changes to `.py` files will be reflected as soon
as the files are saved, but edits to Cython files (`.pxd`, `.pyx`) will
require the `pip install` command below to be run again. Before installing in
editable mode, be sure you have removed any previous installs with
- Install in editable mode. Changes to `.py` files will be reflected as soon as
the files are saved, but edits to Cython files (`.pxd`, `.pyx`) will require
the `pip install` command below to be run again. Before installing in editable
mode, be sure you have removed any previous installs with
`pip uninstall spacy`, which you may need to run multiple times to remove all
traces of earlier installs.
@ -247,8 +242,8 @@ source code and recompiling frequently.
$ pip install --no-build-isolation --editable .
```
- Build in parallel. Starting in v3.4.0, you can specify the number of
build jobs with the environment variable `SPACY_NUM_BUILD_JOBS`:
- Build in parallel. Starting in v3.4.0, you can specify the number of build
jobs with the environment variable `SPACY_NUM_BUILD_JOBS`:
```bash
$ pip install -r requirements.txt
@ -264,7 +259,7 @@ source code and recompiling frequently.
$ python setup.py develop
```
### Building an executable {#executable}
### Building an executable {id="executable"}
The spaCy repository includes a [`Makefile`](%%GITHUB_SPACY/Makefile) that
builds an executable zip file using [`pex`](https://github.com/pantsbuild/pex)
@ -298,7 +293,7 @@ You can configure the build process with the following environment variables:
| `PYVER` | The Python version to build against. This version needs to be available on your build and runtime machines. Defaults to `3.6`. |
| `WHEELHOUSE` | Directory to store the wheel files during compilation. Defaults to `./wheelhouse`. |
### Run tests {#run-tests}
### Run tests {id="run-tests"}
spaCy comes with an [extensive test suite](%%GITHUB_SPACY/spacy/tests). In order
to run the tests, you'll usually want to clone the [repository](%%GITHUB_SPACY)
@ -324,7 +319,7 @@ $ python -m pytest --pyargs %%SPACY_PKG_NAME # basic tests
$ python -m pytest --pyargs %%SPACY_PKG_NAME --slow # basic and slow tests
```
## Troubleshooting guide {#troubleshooting}
## Troubleshooting guide {id="troubleshooting"}
This section collects some of the most common errors you may come across when
installing, loading and using spaCy, as well as their solutions. Also see the
@ -450,8 +445,6 @@ either of these, clone your repository again.
</Accordion>
## Changelog {#changelog}
import Changelog from 'widgets/changelog.js'
## Changelog {id="changelog"}
<Changelog />

138
website/docs/usage/layers-architectures.md → website/docs/usage/layers-architectures.mdx
View File

@ -40,8 +40,7 @@ this config, you won't be able to change it anymore. The architecture is like a
recipe for the network, and you can't change the recipe once the dish has
already been prepared. You have to make a new one.
```ini
### config.cfg (excerpt)
```ini {title="config.cfg (excerpt)"}
[components.tagger]
factory = "tagger"
@ -51,7 +50,7 @@ width = 512
classes = 16
```
## Type signatures {#type-sigs}
## Type signatures {id="type-sigs"}
> #### Example
>
@ -111,11 +110,14 @@ If you're using a modern editor like Visual Studio Code, you can
custom Thinc plugin and get live feedback about mismatched types as you write
code.
[![](../images/thinc_mypy.jpg)](https://thinc.ai/docs/usage-type-checking#linting)
<Image
src="/images/thinc_mypy.jpg"
href="https://thinc.ai/docs/usage-type-checking#linting"
/>
</Accordion>
## Swapping model architectures {#swap-architectures}
## Swapping model architectures {id="swap-architectures"}
If no model is specified for the [`TextCategorizer`](/api/textcategorizer), the
[TextCatEnsemble](/api/architectures#TextCatEnsemble) architecture is used by
@ -123,8 +125,7 @@ default. This architecture combines a simple bag-of-words model with a neural
network, usually resulting in the most accurate results, but at the cost of
speed. The config file for this model would look something like this:
```ini
### config.cfg (excerpt)
```ini {title="config.cfg (excerpt)"}
[components.textcat]
factory = "textcat"
labels = []
@ -162,8 +163,7 @@ use those by swapping out the definition of the textcat's model. For instance,
to use the simple and fast bag-of-words model
[TextCatBOW](/api/architectures#TextCatBOW), you can change the config to:
```ini
### config.cfg (excerpt) {highlight="6-10"}
```ini {title="config.cfg (excerpt)",highlight="6-10"}
[components.textcat]
factory = "textcat"
labels = []
@ -180,7 +180,7 @@ For details on all pre-defined architectures shipped with spaCy and how to
configure them, check out the [model architectures](/api/architectures)
documentation.
### Defining sublayers {#sublayers}
### Defining sublayers {id="sublayers"}
Model architecture functions often accept **sublayers as arguments**, so that
you can try **substituting a different layer** into the network. Depending on
@ -195,8 +195,7 @@ These steps together compute dense, context-sensitive representations of the
tokens, and their combination forms a typical
[`Tok2Vec`](/api/architectures#Tok2Vec) layer:
```ini
### config.cfg (excerpt)
```ini {title="config.cfg (excerpt)"}
[components.tok2vec]
factory = "tok2vec"
@ -217,8 +216,7 @@ a sublayer for another one, for instance changing the first sublayer to a
character embedding with the [CharacterEmbed](/api/architectures#CharacterEmbed)
architecture:
```ini
### config.cfg (excerpt)
```ini {title="config.cfg (excerpt)"}
[components.tok2vec.model.embed]
@architectures = "spacy.CharacterEmbed.v2"
# ...
@ -237,7 +235,7 @@ a transformer. And if you want to define your own solution, all you need to do
is register a ~~Model[List[Doc], List[Floats2d]]~~ architecture function, and
you'll be able to try it out in any of the spaCy components. ​
## Wrapping PyTorch, TensorFlow and other frameworks {#frameworks}
## Wrapping PyTorch, TensorFlow and other frameworks {id="frameworks"}
Thinc allows you to [wrap models](https://thinc.ai/docs/usage-frameworks)
written in other machine learning frameworks like PyTorch, TensorFlow and MXNet
@ -257,8 +255,7 @@ Let's use PyTorch to define a very simple neural network consisting of two
hidden `Linear` layers with `ReLU` activation and dropout, and a
softmax-activated output layer:
```python
### PyTorch model
```python {title="PyTorch model"}
from torch import nn
torch_model = nn.Sequential(
@ -300,7 +297,7 @@ layers, and "native" Thinc layers to do fiddly input and output transformations
and add on task-specific "heads", as efficiency is less of a consideration for
those parts of the network.
### Using wrapped models {#frameworks-usage}
### Using wrapped models {id="frameworks-usage"}
To use our custom model including the PyTorch subnetwork, all we need to do is
register the architecture using the
@ -309,8 +306,7 @@ architecture a name so spaCy knows how to find it, and allows passing in
arguments like hyperparameters via the [config](/usage/training#config). The
full example then becomes:
```python
### Registering the architecture {highlight="9"}
```python {title="Registering the architecture",highlight="9"}
from typing import List
from thinc.types import Floats2d
from thinc.api import Model, PyTorchWrapper, chain, with_array
@ -349,8 +345,7 @@ by specifying it in the config file. In this configuration, all required
parameters for the various subcomponents of the custom architecture are passed
in as settings via the config.
```ini
### config.cfg (excerpt) {highlight="5-5"}
```ini {title="config.cfg (excerpt)",highlight="5-5"}
[components.tagger]
factory = "tagger"
@ -378,13 +373,12 @@ GPU memory allocator accordingly. When `gpu_allocator` is set to "pytorch" or
respective libraries, preventing OOM errors when there's available memory
sitting in the other library's pool.
```ini
### config.cfg (excerpt)
```ini {title="config.cfg (excerpt)"}
[training]
gpu_allocator = "pytorch"
```
## Custom models with Thinc {#thinc}
## Custom models with Thinc {id="thinc"}
Of course it's also possible to define the `Model` from the previous section
entirely in Thinc. The Thinc documentation provides details on the
@ -418,7 +412,7 @@ the PyTorch layers are defined, where `in_features` precedes `out_features`.
</Infobox>
### Shape inference in Thinc {#thinc-shape-inference}
### Shape inference in Thinc {id="thinc-shape-inference"}
It is **not** strictly necessary to define all the input and output dimensions
for each layer, as Thinc can perform
@ -458,8 +452,7 @@ you have to call
[`Model.initialize`](https://thinc.ai/docs/api-model#initialize) with an **input
sample** `X` and an **output sample** `Y` with the correct dimensions:
```python
### Shape inference with initialization {highlight="3,7,10"}
```python {title="Shape inference with initialization",highlight="3,7,10"}
with Model.define_operators({">>": chain}):
layers = (
Relu(hidden_width)
@ -479,7 +472,7 @@ data. In this case, `X` is typically a ~~List[Doc]~~, while `Y` is typically a
functionality is triggered when [`nlp.initialize`](/api/language#initialize) is
called.
### Dropout and normalization in Thinc {#thinc-dropout-norm}
### Dropout and normalization in Thinc {id="thinc-dropout-norm"}
Many of the available Thinc [layers](https://thinc.ai/docs/api-layers) allow you
to define a `dropout` argument that will result in "chaining" an additional
@ -500,7 +493,7 @@ with Model.define_operators({">>": chain}):
model.initialize(X=input_sample, Y=output_sample)
```
## Create new trainable components {#components}
## Create new trainable components {id="components"}
In addition to [swapping out](#swap-architectures) layers in existing
components, you can also implement an entirely new,
@ -518,7 +511,7 @@ overview of the `TrainablePipe` methods used by
</Infobox>
### Example: Entity relation extraction component {#component-rel}
### Example: Entity relation extraction component {id="component-rel"}
This section outlines an example use-case of implementing a **novel relation
extraction component** from scratch. We'll implement a binary relation
@ -537,18 +530,18 @@ two major steps required:
pass through the `nlp` pipeline.
<Project id="tutorials/rel_component">
Run this example use-case by using our project template. It includes all the
code to create the ML model and the pipeline component from scratch.
It also contains two config files to train the model:
one to run on CPU with a Tok2Vec layer, and one for the GPU using a transformer.
The project applies the relation extraction component to identify biomolecular
interactions in a sample dataset, but you can easily swap in your own dataset
for your experiments in any other domain.
Run this example use-case by using our project template. It includes all the
code to create the ML model and the pipeline component from scratch. It also
contains two config files to train the model: one to run on CPU with a Tok2Vec
layer, and one for the GPU using a transformer. The project applies the
relation extraction component to identify biomolecular interactions in a
sample dataset, but you can easily swap in your own dataset for your
experiments in any other domain.
</Project>
<YouTube id="8HL-Ap5_Axo"></YouTube>
#### Step 1: Implementing the Model {#component-rel-model}
#### Step 1: Implementing the Model {id="component-rel-model"}
We need to implement a [`Model`](https://thinc.ai/docs/api-model) that takes a
**list of documents** (~~List[Doc]~~) as input, and outputs a **two-dimensional
@ -561,8 +554,7 @@ matrix** (~~Floats2d~~) of predictions:
> type checks and validation. See the section on [type signatures](#type-sigs)
> for details.
```python
### The model architecture
```python {title="The model architecture"}
@spacy.registry.architectures("rel_model.v1")
def create_relation_model(...) -> Model[List[Doc], Floats2d]:
model = ... # 👈 model will go here
@ -587,8 +579,7 @@ transforms the instance tensor into a final tensor holding the predictions:
> # ...
> ```
```python
### The model architecture {highlight="6"}
```python {title="The model architecture",highlight="6"}
@spacy.registry.architectures("rel_model.v1")
def create_relation_model(
create_instance_tensor: Model[List[Doc], Floats2d],
@ -611,8 +602,7 @@ The `classification_layer` could be something like a
> nO = null
> ```
```python
### The classification layer
```python {title="The classification layer"}
@spacy.registry.architectures("rel_classification_layer.v1")
def create_classification_layer(
nO: int = None, nI: int = None
@ -648,8 +638,7 @@ that has the full implementation.
> # ...
> ```
```python
### The layer that creates the instance tensor
```python {title="The layer that creates the instance tensor"}
@spacy.registry.architectures("rel_instance_tensor.v1")
def create_tensors(
tok2vec: Model[List[Doc], List[Floats2d]],
@ -729,8 +718,7 @@ are within a **maximum distance** (in number of tokens) of each other:
> max_length = 100
> ```
```python
### Candidate generation
```python {title="Candidate generation"}
@spacy.registry.misc("rel_instance_generator.v1")
def create_instances(max_length: int) -> Callable[[Doc], List[Tuple[Span, Span]]]:
def get_candidates(doc: "Doc") -> List[Tuple[Span, Span]]:
@ -748,7 +736,7 @@ This function is added to the [`@misc` registry](/api/top-level#registry) so we
can refer to it from the config, and easily swap it out for any other candidate
generation function.
#### Intermezzo: define how to store the relations data {#component-rel-attribute}
#### Intermezzo: define how to store the relations data {id="component-rel-attribute"}
> #### Example output
>
@ -773,22 +761,20 @@ above 0.5 to be a `True` relation. The ~~Example~~ instances that we'll use as
training data, will include their gold-standard relation annotations in
`example.reference._.rel`.
```python
### Registering the extension attribute
```python {title="Registering the extension attribute"}
from spacy.tokens import Doc
Doc.set_extension("rel", default={})
```
#### Step 2: Implementing the pipeline component {#component-rel-pipe}
#### Step 2: Implementing the pipeline component {id="component-rel-pipe"}
To use our new relation extraction model as part of a custom
[trainable component](/usage/processing-pipelines#trainable-components), we
create a subclass of [`TrainablePipe`](/api/pipe) that holds the model.
![Illustration of Pipe methods](../images/trainable_component.svg)
![Illustration of Pipe methods](/images/trainable_component.svg)
```python
### Pipeline component skeleton
```python {title="Pipeline component skeleton"}
from spacy.pipeline import TrainablePipe
class RelationExtractor(TrainablePipe):
@ -825,8 +811,7 @@ and the name of this component. Additionally, this component, just like the
will predict scores for each label. We add convenience methods to easily
retrieve and add to them.
```python
### The constructor (continued)
```python {title="The constructor (continued)"}
def __init__(self, vocab, model, name="rel"):
"""Create a component instance."""
# ...
@ -855,8 +840,7 @@ will be used to do
layers of the neural network. This is triggered by calling
[`Model.initialize`](https://thinc.ai/api/model#initialize).
```python
### The initialize method {highlight="12,15,18,22"}
```python {title="The initialize method",highlight="12,15,18,22"}
from itertools import islice
def initialize(
@ -896,8 +880,7 @@ update the weights of the model layers. Thinc provides several
[loss functions](https://thinc.ai/docs/api-loss) that can be used for the
implementation of the `get_loss` function.
```python
### The update method {highlight="12-14"}
```python {title="The update method",highlight="12-14"}
def update(
self,
examples: Iterable[Example],
@ -923,8 +906,7 @@ delegate to the internal model's
[predict](https://thinc.ai/docs/api-model#predict) function that takes a batch
of `Doc` objects and returns a ~~Floats2d~~ array:
```python
### The predict method
```python {title="The predict method"}
def predict(self, docs: Iterable[Doc]) -> Floats2d:
predictions = self.model.predict(docs)
return self.model.ops.asarray(predictions)
@ -941,8 +923,7 @@ need to refer to the model's `get_instances` function that defined which pairs
of entities were relevant candidates, so that the predictions can be linked to
those exact entities:
```python
### The set_annotations method {highlight="5-6,10"}
```python {title="The set_annotations method",highlight="5-6,10"}
def set_annotations(self, docs: Iterable[Doc], predictions: Floats2d):
c = 0
get_instances = self.model.attrs["get_instances"]
@ -959,8 +940,7 @@ def set_annotations(self, docs: Iterable[Doc], predictions: Floats2d):
Under the hood, when the pipe is applied to a document, it delegates to the
`predict` and `set_annotations` methods:
```python
### The __call__ method
```python {title="The __call__ method"}
def __call__(self, doc: Doc):
predictions = self.predict([doc])
self.set_annotations([doc], predictions)
@ -971,8 +951,7 @@ There is one more optional method to implement: [`score`](/api/pipe#score)
calculates the performance of your component on a set of examples, and returns
the results as a dictionary:
```python
### The score method
```python {title="The score method"}
def score(self, examples: Iterable[Example]) -> Dict[str, Any]:
prf = PRFScore()
for example in examples:
@ -1011,8 +990,7 @@ assigns it a name and lets you create the component with
> rel_micro_f = 1.0
> ```
```python
### Registering the pipeline component
```python {title="Registering the pipeline component"}
from spacy.language import Language
@Language.factory("relation_extractor")
@ -1024,8 +1002,7 @@ You can extend the decorator to include information such as the type of
annotations that are required for this component to run, the type of annotations
it produces, and the scores that can be calculated:
```python
### Factory annotations {highlight="5-11"}
```python {title="Factory annotations",highlight="5-11"}
from spacy.language import Language
@Language.factory(
@ -1043,11 +1020,10 @@ def make_relation_extractor(nlp, name, model):
```
<Project id="tutorials/rel_component">
Run this example use-case by using our project template. It includes all the
code to create the ML model and the pipeline component from scratch.
It contains two config files to train the model:
one to run on CPU with a Tok2Vec layer, and one for the GPU using a transformer.
The project applies the relation extraction component to identify biomolecular
interactions, but you can easily swap in your own dataset for your experiments
in any other domain.
Run this example use-case by using our project template. It includes all the
code to create the ML model and the pipeline component from scratch. It
contains two config files to train the model: one to run on CPU with a Tok2Vec
layer, and one for the GPU using a transformer. The project applies the
relation extraction component to identify biomolecular interactions, but you
can easily swap in your own dataset for your experiments in any other domain.
</Project>

249
website/docs/usage/linguistic-features.md → website/docs/usage/linguistic-features.mdx
View File

@ -26,9 +26,7 @@ information. That's exactly what spaCy is designed to do: you put in raw text,
and get back a [`Doc`](/api/doc) object, that comes with a variety of
annotations.
## Part-of-speech tagging {#pos-tagging model="tagger, parser"}
import PosDeps101 from 'usage/101/\_pos-deps.md'
## Part-of-speech tagging {id="pos-tagging",model="tagger, parser"}
<PosDeps101 />
@ -40,7 +38,7 @@ in the [models directory](/models).
</Infobox>
## Morphology {#morphology}
## Morphology {id="morphology"}
Inflectional morphology is the process by which a root form of a word is
modified by adding prefixes or suffixes that specify its grammatical function
@ -64,8 +62,7 @@ allows you to access individual morphological features.
> and express that it's a pronoun in the third person.
> 2. Inspect `token.morph` for the other tokens.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -76,14 +73,13 @@ print(token.morph) # 'Case=Nom|Number=Sing|Person=1|PronType=Prs'
print(token.morph.get("PronType")) # ['Prs']
```
### Statistical morphology {#morphologizer new="3" model="morphologizer"}
### Statistical morphology {id="morphologizer",version="3",model="morphologizer"}
spaCy's statistical [`Morphologizer`](/api/morphologizer) component assigns the
morphological features and coarse-grained part-of-speech tags as `Token.morph`
and `Token.pos`.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("de_core_news_sm")
@ -92,7 +88,7 @@ print(doc[2].morph) # 'Case=Nom|Number=Sing|Person=2|PronType=Prs'
print(doc[2].pos_) # 'PRON'
```
### Rule-based morphology {#rule-based-morphology}
### Rule-based morphology {id="rule-based-morphology"}
For languages with relatively simple morphological systems like English, spaCy
can assign morphological features through a rule-based approach, which uses the
@ -108,8 +104,7 @@ coarse-grained part-of-speech tags and morphological features.
[mapping table](#mappings-exceptions) maps the fine-grained tags to a
coarse-grained POS tags and morphological features.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -118,7 +113,7 @@ print(doc[2].morph) # 'Case=Nom|Person=2|PronType=Prs'
print(doc[2].pos_) # 'PRON'
```
## Lemmatization {#lemmatization model="lemmatizer" new="3"}
## Lemmatization {id="lemmatization",model="lemmatizer",version="3"}
spaCy provides two pipeline components for lemmatization:
@ -128,8 +123,7 @@ spaCy provides two pipeline components for lemmatization:
2. The [`EditTreeLemmatizer`](/api/edittreelemmatizer)
<Tag variant="new">3.3</Tag> component provides a trainable lemmatizer.
```python
### {executable="true"}
```python {executable="true"}
import spacy
# English pipelines include a rule-based lemmatizer
@ -160,7 +154,7 @@ provided trained pipelines already include all the required tables, but if you
are creating new pipelines, you'll probably want to install `spacy-lookups-data`
to provide the data when the lemmatizer is initialized.
### Lookup lemmatizer {#lemmatizer-lookup}
### Lookup lemmatizer {id="lemmatizer-lookup"}
For pipelines without a tagger or morphologizer, a lookup lemmatizer can be
added to the pipeline as long as a lookup table is provided, typically through
@ -176,7 +170,7 @@ nlp = spacy.blank("sv")
nlp.add_pipe("lemmatizer", config={"mode": "lookup"})
```
### Rule-based lemmatizer {#lemmatizer-rule}
### Rule-based lemmatizer {id="lemmatizer-rule"}
When training pipelines that include a component that assigns part-of-speech
tags (a morphologizer or a tagger with a [POS mapping](#mappings-exceptions)), a
@ -214,7 +208,7 @@ nlp = spacy.blank("de")
nlp.add_pipe("trainable_lemmatizer", name="lemmatizer")
```
## Dependency Parsing {#dependency-parse model="parser"}
## Dependency Parsing {id="dependency-parse",model="parser"}
spaCy features a fast and accurate syntactic dependency parser, and has a rich
API for navigating the tree. The parser also powers the sentence boundary
@ -232,7 +226,7 @@ different languages, see the label schemes documented in the
</Infobox>
### Noun chunks {#noun-chunks}
### Noun chunks {id="noun-chunks"}
Noun chunks are "base noun phrases" – flat phrases that have a noun as their
head. You can think of noun chunks as a noun plus the words describing the noun
@ -240,8 +234,7 @@ head. You can think of noun chunks as a noun plus the words describing the noun
get the noun chunks in a document, simply iterate over
[`Doc.noun_chunks`](/api/doc#noun_chunks).
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -263,7 +256,7 @@ for chunk in doc.noun_chunks:
| insurance liability | liability | `dobj` | shift |
| manufacturers | manufacturers | `pobj` | toward |
### Navigating the parse tree {#navigating}
### Navigating the parse tree {id="navigating"}
spaCy uses the terms **head** and **child** to describe the words **connected by
a single arc** in the dependency tree. The term **dep** is used for the arc
@ -271,8 +264,7 @@ label, which describes the type of syntactic relation that connects the child to
the head. As with other attributes, the value of `.dep` is a hash value. You can
get the string value with `.dep_`.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -298,17 +290,18 @@ for token in doc:
| toward | `prep` | shift | `NOUN` | manufacturers |
| manufacturers | `pobj` | toward | `ADP` | |
import DisplaCyLong2Html from 'images/displacy-long2.html'
<Iframe title="displaCy visualization of dependencies and entities 2" html={DisplaCyLong2Html} height={450} />
<Iframe
title="displaCy visualization of dependencies and entities 2"
src="/images/displacy-long2.html"
height={450}
/>
Because the syntactic relations form a tree, every word has **exactly one
head**. You can therefore iterate over the arcs in the tree by iterating over
the words in the sentence. This is usually the best way to match an arc of
interest – from below:
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.symbols import nsubj, VERB
@ -340,7 +333,7 @@ for possible_verb in doc:
To iterate through the children, use the `token.children` attribute, which
provides a sequence of [`Token`](/api/token) objects.
#### Iterating around the local tree {#navigating-around}
#### Iterating around the local tree {id="navigating-around"}
A few more convenience attributes are provided for iterating around the local
tree from the token. [`Token.lefts`](/api/token#lefts) and
@ -351,8 +344,7 @@ order. There are also two integer-typed attributes,
[`Token.n_rights`](/api/token#n_rights) that give the number of left and right
children.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -363,8 +355,7 @@ print(doc[2].n_lefts) # 2
print(doc[2].n_rights) # 1
```
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("de_core_news_sm")
@ -387,8 +378,7 @@ sequence of tokens. You can walk up the tree with the
> true for the German pipelines, which have many
> [non-projective dependencies](https://explosion.ai/blog/german-model#word-order).
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -417,8 +407,7 @@ easiest way to create a `Span` object for a syntactic phrase. Note that
`.right_edge` gives a token **within** the subtree – so if you use it as the
end-point of a range, don't forget to `+1`!
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -445,8 +434,7 @@ currency values, i.e. entities labeled as `MONEY`, and then uses the dependency
parse to find the noun phrase they are referring to – for example `"Net income"`
&rarr; `"$9.4 million"`.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -480,7 +468,7 @@ see the usage guide on
</Infobox>
### Visualizing dependencies {#displacy}
### Visualizing dependencies {id="displacy"}
The best way to understand spaCy's dependency parser is interactively. To make
this easier, spaCy comes with a visualization module. You can pass a `Doc` or a
@ -491,8 +479,7 @@ If you want to know how to write rules that hook into some type of syntactic
construction, just plug the sentence into the visualizer and see how spaCy
annotates it.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy import displacy
@ -510,7 +497,7 @@ displaCy in our [online demo](https://explosion.ai/demos/displacy)..
</Infobox>
### Disabling the parser {#disabling}
### Disabling the parser {id="disabling"}
In the [trained pipelines](/models) provided by spaCy, the parser is loaded and
enabled by default as part of the
@ -525,7 +512,7 @@ the `nlp` object. For more details, see the usage guide on
nlp = spacy.load("en_core_web_sm", disable=["parser"])
```
## Named Entity Recognition {#named-entities}
## Named Entity Recognition {id="named-entities"}
spaCy features an extremely fast statistical entity recognition system, that
assigns labels to contiguous spans of tokens. The default
@ -534,13 +521,11 @@ entities, including companies, locations, organizations and products. You can
add arbitrary classes to the entity recognition system, and update the model
with new examples.
### Named Entity Recognition 101 {#named-entities-101}
import NER101 from 'usage/101/\_named-entities.md'
### Named Entity Recognition 101 {id="named-entities-101"}
<NER101 />
### Accessing entity annotations and labels {#accessing-ner}
### Accessing entity annotations and labels {id="accessing-ner"}
The standard way to access entity annotations is the [`doc.ents`](/api/doc#ents)
property, which produces a sequence of [`Span`](/api/span) objects. The entity
@ -569,8 +554,7 @@ on a token, it will return an empty string.
> - `U` – Token is a single-token **unit** entity.
> - `O` – Token is **outside** an entity.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -597,7 +581,7 @@ print(ent_francisco) # ['Francisco', 'I', 'GPE']
| delivery | `2` | `O` | `""` | outside an entity |
| robots | `2` | `O` | `""` | outside an entity |
### Setting entity annotations {#setting-entities}
### Setting entity annotations {id="setting-entities"}
To ensure that the sequence of token annotations remains consistent, you have to
set entity annotations **at the document level**. However, you can't write
@ -605,8 +589,7 @@ directly to the `token.ent_iob` or `token.ent_type` attributes, so the easiest
way to set entities is to use the [`doc.set_ents`](/api/doc#set_ents) function
and create the new entity as a [`Span`](/api/span).
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.tokens import Span
@ -639,15 +622,14 @@ indices, not the character offsets. To create a span from character offsets, use
fb_ent = doc.char_span(0, 2, label="ORG")
```
#### Setting entity annotations from array {#setting-from-array}
#### Setting entity annotations from array {id="setting-from-array"}
You can also assign entity annotations using the
[`doc.from_array`](/api/doc#from_array) method. To do this, you should include
both the `ENT_TYPE` and the `ENT_IOB` attributes in the array you're importing
from.
```python
### {executable="true"}
```python {executable="true"}
import numpy
import spacy
from spacy.attrs import ENT_IOB, ENT_TYPE
@ -664,7 +646,7 @@ doc.from_array(header, attr_array)
print("After", doc.ents) # [London]
```
#### Setting entity annotations in Cython {#setting-cython}
#### Setting entity annotations in Cython {id="setting-cython"}
Finally, you can always write to the underlying struct if you compile a
[Cython](http://cython.org/) function. This is easy to do, and allows you to
@ -686,7 +668,7 @@ cpdef set_entity(Doc doc, int start, int end, attr_t ent_type):
Obviously, if you write directly to the array of `TokenC*` structs, you'll have
responsibility for ensuring that the data is left in a consistent state.
### Built-in entity types {#entity-types}
### Built-in entity types {id="entity-types"}
> #### Tip: Understanding entity types
>
@ -702,7 +684,7 @@ For details on the entity types available in spaCy's trained pipelines, see the
</Infobox>
### Visualizing named entities {#displacy}
### Visualizing named entities {id="displacy"}
The
[displaCy <sup>ENT</sup> visualizer](https://explosion.ai/demos/displacy-ent)
@ -716,8 +698,7 @@ list of `Doc` objects to displaCy and run
For more details and examples, see the
[usage guide on visualizing spaCy](/usage/visualizers).
```python
### Named Entity example
```python {title="Named Entity example"}
import spacy
from spacy import displacy
@ -728,11 +709,13 @@ doc = nlp(text)
displacy.serve(doc, style="ent")
```
import DisplacyEntHtml from 'images/displacy-ent2.html'
<Iframe
title="displaCy visualizer for entities"
src="/images/displacy-ent2.html"
height={180}
/>
<Iframe title="displaCy visualizer for entities" html={DisplacyEntHtml} height={180} />
## Entity Linking {#entity-linking}
## Entity Linking {id="entity-linking"}
To ground the named entities into the "real world", spaCy provides functionality
to perform entity linking, which resolves a textual entity to a unique
@ -740,7 +723,7 @@ identifier from a knowledge base (KB). You can create your own
[`KnowledgeBase`](/api/kb) and [train](/usage/training) a new
[`EntityLinker`](/api/entitylinker) using that custom knowledge base.
### Accessing entity identifiers {#entity-linking-accessing model="entity linking"}
### Accessing entity identifiers {id="entity-linking-accessing",model="entity linking"}
The annotated KB identifier is accessible as either a hash value or as a string,
using the attributes `ent.kb_id` and `ent.kb_id_` of a [`Span`](/api/span)
@ -766,7 +749,7 @@ print(ent_ada_1) # ['Lovelace', 'PERSON', 'Q7259']
print(ent_london_5) # ['London', 'GPE', 'Q84']
```
## Tokenization {#tokenization}
## Tokenization {id="tokenization"}
Tokenization is the task of splitting a text into meaningful segments, called
_tokens_. The input to the tokenizer is a unicode text, and the output is a
@ -785,8 +768,6 @@ during tokenization. This is kind of a core principle of spaCy's `Doc` object:
</Infobox>
import Tokenization101 from 'usage/101/\_tokenization.md'
<Tokenization101 />
<Accordion title="Algorithm details: How spaCy's tokenizer works" id="how-tokenizer-works" spaced>
@ -914,15 +895,14 @@ might make sense to create an entirely custom subclass.
---
### Adding special case tokenization rules {#special-cases}
### Adding special case tokenization rules {id="special-cases"}
Most domains have at least some idiosyncrasies that require custom tokenization
rules. This could be very certain expressions, or abbreviations only used in
this specific field. Here's how to add a special case rule to an existing
[`Tokenizer`](/api/tokenizer) instance:
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.symbols import ORTH
@ -951,7 +931,7 @@ nlp.tokenizer.add_special_case("...gimme...?", [{"ORTH": "...gimme...?"}])
assert len(nlp("...gimme...?")) == 1
```
#### Debugging the tokenizer {#tokenizer-debug new="2.2.3"}
#### Debugging the tokenizer {id="tokenizer-debug",version="2.2.3"}
A working implementation of the pseudo-code above is available for debugging as
[`nlp.tokenizer.explain(text)`](/api/tokenizer#explain). It returns a list of
@ -969,8 +949,7 @@ tokens produced are identical to `nlp.tokenizer()` except for whitespace tokens:
> " SUFFIX
> ```
```python
### {executable="true"}
```python {executable="true"}
from spacy.lang.en import English
nlp = English()
@ -982,7 +961,7 @@ for t in tok_exp:
print(t[1], "\\t", t[0])
```
### Customizing spaCy's Tokenizer class {#native-tokenizers}
### Customizing spaCy's Tokenizer class {id="native-tokenizers"}
Let's imagine you wanted to create a tokenizer for a new language or specific
domain. There are six things you may need to define:
@ -1004,8 +983,7 @@ You shouldn't usually need to create a `Tokenizer` subclass. Standard usage is
to use `re.compile()` to build a regular expression object, and pass its
`.search()` and `.finditer()` methods:
```python
### {executable="true"}
```python {executable="true"}
import re
import spacy
from spacy.tokenizer import Tokenizer
@ -1045,7 +1023,7 @@ only be applied at the **end of a token**, so your expression should end with a
</Infobox>
#### Modifying existing rule sets {#native-tokenizer-additions}
#### Modifying existing rule sets {id="native-tokenizer-additions"}
In many situations, you don't necessarily need entirely custom rules. Sometimes
you just want to add another character to the prefixes, suffixes or infixes. The
@ -1098,8 +1076,7 @@ letters as an infix. If you do not want the tokenizer to split on hyphens
between letters, you can modify the existing infix definition from
[`lang/punctuation.py`](%%GITHUB_SPACY/spacy/lang/punctuation.py):
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.lang.char_classes import ALPHA, ALPHA_LOWER, ALPHA_UPPER
from spacy.lang.char_classes import CONCAT_QUOTES, LIST_ELLIPSES, LIST_ICONS
@ -1138,7 +1115,7 @@ language-specific definitions such as
[`lang/de/punctuation.py`](%%GITHUB_SPACY/spacy/lang/de/punctuation.py) for
German.
### Hooking a custom tokenizer into the pipeline {#custom-tokenizer}
### Hooking a custom tokenizer into the pipeline {id="custom-tokenizer"}
The tokenizer is the first component of the processing pipeline and the only one
that can't be replaced by writing to `nlp.pipeline`. This is because it has a
@ -1146,7 +1123,7 @@ different signature from all the other components: it takes a text and returns a
[`Doc`](/api/doc), whereas all other components expect to already receive a
tokenized `Doc`.
![The processing pipeline](../images/pipeline.svg)
![The processing pipeline](/images/pipeline.svg)
To overwrite the existing tokenizer, you need to replace `nlp.tokenizer` with a
custom function that takes a text and returns a [`Doc`](/api/doc).
@ -1175,7 +1152,7 @@ nlp.tokenizer = my_tokenizer
| `text` | `str` | The raw text to tokenize. |
| **RETURNS** | [`Doc`](/api/doc) | The tokenized document. |
#### Example 1: Basic whitespace tokenizer {#custom-tokenizer-example}
#### Example 1: Basic whitespace tokenizer {id="custom-tokenizer-example"}
Here's an example of the most basic whitespace tokenizer. It takes the shared
vocab, so it can construct `Doc` objects. When it's called on a text, it returns
@ -1183,8 +1160,7 @@ a `Doc` object consisting of the text split on single space characters. We can
then overwrite the `nlp.tokenizer` attribute with an instance of our custom
tokenizer.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.tokens import Doc
@ -1215,7 +1191,7 @@ doc = nlp("What's happened to me? he thought. It wasn't a dream.")
print([token.text for token in doc])
```
#### Example 2: Third-party tokenizers (BERT word pieces) {#custom-tokenizer-example2}
#### Example 2: Third-party tokenizers (BERT word pieces) {id="custom-tokenizer-example2"}
You can use the same approach to plug in any other third-party tokenizers. Your
custom callable just needs to return a `Doc` object with the tokens produced by
@ -1234,8 +1210,7 @@ produced by the tokenizer.
> **training transformer models** in spaCy, as well as helpful utilities for
> aligning word pieces to linguistic tokenization.
```python
### Custom BERT word piece tokenizer
```python {title="Custom BERT word piece tokenizer"}
from tokenizers import BertWordPieceTokenizer
from spacy.tokens import Doc
import spacy
@ -1279,7 +1254,7 @@ tokenizer** it will be using at runtime. See the docs on
</Infobox>
#### Training with custom tokenization {#custom-tokenizer-training new="3"}
#### Training with custom tokenization {id="custom-tokenizer-training",version="3"}
spaCy's [training config](/usage/training#config) describes the settings,
hyperparameters, pipeline and tokenizer used for constructing and training the
@ -1297,8 +1272,7 @@ setting `--code functions.py` when you run [`spacy train`](/api/cli#train).
> @tokenizers = "whitespace_tokenizer"
> ```
```python
### functions.py {highlight="1"}
```python {title="functions.py",highlight="1"}
@spacy.registry.tokenizers("whitespace_tokenizer")
def create_whitespace_tokenizer():
def create_tokenizer(nlp):
@ -1323,8 +1297,7 @@ correct type.
> lowercase = true
> ```
```python
### functions.py {highlight="1"}
```python {title="functions.py",highlight="1"}
@spacy.registry.tokenizers("bert_word_piece_tokenizer")
def create_whitespace_tokenizer(vocab_file: str, lowercase: bool):
def create_tokenizer(nlp):
@ -1348,7 +1321,7 @@ takes a text and returns a `Doc`.
</Infobox>
#### Using pre-tokenized text {#own-annotations}
#### Using pre-tokenized text {id="own-annotations"}
spaCy generally assumes by default that your data is **raw text**. However,
sometimes your data is partially annotated, e.g. with pre-existing tokenization,
@ -1367,8 +1340,7 @@ boolean values, indicating whether each word is followed by a space.
> `Doc` with `words` and `spaces` so that the `doc.text` matches the original
> input text.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.tokens import Doc
@ -1388,7 +1360,7 @@ will assume that all words are followed by a space. Once you have a
part-of-speech tags, syntactic dependencies, named entities and other
attributes.
#### Aligning tokenization {#aligning-tokenization}
#### Aligning tokenization {id="aligning-tokenization"}
spaCy's tokenization is non-destructive and uses language-specific rules
optimized for compatibility with treebank annotations. Other tools and resources
@ -1414,8 +1386,7 @@ token.
> 3. Make `other_tokens` and `spacy_tokens` identical. You'll see that all
> tokens now correspond 1-to-1.
```python
### {executable="true"}
```python {executable="true"}
from spacy.training import Alignment
other_tokens = ["i", "listened", "to", "obama", "'", "s", "podcasts", "."]
@ -1448,7 +1419,7 @@ tokenizations add up to the same string. For example, you'll be able to align
</Infobox>
## Merging and splitting {#retokenization new="2.1"}
## Merging and splitting {id="retokenization",version="2.1"}
The [`Doc.retokenize`](/api/doc#retokenize) context manager lets you merge and
split tokens. Modifications to the tokenization are stored and performed all at
@ -1467,8 +1438,7 @@ root.
> recognized as a named entity, this change will also be reflected in the
> `doc.ents`.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -1537,8 +1507,7 @@ second split subtoken) and "York" should be attached to "in".
> 3. Split the token into three tokens instead of two – for example,
> `["New", "Yo", "rk"]`.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy import displacy
@ -1567,8 +1536,7 @@ the token indices after splitting.
If you don't care about the heads (for example, if you're only running the
tokenizer and not the parser), you can attach each subtoken to itself:
```python
### {highlight="3"}
```python {highlight="3"}
doc = nlp("I live in NewYorkCity")
with doc.retokenize() as retokenizer:
heads = [(doc[3], 0), (doc[3], 1), (doc[3], 2)]
@ -1592,7 +1560,7 @@ with doc.retokenize() as retokenizer:
</Infobox>
### Overwriting custom extension attributes {#retokenization-extensions}
### Overwriting custom extension attributes {id="retokenization-extensions"}
If you've registered custom
[extension attributes](/usage/processing-pipelines#custom-components-attributes),
@ -1624,8 +1592,7 @@ values can't be overwritten. For more details, see the
> you need to provide a list of extension attribute values as the `"_"`
> property, one for each split subtoken.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.tokens import Token
@ -1641,7 +1608,7 @@ with doc.retokenize() as retokenizer:
print("After:", [(token.text, token._.is_musician) for token in doc])
```
## Sentence Segmentation {#sbd}
## Sentence Segmentation {id="sbd"}
A [`Doc`](/api/doc) object's sentences are available via the `Doc.sents`
property. To view a `Doc`'s sentences, you can iterate over the `Doc.sents`, a
@ -1650,8 +1617,7 @@ has sentence boundaries by calling
[`Doc.has_annotation`](/api/doc#has_annotation) with the attribute name
`"SENT_START"`.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -1676,7 +1642,7 @@ spaCy provides four alternatives for sentence segmentation:
processing pipeline can set sentence boundaries by writing to
`Token.is_sent_start`.
### Default: Using the dependency parse {#sbd-parser model="parser"}
### Default: Using the dependency parse {id="sbd-parser",model="parser"}
Unlike other libraries, spaCy uses the dependency parse to determine sentence
boundaries. This is usually the most accurate approach, but it requires a
@ -1686,8 +1652,7 @@ with spaCy's provided trained pipelines. For social media or conversational text
that doesn't follow the same rules, your application may benefit from a custom
trained or rule-based component.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -1701,7 +1666,7 @@ your `Doc` using custom components _before_ it's parsed. Depending on your text,
this may also improve parse accuracy, since the parser is constrained to predict
parses consistent with the sentence boundaries.
### Statistical sentence segmenter {#sbd-senter model="senter" new="3"}
### Statistical sentence segmenter {id="sbd-senter",model="senter",version="3"}
The [`SentenceRecognizer`](/api/sentencerecognizer) is a simple statistical
component that only provides sentence boundaries. Along with being faster and
@ -1721,8 +1686,7 @@ without the parser and then enable the sentence recognizer explicitly with
> which is better at predicting sentence boundaries when punctuation is not
> present.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm", exclude=["parser"])
@ -1732,15 +1696,14 @@ for sent in doc.sents:
print(sent.text)
```
### Rule-based pipeline component {#sbd-component}
### Rule-based pipeline component {id="sbd-component"}
The [`Sentencizer`](/api/sentencizer) component is a
[pipeline component](/usage/processing-pipelines) that splits sentences on
punctuation like `.`, `!` or `?`. You can plug it into your pipeline if you only
need sentence boundaries without dependency parses.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.lang.en import English
@ -1779,8 +1742,7 @@ for unset sentence boundaries. This approach can be useful if you want to
implement **additional** rules specific to your data, while still being able to
take advantage of dependency-based sentence segmentation.
```python
### {executable="true"}
```python {executable="true"}
from spacy.language import Language
import spacy
@ -1802,7 +1764,7 @@ doc = nlp(text)
print("After:", [sent.text for sent in doc.sents])
```
## Mappings & Exceptions {#mappings-exceptions new="3"}
## Mappings & Exceptions {id="mappings-exceptions",version="3"}
The [`AttributeRuler`](/api/attributeruler) manages **rule-based mappings and
exceptions** for all token-level attributes. As the number of
@ -1830,8 +1792,7 @@ The following example shows how the tag and POS `NNP`/`PROPN` can be specified
for the phrase `"The Who"`, overriding the tags provided by the statistical
tagger and the POS tag map.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -1866,13 +1827,11 @@ initialized before training. See the
</Infobox>
## Word vectors and semantic similarity {#vectors-similarity}
import Vectors101 from 'usage/101/\_vectors-similarity.md'
## Word vectors and semantic similarity {id="vectors-similarity"}
<Vectors101 />
### Adding word vectors {#adding-vectors}
### Adding word vectors {id="adding-vectors"}
Custom word vectors can be trained using a number of open-source libraries, such
as [Gensim](https://radimrehurek.com/gensim), [FastText](https://fasttext.cc),
@ -1898,7 +1857,7 @@ access to some nice Latin vectors. You can then pass the directory path to
> doc1.similarity(doc2)
> ```
```cli
```bash
$ wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.la.300.vec.gz
$ python -m spacy init vectors en cc.la.300.vec.gz /tmp/la_vectors_wiki_lg
```
@ -1936,8 +1895,7 @@ the removed words, mapped to `(string, score)` tuples, where `string` is the
entry the removed word was mapped to and `score` the similarity score between
the two words.
```python
### Removed words
```python {title="Removed words"}
{
"Shore": ("coast", 0.732257),
"Precautionary": ("caution", 0.490973),
@ -1958,7 +1916,7 @@ the vector of "leaving", which is identical. If you're using the
option to easily reduce the size of the vectors as you add them to a spaCy
pipeline:
```cli
```bash
$ python -m spacy init vectors en la.300d.vec.tgz /tmp/la_vectors_web_md --prune 10000
```
@ -1968,7 +1926,7 @@ among those retained.
</Accordion>
### Adding vectors individually {#adding-individual-vectors}
### Adding vectors individually {id="adding-individual-vectors"}
The `vector` attribute is a **read-only** numpy or cupy array (depending on
whether you've configured spaCy to use GPU memory), with dtype `float32`. The
@ -1982,8 +1940,7 @@ be slower than approaches that work with the whole vectors table at once, but
it's a great approach for once-off conversions before you save out your `nlp`
object to disk.
```python
### Adding vectors
```python {title="Adding vectors"}
from spacy.vocab import Vocab
vector_data = {
@ -1996,13 +1953,11 @@ for word, vector in vector_data.items():
vocab.set_vector(word, vector)
```
## Language Data {#language-data}
import LanguageData101 from 'usage/101/\_language-data.md'
## Language Data {id="language-data"}
<LanguageData101 />
### Creating a custom language subclass {#language-subclass}
### Creating a custom language subclass {id="language-subclass"}
If you want to customize multiple components of the language data or add support
for a custom language or domain-specific "dialect", you can also implement your
@ -2011,8 +1966,7 @@ own language subclass. The subclass should define two attributes: the `lang`
overview of the available attributes that can be overwritten, see the
[`Language.Defaults`](/api/language#defaults) documentation.
```python
### {executable="true"}
```python {executable="true"}
from spacy.lang.en import English
class CustomEnglishDefaults(English.Defaults):
@ -2050,12 +2004,11 @@ language name, and even train pipelines with it and refer to it in your
> needs to be available during training. You can load a Python file containing
> the code using the `--code` argument:
>
> ```cli
> ```bash
> python -m spacy train config.cfg --code code.py
> ```
```python
### Registering a custom language {highlight="7,12-13"}
```python {title="Registering a custom language",highlight="7,12-13"}
import spacy
from spacy.lang.en import English

72
website/docs/usage/models.md → website/docs/usage/models.mdx
View File

@ -23,9 +23,11 @@ located anywhere on your file system.
## Quickstart {hidden="true"}
import QuickstartModels from 'widgets/quickstart-models.js'
<QuickstartModels title="Quickstart" id="quickstart" description="Install a default trained pipeline package, get the code to load it from within spaCy and an example to test it. For more options, see the section on available packages below." />
<QuickstartModels
title="Quickstart"
id="quickstart"
description="Install a default trained pipeline package, get the code to load it from within spaCy and an example to test it. For more options, see the section on available packages below."
/>
### Usage note
@ -56,7 +58,7 @@ yields the same result as generating it using `spacy.blank()`. In both cases the
default configuration for the chosen language is loaded, and no pretrained
components will be available.
## Language support {#languages}
## Language support {id="languages"}
spaCy currently provides support for the following languages. You can help by
improving the existing [language data](/usage/linguistic-features#language-data)
@ -66,11 +68,9 @@ contribute to development. Also see the
[training documentation](/usage/training) for how to train your own pipelines on
your data.
import Languages from 'widgets/languages.js'
<Languages />
### Multi-language support {#multi-language new="2"}
### Multi-language support {id="multi-language",version="2"}
> ```python
> # Standard import
@ -89,10 +89,10 @@ generic subclass containing only the base language data, can be found in
To train a pipeline using the neutral multi-language class, you can set
`lang = "xx"` in your [training config](/usage/training#config). You can also
import the `MultiLanguage` class directly, or call
\import the `MultiLanguage` class directly, or call
[`spacy.blank("xx")`](/api/top-level#spacy.blank) for lazy-loading.
### Chinese language support {#chinese new="2.3"}
### Chinese language support {id="chinese",version="2.3"}
The Chinese language class supports three word segmentation options, `char`,
`jieba` and `pkuseg`.
@ -113,8 +113,7 @@ The Chinese language class supports three word segmentation options, `char`,
> nlp.tokenizer.initialize(pkuseg_model="mixed")
> ```
```ini
### config.cfg
```ini {title="config.cfg"}
[nlp.tokenizer]
@tokenizers = "spacy.zh.ChineseTokenizer"
segmenter = "char"
@ -155,8 +154,7 @@ local path at runtime. See the usage guide on the
[config lifecycle](/usage/training#config-lifecycle) for more background on
this.
```ini
### config.cfg
```ini {title="config.cfg"}
[initialize]
[initialize.tokenizer]
@ -167,8 +165,7 @@ pkuseg_user_dict = "default"
You can also initialize the tokenizer for a blank language class by calling its
`initialize` method:
```python
### Examples
```python {title="Examples"}
# Initialize the pkuseg tokenizer
cfg = {"segmenter": "pkuseg"}
nlp = Chinese.from_config({"nlp": {"tokenizer": cfg}})
@ -227,7 +224,7 @@ nlp.tokenizer.initialize(pkuseg_model="/path/to/pkuseg_model")
</Accordion>
### Japanese language support {#japanese new=2.3}
### Japanese language support {id="japanese",version="2.3"}
> #### Manual setup
>
@ -247,8 +244,7 @@ segmentation and part-of-speech tagging. The default Japanese language class and
the provided Japanese pipelines use SudachiPy split mode `A`. The tokenizer
config can be used to configure the split mode to `A`, `B` or `C`.
```ini
### config.cfg
```ini {title="config.cfg"}
[nlp.tokenizer]
@tokenizers = "spacy.ja.JapaneseTokenizer"
split_mode = "A"
@ -266,7 +262,7 @@ used for training the current [Japanese pipelines](/models/ja).
</Infobox>
### Korean language support {#korean}
### Korean language support {id="korean"}
> #### mecab-ko tokenizer
>
@ -291,8 +287,7 @@ than MeCab. To configure a Korean pipeline with the rule-based tokenizer:
> nlp = spacy.blank("ko", config=config)
> ```
```ini
### config.cfg
```ini {title="config.cfg"}
[nlp]
lang = "ko"
tokenizer = {"@tokenizers" = "spacy.Tokenizer.v1"}
@ -305,7 +300,7 @@ additional dependencies are required.
</Infobox>
## Installing and using trained pipelines {#download}
## Installing and using trained pipelines {id="download"}
The easiest way to download a trained pipeline is via spaCy's
[`download`](/api/cli#download) command. It takes care of finding the
@ -327,7 +322,7 @@ best-matching package compatible with your spaCy installation.
> + nlp = spacy.load("en_core_web_sm")
> ```
```cli
```bash
# Download best-matching version of a package for your spaCy installation
$ python -m spacy download en_core_web_sm
@ -338,7 +333,7 @@ $ python -m spacy download en_core_web_sm-3.0.0 --direct
The download command will [install the package](/usage/models#download-pip) via
pip and place the package in your `site-packages` directory.
```cli
```bash
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
$ python -m spacy download en_core_web_sm
```
@ -356,18 +351,18 @@ Make sure to **restart your kernel** or runtime after installation (just like
you would when installing other Python packages) to make sure that the installed
pipeline package can be found.
```cli
```bash
!python -m spacy download en_core_web_sm
```
### Installation via pip {#download-pip}
### Installation via pip {id="download-pip"}
To download a trained pipeline directly using
[pip](https://pypi.python.org/pypi/pip), point `pip install` to the URL or local
path of the wheel file or archive. Installing the wheel is usually more
efficient.
> #### Pipeline Package URLs {#pipeline-urls}
> #### Pipeline Package URLs {id="pipeline-urls"}
>
> Pretrained pipeline distributions are hosted on
> [Github Releases](https://github.com/explosion/spacy-models/releases), and you
@ -407,7 +402,7 @@ You can also add the direct download link to your application's
`requirements.txt`. For more details, see the section on
[working with pipeline packages in production](#production).
### Manual download and installation {#download-manual}
### Manual download and installation {id="download-manual"}
In some cases, you might prefer downloading the data manually, for example to
place it into a custom directory. You can download the package via your browser
@ -416,8 +411,7 @@ or configure your own download script using the URL of the archive file. The
archive consists of a package directory that contains another directory with the
pipeline data.
```yaml
### Directory structure {highlight="6"}
```yaml {title="Directory structure",highlight="6"}
└── en_core_web_md-3.0.0.tar.gz # downloaded archive
├── setup.py # setup file for pip installation
├── meta.json # copy of pipeline meta
@ -432,7 +426,7 @@ pipeline data.
You can place the **pipeline package directory** anywhere on your local file
system.
### Installation from Python {#download-python}
### Installation from Python {id="download-python"}
Since the [`spacy download`](/api/cli#download) command installs the pipeline as
a **Python package**, we always recommend running it from the command line, just
@ -453,7 +447,7 @@ import spacy
spacy.cli.download("en_core_web_sm")
```
### Using trained pipelines with spaCy {#usage}
### Using trained pipelines with spaCy {id="usage"}
To load a pipeline package, use [`spacy.load`](/api/top-level#spacy.load) with
the package name or a path to the data directory:
@ -487,14 +481,13 @@ exposes the pipeline's meta data as the attribute `meta`. For example,
</Infobox>
### Importing pipeline packages as modules {#usage-import}
### Importing pipeline packages as modules {id="usage-import"}
If you've installed a trained pipeline via [`spacy download`](/api/cli#download)
or directly via pip, you can also `import` it and then call its `load()` method
with no arguments:
```python
### {executable="true"}
```python {executable="true"}
import en_core_web_sm
nlp = en_core_web_sm.load()
@ -510,7 +503,7 @@ as your code will raise an `ImportError` immediately, instead of failing
somewhere down the line when calling `spacy.load()`. For more details, see the
section on [working with pipeline packages in production](#production).
## Using trained pipelines in production {#production}
## Using trained pipelines in production {id="production"}
If your application depends on one or more trained pipeline packages, you'll
usually want to integrate them into your continuous integration workflow and
@ -519,7 +512,7 @@ and loading pipeline packages, the underlying functionality is entirely based on
native Python packaging. This allows your application to handle a spaCy pipeline
like any other package dependency.
### Downloading and requiring package dependencies {#models-download}
### Downloading and requiring package dependencies {id="models-download"}
spaCy's built-in [`download`](/api/cli#download) command is mostly intended as a
convenient, interactive wrapper. It performs compatibility checks and prints
@ -535,8 +528,7 @@ installation, you can upload the pipeline packages there. pip's
supports both package names to download via a PyPi server, as well as
[direct URLs](#pipeline-urls).
```text
### requirements.txt
```text {title="requirements.txt"}
spacy>=3.0.0,<4.0.0
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.0/en_core_web_sm-3.4.0-py3-none-any.whl
```
@ -547,7 +539,7 @@ each pipeline. If you've [trained](/usage/training) your own pipeline, you can
use the [`spacy package`](/api/cli#package) command to generate the required
meta data and turn it into a loadable package.
### Loading and testing pipeline packages {#models-loading}
### Loading and testing pipeline packages {id="models-loading"}
Pipeline packages are regular Python packages, so you can also import them as a
package using Python's native `import` syntax, and then call the `load` method

133
website/docs/usage/processing-pipelines.md → website/docs/usage/processing-pipelines.mdx
View File

@ -12,11 +12,9 @@ menu:
- ['Plugins & Wrappers', 'plugins']
---
import Pipelines101 from 'usage/101/\_pipelines.md'
<Pipelines101 />
## Processing text {#processing}
## Processing text {id="processing"}
When you call `nlp` on a text, spaCy will **tokenize** it and then **call each
component** on the `Doc`, in order. It then returns the processed `Doc` that you
@ -62,8 +60,7 @@ so we can iterate over them and access the named entity predictions:
> 1. Also disable the `"ner"` component. You'll see that the `doc.ents` are now
> empty, because the entity recognizer didn't run.
```python
### {executable="true"}
```python {executable="true"}
import spacy
texts = [
@ -97,8 +94,7 @@ the input should be a sequence of `(text, context)` tuples and the output will
be a sequence of `(doc, context)` tuples. For example, you can pass metadata in
the context and save it in a [custom attribute](#custom-components-attributes):
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.tokens import Doc
@ -122,7 +118,7 @@ for doc in docs:
print(f"{doc._.text_id}: {doc.text}")
```
### Multiprocessing {#multiprocessing}
### Multiprocessing {id="multiprocessing"}
spaCy includes built-in support for multiprocessing with
[`nlp.pipe`](/api/language#pipe) using the `n_process` option:
@ -170,7 +166,7 @@ the number of threads before loading any models using
</Infobox>
## Pipelines and built-in components {#pipelines}
## Pipelines and built-in components {id="pipelines"}
spaCy makes it very easy to create your own pipelines consisting of reusable
components – this includes spaCy's default tagger, parser and entity recognizer,
@ -248,8 +244,7 @@ tagging pipeline. This is also why the pipeline state is always held by the
together and returns an instance of `Language` with a pipeline set and access to
the binary data:
```python
### spacy.load under the hood
```python {title="spacy.load under the hood"}
lang = "en"
pipeline = ["tok2vec", "tagger", "parser", "ner", "attribute_ruler", "lemmatizer"]
data_path = "path/to/en_core_web_sm/en_core_web_sm-3.0.0"
@ -268,8 +263,7 @@ subsequently to the `Token` and `Span` which are only views of the `Doc`, and
don't own any data themselves. All components return the modified document,
which is then processed by the next component in the pipeline.
```python
### The pipeline under the hood
```python {title="The pipeline under the hood"}
doc = nlp.make_doc("This is a sentence") # Create a Doc from raw text
for name, proc in nlp.pipeline: # Iterate over components in order
doc = proc(doc) # Apply each component
@ -286,7 +280,7 @@ print(nlp.pipe_names)
# ['tok2vec', 'tagger', 'parser', 'ner', 'attribute_ruler', 'lemmatizer']
```
### Built-in pipeline components {#built-in}
### Built-in pipeline components {id="built-in"}
spaCy ships with several built-in pipeline components that are registered with
string names. This means that you can initialize them by calling
@ -321,7 +315,7 @@ available pipeline components and component functions.
| `tok2vec` | [`Tok2Vec`](/api/tok2vec) | Assign token-to-vector embeddings. |
| `transformer` | [`Transformer`](/api/transformer) | Assign the tokens and outputs of a transformer model. |
### Disabling, excluding and modifying components {#disabling}
### Disabling, excluding and modifying components {id="disabling"}
If you don't need a particular component of the pipeline – for example, the
tagger or the parser, you can **disable or exclude** it. This can sometimes make
@ -364,7 +358,8 @@ nlp.enable_pipe("tagger")
In addition to `disable`, `spacy.load()` also accepts `enable`. If `enable` is
set, all components except for those in `enable` are disabled. If `enable` and
`disable` conflict (i.e. the same component is included in both), an error is raised.
`disable` conflict (i.e. the same component is included in both), an error is
raised.
```python
# Load the complete pipeline, but disable all components except for tok2vec and tagger
@ -391,8 +386,7 @@ call its `restore()` method to restore the disabled components when needed. This
can be useful if you want to prevent unnecessary code indentation of large
blocks.
```python
### Disable for block
```python {title="Disable for block"}
# 1. Use as a context manager
with nlp.select_pipes(disable=["tagger", "parser", "lemmatizer"]):
doc = nlp("I won't be tagged and parsed")
@ -460,7 +454,7 @@ run as part of the pipeline.
| `nlp.component_names` | All component names, including disabled components. |
| `nlp.disabled` | Names of components that are currently disabled. |
### Sourcing components from existing pipelines {#sourced-components new="3"}
### Sourcing components from existing pipelines {id="sourced-components",version="3"}
Pipeline components that are independent can also be reused across pipelines.
Instead of adding a new blank component, you can also copy an existing component
@ -503,8 +497,7 @@ vectors available – otherwise, it won't be able to make the same predictions.
> frozen_components = ["ner"]
> ```
```python
### {executable="true"}
```python {executable="true"}
import spacy
# The source pipeline with different components
@ -517,7 +510,7 @@ nlp.add_pipe("ner", source=source_nlp)
print(nlp.pipe_names)
```
### Analyzing pipeline components {#analysis new="3"}
### Analyzing pipeline components {id="analysis",version="3"}
The [`nlp.analyze_pipes`](/api/language#analyze_pipes) method analyzes the
components in the current pipeline and outputs information about them like the
@ -534,8 +527,7 @@ table instead of only returning the structured data.
> `"entity_linker"`. The analysis should now show no problems, because
> requirements are met.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.blank("en")
@ -547,8 +539,7 @@ analysis = nlp.analyze_pipes(pretty=True)
<Accordion title="Example output">
```json
### Structured
```json {title="Structured"}
{
"summary": {
"tagger": {
@ -566,7 +557,12 @@ analysis = nlp.analyze_pipes(pretty=True)
},
"problems": {
"tagger": [],
"entity_linker": ["doc.ents", "doc.sents", "token.ent_iob", "token.ent_type"]
"entity_linker": [
"doc.ents",
"doc.sents",
"token.ent_iob",
"token.ent_type"
]
},
"attrs": {
"token.ent_iob": { "assigns": [], "requires": ["entity_linker"] },
@ -609,7 +605,7 @@ doesn't, the pipeline analysis won't catch that.
</Infobox>
## Creating custom pipeline components {#custom-components}
## Creating custom pipeline components {id="custom-components"}
A pipeline component is a function that receives a `Doc` object, modifies it and
returns it – for example, by using the current weights to make a prediction and
@ -678,7 +674,7 @@ your custom components, and make sure they can be saved and loaded.
</Infobox>
### Examples: Simple stateless pipeline components {#custom-components-simple}
### Examples: Simple stateless pipeline components {id="custom-components-simple"}
The following component receives the `Doc` in the pipeline and prints some
information about it: the number of tokens, the part-of-speech tags of the
@ -699,8 +695,7 @@ component under the name `"info_component"`.
> else. spaCy should now complain that it doesn't know a component of the
> name `"info_component"`.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.language import Language
@ -733,8 +728,7 @@ boundaries.
> to `None` (missing value), the parser will assign sentence boundaries in
> between.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.language import Language
@ -757,7 +751,7 @@ for sent in doc.sents:
print(sent.text)
```
### Component factories and stateful components {#custom-components-factories}
### Component factories and stateful components {id="custom-components-factories"}
Component factories are callables that take settings and return a **pipeline
component function**. This is useful if your component is stateful and if you
@ -787,8 +781,7 @@ All other settings can be passed in by the user via the `config` argument on
[`@Language.factory`](/api/language#factory) decorator also lets you define a
`default_config` that's used as a fallback.
```python
### With config {highlight="4,9"}
```python {title="With config",highlight="4,9"}
import spacy
from spacy.language import Language
@ -837,7 +830,7 @@ make your factory a separate function. That's also how spaCy does it internally.
</Accordion>
### Language-specific factories {#factories-language new="3"}
### Language-specific factories {id="factories-language",version="3"}
There are many use cases where you might want your pipeline components to be
language-specific. Sometimes this requires entirely different implementation per
@ -852,8 +845,7 @@ a token, the `Token.norm_` with an entry from a language-specific lookup table.
It's registered twice under the name `"token_normalizer"` – once using
`@English.factory` and once using `@German.factory`:
```python
### {executable="true"}
```python {executable="true"}
from spacy.lang.en import English
from spacy.lang.de import German
@ -895,7 +887,7 @@ available, falls back to looking up the regular factory name.
</Infobox>
### Example: Stateful component with settings {#example-stateful-components}
### Example: Stateful component with settings {id="example-stateful-components"}
This example shows a **stateful** pipeline component for handling acronyms:
based on a dictionary, it will detect acronyms and their expanded forms in both
@ -922,8 +914,7 @@ case-sensitive.
> should see an entry for the acronyms component, referencing the factory
> `acronyms` and the config settings.
```python
### {executable="true"}
```python {executable="true"}
from spacy.language import Language
from spacy.tokens import Doc
from spacy.matcher import PhraseMatcher
@ -964,7 +955,7 @@ doc = nlp("LOL, be right back")
print(doc._.acronyms)
```
## Initializing and serializing component data {#component-data}
## Initializing and serializing component data {id="component-data"}
Many stateful components depend on **data resources** like dictionaries and
lookup tables that should ideally be **configurable**. For example, it makes
@ -995,7 +986,7 @@ that if a component saves out its config and settings, the
since that's the config the component was created with. It will also fail if the
data is not JSON-serializable.
### Option 1: Using a registered function {#component-data-function}
### Option 1: Using a registered function {id="component-data-function"}
<Infobox>
@ -1025,8 +1016,7 @@ argument, the name:
> batchers. `misc` is intended for miscellaneous functions that don't fit
> anywhere else.
```python
### Registered function for assets {highlight="1"}
```python {title="Registered function for assets",highlight="1"}
@spacy.registry.misc("acronyms.slang_dict.v1")
def create_acronyms_slang_dict():
dictionary = {"lol": "laughing out loud", "brb": "be right back"}
@ -1064,7 +1054,7 @@ the name. Registered functions can also take **arguments**, by the way, that can
be defined in the config as well – you can read more about this in the docs on
[training with custom code](/usage/training#custom-code).
### Option 2: Save data with the pipeline and load it in once on initialization {#component-data-initialization}
### Option 2: Save data with the pipeline and load it in once on initialization {id="component-data-initialization"}
<Infobox>
@ -1094,8 +1084,7 @@ on [serialization methods](/usage/saving-loading/#serialization-methods).
> receive the directory path `/path/acronyms` and can then create files in this
> directory.
```python
### Custom serialization methods {highlight="7-11,13-15"}
```python {title="Custom serialization methods",highlight="7-11,13-15"}
import srsly
from spacy.util import ensure_path
@ -1157,7 +1146,7 @@ pipeline is loaded. For more background on this, see the usage guides on the
[config lifecycle](/usage/training#config-lifecycle) and
[custom initialization](/usage/training#initialization).
![Illustration of pipeline lifecycle](../images/lifecycle.svg)
![Illustration of pipeline lifecycle](/images/lifecycle.svg)
A component's `initialize` method needs to take at least **two named
arguments**: a `get_examples` callback that gives it access to the training
@ -1177,8 +1166,7 @@ be defined via the config – in this case a dictionary `data`.
> path = "/path/to/slang_dict.json"
> ```
```python
### Custom initialize method {highlight="5-6"}
```python {title="Custom initialize method",highlight="5-6"}
class AcronymComponent:
def __init__(self):
self.data = {}
@ -1196,7 +1184,7 @@ object is saved to disk, which will run the component's `to_disk` method. When
the pipeline is loaded back into spaCy later to use it, the `from_disk` method
will load the data back in.
## Python type hints and validation {#type-hints new="3"}
## Python type hints and validation {id="type-hints",version="3"}
spaCy's configs are powered by our machine learning library Thinc's
[configuration system](https://thinc.ai/docs/usage-config), which supports
@ -1241,8 +1229,7 @@ string value.
> and write a type hint for `log_level` that only accepts the exact string
> values `"DEBUG"`, `"INFO"` or `"CRITICAL"`.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.language import Language
from spacy.tokens import Doc
@ -1266,14 +1253,14 @@ nlp.add_pipe("debug", config={"log_level": "DEBUG"})
doc = nlp("This is a text...")
```
## Trainable components {#trainable-components new="3"}
## Trainable components {id="trainable-components",version="3"}
spaCy's [`TrainablePipe`](/api/pipe) class helps you implement your own
trainable components that have their own model instance, make predictions over
`Doc` objects and can be updated using [`spacy train`](/api/cli#train). This
lets you plug fully custom machine learning components into your pipeline.
![Illustration of Pipe methods](../images/trainable_component.svg)
![Illustration of Pipe methods](/images/trainable_component.svg)
You'll need the following:
@ -1331,8 +1318,7 @@ components. It also makes the components more **modular** and lets you
[swap](/usage/layers-architectures#swap-architectures) different architectures
in your config, and re-use model definitions.
```ini
### config.cfg (excerpt)
```ini {title="config.cfg (excerpt)"}
[components]
[components.textcat]
@ -1383,7 +1369,7 @@ into your spaCy pipeline, see the usage guide on
</Infobox>
## Extension attributes {#custom-components-attributes new="2"}
## Extension attributes {id="custom-components-attributes",version="2"}
spaCy allows you to set any custom attributes and methods on the `Doc`, `Span`
and `Token`, which become available as `Doc._`, `Span._` and `Token._` – for
@ -1466,8 +1452,7 @@ particular instance. If an attribute of the same name already exists, or if
you're trying to access an attribute that hasn't been registered, spaCy will
raise an `AttributeError`.
```python
### Example
```python {title="Example"}
from spacy.tokens import Doc, Span, Token
fruits = ["apple", "pear", "banana", "orange", "strawberry"]
@ -1494,7 +1479,7 @@ Once you've registered your custom attribute, you can also use the built-in
especially useful it you want to pass in a string instead of calling
`doc._.my_attr`.
### Example: Pipeline component for GPE entities and country meta data via a REST API {#component-example3}
### Example: Pipeline component for GPE entities and country meta data via a REST API {id="component-example3"}
This example shows the implementation of a pipeline component that fetches
country meta data via the [REST Countries API](https://restcountries.com), sets
@ -1502,8 +1487,7 @@ entity annotations for countries and sets custom attributes on the `Doc` and
`Span` – for example, the capital, latitude/longitude coordinates and even the
country flag.
```python
### {executable="true"}
```python {executable="true"}
import requests
from spacy.lang.en import English
from spacy.language import Language
@ -1570,7 +1554,7 @@ mistakes or foreign-language versions, you could also implement a
`like_country`-style getter function that makes a request to the search API
endpoint and returns the best-matching result.
### User hooks {#custom-components-user-hooks}
### User hooks {id="custom-components-user-hooks"}
While it's generally recommended to use the `Doc._`, `Span._` and `Token._`
proxies to add your own custom attributes, spaCy offers a few exceptions to
@ -1601,8 +1585,7 @@ to `Doc.user_span_hooks` and `Doc.user_token_hooks`.
| `user_token_hooks` | [`Token.similarity`](/api/token#similarity), [`Token.vector`](/api/token#vector), [`Token.has_vector`](/api/token#has_vector), [`Token.vector_norm`](/api/token#vector_norm), [`Token.conjuncts`](/api/token#conjuncts) |
| `user_span_hooks` | [`Span.similarity`](/api/span#similarity), [`Span.vector`](/api/span#vector), [`Span.has_vector`](/api/span#has_vector), [`Span.vector_norm`](/api/span#vector_norm), [`Span.root`](/api/span#root) |
```python
### Add custom similarity hooks
```python {title="Add custom similarity hooks"}
from spacy.language import Language
@ -1626,7 +1609,7 @@ def create_similarity_component(nlp, name, index: int):
return SimilarityModel(name, index)
```
## Developing plugins and wrappers {#plugins}
## Developing plugins and wrappers {id="plugins"}
We're very excited about all the new possibilities for community extensions and
plugins in spaCy, and we can't wait to see what you build with it! To get you
@ -1634,7 +1617,7 @@ started, here are a few tips, tricks and best
practices. [See here](/universe/?category=pipeline) for examples of other spaCy
extensions.
### Usage ideas {#custom-components-usage-ideas}
### Usage ideas {id="custom-components-usage-ideas"}
- **Adding new features and hooking in models.** For example, a sentiment
analysis model, or your preferred solution for lemmatization or sentiment
@ -1650,7 +1633,7 @@ extensions.
exports relevant information about the current state of the processed
document, and insert it at any point of your pipeline.
### Best practices {#custom-components-best-practices}
### Best practices {id="custom-components-best-practices"}
Extensions can claim their own `._` namespace and exist as standalone packages.
If you're developing a tool or library and want to make it easy for others to
@ -1738,7 +1721,7 @@ function that takes a `Doc`, modifies it and returns it.
to help people find it. If you post it on Twitter, feel free to tag
[@spacy_io](https://twitter.com/spacy_io) so we can check it out.
### Wrapping other models and libraries {#wrapping-models-libraries}
### Wrapping other models and libraries {id="wrapping-models-libraries"}
Let's say you have a custom entity recognizer that takes a list of strings and
returns their [BILUO tags](/usage/linguistic-features#accessing-ner). Given an
@ -1760,8 +1743,7 @@ wrapper has to do is compute the entity spans and overwrite the `doc.ents`.
> attributes. By definition, each token can only be part of one entity, so
> overlapping entity spans are not allowed.
```python
### {highlight="1,8-9"}
```python {highlight="1,8-9"}
import your_custom_entity_recognizer
from spacy.training import biluo_tags_to_spans
from spacy.language import Language
@ -1799,8 +1781,7 @@ label scheme than spaCy's default models.
> it fully replaces the `nlp` object instead of providing a pipeline component,
> since it also needs to handle tokenization.
```python
### {highlight="1,11,17-19"}
```python {highlight="1,11,17-19"}
import your_custom_model
from spacy.language import Language
from spacy.symbols import POS, TAG, DEP, HEAD

219
website/docs/usage/projects.md → website/docs/usage/projects.mdx
View File

@ -1,6 +1,6 @@
---
title: Projects
new: 3
version: 3
menu:
- ['Intro & Workflow', 'intro']
- ['Directory & Assets', 'directory']
@ -9,7 +9,7 @@ menu:
- ['Integrations', 'integrations']
---
## Introduction and workflow {#intro hidden="true"}
## Introduction and workflow {id="intro",hidden="true"}
> #### 🪐 Project templates
>
@ -27,7 +27,7 @@ and share your results with your team. spaCy projects can be used via the new
[`spacy project`](/api/cli#project) command and we provide templates in our
[`projects`](https://github.com/explosion/projects) repo.
![Illustration of project workflow and commands](../images/projects.svg)
![Illustration of project workflow and commands](/images/projects.svg)
<Project id="pipelines/tagger_parser_ud">
@ -43,16 +43,34 @@ and experiments, iterate on demos and prototypes and ship your models into
production.
<Grid narrow cols={3}>
<Integration title="DVC" logo="dvc" url="#dvc">Manage and version your data</Integration>
<Integration title="Prodigy" logo="prodigy" url="#prodigy">Create labelled training data</Integration>
<Integration title="Streamlit" logo="streamlit" url="#streamlit">Visualize and demo your pipelines</Integration>
<Integration title="FastAPI" logo="fastapi" url="#fastapi">Serve your models and host APIs</Integration>
<Integration title="Ray" logo="ray" url="#ray">Distributed and parallel training</Integration>
<Integration title="Weights &amp; Biases" logo="wandb" url="#wandb">Track your experiments and results</Integration>
<Integration title="Hugging Face Hub" logo="huggingface_hub" url="#huggingface_hub">Upload your pipelines to the Hugging Face Hub</Integration>
<Integration title="DVC" logo="dvc" url="#dvc">
Manage and version your data
</Integration>
<Integration title="Prodigy" logo="prodigy" url="#prodigy">
Create labelled training data
</Integration>
<Integration title="Streamlit" logo="streamlit" url="#streamlit">
Visualize and demo your pipelines
</Integration>
<Integration title="FastAPI" logo="fastapi" url="#fastapi">
Serve your models and host APIs
</Integration>
<Integration title="Ray" logo="ray" url="#ray">
Distributed and parallel training
</Integration>
<Integration title="Weights &amp; Biases" logo="wandb" url="#wandb">
Track your experiments and results
</Integration>
<Integration
title="Hugging Face Hub"
logo="huggingface_hub"
url="#huggingface_hub"
>
Upload your pipelines to the Hugging Face Hub
</Integration>
</Grid>
### 1. Clone a project template {#clone}
### 1. Clone a project template {id="clone"}
> #### Cloning under the hood
>
@ -64,7 +82,7 @@ project template and copies the files to a local directory. You can then run the
project, e.g. to train a pipeline and edit the commands and scripts to build
fully custom workflows.
```cli
```bash
python -m spacy project clone pipelines/tagger_parser_ud
```
@ -74,7 +92,7 @@ can specify an optional second argument to define the output directory. The
use the spaCy [`projects`](https://github.com/explosion/projects) repo. You can
also use any private repo you have access to with Git.
### 2. Fetch the project assets {#assets}
### 2. Fetch the project assets {id="assets"}
> #### project.yml
>
@ -97,7 +115,7 @@ with. Each project template comes with a `project.yml` that defines the assets
to download and where to put them. The [`spacy project assets`](/api/cli#run)
will fetch the project assets for you:
```cli
```bash
$ cd some_example_project
$ python -m spacy project assets
```
@ -112,7 +130,7 @@ necessarily want to download when you run `spacy project assets`. That's why
assets can be marked as [`extra`](#data-assets-url) - by default, these assets
are not downloaded. If they should be, run `spacy project assets --extra`.
### 3. Run a command {#run}
### 3. Run a command {id="run"}
> #### project.yml
>
@ -135,7 +153,7 @@ Commands consist of one or more steps and can be run with
[`spacy project run`](/api/cli#project-run). The following will run the command
`preprocess` defined in the `project.yml`:
```cli
```bash
$ python -m spacy project run preprocess
```
@ -155,7 +173,7 @@ detected, a corresponding warning is displayed. If you'd like to disable the
dependency check, set `check_requirements: false` in your project's
`project.yml`.
### 4. Run a workflow {#run-workfow}
### 4. Run a workflow {id="run-workfow"}
> #### project.yml
>
@ -175,7 +193,7 @@ pipeline on the converted data and if that's successful, run
installable Python package. The following command runs the workflow named `all`
defined in the `project.yml`, and executes the commands it specifies, in order:
```cli
```bash
$ python -m spacy project run all
```
@ -188,7 +206,7 @@ advanced data pipelines and track your changes in Git, check out the
from a workflow defined in your `project.yml` so you can manage your spaCy
project as a DVC repo.
### 5. Optional: Push to remote storage {#push}
### 5. Optional: Push to remote storage {id="push"}
> ```yaml
> ### project.yml
@ -204,7 +222,7 @@ a remote storage, using protocols like [S3](https://aws.amazon.com/s3/),
you **export** your pipeline packages, **share** work with your team, or **cache
results** to avoid repeating work.
```cli
```bash
$ python -m spacy project push
```
@ -213,9 +231,9 @@ different storages. To download state from a remote storage, you can use the
[`spacy project pull`](/api/cli#project-pull) command. For more details, see the
docs on [remote storage](#remote).
## Project directory and assets {#directory}
## Project directory and assets {id="directory"}
### project.yml {#project-yml}
### project.yml {id="project-yml"}
The `project.yml` defines the assets a project depends on, like datasets and
pretrained weights, as well as a series of commands that can be run separately
@ -277,7 +295,7 @@ pipelines.
| `spacy_version` | Optional spaCy version range like `>=3.0.0,<3.1.0` that the project is compatible with. If it's loaded with an incompatible version, an error is raised when the project is loaded. |
| `check_requirements` <Tag variant="new">3.4.2</Tag> | A flag determining whether to verify that the installed dependencies align with the project's `requirements.txt`. Defaults to `true`. |
### Data assets {#data-assets}
### Data assets {id="data-assets"}
Assets are any files that your project might need, like training and development
corpora or pretrained weights for initializing your model. Assets are defined in
@ -288,7 +306,7 @@ Asset URLs can be a number of different **protocols**: HTTP, HTTPS, FTP, SSH,
and even **cloud storage** such as GCS and S3. You can also download assets from
a **Git repo** instead.
#### Downloading from a URL or cloud storage {#data-assets-url}
#### Downloading from a URL or cloud storage {id="data-assets-url"}
Under the hood, spaCy uses the
[`smart-open`](https://github.com/RaRe-Technologies/smart_open) library so you
@ -318,7 +336,7 @@ dependencies to use certain protocols.
| `checksum` | Optional checksum of the file. If provided, it will be used to verify that the file matches and downloads will be skipped if a local file with the same checksum already exists. |
| `description` | Optional asset description, used in [auto-generated docs](#custom-docs). |
#### Downloading from a Git repo {#data-assets-git}
#### Downloading from a Git repo {id="data-assets-git"}
If a `git` block is provided, the asset is downloaded from the given Git
repository. You can download from any repo that you have access to. Under the
@ -345,7 +363,7 @@ files you need and not the whole repo.
| `checksum` | Optional checksum of the file. If provided, it will be used to verify that the file matches and downloads will be skipped if a local file with the same checksum already exists. |
| `description` | Optional asset description, used in [auto-generated docs](#custom-docs). |
#### Working with private assets {#data-asets-private}
#### Working with private assets {id="data-asets-private"}
> #### project.yml
>
@ -365,7 +383,7 @@ directory themselves. The [`project assets`](/api/cli#project-assets) command
will alert you about missing files and mismatched checksums, so you can ensure
that others are running your project with the same data.
### Dependencies and outputs {#deps-outputs}
### Dependencies and outputs {id="deps-outputs"}
Each command defined in the `project.yml` can optionally define a list of
dependencies and outputs. These are the files the command requires and creates.
@ -374,9 +392,8 @@ For example, a command for training a pipeline may depend on a
it will export a directory `model-best`, which you can then re-use in other
commands.
<!-- prettier-ignore -->
```yaml
### project.yml
{/* prettier-ignore */}
```yaml {title="project.yml"}
commands:
- name: train
help: 'Train a spaCy pipeline using the specified corpus and config'
@ -415,7 +432,7 @@ If you're planning on integrating your spaCy project with DVC, you can also use
`outputs_no_cache` instead of `outputs` to define outputs that won't be cached
or tracked.
### Files and directory structure {#project-files}
### Files and directory structure {id="project-files"}
The `project.yml` can define a list of `directories` that should be created
within a project – for instance, `assets`, `training`, `corpus` and so on. spaCy
@ -427,13 +444,12 @@ directory:
> #### project.yml
>
> <!-- prettier-ignore -->
> {/* prettier-ignore */}
> ```yaml
> directories: ['assets', 'configs', 'corpus', 'metas', 'metrics', 'notebooks', 'packages', 'scripts', 'training']
> ```
```yaml
### Example project directory
```yaml {title="Example project directory"}
├── project.yml # the project settings
├── project.lock # lockfile that tracks inputs/outputs
├── assets/ # downloaded data assets
@ -455,7 +471,7 @@ the only file that's required for a project is the `project.yml`.
---
## Custom scripts and projects {#custom}
## Custom scripts and projects {id="custom"}
The `project.yml` lets you define any custom commands and run them as part of
your training, evaluation or deployment workflows. The `script` section defines
@ -467,8 +483,7 @@ calls into [`pytest`](https://docs.pytest.org/en/latest/), runs your tests and
uses [`pytest-html`](https://github.com/pytest-dev/pytest-html) to export a test
report:
```yaml
### project.yml
```yaml {title="project.yml"}
commands:
- name: test
help: 'Test the trained pipeline'
@ -488,7 +503,7 @@ Setting `no_skip: true` means that the command will always run, even if the
dependencies (the trained pipeline) haven't changed. This makes sense here,
because you typically don't want to skip your tests.
### Writing custom scripts {#custom-scripts}
### Writing custom scripts {id="custom-scripts"}
Your project commands can include any custom scripts – essentially, anything you
can run from the command line. Here's an example of a custom script that uses
@ -504,8 +519,7 @@ that you can define via your `project.yml`:
> types. For instance, `batch_size: int` means that the value provided via the
> command line is converted to an integer.
```python
### scripts/custom_evaluation.py
```python {title="scripts/custom_evaluation.py"}
import typer
def custom_evaluation(batch_size: int = 128, model_path: str, data_path: str):
@ -531,9 +545,8 @@ override settings on the command line – for example using `--vars.batch_size`.
> everything with the same Python (not some other Python installed on your
> system). It also normalizes references to `python3`, `pip3` and `pip`.
<!-- prettier-ignore -->
```yaml
### project.yml
{/* prettier-ignore */}
```yaml {title="project.yml"}
vars:
batch_size: 128
@ -557,8 +570,7 @@ settings on the command line and passing through system-level settings.
> BATCH_SIZE=128 python -m spacy project run evaluate
> ```
```yaml
### project.yml
```yaml {title="project.yml"}
env:
batch_size: BATCH_SIZE
gpu_id: GPU_ID
@ -569,14 +581,14 @@ commands:
- 'python scripts/custom_evaluation.py ${env.batch_size}'
```
### Documenting your project {#custom-docs}
### Documenting your project {id="custom-docs"}
> #### Readme Example
>
> For more examples, see the [`projects`](https://github.com/explosion/projects)
> repo.
>
> ![Screenshot of auto-generated Markdown Readme](../images/project_document.jpg)
> ![Screenshot of auto-generated Markdown Readme](/images/project_document.jpg)
When your custom project is ready and you want to share it with others, you can
use the [`spacy project document`](/api/cli#project-document) command to
@ -586,7 +598,7 @@ in the project and include details on how to run the project, as well as links
to the relevant spaCy documentation to make it easy for others to get started
using your project.
```cli
```bash
$ python -m spacy project document --output README.md
```
@ -600,18 +612,18 @@ up to date.
Note that the contents of an existing file will be **replaced** if no existing
auto-generated docs are found. If you want spaCy to ignore a file and not update
it, you can add the comment marker `<!-- SPACY PROJECT: IGNORE -->` anywhere in
it, you can add the comment marker `{/* SPACY PROJECT: IGNORE */}` anywhere in
your markup.
</Infobox>
### Cloning from your own repo {#custom-repo}
### Cloning from your own repo {id="custom-repo"}
The [`spacy project clone`](/api/cli#project-clone) command lets you customize
the repo to clone from using the `--repo` option. It calls into `git`, so you'll
be able to clone from any repo that you have access to, including private repos.
```cli
```bash
python -m spacy project clone your_project --repo https://github.com/you/repo
```
@ -632,7 +644,7 @@ projects.
</Infobox>
## Remote Storage {#remote}
## Remote Storage {id="remote"}
You can persist your project outputs to a remote storage using the
[`project push`](/api/cli#project-push) command. This can help you **export**
@ -653,12 +665,11 @@ protocols.
> #### Example
>
> ```cli
> ```bash
> $ python -m spacy project pull local
> ```
```yaml
### project.yml
```yaml {title="project.yml"}
remotes:
default: 's3://my-spacy-bucket'
local: '/mnt/scratch/cache'
@ -673,9 +684,9 @@ according to a hash of the command string and the command's dependencies.
Finally, within those directories are files, named according to an MD5 hash of
their contents.
<!-- TODO: update with actual real example? -->
{/* TODO: update with actual real example? */}
<!-- prettier-ignore -->
{/* prettier-ignore */}
```yaml
└── urlencoded_file_path # Path of original file
├── some_command_hash # Hash of command you ran
@ -689,8 +700,7 @@ their contents.
For instance, let's say you had the following command in your `project.yml`:
```yaml
### project.yml
```yaml {title="project.yml"}
- name: train
help: 'Train a spaCy pipeline using the specified corpus and config'
script:
@ -719,7 +729,7 @@ execution context of your output. It would then compute an MD5 hash of the
`training/model-best` directory, and use those three pieces of information to
construct the storage URL.
```cli
```bash
$ python -m spacy project run train
$ python -m spacy project push
```
@ -740,9 +750,12 @@ state, and you don't have to come up with names or version numbers. However,
it's up to you to manage the size of your remote storage, and to remove files
that are no longer relevant to you.
## Integrations {#integrations}
## Integrations {id="integrations"}
### Data Version Control (DVC) {#dvc} <IntegrationLogo name="dvc" title="DVC" width={70} height="auto" align="right" />
{<H3 id="dvc"> Data Version Control (DVC)
<IntegrationLogo name="dvc" title="DVC" width={70} height="auto" align="right"/>
</H3>}
Data assets like training corpora or pretrained weights are at the core of any
NLP project, but they're often difficult to manage: you can't just check them
@ -787,7 +800,7 @@ can then manage your spaCy project like any other DVC project, run
and [`dvc repro`](https://dvc.org/doc/command-reference/repro) to reproduce the
workflow or individual commands.
```cli
```bash
$ python -m spacy project dvc [project_dir] [workflow_name]
```
@ -800,13 +813,14 @@ workflows, but only one can be tracked by DVC.
</Infobox>
<!-- TODO: <Project id="integrations/dvc">
</Project> -->
{/* { TODO: <Project id="integrations/dvc"></Project>} */}
---
### Prodigy {#prodigy} <IntegrationLogo name="prodigy" width={100} height="auto" align="right" />
{<H3 id="prodigy">Prodigy
<IntegrationLogo name="prodigy" width={100} height="auto" align="right" />
</H3>}
[Prodigy](https://prodi.gy) is a modern annotation tool for creating training
data for machine learning models, developed by us. It integrates with spaCy
@ -831,13 +845,12 @@ collected with Prodigy and training a spaCy pipeline:
> #### Example usage
>
> ```cli
> ```bash
> $ python -m spacy project run all
> ```
<!-- prettier-ignore -->
```yaml
### project.yml
{/* prettier-ignore */}
```yaml {title="project.yml"}
vars:
prodigy:
train_dataset: "fashion_brands_training"
@ -869,7 +882,11 @@ commands:
> #### Example train curve output
>
> [![Screenshot of train curve terminal output](../images/prodigy_train_curve.jpg)](https://prodi.gy/docs/recipes#train-curve)
> <Image
> src="/images/prodigy_train_curve.jpg"
> href="https://prodi.gy/docs/recipes#train-curve"
> alt="Screenshot of train curve terminal output"
> />
The [`train-curve`](https://prodi.gy/docs/recipes#train-curve) recipe is another
cool workflow you can include in your project. It will run the training with
@ -877,9 +894,8 @@ different portions of the data, e.g. 25%, 50%, 75% and 100%. As a rule of thumb,
if accuracy increases in the last segment, this could indicate that collecting
more annotations of the same type might improve the model further.
<!-- prettier-ignore -->
```yaml
### project.yml (excerpt)
{/* prettier-ignore */}
```yaml {title="project.yml (excerpt)"}
- name: "train_curve"
help: "Train the model with Prodigy by using different portions of training examples to evaluate if more annotations can potentially improve the performance"
script:
@ -908,7 +924,10 @@ improve performance.
---
### Streamlit {#streamlit} <IntegrationLogo name="streamlit" width={150} height="auto" align="right" />
{<H3 id="streamlit">Streamlit
<IntegrationLogo name="streamlit" width={150} height="auto" align="right" />
</H3>}
[Streamlit](https://streamlit.io) is a Python framework for building interactive
data apps. The [`spacy-streamlit`](https://github.com/explosion/spacy-streamlit)
@ -916,7 +935,7 @@ package helps you integrate spaCy visualizations into your Streamlit apps and
quickly spin up demos to explore your pipelines interactively. It includes a
full embedded visualizer, as well as individual components.
<!-- TODO: update once version is stable -->
{/* TODO: update once version is stable */}
> #### Installation
>
@ -924,7 +943,7 @@ full embedded visualizer, as well as individual components.
> $ pip install spacy-streamlit --pre
> ```
![](../images/spacy-streamlit.png)
![](/images/spacy-streamlit.png)
Using [`spacy-streamlit`](https://github.com/explosion/spacy-streamlit), your
projects can easily define their own scripts that spin up an interactive
@ -941,13 +960,12 @@ and explore your own custom trained pipelines.
> #### Example usage
>
> ```cli
> ```bash
> $ python -m spacy project run visualize
> ```
<!-- prettier-ignore -->
```yaml
### project.yml
{/* prettier-ignore */}
```yaml {title="project.yml"}
commands:
- name: visualize
help: "Visualize the pipeline's output interactively using Streamlit"
@ -967,7 +985,10 @@ https://github.com/explosion/projects/blob/v3/integrations/streamlit/scripts/vis
---
### FastAPI {#fastapi} <IntegrationLogo name="fastapi" width={100} height="auto" align="right" />
{<H3 id="fastapi">FastAPI
<IntegrationLogo name="fastapi" width={100} height="auto" align="right" />
</H3>}
[FastAPI](https://fastapi.tiangolo.com/) is a modern high-performance framework
for building REST APIs with Python, based on Python
@ -986,13 +1007,12 @@ query your API from Python and JavaScript (Vanilla JS and React).
> #### Example usage
>
> ```cli
> ```bash
> $ python -m spacy project run serve
> ```
<!-- prettier-ignore -->
```yaml
### project.yml
{/* prettier-ignore */}
```yaml {title="project.yml"}
- name: "serve"
help: "Serve the models via a FastAPI REST API using the given host and port"
script:
@ -1014,7 +1034,10 @@ https://github.com/explosion/projects/blob/v3/integrations/fastapi/scripts/main.
---
### Weights & Biases {#wandb} <IntegrationLogo name="wandb" width={175} height="auto" align="right" />
{<H3 id="wandb">Weights & Biases
<IntegrationLogo name="wandb" width={175} height="auto" align="right" />
</H3>}
[Weights & Biases](https://www.wandb.com/) is a popular platform for experiment
tracking. spaCy integrates with it out-of-the-box via the
@ -1036,9 +1059,9 @@ and you'll be able to see the impact it has on your results.
> model_log_interval = 1000
> ```
![Screenshot: Visualized training results](../images/wandb1.jpg)
![Screenshot: Visualized training results](/images/wandb1.jpg)
![Screenshot: Parameter importance using config values](../images/wandb2.jpg 'Parameter importance using config values')
![Screenshot: Parameter importance using config values](/images/wandb2.jpg 'Parameter importance using config values')
<Project id="integrations/wandb">
@ -1052,7 +1075,10 @@ logging the results.
---
### Hugging Face Hub {#huggingface_hub} <IntegrationLogo name="huggingface_hub" width={175} height="auto" align="right" />
{<H3 id="huggingface_hub">Hugging Face Hub
<IntegrationLogo name="huggingface_hub" width={175} height="auto" align="right" />
</H3>}
The [Hugging Face Hub](https://huggingface.co/) lets you upload models and share
them with others. It hosts models as Git-based repositories which are storage
@ -1065,7 +1091,7 @@ it's installed.
> #### Installation
>
> ```cli
> ```bash
> $ pip install spacy-huggingface-hub
> # Check that the CLI is registered
> $ python -m spacy huggingface-hub --help
@ -1078,7 +1104,7 @@ package, including the auto-generated pretty `README.md` and the model details
available in the `meta.json`. For examples, check out the
[spaCy pipelines](https://huggingface.co/spacy) we've uploaded.
```cli
```bash
$ huggingface-cli login
$ python -m spacy package ./en_ner_fashion ./output --build wheel
$ cd ./output/en_ner_fashion-0.0.0/dist
@ -1089,16 +1115,15 @@ After uploading, you will see the live URL of your pipeline packages, as well as
the direct URL to the model wheel you can install via `pip install`. You'll also
be able to test your pipeline interactively from your browser:
![Screenshot: interactive NER visualizer](../images/huggingface_hub.jpg)
![Screenshot: interactive NER visualizer](/images/huggingface_hub.jpg)
In your `project.yml`, you can add a command that uploads your trained and
packaged pipeline to the hub. You can either run this as a manual step, or
automatically as part of a workflow. Make sure to set `--build wheel` when
running `spacy package` to build a wheel file for your pipeline package.
<!-- prettier-ignore -->
```yaml
### project.yml
{/* prettier-ignore */}
```yaml {title="project.yml"}
- name: "push_to_hub"
help: "Upload the trained model to the Hugging Face Hub"
script:

200
website/docs/usage/rule-based-matching.md → website/docs/usage/rule-based-matching.mdx
View File

@ -60,7 +60,7 @@ another token that's at least 10 characters long.
</Accordion>
## Token-based matching {#matcher}
## Token-based matching {id="matcher"}
spaCy features a rule-matching engine, the [`Matcher`](/api/matcher), that
operates over tokens, similar to regular expressions. The rules can refer to
@ -72,7 +72,7 @@ To match large terminology lists, you can use the
[`PhraseMatcher`](/api/phrasematcher), which accepts `Doc` objects as match
patterns.
### Adding patterns {#adding-patterns}
### Adding patterns {id="adding-patterns"}
Let's say we want to enable spaCy to find a combination of three tokens:
@ -102,8 +102,7 @@ First, we initialize the `Matcher` with a vocab. The matcher must always share
the same vocab with the documents it will operate on. We can now call
[`matcher.add()`](/api/matcher#add) with an ID and a list of patterns.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.matcher import Matcher
@ -153,7 +152,7 @@ merge _some_ patterns into one token, while adding entity labels for other
pattern types. You shouldn't have to create different matchers for each of those
processes.
#### Available token attributes {#adding-patterns-attributes}
#### Available token attributes {id="adding-patterns-attributes"}
The available token pattern keys correspond to a number of
[`Token` attributes](/api/token#attributes). The supported attributes for
@ -208,7 +207,11 @@ you need to describe fields like this.
<Infobox title="Tip: Try the interactive matcher explorer">
[![Matcher demo](../images/matcher-demo.jpg)](https://explosion.ai/demos/matcher)
<Image
src="/images/matcher-demo.jpg"
href="https://explosion.ai/demos/matcher"
alt="Matcher demo"
/>
The [Matcher Explorer](https://explosion.ai/demos/matcher) lets you test the
rule-based `Matcher` by creating token patterns interactively and running them
@ -218,7 +221,7 @@ spaCy processes your text – and why your pattern matches, or why it doesn't.
</Infobox>
#### Extended pattern syntax and attributes {#adding-patterns-attributes-extended new="2.1"}
#### Extended pattern syntax and attributes {id="adding-patterns-attributes-extended",version="2.1"}
Instead of mapping to a single value, token patterns can also map to a
**dictionary of properties**. For example, to specify that the value of a lemma
@ -251,7 +254,7 @@ following rich comparison attributes are available:
| `INTERSECTS` | Attribute value (for `MORPH` or custom list attributes) has a non-empty intersection with a list. ~~Any~~ |
| `==`, `>=`, `<=`, `>`, `<` | Attribute value is equal, greater or equal, smaller or equal, greater or smaller. ~~Union[int, float]~~ |
#### Regular expressions {#regex new="2.1"}
#### Regular expressions {id="regex",version="2.1"}
In some cases, only matching tokens and token attributes isn't enough – for
example, you might want to match different spellings of a word, without having
@ -287,7 +290,7 @@ token. If you need to match on the whole text instead, see the details on
</Infobox>
##### Matching regular expressions on the full text {#regex-text}
##### Matching regular expressions on the full text {id="regex-text"}
If your expressions apply to multiple tokens, a simple solution is to match on
the `doc.text` with `re.finditer` and use the
@ -301,8 +304,7 @@ more valid tokens, `Doc.char_span` returns `None`.
> `"USA"` is a single token and `Span` objects are **sequences of tokens**. So
> `"US"` cannot be its own span, because it does not end on a token boundary.
```python
### {executable="true"}
```python {executable="true"}
import spacy
import re
@ -349,8 +351,7 @@ the (white)space tokens are split on. That hopefully shouldn't happen, though,
because it'd mean your regex is producing matches with leading or trailing
whitespace.
```python
### {highlight="5-8"}
```python {highlight="5-8"}
span = doc.char_span(start, end)
if span is not None:
print("Found match:", span.text)
@ -364,7 +365,7 @@ else:
</Accordion>
#### Fuzzy matching {#fuzzy new="3.5"}
#### Fuzzy matching {id="fuzzy", version="3.5"}
Fuzzy matching allows you to match tokens with alternate spellings, typos, etc.
without specifying every possible variant.
@ -393,7 +394,7 @@ pattern = [{"_": {"country": {"FUZZY": "Kyrgyzstan"}}}]
pattern = [{"_": {"country": {"FUZZY3": "Kyrgyzstan"}}}]
```
#### Regex and fuzzy matching with lists {#regex-fuzzy-lists new="3.5"}
#### Regex and fuzzy matching with lists {id="regex-fuzzy-lists", version="3.5"}
Starting in spaCy v3.5, both `REGEX` and `FUZZY` can be combined with the
attributes `IN` and `NOT_IN`:
@ -406,7 +407,7 @@ pattern = [{"TEXT": {"REGEX": {"NOT_IN": ["^awe(some)?$", "^wonder(ful)?"]}}}]
---
#### Operators and quantifiers {#quantifiers}
#### Operators and quantifiers {id="quantifiers"}
The matcher also lets you use quantifiers, specified as the `'OP'` key.
Quantifiers let you define sequences of tokens to be matched, e.g. one or more
@ -442,28 +443,28 @@ This quirk in the semantics is corrected in spaCy v2.1.0.
</Infobox>
#### Using wildcard token patterns {#adding-patterns-wildcard new="2"}
#### Using wildcard token patterns {id="adding-patterns-wildcard",version="2"}
While the token attributes offer many options to write highly specific patterns,
you can also use an empty dictionary, `{}` as a wildcard representing **any
token**. This is useful if you know the context of what you're trying to match,
but very little about the specific token and its characters. For example, let's
say you're trying to extract people's user names from your data. All you know is
that they are listed as "User name: {username}". The name itself may contain any
character, but no whitespace – so you'll know it will be handled as one token.
that they are listed as "User name: \{username\}". The name itself may contain
any character, but no whitespace – so you'll know it will be handled as one
token.
```python
[{"ORTH": "User"}, {"ORTH": "name"}, {"ORTH": ":"}, {}]
```
#### Validating and debugging patterns {#pattern-validation new="2.1"}
#### Validating and debugging patterns {id="pattern-validation",version="2.1"}
The `Matcher` can validate patterns against a JSON schema with the option
`validate=True`. This is useful for debugging patterns during development, in
particular for catching unsupported attributes.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.matcher import Matcher
@ -479,7 +480,7 @@ matcher.add("HelloWorld", [pattern])
```
### Adding on_match rules {#on_match}
### Adding on_match rules {id="on_match"}
To move on to a more realistic example, let's say you're working with a large
corpus of blog articles, and you want to match all mentions of "Google I/O"
@ -487,8 +488,7 @@ corpus of blog articles, and you want to match all mentions of "Google I/O"
match on the uppercase versions, avoiding matches with phrases such as "Google
i/o".
```python
### {executable="true"}
```python {executable="true"}
from spacy.lang.en import English
from spacy.matcher import Matcher
from spacy.tokens import Span
@ -551,7 +551,7 @@ you prefer.
| `i` | Index of the current match (`matches[i`]). ~~int~~ |
| `matches` | A list of `(match_id, start, end)` tuples, describing the matches. A match tuple describes a span `doc[start:end`]. ~~List[Tuple[int, int int]]~~ |
### Creating spans from matches {#matcher-spans}
### Creating spans from matches {id="matcher-spans"}
Creating [`Span`](/api/span) objects from the returned matches is a very common
use case. spaCy makes this easy by giving you access to the `start` and `end`
@ -560,8 +560,7 @@ label. As of spaCy v3.0, you can also set `as_spans=True` when calling the
matcher on a `Doc`, which will return a list of [`Span`](/api/span) objects
using the `match_id` as the span label.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.matcher import Matcher
from spacy.tokens import Span
@ -584,7 +583,7 @@ for span in matches:
print(span.text, span.label_)
```
### Using custom pipeline components {#matcher-pipeline}
### Using custom pipeline components {id="matcher-pipeline"}
Let's say your data also contains some annoying pre-processing artifacts, like
leftover HTML line breaks (e.g. `<br>` or `<BR/>`). To make your text easier to
@ -595,8 +594,7 @@ process the text. You can achieve this by adding a
that's called on each `Doc` object, merges the leftover HTML spans and sets an
attribute `bad_html` on the token.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.language import Language
from spacy.matcher import Matcher
@ -662,7 +660,7 @@ and **extension attributes**, see the
</Infobox>
### Example: Using linguistic annotations {#example1}
### Example: Using linguistic annotations {id="example1"}
Let's say you're analyzing user comments and you want to find out what people
are saying about Facebook. You want to start off by finding adjectives following
@ -691,8 +689,7 @@ calculate the start and end of the matched span within the sentence. Using
displaCy in ["manual" mode](/usage/visualizers#manual-usage) lets you pass in a
list of dictionaries containing the text and entities to render.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy import displacy
from spacy.matcher import Matcher
@ -728,7 +725,7 @@ matches = matcher(doc)
displacy.render(matched_sents, style="ent", manual=True)
```
### Example: Phone numbers {#example2}
### Example: Phone numbers {id="example2"}
Phone numbers can have many different formats and matching them is often tricky.
During tokenization, spaCy will leave sequences of numbers intact and only split
@ -763,8 +760,7 @@ set of rules like this is often better than training a model. It'll produce more
predictable results, is much easier to modify and extend, and doesn't require
any training data – only a set of test cases.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.matcher import Matcher
@ -782,7 +778,7 @@ for match_id, start, end in matches:
print(span.text)
```
### Example: Hashtags and emoji on social media {#example3}
### Example: Hashtags and emoji on social media {id="example3"}
Social media posts, especially tweets, can be difficult to work with. They're
very short and often contain various emoji and hashtags. By only looking at the
@ -812,8 +808,7 @@ that you can create a pattern for one or more emoji tokens. Valid hashtags
usually consist of a `#`, plus a sequence of ASCII characters with no
whitespace, making them easy to match as well.
```python
### {executable="true"}
```python {executable="true"}
from spacy.lang.en import English
from spacy.matcher import Matcher
@ -883,8 +878,7 @@ To label the hashtags, we can use a
[custom attribute](/usage/processing-pipelines#custom-components-attributes) set
on the respective token:
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.matcher import Matcher
from spacy.tokens import Token
@ -914,17 +908,16 @@ for token in doc:
print(token.text, token._.is_hashtag)
```
## Efficient phrase matching {#phrasematcher}
## Efficient phrase matching {id="phrasematcher"}
If you need to match large terminology lists, you can also use the
[`PhraseMatcher`](/api/phrasematcher) and create [`Doc`](/api/doc) objects
instead of token patterns, which is much more efficient overall. The `Doc`
patterns can contain single or multiple tokens.
### Adding phrase patterns {#adding-phrase-patterns}
### Adding phrase patterns {id="adding-phrase-patterns"}
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.matcher import PhraseMatcher
@ -967,7 +960,7 @@ as a stream.
</Infobox>
### Matching on other token attributes {#phrasematcher-attrs new="2.1"}
### Matching on other token attributes {id="phrasematcher-attrs",version="2.1"}
By default, the `PhraseMatcher` will match on the verbatim token text, e.g.
`Token.text`. By setting the `attr` argument on initialization, you can change
@ -975,8 +968,7 @@ By default, the `PhraseMatcher` will match on the verbatim token text, e.g.
pattern to the matched `Doc`. For example, using the attribute `LOWER` lets you
match on `Token.lower` and create case-insensitive match patterns:
```python
### {executable="true"}
```python {executable="true"}
from spacy.lang.en import English
from spacy.matcher import PhraseMatcher
@ -1010,8 +1002,7 @@ will be tokenized and you'll be able to find tokens and combinations of tokens
based on a few examples. Here, we're matching on the shapes `ddd.d.d.d` and
`ddd.ddd.d.d`:
```python
### {executable="true"}
```python {executable="true"}
from spacy.lang.en import English
from spacy.matcher import PhraseMatcher
@ -1031,7 +1022,7 @@ to match phrases with the same sequence of punctuation and non-punctuation
tokens as the pattern. But this can easily get confusing and doesn't have much
of an advantage over writing one or two token patterns.
## Dependency Matcher {#dependencymatcher new="3" model="parser"}
## Dependency Matcher {id="dependencymatcher",version="3",model="parser"}
The [`DependencyMatcher`](/api/dependencymatcher) lets you match patterns within
the dependency parse using
@ -1099,7 +1090,7 @@ can be used as `LEFT_ID` in another dict.
</Infobox>
### Dependency matcher operators {#dependencymatcher-operators}
### Dependency matcher operators {id="dependencymatcher-operators"}
The following operators are supported by the `DependencyMatcher`, most of which
come directly from
@ -1120,7 +1111,7 @@ come directly from
| `A $++ B` | `B` is a right sibling of `A`, i.e. `A` and `B` have the same parent and `A.i < B.i`. |
| `A $-- B` | `B` is a left sibling of `A`, i.e. `A` and `B` have the same parent and `A.i > B.i`. |
### Designing dependency matcher patterns {#dependencymatcher-patterns}
### Designing dependency matcher patterns {id="dependencymatcher-patterns"}
Let's say we want to find sentences describing who founded what kind of company:
@ -1145,9 +1136,11 @@ relations and tokens we want to match:
> displacy.serve(doc)
> ```
import DisplaCyDepFoundedHtml from 'images/displacy-dep-founded.html'
<Iframe title="displaCy visualization of dependencies" html={DisplaCyDepFoundedHtml} height={450} />
<Iframe
title="displaCy visualization of dependencies"
src="/images/displacy-dep-founded.html"
height={450}
/>
The relations we're interested in are:
@ -1165,8 +1158,7 @@ head to an immediate dependent as `head > child`.
The simplest dependency matcher pattern will identify and name a single token in
the tree:
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.matcher import DependencyMatcher
@ -1187,8 +1179,7 @@ print(matches) # [(4851363122962674176, [1])]
Now that we have a named anchor token (`anchor_founded`), we can add the founder
as the immediate dependent (`>`) of `founded` with the dependency label `nsubj`:
```python
### Step 1 {highlight="8,10"}
```python {title="Step 1",highlight="8,10"}
pattern = [
{
"RIGHT_ID": "anchor_founded",
@ -1206,8 +1197,7 @@ pattern = [
The direct object (`dobj`) is added in the same way:
```python
### Step 2 {highlight=""}
```python {title="Step 2"}
pattern = [
#...
{
@ -1227,8 +1217,7 @@ tokens into the pattern**. For the final part of our pattern, we'll specify that
the token `founded_object` should have a modifier with the dependency relation
`amod` or `compound`:
```python
### Step 3 {highlight="7"}
```python {title="Step 3",highlight="7"}
pattern = [
# ...
{
@ -1247,12 +1236,11 @@ each new token needs to be linked to an existing token on its left. As for
`founded` in this example, a token may be linked to more than one token on its
right:
![Dependency matcher pattern](../images/dep-match-diagram.svg)
![Dependency matcher pattern](/images/dep-match-diagram.svg)
The full pattern comes together as shown in the example below:
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.matcher import DependencyMatcher
@ -1308,14 +1296,14 @@ of patterns such as `{}` that match any token in the sentence.
</Infobox>
## Rule-based entity recognition {#entityruler new="2.1"}
## Rule-based entity recognition {id="entityruler",version="2.1"}
The [`EntityRuler`](/api/entityruler) is a component that lets you add named
entities based on pattern dictionaries, which makes it easy to combine
rule-based and statistical named entity recognition for even more powerful
pipelines.
### Entity Patterns {#entityruler-patterns}
### Entity Patterns {id="entityruler-patterns"}
Entity patterns are dictionaries with two keys: `"label"`, specifying the label
to assign to the entity if the pattern is matched, and `"pattern"`, the match
@ -1333,7 +1321,7 @@ pattern. The entity ruler accepts two types of patterns:
{"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}
```
### Using the entity ruler {#entityruler-usage}
### Using the entity ruler {id="entityruler-usage"}
The [`EntityRuler`](/api/entityruler) is a pipeline component that's typically
added via [`nlp.add_pipe`](/api/language#add_pipe). When the `nlp` object is
@ -1343,8 +1331,7 @@ matches were to overlap, the pattern matching most tokens takes priority. If
they also happen to be equally long, then the match occurring first in the `Doc`
is chosen.
```python
### {executable="true"}
```python {executable="true"}
from spacy.lang.en import English
nlp = English()
@ -1366,8 +1353,7 @@ entity ruler will only add spans to the `doc.ents` if they don't overlap with
existing entities predicted by the model. To overwrite overlapping entities, you
can set `overwrite_ents=True` on initialization.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -1379,7 +1365,7 @@ doc = nlp("MyCorp Inc. is a company in the U.S.")
print([(ent.text, ent.label_) for ent in doc.ents])
```
#### Validating and debugging EntityRuler patterns {#entityruler-pattern-validation new="2.1.8"}
#### Validating and debugging EntityRuler patterns {id="entityruler-pattern-validation",version="2.1.8"}
The entity ruler can validate patterns against a JSON schema with the config
setting `"validate"`. See details under
@ -1389,14 +1375,13 @@ setting `"validate"`. See details under
ruler = nlp.add_pipe("entity_ruler", config={"validate": True})
```
### Adding IDs to patterns {#entityruler-ent-ids new="2.2.2"}
### Adding IDs to patterns {id="entityruler-ent-ids",version="2.2.2"}
The [`EntityRuler`](/api/entityruler) can also accept an `id` attribute for each
pattern. Using the `id` attribute allows multiple patterns to be associated with
the same entity.
```python
### {executable="true"}
```python {executable="true"}
from spacy.lang.en import English
nlp = English()
@ -1418,15 +1403,14 @@ patterns, the `ent_id_` property of the matched entity is set to the `id` given
in the patterns. So in the example above it's easy to identify that "San
Francisco" and "San Fran" are both the same entity.
### Using pattern files {#entityruler-files}
### Using pattern files {id="entityruler-files"}
The [`to_disk`](/api/entityruler#to_disk) and
[`from_disk`](/api/entityruler#from_disk) let you save and load patterns to and
from JSONL (newline-delimited JSON) files, containing one pattern object per
line.
```json
### patterns.jsonl
```json {title="patterns.jsonl"}
{"label": "ORG", "pattern": "Apple"}
{"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}
```
@ -1463,9 +1447,9 @@ all pipeline components will be restored and deserialized – including the enti
ruler. This lets you ship powerful pipeline packages with binary weights _and_
rules included!
### Using a large number of phrase patterns {#entityruler-large-phrase-patterns new="2.2.4"}
### Using a large number of phrase patterns {id="entityruler-large-phrase-patterns",version="2.2.4"}
<!-- TODO: double-check that this still works if the ruler is added to the pipeline on creation, and include suggestion if needed -->
{/* TODO: double-check that this still works if the ruler is added to the pipeline on creation, and include suggestion if needed */}
When using a large amount of **phrase patterns** (roughly > 10000) it's useful
to understand how the `add_patterns` function of the entity ruler works. For
@ -1491,14 +1475,14 @@ with nlp.select_pipes(enable="tagger"):
ruler.add_patterns(patterns)
```
## Rule-based span matching {#spanruler new="3.3.1"}
## Rule-based span matching {id="spanruler",version="3.3.1"}
The [`SpanRuler`](/api/spanruler) is a generalized version of the entity ruler
that lets you add spans to `doc.spans` or `doc.ents` based on pattern
dictionaries, which makes it easy to combine rule-based and statistical pipeline
components.
### Span patterns {#spanruler-patterns}
### Span patterns {id="spanruler-patterns"}
The [pattern format](#entityruler-patterns) is the same as for the entity ruler:
@ -1514,7 +1498,7 @@ The [pattern format](#entityruler-patterns) is the same as for the entity ruler:
{"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}
```
### Using the span ruler {#spanruler-usage}
### Using the span ruler {id="spanruler-usage"}
The [`SpanRuler`](/api/spanruler) is a pipeline component that's typically added
via [`nlp.add_pipe`](/api/language#add_pipe). When the `nlp` object is called on
@ -1524,8 +1508,7 @@ Unlike in `doc.ents`, overlapping matches are allowed in `doc.spans`, so no
filtering is required, but optional filtering and sorting can be applied to the
spans before they're saved.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.blank("en")
@ -1547,8 +1530,7 @@ always filtered, using [`util.filter_spans`](/api/top-level#util.filter_spans)
by default. See the [`SpanRuler` API docs](/api/spanruler) for more information
about how to customize the sorting and filtering of matched spans.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -1562,14 +1544,13 @@ doc = nlp("MyCorp Inc. is a company in the U.S.")
print([(ent.text, ent.label_) for ent in doc.ents])
```
### Using pattern files {#spanruler-files}
### Using pattern files {id="spanruler-files"}
You can save patterns in a JSONL file (newline-delimited JSON) to load with
[`SpanRuler.initialize`](/api/spanruler#initialize) or
[`SpanRuler.add_patterns`](/api/spanruler#add_patterns).
```json
### patterns.jsonl
```json {title="patterns.jsonl"}
{"label": "ORG", "pattern": "Apple"}
{"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}
```
@ -1593,7 +1574,7 @@ separately and then added through
</Infobox>
## Combining models and rules {#models-rules}
## Combining models and rules {id="models-rules"}
You can combine statistical and rule-based components in a variety of ways.
Rule-based components can be used to improve the accuracy of statistical models,
@ -1604,7 +1585,7 @@ components after a statistical model to correct common errors. Finally,
rule-based components can reference the attributes set by statistical models, in
order to implement more abstract logic.
### Example: Expanding named entities {#models-rules-ner}
### Example: Expanding named entities {id="models-rules-ner"}
When using a trained
[named entity recognition](/usage/linguistic-features/#named-entities) model to
@ -1630,8 +1611,7 @@ or "Dr.". This makes sense, because it makes it easier to resolve the entity
type back to a knowledge base. But what if your application needs the full
names, _including_ the titles?
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -1650,8 +1630,7 @@ expands the entity span by one token. After all, what all titles in this example
have in common is that _if_ they occur, they occur in the **previous token**
right before the person entity.
```python
### {highlight="9-13"}
```python {highlight="9-13"}
from spacy.language import Language
from spacy.tokens import Span
@ -1679,8 +1658,7 @@ register it as a [pipeline component](/usage/processing-pipelines) so it can run
automatically when processing a text. We can use
[`nlp.add_pipe`](/api/language#add_pipe) to add it to the current pipeline.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.language import Language
from spacy.tokens import Span
@ -1729,8 +1707,7 @@ We can now use the [`Span.set_extension`](/api/span#set_extension) method to add
the custom extension attribute `"person_title"`, using `get_person_title` as the
getter function.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.tokens import Span
@ -1749,7 +1726,7 @@ doc = nlp("Dr Alex Smith chaired first board meeting of Acme Corp Inc.")
print([(ent.text, ent.label_, ent._.person_title) for ent in doc.ents])
```
### Example: Using entities, part-of-speech tags and the dependency parse {#models-rules-pos-dep}
### Example: Using entities, part-of-speech tags and the dependency parse {id="models-rules-pos-dep"}
> #### Linguistic features
>
@ -1772,8 +1749,7 @@ tense**, whether company names are attached to it and whether the person is the
subject. All of this information is available in the part-of-speech tags and the
dependency parse.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -1788,7 +1764,7 @@ print([(ent.text, ent.label_) for ent in doc.ents])
> - `VBD`: Verb, past tense.
> - `IN`: Conjunction, subordinating or preposition.
![Visualization of dependency parse](../images/displacy-model-rules.svg "[`spacy.displacy`](/api/top-level#displacy) visualization with `options={'fine_grained': True}` to output the fine-grained part-of-speech tags, i.e. `Token.tag_`")
![Visualization of dependency parse](/images/displacy-model-rules.svg "[`spacy.displacy`](/api/top-level#displacy) visualization with `options={'fine_grained': True}` to output the fine-grained part-of-speech tags, i.e. `Token.tag_`")
In this example, "worked" is the root of the sentence and is a past tense verb.
Its subject is "Alex Smith", the person who worked. "at Acme Corp Inc." is a
@ -1837,8 +1813,7 @@ the entity `Span` – for example `._.orgs` or `._.prev_orgs` and
> retokenizer.merge(ent)
> ```
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.language import Language
from spacy import displacy
@ -1871,12 +1846,11 @@ notice that our current logic fails and doesn't correctly detect the company as
a past organization. That's because the root is a participle and the tense
information is in the attached auxiliary "was":
![Visualization of dependency parse](../images/displacy-model-rules2.svg)
![Visualization of dependency parse](/images/displacy-model-rules2.svg)
To solve this, we can adjust the rules to also check for the above construction:
```python
### {highlight="10-12"}
```python {highlight="10-12"}
@Language.component("extract_person_orgs")
def extract_person_orgs(doc):
person_entities = [ent for ent in doc.ents if ent.label_ == "PERSON"]

87
website/docs/usage/saving-loading.md → website/docs/usage/saving-loading.mdx
View File

@ -8,13 +8,11 @@ menu:
- ['Trained Pipelines', 'models']
---
## Basics {#basics hidden="true"}
import Serialization101 from 'usage/101/\_serialization.md'
## Basics {id="basics",hidden="true"}
<Serialization101 />
### Serializing the pipeline {#pipeline}
### Serializing the pipeline {id="pipeline"}
When serializing the pipeline, keep in mind that this will only save out the
**binary data for the individual components** to allow spaCy to restore them –
@ -30,14 +28,12 @@ contains the pipeline configuration and all the relevant settings.
> dictionary containing the training configuration, pipeline component factories
> and other settings. It is saved out with a pipeline as the `config.cfg`.
```python
### Serialize
```python {title="Serialize"}
config = nlp.config
bytes_data = nlp.to_bytes()
```
```python
### Deserialize
```python {title="Deserialize"}
lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
nlp = lang_cls.from_config(config)
nlp.from_bytes(bytes_data)
@ -49,7 +45,7 @@ the language class, creates and adds the pipeline components based on the config
and _then_ loads in the binary data. You can read more about this process
[here](/usage/processing-pipelines#pipelines).
## Serializing Doc objects efficiently {#docs new="2.2"}
## Serializing Doc objects efficiently {id="docs",version="2.2"}
If you're working with lots of data, you'll probably need to pass analyses
between machines, either to use something like [Dask](https://dask.org) or
@ -64,8 +60,7 @@ collection of `Doc` objects together, and is much more efficient than calling
also control what data gets saved, and you can merge pallets together for easy
map/reduce-style processing.
```python
### {highlight="4,8,9,13,14"}
```python {highlight="4,8,9,13,14"}
import spacy
from spacy.tokens import DocBin
@ -101,7 +96,7 @@ print([doc._.my_custom_attr for doc in docs])
</Infobox>
### Using Pickle {#pickle}
### Using Pickle {id="pickle"}
> #### Example
>
@ -124,8 +119,7 @@ the entire pipeline once. And instead of pickling several `Doc` objects
separately, pickle a list of `Doc` objects. Since they all share a reference to
the _same_ `Vocab` object, it will only be included once.
```python
### Pickling objects with shared data {highlight="8-9"}
```python {title="Pickling objects with shared data",highlight="8-9"}
doc1 = nlp("Hello world")
doc2 = nlp("This is a test")
@ -161,7 +155,7 @@ data = pickle.dumps(span_doc)
</Infobox>
## Implementing serialization methods {#serialization-methods}
## Implementing serialization methods {id="serialization-methods"}
When you call [`nlp.to_disk`](/api/language#to_disk),
[`nlp.from_disk`](/api/language#from_disk) or load a pipeline package, spaCy
@ -201,8 +195,7 @@ the data to and from a JSON file.
> saving out a pipeline with a rule-based entity recognizer and including all
> rules _with_ the component data.
```python
### {highlight="16-23,25-30"}
```python {highlight="16-23,25-30"}
import json
from spacy import Language
from spacy.util import ensure_path
@ -242,8 +235,7 @@ After adding the component to the pipeline and adding some data to it, we can
serialize the `nlp` object to a directory, which will call the custom
component's `to_disk` method.
```python
### {highlight="2-4"}
```python {highlight="2-4"}
nlp = spacy.load("en_core_web_sm")
my_component = nlp.add_pipe("my_component")
my_component.add({"hello": "world"})
@ -254,8 +246,7 @@ The contents of the directory would then look like this.
`CustomComponent.to_disk` converted the data to a JSON string and saved it to a
file `data.json` in its subdirectory:
```yaml
### Directory structure {highlight="2-3"}
```yaml {title="Directory structure",highlight="2-3"}
└── /path/to/pipeline
├── my_component # data serialized by "my_component"
│ └── data.json
@ -292,9 +283,9 @@ custom components to spaCy automatically.
</Infobox>
<!-- ## Initializing components with data {#initialization new="3"} -->
{/* ## Initializing components with data {id="initialization",version="3"} */}
## Using entry points {#entry-points new="2.1"}
## Using entry points {id="entry-points",version="2.1"}
Entry points let you expose parts of a Python package you write to other Python
packages. This lets one application easily customize the behavior of another, by
@ -313,7 +304,7 @@ installed in the same environment – that's it.
| `spacy_lookups` | Group of entry points for custom [`Lookups`](/api/lookups), including lemmatizer data. Used by spaCy's [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) package. |
| [`spacy_displacy_colors`](#entry-points-displacy) | Group of entry points of custom label colors for the [displaCy visualizer](/usage/visualizers#ent). The key name doesn't matter, but it should point to a dict of labels and color values. Useful for custom models that predict different entity types. |
### Custom components via entry points {#entry-points-components}
### Custom components via entry points {id="entry-points-components"}
When you load a pipeline, spaCy will generally use its `config.cfg` to set up
the language class and construct the pipeline. The pipeline is specified as a
@ -343,8 +334,7 @@ snake when it's called:
> └── setup.py # setup file for pip installation
> ```
```python
### snek.py
```python {title="snek.py"}
from spacy.language import Language
snek = """
@ -377,8 +367,7 @@ entry to the factories, you can now expose it in your `setup.py` via the
> the created entry point is named `snek` and points to the function
> `snek_component` in the module `snek`, i.e. `snek.py`.
```python
### setup.py {highlight="5-7"}
```python {title="setup.py",highlight="5-7"}
from setuptools import setup
setup(
@ -459,8 +448,7 @@ class SnekFactory:
return doc
```
```diff
### setup.py
```diff {title="setup.py"}
entry_points={
- "spacy_factories": ["snek = snek:snek_component"]
+ "spacy_factories": ["snek = snek:SnekFactory"]
@ -496,7 +484,7 @@ The above example will serialize the current snake in a `snek.txt` in the data
directory. When a pipeline using the `snek` component is loaded, it will open
the `snek.txt` and make it available to the component.
### Custom language classes via entry points {#entry-points-languages}
### Custom language classes via entry points {id="entry-points-languages"}
To stay with the theme of the previous example and
[this blog post on entry points](https://amir.rachum.com/blog/2017/07/28/python-entry-points/),
@ -505,8 +493,7 @@ custom pipeline – but you don't necessarily want to modify spaCy's code to ad
language. In your package, you could then implement the following
[custom language subclass](/usage/linguistic-features#language-subclass):
```python
### snek.py
```python {title="snek.py"}
from spacy.language import Language
class SnekDefaults(Language.Defaults):
@ -521,8 +508,7 @@ Alongside the `spacy_factories`, there's also an entry point option for
`spacy_languages`, which maps language codes to language-specific `Language`
subclasses:
```diff
### setup.py
```diff {title="setup.py"}
from setuptools import setup
setup(
@ -540,7 +526,7 @@ pipeline packages you [train](/usage/training), which could then specify
`lang = snk` in their `config.cfg` without spaCy raising an error because the
language is not available in the core library.
### Custom displaCy colors via entry points {#entry-points-displacy new="2.2"}
### Custom displaCy colors via entry points {id="entry-points-displacy",version="2.2"}
If you're training a named entity recognition model for a custom domain, you may
end up training different labels that don't have pre-defined colors in the
@ -555,8 +541,7 @@ values.
> [scispaCy](/universe/project/scispacy) and
> [Blackstone](/universe/project/blackstone).
```python
### snek.py
```python {title="snek.py"}
displacy_colors = {"SNEK": "#3dff74", "HUMAN": "#cfc5ff"}
```
@ -564,8 +549,7 @@ Given the above colors, the entry point can be defined as follows. Entry points
need to have a name, so we use the key `colors`. However, the name doesn't
matter and whatever is defined in the entry point group will be used.
```diff
### setup.py
```diff {title="setup.py"}
from setuptools import setup
setup(
@ -580,11 +564,13 @@ After installing the package, the custom colors will be used when visualizing
text with `displacy`. Whenever the label `SNEK` is assigned, it will be
displayed in `#3dff74`.
import DisplaCyEntSnekHtml from 'images/displacy-ent-snek.html'
<Iframe
title="displaCy visualization of entities"
src="/images/displacy-ent-snek.html"
height={100}
/>
<Iframe title="displaCy visualization of entities" html={DisplaCyEntSnekHtml} height={100} />
## Saving, loading and distributing trained pipelines {#models}
## Saving, loading and distributing trained pipelines {id="models"}
After training your pipeline, you'll usually want to save its state, and load it
back later. You can do this with the [`Language.to_disk`](/api/language#to_disk)
@ -629,7 +615,7 @@ Universal Dependencies treebank and generates an installable Python package.
</Project>
### Generating a pipeline package {#models-generating}
### Generating a pipeline package {id="models-generating"}
<Infobox title="Important note" variant="warning">
@ -662,7 +648,7 @@ this, see the [`package`](/api/cli#package) docs.
> }
> ```
```cli
```bash
$ python -m spacy package ./en_example_pipeline ./packages
```
@ -671,8 +657,7 @@ This command will create a pipeline package directory and will run
`.tar.gz` archive of your package that can be installed using `pip install`.
Installing the binary wheel is usually more efficient.
```yaml
### Directory structure
```yaml {title="Directory structure"}
└── /
├── MANIFEST.in # to include meta.json
├── meta.json # pipeline meta data
@ -693,7 +678,7 @@ If you're creating the package manually, keep in mind that the directories need
to be named according to the naming conventions of `lang_name` and
`lang_name-version`.
### Including custom functions and components {#models-custom}
### Including custom functions and components {id="models-custom"}
If your pipeline includes
[custom components](/usage/processing-pipelines#custom-components), model
@ -713,7 +698,7 @@ argument.
> ...
> ```
```cli
```bash
$ python -m spacy package ./en_example_pipeline ./packages --code functions.py
```
@ -744,7 +729,7 @@ spaCy to export the current state of its `nlp` objects via
</Infobox>
### Loading a custom pipeline package {#loading}
### Loading a custom pipeline package {id="loading"}
To load a pipeline from a data directory, you can use
[`spacy.load()`](/api/top-level#spacy.load) with the local path. This will look

102
website/docs/usage/spacy-101.md → website/docs/usage/spacy-101.mdx
View File

@ -30,18 +30,24 @@ quick introduction.
<Infobox title="Take the free interactive course">
[![Advanced NLP with spaCy](../images/course.jpg)](https://course.spacy.io)
<Image
src="/images/course.jpg"
href="https://course.spacy.io"
alt="Advanced NLP with spaCy"
/>
In this course you'll learn how to use spaCy to build advanced natural language
understanding systems, using both rule-based and machine learning approaches. It
includes 55 exercises featuring interactive coding practice, multiple-choice
questions and slide decks.
<p><Button to="https://course.spacy.io" variant="primary">Start the course</Button></p>
<Button to="https://course.spacy.io" variant="primary">
{'Start the course'}
</Button>
</Infobox>
## What's spaCy? {#whats-spacy}
## What's spaCy? {id="whats-spacy"}
<Grid cols={2}>
@ -82,7 +88,7 @@ systems, or to pre-process text for **deep learning**.
</Grid>
### What spaCy isn't {#what-spacy-isnt}
### What spaCy isn't {id="what-spacy-isnt"}
- ❌ **spaCy is not a platform or "an API"**. Unlike a platform, spaCy does not
provide a software as a service, or a web application. It's an open-source
@ -103,7 +109,7 @@ systems, or to pre-process text for **deep learning**.
publishing spaCy and other software is called
[Explosion](https://explosion.ai).
## Features {#features}
## Features {id="features"}
In the documentation, you'll come across mentions of spaCy's features and
capabilities. Some of them refer to linguistic concepts, while others are
@ -124,7 +130,7 @@ related to more general machine learning functionality.
| **Training** | Updating and improving a statistical model's predictions. |
| **Serialization** | Saving objects to files or byte strings. |
### Statistical models {#statistical-models}
### Statistical models {id="statistical-models"}
While some of spaCy's features work independently, others require
[trained pipelines](/models) to be loaded, which enable spaCy to **predict**
@ -148,7 +154,7 @@ are always a good start. They typically include the following components:
and model implementations to use, to put spaCy in the correct state when you
load the pipeline.
## Linguistic annotations {#annotations}
## Linguistic annotations {id="annotations"}
spaCy provides a variety of linguistic annotations to give you **insights into a
text's grammatical structure**. This includes the word types, like the parts of
@ -159,7 +165,7 @@ the website or company in a specific context.
> #### Loading pipelines
>
> ```cli
> ```bash
> $ python -m spacy download en_core_web_sm
>
> >>> import spacy
@ -172,8 +178,7 @@ can load it via [`spacy.load`](/api/top-level#spacy.load). This will return a
usually call it `nlp`. Calling the `nlp` object on a string of text will return
a processed `Doc`:
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -189,9 +194,7 @@ original string, or reconstruct the original by joining the tokens and their
trailing whitespace. This way, you'll never lose any information when processing
text with spaCy.
### Tokenization {#annotations-token}
import Tokenization101 from 'usage/101/\_tokenization.md'
### Tokenization {id="annotations-token"}
<Tokenization101 />
@ -205,9 +208,7 @@ language-specific data**, see the usage guides on
</Infobox>
### Part-of-speech tags and dependencies {#annotations-pos-deps model="parser"}
import PosDeps101 from 'usage/101/\_pos-deps.md'
### Part-of-speech tags and dependencies {id="annotations-pos-deps",model="parser"}
<PosDeps101 />
@ -220,9 +221,7 @@ how to **navigate and use the parse tree** effectively, see the usage guides on
</Infobox>
### Named Entities {#annotations-ner model="ner"}
import NER101 from 'usage/101/\_named-entities.md'
### Named Entities {id="annotations-ner",model="ner"}
<NER101 />
@ -236,9 +235,7 @@ of a model, see the usage guides on
</Infobox>
### Word vectors and similarity {#vectors-similarity model="vectors"}
import Vectors101 from 'usage/101/\_vectors-similarity.md'
### Word vectors and similarity {id="vectors-similarity",model="vectors"}
<Vectors101 />
@ -250,9 +247,7 @@ To learn more about word vectors, how to **customize them** and how to load
</Infobox>
## Pipelines {#pipelines}
import Pipelines101 from 'usage/101/\_pipelines.md'
## Pipelines {id="pipelines"}
<Pipelines101 />
@ -264,13 +259,11 @@ guide on [language processing pipelines](/usage/processing-pipelines).
</Infobox>
## Architecture {#architecture}
import Architecture101 from 'usage/101/\_architecture.md'
## Architecture {id="architecture"}
<Architecture101 />
## Vocab, hashes and lexemes {#vocab}
## Vocab, hashes and lexemes {id="vocab"}
Whenever possible, spaCy tries to store data in a vocabulary, the
[`Vocab`](/api/vocab), that will be **shared by multiple documents**. To save
@ -288,7 +281,7 @@ and part-of-speech tags like "VERB" are also encoded. Internally, spaCy only
> - **StringStore**: The dictionary mapping hash values to strings, for example
> `3197928453018144401` → "coffee".
![Doc, Vocab, Lexeme and StringStore](../images/vocab_stringstore.svg)
![Doc, Vocab, Lexeme and StringStore](/images/vocab_stringstore.svg)
If you process lots of documents containing the word "coffee" in all kinds of
different contexts, storing the exact string "coffee" every time would take up
@ -297,8 +290,7 @@ way too much space. So instead, spaCy hashes the string and stores it in the
**lookup table that works in both directions** – you can look up a string to get
its hash, or a hash to get its string:
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -315,8 +307,7 @@ a word. For example, no matter if "love" is used as a verb or a noun in some
context, its spelling and whether it consists of alphabetic characters won't
ever change. Its hash value will also always be the same.
```python
### {executable="true"}
```python {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
@ -354,8 +345,7 @@ vocabulary. That's why you always need to make sure all objects you create have
access to the same vocabulary. If they don't, spaCy might not be able to find
the strings it needs.
```python
### {executable="true"}
```python {executable="true"}
import spacy
from spacy.tokens import Doc
from spacy.vocab import Vocab
@ -382,9 +372,7 @@ spaCy will also export the `Vocab` when you save a `Doc` or `nlp` object. This
will give you the object and its encoded annotations, plus the "key" to decode
it.
## Serialization {#serialization}
import Serialization101 from 'usage/101/\_serialization.md'
## Serialization {id="serialization"}
<Serialization101 />
@ -395,9 +383,7 @@ guide on [saving and loading](/usage/saving-loading#models).
</Infobox>
## Training {#training}
import Training101 from 'usage/101/\_training.md'
## Training {id="training"}
<Training101 />
@ -409,7 +395,7 @@ data and how to improve spaCy's named models, see the usage guides on
</Infobox>
### Training config and lifecycle {#training-config}
### Training config and lifecycle {id="training-config"}
Training config files include all **settings and hyperparameters** for training
your pipeline. Instead of providing lots of arguments on the command line, you
@ -433,7 +419,7 @@ source of truth", both at **training** and **runtime**.
> initial_rate = 0.01
> ```
![Illustration of pipeline lifecycle](../images/lifecycle.svg)
![Illustration of pipeline lifecycle](/images/lifecycle.svg)
<Infobox title="Training configuration system" emoji="📖">
@ -443,7 +429,7 @@ hyperparameters, see the [training config](/usage/training#config) usage guide.
</Infobox>
### Trainable components {#training-components}
### Trainable components {id="training-components"}
spaCy's [`Pipe`](/api/pipe) class helps you implement your own trainable
components that have their own model instance, make predictions over `Doc`
@ -462,7 +448,7 @@ configured via a single training config.
> width = 128
> ```
![Illustration of Pipe methods](../images/trainable_component.svg)
![Illustration of Pipe methods](/images/trainable_component.svg)
<Infobox title="Custom trainable components" emoji="📖">
@ -474,13 +460,11 @@ for trainable components.
</Infobox>
## Language data {#language-data}
import LanguageData101 from 'usage/101/\_language-data.md'
## Language data {id="language-data"}
<LanguageData101 />
## Community & FAQ {#community-faq}
## Community & FAQ {id="community-faq"}
We're very happy to see the spaCy community grow and include a mix of people
from all kinds of different backgrounds – computational linguistics, data
@ -488,7 +472,7 @@ science, deep learning, research and more. If you'd like to get involved, below
are some answers to the most important questions and resources for further
reading.
### Help, my code isn't working! {#faq-help-code}
### Help, my code isn't working! {id="faq-help-code"}
Bugs suck, and we're doing our best to continuously improve the tests and fix
bugs as soon as possible. Before you submit an issue, do a quick search and
@ -513,10 +497,11 @@ via the following platforms:
questions** and everything related to problems with your specific code. The
Stack Overflow community is much larger than ours, so if your problem can be
solved by others, you'll receive help much quicker.
- [GitHub discussions](https://github.com/explosion/spaCy/discussions): **General
discussion**, **project ideas** and **usage questions**. Meet other community
members to get help with a specific code implementation, discuss ideas for new
projects/plugins, support more languages, and share best practices.
- [GitHub discussions](https://github.com/explosion/spaCy/discussions):
**General discussion**, **project ideas** and **usage questions**. Meet other
community members to get help with a specific code implementation, discuss
ideas for new projects/plugins, support more languages, and share best
practices.
- [GitHub issue tracker](https://github.com/explosion/spaCy/issues): **Bug
reports** and **improvement suggestions**, i.e. everything that's likely
spaCy's fault. This also includes problems with the trained pipelines beyond
@ -533,7 +518,7 @@ and the next time you need help, they might repay the favor.
</Infobox>
### How can I contribute to spaCy? {#faq-contributing}
### How can I contribute to spaCy? {id="faq-contributing"}
You don't have to be an NLP expert or Python pro to contribute, and we're happy
to help you get started. If you're new to spaCy, a good place to start is the
@ -567,7 +552,7 @@ By participating, you are expected to uphold this code.
</Infobox>
### I've built something cool with spaCy – how can I get the word out? {#faq-project-with-spacy}
### I've built something cool with spaCy – how can I get the word out? {id="faq-project-with-spacy"}
First, congrats – we'd love to check it out! When you share your project on
Twitter, don't forget to tag [@spacy_io](https://twitter.com/spacy_io) so we
@ -588,7 +573,8 @@ project is using spaCy, you can grab one of our **spaCy badges** here:
[![Built with spaCy](https://img.shields.io/badge/built%20with-spaCy-09a3d5.svg)](https://spacy.io)
```
<img src={`https://img.shields.io/badge/made%20with%20❤%20and-spaCy-09a3d5.svg`}
<img
src={`https://img.shields.io/badge/made%20with%20❤%20and-spaCy-09a3d5.svg`}
/>
```markdown

256
website/docs/usage/training.md → website/docs/usage/training.mdx
View File

@ -15,15 +15,17 @@ menu:
- ['Internal API', 'api']
---
## Introduction to training {#basics hidden="true"}
import Training101 from 'usage/101/\_training.md'
## Introduction to training {id="basics",hidden="true"}
<Training101 />
<Infobox title="Tip: Try the Prodigy annotation tool">
[![Prodigy: Radically efficient machine teaching](../images/prodigy.jpg)](https://prodi.gy)
<Image
src="/images/prodigy.jpg"
href="https://prodi.gy"
alt="Prodigy: Radically efficient machine teaching"
/>
If you need to label a lot of data, check out [Prodigy](https://prodi.gy), a
new, active learning-powered annotation tool we've developed. Prodigy is fast
@ -34,7 +36,7 @@ ready-to-use spaCy pipelines.
</Infobox>
## Quickstart {#quickstart tag="new"}
## Quickstart {id="quickstart",tag="new"}
The recommended way to train your spaCy pipelines is via the
[`spacy train`](/api/cli#train) command on the command line. It only needs a
@ -69,8 +71,6 @@ config.
> requirements and settings as CLI arguments.
> 2. Run [`train`](/api/cli#train) with the exported config and data.
import QuickstartTraining from 'widgets/quickstart-training.js'
<QuickstartTraining />
After you've saved the starter config to a file `base_config.cfg`, you can use
@ -78,7 +78,7 @@ the [`init fill-config`](/api/cli#init-fill-config) command to fill in the
remaining defaults. Training configs should always be **complete and without
hidden defaults**, to keep your experiments reproducible.
```cli
```bash
$ python -m spacy init fill-config base_config.cfg config.cfg
```
@ -88,7 +88,7 @@ $ python -m spacy init fill-config base_config.cfg config.cfg
> your training and development data, get useful stats, and find problems like
> invalid entity annotations, cyclic dependencies, low data labels and more.
>
> ```cli
> ```bash
> $ python -m spacy debug data config.cfg
> ```
@ -100,7 +100,7 @@ add your data and run [`train`](/api/cli#train) with your config. See the
spaCy's binary `.spacy` format. You can either include the data paths in the
`[paths]` section of your config, or pass them in via the command line.
```cli
```bash
$ python -m spacy train config.cfg --output ./output --paths.train ./train.spacy --paths.dev ./dev.spacy
```
@ -108,7 +108,7 @@ $ python -m spacy train config.cfg --output ./output --paths.train ./train.spacy
>
> Use the `--gpu-id` option to select the GPU:
>
> ```cli
> ```bash
> $ python -m spacy train config.cfg --gpu-id 0
> ```
@ -141,7 +141,7 @@ treebank.
</Project>
## Training config system {#config}
## Training config system {id="config"}
Training config files include all **settings and hyperparameters** for training
your pipeline. Instead of providing lots of arguments on the command line, you
@ -212,7 +212,7 @@ available for the different architectures are documented with the
<YouTube id="BWhh3r6W-qE"></YouTube>
### Config lifecycle at runtime and training {#config-lifecycle}
### Config lifecycle at runtime and training {id="config-lifecycle"}
A pipeline's `config.cfg` is considered the "single source of truth", both at
**training** and **runtime**. Under the hood,
@ -222,7 +222,7 @@ config is available as [`nlp.config`](/api/language#config) and it includes all
information about the pipeline, as well as the settings used to train and
initialize it.
![Illustration of pipeline lifecycle](../images/lifecycle.svg)
![Illustration of pipeline lifecycle](/images/lifecycle.svg)
At runtime spaCy will only use the `[nlp]` and `[components]` blocks of the
config and load all data, including tokenization rules, model weights and other
@ -240,7 +240,7 @@ requiring it to be available at runtime. You can also use this mechanism to
provide data paths to custom pipeline components and custom tokenizers – see the
section on [custom initialization](#initialization) for details.
### Overwriting config settings on the command line {#config-overrides}
### Overwriting config settings on the command line {id="config-overrides"}
The config system means that you can define all settings **in one place** and in
a consistent format. There are no command-line arguments that need to be set,
@ -254,7 +254,7 @@ For cases like this, you can set additional command-line options starting with
`--paths.train ./corpus/train.spacy` sets the `train` value in the `[paths]`
block.
```cli
```bash
$ python -m spacy train config.cfg --paths.train ./corpus/train.spacy --paths.dev ./corpus/dev.spacy --training.batch_size 128
```
@ -271,7 +271,7 @@ reference it across your config and override it on the CLI once.
> [`spacy train`](/api/cli#train) to make spaCy log more info, including which
> overrides were set via the CLI and environment variables.
#### Adding overrides via environment variables {#config-overrides-env}
#### Adding overrides via environment variables {id="config-overrides-env"}
Instead of defining the overrides as CLI arguments, you can also use the
`SPACY_CONFIG_OVERRIDES` environment variable using the same argument syntax.
@ -279,11 +279,11 @@ This is especially useful if you're training models as part of an automated
process. Environment variables **take precedence** over CLI overrides and values
defined in the config file.
```cli
```bash
$ SPACY_CONFIG_OVERRIDES="--system.gpu_allocator pytorch --training.batch_size 128" ./your_script.sh
```
### Reading from standard input {#config-stdin}
### Reading from standard input {id="config-stdin"}
Setting the config path to `-` on the command line lets you read the config from
standard input and pipe it forward from a different process, like
@ -297,11 +297,11 @@ fly without having to save to and load from disk.
> stdout. In a custom script, you can print the string config, e.g.
> `print(nlp.config.to_str())`.
```cli
```bash
$ python -m spacy init config - --lang en --pipeline ner,textcat --optimize accuracy | python -m spacy train - --paths.train ./corpus/train.spacy --paths.dev ./corpus/dev.spacy
```
### Using variable interpolation {#config-interpolation}
### Using variable interpolation {id="config-interpolation"}
Another very useful feature of the config system is that it supports variable
interpolation for both **values and sections**. This means that you only need to
@ -310,8 +310,7 @@ define a setting once and can reference it across your config using the
the `[training]` block, and the whole block of `[training.optimizer]` is reused
in `[pretraining]` and will become `pretraining.optimizer`.
```ini
### config.cfg (excerpt) {highlight="5,18"}
```ini {title="config.cfg (excerpt)",highlight="5,18"}
[system]
seed = 0
@ -354,7 +353,7 @@ that reference this variable.
</Infobox>
## Preparing Training Data {#training-data}
## Preparing Training Data {id="training-data"}
Training data for NLP projects comes in many different formats. For some common
formats such as CoNLL, spaCy provides [converters](/api/cli#convert) you can use
@ -372,8 +371,7 @@ are handled automatically.
Here's an example of creating a `.spacy` file from some NER annotations.
```python
### preprocess.py
```python {title="preprocess.py"}
import spacy
from spacy.tokens import DocBin
@ -407,9 +405,9 @@ convert your data to JSON before creating a `.spacy` file.
</Accordion>
## Customizing the pipeline and training {#config-custom}
## Customizing the pipeline and training {id="config-custom"}
### Defining pipeline components {#config-components}
### Defining pipeline components {id="config-components"}
You typically train a [pipeline](/usage/processing-pipelines) of **one or more
components**. The `[components]` block in the config defines the available
@ -438,8 +436,7 @@ existing weights. This lets you include an already trained component in your
pipeline, or update a trained component with more data specific to your use
case.
```ini
### config.cfg (excerpt)
```ini {title="config.cfg (excerpt)"}
[components]
# "parser" and "ner" are sourced from a trained pipeline
@ -518,7 +515,7 @@ replace_listeners = ["model.tok2vec"]
</Infobox>
### Using predictions from preceding components {#annotating-components new="3.1"}
### Using predictions from preceding components {id="annotating-components",version="3.1"}
By default, components are updated in isolation during training, which means
that they don't see the predictions of any earlier components in the pipeline. A
@ -532,8 +529,7 @@ list of components. For example, the feature `DEP` from the parser could be used
as a tagger feature by including `DEP` in the tok2vec `attrs` and including
`parser` in `annotating_components`:
```ini
### config.cfg (excerpt) {highlight="7,12"}
```ini {title="config.cfg (excerpt)",highlight="7,12"}
[nlp]
pipeline = ["parser", "tagger"]
@ -555,8 +551,7 @@ pipeline is run. The config excerpt below shows how a frozen `ner` component and
a `sentencizer` can provide the required `doc.sents` and `doc.ents` for the
entity linker during training:
```ini
### config.cfg (excerpt)
```ini {title="config.cfg (excerpt)"}
[nlp]
pipeline = ["sentencizer", "ner", "entity_linker"]
@ -580,7 +575,7 @@ now-updated model to the predicted docs.
</Infobox>
### Using registered functions {#config-functions}
### Using registered functions {id="config-functions"}
The training configuration defined in the config file doesn't have to only
consist of static values. Some settings can also be **functions**. For instance,
@ -588,8 +583,7 @@ the `batch_size` can be a number that doesn't change, or a schedule, like a
sequence of compounding values, which has shown to be an effective trick (see
[Smith et al., 2017](https://arxiv.org/abs/1711.00489)).
```ini
### With static value
```ini {title="With static value"}
[training]
batch_size = 128
```
@ -612,8 +606,7 @@ from your configs.
> instance, a learning rate schedule can be provided as the an argument of an
> optimizer.
```ini
### With registered function
```ini {title="With registered function"}
[training.batch_size]
@schedules = "compounding.v1"
start = 100
@ -621,7 +614,7 @@ stop = 1000
compound = 1.001
```
### Model architectures {#model-architectures}
### Model architectures {id="model-architectures"}
> #### 💡 Model type annotations
>
@ -659,7 +652,7 @@ different tasks. For example:
| [TransitionBasedParser](/api/architectures#TransitionBasedParser) | Build a [transition-based parser](https://explosion.ai/blog/parsing-english-in-python) model used in the default [`EntityRecognizer`](/api/entityrecognizer) and [`DependencyParser`](/api/dependencyparser). ~~Model[List[Docs], List[List[Floats2d]]]~~ |
| [TextCatEnsemble](/api/architectures#TextCatEnsemble) | Stacked ensemble of a bag-of-words model and a neural network model with an internal CNN embedding layer. Used in the default [`TextCategorizer`](/api/textcategorizer). ~~Model[List[Doc], Floats2d]~~ |
### Metrics, training output and weighted scores {#metrics}
### Metrics, training output and weighted scores {id="metrics"}
When you train a pipeline using the [`spacy train`](/api/cli#train) command,
you'll see a table showing the metrics after each pass over the data. The
@ -721,7 +714,7 @@ still look good.
</Accordion>
## Custom functions {#custom-functions}
## Custom functions {id="custom-functions"}
Registered functions in the training config files can refer to built-in
implementations, but you can also plug in fully **custom implementations**. All
@ -740,14 +733,14 @@ Each custom function can have any number of arguments that are passed in via the
given parameter is always explicitly set in the config, avoid setting a default
value for it.
### Training with custom code {#custom-code}
### Training with custom code {id="custom-code"}
> ```cli
> ```bash
> ### Training
> $ python -m spacy train config.cfg --code functions.py
> ```
>
> ```cli
> ```bash
> ### Packaging
> $ python -m spacy package ./model-best ./packages --code functions.py
> ```
@ -765,7 +758,7 @@ any custom architectures, functions or
your pipeline and registered when it's loaded. See the documentation on
[saving and loading pipelines](/usage/saving-loading#models-custom) for details.
#### Example: Modifying the nlp object {#custom-code-nlp-callbacks}
#### Example: Modifying the nlp object {id="custom-code-nlp-callbacks"}
For many use cases, you don't necessarily want to implement the whole `Language`
subclass and language data from scratch – it's often enough to make a few small
@ -799,8 +792,7 @@ stop word to the defaults:
> @callbacks = "customize_language_data"
> ```
```python
### functions.py {highlight="3,6"}
```python {title="functions.py",highlight="3,6"}
import spacy
@spacy.registry.callbacks("customize_language_data")
@ -836,8 +828,7 @@ we're adding the arguments `extra_stop_words` (a list of strings) and `debug`
> debug = true
> ```
```python
### functions.py {highlight="5,7-9"}
```python {title="functions.py",highlight="5,7-9"}
from typing import List
import spacy
@ -873,11 +864,11 @@ you can now run [`spacy train`](/api/cli#train) and point the argument `--code`
to your Python file. Before loading the config, spaCy will import the
`functions.py` module and your custom functions will be registered.
```cli
```bash
$ python -m spacy train config.cfg --output ./output --code ./functions.py
```
#### Example: Modifying tokenizer settings {#custom-tokenizer}
#### Example: Modifying tokenizer settings {id="custom-tokenizer"}
Use the `initialize.before_init` callback to modify the tokenizer settings when
training a new pipeline. Write a registered callback that modifies the tokenizer
@ -892,8 +883,7 @@ settings and specify this callback in your config:
> @callbacks = "customize_tokenizer"
> ```
```python
### functions.py
```python {title="functions.py"}
from spacy.util import registry, compile_suffix_regex
@registry.callbacks("customize_tokenizer")
@ -912,7 +902,7 @@ def make_customize_tokenizer():
When training, provide the function above with the `--code` option:
```cli
```bash
$ python -m spacy train config.cfg --code ./functions.py
```
@ -936,7 +926,7 @@ disk.
</Infobox>
#### Example: Custom logging function {#custom-logging}
#### Example: Custom logging function {id="custom-logging"}
During training, the results of each step are passed to a logger function. By
default, these results are written to the console with the
@ -966,8 +956,7 @@ tabular results to a file:
> log_path = "my_file.tab"
> ```
```python
### functions.py
```python {title="functions.py"}
import sys
from typing import IO, Tuple, Callable, Dict, Any, Optional
import spacy
@ -1005,7 +994,7 @@ def custom_logger(log_path):
return setup_logger
```
#### Example: Custom batch size schedule {#custom-code-schedule}
#### Example: Custom batch size schedule {id="custom-code-schedule"}
You can also implement your own batch size schedule to use during training. The
`@spacy.registry.schedules` decorator lets you register that function in the
@ -1019,8 +1008,7 @@ You can also implement your own batch size schedule to use during training. The
> know that a config referencing `v1` means a different function than a config
> referencing `v2`.
```python
### functions.py
```python {title="functions.py"}
import spacy
@spacy.registry.schedules("my_custom_schedule.v1")
@ -1037,15 +1025,14 @@ settings in the block will be passed to the function as keyword arguments. Keep
in mind that the config shouldn't have any hidden defaults and all arguments on
the functions need to be represented in the config.
```ini
### config.cfg (excerpt)
```ini {title="config.cfg (excerpt)"}
[training.batch_size]
@schedules = "my_custom_schedule.v1"
start = 2
factor = 1.005
```
### Defining custom architectures {#custom-architectures}
### Defining custom architectures {id="custom-architectures"}
Built-in pipeline components such as the tagger or named entity recognizer are
constructed with default neural network [models](/api/architectures). You can
@ -1064,8 +1051,7 @@ for more details.
> output_width = 512
> ```
```python
### functions.py
```python {title="functions.py"}
from typing import List
from thinc.types import Floats2d
from thinc.api import Model
@ -1077,7 +1063,7 @@ def custom_neural_network(output_width: int) -> Model[List[Doc], List[Floats2d]]
return create_model(output_width)
```
## Customizing the initialization {#initialization}
## Customizing the initialization {id="initialization"}
When you start training a new model from scratch,
[`spacy train`](/api/cli#train) will call
@ -1120,7 +1106,7 @@ because the component settings required for training (load data from an external
file) wouldn't match the component settings required at runtime (load what's
included with the saved `nlp` object and don't depend on external file).
![Illustration of pipeline lifecycle](../images/lifecycle.svg)
![Illustration of pipeline lifecycle](/images/lifecycle.svg)
<Infobox title="How components save and load data" emoji="📖">
@ -1131,7 +1117,7 @@ initialization is implemented under the hood, see the usage guide on
</Infobox>
#### Initializing labels {#initialization-labels}
#### Initializing labels {id="initialization-labels"}
Built-in pipeline components like the
[`EntityRecognizer`](/api/entityrecognizer) or
@ -1157,7 +1143,7 @@ allow them to initialize faster.
> path = "corpus/labels/ner.json
> ```
```cli
```bash
$ python -m spacy init labels config.cfg ./corpus --paths.train ./corpus/train.spacy
```
@ -1174,20 +1160,20 @@ should always let spaCy **auto-generate the labels** for you.
</Infobox>
## Data utilities {#data}
## Data utilities {id="data"}
spaCy includes various features and utilities to make it easy to train models
using your own data, manage training and evaluation corpora, convert existing
annotations and configure data augmentation strategies for more robust models.
### Converting existing corpora and annotations {#data-convert}
### Converting existing corpora and annotations {id="data-convert"}
If you have training data in a standard format like `.conll` or `.conllu`, the
easiest way to convert it for use with spaCy is to run
[`spacy convert`](/api/cli#convert) and pass it a file and an output directory.
By default, the command will pick the converter based on the file extension.
```cli
```bash
$ python -m spacy convert ./train.gold.conll ./corpus
```
@ -1215,8 +1201,7 @@ especially when packing multiple documents together. You can also create `Doc`
objects manually, so you can write your own custom logic to convert and store
existing annotations for use in spaCy.
```python
### Training data from Doc objects {highlight="6-9"}
```python {title="Training data from Doc objects",highlight="6-9"}
import spacy
from spacy.tokens import Doc, DocBin
@ -1230,7 +1215,7 @@ docbin.add(doc)
docbin.to_disk("./train.spacy")
```
### Working with corpora {#data-corpora}
### Working with corpora {id="data-corpora"}
> #### Example
>
@ -1271,7 +1256,7 @@ By default, the training data is loaded into memory and shuffled before each
epoch. If the corpus is **too large to fit into memory** during training, stream
the corpus using a custom reader as described in the next section.
### Custom data reading and batching {#custom-code-readers-batchers}
### Custom data reading and batching {id="custom-code-readers-batchers"}
Some use-cases require **streaming in data** or manipulating datasets on the
fly, rather than generating all data beforehand and storing it to disk. Instead
@ -1296,8 +1281,7 @@ as **config settings** – in this case, `source`.
> source = "s3://your_bucket/path/data.csv"
> ```
```python
### functions.py {highlight="7-8"}
```python {title="functions.py",highlight="7-8"}
from typing import Callable, Iterator, List
import spacy
from spacy.training import Example
@ -1354,8 +1338,7 @@ training should stop.
> max_steps = 2000
> ```
```python
### functions.py
```python {title="functions.py"}
from typing import Callable, Iterable, Iterator
from spacy import util
import random
@ -1418,8 +1401,7 @@ annotations are the same.
> size = 150
> ```
```python
### functions.py
```python {title="functions.py"}
from typing import Callable, Iterable, Iterator, List
import spacy
from spacy.training import Example
@ -1439,12 +1421,9 @@ def filter_batch(size: int) -> Callable[[Iterable[Example]], Iterator[List[Examp
return create_filtered_batches
```
<!-- TODO:
* Custom corpus class
* Minibatching
-->
{/* TODO: Custom corpus class, Minibatching */}
### Data augmentation {#data-augmentation}
### Data augmentation {id="data-augmentation"}
Data augmentation is the process of applying small **modifications** to the
training data. It can be especially useful for punctuation and case replacement
@ -1458,8 +1437,7 @@ your config. The built-in [`orth_variants`](/api/top-level#orth_variants)
augmenter creates a data augmentation callback that uses orth-variant
replacement.
```ini
### config.cfg (excerpt) {highlight="8,14"}
```ini {title="config.cfg (excerpt)",highlight="8,14"}
[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
@ -1483,12 +1461,18 @@ typically loaded from a JSON file. There are two types of orth variant rules:
`"single"` for single tokens that should be replaced (e.g. hyphens) and
`"paired"` for pairs of tokens (e.g. quotes).
<!-- prettier-ignore -->
```json
### orth_variants.json
```json {title="orth_variants.json"}
{
"single": [{ "tags": ["NFP"], "variants": ["…", "..."] }],
"paired": [{ "tags": ["``", "''"], "variants": [["'", "'"], ["‘", "’"]] }]
"paired": [
{
"tags": ["``", "''"],
"variants": [
["'", "'"],
["‘", "’"]
]
}
]
}
```
@ -1511,7 +1495,7 @@ to apply it to the **training corpus**, not the development data.
</Infobox>
#### Writing custom data augmenters {#data-augmentation-custom}
#### Writing custom data augmenters {id="data-augmentation-custom"}
Using the [`@spacy.augmenters`](/api/top-level#registry) registry, you can also
register your own data augmentation callbacks. The callback should be a function
@ -1572,7 +1556,78 @@ token-based annotations like the dependency parse or entity labels, you'll need
to take care to adjust the `Example` object so its annotations match and remain
valid.
## Internal training API {#api}
## Parallel & distributed training with Ray {id="parallel-training"}
> #### Installation
>
> ```bash
> $ pip install -U %%SPACY_PKG_NAME[ray]%%SPACY_PKG_FLAGS
> # Check that the CLI is registered
> $ python -m spacy ray --help
> ```
[Ray](https://ray.io/) is a fast and simple framework for building and running
**distributed applications**. You can use Ray to train spaCy on one or more
remote machines, potentially speeding up your training process. Parallel
training won't always be faster though – it depends on your batch size, models,
and hardware.
<Infobox variant="warning">
To use Ray with spaCy, you need the
[`spacy-ray`](https://github.com/explosion/spacy-ray) package installed.
Installing the package will automatically add the `ray` command to the spaCy
CLI.
</Infobox>
The [`spacy ray train`](/api/cli#ray-train) command follows the same API as
[`spacy train`](/api/cli#train), with a few extra options to configure the Ray
setup. You can optionally set the `--address` option to point to your Ray
cluster. If it's not set, Ray will run locally.
```bash
python -m spacy ray train config.cfg --n-workers 2
```
<Project id="integrations/ray">
Get started with parallel training using our project template. It trains a
simple model on a Universal Dependencies Treebank and lets you parallelize the
training with Ray.
</Project>
### How parallel training works {id="parallel-training-details"}
Each worker receives a shard of the **data** and builds a copy of the **model
and optimizer** from the [`config.cfg`](#config). It also has a communication
channel to **pass gradients and parameters** to the other workers. Additionally,
each worker is given ownership of a subset of the parameter arrays. Every
parameter array is owned by exactly one worker, and the workers are given a
mapping so they know which worker owns which parameter.
![Illustration of setup](/images/spacy-ray.svg)
As training proceeds, every worker will be computing gradients for **all** of
the model parameters. When they compute gradients for parameters they don't own,
they'll **send them to the worker** that does own that parameter, along with a
version identifier so that the owner can decide whether to discard the gradient.
Workers use the gradients they receive and the ones they compute locally to
update the parameters they own, and then broadcast the updated array and a new
version ID to the other workers.
This training procedure is **asynchronous** and **non-blocking**. Workers always
push their gradient increments and parameter updates, they do not have to pull
them and block on the result, so the transfers can happen in the background,
overlapped with the actual training work. The workers also do not have to stop
and wait for each other ("synchronize") at the start of each batch. This is very
useful for spaCy, because spaCy is often trained on long documents, which means
**batches can vary in size** significantly. Uneven workloads make synchronous
gradient descent inefficient, because if one batch is slow, all of the other
workers are stuck waiting for it to complete before they can continue.
## Internal training API {id="api"}
<Infobox variant="danger">
@ -1586,7 +1641,7 @@ typically give you everything you need to train fully custom pipelines with
</Infobox>
### Training from a Python script {#api-train new="3.2"}
### Training from a Python script {id="api-train",version="3.2"}
If you want to run the training from a Python script instead of using the
[`spacy train`](/api/cli#train) CLI command, you can call into the
@ -1600,7 +1655,7 @@ from spacy.cli.train import train
train("./config.cfg", overrides={"paths.train": "./train.spacy", "paths.dev": "./dev.spacy"})
```
### Internal training loop API {#api-loop}
### Internal training loop API {id="api-loop"}
<Infobox variant="warning">
@ -1691,8 +1746,7 @@ of being dropped.
> - [`nlp.to_disk`](/api/language#to_disk): Save the updated pipeline to a
> directory.
```python
### Example training loop
```python {title="Example training loop"}
optimizer = nlp.initialize()
for itn in range(100):
random.shuffle(train_data)

24
website/docs/usage/v2-1.md → website/docs/usage/v2-1.mdx
View File

@ -6,7 +6,7 @@ menu:
- ['Backwards Incompatibilities', 'incompat']
---
## New Features {#features hidden="true"}
## New Features {id="features",hidden="true"}
spaCy v2.1 has focussed primarily on stability and performance, solidifying the
design changes introduced in [v2.0](/usage/v2). As well as smaller models,
@ -16,7 +16,7 @@ for some exciting new NLP innovations. For the full changelog, see the
For more details and a behind-the-scenes look at the new release,
[see our blog post](https://explosion.ai/blog/spacy-v2-1).
### BERT/ULMFit/Elmo-style pre-training {#pretraining tag="experimental"}
### BERT/ULMFit/Elmo-style pre-training {id="pretraining",tag="experimental"}
> #### Example
>
@ -41,7 +41,7 @@ it.
</Infobox>
### Extended match pattern API {#matcher-api}
### Extended match pattern API {id="matcher-api"}
> #### Example
>
@ -69,7 +69,7 @@ values.
</Infobox>
### Easy rule-based entity recognition {#entity-ruler}
### Easy rule-based entity recognition {id="entity-ruler"}
> #### Example
>
@ -93,7 +93,7 @@ flexibility.
</Infobox>
### Phrase matching with other attributes {#phrasematcher}
### Phrase matching with other attributes {id="phrasematcher"}
> #### Example
>
@ -117,7 +117,7 @@ or `POS` for finding sequences of the same part-of-speech tags.
</Infobox>
### Retokenizer for merging and splitting {#retokenizer}
### Retokenizer for merging and splitting {id="retokenizer"}
> #### Example
>
@ -144,7 +144,7 @@ deprecated.
</Infobox>
### Components and languages via entry points {#entry-points}
### Components and languages via entry points {id="entry-points"}
> #### Example
>
@ -171,7 +171,7 @@ is required.
</Infobox>
### Improved documentation {#docs}
### Improved documentation {id="docs"}
Although it looks pretty much the same, we've rebuilt the entire documentation
using [Gatsby](https://www.gatsbyjs.org/) and [MDX](https://mdxjs.com/). It's
@ -180,9 +180,9 @@ entirely **in Markdown**, without having to compromise on easy-to-use custom UI
components. We're hoping that the Markdown source will make it even easier to
contribute to the documentation. For more details, check out the
[styleguide](/styleguide) and
[source](https://github.com/explosion/spacy/tree/v2.x/website). While
converting the pages to Markdown, we've also fixed a bunch of typos, improved
the existing pages and added some new content:
[source](https://github.com/explosion/spacy/tree/v2.x/website). While converting
the pages to Markdown, we've also fixed a bunch of typos, improved the existing
pages and added some new content:
- **Usage Guide:** [Rule-based Matching](/usage/rule-based-matching)<br/>How to
use the `Matcher`, `PhraseMatcher` and the new `EntityRuler`, and write
@ -200,7 +200,7 @@ the existing pages and added some new content:
- **API:** [`Sentencizer`](/api/sentencizer)
- **API:** [Pipeline functions](/api/pipeline-functions)
## Backwards incompatibilities {#incompat}
## Backwards incompatibilities {id="incompat"}
<Infobox title="Important note on models" variant="warning">

17
website/docs/usage/v2-2.md → website/docs/usage/v2-2.mdx
View File

@ -7,7 +7,7 @@ menu:
- ['Migrating from v2.1', 'migrating']
---
## New Features {#features hidden="true"}
## New Features {id="features",hidden="true"}
spaCy v2.2 features improved statistical models, new pretrained models for
Norwegian and Lithuanian, better Dutch NER, as well as a new mechanism for
@ -22,7 +22,7 @@ full changelog, see the
For more details and a behind-the-scenes look at the new release,
[see our blog post](https://explosion.ai/blog/spacy-v2-2).
### Better pretrained models and more languages {#models}
### Better pretrained models and more languages {id="models"}
> #### Example
>
@ -46,7 +46,7 @@ overall. We've also added new core models for [Norwegian](/models/nb) (MIT) and
</Infobox>
### Text classification scores and CLI training {#train-textcat-cli}
### Text classification scores and CLI training {id="train-textcat-cli"}
> #### Example
>
@ -106,7 +106,7 @@ processing.
</Infobox>
### Serializable lookup tables and smaller installation {#lookups}
### Serializable lookup tables and smaller installation {id="lookups"}
> #### Example
>
@ -140,7 +140,7 @@ languages that don't yet ship with pretrained models.
</Infobox>
### CLI command to debug and validate training data {#debug-data}
### CLI command to debug and validate training data {id="debug-data"}
> #### Example
>
@ -300,7 +300,7 @@ will not be available.
</Infobox>
## Backwards incompatibilities {#incompat}
## Backwards incompatibilities {id="incompat"}
<Infobox title="Important note on models" variant="warning">
@ -379,7 +379,7 @@ check if all of your models are up to date, you can run the
list of dicts. This is mostly internals, but if your code used
`nlp.meta["sources"]`, you might have to update it.
### Migrating from spaCy 2.1 {#migrating}
### Migrating from spaCy 2.1 {id="migrating"}
#### Lemmatization data and lookup tables
@ -389,8 +389,7 @@ with only tokenizers, you now need to install that data explicitly via
setup is required – the package just needs to be installed in the same
environment as spaCy.
```python
### {highlight="3-4"}
```python {highlight="3-4"}
nlp = Turkish()
doc = nlp("Bu bir cümledir.")
# 🚨 This now requires the lookups data to be installed explicitly

12
website/docs/usage/v2-3.md → website/docs/usage/v2-3.mdx
View File

@ -7,7 +7,7 @@ menu:
- ['Migrating from v2.2', 'migrating']
---
## New Features {#features hidden="true"}
## New Features {id="features",hidden="true"}
spaCy v2.3 features new pretrained models for five languages, word vectors for
all language models, and decreased model size and loading times for models with
@ -19,7 +19,7 @@ Model packages with vectors are about **2&times** smaller on disk and load
For more details and a behind-the-scenes look at the new release,
[see our blog post](https://explosion.ai/blog/spacy-v2-3).
### Expanded model families with vectors {#models}
### Expanded model families with vectors {id="models"}
> #### Example
>
@ -45,7 +45,7 @@ and Dutch has been extended to include both UD Dutch Alpino and LassySmall.
</Infobox>
### Chinese {#chinese}
### Chinese {id="chinese"}
> #### Example
>
@ -77,7 +77,7 @@ Python 3.8.
</Infobox>
### Japanese {#japanese}
### Japanese {id="japanese"}
The updated Japanese language class switches to
[`SudachiPy`](https://github.com/WorksApplications/SudachiPy) for word
@ -102,7 +102,7 @@ installing spaCy for Japanese, which is now possible with a single command:
- [`spacy train`](/api/cli#train) on GPU restricts the CPU timing evaluation to
the first iteration
## Backwards incompatibilities {#incompat}
## Backwards incompatibilities {id="incompat"}
<Infobox title="Important note on models" variant="warning">
@ -140,7 +140,7 @@ the [`spacy validate`](/api/cli#validate) command.
spaCy's tokenization and Universal Dependencies multi-word tokens used for
contractions.
### Migrating from spaCy 2.2 {#migrating}
### Migrating from spaCy 2.2 {id="migrating"}
#### Tokenizer settings

44
website/docs/usage/v2.md → website/docs/usage/v2.mdx
View File

@ -17,7 +17,7 @@ spaCy, or just want to brush up on some NLP basics and the details of the
library, check out the [spaCy 101 guide](/usage/spacy-101) that explains the
most important concepts with examples and illustrations.
## Summary {#summary}
## Summary {id="summary"}
<Grid cols={2}>
@ -73,12 +73,12 @@ differentiable. Even if you don't have explicitly annotated data, you can update
spaCy using all the **latest deep learning tricks** like adversarial training,
noise contrastive estimation or reinforcement learning.
## New features {#features}
## New features {id="features"}
This section contains an overview of the most important **new features and
improvements**. The [API docs](/api) include additional deprecation notes.
### Convolutional neural network models {#features-models}
### Convolutional neural network models {id="features-models"}
> #### Example
>
@ -109,7 +109,7 @@ process.
</Infobox>
### Improved processing pipelines {#features-pipelines}
### Improved processing pipelines {id="features-pipelines"}
> #### Example
>
@ -130,7 +130,7 @@ write any **attributes, properties and methods** to the `Doc`, `Token` and
`Span`. You can add data, implement new features, integrate other libraries with
spaCy or plug in your own machine learning models.
![The processing pipeline](../images/pipeline.svg)
![The processing pipeline](/images/pipeline.svg)
<Infobox>
@ -143,7 +143,7 @@ spaCy or plug in your own machine learning models.
</Infobox>
### Text classification {#features-text-classification}
### Text classification {id="features-text-classification"}
> #### Example
>
@ -172,7 +172,7 @@ network to assign position-sensitive vectors to each word in the document.
</Infobox>
### Hash values instead of integer IDs {#features-hash-ids}
### Hash values instead of integer IDs {id="features-hash-ids"}
> #### Example
>
@ -201,7 +201,7 @@ available via `token.orth`.
</Infobox>
### Improved word vectors support {#features-vectors}
### Improved word vectors support {id="features-vectors"}
> #### Example
>
@ -231,7 +231,7 @@ you set the `--prune-vectors` flag. Otherwise, you can use the new
</Infobox>
### Saving, loading and serialization {#features-serializer}
### Saving, loading and serialization {id="features-serializer"}
> #### Example
>
@ -266,7 +266,7 @@ the class directly, e.g. `from spacy.lang.en import English` or use
</Infobox>
### displaCy visualizer with Jupyter support {#features-displacy}
### displaCy visualizer with Jupyter support {id="features-displacy"}
> #### Example
>
@ -291,7 +291,7 @@ notebook.
</Infobox>
### Improved language data and lazy loading {#features-language}
### Improved language data and lazy loading {id="features-language"}
Language-specific data now lives in its own submodule, `spacy.lang`. Languages
are lazy-loaded, i.e. only loaded when you import a `Language` class, or load a
@ -308,7 +308,7 @@ lookup-based lemmatization – and **many new languages**!
</Infobox>
### Revised matcher API and phrase matcher {#features-matcher}
### Revised matcher API and phrase matcher {id="features-matcher"}
> #### Example
>
@ -338,7 +338,7 @@ patterns.
</Infobox>
## Backwards incompatibilities {#incompat}
## Backwards incompatibilities {id="incompat"}
The following modules, classes and methods have changed between v1.x and v2.0.
@ -373,7 +373,7 @@ The following modules, classes and methods have changed between v1.x and v2.0.
| `Doc.read_bytes` | [`Doc.to_bytes`](/api/doc#to_bytes) [`Doc.from_bytes`](/api/doc#from_bytes) [`Doc.to_disk`](/api/doc#to_disk) [`Doc.from_disk`](/api/doc#from_disk) |
| `Token.is_ancestor_of` | [`Token.is_ancestor`](/api/token#is_ancestor) |
### Deprecated {#deprecated}
### Deprecated {id="deprecated"}
The following methods are deprecated. They can still be used, but should be
replaced.
@ -383,7 +383,7 @@ replaced.
| `Tokenizer.tokens_from_list` | [`Doc`](/api/doc) |
| `Span.sent_start` | [`Span.is_sent_start`](/api/span#is_sent_start) |
## Migrating from spaCy 1.x {#migrating}
## Migrating from spaCy 1.x {id="migrating"}
Because we'e made so many architectural changes to the library, we've tried to
**keep breaking changes to a minimum**. A lot of projects follow the philosophy
@ -403,7 +403,7 @@ v2.0.
</Infobox>
### Document processing {#migrating-document-processing}
### Document processing {id="migrating-document-processing"}
The [`Language.pipe`](/api/language#pipe) method allows spaCy to batch
documents, which brings a **significant performance advantage** in v2.0. The new
@ -421,7 +421,7 @@ To make usage easier, there's now a boolean `as_tuples` keyword argument, that
lets you pass in an iterator of `(text, context)` pairs, so you can get back an
iterator of `(doc, context)` tuples.
### Saving, loading and serialization {#migrating-saving-loading}
### Saving, loading and serialization {id="migrating-saving-loading"}
Double-check all calls to `spacy.load()` and make sure they don't use the `path`
keyword argument. If you're only loading in binary data and not a model package
@ -451,7 +451,7 @@ If you've trained models with input from v1.x, you'll need to **retrain them**
with spaCy v2.0. All previous models will not be compatible with the new
version.
### Processing pipelines and language data {#migrating-languages}
### Processing pipelines and language data {id="migrating-languages"}
If you're importing language data or `Language` classes, make sure to change
your import statements to import from `spacy.lang`. If you've added your own
@ -496,7 +496,7 @@ import and instantiate them directly – but it's more convenient to use the new
+ nlp.add_pipe(tagger, first=True)
```
### Training {#migrating-training}
### Training {id="migrating-training"}
All built-in pipeline components are now subclasses of [`Pipe`](/api/pipe),
fully trainable and serializable, and follow the same API. Instead of updating
@ -525,7 +525,7 @@ training.
+ nlp.to_disk("/model")
```
### Attaching custom data to the Doc {#migrating-doc}
### Attaching custom data to the Doc {id="migrating-doc"}
Previously, you had to create a new container in order to attach custom data to
a `Doc` object. This often required converting the `Doc` objects to and from
@ -561,7 +561,7 @@ own extension module.
+ meta = doc._.meta
```
### Strings and hash values {#migrating-strings}
### Strings and hash values {id="migrating-strings"}
The change from integer IDs to hash values may not actually affect your code
very much. However, if you're adding strings to the vocab manually, you now need
@ -577,7 +577,7 @@ be sure that the string-to-hash mapping will always match across vocabularies.
+ other_nlp.vocab.strings["coffee"] # 3197928453018144401
```
### Adding patterns and callbacks to the matcher {#migrating-matcher}
### Adding patterns and callbacks to the matcher {id="migrating-matcher"}
If you're using the matcher, you can now add patterns in one step. This should
be easy to update – simply merge the ID, callback and patterns into one call to

48
website/docs/usage/v3-1.md → website/docs/usage/v3-1.mdx
View File

@ -6,7 +6,7 @@ menu:
- ['Upgrading Notes', 'upgrading']
---
## New Features {#features hidden="true"}
## New Features {id="features",hidden="true"}
It's been great to see the adoption of the new spaCy v3, which introduced
[transformer-based](/usage/embeddings-transformers) pipelines, a new
@ -18,7 +18,7 @@ component for predicting arbitrary and potentially overlapping spans, support
for partial incorrect annotations in the entity recognizer, new trained
pipelines for Catalan and Danish, as well as many bug fixes and improvements.
### Using predicted annotations during training {#predicted-annotations-training}
### Using predicted annotations during training {id="predicted-annotations-training"}
By default, components are updated in isolation during training, which means
that they don't see the predictions of any earlier components in the pipeline.
@ -29,8 +29,7 @@ on the predicted docs during training. This makes it easy to use the predictions
of a previous component in the pipeline as features for a subsequent component,
e.g. the dependency labels in the tagger:
```ini
### config.cfg (excerpt) {highlight="7,12"}
```ini {title="config.cfg (excerpt)",highlight="7,12"}
[nlp]
pipeline = ["parser", "tagger"]
@ -52,7 +51,7 @@ as a feature for a subsequent tagger component in the pipeline.
</Project>
### SpanCategorizer for predicting arbitrary and overlapping spans {#spancategorizer tag="experimental"}
### SpanCategorizer for predicting arbitrary and overlapping spans {id="spancategorizer",tag="experimental"}
A common task in applied NLP is extracting spans of texts from documents,
including longer phrases or nested expressions. Named entity recognition isn't
@ -76,7 +75,11 @@ This project trains a span categorizer for Indonesian NER.
<Infobox title="Tip: Create data with Prodigy's new span annotation UI">
[![Prodigy: example of the new manual spans UI](../images/prodigy_spans-manual.jpg)](https://support.prodi.gy/t/3861)
<Image
src="/images/prodigy_spans-manual.jpg"
href="https://support.prodi.gy/t/3861"
alt="Prodigy: example of the new manual spans UI"
/>
The upcoming version of our annotation tool [Prodigy](https://prodi.gy)
(currently available as a [pre-release](https://support.prodi.gy/t/3861) for all
@ -86,7 +89,7 @@ for spaCy's `SpanCategorizer` component.
</Infobox>
### Update the entity recognizer with partial incorrect annotations {#negative-samples}
### Update the entity recognizer with partial incorrect annotations {id="negative-samples"}
> #### config.cfg (excerpt)
>
@ -116,9 +119,9 @@ train_doc.spans["incorrect_spans"] = [
]
```
<!-- TODO: more details and/or example project? -->
{/* TODO: more details and/or example project? */}
### New pipeline packages for Catalan and Danish {#pipeline-packages}
### New pipeline packages for Catalan and Danish {id="pipeline-packages"}
spaCy v3.1 adds 5 new pipeline packages, including a new core family for Catalan
and a new transformer-based pipeline for Danish using the
@ -140,7 +143,7 @@ your own.
| [`ca_core_news_trf`](/models/ca#ca_core_news_trf) | Catalan | 98.9 | 93.0 | 91.2 |
| [`da_core_news_trf`](/models/da#da_core_news_trf) | Danish | 98.0 | 85.0 | 82.9 |
### Resizable text classification architectures {#resizable-textcat}
### Resizable text classification architectures {id="resizable-textcat"}
Previously, the [`TextCategorizer`](/api/textcategorizer) architectures could
not be resized, meaning that you couldn't add new labels to an already trained
@ -148,18 +151,18 @@ model. In spaCy v3.1, the [TextCatCNN](/api/architectures#TextCatCNN) and
[TextCatBOW](/api/architectures#TextCatBOW) architectures are now resizable,
while ensuring that the predictions for the old labels remain the same.
### CLI command to assemble pipeline from config {#assemble}
### CLI command to assemble pipeline from config {id="assemble"}
The [`spacy assemble`](/api/cli#assemble) command lets you assemble a pipeline
from a config file without additional training. It can be especially useful for
creating a blank pipeline with a custom tokenizer, rule-based components or word
vectors.
```cli
```bash
$ python -m spacy assemble config.cfg ./output
```
### Pretty pipeline package READMEs {#package-readme}
### Pretty pipeline package READMEs {id="package-readme"}
The [`spacy package`](/api/cli#package) command now auto-generates a pretty
`README.md` based on the pipeline information defined in the `meta.json`. This
@ -167,7 +170,7 @@ includes a table with a general overview, as well as the label scheme and
accuracy figures, if available. For an example, see the
[model releases](https://github.com/explosion/spacy-models/releases).
### Support for streaming large or infinite corpora {#streaming-corpora}
### Support for streaming large or infinite corpora {id="streaming-corpora"}
> #### config.cfg (excerpt)
>
@ -193,7 +196,7 @@ available labels in every example. If necessary, you can use the
your components using a representative sample so the model can be initialized
correctly before training.
### New lemmatizers for Catalan and Italian {#pos-lemmatizers}
### New lemmatizers for Catalan and Italian {id="pos-lemmatizers"}
The trained pipelines for [Catalan](/models/ca) and [Italian](/models/it) now
include lemmatizers that use the predicted part-of-speech tags as part of the
@ -203,7 +206,7 @@ sure you have the
[`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) package
installed, which provides the relevant tables.
### Upload your pipelines to the Hugging Face Hub {#huggingface-hub}
### Upload your pipelines to the Hugging Face Hub {id="huggingface-hub"}
The [Hugging Face Hub](https://huggingface.co/) lets you upload models and share
them with others, and it now supports spaCy pipelines out-of-the-box. The new
@ -218,7 +221,7 @@ details, files and interactive visualizers, as well as a direct URL to the wheel
file that you can install via `pip install`. For examples, check out the
[spaCy pipelines](https://huggingface.co/spacy) we've uploaded.
```cli
```bash
$ pip install spacy-huggingface-hub
$ huggingface-cli login
$ python -m spacy package ./en_ner_fashion ./output --build wheel
@ -238,9 +241,9 @@ packaged model has changed. This makes it easy to deploy your models end-to-end.
</Project>
## Notes about upgrading from v3.0 {#upgrading}
## Notes about upgrading from v3.0 {id="upgrading"}
### Pipeline package version compatibility {#version-compat}
### Pipeline package version compatibility {id="version-compat"}
> #### Using legacy implementations
>
@ -280,7 +283,7 @@ In many cases (`spacy train`, `spacy.load()`), the new defaults will be filled
in automatically, but you'll need to fill in the new settings to run
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
### Sourcing pipeline components with vectors {#source-vectors}
### Sourcing pipeline components with vectors {id="source-vectors"}
If you're sourcing a pipeline component that requires static vectors (for
example, a tagger or parser from an `md` or `lg` pretrained pipeline), be sure
@ -289,8 +292,7 @@ spaCy v3.0, a bug allowed vectors to be loaded implicitly through `source`,
however in v3.1 this setting must be provided explicitly as
`[initialize.vectors]`:
```ini
### config.cfg (excerpt)
```ini {title="config.cfg (excerpt)"}
[components.ner]
source = "en_core_web_md"
@ -312,7 +314,7 @@ vectors. If you are sourcing a rule-based component like an entity ruler or
lemmatizer that does not use the vectors as a model feature, then this warning
can be safely ignored.
### Warnings {#warnings}
### Warnings {id="warnings"}
Logger warnings have been converted to Python warnings. Use
[`warnings.filterwarnings`](https://docs.python.org/3/library/warnings.html#warnings.filterwarnings)

32
website/docs/usage/v3-2.md → website/docs/usage/v3-2.mdx
View File

@ -6,7 +6,7 @@ menu:
- ['Upgrading Notes', 'upgrading']
---
## New Features {#features hidden="true"}
## New Features {id="features",hidden="true"}
spaCy v3.2 adds support for [`floret`](https://github.com/explosion/floret)
vectors, makes custom `Doc` creation and scoring easier, and includes many bug
@ -26,31 +26,29 @@ $ pip install spacy[apple]
</Infobox>
### Registered scoring functions {#registered-scoring-functions}
### Registered scoring functions {id="registered-scoring-functions"}
To customize the scoring, you can specify a scoring function for each component
in your config from the new [`scorers` registry](/api/top-level#registry):
```ini
### config.cfg (excerpt) {highlight="3"}
```ini {title="config.cfg (excerpt)",highlight="3"}
[components.tagger]
factory = "tagger"
scorer = {"@scorers":"spacy.tagger_scorer.v1"}
```
### Overwrite settings {#overwrite}
### Overwrite settings {id="overwrite"}
Most pipeline components now include an `overwrite` setting in the config that
determines whether existing annotation in the `Doc` is preserved or overwritten:
```ini
### config.cfg (excerpt) {highlight="3"}
```ini {title="config.cfg (excerpt)",highlight="3"}
[components.tagger]
factory = "tagger"
overwrite = false
```
### Doc input for pipelines {#doc-input}
### Doc input for pipelines {id="doc-input"}
[`nlp`](/api/language#call) and [`nlp.pipe`](/api/language#pipe) accept
[`Doc`](/api/doc) input, skipping the tokenizer if a `Doc` is provided instead
@ -63,13 +61,13 @@ doc._.text_id = 500
doc = nlp(doc)
```
### Support for floret vectors {#vectors}
### Support for floret vectors {id="vectors"}
We recently published [`floret`](https://github.com/explosion/floret), an
extended version of [fastText](https://fasttext.cc) that combines fastText's
subwords with Bloom embeddings for compact, full-coverage vectors. The use of
subwords means that there are no OOV words and due to Bloom embeddings, the
vector table can be kept very small at <100K entries. Bloom embeddings are
vector table can be kept very small at \<100K entries. Bloom embeddings are
already used by [HashEmbed](https://thinc.ai/docs/api-layers#hashembed) in
[tok2vec](/api/architectures#tok2vec-arch) for compact spaCy models.
@ -127,7 +125,7 @@ For the default project settings with 1M (3.3G) tokenized training texts and 50K
</Project>
### Updates for spacy-transformers v1.1 {#spacy-transformers}
### Updates for spacy-transformers v1.1 {id="spacy-transformers"}
[`spacy-transformers`](https://github.com/explosion/spacy-transformers) v1.1 has
been refactored to improve serialization and support of inline transformer
@ -147,7 +145,7 @@ such as `output_attentions`. Additional output is stored under
has been improved by streamlining allocations for tokenizer output and there is
new support for [mixed-precision training](/api/architectures#TransformerModel).
### New transformer package for Japanese {#pipeline-packages}
### New transformer package for Japanese {id="pipeline-packages"}
spaCy v3.2 adds a new transformer pipeline package for Japanese
[`ja_core_news_trf`](/models/ja#ja_core_news_trf), which uses the `basic`
@ -155,7 +153,7 @@ pretokenizer instead of `mecab` to limit the number of dependencies required for
the pipeline. Thanks to Hiroshi Matsuda and the spaCy Japanese community for
their contributions!
### Pipeline and language updates {#pipeline-updates}
### Pipeline and language updates {id="pipeline-updates"}
- All Universal Dependencies training data has been updated to v2.8.
- The Catalan data, tokenizer and lemmatizer have been updated, thanks to Carlos
@ -173,9 +171,9 @@ spaCy v3.2 also features a new Irish lemmatizer, support for `noun_chunks` in
Portuguese, improved `noun_chunks` for Spanish and additional updates for
Bulgarian, Catalan, Sinhala, Tagalog, Tigrinya and Vietnamese.
## Notes about upgrading from v3.1 {#upgrading}
## Notes about upgrading from v3.1 {id="upgrading"}
### Pipeline package version compatibility {#version-compat}
### Pipeline package version compatibility {id="version-compat"}
> #### Using legacy implementations
>
@ -207,7 +205,7 @@ working as expected, you can update the spaCy version requirements in the
To update a config from spaCy v3.1 with the new v3.2 settings, run
[`init fill-config`](/api/cli#init-fill-config):
```cli
```bash
$ python -m spacy init fill-config config-v3.1.cfg config-v3.2.cfg
```
@ -216,7 +214,7 @@ In many cases ([`spacy train`](/api/cli#train),
automatically, but you'll need to fill in the new settings to run
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
## Notes about upgrading from spacy-transformers v1.0 {#upgrading-transformers}
## Notes about upgrading from spacy-transformers v1.0 {id="upgrading-transformers"}
When you're loading a transformer pipeline package trained with
[`spacy-transformers`](https://github.com/explosion/spacy-transformers) v1.0

28
website/docs/usage/v3-3.md → website/docs/usage/v3-3.mdx
View File

@ -6,12 +6,12 @@ menu:
- ['Upgrading Notes', 'upgrading']
---
## New features {#features hidden="true"}
## New features {id="features",hidden="true"}
spaCy v3.3 improves the speed of core pipeline components, adds a new trainable
lemmatizer, and introduces trained pipelines for Finnish, Korean and Swedish.
### Speed improvements {#speed}
### Speed improvements {id="speed"}
v3.3 includes a slew of speed improvements:
@ -50,14 +50,14 @@ sizes on Linux (Intel Xeon W-2265) and OS X (M1) to compare spaCy v3.2 vs. v3.3:
| (+v3.3 trainable lemmatizer) | 1000 | 20047 | 20628 | 2.90% |
| | 10000 | 15921 | 18546 | 16.49% |
### Trainable lemmatizer {#trainable-lemmatizer}
### Trainable lemmatizer {id="trainable-lemmatizer"}
The new [trainable lemmatizer](/api/edittreelemmatizer) component uses
[edit trees](https://explosion.ai/blog/edit-tree-lemmatizer) to transform tokens
into lemmas. Try out the trainable lemmatizer with the
[training quickstart](/usage/training#quickstart)!
### displaCy support for overlapping spans and arcs {#displacy}
### displaCy support for overlapping spans and arcs {id="displacy"}
displaCy now supports overlapping spans with a new
[`span`](/usage/visualizers#span) style and multiple arcs with different labels
@ -77,9 +77,11 @@ doc.spans["custom"] = [Span(doc, 3, 6, "ORG"), Span(doc, 5, 6, "GPE")]
displacy.serve(doc, style="span", options={"spans_key": "custom"})
```
import DisplacySpanHtml from 'images/displacy-span.html'
<Iframe title="displaCy visualizer for overlapping spans" html={DisplacySpanHtml} height={180} />
<Iframe
title="displaCy visualizer for overlapping spans"
src="/images/displacy-span.html"
height={180}
/>
## Additional features and improvements
@ -94,9 +96,9 @@ import DisplacySpanHtml from 'images/displacy-span.html'
Russian, Slovenian, Spanish, Turkish, Ukrainian and Vietnamese.
- New noun chunks for Finnish.
## Trained pipelines {#pipelines}
## Trained pipelines {id="pipelines"}
### New trained pipelines {#new-pipelines}
### New trained pipelines {id="new-pipelines"}
v3.3 introduces new CPU/CNN pipelines for Finnish, Korean and Swedish, which use
the new trainable lemmatizer and
@ -116,7 +118,7 @@ pipelines have compact vectors with no out-of-vocabulary words.
| [`sv_core_news_md`](/models/sv#sv_core_news_md) | Swedish | 96.3 | 78.5 | 79.3 |
| [`sv_core_news_lg`](/models/sv#sv_core_news_lg) | Swedish | 96.3 | 79.1 | 81.1 |
### Pipeline updates {#pipeline-updates}
### Pipeline updates {id="pipeline-updates"}
The following languages switch from lookup or rule-based lemmatizers to the new
trainable lemmatizer: Danish, Dutch, German, Greek, Italian, Lithuanian,
@ -149,7 +151,7 @@ continue using the previous lemmatizer, you can
In addition, the vectors in the English pipelines are deduplicated to improve
the pruned vectors in the `md` models and reduce the `lg` model size.
## Notes about upgrading from v3.2 {#upgrading}
## Notes about upgrading from v3.2 {id="upgrading"}
### Span comparisons
@ -199,7 +201,7 @@ word list similar to the list of in-vocabulary vector keys with default vectors.
- `Vectors.most_similar` is not supported because there's no fixed list of
vectors to compare your vectors to.
### Pipeline package version compatibility {#version-compat}
### Pipeline package version compatibility {id="version-compat"}
> #### Using legacy implementations
>
@ -233,7 +235,7 @@ working as expected, you can update the spaCy version requirements in the
To update a config from spaCy v3.2 with the new v3.3 settings, run
[`init fill-config`](/api/cli#init-fill-config):
```cli
```bash
$ python -m spacy init fill-config config-v3.2.cfg config-v3.3.cfg
```

22
website/docs/usage/v3-4.md → website/docs/usage/v3-4.mdx
View File

@ -6,19 +6,19 @@ menu:
- ['Upgrading Notes', 'upgrading']
---
## New features {#features hidden="true"}
## New features {id="features",hidden="true"}
spaCy v3.4 brings typing and speed improvements along with new vectors for
English CNN pipelines and new trained pipelines for Croatian. This release also
includes prebuilt linux aarch64 wheels for all spaCy dependencies distributed by
Explosion.
### Typing improvements {#typing}
### Typing improvements {id="typing"}
spaCy v3.4 supports pydantic v1.9 and mypy 0.950+ through extensive updates to
types in Thinc v8.1.
### Speed improvements {#speed}
### Speed improvements {id="speed"}
- For the parser, use C `saxpy`/`sgemm` provided by the `Ops` implementation in
order to use Accelerate through `thinc-apple-ops`.
@ -42,9 +42,9 @@ types in Thinc v8.1.
- Support env variable `SPACY_NUM_BUILD_JOBS` to specify the number of build
jobs to run in parallel with `pip`.
## Trained pipelines {#pipelines}
## Trained pipelines {id="pipelines"}
### New trained pipelines {#new-pipelines}
### New trained pipelines {id="new-pipelines"}
v3.4 introduces new CPU/CNN pipelines for Croatian, which use the trainable
lemmatizer and [floret vectors](https://github.com/explosion/floret). Due to the
@ -57,20 +57,20 @@ subwords, the pipelines have compact vectors with no out-of-vocabulary words.
| [`hr_core_news_md`](/models/hr#hr_core_news_md) | 97.3 | 80.1 | 81.8 |
| [`hr_core_news_lg`](/models/hr#hr_core_news_lg) | 97.5 | 80.4 | 83.0 |
### Pipeline updates {#pipeline-updates}
### Pipeline updates {id="pipeline-updates"}
All CNN pipelines have been extended with whitespace augmentation.
The English CNN pipelines have new word vectors:
| Package | Model Version | TAG | Parser LAS | NER F |
| ----------------------------------------------- | ------------- | ---: | ---------: | ----: |
| Package | Model Version | TAG | Parser LAS | NER F |
| --------------------------------------------- | ------------- | ---: | ---------: | ----: |
| [`en_core_web_md`](/models/en#en_core_web_md) | v3.3.0 | 97.3 | 90.1 | 84.6 |
| [`en_core_web_md`](/models/en#en_core_web_md) | v3.4.0 | 97.2 | 90.3 | 85.5 |
| [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.3.0 | 97.4 | 90.1 | 85.3 |
| [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.4.0 | 97.3 | 90.2 | 85.6 |
## Notes about upgrading from v3.3 {#upgrading}
## Notes about upgrading from v3.3 {id="upgrading"}
### Doc.has_vector
@ -99,7 +99,7 @@ word list similar to the list of in-vocabulary vector keys with default vectors.
- `Vectors.most_similar` is not supported because there's no fixed list of
vectors to compare your vectors to.
### Pipeline package version compatibility {#version-compat}
### Pipeline package version compatibility {id="version-compat"}
> #### Using legacy implementations
>
@ -133,7 +133,7 @@ working as expected, you can update the spaCy version requirements in the
To update a config from spaCy v3.3 with the new v3.4 settings, run
[`init fill-config`](/api/cli#init-fill-config):
```cli
```bash
$ python -m spacy init fill-config config-v3.3.cfg config-v3.4.cfg
```

Some files were not shown because too many files have changed in this diff Show More