657af5f91f
* 🚨 Ignore all existing Mypy errors * 🏗 Add Mypy check to CI * Add types-mock and types-requests as dev requirements * Add additional type ignore directives * Add types packages to dev-only list in reqs test * Add types-dataclasses for python 3.6 * Add ignore to pretrain * 🏷 Improve type annotation on `run_command` helper The `run_command` helper previously declared that it returned an `Optional[subprocess.CompletedProcess]`, but it isn't actually possible for the function to return `None`. These changes modify the type annotation of the `run_command` helper and remove all now-unnecessary `# type: ignore` directives. * 🔧 Allow variable type redefinition in limited contexts These changes modify how Mypy is configured to allow variables to have their type automatically redefined under certain conditions. The Mypy documentation contains the following example: ```python def process(items: List[str]) -> None: # 'items' has type List[str] items = [item.split() for item in items] # 'items' now has type List[List[str]] ... ``` This configuration change is especially helpful in reducing the number of `# type: ignore` directives needed to handle the common pattern of: * Accepting a filepath as a string * Overwriting the variable using `filepath = ensure_path(filepath)` These changes enable redefinition and remove all `# type: ignore` directives rendered redundant by this change. * 🏷 Add type annotation to converters mapping * 🚨 Fix Mypy error in convert CLI argument verification * 🏷 Improve type annotation on `resolve_dot_names` helper * 🏷 Add type annotations for `Vocab` attributes `strings` and `vectors` * 🏷 Add type annotations for more `Vocab` attributes * 🏷 Add loose type annotation for gold data compilation * 🏷 Improve `_format_labels` type annotation * 🏷 Fix `get_lang_class` type annotation * 🏷 Loosen return type of `Language.evaluate` * 🏷 Don't accept `Scorer` in `handle_scores_per_type` * 🏷 Add `string_to_list` overloads * 🏷 Fix non-Optional command-line options * 🙈 Ignore redefinition of `wandb_logger` in `loggers.py` * ➕ Install `typing_extensions` in Python 3.8+ The `typing_extensions` package states that it should be used when "writing code that must be compatible with multiple Python versions". Since SpaCy needs to support multiple Python versions, it should be used when newer `typing` module members are required. One example of this is `Literal`, which is available starting with Python 3.8. Previously SpaCy tried to import `Literal` from `typing`, falling back to `typing_extensions` if the import failed. However, Mypy doesn't seem to be able to understand what `Literal` means when the initial import means. Therefore, these changes modify how `compat` imports `Literal` by always importing it from `typing_extensions`. These changes also modify how `typing_extensions` is installed, so that it is a requirement for all Python versions, including those greater than or equal to 3.8. * 🏷 Improve type annotation for `Language.pipe` These changes add a missing overload variant to the type signature of `Language.pipe`. Additionally, the type signature is enhanced to allow type checkers to differentiate between the two overload variants based on the `as_tuple` parameter. Fixes #8772 * ➖ Don't install `typing-extensions` in Python 3.8+ After more detailed analysis of how to implement Python version-specific type annotations using SpaCy, it has been determined that by branching on a comparison against `sys.version_info` can be statically analyzed by Mypy well enough to enable us to conditionally use `typing_extensions.Literal`. This means that we no longer need to install `typing_extensions` for Python versions greater than or equal to 3.8! 🎉 These changes revert previous changes installing `typing-extensions` regardless of Python version and modify how we import the `Literal` type to ensure that Mypy treats it properly. * resolve mypy errors for Strict pydantic types * refactor code to avoid missing return statement * fix types of convert CLI command * avoid list-set confustion in debug_data * fix typo and formatting * small fixes to avoid type ignores * fix types in profile CLI command and make it more efficient * type fixes in projects CLI * put one ignore back * type fixes for render * fix render types - the sequel * fix BaseDefault in language definitions * fix type of noun_chunks iterator - yields tuple instead of span * fix types in language-specific modules * 🏷 Expand accepted inputs of `get_string_id` `get_string_id` accepts either a string (in which case it returns its ID) or an ID (in which case it immediately returns the ID). These changes extend the type annotation of `get_string_id` to indicate that it can accept either strings or IDs. * 🏷 Handle override types in `combine_score_weights` The `combine_score_weights` function allows users to pass an `overrides` mapping to override data extracted from the `weights` argument. Since it allows `Optional` dictionary values, the return value may also include `Optional` dictionary values. These changes update the type annotations for `combine_score_weights` to reflect this fact. * 🏷 Fix tokenizer serialization method signatures in `DummyTokenizer` * 🏷 Fix redefinition of `wandb_logger` These changes fix the redefinition of `wandb_logger` by giving a separate name to each `WandbLogger` version. For backwards-compatibility, `spacy.train` still exports `wandb_logger_v3` as `wandb_logger` for now. * more fixes for typing in language * type fixes in model definitions * 🏷 Annotate `_RandomWords.probs` as `NDArray` * 🏷 Annotate `tok2vec` layers to help Mypy * 🐛 Fix `_RandomWords.probs` type annotations for Python 3.6 Also remove an import that I forgot to move to the top of the module 😅 * more fixes for matchers and other pipeline components * quick fix for entity linker * fixing types for spancat, textcat, etc * bugfix for tok2vec * type annotations for scorer * add runtime_checkable for Protocol * type and import fixes in tests * mypy fixes for training utilities * few fixes in util * fix import * 🐵 Remove unused `# type: ignore` directives * 🏷 Annotate `Language._components` * 🏷 Annotate `spacy.pipeline.Pipe` * add doc as property to span.pyi * small fixes and cleanup * explicit type annotations instead of via comment Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com> Co-authored-by: svlandeg <sofie.vanlandeghem@gmail.com> Co-authored-by: svlandeg <svlandeg@github.com> |
||
---|---|---|
.. | ||
docs | ||
meta | ||
setup | ||
src | ||
.eslintrc | ||
.prettierrc | ||
gatsby-browser.js | ||
gatsby-config.js | ||
gatsby-node.js | ||
package-lock.json | ||
package.json | ||
README.md | ||
runtime.txt | ||
UNIVERSE.md |
spacy.io website and docs
This page contains the documentation and styleguide for the spaCy website. Its rendered version is available at https://spacy.io/styleguide.
The spacy.io website is implemented using Gatsby with Remark and MDX. This allows authoring content in straightforward Markdown without the usual limitations. Standard elements can be overwritten with powerful React components and wherever Markdown syntax isn't enough, JSX components can be used.
Contributing to the site
The docs can always use another example or more detail, and they should always be up to date and not misleading. We always appreciate a pull request. To quickly find the correct file to edit, simply click on the "Suggest edits" button at the bottom of a page.
For more details on editing the site locally, see the installation instructions and markdown reference below.
Logo
import { Logos } from 'widgets/styleguide'
If you would like to use the spaCy logo on your site, please get in touch and ask us first. However, if you want to show support and tell others that your project is using spaCy, you can grab one of our spaCy badges.
Colors
import { Colors, Patterns } from 'widgets/styleguide'
Patterns
Typography
import { H1, H2, H3, H4, H5, Label, InlineList, Comment } from 'components/typography'
Markdown
## Headline 2 ## Headline 2 {#some_id} ## Headline 2 {#some_id tag="method"}
JSX
<H2>Headline 2</H2> <H2 id="some_id">Headline 2</H2> <H2 id="some_id" tag="method">Headline 2</H2>
Headlines are set in HK Grotesk by Hanken Design. All other body text and code uses the best-matching default system font to provide a "native" reading experience. All code uses the JetBrains Mono typeface by JetBrains.
Level 2 headings are automatically wrapped in <section>
elements at compile
time, using a custom
Markdown transformer.
This makes it easier to highlight the section that's currently in the viewpoint
in the sidebar menu.
Headline 1
Headline 2
Headline 3
Headline 4
Headline 5
LabelThe following optional attributes can be set on the headline to modify it. For
example, to add a tag for the documented type or mark features that have been
introduced in a specific version or require statistical models to be loaded.
Tags are also available as standalone <Tag />
components.
Argument | Example | Result |
---|---|---|
tag |
{tag="method"} |
method |
new |
{new="3"} |
3 |
model |
{model="tagger, parser"} |
tagger, parser |
hidden |
{hidden="true"} |
Elements
Links
Markdown
[I am a link](https://spacy.io)
JSX
<Link to="https://spacy.io">I am a link</Link>
Special link styles are used depending on the link URL.
- I am a regular external link
- I am a link to the documentation
- I am a link to an architecture
- I am a link to a model
- I am a link to GitHub
Abbreviations
import { Abbr } from 'components/typography'
JSX
<Abbr title="Explanation">Abbreviation</Abbr>
Some text with an abbreviation. On small screens, I collapse and the explanation text is displayed next to the abbreviation.
Tags
import Tag from 'components/tag'
<Tag>method</Tag> <Tag variant="new">2.1</Tag> <Tag variant="model">tagger, parser</Tag>
Tags can be used together with headlines, or next to properties across the
documentation, and combined with tooltips to provide additional information. An
optional variant
argument can be used for special tags. variant="new"
makes
the tag take a version number to mark new features. Using the component,
visibility of this tag can later be toggled once the feature isn't considered
new anymore. Setting variant="model"
takes a description of model capabilities
and can be used to mark features that require a respective model to be
installed.
method 2 tagger, parser
Buttons
import Button from 'components/button'
<Button to="#" variant="primary">Primary small</Button> <Button to="#" variant="secondary">Secondary small</Button>
Link buttons come in two variants, primary
and secondary
and two sizes, with
an optional large
size modifier. Since they're mostly used as enhanced links,
the buttons are implemented as styled links instead of native button elements.
Primary small Secondary small
Primary large Secondary large
Components
Table
Markdown
| Header 1 | Header 2 | | -------- | -------- | | Column 1 | Column 2 |
JSX
<Table> <Tr><Th>Header 1</Th><Th>Header 2</Th></Tr></thead> <Tr><Td>Column 1</Td><Td>Column 2</Td></Tr> </Table>
Tables are used to present data and API documentation. Certain keywords can be used to mark a footer row with a distinct style, for example to visualize the return values of a documented function.
Header 1 | Header 2 | Header 3 | Header 4 |
---|---|---|---|
Column 1 | Column 2 | Column 3 | Column 4 |
Column 1 | Column 2 | Column 3 | Column 4 |
Column 1 | Column 2 | Column 3 | Column 4 |
Column 1 | Column 2 | Column 3 | Column 4 |
RETURNS | Column 2 | Column 3 | Column 4 |
Tables also support optional "divider" rows that are typically used to denote keyword-only arguments in API documentation. To turn a row into a dividing headline, it should only include content in its first cell, and its value should be italicized:
Markdown
| Header 1 | Header 2 | Header 3 | | -------- | -------- | -------- | | Column 1 | Column 2 | Column 3 | | _Hello_ | | | | Column 1 | Column 2 | Column 3 |
Header 1 | Header 2 | Header 3 |
---|---|---|
Column 1 | Column 2 | Column 3 |
Hello | ||
Column 1 | Column 2 | Column 3 |
Type Annotations
Markdown
~~Model[List[Doc], Floats2d]~~
JSX
<TypeAnnotation>Model[List[Doc], Floats2d]</Typeannotation>
Type annotations are special inline code blocks are used to describe Python
types in the type hints format.
The special component will split the type, apply syntax highlighting and link
all types that specify links in meta/type-annotations.json
. Types can link to
internal or external documentation pages. To make it easy to represent the type
annotations in Markdown, the rendering "hijacks" the ~~
tags that would
typically be converted to a <del>
element – but in this case, text surrounded
by ~~
becomes a type annotation.
Dict[str, List[Union[Doc, Span]]]Model[List[Doc], List[numpy.ndarray]]
Type annotations support a special visual style in tables and will render as a separate row, under the cell text. This allows the API docs to display complex types without taking up too much space in the cell. The type annotation should always be the last element in the row.
Markdown
| Header 1 | Header 2 | | -------- | ----------------------- | | Column 1 | Column 2 ~~List[Doc]~~ |
Name | Description |
---|---|
vocab |
The shared vocabulary. |
model |
The Thinc Model wrapping the transformer. |
set_extra_annotations |
Function that takes a batch of Doc objects and transformer outputs and can set additional annotations on the Doc . |
List
Markdown
1. One 2. Two
JSX
<Ol> <Li>One</Li> <Li>Two</Li> </Ol>
Lists are available as bulleted and numbered. Markdown lists are transformed automatically.
- I am a bulleted list
- I have nice bullets
- Lorem ipsum dolor
- consectetur adipiscing elit
- I am an ordered list
- I have nice numbers
- Lorem ipsum dolor
- consectetur adipiscing elit
Aside
Markdown
> #### Aside title > This is aside text.
JSX
<Aside title="Aside title">This is aside text.</Aside>
Asides can be used to display additional notes and content in the right-hand column. Asides can contain text, code and other elements if needed. Visually, asides are moved to the side on the X-axis, and displayed at the same level they were inserted. On small screens, they collapse and are rendered in their original position, in between the text.
To make them easier to use in Markdown, paragraphs formatted as blockquotes will
turn into asides by default. Level 4 headlines (with a leading ####
) will
become aside titles.
Code Block
Markdown
```python ### This is a title import spacy ```
JSX
<CodeBlock title="This is a title" lang="python"> import spacy </CodeBlock>
Code blocks use the Prism syntax highlighter with a
custom theme. The language can be set individually on each block, and defaults
to raw text with no highlighting. An optional label can be added as the first
line with the prefix ####
(Python-like) and ///
(JavaScript-like). the
indented block as plain text and preserve whitespace.
### Using spaCy
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a sentence.")
for token in doc:
print(token.text, token.pos_)
Code blocks and also specify an optional range of line numbers to highlight by
adding {highlight="..."}
to the headline. Acceptable ranges are spans like
5-7
, but also 5-7,10
or 5-7,10,13-14
.
Markdown
```python ### This is a title {highlight="1-2"} import spacy nlp = spacy.load("en_core_web_sm") ```
### Using the matcher {highlight="5-7"}
import spacy
from spacy.matcher import Matcher
nlp = spacy.load('en_core_web_sm')
matcher = Matcher(nlp.vocab)
pattern = [{"LOWER": "hello"}, {"IS_PUNCT": True}, {"LOWER": "world"}]
matcher.add("HelloWorld", None, pattern)
doc = nlp("Hello, world! Hello world!")
matches = matcher(doc)
Adding {executable="true"}
to the title turns the code into an executable
block, powered by Binder and
Juniper. If JavaScript is disabled, the
interactive widget defaults to a regular code block.
Markdown
```python ### {executable="true"} import spacy nlp = spacy.load("en_core_web_sm") ```
### {executable="true"}
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a sentence.")
for token in doc:
print(token.text, token.pos_)
If a code block only contains a URL to a GitHub file, the raw file contents are embedded automatically and syntax highlighting is applied. The link to the original file is shown at the top of the widget.
Markdown
```python https://github.com/... ```
JSX
<GitHubCode url="https://github.com/..." lang="python" />
https://github.com/explosion/spaCy/tree/master/spacy/language.py
Infobox
import Infobox from 'components/infobox'
JSX
<Infobox title="Information">Regular infobox</Infobox> <Infobox title="Important note" variant="warning">This is a warning.</Infobox> <Infobox title="Be careful!" variant="danger">This is dangerous.</Infobox>
Infoboxes can be used to add notes, updates, warnings or additional information
to a page or section. Semantically, they're implemented and interpreted as an
aside
element. Infoboxes can take an optional title
argument, as well as an
optional variant
(either "warning"
or "danger"
).
If needed, an infobox can contain regular text, inline code
, lists and other
blocks.
If needed, an infobox can contain regular text, inline code
, lists and other
blocks.
If needed, an infobox can contain regular text, inline code
, lists and other
blocks.
Accordion
import Accordion from 'components/accordion'
JSX
<Accordion title="This is an accordion"> Accordion content goes here. </Accordion>
Accordions are collapsible sections that are mostly used for lengthy tables, like the tag and label annotation schemes for different languages. They all need to be presented – but chances are the user doesn't actually care about all of them, especially not at the same time. So it's fairly reasonable to hide them begin a click. This particular implementation was inspired by the amazing Inclusive Components blog.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque enim ante, pretium a orci eget, varius dignissim augue. Nam eu dictum mauris, id tincidunt nisi. Integer commodo pellentesque tincidunt. Nam at turpis finibus tortor gravida sodales tincidunt sit amet est. Nullam euismod arcu in tortor auctor, sit amet dignissim justo congue.
Setup and installation
Before running the setup, make sure your versions of Node and npm are up to date. Node v10.15 or later is required.
# Clone the repository
git clone https://github.com/explosion/spaCy
cd spaCy/website
# Install Gatsby's command-line tool
npm install --global gatsby-cli
# Install the dependencies
npm install
# Start the development server
npm run dev
If you are planning on making edits to the site, you should also set up the
Prettier code formatter. It takes care of formatting
Markdown and other files automatically.
See here for the available
extensions for your code editor. The
.prettierrc
file in the root defines the settings used in this codebase.
Markdown reference
All page content and page meta lives in the .md
files in the /docs
directory. The frontmatter block at the top of each file defines the page title
and other settings like the sidebar menu.
---
title: Page title
---
## Headline starting a section {#some_id}
This is a regular paragraph with a [link](https://spacy.io) and **bold text**.
> #### This is an aside title
>
> This is aside text.
### Subheadline
| Header 1 | Header 2 |
| -------- | -------- |
| Column 1 | Column 2 |
```python
### Code block title {highlight="2-3"}
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Hello world")
```
<Infobox title="Important note" variant="warning">
This is content in the infobox.
</Infobox>
In addition to the native markdown elements, you can use the components
<Infobox />
, <Accordion />
, <Abbr />
and
<Tag />
via their JSX syntax.
Project structure
### Directory structure
├── docs # the actual markdown content
├── meta # JSON-formatted site metadata
| ├── languages.json # supported languages and statistical models
| ├── sidebars.json # sidebar navigations for different sections
| ├── site.json # general site metadata
| └── universe.json # data for the spaCy universe section
├── public # compiled site
├── src # source
| ├── components # React components
| ├── fonts # webfonts
| ├── images # images used in the layout
| ├── plugins # custom plugins to transform Markdown
| ├── styles # CSS modules and global styles
| ├── templates # page layouts
| | ├── docs.js # layout template for documentation pages
| | ├── index.js # global layout template
| | ├── models.js # layout template for model pages
| | └── universe.js # layout templates for universe
| └── widgets # non-reusable components with content, e.g. changelog
├── gatsby-browser.js # browser-specific hooks for Gatsby
├── gatsby-config.js # Gatsby configuration
├── gatsby-node.js # Node-specific hooks for Gatsby
└── package.json # package settings and dependencies
Editorial
- "spaCy" should always be spelled with a lowercase "s" and a capital "C",
unless it specifically refers to the Python package or Python import
spacy
(in which case it should be formatted as code).- ✅ spaCy is a library for advanced NLP in Python.
- ❌ Spacy is a library for advanced NLP in Python.
- ✅ First, you need to install the
spacy
package from pip.
- Mentions of code, like function names, classes, variable names etc. in inline
text should be formatted as
code
.- ✅ "Calling the
nlp
object on a text returns aDoc
."
- ✅ "Calling the
- Objects that have pages in the API docs should be linked – for
example,
Doc
orLanguage.to_disk
. The mentions should still be formatted as code within the link. Links pointing to the API docs will automatically receive a little icon. However, if a paragraph includes many references to the API, the links can easily get messy. In that case, we typically only link the first mention of an object and not any subsequent ones.- ✅ The
Span
andToken
objects are views of aDoc
.Span.as_doc
creates aDoc
object from aSpan
. - ❌ The
Span
andToken
objects are views of aDoc
.Span.as_doc
creates aDoc
object from aSpan
.
- ✅ The
-
Other things we format as code are: references to trained pipeline packages like
en_core_web_sm
or file names likecode.py
ormeta.json
.- ✅ After training, the
config.cfg
is saved to disk.
- ✅ After training, the
-
Type annotations are a special type of code formatting, expressed by wrapping the text in
~~
instead of backticks. The result looks like this:List[Doc]. All references to known types will be linked automatically.- ✅ The model has the input type
List[Doc]and it outputs aList[Array2d].
- ✅ The model has the input type
-
We try to keep links meaningful but short.
- ✅ For details, see the usage guide on training with custom code.
- ❌ For details, see the usage guide on training with custom code.
- ❌ For details, see the usage guide on training with custom code here.