explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2025-11-04 01:48:04 +03:00
Code Issues Packages Projects Releases Wiki Activity

Update projects docs etc.

Browse Source
This commit is contained in:
Ines Montani 2020-07-09 19:43:25 +02:00
parent 175d34d8f9
commit ea01831f6a
15 changed files with 752 additions and 128 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

219
website/docs/api/cli.md
View File

@ -297,60 +297,41 @@ will not be available.
## Train {#train} ## Train {#train}
<!-- TODO: document new training -->
Train a model. Expects data in spaCy's Train a model. Expects data in spaCy's
[JSON format](/api/data-formats#json-input). On each epoch, a model will be [binary format](/api/data-formats#training) and a
saved out to the directory. Accuracy scores and model details will be added to a [config file](/api/data-formats#config) with all settings and hyperparameters.
[`meta.json`](/usage/training#models-generating) to allow packaging the model Will save out the best model from all epochs, as well as the final model. The
using the [`package`](/api/cli#package) command. `--code` argument can be used to provide a Python file that's imported before
the training process starts. This lets you register
[custom functions](/usage/training#custom-models) and architectures and refer to
them in your config, all while still using spaCy's built-in `train` workflow. If
you need to manage complex multi-step training workflows, check out the new
[spaCy projects](/usage/projects).
<Infobox title="New in v3.0" variant="warning">
As of spaCy v3.0, the `train` command doesn't take a long list of command-line
arguments anymore and instead expects a single
[`config.cfg` file](/usage/training#config) containing all settings for the
pipeline, training process and hyperparameters.
</Infobox>
```bash ```bash
$ python -m spacy train [lang] [output_path] [train_path] [dev_path] $ python -m spacy train [train_path] [dev_path] [config_path] [--output]
[--base-model] [--pipeline] [--vectors] [--n-iter] [--n-early-stopping] [--code] [--verbose]
[--n-examples] [--use-gpu] [--version] [--meta-path] [--init-tok2vec]
[--parser-multitasks] [--entity-multitasks] [--gold-preproc] [--noise-level]
[--orth-variant-level] [--learn-tokens] [--textcat-arch] [--textcat-multilabel]
[--textcat-positive-label] [--verbose]
``` ```
| Argument | Type | Description | | Argument | Type | Description |
| --------------------------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `lang` | positional | Model language. | | `train_path` | positional | Location of training data in spaCy's [binary format](/api/data-formats#training). Can be a file or a directory of files. |
| `output_path` | positional | Directory to store model in. Will be created if it doesn't exist. | | `dev_path` | positional | Location of development data for evaluation in spaCy's [binary format](/api/data-formats#training). Can be a file or a directory of files. |
| `train_path` | positional | Location of JSON-formatted training data. Can be a file or a directory of files. | | `config_path` | positional | Path to [training config](/api/data-formats#config) file containing all settings and hyperparameters. |
| `dev_path` | positional | Location of JSON-formatted development data for evaluation. Can be a file or a directory of files. | | `--output`, `-o` | positional | Directory to store model in. Will be created if it doesn't exist. |
| `--base-model`, `-b` <Tag variant="new">2.1</Tag> | option | Optional name of base model to update. Can be any loadable spaCy model. | | `--code`, `-c` | option | Path to Python file with additional code to be imported. Allows [registering custom functions](/usage/training#custom-models) for new architectures. |
| `--pipeline`, `-p` <Tag variant="new">2.1</Tag> | option | Comma-separated names of pipeline components to train. Defaults to `'tagger,parser,ner'`. | | `--verbose`, `-V` | flag | Show more detailed messages during training. |
| `--replace-components`, `-R` | flag | Replace components from the base model. |
| `--vectors`, `-v` | option | Model to load vectors from. |
| `--n-iter`, `-n` | option | Number of iterations (default: `30`). |
| `--n-early-stopping`, `-ne` | option | Maximum number of training epochs without dev accuracy improvement. |
| `--n-examples`, `-ns` | option | Number of examples to use (defaults to `0` for all examples). |
| `--use-gpu`, `-g` | option | GPU ID or `-1` for CPU only (default: `-1`). |
| `--version`, `-V` | option | Model version. Will be written out to the model's `meta.json` after training. |
| `--meta-path`, `-m` <Tag variant="new">2</Tag> | option | Optional path to model [`meta.json`](/usage/training#models-generating). All relevant properties like `lang`, `pipeline` and `spacy_version` will be overwritten. |
| `--init-tok2vec`, `-t2v` <Tag variant="new">2.1</Tag> | option | Path to pretrained weights for the token-to-vector parts of the models. See `spacy pretrain`. Experimental. |
| `--parser-multitasks`, `-pt` | option | Side objectives for parser CNN, e.g. `'dep'` or `'dep,tag'` |
| `--entity-multitasks`, `-et` | option | Side objectives for NER CNN, e.g. `'dep'` or `'dep,tag'` |
| `--width`, `-cw` <Tag variant="new">2.2.4</Tag> | option | Width of CNN layers of `Tok2Vec` component. |
| `--conv-depth`, `-cd` <Tag variant="new">2.2.4</Tag> | option | Depth of CNN layers of `Tok2Vec` component. |
| `--cnn-window`, `-cW` <Tag variant="new">2.2.4</Tag> | option | Window size for CNN layers of `Tok2Vec` component. |
| `--cnn-pieces`, `-cP` <Tag variant="new">2.2.4</Tag> | option | Maxout size for CNN layers of `Tok2Vec` component. |
| `--use-chars`, `-chr` <Tag variant="new">2.2.4</Tag> | flag | Whether to use character-based embedding of `Tok2Vec` component. |
| `--bilstm-depth`, `-lstm` <Tag variant="new">2.2.4</Tag> | option | Depth of BiLSTM layers of `Tok2Vec` component (requires PyTorch). |
| `--embed-rows`, `-er` <Tag variant="new">2.2.4</Tag> | option | Number of embedding rows of `Tok2Vec` component. |
| `--noise-level`, `-nl` | option | Float indicating the amount of corruption for data augmentation. |
| `--orth-variant-level`, `-ovl` <Tag variant="new">2.2</Tag> | option | Float indicating the orthography variation for data augmentation (e.g. `0.3` for making 30% of occurrences of some tokens subject to replacement). |
| `--gold-preproc`, `-G` | flag | Use gold preprocessing. |
| `--learn-tokens`, `-T` | flag | Make parser learn gold-standard tokenization by merging ] subtokens. Typically used for languages like Chinese. |
| `--textcat-multilabel`, `-TML` <Tag variant="new">2.2</Tag> | flag | Text classification classes aren't mutually exclusive (multilabel). |
| `--textcat-arch`, `-ta` <Tag variant="new">2.2</Tag> | option | Text classification model architecture. Defaults to `"bow"`. |
| `--textcat-positive-label`, `-tpl` <Tag variant="new">2.2</Tag> | option | Text classification positive label for binary classes with two labels. |
| `--tag-map-path`, `-tm` <Tag variant="new">2.2.4</Tag> | option | Location of JSON-formatted tag map. |
| `--verbose`, `-VV` <Tag variant="new">2.0.13</Tag> | flag | Show more detailed messages during training. |
| `--help`, `-h` | flag | Show help message and available arguments. | | `--help`, `-h` | flag | Show help message and available arguments. |
| **CREATES** | model, pickle | A spaCy model on each epoch. | | **CREATES** | model | The final model and the best model. |
## Pretrain {#pretrain new="2.1" tag="experimental"} ## Pretrain {#pretrain new="2.1" tag="experimental"}
@ -507,12 +488,13 @@ so you don't have to run `python setup.py sdist` separately anymore.
$ python -m spacy package [input_dir] [output_dir] [--meta-path] [--create-meta] [--force] $ python -m spacy package [input_dir] [output_dir] [--meta-path] [--create-meta] [--force]
``` ```
```bash > #### Example
### Example >
python -m spacy package /input /output > ```bash
cd /output/en_model-0.0.0 > python -m spacy package /input /output
pip install dist/en_model-0.0.0.tar.gz > cd /output/en_model-0.0.0
``` > pip install dist/en_model-0.0.0.tar.gz
> ```
| Argument | Type | Description | | Argument | Type | Description |
| ------------------------------------------------ | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------------------------------------------ | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@ -525,14 +507,143 @@ pip install dist/en_model-0.0.0.tar.gz
| `--help`, `-h` | flag | Show help message and available arguments. | | `--help`, `-h` | flag | Show help message and available arguments. |
| **CREATES** | directory | A Python package containing the spaCy model. | | **CREATES** | directory | A Python package containing the spaCy model. |
## Project {#project} ## Project {#project new="3"}
<!-- TODO: document project command and subcommands. We should probably wait and only finalize this once we've finalized the design --> <!-- TODO: document project command and subcommands. We should probably wait and only finalize this once we've finalized the design -->
The `spacy project` CLI includes subcommands for working with
[spaCy projects](/usage/projects), end-to-end workflows for building and
deploying custom spaCy models.
### project clone {#project-clone} ### project clone {#project-clone}
Clone a project template from a Git repository. Calls into `git` under the hood
and uses the sparse checkout feature, so you're only downloading what you need.
By default, spaCy's
[project templates repo](https://github.com/explosion/projects) is used, but you
can provide any other repo (public or private) that you have access to using the
`--repo` option.
<!-- TODO: update example once we've decided on repo structure -->
```bash
$ python -m spacy project clone [name] [dest] [--repo]
```
> #### Example
>
> ```bash
> $ python -m spacy project clone some_example
> ```
>
> Clone from custom repo:
>
> ```bash
> $ python -m spacy project clone template --repo https://github.com/your_org/your_repo
> ```
| Argument | Type | Description |
| -------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `name` | positional | The name of the template to clone, relative to the repo. Can be a top-level directory or a subdirectory like `dir/template`. |
| `dest` | positional | Where to clone the project. Defaults to current working directory. |
| `--repo`, `-r` | option | The repository to clone from. Can be any public or private Git repo you have access to. |
| `--help`, `-h` | flag | Show help message and available arguments. |
| **CREATES** | directory | The cloned [project directory](/usage/projects#project-files). |
### project assets {#project-assets} ### project assets {#project-assets}
Fetch project assets like datasets and pretrained weights. Assets are defined in
the `assets` section of the [`project.yml`](/usage/projects#project-yml). If a
`checksum` is provided, the file is only downloaded if no local file with the
same checksum exists and spaCy will show an error if the checksum of the
downloaded file doesn't match. If assets don't specify a `url` they're
considered "private" and you have to take care of putting them into the
destination directory yourself. If a local path is provided, the asset is copied
into the current project.
<!-- TODO: update example once we've decided on repo structure -->
```bash
$ python -m spacy project assets [project_dir]
```
> #### Example
>
> ```bash
> $ python -m spacy project assets
> ```
| Argument | Type | Description |
| -------------- | ---------- | ----------------------------------------------------------------- |
| `project_dir` | positional | Path to project directory. Defaults to current working directory. |
| `--help`, `-h` | flag | Show help message and available arguments. |
| **CREATES** | files | Downloaded or copied assets defined in the `project.yml`. |
### project run {#project-run} ### project run {#project-run}
Run a named command or workflow defined in the
[`project.yml`](/usage/projects#project-yml). If a workflow name is specified,
all commands in the workflow are run, in order. If commands define
[dependencies or outputs](/usage/projects#deps-outputs), they will only be
re-run if state has changed. For example, if the input dataset changes, a
preprocessing command that depends on those files will be re-run.
<!-- TODO: update example once we've decided on repo structure -->
```bash
$ python -m spacy project run [subcommand] [project_dir] [--force] [--dry]
```
> #### Example
>
> ```bash
> $ python -m spacy project run train
> ```
| Argument | Type | Description |
| --------------- | ---------- | ----------------------------------------------------------------- |
| `subcommand` | positional | Name of the command or workflow to run. |
| `project_dir` | positional | Path to project directory. Defaults to current working directory. |
| `--force`, `-F` | flag | Force re-running steps, even if nothing changed. |
| `--dry`, `-D` | flag |  Perform a dry run and don't execute scripts. |
| `--help`, `-h` | flag | Show help message and available arguments. |
### project dvc {#project-dvc} ### project dvc {#project-dvc}
Auto-generate [Data Version Control](https://dvc.org) (DVC) config file. Calls
[`dvc run`](https://dvc.org/doc/command-reference/run) with `--no-exec` under
the hood to generate the `dvc.yaml`. A DVC project can only define one pipeline,
so you need to specify one workflow defined in the
[`project.yml`](/usage/projects#project-yml). If no workflow is specified, the
first defined workflow is used. The DVC config will only be updated if the
`project.yml` changed. For details, see the
[DVC integration](/usage/projects#dvc) docs.
<Infobox variant="warning">
This command requires DVC to be installed and initialized in the project
directory, e.g. via [`dvc init`](https://dvc.org/doc/command-reference/init).
You'll also need to add the assets you want to track with
[`dvc add`](https://dvc.org/doc/command-reference/add).
</Infobox>
```bash
$ python -m spacy project dvc [project_dir] [workflow] [--force] [--verbose]
```
> #### Example
>
> ```bash
> git init
> dvc init
> python -m spacy project dvc all
> ```
| Argument | Type | Description |
| ----------------- | ---------- | --------------------------------------------------------------------------------- |
| `project_dir` | positional | Path to project directory. Defaults to current working directory. |
| `workflow` | positional | Name of workflow defined in `project.yml`. Defaults to first workflow if not set. |
| `--force`, `-F` | flag | Force-updating config file. |
| `--verbose`, `-V` | flag |  Print more output generated by DVC. |
| `--help`, `-h` | flag | Show help message and available arguments. |

488
website/docs/usage/projects.md
View File

@ -5,25 +5,29 @@ menu:
- ['Intro & Workflow', 'intro'] - ['Intro & Workflow', 'intro']
- ['Directory & Assets', 'directory'] - ['Directory & Assets', 'directory']
- ['Custom Projects', 'custom'] - ['Custom Projects', 'custom']
- ['Integrations', 'integrations']
--- ---
> #### Project templates > #### 🪐 Project templates
> >
> Our [`projects`](https://github.com/explosion/projects) repo includes various > Our [`projects`](https://github.com/explosion/projects) repo includes various
> project templates for different tasks and models that you can clone and run. > project templates for different NLP tasks, models, workflows and integrations
> that you can clone and run. The easiest way to get started is to pick a
<!-- TODO: write more about templates in aside --> > template, clone it and start modifying it!
spaCy projects let you manage and share **end-to-end spaCy workflows** for spaCy projects let you manage and share **end-to-end spaCy workflows** for
training, packaging and serving your custom models. You can start off by cloning different **use cases and domains**, and orchestrate training, packaging and
a pre-defined project template, adjust it to fit your needs, load in your data, serving your custom models. You can start off by cloning a pre-defined project
train a model, export it as a Python package and share the project templates template, adjust it to fit your needs, load in your data, train a model, export
with your team. Under the hood, project use it as a Python package and share the project templates with your team. spaCy
[Data Version Control](https://dvc.org) (DVC) to track and version inputs and projects can be used via the new [`spacy project`](/api/cli#project) command.
outputs, and make sure you're only re-running what's needed. spaCy projects can For an overview of the available project templates, check out the
be used via the new [`spacy project`](/api/cli#project) command. For an overview [`projects`](https://github.com/explosion/projects) repo. spaCy projects also
of the available project templates, check out the [integrate](#integrations) with many other cool machine learning and data
[`projects`](https://github.com/explosion/projects) repo. science tools to track and manage your data and experiments, iterate on demos
and prototypes and ship your models into production.
<!-- TODO: mention integrations -->
## Introduction and workflow {#intro} ## Introduction and workflow {#intro}
@ -37,18 +41,32 @@ mattis pretium.
</Project> </Project>
spaCy projects make it easy to integrate with many other **awesome tools** in
the data science and machine learning ecosystem to track and manage your data
and experiments, iterate on demos and prototypes and ship your models into
production.
<Grid narrow cols={3}>
<Integration title="DVC" logo="dvc" url="#dvc">Manage and version your data</Integration>
<Integration title="Prodigy" logo="prodigy" url="#prodigy">Create labelled training data</Integration>
<Integration title="Streamlit" logo="streamlit" url="#streamlit">Visualize and demo your models</Integration>
<Integration title="FastAPI" logo="fastapi" url="#fastapi">Serve your models and host APIs</Integration>
<Integration title="Ray" logo="ray" url="#ray">Distributed and parallel training</Integration>
<Integration title="Weights &amp; Biases" logo="wandb" url="#wandb">Track your experiments and results</Integration>
</Grid>
### 1. Clone a project template {#clone} ### 1. Clone a project template {#clone}
> #### Cloning under the hoodimport { ReactComponent as WandBLogo } from '../images/logos/wandb.svg'
>
> To clone a project, spaCy calls into `git` and uses the "sparse checkout"
> feature to only clone the relevant directory or directories.
The [`spacy project clone`](/api/cli#project-clone) command clones an existing The [`spacy project clone`](/api/cli#project-clone) command clones an existing
project template and copies the files to a local directory. You can then run the project template and copies the files to a local directory. You can then run the
project, e.g. to train a model and edit the commands and scripts to build fully project, e.g. to train a model and edit the commands and scripts to build fully
custom workflows. custom workflows.
> #### Cloning under the hood
>
> To clone a project, spaCy calls into `git` and uses the "sparse checkout"
> feature to only clone the relevant directory or directories.
```bash ```bash
$ python -m spacy clone some_example_project $ python -m spacy clone some_example_project
``` ```
@ -59,46 +77,169 @@ can specify an optional second argument to define the output directory. The
to use the spaCy [`projects`](https://github.com/explosion/projects) repo. You to use the spaCy [`projects`](https://github.com/explosion/projects) repo. You
can also use any private repo you have access to with Git. can also use any private repo you have access to with Git.
If you plan on making the project a Git repo, you can set the `--git` flag to
set it up automatically _before_ initializing DVC, so DVC can integrate with
Git. This means that it will automatically add asset files to a `.gitignore` (so
you never check assets into the repo, only the asset meta files).
### 2. Fetch the project assets {#assets} ### 2. Fetch the project assets {#assets}
> #### project.yml
>
> ```yaml
> assets:
> - dest: 'assets/training.spacy'
> url: 'https://example.com/data.spacy'
> checksum: '63373dd656daa1fd3043ce166a59474c'
> ```
Assets are data files your project needs – for example, the training and Assets are data files your project needs – for example, the training and
evaluation data or pretrained vectors and embeddings to initialize your model evaluation data or pretrained vectors and embeddings to initialize your model
with. <!-- TODO: ... --> with. Each project template comes with a `project.yml` that defines the assets
to download and where to put them. The
[`spacy project assets`](/api/cli#project-assets) will fetch the project assets
for you:
```bash ```bash
cd some_example_project cd some_example_project
python -m spacy project assets python -m spacy project assets
``` ```
### 3. Run the steps {#run-all} ### 3. Run a command {#run}
> #### project.yml
>
> ```yaml
> commands:
> - name: preprocess
> help: "Convert the input data to spaCy's format"
> script:
> - 'python -m spacy convert assets/train.conllu corpus/'
> - 'python -m spacy convert assets/eval.conllu corpus/'
> deps:
> - 'assets/train.conllu'
> - 'assets/eval.conllu'
> outputs:
> - 'corpus/train.spacy'
> - 'corpus/eval.spacy'
> ```
Commands consist of one or more steps and can be run with
[`spacy project run`](/api/cli#project-run). The following will run the command
`preprocess` defined in the `project.yml`:
```bash ```bash
$ python -m spacy project run-all $ python -m spacy project run preprocess
``` ```
### 4. Run single commands {#run} Commands can define their expected [dependencies and outputs](#deps-outputs)
using the `deps` (files the commands require) and `outputs` (files the commands
create) keys. This allows your project to track changes and determine whether a
command needs to be re-run. For instance, if your input data changes, you want
to re-run the `preprocess` command. But if nothing changed, this step can be
skipped. You can also set `--force` to force re-running a command, or `--dry` to
perform a "dry run" and see what would happen (without actually running the
script).
### 4. Run a workflow {#run-workfow}
> #### project.yml
>
> ```yaml
> workflows:
> all:
> - preprocess
> - train
> - package
> ```
Workflows are series of commands that are run in order and often depend on each
other. For instance, to generate a packaged model, you might start by converting
your data, then run [`spacy train`](/api/cli#train) to train your model on the
converted data and if that's successful, run [`spacy package`](/api/cli#package)
to turn the best model artifact into an installable Python package. The
following command run the workflow named `all` defined in the `project.yml`, and
execute the commands it specifies, in order:
```bash ```bash
$ python -m spacy project run visualize $ python -m spacy project run all
``` ```
Using the expected [dependencies and outputs](#deps-outputs) defined in the
commands, spaCy can determine whether to re-run a command (if its inputs or
outputs have changed) or whether to skip it. If you're looking to implement more
advanced data pipelines and track your changes in Git, check out the
[Data Version Control (DVC) integration](#dvc). The
[`spacy project dvc`](/api/cli#project-dvc) command generates a DVC config file
from a workflow defined in your `project.yml` so you can manage your spaCy
project as a DVC repo.
## Project directory and assets {#directory} ## Project directory and assets {#directory}
### project.yml {#project-yml} ### project.yml {#project-yml}
The project config, `project.yml`, defines the assets a project depends on, like The `project.yml` defines the assets a project depends on, like datasets and
datasets and pretrained weights, as well as a series of commands that can be run pretrained weights, as well as a series of commands that can be run separately
separately or as a pipeline – for instance, to preprocess the data, convert it or as a workflow – for instance, to preprocess the data, convert it to spaCy's
to spaCy's format, train a model, evaluate it and export metrics, package it and format, train a model, evaluate it and export metrics, package it and spin up a
spin up a quick web demo. It looks pretty similar to a config file used to quick web demo. It looks pretty similar to a config file used to define CI
define CI pipelines. pipelines.
<!-- TODO: include example etc. --> <!-- TODO: update with better (final) example -->
```yaml
https://github.com/explosion/spacy-boilerplates/blob/master/ner_fashion/project.yml
```
| Section | Description |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `variables` | A dictionary of variables that can be referenced in paths, URLs and scripts. For example, `{NAME}` will use the value of the variable `NAME`. |
| `assets` | A list of assets that can be fetched with the [`project assets`](/api/cli#project-assets) command. `url` defines a URL or local path, `dest` is the destination file relative to the project directory, and an optional `checksum` ensures that an error is raised if the file's checksum doesn't match. |
| `workflows` | A dictionary of workflow names, mapped to a list of command names, to execute in order. Workflows can be run with the [`project run`](/api/cli#project-run) command. |
| `commands` | A list of named commands. A command can define an optional help message (shown in the CLI when the user adds `--help`) and the `script`, a list of commands to run. The `deps` and `outputs` let you define the created file the command depends on and produces, respectively. This lets spaCy determine whether a command needs to be re-run because its dependencies or outputs changed. Commands can be run as part of a workflow, or separately with the [`project run`](/api/cli#project-run) command. |
### Dependencies and outputs {#deps-outputs}
Each command defined in the `project.yml` can optionally define a list of
dependencies and outputs. These are the files the commands requires and creates.
For example, a command for training a model may depend on a
[`config.cfg`](/usage/training#config) and the training and evaluation data, and
it will export a directory `model-best`, containing the best model, which you
can then re-use in other commands.
<!-- prettier-ignore -->
```yaml
### project.yml
commands:
- name: train
help: 'Train a spaCy model using the specified corpus and config'
script:
- 'python -m spacy train ./corpus/training.spacy ./corpus/evaluation.spacy ./configs/config.cfg -o training/'
deps:
- 'configs/config.cfg'
- 'corpus/training.spacy'
- 'corpus/evaluation.spacy'
outputs:
- 'training/model-best'
```
> #### Re-running vs. skipping
>
> Under the hood, spaCy uses a `project.lock` lockfile that stores the details
> for each command, as well as its dependencies and outputs and their checksums.
> It's updated on each run. If any of this information changes, the command will
> be re-run. Otherwise, it will be skipped.
If you're running a command and it depends on files that are missing, spaCy will
show you an error. If a command defines dependencies and outputs that haven't
changed since the last run, the command will be skipped. This means that you're
only re-running commands if they need to be re-run. To force re-running a
command or workflow, even if nothing changed, you can set the `--force` flag.
Note that [`spacy project`](/api/cli#project) doesn't compile any dependency
graphs based on the dependencies and outputs, and won't re-run previous steps
automatically. For instance, if you only run the command `train` that depends on
data created by `preprocess` and those files are missing, spaCy will show an
error – it won't just re-run `preprocess`. If you're looking for more advanced
data management, check out the [Data Version Control (DVC) integration](#dvc)
integration. If you're planning on integrating your spaCy project with DVC, you
can also use `outputs_no_cache` instead of `outputs` to define outputs that
won't be cached or tracked.
### Files and directory structure {#project-files} ### Files and directory structure {#project-files}
@ -109,10 +250,9 @@ scripts).
```yaml ```yaml
### Project directory ### Project directory
├── project.yml # the project configuration ├── project.yml # the project settings
├── dvc.yaml # auto-generated Data Version Control config ├── project.lock # lockfile that tracks inputs/outputs
├── dvc.lock # auto-generated Data Version control lock file ├── assets/ # downloaded data assets
├── assets/ # downloaded data assets and DVC meta files
├── metrics/ # output directory for evaluation metrics ├── metrics/ # output directory for evaluation metrics
├── training/ # output directory for trained models ├── training/ # output directory for trained models
├── corpus/ # output directory for training corpus ├── corpus/ # output directory for training corpus
@ -125,13 +265,89 @@ scripts).
└── ... # any other files, like a requirements.txt etc. └── ... # any other files, like a requirements.txt etc.
``` ```
When the project is initialized, spaCy will auto-generate a `dvc.yaml` based on ---
the project config. The file is updated whenever the project config has changed
and includes all commands defined in the `run` section of the project config.
This allows DVC to track the inputs and outputs and know which steps need to be
re-run.
#### Why Data Version Control (DVC)? ## Custom scripts and projects {#custom}
The `project.yml` lets you define any custom commands and run them as part of
your training, evaluation or deployment workflows. The `script` section defines
a list of commands that are called in a subprocess, in order. This lets you
execute other Python scripts or command-line tools. Let's say you've written a
few integration tests that load the best model produced by the training command
and check that it works correctly. You can now define a `test` command that
calls into [`pytest`](https://docs.pytest.org/en/latest/) and runs your tests:
> #### Calling into Python
>
> If any of your command scripts call into `python`, spaCy will take care of
> replacing that with your `sys.executable`, to make sure you're executing
> everything with the same Python (not some other Python installed on your
> system). It also normalizes references to `python3`, `pip3` and `pip`.
```yaml
### project.yml
commands:
- name: test
help: 'Test the trained model'
script:
- 'python -m pytest ./scripts/tests'
deps:
- 'training/model-best'
```
Adding `training/model-best` to the command's `deps` lets you ensure that the
file is available. If not, spaCy will show an error and the command won't run.
<!-- TODO: add another example -->
### Cloning from your own repo {#custom-repo}
The [`spacy project clone`](/api/cli#project-clone) command lets you customize
the repo to clone from using the `--repo` option. It calls into `git`, so you'll
be able to clone from any repo that you have access to, including private repos.
```bash
$ python -m spacy project your_project --repo https://github.com/you/repo
```
At a minimum, a valid project template needs to contain a
[`project.yml`](#project-yml). It can also include
[other files](/usage/projects#project-files), like custom scripts, a
`requirements.txt` listing additional dependencies,
[training configs](/usage/training#config) and model meta templates, or Jupyter
notebooks with usage examples.
<Infobox title="Important note about assets" variant="warning">
It's typically not a good idea to check large data assets, trained models or
other artifacts into a Git repo and you should exclude them from your project
template. If you want to version your data and models, check out
[Data Version Control](#dvc) (DVC), which integrates with spaCy projects.
</Infobox>
### Working with private assets {#private-assets}
For many projects, the datasets and weights you're working with might be
company-internal and not available via a public URL. In that case, you can
specify the destination paths and a checksum, and leave out the URL. When your
teammates clone and run your project, they can place the files in the respective
directory themselves. The [`spacy project assets`](/api/cli#project-assets)
command will alert about missing files and mismatched checksums, so you can
ensure that others are running your project with the same data.
```yaml
### project.yml
assets:
- dest: 'assets/private_training_data.json'
checksum: '63373dd656daa1fd3043ce166a59474c'
- dest: 'assets/private_vectors.bin'
checksum: '5113dc04e03f079525edd8df3f4f39e3'
```
## Integrations {#integrations}
### Data Version Control (DVC) {#dvc} <IntegrationLogo name="dvc" title="DVC" width={70} height="auto" align="right" />
Data assets like training corpora or pretrained weights are at the core of any Data assets like training corpora or pretrained weights are at the core of any
NLP project, but they're often difficult to manage: you can't just check them NLP project, but they're often difficult to manage: you can't just check them
@ -140,19 +356,187 @@ steps that depend on each other, like a preprocessing step that generates your
training data, you need to make sure the data is always up-to-date, and re-run training data, you need to make sure the data is always up-to-date, and re-run
all steps of your process every time, just to be safe. all steps of your process every time, just to be safe.
[Data Version Control (DVC)](https://dvc.org) is a standalone open-source tool [Data Version Control](https://dvc.org) (DVC) is a standalone open-source tool
that integrates into your workflow like Git, builds a dependency graph for your that integrates into your workflow like Git, builds a dependency graph for your
data pipelines and tracks and caches your data files. If you're downloading data data pipelines and tracks and caches your data files. If you're downloading data
from an external source, like a storage bucket, DVC can tell whether the from an external source, like a storage bucket, DVC can tell whether the
resource has changed. It can also determine whether to re-run a step, depending resource has changed. It can also determine whether to re-run a step, depending
on whether its input have changed or not. All metadata can be checked into a Git on whether its input have changed or not. All metadata can be checked into a Git
repo, so you'll always be able to reproduce your experiments. `spacy project` repo, so you'll always be able to reproduce your experiments.
uses DVC under the hood and you typically don't have to think about it if you
don't want to. But if you do want to integrate with DVC more deeply, you can.
Each spaCy project is also a regular DVC project.
#### Checking projects into Git To set up DVC, install the package and initialize your spaCy project as a Git
and DVC repo. You can also
[customize your DVC installation](https://dvc.org/doc/install/macos#install-with-pip)
to include support for remote storage like Google Cloud Storage, S3, Azure, SSH
and more.
```bash
pip install dvc # Install DVC
git init # Initialize a Git repo
dvc init # Initialize a DVC project
```
The [`spacy project dvc`](/api/cli#project-dvc) command creates a `dvc.yaml`
config file based on a workflow defined in your `project.yml`. Whenever you
update your project, you can re-run the command to update your DVC config. You
can then manage your spaCy project like any other DVC project, run
[`dvc add`](https://dvc.org/doc/command-reference/add) to add and track assets
and [`dvc repro`](https://dvc.org/doc/command-reference/repro) to reproduce the
workflow or individual commands.
```bash
$ python -m spacy project dvc [workflow name]
```
<Infobox title="Important note for multiple workflows" variant="warning">
DVC currently expects a single workflow per project, so when creating the config
with [`spacy project dvc`](/api/cli#project-dvc), you need to specify the name
of a workflow defined in your `project.yml`. You can still use multiple
workflows, but only one can be tracked by DVC.
</Infobox>
<Project id="integrations/dvc">
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus interdum
sodales lectus, ut sodales orci ullamcorper id. Sed condimentum neque ut erat
mattis pretium.
</Project>
--- ---
## Custom projects and scripts {#custom} ### Prodigy {#prodigy} <IntegrationLogo name="prodigy" width={100} height="auto" align="right" />
[Prodigy](https://prodi.gy) is a modern annotation tool for creating training
data for machine learning models, developed by us. It integrates with spaCy
out-of-the-box and provides many different
[annotation recipes](https://prodi.gy/docs/recipes) for a variety of NLP tasks,
with and without a model in the loop. If Prodigy is installed in your project,
you can
The following example command starts the Prodigy app using the
[`ner.correct`](https://prodi.gy/docs/recipes#ner-correct) recipe and streams in
suggestions for the given entity labels produced by a pretrained model. You can
then correct the suggestions manually in the UI. After you save and exit the
server, the full dataset is exported in spaCy's format and split into a training
and evaluation set.
<!-- prettier-ignore -->
```yaml
### project.yml
variables:
PRODIGY_DATASET: 'ner_articles'
PRODIGY_LABELS: 'PERSON,ORG,PRODUCT'
PRODIGY_MODEL: 'en_core_web_md'
commands:
- name: annotate
- script:
- 'python -m prodigy ner.correct {PRODIGY_DATASET} ./assets/raw_data.jsonl
{PRODIGY_MODEL} --labels {PRODIGY_LABELS}'
- 'python -m prodigy data-to-spacy ./corpus/train.spacy
./corpus/eval.spacy --ner {PRODIGY_DATASET}'
- deps:
- 'assets/raw_data.jsonl'
- outputs:
- 'corpus/train.spacy'
- 'corpus/eval.spacy'
```
<Project id="integrations/prodigy">
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus interdum
sodales lectus, ut sodales orci ullamcorper id. Sed condimentum neque ut erat
mattis pretium.
</Project>
---
### Streamlit {#streamlit} <IntegrationLogo name="streamlit" width={150} height="auto" align="right" />
<Grid cols={2} gutterBottom={false}>
<div>
[Streamlit](https://streamlit.io) is a Python framework for building interactive
data apps. The [`spacy-streamlit`](https://github.com/explosion/spacy-streamlit)
package helps you integrate spaCy visualizations into your Streamlit apps and
quickly spin up demos to explore your models interactively. It includes a full
embedded visualizer, as well as individual components.
```bash
$ pip install spacy_streamlit
```
</div>
![](../images/spacy-streamlit.png)
</Grid>
Using [`spacy-streamlit`](https://github.com/explosion/spacy-streamlit), your
projects can easily define their own scripts that spin up an interactive
visualizer, using the latest model you trained, or a selection of models so you
can compare their results. The following script starts an
[NER visualizer](/usage/visualizers#ent) and takes two positional command-line
argument you can pass in from your `config.yml`: a comma-separated list of model
paths and an example text to use as the default text.
```python
### scripts/visualize.py
import spacy_streamlit
import sys
DEFAULT_TEXT = sys.argv[2] if len(sys.argv) >= 3 else ""
MODELS = [name.strip() for name in sys.argv[1].split(",")]
spacy_streamlit.visualize(MODELS, DEFAULT_TEXT, visualizers=["ner"])
```
<!-- prettier-ignore -->
```yaml
### project.yml
commands:
- name: visualize
help: "Visualize the model's output interactively using Streamlit"
script:
- 'streamlit run ./scripts/visualize.py ./training/model-best "I like Adidas shoes."'
deps:
- 'training/model-best'
```
<Project id="integrations/streamlit">
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus interdum
sodales lectus, ut sodales orci ullamcorper id. Sed condimentum neque ut erat
mattis pretium.
</Project>
---
### FastAPI {#fastapi} <IntegrationLogo name="fastapi" width={100} height="auto" align="right" />
<!-- TODO: come up with example – there's not much integration needed, but it'd be nice to show an example that addresses some of the main concerns for serving ML (workers etc.) -->
<Project id="integrations/fastapi">
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus interdum
sodales lectus, ut sodales orci ullamcorper id. Sed condimentum neque ut erat
mattis pretium.
</Project>
---
### Ray {#ray} <IntegrationLogo name="ray" width={100} height="auto" align="right" />
<!-- TODO: document -->
---
### Weights & Biases {#wandb} <IntegrationLogo name="wandb" width={175} height="auto" align="right" />
<!-- TODO: decide how we want this to work? Just send results plus config from spacy evaluate in a separate command/script? -->

4
website/docs/usage/visualizers.md
View File

@ -4,7 +4,7 @@ teaser: Visualize dependencies and entities in your browser or in a notebook
new: 2 new: 2
menu: menu:
- ['Dependencies', 'dep'] - ['Dependencies', 'dep']
- ['Entities', 'ent'] - ['Named Entities', 'ent']
- ['Jupyter Notebooks', 'jupyter'] - ['Jupyter Notebooks', 'jupyter']
- ['Rendering HTML', 'html'] - ['Rendering HTML', 'html']
- ['Web app usage', 'webapp'] - ['Web app usage', 'webapp']
@ -356,6 +356,6 @@ Alternatively, if you're using [Streamlit](https://streamlit.io), check out the
helps you integrate spaCy visualizations into your apps. It includes a full helps you integrate spaCy visualizations into your apps. It includes a full
embedded visualizer, as well as individual components. embedded visualizer, as well as individual components.
![](../images/spacy-streamlit.png)] ![](../images/spacy-streamlit.png)
</Grid> </Grid>

13
website/src/components/card.js
View File

@ -1,18 +1,20 @@
import React from 'react' import React from 'react'
import PropTypes from 'prop-types' import PropTypes from 'prop-types'
import classNames from 'classnames'
import Link from './link' import Link from './link'
import { H5 } from './typography' import { H5 } from './typography'
import classes from '../styles/card.module.sass' import classes from '../styles/card.module.sass'
const Card = ({ title, to, image, header, onClick, children }) => ( const Card = ({ title, to, image, header, small, onClick, children }) => (
<div className={classes.root}> <div className={classNames(classes.root, { [classes.small]: !!small })}>
{header && ( {header && (
<Link to={to} onClick={onClick} hidden> <Link to={to} onClick={onClick} hidden>
{header} {header}
</Link> </Link>
)} )}
<H5> {(title || image) && (
<H5 className={classes.title}>
{image && ( {image && (
<div className={classes.image}> <div className={classes.image}>
<img src={image} width={35} alt="" /> <img src={image} width={35} alt="" />
@ -24,6 +26,7 @@ const Card = ({ title, to, image, header, onClick, children }) => (
</Link> </Link>
)} )}
</H5> </H5>
)}
<Link to={to} onClick={onClick} hidden> <Link to={to} onClick={onClick} hidden>
{children} {children}
</Link> </Link>
@ -31,10 +34,10 @@ const Card = ({ title, to, image, header, onClick, children }) => (
) )
Card.propTypes = { Card.propTypes = {
title: PropTypes.string, title: PropTypes.node,
header: PropTypes.node,
to: PropTypes.string, to: PropTypes.string,
image: PropTypes.string, image: PropTypes.string,
card: PropTypes.node,
onClick: PropTypes.func, onClick: PropTypes.func,
children: PropTypes.node, children: PropTypes.node,
} }

2
website/src/components/infobox.js
View File

@ -37,7 +37,7 @@ Infobox.defaultProps = {
} }
Infobox.propTypes = { Infobox.propTypes = {
title: PropTypes.string, title: PropTypes.node,
id: PropTypes.string, id: PropTypes.string,
variant: PropTypes.oneOf(['default', 'warning', 'danger']), variant: PropTypes.oneOf(['default', 'warning', 'danger']),
className: PropTypes.string, className: PropTypes.string,

2
website/src/components/table.js
View File

@ -29,7 +29,7 @@ function getCellContent(children) {
function isDividerRow(children) { function isDividerRow(children) {
if (children.length && children[0].props && children[0].props.name == 'td') { if (children.length && children[0].props && children[0].props.name == 'td') {
const tdChildren = children[0].props.children const tdChildren = children[0].props.children
if (!Array.isArray(tdChildren) && tdChildren.props) { if (tdChildren && !Array.isArray(tdChildren) && tdChildren.props) {
return tdChildren.props.name === 'em' return tdChildren.props.name === 'em'
} }
} }

5
website/src/images/logos/dvc.svg Normal file
View File

@ -0,0 +1,5 @@
<svg xmlns="http://www.w3.org/2000/svg" width="36" height="22" viewBox="0 0 36 22">
<path d="M0 16.584V.696A.44.44 0 01.132.372a.413.413 0 01.3-.132h5.856c2.32 0 4.304.82 5.952 2.46 1.648 1.64 2.472 3.612 2.472 5.916 0 2.32-.824 4.304-2.472 5.952-1.648 1.648-3.632 2.472-5.952 2.472H.432a.413.413 0 01-.3-.132.44.44 0 01-.132-.324zm3.744-3.096h2.304c1.344 0 2.46-.468 3.348-1.404.888-.936 1.332-2.092 1.332-3.468 0-1.36-.444-2.508-1.332-3.444-.888-.936-2.004-1.404-3.348-1.404H3.744v9.72z" fill="#13ADC7"></path>
<path d="M17.121 21.016L9.633 4.864c-.08-.16-.076-.304.012-.432s.22-.192.396-.192h3.192c.208 0 .344.088.408.264l3.96 8.76h.144l3.96-8.76c.064-.176.2-.264.408-.264h3.192c.176 0 .308.064.396.192.088.128.092.272.012.432l-7.536 16.152c-.096.176-.232.264-.408.264h-.24c-.176 0-.312-.088-.408-.264z" fill="#945DD6"></path>
<path d="M23.196 14.784c-1.68-1.68-2.52-3.72-2.52-6.12s.844-4.444 2.532-6.132C24.896.844 26.94 0 29.34 0c2.24 0 4.176.744 5.808 2.232.24.224.248.448.024.672L33.3 4.848c-.208.192-.408.192-.6 0-.912-.816-1.992-1.224-3.24-1.224-1.344 0-2.464.484-3.36 1.452-.896.968-1.344 2.132-1.344 3.492 0 1.344.452 2.492 1.356 3.444.904.952 2.028 1.428 3.372 1.428 1.248 0 2.32-.384 3.216-1.152.224-.192.432-.184.624.024l1.872 1.992c.208.208.2.424-.024.648-1.6 1.552-3.544 2.328-5.832 2.328-2.4 0-4.448-.832-6.144-2.496z" fill="#F46737"></path>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.3 KiB

14
website/src/images/logos/fastapi.svg Normal file
View File

@ -0,0 +1,14 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1309.1" height="241.8" viewBox="0 0 346.4 64">
<g fill="#009688">
<path d="M32 0a32 32 0 100 64 32 32 0 000-64zm-1.7 57.6v-20H19.2l16-31.3v20.1h10.7z"/>
<g stroke-width="2">
<path d="M89.5 59.4V4.2h33.3v6.6H97.3v16.6H120V34H97.3v25.4z" />
<path d="M141.8 54h4.7l3.4-.5V41.2q-.8-.4-2.6-.7-1.8-.3-4.3-.3l-3.6.2q-1.9.3-3.5 1-1.5.8-2.5 2.1t-1 3.5q0 4 2.5 5.6 2.6 1.5 7 1.5zm-.6-37q4.5 0 7.5 1.1 3.1 1.1 5 3.3 1.9 2 2.7 5 .7 2.9.7 6.4v25.9l-2.7.5-3.8.4q-2.1.3-4.7.4-2.5.3-5 .3-3.4 0-6.4-.7-3-.7-5-2.3-2.2-1.6-3.4-4.1-1.2-2.6-1.2-6.1 0-3.5 1.3-6 1.5-2.4 3.8-4 2.4-1.4 5.6-2.2 3.2-.7 6.7-.7l2.3.2q1.2 0 2.3.3l1.9.3 1 .3v-2.1q0-1.8-.3-3.6-.4-1.8-1.4-3.2-1-1.4-3-2.2-1.7-.9-4.5-.9-3.7 0-6.5.6-2.7.4-4 1l-1-6.1q1.5-.7 4.9-1.2 3.3-.7 7.2-.7z" />
<path d="M179 54q4.6 0 6.7-1.1 2.3-1.2 2.3-3.9t-2.2-4.3q-2.1-1.6-7-3.5l-4.7-2q-2.2-1-3.8-2.3-1.6-1.4-2.5-3.3-1-2-1-4.7 0-5.5 4.1-8.7 4-3.3 11-3.3 1.8 0 3.6.3 1.7.1 3.3.4l2.6.6 1.8.6L192 25q-1.2-.6-3.8-1.2-2.5-.8-6.1-.8-3.1 0-5.4 1.3-2.4 1.2-2.4 3.8 0 1.4.5 2.4.6 1 1.6 2L179 34l3.9 1.5q2.9 1 5.2 2.2 2.3 1 4 2.5 1.6 1.4 2.5 3.5.8 2 .8 5 0 5.7-4.3 8.6-4.2 3-12 3-5.6 0-8.7-1l-4.2-1.3 1.3-6.4 4.1 1.4q2.8 1 7.4 1z" />
<path d="M211.8 18h15.7v6.2h-15.7v19.1q0 3.1.5 5.2.5 2 1.4 3.2 1 1 2.4 1.6 1.5.5 3.4.5 3.3 0 5.3-.8t2.9-1l1.4 6q-1.1.6-3.9 1.4-2.8.9-6.4.9-4.2 0-7-1-2.7-1.2-4.4-3.3-1.6-2.2-2.4-5.3-.6-3.2-.6-7.3v-37l7.4-1.3z" />
<path d="M274.3 59.4l-2.5-7-2.5-7.1h-25l-5 14.1h-8l5.9-16.2 5.4-14.2q2.7-6.7 5.4-12.7 2.6-6.2 5.5-12.1h7q3 6 5.5 12 2.7 6.1 5.3 12.8l5.5 14.2 6 16.2zM267.1 39L262 25.5q-2.5-6.5-5.2-12.5-2.8 6-5.3 12.5l-5 13.4z" />
<path d="M304.9 3.6q11.6 0 17.8 4.5 6.3 4.4 6.3 13.1 0 4.8-1.7 8.2-1.7 3.4-5 5.5-3.2 2.1-7.8 3-4.6 1-10.4 1h-6.2v20.5h-7.7V5q3.3-.8 7.3-1 4-.4 7.4-.4zm.6 6.8q-5 0-7.6.2v21.7h5.9q4 0 7.1-.5 3.2-.5 5.4-1.7 2.2-1.3 3.4-3.5 1.2-2.1 1.2-5.5 0-3.1-1.3-5.2-1.2-2-3.3-3.3-2-1.3-4.9-1.7-2.8-.5-5.9-.5z" />
<path d="M338.6 4.2h7.8v55.2h-7.8z" />
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.0 KiB

3
website/src/images/logos/prodigy.svg Normal file
View File

@ -0,0 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 540 158" width="540" height="158">
<path d="M70.6 48.6c7 7.3 10.5 17 10.5 29.2s-3.3 22-10.4 29.2c-7 7.3-16 11-27 11-9.4 0-16.8-2.6-21.7-8v44.7H0V39h20.7v8c4.8-6.3 12.4-9.5 23-9.5 11 0 20 3.7 27 11zM22 76v3.6c0 12 7.2 19.8 18.2 19.8 11.2 0 18.7-8 18.7-21.6S51.3 56.2 40 56.2C29.2 56.2 22 64 22 76zM133.8 59.4c-12.6 0-20.5 7-20.5 17.8v39.3h-22V39h21v8.8c4-6.4 11.3-9.6 21.4-9.6v21.2zM209.5 107c-7.6 7.4-17.5 11.2-29.5 11.2s-22-3.8-29.7-11c-7.6-7.6-11.5-17.3-11.5-29.3 0-12.2 4-22 11.5-29.3 7.8-7.3 17.7-11 29.7-11s22 3.7 29.5 11c7.8 7.3 11.7 17 11.7 29.2 0 11.8-4 21.6-11.7 29zM180 56.3c-5.7 0-10.3 2-13.8 5.8s-5.2 9-5.2 15.7c0 6.7 1.8 12 5.2 15.7 3.4 3.8 8 5.7 13.8 5.7s10.3-1.8 13.8-5.6 5.2-9 5.2-15.7c0-6.8-1.8-12-5.2-15.7-3.5-3.8-8-5.8-13.8-5.8zM313 116.5h-20.5v-8c-4.4 5.6-12.7 9.7-23 9.7-11 0-20-3.8-27-11-7-7.5-10.5-17.2-10.5-29.4s3.5-22 10.3-29.2c7-7.3 16-11 27-11 9.7 0 17 2.6 22 8V0H313v116.5zm-58.8-38.7c0 13.6 7.5 21.4 18.7 21.4 10.8 0 18.2-7.3 18.2-19.8V76c0-12.2-7.3-19.8-18.3-19.8-11.3 0-18.8 8-18.8 21.6zM354 13.6c0 3.6-1.2 6.8-3.8 9.3-5 4.8-13.6 4.8-18.6 0C323.2 15.3 330-.3 341 .3c7.3 0 13 6 13 13.2zm-2 103h-22V39h22v77.5zM425 47v-8h20.6v80.4c0 11.2-3.6 20-10.6 26.8-7 6.7-16.6 10-28.5 10-23.4 0-37-11.4-40-29.8l21.8-.8c1 7.6 7.6 12 17.4 12 11.2 0 18-5.8 18-16.6v-11c-5 5.4-12.4 8-21.8 8-11 0-20-3.7-27-11s-10.4-17-10.4-29.2 3.5-22 10.3-29.2c7-7.3 16-11 27-11 10.6 0 18.3 3 23 9.5zM387 78c0 13.6 7.5 21.6 18.7 21.6 11 0 18.3-7.6 18.3-19.8V76c0-12.2-7.3-19.8-18.3-19.8-11.2 0-18.7 8-18.7 21.6zM488.8 154.8H465l19.8-45L454.5 39h24l18 46.2L514 39h24.3l-49.7 115.8z" fill="#1a1e24"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.6 KiB

4
website/src/images/logos/ray.svg Normal file
View File

@ -0,0 +1,4 @@
<svg xmlns="http://www.w3.org/2000/svg" width="945.3" height="350" viewBox="90 90 850 300">
<path fill="#231f20" d="M549.5 308.7h18.7L534.6 260c20.7-6.7 32.1-22.7 32.1-46.4 0-30.5-20.6-48-56.4-48h-54.4v143h16.3v-45.1h38c3.3 0 6.6-.2 9.7-.4zm-77.3-61v-66.4h38c26 0 40.8 11 40.8 32.7 0 22.3-14.8 33.7-40.7 33.7zm224.2 24.8l16 36.2H730l-64-143.2h-17L585 308.7h17.4l16-36.2zm-7-15.8h-64.2l32-72.6zm106.6 3l54.4-94.2h-16.8l-45.8 74.7-46.4-74.7h-17l55.4 94.7v48.5H796v-49"/>
<path fill="#00adef" d="M204.3 227.2a40.5 40.5 0 0178.7 0H313c1-3.9 2.5-7.6 4.5-11l-53-52.8a40.4 40.4 0 01-49.4-6.2A40.4 40.4 0 01243.7 88a40.5 40.5 0 0134.8 61.4l53 53a40.4 40.4 0 0149.4 6.1 40.5 40.5 0 01-28.7 69.3c-7.6 0-14.7-2.1-20.8-5.8l-53 53a40.4 40.4 0 01-6 49.5 40.5 40.5 0 01-69.3-28.7 40.4 40.4 0 0161.4-34.8l52.9-53c-2-3.3-3.5-7-4.5-10.9H283a40.5 40.5 0 01-78.7 0h-29.9a40.5 40.5 0 110-19.8zM229 360.4a20.7 20.7 0 0035.4-14.7A20.7 20.7 0 00229 331a20.7 20.7 0 000 29.4zm138-108.6a20.7 20.7 0 00-14.8-35.4 20.7 20.7 0 00-14.6 35.4 20.7 20.7 0 0029.3 0zm-108.7-138a20.7 20.7 0 00-35.4 14.7 20.7 20.7 0 0035.4 14.7 20.7 20.7 0 000-29.4zm-138 108.6a20.7 20.7 0 0014.8 35.5 20.7 20.7 0 0014.6-35.5 20.7 20.7 0 00-29.3 0zm108.7 0a20.7 20.7 0 0014.7 35.5 20.7 20.7 0 0014.6-35.5 20.7 20.7 0 00-29.3 0"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.3 KiB

14
website/src/images/logos/streamlit.svg Normal file
View File

@ -0,0 +1,14 @@
<svg xmlns="http://www.w3.org/2000/svg" width="800" height="200" viewBox="130 250 550 100">
<path fill="#262730" d="M300.4 316.8h8c1 8.4 4.3 15.4 18.2 15.4 8.8 0 15.6-5.3 15.6-13.2 0-8-3.6-11.3-16.6-13.5-14.4-2.3-23-6.9-23-19.3 0-10.8 9.2-19.1 22.1-19.1 13.8 0 22.3 6.9 23.6 19.4H341c-1.7-8.9-6.7-12.5-16.2-12.5-9.4 0-14.3 4.8-14.3 11.4 0 6.8 2.5 10.4 16.5 12.5 15.3 2.5 23.4 7.2 23.4 20.5 0 11.8-10.1 20.7-23.7 20.7-18.4 0-25-10.3-26.2-22.3z"/>
<path fill="#262730" d="M362.4 325.4v-33h-7.2v-6.8h7.2v-12.2h8v12.2H382v6.8h-11.7v32.3c0 5 2.2 7.4 6.2 7.4 2.5 0 4.4-.4 6.1-1v6.7c-1.6.6-3.5 1.1-6.8 1.1-9 0-13.5-5.5-13.5-13.4z"/>
<path fill="#262730" d="M390.3 285.6h8v9.2c3.1-5.8 7.5-9.8 16.5-10v7.5c-10 .5-16.5 4.8-16.5 17.4v28.5h-8v-52.6z"/>
<path fill="#262730" d="M454.5 307.4c-.4-11.2-6.1-16-14.4-16-8.1 0-13.7 6-15 16h29.4zm-37.9 5.5V311c0-16.8 9.7-26.4 23.5-26.4 11.6 0 22.6 7 22.6 25.8v3.4h-37.8c.3 11.6 5.8 18.5 16 18.5 7.9 0 12.4-3 13.4-8.9h8c-1.7 10.2-10 15.6-21.4 15.6-14.2 0-24.3-10.5-24.3-26.2z"/>
<path fill="#262730" d="M500.8 320.4v-6.8H494c-10.9 0-17.5 2.7-17.5 10.2 0 5.2 2.5 8.9 9.5 8.9 8.5 0 15-4.4 15-12.3zm-32.3 3.4c0-11.8 11.3-16 25.1-16h7.2V303c0-8-3-11.5-10.7-11.5-7 0-10.8 3-11.6 9.2h-8c1.1-11.6 10.3-15.9 20-15.9 9.6 0 18.3 3 18.3 17.2v36.3h-8v-6.4a18 18 0 01-15.6 7.3c-9.3 0-16.7-4.9-16.7-15.3z"/>
<path fill="#262730" d="M519.4 285.6h8v7.8c2.2-4.8 7.7-8.7 15-8.7 6.3 0 12 2.7 14.6 9.6 3.5-6.7 11-9.6 17.2-9.6 9 0 16.9 5.5 16.9 20.3v33.2h-8v-33.8c0-9-3.7-12.7-10.6-12.7a13 13 0 00-13.3 13.7v32.8h-8v-33.8c0-9-3.7-12.7-10.6-12.7a13 13 0 00-13.3 13.7v32.8h-8v-52.6z"/>
<path fill="#262730" d="M602.1 338.2h8v-72.9h-8v73z"/>
<path fill="#262730" d="M623 338.2h8v-52.6h-8v52.6zm-1.4-67.5c0-3 2.4-5.4 5.3-5.4a5.4 5.4 0 010 10.8 5.4 5.4 0 01-5.3-5.4z"/>
<path fill="#262730" d="M646.6 325.4v-33h-7.2v-6.8h7.2v-12.2h8v12.2h11.7v6.8h-11.7v32.3c0 5 2.2 7.4 6.2 7.4 2.5 0 4.5-.4 6.2-1v6.7c-1.7.6-3.6 1.1-6.8 1.1-9.2 0-13.6-5.5-13.6-13.4z"/>
<path fill="#FFC7C7" d="M276.4 276v.1l-22.2 57.7c-1.5 2.7-3.7 4.4-7 4.4h-84.7c-2 0-4.2-1-5.3-2l116.4-62.5c1.6-1 3.3.6 2.8 2.3z"/>
<path fill="#FF8C8C" d="M254.1 333.8c-1.5 2.8-3.7 4.4-7 4.4h-84.6c-3 0-5.8-1.6-7.1-4.4l47-71.3c1.1-1.6 3.5-1.6 4.7 0l47 71.3z"/>
<path fill="#FF4B4B" d="M252.4 336.2c-1 1.1-2.9 2-5.2 2h-84.7c-3 0-6-1.5-7.1-4.4L133 276c-.6-1.6 1-3.2 2.7-2.5h.1l116.5 62.6z"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.3 KiB

28
website/src/images/logos/wandb.svg Normal file
View File

@ -0,0 +1,28 @@
<svg xmlns="http://www.w3.org/2000/svg" width="700" height="109" fill="none" viewBox="0 0 700 109">
<path fill="#000" d="M181.2 83.4h3.2l12.8-47.1 6.3-1v-3h-17.1v3l6.9 1-9.6 35-11.3-35 7.6-1v-3h-20.5v3l7.5 1L155.5 71l-9.3-34.8 7-.8v-3.1h-20.6v3l6.3.8 13.6 47.3h3.3L168 44l13 39.3z"/>
<path fill="#000" d="M219.2 49.4c4.6 0 8.2 3.9 8.2 8.2 0 2.2-.8 4-4.5 4h-13.7c.8-8.8 5.7-12.2 10-12.2zm.4-3.5c-9.4 0-17.4 7.8-17.4 19.3 0 11.6 6.7 18.6 16.9 18.6 6.9 0 11.7-3.4 14.5-8.7l-1.8-1.5C229 76.8 226 79 221.3 79c-7.3 0-12.1-5-12.3-14.2h24.1c.3-1 .6-2.5.6-4.2 0-9.2-6.2-14.6-14-14.6z"/>
<path fill="#000" d="M249 37.6c2.7 0 5-2 5-4.8 0-3-2.3-4.8-5-4.8-3 0-5.1 1.8-5.1 4.8 0 2.8 2.1 4.8 5 4.8zm8.5 42.5l-5.3-1-.1-12.1V57l.2-10.3-1.1-.8-11.7 4.7v2.6l6 .7c.2 3 .3 5.4.3 9.5V67l-.1 12-5.7 1.1v2.7h17.5v-2.7z"/>
<path fill="#000" d="M278 49c4.8 0 7.5 3.8 7.5 9.1 0 5.6-2.9 9.6-7.7 9.6-4.7 0-7.3-3.8-7.3-9.3 0-5.5 2.8-9.4 7.6-9.4zm-13.7 9.3c0 5.2 2.2 8.7 5.8 10.7-3.7 3.5-5 6-5 8.4 0 3 1.6 4.8 4.9 5.7-5.6 2.8-7.4 5.4-7.4 8.7 0 5.3 4.7 9.1 15.6 9.1 12.4 0 18.7-6.5 18.7-13 0-6.2-3.8-9.8-12.2-9.8h-9.9c-3.5 0-4.3-1.3-4.3-3.4 0-1.7.6-3.1 1.8-4.8 1.7.5 3.6.8 5.6.8 7.9 0 13.7-4.4 13.7-12.4 0-2.7-.6-5-1.7-6.8h7.4v-4.8l-1.1-.8-8.6 3a15 15 0 00-9.6-3c-7.9 0-13.7 4.5-13.7 12.4zm14.2 39.1c-6 0-9.7-2-9.7-6.9 0-2.4.8-4.5 3-6.8l3 .2h8.9c5.4 0 7.5 2.6 7.5 5.9 0 4.2-4.4 7.6-12.7 7.6z"/>
<path fill="#000" d="M306.3 67l-.1 12-5.7 1.1v2.7h17.6v-2.7l-5.3-1-.1-12.1V56.3c3.8-3.5 7-5 10.7-5 4.4 0 6.1 2.3 6.1 9.2V67l-.1 12.1-5.5 1v2.7h17.5v-2.7l-5.3-1L336 67v-7c0-10-3.2-14-9.7-14a20 20 0 00-13.8 7V38.3l.3-10.7-1-.6-11.7 3.4V33l6.2.8V67z"/>
<path fill="#000" d="M356.7 74.4V50.9h10.2v-4h-10l.3-10.2h-4.5l-2 10-6.7 1v3.2h6.2V75.2c0 5.9 3.3 8.6 8.6 8.6 3.8 0 7-1.4 8.9-4l-1.8-2c-2 1.4-3.4 2-5 2-2.7 0-4.2-1.6-4.2-5.4z"/>
<path fill="#000" d="M383 83.8c9.7 0 14.7-5 14.7-10.8 0-4.8-2.9-8.3-10.4-11.1l-2.4-.8c-5-2-7-3.2-7-6.5 0-3.1 2.5-5.2 7.1-5.2 1.8 0 3.6.5 5.2 1.3l1.2 6h4l.3-7.7c-3.3-2-6.5-3-10.6-3-8.5 0-13.2 5-13.2 10.6 0 5.1 3.7 8.2 9.2 10.2l3.5 1.3c4.7 1.7 7 3.3 7 6.6 0 3.4-2.7 5.7-8.5 5.7-2.3 0-4.3-.4-6-1.2l-1.5-6.6h-4.2l.3 8.3c3.7 2 7 3 11.3 3z"/>
<path fill="#000" d="M435 49.4c-2.7-3.5-3.3-5.9-3.3-8.6 0-4 3-7 6.8-7 3.7 0 6.6 2.7 6.6 7 0 4.5-3.3 7.8-8 11-.7-.7-1.4-1.5-2-2.4zM469 83l.2-3.4-7.3-1-6.2-6.5a55.5 55.5 0 007.9-17.7l5.7-1v-3H454v3l6 1c-1.4 5.9-3.5 10.8-6.3 15.5-3-3.2-5.7-6-8.7-9.4a335 335 0 01-6-6.7c7.6-4.2 11.1-7.9 11.1-13 0-5.8-4-9.8-11.5-9.8-7 0-12.4 4.1-12.4 10.8 0 4 1.5 7.8 5.2 12.1l.3.4c-7.5 3.8-11.4 8.9-11.4 15.7 0 7.6 6.5 13.7 16 13.7 7.2 0 12-2.8 15.4-6.4l2.6 3c2.5 2.5 4.9 3.4 8.9 3.4 2 0 3.5-.2 5.7-.7zm-27.2-16.8l7.7 8.7c-2.9 2.5-7 4-11.2 4-7.3 0-11.4-5.5-11.4-11a13 13 0 016.6-11.5c2.5 2.9 4.9 5.6 8.3 9.8z"/>
<path fill="#000" d="M500.8 59.4c0 6.6 0 13.2-.2 19.5l-7 .8v3.1h20.8c14.6 0 20.2-7 20.2-13.8 0-6.1-4.2-11-14.3-12.5 8.4-1.7 11.8-6.5 11.8-12 0-7-6-12.2-16.5-12.2h-22v3l7 .9c.2 6.5.2 13 .2 19.5v3.7zm11.3-1c10.4 0 15 3.6 15 10.6 0 6.7-4.7 10.3-14.4 10.3h-4.6c-.2-6.4-.2-13-.2-20.9h4.2zm1.2-22.7c7.7 0 11.6 2.5 11.6 9.4S521 55 512 55h-4c0-6 0-12.7.2-19.2h5.2z"/>
<path fill="#000" d="M549.4 37.6c2.8 0 5-2 5-4.8 0-3-2.2-4.8-5-4.8s-5 1.8-5 4.8c0 2.8 2.2 4.8 5 4.8zm8.6 42.5l-5.2-1-.2-12.1V57l.2-10.3-1.1-.8-11.6 4.7v2.6l6 .7c.2 3 .2 5.4.2 9.5V67l-.1 12-5.7 1.1v2.7H558v-2.7z"/>
<path fill="#000" d="M591.8 83.6c2.7 0 4.8-1 6.6-4l-1.5-1.6c-.8 1-1.7 1.7-3 1.7-1.6 0-2.7-1.2-2.7-4.5V59.1c0-9.5-3.8-13.2-11.7-13.2-7.7 0-13.3 3.7-14.7 9.4.2 2 1.4 3.1 3.5 3.1 2 0 3.3-1.3 4-3.6l1.4-5c1.6-.3 3-.4 4-.4 5 0 7.2 1.8 7.2 9.2v1.9c-3 .6-6.2 1.6-8.6 2.5-10.2 3.7-12.5 7-12.5 11.4 0 6.2 4.6 9.4 10.1 9.4 4.5 0 7-2 11.1-6a6.6 6.6 0 006.8 5.8zm-21.3-10.4c0-2.2.8-5.1 7.4-7.7 1.6-.6 4.3-1.5 7-2.2v11.6c-4.3 3-5.8 4-8.3 4-3.5 0-6-1.7-6-5.7z"/>
<path fill="#000" d="M614 83.8c9.7 0 14.7-5 14.7-10.8 0-4.8-2.9-8.3-10.4-11.1l-2.4-.8c-5.1-2-7-3.2-7-6.5 0-3.1 2.5-5.2 7-5.2 2 0 3.6.5 5.3 1.3l1.2 6h4l.3-7.7c-3.3-2-6.5-3-10.6-3-8.5 0-13.2 5-13.2 10.6 0 5.1 3.6 8.2 9.2 10.2l3.4 1.3c4.8 1.7 7 3.3 7 6.6 0 3.4-2.7 5.7-8.4 5.7-2.3 0-4.3-.4-6.1-1.2l-1.5-6.6h-4.1l.3 8.3c3.7 2 7 3 11.3 3z"/>
<path fill="#000" d="M652 49.4c4.6 0 8.2 3.9 8.2 8.2 0 2.2-.8 4-4.5 4H642c.8-8.8 5.7-12.2 10-12.2zm.4-3.5c-9.5 0-17.4 7.8-17.4 19.3 0 11.6 6.7 18.6 16.9 18.6 6.8 0 11.7-3.4 14.5-8.7l-1.8-1.5c-2.8 3.2-5.7 5.3-10.5 5.3-7.3 0-12.1-5-12.3-14.2H666c.3-1 .6-2.5.6-4.2 0-9.2-6.2-14.6-14.1-14.6z"/>
<path fill="#000" d="M685.3 83.8c9.7 0 14.7-5 14.7-10.8 0-4.8-2.9-8.3-10.4-11.1l-2.4-.8c-5.1-2-7-3.2-7-6.5 0-3.1 2.5-5.2 7.1-5.2 1.8 0 3.6.5 5.2 1.3l1.2 6h4l.3-7.7c-3.3-2-6.5-3-10.6-3-8.5 0-13.2 5-13.2 10.6 0 5.1 3.7 8.2 9.2 10.2l3.4 1.3c4.8 1.7 7 3.3 7 6.6 0 3.4-2.6 5.7-8.4 5.7-2.3 0-4.3-.4-6.1-1.2l-1.4-6.6h-4.2l.3 8.3c3.7 2 7 3 11.3 3z"/>
<path fill="#FC3" d="M0 35.1a11.2 11.2 0 1022.5 0 11.2 11.2 0 00-22.5 0z"/>
<path fill="#FC3" d="M0 91.3a11.2 11.2 0 1022.5 0 11.2 11.2 0 00-22.5 0z"/>
<path fill="#FC3" d="M4.2 63.2a7 7 0 1014 0 7 7 0 00-14 0z"/>
<path fill="#FC3" d="M4.2 7a7 7 0 1014 0 7 7 0 00-14 0z"/>
<path fill="#FC3" d="M43.1 73.6a11.2 11.2 0 1022.5 0 11.2 11.2 0 00-22.5 0z"/>
<path fill="#FC3" d="M47.3 101.7a7 7 0 1014 0 7 7 0 00-14 0z"/>
<path fill="#FC3" d="M47.3 45.5a7 7 0 1014 0 7 7 0 00-14 0z"/>
<path fill="#FC3" d="M47.3 17.4a7 7 0 1014 0 7 7 0 00-14 0z"/>
<path fill="#FC3" d="M86.2 35.1a11.2 11.2 0 1022.5 0 11.2 11.2 0 00-22.5 0z"/>
<path fill="#FC3" d="M90.5 7a7 7 0 1014 0 7 7 0 00-14 0z"/>
<path fill="#FC3" d="M90.5 63.2a7 7 0 1014 0 7 7 0 00-14 0z"/>
<path fill="#FC3" d="M90.5 91.3a7 7 0 1014 0 7 7 0 00-14 0z"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 5.4 KiB

9
website/src/styles/card.module.sass
View File

@ -5,6 +5,15 @@
font: var(--font-size-md)/var(--line-height-md) var(--font-primary) font: var(--font-size-md)/var(--line-height-md) var(--font-primary)
margin-bottom: var(--spacing-sm) margin-bottom: var(--spacing-sm)
.small
padding: 1.5rem
font-size: var(--font-size-sm)
line-height: var(--line-height-sm)
color: var(--color-dark)
.title
margin-bottom: var(--spacing-xs)
.image .image
$image-size: 35px $image-size: 35px
width: $image-size width: $image-size

3
website/src/templates/index.js
View File

@ -33,6 +33,7 @@ import { YouTube, SoundCloud, Iframe, Image } from '../components/embed'
import Alert from '../components/alert' import Alert from '../components/alert'
import Search from '../components/search' import Search from '../components/search'
import Project from '../widgets/project' import Project from '../widgets/project'
import { Integration, IntegrationLogo } from '../widgets/integration'
const mdxComponents = { const mdxComponents = {
a: Link, a: Link,
@ -75,6 +76,8 @@ const scopeComponents = {
Grid, Grid,
InlineCode, InlineCode,
Project, Project,
Integration,
IntegrationLogo,
} }
const AlertSpace = ({ nightly }) => { const AlertSpace = ({ nightly }) => {

46
website/src/widgets/integration.js Normal file
View File

@ -0,0 +1,46 @@
import React from 'react'
import Card from '../components/card'
import { ReactComponent as DVCLogo } from '../images/logos/dvc.svg'
import { ReactComponent as ProdigyLogo } from '../images/logos/prodigy.svg'
import { ReactComponent as StreamlitLogo } from '../images/logos/streamlit.svg'
import { ReactComponent as FastAPILogo } from '../images/logos/fastapi.svg'
import { ReactComponent as WandBLogo } from '../images/logos/wandb.svg'
import { ReactComponent as RayLogo } from '../images/logos/ray.svg'
const LOGOS = {
dvc: DVCLogo,
prodigy: ProdigyLogo,
streamlit: StreamlitLogo,
fastapi: FastAPILogo,
wandb: WandBLogo,
ray: RayLogo,
}
export const IntegrationLogo = ({ name, title, width, height, maxWidth, align, ...props }) => {
const Logo = LOGOS[name]
if (!Logo) throw new Error(`Unknown logo: ${name}`)
const style = { maxWidth, float: align || 'none' }
return (
<Logo
aria-label={title}
aria-hidden={title ? undefined : 'true'}
width={width}
height={height}
style={style}
{...props}
/>
)
}
export const Integration = ({ height = 30, url, logo, title, children }) => {
const header = logo && (
<IntegrationLogo name={logo} title={title} height={height} width="auto" maxWidth="80%" />
)
return (
<Card title={header} to={url} small>
{children}
</Card>
)
}