spaCy/website
Daniël de Kok 5e297aa20e
Add TrainablePipe.{distill,get_teacher_student_loss} (#12016)
* Add `TrainablePipe.{distill,get_teacher_student_loss}`

This change adds two methods:

- `TrainablePipe::distill` which performs a training step of a
   student pipe on a teacher pipe, giving a batch of `Doc`s.
- `TrainablePipe::get_teacher_student_loss` computes the loss
  of a student relative to the teacher.

The `distill` or `get_teacher_student_loss` methods are also implemented
in the tagger, edit tree lemmatizer, and parser pipes, to enable
distillation in those pipes and as an example for other pipes.

* Fix stray `Beam` import

* Fix incorrect import

* Apply suggestions from code review

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* TrainablePipe.distill: use `Iterable[Example]`

* Add Pipe.is_distillable method

* Add `validate_distillation_examples`

This first calls `validate_examples` and then checks that the
student/teacher tokens are the same.

* Update distill documentation

* Add distill documentation for all pipes that support distillation

* Fix incorrect identifier

* Apply suggestions from code review

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Add comment to explain `is_distillable`

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
2023-01-16 10:25:53 +01:00
..
.vscode Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
docs Add TrainablePipe.{distill,get_teacher_student_loss} (#12016) 2023-01-16 10:25:53 +01:00
meta Merge branch 'copy_master' into copy_v4 2023-01-11 18:40:55 +01:00
pages Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
plugins Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
public Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
setup Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
src Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
.eslintrc Tidy up website and add eslint config [ci skip] 2019-03-12 15:21:58 +01:00
.eslintrc.json Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
.gitignore Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
.nvmrc Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
.prettierignore Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
.prettierrc Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
Dockerfile Docker Image for Website Dev (#10098) 2022-01-20 23:02:13 +01:00
netlify.toml Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
next-sitemap.config.mjs Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
next.config.mjs Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
package-lock.json Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
package.json Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
README.md Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
runtime.txt Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
tsconfig.json Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
UNIVERSE.md Merge branch 'copy_master' into copy_v4 2023-01-11 18:40:55 +01:00

spacy.io website and docs

Netlify Status

The styleguide for the spaCy website is available at spacy.io/styleguide.

Setup and installation

# Clone the repository
git clone https://github.com/explosion/spaCy
cd spaCy/website

# Switch to the correct Node version
#
# If you don't have NVM and don't want to use it, you can manually switch to the Node version
# stated in /.nvmrc and skip this step
nvm use

# Install the dependencies
npm install

# Start the development server
npm run dev

If you are planning on making edits to the site, you should also set up the Prettier code formatter. It takes care of formatting Markdown and other files automatically. See here for the available extensions for your code editor. The .prettierrc file in the root defines the settings used in this codebase.

Building & developing the site with Docker

While it shouldn't be necessary and is not recommended you can run this site in a Docker container.

If you'd like to do this, be sure you do not include your local node_modules folder, since there are some dependencies that need to be built for the image system. Rename it before using.

docker run -it \
  -v $(pwd):/spacy-io/website \
  -p 8000:8000 \
  ghcr.io/explosion/spacy-io \
  gatsby develop -H 0.0.0.0

This will allow you to access the built website at http://0.0.0.0:8000/ in your browser, and still edit code in your editor while having the site reflect those changes.

Note: If you're working on a Mac with an M1 processor, you might see segfault errors from qemu if you use the default image. To fix this use the arm64 tagged image in the docker run command (ghcr.io/explosion/spacy-io:arm64).

Building the Docker image

If you'd like to build the image locally, you can do so like this:

docker build -t spacy-io .

This will take some time, so if you want to use the prebuilt image you'll save a bit of time.

Project structure

├── docs                 # the actual markdown content
├── meta                 # JSON-formatted site metadata
|   ├── dynamicMeta.js   # At build time generated meta data
|   ├── languages.json   # supported languages and statistical models
|   ├── sidebars.json    # sidebar navigations for different sections
|   ├── site.json        # general site metadata
|   ├── type-annotations.json # Type annotations
|   └── universe.json    # data for the spaCy universe section
├── pages                # Next router pages
├── public               # static images and other assets
├── setup                # Jinja setup
├── src                  # source
|   ├── components       # React components
|   ├── fonts            # webfonts
|   ├── images           # images used in the layout
|   ├── plugins          # custom plugins to transform Markdown
|   ├── styles           # CSS modules and global styles
|   ├── templates        # page layouts
|   |   ├── docs.js      # layout template for documentation pages
|   |   ├── index.js     # global layout template
|   |   ├── models.js    # layout template for model pages
|   |   └── universe.js  # layout templates for universe
|   └── widgets          # non-reusable components with content, e.g. changelog
├── .eslintrc.json       # ESLint config file
├── .nvmrc               # NVM config file
|                        # (to support "nvm use" to switch to correct Node version)
|
├── .prettierrc          # Prettier config file
├── next.config.mjs      # Next config file
├── package.json         # package settings and dependencies
└── tsconfig.json        # TypeScript config file