spaCy/website
Raphael Mitsch 304b9331e6
Modify EL batching to doc-wise streaming approach (#12367)
* Convert Candidate from Cython to Python class.

* Format.

* Fix .entity_ typo in _add_activations() usage.

* Change type for mentions to look up entity candidates for to SpanGroup from Iterable[Span].

* Update docs.

* Update spacy/kb/candidate.py

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update doc string of BaseCandidate.__init__().

* Update spacy/kb/candidate.py

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Rename Candidate to InMemoryCandidate, BaseCandidate to Candidate.

* Adjust Candidate to support and mandate numerical entity IDs.

* Format.

* Fix docstring and docs.

* Update website/docs/api/kb.mdx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Rename alias -> mention.

* Refactor Candidate attribute names. Update docs and tests accordingly.

* Refacor Candidate attributes and their usage.

* Format.

* Fix mypy error.

* Update error code in line with v4 convention.

* Modify EL batching system.

* Update leftover get_candidates() mention in docs.

* Format docs.

* Format.

* Update spacy/kb/candidate.py

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Updated error code.

* Simplify interface for int/str representations.

* Update website/docs/api/kb.mdx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Rename 'alias' to 'mention'.

* Port Candidate and InMemoryCandidate to Cython.

* Remove redundant entry in setup.py.

* Add abstract class check.

* Drop storing mention.

* Update spacy/kb/candidate.pxd

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Fix entity_id refactoring problems in docstrings.

* Drop unused InMemoryCandidate._entity_hash.

* Update docstrings.

* Move attributes out of Candidate.

* Partially fix alias/mention terminology usage. Convert Candidate to interface.

* Remove prior_prob from supported properties in Candidate. Introduce KnowledgeBase.supports_prior_probs().

* Update docstrings related to prior_prob.

* Update alias/mention usage in doc(strings).

* Update spacy/ml/models/entity_linker.py

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update spacy/ml/models/entity_linker.py

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Mention -> alias renaming. Drop Candidate.mentions(). Drop InMemoryLookupKB.get_alias_candidates() from docs.

* Update docstrings.

* Fix InMemoryCandidate attribute names.

* Update spacy/kb/kb.pyx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update spacy/ml/models/entity_linker.py

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update W401 test.

* Update spacy/errors.py

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update spacy/kb/kb.pyx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Use Candidate output type for toy generators in the test suite to mimick best practices

* fix docs

* fix import

* Fix merge leftovers.

* Update spacy/kb/kb.pyx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update spacy/kb/kb.pyx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update website/docs/api/kb.mdx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update website/docs/api/entitylinker.mdx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update spacy/kb/kb_in_memory.pyx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update website/docs/api/inmemorylookupkb.mdx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update get_candidates() docstring.

* Reformat imports in entity_linker.py.

* Drop valid_ent_idx_per_doc.

* Update docs.

* Format.

* Simplify doc loop in predict().

* Remove E1044 comment.

* Fix merge errors.

* Format.

* Format.

* Format.

* Fix merge error & tests.

* Format.

* Apply suggestions from code review

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>

* Use type alias.

* isort.

* isort.

* Lint.

* Add typedefs.pyx.

* Fix typedef import.

* Fix type aliases.

* Format.

* Update docstring and type usage.

* Add info on get_candidates(), get_candidates_batched().

* Readd get_candidates info to v3 changelog.

* Update website/docs/api/entitylinker.mdx

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Update factory functions for backwards compatibility.

* Format.

* Ignore mypy error.

* Fix mypy error.

* Format.

* Add test for multiple docs with multiple entities.

---------

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
Co-authored-by: svlandeg <svlandeg@github.com>
2024-04-09 11:39:18 +02:00
..
.vscode Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
docs Modify EL batching to doc-wise streaming approach (#12367) 2024-04-09 11:39:18 +02:00
meta Merge remote-tracking branch 'upstream/master' into maintenance/v4-merge-master-20240119 2024-01-19 12:34:29 +01:00
pages Add spacy-llm docs to website (#12782) 2023-07-24 14:44:47 +02:00
plugins Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
public Inline displaCy visualizations in docs (#13050) [ci skip] 2023-10-06 14:22:43 +02:00
setup Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
src Merge remote-tracking branch 'upstream/master' into maintenance/v4-merge-master-20240119 2024-01-19 12:34:29 +01:00
.dockerignore Update Dockerfile to work with Next.js (#12119) 2023-01-18 18:15:47 +01:00
.eslintrc Tidy up website and add eslint config [ci skip] 2019-03-12 15:21:58 +01:00
.eslintrc.json Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
.gitignore Move all website gitignore settings to website/.gitignore (#12120) 2023-01-18 21:46:19 +01:00
.nvmrc Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
.prettierignore Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
.prettierrc Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
Dockerfile Update Dockerfile to work with Next.js (#12119) 2023-01-18 18:15:47 +01:00
netlify.toml Add headers to netlify.toml [ci skip] 2023-08-30 11:55:23 +02:00
next-sitemap.config.mjs Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
next.config.mjs Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
package-lock.json Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
package.json Make sure to run Python setup before NPM dev mode (#12384) 2023-03-08 11:59:10 +01:00
README.md Update Dockerfile to work with Next.js (#12119) 2023-01-18 18:15:47 +01:00
runtime.txt Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
tsconfig.json Website migration from Gatsby to Next (#12058) 2023-01-11 17:30:07 +01:00
UNIVERSE.md Merge branch 'copy_master' into copy_v4 2023-01-11 18:40:55 +01:00

spacy.io website and docs

Netlify Status

The styleguide for the spaCy website is available at spacy.io/styleguide.

Setup and installation

# Clone the repository
git clone https://github.com/explosion/spaCy
cd spaCy/website

# Switch to the correct Node version
#
# If you don't have NVM and don't want to use it, you can manually switch to the Node version
# stated in /.nvmrc and skip this step
nvm use

# Install the dependencies
npm install

# Start the development server
npm run dev

If you are planning on making edits to the site, you should also set up the Prettier code formatter. It takes care of formatting Markdown and other files automatically. See here for the available extensions for your code editor. The .prettierrc file in the root defines the settings used in this codebase.

Building & developing the site with Docker

While it shouldn't be necessary and is not recommended you can run this site in a Docker container.

If you'd like to do this, be sure you do not include your local node_modules folder, since there are some dependencies that need to be built for the image system. Rename it before using.

First build the Docker image. This only needs to be done on the first run or when changes are made to Dockerfile or the website dependencies:

docker build -t spacy-io .

You can then build and run the website with:

docker run -it \
  --rm \
  -v $(pwd):/home/node/website \
  -p 3000:3000 \
  spacy-io \
  npm run dev -- -H 0.0.0.0

This will allow you to access the built website at http://0.0.0.0:3000/ in your browser, and still edit code in your editor while having the site reflect those changes.

Project structure

├── docs                 # the actual markdown content
├── meta                 # JSON-formatted site metadata
|   ├── dynamicMeta.js   # At build time generated meta data
|   ├── languages.json   # supported languages and statistical models
|   ├── sidebars.json    # sidebar navigations for different sections
|   ├── site.json        # general site metadata
|   ├── type-annotations.json # Type annotations
|   └── universe.json    # data for the spaCy universe section
├── pages                # Next router pages
├── public               # static images and other assets
├── setup                # Jinja setup
├── src                  # source
|   ├── components       # React components
|   ├── fonts            # webfonts
|   ├── images           # images used in the layout
|   ├── plugins          # custom plugins to transform Markdown
|   ├── styles           # CSS modules and global styles
|   ├── templates        # page layouts
|   |   ├── docs.js      # layout template for documentation pages
|   |   ├── index.js     # global layout template
|   |   ├── models.js    # layout template for model pages
|   |   └── universe.js  # layout templates for universe
|   └── widgets          # non-reusable components with content, e.g. changelog
├── .eslintrc.json       # ESLint config file
├── .nvmrc               # NVM config file
|                        # (to support "nvm use" to switch to correct Node version)
|
├── .prettierrc          # Prettier config file
├── next.config.mjs      # Next config file
├── package.json         # package settings and dependencies
└── tsconfig.json        # TypeScript config file