diff --git a/website/docs/usage/_data.json b/website/docs/usage/_data.json index 4621ec8c2..436d14abe 100644 --- a/website/docs/usage/_data.json +++ b/website/docs/usage/_data.json @@ -2,6 +2,7 @@ "sidebar": { "Get started": { "Installation": "./", + "Models": "models", "Lightning tour": "lightning-tour", "Resources": "resources" }, @@ -28,6 +29,11 @@ "index": { "title": "Install spaCy", + "next": "models" + }, + + "models": { + "title": "Models", "next": "lightning-tour" }, diff --git a/website/docs/usage/models.jade b/website/docs/usage/models.jade new file mode 100644 index 000000000..ebe55a87d --- /dev/null +++ b/website/docs/usage/models.jade @@ -0,0 +1,274 @@ +//- 💫 DOCS > USAGE > MODELS + +include ../../_includes/_mixins + +p + | As of v1.7.0, models for spaCy can be installed as #[strong Python packages]. + | This means that they're a component of your application, just like any + | other module. They're versioned and can be defined as a dependency in your + | #[code requirements.txt]. Models can be installed from a download URL or + | a local directory, manually or via #[+a("https://pypi.python.org/pypi/pip") pip]. + | Their data can be located anywhere on your file system. To make a model + | available to spaCy, all you need to do is create a "shortcut link", an + | internal alias that tells spaCy where to find the data files for a specific + | model name. + ++aside-code("Quickstart"). + # Install spaCy and download English model + pip install spacy + python -m spacy.download en + + # Usage in Python + import spacy + nlp = spacy.load('en') + doc = nlp(u'This is a sentence.') + ++h(2, "available") Available models + ++table(["Name", "Size", "Description"]) + +row + +cell #[code en_core_web_md] + +cell 340 MB + +cell Vocab, syntax, entities, word vectors + + +row + +cell #[code en_core_web_sm] + +cell 52 MB + +cell Vocab, syntax, entities, word vectors #[+tag default] + + +row + +cell #[code en_vectors_glove_md] + +cell 693 MB + +cell + | #[+a("http://nlp.stanford.edu/projects/glove/") GloVe] Common + | Crawl vectors + + +row + +cell #[code de_core_news_md] + +cell 637 MB + +cell Vocab, syntax, entities, word vectors #[+tag default] + +p + | Models are now available as #[code .tar.gz] archives #[+a(MODELS_URL) from GitHub], + | attached to individual releases. They can be downloaded and loaded manually, + | or using spaCy's #[code download] and #[code link] commands. All models + | follow the naming convention of #[code [language]_[type]_[genre]_[size]]. + ++button(MODELS_URL + "/releases", true, "primary") View models + ++h(2, "download") Downloading models + ++aside("Downloading models in spaCy < v1.7") + | In older versions of spaCy, you can still use the old download commands. + | This will download and install the models into the #[code spacy/data] + | directory. + + +code.o-no-block. + python -m spacy.en.download all + python -m spacy.de.download all + python -m spacy.en.download glove + + | The old models are now available as Python packages, meaning that you can + | also choose to #[+a("#usage-import") import them] as modules. + +p + | The easiest way to download a model is via spaCy's #[code download] + | command. It takes care of finding the best-matching model compatible with + | your spaCy installation. + ++code(false, "bash"). + # out-of-the-box: download best-matching default model + python -m spacy.download en + python -m spacy.download de + + # download best-matching version of specific model for your spaCy installation + python -m spacy.download en_core_web_md + + # download exact model version (doesn't create shortcut link) + python -m spacy.download en_core_web_md-1.2.0 --direct + +p + | The download command will #[+a("#download-pip") install the model] via + | pip, place the package in your #[code site-packages] directory and create + | a #[+a("#usage") shortcut link] that lets you load the model by name. The + | shortcut link will be the same as the model name used in + | #[code spacy.download]. + ++code(false, "bash"). + pip install spacy + python -m spacy.download en + ++code. + import spacy + nlp = spacy.load('en') + doc = nlp(u'This is a sentence.') + ++h(3, "download-pip") Installation via pip + +p + | To download a model directly using #[+a("https://pypi.python.org/pypi/pip") pip], + | simply point #[code pip install] to the URL or local path of the archive + | file. To find the direct link to a model, head over to the + | #[+a(MODELS_URL + "/releases") model releases], right click on the archive + | link and copy it to your clipboard. + ++code(false, "bash"). + # with external URL + pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_md-1.2.0/en_core_web_md-1.2.0.tar.gz + + # with local file + pip install /Users/you/en_core_web_md-1.2.0.tar.gz + +p + | By default, this will install the model into your #[code site-packages] + | directory. You can then create a #[+a("#usage") shortcut link] for your + | model to load it via #[code spacy.load()], or #[+a("usage-import") import it] + | as a Python module. + ++h(3, "download-manual") Manual download and installation + +p + | In some cases, you might prefer downloading the data manually, for + | example to place it into a custom directory. You can download the model + | via your browser from the #[+a(MODELS_URL) latest releases], or configure + | your own download script using the URL of the archive file. The archive + | consists of a model directory that contains another directory with the + | model data. + ++code("Directory structure", "yaml"). + └── en_core_web_md-1.2.0.tar.gz # downloaded archive + ├── meta.json # model meta data + ├── setup.py # setup file for pip installation + └── en_core_web_md # model directory + ├── __init__.py # init for pip installation + ├── meta.json # model meta data + └── en_core_web_md-1.2.0 # model data + +p + | You can place the model data directory anywhere on your local file system. + | To use it with spaCy, simply create a #[+a("#usage") shortcut link] for + | the directory and assign it a name. + ++h(2, "usage") Using models with spaCy + +p + | While previous versions of spaCy required you to maintain a data directory + | containing the models for each installation, you can now choose how and + | where you want to keep your data files. To load the models conveniently + | from within spaCy, you can use the #[code spacy.link] command to create a + | symlink. This lets you set up custom shortcut links for models so you can + | load them by name. + ++code(false, "bash"). + python -m spacy.link [package name or path] [shortcut] [--force] + +p + | The first argument is the package name (if the model was installed via + | pip), or a local path to the the data directory. The second argument is + | the internal name you want to use for the model. Setting the #[code --force] + | flag will overwrite any existing links. + ++code("Examples", "bash"). + # Create link en_default for model en_core_web_md + python -m spacy.link en_core_web_md en_default + + # Create link my_amazing_model for model data in local directory + python -m spacy.link /Users/you/model my_amazing_model + ++h(3, "usage-loading") Loading models + +p To load a model, use #[code spacy.load()] with the model's shortcut link: + ++code. + import spacy + nlp = spacy.load('en_default') + doc = nlp(u'This is a sentence.') + +p + | You can also use the #[code info()] method to print a model's meta data + | before loading it. Each #[code Language] object returned by #[code spacy.load()] + | also exposes the model's meta data as the property #[code meta]: + ++code. + import spacy + spacy.info('en_default') + # JSON-formatted model meta data + + nlp = spacy.load('en_default') + print(nlp.meta['version']) + # 1.2.0 + ++h(3, "usage-import") Importing models as modules + +p + | If you've installed a model via pip, you can also #[code import] it + | directly and then call its #[code load()] method with no arguments: + ++code. + import spacy + import en_core_web_md + + nlp = en_core_web_md.load() + doc = nlp(u'This is a sentence.') + +p This should also work for older models in previous versions of spaCy. + ++h(2, "own-models") Using your own models + +p + | If you've trained your own model, for example for + | #[+a("/docs/usage/adding-languages") additional languages], you can + | create a shortuct link for it by pointing #[code spacy.link] to the + | model's data directory. To allow your model to be downloaded and + | installed via pip, you'll also need to generate a package for it. + ++infobox("Important note") + | The model packages are #[strong not suitable] for the public + | #[+a("https://pypi.python.org") pypi.python.org] directory, which is not + | designed for binary data and files over 50 MB. However, if your company + | is running an internal installation of pypi, publishing your models on + | there can be a convenient solution to share them with your team. + +p The model directory should look like this: + ++code("Directory structure", "yaml"). + └── / + ├── MANIFEST.in # to include meta.json + ├── meta.json # model meta data + ├── setup.py # setup file for pip installation + └── en_core_web_md # model directory + ├── __init__.py # init for pip installation + └── en_core_web_md-1.2.0 # model data + +p + | You can find templates for all files in our + | #[+a(gh("spacy-dev-resouces", "templates/model")) spaCy dev resources]. + | Unless you want to customise installation and loading, the only file + | you'll need to modify is #[code meta.json], which includes the model's + | meta data. It will later be copied into the package and data directory. + ++code("meta.json", "json"). + { + "name": "core_web_md", + "lang": "en", + "version": "1.2.0", + "spacy_version": "1.7.0", + "description": "English model for spaCy", + "author": "Explosion AI", + "email": "contact@explosion.ai", + "license": "MIT" + } + +p + | Keep in mind that the directories need to be named according to the + | naming conventions. The #[code lang] setting is also used to create the + | respective #[code Language] class in spaCy, which will later be returned + | by the model's #[code load()] method. + +p + | To generate the package, run the following command from within the + | directory. This will create a #[code .tar.gz] archive in a directory + | #[code /dist]. + ++code(false, "bash"). + python setup.py sdist