spaCy/website/docs/usage/models.jade

246 lines
9.7 KiB
Plaintext
Raw Normal View History

2017-03-16 23:53:31 +03:00
//- 💫 DOCS > USAGE > MODELS
include ../../_includes/_mixins
p
| As of v1.7.0, models for spaCy can be installed as #[strong Python packages].
| This means that they're a component of your application, just like any
| other module. They're versioned and can be defined as a dependency in your
| #[code requirements.txt]. Models can be installed from a download URL or
| a local directory, manually or via #[+a("https://pypi.python.org/pypi/pip") pip].
| Their data can be located anywhere on your file system. To make a model
| available to spaCy, all you need to do is create a "shortcut link", an
| internal alias that tells spaCy where to find the data files for a specific
| model name.
+aside-code("Quickstart").
# Install spaCy and download English model
pip install spacy
2017-03-18 17:24:42 +03:00
python -m spacy download en
2017-03-16 23:53:31 +03:00
# Usage in Python
import spacy
nlp = spacy.load('en')
doc = nlp(u'This is a sentence.')
+infobox("Important note")
| Due to improvements in the English lemmatizer in v1.7.0, you need to
| #[strong download the new English models]. The German model is still
| compatible. If you've trained statistical models that use spaCy's
| annotations, you should #[strong retrain your models after updating spaCy].
| If you don't retrain your models, you may suffer train/test skew, which
| might decrease your accuracy.
2017-03-16 23:53:31 +03:00
+h(2, "available") Available models
2017-03-16 23:53:31 +03:00
2017-04-26 21:49:48 +03:00
include _models-list
2017-03-16 23:53:31 +03:00
+h(2, "download") Downloading models
+aside("Downloading models in spaCy < v1.7")
| In older versions of spaCy, you can still use the old download commands.
| This will download and install the models into the #[code spacy/data]
| directory.
+code.o-no-block.
python -m spacy.en.download all
python -m spacy.de.download all
python -m spacy.en.download glove
2017-03-17 19:01:16 +03:00
| The old models are also #[+a(gh("spacy") + "/tree/v1.6.0") attached to the v1.6.0 release].
| To download and install them manually, unpack the archive, drop the
| contained directory into #[code spacy/data] and load the model via
| #[code spacy.load('en')] or #[code spacy.load('de')].
2017-03-16 23:53:31 +03:00
p
| The easiest way to download a model is via spaCy's #[code download]
| command. It takes care of finding the best-matching model compatible with
| your spaCy installation.
+code(false, "bash").
# out-of-the-box: download best-matching default model
2017-03-18 17:24:42 +03:00
python -m spacy download en
python -m spacy download de
2017-04-27 13:08:45 +03:00
python -m spacy download fr
2017-03-16 23:53:31 +03:00
# download best-matching version of specific model for your spaCy installation
2017-03-18 17:24:42 +03:00
python -m spacy download en_core_web_md
2017-03-16 23:53:31 +03:00
# download exact model version (doesn't create shortcut link)
python -m spacy download en_core_web_md-1.2.1 --direct
2017-03-16 23:53:31 +03:00
p
| The download command will #[+a("#download-pip") install the model] via
| pip, place the package in your #[code site-packages] directory and create
| a #[+a("#usage") shortcut link] that lets you load the model by name. The
| shortcut link will be the same as the model name used in
| #[code spacy.download].
+code(false, "bash").
pip install spacy
2017-03-18 17:24:42 +03:00
python -m spacy download en
2017-03-16 23:53:31 +03:00
+code.
import spacy
nlp = spacy.load('en')
doc = nlp(u'This is a sentence.')
+h(3, "download-pip") Installation via pip
p
| To download a model directly using #[+a("https://pypi.python.org/pypi/pip") pip],
| simply point #[code pip install] to the URL or local path of the archive
| file. To find the direct link to a model, head over to the
2017-03-17 00:09:43 +03:00
| #[+a(gh("spacy-models") + "/releases") model releases], right click on the archive
2017-03-16 23:53:31 +03:00
| link and copy it to your clipboard.
+code(false, "bash").
# with external URL
pip install #{gh("spacy-models")}/releases/download/en_core_web_md-1.2.1/en_core_web_md-1.2.1.tar.gz
2017-03-16 23:53:31 +03:00
# with local file
pip install /Users/you/en_core_web_md-1.2.1.tar.gz
2017-03-16 23:53:31 +03:00
p
| By default, this will install the model into your #[code site-packages]
| directory. You can then create a #[+a("#usage") shortcut link] for your
| model to load it via #[code spacy.load()], or #[+a("usage-import") import it]
| as a Python module.
+h(3, "download-manual") Manual download and installation
p
| In some cases, you might prefer downloading the data manually, for
| example to place it into a custom directory. You can download the model
2017-03-17 00:09:43 +03:00
| via your browser from the #[+a(gh("spacy-models")) latest releases], or configure
2017-03-16 23:53:31 +03:00
| your own download script using the URL of the archive file. The archive
| consists of a model directory that contains another directory with the
| model data.
+code("Directory structure", "yaml").
└── en_core_web_md-1.2.0.tar.gz # downloaded archive
├── meta.json # model meta data
├── setup.py # setup file for pip installation
└── en_core_web_md # model directory
├── __init__.py # init for pip installation
├── meta.json # model meta data
└── en_core_web_md-1.2.0 # model data
p
| You can place the model data directory anywhere on your local file system.
2017-03-17 18:09:56 +03:00
| To use it with spaCy, simply assign it a name by creating a
| #[+a("#usage") shortcut link] for the data directory.
2017-03-16 23:53:31 +03:00
+h(2, "usage") Using models with spaCy
p
| While previous versions of spaCy required you to maintain a data directory
| containing the models for each installation, you can now choose how and
| where you want to keep your data files. To load the models conveniently
| from within spaCy, you can use the #[code spacy.link] command to create a
| symlink. This lets you set up custom shortcut links for models so you can
| load them by name.
+code(false, "bash").
2017-03-18 17:24:42 +03:00
python -m spacy link [package name or path] [shortcut] [--force]
2017-03-16 23:53:31 +03:00
p
| The first argument is the package name (if the model was installed via
| pip), or a local path to the the data directory. The second argument is
| the internal name you want to use for the model. Setting the #[code --force]
| flag will overwrite any existing links.
+code("Examples", "bash").
2017-03-17 01:23:35 +03:00
# set up shortcut link to load installed package as "en_default"
2017-03-18 17:24:42 +03:00
python -m spacy link en_core_web_md en_default
2017-03-16 23:53:31 +03:00
2017-03-17 00:09:43 +03:00
# set up shortcut link to load local model as "my_amazing_model"
2017-03-18 17:24:42 +03:00
python -m spacy link /Users/you/model my_amazing_model
2017-03-16 23:53:31 +03:00
+h(3, "usage-loading") Loading models
2017-03-17 21:26:37 +03:00
p
| To load a model, use #[code spacy.load()] with the model's shortcut link.
2017-03-16 23:53:31 +03:00
+code.
import spacy
nlp = spacy.load('en_default')
doc = nlp(u'This is a sentence.')
p
2017-03-18 17:24:42 +03:00
| You can also use the #[info] command or #[code info()] method to print a model's meta data
2017-03-16 23:53:31 +03:00
| before loading it. Each #[code Language] object returned by #[code spacy.load()]
2017-03-18 17:24:42 +03:00
| also exposes the model's meta data as the attribute #[code meta].
+code(false, "bash").
python -m spacy info en
# model meta data
2017-03-16 23:53:31 +03:00
+code.
import spacy
spacy.info('en_default')
2017-03-18 17:24:42 +03:00
# model meta data
2017-03-16 23:53:31 +03:00
nlp = spacy.load('en_default')
print(nlp.meta['version'])
# 1.2.0
+h(3, "usage-import") Importing models as modules
p
| If you've installed a model via pip, you can also #[code import] it
| directly and then call its #[code load()] method with no arguments:
+code.
import spacy
import en_core_web_md
nlp = en_core_web_md.load()
doc = nlp(u'This is a sentence.')
+h(3, "models-download") Downloading and requiring model dependencies
p
| spaCy's built-in #[+api("cli#download") #[code download]] command
| is mostly intended as a convenient, interactive wrapper. It performs
| compatibility checks and prints detailed error messages and warnings.
| However, if you're downloading models as part of an automated build
| process, this only adds an unnecessary layer of complexity. If you know
| which models your application needs, you should be specifying them directly.
+aside("Prevent re-downloading models")
| If you're installing a model from a URL, pip will usually re-download and
| re-install the package, even if you already have a matching
| version installed. To prevent this, simply add #[code #egg=] and the
| package name after the URL, e.g. #[code #egg=en_core_web_sm] or
| #[code #egg=en_core_web_sm-1.2.0]. This tells pip which package and version
| you're trying to download, and will skip the package if a matching
| installation is found.
p
| Because all models are valid Python packages, you can add them to your
| application's #[code requirements.txt]. If you're running your own
| internal PyPi installation, you can simply upload the models there. pip's
| #[+a("https://pip.pypa.io/en/latest/reference/pip_install/#requirements-file-format") requirements file format]
| supports both package names to download via a PyPi server, as well as direct
| URLs.
+code("requirements.txt", "text").
spacy&gt;=1.8.0,&lt;2.0.0
-e #{gh("spacy-models")}/releases/download/en_core_web_sm-1.2.0/en_core_web_sm-1.2.0.tar.gz#egg=en_core_web_sm-1.2.0
2017-03-16 23:53:31 +03:00
+h(2, "own-models") Using your own models
p
| If you've trained your own model, for example for
| #[+a("/docs/usage/adding-languages") additional languages] or
| #[+a("/docs/usage/training-ner") custom named entities], you can save its
| state using the #[code Language.save_to_directory()] method. To make the
| model more convenient to deploy, we recommend wrapping it as a Python
| package.
+infobox("Saving and loading models")
| For more information and a detailed guide on how to package your model,
| see the documentation on
| #[+a("/docs/usage/saving-loading") saving and loading models].