spaCy/website/docs/usage/models.jade
2017-05-22 13:55:01 +02:00

216 lines
9.4 KiB
Plaintext

//- 💫 DOCS > USAGE > MODELS
include ../../_includes/_mixins
p
| As of v1.7.0, models for spaCy can be installed as #[strong Python packages].
| This means that they're a component of your application, just like any
| other module. They're versioned and can be defined as a dependency in your
| #[code requirements.txt]. Models can be installed from a download URL or
| a local directory, manually or via #[+a("https://pypi.python.org/pypi/pip") pip].
| Their data can be located anywhere on your file system.
+aside("Important note")
| If you're upgrading to spaCy v1.7.x or v2.x, you need to
| #[strong download the new models]. If you've trained statistical models
| that use spaCy's annotations, you should #[strong retrain your models]
| after updating spaCy. If you don't retrain, you may suffer train/test
| skew, which might decrease your accuracy.
+quickstart(QUICKSTART_MODELS, "Quickstart", "Install a default model, get the code to load it from within spaCy and an example to test it. For more options, see the section on available models below.")
- var examples = {en: "This is a sentence.", de: "Dies ist ein Satz.", fr: "C'est une phrase."}
for models, lang in MODELS
- var package = (models.length == 1) ? models[0] : models.find(function(m) { return m.def })
+qs({lang: lang}) python -m spacy download #{lang}
+qs({lang: lang}, "divider")
+qs({lang: lang, load: "module"}, "python") import #{package.id}
+qs({lang: lang, load: "module"}, "python") nlp = #{package.id}.load()
+qs({lang: lang, load: "spacy"}, "python") nlp = spacy.load('#{lang}')
+qs({lang: lang, config: "example"}, "python") doc = nlp(u"#{examples[lang]}")
+qs({lang: lang, config: "example"}, "python") print([(w.text, w.pos_) for w in doc])
+h(2, "available") Available models
include _models-list
+h(2, "download") Downloading models
+aside("Downloading models in spaCy < v1.7")
| In older versions of spaCy, you can still use the old download commands.
| This will download and install the models into the #[code spacy/data]
| directory.
+code.o-no-block.
python -m spacy.en.download all
python -m spacy.de.download all
python -m spacy.en.download glove
| The old models are also #[+a(gh("spacy") + "/tree/v1.6.0") attached to the v1.6.0 release].
| To download and install them manually, unpack the archive, drop the
| contained directory into #[code spacy/data].
p
| The easiest way to download a model is via spaCy's
| #[+api("cli#download") #[code download]] command. It takes care of
| finding the best-matching model compatible with your spaCy installation.
- var models = Object.keys(MODELS).map(function(lang) { return "python -m spacy download " + lang })
+code(false, "bash").
# out-of-the-box: download best-matching default model
#{Object.keys(MODELS).map(function(l) {return "python -m spacy download " + l}).join('\n')}
# download best-matching version of specific model for your spaCy installation
python -m spacy download en_core_web_md
# download exact model version (doesn't create shortcut link)
python -m spacy download en_core_web_md-1.2.0 --direct
p
| The download command will #[+a("#download-pip") install the model] via
| pip, place the package in your #[code site-packages] directory and create
| a #[+a("#usage") shortcut link] that lets you load the model by a custom
| name. The shortcut link will be the same as the model name used in
| #[code spacy.download].
+code(false, "bash").
pip install spacy
python -m spacy download en
+code.
import spacy
nlp = spacy.load('en')
doc = nlp(u'This is a sentence.')
+h(3, "download-pip") Installation via pip
p
| To download a model directly using #[+a("https://pypi.python.org/pypi/pip") pip],
| simply point #[code pip install] to the URL or local path of the archive
| file. To find the direct link to a model, head over to the
| #[+a(gh("spacy-models") + "/releases") model releases], right click on the archive
| link and copy it to your clipboard.
+code(false, "bash").
# with external URL
pip install #{gh("spacy-models")}/releases/download/en_core_web_md-1.2.0/en_core_web_md-1.2.0.tar.gz
# with local file
pip install /Users/you/en_core_web_md-1.2.0.tar.gz
p
| By default, this will install the model into your #[code site-packages]
| directory. You can then use #[code spacy.load()] to load it via its
| package name, create a #[+a("#usage-link") shortcut link] to assign it a
| custom name, or #[+a("usage-import") import it] explicitly as a module.
+h(3, "download-manual") Manual download and installation
p
| In some cases, you might prefer downloading the data manually, for
| example to place it into a custom directory. You can download the model
| via your browser from the #[+a(gh("spacy-models")) latest releases], or configure
| your own download script using the URL of the archive file. The archive
| consists of a model directory that contains another directory with the
| model data.
+code("Directory structure", "yaml").
└── en_core_web_md-1.2.0.tar.gz # downloaded archive
├── meta.json # model meta data
├── setup.py # setup file for pip installation
└── en_core_web_md # model directory
├── __init__.py # init for pip installation
├── meta.json # model meta data
└── en_core_web_md-1.2.0 # model data
p
| You can place the model data directory anywhere on your local file system.
| To use it with spaCy, simply assign it a name by creating a
| #[+a("#usage") shortcut link] for the data directory.
+h(2, "usage") Using models with spaCy
p
| To load a model, use #[+api("spacy#load") #[code spacy.load()]] with the
| model's shortcut link, package name or a path to the data directory:
+code.
import spacy
nlp = spacy.load('en') # load model with shortcut link "en"
nlp = spacy.load('en_core_web_sm') # load model package "en_core_web_sm"
nlp = spacy.load('/path/to/model') # load model from a directory
doc = nlp(u'This is a sentence.')
+infobox("Tip: Preview model info")
| You can use the #[+api("cli#info") #[code info]] command or
| #[+api("spacy#info") #[code spacy.info()]] method to print a model's meta data
| before loading it. Each #[code Language] object with a loaded model also
| exposes the model's meta data as the attribute #[code meta]. For example,
| #[code nlp.meta['version']] will return the model's version.
+h(3, "usage-link") Using custom shortcut links
p
| While previous versions of spaCy required you to maintain a data directory
| containing the models for each installation, you can now choose
| #[strong how and where you want to keep your data]. For example, you could
| download all models manually and put them into a local directory.
| Whenever your spaCy projects need a models, you create a shortcut link to
| tell spaCy to load it from there. This means you'll never end up with
| duplicate data.
p
| The #[+api("cli#link") #[code link]] command will create a symlink
| in the #[code spacy/data] directory.
+code(false, "bash").
python -m spacy link [package name or path] [shortcut] [--force]
p
| The first argument is the package name (if the model was installed via
| pip), or a local path to the the data directory. The second argument is
| the internal name you want to use for the model. Setting the #[code --force]
| flag will overwrite any existing links.
+code("Examples", "bash").
# set up shortcut link to load installed package as "en_default"
python -m spacy link en_core_web_md en_default
# set up shortcut link to load local model as "my_amazing_model"
python -m spacy link /Users/you/model my_amazing_model
+infobox("Important note")
| In order to create a symlink, your user needs the required permissions.
| If you've installed spaCy to a system directory and don't have admin
| privileges, the #[code spacy link] command may fail. The easiest solution
| is to re-run the command as admin, or use a #[code virtualenv]. For more
| info on this, see the
| #[+a("/docs/usage/troubleshooting#symlink-privilege") troubleshooting guide].
+h(3, "usage-import") Importing models as modules
p
| If you've installed a model via pip, you can also #[code import] it
| directly and then call its #[code load()] method with no arguments:
+code.
import spacy
import en_core_web_md
nlp = en_core_web_md.load()
doc = nlp(u'This is a sentence.')
+h(2, "own-models") Using your own models
p
| If you've trained your own model, for example for
| #[+a("/docs/usage/adding-languages") additional languages] or
| #[+a("/docs/usage/train-ner") custom named entities], you can save its
| state using the #[+api("language#to_disk") #[code Language.to_disk()]]
| method. To make the model more convenient to deploy, we recommend
| wrapping it as a Python package.
+infobox("Saving and loading models")
| For more information and a detailed guide on how to package your model,
| see the documentation on
| #[+a("/docs/usage/saving-loading") saving and loading models].