spaCy/website/api/cli.jade

669 lines
21 KiB
Plaintext
Raw Normal View History

2017-10-30 21:39:26 +03:00
//- 💫 DOCS > API > COMMAND LINE INTERFACE
include ../_includes/_mixins
2017-03-18 17:24:50 +03:00
p
| As of v1.7.0, spaCy comes with new command line helpers to download and
| link models and show useful debugging information. For a list of available
| commands, type #[code spacy --help].
2017-10-03 15:27:22 +03:00
+h(3, "download") Download
2017-03-18 17:24:50 +03:00
p
2017-10-03 15:27:22 +03:00
| Download #[+a("/usage/models") models] for spaCy. The downloader finds the
2017-03-18 17:24:50 +03:00
| best-matching compatible version, uses pip to download the model as a
| package and automatically creates a
2017-10-03 15:27:22 +03:00
| #[+a("/usage/models#usage") shortcut link] to load the model by name.
2017-03-18 17:24:50 +03:00
| Direct downloads don't perform any compatibility checks and require the
| model name to be specified with its version (e.g., #[code en_core_web_sm-1.2.0]).
+aside("Downloading best practices")
| The #[code download] command is mostly intended as a convenient,
| interactive wrapper it performs compatibility checks and prints
| detailed messages in case things go wrong. It's #[strong not recommended]
| to use this command as part of an automated process. If you know which
| model your project needs, you should consider a
| #[+a("/usage/models#download-pip") direct download via pip], or
| uploading the model to a local PyPi installation and fetching it straight
| from there. This will also allow you to add it as a versioned package
| dependency to your project.
+code(false, "bash", "$").
python -m spacy download [model] [--direct]
2017-03-18 17:24:50 +03:00
+table(["Argument", "Type", "Description"])
+row
+cell #[code model]
+cell positional
+cell Model name or shortcut (#[code en], #[code de], #[code vectors]).
+row
+cell #[code --direct], #[code -d]
+cell flag
+cell Force direct download of exact model version.
+row
+cell #[code --help], #[code -h]
+cell flag
+cell Show help message and available arguments.
+row("foot")
+cell creates
+cell directory, symlink
+cell
| The installed model package in your #[code site-packages]
| directory and a shortcut link as a symlink in #[code spacy/data].
2017-10-03 15:27:22 +03:00
+h(3, "link") Link
2017-03-18 17:24:50 +03:00
p
2017-10-03 15:27:22 +03:00
| Create a #[+a("/usage/models#usage") shortcut link] for a model,
2017-03-18 17:24:50 +03:00
| either a Python package or a local directory. This will let you load
2017-05-21 02:13:05 +03:00
| models from any location using a custom name via
| #[+api("spacy#load") #[code spacy.load()]].
2017-03-18 17:24:50 +03:00
2017-06-04 14:55:00 +03:00
+infobox("Important note")
| In spaCy v1.x, you had to use the model data directory to set up a shortcut
| link for a local path. As of v2.0, spaCy expects all shortcut links to
| be #[strong loadable model packages]. If you want to load a data directory,
| call #[+api("spacy#load") #[code spacy.load()]] or
| #[+api("language#from_disk") #[code Language.from_disk()]] with the path,
| or use the #[+api("cli#package") #[code package]] command to create a
| model package.
+code(false, "bash", "$").
python -m spacy link [origin] [link_name] [--force]
2017-03-18 17:24:50 +03:00
+table(["Argument", "Type", "Description"])
+row
+cell #[code origin]
+cell positional
+cell Model name if package, or path to local directory.
+row
+cell #[code link_name]
+cell positional
+cell Name of the shortcut link to create.
+row
+cell #[code --force], #[code -f]
+cell flag
+cell Force overwriting of existing link.
+row
+cell #[code --help], #[code -h]
+cell flag
+cell Show help message and available arguments.
+row("foot")
+cell creates
+cell symlink
+cell
| A shortcut link of the given name as a symlink in
| #[code spacy/data].
2017-10-03 15:27:22 +03:00
+h(3, "info") Info
2017-03-18 17:24:50 +03:00
p
| Print information about your spaCy installation, models and local setup,
| and generate #[+a("https://en.wikipedia.org/wiki/Markdown") Markdown]-formatted
| markup to copy-paste into #[+a(gh("spacy") + "/issues") GitHub issues].
2017-03-18 20:19:50 +03:00
+code(false, "bash").
python -m spacy info [--markdown]
python -m spacy info [model] [--markdown]
2017-03-18 17:24:50 +03:00
+table(["Argument", "Type", "Description"])
+row
+cell #[code model]
+cell positional
2017-05-20 13:58:05 +03:00
+cell A model, i.e. shortcut link, package name or path (optional).
2017-03-18 17:24:50 +03:00
+row
+cell #[code --markdown], #[code -md]
+cell flag
+cell Print information as Markdown.
+row
+cell #[code --help], #[code -h]
+cell flag
+cell Show help message and available arguments.
2017-03-21 13:19:21 +03:00
+row("foot")
+cell prints
+cell #[code stdout]
+cell Information about your spaCy installation.
2017-10-13 00:36:44 +03:00
+h(3, "validate") Validate
+tag-new(2)
p
| Find all models installed in the current environment (both packages and
| shortcut links) and check whether they are compatible with the currently
| installed version of spaCy. Should be run after upgrading spaCy via
| #[code pip install -U spacy] to ensure that all installed models are
| can be used with the new version. The command is also useful to detect
| out-of-sync model links resulting from links created in different virtual
| environments. It will a list of models, the installed versions, the
| latest compatible version (if out of date) and the commands for updating.
+aside("Automated validation")
| You can also use the #[code validate] command as part of your build
| process or test suite, to ensure all models are up to date before
| proceeding. If incompatible models or shortcut links are found, it will
| return #[code 1].
2017-10-13 00:36:44 +03:00
+code(false, "bash", "$").
python -m spacy validate
2017-10-13 00:36:44 +03:00
+table(["Argument", "Type", "Description"])
+row("foot")
+cell prints
+cell #[code stdout]
+cell Details about the compatibility of your installed models.
2017-10-03 15:27:22 +03:00
+h(3, "convert") Convert
2017-03-21 13:19:21 +03:00
p
2017-10-03 15:27:22 +03:00
| Convert files into spaCy's #[+a("/api/annotation#json-input") JSON format]
| for use with the #[code train] command and other experiment management
2017-10-27 15:37:42 +03:00
| functions. The converter can be specified on the command line, or
| chosen based on the file extension of the input file.
2017-03-21 13:19:21 +03:00
2017-10-03 15:27:22 +03:00
+code(false, "bash", "$", false, false, true).
python -m spacy convert [input_file] [output_dir] [--converter] [--n-sents]
2017-10-27 15:37:42 +03:00
[--morphology]
2017-03-21 13:19:21 +03:00
+table(["Argument", "Type", "Description"])
+row
+cell #[code input_file]
2017-03-21 13:19:21 +03:00
+cell positional
+cell Input file.
2017-03-21 13:19:21 +03:00
+row
+cell #[code output_dir]
+cell positional
+cell Output directory for converted JSON file.
2017-03-21 13:19:21 +03:00
2017-10-27 15:37:42 +03:00
+row
+cell #[code converter], #[code -c]
+cell option
+cell #[+tag-new(2)] Name of converter to use (see below).
2017-03-21 13:19:21 +03:00
+row
2017-05-21 02:13:05 +03:00
+cell #[code --n-sents], #[code -n]
+cell option
+cell Number of sentences per document.
+row
+cell #[code --morphology], #[code -m]
+cell option
+cell Enable appending morphology to tags.
+row
+cell #[code --help], #[code -h]
2017-03-21 13:19:21 +03:00
+cell flag
+cell Show help message and available arguments.
+row("foot")
+cell creates
+cell JSON
+cell Data in spaCy's #[+a("/api/annotation#json-input") JSON format].
2017-10-27 15:37:42 +03:00
p The following converters are available:
+table(["ID", "Description"])
+row
+cell #[code auto]
+cell Automatically pick converter based on file extension (default).
+row
+cell #[code conllu], #[code conll]
+cell Universal Dependencies #[code .conllu] or #[code .conll] format.
+row
+cell #[code ner]
+cell Tab-based named entity recognition format.
+row
+cell #[code iob]
+cell IOB named entity recognition format.
2017-10-03 15:27:22 +03:00
+h(3, "train") Train
2017-03-26 16:33:48 +03:00
p
| Train a model. Expects data in spaCy's
2017-10-03 15:27:22 +03:00
| #[+a("/api/annotation#json-input") JSON format]. On each epoch, a model
| will be saved out to the directory. Accuracy scores and model details
| will be added to a #[+a("/usage/training#models-generating") #[code meta.json]]
| to allow packaging the model using the
| #[+api("cli#package") #[code package]] command.
2017-03-26 16:33:48 +03:00
2017-10-03 15:27:22 +03:00
+code(false, "bash", "$", false, false, true).
python -m spacy train [lang] [output_dir] [train_data] [dev_data] [--n-iter]
[--n-sents] [--use-gpu] [--meta-path] [--vectors] [--no-tagger] [--no-parser]
[--no-entities] [--gold-preproc]
2017-03-26 16:33:48 +03:00
+table(["Argument", "Type", "Description"])
+row
+cell #[code lang]
+cell positional
+cell Model language.
+row
+cell #[code output_dir]
+cell positional
+cell Directory to store model in.
+row
+cell #[code train_data]
+cell positional
+cell Location of JSON-formatted training data.
+row
+cell #[code dev_data]
+cell positional
+cell Location of JSON-formatted development data for evaluation.
2017-03-26 16:33:48 +03:00
+row
2017-05-21 02:13:05 +03:00
+cell #[code --n-iter], #[code -n]
2017-03-26 16:33:48 +03:00
+cell option
+cell Number of iterations (default: #[code 20]).
2017-03-26 16:33:48 +03:00
+row
+cell #[code --n-sents], #[code -ns]
2017-05-21 02:13:05 +03:00
+cell option
+cell Number of sentences (default: #[code 0]).
2017-03-26 16:33:48 +03:00
+row
2017-06-05 00:45:14 +03:00
+cell #[code --use-gpu], #[code -g]
+cell option
2017-05-21 02:13:05 +03:00
+cell Use GPU.
2017-10-03 15:27:22 +03:00
+row
+cell #[code --vectors], #[code -v]
+cell option
+cell Model to load vectors from.
+row
+cell #[code --meta-path], #[code -m]
+cell option
+cell
| #[+tag-new(2)] Optional path to model
| #[+a("/usage/training#models-generating") #[code meta.json]].
| All relevant properties like #[code lang], #[code pipeline] and
| #[code spacy_version] will be overwritten.
+row
+cell #[code --version], #[code -V]
+cell option
+cell
| Model version. Will be written out to the model's
| #[code meta.json] after training.
2017-05-21 02:13:05 +03:00
+row
+cell #[code --no-tagger], #[code -T]
2017-03-26 16:33:48 +03:00
+cell flag
+cell Don't train tagger.
+row
2017-05-21 02:13:05 +03:00
+cell #[code --no-parser], #[code -P]
2017-03-26 16:33:48 +03:00
+cell flag
+cell Don't train parser.
+row
+cell #[code --no-entities], #[code -N]
2017-03-26 16:33:48 +03:00
+cell flag
+cell Don't train NER.
2017-10-03 15:27:22 +03:00
+row
+cell #[code --gold-preproc], #[code -G]
+cell flag
+cell Use gold preprocessing.
2017-03-26 16:33:48 +03:00
+row
+cell #[code --help], #[code -h]
+cell flag
+cell Show help message and available arguments.
+row("foot")
+cell creates
+cell model, pickle
+cell A spaCy model on each epoch, and a final #[code .pickle] file.
2017-10-03 15:27:22 +03:00
+h(4, "train-hyperparams") Environment variables for hyperparameters
+tag-new(2)
p
| spaCy lets you set hyperparameters for training via environment variables.
| This is useful, because it keeps the command simple and allows you to
| #[+a("https://askubuntu.com/questions/17536/how-do-i-create-a-permanent-bash-alias/17537#17537") create an alias]
| for your custom #[code train] command while still being able to easily
| tweak the hyperparameters. For example:
+code(false, "bash", "$").
parser_hidden_depth=2 parser_maxout_pieces=1 spacy train [...]
+code("Usage with alias", "bash", "$").
alias train-parser="spacy train en /output /data /train /dev -n 1000"
parser_maxout_pieces=1 train-parser
+table(["Name", "Description", "Default"])
+row
+cell #[code dropout_from]
2017-10-03 15:27:22 +03:00
+cell Initial dropout rate.
+cell #[code 0.2]
+row
+cell #[code dropout_to]
2017-10-03 15:27:22 +03:00
+cell Final dropout rate.
+cell #[code 0.2]
+row
+cell #[code dropout_decay]
2017-10-03 15:27:22 +03:00
+cell Rate of dropout change.
+cell #[code 0.0]
+row
+cell #[code batch_from]
2017-10-03 15:27:22 +03:00
+cell Initial batch size.
+cell #[code 1]
+row
+cell #[code batch_to]
2017-10-03 15:27:22 +03:00
+cell Final batch size.
+cell #[code 64]
+row
+cell #[code batch_compound]
2017-10-03 15:27:22 +03:00
+cell Rate of batch size acceleration.
+cell #[code 1.001]
+row
+cell #[code token_vector_width]
2017-10-03 15:27:22 +03:00
+cell Width of embedding tables and convolutional layers.
+cell #[code 128]
+row
+cell #[code embed_size]
2017-10-03 15:27:22 +03:00
+cell Number of rows in embedding tables.
+cell #[code 7500]
//- +row
//- +cell #[code parser_maxout_pieces]
//- +cell Number of pieces in the parser's and NER's first maxout layer.
//- +cell #[code 2]
//- +row
//- +cell #[code parser_hidden_depth]
//- +cell Number of hidden layers in the parser and NER.
//- +cell #[code 1]
+row
+cell #[code hidden_width]
2017-10-03 15:27:22 +03:00
+cell Size of the parser's and NER's hidden layers.
+cell #[code 128]
//- +row
//- +cell #[code history_feats]
//- +cell Number of previous action ID features for parser and NER.
//- +cell #[code 128]
2017-10-07 15:10:10 +03:00
//- +row
//- +cell #[code history_width]
//- +cell Number of embedding dimensions for each action ID.
//- +cell #[code 128]
2017-10-07 15:10:10 +03:00
+row
+cell #[code learn_rate]
2017-10-03 15:27:22 +03:00
+cell Learning rate.
+cell #[code 0.001]
+row
+cell #[code optimizer_B1]
2017-10-03 15:27:22 +03:00
+cell Momentum for the Adam solver.
+cell #[code 0.9]
+row
+cell #[code optimizer_B2]
2017-10-03 15:27:22 +03:00
+cell Adagrad-momentum for the Adam solver.
+cell #[code 0.999]
+row
+cell #[code optimizer_eps]
2017-10-03 15:27:22 +03:00
+cell Epsylon value for the Adam solver.
+cell #[code 1e-08]
+row
+cell #[code L2_penalty]
2017-10-03 15:27:22 +03:00
+cell L2 regularisation penalty.
+cell #[code 1e-06]
+row
+cell #[code grad_norm_clip]
2017-10-03 15:27:22 +03:00
+cell Gradient L2 norm constraint.
+cell #[code 1.0]
+h(3, "vocab") Vocab
+tag-new(2)
p
2017-10-30 21:39:26 +03:00
| Compile a vocabulary from a
| #[+a("/api/annotation#vocab-jsonl") lexicon JSONL] file and optional
| word vectors. Will save out a valid spaCy model that you can load via
| #[+api("spacy#load") #[code spacy.load]] or package using the
| #[+api("cli#package") #[code package]] command.
+code(false, "bash", "$").
python -m spacy vocab [lang] [output_dir] [lexemes_loc] [vectors_loc]
+table(["Argument", "Type", "Description"])
+row
+cell #[code lang]
+cell positional
+cell
| Model language
| #[+a("https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes") ISO code],
| e.g. #[code en].
+row
+cell #[code output_dir]
+cell positional
+cell Model output directory. Will be created if it doesn't exist.
+row
+cell #[code lexemes_loc]
+cell positional
2017-10-30 21:39:26 +03:00
+cell
| Location of lexical data in spaCy's
| #[+a("/api/annotation#vocab-jsonl") JSONL format].
+row
+cell #[code vectors_loc]
+cell positional
+cell Optional location of vectors data as numpy #[code .npz] file.
+row("foot")
+cell creates
2017-12-07 12:14:37 +03:00
+cell model
+cell A spaCy model containing the vocab and vectors.
+h(3, "init-model") Init Model
+tag-new(2)
p
| Create a new model directory from raw data, like word frequencies, Brown
| clusters and word vectors. This command is similar to the
| #[code spacy model] command in v1.x.
+code(false, "bash", "$", false, false, true).
python -m spacy init-model [lang] [output_dir] [freqs_loc] [--clusters-loc] [--vectors-loc] [--prune-vectors]
+table(["Argument", "Type", "Description"])
+row
+cell #[code lang]
+cell positional
+cell
| Model language
| #[+a("https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes") ISO code],
| e.g. #[code en].
+row
+cell #[code output_dir]
+cell positional
+cell Model output directory. Will be created if it doesn't exist.
+row
+cell #[code freqs_loc]
+cell positional
+cell
| Location of word frequencies file. Should be a tab-separated
| file with three columns: frequency, document frequency and
| frequency count.
+row
+cell #[code --clusters-loc], #[code -c]
+cell option
+cell
| Optional location of clusters file. Should be a tab-separated
| file with three columns: cluster, word and frequency.
+row
+cell #[code --vectors-loc], #[code -v]
+cell option
+cell
| Optional location of vectors file. Should be a tab-separated
2018-04-10 22:42:54 +03:00
| file in Word2Vec format where the first column contains the word
| and the remaining columns the values. File can be provided in
| #[code .txt] format or as a zipped text file in #[code .zip] or
| #[code .tar.gz] format.
2017-12-07 12:14:37 +03:00
+row
+cell #[code --prune-vectors], #[code -V]
+cell flag
+cell
| Number of vectors to prune the vocabulary to. Defaults to
| #[code -1] for no pruning.
+row("foot")
+cell creates
+cell model
+cell A spaCy model containing the vocab and vectors.
+h(3, "evaluate") Evaluate
+tag-new(2)
p
| Evaluate a model's accuracy and speed on JSON-formatted annotated data.
| Will print the results and optionally export
| #[+a("/usage/visualizers") displaCy visualizations] of a sample set of
| parses to #[code .html] files. Visualizations for the dependency parse
| and NER will be exported as separate files if the respective component
| is present in the model's pipeline.
+code(false, "bash", "$", false, false, true).
python -m spacy evaluate [model] [data_path] [--displacy-path] [--displacy-limit] [--gpu-id] [--gold-preproc]
+table(["Argument", "Type", "Description"])
+row
+cell #[code model]
+cell positional
+cell
| Model to evaluate. Can be a package or shortcut link name, or a
| path to a model data directory.
+row
+cell #[code data_path]
+cell positional
+cell Location of JSON-formatted evaluation data.
+row
+cell #[code --displacy-path], #[code -dp]
+cell option
+cell
| Directory to output rendered parses as HTML. If not set, no
| visualizations will be generated.
+row
+cell #[code --displacy-limit], #[code -dl]
+cell option
+cell
| Number of parses to generate per file. Defaults to #[code 25].
| Keep in mind that a significantly higher number might cause the
| #[code .html] files to render slowly.
+row
+cell #[code --gpu-id], #[code -g]
+cell option
+cell GPU to use, if any. Defaults to #[code -1] for CPU.
+row
+cell #[code --gold-preproc], #[code -G]
+cell flag
+cell Use gold preprocessing.
+row("foot")
+cell prints / creates
+cell #[code stdout], HTML
+cell Training results and optional displaCy visualizations.
2017-10-03 15:27:22 +03:00
+h(3, "package") Package
p
2017-10-03 15:27:22 +03:00
| Generate a #[+a("/usage/training#models-generating") model Python package]
2017-04-16 14:37:24 +03:00
| from an existing model data directory. All data files are copied over.
| If the path to a #[code meta.json] is supplied, or a #[code meta.json] is
| found in the input directory, this file is used. Otherwise, the data can
| be entered directly from the command line. After packaging, you can run
| #[code python setup.py sdist] from the newly created directory to turn
| your model into an installable archive file.
2017-10-03 15:27:22 +03:00
+code(false, "bash", "$", false, false, true).
python -m spacy package [input_dir] [output_dir] [--meta-path] [--create-meta] [--force]
+aside-code("Example", "bash").
python -m spacy package /input /output
cd /output/en_model-0.0.0
python setup.py sdist
pip install dist/en_model-0.0.0.tar.gz
+table(["Argument", "Type", "Description"])
+row
+cell #[code input_dir]
+cell positional
+cell Path to directory containing model data.
+row
+cell #[code output_dir]
+cell positional
+cell Directory to create package folder in.
2017-04-16 14:37:24 +03:00
+row
2017-08-14 17:45:42 +03:00
+cell #[code --meta-path], #[code -m]
2017-04-16 14:37:24 +03:00
+cell option
+cell #[+tag-new(2)] Path to #[code meta.json] file (optional).
2017-04-16 14:37:24 +03:00
2017-08-14 17:45:42 +03:00
+row
+cell #[code --create-meta], #[code -c]
+cell flag
+cell
| #[+tag-new(2)] Create a #[code meta.json] file on the command
| line, even if one already exists in the directory. If an
| existing file is found, its entries will be shown as the defaults
| in the command line prompt.
+row
+cell #[code --force], #[code -f]
+cell flag
+cell Force overwriting of existing folder in output directory.
+row
+cell #[code --help], #[code -h]
+cell flag
+cell Show help message and available arguments.
+row("foot")
+cell creates
+cell directory
+cell A Python package containing the spaCy model.