spaCy/website/usage/_processing-pipelines/_pipelines.jade

155 lines
7.0 KiB
Plaintext
Raw Normal View History

2017-10-03 15:26:20 +03:00
//- 💫 DOCS > USAGE > PROCESSING PIPELINES > PIPELINES
2016-10-31 21:04:15 +03:00
p
| spaCy makes it very easy to create your own pipelines consisting of
2017-11-01 21:49:04 +03:00
| reusable components this includes spaCy's default tagger,
| parser and entity regcognizer, but also your own custom processing
| functions. A pipeline component can be added to an already existing
| #[code nlp] object, specified when initialising a #[code Language] class,
| or defined within a
| #[+a("/usage/training#saving-loading") model package].
2016-10-31 21:04:15 +03:00
p
| When you load a model, spaCy first consults the model's
| #[+a("/usage/training#saving-loading") #[code meta.json]]. The
| meta typically includes the model details, the ID of a language class,
| and an optional list of pipeline components. spaCy then does the
| following:
+aside-code("meta.json (excerpt)", "json").
{
"name": "example_model",
"lang": "en"
"description": "Example model for spaCy",
2017-10-07 15:05:59 +03:00
"pipeline": ["tagger", "parser"]
}
+list("numbers")
+item
2017-10-07 15:05:59 +03:00
| Load the #[strong language class and data] for the given ID via
| #[+api("top-level#util.get_lang_class") #[code get_lang_class]] and initialise
2017-10-07 15:05:59 +03:00
| it. The #[code Language] class contains the shared vocabulary,
| tokenization rules and the language-specific annotation scheme.
+item
2017-10-07 15:05:59 +03:00
| Iterate over the #[strong pipeline names] and create each component
| using #[+api("language#create_pipe") #[code create_pipe]], which
| looks them up in #[code Language.factories].
+item
2017-10-07 15:05:59 +03:00
| Add each pipeline component to the pipeline in order, using
| #[+api("language#add_pipe") #[code add_pipe]].
+item
2017-10-07 15:05:59 +03:00
| Make the #[strong model data] available to the #[code Language] class
| by calling #[+api("language#from_disk") #[code from_disk]] with the
| path to the model data ditectory.
p
| So when you call this...
2016-10-31 21:04:15 +03:00
+code.
nlp = spacy.load('en')
2016-10-31 21:04:15 +03:00
p
2017-11-01 21:49:04 +03:00
| ... the model tells spaCy to use the language #[code "en"] and the
| pipeline #[code.u-break ["tagger", "parser", "ner"]]. spaCy will then
| initialise #[code spacy.lang.en.English], and create each pipeline
2017-10-07 15:05:59 +03:00
| component and add it to the processing pipeline. It'll then load in the
| model's data from its data ditectory and return the modified
| #[code Language] class for you to use as the #[code nlp] object.
2017-10-03 15:26:20 +03:00
p
| Fundamentally, a #[+a("/models") spaCy model] consists of three
| components: #[strong the weights], i.e. binary data loaded in from a
| directory, a #[strong pipeline] of functions called in order,
| and #[strong language data] like the tokenization rules and annotation
| scheme. All of this is specific to each model, and defined in the
| model's #[code meta.json] for example, a Spanish NER model requires
| different weights, language data and pipeline components than an English
| parsing and tagging model. This is also why the pipeline state is always
| held by the #[code Language] class.
| #[+api("spacy#load") #[code spacy.load]] puts this all together and
| returns an instance of #[code Language] with a pipeline set and access
| to the binary data:
+code("spacy.load under the hood").
lang = 'en'
2017-11-01 21:49:04 +03:00
pipeline = ['tagger', 'parser', 'ner']
2017-10-03 15:26:20 +03:00
data_path = 'path/to/en_core_web_sm/en_core_web_sm-2.0.0'
2017-10-07 15:05:59 +03:00
cls = spacy.util.get_lang_class(lang) # 1. get Language instance, e.g. English()
nlp = cls() # 2. initialise it
for name in pipeline:
component = nlp.create_pipe(name) # 3. create the pipeline components
nlp.add_pipe(component) # 4. add the component to the pipeline
nlp.from_disk(model_data_path) # 5. load in the binary data
2016-10-31 21:04:15 +03:00
p
| When you call #[code nlp] on a text, spaCy will #[strong tokenize] it and
| then #[strong call each component] on the #[code Doc], in order.
2017-10-03 15:26:20 +03:00
| Since the model data is loaded, the components can access it to assign
| annotations to the #[code Doc] object, and subsequently to the
| #[code Token] and #[code Span] which are only views of the #[code Doc],
| and don't own any data themselves. All components return the modified
| document, which is then processed by the component next in the pipeline.
2016-10-31 21:04:15 +03:00
+code("The pipeline under the hood").
2017-10-07 15:05:59 +03:00
doc = nlp.make_doc(u'This is a sentence') # create a Doc from raw text
for name, proc in nlp.pipeline: # iterate over components in order
doc = proc(doc) # apply each component
p
2017-10-07 15:05:59 +03:00
| The current processing pipeline is available as #[code nlp.pipeline],
| which returns a list of #[code (name, component)] tuples, or
| #[code nlp.pipe_names], which only returns a list of human-readable
| component names.
2016-10-31 21:04:15 +03:00
+code.
2017-10-07 15:05:59 +03:00
nlp.pipeline
# [('tagger', <spacy.pipeline.Tagger>), ('parser', <spacy.pipeline.DependencyParser>), ('ner', <spacy.pipeline.EntityRecognizer>)]
nlp.pipe_names
# ['tagger', 'parser', 'ner']
2016-10-31 21:04:15 +03:00
2017-10-07 15:05:59 +03:00
+h(3, "disabling") Disabling and modifying pipeline components
p
| If you don't need a particular component of the pipeline for
| example, the tagger or the parser, you can disable loading it. This can
| sometimes make a big difference and improve loading speed. Disabled
2017-05-29 15:21:00 +03:00
| component names can be provided to #[+api("spacy#load") #[code spacy.load()]],
| #[+api("language#from_disk") #[code Language.from_disk()]] or the
2017-05-26 13:46:29 +03:00
| #[code nlp] object itself as a list:
+code.
nlp = spacy.load('en', disable['parser', 'tagger'])
2017-11-01 21:49:04 +03:00
nlp = English().from_disk('/model', disable=['ner'])
p
2017-10-07 15:05:59 +03:00
| You can also use the #[+api("language#remove_pipe") #[code remove_pipe]]
| method to remove pipeline components from an existing pipeline, the
| #[+api("language#rename_pipe") #[code rename_pipe]] method to rename them,
| or the #[+api("language#replace_pipe") #[code replace_pipe]] method
| to replace them with a custom component entirely (more details on this
| in the section on #[+a("#custom-components") custom components].
+code.
2017-10-07 15:05:59 +03:00
nlp.remove_pipe('parser')
nlp.rename_pipe('ner', 'entityrecognizer')
nlp.replace_pipe('tagger', my_custom_tagger)
+infobox("Important note: disabling pipeline components")
.o-block
| Since spaCy v2.0 comes with better support for customising the
| processing pipeline components, the #[code parser], #[code tagger]
| and #[code entity] keyword arguments have been replaced with
| #[code disable], which takes a list of pipeline component names.
2017-10-07 15:05:59 +03:00
| This lets you disable pre-defined components when loading
| a model, or initialising a Language class via
2017-11-06 15:20:36 +03:00
| #[+api("language#from_disk") #[code from_disk]].
2017-10-07 15:05:59 +03:00
+code-new.
2017-10-07 15:05:59 +03:00
nlp = spacy.load('en', disable=['ner'])
nlp.remove_pipe('parser')
doc = nlp(u"I don't want parsed")
+code-old.
2017-05-25 01:56:16 +03:00
nlp = spacy.load('en', tagger=False, entity=False)
doc = nlp(u"I don't want parsed", parse=False)