mirror of
https://github.com/explosion/spaCy.git
synced 2024-09-21 19:39:13 +03:00
127 lines
4.8 KiB
Plaintext
127 lines
4.8 KiB
Plaintext
//- 💫 DOCS > USAGE > PROCESSING PIPELINES > EXAMPLES
|
||
|
||
p
|
||
| To see real-world examples of pipeline factories and components in action,
|
||
| you can have a look at the source of spaCy's built-in components, e.g.
|
||
| the #[+api("tagger") #[code Tagger]], #[+api("parser") #[code Parser]] or
|
||
| #[+api("entityrecognizer") #[code EntityRecongnizer]].
|
||
|
||
+h(3, "example1") Example: Custom sentence segmentation logic
|
||
|
||
p
|
||
| Let's say you want to implement custom logic to improve spaCy's sentence
|
||
| boundary detection. Currently, sentence segmentation is based on the
|
||
| dependency parse, which doesn't always produce ideal results. The custom
|
||
| logic should therefore be applied #[strong after] tokenization, but
|
||
| #[strong before] the dependency parsing – this way, the parser can also
|
||
| take advantage of the sentence boundaries.
|
||
|
||
+code.
|
||
def sbd_component(doc):
|
||
for i, token in enumerate(doc[:-2]):
|
||
# define sentence start if period + titlecase token
|
||
if token.text == '.' and doc[i+1].is_title:
|
||
doc[i+1].sent_start = True
|
||
return doc
|
||
|
||
p
|
||
| In this case, we simply want to add the component to the existing
|
||
| pipeline of the English model. We can do this by inserting it at index 0
|
||
| of #[code nlp.pipeline]:
|
||
|
||
+code.
|
||
nlp = spacy.load('en')
|
||
nlp.pipeline.insert(0, sbd_component)
|
||
|
||
p
|
||
| When you call #[code nlp] on some text, spaCy will tokenize it to create
|
||
| a #[code Doc] object, and first call #[code sbd_component] on it, followed
|
||
| by the model's default pipeline.
|
||
|
||
+h(3, "example2") Example: Sentiment model
|
||
|
||
p
|
||
| Let's say you have trained your own document sentiment model on English
|
||
| text. After tokenization, you want spaCy to first execute the
|
||
| #[strong default tensorizer], followed by a custom
|
||
| #[strong sentiment component] that adds a #[code .sentiment]
|
||
| property to the #[code Doc], containing your model's sentiment precition.
|
||
|
||
p
|
||
| Your component class will have a #[code from_disk()] method that spaCy
|
||
| calls to load the model data. When called, the component will compute
|
||
| the sentiment score, add it to the #[code Doc] and return the modified
|
||
| document. Optionally, the component can include an #[code update()] method
|
||
| to allow training the model.
|
||
|
||
+code.
|
||
import pickle
|
||
from pathlib import Path
|
||
|
||
class SentimentComponent(object):
|
||
def __init__(self, vocab):
|
||
self.weights = None
|
||
|
||
def __call__(self, doc):
|
||
doc.sentiment = sum(self.weights*doc.vector) # set sentiment property
|
||
return doc
|
||
|
||
def from_disk(self, path): # path = model path + factory ID ('sentiment')
|
||
self.weights = pickle.load(Path(path) / 'weights.bin') # load weights
|
||
return self
|
||
|
||
def update(self, doc, gold): # update weights – allows training!
|
||
prediction = sum(self.weights*doc.vector)
|
||
self.weights -= 0.001*doc.vector*(prediction-gold.sentiment)
|
||
|
||
p
|
||
| The factory will initialise the component with the #[code Vocab] object.
|
||
| To be able to add it to your model's pipeline as #[code 'sentiment'],
|
||
| it also needs to be registered via
|
||
| #[+api("spacy#set_factory") #[code set_factory()]].
|
||
|
||
+code.
|
||
def sentiment_factory(vocab):
|
||
component = SentimentComponent(vocab) # initialise component
|
||
return component
|
||
|
||
spacy.set_factory('sentiment', sentiment_factory)
|
||
|
||
p
|
||
| The above code should be #[strong shipped with your model]. You can use
|
||
| the #[+api("cli#package") #[code package]] command to create all required
|
||
| files and directories. The model package will include an
|
||
| #[+src(gh("spacy-dev-resources", "templates/model/en_model_name/__init__.py")) #[code __init__.py]]
|
||
| with a #[code load()] method, that will initialise the language class with
|
||
| the model's pipeline and call the #[code from_disk()] method to load
|
||
| the model data.
|
||
|
||
p
|
||
| In the model package's meta.json, specify the language class and pipeline
|
||
| IDs:
|
||
|
||
+code("meta.json (excerpt)", "json").
|
||
{
|
||
"name": "sentiment_model",
|
||
"lang": "en",
|
||
"version": "1.0.0",
|
||
"spacy_version": ">=2.0.0,<3.0.0",
|
||
"pipeline": ["tensorizer", "sentiment"]
|
||
}
|
||
|
||
p
|
||
| When you load your new model, spaCy will call the model's #[code load()]
|
||
| method. This will return a #[code Language] object with a pipeline
|
||
| containing the default tensorizer, and the sentiment component returned
|
||
| by your custom #[code "sentiment"] factory.
|
||
|
||
+code.
|
||
nlp = spacy.load('en_sentiment_model')
|
||
doc = nlp(u'I love pizza')
|
||
assert doc.sentiment
|
||
|
||
+infobox("Saving and loading models")
|
||
| For more information and a detailed guide on how to package your model,
|
||
| see the documentation on
|
||
| #[+a("/usage/training#saving-loading") saving and loading models].
|