spaCy/spacy/pipeline.pyx

910 lines
33 KiB
Cython
Raw Normal View History

Update draft of parser neural network model Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU. Outline of the model: We first predict context-sensitive vectors for each word in the input: (embed_lower | embed_prefix | embed_suffix | embed_shape) >> Maxout(token_width) >> convolution ** 4 This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features. To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a representation that's one affine transform from this informative lexical information. This is obviously good for the parser (which backprops to the convolutions too). The parser model makes a state vector by concatenating the vector representations for its context tokens. Current results suggest few context tokens works well. Maybe this is a bug. The current context tokens: * S0, S1, S2: Top three words on the stack * B0, B1: First two words of the buffer * S0L1, S0L2: Leftmost and second leftmost children of S0 * S0R1, S0R2: Rightmost and second rightmost children of S0 * S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0 This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately, there's a way to structure the computation to save some expense (and make it more GPU friendly). The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN -- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model is so big.) This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity. The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier. We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle in CUDA to train. Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to be 0 cost. This is defined as: (exp(score) / Z) - (exp(score) / gZ) Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly, but so far this isn't working well. Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit greatly from the pre-computation trick.
2017-05-13 00:09:15 +03:00
# cython: infer_types=True
# cython: profile=True
# coding: utf8
from __future__ import unicode_literals
2017-05-07 19:04:24 +03:00
import numpy
Update draft of parser neural network model Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU. Outline of the model: We first predict context-sensitive vectors for each word in the input: (embed_lower | embed_prefix | embed_suffix | embed_shape) >> Maxout(token_width) >> convolution ** 4 This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features. To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a representation that's one affine transform from this informative lexical information. This is obviously good for the parser (which backprops to the convolutions too). The parser model makes a state vector by concatenating the vector representations for its context tokens. Current results suggest few context tokens works well. Maybe this is a bug. The current context tokens: * S0, S1, S2: Top three words on the stack * B0, B1: First two words of the buffer * S0L1, S0L2: Leftmost and second leftmost children of S0 * S0R1, S0R2: Rightmost and second rightmost children of S0 * S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0 This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately, there's a way to structure the computation to save some expense (and make it more GPU friendly). The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN -- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model is so big.) This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity. The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier. We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle in CUDA to train. Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to be 0 cost. This is defined as: (exp(score) / Z) - (exp(score) / gZ) Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly, but so far this isn't working well. Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit greatly from the pre-computation trick.
2017-05-13 00:09:15 +03:00
cimport numpy as np
import cytoolz
2017-05-29 16:40:45 +03:00
from collections import OrderedDict
import ujson
import msgpack
2017-10-27 21:29:08 +03:00
from thinc.api import chain
2017-11-03 22:20:26 +03:00
from thinc.v2v import Affine, SELU, Softmax
2017-10-27 21:29:08 +03:00
from thinc.t2v import Pooling, max_pool, mean_pool
from thinc.neural.util import to_categorical, copy_array
from thinc.neural._classes.difference import Siamese, CauchySimilarity
2017-05-08 15:53:45 +03:00
from .tokens.doc cimport Doc
from .syntax.nn_parser cimport Parser
from .syntax import nonproj
from .syntax.ner cimport BiluoPushDown
from .syntax.arc_eager cimport ArcEager
from .morphology cimport Morphology
from .vocab cimport Vocab
2017-05-22 13:17:44 +03:00
from .syntax import nonproj
from .compat import json_dumps
Update draft of parser neural network model Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU. Outline of the model: We first predict context-sensitive vectors for each word in the input: (embed_lower | embed_prefix | embed_suffix | embed_shape) >> Maxout(token_width) >> convolution ** 4 This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features. To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a representation that's one affine transform from this informative lexical information. This is obviously good for the parser (which backprops to the convolutions too). The parser model makes a state vector by concatenating the vector representations for its context tokens. Current results suggest few context tokens works well. Maybe this is a bug. The current context tokens: * S0, S1, S2: Top three words on the stack * B0, B1: First two words of the buffer * S0L1, S0L2: Leftmost and second leftmost children of S0 * S0R1, S0R2: Rightmost and second rightmost children of S0 * S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0 This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately, there's a way to structure the computation to save some expense (and make it more GPU friendly). The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN -- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model is so big.) This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity. The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier. We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle in CUDA to train. Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to be 0 cost. This is defined as: (exp(score) / Z) - (exp(score) / gZ) Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly, but so far this isn't working well. Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit greatly from the pre-computation trick.
2017-05-13 00:09:15 +03:00
2017-10-27 21:29:08 +03:00
from .attrs import POS
from .parts_of_speech import X
2017-10-27 21:29:08 +03:00
from ._ml import Tok2Vec, build_text_classifier, build_tagger_model
2017-11-03 22:20:26 +03:00
from ._ml import link_vectors_to_models, zero_init, flatten
from ._ml import create_default_optimizer
2017-10-27 21:29:08 +03:00
from . import util
2017-09-02 13:53:38 +03:00
class SentenceSegmenter(object):
2017-09-25 19:37:13 +03:00
"""A simple spaCy hook, to allow custom sentence boundary detection logic
2017-10-27 21:29:08 +03:00
(that doesn't require the dependency parse). To change the sentence
boundary detection strategy, pass a generator function `strategy` on
initialization, or assign a new strategy to the .strategy attribute.
2017-09-02 13:53:38 +03:00
Sentence detection strategies should be generators that take `Doc` objects
and yield `Span` objects for each sentence.
2017-09-25 19:37:13 +03:00
"""
2017-09-02 13:53:38 +03:00
name = 'sbd'
def __init__(self, vocab, strategy=None):
self.vocab = vocab
if strategy is None or strategy == 'on_punct':
strategy = self.split_on_punct
self.strategy = strategy
def __call__(self, doc):
doc.user_hooks['sents'] = self.strategy
return doc
2017-09-02 13:53:38 +03:00
@staticmethod
def split_on_punct(doc):
start = 0
seen_period = False
for i, word in enumerate(doc):
if seen_period and not word.is_punct:
2017-10-27 21:29:08 +03:00
yield doc[start:word.i]
2017-09-02 13:53:38 +03:00
start = word.i
seen_period = False
elif word.text in ['.', '!', '?']:
seen_period = True
if start < len(doc):
2017-10-27 21:29:08 +03:00
yield doc[start:len(doc)]
2017-09-02 13:53:38 +03:00
2017-10-26 13:40:40 +03:00
class Pipe(object):
2017-10-27 21:29:08 +03:00
"""This class is not instantiated directly. Components inherit from it, and
it defines the interface that components should follow to function as
components in a spaCy analysis pipeline.
"""
name = None
@classmethod
def Model(cls, *shape, **kwargs):
2017-09-25 19:37:13 +03:00
"""Initialize a model for the pipe."""
raise NotImplementedError
def __init__(self, vocab, model=True, **cfg):
2017-09-25 19:37:13 +03:00
"""Create a new pipe instance."""
raise NotImplementedError
def __call__(self, doc):
2017-09-25 19:37:13 +03:00
"""Apply the pipe to one document. The document is
2017-09-25 17:20:49 +03:00
modified in-place, and returned.
2017-09-25 19:37:13 +03:00
2017-09-25 17:20:49 +03:00
Both __call__ and pipe should delegate to the `predict()`
and `set_annotations()` methods.
2017-09-25 19:37:13 +03:00
"""
2017-11-03 13:20:05 +03:00
scores, tensors = self.predict([doc])
self.set_annotations([doc], scores, tensors=tensors)
return doc
def pipe(self, stream, batch_size=128, n_threads=-1):
2017-09-25 19:37:13 +03:00
"""Apply the pipe to a stream of documents.
2017-09-25 17:20:49 +03:00
Both __call__ and pipe should delegate to the `predict()`
and `set_annotations()` methods.
2017-09-25 19:37:13 +03:00
"""
for docs in cytoolz.partition_all(batch_size, stream):
docs = list(docs)
2017-11-03 13:20:05 +03:00
scores, tensors = self.predict(docs)
self.set_annotations(docs, scores, tensor=tensors)
yield from docs
def predict(self, docs):
2017-09-25 19:37:13 +03:00
"""Apply the pipeline's model to a batch of docs, without
2017-09-25 17:20:49 +03:00
modifying them.
2017-09-25 19:37:13 +03:00
"""
raise NotImplementedError
2017-11-03 13:20:05 +03:00
def set_annotations(self, docs, scores, tensors=None):
2017-09-25 19:37:13 +03:00
"""Modify a batch of documents, using pre-computed scores."""
raise NotImplementedError
2017-09-25 17:20:49 +03:00
def update(self, docs, golds, drop=0., sgd=None, losses=None):
2017-09-25 19:37:13 +03:00
"""Learn from a batch of documents and gold-standard information,
2017-09-25 17:20:49 +03:00
updating the pipe's model.
Delegates to predict() and get_loss().
2017-09-25 19:37:13 +03:00
"""
raise NotImplementedError
def get_loss(self, docs, golds, scores):
2017-09-25 19:37:13 +03:00
"""Find the loss and gradient of loss for the batch of
documents and their predicted scores."""
raise NotImplementedError
def add_label(self, label):
"""Add an output label, to be predicted by the model.
It's possible to extend pre-trained models with new labels,
but care should be taken to avoid the "catastrophic forgetting"
problem.
"""
raise NotImplementedError
def create_optimizer(self):
return create_default_optimizer(self.model.ops,
**self.cfg.get('optimizer', {}))
def begin_training(self, gold_tuples=tuple(), pipeline=None, sgd=None):
2017-09-25 19:37:13 +03:00
"""Initialize the pipe for training, using data exampes if available.
If no model has been initialized yet, the model is added."""
if self.model is True:
2017-09-25 17:20:49 +03:00
self.model = self.Model(**self.cfg)
2017-09-25 17:22:07 +03:00
link_vectors_to_models(self.vocab)
if sgd is None:
sgd = self.create_optimizer()
return sgd
def use_params(self, params):
2017-10-27 21:29:08 +03:00
"""Modify the pipe's model, to use the given parameter values."""
with self.model.use_params(params):
yield
def to_bytes(self, **exclude):
2017-09-25 19:37:13 +03:00
"""Serialize the pipe to a bytestring."""
serialize = OrderedDict()
serialize['cfg'] = lambda: json_dumps(self.cfg)
if self.model in (True, False, None):
serialize['model'] = lambda: self.model
else:
serialize['model'] = self.model.to_bytes
serialize['vocab'] = self.vocab.to_bytes
return util.to_bytes(serialize, exclude)
def from_bytes(self, bytes_data, **exclude):
2017-09-25 19:37:13 +03:00
"""Load the pipe from a bytestring."""
2017-09-02 16:17:20 +03:00
def load_model(b):
if self.model is True:
self.cfg['pretrained_dims'] = self.vocab.vectors_length
2017-09-02 16:17:20 +03:00
self.model = self.Model(**self.cfg)
self.model.from_bytes(b)
deserialize = OrderedDict((
2017-09-02 16:17:20 +03:00
('cfg', lambda b: self.cfg.update(ujson.loads(b))),
2017-09-26 14:45:14 +03:00
('vocab', lambda b: self.vocab.from_bytes(b)),
('model', load_model),
))
util.from_bytes(bytes_data, deserialize, exclude)
return self
def to_disk(self, path, **exclude):
2017-09-25 19:37:13 +03:00
"""Serialize the pipe to disk."""
serialize = OrderedDict()
serialize['cfg'] = lambda p: p.open('w').write(json_dumps(self.cfg))
serialize['vocab'] = lambda p: self.vocab.to_disk(p)
if self.model not in (None, True, False):
serialize['model'] = lambda p: p.open('wb').write(self.model.to_bytes())
util.to_disk(path, serialize, exclude)
def from_disk(self, path, **exclude):
2017-09-25 19:37:13 +03:00
"""Load the pipe from disk."""
2017-09-02 16:17:20 +03:00
def load_model(p):
if self.model is True:
self.cfg['pretrained_dims'] = self.vocab.vectors_length
2017-09-02 16:17:20 +03:00
self.model = self.Model(**self.cfg)
self.model.from_bytes(p.open('rb').read())
deserialize = OrderedDict((
2017-09-02 16:17:20 +03:00
('cfg', lambda p: self.cfg.update(_load_cfg(p))),
2017-07-23 01:33:43 +03:00
('vocab', lambda p: self.vocab.from_disk(p)),
('model', load_model),
))
util.from_disk(path, deserialize, exclude)
return self
2017-07-23 15:11:07 +03:00
def _load_cfg(path):
if path.exists():
return ujson.load(path.open())
else:
return {}
2017-10-26 13:40:40 +03:00
class Tensorizer(Pipe):
2017-05-19 01:00:02 +03:00
"""Assign position-sensitive vectors to tokens, using a CNN or RNN."""
2017-05-31 14:42:39 +03:00
name = 'tensorizer'
Update draft of parser neural network model Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU. Outline of the model: We first predict context-sensitive vectors for each word in the input: (embed_lower | embed_prefix | embed_suffix | embed_shape) >> Maxout(token_width) >> convolution ** 4 This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features. To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a representation that's one affine transform from this informative lexical information. This is obviously good for the parser (which backprops to the convolutions too). The parser model makes a state vector by concatenating the vector representations for its context tokens. Current results suggest few context tokens works well. Maybe this is a bug. The current context tokens: * S0, S1, S2: Top three words on the stack * B0, B1: First two words of the buffer * S0L1, S0L2: Leftmost and second leftmost children of S0 * S0R1, S0R2: Rightmost and second rightmost children of S0 * S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0 This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately, there's a way to structure the computation to save some expense (and make it more GPU friendly). The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN -- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model is so big.) This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity. The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier. We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle in CUDA to train. Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to be 0 cost. This is defined as: (exp(score) / Z) - (exp(score) / gZ) Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly, but so far this isn't working well. Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit greatly from the pre-computation trick.
2017-05-13 00:09:15 +03:00
@classmethod
2017-11-03 22:20:26 +03:00
def Model(cls, output_size=300, input_size=384, **cfg):
2017-05-19 01:00:02 +03:00
"""Create a new statistical model for the class.
width (int): Output size of the model.
embed_size (int): Number of vectors in the embedding table.
**cfg: Config parameters.
RETURNS (Model): A `thinc.neural.Model` or similar instance.
"""
2017-11-03 22:20:26 +03:00
model = chain(
SELU(output_size, input_size),
SELU(output_size, output_size),
zero_init(Affine(output_size, output_size)))
return model
Update draft of parser neural network model Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU. Outline of the model: We first predict context-sensitive vectors for each word in the input: (embed_lower | embed_prefix | embed_suffix | embed_shape) >> Maxout(token_width) >> convolution ** 4 This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features. To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a representation that's one affine transform from this informative lexical information. This is obviously good for the parser (which backprops to the convolutions too). The parser model makes a state vector by concatenating the vector representations for its context tokens. Current results suggest few context tokens works well. Maybe this is a bug. The current context tokens: * S0, S1, S2: Top three words on the stack * B0, B1: First two words of the buffer * S0L1, S0L2: Leftmost and second leftmost children of S0 * S0R1, S0R2: Rightmost and second rightmost children of S0 * S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0 This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately, there's a way to structure the computation to save some expense (and make it more GPU friendly). The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN -- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model is so big.) This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity. The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier. We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle in CUDA to train. Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to be 0 cost. This is defined as: (exp(score) / Z) - (exp(score) / gZ) Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly, but so far this isn't working well. Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit greatly from the pre-computation trick.
2017-05-13 00:09:15 +03:00
def __init__(self, vocab, model=True, **cfg):
2017-05-19 01:00:02 +03:00
"""Construct a new statistical model. Weights are not allocated on
initialisation.
2017-10-27 21:29:08 +03:00
vocab (Vocab): A `Vocab` instance. The model must share the same
`Vocab` instance with the `Doc` objects it will process.
2017-05-19 01:00:02 +03:00
model (Model): A `Model` instance or `True` allocate one later.
**cfg: Config parameters.
EXAMPLE:
>>> from spacy.pipeline import TokenVectorEncoder
>>> tok2vec = TokenVectorEncoder(nlp.vocab)
>>> tok2vec.model = tok2vec.Model(128, 5000)
"""
self.vocab = vocab
2017-05-18 12:29:51 +03:00
self.model = model
2017-11-03 22:20:26 +03:00
self.input_models = []
2017-07-23 01:52:47 +03:00
self.cfg = dict(cfg)
self.cfg['pretrained_dims'] = self.vocab.vectors.data.shape[1]
2017-09-22 17:38:22 +03:00
self.cfg.setdefault('cnn_maxout_pieces', 3)
2017-05-28 16:11:58 +03:00
def __call__(self, doc):
2017-05-19 01:00:02 +03:00
"""Add context-sensitive vectors to a `Doc`, e.g. from a CNN or LSTM
model. Vectors are set to the `Doc.tensor` attribute.
docs (Doc or iterable): One or more documents to add vectors to.
RETURNS (dict or None): Intermediate computations.
"""
2017-05-28 16:11:58 +03:00
tokvecses = self.predict([doc])
self.set_annotations([doc], tokvecses)
return doc
2017-05-18 16:30:59 +03:00
def pipe(self, stream, batch_size=128, n_threads=-1):
2017-05-19 01:00:02 +03:00
"""Process `Doc` objects as a stream.
stream (iterator): A sequence of `Doc` objects to process.
batch_size (int): Number of `Doc` objects to group.
n_threads (int): Number of threads.
2017-05-21 21:46:23 +03:00
YIELDS (iterator): A sequence of `Doc` objects, in order of input.
2017-05-19 01:00:02 +03:00
"""
for docs in cytoolz.partition_all(batch_size, stream):
2017-05-22 01:52:01 +03:00
docs = list(docs)
2017-11-03 22:20:26 +03:00
tensors = self.predict(docs)
self.set_annotations(docs, tensors)
yield from docs
2017-05-18 12:29:51 +03:00
def predict(self, docs):
2017-05-19 01:00:02 +03:00
"""Return a single tensor for a batch of documents.
docs (iterable): A sequence of `Doc` objects.
2017-10-27 21:29:08 +03:00
RETURNS (object): Vector representations for each token in the docs.
2017-05-19 01:00:02 +03:00
"""
2017-11-03 22:20:26 +03:00
inputs = self.model.ops.flatten([doc.tensor for doc in docs])
outputs = self.model(inputs)
return self.model.ops.unflatten(outputs, [len(d) for d in docs])
2017-11-03 22:20:26 +03:00
def set_annotations(self, docs, tensors):
2017-05-19 01:00:02 +03:00
"""Set the tensor attribute for a batch of documents.
docs (iterable): A sequence of `Doc` objects.
2017-11-03 22:20:26 +03:00
tensors (object): Vector representation for each token in the docs.
2017-05-19 01:00:02 +03:00
"""
2017-11-03 22:20:26 +03:00
for doc, tensor in zip(docs, tensors):
assert tensor.shape[0] == len(doc)
doc.tensor = tensor
2017-05-25 04:09:51 +03:00
def update(self, docs, golds, state=None, drop=0., sgd=None, losses=None):
2017-05-19 01:00:02 +03:00
"""Update the model.
docs (iterable): A batch of `Doc` objects.
golds (iterable): A batch of `GoldParse` objects.
drop (float): The droput rate.
sgd (callable): An optimizer.
2017-05-19 01:00:02 +03:00
RETURNS (dict): Results from the update.
"""
if isinstance(docs, Doc):
docs = [docs]
2017-11-03 22:20:26 +03:00
inputs = []
bp_inputs = []
for tok2vec in self.input_models:
tensor, bp_tensor = tok2vec.begin_update(docs, drop=drop)
inputs.append(tensor)
bp_inputs.append(bp_tensor)
inputs = self.model.ops.xp.hstack(inputs)
scores, bp_scores = self.model.begin_update(inputs, drop=drop)
loss, d_scores = self.get_loss(docs, golds, scores)
d_inputs = bp_scores(d_scores, sgd=sgd)
d_inputs = self.model.ops.xp.split(d_inputs, len(self.input_models), axis=1)
2017-11-05 14:25:10 +03:00
for d_input, bp_input in zip(d_inputs, bp_inputs):
2017-11-03 22:20:26 +03:00
bp_input(d_input, sgd=sgd)
if losses is not None:
losses.setdefault(self.name, 0.)
losses[self.name] += loss
return loss
def get_loss(self, docs, golds, prediction):
target = []
i = 0
for doc in docs:
vectors = self.model.ops.xp.vstack([w.vector for w in doc])
target.append(vectors)
target = self.model.ops.xp.vstack(target)
d_scores = (prediction - target) / prediction.shape[0]
loss = (d_scores**2).sum()
return loss, d_scores
def begin_training(self, gold_tuples=tuple(), pipeline=None, sgd=None):
"""Allocate models, pre-process training data and acquire an
2017-05-19 01:00:02 +03:00
optimizer.
gold_tuples (iterable): Gold-standard training data.
pipeline (list): The pipeline the model is part of.
"""
2017-11-03 22:20:26 +03:00
for name, model in pipeline:
if getattr(model, 'tok2vec', None):
self.input_models.append(model.tok2vec)
2017-05-18 12:29:51 +03:00
if self.model is True:
2017-11-03 22:20:26 +03:00
self.cfg['input_size'] = 384
self.cfg['output_size'] = 300
self.model = self.Model(**self.cfg)
link_vectors_to_models(self.vocab)
if sgd is None:
sgd = self.create_optimizer()
return sgd
2017-05-18 12:29:51 +03:00
2017-05-29 02:37:57 +03:00
2017-10-26 13:40:40 +03:00
class Tagger(Pipe):
2017-06-01 18:37:53 +03:00
name = 'tagger'
2017-10-27 21:29:08 +03:00
2017-07-23 01:52:47 +03:00
def __init__(self, vocab, model=True, **cfg):
self.vocab = vocab
self.model = model
2017-07-23 01:52:47 +03:00
self.cfg = dict(cfg)
self.cfg.setdefault('cnn_maxout_pieces', 2)
2017-10-27 21:29:08 +03:00
self.cfg.setdefault('pretrained_dims',
self.vocab.vectors.data.shape[1])
@property
def labels(self):
return self.vocab.morphology.tag_names
2017-11-03 22:20:26 +03:00
@property
def tok2vec(self):
if self.model in (None, True, False):
return None
else:
return chain(self.model.tok2vec, flatten)
def __call__(self, doc):
2017-11-03 13:20:05 +03:00
tags, tokvecs = self.predict([doc])
self.set_annotations([doc], tags, tensors=tokvecs)
2017-05-28 16:11:58 +03:00
return doc
def pipe(self, stream, batch_size=128, n_threads=-1):
for docs in cytoolz.partition_all(batch_size, stream):
docs = list(docs)
2017-11-03 13:20:05 +03:00
tag_ids, tokvecs = self.predict(docs)
self.set_annotations(docs, tag_ids, tensors=tokvecs)
yield from docs
def predict(self, docs):
2017-11-03 13:20:05 +03:00
tokvecs = self.model.tok2vec(docs)
scores = self.model.softmax(tokvecs)
2017-11-03 15:29:36 +03:00
guesses = []
for doc_scores in scores:
doc_guesses = doc_scores.argmax(axis=1)
if not isinstance(doc_guesses, numpy.ndarray):
doc_guesses = doc_guesses.get()
guesses.append(doc_guesses)
2017-11-03 13:20:05 +03:00
return guesses, tokvecs
2017-11-03 13:20:05 +03:00
def set_annotations(self, docs, batch_tag_ids, tensors=None):
if isinstance(docs, Doc):
docs = [docs]
cdef Doc doc
cdef int idx = 0
2017-05-18 12:29:51 +03:00
cdef Vocab vocab = self.vocab
2017-05-08 15:53:45 +03:00
for i, doc in enumerate(docs):
2017-05-21 17:05:34 +03:00
doc_tag_ids = batch_tag_ids[i]
if hasattr(doc_tag_ids, 'get'):
doc_tag_ids = doc_tag_ids.get()
2017-05-18 12:29:51 +03:00
for j, tag_id in enumerate(doc_tag_ids):
2017-06-04 23:52:42 +03:00
# Don't clobber preset POS tags
if doc.c[j].tag == 0 and doc.c[j].pos == 0:
2017-11-06 14:36:05 +03:00
# Don't clobber preset lemmas
lemma = doc.c[j].lemma
vocab.morphology.assign_tag_id(&doc.c[j], tag_id)
2017-11-06 18:56:19 +03:00
if lemma != 0 and lemma != doc.c[j].lex.orth:
2017-11-06 14:36:05 +03:00
doc.c[j].lemma = lemma
2017-05-08 15:53:45 +03:00
idx += 1
2017-11-03 13:20:05 +03:00
if tensors is not None:
2017-11-05 17:34:40 +03:00
if isinstance(doc.tensor, numpy.ndarray) \
and not isinstance(tensors[i], numpy.ndarray):
doc.extend_tensor(tensors[i].get())
else:
doc.extend_tensor(tensors[i])
2017-06-05 02:35:07 +03:00
doc.is_tagged = True
2017-05-08 15:53:45 +03:00
def update(self, docs, golds, drop=0., sgd=None, losses=None):
2017-08-20 15:42:23 +03:00
if losses is not None and self.name not in losses:
losses[self.name] = 0.
tag_scores, bp_tag_scores = self.model.begin_update(docs, drop=drop)
loss, d_tag_scores = self.get_loss(docs, golds, tag_scores)
2017-09-23 03:58:06 +03:00
bp_tag_scores(d_tag_scores, sgd=sgd)
2017-05-18 12:29:51 +03:00
2017-08-20 15:42:23 +03:00
if losses is not None:
losses[self.name] += loss
def get_loss(self, docs, golds, scores):
scores = self.model.ops.flatten(scores)
tag_index = {tag: i for i, tag in enumerate(self.labels)}
2017-05-18 12:29:51 +03:00
cdef int idx = 0
Update draft of parser neural network model Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU. Outline of the model: We first predict context-sensitive vectors for each word in the input: (embed_lower | embed_prefix | embed_suffix | embed_shape) >> Maxout(token_width) >> convolution ** 4 This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features. To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a representation that's one affine transform from this informative lexical information. This is obviously good for the parser (which backprops to the convolutions too). The parser model makes a state vector by concatenating the vector representations for its context tokens. Current results suggest few context tokens works well. Maybe this is a bug. The current context tokens: * S0, S1, S2: Top three words on the stack * B0, B1: First two words of the buffer * S0L1, S0L2: Leftmost and second leftmost children of S0 * S0R1, S0R2: Rightmost and second rightmost children of S0 * S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0 This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately, there's a way to structure the computation to save some expense (and make it more GPU friendly). The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN -- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model is so big.) This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity. The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier. We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle in CUDA to train. Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to be 0 cost. This is defined as: (exp(score) / Z) - (exp(score) / gZ) Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly, but so far this isn't working well. Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit greatly from the pre-computation trick.
2017-05-13 00:09:15 +03:00
correct = numpy.zeros((scores.shape[0],), dtype='i')
guesses = scores.argmax(axis=1)
Update draft of parser neural network model Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU. Outline of the model: We first predict context-sensitive vectors for each word in the input: (embed_lower | embed_prefix | embed_suffix | embed_shape) >> Maxout(token_width) >> convolution ** 4 This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features. To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a representation that's one affine transform from this informative lexical information. This is obviously good for the parser (which backprops to the convolutions too). The parser model makes a state vector by concatenating the vector representations for its context tokens. Current results suggest few context tokens works well. Maybe this is a bug. The current context tokens: * S0, S1, S2: Top three words on the stack * B0, B1: First two words of the buffer * S0L1, S0L2: Leftmost and second leftmost children of S0 * S0R1, S0R2: Rightmost and second rightmost children of S0 * S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0 This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately, there's a way to structure the computation to save some expense (and make it more GPU friendly). The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN -- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model is so big.) This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity. The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier. We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle in CUDA to train. Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to be 0 cost. This is defined as: (exp(score) / Z) - (exp(score) / gZ) Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly, but so far this isn't working well. Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit greatly from the pre-computation trick.
2017-05-13 00:09:15 +03:00
for gold in golds:
for tag in gold.tags:
if tag is None:
correct[idx] = guesses[idx]
else:
correct[idx] = tag_index[tag]
Update draft of parser neural network model Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU. Outline of the model: We first predict context-sensitive vectors for each word in the input: (embed_lower | embed_prefix | embed_suffix | embed_shape) >> Maxout(token_width) >> convolution ** 4 This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features. To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a representation that's one affine transform from this informative lexical information. This is obviously good for the parser (which backprops to the convolutions too). The parser model makes a state vector by concatenating the vector representations for its context tokens. Current results suggest few context tokens works well. Maybe this is a bug. The current context tokens: * S0, S1, S2: Top three words on the stack * B0, B1: First two words of the buffer * S0L1, S0L2: Leftmost and second leftmost children of S0 * S0R1, S0R2: Rightmost and second rightmost children of S0 * S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0 This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately, there's a way to structure the computation to save some expense (and make it more GPU friendly). The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN -- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model is so big.) This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity. The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier. We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle in CUDA to train. Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to be 0 cost. This is defined as: (exp(score) / Z) - (exp(score) / gZ) Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly, but so far this isn't working well. Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit greatly from the pre-computation trick.
2017-05-13 00:09:15 +03:00
idx += 1
2017-05-18 16:30:59 +03:00
correct = self.model.ops.xp.array(correct, dtype='i')
Update draft of parser neural network model Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU. Outline of the model: We first predict context-sensitive vectors for each word in the input: (embed_lower | embed_prefix | embed_suffix | embed_shape) >> Maxout(token_width) >> convolution ** 4 This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features. To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a representation that's one affine transform from this informative lexical information. This is obviously good for the parser (which backprops to the convolutions too). The parser model makes a state vector by concatenating the vector representations for its context tokens. Current results suggest few context tokens works well. Maybe this is a bug. The current context tokens: * S0, S1, S2: Top three words on the stack * B0, B1: First two words of the buffer * S0L1, S0L2: Leftmost and second leftmost children of S0 * S0R1, S0R2: Rightmost and second rightmost children of S0 * S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0 This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately, there's a way to structure the computation to save some expense (and make it more GPU friendly). The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN -- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model is so big.) This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity. The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier. We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle in CUDA to train. Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to be 0 cost. This is defined as: (exp(score) / Z) - (exp(score) / gZ) Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly, but so far this isn't working well. Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit greatly from the pre-computation trick.
2017-05-13 00:09:15 +03:00
d_scores = scores - to_categorical(correct, nb_classes=scores.shape[1])
2017-05-28 02:32:46 +03:00
d_scores /= d_scores.shape[0]
2017-05-18 12:29:51 +03:00
loss = (d_scores**2).sum()
d_scores = self.model.ops.unflatten(d_scores, [len(d) for d in docs])
2017-05-18 16:30:59 +03:00
return float(loss), d_scores
def begin_training(self, gold_tuples=tuple(), pipeline=None, sgd=None):
2017-05-18 16:30:59 +03:00
orig_tag_map = dict(self.vocab.morphology.tag_map)
new_tag_map = {}
for raw_text, annots_brackets in gold_tuples:
for annots, brackets in annots_brackets:
ids, words, tags, heads, deps, ents = annots
for tag in tags:
2017-05-18 16:30:59 +03:00
if tag in orig_tag_map:
new_tag_map[tag] = orig_tag_map[tag]
else:
new_tag_map[tag] = {POS: X}
cdef Vocab vocab = self.vocab
2017-06-01 11:04:36 +03:00
if new_tag_map:
vocab.morphology = Morphology(vocab.strings, new_tag_map,
2017-06-05 00:34:32 +03:00
vocab.morphology.lemmatizer,
exc=vocab.morphology.exc)
2017-05-29 21:23:47 +03:00
if self.model is True:
self.cfg['pretrained_dims'] = self.vocab.vectors.data.shape[1]
2017-09-21 21:07:26 +03:00
self.model = self.Model(self.vocab.morphology.n_tags, **self.cfg)
link_vectors_to_models(self.vocab)
if sgd is None:
sgd = self.create_optimizer()
return sgd
2017-05-29 21:23:47 +03:00
@classmethod
2017-09-23 03:58:06 +03:00
def Model(cls, n_tags, **cfg):
return build_tagger_model(n_tags, **cfg)
def add_label(self, label, values=None):
if label in self.labels:
return 0
if self.model not in (True, False, None):
# Here's how the model resizing will work, once the
# neuron-to-tag mapping is no longer controlled by
# the Morphology class, which sorts the tag names.
# The sorting makes adding labels difficult.
# smaller = self.model._layers[-1]
# larger = Softmax(len(self.labels)+1, smaller.nI)
# copy_array(larger.W[:smaller.nO], smaller.W)
# copy_array(larger.b[:smaller.nO], smaller.b)
# self.model._layers[-1] = larger
raise ValueError(
"Resizing pre-trained Tagger models is not "
"currently supported.")
tag_map = dict(self.vocab.morphology.tag_map)
if values is None:
values = {POS: "X"}
tag_map[label] = values
self.vocab.morphology = Morphology(
self.vocab.strings, tag_map=tag_map,
lemmatizer=self.vocab.morphology.lemmatizer,
exc=self.vocab.morphology.exc)
return 1
2017-05-18 16:30:59 +03:00
def use_params(self, params):
with self.model.use_params(params):
yield
2017-05-29 11:14:20 +03:00
def to_bytes(self, **exclude):
serialize = OrderedDict()
if self.model in (None, True, False):
serialize['model'] = lambda: self.model
else:
serialize['model'] = self.model.to_bytes
serialize['vocab'] = self.vocab.to_bytes
2017-10-27 21:29:08 +03:00
serialize['tag_map'] = lambda: msgpack.dumps(
self.vocab.morphology.tag_map, use_bin_type=True, encoding='utf8')
2017-05-29 11:14:20 +03:00
return util.to_bytes(serialize, exclude)
def from_bytes(self, bytes_data, **exclude):
2017-05-29 21:23:47 +03:00
def load_model(b):
if self.model is True:
2017-10-27 21:29:08 +03:00
token_vector_width = util.env_opt(
'token_vector_width',
self.cfg.get('token_vector_width', 128))
self.model = self.Model(self.vocab.morphology.n_tags,
**self.cfg)
self.model.from_bytes(b)
def load_tag_map(b):
2017-06-02 18:29:21 +03:00
tag_map = msgpack.loads(b, encoding='utf8')
self.vocab.morphology = Morphology(
self.vocab.strings, tag_map=tag_map,
2017-06-05 00:34:32 +03:00
lemmatizer=self.vocab.morphology.lemmatizer,
exc=self.vocab.morphology.exc)
2017-05-30 01:53:06 +03:00
deserialize = OrderedDict((
('vocab', lambda b: self.vocab.from_bytes(b)),
('tag_map', load_tag_map),
('cfg', lambda b: self.cfg.update(ujson.loads(b))),
2017-05-30 01:53:06 +03:00
('model', lambda b: load_model(b)),
))
2017-05-29 21:23:47 +03:00
util.from_bytes(bytes_data, deserialize, exclude)
2017-05-29 11:14:20 +03:00
return self
def to_disk(self, path, **exclude):
self.cfg['pretrained_dims'] = self.vocab.vectors.data.shape[1]
serialize = OrderedDict((
('vocab', lambda p: self.vocab.to_disk(p)),
2017-06-02 18:29:21 +03:00
('tag_map', lambda p: p.open('wb').write(msgpack.dumps(
self.vocab.morphology.tag_map,
use_bin_type=True,
encoding='utf8'))),
('model', lambda p: p.open('wb').write(self.model.to_bytes())),
2017-07-23 01:33:43 +03:00
('cfg', lambda p: p.open('w').write(json_dumps(self.cfg)))
))
util.to_disk(path, serialize, exclude)
def from_disk(self, path, **exclude):
def load_model(p):
if self.model is True:
2017-09-21 21:07:26 +03:00
self.model = self.Model(self.vocab.morphology.n_tags, **self.cfg)
self.model.from_bytes(p.open('rb').read())
def load_tag_map(p):
2017-06-02 18:29:21 +03:00
with p.open('rb') as file_:
tag_map = msgpack.loads(file_.read(), encoding='utf8')
self.vocab.morphology = Morphology(
self.vocab.strings, tag_map=tag_map,
2017-06-05 00:34:32 +03:00
lemmatizer=self.vocab.morphology.lemmatizer,
exc=self.vocab.morphology.exc)
deserialize = OrderedDict((
('cfg', lambda p: self.cfg.update(_load_cfg(p))),
('vocab', lambda p: self.vocab.from_disk(p)),
('tag_map', load_tag_map),
('model', load_model),
))
util.from_disk(path, deserialize, exclude)
return self
2017-05-29 11:14:20 +03:00
class MultitaskObjective(Tagger):
2017-10-27 21:29:08 +03:00
"""Experimental: Assist training of a parser or tagger, by training a
side-objective.
"""
2017-05-22 01:52:30 +03:00
name = 'nn_labeller'
2017-10-27 21:29:08 +03:00
def __init__(self, vocab, model=True, target='dep_tag_offset', **cfg):
2017-05-22 01:52:30 +03:00
self.vocab = vocab
self.model = model
if target == 'dep':
self.make_label = self.make_dep
elif target == 'tag':
self.make_label = self.make_tag
elif target == 'ent':
self.make_label = self.make_ent
elif target == 'dep_tag_offset':
self.make_label = self.make_dep_tag_offset
elif target == 'ent_tag':
self.make_label = self.make_ent_tag
elif hasattr(target, '__call__'):
self.make_label = target
else:
2017-10-27 21:29:08 +03:00
raise ValueError("MultitaskObjective target should be function or "
"one of: dep, tag, ent, dep_tag_offset, ent_tag.")
2017-07-23 01:52:47 +03:00
self.cfg = dict(cfg)
self.cfg.setdefault('cnn_maxout_pieces', 2)
2017-10-27 21:29:08 +03:00
self.cfg.setdefault('pretrained_dims',
self.vocab.vectors.data.shape[1])
2017-07-23 01:52:47 +03:00
@property
def labels(self):
return self.cfg.setdefault('labels', {})
2017-07-23 01:52:47 +03:00
@labels.setter
def labels(self, value):
self.cfg['labels'] = value
2017-05-22 01:52:30 +03:00
2017-11-03 13:20:05 +03:00
def set_annotations(self, docs, dep_ids, tensors=None):
2017-05-22 01:52:30 +03:00
pass
def begin_training(self, gold_tuples=tuple(), pipeline=None, tok2vec=None,
sgd=None):
2017-05-22 13:17:44 +03:00
gold_tuples = nonproj.preprocess_training_data(gold_tuples)
2017-05-22 01:52:30 +03:00
for raw_text, annots_brackets in gold_tuples:
for annots, brackets in annots_brackets:
ids, words, tags, heads, deps, ents = annots
for i in range(len(ids)):
label = self.make_label(i, words, tags, heads, deps, ents)
if label is not None and label not in self.labels:
self.labels[label] = len(self.labels)
2017-05-29 21:23:47 +03:00
if self.model is True:
2017-09-27 19:43:58 +03:00
token_vector_width = util.env_opt('token_vector_width')
self.model = chain(
tok2vec,
2017-09-27 19:43:58 +03:00
Softmax(len(self.labels), token_vector_width)
)
link_vectors_to_models(self.vocab)
if sgd is None:
sgd = self.create_optimizer()
return sgd
2017-05-29 21:23:47 +03:00
@classmethod
def Model(cls, n_tags, tok2vec=None, **cfg):
return build_tagger_model(n_tags, tok2vec=tok2vec, **cfg)
2017-05-22 01:52:30 +03:00
def get_loss(self, docs, golds, scores):
cdef int idx = 0
correct = numpy.zeros((scores.shape[0],), dtype='i')
guesses = scores.argmax(axis=1)
for gold in golds:
for i in range(len(gold.labels)):
label = self.make_label(i, gold.words, gold.tags, gold.heads,
gold.labels, gold.ents)
if label is None or label not in self.labels:
2017-05-22 01:52:30 +03:00
correct[idx] = guesses[idx]
else:
correct[idx] = self.labels[label]
2017-05-22 01:52:30 +03:00
idx += 1
correct = self.model.ops.xp.array(correct, dtype='i')
d_scores = scores - to_categorical(correct, nb_classes=scores.shape[1])
2017-05-28 02:32:46 +03:00
d_scores /= d_scores.shape[0]
2017-05-22 01:52:30 +03:00
loss = (d_scores**2).sum()
return float(loss), d_scores
@staticmethod
def make_dep(i, words, tags, heads, deps, ents):
if deps[i] is None or heads[i] is None:
return None
return deps[i]
@staticmethod
def make_tag(i, words, tags, heads, deps, ents):
return tags[i]
@staticmethod
def make_ent(i, words, tags, heads, deps, ents):
if ents is None:
return None
return ents[i]
@staticmethod
def make_dep_tag_offset(i, words, tags, heads, deps, ents):
if deps[i] is None or heads[i] is None:
return None
offset = heads[i] - i
offset = min(offset, 2)
offset = max(offset, -2)
return '%s-%s:%d' % (deps[i], tags[i], offset)
@staticmethod
def make_ent_tag(i, words, tags, heads, deps, ents):
if ents is None or ents[i] is None:
return None
else:
return '%s-%s' % (tags[i], ents[i])
2017-10-26 13:40:40 +03:00
class SimilarityHook(Pipe):
"""
2017-10-27 21:29:08 +03:00
Experimental: A pipeline component to install a hook for supervised
similarity into `Doc` objects. Requires a `Tensorizer` to pre-process
documents. The similarity model can be any object obeying the Thinc `Model`
interface. By default, the model concatenates the elementwise mean and
elementwise max of the two tensors, and compares them using the
Cauchy-like similarity function from Chen (2013):
2017-10-27 21:29:08 +03:00
>>> similarity = 1. / (1. + (W * (vec1-vec2)**2).sum())
Where W is a vector of dimension weights, initialized to 1.
"""
name = 'similarity'
2017-10-27 21:29:08 +03:00
2017-07-23 01:52:47 +03:00
def __init__(self, vocab, model=True, **cfg):
self.vocab = vocab
self.model = model
2017-07-23 01:52:47 +03:00
self.cfg = dict(cfg)
@classmethod
def Model(cls, length):
return Siamese(Pooling(max_pool, mean_pool), CauchySimilarity(length))
def __call__(self, doc):
2017-09-25 19:37:13 +03:00
"""Install similarity hook"""
doc.user_hooks['similarity'] = self.predict
return doc
def pipe(self, docs, **kwargs):
for doc in docs:
yield self(doc)
def predict(self, doc1, doc2):
return self.model.predict([(doc1, doc2)])
def update(self, doc1_doc2, golds, sgd=None, drop=0.):
sims, bp_sims = self.model.begin_update(doc1_doc2, drop=drop)
def begin_training(self, _=tuple(), pipeline=None, sgd=None):
2017-10-27 21:29:08 +03:00
"""Allocate model, using width from tensorizer in pipeline.
gold_tuples (iterable): Gold-standard training data.
pipeline (list): The pipeline the model is part of.
"""
if self.model is True:
self.model = self.Model(pipeline[0].model.nO)
2017-09-22 17:38:22 +03:00
link_vectors_to_models(self.vocab)
if sgd is None:
sgd = self.create_optimizer()
return sgd
2017-10-26 13:40:40 +03:00
class TextCategorizer(Pipe):
2017-07-22 02:14:07 +03:00
name = 'textcat'
@classmethod
def Model(cls, nr_class=1, width=64, **cfg):
return build_text_classifier(nr_class, width, **cfg)
def __init__(self, vocab, model=True, **cfg):
self.vocab = vocab
self.model = model
2017-07-23 01:52:47 +03:00
self.cfg = dict(cfg)
2017-07-23 01:33:43 +03:00
@property
def labels(self):
return self.cfg.setdefault('labels', [])
2017-07-23 01:33:43 +03:00
@labels.setter
def labels(self, value):
self.cfg['labels'] = value
def __call__(self, doc):
2017-11-03 13:20:05 +03:00
scores, tensors = self.predict([doc])
self.set_annotations([doc], scores, tensors=tensors)
return doc
def pipe(self, stream, batch_size=128, n_threads=-1):
for docs in cytoolz.partition_all(batch_size, stream):
docs = list(docs)
2017-11-03 13:20:05 +03:00
scores, tensors = self.predict(docs)
self.set_annotations(docs, scores, tensors=tensors)
yield from docs
def predict(self, docs):
scores = self.model(docs)
scores = self.model.ops.asarray(scores)
2017-11-05 14:25:10 +03:00
tensors = [doc.tensor for doc in docs]
return scores, tensors
2017-11-03 13:20:05 +03:00
def set_annotations(self, docs, scores, tensors=None):
for i, doc in enumerate(docs):
for j, label in enumerate(self.labels):
doc.cats[label] = float(scores[i, j])
def update(self, docs, golds, state=None, drop=0., sgd=None, losses=None):
scores, bp_scores = self.model.begin_update(docs, drop=drop)
loss, d_scores = self.get_loss(docs, golds, scores)
bp_scores(d_scores, sgd=sgd)
if losses is not None:
losses.setdefault(self.name, 0.0)
losses[self.name] += loss
def get_loss(self, docs, golds, scores):
truths = numpy.zeros((len(golds), len(self.labels)), dtype='f')
not_missing = numpy.ones((len(golds), len(self.labels)), dtype='f')
for i, gold in enumerate(golds):
for j, label in enumerate(self.labels):
if label in gold.cats:
truths[i, j] = gold.cats[label]
else:
not_missing[i, j] = 0.
truths = self.model.ops.asarray(truths)
not_missing = self.model.ops.asarray(not_missing)
d_scores = (scores-truths) / scores.shape[0]
d_scores *= not_missing
mean_square_error = ((scores-truths)**2).sum(axis=1).mean()
return mean_square_error, d_scores
def add_label(self, label):
if label in self.labels:
return 0
2017-11-01 19:06:43 +03:00
if self.model not in (None, True, False):
smaller = self.model._layers[-1]
larger = Affine(len(self.labels)+1, smaller.nI)
copy_array(larger.W[:smaller.nO], smaller.W)
copy_array(larger.b[:smaller.nO], smaller.b)
self.model._layers[-1] = larger
self.labels.append(label)
return 1
def begin_training(self, gold_tuples=tuple(), pipeline=None, sgd=None):
2017-09-02 15:56:30 +03:00
if pipeline and getattr(pipeline[0], 'name', None) == 'tensorizer':
token_vector_width = pipeline[0].model.nO
else:
token_vector_width = 64
if self.model is True:
self.cfg['pretrained_dims'] = self.vocab.vectors_length
2017-09-02 15:56:30 +03:00
self.model = self.Model(len(self.labels), token_vector_width,
**self.cfg)
2017-09-22 17:38:22 +03:00
link_vectors_to_models(self.vocab)
if sgd is None:
sgd = self.create_optimizer()
return sgd
cdef class DependencyParser(Parser):
name = 'parser'
TransitionSystem = ArcEager
@property
def postprocesses(self):
return [nonproj.deprojectivize]
def init_multitask_objectives(self, gold_tuples, pipeline, sgd=None, **cfg):
2017-09-27 19:44:16 +03:00
for target in []:
labeller = MultitaskObjective(self.vocab, target=target)
tok2vec = self.model[0]
2017-10-27 21:29:08 +03:00
labeller.begin_training(gold_tuples, pipeline=pipeline,
tok2vec=tok2vec, sgd=sgd)
pipeline.append(labeller)
self._multitasks.append(labeller)
def __reduce__(self):
2017-10-27 21:29:08 +03:00
return (DependencyParser, (self.vocab, self.moves, self.model),
None, None)
cdef class EntityRecognizer(Parser):
2017-05-31 14:42:39 +03:00
name = 'ner'
TransitionSystem = BiluoPushDown
nr_feature = 6
def init_multitask_objectives(self, gold_tuples, pipeline, sgd=None, **cfg):
for target in []:
labeller = MultitaskObjective(self.vocab, target=target)
tok2vec = self.model[0]
2017-10-27 21:29:08 +03:00
labeller.begin_training(gold_tuples, pipeline=pipeline,
tok2vec=tok2vec)
pipeline.append(labeller)
self._multitasks.append(labeller)
def __reduce__(self):
2017-10-27 21:29:08 +03:00
return (EntityRecognizer, (self.vocab, self.moves, self.model),
None, None)
2017-03-15 17:27:41 +03:00
__all__ = ['Tagger', 'DependencyParser', 'EntityRecognizer', 'Tensorizer']