mirror of
https://github.com/explosion/spaCy.git
synced 2024-12-25 17:36:30 +03:00
827b5af697
Model is good, but code is messy. Currently requires Chainer, which may cause the build to fail on machines without a GPU. Outline of the model: We first predict context-sensitive vectors for each word in the input: (embed_lower | embed_prefix | embed_suffix | embed_shape) >> Maxout(token_width) >> convolution ** 4 This convolutional layer is shared between the tagger and the parser. This prevents the parser from needing tag features. To boost the representation, we make a "super tag" with POS, morphology and dependency label. The tagger predicts this by adding a softmax layer onto the convolutional layer --- so, we're teaching the convolutional layer to give us a representation that's one affine transform from this informative lexical information. This is obviously good for the parser (which backprops to the convolutions too). The parser model makes a state vector by concatenating the vector representations for its context tokens. Current results suggest few context tokens works well. Maybe this is a bug. The current context tokens: * S0, S1, S2: Top three words on the stack * B0, B1: First two words of the buffer * S0L1, S0L2: Leftmost and second leftmost children of S0 * S0R1, S0R2: Rightmost and second rightmost children of S0 * S1L1, S1L2, S1R2, S1R, B0L1, B0L2: Likewise for S1 and B0 This makes the state vector quite long: 13*T, where T is the token vector width (128 is working well). Fortunately, there's a way to structure the computation to save some expense (and make it more GPU friendly). The parser typically visits 2*N states for a sentence of length N (although it may visit more, if it back-tracks with a non-monotonic transition). A naive implementation would require 2*N (B, 13*T) @ (13*T, H) matrix multiplications for a batch of size B. We can instead perform one (B*N, T) @ (T, 13*H) multiplication, to pre-compute the hidden weights for each positional feature wrt the words in the batch. (Note that our token vectors come from the CNN -- so we can't play this trick over the vocabulary. That's how Stanford's NN parser works --- and why its model is so big.) This pre-computation strategy allows a nice compromise between GPU-friendliness and implementation simplicity. The CNN and the wide lower layer are computed on the GPU, and then the precomputed hidden weights are moved to the CPU, before we start the transition-based parsing process. This makes a lot of things much easier. We don't have to worry about variable-length batch sizes, and we don't have to implement the dynamic oracle in CUDA to train. Currently the parser's loss function is multilabel log loss, as the dynamic oracle allows multiple states to be 0 cost. This is defined as: (exp(score) / Z) - (exp(score) / gZ) Where gZ is the sum of the scores assigned to gold classes. I'm very interested in regressing on the cost directly, but so far this isn't working well. Machinery is in place for beam-search, which has been working well for the linear model. Beam search should benefit greatly from the pre-computation trick. |
||
|---|---|---|
| .. | ||
| bn | ||
| cli | ||
| data | ||
| de | ||
| en | ||
| es | ||
| fi | ||
| fr | ||
| he | ||
| hu | ||
| it | ||
| ja | ||
| language_data | ||
| munge | ||
| nb | ||
| nl | ||
| pt | ||
| serialize | ||
| sv | ||
| syntax | ||
| tests | ||
| tokens | ||
| zh | ||
| __init__.pxd | ||
| __init__.py | ||
| __main__.py | ||
| _ml.py | ||
| about.py | ||
| attrs.pxd | ||
| attrs.pyx | ||
| cfile.pxd | ||
| cfile.pyx | ||
| compat.py | ||
| deprecated.py | ||
| gold.pxd | ||
| gold.pyx | ||
| language.py | ||
| lemmatizer.py | ||
| lexeme.pxd | ||
| lexeme.pyx | ||
| matcher.pyx | ||
| morphology.pxd | ||
| morphology.pyx | ||
| orth.pxd | ||
| orth.pyx | ||
| parts_of_speech.pxd | ||
| parts_of_speech.pyx | ||
| pipeline.pxd | ||
| pipeline.pyx | ||
| scorer.py | ||
| strings.pxd | ||
| strings.pyx | ||
| structs.pxd | ||
| symbols.pxd | ||
| symbols.pyx | ||
| tagger.pxd | ||
| tagger.pyx | ||
| tokenizer.pxd | ||
| tokenizer.pyx | ||
| train.py | ||
| typedefs.pxd | ||
| typedefs.pyx | ||
| util.py | ||
| vocab.pxd | ||
| vocab.pyx | ||