spaCy/website/api/_architecture/_nn-model.jade

//- 💫 DOCS > API > ARCHITECTURE > NN MODEL ARCHITECTURE

p
    |  spaCy's statistical models have been custom-designed to give a
    |  high-performance mix of speed and accuracy. The current architecture
    |  hasn't been published yet, but in the meantime we prepared a video that
    |  explains how the models work, with particular focus on NER.

+youtube("sqDHBH9IjRU")

p
    |  The parsing model is a blend of recent results. The two recent
    |  inspirations have been the work of Eli Klipperwasser and Yoav Goldberg at
    |  Bar Ilan#[+fn(1)], and the SyntaxNet team from Google. The foundation of
    |  the parser is still based on the work of Joakim Nivre#[+fn(2)], who
    |  introduced the transition-based framework#[+fn(3)], the arc-eager
    |  transition system, and the imitation learning objective. The model is
    |  implemented using #[+a(gh("thinc")) Thinc], spaCy's machine learning
    |  library. We first predict context-sensitive vectors for each word in the
    |  input:

+code.
    (embed_lower | embed_prefix | embed_suffix | embed_shape)
        >> Maxout(token_width)
        >> convolution ** 4

p
    |  This convolutional layer is shared between the tagger, parser and NER,
    |  and will also be shared by the future neural lemmatizer. Because the
    |  parser shares these layers with the tagger, the parser does not require
    |  tag features. I got this trick from David Weiss's "Stack Combination"
    |  paper#[+fn(4)].

p
    |  To boost the representation, the tagger actually predicts a "super tag"
    |  with POS, morphology and dependency label#[+fn(5)]. The tagger predicts
    |  these supertags by adding a softmax layer onto the convolutional layer –
    |  so, we're teaching the convolutional layer to give us a representation
    |  that's one affine transform from this informative lexical information.
    |  This is obviously good for the parser (which backprops to the
    |  convolutions too). The parser model makes a state vector by concatenating
    |  the vector representations for its context tokens.  The current context
    |  tokens:

+table
    +row
        +cell #[code S0], #[code S1], #[code S2]
        +cell Top three words on the stack.

    +row
        +cell #[code B0], #[code B1]
        +cell First two words of the buffer.

    +row
        +cell
            |  #[code S0L1], #[code S1L1], #[code S2L1], #[code B0L1],
            |  #[code B1L1]#[br]
            |  #[code S0L2], #[code S1L2], #[code S2L2], #[code B0L2],
            |  #[code B1L2]
        +cell
            |  Leftmost and second leftmost children of #[code S0], #[code S1],
            |  #[code S2], #[code B0] and #[code B1].

    +row
        +cell
            |  #[code S0R1], #[code S1R1], #[code S2R1], #[code B0R1],
            |  #[code B1R1]#[br]
            |  #[code S0R2], #[code S1R2], #[code S2R2], #[code B0R2],
            |  #[code B1R2]
        +cell
            |  Rightmost and second rightmost children of #[code S0], #[code S1],
            |  #[code S2], #[code B0] and #[code B1].

p
    |  This makes the state vector quite long: #[code 13*T], where #[code T] is
    |  the token vector width (128 is working well). Fortunately, there's a way
    |  to structure the computation to save some expense (and make it more
    |  GPU-friendly).

p
    |  The parser typically visits #[code 2*N] states for a sentence of length
    |  #[code N] (although it may visit more, if it back-tracks with a
    |  non-monotonic transition#[+fn(4)]). A naive implementation would require
    |  #[code 2*N (B, 13*T) @ (13*T, H)] matrix multiplications for a batch of
    |  size #[code B]. We can instead perform one #[code (B*N, T) @ (T, 13*H)]
    |  multiplication, to pre-compute the hidden weights for each positional
    |  feature with respect to the words in the batch. (Note that our token
    |  vectors come from the CNN — so we can't play this trick over the
    |  vocabulary. That's how Stanford's NN parser#[+fn(3)] works — and why its
    |  model is so big.)

p
    |  This pre-computation strategy allows a nice compromise between
    |  GPU-friendliness and implementation simplicity. The CNN and the wide
    |  lower layer are computed on the GPU, and then the precomputed hidden
    |  weights are moved to the CPU, before we start the transition-based
    |  parsing process. This makes a lot of things much easier. We don't have to
    |  worry about variable-length batch sizes, and we don't have to implement
    |  the dynamic oracle in CUDA to train.

p
    |  Currently the parser's loss function is multilabel log loss#[+fn(6)], as
    |  the dynamic oracle allows multiple states to be 0 cost. This is defined
    |  as follows, where #[code gZ] is the sum of the scores assigned to gold
    |  classes:

+code.
    (exp(score) / Z) - (exp(score) / gZ)

+bibliography
    +item
        |  #[+a("https://www.semanticscholar.org/paper/Simple-and-Accurate-Dependency-Parsing-Using-Bidir-Kiperwasser-Goldberg/3cf31ecb2724b5088783d7c96a5fc0d5604cbf41") Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations]
        br
        |  Eliyahu Kiperwasser, Yoav Goldberg. (2016)

    +item
        |  #[+a("https://www.semanticscholar.org/paper/A-Dynamic-Oracle-for-Arc-Eager-Dependency-Parsing-Goldberg-Nivre/22697256ec19ecc3e14fcfc63624a44cf9c22df4") A Dynamic Oracle for Arc-Eager Dependency Parsing]
        br
        |  Yoav Goldberg, Joakim Nivre (2012)

    +item
        |  #[+a("https://explosion.ai/blog/parsing-english-in-python") Parsing English in 500 Lines of Python]
        br
        |  Matthew Honnibal (2013)

    +item
        |  #[+a("https://www.semanticscholar.org/paper/Stack-propagation-Improved-Representation-Learning-Zhang-Weiss/0c133f79b23e8c680891d2e49a66f0e3d37f1466") Stack-propagation: Improved Representation Learning for Syntax]
        br
        |  Yuan Zhang, David Weiss (2016)

    +item
        |  #[+a("https://www.semanticscholar.org/paper/Deep-multi-task-learning-with-low-level-tasks-supe-S%C3%B8gaard-Goldberg/03ad06583c9721855ccd82c3d969a01360218d86") Deep multi-task learning with low level tasks supervised at lower layers]
        br
        |  Anders Søgaard, Yoav Goldberg (2016)

    +item
        |  #[+a("https://www.semanticscholar.org/paper/An-Improved-Non-monotonic-Transition-System-for-De-Honnibal-Johnson/4094cee47ade13b77b5ab4d2e6cb9dd2b8a2917c") An Improved Non-monotonic Transition System for Dependency Parsing]
        br
        |  Matthew Honnibal, Mark Johnson (2015)

    +item
        |  #[+a("http://cs.stanford.edu/people/danqi/papers/emnlp2014.pdf") A Fast and Accurate Dependency Parser using Neural Networks]
        br
        |  Danqi Cheng, Christopher D. Manning (2014)

    +item
        |  #[+a("https://www.semanticscholar.org/paper/Parsing-the-Wall-Street-Journal-using-a-Lexical-Fu-Riezler-King/0ad07862a91cd59b7eb5de38267e47725a62b8b2") Parsing the Wall Street Journal using a Lexical-Functional Grammar and Discriminative Estimation Techniques]
        br
        |  Stefan Riezler et al. (2002)