explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-12-26 01:46:28 +03:00
Code Issues Packages Projects Releases Wiki Activity

Fix out-of-bounds access in NER training

Browse Source 
The helper method state.B(1) gets the index of the first token of the
buffer, or -1 if no such token exists. Normally this is safe because we
pass this to functions like state.safe_get(), which returns an empty
token. Here we used it directly as an array index, which is not okay!

This error may have been the cause of out-of-bounds access errors during
training. Similar errors may still be around, so much be hunted down.
Hunting this one down took a long time...I printed out values across
training runs and diffed, looking for points of divergence between
runs, when no randomness should be allowed.
This commit is contained in:
Matthew Honnibal 2018-10-27 00:46:30 +02:00
parent c6a320cad4
commit ad068f51be
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
spacy/syntax/ner.pyx
View File

@ -338,7 +338,7 @@ cdef class In:
@staticmethod @staticmethod
cdef weight_t cost(StateClass s, const GoldParseC* gold, attr_t label) nogil: cdef weight_t cost(StateClass s, const GoldParseC* gold, attr_t label) nogil:
move = IN move = IN
cdef int next_act = gold.ner[s.B(1)].move if s.B(0) < s.c.length else OUT cdef int next_act = gold.ner[s.B(1)].move if s.B(1) >= 0 else OUT
cdef int g_act = gold.ner[s.B(0)].move cdef int g_act = gold.ner[s.B(0)].move
cdef attr_t g_tag = gold.ner[s.B(0)].label cdef attr_t g_tag = gold.ner[s.B(0)].label
cdef bint is_sunk = _entity_is_sunk(s, gold.ner) cdef bint is_sunk = _entity_is_sunk(s, gold.ner)