mirror of
https://github.com/explosion/spaCy.git
synced 2026-01-11 19:21:15 +03:00
3d182fbc43
Entries in GoldParse.{words, heads, tags, deps, ner} can now be lists
instead of single values, to handle getting the analysis for fused
tokens. For instance, let's say we have a token like "hows", while the
gold-standard has two tokens, ["how", "s"]. We need to store the gold
data for each of the two subtokens.
Example gold.words: [["how", "s"], "it", "going"]
Things get more complicated for heads, as we need to address particular
subtokens. Let's say the gold heads for ["how", "s", "it", "going"] is
[1, 1, 3, 1], i.e. the root "s" is within the subtoken. The gold.heads
list would be:
[[(0, 1), (0, 1)], 2, (0, 1)]
The tuples indicate token 0, subtoken 1. A helper method
_flatten_fused_heads is available that unpacks the above to
[1, 1, 3, 1].
|
||
|---|---|---|
| .. | ||
| cli | ||
| data | ||
| displacy | ||
| lang | ||
| syntax | ||
| tests | ||
| tokens | ||
| __init__.pxd | ||
| __init__.py | ||
| __main__.py | ||
| _align.pyx | ||
| _matcher2_notes.py | ||
| _ml.py | ||
| about.py | ||
| attrs.pxd | ||
| attrs.pyx | ||
| compat.py | ||
| glossary.py | ||
| gold.pxd | ||
| gold.pyx | ||
| language.py | ||
| lemmatizer.py | ||
| lexeme.pxd | ||
| lexeme.pyx | ||
| matcher.pyx | ||
| morphology.pxd | ||
| morphology.pyx | ||
| parts_of_speech.pxd | ||
| parts_of_speech.pyx | ||
| pipeline.pxd | ||
| pipeline.pyx | ||
| scorer.py | ||
| strings.pxd | ||
| strings.pyx | ||
| structs.pxd | ||
| symbols.pxd | ||
| symbols.pyx | ||
| tokenizer.pxd | ||
| tokenizer.pyx | ||
| typedefs.pxd | ||
| typedefs.pyx | ||
| util.py | ||
| vectors.pyx | ||
| vocab.pxd | ||
| vocab.pyx | ||