spaCy/website/docs/api/example.md

title	teaser	tag	source	new
Example	A training instance	class	spacy/gold/example.pyx	3.0

An Example holds the information for one training instance. It stores two Doc objects: one for holding the gold-standard reference data, and one for holding the predictions of the pipeline. An Alignment object stores the alignment between these two documents, as they can differ in tokenization.

Example.__init__

Construct an Example object from the predicted document and the reference document. If alignment is None, it will be initialized from the words in both documents.

Example

from spacy.tokens import Doc
from spacy.gold import Example

words = ["hello", "world", "!"]
spaces = [True, False, False]
predicted = Doc(nlp.vocab, words=words, spaces=spaces)
reference = parse_gold_doc(my_data)
example = Example(predicted, reference)

Name	Description
predicted	The document containing (partial) predictions. Can not be None. Doc
reference	The document containing gold-standard annotations. Can not be None. Doc
keyword-only
alignment	An object holding the alignment between the tokens of the predicted and reference documents. Optional[Alignment]

Example.from_dict

Construct an Example object from the predicted document and the reference annotations provided as a dictionary. For more details on the required format, see the training format documentation.

Example

from spacy.tokens import Doc
from spacy.gold import Example

predicted = Doc(vocab, words=["Apply", "some", "sunscreen"])
token_ref = ["Apply", "some", "sun", "screen"]
tags_ref = ["VERB", "DET", "NOUN", "NOUN"]
example = Example.from_dict(predicted, {"words": token_ref, "tags": tags_ref})

Name	Description
predicted	The document containing (partial) predictions. Can not be None. Doc
example_dict	Dict[str, obj]
RETURNS	The newly constructed object. Example

Example.text

The text of the predicted document in this Example.

Example

raw_text = example.text

Name	Description
RETURNS	The text of the predicted document. str

Example.predicted

The Doc holding the predictions. Occasionally also referred to as example.x.

Example

docs = [eg.predicted for eg in examples]
predictions, _ = model.begin_update(docs)
set_annotations(docs, predictions)

Name	Description
RETURNS	The document containing (partial) predictions. Doc

Example.reference

The Doc holding the gold-standard annotations. Occasionally also referred to as example.y.

Example

for i, eg in enumerate(examples):
    for j, label in enumerate(all_labels):
        gold_labels[i][j] = eg.reference.cats.get(label, 0.0)

Name	Description
RETURNS	The document containing gold-standard annotations. Doc

Example.alignment

The Alignment object mapping the tokens of the predicted document to those of the reference document.

Example

tokens_x = ["Apply", "some", "sunscreen"]
x = Doc(vocab, words=tokens_x)
tokens_y = ["Apply", "some", "sun", "screen"]
example = Example.from_dict(x, {"words": tokens_y})
alignment = example.alignment
assert list(alignment.y2x.data) == [[0], [1], [2], [2]]

Name	Description
RETURNS	The document containing gold-standard annotations. Alignment

Example.get_aligned

Get the aligned view of a certain token attribute, denoted by its int ID or string name.

Example

predicted = Doc(vocab, words=["Apply", "some", "sunscreen"])
token_ref = ["Apply", "some", "sun", "screen"]
tags_ref = ["VERB", "DET", "NOUN", "NOUN"]
example = Example.from_dict(predicted, {"words": token_ref, "tags": tags_ref})
assert example.get_aligned("TAG", as_string=True) == ["VERB", "DET", "NOUN"]

Name	Description
field	Attribute ID or string name. Union[int, str]
as_string	Whether or not to return the list of values as strings. Defaults to False. bool
RETURNS	List of integer values, or string values if as_string is True. Union[List[int], List[str]]

Example.get_aligned_parse

Get the aligned view of the dependency parse. If projectivize is set to True, non-projective dependency trees are made projective through the Pseudo-Projective Dependency Parsing algorithm by Nivre and Nilsson (2005).

Example

doc = nlp("He pretty quickly walks away")
example = Example.from_dict(doc, {"heads": [3, 2, 3, 0, 2]})
proj_heads, proj_labels = example.get_aligned_parse(projectivize=True)
assert proj_heads == [3, 2, 3, 0, 3]

Name	Description
projectivize	Whether or not to projectivize the dependency trees. Defaults to True. bool
RETURNS	List of integer values, or string values if as_string is True. Union[List[int], List[str]]

Example.get_aligned_ner

Get the aligned view of the NER BILUO tags.

Example

words = ["Mrs", "Smith", "flew", "to", "New York"]
doc = Doc(en_vocab, words=words)
entities = [(0, 9, "PERSON"), (18, 26, "LOC")]
gold_words = ["Mrs Smith", "flew", "to", "New", "York"]
example = Example.from_dict(doc, {"words": gold_words, "entities": entities})
ner_tags = example.get_aligned_ner()
assert ner_tags == ["B-PERSON", "L-PERSON", "O", "O", "U-LOC"]

Name	Description
RETURNS	List of BILUO values, denoting whether tokens are part of an NER annotation or not. List[str]

Example.get_aligned_spans_y2x

Get the aligned view of any set of Span objects defined over Example.reference. The resulting span indices will align to the tokenization in Example.predicted.

Example

words = ["Mr and Mrs Smith", "flew", "to", "New York"]
doc = Doc(en_vocab, words=words)
entities = [(0, 16, "PERSON")]
tokens_ref = ["Mr", "and", "Mrs", "Smith", "flew", "to", "New", "York"]
example = Example.from_dict(doc, {"words": tokens_ref, "entities": entities})
ents_ref = example.reference.ents
assert [(ent.start, ent.end) for ent in ents_ref] == [(0, 4)]
ents_y2x = example.get_aligned_spans_y2x(ents_ref)
assert [(ent.start, ent.end) for ent in ents_y2x] == [(0, 1)]

Name	Description
y_spans	Span objects aligned to the tokenization of reference. Iterable[Span]
RETURNS	Span objects aligned to the tokenization of predicted. List[Span]

Example.get_aligned_spans_x2y

Get the aligned view of any set of Span objects defined over Example.predicted. The resulting span indices will align to the tokenization in Example.reference. This method is particularly useful to assess the accuracy of predicted entities against the original gold-standard annotation.

Example

nlp.add_pipe("my_ner")
doc = nlp("Mr and Mrs Smith flew to New York")
tokens_ref = ["Mr and Mrs", "Smith", "flew", "to", "New York"]
example = Example.from_dict(doc, {"words": tokens_ref})
ents_pred = example.predicted.ents
# Assume the NER model has found "Mr and Mrs Smith" as a named entity
assert [(ent.start, ent.end) for ent in ents_pred] == [(0, 4)]
ents_x2y = example.get_aligned_spans_x2y(ents_pred)
assert [(ent.start, ent.end) for ent in ents_x2y] == [(0, 2)]

Name	Description
x_spans	Span objects aligned to the tokenization of predicted. Iterable[Span]
RETURNS	Span objects aligned to the tokenization of reference. List[Span]

Example.to_dict

Return a dictionary representation of the reference annotation contained in this Example.

Example

eg_dict = example.to_dict()

Name	Description
RETURNS	Dictionary representation of the reference annotation. Dict[str, Any]

Example.split_sents

Split one Example into multiple Example objects, one for each sentence.

Example

doc = nlp("I went yesterday had lots of fun")
tokens_ref = ["I", "went", "yesterday", "had", "lots", "of", "fun"]
sents_ref = [True, False, False, True, False, False, False]
example = Example.from_dict(doc, {"words": tokens_ref, "sent_starts": sents_ref})
split_examples = example.split_sents()
assert split_examples[0].text == "I went yesterday "
assert split_examples[1].text == "had lots of fun"

Name	Description
RETURNS	List of Example objects, one for each original sentence. List[Example]

Alignment

Calculate alignment tables between two tokenizations.

Alignment attributes {#alignment-attributes"}

Name	Description
x2y	The Ragged object holding the alignment from x to y. Ragged
y2x	The Ragged object holding the alignment from y to x. Ragged

The current implementation of the alignment algorithm assumes that both tokenizations add up to the same string. For example, you'll be able to align ["I", "'", "m"] and ["I", "'m"], which both add up to "I'm", but not ["I", "'m"] and ["I", "am"].

Example

from spacy.gold import Alignment

bert_tokens = ["obama", "'", "s", "podcast"]
spacy_tokens = ["obama", "'s", "podcast"]
alignment = Alignment.from_strings(bert_tokens, spacy_tokens)
a2b = alignment.x2y
assert list(a2b.dataXd) == [0, 1, 1, 2]

If a2b.dataXd[1] == a2b.dataXd[2] == 1, that means that A[1] ("'") and A[2] ("s") both align to B[1] ("'s").

Alignment.from_strings

Name	Description
A	String values of candidate tokens to align. List[str]
B	String values of reference tokens to align. List[str]
RETURNS	An Alignment object describing the alignment. Alignment