spaCy/website/docs/api/docbin.md

title	tag	new	teaser	source
DocBin	class	2.2	Pack Doc objects for binary serialization	spacy/tokens/_serialize.py

The DocBin class lets you efficiently serialize the information from a collection of Doc objects. You can control which information is serialized by passing a list of attribute IDs, and optionally also specify whether the user data is serialized. The DocBin is faster and produces smaller data sizes than pickle, and allows you to deserialize without executing arbitrary Python code. A notable downside to this format is that you can't easily extract just one document from the DocBin. The serialization format is gzipped msgpack, where the msgpack object has the following structure:

### msgpack object structrue
{
    "version": str,           # DocBin version number
    "attrs": List[uint64],    # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE]
    "tokens": bytes,          # Serialized numpy uint64 array with the token data
    "spaces": bytes,          # Serialized numpy boolean array with spaces data
    "lengths": bytes,         # Serialized numpy int32 array with the doc lengths
    "strings": List[str]      # List of unique strings in the token data
}

Strings for the words, tags, labels etc are represented by 64-bit hashes in the token data, and every string that occurs at least once is passed via the strings object. This means the storage is more efficient if you pack more documents together, because you have less duplication in the strings. For usage examples, see the docs on serializing Doc objects.

DocBin.__init__

Create a DocBin object to hold serialized annotations.

Example

from spacy.tokens import DocBin
doc_bin = DocBin(attrs=["ENT_IOB", "ENT_TYPE"])

Argument	Description
attrs	List of attributes to serialize. ORTH (hash of token text) and SPACY (whether the token is followed by whitespace) are always serialized, so they're not required. Defaults to ("ORTH", "TAG", "HEAD", "DEP", "ENT_IOB", "ENT_TYPE", "ENT_KB_ID", "LEMMA", "MORPH", "POS"). Iterable[str]
store_user_data	Whether to write the Doc.user_data and the values of custom extension attributes to file/bytes. Defaults to False. bool
docs	Doc objects to add on initialization. Iterable[Doc]

DocBin._\len__

Get the number of Doc objects that were added to the DocBin.

Example

doc_bin = DocBin(attrs=["LEMMA"])
doc = nlp("This is a document to serialize.")
doc_bin.add(doc)
assert len(doc_bin) == 1

Argument	Description
RETURNS	The number of Docs added to the DocBin. int

DocBin.add

Add a Doc's annotations to the DocBin for serialization.

Example

doc_bin = DocBin(attrs=["LEMMA"])
doc = nlp("This is a document to serialize.")
doc_bin.add(doc)

Argument	Description
doc	The Doc object to add. Doc

DocBin.get_docs

Recover Doc objects from the annotations, using the given vocab.

Example

docs = list(doc_bin.get_docs(nlp.vocab))

Argument	Description
vocab	The shared vocab. Vocab
YIELDS	The Doc objects. Doc

DocBin.merge

Extend the annotations of this DocBin with the annotations from another. Will raise an error if the pre-defined attrs of the two DocBins don't match.

Example

doc_bin1 = DocBin(attrs=["LEMMA", "POS"])
doc_bin1.add(nlp("Hello world"))
doc_bin2 = DocBin(attrs=["LEMMA", "POS"])
doc_bin2.add(nlp("This is a sentence"))
doc_bin1.merge(doc_bin2)
assert len(doc_bin1) == 2

Argument	Description
other	The DocBin to merge into the current bin. DocBin

DocBin.to_bytes

Serialize the DocBin's annotations to a bytestring.

Example

docs = [nlp("Hello world!")]
doc_bin = DocBin(docs=docs)
doc_bin_bytes = doc_bin.to_bytes()

Argument	Description
RETURNS	The serialized DocBin. bytes

DocBin.from_bytes

Deserialize the DocBin's annotations from a bytestring.

Example

doc_bin_bytes = doc_bin.to_bytes()
new_doc_bin = DocBin().from_bytes(doc_bin_bytes)

Argument	Description
bytes_data	The data to load from. bytes
RETURNS	The loaded DocBin. DocBin

DocBin.to_disk

Save the serialized DocBin to a file. Typically uses the .spacy extension and the result can be used as the input data for spacy train.

Example

docs = [nlp("Hello world!")]
doc_bin = DocBin(docs=docs)
doc_bin.to_disk("./data.spacy")

Argument	Description
path	The file path, typically with the .spacy extension. Union[str, Path]

DocBin.from_disk

Load a serialized DocBin from a file. Typically uses the .spacy extension.

Example

doc_bin = DocBin().from_disk("./data.spacy")

Argument	Description
path	The file path, typically with the .spacy extension. Union[str, Path]
RETURNS	The loaded DocBin. DocBin