spaCy/spacy/tokens/_serialize.py

205 lines
7.5 KiB
Python
Raw Normal View History

2018-08-22 14:12:51 +03:00
import numpy
import zlib
import srsly
Update spaCy for thinc 8.0.0 (#4920) * Add load_from_config function * Add train_from_config script * Merge configs and expose via spacy.config * Fix script * Suggest create_evaluation_callback * Hard-code for NER * Fix errors * Register command * Add TODO * Update train-from-config todos * Fix imports * Allow delayed setting of parser model nr_class * Get train-from-config working * Tidy up and fix scores and printing * Hide traceback if cancelled * Fix weighted score formatting * Fix score formatting * Make output_path optional * Add Tok2Vec component * Tidy up and add tok2vec_tensors * Add option to copy docs in nlp.update * Copy docs in nlp.update * Adjust nlp.update() for set_annotations * Don't shuffle pipes in nlp.update, decruft * Support set_annotations arg in component update * Support set_annotations in parser update * Add get_gradients method * Add get_gradients to parser * Update errors.py * Fix problems caused by merge * Add _link_components method in nlp * Add concept of 'listeners' and ControlledModel * Support optional attributes arg in ControlledModel * Try having tok2vec component in pipeline * Fix tok2vec component * Fix config * Fix tok2vec * Update for Example * Update for Example * Update config * Add eg2doc util * Update and add schemas/types * Update schemas * Fix nlp.update * Fix tagger * Remove hacks from train-from-config * Remove hard-coded config str * Calculate loss in tok2vec component * Tidy up and use function signatures instead of models * Support union types for registry models * Minor cleaning in Language.update * Make ControlledModel specifically Tok2VecListener * Fix train_from_config * Fix tok2vec * Tidy up * Add function for bilstm tok2vec * Fix type * Fix syntax * Fix pytorch optimizer * Add example configs * Update for thinc describe changes * Update for Thinc changes * Update for dropout/sgd changes * Update for dropout/sgd changes * Unhack gradient update * Work on refactoring _ml * Remove _ml.py module * WIP upgrade cli scripts for thinc * Move some _ml stuff to util * Import link_vectors from util * Update train_from_config * Import from util * Import from util * Temporarily add ml.component_models module * Move ml methods * Move typedefs * Update load vectors * Update gitignore * Move imports * Add PrecomputableAffine * Fix imports * Fix imports * Fix imports * Fix missing imports * Update CLI scripts * Update spacy.language * Add stubs for building the models * Update model definition * Update create_default_optimizer * Fix import * Fix comment * Update imports in tests * Update imports in spacy.cli * Fix import * fix obsolete thinc imports * update srsly pin * from thinc to ml_datasets for example data such as imdb * update ml_datasets pin * using STATE.vectors * small fix * fix Sentencizer.pipe * black formatting * rename Affine to Linear as in thinc * set validate explicitely to True * rename with_square_sequences to with_list2padded * rename with_flatten to with_list2array * chaining layernorm * small fixes * revert Optimizer import * build_nel_encoder with new thinc style * fixes using model's get and set methods * Tok2Vec in component models, various fixes * fix up legacy tok2vec code * add model initialize calls * add in build_tagger_model * small fixes * setting model dims * fixes for ParserModel * various small fixes * initialize thinc Models * fixes * consistent naming of window_size * fixes, removing set_dropout * work around Iterable issue * remove legacy tok2vec * util fix * fix forward function of tok2vec listener * more fixes * trying to fix PrecomputableAffine (not succesful yet) * alloc instead of allocate * add morphologizer * rename residual * rename fixes * Fix predict function * Update parser and parser model * fixing few more tests * Fix precomputable affine * Update component model * Update parser model * Move backprop padding to own function, for test * Update test * Fix p. affine * Update NEL * build_bow_text_classifier and extract_ngrams * Fix parser init * Fix test add label * add build_simple_cnn_text_classifier * Fix parser init * Set gpu off by default in example * Fix tok2vec listener * Fix parser model * Small fixes * small fix for PyTorchLSTM parameters * revert my_compounding hack (iterable fixed now) * fix biLSTM * Fix uniqued * PyTorchRNNWrapper fix * small fixes * use helper function to calculate cosine loss * small fixes for build_simple_cnn_text_classifier * putting dropout default at 0.0 to ensure the layer gets built * using thinc util's set_dropout_rate * moving layer normalization inside of maxout definition to optimize dropout * temp debugging in NEL * fixed NEL model by using init defaults ! * fixing after set_dropout_rate refactor * proper fix * fix test_update_doc after refactoring optimizers in thinc * Add CharacterEmbed layer * Construct tagger Model * Add missing import * Remove unused stuff * Work on textcat * fix test (again :)) after optimizer refactor * fixes to allow reading Tagger from_disk without overwriting dimensions * don't build the tok2vec prematuraly * fix CharachterEmbed init * CharacterEmbed fixes * Fix CharacterEmbed architecture * fix imports * renames from latest thinc update * one more rename * add initialize calls where appropriate * fix parser initialization * Update Thinc version * Fix errors, auto-format and tidy up imports * Fix validation * fix if bias is cupy array * revert for now * ensure it's a numpy array before running bp in ParserStepModel * no reason to call require_gpu twice * use CupyOps.to_numpy instead of cupy directly * fix initialize of ParserModel * remove unnecessary import * fixes for CosineDistance * fix device renaming * use refactored loss functions (Thinc PR 251) * overfitting test for tagger * experimental settings for the tagger: avoid zero-init and subword normalization * clean up tagger overfitting test * use previous default value for nP * remove toy config * bringing layernorm back (had a bug - fixed in thinc) * revert setting nP explicitly * remove setting default in constructor * restore values as they used to be * add overfitting test for NER * add overfitting test for dep parser * add overfitting test for textcat * fixing init for linear (previously affine) * larger eps window for textcat * ensure doc is not None * Require newer thinc * Make float check vaguer * Slop the textcat overfit test more * Fix textcat test * Fix exclusive classes for textcat * fix after renaming of alloc methods * fixing renames and mandatory arguments (staticvectors WIP) * upgrade to thinc==8.0.0.dev3 * refer to vocab.vectors directly instead of its name * rename alpha to learn_rate * adding hashembed and staticvectors dropout * upgrade to thinc 8.0.0.dev4 * add name back to avoid warning W020 * thinc dev4 * update srsly * using thinc 8.0.0a0 ! Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com> Co-authored-by: Ines Montani <ines@ines.io>
2020-01-29 19:06:46 +03:00
from thinc.backends import NumpyOps
2018-08-22 14:12:51 +03:00
from ..compat import copy_reg
2018-09-28 15:27:24 +03:00
from ..tokens import Doc
from ..attrs import SPACY, ORTH, intify_attr
2019-09-18 21:23:21 +03:00
from ..errors import Errors
2018-08-22 14:12:51 +03:00
2019-09-18 16:15:37 +03:00
class DocBin(object):
"""Pack Doc objects for binary serialization.
2019-09-18 16:15:37 +03:00
The DocBin class lets you efficiently serialize the information from a
2019-09-18 14:25:47 +03:00
collection of Doc objects. You can control which information is serialized
by passing a list of attribute IDs, and optionally also specify whether the
2019-09-18 16:15:37 +03:00
user data is serialized. The DocBin is faster and produces smaller data
2019-09-18 14:25:47 +03:00
sizes than pickle, and allows you to deserialize without executing arbitrary
Python code.
2019-09-18 14:25:47 +03:00
The serialization format is gzipped msgpack, where the msgpack object has
the following structure:
2019-09-18 14:25:47 +03:00
{
"attrs": List[uint64], # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE]
"tokens": bytes, # Serialized numpy uint64 array with the token data
"spaces": bytes, # Serialized numpy boolean array with spaces data
"lengths": bytes, # Serialized numpy int32 array with the doc lengths
"strings": List[unicode] # List of unique strings in the token data
}
Strings for the words, tags, labels etc are represented by 64-bit hashes in
the token data, and every string that occurs at least once is passed via the
strings object. This means the storage is more efficient if you pack more
documents together, because you have less duplication in the strings.
A notable downside to this format is that you can't easily extract just one
2019-09-18 21:23:21 +03:00
document from the DocBin.
2019-09-18 14:25:47 +03:00
"""
2019-07-10 20:37:20 +03:00
def __init__(self, attrs=None, store_user_data=False):
2019-09-18 21:23:21 +03:00
"""Create a DocBin object to hold serialized annotations.
attrs (list): List of attributes to serialize. 'orth' and 'spacy' are
always serialized, so they're not required. Defaults to None.
2019-09-18 21:23:21 +03:00
store_user_data (bool): Whether to include the `Doc.user_data`.
RETURNS (DocBin): The newly constructed object.
DOCS: https://spacy.io/api/docbin#init
"""
2018-08-22 14:12:51 +03:00
attrs = attrs or []
attrs = sorted([intify_attr(attr) for attr in attrs])
2019-07-10 20:37:20 +03:00
self.attrs = [attr for attr in attrs if attr != ORTH and attr != SPACY]
2019-09-18 21:23:21 +03:00
self.attrs.insert(0, ORTH) # Ensure ORTH is always attrs[0]
2018-08-22 14:12:51 +03:00
self.tokens = []
self.spaces = []
self.cats = []
2019-07-10 20:37:20 +03:00
self.user_data = []
2018-08-22 14:12:51 +03:00
self.strings = set()
2019-07-10 20:37:20 +03:00
self.store_user_data = store_user_data
2018-08-22 14:12:51 +03:00
2019-09-18 21:23:21 +03:00
def __len__(self):
"""RETURNS: The number of Doc objects added to the DocBin."""
return len(self.tokens)
2018-08-22 14:12:51 +03:00
def add(self, doc):
2019-09-18 21:23:21 +03:00
"""Add a Doc's annotations to the DocBin for serialization.
doc (Doc): The Doc object to add.
DOCS: https://spacy.io/api/docbin#add
"""
2018-08-22 14:12:51 +03:00
array = doc.to_array(self.attrs)
if len(array.shape) == 1:
array = array.reshape((array.shape[0], 1))
self.tokens.append(array)
spaces = doc.to_array(SPACY)
2019-09-18 21:23:21 +03:00
assert array.shape[0] == spaces.shape[0] # this should never happen
2018-08-22 14:12:51 +03:00
spaces = spaces.reshape((spaces.shape[0], 1))
self.spaces.append(numpy.asarray(spaces, dtype=bool))
self.strings.update(w.text for w in doc)
self.cats.append(doc.cats)
2019-07-10 20:37:20 +03:00
if self.store_user_data:
self.user_data.append(srsly.msgpack_dumps(doc.user_data))
2018-08-22 14:12:51 +03:00
def get_docs(self, vocab):
2019-09-18 21:23:21 +03:00
"""Recover Doc objects from the annotations, using the given vocab.
vocab (Vocab): The shared vocab.
YIELDS (Doc): The Doc objects.
DOCS: https://spacy.io/api/docbin#get_docs
"""
2018-08-22 14:12:51 +03:00
for string in self.strings:
vocab[string]
orth_col = self.attrs.index(ORTH)
2019-07-10 20:37:20 +03:00
for i in range(len(self.tokens)):
tokens = self.tokens[i]
spaces = self.spaces[i]
2018-08-22 14:12:51 +03:00
words = [vocab.strings[orth] for orth in tokens[:, orth_col]]
doc = Doc(vocab, words=words, spaces=spaces)
doc = doc.from_array(self.attrs, tokens)
doc.cats = self.cats[i]
2019-07-10 20:37:20 +03:00
if self.store_user_data:
user_data = srsly.msgpack_loads(self.user_data[i], use_list=False)
doc.user_data.update(user_data)
2018-08-22 14:12:51 +03:00
yield doc
def merge(self, other):
2019-09-18 21:23:21 +03:00
"""Extend the annotations of this DocBin with the annotations from
another. Will raise an error if the pre-defined attrs of the two
DocBins don't match.
other (DocBin): The DocBin to merge into the current bin.
DOCS: https://spacy.io/api/docbin#merge
"""
if self.attrs != other.attrs:
raise ValueError(Errors.E166.format(current=self.attrs, other=other.attrs))
2018-08-22 14:12:51 +03:00
self.tokens.extend(other.tokens)
self.spaces.extend(other.spaces)
self.strings.update(other.strings)
self.cats.extend(other.cats)
2019-07-10 20:37:20 +03:00
if self.store_user_data:
self.user_data.extend(other.user_data)
2018-08-22 14:12:51 +03:00
def to_bytes(self):
2019-09-18 21:23:21 +03:00
"""Serialize the DocBin's annotations to a bytestring.
RETURNS (bytes): The serialized DocBin.
DOCS: https://spacy.io/api/docbin#to_bytes
"""
2018-08-22 14:12:51 +03:00
for tokens in self.tokens:
2019-09-18 21:23:21 +03:00
assert len(tokens.shape) == 2, tokens.shape # this should never happen
2018-08-22 14:12:51 +03:00
lengths = [len(tokens) for tokens in self.tokens]
msg = {
"attrs": self.attrs,
"tokens": numpy.vstack(self.tokens).tobytes("C"),
"spaces": numpy.vstack(self.spaces).tobytes("C"),
"lengths": numpy.asarray(lengths, dtype="int32").tobytes("C"),
"strings": list(self.strings),
"cats": self.cats,
2018-08-22 14:12:51 +03:00
}
2019-07-10 20:37:20 +03:00
if self.store_user_data:
msg["user_data"] = self.user_data
return zlib.compress(srsly.msgpack_dumps(msg))
2018-08-22 14:12:51 +03:00
2019-09-18 21:23:21 +03:00
def from_bytes(self, bytes_data):
"""Deserialize the DocBin's annotations from a bytestring.
bytes_data (bytes): The data to load from.
RETURNS (DocBin): The loaded DocBin.
DOCS: https://spacy.io/api/docbin#from_bytes
"""
msg = srsly.msgpack_loads(zlib.decompress(bytes_data))
self.attrs = msg["attrs"]
self.strings = set(msg["strings"])
lengths = numpy.frombuffer(msg["lengths"], dtype="int32")
flat_spaces = numpy.frombuffer(msg["spaces"], dtype=bool)
flat_tokens = numpy.frombuffer(msg["tokens"], dtype="uint64")
2018-08-22 14:12:51 +03:00
shape = (flat_tokens.size // len(self.attrs), len(self.attrs))
flat_tokens = flat_tokens.reshape(shape)
flat_spaces = flat_spaces.reshape((flat_spaces.size, 1))
self.tokens = NumpyOps().unflatten(flat_tokens, lengths)
self.spaces = NumpyOps().unflatten(flat_spaces, lengths)
self.cats = msg["cats"]
2019-07-10 20:37:20 +03:00
if self.store_user_data and "user_data" in msg:
self.user_data = list(msg["user_data"])
2018-08-22 14:12:51 +03:00
for tokens in self.tokens:
2019-09-18 21:23:21 +03:00
assert len(tokens.shape) == 2, tokens.shape # this should never happen
2018-08-22 14:12:51 +03:00
return self
2019-09-18 16:15:37 +03:00
def merge_bins(bins):
2019-07-10 20:37:20 +03:00
merged = None
2019-09-18 16:15:37 +03:00
for byte_string in bins:
2019-07-10 20:37:20 +03:00
if byte_string is not None:
2019-09-18 16:15:37 +03:00
doc_bin = DocBin(store_user_data=True).from_bytes(byte_string)
2019-07-10 20:37:20 +03:00
if merged is None:
2019-09-18 16:15:37 +03:00
merged = doc_bin
2019-07-10 20:37:20 +03:00
else:
2019-09-18 16:15:37 +03:00
merged.merge(doc_bin)
2019-07-10 20:37:20 +03:00
if merged is not None:
return merged.to_bytes()
else:
2019-07-11 12:49:36 +03:00
return b""
2018-08-22 14:12:51 +03:00
def pickle_bin(doc_bin):
return (unpickle_bin, (doc_bin.to_bytes(),))
2018-08-22 14:12:51 +03:00
2019-09-18 16:15:37 +03:00
def unpickle_bin(byte_string):
return DocBin().from_bytes(byte_string)
2018-08-22 14:12:51 +03:00
2019-09-18 16:15:37 +03:00
copy_reg.pickle(DocBin, pickle_bin, unpickle_bin)
2019-07-10 20:37:20 +03:00
# Compatibility, as we had named it this previously.
2019-09-18 16:15:37 +03:00
Binder = DocBin
2019-07-10 20:37:20 +03:00
2019-09-18 16:15:37 +03:00
__all__ = ["DocBin"]