include ../../_includes/_mixins p By default spaCy loads a #[code data/vocab/vec.bin] file, where the #[em data] directory is within the #[code spacy.en] module directory. This file can be replaced, to customize the word vectors that spaCy loads. You can also replace the word vectors at run-time. +h(2, "replacing-vec-bin") Replacing vec.bin p The function #[code spacy.vocab.write_binary_vectors] creates a word vectors file in spaCy's binary data format. It expects a #[code bz2] file in the following format: +code. word_key1 0.92 0.45 -0.9 0.0 word_key2 0.3 0.1 0.6 0.3 ... p That is, each line is a single entry. Each entry consists of a key string, followed by a sequence of floats. Each entry should have the same number of floats. p The following example script will replace the #[code vec.bin] file with vectors read from a #[code bz2] archive: +code. from spacy.vocab import write_binary_vectors import spacy.en from os import path def main(bz2_loc, bin_loc=None): if bin_loc is None: bin_loc = path.join(path.dirname(spacy.en.__file__), 'data', 'vocab', 'vec.bin') write_binary_vectors(bz2_loc, bin_loc) if __name__ == '__main__': plac.call(main) +h(2, "replace-vectors-archive") Replace the Vectors at Run-Time, From an Archive p Since v0.93, instances of #[code Vocab] allow new vectors to be loaded from #[code bz2] archive files. This allows vectors to be loaded as follows: +code. >>> from spacy.en import English >>> nlp = English() >>> n_dimensions = nlp.vocab.load_vectors('glove.840B.300d.txt.bz2') >>> n_dimensions 300 +h(2, "replace-vectors-per-word") Replace Vectors at Run-Time, Per Word p Since v0.93, you can assign to the #[code .vector] attribute of #[code Lexeme] instances. Tokens of that lexical type will then inherit the updated vector. For instance: +code. >>> from spacy.en import English >>> nlp = English() >>> apples, oranges = nlp(u'apples oranges') <type 'spacy.tokens.token.Token'> >>> apples_lexeme = nlp.vocab[u'apples'] >>> type(apples), type(apples_lexeme) (<type 'spacy.tokens.token.Token'>, <type 'spacy.lexeme.Lexeme'>) >>> sum(apples.vector) 0.56299778164247982 >>> apples_lexeme.vector *= 2 >>> sum(apples.vector) 1.1259955632849596 p All tokens which have the #[code orth] attribute #[em apples] will inherit the updated vector. p Note that the updated vectors won't persist after exit, unless you persist them yourself, and then replace the #[code vec.bin] file as described above. p A popular source of word vectors are the #[a(href="http://nlp.stanford.edu/projects/glove/" target="_blank") GloVe word vectors], particularly those calculated off the #[a(href="https://commoncrawl.org/" target="_blank") Common Crawl]. Note that the provided vector file has a few entries which are not valid UTF8 strings. These should be filtered out. p Future versions of spaCy will allow you to provide a file-like object, instead of a location of a #[bz2] file.