spaCy.io

Load the linguistic analysis pipeline. Loading may take up to a minute, and the instance consumes 2 to 3 gigabytes of memory. The pipeline class is responsible for loading and saving the components, and applying them in sequence. Each component can be passed as an argument to the __init__ function, or left as None, in which case it will be loaded from a classmethod, named e.g. default_vocab.

Common usage is to accept all defaults, in which case loading is simply:

nlp = spacy.en.English()

To keep the default components, but load data from a specified directory, use:

nlp = English(data_dir=u'path/to/data_directory')

To disable (and avoid loading) parts of the processing pipeline:

nlp = English(parser=False, tagger=False, entity=False)

• data_dir – The data directory. If None, value is obtained via the default_data_dir() method.
• vocab –The vocab object, which should be an instance of class spacy.vocab.Vocab. If None, the object is obtained from the default_vocab() class method. The vocab object manages all of the language specific rules and definitions, maintains the cache of lexical types, and manages the word vectors. Because the vocab owns this important data, most objects hold a reference to the vocab.
• tokenizer – The tokenizer, which should be a callable that accepts a unicode string, and returns a Doc object. If set to None, the default tokenizer is constructed from the default_tokenizer() method.
• tagger – The part-of-speech tagger, which should be a callable that accepts a Doc object, and sets the part-of-speech tags in-place. If set to None, the default tagger is constructed from the default_tagger() method.
• parser – The dependency parser, which should be a callable that accepts a Doc object, and sets the syntactic heads and dependency labels in-place. If set to None, the default parser is constructed from the default_parser() method.
• entity – The named entity recognizer, which should be a callable that accepts a Doc object, and sets the named entity annotations in-place. If set to None, the default entity recognizer is constructed from the default_entity() method.
• matcher – The pattern matcher, which should be a callable that accepts a Doc object, and sets annotations in-place. If set to None, the default matcher is constructed from the default_matcher() method.

The main entry point to spaCy. Takes raw unicode text, and returns a Doc object, which can be iterated to access Token and Span objects. spaCy's models are all linear-time, so you can supply documents of arbitrary length, e.g. whole novels.

• text (unicode) –The text to be processed. spaCy expects raw unicode txt – you don't necessarily need to, say, split it into paragraphs. However, depending on your documents, you might be better off applying custom pre-processing. Non-text formatting, e.g. from HTML mark-up, should be removed before sending the document to spaCy. If your documents have a consistent format, you may be able to improve accuracy by pre-processing. For instance, if the first word of your documents are always in upper-case, it may be helpful to normalize them before supplying them to spaCy.
• tag (bool) –Whether to apply the part-of-speech tagger. Required for parsing and entity recognition.
• parse (bool) – Whether to apply the syntactic dependency parser.
• entity (bool) –Whether to apply the named entity recognizer.

# from spacy.en import English
# nlp = English()
doc = nlp('Some text.') # Applies tagger, parser, entity
doc = nlp('Some text.', parse=False) # Applies tagger and entity, not parser
doc = nlp('Some text.', entity=False) # Applies tagger and parser, not entity
doc = nlp('Some text.', tag=False) # Does not apply tagger, entity or parser
doc = nlp('') # Zero-length tokens, not an error
# doc = nlp(b'Some text') <-- Error: need unicode
doc = nlp(b'Some text'.decode('utf8')) # Encode to unicode first.

Parse a sequence of texts into a sequence of Doc objects. Accepts a generator as input, and produces a generator as output. spaCy releases the global interpreter lock around the parser and named entity recognizer, allowing shared-memory parallelism via OpenMP. However, OpenMP is not supported on OSX — so multiple threads will only be used on Linux and Windows.

Internally, .pipe accumulates a buffer of batch_size texts, works on them with n_threads workers in parallel, and then yields the Doc objects one by one. Increasing batch_size results in higher latency (a longer time before the first document is yielded), and higher memory used (for the texts in the buffer), but can allow better parallelism.

texts = [u'One document.', u'...', u'Lots of documents']
# .pipe streams input, and produces streaming output
iter_texts = (texts[i % 3] for i in xrange(100000000))
for i, doc in enumerate(nlp.pipe(iter_texts, batch_size=50, n_threads=4)):
    assert doc.is_parsed
    if i == 100:
        break

• doc[i] Get the Token object at position i, where i is an integer. Negative indexing is supported, and follows the usual Python semantics, i.e. doc[-2] is doc[len(doc) - 2].
• doc[start : end] Get a Span object, starting at position start and ending at position end. For instance, doc[2:5] produces a span consisting of tokens 2, 3 and 4. Stepped slices (e.g. doc[start : end : step]) are not supported, as Span objects must be contiguous (cannot have gaps).
• for token in docIterate over Token objects, from which the annotations can be easily accessed. This is the main way of accessing Token objects, which are the main way annotations are accessed from Python. If faster-than-Python speeds are required, you can instead access the annotations as a numpy array, or access the underlying C data directly from Cython, via Doc.data, an array of TokenC structs. The C API has not yet been finalized, and is subject to change.
• len(doc) The number of tokens in the document.

• lemma / lemma_The "base" of the word, with no inflectional suffixes, e.g. the lemma of "developing" is "develop", the lemma of "geese" is "goose", etc. Note that derivational suffixes are not stripped, e.g. the lemma of "instutitions" is "institution", not "institute". Lemmatization is performed using the WordNet data, but extended to also cover closed-class words such as pronouns. By default, the WN lemmatizer returns "hi" as the lemma of "his". We assign pronouns the lemma -PRON-.

• orth / orth_The form of the word with no string normalization or processing, as it appears in the string, without trailing whitespace.
• lower / lower_The form of the word, but forced to lower-case, i.e. lower = word.orth_.lower()
• shape / shape_A transform of the word's string, to show orthographic features. The characters a-z are mapped to x, A-Z is mapped to X, 0-9 is mapped to d. After these mappings, sequences of 4 or more of the same character are truncated to length 4. Examples: C3Po --> XdXx, favorite --> xxxx, :) --> :)
• prefix / prefix_A length-N substring from the start of the word. Length may vary by language; currently for English n=1, i.e. prefix = word.orth_[:1]
• suffix / suffix_A length-N substring from the end of the word. Length may vary by language; currently for English n=3, i.e. suffix = word.orth_[-3:]

• is_alpha Equivalent to word.orth_.isalpha()
• is_ascii Equivalent to any(ord(c) >= 128 for c in word.orth_)
• is_digit Equivalent to word.orth_.isdigit()
• is_lower Equivalent to word.orth_.islower()
• is_title Equivalent to word.orth_.istitle()
• is_punct Equivalent to word.orth_.ispunct()
• is_space Equivalent to word.orth_.isspace()
• like_url Does the word resembles a URL?
• like_num Does the word represent a number? e.g. “10.9”, “10”, “ten”, etc
• like_email Does the word resemble an email?
• is_oov Is the word out-of-vocabulary?
• is_stopIs the word part of a "stop list"? Stop lists are used to improve the quality of topic models, by filtering out common, domain-general words.

• prob The unigram log-probability of the word, estimated from counts from a large corpus, smoothed using Simple Good Turing estimation.
• cluster The Brown cluster ID of the word. These are often useful features for linear models. If you’re using a non-linear model, particularly a neural net or random forest, consider using the real-valued word representation vector, in Token.repvec, instead.
• vector A “word embedding” representation: a dense real-valued vector that supports similarity queries between words. By default, spaCy currently loads vectors produced by the Levy and Goldberg (2014) dependency-based word2vec model.
• has_vectorA boolean value indicating whether a vector.

• idxStart index of the token in the string
• len(token)Length of the token's orth string, in unicode code-points.
• unicode(token)Same as token.orth_
• str(token)In Python 3, returns token.orth_. In Python 2, returnstoken.orth_.encode('utf8')
• textAn alias for token.orth_.
• text_with_wstoken.orth_ + token.whitespace_, i.e. the form of the word as it appears in the string, trailing whitespace. This is useful when you need to use linguistic features to add inline mark-up to the string.
• whitespace_The number of immediate syntactic children following the word in the string.

• pos / pos_A coarse-grained, less detailed tag that represents the word-class of the token. The set of .pos tags are consistent across languages. The available tags are ADJ, ADP, ADV, AUX, CONJ, DET, INTJ, NOUN, NUM, PART, PRON, PROPN, PUNCT, SCONJ, SYM, VERB, X, EOL, SPACE.

• tag / tag_A fine-grained, more detailed tag that represents the word-class and some basic morphological information for the token. These tags are primarily designed to be good features for subsequent models, particularly the syntactic parser. They are language and treebank dependent. The tagger is trained to predict these fine-grained tags, and then a mapping table is used to reduce them to the coarse-grained .pos tags.

• headThe immediate syntactic head of the token. If the token is the root of its sentence, it is the token itself, i.e. root_token.head is root_token
• childrenAn iterator that yields from lefts, and then yields from rights.
• subtreeAn iterator for the part of the sentence syntactically governed by the word, including the word itself.
• left_edgeThe leftmost edge of the token's subtree
• right_edgeThe rightmost edge of the token's subtree

• ent_typeIf the token is part of an entity, its entity type.
• ent_iobThe IOB (inside, outside, begin) entity recognition tag for the token.

• doc[start : end]
• for entity in doc.ents
• for sentence in doc.sents
• for noun_phrase in doc.noun_chunks
• span = Span(doc, start, end, label=0)

• text_with_wsThe form of the span as it appears in the string, trailing whitespace. This is useful when you need to use linguistic features to add inline mark-up to the string.
• lemma / lemma_Whitespace-concatenated lemmas of each token in the span.
• label / label_The span label, used particularly for named entities.

• orth / orth_The form of the word with no string normalization or processing, as it appears in the string, without trailing whitespace.
• lower / lower_The form of the word, but forced to lower-case, i.e. lower = word.orth_.lower()
• shape / shape_A transform of the word's string, to show orthographic features. The characters a-z are mapped to x, A-Z is mapped to X, 0-9 is mapped to d. After these mappings, sequences of 4 or more of the same character are truncated to length 4. Examples: C3Po --> XdXx, favorite --> xxxx, :) --> :)
• prefix / prefix_A length-N substring from the start of the word. Length may vary by language; currently for English n=1, i.e. prefix = word.orth_[:1]
• suffix / suffix_A length-N substring from the end of the word. Length may vary by language; currently for English n=3, i.e. suffix = word.orth_[-3:]

• is_alpha Equivalent to word.orth_.isalpha()
• is_ascii Equivalent to any(ord(c) >= 128 for c in word.orth_)
• is_digit Equivalent to word.orth_.isdigit()
• is_lower Equivalent to word.orth_.islower()
• is_title Equivalent to word.orth_.istitle()
• is_punct Equivalent to word.orth_.ispunct()
• is_space Equivalent to word.orth_.isspace()
• like_url Does the word resembles a URL?
• like_num Does the word represent a number? e.g. “10.9”, “10”, “ten”, etc
• like_email Does the word resemble an email?
• is_oov Is the word out-of-vocabulary?
• is_stopIs the word part of a "stop list"? Stop lists are used to improve the quality of topic models, by filtering out common, domain-general words.

• prob The unigram log-probability of the word, estimated from counts from a large corpus, smoothed using Simple Good Turing estimation.
• cluster The Brown cluster ID of the word. These are often useful features for linear models. If you’re using a non-linear model, particularly a neural net or random forest, consider using the real-valued word representation vector, in Token.repvec, instead.
• vector A “word embedding” representation: a dense real-valued vector that supports similarity queries between words. By default, spaCy currently loads vectors produced by the Levy and Goldberg (2014) dependency-based word2vec model.
• has_vectorA boolean value indicating whether a vector.

• lexeme = vocab[string]
• lexeme = vocab[i]

• nlp.vocab
• doc.vocab
• span.vocab
• token.vocab
• lexeme.vocab

StringStore.__init__ takes no arguments, so a new instance can be constructed as follows:

string_store = StringStore()

However, in practice you'll usually use the instance owned by the language's vocab object, which all classes hold a reference to:

• english.vocab.strings
• doc.vocab.strings
• span.vocab.strings
• token.vocab.strings
• lexeme.vocab.strings

If you create another instance, it will map strings to different integers – which is usually not what you want.