spaCy/website/docs/usage/101/_pos-deps.md
Adriane Boyd f42c9026f5
Update v2.3.x branch (#5636)
* Fix typos and auto-format [ci skip]

* Add pkuseg warnings and auto-format [ci skip]

* Update Binder URL [ci skip]

* Update Binder version [ci skip]

* Update alignment example for new gold.align

* Update POS in tagging example

* Fix numpy.zeros() dtype for Doc.from_array

* Change example title to Dr.

Change example title to Dr. so the current model does exclude the title
in the initial example.

* Fix spacy convert argument

* Warning for sudachipy 0.4.5 (#5611)

* Create myavrum.md (#5612)

* Update lex_attrs.py (#5608)

* Create mahnerak.md (#5615)

* Some changes for Armenian (#5616)

* Fixing numericals

* We need a Armenian question sign to make the sentence a question

* Add Nepali Language  (#5622)

* added support for nepali lang

* added examples and test files

* added spacy contributor agreement

* Japanese model: add user_dict entries and small refactor (#5573)

* user_dict fields: adding inflections, reading_forms, sub_tokens
deleting: unidic_tags
improve code readability around the token alignment procedure

* add test cases, replace fugashi with sudachipy in conftest

* move bunsetu.py to spaCy Universe as a pipeline component BunsetuRecognizer

* tag is space -> both surface and tag are spaces

* consider len(text)==0

* Add warnings example in v2.3 migration guide (#5627)

* contribute (#5632)

* Fix polarity of Token.is_oov and Lexeme.is_oov (#5634)

Fix `Token.is_oov` and `Lexeme.is_oov` so they return `True` when the
lexeme does **not** have a vector.

* Extend what's new in v2.3 with vocab / is_oov (#5635)

* Skip vocab in component config overrides (#5624)

* Fix backslashes in warnings config diff (#5640)

Fix backslashes in warnings config diff in v2.3 migration section.

* Disregard special tag  _SP in check for new tag map (#5641)

* Skip special tag  _SP in check for new tag map

In `Tagger.begin_training()` check for new tags aside from `_SP` in the
new tag map initialized from the provided gold tuples when determining
whether to reinitialize the morphology with the new tag map.

* Simplify _SP check

Co-authored-by: Ines Montani <ines@ines.io>
Co-authored-by: Marat M. Yavrumyan <myavrum@ysu.am>
Co-authored-by: Karen Hambardzumyan <mahnerak@gmail.com>
Co-authored-by: Rameshh <30867740+rameshhpathak@users.noreply.github.com>
Co-authored-by: Hiroshi Matsuda <40782025+hiroshi-matsuda-rit@users.noreply.github.com>
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
2020-06-29 14:13:12 +02:00

3.2 KiB
Raw Blame History

After tokenization, spaCy can parse and tag a given Doc. This is where the statistical model comes in, which enables spaCy to make a prediction of which tag or label most likely applies in this context. A model consists of binary data and is produced by showing a system enough examples for it to make predictions that generalize across the language for example, a word following "the" in English is most likely a noun.

Linguistic annotations are available as Token attributes. Like many NLP libraries, spaCy encodes all strings to hash values to reduce memory usage and improve efficiency. So to get the readable string representation of an attribute, we need to add an underscore _ to its name:

### {executable="true"}
import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for token in doc:
    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,
            token.shape_, token.is_alpha, token.is_stop)
  • Text: The original word text.
  • Lemma: The base form of the word.
  • POS: The simple UPOS part-of-speech tag.
  • Tag: The detailed part-of-speech tag.
  • Dep: Syntactic dependency, i.e. the relation between tokens.
  • Shape: The word shape capitalization, punctuation, digits.
  • is alpha: Is the token an alpha character?
  • is stop: Is the token part of a stop list, i.e. the most common words of the language?
Text Lemma POS Tag Dep Shape alpha stop
Apple apple PROPN NNP nsubj Xxxxx True False
is be AUX VBZ aux xx True True
looking look VERB VBG ROOT xxxx True False
at at ADP IN prep xx True True
buying buy VERB VBG pcomp xxxx True False
U.K. u.k. PROPN NNP compound X.X. False False
startup startup NOUN NN dobj xxxx True False
for for ADP IN prep xxx True True
$ $ SYM $ quantmod $ False False
1 1 NUM CD compound d False False
billion billion NUM CD pobj xxxx True False

Tip: Understanding tags and labels

Most of the tags and labels look pretty abstract, and they vary between languages. spacy.explain will show you a short description for example, spacy.explain("VBZ") returns "verb, 3rd person singular present".

Using spaCy's built-in displaCy visualizer, here's what our example sentence and its dependencies look like:

import DisplaCyLongHtml from 'images/displacy-long.html'; import { Iframe } from 'components/embed'