mirror of
https://github.com/explosion/spaCy.git
synced 2024-12-26 01:46:28 +03:00
Always compare lowercase package names
Otherwise, is_package will return False if model name contains uppercase characters. See this issue: https://support.prodi.gy/t/saving-a-trained-ner-model-as-a-loadable-modu le/46/6
This commit is contained in:
ines
parent
cdb2d83e16
commit
7d77dc490f
|
|
@ -181,9 +181,10 @@ def is_package(name):
|
||||||
name (unicode): Name of package.
|
name (unicode): Name of package.
|
||||||
RETURNS (bool): True if installed package, False if not.
|
RETURNS (bool): True if installed package, False if not.
|
||||||
"""
|
"""
|
||||||
|
name = name.lower() # compare package name against lowercase name
|
||||||
packages = pkg_resources.working_set.by_key.keys()
|
packages = pkg_resources.working_set.by_key.keys()
|
||||||
for package in packages:
|
for package in packages:
|
||||||
if package.replace('-', '_') == name:
|
if package.lower().replace('-', '_') == name:
|
||||||
return True
|
return True
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
@ -194,6 +195,7 @@ def get_package_path(name):
|
||||||
name (unicode): Package name.
|
name (unicode): Package name.
|
||||||
RETURNS (Path): Path to installed package.
|
RETURNS (Path): Path to installed package.
|
||||||
"""
|
"""
|
||||||
|
name = name.lower() # use lowercase version to be safe
|
||||||
# Here we're importing the module just to find it. This is worryingly
|
# Here we're importing the module just to find it. This is worryingly
|
||||||
# indirect, but it's otherwise very difficult to find the package.
|
# indirect, but it's otherwise very difficult to find the package.
|
||||||
pkg = importlib.import_module(name)
|
pkg = importlib.import_module(name)
|
||||||
|
|
|
||||||
300
website/usage/spacy-101.jade
Normal file
300
website/usage/spacy-101.jade
Normal file
|
|
@ -0,0 +1,300 @@
|
||||||
|
//- 💫 DOCS > USAGE > SPACY 101
|
||||||
|
|
||||||
|
include ../_includes/_mixins
|
||||||
|
|
||||||
|
p
|
||||||
|
| Whether you're new to spaCy, or just want to brush up on some
|
||||||
|
| NLP basics and implementation details – this page should have you covered.
|
||||||
|
| Each section will explain one of spaCy's features in simple terms and
|
||||||
|
| with examples or illustrations. Some sections will also reappear across
|
||||||
|
| the usage guides as a quick introduction.
|
||||||
|
|
||||||
|
+aside("Help us improve the docs")
|
||||||
|
| Did you spot a mistake or come across explanations that
|
||||||
|
| are unclear? We always appreciate improvement
|
||||||
|
| #[+a(gh("spaCy") + "/issues") suggestions] or
|
||||||
|
| #[+a(gh("spaCy") + "/pulls") pull requests]. You can find a "Suggest
|
||||||
|
| edits" link at the bottom of each page that points you to the source.
|
||||||
|
|
||||||
|
+h(2, "whats-spacy") What's spaCy?
|
||||||
|
|
||||||
|
+grid.o-no-block
|
||||||
|
+grid-col("half")
|
||||||
|
p
|
||||||
|
| spaCy is a #[strong free, open-source library] for advanced
|
||||||
|
| #[strong Natural Language Processing] (NLP) in Python.
|
||||||
|
|
||||||
|
p
|
||||||
|
| If you're working with a lot of text, you'll eventually want to
|
||||||
|
| know more about it. For example, what's it about? What do the
|
||||||
|
| words mean in context? Who is doing what to whom? What companies
|
||||||
|
| and products are mentioned? Which texts are similar to each other?
|
||||||
|
|
||||||
|
p
|
||||||
|
| spaCy is designed specifically for #[strong production use] and
|
||||||
|
| helps you build applications that process and "understand"
|
||||||
|
| large volumes of text. It can be used to build
|
||||||
|
| #[strong information extraction] or
|
||||||
|
| #[strong natural language understanding] systems, or to
|
||||||
|
| pre-process text for #[strong deep learning].
|
||||||
|
|
||||||
|
+table-of-contents
|
||||||
|
+item #[+a("#features") Features]
|
||||||
|
+item #[+a("#annotations") Linguistic annotations]
|
||||||
|
+item #[+a("#annotations-token") Tokenization]
|
||||||
|
+item #[+a("#annotations-pos-deps") POS tags and dependencies]
|
||||||
|
+item #[+a("#annotations-ner") Named entities]
|
||||||
|
+item #[+a("#vectors-similarity") Word vectors and similarity]
|
||||||
|
+item #[+a("#pipelines") Pipelines]
|
||||||
|
+item #[+a("#vocab") Vocab, hashes and lexemes]
|
||||||
|
+item #[+a("#serialization") Serialization]
|
||||||
|
+item #[+a("#training") Training]
|
||||||
|
+item #[+a("#language-data") Language data]
|
||||||
|
+item #[+a("#lightning-tour") Lightning tour]
|
||||||
|
+item #[+a("#architecture") Architecture]
|
||||||
|
+item #[+a("#community") Community & FAQ]
|
||||||
|
|
||||||
|
+h(3, "what-spacy-isnt") What spaCy isn't
|
||||||
|
|
||||||
|
+list
|
||||||
|
+item #[strong spaCy is not a platform or "an API"].
|
||||||
|
| Unlike a platform, spaCy does not provide a software as a service, or
|
||||||
|
| a web application. It's an open-source library designed to help you
|
||||||
|
| build NLP applications, not a consumable service.
|
||||||
|
+item #[strong spaCy is not an out-of-the-box chat bot engine].
|
||||||
|
| While spaCy can be used to power conversational applications, it's
|
||||||
|
| not designed specifically for chat bots, and only provides the
|
||||||
|
| underlying text processing capabilities.
|
||||||
|
+item #[strong spaCy is not research software].
|
||||||
|
| It's built on the latest research, but it's designed to get
|
||||||
|
| things done. This leads to fairly different design decisions than
|
||||||
|
| #[+a("https://github./nltk/nltk") NLTK]
|
||||||
|
| or #[+a("https://stanfordnlp.github.io/CoreNLP/") CoreNLP], which were
|
||||||
|
| created as platforms for teaching and research. The main difference
|
||||||
|
| is that spaCy is integrated and opinionated. spaCy tries to avoid asking
|
||||||
|
| the user to choose between multiple algorithms that deliver equivalent
|
||||||
|
| functionality. Keeping the menu small lets spaCy deliver generally better
|
||||||
|
| performance and developer experience.
|
||||||
|
+item #[strong spaCy is not a company].
|
||||||
|
| It's an open-source library. Our company publishing spaCy and other
|
||||||
|
| software is called #[+a(COMPANY_URL, true) Explosion AI].
|
||||||
|
|
||||||
|
+section("features")
|
||||||
|
+h(2, "features") Features
|
||||||
|
|
||||||
|
p
|
||||||
|
| In the documentation, you'll come across mentions of spaCy's
|
||||||
|
| features and capabilities. Some of them refer to linguistic concepts,
|
||||||
|
| while others are related to more general machine learning
|
||||||
|
| functionality.
|
||||||
|
|
||||||
|
+aside
|
||||||
|
| If one of spaCy's functionalities #[strong needs a model], it means
|
||||||
|
| that you need to have one of the available
|
||||||
|
| #[+a("/models") statistical models] installed. Models are used
|
||||||
|
| to #[strong predict] linguistic annotations – for example, if a word
|
||||||
|
| is a verb or a noun.
|
||||||
|
|
||||||
|
+table(["Name", "Description", "Needs model"])
|
||||||
|
+row
|
||||||
|
+cell #[strong Tokenization]
|
||||||
|
+cell Segmenting text into words, punctuations marks etc.
|
||||||
|
+cell #[+procon("con")]
|
||||||
|
|
||||||
|
+row
|
||||||
|
+cell #[strong Part-of-speech] (POS) #[strong Tagging]
|
||||||
|
+cell Assigning word types to tokens, like verb or noun.
|
||||||
|
+cell #[+procon("pro")]
|
||||||
|
|
||||||
|
+row
|
||||||
|
+cell #[strong Dependency Parsing]
|
||||||
|
+cell
|
||||||
|
| Assigning syntactic dependency labels, describing the
|
||||||
|
| relations between individual tokens, like subject or object.
|
||||||
|
+cell #[+procon("pro")]
|
||||||
|
|
||||||
|
+row
|
||||||
|
+cell #[strong Lemmatization]
|
||||||
|
+cell
|
||||||
|
| Assigning the base forms of words. For example, the lemma of
|
||||||
|
| "was" is "be", and the lemma of "rats" is "rat".
|
||||||
|
+cell #[+procon("pro")]
|
||||||
|
|
||||||
|
+row
|
||||||
|
+cell #[strong Sentence Boundary Detection] (SBD)
|
||||||
|
+cell Finding and segmenting individual sentences.
|
||||||
|
+cell #[+procon("pro")]
|
||||||
|
|
||||||
|
+row
|
||||||
|
+cell #[strong Named Entity Recongition] (NER)
|
||||||
|
+cell
|
||||||
|
| Labelling named "real-world" objects, like persons, companies
|
||||||
|
| or locations.
|
||||||
|
+cell #[+procon("pro")]
|
||||||
|
|
||||||
|
+row
|
||||||
|
+cell #[strong Similarity]
|
||||||
|
+cell
|
||||||
|
| Comparing words, text spans and documents and how similar
|
||||||
|
| they are to each other.
|
||||||
|
+cell #[+procon("pro")]
|
||||||
|
|
||||||
|
+row
|
||||||
|
+cell #[strong Text Classification]
|
||||||
|
+cell
|
||||||
|
| Assigning categories or labels to a whole document, or parts
|
||||||
|
| of a document.
|
||||||
|
+cell #[+procon("pro")]
|
||||||
|
|
||||||
|
+row
|
||||||
|
+cell #[strong Rule-based Matching]
|
||||||
|
+cell
|
||||||
|
| Finding sequences of tokens based on their texts and
|
||||||
|
| linguistic annotations, similar to regular expressions.
|
||||||
|
+cell #[+procon("con")]
|
||||||
|
|
||||||
|
+row
|
||||||
|
+cell #[strong Training]
|
||||||
|
+cell Updating and improving a statistical model's predictions.
|
||||||
|
+cell #[+procon("neutral")]
|
||||||
|
|
||||||
|
+row
|
||||||
|
+cell #[strong Serialization]
|
||||||
|
+cell Saving objects to files or byte strings.
|
||||||
|
+cell #[+procon("neutral")]
|
||||||
|
|
||||||
|
+h(2, "annotations") Linguistic annotations
|
||||||
|
|
||||||
|
p
|
||||||
|
| spaCy provides a variety of linguistic annotations to give you
|
||||||
|
| #[strong insights into a text's grammatical structure]. This
|
||||||
|
| includes the word types, like the parts of speech, and how the words
|
||||||
|
| are related to each other. For example, if you're analysing text, it
|
||||||
|
| makes a huge difference whether a noun is the subject of a sentence,
|
||||||
|
| or the object – or whether "google" is used as a verb, or refers to
|
||||||
|
| the website or company in a specific context.
|
||||||
|
|
||||||
|
p
|
||||||
|
| Once you've downloaded and installed a #[+a("/usage/models") model],
|
||||||
|
| you can load it via #[+api("spacy#load") #[code spacy.load()]]. This will
|
||||||
|
| return a #[code Language] object contaning all components and data needed
|
||||||
|
| to process text. We usually call it #[code nlp]. Calling the #[code nlp]
|
||||||
|
| object on a string of text will return a processed #[code Doc]:
|
||||||
|
|
||||||
|
+code.
|
||||||
|
import spacy
|
||||||
|
|
||||||
|
nlp = spacy.load('en')
|
||||||
|
doc = nlp(u'Apple is looking at buying U.K. startup for $1 billion')
|
||||||
|
|
||||||
|
p
|
||||||
|
| Even though a #[code Doc] is processed – e.g. split into individual words
|
||||||
|
| and annotated – it still holds #[strong all information of the original text],
|
||||||
|
| like whitespace characters. You can always get the offset of a token into the
|
||||||
|
| original string, or reconstruct the original by joining the tokens and their
|
||||||
|
| trailing whitespace. This way, you'll never lose any information
|
||||||
|
| when processing text with spaCy.
|
||||||
|
|
||||||
|
+h(3, "annotations-token") Tokenization
|
||||||
|
|
||||||
|
include _spacy-101/_tokenization
|
||||||
|
|
||||||
|
+infobox
|
||||||
|
| To learn more about how spaCy's tokenization rules work in detail,
|
||||||
|
| how to #[strong customise and replace] the default tokenizer and how to
|
||||||
|
| #[strong add language-specific data], see the usage guides on
|
||||||
|
| #[+a("/usage/adding-languages") adding languages] and
|
||||||
|
| #[+a("/usage/linguistic-features#tokenization") customising the tokenizer].
|
||||||
|
|
||||||
|
+h(3, "annotations-pos-deps") Part-of-speech tags and dependencies
|
||||||
|
+tag-model("dependency parse")
|
||||||
|
|
||||||
|
include _spacy-101/_pos-deps
|
||||||
|
|
||||||
|
+infobox
|
||||||
|
| To learn more about #[strong part-of-speech tagging] and rule-based
|
||||||
|
| morphology, and how to #[strong navigate and use the parse tree]
|
||||||
|
| effectively, see the usage guides on
|
||||||
|
| #[+a("/usage/linguistic-features#pos-tagging") part-of-speech tagging] and
|
||||||
|
| #[+a("/usage/linguistic-features#dependency-parse") using the dependency parse].
|
||||||
|
|
||||||
|
+h(3, "annotations-ner") Named Entities
|
||||||
|
+tag-model("named entities")
|
||||||
|
|
||||||
|
include _spacy-101/_named-entities
|
||||||
|
|
||||||
|
+infobox
|
||||||
|
| To learn more about entity recognition in spaCy, how to
|
||||||
|
| #[strong add your own entities] to a document and how to
|
||||||
|
| #[strong train and update] the entity predictions of a model, see the
|
||||||
|
| usage guides on
|
||||||
|
| #[+a("/usage/linguistic-features#named-entities") named entity recognition] and
|
||||||
|
| #[+a("/usage/training#ner") training the named entity recognizer].
|
||||||
|
|
||||||
|
+h(2, "vectors-similarity") Word vectors and similarity
|
||||||
|
+tag-model("vectors")
|
||||||
|
|
||||||
|
include _spacy-101/_similarity
|
||||||
|
|
||||||
|
include _spacy-101/_word-vectors
|
||||||
|
|
||||||
|
+infobox
|
||||||
|
| To learn more about word vectors, how to #[strong customise them] and
|
||||||
|
| how to load #[strong your own vectors] into spaCy, see the usage
|
||||||
|
| guide on
|
||||||
|
| #[+a("/usage/word-vectors-similarities") using word vectors and semantic similarities].
|
||||||
|
|
||||||
|
+h(2, "pipelines") Pipelines
|
||||||
|
|
||||||
|
include _spacy-101/_pipelines
|
||||||
|
|
||||||
|
+infobox
|
||||||
|
| To learn more about #[strong how processing pipelines work] in detail,
|
||||||
|
| how to enable and disable their components, and how to
|
||||||
|
| #[strong create your own], see the usage guide on
|
||||||
|
| #[+a("/usage/processing-pipelines") language processing pipelines].
|
||||||
|
|
||||||
|
+h(2, "vocab") Vocab, hashes and lexemes
|
||||||
|
|
||||||
|
include _spacy-101/_vocab
|
||||||
|
|
||||||
|
+h(2, "serialization") Serialization
|
||||||
|
|
||||||
|
include _spacy-101/_serialization
|
||||||
|
|
||||||
|
+infobox
|
||||||
|
| To learn more about how to #[strong save and load your own models],
|
||||||
|
| see the usage guide on
|
||||||
|
| #[+a("/usage/training#saving-loading") saving and loading].
|
||||||
|
|
||||||
|
+h(2, "training") Training
|
||||||
|
|
||||||
|
include _spacy-101/_training
|
||||||
|
|
||||||
|
+infobox
|
||||||
|
| To learn more about #[strong training and updating] models, how to create
|
||||||
|
| training data and how to improve spaCy's named entity recognition models,
|
||||||
|
| see the usage guides on #[+a("/usage/training") training].
|
||||||
|
|
||||||
|
+h(2, "language-data") Language data
|
||||||
|
|
||||||
|
include _spacy-101/_language-data
|
||||||
|
|
||||||
|
+infobox
|
||||||
|
| To learn more about the individual components of the language data and
|
||||||
|
| how to #[strong add a new language] to spaCy in preparation for training
|
||||||
|
| a language model, see the usage guide on
|
||||||
|
| #[+a("/usage/adding-languages") adding languages].
|
||||||
|
|
||||||
|
|
||||||
|
+section("lightning-tour")
|
||||||
|
+h(2, "lightning-tour") Lightning tour
|
||||||
|
include _spacy-101/_lightning-tour
|
||||||
|
|
||||||
|
+section("architecture")
|
||||||
|
+h(2, "architecture") Architecture
|
||||||
|
include _spacy-101/_architecture
|
||||||
|
|
||||||
|
+section("community-faq")
|
||||||
|
+h(2, "community") Community & FAQ
|
||||||
|
include _spacy-101/_community-faq
|
||||||
Loading…
Reference in New Issue
Block a user