spaCy/website/docs/usage/v2.jade

409 lines
16 KiB
Plaintext
Raw Normal View History

2017-05-17 13:00:11 +03:00
//- 💫 DOCS > USAGE > WHAT'S NEW IN V2.0
include ../../_includes/_mixins
2017-05-20 15:00:21 +03:00
p
p
| On this page, you'll find a summary of the #[+a("#features") new features],
| information on the #[+a("#incompat") backwards incompatibilities],
| including a handy overview of what's been renamed or deprecated.
| To help you make the most of v2.0, we also
| #[strong re-wrote almost all of the usage guides and API docs], and added
| more real-world examples. If you're new to spaCy, or just want to brush
| up on some NLP basics and the details of the library, check out
| the #[+a("/docs/usage/spacy-101") spaCy 101 guide] that explains the most
| important concepts with examples and illustrations.
+h(2, "features") New features
p
| This section contains an overview of the most important
| #[strong new features and improvements]. The #[+a("/docs/api") API docs]
| include additional deprecation notes. New methods and functions that
| were introduced in this version are marked with a #[+tag-new(2)] tag.
2017-05-24 21:54:37 +03:00
+h(3, "features-pipelines") Improved processing pipelines
2017-05-20 15:00:21 +03:00
+aside-code("Example").
2017-05-24 21:54:37 +03:00
# Modify an existing pipeline
nlp = spacy.load('en')
nlp.pipeline.append(my_component)
# Register a factory to create a component
spacy.set_factory('my_factory', my_factory)
nlp = Language(pipeline=['my_factory', mycomponent])
2017-05-20 15:00:21 +03:00
p
2017-05-28 02:30:12 +03:00
| It's now much easier to #[strong customise the pipeline] with your own
| components, functions that receive a #[code Doc] object, modify and
| return it. If your component is stateful, you can define and register a
| factory which receives the shared #[code Vocab] object and returns a
|  component. spaCy's default components can be added to your pipeline by
| using their string IDs. This way, you won't have to worry about finding
| and implementing them simply add #[code "tagger"] to the pipeline,
2017-05-24 21:54:37 +03:00
| and spaCy will know what to do.
2017-05-20 15:00:21 +03:00
2017-05-28 02:30:12 +03:00
+image
include ../../assets/img/docs/pipeline.svg
2017-05-20 15:00:21 +03:00
+infobox
2017-05-24 21:54:37 +03:00
| #[strong API:] #[+api("language") #[code Language]]
| #[strong Usage:] #[+a("/docs/usage/language-processing-pipeline") Processing text]
2017-05-20 15:00:21 +03:00
+h(3, "features-hash-ids") Hash values instead of integer IDs
+aside-code("Example").
doc = nlp(u'I love coffee')
2017-05-29 02:09:26 +03:00
assert doc.vocab.strings[u'coffee'] == 3197928453018144401
assert doc.vocab.strings[3197928453018144401] == u'coffee'
2017-05-28 20:42:44 +03:00
beer_hash = doc.vocab.strings.add(u'beer')
assert doc.vocab.strings[u'beer'] == beer_hash
assert doc.vocab.strings[beer_hash] == u'beer'
p
| The #[+api("stringstore") #[code StringStore]] now resolves all strings
| to hash values instead of integer IDs. This means that the string-to-int
| mapping #[strong no longer depends on the vocabulary state], making a lot
| of workflows much simpler, especially during training. Unlike integer IDs
| in spaCy v1.x, hash values will #[strong always match] even across
| models. Strings can now be added explicitly using the new
| #[+api("stringstore#add") #[code Stringstore.add]] method. A token's hash
| is available via #[code token.orth].
+infobox
| #[strong API:] #[+api("stringstore") #[code StringStore]]
| #[strong Usage:] #[+a("/docs/usage/spacy-101#vocab") Vocab, hashes and lexemes 101]
2017-05-24 21:54:37 +03:00
+h(3, "features-serializer") Saving, loading and serialization
2017-05-20 15:00:21 +03:00
+aside-code("Example").
nlp = spacy.load('en') # shortcut link
nlp = spacy.load('en_core_web_sm') # package
nlp = spacy.load('/path/to/en') # unicode path
nlp = spacy.load(Path('/path/to/en')) # pathlib Path
2017-05-24 21:54:37 +03:00
nlp.to_disk('/path/to/nlp')
nlp = English().from_disk('/path/to/nlp')
p
| spay's serialization API has been made consistent across classes and
2017-05-26 14:22:45 +03:00
| objects. All container classes, i.e. #[code Language], #[code Doc],
| #[code Vocab] and #[code StringStore] now have a #[code to_bytes()],
| #[code from_bytes()], #[code to_disk()] and #[code from_disk()] method
| that supports the Pickle protocol.
2017-05-24 21:54:37 +03:00
2017-05-20 15:00:21 +03:00
p
| The improved #[code spacy.load] makes loading models easier and more
| transparent. You can load a model by supplying its
| #[+a("/docs/usage/models#usage") shortcut link], the name of an installed
2017-05-24 21:54:37 +03:00
| #[+a("/docs/usage/saving-loading#generating") model package] or a path.
2017-05-20 15:00:21 +03:00
| The #[code Language] class to initialise will be determined based on the
2017-05-24 21:54:37 +03:00
| model's settings. For a blank language, you can import the class directly,
| e.g. #[code from spacy.lang.en import English].
2017-05-20 15:00:21 +03:00
+infobox
2017-05-24 21:54:37 +03:00
| #[strong API:] #[+api("spacy#load") #[code spacy.load]], #[+api("binder") #[code Binder]]
2017-05-20 15:00:21 +03:00
| #[strong Usage:] #[+a("/docs/usage/saving-loading") Saving and loading]
2017-05-24 21:54:37 +03:00
+h(3, "features-displacy") displaCy visualizer with Jupyter support
+aside-code("Example").
from spacy import displacy
doc = nlp(u'This is a sentence about Facebook.')
displacy.serve(doc, style='dep') # run the web server
html = displacy.render(doc, style='ent') # generate HTML
p
| Our popular dependency and named entity visualizers are now an official
| part of the spaCy library! displaCy can run a simple web server, or
| generate raw HTML markup or SVG files to be exported. You can pass in one
| or more docs, and customise the style. displaCy also auto-detects whether
| you're running #[+a("https://jupyter.org") Jupyter] and will render the
| visualizations in your notebook.
+infobox
| #[strong API:] #[+api("displacy") #[code displacy]]
| #[strong Usage:] #[+a("/docs/usage/visualizers") Visualizing spaCy]
2017-05-24 00:18:53 +03:00
+h(3, "features-language") Improved language data and lazy loading
p
| Language-specfic data now lives in its own submodule, #[code spacy.lang].
| Languages are lazy-loaded, i.e. only loaded when you import a
| #[code Language] class, or load a model that initialises one. This allows
| languages to contain more custom data, e.g. lemmatizer lookup tables, or
| complex regular expressions. The language data has also been tidied up
2017-05-28 02:30:12 +03:00
| and simplified. spaCy now also supports simple lookup-based lemmatization.
2017-05-20 15:00:21 +03:00
+infobox
| #[strong API:] #[+api("language") #[code Language]]
2017-05-24 21:54:37 +03:00
| #[strong Code:] #[+src(gh("spaCy", "spacy/lang")) spacy/lang]
2017-05-20 15:00:21 +03:00
| #[strong Usage:] #[+a("/docs/usage/adding-languages") Adding languages]
+h(3, "features-matcher") Revised matcher API
+aside-code("Example").
from spacy.matcher import Matcher
matcher = Matcher(nlp.vocab)
2017-05-28 02:30:12 +03:00
matcher.add('HEARTS', None, [{'ORTH': '❤️', 'OP': '+'}])
assert len(matcher) == 1
2017-05-28 02:30:12 +03:00
assert 'HEARTS' in matcher
2017-05-20 15:00:21 +03:00
p
| Patterns can now be added to the matcher by calling
| #[+api("matcher-add") #[code matcher.add()]] with a match ID, an optional
| callback function to be invoked on each match, and one or more patterns.
| This allows you to write powerful, pattern-specific logic using only one
| matcher. For example, you might only want to merge some entity types,
| and set custom flags for other matched patterns.
+infobox
| #[strong API:] #[+api("matcher") #[code Matcher]]
| #[strong Usage:] #[+a("/docs/usage/rule-based-matching") Rule-based matching]
+h(3, "features-models") Neural network models for English, German, French, Spanish and multi-language NER
+aside-code("Example", "bash").
python -m spacy download en # default English model
python -m spacy download de # default German model
python -m spacy download fr # default French model
python -m spacy download es # default Spanish model
python -m spacy download xx_ent_web_md # multi-language NER
2017-05-20 15:00:21 +03:00
p
| spaCy v2.0 comes with new and improved neural network models for English,
| German, French and Spanish, as well as a multi-language named entity
| recognition model trained on Wikipedia. #[strong GPU usage] is now
| supported via #[+a("http://chainer.org") Chainer]'s CuPy module.
2017-05-20 15:00:21 +03:00
+infobox
| #[strong Details:] #[+a("/docs/api/language-models") Languages],
| #[+src(gh("spacy-models")) spacy-models]
| #[strong Usage:] #[+a("/docs/usage/models") Models],
| #[+a("/docs/usage#gpu") Using spaCy with GPU]
2017-05-20 15:00:21 +03:00
2017-05-17 13:00:11 +03:00
+h(2, "incompat") Backwards incompatibilities
2017-05-20 15:00:21 +03:00
+table(["Old", "New"])
2017-05-24 00:18:53 +03:00
+row
+cell
| #[code spacy.en]
| #[code spacy.xx]
+cell
| #[code spacy.lang.en]
| #[code spacy.lang.xx]
+row
+cell #[code spacy.orth]
+cell #[code spacy.lang.xx.lex_attrs]
+row
+cell #[code cli.model]
+cell -
2017-05-20 15:00:21 +03:00
+row
+cell #[code Language.save_to_directory]
+cell #[+api("language#to_disk") #[code Language.to_disk]]
+row
2017-05-28 02:30:12 +03:00
+cell #[code Language.create_make_doc]
+cell #[+api("language#attributes") #[code Language.tokenizer]]
2017-05-20 15:00:21 +03:00
+row
+cell
| #[code Vocab.load]
| #[code Vocab.load_lexemes]
| #[code Vocab.load_vectors]
| #[code Vocab.load_vectors_from_bin_loc]
+cell
| #[+api("vocab#from_disk") #[code Vocab.from_disk]]
| #[+api("vocab#from_bytes") #[code Vocab.from_bytes]]
+row
+cell
| #[code Vocab.dump]
| #[code Vocab.dump_vectors]
+cell
| #[+api("vocab#to_disk") #[code Vocab.to_disk]]
| #[+api("vocab#to_bytes") #[code Vocab.to_bytes]]
+row
+cell
| #[code StringStore.load]
+cell
| #[+api("stringstore#from_disk") #[code StringStore.from_disk]]
| #[+api("stringstore#from_bytes") #[code StringStore.from_bytes]]
+row
+cell
| #[code StringStore.dump]
+cell
| #[+api("stringstore#to_disk") #[code StringStore.to_disk]]
| #[+api("stringstore#to_bytes") #[code StringStore.to_bytes]]
2017-05-28 02:30:12 +03:00
+row
+cell #[code Tokenizer.load]
+cell -
+row
+cell #[code Tagger.load]
+cell
| #[+api("tagger#from_disk") #[code Tagger.from_disk]]
| #[+api("tagger#from_bytes") #[code Tagger.from_bytes]]
+row
+cell #[code DependencyParser.load]
+cell
| #[+api("dependencyparser#from_disk") #[code DependencyParser.from_disk]]
| #[+api("dependencyparser#from_bytes") #[code DependencyParser.from_bytes]]
+row
+cell #[code EntityRecognizer.load]
+cell
| #[+api("entityrecognizer#from_disk") #[code EntityRecognizer.from_disk]]
| #[+api("entityrecognizer#from_bytes") #[code EntityRecognizer.from_bytes]]
2017-05-20 15:00:21 +03:00
+row
+cell #[code Matcher.load]
+cell -
+row
+cell
| #[code Matcher.add_pattern]
| #[code Matcher.add_entity]
+cell #[+api("matcher#add") #[code Matcher.add]]
+row
+cell #[code Matcher.get_entity]
+cell #[+api("matcher#get") #[code Matcher.get]]
2017-05-20 15:00:21 +03:00
+row
+cell #[code Matcher.has_entity]
+cell #[+api("matcher#contains") #[code Matcher.__contains__]]
2017-05-20 15:00:21 +03:00
+row
+cell #[code Doc.read_bytes]
2017-05-28 02:30:12 +03:00
+cell #[+api("binder") #[code Binder]]
2017-05-20 15:00:21 +03:00
+row
+cell #[code Token.is_ancestor_of]
+cell #[+api("token#is_ancestor") #[code Token.is_ancestor]]
2017-05-24 00:40:04 +03:00
+h(2, "migrating") Migrating from spaCy 1.x
p
| If you've mostly been using spaCy for basic text processing, chances are
| you won't even have to change your code at all. For all other cases,
| we've tried to focus...
2017-05-24 00:40:04 +03:00
+infobox("Some tips")
| Before migrating, we strongly recommend writing a few
| #[strong simple tests] specific to how you're using spaCy in your
| application. This makes it easier to check whether your code requires
| changes, and if so, which parts are affected.
| (By the way, feel free contribute your tests to
| #[+src(gh("spaCy", "spacy/tests")) our test suite] this will also ensure
| we never accidentally introduce a bug in a workflow that's
| important to you.) If you've trained your own models, keep in mind that
| your train and runtime inputs must match. This means you'll have to
| #[strong retrain your models] with spaCy v2.0 to make them compatible.
2017-05-20 15:00:21 +03:00
2017-05-24 00:40:04 +03:00
+h(3, "migrating-saving-loading") Saving, loading and serialization
p
| Double-check all calls to #[code spacy.load()] and make sure they don't
2017-05-25 01:56:35 +03:00
| use the #[code path] keyword argument. If you're only loading in binary
| data and not a model package that can construct its own #[code Language]
| class and pipeline, you should now use the
| #[+api("language#from_disk") #[code Language.from_disk()]] method.
2017-05-24 00:40:04 +03:00
2017-05-25 01:56:35 +03:00
+code-new.
nlp = spacy.load('/model')
nlp = English().from_disk('/model/data')
2017-05-24 00:40:04 +03:00
+code-old nlp = spacy.load('en', path='/model')
p
| Review all other code that writes state to disk or bytes.
| All containers, now share the same, consistent API for saving and
| loading. Replace saving with #[code to_disk()] or #[code to_bytes()], and
| loading with #[code from_disk()] and #[code from_bytes()].
+code-new.
nlp.to_disk('/model')
nlp.vocab.to_disk('/vocab')
+code-old.
nlp.save_to_directory('/model')
nlp.vocab.dump('/vocab')
+h(3, "migrating-strings") Strings and hash values
p
| The change from integer IDs to hash values may not actually affect your
| code very much. However, if you're adding strings to the vocab manually,
| you now need to call #[+api("stringstore#add") #[code StringStore.add()]]
| explicitly. You can also now be sure that the string-to-hash mapping will
| always match across vocabularies.
+code-new.
nlp.vocab.strings.add(u'coffee')
2017-05-29 02:09:26 +03:00
nlp.vocab.strings[u'coffee'] # 3197928453018144401
other_nlp.vocab.strings[u'coffee'] # 3197928453018144401
+code-old.
nlp.vocab.strings[u'coffee'] # 3672
other_nlp.vocab.strings[u'coffee'] # 40259
2017-05-24 00:40:04 +03:00
+h(3, "migrating-languages") Processing pipelines and language data
p
| If you're importing language data or #[code Language] classes, make sure
| to change your import statements to import from #[code spacy.lang]. If
| you've added your own custom language, it needs to be moved to
2017-05-25 01:56:35 +03:00
| #[code spacy/lang/xx] and adjusted accordingly.
2017-05-24 00:40:04 +03:00
+code-new from spacy.lang.en import English
+code-old from spacy.en import English
p
2017-05-25 01:56:35 +03:00
| If you've been using custom pipeline components, check out the new
| guide on #[+a("/docs/usage/language-processing-pipelines") processing pipelines].
| Appending functions to the pipeline still works but you might be able
| to make this more convenient by registering "component factories".
| Components of the processing pipeline can now be disabled by passing a
| list of their names to the #[code disable] keyword argument on loading
| or processing.
+code-new.
nlp = spacy.load('en', disable=['tagger', 'ner'])
doc = nlp(u"I don't want parsed", disable=['parser'])
+code-old.
nlp = spacy.load('en', tagger=False, entity=False)
doc = nlp(u"I don't want parsed", parse=False)
2017-05-24 00:40:04 +03:00
+h(3, "migrating-matcher") Adding patterns and callbacks to the matcher
p
| If you're using the matcher, you can now add patterns in one step. This
| should be easy to update simply merge the ID, callback and patterns
| into one call to #[+api("matcher#add") #[code Matcher.add()]].
2017-05-24 00:40:04 +03:00
+code-new.
matcher.add('GoogleNow', merge_phrases, [{ORTH: 'Google'}, {ORTH: 'Now'}])
+code-old.
matcher.add_entity('GoogleNow', on_match=merge_phrases)
matcher.add_pattern('GoogleNow', [{ORTH: 'Google'}, {ORTH: 'Now'}])
p
| If you've been using #[strong acceptor functions], you'll need to move
| this logic into the
| #[+a("/docs/usage/rule-based-matching#on_match") #[code on_match] callbacks].
| The callback function is invoked on every match and will give you access to
| the doc, the index of the current match and all total matches. This lets
| you both accept or reject the match, and define the actions to be
| triggered.