diff --git a/.github/contributors/luvogels.md b/.github/contributors/luvogels.md new file mode 100644 index 000000000..c967c1cd2 --- /dev/null +++ b/.github/contributors/luvogels.md @@ -0,0 +1,106 @@ +# spaCy contributor agreement + +This spaCy Contributor Agreement (**"SCA"**) is based on the +[Oracle Contributor Agreement](http://www.oracle.com/technetwork/oca-405177.pdf). +The SCA applies to any contribution that you make to any product or project +managed by us (the **"project"**), and sets out the intellectual property rights +you grant to us in the contributed materials. The term **"us"** shall mean +[ExplosionAI UG (haftungsbeschränkt)](https://explosion.ai/legal). The term +**"you"** shall mean the person or entity identified below. + +If you agree to be bound by these terms, fill in the information requested +below and include the filled-in version with your first pull request, under the +folder [`.github/contributors/`](/.github/contributors/). The name of the file +should be your GitHub username, with the extension `.md`. For example, the user +example_user would create the file `.github/contributors/example_user.md`. + +Read this agreement carefully before signing. These terms and conditions +constitute a binding legal agreement. + +## Contributor Agreement + +1. The term "contribution" or "contributed materials" means any source code, +object code, patch, tool, sample, graphic, specification, manual, +documentation, or any other material posted or submitted by you to the project. + +2. With respect to any worldwide copyrights, or copyright applications and +registrations, in your contribution: + + * you hereby assign to us joint ownership, and to the extent that such + assignment is or becomes invalid, ineffective or unenforceable, you hereby + grant to us a perpetual, irrevocable, non-exclusive, worldwide, no-charge, + royalty-free, unrestricted license to exercise all rights under those + copyrights. This includes, at our option, the right to sublicense these same + rights to third parties through multiple levels of sublicensees or other + licensing arrangements; + + * you agree that each of us can do all things in relation to your + contribution as if each of us were the sole owners, and if one of us makes + a derivative work of your contribution, the one who makes the derivative + work (or has it made will be the sole owner of that derivative work; + + * you agree that you will not assert any moral rights in your contribution + against us, our licensees or transferees; + + * you agree that we may register a copyright in your contribution and + exercise all ownership rights associated with it; and + + * you agree that neither of us has any duty to consult with, obtain the + consent of, pay or render an accounting to the other for any use or + distribution of your contribution. + +3. With respect to any patents you own, or that you can license without payment +to any third party, you hereby grant to us a perpetual, irrevocable, +non-exclusive, worldwide, no-charge, royalty-free license to: + + * make, have made, use, sell, offer to sell, import, and otherwise transfer + your contribution in whole or in part, alone or in combination with or + included in any product, work or materials arising out of the project to + which your contribution was submitted, and + + * at our option, to sublicense these same rights to third parties through + multiple levels of sublicensees or other licensing arrangements. + +4. Except as set out above, you keep all right, title, and interest in your +contribution. The rights that you grant to us under these terms are effective +on the date you first submitted a contribution to us, even if your submission +took place before the date you sign these terms. + +5. You covenant, represent, warrant and agree that: + + * Each contribution that you submit is and shall be an original work of + authorship and you can legally grant the rights set out in this SCA; + + * to the best of your knowledge, each contribution will not violate any + third party's copyrights, trademarks, patents, or other intellectual + property rights; and + + * each contribution shall be in compliance with U.S. export control laws and + other applicable export and import laws. You agree to notify us if you + become aware of any circumstance which would make any of the foregoing + representations inaccurate in any respect. We may publicly disclose your + participation in the project, including the fact that you have signed the SCA. + +6. This SCA is governed by the laws of the State of California and applicable +U.S. Federal law. Any choice of law rules will not apply. + +7. Please place an “x” on one of the applicable statement below. Please do NOT +mark both statements: + + * [x] I am signing on behalf of myself as an individual and no other person + or entity, including my employer, has or will have rights with respect my + contributions. + + * [ ] I am signing on behalf of my employer or a legal entity and I have the + actual authority to contractually bind that entity. + +## Contributor Details + +| Field | Entry | +|------------------------------- | -------------------- | +| Name | Leif Uwe Vogelsang | +| Company name (if applicable) | | +| Title or role (if applicable) | | +| Date | 4/27/2017 | +| GitHub username | luvogels | +| Website (optional) | | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 327f3d58e..8a9ab517b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,9 +40,10 @@ To distinguish issues that are opened by us, the maintainers, we usually add a | [`performance`](https://github.com/explosion/spaCy/labels/performance) | Accuracy, speed and memory use problems | | [`tests`](https://github.com/explosion/spaCy/labels/tests) | Missing or incorrect [tests](spacy/tests) | | [`docs`](https://github.com/explosion/spaCy/labels/docs), [`examples`](https://github.com/explosion/spaCy/labels/examples) | Issues related to the [documentation](https://spacy.io/docs) and [examples](spacy/examples) | -| [`models`](https://github.com/explosion/spaCy/labels/models), [`english`](https://github.com/explosion/spaCy/labels/english), [`german`](https://github.com/explosion/spaCy/labels/german) | Issues related to the specific [models](https://github.com/explosion/spacy-models), languages and data | +| [`models`](https://github.com/explosion/spaCy/labels/models), `language / [name]` | Issues related to the specific [models](https://github.com/explosion/spacy-models), languages and data | | [`linux`](https://github.com/explosion/spaCy/labels/linux), [`osx`](https://github.com/explosion/spaCy/labels/osx), [`windows`](https://github.com/explosion/spaCy/labels/windows) | Issues related to the specific operating systems | | [`pip`](https://github.com/explosion/spaCy/labels/pip), [`conda`](https://github.com/explosion/spaCy/labels/conda) | Issues related to the specific package managers | +| [`wip`](https://github.com/explosion/spaCy/labels/wip) | Work in progress | | [`duplicate`](https://github.com/explosion/spaCy/labels/duplicate) | Duplicates, i.e. issues that have been reported before | | [`meta`](https://github.com/explosion/spaCy/labels/meta) | Meta topics, e.g. repo organisation and issue management | | [`help wanted`](https://github.com/explosion/spaCy/labels/help%20wanted), [`help wanted (easy)`](https://github.com/explosion/spaCy/labels/help%20wanted%20%28easy%29) | Requests for contributions | @@ -75,7 +76,7 @@ example_user would create the file `.github/contributors/example_user.md`. ### Fixing bugs -When fixing a bug, first create an [issue](https://github.com/explosion/spaCy/issues) if one does not already exist. The description text can be very short – we don't want to make this too bureaucratic. +When fixing a bug, first create an [issue](https://github.com/explosion/spaCy/issues) if one does not already exist. The description text can be very short – we don't want to make this too bureaucratic. Next, create a test file named `test_issue[ISSUE NUMBER].py` in the [`spacy/tests/regression`](spacy/tests/regression) folder. Test for the bug you're fixing, and make sure the test fails. Next, add and commit your test file referencing the issue number in the commit message. Finally, fix the bug, make sure your test passes and reference the issue in your commit message. @@ -87,7 +88,16 @@ Code should loosely follow [pep8](https://www.python.org/dev/peps/pep-0008/). Re ### Python conventions -All Python code must be written in an **intersection of Python 2 and Python 3**. This is easy in Cython, but somewhat ugly in Python. We could use some extra utilities for this. Please pay particular attention to code that serialises json objects. +All Python code must be written in an **intersection of Python 2 and Python 3**. This is easy in Cython, but somewhat ugly in Python. Logic that deals with Python or platform compatibility should only live in [`spacy.compat`](spacy/compat.py). To distinguish them from the builtin functions, replacement functions are suffixed with an undersocre, for example `unicode_`. If you need to access the user's version or platform information, for example to show more specific error messages, you can use the `is_config()` helper function. + +```python +from .compat import unicode_, json_dumps, is_config + +compatible_unicode = unicode_('hello world') +compatible_json = json_dumps({'key': 'value'}) +if is_config(windows=True, python2=True): + print("You are using Python 2 on Windows.") +``` Code that interacts with the file-system should accept objects that follow the `pathlib.Path` API, without assuming that the object inherits from `pathlib.Path`. If the function is user-facing and takes a path as an argument, it should check whether the path is provided as a string. Strings should be converted to `pathlib.Path` objects. @@ -95,6 +105,8 @@ At the time of writing (v1.7), spaCy's serialization and deserialization functio Although spaCy uses a lot of classes, inheritance is viewed with some suspicion — it's seen as a mechanism of last resort. You should discuss plans to extend the class hierarchy before implementing. +We have a number of conventions around variable naming that are still being documented, and aren't 100% strict. A general policy is that instances of the class `Doc` should by default be called `doc`, `Token` `token`, `Lexeme` `lex`, `Vocab` `vocab` and `Language` `nlp`. You should avoid naming variables that are of other types these names. For instance, don't name a text string `doc` — you should usually call this `text`. Two general code style preferences further help with naming. First, lean away from introducing temporary variables, as these clutter your namespace. This is one reason why comprehension expressions are often preferred. Second, keep your functions shortish, so that can work in a smaller scope. Of course, this is a question of trade-offs. + ### Cython conventions spaCy's core data structures are implemented as [Cython](http://cython.org/) `cdef` classes. Memory is managed through the `cymem.cymem.Pool` class, which allows you to allocate memory which will be freed when the `Pool` object is garbage collected. This means you usually don't have to worry about freeing memory. You just have to decide which Python object owns the memory, and make it own the `Pool`. When that object goes out of scope, the memory will be freed. You do have to take care that no pointers outlive the object that owns them — but this is generally quite easy. @@ -126,7 +138,7 @@ cdef int c_total(const int* int_array, int length) nogil: return total ``` -If this is confusing, consider that the compiler couldn't deal with `for item in int_array:` — there's no length attached to a raw pointer, so how could we figure out where to stop? The length is provided in the slice notation as a solution to this. Note that we don't have to declare the type of `item` in the code above -- the compiler can easily infer it. This gives us tidy code that looks quite like Python, but is exactly as fast as C — because we've made sure the compilation to C is trivial. +If this is confusing, consider that the compiler couldn't deal with `for item in int_array:` — there's no length attached to a raw pointer, so how could we figure out where to stop? The length is provided in the slice notation as a solution to this. Note that we don't have to declare the type of `item` in the code above — the compiler can easily infer it. This gives us tidy code that looks quite like Python, but is exactly as fast as C — because we've made sure the compilation to C is trivial. Your functions cannot be declared `nogil` if they need to create Python objects or call Python functions. This is perfectly okay — you shouldn't torture your code just to get `nogil` functions. However, if your function isn't `nogil`, you should compile your module with `cython -a --cplus my_module.pyx` and open the resulting `my_module.html` file in a browser. This will let you see how Cython is compiling your code. Calls into the Python run-time will be in bright yellow. This lets you easily see whether Cython is able to correctly type your code, or whether there are unexpected problems. @@ -166,7 +178,7 @@ harp server The docs can always use another example or more detail, and they should always be up to date and not misleading. To quickly find the correct file to edit, simply click on the "Suggest edits" button at the bottom of a page. -To make it easy to add content components, we use a [collection of custom mixins](_includes/_mixins.jade), like `+table`, `+list` or `+code`. +To make it easy to add content components, we use a [collection of custom mixins](_includes/_mixins.jade), like `+table`, `+list` or `+code`. 📖 **For more info and troubleshooting guides, check out the [website README](website).** diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index d40dedf55..b64dc8db3 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -6,6 +6,7 @@ This is a list of everyone who has made significant contributions to spaCy, in a * Andreas Grivas, [@andreasgrv](https://github.com/andreasgrv) * Andrew Poliakov, [@pavlin99th](https://github.com/pavlin99th) * Aniruddha Adhikary [@aniruddha-adhikary](https://github.com/aniruddha-adhikary) +* Ben Eyal, [@beneyal](https://github.com/beneyal) * Bhargav Srinivasa, [@bhargavvader](https://github.com/bhargavvader) * Bruno P. Kinoshita, [@kinow](https://github.com/kinow) * Chris DuBois, [@chrisdubois](https://github.com/chrisdubois) @@ -28,6 +29,7 @@ This is a list of everyone who has made significant contributions to spaCy, in a * Juan Miguel Cejuela, [@juanmirocks](https://github.com/juanmirocks) * Kendrick Tan, [@kendricktan](https://github.com/kendricktan) * Kyle P. Johnson, [@kylepjohnson](https://github.com/kylepjohnson) +* Leif Uwe Vogelsang, [@luvogels](https://github.com/luvogels) * Liling Tan, [@alvations](https://github.com/alvations) * Magnus Burton, [@magnusburton](https://github.com/magnusburton) * Mark Amery, [@ExplodingCabbage](https://github.com/ExplodingCabbage) @@ -50,4 +52,5 @@ This is a list of everyone who has made significant contributions to spaCy, in a * Willem van Hage, [@wrvhage](https://github.com/wrvhage) * Wolfgang Seeker, [@wbwseeker](https://github.com/wbwseeker) * Yanhao Yang, [@YanhaoYang](https://github.com/YanhaoYang) +* Yasuaki Uechi, [@uetchy](https://github.com/uetchy) * Yubing Dong, [@tomtung](https://github.com/tomtung) diff --git a/README.rst b/README.rst index d08860fb8..24b0c232a 100644 --- a/README.rst +++ b/README.rst @@ -4,9 +4,10 @@ spaCy: Industrial-strength NLP spaCy is a library for advanced natural language processing in Python and Cython. spaCy is built on the very latest research, but it isn't researchware. It was designed from day one to be used in real products. spaCy currently supports -English and German, as well as tokenization for Chinese, Spanish, Italian, French, -Portuguese, Dutch, Swedish, Finnish, Hungarian, Bengali and Hebrew. It's commercial -open-source software, released under the MIT license. +English, German and French, as well as tokenization for Spanish, Italian, +Portuguese, Dutch, Swedish, Finnish, Norwegian, Hungarian, Bengali, Hebrew, +Chinese and Japanese. It's commercial open-source software, released under the +MIT license. 📊 **Help us improve the library!** `Take the spaCy user survey `_. @@ -320,6 +321,8 @@ and ``--model`` are optional and enable additional tests: =========== ============== =========== Version Date Description =========== ============== =========== +`v1.8.2`_ ``2017-04-26`` French model and small improvements +`v1.8.1`_ ``2017-04-23`` Saving, loading and training bug fixes `v1.8.0`_ ``2017-04-16`` Better NER training, saving and loading `v1.7.5`_ ``2017-04-07`` Bug fixes and new CLI commands `v1.7.3`_ ``2017-03-26`` Alpha support for Hebrew, new CLI commands and bug fixes @@ -351,6 +354,8 @@ Version Date Description `v0.93`_ ``2015-09-22`` Bug fixes to word vectors =========== ============== =========== +.. _v1.8.2: https://github.com/explosion/spaCy/releases/tag/v1.8.2 +.. _v1.8.1: https://github.com/explosion/spaCy/releases/tag/v1.8.1 .. _v1.8.0: https://github.com/explosion/spaCy/releases/tag/v1.8.0 .. _v1.7.5: https://github.com/explosion/spaCy/releases/tag/v1.7.5 .. _v1.7.3: https://github.com/explosion/spaCy/releases/tag/v1.7.3 diff --git a/examples/keras_parikh_entailment/keras_decomposable_attention.py b/examples/keras_parikh_entailment/keras_decomposable_attention.py index c8aaffd25..f9f912501 100644 --- a/examples/keras_parikh_entailment/keras_decomposable_attention.py +++ b/examples/keras_parikh_entailment/keras_decomposable_attention.py @@ -256,9 +256,9 @@ def test_fit_model(): settings = {'lr': 0.001, 'dropout': 0.2, 'gru_encode':True} model = build_model(vectors, shape, settings) - train_X = _generate_X(20, shape[0], vectors.shape[1]) + train_X = _generate_X(20, shape[0], vectors.shape[0]) train_Y = _generate_Y(20, shape[2]) - dev_X = _generate_X(15, shape[0], vectors.shape[1]) + dev_X = _generate_X(15, shape[0], vectors.shape[0]) dev_Y = _generate_Y(15, shape[2]) model.fit(train_X, train_Y, validation_data=(dev_X, dev_Y), nb_epoch=5, diff --git a/examples/training/train_new_entity_type.py b/examples/training/train_new_entity_type.py index d5d9492f1..4eae11c75 100644 --- a/examples/training/train_new_entity_type.py +++ b/examples/training/train_new_entity_type.py @@ -1,4 +1,5 @@ #!/usr/bin/env python +# coding: utf8 """ Example of training an additional entity type @@ -26,14 +27,13 @@ For more details, see the documentation: Developed for: spaCy 1.7.6 Last tested for: spaCy 1.7.6 """ -# coding: utf8 from __future__ import unicode_literals, print_function import random from pathlib import Path +import random import spacy -from spacy.pipeline import EntityRecognizer from spacy.gold import GoldParse from spacy.tagger import Tagger @@ -44,20 +44,44 @@ def train_ner(nlp, train_data, output_dir): doc = nlp.make_doc(raw_text) for word in doc: _ = nlp.vocab[word.orth] - - for itn in range(20): + random.seed(0) + # You may need to change the learning rate. It's generally difficult to + # guess what rate you should set, especially when you have limited data. + nlp.entity.model.learn_rate = 0.001 + for itn in range(1000): random.shuffle(train_data) + loss = 0. for raw_text, entity_offsets in train_data: gold = GoldParse(doc, entities=entity_offsets) + # By default, the GoldParse class assumes that the entities + # described by offset are complete, and all other words should + # have the tag 'O'. You can tell it to make no assumptions + # about the tag of a word by giving it the tag '-'. + # However, this allows a trivial solution to the current + # learning problem: if words are either 'any tag' or 'ANIMAL', + # the model can learn that all words can be tagged 'ANIMAL'. + #for i in range(len(gold.ner)): + #if not gold.ner[i].endswith('ANIMAL'): + # gold.ner[i] = '-' doc = nlp.make_doc(raw_text) nlp.tagger(doc) - loss = nlp.entity.update(doc, gold) + # As of 1.9, spaCy's parser now lets you supply a dropout probability + # This might help the model generalize better from only a few + # examples. + loss += nlp.entity.update(doc, gold, drop=0.9) + if loss == 0: + break + # This step averages the model's weights. This may or may not be good for + # your situation --- it's empirical. nlp.end_training() if output_dir: + if not output_dir.exists(): + output_dir.mkdir() nlp.save_to_directory(output_dir) def main(model_name, output_directory=None): + print("Loading initial model", model_name) nlp = spacy.load(model_name) if output_directory is not None: output_directory = Path(output_directory) @@ -78,16 +102,23 @@ def main(model_name, output_directory=None): ( "they pretend to care about your feelings, those horses", [(48, 54, 'ANIMAL')] + ), + ( + "horses?", + [(0, 6, 'ANIMAL')] ) + ] nlp.entity.add_label('ANIMAL') - ner = train_ner(nlp, train_data, output_directory) + train_ner(nlp, train_data, output_directory) # Test that the entity is recognized doc = nlp('Do you like horses?') + print("Ents in 'Do you like horses?':") for ent in doc.ents: print(ent.label_, ent.text) if output_directory: + print("Loading from", output_directory) nlp2 = spacy.load('en', path=output_directory) nlp2.entity.add_label('ANIMAL') doc2 = nlp2('Do you like horses?') diff --git a/requirements.txt b/requirements.txt index 6212ab3cd..42910d1be 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11,4 +11,5 @@ ujson>=1.35 dill>=0.2,<0.3 requests>=2.13.0,<3.0.0 regex==2017.4.5 +ftfy>=4.4.2,<5.0.0 pytest>=3.0.6,<4.0.0 diff --git a/setup.py b/setup.py old mode 100644 new mode 100755 index f7cd0ddcb..a112a6e80 --- a/setup.py +++ b/setup.py @@ -8,51 +8,13 @@ import contextlib from distutils.command.build_ext import build_ext from distutils.sysconfig import get_python_inc from distutils import ccompiler, msvccompiler - -try: - from setuptools import Extension, setup -except ImportError: - from distutils.core import Extension, setup +from setuptools import Extension, setup, find_packages PACKAGE_DATA = {'': ['*.pyx', '*.pxd', '*.txt', '*.tokens']} -PACKAGES = [ - 'spacy', - 'spacy.data', - 'spacy.cli', - 'spacy.tokens', - 'spacy.en', - 'spacy.de', - 'spacy.zh', - 'spacy.es', - 'spacy.fr', - 'spacy.it', - 'spacy.hu', - 'spacy.pt', - 'spacy.nl', - 'spacy.sv', - 'spacy.fi', - 'spacy.bn', - 'spacy.he', - 'spacy.en.lemmatizer', - 'spacy.cli.converters', - 'spacy.language_data', - 'spacy.serialize', - 'spacy.syntax', - 'spacy.munge', - 'spacy.tests', - 'spacy.tests.matcher', - 'spacy.tests.parser', - 'spacy.tests.serialize', - 'spacy.tests.spans', - 'spacy.tests.stringstore', - 'spacy.tests.tagger', - 'spacy.tests.tokenizer', - 'spacy.tests.doc', - 'spacy.tests.vectors', - 'spacy.tests.vocab'] +PACKAGES = find_packages() MOD_NAMES = [ @@ -105,15 +67,6 @@ LINK_OPTIONS = { # I don't understand this very well yet. See Issue #267 # Fingers crossed! -#if os.environ.get('USE_OPENMP') == '1': -# compile_options['msvc'].append('/openmp') -# -# -#if not sys.platform.startswith('darwin'): -# compile_options['other'].append('-fopenmp') -# link_options['other'].append('-fopenmp') -# - USE_OPENMP_DEFAULT = '1' if sys.platform != 'darwin' else None if os.environ.get('USE_OPENMP', USE_OPENMP_DEFAULT) == '1': if sys.platform == 'darwin': @@ -129,6 +82,7 @@ if os.environ.get('USE_OPENMP', USE_OPENMP_DEFAULT) == '1': COMPILE_OPTIONS['other'].append('-fopenmp') LINK_OPTIONS['other'].append('-fopenmp') + # By subclassing build_extensions we have the actual compiler that will be used which is really known only after finalize_options # http://stackoverflow.com/questions/724664/python-distutils-how-to-get-a-compiler-that-is-going-to-be-used class build_ext_options: @@ -248,7 +202,8 @@ def setup_package(): 'ujson>=1.35', 'dill>=0.2,<0.3', 'requests>=2.13.0,<3.0.0', - 'regex==2017.4.5'], + 'regex==2017.4.5', + 'ftfy>=4.4.2,<5.0.0'], classifiers=[ 'Development Status :: 5 - Production/Stable', 'Environment :: Console', diff --git a/spacy/__init__.py b/spacy/__init__.py index efd6c00c0..2308ce7e4 100644 --- a/spacy/__init__.py +++ b/spacy/__init__.py @@ -3,14 +3,15 @@ from __future__ import unicode_literals from . import util from .deprecated import resolve_model_name -from .cli import info +from .cli.info import info +from .glossary import explain -from . import en, de, zh, es, it, hu, fr, pt, nl, sv, fi, bn, he +from . import en, de, zh, es, it, hu, fr, pt, nl, sv, fi, bn, he, nb, ja _languages = (en.English, de.German, es.Spanish, pt.Portuguese, fr.French, it.Italian, hu.Hungarian, zh.Chinese, nl.Dutch, sv.Swedish, - fi.Finnish, bn.Bengali, he.Hebrew) + fi.Finnish, bn.Bengali, he.Hebrew, nb.Norwegian, ja.Japanese) for _lang in _languages: diff --git a/spacy/about.py b/spacy/about.py index 5e438c7af..ad4a021c2 100644 --- a/spacy/about.py +++ b/spacy/about.py @@ -3,7 +3,7 @@ # https://github.com/pypa/warehouse/blob/master/warehouse/__about__.py __title__ = 'spacy' -__version__ = '1.8.0' +__version__ = '1.8.2' __summary__ = 'Industrial-strength Natural Language Processing (NLP) with Python and Cython' __uri__ = 'https://spacy.io' __author__ = 'Matthew Honnibal' @@ -13,4 +13,4 @@ __license__ = 'MIT' __docs__ = 'https://spacy.io/docs/usage' __download_url__ = 'https://github.com/explosion/spacy-models/releases/download' __compatibility__ = 'https://raw.githubusercontent.com/explosion/spacy-models/master/compatibility.json' -__shortcuts__ = {'en': 'en_core_web_sm', 'de': 'de_core_news_md', 'vectors': 'en_vectors_glove_md'} +__shortcuts__ = 'https://raw.githubusercontent.com/explosion/spacy-models/master/shortcuts.json' diff --git a/spacy/cli/converters/conllu2json.py b/spacy/cli/converters/conllu2json.py index 3c5ebb0e4..cf473b4a0 100644 --- a/spacy/cli/converters/conllu2json.py +++ b/spacy/cli/converters/conllu2json.py @@ -2,6 +2,7 @@ from __future__ import unicode_literals import json +from ...compat import json_dumps from ... import util @@ -29,7 +30,8 @@ def conllu2json(input_path, output_path, n_sents=10, use_morphology=False): output_filename = input_path.parts[-1].replace(".conllu", ".json") output_file = output_path / output_filename - json.dump(docs, output_file.open('w', encoding='utf-8'), indent=2) + with output_file.open('w', encoding='utf-8') as f: + f.write(json_dumps(docs)) util.print_msg("Created {} documents".format(len(docs)), title="Generated output file {}".format(output_file)) diff --git a/spacy/cli/download.py b/spacy/cli/download.py index 06333eabf..0419de118 100644 --- a/spacy/cli/download.py +++ b/spacy/cli/download.py @@ -17,37 +17,45 @@ def download(model=None, direct=False): if direct: download_model('{m}/{m}.tar.gz'.format(m=model)) else: - model_name = about.__shortcuts__[model] if model in about.__shortcuts__ else model + model_name = check_shortcut(model) compatibility = get_compatibility() version = get_version(model_name, compatibility) download_model('{m}-{v}/{m}-{v}.tar.gz'.format(m=model_name, v=version)) link_package(model_name, model, force=True) -def get_compatibility(): - version = about.__version__ - r = requests.get(about.__compatibility__) +def get_json(url, desc): + r = requests.get(url) if r.status_code != 200: util.sys_exit( - "Couldn't fetch compatibility table. Please find the right model for " - "your spaCy installation (v{v}), and download it manually:".format(v=version), + "Couldn't fetch {d}. Please find the right model for your spaCy " + "installation (v{v}), and download it manually:".format(d=desc, v=about.__version__), "python -m spacy.download [full model name + version] --direct", title="Server error ({c})".format(c=r.status_code)) + return r.json() - comp = r.json()['spacy'] + +def check_shortcut(model): + shortcuts = get_json(about.__shortcuts__, "available shortcuts") + return shortcuts.get(model, model) + + +def get_compatibility(): + version = about.__version__ + comp_table = get_json(about.__compatibility__, "compatibility table") + comp = comp_table['spacy'] if version not in comp: util.sys_exit( "No compatible models found for v{v} of spaCy.".format(v=version), title="Compatibility error") - else: - return comp[version] + return comp[version] def get_version(model, comp): if model not in comp: util.sys_exit( "No compatible model found for " - "{m} (spaCy v{v}).".format(m=model, v=about.__version__), + "'{m}' (spaCy v{v}).".format(m=model, v=about.__version__), title="Compatibility error") return comp[model][0] diff --git a/spacy/cli/model.py b/spacy/cli/model.py index d697df05b..3b9a77b93 100644 --- a/spacy/cli/model.py +++ b/spacy/cli/model.py @@ -8,6 +8,7 @@ from pathlib import Path from preshed.counter import PreshCounter from ..vocab import write_binary_vectors +from ..compat import fix_text from .. import util @@ -41,7 +42,7 @@ def create_model(model_path, vectors_path, vocab, oov_prob): with oov_path.open('w') as f: f.write('%f' % oov_prob) if vectors_path: - vectors_dest = model_path / 'vec.bin' + vectors_dest = vocab_path / 'vec.bin' write_binary_vectors(vectors_path.as_posix(), vectors_dest.as_posix()) @@ -76,6 +77,7 @@ def read_clusters(clusters_path): for line in f: try: cluster, word, freq = line.split() + word = fix_text(word) except ValueError: continue # If the clusterer has only seen the word a few times, its diff --git a/spacy/cli/train.py b/spacy/cli/train.py index 3900c7f39..ce4c77416 100644 --- a/spacy/cli/train.py +++ b/spacy/cli/train.py @@ -2,6 +2,7 @@ from __future__ import unicode_literals, division, print_function import json +from collections import defaultdict from ..util import ensure_path from ..scorer import Scorer @@ -62,10 +63,10 @@ def train_model(Language, train_data, dev_data, output_path, tagger_cfg, parser_ for itn, epoch in enumerate(trainer.epochs(n_iter, augment_data=None)): for doc, gold in epoch: trainer.update(doc, gold) - dev_scores = trainer.evaluate(dev_data) if dev_data else [] + dev_scores = trainer.evaluate(dev_data).scores if dev_data else defaultdict(float) print_progress(itn, trainer.nlp.parser.model.nr_weight, trainer.nlp.parser.model.nr_active_feat, - **dev_scores.scores) + **dev_scores) def evaluate(Language, gold_tuples, output_path): diff --git a/spacy/compat.py b/spacy/compat.py index d216994cc..1ca8a59fe 100644 --- a/spacy/compat.py +++ b/spacy/compat.py @@ -2,6 +2,7 @@ from __future__ import unicode_literals import six +import ftfy import sys import ujson @@ -22,6 +23,8 @@ is_windows = sys.platform.startswith('win') is_linux = sys.platform.startswith('linux') is_osx = sys.platform == 'darwin' +fix_text = ftfy.fix_text + if is_python2: bytes_ = str diff --git a/spacy/es/tag_map.py b/spacy/es/tag_map.py index bdeb7250f..dce29c921 100644 --- a/spacy/es/tag_map.py +++ b/spacy/es/tag_map.py @@ -304,4 +304,5 @@ TAG_MAP = { "VERB__VerbForm=Ger": {"morph": "VerbForm=Ger", "pos": "VERB"}, "VERB__VerbForm=Inf": {"morph": "VerbForm=Inf", "pos": "VERB"}, "X___": {"morph": "_", "pos": "X"}, + "SP": {"morph": "_", "pos": "SPACE"} } diff --git a/spacy/fr/tokenizer_exceptions.py b/spacy/fr/tokenizer_exceptions.py index 43806e270..eef7d789d 100644 --- a/spacy/fr/tokenizer_exceptions.py +++ b/spacy/fr/tokenizer_exceptions.py @@ -13,7 +13,7 @@ from ..symbols import * import os import io -import re +import regex as re def get_exceptions(): diff --git a/spacy/glossary.py b/spacy/glossary.py new file mode 100644 index 000000000..4df5264a6 --- /dev/null +++ b/spacy/glossary.py @@ -0,0 +1,294 @@ +# coding: utf8 +from __future__ import unicode_literals + + +def explain(term): + if term in GLOSSARY: + return GLOSSARY[term] + + +GLOSSARY = { + # POS tags + # Universal POS Tags + # http://universaldependencies.org/u/pos/ + + 'ADJ': 'adjective', + 'ADP': 'adposition', + 'ADV': 'adverb', + 'AUX': 'auxiliary', + 'CONJ': 'conjunction', + 'CCONJ': 'coordinating conjunction', + 'DET': 'determiner', + 'INTJ': 'interjection', + 'NOUN': 'noun', + 'NUM': 'numeral', + 'PART': 'particle', + 'PRON': 'pronoun', + 'PROPN': 'proper noun', + 'PUNCT': 'punctuation', + 'SCONJ': 'subordinating conjunction', + 'SYM': 'symbol', + 'VERB': 'verb', + 'X': 'other', + 'EOL': 'end of line', + 'SPACE': 'space', + + + # POS tags (English) + # OntoNotes 5 / Penn Treebank + # https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html + + '.': 'punctuation mark, sentence closer', + ',': 'punctuation mark, comma', + '-LRB-': 'left round bracket', + '-RRB-': 'right round bracket', + '``': 'opening quotation mark', + '""': 'closing quotation mark', + "''": 'closing quotation mark', + ':': 'punctuation mark, colon or ellipsis', + '$': 'symbol, currency', + '#': 'symbol, number sign', + 'AFX': 'affix', + 'CC': 'conjunction, coordinating', + 'CD': 'cardinal number', + 'DT': 'determiner', + 'EX': 'existential there', + 'FW': 'foreign word', + 'HYPH': 'punctuation mark, hyphen', + 'IN': 'conjunction, subordinating or preposition', + 'JJ': 'adjective', + 'JJR': 'adjective, comparative', + 'JJS': 'adjective, superlative', + 'LS': 'list item marker', + 'MD': 'verb, modal auxillary', + 'NIL': 'missing tag', + 'NN': 'noun, singular or mass', + 'NNP': 'noun, proper singular', + 'NNPS': 'noun, proper plural', + 'NNS': 'noun, plural', + 'PDT': 'predeterminer', + 'POS': 'possessive ending', + 'PRP': 'pronoun, personal', + 'PRP$': 'pronoun, possessive', + 'RB': 'adverb', + 'RBR': 'adverb, comparative', + 'RBS': 'adverb, superlative', + 'RP': 'adverb, particle', + 'TO': 'infinitival to', + 'UH': 'interjection', + 'VB': 'verb, base form', + 'VBD': 'verb, past tense', + 'VBG': 'verb, gerund or present participle', + 'VBN': 'verb, past participle', + 'VBP': 'verb, non-3rd person singular present', + 'VBZ': 'verb, 3rd person singular present', + 'WDT': 'wh-determiner', + 'WP': 'wh-pronoun, personal', + 'WP$': 'wh-pronoun, possessive', + 'WRB': 'wh-adverb', + 'SP': 'space', + 'ADD': 'email', + 'NFP': 'superfluous punctuation', + 'GW': 'additional word in multi-word expression', + 'XX': 'unknown', + 'BES': 'auxillary "be"', + 'HVS': 'forms of "have"', + + + # POS Tags (German) + # TIGER Treebank + # http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/TIGERCorpus/annotation/tiger_introduction.pdf + + '$(': 'other sentence-internal punctuation mark', + '$,': 'comma', + '$.': 'sentence-final punctuation mark', + 'ADJA': 'adjective, attributive', + 'ADJD': 'adjective, adverbial or predicative', + 'APPO': 'postposition', + 'APRP': 'preposition; circumposition left', + 'APPRART': 'preposition with article', + 'APZR': 'circumposition right', + 'ART': 'definite or indefinite article', + 'CARD': 'cardinal number', + 'FM': 'foreign language material', + 'ITJ': 'interjection', + 'KOKOM': 'comparative conjunction', + 'KON': 'coordinate conjunction', + 'KOUI': 'subordinate conjunction with "zu" and infinitive', + 'KOUS': 'subordinate conjunction with sentence', + 'NE': 'proper noun', + 'NNE': 'proper noun', + 'PAV': 'pronominal adverb', + 'PROAV': 'pronominal adverb', + 'PDAT': 'attributive demonstrative pronoun', + 'PDS': 'substituting demonstrative pronoun', + 'PIAT': 'attributive indefinite pronoun without determiner', + 'PIDAT': 'attributive indefinite pronoun with determiner', + 'PIS': 'substituting indefinite pronoun', + 'PPER': 'non-reflexive personal pronoun', + 'PPOSAT': 'attributive possessive pronoun', + 'PPOSS': 'substituting possessive pronoun', + 'PRELAT': 'attributive relative pronoun', + 'PRELS': 'substituting relative pronoun', + 'PRF': 'reflexive personal pronoun', + 'PTKA': 'particle with adjective or adverb', + 'PTKANT': 'answer particle', + 'PTKNEG': 'negative particle', + 'PTKVZ': 'separable verbal particle', + 'PTKZU': '"zu" before infinitive', + 'PWAT': 'attributive interrogative pronoun', + 'PWAV': 'adverbial interrogative or relative pronoun', + 'PWS': 'substituting interrogative pronoun', + 'TRUNC': 'word remnant', + 'VAFIN': 'finite verb, auxiliary', + 'VAIMP': 'imperative, auxiliary', + 'VAINF': 'infinitive, auxiliary', + 'VAPP': 'perfect participle, auxiliary', + 'VMFIN': 'finite verb, modal', + 'VMINF': 'infinitive, modal', + 'VMPP': 'perfect participle, modal', + 'VVFIN': 'finite verb, full', + 'VVIMP': 'imperative, full', + 'VVINF': 'infinitive, full', + 'VVIZU': 'infinitive with "zu", full', + 'VVPP': 'perfect participle, full', + 'XY': 'non-word containing non-letter', + + + # Noun chunks + + 'NP': 'noun phrase', + 'PP': 'prepositional phrase', + 'VP': 'verb phrase', + 'ADVP': 'adverb phrase', + 'ADJP': 'adjective phrase', + 'SBAR': 'subordinating conjunction', + 'PRT': 'particle', + 'PNP': 'prepositional noun phrase', + + + # Dependency Labels (English) + # ClearNLP / Universal Dependencies + # https://github.com/clir/clearnlp-guidelines/blob/master/md/specifications/dependency_labels.md + + 'acomp': 'adjectival complement', + 'advcl': 'adverbial clause modifier', + 'advmod': 'adverbial modifier', + 'agent': 'agent', + 'amod': 'adjectival modifier', + 'appos': 'appositional modifier', + 'attr': 'attribute', + 'aux': 'auxiliary', + 'auxpass': 'auxiliary (passive)', + 'cc': 'coordinating conjunction', + 'ccomp': 'clausal complement', + 'complm': 'complementizer', + 'conj': 'conjunct', + 'cop': 'copula', + 'csubj': 'clausal subject', + 'csubjpass': 'clausal subject (passive)', + 'dep': 'unclassified dependent', + 'det': 'determiner', + 'dobj': 'direct object', + 'expl': 'expletive', + 'hmod': 'modifier in hyphenation', + 'hyph': 'hyphen', + 'infmod': 'infinitival modifier', + 'intj': 'interjection', + 'iobj': 'indirect object', + 'mark': 'marker', + 'meta': 'meta modifier', + 'neg': 'negation modifier', + 'nmod': 'modifier of nominal', + 'nn': 'noun compound modifier', + 'npadvmod': 'noun phrase as adverbial modifier', + 'nsubj': 'nominal subject', + 'nsubjpass': 'nominal subject (passive)', + 'num': 'number modifier', + 'number': 'number compound modifier', + 'oprd': 'object predicate', + 'obj': 'object', + 'obl': 'oblique nominal', + 'parataxis': 'parataxis', + 'partmod': 'participal modifier', + 'pcomp': 'complement of preposition', + 'pobj': 'object of preposition', + 'poss': 'possession modifier', + 'possessive': 'possessive modifier', + 'preconj': 'pre-correlative conjunction', + 'prep': 'prepositional modifier', + 'prt': 'particle', + 'punct': 'punctuation', + 'quantmod': 'modifier of quantifier', + 'rcmod': 'relative clause modifier', + 'root': 'root', + 'xcomp': 'open clausal complement', + + + # Dependency labels (German) + # TIGER Treebank + # http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/TIGERCorpus/annotation/tiger_introduction.pdf + # currently missing: 'cc' (comparative complement) because of conflict + # with English labels + + 'ac': 'adpositional case marker', + 'adc': 'adjective component', + 'ag': 'genitive attribute', + 'ams': 'measure argument of adjective', + 'app': 'apposition', + 'avc': 'adverbial phrase component', + 'cd': 'coordinating conjunction', + 'cj': 'conjunct', + 'cm': 'comparative conjunction', + 'cp': 'complementizer', + 'cvc': 'collocational verb construction', + 'da': 'dative', + 'dh': 'discourse-level head', + 'dm': 'discourse marker', + 'ep': 'expletive es', + 'hd': 'head', + 'ju': 'junctor', + 'mnr': 'postnominal modifier', + 'mo': 'modifier', + 'ng': 'negation', + 'nk': 'noun kernel element', + 'nmc': 'numerical component', + 'oa': 'accusative object', + 'oa': 'second accusative object', + 'oc': 'clausal object', + 'og': 'genitive object', + 'op': 'prepositional object', + 'par': 'parenthetical element', + 'pd': 'predicate', + 'pg': 'phrasal genitive', + 'ph': 'placeholder', + 'pm': 'morphological particle', + 'pnc': 'proper noun component', + 'rc': 'relative clause', + 're': 'repeated element', + 'rs': 'reported speech', + 'sb': 'subject', + + + # Named Entity Recognition + # OntoNotes 5 + # https://catalog.ldc.upenn.edu/docs/LDC2013T19/OntoNotes-Release-5.0.pdf + + 'PERSON': 'People, including fictional', + 'NORP': 'Nationalities or religious or political groups', + 'FACILITY': 'Buildings, airports, highways, bridges, etc.', + 'ORG': 'Companies, agencies, institutions, etc.', + 'GPE': 'Countries, cities, states', + 'LOC': 'Non-GPE locations, mountain ranges, bodies of water', + 'PRODUCT': 'Objects, vehicles, foods, etc. (not services)', + 'EVENT': 'Named hurricanes, battles, wars, sports events, etc.', + 'WORK_OF_ART': 'Titles of books, songs, etc.', + 'LANGUAGE': 'Any named language', + 'DATE': 'Absolute or relative dates or periods', + 'TIME': 'Times smaller than a day', + 'PERCENT': 'Percentage, including "%"', + 'MONEY': 'Monetary values, including unit', + 'QUANTITY': 'Measurements, as of weight or distance', + 'ORDINAL': '"first", "second", etc.', + 'CARDINAL': 'Numerals that do not fall under another type' +} diff --git a/spacy/hu/tokenizer_exceptions.py b/spacy/hu/tokenizer_exceptions.py index 85cf72ec9..a6dc47511 100644 --- a/spacy/hu/tokenizer_exceptions.py +++ b/spacy/hu/tokenizer_exceptions.py @@ -1,7 +1,7 @@ # coding: utf8 from __future__ import unicode_literals -import re +import regex as re from spacy.language_data.punctuation import ALPHA_LOWER, CURRENCY from ..language_data.tokenizer_exceptions import _URL_PATTERN diff --git a/spacy/ja/__init__.py b/spacy/ja/__init__.py new file mode 100644 index 000000000..07e40ada6 --- /dev/null +++ b/spacy/ja/__init__.py @@ -0,0 +1,23 @@ +# encoding: utf8 +from __future__ import unicode_literals, print_function + +from os import path + +from ..language import Language +from ..attrs import LANG +from ..tokens import Doc + +from .language_data import * + + +class Japanese(Language): + lang = 'ja' + + def make_doc(self, text): + try: + from janome.tokenizer import Tokenizer + except ImportError: + raise ImportError("The Japanese tokenizer requires the Janome library: " + "https://github.com/mocobeta/janome") + words = [x.surface for x in Tokenizer().tokenize(text)] + return Doc(self.vocab, words=words, spaces=[False]*len(words)) diff --git a/spacy/ja/language_data.py b/spacy/ja/language_data.py new file mode 100644 index 000000000..007ed2b4e --- /dev/null +++ b/spacy/ja/language_data.py @@ -0,0 +1,23 @@ +# encoding: utf8 +from __future__ import unicode_literals + + +# import base language data +from .. import language_data as base + + +# import util functions +from ..language_data import update_exc, strings_to_exc + + +# import language-specific data from files +from .tag_map import TAG_MAP +from .stop_words import STOP_WORDS + + +TAG_MAP = dict(TAG_MAP) +STOP_WORDS = set(STOP_WORDS) + + +# export +__all__ = ["TAG_MAP", "STOP_WORDS"] diff --git a/spacy/ja/stop_words.py b/spacy/ja/stop_words.py new file mode 100644 index 000000000..45bb7a4d8 --- /dev/null +++ b/spacy/ja/stop_words.py @@ -0,0 +1,9 @@ +# encoding: utf8 +from __future__ import unicode_literals + + +# stop words as whitespace-separated list +STOP_WORDS = set(""" +。 +、 +""".split()) diff --git a/spacy/ja/tag_map.py b/spacy/ja/tag_map.py new file mode 100644 index 000000000..f5b6b5040 --- /dev/null +++ b/spacy/ja/tag_map.py @@ -0,0 +1,24 @@ +# encoding: utf8 +from __future__ import unicode_literals + +from ..symbols import * + + +TAG_MAP = { + "ADV": {POS: ADV}, + "NOUN": {POS: NOUN}, + "ADP": {POS: ADP}, + "PRON": {POS: PRON}, + "SCONJ": {POS: SCONJ}, + "PROPN": {POS: PROPN}, + "DET": {POS: DET}, + "SYM": {POS: SYM}, + "INTJ": {POS: INTJ}, + "PUNCT": {POS: PUNCT}, + "NUM": {POS: NUM}, + "AUX": {POS: AUX}, + "X": {POS: X}, + "CONJ": {POS: CONJ}, + "ADJ": {POS: ADJ}, + "VERB": {POS: VERB} +} diff --git a/spacy/language.py b/spacy/language.py index f47b1d0cc..da02df53d 100644 --- a/spacy/language.py +++ b/spacy/language.py @@ -2,7 +2,6 @@ from __future__ import absolute_import, unicode_literals from contextlib import contextmanager import shutil -import ujson from .tokenizer import Tokenizer from .vocab import Vocab @@ -15,7 +14,7 @@ from .syntax.nonproj import PseudoProjectivity from .pipeline import DependencyParser, EntityRecognizer from .syntax.arc_eager import ArcEager from .syntax.ner import BiluoPushDown -from .compat import unicode_ +from .compat import json_dumps from .attrs import IS_STOP from . import attrs from . import orth @@ -188,15 +187,16 @@ class Language(object): @classmethod def setup_directory(cls, path, **configs): + """ + Initialise a model directory. + """ for name, config in configs.items(): directory = path / name if directory.exists(): shutil.rmtree(str(directory)) directory.mkdir() - with (directory / 'config.json').open('wb') as file_: - data = ujson.dumps(config, indent=2) - if isinstance(data, unicode_): - data = data.encode('utf8') + with (directory / 'config.json').open('w') as file_: + data = json_dumps(config) file_.write(data) if not (path / 'vocab').exists(): (path / 'vocab').mkdir() @@ -242,6 +242,15 @@ class Language(object): self.save_to_directory(path) def __init__(self, **overrides): + """ + Create or load the pipeline. + + Arguments: + **overrides: Keyword arguments indicating which defaults to override. + + Returns: + Language: The newly constructed object. + """ if 'data_dir' in overrides and 'path' not in overrides: raise ValueError("The argument 'data_dir' has been renamed to 'path'") path = util.ensure_path(overrides.get('path', True)) @@ -295,7 +304,7 @@ class Language(object): and can contain arbtrary whitespace. Alignment into the original string is preserved. - Args: + Argsuments: text (unicode): The text to be processed. Returns: @@ -344,12 +353,21 @@ class Language(object): yield doc def save_to_directory(self, path): + """ + Save the Vocab, StringStore and pipeline to a directory. + + Arguments: + path (string or pathlib path): Path to save the model. + """ configs = { 'pos': self.tagger.cfg if self.tagger else {}, 'deps': self.parser.cfg if self.parser else {}, 'ner': self.entity.cfg if self.entity else {}, } + path = util.ensure_path(path) + if not path.exists(): + path.mkdir() self.setup_directory(path, **configs) strings_loc = path / 'vocab' / 'strings.json' diff --git a/spacy/language_data/punctuation.py b/spacy/language_data/punctuation.py index f94d91e80..f23b15bbc 100644 --- a/spacy/language_data/punctuation.py +++ b/spacy/language_data/punctuation.py @@ -1,21 +1,8 @@ # coding: utf8 from __future__ import unicode_literals -import re - - -_ALPHA_LOWER = """ -a ä à á â ǎ æ ã å ā ă ą b c ç ć č ĉ ċ c̄ d ð ď e é è ê ë ė ȅ ȩ ẽ ę f g ĝ ğ h i ı -î ï í ī ì ȉ ǐ į ĩ j k ķ l ł ļ m n ñ ń ň ņ o ö ó ò ő ô õ œ ø ō ő ǒ ơ p q r ř ŗ s -ß ś š ş ŝ t ť u ú û ù ú ū ű ǔ ů ų ư v w ŵ x y ÿ ý ỳ ŷ ỹ z ź ž ż þ -""" - - -_ALPHA_UPPER = """ -A Ä À Á  Ǎ Æ Ã Å Ā Ă Ą B C Ç Ć Č Ĉ Ċ C̄ D Ð Ď E É È Ê Ë Ė Ȅ Ȩ Ẽ Ę F G Ĝ Ğ H I İ -Î Ï Í Ī Ì Ȉ Ǐ Į Ĩ J K Ķ L Ł Ļ M N Ñ Ń Ň Ņ O Ö Ó Ò Ő Ô Õ Œ Ø Ō Ő Ǒ Ơ P Q R Ř Ŗ S -Ś Š Ş Ŝ T Ť U Ú Û Ù Ú Ū Ű Ǔ Ů Ų Ư V W Ŵ X Y Ÿ Ý Ỳ Ŷ Ỹ Z Ź Ž Ż Þ -""" +import regex as re +re.DEFAULT_VERSION = re.VERSION1 _UNITS = """ @@ -57,9 +44,16 @@ LIST_PUNCT = list(_PUNCT.strip().split()) LIST_HYPHENS = list(_HYPHENS.strip().split()) -ALPHA_LOWER = _ALPHA_LOWER.strip().replace(' ', '').replace('\n', '') -ALPHA_UPPER = _ALPHA_UPPER.strip().replace(' ', '').replace('\n', '') -ALPHA = ALPHA_LOWER + ALPHA_UPPER +BENGALI = r'[\p{L}&&\p{Bengali}]' +HEBREW = r'[\p{L}&&\p{Hebrew}]' +LATIN_LOWER = r'[\p{Ll}&&\p{Latin}]' +LATIN_UPPER = r'[\p{Lu}&&\p{Latin}]' +LATIN = r'[[\p{Ll}||\p{Lu}]&&\p{Latin}]' + + +ALPHA_LOWER = '[{}]'.format('||'.join([BENGALI, HEBREW, LATIN_LOWER])) +ALPHA_UPPER = '[{}]'.format('||'.join([BENGALI, HEBREW, LATIN_UPPER])) +ALPHA = '[{}]'.format('||'.join([BENGALI, HEBREW, LATIN])) QUOTES = _QUOTES.strip().replace(' ', '|') diff --git a/spacy/nb/__init__.py b/spacy/nb/__init__.py new file mode 100644 index 000000000..7d4b471a2 --- /dev/null +++ b/spacy/nb/__init__.py @@ -0,0 +1,25 @@ +# encoding: utf8 +from __future__ import unicode_literals, print_function + +from os import path + +from ..language import Language +from ..attrs import LANG + + +# Import language-specific data +from .language_data import * + + +# create Language subclass +class Norwegian(Language): + lang = 'nb' # ISO code + + class Defaults(Language.Defaults): + lex_attr_getters = dict(Language.Defaults.lex_attr_getters) + lex_attr_getters[LANG] = lambda text: 'nb' + + # override defaults + tokenizer_exceptions = TOKENIZER_EXCEPTIONS + #tag_map = TAG_MAP + stop_words = STOP_WORDS diff --git a/spacy/nb/language_data.py b/spacy/nb/language_data.py new file mode 100644 index 000000000..248b09fc7 --- /dev/null +++ b/spacy/nb/language_data.py @@ -0,0 +1,28 @@ +# encoding: utf8 +from __future__ import unicode_literals + + +# import base language data +from .. import language_data as base + + +# import util functions +from ..language_data import update_exc, strings_to_exc, expand_exc + +# import language-specific data from files +#from .tag_map import TAG_MAP +from .stop_words import STOP_WORDS +from .tokenizer_exceptions import TOKENIZER_EXCEPTIONS, ORTH_ONLY +from .morph_rules import MORPH_RULES + +TOKENIZER_EXCEPTIONS = dict(TOKENIZER_EXCEPTIONS) +#TAG_MAP = dict(TAG_MAP) +STOP_WORDS = set(STOP_WORDS) + +# customize tokenizer exceptions +update_exc(TOKENIZER_EXCEPTIONS, strings_to_exc(ORTH_ONLY)) +update_exc(TOKENIZER_EXCEPTIONS, expand_exc(TOKENIZER_EXCEPTIONS, "'", "’")) +update_exc(TOKENIZER_EXCEPTIONS, strings_to_exc(base.EMOTICONS)) + +# export +__all__ = ["TOKENIZER_EXCEPTIONS", "STOP_WORDS", "MORPH_RULES"] diff --git a/spacy/nb/morph_rules.py b/spacy/nb/morph_rules.py new file mode 100644 index 000000000..38498513b --- /dev/null +++ b/spacy/nb/morph_rules.py @@ -0,0 +1,67 @@ +# encoding: utf8 +# norwegian bokmål +from __future__ import unicode_literals + +from ..symbols import * +from ..language_data import PRON_LEMMA + +# Used the table of pronouns at https://no.wiktionary.org/wiki/Tillegg:Pronomen_i_norsk + +MORPH_RULES = { + "PRP": { + "jeg": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Sing", "Case": "Nom"}, + "meg": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Sing", "Case": "Acc"}, + "du": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Two", "Number": "Sing", "Case": "Nom"}, + "deg": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Two", "Number": "Sing", "Case": "Acc"}, + "han": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Gender": "Masc", "Case": "Nom"}, + "ham": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Gender": "Masc", "Case": "Acc"}, + "han": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Gender": "Masc", "Case": "Acc"}, + "hun": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Gender": "Fem", "Case": "Nom"}, + "henne": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Gender": "Fem", "Case": "Acc"}, + "den": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Gender": "Neut"}, + "det": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Gender": "Neut"}, + "seg": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Reflex": "Yes"}, + "vi": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Plur", "Case": "Nom"}, + "oss": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Plur", "Case": "Acc"}, + "dere": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Two", "Number": "Plur", "Case": "Nom"}, + "de": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Plur", "Case": "Nom"}, + "dem": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Plur", "Case": "Acc"}, + "seg": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Plur", "Reflex": "Yes"}, + + "min": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Sing", "Poss": "Yes", "Gender": "Masc"}, + "mi": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Sing", "Poss": "Yes", "Gender": "Fem"}, + "mitt": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Sing", "Poss": "Yes", "Gender": "Neu"}, + "mine": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Plur", "Poss": "Yes"}, + "din": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Two", "Number": "Sing", "Poss": "Yes", "Gender": "Masc"}, + "di": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Two", "Number": "Sing", "Poss": "Yes", "Gender": "Fem"}, + "ditt": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Two", "Number": "Sing", "Poss": "Yes", "Gender": "Neu"}, + "dine": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Two", "Number": "Plur", "Poss": "Yes"}, + "hans": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Poss": "Yes", "Gender": "Masc"}, + "hennes": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Poss": "Yes", "Gender": "Fem"}, + "dens": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Poss": "Yes", "Gender": "Neu"}, + "dets": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "Three", "Number": "Sing", "Poss": "Yes", "Gender": "Neu"}, + "vår": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Plur", "Poss": "Yes"}, + "vårt": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Plur", "Poss": "Yes"}, + "våre": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Plur", "Poss": "Yes", "Gender":"Neu"}, + "deres": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Plur", "Poss": "Yes", "Gender":"Neu", "Reflex":"Yes"}, + "sin": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Sing", "Poss": "Yes", "Gender":"Masc", "Reflex":"Yes"}, + "si": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Sing", "Poss": "Yes", "Gender":"Fem", "Reflex":"Yes"}, + "sitt": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Sing", "Poss": "Yes", "Gender":"Neu", "Reflex":"Yes"}, + "sine": {LEMMA: PRON_LEMMA, "PronType": "Prs", "Person": "One", "Number": "Plur", "Poss": "Yes", "Reflex":"Yes"}, + }, + + "VBZ": { + "er": {LEMMA: "be", "VerbForm": "Fin", "Person": "One", "Tense": "Pres", "Mood": "Ind"}, + "er": {LEMMA: "be", "VerbForm": "Fin", "Person": "Two", "Tense": "Pres", "Mood": "Ind"}, + "er": {LEMMA: "be", "VerbForm": "Fin", "Person": "Three", "Tense": "Pres", "Mood": "Ind"}, + }, + + "VBP": { + "er": {LEMMA: "be", "VerbForm": "Fin", "Tense": "Pres", "Mood": "Ind"} + }, + + "VBD": { + "var": {LEMMA: "be", "VerbForm": "Fin", "Tense": "Past", "Number": "Sing"}, + "vært": {LEMMA: "be", "VerbForm": "Fin", "Tense": "Past", "Number": "Plur"} + } +} diff --git a/spacy/nb/stop_words.py b/spacy/nb/stop_words.py new file mode 100644 index 000000000..721ba2e47 --- /dev/null +++ b/spacy/nb/stop_words.py @@ -0,0 +1,49 @@ +# encoding: utf8 +from __future__ import unicode_literals + +STOP_WORDS = set(""" +alle allerede alt and andre annen annet at av + +bak bare bedre beste blant ble bli blir blitt bris by både + +da dag de del dem den denne der dermed det dette disse drept du + +eller en enn er et ett etter + +fem fikk fire fjor flere folk for fortsatt fotball fra fram frankrike fredag funnet få får fått før først første + +gang gi gikk gjennom gjorde gjort gjør gjøre god godt grunn gå går + +ha hadde ham han hans har hele helt henne hennes her hun hva hvor hvordan hvorfor + +i ifølge igjen ikke ingen inn + +ja jeg + +kamp kampen kan kl klart kom komme kommer kontakt kort kroner kunne kveld kvinner + +la laget land landet langt leder ligger like litt løpet lørdag + +man mandag mange mannen mars med meg mellom men mener menn mennesker mens mer millioner minutter mot msci mye må mål måtte + +ned neste noe noen nok norge norsk norske ntb ny nye nå når + +og også om onsdag opp opplyser oslo oss over + +personer plass poeng politidistrikt politiet president prosent på + +regjeringen runde rundt russland + +sa saken samme sammen samtidig satt se seg seks selv senere september ser sett siden sier sin sine siste sitt skal skriver skulle slik som sted stedet stor store står sverige svært så søndag + +ta tatt tid tidligere til tilbake tillegg tirsdag to tok torsdag tre tror tyskland + +under usa ut uten utenfor + +vant var ved veldig vi videre viktig vil ville viser vår være vært + +å år + +ønsker + +""".split()) diff --git a/spacy/nb/tokenizer_exceptions.py b/spacy/nb/tokenizer_exceptions.py new file mode 100644 index 000000000..ea7658c87 --- /dev/null +++ b/spacy/nb/tokenizer_exceptions.py @@ -0,0 +1,175 @@ +# encoding: utf8 +# Norwegian bokmaål +from __future__ import unicode_literals + +from ..symbols import * +from ..language_data import PRON_LEMMA + + +TOKENIZER_EXCEPTIONS = { + "jan.": [ + {ORTH: "jan.", LEMMA: "januar"} + ], + + "feb.": [ + {ORTH: "feb.", LEMMA: "februar"} + ], + + "jul.": [ + {ORTH: "jul.", LEMMA: "juli"} + ] +} + + +ORTH_ONLY = ["adm.dir.", + "a.m.", + "Aq.", + "b.c.", + "bl.a.", + "bla.", + "bm.", + "bto.", + "ca.", + "cand.mag.", + "c.c.", + "co.", + "d.d.", + "dept.", + "d.m.", + "dr.philos.", + "dvs.", + "d.y.", + "E. coli", + "eg.", + "ekskl.", + "e.Kr.", + "el.", + "e.l.", + "et.", + "etg.", + "ev.", + "evt.", + "f.", + "f.eks.", + "fhv.", + "fk.", + "f.Kr.", + "f.o.m.", + "foreg.", + "fork.", + "fv.", + "fvt.", + "g.", + "gt.", + "gl.", + "gno.", + "gnr.", + "grl.", + "hhv.", + "hoh.", + "hr.", + "h.r.adv.", + "ifb.", + "ifm.", + "iht.", + "inkl.", + "istf.", + "jf.", + "jr.", + "jun.", + "kfr.", + "kgl.res.", + "kl.", + "komm.", + "kst.", + "lø.", + "ma.", + "mag.art.", + "m.a.o.", + "md.", + "mfl.", + "mill.", + "min.", + "m.m.", + "mnd.", + "moh.", + "Mr.", + "muh.", + "mv.", + "mva.", + "ndf.", + "no.", + "nov.", + "nr.", + "nto.", + "nyno.", + "n.å.", + "o.a.", + "off.", + "ofl.", + "okt.", + "o.l.", + "on.", + "op.", + "osv.", + "ovf.", + "p.", + "p.a.", + "Pb.", + "pga.", + "ph.d.", + "pkt.", + "p.m.", + "pr.", + "pst.", + "p.t.", + "red.anm.", + "ref.", + "res.", + "res.kap.", + "resp.", + "rv.", + "s.", + "s.d.", + "sen.", + "sep.", + "siviling.", + "sms.", + "spm.", + "sr.", + "sst.", + "st.", + "stip.", + "stk.", + "st.meld.", + "st.prp.", + "stud.", + "s.u.", + "sv.", + "sø.", + "s.å.", + "såk.", + "temp.", + "ti.", + "tils.", + "tilsv.", + "tl;dr", + "tlf.", + "to.", + "t.o.m.", + "ult.", + "utg.", + "v.", + "vedk.", + "vedr.", + "vg.", + "vgs.", + "vha.", + "vit.ass.", + "vn.", + "vol.", + "vs.", + "vsa.", + "årg.", + "årh." +] diff --git a/spacy/syntax/iterators.pyx b/spacy/syntax/iterators.pyx index c3ba2ca92..e1c44da7f 100644 --- a/spacy/syntax/iterators.pyx +++ b/spacy/syntax/iterators.pyx @@ -10,15 +10,22 @@ def english_noun_chunks(obj): Works on both Doc and Span. """ labels = ['nsubj', 'dobj', 'nsubjpass', 'pcomp', 'pobj', - 'attr', 'ROOT', 'root'] + 'attr', 'ROOT'] doc = obj.doc # Ensure works on both Doc and Span. np_deps = [doc.vocab.strings[label] for label in labels] conj = doc.vocab.strings['conj'] np_label = doc.vocab.strings['NP'] + seen = set() for i, word in enumerate(obj): if word.pos not in (NOUN, PROPN, PRON): continue + # Prevent nested chunks from being produced + if word.i in seen: + continue if word.dep in np_deps: + if any(w.i in seen for w in word.subtree): + continue + seen.update(j for j in range(word.left_edge.i, word.i+1)) yield word.left_edge.i, word.i+1, np_label elif word.dep == conj: head = word.head @@ -26,6 +33,9 @@ def english_noun_chunks(obj): head = head.head # If the head is an NP, and we're coordinated to it, we're an NP if head.dep in np_deps: + if any(w.i in seen for w in word.subtree): + continue + seen.update(j for j in range(word.left_edge.i, word.i+1)) yield word.left_edge.i, word.i+1, np_label diff --git a/spacy/syntax/parser.pyx b/spacy/syntax/parser.pyx index 0bc9cb4ef..b9de1e114 100644 --- a/spacy/syntax/parser.pyx +++ b/spacy/syntax/parser.pyx @@ -11,6 +11,8 @@ import ujson cimport cython cimport cython.parallel +import numpy.random + from cpython.ref cimport PyObject, Py_INCREF, Py_XDECREF from cpython.exc cimport PyErr_CheckSignals from libc.stdint cimport uint32_t, uint64_t @@ -146,7 +148,7 @@ cdef class Parser: if 'labels' in cfg and 'actions' not in cfg: cfg['actions'] = cfg.pop('labels') # TODO: remove this shim when we don't have to support older data - for action_name, labels in dict(cfg['actions']).items(): + for action_name, labels in dict(cfg.get('actions', {})).items(): # We need this to be sorted if isinstance(labels, dict): labels = list(sorted(labels.keys())) @@ -174,13 +176,14 @@ cdef class Parser: if TransitionSystem is None: TransitionSystem = self.TransitionSystem self.vocab = vocab - actions = TransitionSystem.get_actions(**cfg) - self.moves = TransitionSystem(vocab.strings, actions) + cfg['actions'] = TransitionSystem.get_actions(**cfg) + self.moves = TransitionSystem(vocab.strings, cfg['actions']) # TODO: Remove this when we no longer need to support old-style models if isinstance(cfg.get('features'), basestring): cfg['features'] = get_templates(cfg['features']) elif 'features' not in cfg: cfg['features'] = self.feature_templates + self.model = ParserModel(cfg['features']) self.model.l1_penalty = cfg.get('L1', 0.0) self.model.learn_rate = cfg.get('learn_rate', 0.001) @@ -302,7 +305,7 @@ cdef class Parser: free(eg.is_valid) return 0 - def update(self, Doc tokens, GoldParse gold, itn=0): + def update(self, Doc tokens, GoldParse gold, itn=0, double drop=0.0): """ Update the statistical model. @@ -324,9 +327,11 @@ cdef class Parser: nr_feat=self.model.nr_feat) cdef weight_t loss = 0 cdef Transition action + cdef double dropout_rate = self.cfg.get('dropout', drop) while not stcls.is_final(): eg.c.nr_feat = self.model.set_featuresC(eg.c.atoms, eg.c.features, stcls.c) + dropout(eg.c.features, eg.c.nr_feat, dropout_rate) self.moves.set_costs(eg.c.is_valid, eg.c.costs, stcls, gold) self.model.set_scoresC(eg.c.scores, eg.c.features, eg.c.nr_feat) guess = VecVec.arg_max_if_true(eg.c.scores, eg.c.is_valid, eg.c.nr_class) @@ -377,6 +382,18 @@ cdef class Parser: self.cfg.setdefault('extra_labels', []).append(label) +cdef int dropout(FeatureC* feats, int nr_feat, float prob) except -1: + if prob <= 0 or prob >= 1.: + return 0 + cdef double[::1] py_probs = numpy.random.uniform(0., 1., nr_feat) + cdef double* probs = &py_probs[0] + for i in range(nr_feat): + if probs[i] >= prob: + feats[i].value /= prob + else: + feats[i].value = 0. + + cdef class StepwiseState: cdef readonly StateClass stcls cdef readonly Example eg diff --git a/spacy/tests/conftest.py b/spacy/tests/conftest.py index 37d3180d0..b8ada1d9a 100644 --- a/spacy/tests/conftest.py +++ b/spacy/tests/conftest.py @@ -13,6 +13,8 @@ from ..hu import Hungarian from ..fi import Finnish from ..bn import Bengali from ..he import Hebrew +from ..nb import Norwegian + from ..tokens import Doc from ..strings import StringStore @@ -26,7 +28,7 @@ import pytest LANGUAGES = [English, German, Spanish, Italian, French, Portuguese, Dutch, - Swedish, Hungarian, Finnish, Bengali] + Swedish, Hungarian, Finnish, Bengali, Norwegian] @pytest.fixture(params=LANGUAGES) @@ -88,6 +90,9 @@ def bn_tokenizer(): def he_tokenizer(): return Hebrew.Defaults.create_tokenizer() +@pytest.fixture +def nb_tokenizer(): + return Norwegian.Defaults.create_tokenizer() @pytest.fixture def stringstore(): diff --git a/spacy/tests/he/test_tokenizer.py b/spacy/tests/he/test_tokenizer.py index a6c65805a..62ae84223 100644 --- a/spacy/tests/he/test_tokenizer.py +++ b/spacy/tests/he/test_tokenizer.py @@ -3,15 +3,21 @@ from __future__ import unicode_literals import pytest -ABBREVIATION_TESTS = [ - ('פייתון היא שפת תכנות דינמית', ['פייתון', 'היא', 'שפת', 'תכנות', 'דינמית']) -] -TESTCASES = ABBREVIATION_TESTS - - -@pytest.mark.parametrize('text,expected_tokens', TESTCASES) -def test_tokenizer_handles_testcases(he_tokenizer, text, expected_tokens): +@pytest.mark.parametrize('text,expected_tokens', + [('פייתון היא שפת תכנות דינמית', ['פייתון', 'היא', 'שפת', 'תכנות', 'דינמית'])]) +def test_tokenizer_handles_abbreviation(he_tokenizer, text, expected_tokens): tokens = he_tokenizer(text) token_list = [token.text for token in tokens if not token.is_space] - assert expected_tokens == token_list \ No newline at end of file + assert expected_tokens == token_list + + +@pytest.mark.parametrize('text,expected_tokens', [ + ('עקבת אחריו בכל רחבי המדינה.', ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה', '.']), + ('עקבת אחריו בכל רחבי המדינה?', ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה', '?']), + ('עקבת אחריו בכל רחבי המדינה!', ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה', '!']), + ('עקבת אחריו בכל רחבי המדינה..', ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה', '..']), + ('עקבת אחריו בכל רחבי המדינה...', ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה', '...'])]) +def test_tokenizer_handles_punct(he_tokenizer, text, expected_tokens): + tokens = he_tokenizer(text) + assert expected_tokens == [token.text for token in tokens] diff --git a/spacy/tests/nb/__init__.py b/spacy/tests/nb/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/spacy/tests/nb/test_tokenizer.py b/spacy/tests/nb/test_tokenizer.py new file mode 100644 index 000000000..b55901339 --- /dev/null +++ b/spacy/tests/nb/test_tokenizer.py @@ -0,0 +1,17 @@ +# coding: utf8 +from __future__ import unicode_literals + +import pytest + + +NB_TOKEN_EXCEPTION_TESTS = [ + ('Smørsausen brukes bl.a. til fisk', ['Smørsausen', 'brukes', 'bl.a.', 'til', 'fisk']), + ('Jeg kommer først kl. 13 pga. diverse forsinkelser', ['Jeg', 'kommer', 'først', 'kl.', '13', 'pga.', 'diverse', 'forsinkelser']) +] + + +@pytest.mark.parametrize('text,expected_tokens', NB_TOKEN_EXCEPTION_TESTS) +def test_tokenizer_handles_exception_cases(nb_tokenizer, text, expected_tokens): + tokens = nb_tokenizer(text) + token_list = [token.text for token in tokens if not token.is_space] + assert expected_tokens == token_list diff --git a/spacy/tests/regression/test_issue758.py b/spacy/tests/regression/test_issue758.py index c0bbb6945..a059f095f 100644 --- a/spacy/tests/regression/test_issue758.py +++ b/spacy/tests/regression/test_issue758.py @@ -1,3 +1,4 @@ +from __future__ import unicode_literals from ... import load as load_spacy from ...attrs import LEMMA from ...matcher import merge_phrase diff --git a/spacy/tests/regression/test_issue910.py b/spacy/tests/regression/test_issue910.py index 9b2b2287b..3790d429b 100644 --- a/spacy/tests/regression/test_issue910.py +++ b/spacy/tests/regression/test_issue910.py @@ -70,7 +70,6 @@ def temp_save_model(model): -@pytest.mark.xfail @pytest.mark.models def test_issue910(train_data, additional_entity_types): '''Test that adding entities and resuming training works passably OK. @@ -85,11 +84,10 @@ def test_issue910(train_data, additional_entity_types): ents_before_train = [(ent.label_, ent.text) for ent in doc.ents] # Fine tune the ner model for entity_type in additional_entity_types: - if entity_type not in nlp.entity.cfg['actions']['1']: - nlp.entity.add_label(entity_type) + nlp.entity.add_label(entity_type) - nlp.entity.learn_rate = 0.001 - for itn in range(4): + nlp.entity.model.learn_rate = 0.001 + for itn in range(10): random.shuffle(train_data) for raw_text, entity_offsets in train_data: doc = nlp.make_doc(raw_text) @@ -101,13 +99,12 @@ def test_issue910(train_data, additional_entity_types): # Load the fine tuned model loaded_ner = EntityRecognizer.load(model_dir, nlp.vocab) - for entity_type in additional_entity_types: - if entity_type not in loaded_ner.cfg['actions']['1']: - loaded_ner.add_label(entity_type) - - doc = nlp(u"I am looking for a restaurant in Berlin", entity=False) - nlp.tagger(doc) - loaded_ner(doc) - - ents_after_train = [(ent.label_, ent.text) for ent in doc.ents] - assert ents_before_train == ents_after_train + for raw_text, entity_offsets in train_data: + doc = nlp.make_doc(raw_text) + nlp.tagger(doc) + loaded_ner(doc) + ents = {(ent.start_char, ent.end_char): ent.label_ for ent in doc.ents} + for start, end, label in entity_offsets: + if (start, end) not in ents: + print(ents) + assert ents[(start, end)] == label diff --git a/spacy/tests/regression/test_issue995.py b/spacy/tests/regression/test_issue995.py new file mode 100644 index 000000000..633e96fb5 --- /dev/null +++ b/spacy/tests/regression/test_issue995.py @@ -0,0 +1,22 @@ +from __future__ import unicode_literals + +import pytest +from ... import load as load_spacy + +@pytest.fixture +def doc(): + nlp = load_spacy('en') + return nlp('Does flight number three fifty-four require a connecting flight' + ' to get to Boston?') + + +@pytest.mark.models +def test_issue955(doc): + '''Test that we don't have any nested noun chunks''' + seen_tokens = set() + for np in doc.noun_chunks: + print(np.text, np.root.text, np.root.dep_, np.root.tag_) + for word in np: + key = (word.i, word.text) + assert key not in seen_tokens + seen_tokens.add(key) diff --git a/spacy/tests/regression/test_issue999.py b/spacy/tests/regression/test_issue999.py new file mode 100644 index 000000000..d0c861cb1 --- /dev/null +++ b/spacy/tests/regression/test_issue999.py @@ -0,0 +1,78 @@ +from __future__ import unicode_literals +import os +import random +import contextlib +import shutil +import pytest +import tempfile +from pathlib import Path + + +import pathlib +from ...gold import GoldParse +from ...pipeline import EntityRecognizer +from ...language import Language + +try: + unicode +except NameError: + unicode = str + + +@pytest.fixture +def train_data(): + return [ + ["hey",[]], + ["howdy",[]], + ["hey there",[]], + ["hello",[]], + ["hi",[]], + ["i'm looking for a place to eat",[]], + ["i'm looking for a place in the north of town",[[31,36,"location"]]], + ["show me chinese restaurants",[[8,15,"cuisine"]]], + ["show me chines restaurants",[[8,14,"cuisine"]]], + ] + + +@contextlib.contextmanager +def temp_save_model(model): + model_dir = Path(tempfile.mkdtemp()) + model.save_to_directory(model_dir) + yield model_dir + shutil.rmtree(model_dir.as_posix()) + + +def test_issue999(train_data): + '''Test that adding entities and resuming training works passably OK. + There are two issues here: + + 1) We have to readd labels. This isn't very nice. + 2) There's no way to set the learning rate for the weight update, so we + end up out-of-scale, causing it to learn too fast. + ''' + nlp = Language(path=None, entity=False, tagger=False, parser=False) + nlp.entity = EntityRecognizer(nlp.vocab, features=Language.Defaults.entity_features) + for _, offsets in train_data: + for start, end, ent_type in offsets: + nlp.entity.add_label(ent_type) + nlp.entity.model.learn_rate = 0.001 + for itn in range(100): + random.shuffle(train_data) + for raw_text, entity_offsets in train_data: + doc = nlp.make_doc(raw_text) + gold = GoldParse(doc, entities=entity_offsets) + loss = nlp.entity.update(doc, gold) + + with temp_save_model(nlp) as model_dir: + nlp2 = Language(path=model_dir) + + for raw_text, entity_offsets in train_data: + doc = nlp2(raw_text) + ents = {(ent.start_char, ent.end_char): ent.label_ for ent in doc.ents} + for start, end, label in entity_offsets: + if (start, end) in ents: + assert ents[(start, end)] == label + break + else: + if entity_offsets: + raise Exception(ents) diff --git a/spacy/tests/spans/test_span.py b/spacy/tests/spans/test_span.py index 14c176edc..d22fa52ae 100644 --- a/spacy/tests/spans/test_span.py +++ b/spacy/tests/spans/test_span.py @@ -77,3 +77,15 @@ def test_spans_override_sentiment(en_tokenizer): assert doc[:2].sentiment == 10.0 assert doc[-2:].sentiment == 10.0 assert doc[:-1].sentiment == 10.0 + + +def test_spans_are_hashable(en_tokenizer): + """Test spans can be hashed.""" + text = "good stuff bad stuff" + tokens = en_tokenizer(text) + span1 = tokens[:2] + span2 = tokens[2:4] + assert hash(span1) != hash(span2) + span3 = tokens[0:2] + assert hash(span3) == hash(span1) + diff --git a/spacy/tests/test_download.py b/spacy/tests/test_cli.py similarity index 79% rename from spacy/tests/test_download.py rename to spacy/tests/test_cli.py index 4bbe190df..189a238fc 100644 --- a/spacy/tests/test_download.py +++ b/spacy/tests/test_cli.py @@ -2,17 +2,18 @@ from __future__ import unicode_literals from ..cli.download import download, get_compatibility, get_version, check_error_depr + import pytest @pytest.mark.parametrize('model', ['en_core_web_md']) -def test_download_get_matching_version_succeeds(model): +def test_cli_download_get_matching_version_succeeds(model): comp = { model: ['1.7.0', '0.100.0'] } assert get_version(model, comp) @pytest.mark.parametrize('model', ['en_core_web_md']) -def test_download_get_matching_version_fails(model): +def test_cli_download_get_matching_version_fails(model): diff_model = 'test_' + model comp = { diff_model: ['1.7.0', '0.100.0'] } with pytest.raises(SystemExit): @@ -20,6 +21,6 @@ def test_download_get_matching_version_fails(model): @pytest.mark.parametrize('model', [False, None, '', 'all']) -def test_download_no_model_depr_error(model): +def test_cli_download_no_model_depr_error(model): with pytest.raises(SystemExit): check_error_depr(model) diff --git a/spacy/tests/test_misc.py b/spacy/tests/test_misc.py new file mode 100644 index 000000000..41c4efb8a --- /dev/null +++ b/spacy/tests/test_misc.py @@ -0,0 +1,13 @@ +# coding: utf-8 +from __future__ import unicode_literals + +from ..util import ensure_path + +from pathlib import Path +import pytest + + +@pytest.mark.parametrize('text', ['hello/world', 'hello world']) +def test_util_ensure_path_succeeds(text): + path = ensure_path(text) + assert isinstance(path, Path) diff --git a/spacy/tokens/span.pyx b/spacy/tokens/span.pyx index 7e2f62171..fb1e5c732 100644 --- a/spacy/tokens/span.pyx +++ b/spacy/tokens/span.pyx @@ -66,6 +66,10 @@ cdef class Span: elif op == 5: return self.start_char >= other.start_char + def __hash__(self): + return hash((self.doc, self.label, self.start_char, self.end_char)) + + def __len__(self): self._recalculate_indices() if self.end < self.start: diff --git a/spacy/util.py b/spacy/util.py index 0ccdfbd72..0c7136522 100644 --- a/spacy/util.py +++ b/spacy/util.py @@ -2,7 +2,7 @@ from __future__ import unicode_literals, print_function import ujson -import re +import regex as re from pathlib import Path import sys import textwrap diff --git a/website/_harp.json b/website/_harp.json index 03fcbb956..672640405 100644 --- a/website/_harp.json +++ b/website/_harp.json @@ -55,7 +55,7 @@ } }, - "V_CSS": "1.4", + "V_CSS": "1.6", "V_JS": "1.2", "DEFAULT_SYNTAX": "python", "ANALYTICS": "UA-58931649-1", diff --git a/website/_includes/_mixins.jade b/website/_includes/_mixins.jade index ba5a9297e..2f89b0ec4 100644 --- a/website/_includes/_mixins.jade +++ b/website/_includes/_mixins.jade @@ -198,8 +198,8 @@ mixin table(head) //- Table row (only used within +table) -mixin row() - tr.c-table__row&attributes(attributes) +mixin row(...style) + tr.c-table__row(class=prefixArgs(style, "c-table__row"))&attributes(attributes) block @@ -283,3 +283,39 @@ mixin card-item(title, details) if details.author br span.u-text-small.u-color-subtle by #{details.author} + + +//- Table row for models table + +mixin model-row(name, lang, procon, size, license, default_model, divider) + - var licenses = { "CC BY-SA": "https://creativecommons.org/licenses/by-sa/3.0/", "CC BY-NC": "https://creativecommons.org/licenses/by-nc/3.0/" } + + +row(divider ? "divider": null) + +cell #[code=name] + if default_model + | #[span.u-color-theme(title="default model") #[+icon("star", 16)]] + +cell=lang + each icon in procon + +cell.u-text-center #[+procon(icon ? "pro" : "con")] + +cell.u-text-right=size + +cell + if license in licenses + +a(licenses[license])=license + + +//- Table rows for annotation specs + +mixin pos-row(tag, pos, morph, desc) + +row + +cell #[code=tag] + +cell #[code=pos] + +cell + each m in morph.split(" ") + if m + | #[code=m] + +cell.u-text-small=desc + +mixin dep-row(label, desc) + +row + +cell #[code=label] + +cell=desc diff --git a/website/assets/css/_components/_code.sass b/website/assets/css/_components/_code.sass index fd7c20e33..83462ef72 100644 --- a/website/assets/css/_components/_code.sass +++ b/website/assets/css/_components/_code.sass @@ -34,6 +34,7 @@ margin: 0 border-radius: 1px box-decoration-break: clone + white-space: nowrap .c-aside__content & background: lighten($color-front, 10) diff --git a/website/assets/css/_components/_tables.sass b/website/assets/css/_components/_tables.sass index f87169fb1..6ade4f6a1 100644 --- a/website/assets/css/_components/_tables.sass +++ b/website/assets/css/_components/_tables.sass @@ -20,6 +20,9 @@ @extend .u-text-label color: $color-theme + &.c-table__row--divider + border-top: 2px solid $color-theme + //- Table cell diff --git a/website/assets/css/_variables.sass b/website/assets/css/_variables.sass index 1c38d114a..5f9453ea6 100644 --- a/website/assets/css/_variables.sass +++ b/website/assets/css/_variables.sass @@ -34,7 +34,7 @@ $color-dark: lighten($color-front, 20) !default $color-theme: map-get($colors, $theme) $color-theme-dark: darken(map-get($colors, $theme), 5) -$color-theme-light: saturate(lighten(map-get($colors, $theme), 35), 15) +$color-theme-light: saturate(lighten(map-get($colors, $theme), 35), 5) $color-subtle: #ddd !default $color-subtle-light: #f6f6f6 !default diff --git a/website/assets/img/icons.svg b/website/assets/img/icons.svg index 224224084..f62901592 100644 --- a/website/assets/img/icons.svg +++ b/website/assets/img/icons.svg @@ -24,5 +24,8 @@ + + + diff --git a/website/docs/api/_annotation/_dep-labels.jade b/website/docs/api/_annotation/_dep-labels.jade new file mode 100644 index 000000000..9e1e89324 --- /dev/null +++ b/website/docs/api/_annotation/_dep-labels.jade @@ -0,0 +1,113 @@ +//- 💫 DOCS > API > ANNOTATION > DEPENDENCY LABELS + ++infobox("Tip") + | In spaCy v1.8.3+, you can also use #[code spacy.explain()] to get the + | description for the string representation of a label. For example, + | #[code spacy.explain("prt")] will return "particle". + ++h(3, "dependency-parsing-english") English dependency labels + +p + | The English dependency labels use the #[+a("http://www.clearnlp.com") ClearNLP] + | #[+a("http://www.mathcs.emory.edu/~choi/doc/clear-dependency-2012.pdf") CLEAR Style]. + ++table(["Label", "Description"]) + +dep-row("acomp", "adjectival complement") + +dep-row("advcl", "adverbial clause modifier") + +dep-row("advmod", "adverbial modifier") + +dep-row("agent", "agent") + +dep-row("amod", "adjectival modifier") + +dep-row("appos", "appositional modifier") + +dep-row("attr", "attribute") + +dep-row("aux", "auxiliary") + +dep-row("auxpass", "auxiliary (passive)") + +dep-row("cc", "coordinating conjunction") + +dep-row("ccomp", "clausal complement") + +dep-row("complm", "complementizer") + +dep-row("conj", "conjunct") + +dep-row("cop", "copula") + +dep-row("csubj", "clausal subject") + +dep-row("csubjpass", "clausal subject (passive)") + +dep-row("dep", "unclassified dependent") + +dep-row("det", "determiner") + +dep-row("dobj", "direct object") + +dep-row("expl", "expletive") + +dep-row("hmod", "modifier in hyphenation") + +dep-row("hyph", "hyphen") + +dep-row("infmod", "infinitival modifier") + +dep-row("intj", "interjection") + +dep-row("iobj", "indirect object") + +dep-row("mark", "marker") + +dep-row("meta", "meta modifier") + +dep-row("neg", "negation modifier") + +dep-row("nmod", "modifier of nominal") + +dep-row("nn", "noun compound modifier") + +dep-row("npadvmod", "noun phrase as adverbial modifier") + +dep-row("nsubj", "nominal subject") + +dep-row("nsubjpass", "nominal subject (passive)") + +dep-row("num", "number modifier") + +dep-row("number", "number compound modifier") + +dep-row("oprd", "object predicate") + +dep-row("obj", "object") + +dep-row("obl", "oblique nominal") + +dep-row("parataxis", "parataxis") + +dep-row("partmod", "participal modifier") + +dep-row("pcomp", "complement of preposition") + +dep-row("pobj", "object of preposition") + +dep-row("poss", "possession modifier") + +dep-row("possessive", "possessive modifier") + +dep-row("preconj", "pre-correlative conjunction") + +dep-row("prep", "prepositional modifier") + +dep-row("prt", "particle") + +dep-row("punct", "punctuation") + +dep-row("quantmod", "modifier of quantifier") + +dep-row("rcmod", "relative clause modifier") + +dep-row("root", "root") + +dep-row("xcomp", "open clausal complement") + ++h(3, "dependency-parsing-german") German dependency labels + +p + | The German dependency labels use the + | #[+a("http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/TIGERCorpus/annotation/index.html") TIGER Treebank] + | annotation scheme. + ++table(["Label", "Description"]) + +dep-row("ac", "adpositional case marker") + +dep-row("adc", "adjective component") + +dep-row("ag", "genitive attribute") + +dep-row("ams", "measure argument of adjective") + +dep-row("app", "apposition") + +dep-row("avc", "adverbial phrase component") + +dep-row("cc", "comparative complement") + +dep-row("cd", "coordinating conjunction") + +dep-row("cj", "conjunct") + +dep-row("cm", "comparative conjunction") + +dep-row("cp", "complementizer") + +dep-row("cvc", "collocational verb construction") + +dep-row("da", "dative") + +dep-row("dh", "discourse-level head") + +dep-row("dm", "discourse marker") + +dep-row("ep", "expletive es") + +dep-row("hd", "head") + +dep-row("ju", "junctor") + +dep-row("mnr", "postnominal modifier") + +dep-row("mo", "modifier") + +dep-row("ng", "negation") + +dep-row("nk", "noun kernel element") + +dep-row("nmc", "numerical component") + +dep-row("oa", "accusative object") + +dep-row("oa", "second accusative object") + +dep-row("oc", "clausal object") + +dep-row("og", "genitive object") + +dep-row("op", "prepositional object") + +dep-row("par", "parenthetical element") + +dep-row("pd", "predicate") + +dep-row("pg", "phrasal genitive") + +dep-row("ph", "placeholder") + +dep-row("pm", "morphological particle") + +dep-row("pnc", "proper noun component") + +dep-row("rc", "relative clause") + +dep-row("re", "repeated element") + +dep-row("rs", "reported speech") + +dep-row("sb", "subject") diff --git a/website/docs/api/_annotation/_named-entities.jade b/website/docs/api/_annotation/_named-entities.jade index 476659d4a..68b3bd17d 100644 --- a/website/docs/api/_annotation/_named-entities.jade +++ b/website/docs/api/_annotation/_named-entities.jade @@ -1,5 +1,10 @@ //- 💫 DOCS > API > ANNOTATION > NAMED ENTITIES ++infobox("Tip") + | In spaCy v1.8.3+, you can also use #[code spacy.explain()] to get the + | description for the string representation of an entity label. For example, + | #[code spacy.explain("LANGUAGE")] will return "any named language". + +table([ "Type", "Description" ]) +row +cell #[code PERSON] diff --git a/website/docs/api/_annotation/_pos-tags.jade b/website/docs/api/_annotation/_pos-tags.jade index d3f561c3f..d3ceef777 100644 --- a/website/docs/api/_annotation/_pos-tags.jade +++ b/website/docs/api/_annotation/_pos-tags.jade @@ -1,136 +1,139 @@ //- 💫 DOCS > API > ANNOTATION > POS TAGS -mixin pos-row(...row) - +row - each cell in row - +cell - each item in cell.split(" ") - if item - | #[code=item] ++infobox("Tip") + | In spaCy v1.8.3+, you can also use #[code spacy.explain()] to get the + | description for the string representation of a tag. For example, + | #[code spacy.explain("RB")] will return "adverb". + ++h(3, "pos-tagging-english") English part-of-speech tag scheme p - | The part-of-speech tagger uses the + | The English part-of-speech tagger uses the | #[+a("https://catalog.ldc.upenn.edu/LDC2013T19") OntoNotes 5] version of | the Penn Treebank tag set. We also map the tags to the simpler Google | Universal POS tag set. -+h(3, "pos-tagging-english") English part-of-speech tag scheme - -+table(["Tag", "POS", "Morphology"]) - +pos-row("-LRB-", "PUNCT", "PunctType=brck PunctSide=ini") - +pos-row("-PRB-", "PUNCT", "PunctType=brck PunctSide=fin") - +pos-row(",", "PUNCT", "PunctType=comm") - +pos-row(":", "PUNCT", "") - +pos-row(".", "PUNCT", "PunctType=peri") - +pos-row("''", "PUNCT", "PunctType=quot PunctSide=fin") - +pos-row("\"\"", "PUNCT", "PunctType=quot PunctSide=fin") - +pos-row("#", "SYM", "SymType=numbersign") - +pos-row("``", "PUNCT", "PunctType=quot PunctSide=ini") - +pos-row("$", "SYM", "SymType=currency") - +pos-row("ADD", "X", "") - +pos-row("AFX", "ADJ", "Hyph=yes") - +pos-row("BES", "VERB", "") - +pos-row("CC", "CONJ", "ConjType=coor") - +pos-row("CD", "NUM", "NumType=card") - +pos-row("DT", "DET", "") - +pos-row("EX", "ADV", "AdvType=ex") - +pos-row("FW", "X", "Foreign=yes") - +pos-row("GW", "X", "") - +pos-row("HVS", "VERB", "") - +pos-row("HYPH", "PUNCT", "PunctType=dash") - +pos-row("IN", "ADP", "") - +pos-row("JJ", "ADJ", "Degree=pos") - +pos-row("JJR", "ADJ", "Degree=comp") - +pos-row("JJS", "ADJ", "Degree=sup") - +pos-row("LS", "PUNCT", "NumType=ord") - +pos-row("MD", "VERB", "VerbType=mod") - +pos-row("NFP", "PUNCT", "") - +pos-row("NIL", "", "") - +pos-row("NN", "NOUN", "Number=sing") - +pos-row("NNP", "PROPN", "NounType=prop Number=sign") - +pos-row("NNPS", "PROPN", "NounType=prop Number=plur") - +pos-row("NNS", "NOUN", "Number=plur") - +pos-row("PDT", "ADJ", "AdjType=pdt PronType=prn") - +pos-row("POS", "PART", "Poss=yes") - +pos-row("PRP", "PRON", "PronType=prs") - +pos-row("PRP$", "ADJ", "PronType=prs Poss=yes") - +pos-row("RB", "ADV", "Degree=pos") - +pos-row("RBR", "ADV", "Degree=comp") - +pos-row("RBS", "ADV", "Degree=sup") - +pos-row("RP", "PART", "") - +pos-row("SP", "SPACE", "") - +pos-row("SYM", "SYM", "") - +pos-row("TO", "PART", "PartType=inf VerbForm=inf") - +pos-row("UH", "INTJ", "") - +pos-row("VB", "VERB", "VerbForm=inf") - +pos-row("VBD", "VERB", "VerbForm=fin Tense=past") - +pos-row("VBG", "VERB", "VerbForm=part Tense=pres Aspect=prog") - +pos-row("VBN", "VERB", "VerbForm=part Tense=past Aspect=perf") - +pos-row("VBP", "VERB", "VerbForm=fin Tense=pres") - +pos-row("VBZ", "VERB", "VerbForm=fin Tense=pres Number=sing Person=3") - +pos-row("WDT", "ADJ", "PronType=int|rel") - +pos-row("WP", "NOUN", "PronType=int|rel") - +pos-row("WP$", "ADJ", "Poss=yes PronType=int|rel") - +pos-row("WRB", "ADV", "PronType=int|rel") - +pos-row("XX", "X", "") ++table(["Tag", "POS", "Morphology", "Description"]) + +pos-row("-LRB-", "PUNCT", "PunctType=brck PunctSide=ini", "left round bracket") + +pos-row("-PRB-", "PUNCT", "PunctType=brck PunctSide=fin", "right round bracket") + +pos-row(",", "PUNCT", "PunctType=comm", "punctuation mark, comma") + +pos-row(":", "PUNCT", "", "punctuation mark, colon or ellipsis") + +pos-row(".", "PUNCT", "PunctType=peri", "punctuation mark, sentence closer") + +pos-row("''", "PUNCT", "PunctType=quot PunctSide=fin", "closing quotation mark") + +pos-row("\"\"", "PUNCT", "PunctType=quot PunctSide=fin", "closing quotation mark") + +pos-row("#", "SYM", "SymType=numbersign", "symbol, number sign") + +pos-row("``", "PUNCT", "PunctType=quot PunctSide=ini", "opening quotation mark") + +pos-row("$", "SYM", "SymType=currency", "symbol, currency") + +pos-row("ADD", "X", "", "email") + +pos-row("AFX", "ADJ", "Hyph=yes", "affix") + +pos-row("BES", "VERB", "", 'auxillary "be"') + +pos-row("CC", "CONJ", "ConjType=coor", "conjunction, coordinating") + +pos-row("CD", "NUM", "NumType=card", "cardinal number") + +pos-row("DT", "DET", "determiner") + +pos-row("EX", "ADV", "AdvType=ex", "existential there") + +pos-row("FW", "X", "Foreign=yes", "foreign word") + +pos-row("GW", "X", "", "additional word in multi-word expression") + +pos-row("HVS", "VERB", "", 'forms of "have"') + +pos-row("HYPH", "PUNCT", "PunctType=dash", "punctuation mark, hyphen") + +pos-row("IN", "ADP", "", "conjunction, subordinating or preposition") + +pos-row("JJ", "ADJ", "Degree=pos", "adjective") + +pos-row("JJR", "ADJ", "Degree=comp", "adjective, comparative") + +pos-row("JJS", "ADJ", "Degree=sup", "adjective, superlative") + +pos-row("LS", "PUNCT", "NumType=ord", "list item marker") + +pos-row("MD", "VERB", "VerbType=mod", "verb, modal auxillary") + +pos-row("NFP", "PUNCT", "", "superfluous punctuation") + +pos-row("NIL", "", "", "missing tag") + +pos-row("NN", "NOUN", "Number=sing", "noun, singular or mass") + +pos-row("NNP", "PROPN", "NounType=prop Number=sign", "noun, proper singular") + +pos-row("NNPS", "PROPN", "NounType=prop Number=plur", "noun, proper plural") + +pos-row("NNS", "NOUN", "Number=plur", "noun, plural") + +pos-row("PDT", "ADJ", "AdjType=pdt PronType=prn", "predeterminer") + +pos-row("POS", "PART", "Poss=yes", "possessive ending") + +pos-row("PRP", "PRON", "PronType=prs", "pronoun, personal") + +pos-row("PRP$", "ADJ", "PronType=prs Poss=yes", "pronoun, possessive") + +pos-row("RB", "ADV", "Degree=pos", "adverb") + +pos-row("RBR", "ADV", "Degree=comp", "adverb, comparative") + +pos-row("RBS", "ADV", "Degree=sup", "adverb, superlative") + +pos-row("RP", "PART", "", "adverb, particle") + +pos-row("SP", "SPACE", "", "space") + +pos-row("SYM", "SYM", "", "symbol") + +pos-row("TO", "PART", "PartType=inf VerbForm=inf", "infinitival to") + +pos-row("UH", "INTJ", "", "interjection") + +pos-row("VB", "VERB", "VerbForm=inf", "verb, base form") + +pos-row("VBD", "VERB", "VerbForm=fin Tense=past", "verb, past tense") + +pos-row("VBG", "VERB", "VerbForm=part Tense=pres Aspect=prog", "verb, gerund or present participle") + +pos-row("VBN", "VERB", "VerbForm=part Tense=past Aspect=perf", "verb, past participle") + +pos-row("VBP", "VERB", "VerbForm=fin Tense=pres", "verb, non-3rd person singular present") + +pos-row("VBZ", "VERB", "VerbForm=fin Tense=pres Number=sing Person=3", "verb, 3rd person singular present") + +pos-row("WDT", "ADJ", "PronType=int|rel", "wh-determiner") + +pos-row("WP", "NOUN", "PronType=int|rel", "wh-pronoun, personal") + +pos-row("WP$", "ADJ", "Poss=yes PronType=int|rel", "wh-pronoun, possessive") + +pos-row("WRB", "ADV", "PronType=int|rel", "wh-adverb") + +pos-row("XX", "X", "", "unknown") +h(3, "pos-tagging-german") German part-of-speech tag scheme -+table(["Tag", "POS", "Morphology"]) - +pos-row("$(", "PUNCT", "PunctType=brck") - +pos-row("$,", "PUNCT", "PunctType=comm") - +pos-row("$.", "PUNCT", "PunctType=peri") - +pos-row("ADJA", "ADJ", "") - +pos-row("ADJD", "ADJ", "Variant=short") - +pos-row("ADV", "ADV", "") - +pos-row("APPO", "ADP", "AdpType=post") - +pos-row("APPR", "ADP", "AdpType=prep") - +pos-row("APPRART", "ADP", "AdpType=prep PronType=art") - +pos-row("APZR", "ADP", "AdpType=circ") - +pos-row("ART", "DET", "PronType=art") - +pos-row("CARD", "NUM", "NumType=card") - +pos-row("FM", "X", "Foreign=yes") - +pos-row("ITJ", "INTJ", "") - +pos-row("KOKOM", "CONJ", "ConjType=comp") - +pos-row("KON", "CONJ", "") - +pos-row("KOUI", "SCONJ", "") - +pos-row("KOUS", "SCONJ", "") - +pos-row("NE", "PROPN", "") - +pos-row("NNE", "PROPN", "") - +pos-row("NN", "NOUN", "") - +pos-row("PAV", "ADV", "PronType=dem") - +pos-row("PROAV", "ADV", "PronType=dem") - +pos-row("PDAT", "DET", "PronType=dem") - +pos-row("PDS", "PRON", "PronType=dem") - +pos-row("PIAT", "DET", "PronType=ind|neg|tot") - +pos-row("PIDAT", "DET", "AdjType=pdt PronType=ind|neg|tot") - +pos-row("PIS", "PRON", "PronType=ind|neg|tot") - +pos-row("PPER", "PRON", "PronType=prs") - +pos-row("PPOSAT", "DET", "Poss=yes PronType=prs") - +pos-row("PPOSS", "PRON", "PronType=rel") - +pos-row("PRELAT", "DET", "PronType=rel") - +pos-row("PRELS", "PRON", "PronType=rel") - +pos-row("PRF", "PRON", "PronType=prs Reflex=yes") - +pos-row("PTKA", "PART", "") - +pos-row("PTKANT", "PART", "PartType=res") - +pos-row("PTKNEG", "PART", "Negative=yes") - +pos-row("PTKVZ", "PART", "PartType=vbp") - +pos-row("PTKZU", "PART", "PartType=inf") - +pos-row("PWAT", "DET", "PronType=int") - +pos-row("PWAV", "ADV", "PronType=int") - +pos-row("PWS", "PRON", "PronType=int") - +pos-row("TRUNC", "X", "Hyph=yes") - +pos-row("VAFIN", "AUX", "Mood=ind VerbForm=fin") - +pos-row("VAIMP", "AUX", "Mood=imp VerbForm=fin") - +pos-row("VAINF", "AUX", "VerbForm=inf") - +pos-row("VAPP", "AUX", "Aspect=perf VerbForm=fin") - +pos-row("VMFIN", "VERB", "Mood=ind VerbForm=fin VerbType=mod") - +pos-row("VMINF", "VERB", "VerbForm=fin VerbType=mod") - +pos-row("VMPP", "VERB", "Aspect=perf VerbForm=part VerbType=mod") - +pos-row("VVFIN", "VERB", "Mood=ind VerbForm=fin") - +pos-row("VVIMP", "VERB", "Mood=imp VerbForm=fin") - +pos-row("VVINF", "VERB", "VerbForm=inf") - +pos-row("VVIZU", "VERB", "VerbForm=inf") - +pos-row("VVPP", "VERB", "Aspect=perf VerbForm=part") - +pos-row("XY", "X", "") - +pos-row("SP", "SPACE", "") +p + | The German part-of-speech tagger uses the + | #[+a("http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/TIGERCorpus/annotation/index.html") TIGER Treebank] + | annotation scheme. We also map the tags to the simpler Google + | Universal POS tag set. + ++table(["Tag", "POS", "Morphology", "Description"]) + +pos-row("$(", "PUNCT", "PunctType=brck", "other sentence-internal punctuation mark") + +pos-row("$,", "PUNCT", "PunctType=comm", "comma") + +pos-row("$.", "PUNCT", "PunctType=peri", "sentence-final punctuation mark") + +pos-row("ADJA", "ADJ", "", "adjective, attributive") + +pos-row("ADJD", "ADJ", "Variant=short", "adjective, adverbial or predicative") + +pos-row("ADV", "ADV", "", "adverb") + +pos-row("APPO", "ADP", "AdpType=post", "postposition") + +pos-row("APPR", "ADP", "AdpType=prep", "preposition; circumposition left") + +pos-row("APPRART", "ADP", "AdpType=prep PronType=art", "preposition with article") + +pos-row("APZR", "ADP", "AdpType=circ", "circumposition right") + +pos-row("ART", "DET", "PronType=art", "definite or indefinite article") + +pos-row("CARD", "NUM", "NumType=card", "cardinal number") + +pos-row("FM", "X", "Foreign=yes", "foreign language material") + +pos-row("ITJ", "INTJ", "", "interjection") + +pos-row("KOKOM", "CONJ", "ConjType=comp", "comparative conjunction") + +pos-row("KON", "CONJ", "", "coordinate conjunction") + +pos-row("KOUI", "SCONJ", "", 'subordinate conjunction with "zu" and infinitive') + +pos-row("KOUS", "SCONJ", "", "subordinate conjunction with sentence") + +pos-row("NE", "PROPN", "", "proper noun") + +pos-row("NNE", "PROPN", "", "proper noun") + +pos-row("NN", "NOUN", "", "noun, singular or mass") + +pos-row("PAV", "ADV", "PronType=dem", "pronominal adverb") + +pos-row("PROAV", "ADV", "PronType=dem", "pronominal adverb") + +pos-row("PDAT", "DET", "PronType=dem", "attributive demonstrative pronoun") + +pos-row("PDS", "PRON", "PronType=dem", "substituting demonstrative pronoun") + +pos-row("PIAT", "DET", "PronType=ind|neg|tot", "attributive indefinite pronoun without determiner") + +pos-row("PIDAT", "DET", "AdjType=pdt PronType=ind|neg|tot", "attributive indefinite pronoun with determiner") + +pos-row("PIS", "PRON", "PronType=ind|neg|tot", "substituting indefinite pronoun") + +pos-row("PPER", "PRON", "PronType=prs", "non-reflexive personal pronoun") + +pos-row("PPOSAT", "DET", "Poss=yes PronType=prs", "attributive possessive pronoun") + +pos-row("PPOSS", "PRON", "PronType=rel", "substituting possessive pronoun") + +pos-row("PRELAT", "DET", "PronType=rel", "attributive relative pronoun") + +pos-row("PRELS", "PRON", "PronType=rel", "substituting relative pronoun") + +pos-row("PRF", "PRON", "PronType=prs Reflex=yes", "reflexive personal pronoun") + +pos-row("PTKA", "PART", "", "particle with adjective or adverb") + +pos-row("PTKANT", "PART", "PartType=res", "answer particle") + +pos-row("PTKNEG", "PART", "Negative=yes", "negative particle") + +pos-row("PTKVZ", "PART", "PartType=vbp", "separable verbal particle") + +pos-row("PTKZU", "PART", "PartType=inf", '"zu" before infinitive') + +pos-row("PWAT", "DET", "PronType=int", "attributive interrogative pronoun") + +pos-row("PWAV", "ADV", "PronType=int", "adverbial interrogative or relative pronoun") + +pos-row("PWS", "PRON", "PronType=int", "substituting interrogative pronoun") + +pos-row("TRUNC", "X", "Hyph=yes", "word remnant") + +pos-row("VAFIN", "AUX", "Mood=ind VerbForm=fin", "finite verb, auxiliary") + +pos-row("VAIMP", "AUX", "Mood=imp VerbForm=fin", "imperative, auxiliary") + +pos-row("VAINF", "AUX", "VerbForm=inf", "infinitive, auxiliary") + +pos-row("VAPP", "AUX", "Aspect=perf VerbForm=fin", "perfect participle, auxiliary") + +pos-row("VMFIN", "VERB", "Mood=ind VerbForm=fin VerbType=mod", "finite verb, modal") + +pos-row("VMINF", "VERB", "VerbForm=fin VerbType=mod", "infinitive, modal") + +pos-row("VMPP", "VERB", "Aspect=perf VerbForm=part VerbType=mod", "perfect participle, modal") + +pos-row("VVFIN", "VERB", "Mood=ind VerbForm=fin", "finite verb, full") + +pos-row("VVIMP", "VERB", "Mood=imp VerbForm=fin", "imperative, full") + +pos-row("VVINF", "VERB", "VerbForm=inf", "infinitive, full") + +pos-row("VVIZU", "VERB", "VerbForm=inf", 'infinitive with "zu", full') + +pos-row("VVPP", "VERB", "Aspect=perf VerbForm=part", "perfect participle, full") + +pos-row("XY", "X", "", "non-word containing non-letter") + +pos-row("SP", "SPACE", "", "space") diff --git a/website/docs/api/annotation.jade b/website/docs/api/annotation.jade index adc6b28f7..8c6b8fb10 100644 --- a/website/docs/api/annotation.jade +++ b/website/docs/api/annotation.jade @@ -65,16 +65,7 @@ p +h(2, "dependency-parsing") Syntactic Dependency Parsing -+table(["Language", "Converter", "Scheme"]) - +row - +cell English - +cell #[+a("http://www.clearnlp.com") ClearNLP] - +cell #[+a("http://www.mathcs.emory.edu/~choi/doc/clear-dependency-2012.pdf") CLEAR Style] - - +row - +cell German - +cell #[+a("https://github.com/wbwseeker/tiger2dep") TIGER] - +cell #[+a("http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/TIGERCorpus/annotation/index.html") TIGER] +include _annotation/_dep-labels +h(2, "named-entities") Named Entity Recognition diff --git a/website/docs/api/language-models.jade b/website/docs/api/language-models.jade index 083a501be..3bce7272f 100644 --- a/website/docs/api/language-models.jade +++ b/website/docs/api/language-models.jade @@ -7,6 +7,7 @@ p spaCy currently supports the following languages and capabilities: +aside-code("Download language models", "bash"). python -m spacy download en python -m spacy download de + python -m spacy download fr +table([ "Language", "Token", "SBD", "Lemma", "POS", "NER", "Dep", "Vector", "Sentiment"]) +row @@ -19,6 +20,14 @@ p spaCy currently supports the following languages and capabilities: each icon in [ "pro", "pro", "con", "pro", "pro", "pro", "pro", "con" ] +cell.u-text-center #[+procon(icon)] + +row + +cell French #[code fr] + each icon in [ "pro", "pro", "con", "pro", "con", "pro", "pro", "con" ] + +cell.u-text-center #[+procon(icon)] + ++h(2, "available") Available models + +include ../usage/_models-list +h(2, "alpha-support") Alpha support @@ -26,14 +35,15 @@ p | Work has started on the following languages. You can help by improving | the existing language data and extending the tokenization patterns. ++aside("Dependencies") + | Some language tokenizers require external dependencies. To use #[strong Chinese], + | you need to have #[+a("https://github.com/fxsjy/jieba") Jieba] installed. + | The #[strong Japanese] tokenizer requires + | #[+a("https://github.com/mocobeta/janome") Janome]. + +table([ "Language", "Source" ]) - each language, code in { zh: "Chinese", es: "Spanish", it: "Italian", fr: "French", pt: "Portuguese", nl: "Dutch", sv: "Swedish", fi: "Finnish", hu: "Hungarian", bn: "Bengali", he: "Hebrew" } + each language, code in { es: "Spanish", it: "Italian", pt: "Portuguese", nl: "Dutch", sv: "Swedish", fi: "Finnish", nb: "Norwegian Bokmål", hu: "Hungarian", bn: "Bengali", he: "Hebrew", zh: "Chinese", ja: "Japanese" } +row +cell #{language} #[code=code] +cell +src(gh("spaCy", "spacy/" + code)) spacy/#{code} - -p - | Chinese tokenization requires the - | #[+a("https://github.com/fxsjy/jieba") Jieba] library. Statistical - | models are coming soon. diff --git a/website/docs/api/language.jade b/website/docs/api/language.jade index 93e7ff213..d7090c870 100644 --- a/website/docs/api/language.jade +++ b/website/docs/api/language.jade @@ -55,14 +55,14 @@ p Create or load the pipeline. +table(["Name", "Type", "Description"]) +row - +cell #[code **kwrags] + +cell #[code **overrides] +cell - +cell Keyword arguments indicating which defaults to override. +footrow +cell return +cell #[code Language] - +cell #[code self] + +cell The newly constructed object. +h(2, "call") Language.__call__ +tag method @@ -136,3 +136,19 @@ p +cell yield +cell #[code Doc] +cell Containers for accessing the linguistic annotations. + ++h(2, "save_to_directory") Language.save_to_directory + +tag method + +p Save the #[code Vocab], #[code StringStore] and pipeline to a directory. + ++table(["Name", "Type", "Description"]) + +row + +cell #[code path] + +cell string or pathlib path + +cell Path to save the model. + + +footrow + +cell return + +cell #[code None] + +cell - diff --git a/website/docs/usage/_models-list.jade b/website/docs/usage/_models-list.jade new file mode 100644 index 000000000..942de28c4 --- /dev/null +++ b/website/docs/usage/_models-list.jade @@ -0,0 +1,27 @@ +//- 💫 DOCS > USAGE > MODELS LIST + +include ../../_includes/_mixins + +p + | Model differences are mostly statistical. In general, we do expect larger + | models to be "better" and more accurate overall. Ultimately, it depends on + | your use case and requirements, and we recommend starting with the default + | models (marked with a star below). + ++aside + | Models are now available as #[code .tar.gz] archives #[+a(gh("spacy-models")) from GitHub], + | attached to individual releases. They can be downloaded and loaded manually, + | or using spaCy's #[code download] and #[code link] commands. All models + | follow the naming convention of #[code [language]_[type]_[genre]_[size]]. + | #[br]#[br] + + +button(gh("spacy-models"), true, "primary").u-text-tag + | View model releases + ++table(["Name", "Language", "Voc", "Dep", "Ent", "Vec", "Size", "License"]) + +model-row("en_core_web_sm", "English", [1, 1, 1, 1], "50 MB", "CC BY-SA", true) + +model-row("en_core_web_md", "English", [1, 1, 1, 1], "1 GB", "CC BY-SA") + +model-row("en_depent_web_md", "English", [1, 1, 1, 0], "328 MB", "CC BY-SA") + +model-row("en_vectors_glove_md", "English", [1, 0, 0, 1], "727 MB", "CC BY-SA") + +model-row("de_core_news_md", "German", [1, 1, 1, 1], "645 MB", "CC BY-SA", true, true) + +model-row("fr_depvec_web_lg", "French", [1, 1, 0, 1], "1.33 GB", "CC BY-NC", true, true) diff --git a/website/docs/usage/adding-languages.jade b/website/docs/usage/adding-languages.jade index 0c98cc5ca..30c4486b0 100644 --- a/website/docs/usage/adding-languages.jade +++ b/website/docs/usage/adding-languages.jade @@ -27,9 +27,10 @@ p | #[a(href="#brown-clusters") Brown clusters] and | #[a(href="#word-vectors") word vectors]. + +item + | #[strong Set up] a #[a(href="#model-directory") model direcory] and #[strong train] the #[a(href="#train-tagger-parser") tagger and parser]. + p - | Once you have the tokenizer and vocabulary, you can - | #[+a("/docs/usage/training") train the tagger, parser and entity recognizer]. | For some languages, you may also want to develop a solution for | lemmatization and morphological analysis. @@ -98,6 +99,17 @@ p | so that Python functions can be used to help you generalise and combine | the data as you require. ++infobox("For languages with non-latin characters") + | In order for the tokenizer to split suffixes, prefixes and infixes, spaCy + | needs to know the language's character set. If the language you're adding + | uses non-latin characters, you might need to add the required character + | classes to the global + | #[+src(gh("spacy", "spacy/language_data/punctuation.py")) punctuation.py]. + | spaCy uses the #[+a("https://pypi.python.org/pypi/regex/") #[code regex] library] + | to keep this simple and readable. If the language requires very specific + | punctuation rules, you should consider overwriting the default regular + | expressions with your own in the language's #[code Defaults]. + +h(3, "stop-words") Stop words p @@ -395,12 +407,111 @@ p | by linear models, while the word vectors are useful for lexical | similarity models and deep learning. ++h(3, "word-frequencies") Word frequencies + +p + | To generate the word frequencies from a large, raw corpus, you can use the + | #[+src(gh("spacy-dev-resources", "training/word_freqs.py")) word_freqs.py] + | script from the spaCy developer resources. Note that your corpus should + | not be preprocessed (i.e. you need punctuation for example). The + | #[+a("/docs/usage/cli#model") #[code model] command] expects a + | tab-separated word frequencies file with three columns: + ++list("numbers") + +item The number of times the word occurred in your language sample. + +item The number of distinct documents the word occurred in. + +item The word itself. + +p + | An example word frequencies file could look like this: + ++code("es_word_freqs.txt", "text"). + 6361109 111 Aunque + 23598543 111 aunque + 10097056 111 claro + 193454 111 aro + 7711123 111 viene + 12812323 111 mal + 23414636 111 momento + 2014580 111 felicidad + 233865 111 repleto + 15527 111 eto + 235565 111 deliciosos + 17259079 111 buena + 71155 111 Anímate + 37705 111 anímate + 33155 111 cuéntanos + 2389171 111 cuál + 961576 111 típico + +p + | You should make sure you use the spaCy tokenizer for your + | language to segment the text for your word frequencies. This will ensure + | that the frequencies refer to the same segmentation standards you'll be + | using at run-time. For instance, spaCy's English tokenizer segments + | "can't" into two tokens. If we segmented the text by whitespace to + | produce the frequency counts, we'll have incorrect frequency counts for + | the tokens "ca" and "n't". + ++h(3, "brown-clusters") Training the Brown clusters + +p + | spaCy's tagger, parser and entity recognizer are designed to use + | distributional similarity features provided by the + | #[+a("https://github.com/percyliang/brown-cluster") Brown clustering algorithm]. + | You should train a model with between 500 and 1000 clusters. A minimum + | frequency threshold of 10 usually works well. + +p + | An example clusters file could look like this: + ++code("es_clusters.data", "text"). + 0000 Vestigial 1 + 0000 Vesturland 1 + 0000 Veyreau 1 + 0000 Veynes 1 + 0000 Vexilografía 1 + 0000 Vetrigne 1 + 0000 Vetónica 1 + 0000 Asunden 1 + 0000 Villalambrús 1 + 0000 Vichuquén 1 + 0000 Vichtis 1 + 0000 Vichigasta 1 + 0000 VAAH 1 + 0000 Viciebsk 1 + 0000 Vicovaro 1 + 0000 Villardeveyo 1 + 0000 Vidala 1 + 0000 Videoguard 1 + 0000 Vedás 1 + 0000 Videocomunicado 1 + 0000 VideoCrypt 1 + ++h(3, "word-vectors") Training the word vectors + +p + | #[+a("https://en.wikipedia.org/wiki/Word2vec") Word2vec] and related + | algorithms let you train useful word similarity models from unlabelled + | text. This is a key part of using + | #[+a("/docs/usage/deep-learning") deep learning] for NLP with limited + | labelled data. The vectors are also useful by themselves – they power + | the #[code .similarity()] methods in spaCy. For best results, you should + | pre-process the text with spaCy before training the Word2vec model. This + | ensures your tokenization will match. + +p + | You can use our + | #[+src(gh("spacy-dev-resources", "training/word_vectors.py")) word vectors training script], + | which pre-processes the text with your language-specific tokenizer and + | trains the model using #[+a("https://radimrehurek.com/gensim/") Gensim]. + | The #[code vectors.bin] file should consist of one word and vector per line. + ++h(2, "model-directory") Setting up a model directory + p | Once you've collected the word frequencies, Brown clusters and word | vectors files, you can use the - | #[+src(gh("spacy-dev-resources", "training/init.py")) init.py] - | script from our - | #[+a(gh("spacy-dev-resources")) developer resources], or use the new | #[+a("/docs/usage/cli#model") #[code model] command] to create a data | directory: @@ -427,49 +538,22 @@ p | loaded. By default, the command expects to be able to find your language | class using #[code spacy.util.get_lang_class(lang_id)]. -+h(3, "word-frequencies") Word frequencies + ++h(2, "train-tagger-parser") Training the tagger and parser p - | The #[+a("/docs/usage/cli#model") #[code model] command] expects a - | tab-separated word frequencies file with three columns: - -+list("numbers") - +item The number of times the word occurred in your language sample. - +item The number of distinct documents the word occurred in. - +item The word itself. + | You can now train the model using a corpus for your language annotated + | with #[+a("http://universaldependencies.org/") Universal Dependencies]. + | If your corpus uses the + | #[+a("http://universaldependencies.org/docs/format.html") CoNLL-U] format, + | i.e. files with the extension #[code .conllu], you can use the + | #[+a("/docs/usage/cli#convert") #[code convert] command] to convert it to + | spaCy's #[+a("/docs/api/annotation#json-input") JSON format] for training. p - | You should make sure you use the spaCy tokenizer for your - | language to segment the text for your word frequencies. This will ensure - | that the frequencies refer to the same segmentation standards you'll be - | using at run-time. For instance, spaCy's English tokenizer segments - | "can't" into two tokens. If we segmented the text by whitespace to - | produce the frequency counts, we'll have incorrect frequency counts for - | the tokens "ca" and "n't". + | Once you have your UD corpus transformed into JSON, you can train your + | model use the using spaCy's + | #[+a("/docs/usage/cli#train") #[code train] command]: -+h(3, "brown-clusters") Training the Brown clusters - -p - | spaCy's tagger, parser and entity recognizer are designed to use - | distributional similarity features provided by the - | #[+a("https://github.com/percyliang/brown-cluster") Brown clustering algorithm]. - | You should train a model with between 500 and 1000 clusters. A minimum - | frequency threshold of 10 usually works well. - -+h(3, "word-vectors") Training the word vectors - -p - | #[+a("https://en.wikipedia.org/wiki/Word2vec") Word2vec] and related - | algorithms let you train useful word similarity models from unlabelled - | text. This is a key part of using - | #[+a("/docs/usage/deep-learning") deep learning] for NLP with limited - | labelled data. The vectors are also useful by themselves – they power - | the #[code .similarity()] methods in spaCy. For best results, you should - | pre-process the text with spaCy before training the Word2vec model. This - | ensures your tokenization will match. - -p - | You can use our - | #[+src(gh("spacy-dev-resources", "training/word_vectors.py")) word vectors training script], - | which pre-processes the text with your language-specific tokenizer and - | trains the model using #[+a("https://radimrehurek.com/gensim/") Gensim]. ++code(false, "bash"). + python -m spacy train [lang] [output_dir] [train_data] [dev_data] [--n_iter] [--parser_L1] [--no_tagger] [--no_parser] [--no_ner] diff --git a/website/docs/usage/deep-learning.jade b/website/docs/usage/deep-learning.jade index a1252a4fe..fec01b4ba 100644 --- a/website/docs/usage/deep-learning.jade +++ b/website/docs/usage/deep-learning.jade @@ -4,9 +4,9 @@ include ../../_includes/_mixins p | In this example, we'll be using #[+a("https://keras.io/") Keras], as - | it's the most popular deep learning library for Python. Let's assume - | you've written a custom sentiment analysis model that predicts whether a - | document is positive or negative. Now you want to find which entities + | it's the most popular deep learning library for Python. Using Keras, + | we will write a custom sentiment analysis model that predicts whether a + | document is positive or negative. Then, we will use it to find which entities | are commonly associated with positive or negative documents. Here's a | quick example of how that can look at runtime. diff --git a/website/docs/usage/models.jade b/website/docs/usage/models.jade index 9d50dcbc0..9bb75ba9a 100644 --- a/website/docs/usage/models.jade +++ b/website/docs/usage/models.jade @@ -13,14 +13,6 @@ p | internal alias that tells spaCy where to find the data files for a specific | model name. -+infobox("Important note") - | Due to improvements in the English lemmatizer in v1.7.0, you need to - | #[strong download the new English models]. The German model is still - | compatible. If you've trained statistical models that use spaCy's - | annotations, you should #[strong retrain your models after updating spaCy]. - | If you don't retrain your models, you may suffer train/test skew, which - | might decrease your accuracy. - +aside-code("Quickstart"). # Install spaCy and download English model pip install spacy @@ -31,43 +23,17 @@ p nlp = spacy.load('en') doc = nlp(u'This is a sentence.') ++infobox("Important note") + | Due to improvements in the English lemmatizer in v1.7.0, you need to + | #[strong download the new English models]. The German model is still + | compatible. If you've trained statistical models that use spaCy's + | annotations, you should #[strong retrain your models after updating spaCy]. + | If you don't retrain your models, you may suffer train/test skew, which + | might decrease your accuracy. + +h(2, "available") Available models -+table(["Name", "Size", "Description"]) - +row - +cell #[code en_core_web_sm] - +cell 50 MB - +cell Vocab, syntax, entities, word vectors #[+tag default] - - +row - +cell #[code en_core_web_md] - +cell 1 GB - +cell Vocab, syntax, entities, word vectors - - +row - +cell #[code en_depent_web_md] - +cell 328 MB - +cell Vocab, syntax, entities - - +row - +cell #[code en_vectors_glove_md] - +cell 727 MB - +cell - | #[+a("http://nlp.stanford.edu/projects/glove/") GloVe] Common - | Crawl vectors - - +row - +cell #[code de_core_news_md] - +cell 645 MB - +cell Vocab, syntax, entities, word vectors #[+tag default] - -p - | Models are now available as #[code .tar.gz] archives #[+a(gh("spacy-models")) from GitHub], - | attached to individual releases. They can be downloaded and loaded manually, - | or using spaCy's #[code download] and #[code link] commands. All models - | follow the naming convention of #[code [language]_[type]_[genre]_[size]]. - -+button(gh("spacy-models") + "/releases", true, "primary") View models +include _models-list +h(2, "download") Downloading models @@ -95,6 +61,7 @@ p # out-of-the-box: download best-matching default model python -m spacy download en python -m spacy download de + python -m spacy download fr # download best-matching version of specific model for your spaCy installation python -m spacy download en_core_web_md diff --git a/website/docs/usage/saving-loading.jade b/website/docs/usage/saving-loading.jade index 063c5dc50..c4eb08f04 100644 --- a/website/docs/usage/saving-loading.jade +++ b/website/docs/usage/saving-loading.jade @@ -2,7 +2,8 @@ include ../../_includes/_mixins p | After training your model, you'll usually want to save its state, and load - | it back later. You can do this with the #[code Language.save_to_directory()] + | it back later. You can do this with the + | #[+api("language#save_to_directory") #[code Language.save_to_directory()]] | method: +code. @@ -23,7 +24,7 @@ p | there can be a convenient solution to share them with your team. p - | spaCy comes with a handy CLI command that will create all required files, + | spaCy comes with a handy CLI command that will create all required files, | and walk you through generating the meta data. You can also create the | meta.json manually and place it in the model data directory, or supply a | path to it using the #[code --meta] flag. For more info on this, see the @@ -60,7 +61,7 @@ p | #[+a(gh("spacy-dev-resouces", "templates/model")) spaCy dev resources]. | If you're creating the package manually, keep in mind that the directories | need to be named according to the naming conventions of - | #[code [language]_[type]] and #[code [language]_[type]-[version]]. The + | #[code [language]_[name]] and #[code [language]_[name]-[version]]. The | #[code lang] setting in the meta.json is also used to create the | respective #[code Language] class in spaCy, which will later be returned | by the model's #[code load()] method.