Merge branch 'develop' of https://github.com/explosion/spaCy into develop

spacy/tests/README.md

@@ -13,21 +13,32 @@ Tests for spaCy modules and classes live in their own directories of the same na
 2. [Dos and don'ts](#dos-and-donts)
 3. [Parameters](#parameters)
 4. [Fixtures](#fixtures)
-5. [Helpers and utilities](#helpers-and-utilities)
+5. [Testing models](#testing-models)
-6. [Contributing to the tests](#contributing-to-the-tests)
+6. [Helpers and utilities](#helpers-and-utilities)
+7. [Contributing to the tests](#contributing-to-the-tests)
 
 
 ## Running the tests
 
+To show print statements, run the tests with `py.test -s`. To abort after the
+first failure, run them with `py.test -x`.
+
 ```bash
-py.test spacy                    # run basic tests
+py.test spacy                        # run basic tests
-py.test spacy --models           # run basic and model tests
+py.test spacy --models --en          # run basic and English model tests
-py.test spacy --slow             # run basic and slow tests
+py.test spacy --models --all         # run basic and all model tests
-py.test spacy --models --slow    # run all tests
+py.test spacy --slow                 # run basic and slow tests
+py.test spacy --models --all --slow  # run all tests
 ```
 
-To show print statements, run the tests with `py.test -s`. To abort after the first failure, run them with `py.test -x`.
+You can also run tests in a specific file or directory, or even only one
+specific test:
 
+```bash
+py.test spacy/tests/tokenizer  # run all tests in directory
+py.test spacy/tests/tokenizer/test_exceptions.py # run all tests in file
+py.test spacy/tests/tokenizer/test_exceptions.py::test_tokenizer_handles_emoji # run specific test
+```
 
 ## Dos and don'ts
 
@@ -83,14 +94,9 @@ These are the main fixtures that are currently available:
 | Fixture | Description |
 | --- | --- |
 | `tokenizer` | Creates **all available** language tokenizers and runs the test for **each of them**. |
-| `en_tokenizer` | Creates an English `Tokenizer` object. |
+| `en_tokenizer`, `de_tokenizer`, ... | Creates an English, German etc. tokenizer. |
-| `de_tokenizer` | Creates a German `Tokenizer` object. |
+| `en_vocab`, `en_entityrecognizer`, ... | Creates an instance of the English `Vocab`, `EntityRecognizer` object etc. |
-| `hu_tokenizer` | Creates a Hungarian `Tokenizer` object. |
+|  `EN`, `DE`, ... |  Creates a language class with a loaded model. For more info, see [Testing models](#testing-models). |
-| `en_vocab` | Creates an English `Vocab` object. |
-| `en_entityrecognizer` | Creates an English `EntityRecognizer` object. |
-| `lemmatizer` | Creates a `Lemmatizer` object from the installed language data (`None` if no data is found).
-| `EN` | Creates an instance of `English`. Only use for tests that require the models. |
-| `DE` | Creates an instance of `German`. Only use for tests that require the models. |
 | `text_file` | Creates an instance of `StringIO` to simulate reading from and writing to files. |
 | `text_file_b` | Creates an instance of `ByteIO` to simulate reading from and writing to files. |
 
@@ -103,6 +109,48 @@ def test_module_do_something(en_tokenizer):
 
 If all tests in a file require a specific configuration, or use the same complex example, it can be helpful to create a separate fixture. This fixture should be added at the top of each file. Make sure to use descriptive names for these fixtures and don't override any of the global fixtures listed above. **From looking at a test, it should immediately be clear which fixtures are used, and where they are coming from.**
 
+## Testing models
+
+Models should only be loaded and tested **if absolutely necessary** – for example, if you're specifically testing a model's performance, or if your test is related to model loading. If you only need an annotated `Doc`, you should use the `get_doc()` helper function to create it manually instead.
+
+To specify which language models a test is related to, set the language ID as an argument of `@pytest.mark.models`. This allows you to later run the tests with `--models --en`. You can then use the `EN` [fixture](#fixtures) to get a language
+class with a loaded model.
+
+```python
+@pytest.mark.models('en')
+def test_english_model(EN):
+    doc = EN(u'This is a test')
+```
+
+> ⚠️ **Important note:** In order to test models, they need to be installed as a packge. The [conftest.py](conftest.py) includes a list of all available models, mapped to their IDs, e.g. `en`. Unless otherwise specified, each model that's installed in your environment will be imported and tested. If you don't have a model installed, **the test will be skipped**.
+
+Under the hood, `pytest.importorskip` is used to import a model package and skip the test if the package is not installed. The `EN` fixture for example gets all
+available models for `en`, [parametrizes](#parameters) them to run the test for *each of them*, and uses `load_test_model()` to import the model and run the test, or skip it if the model is not installed.
+
+### Testing specific models
+
+Using the `load_test_model()` helper function, you can also write tests for specific models, or combinations of them:
+
+```python
+from .util import load_test_model
+
+@pytest.mark.models('en')
+def test_en_md_only():
+    nlp = load_test_model('en_core_web_md')
+    # test something specific to en_core_web_md
+
+@pytest.mark.models('en', 'fr')
+@pytest.mark.parametrize('model', ['en_core_web_md', 'fr_depvec_web_lg'])
+def test_different_models(model):
+    nlp = load_test_model(model)
+    # test something specific to the parametrized models
+```
+
+### Known issues and future improvements
+
+Using `importorskip` on a list of model packages is not ideal and we're looking to improve this in the future. But at the moment, it's the best way to ensure that tests are performed on specific model packages only, and that you'll always be able to run the tests, even if you don't have *all available models* installed. (If the tests made a call to `spacy.load('en')` instead, this would load whichever model you've created an `en` shortcut for. This may be one of spaCy's default models, but it could just as easily be your own custom English model.)
+
+The current setup also doesn't provide an easy way to only run tests on specific model versions. The `minversion` keyword argument on `pytest.importorskip` can take care of this, but it currently only checks for the package's `__version__` attribute. An alternative solution would be to load a model package's meta.json and skip if the model's version does not match the one specified in the test.
 
 ## Helpers and utilities
 
@@ -152,11 +200,11 @@ print([token.dep_ for token in doc])
 
 **Note:** There's currently no way of setting the serializer data for the parser without loading the models. If this is relevant to your test, constructing the `Doc` via `get_doc()` won't work.
 
-
 ### Other utilities
 
 | Name | Description |
 | --- | --- |
+| `load_test_model` | Load a model if it's installed as a package, otherwise skip test. |
 | `apply_transition_sequence(parser, doc, sequence)` | Perform a series of pre-specified transitions, to put the parser in a desired state. |
 | `add_vecs_to_vocab(vocab, vectors)` | Add list of vector tuples (`[("text", [1, 2, 3])]`) to given vocab. All vectors need to have the same length. |
 | `get_cosine(vec1, vec2)` | Get cosine for two given vectors. |

spacy/tests/util.py

@@ -12,6 +12,7 @@ MODELS = {}
 
 
 def load_test_model(model):
+    """Load a model if it's installed as a package, otherwise skip."""
     if model not in MODELS:
         module = pytest.importorskip(model)
         MODELS[model] = module.load()

website/_includes/_mixins-base.jade

@@ -108,8 +108,8 @@ mixin quickstart(groups, headline, description, hide_results)
                                 |  #[+help(group.help)]
                     .c-quickstart__fields
                         for option in group.options
-                            input.c-quickstart__input(class="c-quickstart__input--" + (group.input_style ? group.input_style : group.multiple ? "check" : "radio") type=group.multiple ? "checkbox" : "radio" name=group.id id=option.id value=option.id checked=option.checked)
+                            input.c-quickstart__input(class="c-quickstart__input--" + (group.input_style ? group.input_style : group.multiple ? "check" : "radio") type=group.multiple ? "checkbox" : "radio" name=group.id id="qs-#{option.id}" value=option.id checked=option.checked)
-                            label.c-quickstart__label(for=option.id)!=option.title
+                            label.c-quickstart__label(for="qs-#{option.id}")!=option.title
                                 if option.meta
                                     |  #[span.c-quickstart__label__meta (#{option.meta})]
                                 if option.help

website/docs/usage/index.jade

@@ -354,12 +354,14 @@ p
     python -c "import os; import spacy; print(os.path.dirname(spacy.__file__))"
 
 p
-    |  Then run #[code pytest] on that directory. The flags #[code --vectors],
+    |  Then run #[code pytest] on that directory. The flags #[code --slow] and
-    |  #[code --slow] and #[code --model] are optional and enable additional
+    |  #[code --model] are optional and enable additional tests.
-    |  tests:
 
 +code(false, "bash").
     # make sure you are using recent pytest version
     python -m pip install -U pytest
 
-    python -m pytest <spacy-directory> --vectors --models --slow
+    python -m pytest <spacy-directory>                 # basic tests
+    python -m pytest <spacy-directory> --slow          # basic and slow tests
+    python -m pytest <spacy-directory> --models --all  # basic and all model tests
+    python -m pytest <spacy-directory> --models --en   # basic and English model tests

website/docs/usage/spacy-101.jade

@@ -2,6 +2,13 @@
 
 include ../../_includes/_mixins
 
+p
+    |  Whether you're new to spaCy, or just want to brush up on some
+    |  NLP basics and implementation details – this page should have you covered.
+    |  Each section will explain one of spaCy's features in simple terms and
+    |  with examples or illustrations. Some sections will also reappear across
+    |  the usage guides as a quick introcution.
+
 +aside("Help us improve the docs")
     |  Did you spot a mistake or come across explanations that
     |  are unclear? We always appreciate improvement
@@ -13,6 +20,23 @@ include ../../_includes/_mixins
 
 +grid.o-no-block
     +grid-col("half")
+        p
+            |  spaCy is a #[strong free, open-source library] for advanced
+            |  #[strong Natural Language Processing] (NLP) in Python.
+
+        p
+            |  If you're working with a lot of text, you'll eventually want to
+            |  know more about it. For example, what's it about? What do the
+            |  words mean in context? Who is doing what to whom? What companies
+            |  and products are mentioned? Which texts are similar to each other?
+
+        p
+            |  spaCy is designed specifically for #[strong production use] and
+            |  helps you build applications that process and "understand"
+            |  large volumes of text. It can be used to build
+            |  #[strong information extraction] or
+            |  #[strong natural language understanding] systems, or to
+            |  pre-process text for #[strong deep learning].
 
     +grid-col("half")
         +infobox
@@ -31,6 +55,29 @@ include ../../_includes/_mixins
                 +item #[+a("#architecture") Architecture]
                 +item #[+a("#community") Community & FAQ]
 
++h(3, "what-spacy-isnt") What spaCy isn't
+
++list
+    +item #[strong spaCy is not a platform or "an API"].
+        |  Unlike a platform, spaCy does not provide a software as a service, or
+        |  a web application. It's an open-source library designed to help you
+        |  build NLP applications, not a consumable service.
+    +item #[strong spaCy is not an out-of-the-box chat bot engine].
+        |  While spaCy can be used to power conversational applications, it's
+        |  not designed specifically for chat bots, and only provides the
+        |  underlying text processing capabilities.
+    +item #[strong spaCy is not research software].
+        |  It's is built on the latest research, but unlike
+        |  #[+a("https://github./nltk/nltk") NLTK], which is intended for
+        |  teaching and research, spaCy follows a more opinionated approach and
+        |  focuses on production usage. Its aim is to provide you with the best
+        |  possible general-purpose solution for text processing and machine learning
+        |  with text input – but this also means that there's only one implementation
+        |  of each component.
+    +item #[strong spaCy is not a company].
+        |  It's an open-source library. Our company publishing spaCy and other
+        |  software is called #[+a(COMPANY_URL, true) Explosion AI].
+
 +h(2, "features") Features
 
 p