5ae63b1fbd
* Add lang folder for la (Latin) * Add Latin lang classes * Add minimal tokenizer exceptions * Add minimal stopwords * Add minimal lex_attrs * Update stopwords, tokenizer exceptions * Add la tests; register la_tokenizer in conftest.py * Update spacy/lang/la/lex_attrs.py Remove duplicate form in Latin lex_attrs Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> * Update natto-py version spec (#11222) * Update natto-py version spec * Update setup.cfg Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com> Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com> * Add scorer to textcat API docs config settings (#11263) * Update docs for pipeline initialize() methods (#11221) * Update documentation for dependency parser * Update documentation for trainable_lemmatizer * Update documentation for entity_linker * Update documentation for ner * Update documentation for morphologizer * Update documentation for senter * Update documentation for spancat * Update documentation for tagger * Update documentation for textcat * Update documentation for tok2vec * Run prettier on edited files * Apply similar changes in transformer docs * Remove need to say annotated example explicitly I removed the need to say "Must contain at least one annotated Example" because it's often a given that Examples will contain some gold-standard annotation. * Run prettier on transformer docs * chore: add 'concepCy' to spacy universe (#11255) * chore: add 'concepCy' to spacy universe * docs: add 'slogan' to concepCy * Support full prerelease versions in the compat table (#11228) * Support full prerelease versions in the compat table * Fix types * adding spans to doc_annotation in Example.to_dict (#11261) * adding spans to doc_annotation in Example.to_dict * to_dict compatible with from_dict: tuples instead of spans * use strings for label and kb_id * Simplify test * Update data formats docs Co-authored-by: Stefanie Wolf <stefanie.wolf@vitecsoftware.com> Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com> * Fix regex invalid escape sequences (#11276) * Add W605 to the errors raised by flake8 in the CI (#11283) * Clean up automated label-based issue handling (#11284) * Clean up automated label-based issue handline 1. upgrade tiangolo/issue-manager to latest 2. move needs-more-info to tiangolo 3. change needs-more-info close time to 7 days 4. delete old needs-more-info config * Use old, longer message * Fix label name * Fix Dutch noun chunks to skip overlapping spans (#11275) * Add test for overlapping noun chunks * Skip overlapping noun chunks * Update spacy/tests/lang/nl/test_noun_chunks.py Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> * Docs: displaCy documentation - data types, `parse_{deps,ents,spans}`, spans example (#10950) * add in spans example and parse references * rm autoformatter * rm extra ents copy * TypedDict draft * type fixes * restore non-documentation files * docs update * fix spans example * fix hyperlinks * add parse example * example fix + argument fix * fix api arg in docs * fix bad variable replacement * fix spacing in style Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> * fix spacing on table * fix spacing on table * rm temp files Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> * include span_ruler for default warning filter (#11333) * Add uk pipelines to website (#11332) * Check for . in factory names (#11336) * Make fixes for PR #11349 * Fix roman numeral coverage in #11349 Co-authored-by: Patrick J. Burns <patricks@diyclassics.org> Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com> Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com> Co-authored-by: Lj Miranda <12949683+ljvmiranda921@users.noreply.github.com> Co-authored-by: Jules Belveze <32683010+JulesBelveze@users.noreply.github.com> Co-authored-by: stefawolf <wlf.ste@gmail.com> Co-authored-by: Stefanie Wolf <stefanie.wolf@vitecsoftware.com> Co-authored-by: Peter Baumgartner <5107405+pmbaumgartner@users.noreply.github.com> |
||
---|---|---|
.. | ||
doc | ||
lang | ||
matcher | ||
morphology | ||
package | ||
parser | ||
pipeline | ||
serialize | ||
tokenizer | ||
training | ||
vocab_vectors | ||
__init__.py | ||
conftest.py | ||
enable_gpu.py | ||
README.md | ||
test_architectures.py | ||
test_cli.py | ||
test_displacy.py | ||
test_errors.py | ||
test_language.py | ||
test_misc.py | ||
test_models.py | ||
test_pickles.py | ||
test_scorer.py | ||
test_ty.py | ||
util.py |
spaCy tests
spaCy uses the pytest framework for testing. For more info on this, see the pytest documentation.
Tests for spaCy modules and classes live in their own directories of the same name. For example, tests for the Tokenizer
can be found in /tests/tokenizer
. All test modules (i.e. directories) also need to be listed in spaCy's setup.py
. To be interpreted and run, all test files and test functions need to be prefixed with test_
.
⚠️ Important note: As part of our new model training infrastructure, we've moved all model tests to the
spacy-models
repository. This allows us to test the models separately from the core library functionality.
Table of contents
- Running the tests
- Dos and don'ts
- Parameters
- Fixtures
- Helpers and utilities
- Contributing to the tests
Running the tests
To show print statements, run the tests with py.test -s
. To abort after the
first failure, run them with py.test -x
.
py.test spacy # run basic tests
py.test spacy --slow # run basic and slow tests
You can also run tests in a specific file or directory, or even only one specific test:
py.test spacy/tests/tokenizer # run all tests in directory
py.test spacy/tests/tokenizer/test_exceptions.py # run all tests in file
py.test spacy/tests/tokenizer/test_exceptions.py::test_tokenizer_handles_emoji # run specific test
Dos and don'ts
To keep the behavior of the tests consistent and predictable, we try to follow a few basic conventions:
- Test names should follow a pattern of
test_[module]_[tested behaviour]
. For example:test_tokenizer_keeps_email
ortest_spans_override_sentiment
. - If you're testing for a bug reported in a specific issue, always create a regression test. Regression tests should be named
test_issue[ISSUE NUMBER]
and live in theregression
directory. - Only use
@pytest.mark.xfail
for tests that should pass, but currently fail. To test for desired negative behavior, useassert not
in your test. - Very extensive tests that take a long time to run should be marked with
@pytest.mark.slow
. If your slow test is testing important behavior, consider adding an additional simpler version. - If tests require loading the models, they should be added to the
spacy-models
tests. - Before requiring the models, always make sure there is no other way to test the particular behavior. In a lot of cases, it's sufficient to simply create a
Doc
object manually. See the section on helpers and utility functions for more info on this. - Avoid unnecessary imports. There should never be a need to explicitly import spaCy at the top of a file, and many components are available as fixtures. You should also avoid wildcard imports (
from module import *
). - If you're importing from spaCy, always use absolute imports. For example:
from spacy.language import Language
. - Try to keep the tests readable and concise. Use clear and descriptive variable names (
doc
,tokens
andtext
are great), keep it short and only test for one behavior at a time.
Parameters
If the test cases can be extracted from the test, always parametrize
them instead of hard-coding them into the test:
@pytest.mark.parametrize('text', ["google.com", "spacy.io"])
def test_tokenizer_keep_urls(tokenizer, text):
tokens = tokenizer(text)
assert len(tokens) == 1
This will run the test once for each text
value. Even if you're only testing one example, it's usually best to specify it as a parameter. This will later make it easier for others to quickly add additional test cases without having to modify the test.
You can also specify parameters as tuples to test with multiple values per test:
@pytest.mark.parametrize('text,length', [("U.S.", 1), ("us.", 2), ("(U.S.", 2)])
To test for combinations of parameters, you can add several parametrize
markers:
@pytest.mark.parametrize('text', ["A test sentence", "Another sentence"])
@pytest.mark.parametrize('punct', ['.', '!', '?'])
This will run the test with all combinations of the two parameters text
and punct
. Use this feature sparingly, though, as it can easily cause unnecessary or undesired test bloat.
Fixtures
Fixtures to create instances of spaCy objects and other components should only be defined once in the global conftest.py
. We avoid having per-directory conftest files, as this can easily lead to confusion.
These are the main fixtures that are currently available:
Fixture | Description |
---|---|
tokenizer |
Basic, language-independent tokenizer. Identical to the xx language class. |
en_tokenizer , de_tokenizer , ... |
Creates an English, German etc. tokenizer. |
en_vocab |
Creates an instance of the English Vocab . |
The fixtures can be used in all tests by simply setting them as an argument, like this:
def test_module_do_something(en_tokenizer):
tokens = en_tokenizer("Some text here")
If all tests in a file require a specific configuration, or use the same complex example, it can be helpful to create a separate fixture. This fixture should be added at the top of each file. Make sure to use descriptive names for these fixtures and don't override any of the global fixtures listed above. From looking at a test, it should immediately be clear which fixtures are used, and where they are coming from.
Helpers and utilities
Our new test setup comes with a few handy utility functions that can be imported from util.py
.
Constructing a Doc
object manually
Loading the models is expensive and not necessary if you're not actually testing the model performance. If all you need is a Doc
object with annotations like heads, POS tags or the dependency parse, you can construct it manually.
def test_doc_token_api_strings(en_vocab):
words = ["Give", "it", "back", "!", "He", "pleaded", "."]
pos = ['VERB', 'PRON', 'PART', 'PUNCT', 'PRON', 'VERB', 'PUNCT']
heads = [0, 0, 0, 0, 5, 5, 5]
deps = ['ROOT', 'dobj', 'prt', 'punct', 'nsubj', 'ROOT', 'punct']
doc = Doc(en_vocab, words=words, pos=pos, heads=heads, deps=deps)
assert doc[0].text == 'Give'
assert doc[0].lower_ == 'give'
assert doc[0].pos_ == 'VERB'
assert doc[0].dep_ == 'ROOT'
Other utilities
Name | Description |
---|---|
apply_transition_sequence(parser, doc, sequence) |
Perform a series of pre-specified transitions, to put the parser in a desired state. |
add_vecs_to_vocab(vocab, vectors) |
Add list of vector tuples ([("text", [1, 2, 3])] ) to given vocab. All vectors need to have the same length. |
get_cosine(vec1, vec2) |
Get cosine for two given vectors. |
assert_docs_equal(doc1, doc2) |
Compare two Doc objects and assert that they're equal. Tests for tokens, tags, dependencies and entities. |
Contributing to the tests
There's still a long way to go to finally reach 100% test coverage – and we'd appreciate your help! 🙌 You can open an issue on our issue tracker and label it tests
, or make a pull request to this repository.
📖 For more information on contributing to spaCy in general, check out our contribution guidelines.