spaCy/spacy/tests
Matthew Honnibal 8c29268749
Improve spacy.gold (no GoldParse, no json format!) (#5555)
* Update errors

* Remove beam for now (maybe)

Remove beam_utils

Update setup.py

Remove beam

* Remove GoldParse

WIP on removing goldparse

Get ArcEager compiling after GoldParse excise

Update setup.py

Get spacy.syntax compiling after removing GoldParse

Rename NewExample -> Example and clean up

Clean html files

Start updating tests

Update Morphologizer

* fix error numbers

* fix merge conflict

* informative error when calling to_array with wrong field

* fix error catching

* fixing language and scoring tests

* start testing get_aligned

* additional tests for new get_aligned function

* Draft create_gold_state for arc_eager oracle

* Fix import

* Fix import

* Remove TokenAnnotation code from nonproj

* fixing NER one-to-many alignment

* Fix many-to-one IOB codes

* fix test for misaligned

* attempt to fix cases with weird spaces

* fix spaces

* test_gold_biluo_different_tokenization works

* allow None as BILUO annotation

* fixed some tests + WIP roundtrip unit test

* add spaces to json output format

* minibatch utiltiy can deal with strings, docs or examples

* fix augment (needs further testing)

* various fixes in scripts - needs to be further tested

* fix test_cli

* cleanup

* correct silly typo

* add support for MORPH in to/from_array, fix morphologizer overfitting test

* fix tagger

* fix entity linker

* ensure test keeps working with non-linked entities

* pipe() takes docs, not examples

* small bug fix

* textcat bugfix

* throw informative error when running the components with the wrong type of objects

* fix parser tests to work with example (most still failing)

* fix BiluoPushDown parsing entities

* small fixes

* bugfix tok2vec

* fix renames and simple_ner labels

* various small fixes

* prevent writing dummy values like deps because that could interfer with sent_start values

* fix the fix

* implement split_sent with aligned SENT_START attribute

* test for split sentences with various alignment issues, works

* Return ArcEagerGoldParse from ArcEager

* Update parser and NER gold stuff

* Draft new GoldCorpus class

* add links to to_dict

* clean up

* fix test checking for variants

* Fix oracles

* Start updating converters

* Move converters under spacy.gold

* Move things around

* Fix naming

* Fix name

* Update converter to produce DocBin

* Update converters

* Allow DocBin to take list of Doc objects.

* Make spacy convert output docbin

* Fix import

* Fix docbin

* Fix compile in ArcEager

* Fix import

* Serialize all attrs by default

* Update converter

* Remove jsonl converter

* Add json2docs converter

* Draft Corpus class for DocBin

* Work on train script

* Update Corpus

* Update DocBin

* Allocate Doc before starting to add words

* Make doc.from_array several times faster

* Update train.py

* Fix Corpus

* Fix parser model

* Start debugging arc_eager oracle

* Update header

* Fix parser declaration

* Xfail some tests

* Skip tests that cause crashes

* Skip test causing segfault

* Remove GoldCorpus

* Update imports

* Update after removing GoldCorpus

* Fix module name of corpus

* Fix mimport

* Work on parser oracle

* Update arc_eager oracle

* Restore ArcEager.get_cost function

* Update transition system

* Update test_arc_eager_oracle

* Remove beam test

* Update test

* Unskip

* Unskip tests

* add links to to_dict

* clean up

* fix test checking for variants

* Allow DocBin to take list of Doc objects.

* Fix compile in ArcEager

* Serialize all attrs by default

Move converters under spacy.gold

Move things around

Fix naming

Fix name

Update converter to produce DocBin

Update converters

Make spacy convert output docbin

Fix import

Fix docbin

Fix import

Update converter

Remove jsonl converter

Add json2docs converter

* Allocate Doc before starting to add words

* Make doc.from_array several times faster

* Start updating converters

* Work on train script

* Draft Corpus class for DocBin

Update Corpus

Fix Corpus

* Update DocBin

Add missing strings when serializing

* Update train.py

* Fix parser model

* Start debugging arc_eager oracle

* Update header

* Fix parser declaration

* Xfail some tests

Skip tests that cause crashes

Skip test causing segfault

* Remove GoldCorpus

Update imports

Update after removing GoldCorpus

Fix module name of corpus

Fix mimport

* Work on parser oracle

Update arc_eager oracle

Restore ArcEager.get_cost function

Update transition system

* Update tests

Remove beam test

Update test

Unskip

Unskip tests

* Add get_aligned_parse method in Example

Fix Example.get_aligned_parse

* Add kwargs to Corpus.dev_dataset to match train_dataset

* Update nonproj

* Use get_aligned_parse in ArcEager

* Add another arc-eager oracle test

* Remove Example.doc property

Remove Example.doc

Remove Example.doc

Remove Example.doc

Remove Example.doc

* Update ArcEager oracle

Fix Break oracle

* Debugging

* Fix Corpus

* Fix eg.doc

* Format

* small fixes

* limit arg for Corpus

* fix test_roundtrip_docs_to_docbin

* fix test_make_orth_variants

* fix add_label test

* Update tests

* avoid writing temp dir in json2docs, fixing 4402 test

* Update test

* Add missing costs to NER oracle

* Update test

* Work on Example.get_aligned_ner method

* Clean up debugging

* Xfail tests

* Remove prints

* Remove print

* Xfail some tests

* Replace unseen labels for parser

* Update test

* Update test

* Xfail test

* Fix Corpus

* fix imports

* fix docs_to_json

* various small fixes

* cleanup

* Support gold_preproc in Corpus

* Support gold_preproc

* Pass gold_preproc setting into corpus

* Remove debugging

* Fix gold_preproc

* Fix json2docs converter

* Fix convert command

* Fix flake8

* Fix import

* fix output_dir (converted to Path by typer)

* fix var

* bugfix: update states after creating golds to avoid out of bounds indexing

* Improve efficiency of ArEager oracle

* pull merge_sent into iob2docs to avoid Doc creation for each line

* fix asserts

* bugfix excl Span.end in iob2docs

* Support max_length in Corpus

* Fix arc_eager oracle

* Filter out uannotated sentences in NER

* Remove debugging in parser

* Simplify NER alignment

* Fix conversion of NER data

* Fix NER init_gold_batch

* Tweak efficiency of precomputable affine

* Update onto-json default

* Update gold test for NER

* Fix parser test

* Update test

* Add NER data test

* Fix convert for single file

* Fix test

* Hack scorer to avoid evaluating non-nered data

* Fix handling of NER data in Example

* Output unlabelled spans from O biluo tags in iob_utils

* Fix unset variable

* Return kept examples from init_gold_batch

* Return examples from init_gold_batch

* Dont return Example from init_gold_batch

* Set spaces on gold doc after conversion

* Add test

* Fix spaces reading

* Improve NER alignment

* Improve handling of missing values in NER

* Restore the 'cutting' in parser training

* Add assertion

* Print epochs

* Restore random cuts in parser/ner training

* Implement Doc.copy

* Implement Example.copy

* Copy examples at the start of Language.update

* Don't unset example docs

* Tweak parser model slightly

* attempt to fix _guess_spaces

* _add_entities_to_doc first, so that links don't get overwritten

* fixing get_aligned_ner for one-to-many

* fix indexing into x_text

* small fix biluo_tags_from_offsets

* Add onto-ner config

* Simplify NER alignment

* Fix NER scoring for partially annotated documents

* fix indexing into x_text

* fix test_cli failing tests by ignoring spans in doc.ents with empty label

* Fix limit

* Improve NER alignment

* Fix count_train

* Remove print statement

* fix tests, we're not having nothing but None

* fix clumsy fingers

* Fix tests

* Fix doc.ents

* Remove empty docs in Corpus and improve limit

* Update config

Co-authored-by: svlandeg <sofie.vanlandeghem@gmail.com>
2020-06-26 19:34:12 +02:00
..
doc Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00
lang Tidy up and auto-format 2020-06-21 22:38:04 +02:00
matcher Merge branch 'develop' into master-tmp 2020-06-20 15:52:00 +02:00
morphology Tidy up and auto-format 2020-02-18 15:38:18 +01:00
package Tidy up and auto-format 2020-06-21 22:38:04 +02:00
parser Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00
pipeline Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00
regression Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00
serialize Remove references to Python 2 / is_python2 2020-06-20 16:11:13 +02:00
tokenizer Merge branch 'develop' into master-tmp 2020-06-20 15:52:00 +02:00
vocab_vectors Remove references to Python 2 / is_python2 2020-06-20 16:11:13 +02:00
__init__.py Revert #4334 2019-09-29 17:32:12 +02:00
conftest.py Merge branch 'develop' into master-tmp 2020-06-20 15:52:00 +02:00
README.md Revert #4334 2019-09-29 17:32:12 +02:00
test_architectures.py Tidy up and auto-format 2020-02-18 15:38:18 +01:00
test_cli.py Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00
test_displacy.py Merge branch 'master' into tmp/sync 2020-03-26 13:38:14 +01:00
test_errors.py fixup! Fix ErrorsWithCodes().__class__ return value 2020-05-14 15:45:58 +02:00
test_gold.py Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00
test_language.py Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00
test_lemmatizer.py add additional test back in (it works now) 2020-06-20 23:23:57 +02:00
test_misc.py Tidy up and auto-format 2020-06-21 22:38:04 +02:00
test_new_example.py Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00
test_pickles.py Drop Python 2.7 and 3.5 (#4828) 2019-12-22 01:53:56 +01:00
test_scorer.py Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00
test_tok2vec.py make dropout in embed layers configurable 2020-06-03 11:50:16 +02:00
test_util.py Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00
util.py Improve spacy.gold (no GoldParse, no json format!) (#5555) 2020-06-26 19:34:12 +02:00

spaCy tests

spaCy uses the pytest framework for testing. For more info on this, see the pytest documentation.

Tests for spaCy modules and classes live in their own directories of the same name. For example, tests for the Tokenizer can be found in /tests/tokenizer. All test modules (i.e. directories) also need to be listed in spaCy's setup.py. To be interpreted and run, all test files and test functions need to be prefixed with test_.

⚠️ Important note: As part of our new model training infrastructure, we've moved all model tests to the spacy-models repository. This allows us to test the models separately from the core library functionality.

Table of contents

  1. Running the tests
  2. Dos and don'ts
  3. Parameters
  4. Fixtures
  5. Helpers and utilities
  6. Contributing to the tests

Running the tests

To show print statements, run the tests with py.test -s. To abort after the first failure, run them with py.test -x.

py.test spacy                        # run basic tests
py.test spacy --slow                 # run basic and slow tests

You can also run tests in a specific file or directory, or even only one specific test:

py.test spacy/tests/tokenizer  # run all tests in directory
py.test spacy/tests/tokenizer/test_exceptions.py # run all tests in file
py.test spacy/tests/tokenizer/test_exceptions.py::test_tokenizer_handles_emoji # run specific test

Dos and don'ts

To keep the behaviour of the tests consistent and predictable, we try to follow a few basic conventions:

  • Test names should follow a pattern of test_[module]_[tested behaviour]. For example: test_tokenizer_keeps_email or test_spans_override_sentiment.
  • If you're testing for a bug reported in a specific issue, always create a regression test. Regression tests should be named test_issue[ISSUE NUMBER] and live in the regression directory.
  • Only use @pytest.mark.xfail for tests that should pass, but currently fail. To test for desired negative behaviour, use assert not in your test.
  • Very extensive tests that take a long time to run should be marked with @pytest.mark.slow. If your slow test is testing important behaviour, consider adding an additional simpler version.
  • If tests require loading the models, they should be added to the spacy-models tests.
  • Before requiring the models, always make sure there is no other way to test the particular behaviour. In a lot of cases, it's sufficient to simply create a Doc object manually. See the section on helpers and utility functions for more info on this.
  • Avoid unnecessary imports. There should never be a need to explicitly import spaCy at the top of a file, and many components are available as fixtures. You should also avoid wildcard imports (from module import *).
  • If you're importing from spaCy, always use absolute imports. For example: from spacy.language import Language.
  • Don't forget the unicode declarations at the top of each file. This way, unicode strings won't have to be prefixed with u.
  • Try to keep the tests readable and concise. Use clear and descriptive variable names (doc, tokens and text are great), keep it short and only test for one behaviour at a time.

Parameters

If the test cases can be extracted from the test, always parametrize them instead of hard-coding them into the test:

@pytest.mark.parametrize('text', ["google.com", "spacy.io"])
def test_tokenizer_keep_urls(tokenizer, text):
    tokens = tokenizer(text)
    assert len(tokens) == 1

This will run the test once for each text value. Even if you're only testing one example, it's usually best to specify it as a parameter. This will later make it easier for others to quickly add additional test cases without having to modify the test.

You can also specify parameters as tuples to test with multiple values per test:

@pytest.mark.parametrize('text,length', [("U.S.", 1), ("us.", 2), ("(U.S.", 2)])

To test for combinations of parameters, you can add several parametrize markers:

@pytest.mark.parametrize('text', ["A test sentence", "Another sentence"])
@pytest.mark.parametrize('punct', ['.', '!', '?'])

This will run the test with all combinations of the two parameters text and punct. Use this feature sparingly, though, as it can easily cause unneccessary or undesired test bloat.

Fixtures

Fixtures to create instances of spaCy objects and other components should only be defined once in the global conftest.py. We avoid having per-directory conftest files, as this can easily lead to confusion.

These are the main fixtures that are currently available:

Fixture Description
tokenizer Basic, language-independent tokenizer. Identical to the xx language class.
en_tokenizer, de_tokenizer, ... Creates an English, German etc. tokenizer.
en_vocab Creates an instance of the English Vocab.

The fixtures can be used in all tests by simply setting them as an argument, like this:

def test_module_do_something(en_tokenizer):
    tokens = en_tokenizer("Some text here")

If all tests in a file require a specific configuration, or use the same complex example, it can be helpful to create a separate fixture. This fixture should be added at the top of each file. Make sure to use descriptive names for these fixtures and don't override any of the global fixtures listed above. From looking at a test, it should immediately be clear which fixtures are used, and where they are coming from.

Helpers and utilities

Our new test setup comes with a few handy utility functions that can be imported from util.py.

Constructing a Doc object manually with get_doc()

Loading the models is expensive and not necessary if you're not actually testing the model performance. If all you need ia a Doc object with annotations like heads, POS tags or the dependency parse, you can use get_doc() to construct it manually.

def test_doc_token_api_strings(en_tokenizer):
    text = "Give it back! He pleaded."
    pos = ['VERB', 'PRON', 'PART', 'PUNCT', 'PRON', 'VERB', 'PUNCT']
    heads = [0, -1, -2, -3, 1, 0, -1]
    deps = ['ROOT', 'dobj', 'prt', 'punct', 'nsubj', 'ROOT', 'punct']

    tokens = en_tokenizer(text)
    doc = get_doc(tokens.vocab, [t.text for t in tokens], pos=pos, heads=heads, deps=deps)
    assert doc[0].text == 'Give'
    assert doc[0].lower_ == 'give'
    assert doc[0].pos_ == 'VERB'
    assert doc[0].dep_ == 'ROOT'

You can construct a Doc with the following arguments:

Argument Description
vocab Vocab instance to use. If you're tokenizing before creating a Doc, make sure to use the tokenizer's vocab. Otherwise, you can also use the en_vocab fixture. (required)
words List of words, for example [t.text for t in tokens]. (required)
heads List of heads as integers.
pos List of POS tags as text values.
tag List of tag names as text values.
dep List of dependencies as text values.
ents List of entity tuples with start, end, label (for example (0, 2, 'PERSON')). The label will be looked up in vocab.strings[label].

Here's how to quickly get these values from within spaCy:

doc = nlp(u'Some text here')
print([token.head.i-token.i for token in doc])
print([token.tag_ for token in doc])
print([token.pos_ for token in doc])
print([token.dep_ for token in doc])
print([(ent.start, ent.end, ent.label_) for ent in doc.ents])

Note: There's currently no way of setting the serializer data for the parser without loading the models. If this is relevant to your test, constructing the Doc via get_doc() won't work.

Other utilities

Name Description
apply_transition_sequence(parser, doc, sequence) Perform a series of pre-specified transitions, to put the parser in a desired state.
add_vecs_to_vocab(vocab, vectors) Add list of vector tuples ([("text", [1, 2, 3])]) to given vocab. All vectors need to have the same length.
get_cosine(vec1, vec2) Get cosine for two given vectors.
assert_docs_equal(doc1, doc2) Compare two Doc objects and assert that they're equal. Tests for tokens, tags, dependencies and entities.

Contributing to the tests

There's still a long way to go to finally reach 100% test coverage and we'd appreciate your help! 🙌 You can open an issue on our issue tracker and label it tests, or make a pull request to this repository.

📖 For more information on contributing to spaCy in general, check out our contribution guidelines.