Commit Graph

15910 Commits

Author SHA1 Message Date
Duygu Altinok
7b98aa4c16
Corrected broken (#9505) 2021-10-20 17:31:59 +02:00
Edward
014da12f1d
Dont add tok2vec when efficiency textcat (#9502) 2021-10-20 17:30:19 +02:00
Ryn Daniels
ddc1bf5b8b
Merge pull request #9518 from explosion/rfd-robot-slowtests
Enable the test_slow command for explosionbot
2021-10-20 12:44:20 +02:00
Daniël de Kok
1f05f56433
Add the spacy.models_with_nvtx_range.v1 callback (#9124)
* Add the spacy.models_with_nvtx_range.v1 callback

This callback recursively adds NVTX ranges to the Models in each pipe in
a pipeline.

* Fix create_models_with_nvtx_range type signature

* NVTX range: wrap models of all trainable pipes jointly

This avoids that (sub-)models that are shared between pipes get wrapped
twice.

* NVTX range callback: make color configurable

Add forward_color and backprop_color options to set the color for the
NVTX range.

* Move create_models_with_nvtx_range to spacy.ml

* Update create_models_with_nvtx_range for thinc changes

with_nvtx_range now updates an existing node, rather than returning a
wrapper node. So, we can simply walk over the nodes and update them.

* NVTX: use after_pipeline_creation in example
2021-10-20 11:59:48 +02:00
Ryn Daniels
66b474ce05
Merge branch 'master' into rfd-robot-slowtests 2021-10-20 11:56:01 +02:00
Ryn Daniels
393e187f2c Enable the test_slow command for explosionbot 2021-10-20 11:20:57 +02:00
Ines Montani
5facdb031c
Merge pull request #9506 from explosion/tests/conftest-options 2021-10-20 10:33:43 +02:00
Sofie Van Landeghem
b758270654
bump thinc to 8.0.11 (#9516) 2021-10-20 10:32:09 +02:00
Adriane Boyd
3f181b73d0
Add ja_core_news_trf to website (#9515) 2021-10-20 10:18:02 +02:00
Paul O'Leary McCann
ef4d4f793b Clarify how to change base Transformer model (#9498)
* Add note about how the model name is used

* Add link to TransformersModel docs, separate paragraph

* Local link

* Revise docs

* Update website/docs/usage/embeddings-transformers.md

* Update website/docs/usage/embeddings-transformers.md

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
2021-10-19 23:32:14 +02:00
Paul O'Leary McCann
222cf9b6d2
Clarify how to change base Transformer model (#9498)
* Add note about how the model name is used

* Add link to TransformersModel docs, separate paragraph

* Local link

* Revise docs

* Update website/docs/usage/embeddings-transformers.md

* Update website/docs/usage/embeddings-transformers.md

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
2021-10-19 23:28:20 +02:00
Ines Montani
ad9f57cbbf Allow conftest.py to run twice for build envs 2021-10-19 15:13:25 +02:00
Sofie Van Landeghem
da578c3d3b
Fix kb.set_entities (#9463)
* avoid creating _vectors_table when also using c_add_vector

* write to self._vectors_table directly in set_entities
2021-10-19 09:39:17 +02:00
Lj Miranda
068cae7755
Include .pyi files in MANIFEST.in (#9500) 2021-10-19 09:05:37 +02:00
Lj Miranda
2bcd383685
Replace previous lock threads with GH action (#9499) 2021-10-19 09:03:59 +02:00
Adriane Boyd
e66fddf934 Minor updates to spacy-transformers docs for v1.1.0 (#9496) 2021-10-18 14:55:22 +02:00
Adriane Boyd
a6424bcea9
Minor updates to spacy-transformers docs for v1.1.0 (#9496) 2021-10-18 14:55:02 +02:00
Adriane Boyd
404aff08e3 Update docs for spacy-transformers v1.1 data classes (#9361) 2021-10-18 14:17:48 +02:00
Sofie Van Landeghem
eaa6798c66 Docs for new spacy-trf architectures (#8954)
* use TransformerModel.v2 in quickstart

* update docs for new transformer architectures

* bump spacy_transformers to 1.1.0

* Add new arguments spacy-transformers.TransformerModel.v3

* Mention that mixed-precision support is experimental

* Describe delta transformers.Tok2VecTransformer versions

* add dot

* add dot, again

* Update some more TransformerModel references v2 -> v3

* Add mixed-precision options to the training quickstart

Disable mixed-precision training/prediction by default.

* Update setup.cfg

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>

* Apply suggestions from code review

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>

* Update website/docs/usage/embeddings-transformers.md

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>

Co-authored-by: Daniël de Kok <me@danieldk.eu>
Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
2021-10-18 14:17:43 +02:00
Adriane Boyd
9b86209a4a
Update docs for spacy-transformers v1.1 data classes (#9361) 2021-10-18 14:16:58 +02:00
Sofie Van Landeghem
3fd3531e12
Docs for new spacy-trf architectures (#8954)
* use TransformerModel.v2 in quickstart

* update docs for new transformer architectures

* bump spacy_transformers to 1.1.0

* Add new arguments spacy-transformers.TransformerModel.v3

* Mention that mixed-precision support is experimental

* Describe delta transformers.Tok2VecTransformer versions

* add dot

* add dot, again

* Update some more TransformerModel references v2 -> v3

* Add mixed-precision options to the training quickstart

Disable mixed-precision training/prediction by default.

* Update setup.cfg

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>

* Apply suggestions from code review

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>

* Update website/docs/usage/embeddings-transformers.md

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>

Co-authored-by: Daniël de Kok <me@danieldk.eu>
Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
2021-10-18 14:15:06 +02:00
Edward
a7cb8de0d7
Fix assertion error in staticvectors (#9481)
* Fix assertion error in staticvectors

* Update spacy/ml/staticvectors.py

* Update spacy/ml/staticvectors.py

Co-authored-by: Ines Montani <ines@ines.io>

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
Co-authored-by: Ines Montani <ines@ines.io>
2021-10-18 09:10:45 +02:00
Adriane Boyd
74ec37f7a8
Switch back to exclude pr paths (#9486) 2021-10-16 23:48:55 +02:00
Adriane Boyd
fca242b34e
Avoid initial wildcards in PR CI filter (#9477)
* Avoid initial wildcards in PR CI filter

* Adjust wildcard patterns
2021-10-15 17:32:00 +02:00
Adriane Boyd
271e8e7856
Skip compat table tests for prerelease versions (#9476) 2021-10-15 14:28:02 +02:00
github-actions[bot]
29e83f0819
Auto-format code with black (#9474)
* Auto-format code with black

* Update spacy/pipeline/pipe.pyi

Co-authored-by: explosion-bot <explosion-bot@users.noreply.github.com>
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
2021-10-15 11:36:49 +02:00
Aviora
9a824255d3
Add examples and num_words for Vietnamese (#9412)
* add examples and num_words

* add contributor agreement

* Update spacy/lang/vi/examples.py

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* consistent format

add empty line at the end of file

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
2021-10-14 19:15:51 +02:00
Adriane Boyd
b5143b1b84
Minor fixes to convert CLI (#9465)
* Provide default value for `msg`
* Compare paths correctly for file conversion
2021-10-14 18:37:34 +02:00
Connor Brinton
657af5f91f
🏷 Add Mypy check to CI and ignore all existing Mypy errors (#9167)
* 🚨 Ignore all existing Mypy errors

* 🏗 Add Mypy check to CI

* Add types-mock and types-requests as dev requirements

* Add additional type ignore directives

* Add types packages to dev-only list in reqs test

* Add types-dataclasses for python 3.6

* Add ignore to pretrain

* 🏷 Improve type annotation on `run_command` helper

The `run_command` helper previously declared that it returned an
`Optional[subprocess.CompletedProcess]`, but it isn't actually possible
for the function to return `None`. These changes modify the type
annotation of the `run_command` helper and remove all now-unnecessary
`# type: ignore` directives.

* 🔧 Allow variable type redefinition in limited contexts

These changes modify how Mypy is configured to allow variables to have
their type automatically redefined under certain conditions. The Mypy
documentation contains the following example:

```python
def process(items: List[str]) -> None:
    # 'items' has type List[str]
    items = [item.split() for item in items]
    # 'items' now has type List[List[str]]
    ...
```

This configuration change is especially helpful in reducing the number
of `# type: ignore` directives needed to handle the common pattern of:
* Accepting a filepath as a string
* Overwriting the variable using `filepath = ensure_path(filepath)`

These changes enable redefinition and remove all `# type: ignore`
directives rendered redundant by this change.

* 🏷 Add type annotation to converters mapping

* 🚨 Fix Mypy error in convert CLI argument verification

* 🏷 Improve type annotation on `resolve_dot_names` helper

* 🏷 Add type annotations for `Vocab` attributes `strings` and `vectors`

* 🏷 Add type annotations for more `Vocab` attributes

* 🏷 Add loose type annotation for gold data compilation

* 🏷 Improve `_format_labels` type annotation

* 🏷 Fix `get_lang_class` type annotation

* 🏷 Loosen return type of `Language.evaluate`

* 🏷 Don't accept `Scorer` in `handle_scores_per_type`

* 🏷 Add `string_to_list` overloads

* 🏷 Fix non-Optional command-line options

* 🙈 Ignore redefinition of `wandb_logger` in `loggers.py`

*  Install `typing_extensions` in Python 3.8+

The `typing_extensions` package states that it should be used when
"writing code that must be compatible with multiple Python versions".
Since SpaCy needs to support multiple Python versions, it should be used
when newer `typing` module members are required. One example of this is
`Literal`, which is available starting with Python 3.8.

Previously SpaCy tried to import `Literal` from `typing`, falling back
to `typing_extensions` if the import failed. However, Mypy doesn't seem
to be able to understand what `Literal` means when the initial import
means. Therefore, these changes modify how `compat` imports `Literal` by
always importing it from `typing_extensions`.

These changes also modify how `typing_extensions` is installed, so that
it is a requirement for all Python versions, including those greater
than or equal to 3.8.

* 🏷 Improve type annotation for `Language.pipe`

These changes add a missing overload variant to the type signature of
`Language.pipe`. Additionally, the type signature is enhanced to allow
type checkers to differentiate between the two overload variants based
on the `as_tuple` parameter.

Fixes #8772

*  Don't install `typing-extensions` in Python 3.8+

After more detailed analysis of how to implement Python version-specific
type annotations using SpaCy, it has been determined that by branching
on a comparison against `sys.version_info` can be statically analyzed by
Mypy well enough to enable us to conditionally use
`typing_extensions.Literal`. This means that we no longer need to
install `typing_extensions` for Python versions greater than or equal to
3.8! 🎉

These changes revert previous changes installing `typing-extensions`
regardless of Python version and modify how we import the `Literal` type
to ensure that Mypy treats it properly.

* resolve mypy errors for Strict pydantic types

* refactor code to avoid missing return statement

* fix types of convert CLI command

* avoid list-set confustion in debug_data

* fix typo and formatting

* small fixes to avoid type ignores

* fix types in profile CLI command and make it more efficient

* type fixes in projects CLI

* put one ignore back

* type fixes for render

* fix render types - the sequel

* fix BaseDefault in language definitions

* fix type of noun_chunks iterator - yields tuple instead of span

* fix types in language-specific modules

* 🏷 Expand accepted inputs of `get_string_id`

`get_string_id` accepts either a string (in which case it returns its 
ID) or an ID (in which case it immediately returns the ID). These 
changes extend the type annotation of `get_string_id` to indicate that 
it can accept either strings or IDs.

* 🏷 Handle override types in `combine_score_weights`

The `combine_score_weights` function allows users to pass an `overrides` 
mapping to override data extracted from the `weights` argument. Since it 
allows `Optional` dictionary values, the return value may also include 
`Optional` dictionary values.

These changes update the type annotations for `combine_score_weights` to 
reflect this fact.

* 🏷 Fix tokenizer serialization method signatures in `DummyTokenizer`

* 🏷 Fix redefinition of `wandb_logger`

These changes fix the redefinition of `wandb_logger` by giving a 
separate name to each `WandbLogger` version. For 
backwards-compatibility, `spacy.train` still exports `wandb_logger_v3` 
as `wandb_logger` for now.

* more fixes for typing in language

* type fixes in model definitions

* 🏷 Annotate `_RandomWords.probs` as `NDArray`

* 🏷 Annotate `tok2vec` layers to help Mypy

* 🐛 Fix `_RandomWords.probs` type annotations for Python 3.6

Also remove an import that I forgot to move to the top of the module 😅

* more fixes for matchers and other pipeline components

* quick fix for entity linker

* fixing types for spancat, textcat, etc

* bugfix for tok2vec

* type annotations for scorer

* add runtime_checkable for Protocol

* type and import fixes in tests

* mypy fixes for training utilities

* few fixes in util

* fix import

* 🐵 Remove unused `# type: ignore` directives

* 🏷 Annotate `Language._components`

* 🏷 Annotate `spacy.pipeline.Pipe`

* add doc as property to span.pyi

* small fixes and cleanup

* explicit type annotations instead of via comment

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
Co-authored-by: svlandeg <sofie.vanlandeghem@gmail.com>
Co-authored-by: svlandeg <svlandeg@github.com>
2021-10-14 15:21:40 +02:00
Adriane Boyd
631c170fa5
Adjust PR CI includes (#9464) 2021-10-14 13:46:09 +02:00
Adriane Boyd
8db574e0b5
Temporarily ignore W095 in assemble CLI CI test (#9460)
* Temporarily ignore W095 in assemble CLI CI test

* Adjust PR CI includes
2021-10-14 13:27:39 +02:00
Adriane Boyd
fe6d63aedc
Merge pull request #9459 from adrianeboyd/chore/v3.2.0.dev0
Set version to v3.2.0.dev0
2021-10-14 11:07:19 +02:00
Adriane Boyd
bd6433bbab Temporarily use v3.1.0 models in CI 2021-10-14 10:31:11 +02:00
Adriane Boyd
8a018f5207 Set version to v3.2.0.dev0 2021-10-14 10:31:11 +02:00
Adriane Boyd
d8e8ffdc5c
Merge pull request #9458 from adrianeboyd/chore/update-develop-from-master-v3.1-3
Update develop from master for v3.2.0
2021-10-14 10:30:33 +02:00
Adriane Boyd
d98d525bc8 Merge remote-tracking branch 'upstream/master' into chore/update-develop-from-master-v3.1-3 2021-10-14 09:41:46 +02:00
Edward
9dfb12e29f Update universe example codes (#9422)
* Update universe plugins

* Adjust azure trigger

* Add init to tests/universe

* deliberatly trying to break the universe to see if the CI catches it

* revert

Co-authored-by: svlandeg <svlandeg@github.com>
2021-10-14 09:37:05 +02:00
Paul O'Leary McCann
a3b7519aba
Fix JA Morph Values (#9449)
* Don't set empty / weird values in morph

* Update tests to handy empty morph values

* Fix everything

* Replace potentially problematic characters

* Fix test
2021-10-14 09:21:36 +02:00
Ines Montani
c48564688f
Merge pull request #9423 from explosion/tests/issue-marker 2021-10-13 16:53:40 +02:00
Edward
72711dc2c9
Update universe example codes (#9422)
* Update universe plugins

* Adjust azure trigger

* Add init to tests/universe

* deliberatly trying to break the universe to see if the CI catches it

* revert

Co-authored-by: svlandeg <svlandeg@github.com>
2021-10-13 16:29:19 +02:00
Jette16
78365452d3
Moved test for universe into .github folder (#9447)
* Moved universe-test into .github folder

* Cleaned code

* CHanged a file name
2021-10-13 14:13:06 +02:00
Sofie Van Landeghem
d2645b2e03
Fix test for spancat (#9446)
* fix test for spancat

* increase tolerance for almost equal checks

* Update spacy/tests/test_models.py

* Update spacy/tests/test_models.py
2021-10-13 10:48:35 +02:00
Sofie Van Landeghem
2e3d6b8b5a
Fix test for spancat (#9446)
* fix test for spancat

* increase tolerance for almost equal checks

* Update spacy/tests/test_models.py

* Update spacy/tests/test_models.py
2021-10-13 10:47:56 +02:00
Sofie Van Landeghem
5e8e8525f0
fix W108 filter (#9438)
* remove text argument from W108 to enable 'once' filtering

* include the option of partial POS annotation

* fix typo

* Update spacy/errors.py

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
2021-10-12 19:56:44 +02:00
Lj Miranda
6425b9a1c4
Include JsonlCorpus from the imports (#9431) 2021-10-12 15:39:14 +02:00
Ryn Daniels
bb6623bb2d
Merge pull request #9426 from explosion/rfd-bot-config
Add allowed_teams to the explosion-bot config
2021-10-12 09:50:39 +02:00
Ryn Daniels
2fb420ec23 Add allowed_teams to the explosion-bot config 2021-10-11 18:20:48 +02:00
Ryn Daniels
f64e39fa49
Install explosionbot as a github action (#9420) 2021-10-11 15:43:27 +02:00
Paul O'Leary McCann
efe5beefe0
Add test for case where parser overwrite annotations (#9406)
* Add test for case where parser overwrite annotations

* Move test to its own file

Also add note about how other tokens modify results.

* Fix xfail decorator
2021-10-11 14:57:45 +02:00
Ines Montani
1fa7c4e73b Support issue marker via pytest 2021-10-11 13:56:24 +02:00