+
+# Developer Documentation
+
+This directory includes additional documentation and explanations of spaCy's internals. It's mostly intended for the spaCy core development team and contributors interested in the more complex parts of the library. The documents generally focus on more abstract implementation details and how specific methods and algorithms work, and they assume knowledge of what's already available in the [usage documentation](https://spacy.io/usage) and [API reference](https://spacy.io/api).
+
+If you're looking to contribute to spaCy, make sure to check out the documentation and [contributing guide](https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md) first.
diff --git a/extra/DEVELOPER_DOCS/StringStore-Vocab.md b/extra/DEVELOPER_DOCS/StringStore-Vocab.md
new file mode 100644
index 000000000..866ba2aae
--- /dev/null
+++ b/extra/DEVELOPER_DOCS/StringStore-Vocab.md
@@ -0,0 +1,216 @@
+# StringStore & Vocab
+
+> Reference: `spacy/strings.pyx`
+> Reference: `spacy/vocab.pyx`
+
+## Overview
+
+spaCy represents mosts strings internally using a `uint64` in Cython which
+corresponds to a hash. The magic required to make this largely transparent is
+handled by the `StringStore`, and is integrated into the pipelines using the
+`Vocab`, which also connects it to some other information.
+
+These are mostly internal details that average library users should never have
+to think about. On the other hand, when developing a component it's normal to
+interact with the Vocab for lexeme data or word vectors, and it's not unusual
+to add labels to the `StringStore`.
+
+## StringStore
+
+### Overview
+
+The `StringStore` is a `cdef class` that looks a bit like a two-way dictionary,
+though it is not a subclass of anything in particular.
+
+The main functionality of the `StringStore` is that `__getitem__` converts
+hashes into strings or strings into hashes.
+
+The full details of the conversion are complicated. Normally you shouldn't have
+to worry about them, but the first applicable case here is used to get the
+return value:
+
+1. 0 and the empty string are special cased to each other
+2. internal symbols use a lookup table (`SYMBOLS_BY_STR`)
+3. normal strings or bytes are hashed
+4. internal symbol IDs in `SYMBOLS_BY_INT` are handled
+5. anything not yet handled is used as a hash to lookup a string
+
+For the symbol enums, see [`symbols.pxd`](https://github.com/explosion/spaCy/blob/master/spacy/symbols.pxd).
+
+Almost all strings in spaCy are stored in the `StringStore`. This naturally
+includes tokens, but also includes things like labels (not just NER/POS/dep,
+but also categories etc.), lemmas, lowercase forms, word shapes, and so on. One
+of the main results of this is that tokens can be represented by a compact C
+struct ([`LexemeC`](https://spacy.io/api/cython-structs#lexemec)/[`TokenC`](https://github.com/explosion/spaCy/issues/4854)) that mostly consists of string hashes. This also means that converting
+input for the models is straightforward, and there's not a token mapping step
+like in many machine learning frameworks. Additionally, because the token IDs
+in spaCy are based on hashes, they are consistent across environments or
+models.
+
+One pattern you'll see a lot in spaCy APIs is that `something.value` returns an
+`int` and `something.value_` returns a string. That's implemented using the
+`StringStore`. Typically the `int` is stored in a C struct and the string is
+generated via a property that calls into the `StringStore` with the `int`.
+
+Besides `__getitem__`, the `StringStore` has functions to return specifically a
+string or specifically a hash, regardless of whether the input was a string or
+hash to begin with, though these are only used occasionally.
+
+### Implementation Details: Hashes and Allocations
+
+Hashes are 64-bit and are computed using [murmurhash][] on UTF-8 bytes. There is no
+mechanism for detecting and avoiding collisions. To date there has never been a
+reproducible collision or user report about any related issues.
+
+[murmurhash]: https://github.com/explosion/murmurhash
+
+The empty string is not hashed, it's just converted to/from 0.
+
+A small number of strings use indices into a lookup table (so low integers)
+rather than hashes. This is mostly Universal Dependencies labels or other
+strings considered "core" in spaCy. This was critical in v1, which hadn't
+introduced hashing yet. Since v2 it's important for items in `spacy.attrs`,
+especially lexeme flags, but is otherwise only maintained for backwards
+compatibility.
+
+You can call `strings["mystring"]` with a string the `StringStore` has never seen
+before and it will return a hash. But in order to do the reverse operation, you
+need to call `strings.add("mystring")` first. Without a call to `add` the
+string will not be interned.
+
+Example:
+
+```
+from spacy.strings import StringStore
+
+ss = StringStore()
+hashval = ss["spacy"] # 10639093010105930009
+try:
+ # this won't work
+ ss[hashval]
+except KeyError:
+ print(f"key {hashval} unknown in the StringStore.")
+
+ss.add("spacy")
+assert ss[hashval] == "spacy" # it works now
+
+# There is no `.keys` property, but you can iterate over keys
+# The empty string will never be in the list of keys
+for key in ss:
+ print(key)
+```
+
+In normal use nothing is ever removed from the `StringStore`. In theory this
+means that if you do something like iterate through all hex values of a certain
+length you can have explosive memory usage. In practice this has never been an
+issue. (Note that this is also different from using `sys.intern` to intern
+Python strings, which does not guarantee they won't be garbage collected later.)
+
+Strings are stored in the `StringStore` in a peculiar way: each string uses a
+union that is either an eight-byte `char[]` or a `char*`. Short strings are
+stored directly in the `char[]`, while longer strings are stored in allocated
+memory and prefixed with their length. This is a strategy to reduce indirection
+and memory fragmentation. See `decode_Utf8Str` and `_allocate` in
+`strings.pyx` for the implementation.
+
+### When to Use the StringStore?
+
+While you can ignore the `StringStore` in many cases, there are situations where
+you should make use of it to avoid errors.
+
+Any time you introduce a string that may be set on a `Doc` field that has a hash,
+you should add the string to the `StringStore`. This mainly happens when adding
+labels in components, but there are some other cases:
+
+- syntax iterators, mainly `get_noun_chunks`
+- external data used in components, like the `KnowledgeBase` in the `entity_linker`
+- labels used in tests
+
+## Vocab
+
+The `Vocab` is a core component of a `Language` pipeline. Its main function is
+to manage `Lexeme`s, which are structs that contain information about a token
+that depends only on its surface form, without context. `Lexeme`s store much of
+the data associated with `Token`s. As a side effect of this the `Vocab` also
+manages the `StringStore` for a pipeline and a grab-bag of other data.
+
+These are things stored in the vocab:
+
+- `Lexeme`s
+- `StringStore`
+- `Morphology`: manages info used in `MorphAnalysis` objects
+- `vectors`: basically a dict for word vectors
+- `lookups`: language specific data like lemmas
+- `writing_system`: language specific metadata
+- `get_noun_chunks`: a syntax iterator
+- lex attribute getters: functions like `is_punct`, set in language defaults
+- `cfg`: **not** the pipeline config, this is mostly unused
+- `_unused_object`: Formerly an unused object, kept around until v4 for compatability
+
+Some of these, like the Morphology and Vectors, are complex enough that they
+need their own explanations. Here we'll just look at Vocab-specific items.
+
+### Lexemes
+
+A `Lexeme` is a type that mainly wraps a `LexemeC`, a struct consisting of ints
+that identify various context-free token attributes. Lexemes are the core data
+of the `Vocab`, and can be accessed using `__getitem__` on the `Vocab`. The memory
+for storing `LexemeC` objects is managed by a pool that belongs to the `Vocab`.
+
+Note that `__getitem__` on the `Vocab` works much like the `StringStore`, in
+that it accepts a hash or id, with one important difference: if you do a lookup
+using a string, that value is added to the `StringStore` automatically.
+
+The attributes stored in a `LexemeC` are:
+
+- orth (the raw text)
+- lower
+- norm
+- shape
+- prefix
+- suffix
+
+Most of these are straightforward. All of them can be customized, and (except
+`orth`) probably should be since the defaults are based on English, but in
+practice this is rarely done at present.
+
+### Lookups
+
+This is basically a dict of dicts, implemented using a `Table` for each
+sub-dict, that stores lemmas and other language-specific lookup data.
+
+A `Table` is a subclass of `OrderedDict` used for string-to-string data. It uses
+Bloom filters to speed up misses and has some extra serialization features.
+Tables are not used outside of the lookups.
+
+### Lex Attribute Getters
+
+Lexical Attribute Getters like `is_punct` are defined on a per-language basis,
+much like lookups, but take the form of functions rather than string-to-string
+dicts, so they're stored separately.
+
+### Writing System
+
+This is a dict with three attributes:
+
+- `direction`: ltr or rtl (default ltr)
+- `has_case`: bool (default `True`)
+- `has_letters`: bool (default `True`, `False` only for CJK for now)
+
+Currently these are not used much - the main use is that `direction` is used in
+visualizers, though `rtl` doesn't quite work (see
+[#4854](https://github.com/explosion/spaCy/issues/4854)). In the future they
+could be used when choosing hyperparameters for subwords, controlling word
+shape generation, and similar tasks.
+
+### Other Vocab Members
+
+The Vocab is kind of the default place to store things from `Language.defaults`
+that don't belong to the Tokenizer. The following properties are in the Vocab
+just because they don't have anywhere else to go.
+
+- `get_noun_chunks`
+- `cfg`: This is a dict that just stores `oov_prob` (hardcoded to `-20`)
+- `_unused_object`: Leftover C member, should be removed in next major version
+
+
diff --git a/licenses/3rd_party_licenses.txt b/licenses/3rd_party_licenses.txt
index 7bc3d4547..d58da9c4a 100644
--- a/licenses/3rd_party_licenses.txt
+++ b/licenses/3rd_party_licenses.txt
@@ -104,3 +104,26 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+
+
+importlib_metadata
+------------------
+
+* Files: util.py
+
+The implementation of packages_distributions() is adapted from
+importlib_metadata, which is distributed under the following license:
+
+Copyright 2017-2019 Jason R. Coombs, Barry Warsaw
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
diff --git a/pyproject.toml b/pyproject.toml
index 6d2dd2030..f81484d43 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -5,7 +5,7 @@ requires = [
"cymem>=2.0.2,<2.1.0",
"preshed>=3.0.2,<3.1.0",
"murmurhash>=0.28.0,<1.1.0",
- "thinc>=8.0.7,<8.1.0",
+ "thinc>=8.0.12,<8.1.0",
"blis>=0.4.0,<0.8.0",
"pathy",
"numpy>=1.15.0",
diff --git a/requirements.txt b/requirements.txt
index b626c691a..8d7372cfe 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,15 +1,16 @@
# Our libraries
-spacy-legacy>=3.0.7,<3.1.0
+spacy-legacy>=3.0.8,<3.1.0
+spacy-loggers>=1.0.0,<2.0.0
cymem>=2.0.2,<2.1.0
preshed>=3.0.2,<3.1.0
-thinc>=8.0.7,<8.1.0
+thinc>=8.0.12,<8.1.0
blis>=0.4.0,<0.8.0
ml_datasets>=0.2.0,<0.3.0
murmurhash>=0.28.0,<1.1.0
wasabi>=0.8.1,<1.1.0
srsly>=2.4.1,<3.0.0
-catalogue>=2.0.4,<2.1.0
-typer>=0.3.0,<0.4.0
+catalogue>=2.0.6,<2.1.0
+typer>=0.3.0,<0.5.0
pathy>=0.3.5
# Third party dependencies
numpy>=1.15.0
@@ -17,6 +18,7 @@ requests>=2.13.0,<3.0.0
tqdm>=4.38.0,<5.0.0
pydantic>=1.7.4,!=1.8,!=1.8.1,<1.9.0
jinja2
+langcodes>=3.2.0,<4.0.0
# Official Python utilities
setuptools
packaging>=20.0
@@ -29,3 +31,7 @@ pytest-timeout>=1.3.0,<2.0.0
mock>=2.0.0,<3.0.0
flake8>=3.8.0,<3.10.0
hypothesis>=3.27.0,<7.0.0
+mypy==0.910
+types-dataclasses>=0.1.3; python_version < "3.7"
+types-mock>=0.1.1
+types-requests
diff --git a/setup.cfg b/setup.cfg
index afc4c4ed1..586a044ff 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -21,6 +21,7 @@ classifiers =
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
+ Programming Language :: Python :: 3.10
Topic :: Scientific/Engineering
project_urls =
Release notes = https://github.com/explosion/spaCy/releases
@@ -37,19 +38,20 @@ setup_requires =
cymem>=2.0.2,<2.1.0
preshed>=3.0.2,<3.1.0
murmurhash>=0.28.0,<1.1.0
- thinc>=8.0.7,<8.1.0
+ thinc>=8.0.12,<8.1.0
install_requires =
# Our libraries
- spacy-legacy>=3.0.7,<3.1.0
+ spacy-legacy>=3.0.8,<3.1.0
+ spacy-loggers>=1.0.0,<2.0.0
murmurhash>=0.28.0,<1.1.0
cymem>=2.0.2,<2.1.0
preshed>=3.0.2,<3.1.0
- thinc>=8.0.7,<8.1.0
+ thinc>=8.0.12,<8.1.0
blis>=0.4.0,<0.8.0
wasabi>=0.8.1,<1.1.0
srsly>=2.4.1,<3.0.0
- catalogue>=2.0.4,<2.1.0
- typer>=0.3.0,<0.4.0
+ catalogue>=2.0.6,<2.1.0
+ typer>=0.3.0,<0.5.0
pathy>=0.3.5
# Third-party dependencies
tqdm>=4.38.0,<5.0.0
@@ -61,6 +63,7 @@ install_requires =
setuptools
packaging>=20.0
typing_extensions>=3.7.4,<4.0.0.0; python_version < "3.8"
+ langcodes>=3.2.0,<4.0.0
[options.entry_points]
console_scripts =
@@ -68,37 +71,45 @@ console_scripts =
[options.extras_require]
lookups =
- spacy_lookups_data>=1.0.2,<1.1.0
+ spacy_lookups_data>=1.0.3,<1.1.0
transformers =
- spacy_transformers>=1.0.1,<1.1.0
+ spacy_transformers>=1.1.2,<1.2.0
ray =
spacy_ray>=0.1.0,<1.0.0
cuda =
- cupy>=5.0.0b4,<10.0.0
+ cupy>=5.0.0b4,<11.0.0
cuda80 =
- cupy-cuda80>=5.0.0b4,<10.0.0
+ cupy-cuda80>=5.0.0b4,<11.0.0
cuda90 =
- cupy-cuda90>=5.0.0b4,<10.0.0
+ cupy-cuda90>=5.0.0b4,<11.0.0
cuda91 =
- cupy-cuda91>=5.0.0b4,<10.0.0
+ cupy-cuda91>=5.0.0b4,<11.0.0
cuda92 =
- cupy-cuda92>=5.0.0b4,<10.0.0
+ cupy-cuda92>=5.0.0b4,<11.0.0
cuda100 =
- cupy-cuda100>=5.0.0b4,<10.0.0
+ cupy-cuda100>=5.0.0b4,<11.0.0
cuda101 =
- cupy-cuda101>=5.0.0b4,<10.0.0
+ cupy-cuda101>=5.0.0b4,<11.0.0
cuda102 =
- cupy-cuda102>=5.0.0b4,<10.0.0
+ cupy-cuda102>=5.0.0b4,<11.0.0
cuda110 =
- cupy-cuda110>=5.0.0b4,<10.0.0
+ cupy-cuda110>=5.0.0b4,<11.0.0
cuda111 =
- cupy-cuda111>=5.0.0b4,<10.0.0
+ cupy-cuda111>=5.0.0b4,<11.0.0
cuda112 =
- cupy-cuda112>=5.0.0b4,<10.0.0
+ cupy-cuda112>=5.0.0b4,<11.0.0
+cuda113 =
+ cupy-cuda113>=5.0.0b4,<11.0.0
+cuda114 =
+ cupy-cuda114>=5.0.0b4,<11.0.0
+cuda115 =
+ cupy-cuda115>=5.0.0b4,<11.0.0
+apple =
+ thinc-apple-ops>=0.0.4,<1.0.0
# Language tokenizers with external dependencies
ja =
- sudachipy>=0.4.9
- sudachidict_core>=20200330
+ sudachipy>=0.5.2,!=0.6.1
+ sudachidict_core>=20211220
ko =
natto-py==0.9.0
th =
@@ -122,9 +133,11 @@ exclude =
[tool:pytest]
markers =
- slow
+ slow: mark a test as slow
+ issue: reference specific issue
[mypy]
ignore_missing_imports = True
no_implicit_optional = True
plugins = pydantic.mypy, thinc.mypy
+allow_redefinition = True
diff --git a/setup.py b/setup.py
index fcc124a43..03a1e01dd 100755
--- a/setup.py
+++ b/setup.py
@@ -81,6 +81,7 @@ COPY_FILES = {
ROOT / "setup.cfg": PACKAGE_ROOT / "tests" / "package",
ROOT / "pyproject.toml": PACKAGE_ROOT / "tests" / "package",
ROOT / "requirements.txt": PACKAGE_ROOT / "tests" / "package",
+ ROOT / "website" / "meta" / "universe.json": PACKAGE_ROOT / "tests" / "universe",
}
diff --git a/spacy/__init__.py b/spacy/__init__.py
index f20c32eb5..ca47edc94 100644
--- a/spacy/__init__.py
+++ b/spacy/__init__.py
@@ -5,7 +5,7 @@ import sys
# set library-specific custom warning handling before doing anything else
from .errors import setup_default_warnings
-setup_default_warnings()
+setup_default_warnings() # noqa: E402
# These are imported as part of the API
from thinc.api import prefer_gpu, require_gpu, require_cpu # noqa: F401
diff --git a/spacy/about.py b/spacy/about.py
index 499133cc0..c253d5052 100644
--- a/spacy/about.py
+++ b/spacy/about.py
@@ -1,6 +1,6 @@
# fmt: off
__title__ = "spacy"
-__version__ = "3.1.0"
+__version__ = "3.2.1"
__download_url__ = "https://github.com/explosion/spacy-models/releases/download"
__compatibility__ = "https://raw.githubusercontent.com/explosion/spacy-models/master/compatibility.json"
__projects__ = "https://github.com/explosion/projects"
diff --git a/spacy/attrs.pyx b/spacy/attrs.pyx
index 9122de17b..dc8eed7c3 100644
--- a/spacy/attrs.pyx
+++ b/spacy/attrs.pyx
@@ -1,3 +1,6 @@
+from .errors import Errors
+
+IOB_STRINGS = ("", "I", "O", "B")
IDS = {
"": NULL_ATTR,
@@ -64,7 +67,6 @@ IDS = {
"FLAG61": FLAG61,
"FLAG62": FLAG62,
"FLAG63": FLAG63,
-
"ID": ID,
"ORTH": ORTH,
"LOWER": LOWER,
@@ -72,7 +74,6 @@ IDS = {
"SHAPE": SHAPE,
"PREFIX": PREFIX,
"SUFFIX": SUFFIX,
-
"LENGTH": LENGTH,
"LEMMA": LEMMA,
"POS": POS,
@@ -87,7 +88,7 @@ IDS = {
"SPACY": SPACY,
"LANG": LANG,
"MORPH": MORPH,
- "IDX": IDX
+ "IDX": IDX,
}
@@ -109,28 +110,66 @@ def intify_attrs(stringy_attrs, strings_map=None, _do_deprecated=False):
"""
inty_attrs = {}
if _do_deprecated:
- if 'F' in stringy_attrs:
+ if "F" in stringy_attrs:
stringy_attrs["ORTH"] = stringy_attrs.pop("F")
- if 'L' in stringy_attrs:
+ if "L" in stringy_attrs:
stringy_attrs["LEMMA"] = stringy_attrs.pop("L")
- if 'pos' in stringy_attrs:
+ if "pos" in stringy_attrs:
stringy_attrs["TAG"] = stringy_attrs.pop("pos")
- if 'morph' in stringy_attrs:
- morphs = stringy_attrs.pop('morph')
- if 'number' in stringy_attrs:
- stringy_attrs.pop('number')
- if 'tenspect' in stringy_attrs:
- stringy_attrs.pop('tenspect')
+ if "morph" in stringy_attrs:
+ morphs = stringy_attrs.pop("morph")
+ if "number" in stringy_attrs:
+ stringy_attrs.pop("number")
+ if "tenspect" in stringy_attrs:
+ stringy_attrs.pop("tenspect")
morph_keys = [
- 'PunctType', 'PunctSide', 'Other', 'Degree', 'AdvType', 'Number',
- 'VerbForm', 'PronType', 'Aspect', 'Tense', 'PartType', 'Poss',
- 'Hyph', 'ConjType', 'NumType', 'Foreign', 'VerbType', 'NounType',
- 'Gender', 'Mood', 'Negative', 'Tense', 'Voice', 'Abbr',
- 'Derivation', 'Echo', 'Foreign', 'NameType', 'NounType', 'NumForm',
- 'NumValue', 'PartType', 'Polite', 'StyleVariant',
- 'PronType', 'AdjType', 'Person', 'Variant', 'AdpType',
- 'Reflex', 'Negative', 'Mood', 'Aspect', 'Case',
- 'Polarity', 'PrepCase', 'Animacy' # U20
+ "PunctType",
+ "PunctSide",
+ "Other",
+ "Degree",
+ "AdvType",
+ "Number",
+ "VerbForm",
+ "PronType",
+ "Aspect",
+ "Tense",
+ "PartType",
+ "Poss",
+ "Hyph",
+ "ConjType",
+ "NumType",
+ "Foreign",
+ "VerbType",
+ "NounType",
+ "Gender",
+ "Mood",
+ "Negative",
+ "Tense",
+ "Voice",
+ "Abbr",
+ "Derivation",
+ "Echo",
+ "Foreign",
+ "NameType",
+ "NounType",
+ "NumForm",
+ "NumValue",
+ "PartType",
+ "Polite",
+ "StyleVariant",
+ "PronType",
+ "AdjType",
+ "Person",
+ "Variant",
+ "AdpType",
+ "Reflex",
+ "Negative",
+ "Mood",
+ "Aspect",
+ "Case",
+ "Polarity",
+ "PrepCase",
+ "Animacy", # U20
]
for key in morph_keys:
if key in stringy_attrs:
@@ -142,8 +181,13 @@ def intify_attrs(stringy_attrs, strings_map=None, _do_deprecated=False):
for name, value in stringy_attrs.items():
int_key = intify_attr(name)
if int_key is not None:
- if strings_map is not None and isinstance(value, basestring):
- if hasattr(strings_map, 'add'):
+ if int_key == ENT_IOB:
+ if value in IOB_STRINGS:
+ value = IOB_STRINGS.index(value)
+ elif isinstance(value, str):
+ raise ValueError(Errors.E1025.format(value=value))
+ if strings_map is not None and isinstance(value, str):
+ if hasattr(strings_map, "add"):
value = strings_map.add(value)
else:
value = strings_map[value]
diff --git a/spacy/cli/_util.py b/spacy/cli/_util.py
index ed1e840a5..fb680d888 100644
--- a/spacy/cli/_util.py
+++ b/spacy/cli/_util.py
@@ -1,4 +1,5 @@
-from typing import Dict, Any, Union, List, Optional, Tuple, Iterable, TYPE_CHECKING
+from typing import Dict, Any, Union, List, Optional, Tuple, Iterable
+from typing import TYPE_CHECKING, overload
import sys
import shutil
from pathlib import Path
@@ -15,6 +16,7 @@ from thinc.util import has_cupy, gpu_is_available
from configparser import InterpolationError
import os
+from ..compat import Literal
from ..schemas import ProjectConfigSchema, validate
from ..util import import_file, run_command, make_tempdir, registry, logger
from ..util import is_compatible_version, SimpleFrozenDict, ENV_VARS
@@ -260,15 +262,16 @@ def get_checksum(path: Union[Path, str]) -> str:
RETURNS (str): The checksum.
"""
path = Path(path)
+ if not (path.is_file() or path.is_dir()):
+ msg.fail(f"Can't get checksum for {path}: not a file or directory", exits=1)
if path.is_file():
return hashlib.md5(Path(path).read_bytes()).hexdigest()
- if path.is_dir():
+ else:
# TODO: this is currently pretty slow
dir_checksum = hashlib.md5()
for sub_file in sorted(fp for fp in path.rglob("*") if fp.is_file()):
dir_checksum.update(sub_file.read_bytes())
return dir_checksum.hexdigest()
- msg.fail(f"Can't get checksum for {path}: not a file or directory", exits=1)
@contextmanager
@@ -397,7 +400,11 @@ def git_checkout(
run_command(cmd, capture=True)
# We need Path(name) to make sure we also support subdirectories
try:
- shutil.copytree(str(tmp_dir / Path(subpath)), str(dest))
+ source_path = tmp_dir / Path(subpath)
+ if not is_subpath_of(tmp_dir, source_path):
+ err = f"'{subpath}' is a path outside of the cloned repository."
+ msg.fail(err, repo, exits=1)
+ shutil.copytree(str(source_path), str(dest))
except FileNotFoundError:
err = f"Can't clone {subpath}. Make sure the directory exists in the repo (branch '{branch}')"
msg.fail(err, repo, exits=1)
@@ -445,8 +452,14 @@ def git_sparse_checkout(repo, subpath, dest, branch):
# And finally, we can checkout our subpath
cmd = f"git -C {tmp_dir} checkout {branch} {subpath}"
run_command(cmd, capture=True)
- # We need Path(name) to make sure we also support subdirectories
- shutil.move(str(tmp_dir / Path(subpath)), str(dest))
+
+ # Get a subdirectory of the cloned path, if appropriate
+ source_path = tmp_dir / Path(subpath)
+ if not is_subpath_of(tmp_dir, source_path):
+ err = f"'{subpath}' is a path outside of the cloned repository."
+ msg.fail(err, repo, exits=1)
+
+ shutil.move(str(source_path), str(dest))
def get_git_version(
@@ -458,12 +471,15 @@ def get_git_version(
RETURNS (Tuple[int, int]): The version as a (major, minor) tuple. Returns
(0, 0) if the version couldn't be determined.
"""
- ret = run_command("git --version", capture=True)
+ try:
+ ret = run_command("git --version", capture=True)
+ except:
+ raise RuntimeError(error)
stdout = ret.stdout.strip()
if not stdout or not stdout.startswith("git version"):
- return (0, 0)
+ return 0, 0
version = stdout[11:].strip().split(".")
- return (int(version[0]), int(version[1]))
+ return int(version[0]), int(version[1])
def _http_to_git(repo: str) -> str:
@@ -477,6 +493,29 @@ def _http_to_git(repo: str) -> str:
return repo
+def is_subpath_of(parent, child):
+ """
+ Check whether `child` is a path contained within `parent`.
+ """
+ # Based on https://stackoverflow.com/a/37095733 .
+
+ # In Python 3.9, the `Path.is_relative_to()` method will supplant this, so
+ # we can stop using crusty old os.path functions.
+ parent_realpath = os.path.realpath(parent)
+ child_realpath = os.path.realpath(child)
+ return os.path.commonpath([parent_realpath, child_realpath]) == parent_realpath
+
+
+@overload
+def string_to_list(value: str, intify: Literal[False] = ...) -> List[str]:
+ ...
+
+
+@overload
+def string_to_list(value: str, intify: Literal[True]) -> List[int]:
+ ...
+
+
def string_to_list(value: str, intify: bool = False) -> Union[List[str], List[int]]:
"""Parse a comma-separated string to a list and account for various
formatting options. Mostly used to handle CLI arguments that take a list of
@@ -487,7 +526,7 @@ def string_to_list(value: str, intify: bool = False) -> Union[List[str], List[in
RETURNS (Union[List[str], List[int]]): A list of strings or ints.
"""
if not value:
- return []
+ return [] # type: ignore[return-value]
if value.startswith("[") and value.endswith("]"):
value = value[1:-1]
result = []
@@ -499,7 +538,7 @@ def string_to_list(value: str, intify: bool = False) -> Union[List[str], List[in
p = p[1:-1]
p = p.strip()
if intify:
- p = int(p)
+ p = int(p) # type: ignore[assignment]
result.append(p)
return result
diff --git a/spacy/cli/convert.py b/spacy/cli/convert.py
index c84aa6431..04eb7078f 100644
--- a/spacy/cli/convert.py
+++ b/spacy/cli/convert.py
@@ -1,4 +1,4 @@
-from typing import Optional, Any, List, Union
+from typing import Callable, Iterable, Mapping, Optional, Any, List, Union
from enum import Enum
from pathlib import Path
from wasabi import Printer
@@ -9,7 +9,7 @@ import itertools
from ._util import app, Arg, Opt
from ..training import docs_to_json
-from ..tokens import DocBin
+from ..tokens import Doc, DocBin
from ..training.converters import iob_to_docs, conll_ner_to_docs, json_to_docs
from ..training.converters import conllu_to_docs
@@ -19,7 +19,7 @@ from ..training.converters import conllu_to_docs
# entry to this dict with the file extension mapped to the converter function
# imported from /converters.
-CONVERTERS = {
+CONVERTERS: Mapping[str, Callable[..., Iterable[Doc]]] = {
"conllubio": conllu_to_docs,
"conllu": conllu_to_docs,
"conll": conll_ner_to_docs,
@@ -66,19 +66,16 @@ def convert_cli(
DOCS: https://spacy.io/api/cli#convert
"""
- if isinstance(file_type, FileTypes):
- # We get an instance of the FileTypes from the CLI so we need its string value
- file_type = file_type.value
input_path = Path(input_path)
- output_dir = "-" if output_dir == Path("-") else output_dir
+ output_dir: Union[str, Path] = "-" if output_dir == Path("-") else output_dir
silent = output_dir == "-"
msg = Printer(no_print=silent)
- verify_cli_args(msg, input_path, output_dir, file_type, converter, ner_map)
+ verify_cli_args(msg, input_path, output_dir, file_type.value, converter, ner_map)
converter = _get_converter(msg, converter, input_path)
convert(
input_path,
output_dir,
- file_type=file_type,
+ file_type=file_type.value,
n_sents=n_sents,
seg_sents=seg_sents,
model=model,
@@ -94,7 +91,7 @@ def convert_cli(
def convert(
- input_path: Union[str, Path],
+ input_path: Path,
output_dir: Union[str, Path],
*,
file_type: str = "json",
@@ -108,13 +105,14 @@ def convert(
lang: Optional[str] = None,
concatenate: bool = False,
silent: bool = True,
- msg: Optional[Printer],
+ msg: Optional[Printer] = None,
) -> None:
+ input_path = Path(input_path)
if not msg:
msg = Printer(no_print=silent)
ner_map = srsly.read_json(ner_map) if ner_map is not None else None
doc_files = []
- for input_loc in walk_directory(Path(input_path), converter):
+ for input_loc in walk_directory(input_path, converter):
with input_loc.open("r", encoding="utf-8") as infile:
input_data = infile.read()
# Use converter function to convert data
@@ -141,7 +139,7 @@ def convert(
else:
db = DocBin(docs=docs, store_user_data=True)
len_docs = len(db)
- data = db.to_bytes()
+ data = db.to_bytes() # type: ignore[assignment]
if output_dir == "-":
_print_docs_to_stdout(data, file_type)
else:
@@ -220,13 +218,12 @@ def walk_directory(path: Path, converter: str) -> List[Path]:
def verify_cli_args(
msg: Printer,
- input_path: Union[str, Path],
+ input_path: Path,
output_dir: Union[str, Path],
- file_type: FileTypes,
+ file_type: str,
converter: str,
ner_map: Optional[Path],
):
- input_path = Path(input_path)
if file_type not in FILE_TYPES_STDOUT and output_dir == "-":
msg.fail(
f"Can't write .{file_type} data to stdout. Please specify an output directory.",
@@ -244,13 +241,13 @@ def verify_cli_args(
msg.fail("No input files in directory", input_path, exits=1)
file_types = list(set([loc.suffix[1:] for loc in input_locs]))
if converter == "auto" and len(file_types) >= 2:
- file_types = ",".join(file_types)
- msg.fail("All input files must be same type", file_types, exits=1)
+ file_types_str = ",".join(file_types)
+ msg.fail("All input files must be same type", file_types_str, exits=1)
if converter != "auto" and converter not in CONVERTERS:
msg.fail(f"Can't find converter for {converter}", exits=1)
-def _get_converter(msg, converter, input_path):
+def _get_converter(msg, converter, input_path: Path):
if input_path.is_dir():
input_path = walk_directory(input_path, converter)[0]
if converter == "auto":
diff --git a/spacy/cli/debug_config.py b/spacy/cli/debug_config.py
index 56ee12336..409fac4ed 100644
--- a/spacy/cli/debug_config.py
+++ b/spacy/cli/debug_config.py
@@ -25,7 +25,7 @@ def debug_config_cli(
show_vars: bool = Opt(False, "--show-variables", "-V", help="Show an overview of all variables referenced in the config and their values. This will also reflect variables overwritten on the CLI.")
# fmt: on
):
- """Debug a config.cfg file and show validation errors. The command will
+ """Debug a config file and show validation errors. The command will
create all objects in the tree and validate them. Note that some config
validation errors are blocking and will prevent the rest of the config from
being resolved. This means that you may not see all validation errors at
diff --git a/spacy/cli/debug_data.py b/spacy/cli/debug_data.py
index b4119abdf..ab7c20d48 100644
--- a/spacy/cli/debug_data.py
+++ b/spacy/cli/debug_data.py
@@ -1,4 +1,5 @@
-from typing import List, Sequence, Dict, Any, Tuple, Optional, Set
+from typing import Any, Dict, Iterable, List, Optional, Sequence, Set, Tuple, Union
+from typing import cast, overload
from pathlib import Path
from collections import Counter
import sys
@@ -13,10 +14,11 @@ from ..training.initialize import get_sourced_components
from ..schemas import ConfigSchemaTraining
from ..pipeline._parser_internals import nonproj
from ..pipeline._parser_internals.nonproj import DELIMITER
-from ..pipeline import Morphologizer
+from ..pipeline import Morphologizer, SpanCategorizer
from ..morphology import Morphology
from ..language import Language
from ..util import registry, resolve_dot_names
+from ..compat import Literal
from .. import util
@@ -101,13 +103,14 @@ def debug_data(
# Create the gold corpus to be able to better analyze data
dot_names = [T["train_corpus"], T["dev_corpus"]]
train_corpus, dev_corpus = resolve_dot_names(config, dot_names)
+
+ nlp.initialize(lambda: train_corpus(nlp))
+ msg.good("Pipeline can be initialized with data")
+
train_dataset = list(train_corpus(nlp))
dev_dataset = list(dev_corpus(nlp))
msg.good("Corpus is loadable")
- nlp.initialize(lambda: train_dataset)
- msg.good("Pipeline can be initialized with data")
-
# Create all gold data here to avoid iterating over the train_dataset constantly
gold_train_data = _compile_gold(train_dataset, factory_names, nlp, make_proj=True)
gold_train_unpreprocessed_data = _compile_gold(
@@ -200,7 +203,7 @@ def debug_data(
has_low_data_warning = False
has_no_neg_warning = False
has_ws_ents_error = False
- has_punct_ents_warning = False
+ has_boundary_cross_ents_warning = False
msg.divider("Named Entity Recognition")
msg.info(f"{len(model_labels)} label(s)")
@@ -227,10 +230,6 @@ def debug_data(
msg.fail(f"{gold_train_data['ws_ents']} invalid whitespace entity spans")
has_ws_ents_error = True
- if gold_train_data["punct_ents"]:
- msg.warn(f"{gold_train_data['punct_ents']} entity span(s) with punctuation")
- has_punct_ents_warning = True
-
for label in labels:
if label_counts[label] <= NEW_LABEL_THRESHOLD:
msg.warn(
@@ -244,14 +243,20 @@ def debug_data(
msg.warn(f"No examples for texts WITHOUT new label '{label}'")
has_no_neg_warning = True
+ if gold_train_data["boundary_cross_ents"]:
+ msg.warn(
+ f"{gold_train_data['boundary_cross_ents']} entity span(s) crossing sentence boundaries"
+ )
+ has_boundary_cross_ents_warning = True
+
if not has_low_data_warning:
msg.good("Good amount of examples for all labels")
if not has_no_neg_warning:
msg.good("Examples without occurrences available for all labels")
if not has_ws_ents_error:
msg.good("No entities consisting of or starting/ending with whitespace")
- if not has_punct_ents_warning:
- msg.good("No entities consisting of or starting/ending with punctuation")
+ if not has_boundary_cross_ents_warning:
+ msg.good("No entities crossing sentence boundaries")
if has_low_data_warning:
msg.text(
@@ -267,15 +272,9 @@ def debug_data(
show=verbose,
)
if has_ws_ents_error:
- msg.text(
- "As of spaCy v2.1.0, entity spans consisting of or starting/ending "
- "with whitespace characters are considered invalid."
- )
-
- if has_punct_ents_warning:
msg.text(
"Entity spans consisting of or starting/ending "
- "with punctuation can not be trained with a noise level > 0."
+ "with whitespace characters are considered invalid."
)
if "textcat" in factory_names:
@@ -377,10 +376,11 @@ def debug_data(
if "tagger" in factory_names:
msg.divider("Part-of-speech Tagging")
- labels = [label for label in gold_train_data["tags"]]
+ label_list = [label for label in gold_train_data["tags"]]
model_labels = _get_labels_from_model(nlp, "tagger")
- msg.info(f"{len(labels)} label(s) in train data")
- missing_labels = model_labels - set(labels)
+ msg.info(f"{len(label_list)} label(s) in train data")
+ labels = set(label_list)
+ missing_labels = model_labels - labels
if missing_labels:
msg.warn(
"Some model labels are not present in the train data. The "
@@ -394,10 +394,11 @@ def debug_data(
if "morphologizer" in factory_names:
msg.divider("Morphologizer (POS+Morph)")
- labels = [label for label in gold_train_data["morphs"]]
+ label_list = [label for label in gold_train_data["morphs"]]
model_labels = _get_labels_from_model(nlp, "morphologizer")
- msg.info(f"{len(labels)} label(s) in train data")
- missing_labels = model_labels - set(labels)
+ msg.info(f"{len(label_list)} label(s) in train data")
+ labels = set(label_list)
+ missing_labels = model_labels - labels
if missing_labels:
msg.warn(
"Some model labels are not present in the train data. The "
@@ -564,7 +565,7 @@ def _compile_gold(
nlp: Language,
make_proj: bool,
) -> Dict[str, Any]:
- data = {
+ data: Dict[str, Any] = {
"ner": Counter(),
"cats": Counter(),
"tags": Counter(),
@@ -573,7 +574,7 @@ def _compile_gold(
"words": Counter(),
"roots": Counter(),
"ws_ents": 0,
- "punct_ents": 0,
+ "boundary_cross_ents": 0,
"n_words": 0,
"n_misaligned_words": 0,
"words_missing_vectors": Counter(),
@@ -608,19 +609,11 @@ def _compile_gold(
if label.startswith(("B-", "U-", "L-")) and doc[i].is_space:
# "Illegal" whitespace entity
data["ws_ents"] += 1
- if label.startswith(("B-", "U-", "L-")) and doc[i].text in [
- ".",
- "'",
- "!",
- "?",
- ",",
- ]:
- # punctuation entity: could be replaced by whitespace when training with noise,
- # so add a warning to alert the user to this unexpected side effect.
- data["punct_ents"] += 1
if label.startswith(("B-", "U-")):
combined_label = label.split("-")[1]
data["ner"][combined_label] += 1
+ if gold[i].is_sent_start and label.startswith(("I-", "L-")):
+ data["boundary_cross_ents"] += 1
elif label == "-":
data["ner"]["-"] += 1
if "textcat" in factory_names or "textcat_multilabel" in factory_names:
@@ -669,10 +662,28 @@ def _compile_gold(
return data
-def _format_labels(labels: List[Tuple[str, int]], counts: bool = False) -> str:
+@overload
+def _format_labels(labels: Iterable[str], counts: Literal[False] = False) -> str:
+ ...
+
+
+@overload
+def _format_labels(
+ labels: Iterable[Tuple[str, int]],
+ counts: Literal[True],
+) -> str:
+ ...
+
+
+def _format_labels(
+ labels: Union[Iterable[str], Iterable[Tuple[str, int]]],
+ counts: bool = False,
+) -> str:
if counts:
- return ", ".join([f"'{l}' ({c})" for l, c in labels])
- return ", ".join([f"'{l}'" for l in labels])
+ return ", ".join(
+ [f"'{l}' ({c})" for l, c in cast(Iterable[Tuple[str, int]], labels)]
+ )
+ return ", ".join([f"'{l}'" for l in cast(Iterable[str], labels)])
def _get_examples_without_label(data: Sequence[Example], label: str) -> int:
@@ -688,8 +699,30 @@ def _get_examples_without_label(data: Sequence[Example], label: str) -> int:
return count
-def _get_labels_from_model(nlp: Language, pipe_name: str) -> Set[str]:
- if pipe_name not in nlp.pipe_names:
- return set()
- pipe = nlp.get_pipe(pipe_name)
- return set(pipe.labels)
+def _get_labels_from_model(nlp: Language, factory_name: str) -> Set[str]:
+ pipe_names = [
+ pipe_name
+ for pipe_name in nlp.pipe_names
+ if nlp.get_pipe_meta(pipe_name).factory == factory_name
+ ]
+ labels: Set[str] = set()
+ for pipe_name in pipe_names:
+ pipe = nlp.get_pipe(pipe_name)
+ labels.update(pipe.labels)
+ return labels
+
+
+def _get_labels_from_spancat(nlp: Language) -> Dict[str, Set[str]]:
+ pipe_names = [
+ pipe_name
+ for pipe_name in nlp.pipe_names
+ if nlp.get_pipe_meta(pipe_name).factory == "spancat"
+ ]
+ labels: Dict[str, Set[str]] = {}
+ for pipe_name in pipe_names:
+ pipe = nlp.get_pipe(pipe_name)
+ assert isinstance(pipe, SpanCategorizer)
+ if pipe.key not in labels:
+ labels[pipe.key] = set()
+ labels[pipe.key].update(pipe.labels)
+ return labels
diff --git a/spacy/cli/evaluate.py b/spacy/cli/evaluate.py
index 378911a20..0d08d2c5e 100644
--- a/spacy/cli/evaluate.py
+++ b/spacy/cli/evaluate.py
@@ -136,7 +136,7 @@ def evaluate(
def handle_scores_per_type(
- scores: Union[Scorer, Dict[str, Any]],
+ scores: Dict[str, Any],
data: Dict[str, Any] = {},
*,
spans_key: str = "sc",
diff --git a/spacy/cli/info.py b/spacy/cli/info.py
index 8cc7018ff..e6a1cb616 100644
--- a/spacy/cli/info.py
+++ b/spacy/cli/info.py
@@ -15,7 +15,7 @@ def info_cli(
model: Optional[str] = Arg(None, help="Optional loadable spaCy pipeline"),
markdown: bool = Opt(False, "--markdown", "-md", help="Generate Markdown for GitHub issues"),
silent: bool = Opt(False, "--silent", "-s", "-S", help="Don't print anything (just return)"),
- exclude: Optional[str] = Opt("labels", "--exclude", "-e", help="Comma-separated keys to exclude from the print-out"),
+ exclude: str = Opt("labels", "--exclude", "-e", help="Comma-separated keys to exclude from the print-out"),
# fmt: on
):
"""
@@ -61,7 +61,7 @@ def info(
return raw_data
-def info_spacy() -> Dict[str, any]:
+def info_spacy() -> Dict[str, Any]:
"""Generate info about the current spaCy intallation.
RETURNS (dict): The spaCy info.
diff --git a/spacy/cli/init_config.py b/spacy/cli/init_config.py
index 55622452b..d4cd939c2 100644
--- a/spacy/cli/init_config.py
+++ b/spacy/cli/init_config.py
@@ -27,9 +27,9 @@ class Optimizations(str, Enum):
@init_cli.command("config")
def init_config_cli(
# fmt: off
- output_file: Path = Arg(..., help="File to save config.cfg to or - for stdout (will only output config and no additional logging info)", allow_dash=True),
- lang: Optional[str] = Opt("en", "--lang", "-l", help="Two-letter code of the language to use"),
- pipeline: Optional[str] = Opt("tagger,parser,ner", "--pipeline", "-p", help="Comma-separated names of trainable pipeline components to include (without 'tok2vec' or 'transformer')"),
+ output_file: Path = Arg(..., help="File to save the config to or - for stdout (will only output config and no additional logging info)", allow_dash=True),
+ lang: str = Opt("en", "--lang", "-l", help="Two-letter code of the language to use"),
+ pipeline: str = Opt("tagger,parser,ner", "--pipeline", "-p", help="Comma-separated names of trainable pipeline components to include (without 'tok2vec' or 'transformer')"),
optimize: Optimizations = Opt(Optimizations.efficiency.value, "--optimize", "-o", help="Whether to optimize for efficiency (faster inference, smaller model, lower memory consumption) or higher accuracy (potentially larger and slower model). This will impact the choice of architecture, pretrained weights and related hyperparameters."),
gpu: bool = Opt(False, "--gpu", "-G", help="Whether the model can run on GPU. This will impact the choice of architecture, pretrained weights and related hyperparameters."),
pretraining: bool = Opt(False, "--pretraining", "-pt", help="Include config for pretraining (with 'spacy pretrain')"),
@@ -37,15 +37,13 @@ def init_config_cli(
# fmt: on
):
"""
- Generate a starter config.cfg for training. Based on your requirements
+ Generate a starter config file for training. Based on your requirements
specified via the CLI arguments, this command generates a config with the
optimal settings for your use case. This includes the choice of architecture,
pretrained weights and related hyperparameters.
DOCS: https://spacy.io/api/cli#init-config
"""
- if isinstance(optimize, Optimizations): # instance of enum from the CLI
- optimize = optimize.value
pipeline = string_to_list(pipeline)
is_stdout = str(output_file) == "-"
if not is_stdout and output_file.exists() and not force_overwrite:
@@ -57,7 +55,7 @@ def init_config_cli(
config = init_config(
lang=lang,
pipeline=pipeline,
- optimize=optimize,
+ optimize=optimize.value,
gpu=gpu,
pretraining=pretraining,
silent=is_stdout,
@@ -68,15 +66,15 @@ def init_config_cli(
@init_cli.command("fill-config")
def init_fill_config_cli(
# fmt: off
- base_path: Path = Arg(..., help="Base config to fill", exists=True, dir_okay=False),
- output_file: Path = Arg("-", help="File to save config.cfg to (or - for stdout)", allow_dash=True),
+ base_path: Path = Arg(..., help="Path to base config to fill", exists=True, dir_okay=False),
+ output_file: Path = Arg("-", help="Path to output .cfg file (or - for stdout)", allow_dash=True),
pretraining: bool = Opt(False, "--pretraining", "-pt", help="Include config for pretraining (with 'spacy pretrain')"),
diff: bool = Opt(False, "--diff", "-D", help="Print a visual diff highlighting the changes"),
code_path: Optional[Path] = Opt(None, "--code-path", "--code", "-c", help="Path to Python file with additional code (registered functions) to be imported"),
# fmt: on
):
"""
- Fill partial config.cfg with default values. Will add all missing settings
+ Fill partial config file with default values. Will add all missing settings
from the default config and will create all objects, check the registered
functions for their default values and update the base config. This command
can be used with a config generated via the training quickstart widget:
@@ -175,8 +173,8 @@ def init_config(
"Pipeline": ", ".join(pipeline),
"Optimize for": optimize,
"Hardware": variables["hardware"].upper(),
- "Transformer": template_vars.transformer.get("name")
- if template_vars.use_transformer
+ "Transformer": template_vars.transformer.get("name") # type: ignore[attr-defined]
+ if template_vars.use_transformer # type: ignore[attr-defined]
else None,
}
msg.info("Generated config template specific for your use case")
diff --git a/spacy/cli/init_pipeline.py b/spacy/cli/init_pipeline.py
index 2a920cdda..d53a61b8e 100644
--- a/spacy/cli/init_pipeline.py
+++ b/spacy/cli/init_pipeline.py
@@ -20,6 +20,7 @@ def init_vectors_cli(
output_dir: Path = Arg(..., help="Pipeline output directory"),
prune: int = Opt(-1, "--prune", "-p", help="Optional number of vectors to prune to"),
truncate: int = Opt(0, "--truncate", "-t", help="Optional number of vectors to truncate to when reading in vectors file"),
+ mode: str = Opt("default", "--mode", "-m", help="Vectors mode: default or floret"),
name: Optional[str] = Opt(None, "--name", "-n", help="Optional name for the word vectors, e.g. en_core_web_lg.vectors"),
verbose: bool = Opt(False, "--verbose", "-V", "-VV", help="Display more information for debugging purposes"),
jsonl_loc: Optional[Path] = Opt(None, "--lexemes-jsonl", "-j", help="Location of JSONL-formatted attributes file", hidden=True),
@@ -34,7 +35,14 @@ def init_vectors_cli(
nlp = util.get_lang_class(lang)()
if jsonl_loc is not None:
update_lexemes(nlp, jsonl_loc)
- convert_vectors(nlp, vectors_loc, truncate=truncate, prune=prune, name=name)
+ convert_vectors(
+ nlp,
+ vectors_loc,
+ truncate=truncate,
+ prune=prune,
+ name=name,
+ mode=mode,
+ )
msg.good(f"Successfully converted {len(nlp.vocab.vectors)} vectors")
nlp.to_disk(output_dir)
msg.good(
diff --git a/spacy/cli/package.py b/spacy/cli/package.py
index 5dfe67296..f9d2a9af2 100644
--- a/spacy/cli/package.py
+++ b/spacy/cli/package.py
@@ -1,7 +1,10 @@
-from typing import Optional, Union, Any, Dict, List, Tuple
+from typing import Optional, Union, Any, Dict, List, Tuple, cast
import shutil
from pathlib import Path
from wasabi import Printer, MarkdownRenderer, get_raw_input
+from thinc.api import Config
+from collections import defaultdict
+from catalogue import RegistryError
import srsly
import sys
@@ -99,6 +102,12 @@ def package(
msg.fail("Can't load pipeline meta.json", meta_path, exits=1)
meta = srsly.read_json(meta_path)
meta = get_meta(input_dir, meta)
+ if meta["requirements"]:
+ msg.good(
+ f"Including {len(meta['requirements'])} package requirement(s) from "
+ f"meta and config",
+ ", ".join(meta["requirements"]),
+ )
if name is not None:
meta["name"] = name
if version is not None:
@@ -139,6 +148,9 @@ def package(
readme = generate_readme(meta)
create_file(readme_path, readme)
create_file(package_path / model_name_v / "README.md", readme)
+ msg.good("Generated README.md from meta.json")
+ else:
+ msg.info("Using existing README.md from pipeline directory")
imports = []
for code_path in code_paths:
imports.append(code_path.stem)
@@ -172,6 +184,64 @@ def has_wheel() -> bool:
return False
+def get_third_party_dependencies(
+ config: Config, exclude: List[str] = util.SimpleFrozenList()
+) -> List[str]:
+ """If the config includes references to registered functions that are
+ provided by third-party packages (spacy-transformers, other libraries), we
+ want to include them in meta["requirements"] so that the package specifies
+ them as dependencies and the user won't have to do it manually.
+
+ We do this by:
+ - traversing the config to check for registered function (@ keys)
+ - looking up the functions and getting their module
+ - looking up the module version and generating an appropriate version range
+
+ config (Config): The pipeline config.
+ exclude (list): List of packages to exclude (e.g. that already exist in meta).
+ RETURNS (list): The versioned requirements.
+ """
+ own_packages = ("spacy", "spacy-legacy", "spacy-nightly", "thinc", "srsly")
+ distributions = util.packages_distributions()
+ funcs = defaultdict(set)
+ # We only want to look at runtime-relevant sections, not [training] or [initialize]
+ for section in ("nlp", "components"):
+ for path, value in util.walk_dict(config[section]):
+ if path[-1].startswith("@"): # collect all function references by registry
+ funcs[path[-1][1:]].add(value)
+ for component in config.get("components", {}).values():
+ if "factory" in component:
+ funcs["factories"].add(component["factory"])
+ modules = set()
+ lang = config["nlp"]["lang"]
+ for reg_name, func_names in funcs.items():
+ for func_name in func_names:
+ # Try the lang-specific version and fall back
+ try:
+ func_info = util.registry.find(reg_name, lang + "." + func_name)
+ except RegistryError:
+ try:
+ func_info = util.registry.find(reg_name, func_name)
+ except RegistryError as regerr:
+ # lang-specific version being absent is not actually an issue
+ raise regerr from None
+ module_name = func_info.get("module") # type: ignore[attr-defined]
+ if module_name: # the code is part of a module, not a --code file
+ modules.add(func_info["module"].split(".")[0]) # type: ignore[index]
+ dependencies = []
+ for module_name in modules:
+ if module_name in distributions:
+ dist = distributions.get(module_name)
+ if dist:
+ pkg = dist[0]
+ if pkg in own_packages or pkg in exclude:
+ continue
+ version = util.get_package_version(pkg)
+ version_range = util.get_minor_version_range(version) # type: ignore[arg-type]
+ dependencies.append(f"{pkg}{version_range}")
+ return dependencies
+
+
def get_build_formats(formats: List[str]) -> Tuple[bool, bool]:
supported = ["sdist", "wheel", "none"]
for form in formats:
@@ -192,7 +262,7 @@ def create_file(file_path: Path, contents: str) -> None:
def get_meta(
model_path: Union[str, Path], existing_meta: Dict[str, Any]
) -> Dict[str, Any]:
- meta = {
+ meta: Dict[str, Any] = {
"lang": "en",
"name": "pipeline",
"version": "0.0.0",
@@ -202,9 +272,10 @@ def get_meta(
"url": "",
"license": "MIT",
}
- meta.update(existing_meta)
nlp = util.load_model_from_path(Path(model_path))
- meta["spacy_version"] = util.get_model_version_range(about.__version__)
+ meta.update(nlp.meta)
+ meta.update(existing_meta)
+ meta["spacy_version"] = util.get_minor_version_range(about.__version__)
meta["vectors"] = {
"width": nlp.vocab.vectors_length,
"vectors": len(nlp.vocab.vectors),
@@ -213,6 +284,11 @@ def get_meta(
}
if about.__title__ != "spacy":
meta["parent_package"] = about.__title__
+ meta.setdefault("requirements", [])
+ # Update the requirements with all third-party packages in the config
+ existing_reqs = [util.split_requirement(req)[0] for req in meta["requirements"]]
+ reqs = get_third_party_dependencies(nlp.config, exclude=existing_reqs)
+ meta["requirements"].extend(reqs)
return meta
@@ -258,8 +334,8 @@ def generate_readme(meta: Dict[str, Any]) -> str:
license_name = meta.get("license")
sources = _format_sources(meta.get("sources"))
description = meta.get("description")
- label_scheme = _format_label_scheme(meta.get("labels"))
- accuracy = _format_accuracy(meta.get("performance"))
+ label_scheme = _format_label_scheme(cast(Dict[str, Any], meta.get("labels")))
+ accuracy = _format_accuracy(cast(Dict[str, Any], meta.get("performance")))
table_data = [
(md.bold("Name"), md.code(name)),
(md.bold("Version"), md.code(version)),
@@ -331,7 +407,7 @@ def _format_label_scheme(data: Dict[str, Any]) -> str:
continue
col1 = md.bold(md.code(pipe))
col2 = ", ".join(
- [md.code(label.replace("|", "\\|")) for label in labels]
+ [md.code(str(label).replace("|", "\\|")) for label in labels]
) # noqa: W605
label_data.append((col1, col2))
n_labels += len(labels)
diff --git a/spacy/cli/profile.py b/spacy/cli/profile.py
index f4f0d3caf..3c282c73d 100644
--- a/spacy/cli/profile.py
+++ b/spacy/cli/profile.py
@@ -32,7 +32,7 @@ def profile_cli(
DOCS: https://spacy.io/api/cli#debug-profile
"""
- if ctx.parent.command.name == NAME: # called as top-level command
+ if ctx.parent.command.name == NAME: # type: ignore[union-attr] # called as top-level command
msg.warn(
"The profile command is now available via the 'debug profile' "
"subcommand. You can run python -m spacy debug --help for an "
@@ -42,9 +42,9 @@ def profile_cli(
def profile(model: str, inputs: Optional[Path] = None, n_texts: int = 10000) -> None:
-
if inputs is not None:
- inputs = _read_inputs(inputs, msg)
+ texts = _read_inputs(inputs, msg)
+ texts = list(itertools.islice(texts, n_texts))
if inputs is None:
try:
import ml_datasets
@@ -56,16 +56,13 @@ def profile(model: str, inputs: Optional[Path] = None, n_texts: int = 10000) ->
exits=1,
)
- n_inputs = 25000
- with msg.loading("Loading IMDB dataset via Thinc..."):
- imdb_train, _ = ml_datasets.imdb()
- inputs, _ = zip(*imdb_train)
- msg.info(f"Loaded IMDB dataset and using {n_inputs} examples")
- inputs = inputs[:n_inputs]
+ with msg.loading("Loading IMDB dataset via ml_datasets..."):
+ imdb_train, _ = ml_datasets.imdb(train_limit=n_texts, dev_limit=0)
+ texts, _ = zip(*imdb_train)
+ msg.info(f"Loaded IMDB dataset and using {n_texts} examples")
with msg.loading(f"Loading pipeline '{model}'..."):
nlp = load_model(model)
msg.good(f"Loaded pipeline '{model}'")
- texts = list(itertools.islice(inputs, n_texts))
cProfile.runctx("parse_texts(nlp, texts)", globals(), locals(), "Profile.prof")
s = pstats.Stats("Profile.prof")
msg.divider("Profile stats")
@@ -87,7 +84,7 @@ def _read_inputs(loc: Union[Path, str], msg: Printer) -> Iterator[str]:
if not input_path.exists() or not input_path.is_file():
msg.fail("Not a valid input data file", loc, exits=1)
msg.info(f"Using data from {input_path.parts[-1]}")
- file_ = input_path.open()
+ file_ = input_path.open() # type: ignore[assignment]
for line in file_:
data = srsly.json_loads(line)
text = data["text"]
diff --git a/spacy/cli/project/assets.py b/spacy/cli/project/assets.py
index b49e18608..5e0cdfdf2 100644
--- a/spacy/cli/project/assets.py
+++ b/spacy/cli/project/assets.py
@@ -1,18 +1,25 @@
-from typing import Optional
+from typing import Any, Dict, Optional
from pathlib import Path
from wasabi import msg
+import os
import re
import shutil
import requests
+import typer
from ...util import ensure_path, working_dir
from .._util import project_cli, Arg, Opt, PROJECT_FILE, load_project_config
from .._util import get_checksum, download_file, git_checkout, get_git_version
+from .._util import SimpleFrozenDict, parse_config_overrides
-@project_cli.command("assets")
+@project_cli.command(
+ "assets",
+ context_settings={"allow_extra_args": True, "ignore_unknown_options": True},
+)
def project_assets_cli(
# fmt: off
+ ctx: typer.Context, # This is only used to read additional arguments
project_dir: Path = Arg(Path.cwd(), help="Path to cloned project. Defaults to current working directory.", exists=True, file_okay=False),
sparse_checkout: bool = Opt(False, "--sparse", "-S", help="Use sparse checkout for assets provided via Git, to only check out and clone the files needed. Requires Git v22.2+.")
# fmt: on
@@ -24,16 +31,22 @@ def project_assets_cli(
DOCS: https://spacy.io/api/cli#project-assets
"""
- project_assets(project_dir, sparse_checkout=sparse_checkout)
+ overrides = parse_config_overrides(ctx.args)
+ project_assets(project_dir, overrides=overrides, sparse_checkout=sparse_checkout)
-def project_assets(project_dir: Path, *, sparse_checkout: bool = False) -> None:
+def project_assets(
+ project_dir: Path,
+ *,
+ overrides: Dict[str, Any] = SimpleFrozenDict(),
+ sparse_checkout: bool = False,
+) -> None:
"""Fetch assets for a project using DVC if possible.
project_dir (Path): Path to project directory.
"""
project_path = ensure_path(project_dir)
- config = load_project_config(project_path)
+ config = load_project_config(project_path, overrides=overrides)
assets = config.get("assets", {})
if not assets:
msg.warn(f"No assets specified in {PROJECT_FILE}", exits=0)
@@ -59,6 +72,15 @@ def project_assets(project_dir: Path, *, sparse_checkout: bool = False) -> None:
shutil.rmtree(dest)
else:
dest.unlink()
+ if "repo" not in asset["git"] or asset["git"]["repo"] is None:
+ msg.fail(
+ "A git asset must include 'repo', the repository address.", exits=1
+ )
+ if "path" not in asset["git"] or asset["git"]["path"] is None:
+ msg.fail(
+ "A git asset must include 'path' - use \"\" to get the entire repository.",
+ exits=1,
+ )
git_checkout(
asset["git"]["repo"],
asset["git"]["path"],
@@ -108,11 +130,17 @@ def fetch_asset(
the asset failed.
"""
dest_path = (project_path / dest).resolve()
- if dest_path.exists() and checksum:
+ if dest_path.exists():
# If there's already a file, check for checksum
- if checksum == get_checksum(dest_path):
- msg.good(f"Skipping download with matching checksum: {dest}")
- return dest_path
+ if checksum:
+ if checksum == get_checksum(dest_path):
+ msg.good(f"Skipping download with matching checksum: {dest}")
+ return
+ else:
+ # If there's not a checksum, make sure the file is a possibly valid size
+ if os.path.getsize(dest_path) == 0:
+ msg.warn(f"Asset exists but with size of 0 bytes, deleting: {dest}")
+ os.remove(dest_path)
# We might as well support the user here and create parent directories in
# case the asset dir isn't listed as a dir to create in the project.yml
if not dest_path.parent.exists():
@@ -129,7 +157,6 @@ def fetch_asset(
msg.good(f"Copied local asset {dest}")
else:
msg.fail(f"Download failed: {dest}", e)
- return
if checksum and checksum != get_checksum(dest_path):
msg.fail(f"Checksum doesn't match value defined in {PROJECT_FILE}: {dest}")
diff --git a/spacy/cli/project/clone.py b/spacy/cli/project/clone.py
index 72d4004f8..360ee3428 100644
--- a/spacy/cli/project/clone.py
+++ b/spacy/cli/project/clone.py
@@ -80,9 +80,9 @@ def check_clone(name: str, dest: Path, repo: str) -> None:
repo (str): URL of the repo to clone from.
"""
git_err = (
- f"Cloning spaCy project templates requires Git and the 'git' command. ",
+ f"Cloning spaCy project templates requires Git and the 'git' command. "
f"To clone a project without Git, copy the files from the '{name}' "
- f"directory in the {repo} to {dest} manually.",
+ f"directory in the {repo} to {dest} manually."
)
get_git_version(error=git_err)
if not dest:
diff --git a/spacy/cli/project/dvc.py b/spacy/cli/project/dvc.py
index 7e37712c3..83dc5efbf 100644
--- a/spacy/cli/project/dvc.py
+++ b/spacy/cli/project/dvc.py
@@ -143,8 +143,8 @@ def run_dvc_commands(
easier to pass flags like --quiet that depend on a variable or
command-line setting while avoiding lots of nested conditionals.
"""
- for command in commands:
- command = split_command(command)
+ for c in commands:
+ command = split_command(c)
dvc_command = ["dvc", *command]
# Add the flags if they are set to True
for flag, is_active in flags.items():
diff --git a/spacy/cli/project/pull.py b/spacy/cli/project/pull.py
index b88387a9f..6e3cde88c 100644
--- a/spacy/cli/project/pull.py
+++ b/spacy/cli/project/pull.py
@@ -2,7 +2,7 @@ from pathlib import Path
from wasabi import msg
from .remote_storage import RemoteStorage
from .remote_storage import get_command_hash
-from .._util import project_cli, Arg
+from .._util import project_cli, Arg, logger
from .._util import load_project_config
from .run import update_lockfile
@@ -39,11 +39,15 @@ def project_pull(project_dir: Path, remote: str, *, verbose: bool = False):
# in the list.
while commands:
for i, cmd in enumerate(list(commands)):
+ logger.debug(f"CMD: {cmd['name']}.")
deps = [project_dir / dep for dep in cmd.get("deps", [])]
if all(dep.exists() for dep in deps):
cmd_hash = get_command_hash("", "", deps, cmd["script"])
for output_path in cmd.get("outputs", []):
url = storage.pull(output_path, command_hash=cmd_hash)
+ logger.debug(
+ f"URL: {url} for {output_path} with command hash {cmd_hash}"
+ )
yield url, output_path
out_locs = [project_dir / out for out in cmd.get("outputs", [])]
@@ -53,6 +57,8 @@ def project_pull(project_dir: Path, remote: str, *, verbose: bool = False):
# we iterate over the loop again.
commands.pop(i)
break
+ else:
+ logger.debug(f"Dependency missing. Skipping {cmd['name']} outputs.")
else:
# If we didn't break the for loop, break the while loop.
break
diff --git a/spacy/cli/project/push.py b/spacy/cli/project/push.py
index 44050b716..bc779e9cd 100644
--- a/spacy/cli/project/push.py
+++ b/spacy/cli/project/push.py
@@ -3,7 +3,7 @@ from wasabi import msg
from .remote_storage import RemoteStorage
from .remote_storage import get_content_hash, get_command_hash
from .._util import load_project_config
-from .._util import project_cli, Arg
+from .._util import project_cli, Arg, logger
@project_cli.command("push")
@@ -37,12 +37,15 @@ def project_push(project_dir: Path, remote: str):
remote = config["remotes"][remote]
storage = RemoteStorage(project_dir, remote)
for cmd in config.get("commands", []):
+ logger.debug(f"CMD: cmd['name']")
deps = [project_dir / dep for dep in cmd.get("deps", [])]
if any(not dep.exists() for dep in deps):
+ logger.debug(f"Dependency missing. Skipping {cmd['name']} outputs")
continue
cmd_hash = get_command_hash(
"", "", [project_dir / dep for dep in cmd.get("deps", [])], cmd["script"]
)
+ logger.debug(f"CMD_HASH: {cmd_hash}")
for output_path in cmd.get("outputs", []):
output_loc = project_dir / output_path
if output_loc.exists() and _is_not_empty_dir(output_loc):
@@ -51,6 +54,9 @@ def project_push(project_dir: Path, remote: str):
command_hash=cmd_hash,
content_hash=get_content_hash(output_loc),
)
+ logger.debug(
+ f"URL: {url} for output {output_path} with cmd_hash {cmd_hash}"
+ )
yield output_path, url
diff --git a/spacy/cli/project/remote_storage.py b/spacy/cli/project/remote_storage.py
index 6056458e2..336a4bcb3 100644
--- a/spacy/cli/project/remote_storage.py
+++ b/spacy/cli/project/remote_storage.py
@@ -41,7 +41,7 @@ class RemoteStorage:
raise IOError(f"Cannot push {loc}: does not exist.")
url = self.make_url(path, command_hash, content_hash)
if url.exists():
- return None
+ return url
tmp: Path
with make_tempdir() as tmp:
tar_loc = tmp / self.encode_name(str(path))
@@ -131,8 +131,10 @@ def get_command_hash(
currently installed packages, whatever environment variables have been marked
as relevant, and the command.
"""
- check_commit = check_bool_env_var(ENV_VARS.PROJECT_USE_GIT_VERSION)
- spacy_v = GIT_VERSION if check_commit else get_minor_version(about.__version__)
+ if check_bool_env_var(ENV_VARS.PROJECT_USE_GIT_VERSION):
+ spacy_v = GIT_VERSION
+ else:
+ spacy_v = str(get_minor_version(about.__version__) or "")
dep_checksums = [get_checksum(dep) for dep in sorted(deps)]
hashes = [spacy_v, site_hash, env_hash] + dep_checksums
hashes.extend(cmd)
diff --git a/spacy/cli/project/run.py b/spacy/cli/project/run.py
index 5339d2a21..734803bc4 100644
--- a/spacy/cli/project/run.py
+++ b/spacy/cli/project/run.py
@@ -1,6 +1,7 @@
from typing import Optional, List, Dict, Sequence, Any, Iterable
from pathlib import Path
from wasabi import msg
+from wasabi.util import locale_escape
import sys
import srsly
import typer
@@ -57,6 +58,7 @@ def project_run(
project_dir (Path): Path to project directory.
subcommand (str): Name of command to run.
+ overrides (Dict[str, Any]): Optional config overrides.
force (bool): Force re-running, even if nothing changed.
dry (bool): Perform a dry run and don't execute commands.
capture (bool): Whether to capture the output and errors of individual commands.
@@ -68,11 +70,18 @@ def project_run(
config = load_project_config(project_dir, overrides=overrides)
commands = {cmd["name"]: cmd for cmd in config.get("commands", [])}
workflows = config.get("workflows", {})
- validate_subcommand(commands.keys(), workflows.keys(), subcommand)
+ validate_subcommand(list(commands.keys()), list(workflows.keys()), subcommand)
if subcommand in workflows:
msg.info(f"Running workflow '{subcommand}'")
for cmd in workflows[subcommand]:
- project_run(project_dir, cmd, force=force, dry=dry, capture=capture)
+ project_run(
+ project_dir,
+ cmd,
+ overrides=overrides,
+ force=force,
+ dry=dry,
+ capture=capture,
+ )
else:
cmd = commands[subcommand]
for dep in cmd.get("deps", []):
@@ -107,7 +116,7 @@ def print_run_help(project_dir: Path, subcommand: Optional[str] = None) -> None:
workflows = config.get("workflows", {})
project_loc = "" if is_cwd(project_dir) else project_dir
if subcommand:
- validate_subcommand(commands.keys(), workflows.keys(), subcommand)
+ validate_subcommand(list(commands.keys()), list(workflows.keys()), subcommand)
print(f"Usage: {COMMAND} project run {subcommand} {project_loc}")
if subcommand in commands:
help_text = commands[subcommand].get("help")
@@ -127,7 +136,7 @@ def print_run_help(project_dir: Path, subcommand: Optional[str] = None) -> None:
print("")
title = config.get("title")
if title:
- print(f"{title}\n")
+ print(f"{locale_escape(title)}\n")
if config_commands:
print(f"Available commands in {PROJECT_FILE}")
print(f"Usage: {COMMAND} project run [COMMAND] {project_loc}")
@@ -155,8 +164,8 @@ def run_commands(
when you want to turn over execution to the command, and capture=True
when you want to run the command more like a function.
"""
- for command in commands:
- command = split_command(command)
+ for c in commands:
+ command = split_command(c)
# Not sure if this is needed or a good idea. Motivation: users may often
# use commands in their config that reference "python" and we want to
# make sure that it's always executing the same Python that spaCy is
@@ -212,6 +221,9 @@ def check_rerun(
strict_version (bool):
RETURNS (bool): Whether to re-run the command.
"""
+ # Always rerun if no-skip is set
+ if command.get("no_skip", False):
+ return True
lock_path = project_dir / PROJECT_LOCK
if not lock_path.exists(): # We don't have a lockfile, run command
return True
@@ -282,7 +294,7 @@ def get_lock_entry(project_dir: Path, command: Dict[str, Any]) -> Dict[str, Any]
}
-def get_fileinfo(project_dir: Path, paths: List[str]) -> List[Dict[str, str]]:
+def get_fileinfo(project_dir: Path, paths: List[str]) -> List[Dict[str, Optional[str]]]:
"""Generate the file information for a list of paths (dependencies, outputs).
Includes the file path and the file's checksum.
diff --git a/spacy/cli/templates/quickstart_training.jinja b/spacy/cli/templates/quickstart_training.jinja
index 339fb1e96..b78806fec 100644
--- a/spacy/cli/templates/quickstart_training.jinja
+++ b/spacy/cli/templates/quickstart_training.jinja
@@ -16,7 +16,10 @@ gpu_allocator = null
[nlp]
lang = "{{ lang }}"
-{%- if "tagger" in components or "morphologizer" in components or "parser" in components or "ner" in components or "entity_linker" in components or (("textcat" in components or "textcat_multilabel" in components) and optimize == "accuracy") -%}
+{%- set has_textcat = ("textcat" in components or "textcat_multilabel" in components) -%}
+{%- set with_accuracy = optimize == "accuracy" -%}
+{%- set has_accurate_textcat = has_textcat and with_accuracy -%}
+{%- if ("tagger" in components or "morphologizer" in components or "parser" in components or "ner" in components or "entity_linker" in components or has_accurate_textcat) -%}
{%- set full_pipeline = ["transformer" if use_transformer else "tok2vec"] + components %}
{%- else -%}
{%- set full_pipeline = components %}
@@ -32,7 +35,7 @@ batch_size = {{ 128 if hardware == "gpu" else 1000 }}
factory = "transformer"
[components.transformer.model]
-@architectures = "spacy-transformers.TransformerModel.v1"
+@architectures = "spacy-transformers.TransformerModel.v3"
name = "{{ transformer["name"] }}"
tokenizer_config = {"use_fast": true}
@@ -198,7 +201,7 @@ no_output_layer = false
{# NON-TRANSFORMER PIPELINE #}
{% else -%}
-
+{% if "tok2vec" in full_pipeline -%}
[components.tok2vec]
factory = "tok2vec"
@@ -223,6 +226,7 @@ width = {{ 96 if optimize == "efficiency" else 256 }}
depth = {{ 4 if optimize == "efficiency" else 8 }}
window_size = 1
maxout_pieces = 3
+{% endif -%}
{% if "morphologizer" in components %}
[components.morphologizer]
diff --git a/spacy/cli/templates/quickstart_training_recommendations.yml b/spacy/cli/templates/quickstart_training_recommendations.yml
index dac3a26c1..a7bf9b74a 100644
--- a/spacy/cli/templates/quickstart_training_recommendations.yml
+++ b/spacy/cli/templates/quickstart_training_recommendations.yml
@@ -41,10 +41,10 @@ da:
word_vectors: da_core_news_lg
transformer:
efficiency:
- name: DJSammy/bert-base-danish-uncased_BotXO,ai
+ name: Maltehb/danish-bert-botxo
size_factor: 3
accuracy:
- name: DJSammy/bert-base-danish-uncased_BotXO,ai
+ name: Maltehb/danish-bert-botxo
size_factor: 3
de:
word_vectors: de_core_news_lg
diff --git a/spacy/cli/train.py b/spacy/cli/train.py
index 2932edd3b..cc22cbba6 100644
--- a/spacy/cli/train.py
+++ b/spacy/cli/train.py
@@ -1,4 +1,4 @@
-from typing import Optional
+from typing import Optional, Dict, Any, Union
from pathlib import Path
from wasabi import msg
import typer
@@ -7,7 +7,7 @@ import sys
from ._util import app, Arg, Opt, parse_config_overrides, show_validation_error
from ._util import import_code, setup_gpu
-from ..training.loop import train
+from ..training.loop import train as train_nlp
from ..training.initialize import init_nlp
from .. import util
@@ -40,14 +40,30 @@ def train_cli(
DOCS: https://spacy.io/api/cli#train
"""
util.logger.setLevel(logging.DEBUG if verbose else logging.INFO)
+ overrides = parse_config_overrides(ctx.args)
+ import_code(code_path)
+ train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
+
+
+def train(
+ config_path: Union[str, Path],
+ output_path: Optional[Union[str, Path]] = None,
+ *,
+ use_gpu: int = -1,
+ overrides: Dict[str, Any] = util.SimpleFrozenDict(),
+):
+ config_path = util.ensure_path(config_path)
+ output_path = util.ensure_path(output_path)
# Make sure all files and paths exists if they are needed
if not config_path or (str(config_path) != "-" and not config_path.exists()):
msg.fail("Config file not found", config_path, exits=1)
- if output_path is not None and not output_path.exists():
- output_path.mkdir(parents=True)
- msg.good(f"Created output directory: {output_path}")
- overrides = parse_config_overrides(ctx.args)
- import_code(code_path)
+ if not output_path:
+ msg.info("No output directory provided")
+ else:
+ if not output_path.exists():
+ output_path.mkdir(parents=True)
+ msg.good(f"Created output directory: {output_path}")
+ msg.info(f"Saving to output directory: {output_path}")
setup_gpu(use_gpu)
with show_validation_error(config_path):
config = util.load_config(config_path, overrides=overrides, interpolate=False)
@@ -56,4 +72,4 @@ def train_cli(
nlp = init_nlp(config, use_gpu=use_gpu)
msg.good("Initialized pipeline")
msg.divider("Training pipeline")
- train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
+ train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
diff --git a/spacy/cli/validate.py b/spacy/cli/validate.py
index a727e380e..a918e9a39 100644
--- a/spacy/cli/validate.py
+++ b/spacy/cli/validate.py
@@ -99,7 +99,7 @@ def get_model_pkgs(silent: bool = False) -> Tuple[dict, dict]:
warnings.filterwarnings("ignore", message="\\[W09[45]")
model_meta = get_model_meta(model_path)
spacy_version = model_meta.get("spacy_version", "n/a")
- is_compat = is_compatible_version(about.__version__, spacy_version)
+ is_compat = is_compatible_version(about.__version__, spacy_version) # type: ignore[assignment]
pkgs[pkg_name] = {
"name": package,
"version": version,
diff --git a/spacy/compat.py b/spacy/compat.py
index 6eca18b80..89132735d 100644
--- a/spacy/compat.py
+++ b/spacy/compat.py
@@ -5,12 +5,12 @@ from thinc.util import copy_array
try:
import cPickle as pickle
except ImportError:
- import pickle
+ import pickle # type: ignore[no-redef]
try:
import copy_reg
except ImportError:
- import copyreg as copy_reg
+ import copyreg as copy_reg # type: ignore[no-redef]
try:
from cupy.cuda.stream import Stream as CudaStream
@@ -22,10 +22,18 @@ try:
except ImportError:
cupy = None
+if sys.version_info[:2] >= (3, 8): # Python 3.8+
+ from typing import Literal, Protocol, runtime_checkable
+else:
+ from typing_extensions import Literal, Protocol, runtime_checkable # noqa: F401
+
+# Important note: The importlib_metadata "backport" includes functionality
+# that's not part of the built-in importlib.metadata. We should treat this
+# import like the built-in and only use what's available there.
try: # Python 3.8+
- from typing import Literal
+ import importlib.metadata as importlib_metadata
except ImportError:
- from typing_extensions import Literal # noqa: F401
+ from catalogue import _importlib_metadata as importlib_metadata # type: ignore[no-redef] # noqa: F401
from thinc.api import Optimizer # noqa: F401
diff --git a/spacy/default_config.cfg b/spacy/default_config.cfg
index ceb7357fc..86a72926e 100644
--- a/spacy/default_config.cfg
+++ b/spacy/default_config.cfg
@@ -68,12 +68,14 @@ seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
-# Controls early-stopping. 0 disables early stopping.
+# Controls early-stopping, i.e., the number of steps to continue without
+# improvement before stopping. 0 disables early stopping.
patience = 1600
# Number of epochs. 0 means unlimited. If >= 0, train corpus is loaded once in
# memory and shuffled within the training loop. -1 means stream train corpus
# rather than loading in memory with no shuffling within the training loop.
max_epochs = 0
+# Maximum number of update steps to train for. 0 means an unlimited number of steps.
max_steps = 20000
eval_frequency = 200
# Control how scores are printed and checkpoints are evaluated.
diff --git a/spacy/default_config_pretraining.cfg b/spacy/default_config_pretraining.cfg
index 16f767772..d70ecf04c 100644
--- a/spacy/default_config_pretraining.cfg
+++ b/spacy/default_config_pretraining.cfg
@@ -5,6 +5,7 @@ raw_text = null
max_epochs = 1000
dropout = 0.2
n_save_every = null
+n_save_epoch = null
component = "tok2vec"
layer = ""
corpus = "corpora.pretrain"
diff --git a/spacy/displacy/__init__.py b/spacy/displacy/__init__.py
index 78b83f2e5..25d530c83 100644
--- a/spacy/displacy/__init__.py
+++ b/spacy/displacy/__init__.py
@@ -18,7 +18,7 @@ RENDER_WRAPPER = None
def render(
- docs: Union[Iterable[Union[Doc, Span]], Doc, Span],
+ docs: Union[Iterable[Union[Doc, Span, dict]], Doc, Span, dict],
style: str = "dep",
page: bool = False,
minify: bool = False,
@@ -28,7 +28,8 @@ def render(
) -> str:
"""Render displaCy visualisation.
- docs (Union[Iterable[Doc], Doc]): Document(s) to visualise.
+ docs (Union[Iterable[Union[Doc, Span, dict]], Doc, Span, dict]]): Document(s) to visualise.
+ a 'dict' is only allowed here when 'manual' is set to True
style (str): Visualisation style, 'dep' or 'ent'.
page (bool): Render markup as full HTML page.
minify (bool): Minify HTML markup.
@@ -53,8 +54,8 @@ def render(
raise ValueError(Errors.E096)
renderer_func, converter = factories[style]
renderer = renderer_func(options=options)
- parsed = [converter(doc, options) for doc in docs] if not manual else docs
- _html["parsed"] = renderer.render(parsed, page=page, minify=minify).strip()
+ parsed = [converter(doc, options) for doc in docs] if not manual else docs # type: ignore
+ _html["parsed"] = renderer.render(parsed, page=page, minify=minify).strip() # type: ignore
html = _html["parsed"]
if RENDER_WRAPPER is not None:
html = RENDER_WRAPPER(html)
@@ -133,7 +134,7 @@ def parse_deps(orig_doc: Doc, options: Dict[str, Any] = {}) -> Dict[str, Any]:
"lemma": np.root.lemma_,
"ent_type": np.root.ent_type_,
}
- retokenizer.merge(np, attrs=attrs)
+ retokenizer.merge(np, attrs=attrs) # type: ignore[arg-type]
if options.get("collapse_punct", True):
spans = []
for word in doc[:-1]:
@@ -148,7 +149,7 @@ def parse_deps(orig_doc: Doc, options: Dict[str, Any] = {}) -> Dict[str, Any]:
with doc.retokenize() as retokenizer:
for span, tag, lemma, ent_type in spans:
attrs = {"tag": tag, "lemma": lemma, "ent_type": ent_type}
- retokenizer.merge(span, attrs=attrs)
+ retokenizer.merge(span, attrs=attrs) # type: ignore[arg-type]
fine_grained = options.get("fine_grained")
add_lemma = options.get("add_lemma")
words = [
@@ -180,11 +181,19 @@ def parse_deps(orig_doc: Doc, options: Dict[str, Any] = {}) -> Dict[str, Any]:
def parse_ents(doc: Doc, options: Dict[str, Any] = {}) -> Dict[str, Any]:
"""Generate named entities in [{start: i, end: i, label: 'label'}] format.
- doc (Doc): Document do parse.
+ doc (Doc): Document to parse.
+ options (Dict[str, Any]): NER-specific visualisation options.
RETURNS (dict): Generated entities keyed by text (original text) and ents.
"""
+ kb_url_template = options.get("kb_url_template", None)
ents = [
- {"start": ent.start_char, "end": ent.end_char, "label": ent.label_}
+ {
+ "start": ent.start_char,
+ "end": ent.end_char,
+ "label": ent.label_,
+ "kb_id": ent.kb_id_ if ent.kb_id_ else "",
+ "kb_url": kb_url_template.format(ent.kb_id_) if kb_url_template else "#",
+ }
for ent in doc.ents
]
if not ents:
diff --git a/spacy/displacy/render.py b/spacy/displacy/render.py
index ba56beca3..a032d843b 100644
--- a/spacy/displacy/render.py
+++ b/spacy/displacy/render.py
@@ -3,7 +3,7 @@ import uuid
from .templates import TPL_DEP_SVG, TPL_DEP_WORDS, TPL_DEP_WORDS_LEMMA, TPL_DEP_ARCS
from .templates import TPL_ENT, TPL_ENT_RTL, TPL_FIGURE, TPL_TITLE, TPL_PAGE
-from .templates import TPL_ENTS
+from .templates import TPL_ENTS, TPL_KB_LINK
from ..util import minify_html, escape_html, registry
from ..errors import Errors
@@ -18,7 +18,7 @@ DEFAULT_LABEL_COLORS = {
"LOC": "#ff9561",
"PERSON": "#aa9cfc",
"NORP": "#c887fb",
- "FACILITY": "#9cc9cc",
+ "FAC": "#9cc9cc",
"EVENT": "#ffeb80",
"LAW": "#ff8197",
"LANGUAGE": "#ff8197",
@@ -305,7 +305,7 @@ class EntityRenderer:
"""Render entities in text.
text (str): Original text.
- spans (list): Individual entity spans and their start, end and label.
+ spans (list): Individual entity spans and their start, end, label, kb_id and kb_url.
title (str / None): Document title set in Doc.user_data['title'].
"""
markup = ""
@@ -314,6 +314,9 @@ class EntityRenderer:
label = span["label"]
start = span["start"]
end = span["end"]
+ kb_id = span.get("kb_id", "")
+ kb_url = span.get("kb_url", "#")
+ kb_link = TPL_KB_LINK.format(kb_id=kb_id, kb_url=kb_url) if kb_id else ""
additional_params = span.get("params", {})
entity = escape_html(text[start:end])
fragments = text[offset:start].split("\n")
@@ -323,7 +326,12 @@ class EntityRenderer:
markup += ""
if self.ents is None or label.upper() in self.ents:
color = self.colors.get(label.upper(), self.default_color)
- ent_settings = {"label": label, "text": entity, "bg": color}
+ ent_settings = {
+ "label": label,
+ "text": entity,
+ "bg": color,
+ "kb_link": kb_link,
+ }
ent_settings.update(additional_params)
markup += self.ent_template.format(**ent_settings)
else:
diff --git a/spacy/displacy/templates.py b/spacy/displacy/templates.py
index b9cbf717b..e7d3d4266 100644
--- a/spacy/displacy/templates.py
+++ b/spacy/displacy/templates.py
@@ -51,17 +51,22 @@ TPL_ENTS = """
TPL_ENT = """
{text}
- {label}
+ {label}{kb_link}
"""
TPL_ENT_RTL = """
{text}
- {label}
+ {label}{kb_link}
"""
+# Important: this needs to start with a space!
+TPL_KB_LINK = """
+ {kb_id}
+"""
+
TPL_PAGE = """
diff --git a/spacy/errors.py b/spacy/errors.py
index 2173dd58a..390612123 100644
--- a/spacy/errors.py
+++ b/spacy/errors.py
@@ -1,18 +1,13 @@
import warnings
-def add_codes(err_cls):
- """Add error codes to string messages via class attribute names."""
-
- class ErrorsWithCodes(err_cls):
- def __getattribute__(self, code):
- msg = super(ErrorsWithCodes, self).__getattribute__(code)
- if code.startswith("__"): # python system attributes like __class__
- return msg
- else:
- return "[{code}] {msg}".format(code=code, msg=msg)
-
- return ErrorsWithCodes()
+class ErrorsWithCodes(type):
+ def __getattribute__(self, code):
+ msg = super().__getattribute__(code)
+ if code.startswith("__"): # python system attributes like __class__
+ return msg
+ else:
+ return "[{code}] {msg}".format(code=code, msg=msg)
def setup_default_warnings():
@@ -25,7 +20,10 @@ def setup_default_warnings():
filter_warning("once", error_msg=Warnings.W036.format(name=pipe))
# warn once about lemmatizer without required POS
- filter_warning("once", error_msg="[W108]")
+ filter_warning("once", error_msg=Warnings.W108)
+
+ # floret vector table cannot be modified
+ filter_warning("once", error_msg="[W114]")
def filter_warning(action: str, error_msg: str):
@@ -44,8 +42,7 @@ def _escape_warning_msg(msg):
# fmt: off
-@add_codes
-class Warnings:
+class Warnings(metaclass=ErrorsWithCodes):
W005 = ("Doc object not parsed. This means displaCy won't be able to "
"generate a dependency visualization for it. Make sure the Doc "
"was processed with a model that supports dependency parsing, and "
@@ -116,13 +113,11 @@ class Warnings:
# New warnings added in v3.x
W086 = ("Component '{listener}' will be (re)trained, but it needs the component "
- "'{name}' which is frozen. You can either freeze both, or neither "
- "of the two. If you're sourcing the component from "
- "an existing pipeline, you can use the `replace_listeners` setting in "
- "the config block to replace its token-to-vector listener with a copy "
- "and make it independent. For example, `replace_listeners = "
- "[\"model.tok2vec\"]` See the documentation for details: "
- "https://spacy.io/usage/training#config-components-listeners")
+ "'{name}' which is frozen. If you want to prevent retraining '{name}' "
+ "but want to train '{listener}' on top of it, you should add '{name}' to the "
+ "list of 'annotating_components' in the 'training' block in the config. "
+ "See the documentation for details: "
+ "https://spacy.io/usage/training#annotating-components")
W087 = ("Component '{name}' will be (re)trained, but the component '{listener}' "
"depends on it via a listener and is frozen. This means that the "
"performance of '{listener}' will be degraded. You can either freeze "
@@ -172,8 +167,8 @@ class Warnings:
"call the {matcher} on each Doc object.")
W107 = ("The property `Doc.{prop}` is deprecated. Use "
"`Doc.has_annotation(\"{attr}\")` instead.")
- W108 = ("The rule-based lemmatizer did not find POS annotation for the "
- "token '{text}'. Check that your pipeline includes components that "
+ W108 = ("The rule-based lemmatizer did not find POS annotation for one or "
+ "more tokens. Check that your pipeline includes components that "
"assign token.pos, typically 'tagger'+'attribute_ruler' or "
"'morphologizer'.")
W109 = ("Unable to save user hooks while serializing the doc. Re-add any "
@@ -192,10 +187,14 @@ class Warnings:
"vectors. This is almost certainly a mistake.")
W113 = ("Sourced component '{name}' may not work as expected: source "
"vectors are not identical to current pipeline vectors.")
+ W114 = ("Using multiprocessing with GPU models is not recommended and may "
+ "lead to errors.")
+ W115 = ("Skipping {method}: the floret vector table cannot be modified. "
+ "Vectors are calculated from character ngrams.")
+ W116 = ("Unable to clean attribute '{attr}'.")
-@add_codes
-class Errors:
+class Errors(metaclass=ErrorsWithCodes):
E001 = ("No component '{name}' found in pipeline. Available names: {opts}")
E002 = ("Can't find factory for '{name}' for language {lang} ({lang_code}). "
"This usually happens when spaCy calls `nlp.{method}` with a custom "
@@ -284,7 +283,7 @@ class Errors:
"you forget to call the `set_extension` method?")
E047 = ("Can't assign a value to unregistered extension attribute "
"'{name}'. Did you forget to call the `set_extension` method?")
- E048 = ("Can't import language {lang} from spacy.lang: {err}")
+ E048 = ("Can't import language {lang} or any matching language from spacy.lang: {err}")
E050 = ("Can't find model '{name}'. It doesn't seem to be a Python "
"package or a valid path to a data directory.")
E052 = ("Can't find model directory: {path}")
@@ -356,8 +355,8 @@ class Errors:
E098 = ("Invalid pattern: expected both RIGHT_ID and RIGHT_ATTRS.")
E099 = ("Invalid pattern: the first node of pattern should be an anchor "
"node. The node should only contain RIGHT_ID and RIGHT_ATTRS.")
- E100 = ("Nodes other than the anchor node should all contain LEFT_ID, "
- "REL_OP and RIGHT_ID.")
+ E100 = ("Nodes other than the anchor node should all contain {required}, "
+ "but these are missing: {missing}")
E101 = ("RIGHT_ID should be a new node and LEFT_ID should already have "
"have been declared in previous edges.")
E102 = ("Can't merge non-disjoint spans. '{token}' is already part of "
@@ -518,9 +517,24 @@ class Errors:
E199 = ("Unable to merge 0-length span at `doc[{start}:{end}]`.")
E200 = ("Can't yet set {attr} from Span. Vote for this feature on the "
"issue tracker: http://github.com/explosion/spaCy/issues")
- E202 = ("Unsupported alignment mode '{mode}'. Supported modes: {modes}.")
+ E202 = ("Unsupported {name} mode '{mode}'. Supported modes: {modes}.")
# New errors added in v3.x
+ E858 = ("The {mode} vector table does not support this operation. "
+ "{alternative}")
+ E859 = ("The floret vector table cannot be modified.")
+ E860 = ("Can't truncate fasttext-bloom vectors.")
+ E861 = ("No 'keys' should be provided when initializing floret vectors "
+ "with 'minn' and 'maxn'.")
+ E862 = ("'hash_count' must be between 1-4 for floret vectors.")
+ E863 = ("'maxn' must be greater than or equal to 'minn'.")
+ E864 = ("The complete vector table 'data' is required to initialize floret "
+ "vectors.")
+ E865 = ("A SpanGroup is not functional after the corresponding Doc has "
+ "been garbage collected. To keep using the spans, make sure that "
+ "the corresponding Doc object is still available in the scope of "
+ "your function.")
+ E866 = ("Expected a string or 'Doc' as input, but got: {type}.")
E867 = ("The 'textcat' component requires at least two labels because it "
"uses mutually exclusive classes where exactly one label is True "
"for each doc. For binary classification tasks, you can use two "
@@ -628,7 +642,7 @@ class Errors:
E912 = ("Failed to initialize lemmatizer. Missing lemmatizer table(s) found "
"for mode '{mode}'. Required tables: {tables}. Found: {found}.")
E913 = ("Corpus path can't be None. Maybe you forgot to define it in your "
- "config.cfg or override it on the CLI?")
+ ".cfg file or override it on the CLI?")
E914 = ("Executing {name} callback failed. Expected the function to "
"return the nlp object but got: {value}. Maybe you forgot to return "
"the modified object in your function?")
@@ -655,7 +669,9 @@ class Errors:
"{nO} - cannot add any more labels.")
E923 = ("It looks like there is no proper sample data to initialize the "
"Model of component '{name}'. To check your input data paths and "
- "annotation, run: python -m spacy debug data config.cfg")
+ "annotation, run: python -m spacy debug data config.cfg "
+ "and include the same config override values you would specify "
+ "for the 'spacy train' command.")
E924 = ("The '{name}' component does not seem to be initialized properly. "
"This is likely a bug in spaCy, so feel free to open an issue: "
"https://github.com/explosion/spaCy/issues")
@@ -790,7 +806,7 @@ class Errors:
"to token boundaries.")
E982 = ("The `Token.ent_iob` attribute should be an integer indexing "
"into {values}, but found {value}.")
- E983 = ("Invalid key for '{dict}': {key}. Available keys: "
+ E983 = ("Invalid key(s) for '{dict}': {key}. Available keys: "
"{keys}")
E984 = ("Invalid component config for '{name}': component block needs either "
"a key `factory` specifying the registered function used to "
@@ -864,7 +880,21 @@ class Errors:
E1018 = ("Knowledge base for component '{name}' is not set. "
"Make sure either `nel.initialize` or `nel.set_kb` "
"is called with a `kb_loader` function.")
-
+ E1019 = ("`noun_chunks` requires the pos tagging, which requires a "
+ "statistical model to be installed and loaded. For more info, see "
+ "the documentation:\nhttps://spacy.io/usage/models")
+ E1020 = ("No `epoch_resume` value specified and could not infer one from "
+ "filename. Specify an epoch to resume from.")
+ E1021 = ("`pos` value \"{pp}\" is not a valid Universal Dependencies tag. "
+ "Non-UD tags should use the `tag` property.")
+ E1022 = ("Words must be of type str or int, but input is of type '{wtype}'")
+ E1023 = ("Couldn't read EntityRuler from the {path}. This file doesn't "
+ "exist.")
+ E1024 = ("A pattern with ID \"{ent_id}\" is not present in EntityRuler "
+ "patterns.")
+ E1025 = ("Cannot intify the value '{value}' as an IOB string. The only "
+ "supported values are: 'I', 'O', 'B' and ''")
+
# Deprecated model shortcuts, only used in errors and warnings
OLD_MODEL_SHORTCUTS = {
diff --git a/spacy/glossary.py b/spacy/glossary.py
index 0dc075ca7..e45704fc5 100644
--- a/spacy/glossary.py
+++ b/spacy/glossary.py
@@ -95,6 +95,7 @@ GLOSSARY = {
"XX": "unknown",
"BES": 'auxiliary "be"',
"HVS": 'forms of "have"',
+ "_SP": "whitespace",
# POS Tags (German)
# TIGER Treebank
# http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/TIGERCorpus/annotation/tiger_introduction.pdf
diff --git a/spacy/kb.pyx b/spacy/kb.pyx
index d8514b54c..9a765c8e4 100644
--- a/spacy/kb.pyx
+++ b/spacy/kb.pyx
@@ -1,5 +1,5 @@
# cython: infer_types=True, profile=True
-from typing import Iterator, Iterable
+from typing import Iterator, Iterable, Callable, Dict, Any
import srsly
from cymem.cymem cimport Pool
@@ -96,6 +96,8 @@ cdef class KnowledgeBase:
def initialize_entities(self, int64_t nr_entities):
self._entry_index = PreshMap(nr_entities + 1)
self._entries = entry_vec(nr_entities + 1)
+
+ def initialize_vectors(self, int64_t nr_entities):
self._vectors_table = float_matrix(nr_entities + 1)
def initialize_aliases(self, int64_t nr_aliases):
@@ -122,7 +124,7 @@ cdef class KnowledgeBase:
def get_alias_strings(self):
return [self.vocab.strings[x] for x in self._alias_index]
- def add_entity(self, unicode entity, float freq, vector[float] entity_vector):
+ def add_entity(self, str entity, float freq, vector[float] entity_vector):
"""
Add an entity to the KB, optionally specifying its log probability based on corpus frequency
Return the hash of the entity ID/name at the end.
@@ -154,6 +156,7 @@ cdef class KnowledgeBase:
nr_entities = len(set(entity_list))
self.initialize_entities(nr_entities)
+ self.initialize_vectors(nr_entities)
i = 0
cdef KBEntryC entry
@@ -172,8 +175,8 @@ cdef class KnowledgeBase:
entry.entity_hash = entity_hash
entry.freq = freq_list[i]
- vector_index = self.c_add_vector(entity_vector=vector_list[i])
- entry.vector_index = vector_index
+ self._vectors_table[i] = entity_vector
+ entry.vector_index = i
entry.feats_row = -1 # Features table currently not implemented
@@ -182,15 +185,15 @@ cdef class KnowledgeBase:
i += 1
- def contains_entity(self, unicode entity):
+ def contains_entity(self, str entity):
cdef hash_t entity_hash = self.vocab.strings.add(entity)
return entity_hash in self._entry_index
- def contains_alias(self, unicode alias):
+ def contains_alias(self, str alias):
cdef hash_t alias_hash = self.vocab.strings.add(alias)
return alias_hash in self._alias_index
- def add_alias(self, unicode alias, entities, probabilities):
+ def add_alias(self, str alias, entities, probabilities):
"""
For a given alias, add its potential entities and prior probabilies to the KB.
Return the alias_hash at the end
@@ -236,7 +239,7 @@ cdef class KnowledgeBase:
raise RuntimeError(Errors.E891.format(alias=alias))
return alias_hash
- def append_alias(self, unicode alias, unicode entity, float prior_prob, ignore_warnings=False):
+ def append_alias(self, str alias, str entity, float prior_prob, ignore_warnings=False):
"""
For an alias already existing in the KB, extend its potential entities with one more.
Throw a warning if either the alias or the entity is unknown,
@@ -283,7 +286,7 @@ cdef class KnowledgeBase:
alias_entry.probs = probs
self._aliases_table[alias_index] = alias_entry
- def get_alias_candidates(self, unicode alias) -> Iterator[Candidate]:
+ def get_alias_candidates(self, str alias) -> Iterator[Candidate]:
"""
Return candidate entities for an alias. Each candidate defines the entity, the original alias,
and the prior probability of that alias resolving to that entity.
@@ -304,7 +307,7 @@ cdef class KnowledgeBase:
for (entry_index, prior_prob) in zip(alias_entry.entry_indices, alias_entry.probs)
if entry_index != 0]
- def get_vector(self, unicode entity):
+ def get_vector(self, str entity):
cdef hash_t entity_hash = self.vocab.strings[entity]
# Return an empty list if this entity is unknown in this KB
@@ -314,7 +317,7 @@ cdef class KnowledgeBase:
return self._vectors_table[self._entries[entry_index].vector_index]
- def get_prior_prob(self, unicode entity, unicode alias):
+ def get_prior_prob(self, str entity, str alias):
""" Return the prior probability of a given alias being linked to a given entity,
or return 0.0 when this combination is not known in the knowledge base"""
cdef hash_t alias_hash = self.vocab.strings[alias]
@@ -386,6 +389,7 @@ cdef class KnowledgeBase:
nr_aliases = header[1]
entity_vector_length = header[2]
self.initialize_entities(nr_entities)
+ self.initialize_vectors(nr_entities)
self.initialize_aliases(nr_aliases)
self.entity_vector_length = entity_vector_length
@@ -446,7 +450,7 @@ cdef class KnowledgeBase:
raise ValueError(Errors.E929.format(loc=path))
if not path.is_dir():
raise ValueError(Errors.E928.format(loc=path))
- deserialize = {}
+ deserialize: Dict[str, Callable[[Any], Any]] = {}
deserialize["contents"] = lambda p: self.read_contents(p)
deserialize["strings.json"] = lambda p: self.vocab.strings.from_disk(p)
util.from_disk(path, deserialize, exclude)
@@ -509,6 +513,7 @@ cdef class KnowledgeBase:
reader.read_header(&nr_entities, &entity_vector_length)
self.initialize_entities(nr_entities)
+ self.initialize_vectors(nr_entities)
self.entity_vector_length = entity_vector_length
# STEP 1: load entity vectors
@@ -582,7 +587,7 @@ cdef class Writer:
def __init__(self, path):
assert isinstance(path, Path)
content = bytes(path)
- cdef bytes bytes_loc = content.encode('utf8') if type(content) == unicode else content
+ cdef bytes bytes_loc = content.encode('utf8') if type(content) == str else content
self._fp = fopen(+ If you've come across a universe project that isn't working or is + incompatible with the reported spaCy version, let us know by{' '} + + opening a discussion thread + + . +
+
@@ -168,25 +181,41 @@ UniverseContent.propTypes = {
mdxComponents: PropTypes.object,
}
+const SpaCyVersion = ({ version }) => {
+ const versions = !Array.isArray(version) ? [version] : version
+ return versions.map((v, i) => (
+ <>
+