mirror of
https://github.com/explosion/spaCy.git
synced 2025-01-24 08:14:15 +03:00
Merge pull request #11572 from svlandeg/copy_v4
update v4 with latest from master and develop
This commit is contained in:
Sofie Van Landeghem
GitHub
commit
c6704f368c
2
.github/ISSUE_TEMPLATE/01_bugs.md
vendored
2
.github/ISSUE_TEMPLATE/01_bugs.md
vendored
|
|
@ -10,7 +10,7 @@ about: Use this template if you came across a bug or unexpected behaviour differ
|
|||
<!-- Include a code example or the steps that led to the problem. Please try to be as specific as possible. -->
|
||||
|
||||
## Your Environment
|
||||
<!-- Include details of your environment. If you're using spaCy 1.7+, you can also type `python -m spacy info --markdown` and copy-paste the result here.-->
|
||||
<!-- Include details of your environment. You can also type `python -m spacy info --markdown` and copy-paste the result here.-->
|
||||
* Operating System:
|
||||
* Python Version Used:
|
||||
* spaCy Version Used:
|
||||
|
|
|
|||
2
.github/azure-steps.yml
vendored
2
.github/azure-steps.yml
vendored
|
|
@ -27,7 +27,7 @@ steps:
|
|||
|
||||
- script: python -m mypy spacy
|
||||
displayName: 'Run mypy'
|
||||
condition: ne(variables['python_version'], '3.10')
|
||||
condition: ne(variables['python_version'], '3.6')
|
||||
|
||||
- task: DeleteFiles@1
|
||||
inputs:
|
||||
|
|
|
|||
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -24,6 +24,7 @@ quickstart-training-generator.js
|
|||
cythonize.json
|
||||
spacy/*.html
|
||||
*.cpp
|
||||
*.c
|
||||
*.so
|
||||
|
||||
# Vim / VSCode / editors
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ jobs:
|
|||
inputs:
|
||||
versionSpec: "3.7"
|
||||
- script: |
|
||||
pip install flake8==3.9.2
|
||||
pip install flake8==5.0.4
|
||||
python -m flake8 spacy --count --select=E901,E999,F821,F822,F823,W605 --show-source --statistics
|
||||
displayName: "flake8"
|
||||
|
||||
|
|
|
|||
82
extra/DEVELOPER_DOCS/Satellite Packages.md
Normal file
82
extra/DEVELOPER_DOCS/Satellite Packages.md
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
# spaCy Satellite Packages
|
||||
|
||||
This is a list of all the active repos relevant to spaCy besides the main one, with short descriptions, history, and current status. Archived repos will not be covered.
|
||||
|
||||
## Always Included in spaCy
|
||||
|
||||
These packages are always pulled in when you install spaCy. Most of them are direct dependencies, but some are transitive dependencies through other packages.
|
||||
|
||||
- [spacy-legacy](https://github.com/explosion/spacy-legacy): When an architecture in spaCy changes enough to get a new version, the old version is frozen and moved to spacy-legacy. This allows us to keep the core library slim while also preserving backwards compatability.
|
||||
- [thinc](https://github.com/explosion/thinc): Thinc is the machine learning library that powers trainable components in spaCy. It wraps backends like Numpy, PyTorch, and Tensorflow to provide a functional interface for specifying architectures.
|
||||
- [catalogue](https://github.com/explosion/catalogue): Small library for adding function registries, like those used for model architectures in spaCy.
|
||||
- [confection](https://github.com/explosion/confection): This library contains the functionality for config parsing that was formerly contained directly in Thinc.
|
||||
- [spacy-loggers](https://github.com/explosion/spacy-loggers): Contains loggers beyond the default logger available in spaCy's core code base. This includes loggers integrated with third-party services, which may differ in release cadence from spaCy itself.
|
||||
- [wasabi](https://github.com/explosion/wasabi): A command line formatting library, used for terminal output in spaCy.
|
||||
- [srsly](https://github.com/explosion/srsly): A wrapper that vendors several serialization libraries for spaCy. Includes parsers for JSON, JSONL, MessagePack, (extended) Pickle, and YAML.
|
||||
- [preshed](https://github.com/explosion/preshed): A Cython library for low-level data structures like hash maps, used for memory efficient data storage.
|
||||
- [cython-blis](https://github.com/explosion/cython-blis): Fast matrix multiplication using BLIS without depending on system libraries. Required by Thinc, rather than spaCy directly.
|
||||
- [murmurhash](https://github.com/explosion/murmurhash): A wrapper library for a C++ murmurhash implementation, used for string IDs in spaCy and preshed.
|
||||
- [cymem](https://github.com/explosion/cymem): A small library for RAII-style memory management in Cython.
|
||||
|
||||
## Optional Extensions for spaCy
|
||||
|
||||
These are repos that can be used by spaCy but aren't part of a default installation. Many of these are wrappers to integrate various kinds of third-party libraries.
|
||||
|
||||
- [spacy-transformers](https://github.com/explosion/spacy-transformers): A wrapper for the [HuggingFace Transformers](https://huggingface.co/docs/transformers/index) library, this handles the extensive conversion necessary to coordinate spaCy's powerful `Doc` representation, training pipeline, and the Transformer embeddings. When released, this was known as `spacy-pytorch-transformers`, but it changed to the current name when HuggingFace update the name of their library as well.
|
||||
- [spacy-huggingface-hub](https://github.com/explosion/spacy-huggingface-hub): This package has a CLI script for uploading a packaged spaCy pipeline (created with `spacy package`) to the [Hugging Face Hub](https://huggingface.co/models).
|
||||
- [spacy-alignments](https://github.com/explosion/spacy-alignments): A wrapper for the tokenizations library (mentioned below) with a modified build system to simplify cross-platform wheel creation. Used in spacy-transformers for aligning spaCy and HuggingFace tokenizations.
|
||||
- [spacy-experimental](https://github.com/explosion/spacy-experimental): Experimental components that are not quite ready for inclusion in the main spaCy library. Usually there are unresolved questions around their APIs, so the experimental library allows us to expose them to the community for feedback before fully integrating them.
|
||||
- [spacy-lookups-data](https://github.com/explosion/spacy-lookups-data): A repository of linguistic data, such as lemmas, that takes up a lot of disk space. Originally created to reduce the size of the spaCy core library. This is mainly useful if you want the data included but aren't using a pretrained pipeline; for the affected languages, the relevant data is included in pretrained pipelines directly.
|
||||
- [coreferee](https://github.com/explosion/coreferee): Coreference resolution for English, French, German and Polish, optimised for limited training data and easily extensible for further languages. Used as a spaCy pipeline component.
|
||||
- [spacy-stanza](https://github.com/explosion/spacy-stanza): This is a wrapper that allows the use of Stanford's Stanza library in spaCy.
|
||||
- [spacy-streamlit](https://github.com/explosion/spacy-streamlit): A wrapper for the Streamlit dashboard building library to help with integrating [displaCy](https://spacy.io/api/top-level/#displacy).
|
||||
- [spacymoji](https://github.com/explosion/spacymoji): A library to add extra support for emoji to spaCy, such as including character names.
|
||||
- [thinc-apple-ops](https://github.com/explosion/thinc-apple-ops): A special backend for OSX that uses Apple's native libraries for improved performance.
|
||||
- [os-signpost](https://github.com/explosion/os-signpost): A Python package that allows you to use the `OSSignposter` API in OSX for performance analysis.
|
||||
- [spacy-ray](https://github.com/explosion/spacy-ray): A wrapper to integrate spaCy with Ray, a distributed training framework. Currently a work in progress.
|
||||
|
||||
## Prodigy
|
||||
|
||||
[Prodigy](https://prodi.gy) is Explosion's easy to use and highly customizable tool for annotating data. Prodigy itself requires a license, but the repos below contain documentation, examples, and editor or notebook integrations.
|
||||
|
||||
- [prodigy-recipes](https://github.com/explosion/prodigy-recipes): Sample recipes for Prodigy, along with notebooks and other examples of usage.
|
||||
- [vscode-prodigy](https://github.com/explosion/vscode-prodigy): A VS Code extension that lets you run Prodigy inside VS Code.
|
||||
- [jupyterlab-prodigy](https://github.com/explosion/jupyterlab-prodigy): An extension for JupyterLab that lets you run Prodigy inside JupyterLab.
|
||||
|
||||
## Independent Tools or Projects
|
||||
|
||||
These are tools that may be related to or use spaCy, but are functional independent projects in their own right as well.
|
||||
|
||||
- [floret](https://github.com/explosion/floret): A modification of fastText to use Bloom Embeddings. Can be used to add vectors with subword features to spaCy, and also works independently in the same manner as fastText.
|
||||
- [sense2vec](https://github.com/explosion/sense2vec): A library to make embeddings of noun phrases or words coupled with their part of speech. This library uses spaCy.
|
||||
- [spacy-vectors-builder](https://github.com/explosion/spacy-vectors-builder): This is a spaCy project that builds vectors using floret and a lot of input text. It handles downloading the input data as well as the actual building of vectors.
|
||||
- [holmes-extractor](https://github.com/explosion/holmes-extractor): Information extraction from English and German texts based on predicate logic. Uses spaCy.
|
||||
- [healthsea](https://github.com/explosion/healthsea): Healthsea is a project to extract information from comments about health supplements. Structurally, it's a self-contained, large spaCy project.
|
||||
- [spacy-pkuseg](https://github.com/explosion/spacy-pkuseg): A fork of the pkuseg Chinese tokenizer. Used for Chinese support in spaCy, but also works independently.
|
||||
- [ml-datasets](https://github.com/explosion/ml-datasets): This repo includes loaders for several standard machine learning datasets, like MNIST or WikiNER, and has historically been used in spaCy example code and documentation.
|
||||
|
||||
## Documentation and Informational Repos
|
||||
|
||||
These repos are used to support the spaCy docs or otherwise present information about spaCy or other Explosion projects.
|
||||
|
||||
- [projects](https://github.com/explosion/projects): The projects repo is used to show detailed examples of spaCy usage. Individual projects can be checked out using the spaCy command line tool, rather than checking out the projects repo directly.
|
||||
- [spacy-course](https://github.com/explosion/spacy-course): Home to the interactive spaCy course for learning about how to use the library and some basic NLP principles.
|
||||
- [spacy-io-binder](https://github.com/explosion/spacy-io-binder): Home to the notebooks used for interactive examples in the documentation.
|
||||
|
||||
## Organizational / Meta
|
||||
|
||||
These repos are used for organizing data around spaCy, but are not something an end user would need to install as part of using the library.
|
||||
|
||||
- [spacy-models](https://github.com/explosion/spacy-models): This repo contains metadata (but not training data) for all the spaCy models. This includes information about where their training data came from, version compatability, and performance information. It also includes tests for the model packages, and the built models are hosted as releases of this repo.
|
||||
- [wheelwright](https://github.com/explosion/wheelwright): A tool for automating our PyPI builds and releases.
|
||||
- [ec2buildwheel](https://github.com/explosion/ec2buildwheel): A small project that allows you to build Python packages in the manner of cibuildwheel, but on any EC2 image. Used by wheelwright.
|
||||
|
||||
## Other
|
||||
|
||||
Repos that don't fit in any of the above categories.
|
||||
|
||||
- [blis](https://github.com/explosion/blis): A fork of the official BLIS library. The main branch is not updated, but work continues in various branches. This is used for cython-blis.
|
||||
- [tokenizations](https://github.com/explosion/tokenizations): A library originally by Yohei Tamura to align strings with tolerance to some variations in features like case and diacritics, used for aligning tokens and wordpieces. Adopted and maintained by Explosion, but usually spacy-alignments is used instead.
|
||||
- [conll-2012](https://github.com/explosion/conll-2012): A repo to hold some slightly cleaned up versions of the official scripts for the CoNLL 2012 shared task involving coreference resolution. Used in the coref project.
|
||||
- [fastapi-explosion-extras](https://github.com/explosion/fastapi-explosion-extras): Some small tweaks to FastAPI used at Explosion.
|
||||
|
||||
|
|
@ -127,3 +127,34 @@ distributed under the License is distributed on an "AS IS" BASIS,
|
|||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
|
||||
polyleven
|
||||
---------
|
||||
|
||||
* Files: spacy/matcher/polyleven.c
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2021 Fujimoto Seiji <fujimoto@ceptord.net>
|
||||
Copyright (c) 2021 Max Bachmann <kontakt@maxbachmann.de>
|
||||
Copyright (c) 2022 Nick Mazuk
|
||||
Copyright (c) 2022 Michael Weiss <code@mweiss.ch>
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
|
|||
|
|
@ -30,9 +30,10 @@ pytest-timeout>=1.3.0,<2.0.0
|
|||
mock>=2.0.0,<3.0.0
|
||||
flake8>=3.8.0,<3.10.0
|
||||
hypothesis>=3.27.0,<7.0.0
|
||||
mypy>=0.910,<0.970; platform_machine!='aarch64'
|
||||
mypy>=0.980,<0.990; platform_machine != "aarch64" and python_version >= "3.7"
|
||||
types-dataclasses>=0.1.3; python_version < "3.7"
|
||||
types-mock>=0.1.1
|
||||
types-setuptools>=57.0.0
|
||||
types-requests
|
||||
types-setuptools>=57.0.0
|
||||
black>=22.0,<23.0
|
||||
|
|
|
|||
36
setup.cfg
36
setup.cfg
|
|
@ -68,37 +68,41 @@ transformers =
|
|||
ray =
|
||||
spacy_ray>=0.1.0,<1.0.0
|
||||
cuda =
|
||||
cupy>=5.0.0b4,<11.0.0
|
||||
cupy>=5.0.0b4,<12.0.0
|
||||
cuda80 =
|
||||
cupy-cuda80>=5.0.0b4,<11.0.0
|
||||
cupy-cuda80>=5.0.0b4,<12.0.0
|
||||
cuda90 =
|
||||
cupy-cuda90>=5.0.0b4,<11.0.0
|
||||
cupy-cuda90>=5.0.0b4,<12.0.0
|
||||
cuda91 =
|
||||
cupy-cuda91>=5.0.0b4,<11.0.0
|
||||
cupy-cuda91>=5.0.0b4,<12.0.0
|
||||
cuda92 =
|
||||
cupy-cuda92>=5.0.0b4,<11.0.0
|
||||
cupy-cuda92>=5.0.0b4,<12.0.0
|
||||
cuda100 =
|
||||
cupy-cuda100>=5.0.0b4,<11.0.0
|
||||
cupy-cuda100>=5.0.0b4,<12.0.0
|
||||
cuda101 =
|
||||
cupy-cuda101>=5.0.0b4,<11.0.0
|
||||
cupy-cuda101>=5.0.0b4,<12.0.0
|
||||
cuda102 =
|
||||
cupy-cuda102>=5.0.0b4,<11.0.0
|
||||
cupy-cuda102>=5.0.0b4,<12.0.0
|
||||
cuda110 =
|
||||
cupy-cuda110>=5.0.0b4,<11.0.0
|
||||
cupy-cuda110>=5.0.0b4,<12.0.0
|
||||
cuda111 =
|
||||
cupy-cuda111>=5.0.0b4,<11.0.0
|
||||
cupy-cuda111>=5.0.0b4,<12.0.0
|
||||
cuda112 =
|
||||
cupy-cuda112>=5.0.0b4,<11.0.0
|
||||
cupy-cuda112>=5.0.0b4,<12.0.0
|
||||
cuda113 =
|
||||
cupy-cuda113>=5.0.0b4,<11.0.0
|
||||
cupy-cuda113>=5.0.0b4,<12.0.0
|
||||
cuda114 =
|
||||
cupy-cuda114>=5.0.0b4,<11.0.0
|
||||
cupy-cuda114>=5.0.0b4,<12.0.0
|
||||
cuda115 =
|
||||
cupy-cuda115>=5.0.0b4,<11.0.0
|
||||
cupy-cuda115>=5.0.0b4,<12.0.0
|
||||
cuda116 =
|
||||
cupy-cuda116>=5.0.0b4,<11.0.0
|
||||
cupy-cuda116>=5.0.0b4,<12.0.0
|
||||
cuda117 =
|
||||
cupy-cuda117>=5.0.0b4,<11.0.0
|
||||
cupy-cuda117>=5.0.0b4,<12.0.0
|
||||
cuda11x =
|
||||
cupy-cuda11x>=11.0.0,<12.0.0
|
||||
cuda-autodetect =
|
||||
cupy-wheel>=11.0.0,<12.0.0
|
||||
apple =
|
||||
thinc-apple-ops>=0.1.0.dev0,<1.0.0
|
||||
# Language tokenizers with external dependencies
|
||||
|
|
|
|||
15
setup.py
15
setup.py
|
|
@ -30,7 +30,9 @@ MOD_NAMES = [
|
|||
"spacy.lexeme",
|
||||
"spacy.vocab",
|
||||
"spacy.attrs",
|
||||
"spacy.kb",
|
||||
"spacy.kb.candidate",
|
||||
"spacy.kb.kb",
|
||||
"spacy.kb.kb_in_memory",
|
||||
"spacy.ml.parser_model",
|
||||
"spacy.morphology",
|
||||
"spacy.pipeline.dep_parser",
|
||||
|
|
@ -205,6 +207,17 @@ def setup_package():
|
|||
get_python_inc(plat_specific=True),
|
||||
]
|
||||
ext_modules = []
|
||||
ext_modules.append(
|
||||
Extension(
|
||||
"spacy.matcher.levenshtein",
|
||||
[
|
||||
"spacy/matcher/levenshtein.pyx",
|
||||
"spacy/matcher/polyleven.c",
|
||||
],
|
||||
language="c",
|
||||
include_dirs=include_dirs,
|
||||
)
|
||||
)
|
||||
for name in MOD_NAMES:
|
||||
mod_path = name.replace(".", "/") + ".pyx"
|
||||
ext = Extension(
|
||||
|
|
|
|||
|
|
@ -31,9 +31,9 @@ def load(
|
|||
name: Union[str, Path],
|
||||
*,
|
||||
vocab: Union[Vocab, bool] = True,
|
||||
disable: Union[str, Iterable[str]] = util.SimpleFrozenList(),
|
||||
enable: Union[str, Iterable[str]] = util.SimpleFrozenList(),
|
||||
exclude: Union[str, Iterable[str]] = util.SimpleFrozenList(),
|
||||
disable: Union[str, Iterable[str]] = util._DEFAULT_EMPTY_PIPES,
|
||||
enable: Union[str, Iterable[str]] = util._DEFAULT_EMPTY_PIPES,
|
||||
exclude: Union[str, Iterable[str]] = util._DEFAULT_EMPTY_PIPES,
|
||||
config: Union[Dict[str, Any], Config] = util.SimpleFrozenDict(),
|
||||
) -> Language:
|
||||
"""Load a spaCy model from an installed package or a local path.
|
||||
|
|
|
|||
|
|
@ -573,3 +573,12 @@ def setup_gpu(use_gpu: int, silent=None) -> None:
|
|||
local_msg.info("Using CPU")
|
||||
if gpu_is_available():
|
||||
local_msg.info("To switch to GPU 0, use the option: --gpu-id 0")
|
||||
|
||||
|
||||
def _format_number(number: Union[int, float], ndigits: int = 2) -> str:
|
||||
"""Formats a number (float or int) rounding to `ndigits`, without truncating trailing 0s,
|
||||
as happens with `round(number, ndigits)`"""
|
||||
if isinstance(number, float):
|
||||
return f"{number:.{ndigits}f}"
|
||||
else:
|
||||
return str(number)
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ import typer
|
|||
import math
|
||||
|
||||
from ._util import app, Arg, Opt, show_validation_error, parse_config_overrides
|
||||
from ._util import import_code, debug_cli
|
||||
from ._util import import_code, debug_cli, _format_number
|
||||
from ..training import Example, remove_bilu_prefix
|
||||
from ..training.initialize import get_sourced_components
|
||||
from ..schemas import ConfigSchemaTraining
|
||||
|
|
@ -989,7 +989,8 @@ def _get_kl_divergence(p: Counter, q: Counter) -> float:
|
|||
def _format_span_row(span_data: List[Dict], labels: List[str]) -> List[Any]:
|
||||
"""Compile into one list for easier reporting"""
|
||||
d = {
|
||||
label: [label] + list(round(d[label], 2) for d in span_data) for label in labels
|
||||
label: [label] + list(_format_number(d[label]) for d in span_data)
|
||||
for label in labels
|
||||
}
|
||||
return list(d.values())
|
||||
|
||||
|
|
@ -1004,6 +1005,10 @@ def _get_span_characteristics(
|
|||
label: _gmean(l)
|
||||
for label, l in compiled_gold["spans_length"][spans_key].items()
|
||||
}
|
||||
spans_per_type = {
|
||||
label: len(spans)
|
||||
for label, spans in compiled_gold["spans_per_type"][spans_key].items()
|
||||
}
|
||||
min_lengths = [min(l) for l in compiled_gold["spans_length"][spans_key].values()]
|
||||
max_lengths = [max(l) for l in compiled_gold["spans_length"][spans_key].values()]
|
||||
|
||||
|
|
@ -1031,6 +1036,7 @@ def _get_span_characteristics(
|
|||
return {
|
||||
"sd": span_distinctiveness,
|
||||
"bd": sb_distinctiveness,
|
||||
"spans_per_type": spans_per_type,
|
||||
"lengths": span_length,
|
||||
"min_length": min(min_lengths),
|
||||
"max_length": max(max_lengths),
|
||||
|
|
@ -1045,12 +1051,15 @@ def _get_span_characteristics(
|
|||
|
||||
def _print_span_characteristics(span_characteristics: Dict[str, Any]):
|
||||
"""Print all span characteristics into a table"""
|
||||
headers = ("Span Type", "Length", "SD", "BD")
|
||||
headers = ("Span Type", "Length", "SD", "BD", "N")
|
||||
# Wasabi has this at 30 by default, but we might have some long labels
|
||||
max_col = max(30, max(len(label) for label in span_characteristics["labels"]))
|
||||
# Prepare table data with all span characteristics
|
||||
table_data = [
|
||||
span_characteristics["lengths"],
|
||||
span_characteristics["sd"],
|
||||
span_characteristics["bd"],
|
||||
span_characteristics["spans_per_type"],
|
||||
]
|
||||
table = _format_span_row(
|
||||
span_data=table_data, labels=span_characteristics["labels"]
|
||||
|
|
@ -1061,8 +1070,18 @@ def _print_span_characteristics(span_characteristics: Dict[str, Any]):
|
|||
span_characteristics["avg_sd"],
|
||||
span_characteristics["avg_bd"],
|
||||
]
|
||||
footer = ["Wgt. Average"] + [str(round(f, 2)) for f in footer_data]
|
||||
msg.table(table, footer=footer, header=headers, divider=True)
|
||||
|
||||
footer = (
|
||||
["Wgt. Average"] + ["{:.2f}".format(round(f, 2)) for f in footer_data] + ["-"]
|
||||
)
|
||||
msg.table(
|
||||
table,
|
||||
footer=footer,
|
||||
header=headers,
|
||||
divider=True,
|
||||
aligns=["l"] + ["r"] * (len(footer_data) + 1),
|
||||
max_col=max_col,
|
||||
)
|
||||
|
||||
|
||||
def _get_spans_length_freq_dist(
|
||||
|
|
|
|||
|
|
@ -147,6 +147,7 @@ def info_installed_model_url(model: str) -> Optional[str]:
|
|||
# something else, like no file or invalid JSON
|
||||
return None
|
||||
|
||||
|
||||
def info_model_url(model: str) -> Dict[str, Any]:
|
||||
"""Return the download URL for the latest version of a pipeline."""
|
||||
version = get_latest_version(model)
|
||||
|
|
|
|||
|
|
@ -299,8 +299,8 @@ def get_meta(
|
|||
}
|
||||
nlp = util.load_model_from_path(Path(model_path))
|
||||
meta.update(nlp.meta)
|
||||
meta.update(existing_meta)
|
||||
meta["spacy_version"] = util.get_minor_version_range(about.__version__)
|
||||
meta.update(existing_meta)
|
||||
meta["vectors"] = {
|
||||
"width": nlp.vocab.vectors_length,
|
||||
"vectors": len(nlp.vocab.vectors),
|
||||
|
|
|
|||
|
|
@ -1,5 +1,8 @@
|
|||
from typing import Optional, List, Dict, Sequence, Any, Iterable
|
||||
from typing import Optional, List, Dict, Sequence, Any, Iterable, Tuple
|
||||
import os.path
|
||||
from pathlib import Path
|
||||
|
||||
import pkg_resources
|
||||
from wasabi import msg
|
||||
from wasabi.util import locale_escape
|
||||
import sys
|
||||
|
|
@ -71,6 +74,12 @@ def project_run(
|
|||
commands = {cmd["name"]: cmd for cmd in config.get("commands", [])}
|
||||
workflows = config.get("workflows", {})
|
||||
validate_subcommand(list(commands.keys()), list(workflows.keys()), subcommand)
|
||||
|
||||
req_path = project_dir / "requirements.txt"
|
||||
if config.get("check_requirements", True) and os.path.exists(req_path):
|
||||
with req_path.open() as requirements_file:
|
||||
_check_requirements([req.replace("\n", "") for req in requirements_file])
|
||||
|
||||
if subcommand in workflows:
|
||||
msg.info(f"Running workflow '{subcommand}'")
|
||||
for cmd in workflows[subcommand]:
|
||||
|
|
@ -195,6 +204,8 @@ def validate_subcommand(
|
|||
msg.fail(f"No commands or workflows defined in {PROJECT_FILE}", exits=1)
|
||||
if subcommand not in commands and subcommand not in workflows:
|
||||
help_msg = []
|
||||
if subcommand in ["assets", "asset"]:
|
||||
help_msg.append("Did you mean to run: python -m spacy project assets?")
|
||||
if commands:
|
||||
help_msg.append(f"Available commands: {', '.join(commands)}")
|
||||
if workflows:
|
||||
|
|
@ -308,3 +319,32 @@ def get_fileinfo(project_dir: Path, paths: List[str]) -> List[Dict[str, Optional
|
|||
md5 = get_checksum(file_path) if file_path.exists() else None
|
||||
data.append({"path": path, "md5": md5})
|
||||
return data
|
||||
|
||||
|
||||
def _check_requirements(requirements: List[str]) -> Tuple[bool, bool]:
|
||||
"""Checks whether requirements are installed and free of version conflicts.
|
||||
requirements (List[str]): List of requirements.
|
||||
RETURNS (Tuple[bool, bool]): Whether (1) any packages couldn't be imported, (2) any packages with version conflicts
|
||||
exist.
|
||||
"""
|
||||
|
||||
failed_pkgs_msgs: List[str] = []
|
||||
conflicting_pkgs_msgs: List[str] = []
|
||||
|
||||
for req in requirements:
|
||||
try:
|
||||
pkg_resources.require(req)
|
||||
except pkg_resources.DistributionNotFound as dnf:
|
||||
failed_pkgs_msgs.append(dnf.report())
|
||||
except pkg_resources.VersionConflict as vc:
|
||||
conflicting_pkgs_msgs.append(vc.report())
|
||||
|
||||
if len(failed_pkgs_msgs) or len(conflicting_pkgs_msgs):
|
||||
msg.warn(
|
||||
title="Missing requirements or requirement conflicts detected. Make sure your Python environment is set up "
|
||||
"correctly and you installed all requirements specified in your project's requirements.txt: "
|
||||
)
|
||||
for pgk_msg in failed_pkgs_msgs + conflicting_pkgs_msgs:
|
||||
msg.text(pgk_msg)
|
||||
|
||||
return len(failed_pkgs_msgs) > 0, len(conflicting_pkgs_msgs) > 0
|
||||
|
|
|
|||
|
|
@ -271,13 +271,8 @@ factory = "tok2vec"
|
|||
[components.tok2vec.model.embed]
|
||||
@architectures = "spacy.MultiHashEmbed.v2"
|
||||
width = ${components.tok2vec.model.encode.width}
|
||||
{% if has_letters -%}
|
||||
attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"]
|
||||
rows = [5000, 2500, 2500, 2500]
|
||||
{% else -%}
|
||||
attrs = ["ORTH", "SHAPE"]
|
||||
rows = [5000, 2500]
|
||||
{% endif -%}
|
||||
rows = [5000, 1000, 2500, 2500]
|
||||
include_static_vectors = {{ "true" if optimize == "accuracy" else "false" }}
|
||||
|
||||
[components.tok2vec.model.encode]
|
||||
|
|
|
|||
|
|
@ -271,4 +271,3 @@ zh:
|
|||
accuracy:
|
||||
name: bert-base-chinese
|
||||
size_factor: 3
|
||||
has_letters: false
|
||||
|
|
|
|||
|
|
@ -212,6 +212,8 @@ class Warnings(metaclass=ErrorsWithCodes):
|
|||
W121 = ("Attempting to trace non-existent method '{method}' in pipe '{pipe}'")
|
||||
W122 = ("Couldn't trace method '{method}' in pipe '{pipe}'. This can happen if the pipe class "
|
||||
"is a Cython extension type.")
|
||||
W123 = ("Argument {arg} with value {arg_value} is used instead of {config_value} as specified in the config. Be "
|
||||
"aware that this might affect other components in your pipeline.")
|
||||
|
||||
|
||||
class Errors(metaclass=ErrorsWithCodes):
|
||||
|
|
@ -538,6 +540,8 @@ class Errors(metaclass=ErrorsWithCodes):
|
|||
E199 = ("Unable to merge 0-length span at `doc[{start}:{end}]`.")
|
||||
E200 = ("Can't set {attr} from Span.")
|
||||
E202 = ("Unsupported {name} mode '{mode}'. Supported modes: {modes}.")
|
||||
E203 = ("If the {name} embedding layer is not updated "
|
||||
"during training, make sure to include it in 'annotating components'")
|
||||
|
||||
# New errors added in v3.x
|
||||
E853 = ("Unsupported component factory name '{name}'. The character '.' is "
|
||||
|
|
@ -707,11 +711,11 @@ class Errors(metaclass=ErrorsWithCodes):
|
|||
"need to modify the pipeline, use the built-in methods like "
|
||||
"`nlp.add_pipe`, `nlp.remove_pipe`, `nlp.disable_pipe` or "
|
||||
"`nlp.enable_pipe` instead.")
|
||||
E927 = ("Can't write to frozen list Maybe you're trying to modify a computed "
|
||||
E927 = ("Can't write to frozen list. Maybe you're trying to modify a computed "
|
||||
"property or default function argument?")
|
||||
E928 = ("A KnowledgeBase can only be serialized to/from from a directory, "
|
||||
E928 = ("An InMemoryLookupKB can only be serialized to/from from a directory, "
|
||||
"but the provided argument {loc} points to a file.")
|
||||
E929 = ("Couldn't read KnowledgeBase from {loc}. The path does not seem to exist.")
|
||||
E929 = ("Couldn't read InMemoryLookupKB from {loc}. The path does not seem to exist.")
|
||||
E930 = ("Received invalid get_examples callback in `{method}`. "
|
||||
"Expected function that returns an iterable of Example objects but "
|
||||
"got: {obj}")
|
||||
|
|
@ -937,10 +941,17 @@ class Errors(metaclass=ErrorsWithCodes):
|
|||
E1040 = ("Doc.from_json requires all tokens to have the same attributes. "
|
||||
"Some tokens do not contain annotation for: {partial_attrs}")
|
||||
E1041 = ("Expected a string, Doc, or bytes as input, but got: {type}")
|
||||
E1042 = ("Function was called with `{arg1}`={arg1_values} and "
|
||||
"`{arg2}`={arg2_values} but these arguments are conflicting.")
|
||||
E1042 = ("`enable={enable}` and `disable={disable}` are inconsistent with each other.\nIf you only passed "
|
||||
"one of `enable` or `disable`, the other argument is specified in your pipeline's configuration.\nIn that "
|
||||
"case pass an empty list for the previously not specified argument to avoid this error.")
|
||||
E1043 = ("Expected None or a value in range [{range_start}, {range_end}] for entity linker threshold, but got "
|
||||
"{value}.")
|
||||
E1044 = ("Expected `candidates_batch_size` to be >= 1, but got: {value}")
|
||||
E1045 = ("Encountered {parent} subclass without `{parent}.{method}` "
|
||||
"method in '{name}'. If you want to use this method, make "
|
||||
"sure it's overwritten on the subclass.")
|
||||
E1046 = ("{cls_name} is an abstract class and cannot be instantiated. If you are looking for spaCy's default "
|
||||
"knowledge base, use `InMemoryLookupKB`.")
|
||||
|
||||
# v4 error strings
|
||||
E4000 = ("Expected a Doc as input, but got: '{type}'")
|
||||
|
|
|
|||
3
spacy/kb/__init__.py
Normal file
3
spacy/kb/__init__.py
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
from .kb import KnowledgeBase
|
||||
from .kb_in_memory import InMemoryLookupKB
|
||||
from .candidate import Candidate, get_candidates, get_candidates_batch
|
||||
12
spacy/kb/candidate.pxd
Normal file
12
spacy/kb/candidate.pxd
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
from .kb cimport KnowledgeBase
|
||||
from libcpp.vector cimport vector
|
||||
from ..typedefs cimport hash_t
|
||||
|
||||
# Object used by the Entity Linker that summarizes one entity-alias candidate combination.
|
||||
cdef class Candidate:
|
||||
cdef readonly KnowledgeBase kb
|
||||
cdef hash_t entity_hash
|
||||
cdef float entity_freq
|
||||
cdef vector[float] entity_vector
|
||||
cdef hash_t alias_hash
|
||||
cdef float prior_prob
|
||||
74
spacy/kb/candidate.pyx
Normal file
74
spacy/kb/candidate.pyx
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# cython: infer_types=True, profile=True
|
||||
|
||||
from typing import Iterable
|
||||
from .kb cimport KnowledgeBase
|
||||
from ..tokens import Span
|
||||
|
||||
cdef class Candidate:
|
||||
"""A `Candidate` object refers to a textual mention (`alias`) that may or may not be resolved
|
||||
to a specific `entity` from a Knowledge Base. This will be used as input for the entity linking
|
||||
algorithm which will disambiguate the various candidates to the correct one.
|
||||
Each candidate (alias, entity) pair is assigned a certain prior probability.
|
||||
|
||||
DOCS: https://spacy.io/api/kb/#candidate-init
|
||||
"""
|
||||
|
||||
def __init__(self, KnowledgeBase kb, entity_hash, entity_freq, entity_vector, alias_hash, prior_prob):
|
||||
self.kb = kb
|
||||
self.entity_hash = entity_hash
|
||||
self.entity_freq = entity_freq
|
||||
self.entity_vector = entity_vector
|
||||
self.alias_hash = alias_hash
|
||||
self.prior_prob = prior_prob
|
||||
|
||||
@property
|
||||
def entity(self) -> int:
|
||||
"""RETURNS (uint64): hash of the entity's KB ID/name"""
|
||||
return self.entity_hash
|
||||
|
||||
@property
|
||||
def entity_(self) -> str:
|
||||
"""RETURNS (str): ID/name of this entity in the KB"""
|
||||
return self.kb.vocab.strings[self.entity_hash]
|
||||
|
||||
@property
|
||||
def alias(self) -> int:
|
||||
"""RETURNS (uint64): hash of the alias"""
|
||||
return self.alias_hash
|
||||
|
||||
@property
|
||||
def alias_(self) -> str:
|
||||
"""RETURNS (str): ID of the original alias"""
|
||||
return self.kb.vocab.strings[self.alias_hash]
|
||||
|
||||
@property
|
||||
def entity_freq(self) -> float:
|
||||
return self.entity_freq
|
||||
|
||||
@property
|
||||
def entity_vector(self) -> Iterable[float]:
|
||||
return self.entity_vector
|
||||
|
||||
@property
|
||||
def prior_prob(self) -> float:
|
||||
return self.prior_prob
|
||||
|
||||
|
||||
def get_candidates(kb: KnowledgeBase, mention: Span) -> Iterable[Candidate]:
|
||||
"""
|
||||
Return candidate entities for a given mention and fetching appropriate entries from the index.
|
||||
kb (KnowledgeBase): Knowledge base to query.
|
||||
mention (Span): Entity mention for which to identify candidates.
|
||||
RETURNS (Iterable[Candidate]): Identified candidates.
|
||||
"""
|
||||
return kb.get_candidates(mention)
|
||||
|
||||
|
||||
def get_candidates_batch(kb: KnowledgeBase, mentions: Iterable[Span]) -> Iterable[Iterable[Candidate]]:
|
||||
"""
|
||||
Return candidate entities for the given mentions and fetching appropriate entries from the index.
|
||||
kb (KnowledgeBase): Knowledge base to query.
|
||||
mention (Iterable[Span]): Entity mentions for which to identify candidates.
|
||||
RETURNS (Iterable[Iterable[Candidate]]): Identified candidates.
|
||||
"""
|
||||
return kb.get_candidates_batch(mentions)
|
||||
10
spacy/kb/kb.pxd
Normal file
10
spacy/kb/kb.pxd
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
"""Knowledge-base for entity or concept linking."""
|
||||
|
||||
from cymem.cymem cimport Pool
|
||||
from libc.stdint cimport int64_t
|
||||
from ..vocab cimport Vocab
|
||||
|
||||
cdef class KnowledgeBase:
|
||||
cdef Pool mem
|
||||
cdef readonly Vocab vocab
|
||||
cdef readonly int64_t entity_vector_length
|
||||
108
spacy/kb/kb.pyx
Normal file
108
spacy/kb/kb.pyx
Normal file
|
|
@ -0,0 +1,108 @@
|
|||
# cython: infer_types=True, profile=True
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Tuple, Union
|
||||
from cymem.cymem cimport Pool
|
||||
|
||||
from .candidate import Candidate
|
||||
from ..tokens import Span
|
||||
from ..util import SimpleFrozenList
|
||||
from ..errors import Errors
|
||||
|
||||
|
||||
cdef class KnowledgeBase:
|
||||
"""A `KnowledgeBase` instance stores unique identifiers for entities and their textual aliases,
|
||||
to support entity linking of named entities to real-world concepts.
|
||||
This is an abstract class and requires its operations to be implemented.
|
||||
|
||||
DOCS: https://spacy.io/api/kb
|
||||
"""
|
||||
|
||||
def __init__(self, vocab: Vocab, entity_vector_length: int):
|
||||
"""Create a KnowledgeBase."""
|
||||
# Make sure abstract KB is not instantiated.
|
||||
if self.__class__ == KnowledgeBase:
|
||||
raise TypeError(
|
||||
Errors.E1046.format(cls_name=self.__class__.__name__)
|
||||
)
|
||||
|
||||
self.vocab = vocab
|
||||
self.entity_vector_length = entity_vector_length
|
||||
self.mem = Pool()
|
||||
|
||||
def get_candidates_batch(self, mentions: Iterable[Span]) -> Iterable[Iterable[Candidate]]:
|
||||
"""
|
||||
Return candidate entities for specified texts. Each candidate defines the entity, the original alias,
|
||||
and the prior probability of that alias resolving to that entity.
|
||||
If no candidate is found for a given text, an empty list is returned.
|
||||
mentions (Iterable[Span]): Mentions for which to get candidates.
|
||||
RETURNS (Iterable[Iterable[Candidate]]): Identified candidates.
|
||||
"""
|
||||
return [self.get_candidates(span) for span in mentions]
|
||||
|
||||
def get_candidates(self, mention: Span) -> Iterable[Candidate]:
|
||||
"""
|
||||
Return candidate entities for specified text. Each candidate defines the entity, the original alias,
|
||||
and the prior probability of that alias resolving to that entity.
|
||||
If the no candidate is found for a given text, an empty list is returned.
|
||||
mention (Span): Mention for which to get candidates.
|
||||
RETURNS (Iterable[Candidate]): Identified candidates.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
Errors.E1045.format(parent="KnowledgeBase", method="get_candidates", name=self.__name__)
|
||||
)
|
||||
|
||||
def get_vectors(self, entities: Iterable[str]) -> Iterable[Iterable[float]]:
|
||||
"""
|
||||
Return vectors for entities.
|
||||
entity (str): Entity name/ID.
|
||||
RETURNS (Iterable[Iterable[float]]): Vectors for specified entities.
|
||||
"""
|
||||
return [self.get_vector(entity) for entity in entities]
|
||||
|
||||
def get_vector(self, str entity) -> Iterable[float]:
|
||||
"""
|
||||
Return vector for entity.
|
||||
entity (str): Entity name/ID.
|
||||
RETURNS (Iterable[float]): Vector for specified entity.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
Errors.E1045.format(parent="KnowledgeBase", method="get_vector", name=self.__name__)
|
||||
)
|
||||
|
||||
def to_bytes(self, **kwargs) -> bytes:
|
||||
"""Serialize the current state to a binary string.
|
||||
RETURNS (bytes): Current state as binary string.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
Errors.E1045.format(parent="KnowledgeBase", method="to_bytes", name=self.__name__)
|
||||
)
|
||||
|
||||
def from_bytes(self, bytes_data: bytes, *, exclude: Tuple[str] = tuple()):
|
||||
"""Load state from a binary string.
|
||||
bytes_data (bytes): KB state.
|
||||
exclude (Tuple[str]): Properties to exclude when restoring KB.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
Errors.E1045.format(parent="KnowledgeBase", method="from_bytes", name=self.__name__)
|
||||
)
|
||||
|
||||
def to_disk(self, path: Union[str, Path], exclude: Iterable[str] = SimpleFrozenList()) -> None:
|
||||
"""
|
||||
Write KnowledgeBase content to disk.
|
||||
path (Union[str, Path]): Target file path.
|
||||
exclude (Iterable[str]): List of components to exclude.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
Errors.E1045.format(parent="KnowledgeBase", method="to_disk", name=self.__name__)
|
||||
)
|
||||
|
||||
def from_disk(self, path: Union[str, Path], exclude: Iterable[str] = SimpleFrozenList()) -> None:
|
||||
"""
|
||||
Load KnowledgeBase content from disk.
|
||||
path (Union[str, Path]): Target file path.
|
||||
exclude (Iterable[str]): List of components to exclude.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
Errors.E1045.format(parent="KnowledgeBase", method="from_disk", name=self.__name__)
|
||||
)
|
||||
|
|
@ -1,14 +1,12 @@
|
|||
"""Knowledge-base for entity or concept linking."""
|
||||
from cymem.cymem cimport Pool
|
||||
from preshed.maps cimport PreshMap
|
||||
from libcpp.vector cimport vector
|
||||
from libc.stdint cimport int32_t, int64_t
|
||||
from libc.stdio cimport FILE
|
||||
|
||||
from .vocab cimport Vocab
|
||||
from .typedefs cimport hash_t
|
||||
from .structs cimport KBEntryC, AliasC
|
||||
|
||||
from ..typedefs cimport hash_t
|
||||
from ..structs cimport KBEntryC, AliasC
|
||||
from .kb cimport KnowledgeBase
|
||||
|
||||
ctypedef vector[KBEntryC] entry_vec
|
||||
ctypedef vector[AliasC] alias_vec
|
||||
|
|
@ -16,21 +14,7 @@ ctypedef vector[float] float_vec
|
|||
ctypedef vector[float_vec] float_matrix
|
||||
|
||||
|
||||
# Object used by the Entity Linker that summarizes one entity-alias candidate combination.
|
||||
cdef class Candidate:
|
||||
cdef readonly KnowledgeBase kb
|
||||
cdef hash_t entity_hash
|
||||
cdef float entity_freq
|
||||
cdef vector[float] entity_vector
|
||||
cdef hash_t alias_hash
|
||||
cdef float prior_prob
|
||||
|
||||
|
||||
cdef class KnowledgeBase:
|
||||
cdef Pool mem
|
||||
cdef readonly Vocab vocab
|
||||
cdef int64_t entity_vector_length
|
||||
|
||||
cdef class InMemoryLookupKB(KnowledgeBase):
|
||||
# This maps 64bit keys (hash of unique entity string)
|
||||
# to 64bit values (position of the _KBEntryC struct in the _entries vector).
|
||||
# The PreshMap is pretty space efficient, as it uses open addressing. So
|
||||
|
|
@ -1,8 +1,7 @@
|
|||
# cython: infer_types=True, profile=True
|
||||
from typing import Iterator, Iterable, Callable, Dict, Any
|
||||
from typing import Iterable, Callable, Dict, Any, Union
|
||||
|
||||
import srsly
|
||||
from cymem.cymem cimport Pool
|
||||
from preshed.maps cimport PreshMap
|
||||
from cpython.exc cimport PyErr_SetFromErrno
|
||||
from libc.stdio cimport fopen, fclose, fread, fwrite, feof, fseek
|
||||
|
|
@ -12,85 +11,28 @@ from libcpp.vector cimport vector
|
|||
from pathlib import Path
|
||||
import warnings
|
||||
|
||||
from .typedefs cimport hash_t
|
||||
from .errors import Errors, Warnings
|
||||
from . import util
|
||||
from .util import SimpleFrozenList, ensure_path
|
||||
|
||||
cdef class Candidate:
|
||||
"""A `Candidate` object refers to a textual mention (`alias`) that may or may not be resolved
|
||||
to a specific `entity` from a Knowledge Base. This will be used as input for the entity linking
|
||||
algorithm which will disambiguate the various candidates to the correct one.
|
||||
Each candidate (alias, entity) pair is assigned to a certain prior probability.
|
||||
|
||||
DOCS: https://spacy.io/api/kb/#candidate_init
|
||||
"""
|
||||
|
||||
def __init__(self, KnowledgeBase kb, entity_hash, entity_freq, entity_vector, alias_hash, prior_prob):
|
||||
self.kb = kb
|
||||
self.entity_hash = entity_hash
|
||||
self.entity_freq = entity_freq
|
||||
self.entity_vector = entity_vector
|
||||
self.alias_hash = alias_hash
|
||||
self.prior_prob = prior_prob
|
||||
|
||||
@property
|
||||
def entity(self):
|
||||
"""RETURNS (uint64): hash of the entity's KB ID/name"""
|
||||
return self.entity_hash
|
||||
|
||||
@property
|
||||
def entity_(self):
|
||||
"""RETURNS (str): ID/name of this entity in the KB"""
|
||||
return self.kb.vocab.strings[self.entity_hash]
|
||||
|
||||
@property
|
||||
def alias(self):
|
||||
"""RETURNS (uint64): hash of the alias"""
|
||||
return self.alias_hash
|
||||
|
||||
@property
|
||||
def alias_(self):
|
||||
"""RETURNS (str): ID of the original alias"""
|
||||
return self.kb.vocab.strings[self.alias_hash]
|
||||
|
||||
@property
|
||||
def entity_freq(self):
|
||||
return self.entity_freq
|
||||
|
||||
@property
|
||||
def entity_vector(self):
|
||||
return self.entity_vector
|
||||
|
||||
@property
|
||||
def prior_prob(self):
|
||||
return self.prior_prob
|
||||
from ..tokens import Span
|
||||
from ..typedefs cimport hash_t
|
||||
from ..errors import Errors, Warnings
|
||||
from .. import util
|
||||
from ..util import SimpleFrozenList, ensure_path
|
||||
from ..vocab cimport Vocab
|
||||
from .kb cimport KnowledgeBase
|
||||
from .candidate import Candidate as Candidate
|
||||
|
||||
|
||||
def get_candidates(KnowledgeBase kb, span) -> Iterator[Candidate]:
|
||||
"""
|
||||
Return candidate entities for a given span by using the text of the span as the alias
|
||||
and fetching appropriate entries from the index.
|
||||
This particular function is optimized to work with the built-in KB functionality,
|
||||
but any other custom candidate generation method can be used in combination with the KB as well.
|
||||
"""
|
||||
return kb.get_alias_candidates(span.text)
|
||||
|
||||
|
||||
cdef class KnowledgeBase:
|
||||
"""A `KnowledgeBase` instance stores unique identifiers for entities and their textual aliases,
|
||||
cdef class InMemoryLookupKB(KnowledgeBase):
|
||||
"""An `InMemoryLookupKB` instance stores unique identifiers for entities and their textual aliases,
|
||||
to support entity linking of named entities to real-world concepts.
|
||||
|
||||
DOCS: https://spacy.io/api/kb
|
||||
DOCS: https://spacy.io/api/kb_in_memory
|
||||
"""
|
||||
|
||||
def __init__(self, Vocab vocab, entity_vector_length):
|
||||
"""Create a KnowledgeBase."""
|
||||
self.mem = Pool()
|
||||
self.entity_vector_length = entity_vector_length
|
||||
"""Create an InMemoryLookupKB."""
|
||||
super().__init__(vocab, entity_vector_length)
|
||||
self._entry_index = PreshMap()
|
||||
self._alias_index = PreshMap()
|
||||
self.vocab = vocab
|
||||
self._create_empty_vectors(dummy_hash=self.vocab.strings[""])
|
||||
|
||||
def _initialize_entities(self, int64_t nr_entities):
|
||||
|
|
@ -104,11 +46,6 @@ cdef class KnowledgeBase:
|
|||
self._alias_index = PreshMap(nr_aliases + 1)
|
||||
self._aliases_table = alias_vec(nr_aliases + 1)
|
||||
|
||||
@property
|
||||
def entity_vector_length(self):
|
||||
"""RETURNS (uint64): length of the entity vectors"""
|
||||
return self.entity_vector_length
|
||||
|
||||
def __len__(self):
|
||||
return self.get_size_entities()
|
||||
|
||||
|
|
@ -286,7 +223,10 @@ cdef class KnowledgeBase:
|
|||
alias_entry.probs = probs
|
||||
self._aliases_table[alias_index] = alias_entry
|
||||
|
||||
def get_alias_candidates(self, str alias) -> Iterator[Candidate]:
|
||||
def get_candidates(self, mention: Span) -> Iterable[Candidate]:
|
||||
return self.get_alias_candidates(mention.text) # type: ignore
|
||||
|
||||
def get_alias_candidates(self, str alias) -> Iterable[Candidate]:
|
||||
"""
|
||||
Return candidate entities for an alias. Each candidate defines the entity, the original alias,
|
||||
and the prior probability of that alias resolving to that entity.
|
||||
|
|
@ -280,7 +280,7 @@ _currency = (
|
|||
_punct = (
|
||||
r"… …… , : ; \! \? ¿ ؟ ¡ \( \) \[ \] \{ \} < > _ # \* & 。 ？ ！ ， 、 ； ： ～ · । ، ۔ ؛ ٪"
|
||||
)
|
||||
_quotes = r'\' " ” “ ` ‘ ´ ’ ‚ , „ » « 「 」 『 』 （ ） 〔 〕 【 】 《 》 〈 〉'
|
||||
_quotes = r'\' " ” “ ` ‘ ´ ’ ‚ , „ » « 「 」 『 』 （ ） 〔 〕 【 】 《 》 〈 〉 〈 〉 ⟦ ⟧'
|
||||
_hyphens = "- – — -- --- —— ~"
|
||||
|
||||
# Various symbols like dingbats, but also emoji
|
||||
|
|
|
|||
|
|
@ -1,11 +1,15 @@
|
|||
from .tokenizer_exceptions import TOKENIZER_EXCEPTIONS
|
||||
from .stop_words import STOP_WORDS
|
||||
from .lex_attrs import LEX_ATTRS
|
||||
from .punctuation import TOKENIZER_PREFIXES, TOKENIZER_SUFFIXES, TOKENIZER_INFIXES
|
||||
from ...language import Language, BaseDefaults
|
||||
|
||||
|
||||
class AncientGreekDefaults(BaseDefaults):
|
||||
tokenizer_exceptions = TOKENIZER_EXCEPTIONS
|
||||
prefixes = TOKENIZER_PREFIXES
|
||||
suffixes = TOKENIZER_SUFFIXES
|
||||
infixes = TOKENIZER_INFIXES
|
||||
lex_attr_getters = LEX_ATTRS
|
||||
stop_words = STOP_WORDS
|
||||
|
||||
|
|
|
|||
46
spacy/lang/grc/punctuation.py
Normal file
46
spacy/lang/grc/punctuation.py
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
from ..char_classes import LIST_PUNCT, LIST_ELLIPSES, LIST_QUOTES, LIST_CURRENCY
|
||||
from ..char_classes import LIST_ICONS, ALPHA_LOWER, ALPHA_UPPER, ALPHA, HYPHENS
|
||||
from ..char_classes import CONCAT_QUOTES
|
||||
|
||||
_prefixes = (
|
||||
[
|
||||
"†",
|
||||
"⸏",
|
||||
]
|
||||
+ LIST_PUNCT
|
||||
+ LIST_ELLIPSES
|
||||
+ LIST_QUOTES
|
||||
+ LIST_CURRENCY
|
||||
+ LIST_ICONS
|
||||
)
|
||||
|
||||
_suffixes = (
|
||||
LIST_PUNCT
|
||||
+ LIST_ELLIPSES
|
||||
+ LIST_QUOTES
|
||||
+ LIST_ICONS
|
||||
+ [
|
||||
"†",
|
||||
"⸎",
|
||||
r"(?<=[\u1F00-\u1FFF\u0370-\u03FF])[\-\.⸏]",
|
||||
]
|
||||
)
|
||||
|
||||
_infixes = (
|
||||
LIST_ELLIPSES
|
||||
+ LIST_ICONS
|
||||
+ [
|
||||
r"(?<=[0-9])[+\-\*^](?=[0-9-])",
|
||||
r"(?<=[{al}{q}])\.(?=[{au}{q}])".format(
|
||||
al=ALPHA_LOWER, au=ALPHA_UPPER, q=CONCAT_QUOTES
|
||||
),
|
||||
r"(?<=[{a}]),(?=[{a}])".format(a=ALPHA),
|
||||
r"(?<=[{a}0-9])(?:{h})(?=[{a}])".format(a=ALPHA, h=HYPHENS),
|
||||
r"(?<=[{a}0-9])[:<>=/](?=[{a}])".format(a=ALPHA),
|
||||
r"(?<=[\u1F00-\u1FFF\u0370-\u03FF])—",
|
||||
]
|
||||
)
|
||||
|
||||
TOKENIZER_PREFIXES = _prefixes
|
||||
TOKENIZER_SUFFIXES = _suffixes
|
||||
TOKENIZER_INFIXES = _infixes
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
from typing import Iterator, Optional, Any, Dict, Callable, Iterable, Collection
|
||||
from typing import Iterator, Optional, Any, Dict, Callable, Iterable
|
||||
from typing import Union, Tuple, List, Set, Pattern, Sequence
|
||||
from typing import NoReturn, TYPE_CHECKING, TypeVar, cast, overload
|
||||
|
||||
|
|
@ -10,6 +10,7 @@ from contextlib import contextmanager
|
|||
from copy import deepcopy
|
||||
from pathlib import Path
|
||||
import warnings
|
||||
|
||||
from thinc.api import get_current_ops, Config, CupyOps, Optimizer
|
||||
import srsly
|
||||
import multiprocessing as mp
|
||||
|
|
@ -24,7 +25,7 @@ from .pipe_analysis import validate_attrs, analyze_pipes, print_pipe_analysis
|
|||
from .training import Example, validate_examples
|
||||
from .training.initialize import init_vocab, init_tok2vec
|
||||
from .scorer import Scorer
|
||||
from .util import registry, SimpleFrozenList, _pipe, raise_error
|
||||
from .util import registry, SimpleFrozenList, _pipe, raise_error, _DEFAULT_EMPTY_PIPES
|
||||
from .util import SimpleFrozenDict, combine_score_weights, CONFIG_SECTION_ORDER
|
||||
from .util import warn_if_jupyter_cupy
|
||||
from .lang.tokenizer_exceptions import URL_MATCH, BASE_EXCEPTIONS
|
||||
|
|
@ -1698,9 +1699,9 @@ class Language:
|
|||
config: Union[Dict[str, Any], Config] = {},
|
||||
*,
|
||||
vocab: Union[Vocab, bool] = True,
|
||||
disable: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
enable: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
exclude: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
disable: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
enable: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
exclude: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
meta: Dict[str, Any] = SimpleFrozenDict(),
|
||||
auto_fill: bool = True,
|
||||
validate: bool = True,
|
||||
|
|
@ -1727,12 +1728,6 @@ class Language:
|
|||
|
||||
DOCS: https://spacy.io/api/language#from_config
|
||||
"""
|
||||
if isinstance(disable, str):
|
||||
disable = [disable]
|
||||
if isinstance(enable, str):
|
||||
enable = [enable]
|
||||
if isinstance(exclude, str):
|
||||
exclude = [exclude]
|
||||
if auto_fill:
|
||||
config = Config(
|
||||
cls.default_config, section_order=CONFIG_SECTION_ORDER
|
||||
|
|
@ -1877,9 +1872,38 @@ class Language:
|
|||
nlp.vocab.from_bytes(vocab_b)
|
||||
|
||||
# Resolve disabled/enabled settings.
|
||||
if isinstance(disable, str):
|
||||
disable = [disable]
|
||||
if isinstance(enable, str):
|
||||
enable = [enable]
|
||||
if isinstance(exclude, str):
|
||||
exclude = [exclude]
|
||||
|
||||
def fetch_pipes_status(value: Iterable[str], key: str) -> Iterable[str]:
|
||||
"""Fetch value for `enable` or `disable` w.r.t. the specified config and passed arguments passed to
|
||||
.load(). If both arguments and config specified values for this field, the passed arguments take precedence
|
||||
and a warning is printed.
|
||||
value (Iterable[str]): Passed value for `enable` or `disable`.
|
||||
key (str): Key for field in config (either "enabled" or "disabled").
|
||||
RETURN (Iterable[str]):
|
||||
"""
|
||||
# We assume that no argument was passed if the value is the specified default value.
|
||||
if id(value) == id(_DEFAULT_EMPTY_PIPES):
|
||||
return config["nlp"].get(key, [])
|
||||
else:
|
||||
if len(config["nlp"].get(key, [])):
|
||||
warnings.warn(
|
||||
Warnings.W123.format(
|
||||
arg=key[:-1],
|
||||
arg_value=value,
|
||||
config_value=config["nlp"][key],
|
||||
)
|
||||
)
|
||||
return value
|
||||
|
||||
disabled_pipes = cls._resolve_component_status(
|
||||
[*config["nlp"]["disabled"], *disable],
|
||||
[*config["nlp"].get("enabled", []), *enable],
|
||||
fetch_pipes_status(disable, "disabled"),
|
||||
fetch_pipes_status(enable, "enabled"),
|
||||
config["nlp"]["pipeline"],
|
||||
)
|
||||
nlp._disabled = set(p for p in disabled_pipes if p not in exclude)
|
||||
|
|
@ -2064,14 +2088,7 @@ class Language:
|
|||
pipe_name for pipe_name in pipe_names if pipe_name not in enable
|
||||
]
|
||||
if disable and disable != to_disable:
|
||||
raise ValueError(
|
||||
Errors.E1042.format(
|
||||
arg1="enable",
|
||||
arg2="disable",
|
||||
arg1_values=enable,
|
||||
arg2_values=disable,
|
||||
)
|
||||
)
|
||||
raise ValueError(Errors.E1042.format(enable=enable, disable=disable))
|
||||
|
||||
return tuple(to_disable)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
from .matcher import Matcher
|
||||
from .phrasematcher import PhraseMatcher
|
||||
from .dependencymatcher import DependencyMatcher
|
||||
from .levenshtein import levenshtein
|
||||
|
||||
__all__ = ["Matcher", "PhraseMatcher", "DependencyMatcher"]
|
||||
__all__ = ["Matcher", "PhraseMatcher", "DependencyMatcher", "levenshtein"]
|
||||
|
|
|
|||
15
spacy/matcher/levenshtein.pyx
Normal file
15
spacy/matcher/levenshtein.pyx
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
# cython: profile=True, binding=True, infer_types=True
|
||||
from cpython.object cimport PyObject
|
||||
from libc.stdint cimport int64_t
|
||||
|
||||
from typing import Optional
|
||||
|
||||
|
||||
cdef extern from "polyleven.c":
|
||||
int64_t polyleven(PyObject *o1, PyObject *o2, int64_t k)
|
||||
|
||||
|
||||
cpdef int64_t levenshtein(a: str, b: str, k: Optional[int] = None):
|
||||
if k is None:
|
||||
k = -1
|
||||
return polyleven(<PyObject*>a, <PyObject*>b, k)
|
||||
384
spacy/matcher/polyleven.c
Normal file
384
spacy/matcher/polyleven.c
Normal file
|
|
@ -0,0 +1,384 @@
|
|||
/*
|
||||
* Adapted from Polyleven (https://ceptord.net/)
|
||||
*
|
||||
* Source: https://github.com/fujimotos/polyleven/blob/c3f95a080626c5652f0151a2e449963288ccae84/polyleven.c
|
||||
*
|
||||
* Copyright (c) 2021 Fujimoto Seiji <fujimoto@ceptord.net>
|
||||
* Copyright (c) 2021 Max Bachmann <kontakt@maxbachmann.de>
|
||||
* Copyright (c) 2022 Nick Mazuk
|
||||
* Copyright (c) 2022 Michael Weiss <code@mweiss.ch>
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
* of this software and associated documentation files (the "Software"), to deal
|
||||
* in the Software without restriction, including without limitation the rights
|
||||
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
* copies of the Software, and to permit persons to whom the Software is
|
||||
* furnished to do so, subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
* SOFTWARE.
|
||||
*/
|
||||
|
||||
#include <Python.h>
|
||||
#include <stdint.h>
|
||||
|
||||
#define MIN(a,b) ((a) < (b) ? (a) : (b))
|
||||
#define MAX(a,b) ((a) > (b) ? (a) : (b))
|
||||
#define CDIV(a,b) ((a) / (b) + ((a) % (b) > 0))
|
||||
#define BIT(i,n) (((i) >> (n)) & 1)
|
||||
#define FLIP(i,n) ((i) ^ ((uint64_t) 1 << (n)))
|
||||
#define ISASCII(kd) ((kd) == PyUnicode_1BYTE_KIND)
|
||||
|
||||
/*
|
||||
* Bare bone of PyUnicode
|
||||
*/
|
||||
struct strbuf {
|
||||
void *ptr;
|
||||
int kind;
|
||||
int64_t len;
|
||||
};
|
||||
|
||||
static void strbuf_init(struct strbuf *s, PyObject *o)
|
||||
{
|
||||
s->ptr = PyUnicode_DATA(o);
|
||||
s->kind = PyUnicode_KIND(o);
|
||||
s->len = PyUnicode_GET_LENGTH(o);
|
||||
}
|
||||
|
||||
#define strbuf_read(s, i) PyUnicode_READ((s)->kind, (s)->ptr, (i))
|
||||
|
||||
/*
|
||||
* An encoded mbleven model table.
|
||||
*
|
||||
* Each 8-bit integer represents an edit sequence, with using two
|
||||
* bits for a single operation.
|
||||
*
|
||||
* 01 = DELETE, 10 = INSERT, 11 = REPLACE
|
||||
*
|
||||
* For example, 13 is '1101' in binary notation, so it means
|
||||
* DELETE + REPLACE.
|
||||
*/
|
||||
static const uint8_t MBLEVEN_MATRIX[] = {
|
||||
3, 0, 0, 0, 0, 0, 0, 0,
|
||||
1, 0, 0, 0, 0, 0, 0, 0,
|
||||
15, 9, 6, 0, 0, 0, 0, 0,
|
||||
13, 7, 0, 0, 0, 0, 0, 0,
|
||||
5, 0, 0, 0, 0, 0, 0, 0,
|
||||
63, 39, 45, 57, 54, 30, 27, 0,
|
||||
61, 55, 31, 37, 25, 22, 0, 0,
|
||||
53, 29, 23, 0, 0, 0, 0, 0,
|
||||
21, 0, 0, 0, 0, 0, 0, 0,
|
||||
};
|
||||
|
||||
#define MBLEVEN_MATRIX_GET(k, d) ((((k) + (k) * (k)) / 2 - 1) + (d)) * 8
|
||||
|
||||
static int64_t mbleven_ascii(char *s1, int64_t len1,
|
||||
char *s2, int64_t len2, int k)
|
||||
{
|
||||
int pos;
|
||||
uint8_t m;
|
||||
int64_t i, j, c, r;
|
||||
|
||||
pos = MBLEVEN_MATRIX_GET(k, len1 - len2);
|
||||
r = k + 1;
|
||||
|
||||
while (MBLEVEN_MATRIX[pos]) {
|
||||
m = MBLEVEN_MATRIX[pos++];
|
||||
i = j = c = 0;
|
||||
while (i < len1 && j < len2) {
|
||||
if (s1[i] != s2[j]) {
|
||||
c++;
|
||||
if (!m) break;
|
||||
if (m & 1) i++;
|
||||
if (m & 2) j++;
|
||||
m >>= 2;
|
||||
} else {
|
||||
i++;
|
||||
j++;
|
||||
}
|
||||
}
|
||||
c += (len1 - i) + (len2 - j);
|
||||
r = MIN(r, c);
|
||||
if (r < 2) {
|
||||
return r;
|
||||
}
|
||||
}
|
||||
return r;
|
||||
}
|
||||
|
||||
static int64_t mbleven(PyObject *o1, PyObject *o2, int64_t k)
|
||||
{
|
||||
int pos;
|
||||
uint8_t m;
|
||||
int64_t i, j, c, r;
|
||||
struct strbuf s1, s2;
|
||||
|
||||
strbuf_init(&s1, o1);
|
||||
strbuf_init(&s2, o2);
|
||||
|
||||
if (s1.len < s2.len)
|
||||
return mbleven(o2, o1, k);
|
||||
|
||||
if (k > 3)
|
||||
return -1;
|
||||
|
||||
if (k < s1.len - s2.len)
|
||||
return k + 1;
|
||||
|
||||
if (ISASCII(s1.kind) && ISASCII(s2.kind))
|
||||
return mbleven_ascii(s1.ptr, s1.len, s2.ptr, s2.len, k);
|
||||
|
||||
pos = MBLEVEN_MATRIX_GET(k, s1.len - s2.len);
|
||||
r = k + 1;
|
||||
|
||||
while (MBLEVEN_MATRIX[pos]) {
|
||||
m = MBLEVEN_MATRIX[pos++];
|
||||
i = j = c = 0;
|
||||
while (i < s1.len && j < s2.len) {
|
||||
if (strbuf_read(&s1, i) != strbuf_read(&s2, j)) {
|
||||
c++;
|
||||
if (!m) break;
|
||||
if (m & 1) i++;
|
||||
if (m & 2) j++;
|
||||
m >>= 2;
|
||||
} else {
|
||||
i++;
|
||||
j++;
|
||||
}
|
||||
}
|
||||
c += (s1.len - i) + (s2.len - j);
|
||||
r = MIN(r, c);
|
||||
if (r < 2) {
|
||||
return r;
|
||||
}
|
||||
}
|
||||
return r;
|
||||
}
|
||||
|
||||
/*
|
||||
* Data structure to store Peq (equality bit-vector).
|
||||
*/
|
||||
struct blockmap_entry {
|
||||
uint32_t key[128];
|
||||
uint64_t val[128];
|
||||
};
|
||||
|
||||
struct blockmap {
|
||||
int64_t nr;
|
||||
struct blockmap_entry *list;
|
||||
};
|
||||
|
||||
#define blockmap_key(c) ((c) | 0x80000000U)
|
||||
#define blockmap_hash(c) ((c) % 128)
|
||||
|
||||
static int blockmap_init(struct blockmap *map, struct strbuf *s)
|
||||
{
|
||||
int64_t i;
|
||||
struct blockmap_entry *be;
|
||||
uint32_t c, k;
|
||||
uint8_t h;
|
||||
|
||||
map->nr = CDIV(s->len, 64);
|
||||
map->list = calloc(1, map->nr * sizeof(struct blockmap_entry));
|
||||
if (map->list == NULL) {
|
||||
PyErr_NoMemory();
|
||||
return -1;
|
||||
}
|
||||
|
||||
for (i = 0; i < s->len; i++) {
|
||||
be = &(map->list[i / 64]);
|
||||
c = strbuf_read(s, i);
|
||||
h = blockmap_hash(c);
|
||||
k = blockmap_key(c);
|
||||
|
||||
while (be->key[h] && be->key[h] != k)
|
||||
h = blockmap_hash(h + 1);
|
||||
be->key[h] = k;
|
||||
be->val[h] |= (uint64_t) 1 << (i % 64);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void blockmap_clear(struct blockmap *map)
|
||||
{
|
||||
if (map->list)
|
||||
free(map->list);
|
||||
map->list = NULL;
|
||||
map->nr = 0;
|
||||
}
|
||||
|
||||
static uint64_t blockmap_get(struct blockmap *map, int block, uint32_t c)
|
||||
{
|
||||
struct blockmap_entry *be;
|
||||
uint8_t h;
|
||||
uint32_t k;
|
||||
|
||||
h = blockmap_hash(c);
|
||||
k = blockmap_key(c);
|
||||
|
||||
be = &(map->list[block]);
|
||||
while (be->key[h] && be->key[h] != k)
|
||||
h = blockmap_hash(h + 1);
|
||||
return be->key[h] == k ? be->val[h] : 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Myers' bit-parallel algorithm
|
||||
*
|
||||
* See: G. Myers. "A fast bit-vector algorithm for approximate string
|
||||
* matching based on dynamic programming." Journal of the ACM, 1999.
|
||||
*/
|
||||
static int64_t myers1999_block(struct strbuf *s1, struct strbuf *s2,
|
||||
struct blockmap *map)
|
||||
{
|
||||
uint64_t Eq, Xv, Xh, Ph, Mh, Pv, Mv, Last;
|
||||
uint64_t *Mhc, *Phc;
|
||||
int64_t i, b, hsize, vsize, Score;
|
||||
uint8_t Pb, Mb;
|
||||
|
||||
hsize = CDIV(s1->len, 64);
|
||||
vsize = CDIV(s2->len, 64);
|
||||
Score = s2->len;
|
||||
|
||||
Phc = malloc(hsize * 2 * sizeof(uint64_t));
|
||||
if (Phc == NULL) {
|
||||
PyErr_NoMemory();
|
||||
return -1;
|
||||
}
|
||||
Mhc = Phc + hsize;
|
||||
memset(Phc, -1, hsize * sizeof(uint64_t));
|
||||
memset(Mhc, 0, hsize * sizeof(uint64_t));
|
||||
Last = (uint64_t)1 << ((s2->len - 1) % 64);
|
||||
|
||||
for (b = 0; b < vsize; b++) {
|
||||
Mv = 0;
|
||||
Pv = (uint64_t) -1;
|
||||
Score = s2->len;
|
||||
|
||||
for (i = 0; i < s1->len; i++) {
|
||||
Eq = blockmap_get(map, b, strbuf_read(s1, i));
|
||||
|
||||
Pb = BIT(Phc[i / 64], i % 64);
|
||||
Mb = BIT(Mhc[i / 64], i % 64);
|
||||
|
||||
Xv = Eq | Mv;
|
||||
Xh = ((((Eq | Mb) & Pv) + Pv) ^ Pv) | Eq | Mb;
|
||||
|
||||
Ph = Mv | ~ (Xh | Pv);
|
||||
Mh = Pv & Xh;
|
||||
|
||||
if (Ph & Last) Score++;
|
||||
if (Mh & Last) Score--;
|
||||
|
||||
if ((Ph >> 63) ^ Pb)
|
||||
Phc[i / 64] = FLIP(Phc[i / 64], i % 64);
|
||||
|
||||
if ((Mh >> 63) ^ Mb)
|
||||
Mhc[i / 64] = FLIP(Mhc[i / 64], i % 64);
|
||||
|
||||
Ph = (Ph << 1) | Pb;
|
||||
Mh = (Mh << 1) | Mb;
|
||||
|
||||
Pv = Mh | ~ (Xv | Ph);
|
||||
Mv = Ph & Xv;
|
||||
}
|
||||
}
|
||||
free(Phc);
|
||||
return Score;
|
||||
}
|
||||
|
||||
static int64_t myers1999_simple(uint8_t *s1, int64_t len1, uint8_t *s2, int64_t len2)
|
||||
{
|
||||
uint64_t Peq[256];
|
||||
uint64_t Eq, Xv, Xh, Ph, Mh, Pv, Mv, Last;
|
||||
int64_t i;
|
||||
int64_t Score = len2;
|
||||
|
||||
memset(Peq, 0, sizeof(Peq));
|
||||
|
||||
for (i = 0; i < len2; i++)
|
||||
Peq[s2[i]] |= (uint64_t) 1 << i;
|
||||
|
||||
Mv = 0;
|
||||
Pv = (uint64_t) -1;
|
||||
Last = (uint64_t) 1 << (len2 - 1);
|
||||
|
||||
for (i = 0; i < len1; i++) {
|
||||
Eq = Peq[s1[i]];
|
||||
|
||||
Xv = Eq | Mv;
|
||||
Xh = (((Eq & Pv) + Pv) ^ Pv) | Eq;
|
||||
|
||||
Ph = Mv | ~ (Xh | Pv);
|
||||
Mh = Pv & Xh;
|
||||
|
||||
if (Ph & Last) Score++;
|
||||
if (Mh & Last) Score--;
|
||||
|
||||
Ph = (Ph << 1) | 1;
|
||||
Mh = (Mh << 1);
|
||||
|
||||
Pv = Mh | ~ (Xv | Ph);
|
||||
Mv = Ph & Xv;
|
||||
}
|
||||
return Score;
|
||||
}
|
||||
|
||||
static int64_t myers1999(PyObject *o1, PyObject *o2)
|
||||
{
|
||||
struct strbuf s1, s2;
|
||||
struct blockmap map;
|
||||
int64_t ret;
|
||||
|
||||
strbuf_init(&s1, o1);
|
||||
strbuf_init(&s2, o2);
|
||||
|
||||
if (s1.len < s2.len)
|
||||
return myers1999(o2, o1);
|
||||
|
||||
if (ISASCII(s1.kind) && ISASCII(s2.kind) && s2.len < 65)
|
||||
return myers1999_simple(s1.ptr, s1.len, s2.ptr, s2.len);
|
||||
|
||||
if (blockmap_init(&map, &s2))
|
||||
return -1;
|
||||
|
||||
ret = myers1999_block(&s1, &s2, &map);
|
||||
blockmap_clear(&map);
|
||||
return ret;
|
||||
}
|
||||
|
||||
/*
|
||||
* Interface functions
|
||||
*/
|
||||
static int64_t polyleven(PyObject *o1, PyObject *o2, int64_t k)
|
||||
{
|
||||
int64_t len1, len2;
|
||||
|
||||
len1 = PyUnicode_GET_LENGTH(o1);
|
||||
len2 = PyUnicode_GET_LENGTH(o2);
|
||||
|
||||
if (len1 < len2)
|
||||
return polyleven(o2, o1, k);
|
||||
|
||||
if (k == 0)
|
||||
return PyUnicode_Compare(o1, o2) ? 1 : 0;
|
||||
|
||||
if (0 < k && k < len1 - len2)
|
||||
return k + 1;
|
||||
|
||||
if (len2 == 0)
|
||||
return len1;
|
||||
|
||||
if (0 < k && k < 4)
|
||||
return mbleven(o1, o2, k);
|
||||
|
||||
return myers1999(o1, o2);
|
||||
}
|
||||
|
|
@ -89,11 +89,14 @@ def pipes_with_nvtx_range(
|
|||
types.MethodType(nvtx_range_wrapper_for_pipe_method, pipe), func
|
||||
)
|
||||
|
||||
# Try to preserve the original function signature.
|
||||
# We need to preserve the original function signature so that
|
||||
# the original parameters are passed to pydantic for validation downstream.
|
||||
try:
|
||||
wrapped_func.__signature__ = inspect.signature(func) # type: ignore
|
||||
except:
|
||||
pass
|
||||
# Can fail for Cython methods that do not have bindings.
|
||||
warnings.warn(Warnings.W122.format(method=name, pipe=pipe.name))
|
||||
continue
|
||||
|
||||
try:
|
||||
setattr(
|
||||
|
|
|
|||
|
|
@ -1,11 +1,12 @@
|
|||
from pathlib import Path
|
||||
from typing import Optional, Callable, Iterable, List, Tuple
|
||||
from thinc.types import Floats2d
|
||||
from thinc.api import chain, clone, list2ragged, reduce_mean, residual
|
||||
from thinc.api import Model, Maxout, Linear, noop, tuplify, Ragged
|
||||
from thinc.api import chain, list2ragged, reduce_mean, residual
|
||||
from thinc.api import Model, Maxout, Linear, tuplify, Ragged
|
||||
|
||||
from ...util import registry
|
||||
from ...kb import KnowledgeBase, Candidate, get_candidates
|
||||
from ...kb import KnowledgeBase, InMemoryLookupKB
|
||||
from ...kb import Candidate, get_candidates, get_candidates_batch
|
||||
from ...vocab import Vocab
|
||||
from ...tokens import Span, Doc
|
||||
from ..extract_spans import extract_spans
|
||||
|
|
@ -78,9 +79,11 @@ def span_maker_forward(model, docs: List[Doc], is_train) -> Tuple[Ragged, Callab
|
|||
|
||||
|
||||
@registry.misc("spacy.KBFromFile.v1")
|
||||
def load_kb(kb_path: Path) -> Callable[[Vocab], KnowledgeBase]:
|
||||
def kb_from_file(vocab):
|
||||
kb = KnowledgeBase(vocab, entity_vector_length=1)
|
||||
def load_kb(
|
||||
kb_path: Path,
|
||||
) -> Callable[[Vocab], KnowledgeBase]:
|
||||
def kb_from_file(vocab: Vocab):
|
||||
kb = InMemoryLookupKB(vocab, entity_vector_length=1)
|
||||
kb.from_disk(kb_path)
|
||||
return kb
|
||||
|
||||
|
|
@ -88,9 +91,11 @@ def load_kb(kb_path: Path) -> Callable[[Vocab], KnowledgeBase]:
|
|||
|
||||
|
||||
@registry.misc("spacy.EmptyKB.v1")
|
||||
def empty_kb(entity_vector_length: int) -> Callable[[Vocab], KnowledgeBase]:
|
||||
def empty_kb_factory(vocab):
|
||||
return KnowledgeBase(vocab=vocab, entity_vector_length=entity_vector_length)
|
||||
def empty_kb(
|
||||
entity_vector_length: int,
|
||||
) -> Callable[[Vocab], KnowledgeBase]:
|
||||
def empty_kb_factory(vocab: Vocab):
|
||||
return InMemoryLookupKB(vocab=vocab, entity_vector_length=entity_vector_length)
|
||||
|
||||
return empty_kb_factory
|
||||
|
||||
|
|
@ -98,3 +103,10 @@ def empty_kb(entity_vector_length: int) -> Callable[[Vocab], KnowledgeBase]:
|
|||
@registry.misc("spacy.CandidateGenerator.v1")
|
||||
def create_candidates() -> Callable[[KnowledgeBase, Span], Iterable[Candidate]]:
|
||||
return get_candidates
|
||||
|
||||
|
||||
@registry.misc("spacy.CandidateBatchGenerator.v1")
|
||||
def create_candidates_batch() -> Callable[
|
||||
[KnowledgeBase, Iterable[Span]], Iterable[Iterable[Candidate]]
|
||||
]:
|
||||
return get_candidates_batch
|
||||
|
|
|
|||
|
|
@ -60,9 +60,11 @@ DEFAULT_NEL_MODEL = Config().from_str(default_model_config)["model"]
|
|||
"incl_context": True,
|
||||
"entity_vector_length": 64,
|
||||
"get_candidates": {"@misc": "spacy.CandidateGenerator.v1"},
|
||||
"get_candidates_batch": {"@misc": "spacy.CandidateBatchGenerator.v1"},
|
||||
"overwrite": True,
|
||||
"scorer": {"@scorers": "spacy.entity_linker_scorer.v1"},
|
||||
"use_gold_ents": True,
|
||||
"candidates_batch_size": 1,
|
||||
"threshold": None,
|
||||
"save_activations": False,
|
||||
},
|
||||
|
|
@ -83,9 +85,13 @@ def make_entity_linker(
|
|||
incl_context: bool,
|
||||
entity_vector_length: int,
|
||||
get_candidates: Callable[[KnowledgeBase, Span], Iterable[Candidate]],
|
||||
get_candidates_batch: Callable[
|
||||
[KnowledgeBase, Iterable[Span]], Iterable[Iterable[Candidate]]
|
||||
],
|
||||
overwrite: bool,
|
||||
scorer: Optional[Callable],
|
||||
use_gold_ents: bool,
|
||||
candidates_batch_size: int,
|
||||
threshold: Optional[float] = None,
|
||||
save_activations: bool,
|
||||
):
|
||||
|
|
@ -99,18 +105,22 @@ def make_entity_linker(
|
|||
incl_prior (bool): Whether or not to include prior probabilities from the KB in the model.
|
||||
incl_context (bool): Whether or not to include the local context in the model.
|
||||
entity_vector_length (int): Size of encoding vectors in the KB.
|
||||
get_candidates (Callable[[KnowledgeBase, "Span"], Iterable[Candidate]]): Function that
|
||||
get_candidates (Callable[[KnowledgeBase, Span], Iterable[Candidate]]): Function that
|
||||
produces a list of candidates, given a certain knowledge base and a textual mention.
|
||||
get_candidates_batch (
|
||||
Callable[[KnowledgeBase, Iterable[Span]], Iterable[Iterable[Candidate]]], Iterable[Candidate]]
|
||||
): Function that produces a list of candidates, given a certain knowledge base and several textual mentions.
|
||||
scorer (Optional[Callable]): The scoring method.
|
||||
use_gold_ents (bool): Whether to copy entities from gold docs or not. If false, another
|
||||
component must provide entity annotations.
|
||||
candidates_batch_size (int): Size of batches for entity candidate generation.
|
||||
threshold (Optional[float]): Confidence threshold for entity predictions. If confidence is below the threshold,
|
||||
prediction is discarded. If None, predictions are not filtered by any threshold.
|
||||
save_activations (bool): save model activations in Doc when annotating.
|
||||
"""
|
||||
|
||||
if not model.attrs.get("include_span_maker", False):
|
||||
# The only difference in arguments here is that use_gold_ents is not available
|
||||
# The only difference in arguments here is that use_gold_ents and threshold aren't available.
|
||||
return EntityLinker_v1(
|
||||
nlp.vocab,
|
||||
model,
|
||||
|
|
@ -134,9 +144,11 @@ def make_entity_linker(
|
|||
incl_context=incl_context,
|
||||
entity_vector_length=entity_vector_length,
|
||||
get_candidates=get_candidates,
|
||||
get_candidates_batch=get_candidates_batch,
|
||||
overwrite=overwrite,
|
||||
scorer=scorer,
|
||||
use_gold_ents=use_gold_ents,
|
||||
candidates_batch_size=candidates_batch_size,
|
||||
threshold=threshold,
|
||||
save_activations=save_activations,
|
||||
)
|
||||
|
|
@ -171,9 +183,13 @@ class EntityLinker(TrainablePipe):
|
|||
incl_context: bool,
|
||||
entity_vector_length: int,
|
||||
get_candidates: Callable[[KnowledgeBase, Span], Iterable[Candidate]],
|
||||
get_candidates_batch: Callable[
|
||||
[KnowledgeBase, Iterable[Span]], Iterable[Iterable[Candidate]]
|
||||
],
|
||||
overwrite: bool = BACKWARD_OVERWRITE,
|
||||
scorer: Optional[Callable] = entity_linker_score,
|
||||
use_gold_ents: bool,
|
||||
candidates_batch_size: int,
|
||||
threshold: Optional[float] = None,
|
||||
save_activations: bool = False,
|
||||
) -> None:
|
||||
|
|
@ -190,10 +206,14 @@ class EntityLinker(TrainablePipe):
|
|||
entity_vector_length (int): Size of encoding vectors in the KB.
|
||||
get_candidates (Callable[[KnowledgeBase, Span], Iterable[Candidate]]): Function that
|
||||
produces a list of candidates, given a certain knowledge base and a textual mention.
|
||||
scorer (Optional[Callable]): The scoring method. Defaults to
|
||||
Scorer.score_links.
|
||||
get_candidates_batch (
|
||||
Callable[[KnowledgeBase, Iterable[Span]], Iterable[Iterable[Candidate]]],
|
||||
Iterable[Candidate]]
|
||||
): Function that produces a list of candidates, given a certain knowledge base and several textual mentions.
|
||||
scorer (Optional[Callable]): The scoring method. Defaults to Scorer.score_links.
|
||||
use_gold_ents (bool): Whether to copy entities from gold docs or not. If false, another
|
||||
component must provide entity annotations.
|
||||
candidates_batch_size (int): Size of batches for entity candidate generation.
|
||||
threshold (Optional[float]): Confidence threshold for entity predictions. If confidence is below the
|
||||
threshold, prediction is discarded. If None, predictions are not filtered by any threshold.
|
||||
DOCS: https://spacy.io/api/entitylinker#init
|
||||
|
|
@ -216,23 +236,28 @@ class EntityLinker(TrainablePipe):
|
|||
self.incl_prior = incl_prior
|
||||
self.incl_context = incl_context
|
||||
self.get_candidates = get_candidates
|
||||
self.get_candidates_batch = get_candidates_batch
|
||||
self.cfg: Dict[str, Any] = {"overwrite": overwrite}
|
||||
self.distance = CosineDistance(normalize=False)
|
||||
# how many neighbour sentences to take into account
|
||||
# create an empty KB by default. If you want to load a predefined one, specify it in 'initialize'.
|
||||
# create an empty KB by default
|
||||
self.kb = empty_kb(entity_vector_length)(self.vocab)
|
||||
self.scorer = scorer
|
||||
self.use_gold_ents = use_gold_ents
|
||||
self.candidates_batch_size = candidates_batch_size
|
||||
self.threshold = threshold
|
||||
self.save_activations = save_activations
|
||||
|
||||
if candidates_batch_size < 1:
|
||||
raise ValueError(Errors.E1044)
|
||||
|
||||
def set_kb(self, kb_loader: Callable[[Vocab], KnowledgeBase]):
|
||||
"""Define the KB of this pipe by providing a function that will
|
||||
create it using this object's vocab."""
|
||||
if not callable(kb_loader):
|
||||
raise ValueError(Errors.E885.format(arg_type=type(kb_loader)))
|
||||
|
||||
self.kb = kb_loader(self.vocab)
|
||||
self.kb = kb_loader(self.vocab) # type: ignore
|
||||
|
||||
def validate_kb(self) -> None:
|
||||
# Raise an error if the knowledge base is not initialized.
|
||||
|
|
@ -254,8 +279,8 @@ class EntityLinker(TrainablePipe):
|
|||
get_examples (Callable[[], Iterable[Example]]): Function that
|
||||
returns a representative sample of gold-standard Example objects.
|
||||
nlp (Language): The current nlp object the component is part of.
|
||||
kb_loader (Callable[[Vocab], KnowledgeBase]): A function that creates a KnowledgeBase from a Vocab instance.
|
||||
Note that providing this argument, will overwrite all data accumulated in the current KB.
|
||||
kb_loader (Callable[[Vocab], KnowledgeBase]): A function that creates a KnowledgeBase from a Vocab
|
||||
instance. Note that providing this argument will overwrite all data accumulated in the current KB.
|
||||
Use this only when loading a KB as-such from file.
|
||||
|
||||
DOCS: https://spacy.io/api/entitylinker#initialize
|
||||
|
|
@ -439,15 +464,40 @@ class EntityLinker(TrainablePipe):
|
|||
docs_ents.append(Ragged(xp.zeros(0, dtype="uint64"), ops.alloc1i(0)))
|
||||
continue
|
||||
sentences = [s for s in doc.sents]
|
||||
# Looping through each entity (TODO: rewrite)
|
||||
for ent in doc.ents:
|
||||
|
||||
# Loop over entities in batches.
|
||||
for ent_idx in range(0, len(doc.ents), self.candidates_batch_size):
|
||||
ent_batch = doc.ents[ent_idx : ent_idx + self.candidates_batch_size]
|
||||
|
||||
# Look up candidate entities.
|
||||
valid_ent_idx = [
|
||||
idx
|
||||
for idx in range(len(ent_batch))
|
||||
if ent_batch[idx].label_ not in self.labels_discard
|
||||
]
|
||||
|
||||
batch_candidates = list(
|
||||
self.get_candidates_batch(
|
||||
self.kb, [ent_batch[idx] for idx in valid_ent_idx]
|
||||
)
|
||||
if self.candidates_batch_size > 1
|
||||
else [
|
||||
self.get_candidates(self.kb, ent_batch[idx])
|
||||
for idx in valid_ent_idx
|
||||
]
|
||||
)
|
||||
|
||||
# Looping through each entity in batch (TODO: rewrite)
|
||||
for j, ent in enumerate(ent_batch):
|
||||
sent_index = sentences.index(ent.sent)
|
||||
assert sent_index >= 0
|
||||
|
||||
if self.incl_context:
|
||||
# get n_neighbour sentences, clipped to the length of the document
|
||||
start_sentence = max(0, sent_index - self.n_sents)
|
||||
end_sentence = min(len(sentences) - 1, sent_index + self.n_sents)
|
||||
end_sentence = min(
|
||||
len(sentences) - 1, sent_index + self.n_sents
|
||||
)
|
||||
start_token = sentences[start_sentence].start
|
||||
end_token = sentences[end_sentence].end
|
||||
sent_doc = doc[start_token:end_token].as_doc()
|
||||
|
|
@ -466,7 +516,7 @@ class EntityLinker(TrainablePipe):
|
|||
ents=[0],
|
||||
)
|
||||
else:
|
||||
candidates = list(self.get_candidates(self.kb, ent))
|
||||
candidates = list(batch_candidates[j])
|
||||
if not candidates:
|
||||
# no prediction possible for this entity - setting to NIL
|
||||
final_kb_ids.append(self.NIL)
|
||||
|
|
@ -514,7 +564,8 @@ class EntityLinker(TrainablePipe):
|
|||
scores = prior_probs + sims - (prior_probs * sims)
|
||||
final_kb_ids.append(
|
||||
candidates[scores.argmax().item()].entity_
|
||||
if self.threshold is None or scores.max() >= self.threshold
|
||||
if self.threshold is None
|
||||
or scores.max() >= self.threshold
|
||||
else EntityLinker.NIL
|
||||
)
|
||||
self._add_activations(
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
import warnings
|
||||
from typing import Optional, Union, List, Dict, Tuple, Iterable, Any, Callable, Sequence
|
||||
from typing import cast
|
||||
import warnings
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
import srsly
|
||||
|
|
@ -317,7 +316,7 @@ class EntityRuler(Pipe):
|
|||
phrase_pattern["id"] = ent_id
|
||||
phrase_patterns.append(phrase_pattern)
|
||||
for entry in token_patterns + phrase_patterns: # type: ignore[operator]
|
||||
label = entry["label"]
|
||||
label = entry["label"] # type: ignore
|
||||
if "id" in entry:
|
||||
ent_label = label
|
||||
label = self._create_label(label, entry["id"])
|
||||
|
|
|
|||
|
|
@ -68,8 +68,7 @@ class EntityLinker_v1(TrainablePipe):
|
|||
entity_vector_length (int): Size of encoding vectors in the KB.
|
||||
get_candidates (Callable[[KnowledgeBase, Span], Iterable[Candidate]]): Function that
|
||||
produces a list of candidates, given a certain knowledge base and a textual mention.
|
||||
scorer (Optional[Callable]): The scoring method. Defaults to
|
||||
Scorer.score_links.
|
||||
scorer (Optional[Callable]): The scoring method. Defaults to Scorer.score_links.
|
||||
DOCS: https://spacy.io/api/entitylinker#init
|
||||
"""
|
||||
self.vocab = vocab
|
||||
|
|
@ -115,7 +114,7 @@ class EntityLinker_v1(TrainablePipe):
|
|||
get_examples (Callable[[], Iterable[Example]]): Function that
|
||||
returns a representative sample of gold-standard Example objects.
|
||||
nlp (Language): The current nlp object the component is part of.
|
||||
kb_loader (Callable[[Vocab], KnowledgeBase]): A function that creates a KnowledgeBase from a Vocab instance.
|
||||
kb_loader (Callable[[Vocab], KnowledgeBase]): A function that creates an InMemoryLookupKB from a Vocab instance.
|
||||
Note that providing this argument, will overwrite all data accumulated in the current KB.
|
||||
Use this only when loading a KB as-such from file.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# cython: infer_types=True, profile=True
|
||||
# cython: infer_types=True, profile=True, binding=True
|
||||
from typing import Optional, Tuple, Iterable, Iterator, Callable, Union, Dict
|
||||
import srsly
|
||||
import warnings
|
||||
|
|
|
|||
|
|
@ -123,9 +123,6 @@ class Tok2Vec(TrainablePipe):
|
|||
width = self.model.get_dim("nO")
|
||||
return [self.model.ops.alloc((0, width)) for doc in docs]
|
||||
tokvecs = self.model.predict(docs)
|
||||
batch_id = Tok2VecListener.get_batch_id(docs)
|
||||
for listener in self.listeners:
|
||||
listener.receive(batch_id, tokvecs, _empty_backprop)
|
||||
return tokvecs
|
||||
|
||||
def set_annotations(self, docs: Sequence[Doc], tokvecses) -> None:
|
||||
|
|
@ -286,6 +283,17 @@ class Tok2VecListener(Model):
|
|||
def forward(model: Tok2VecListener, inputs, is_train: bool):
|
||||
"""Supply the outputs from the upstream Tok2Vec component."""
|
||||
if is_train:
|
||||
# This might occur during training when the tok2vec layer is frozen / hasn't been updated.
|
||||
# In that case, it should be set to "annotating" so we can retrieve the embeddings from the doc.
|
||||
if model._batch_id is None:
|
||||
outputs = []
|
||||
for doc in inputs:
|
||||
if doc.tensor.size == 0:
|
||||
raise ValueError(Errors.E203.format(name="tok2vec"))
|
||||
else:
|
||||
outputs.append(doc.tensor)
|
||||
return outputs, _empty_backprop
|
||||
else:
|
||||
model.verify_inputs(inputs)
|
||||
return model._outputs, model._backprop
|
||||
else:
|
||||
|
|
@ -306,7 +314,7 @@ def forward(model: Tok2VecListener, inputs, is_train: bool):
|
|||
outputs.append(model.ops.alloc2f(len(doc), width))
|
||||
else:
|
||||
outputs.append(doc.tensor)
|
||||
return outputs, lambda dX: []
|
||||
return outputs, _empty_backprop
|
||||
|
||||
|
||||
def _empty_backprop(dX): # for pickling
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# cython: infer_types=True, profile=True
|
||||
# cython: infer_types=True, profile=True, binding=True
|
||||
from typing import Iterable, Iterator, Optional, Dict, Tuple, Callable
|
||||
import srsly
|
||||
from thinc.api import set_dropout_rate, Model, Optimizer
|
||||
|
|
|
|||
|
|
@ -82,6 +82,21 @@ def test_issue2396(en_vocab):
|
|||
assert (span.get_lca_matrix() == matrix).all()
|
||||
|
||||
|
||||
@pytest.mark.issue(11499)
|
||||
def test_init_args_unmodified(en_vocab):
|
||||
words = ["A", "sentence"]
|
||||
ents = ["B-TYPE1", ""]
|
||||
sent_starts = [True, False]
|
||||
Doc(
|
||||
vocab=en_vocab,
|
||||
words=words,
|
||||
ents=ents,
|
||||
sent_starts=sent_starts,
|
||||
)
|
||||
assert ents == ["B-TYPE1", ""]
|
||||
assert sent_starts == [True, False]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text", ["-0.23", "+123,456", "±1"])
|
||||
@pytest.mark.parametrize("lang_cls", [English, MultiLanguage])
|
||||
@pytest.mark.issue(2782)
|
||||
|
|
|
|||
18
spacy/tests/lang/grc/test_tokenizer.py
Normal file
18
spacy/tests/lang/grc/test_tokenizer.py
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
import pytest
|
||||
|
||||
|
||||
# fmt: off
|
||||
GRC_TOKEN_EXCEPTION_TESTS = [
|
||||
("τὸ 〈τῆς〉 φιλοσοφίας ἔργον ἔνιοί φασιν ἀπὸ ⟦βαρβάρων⟧ ἄρξαι.", ["τὸ", "〈", "τῆς", "〉", "φιλοσοφίας", "ἔργον", "ἔνιοί", "φασιν", "ἀπὸ", "⟦", "βαρβάρων", "⟧", "ἄρξαι", "."]),
|
||||
("τὴν δὲ τῶν Αἰγυπτίων φιλοσοφίαν εἶναι τοιαύτην περί τε †θεῶν† καὶ ὑπὲρ δικαιοσύνης.", ["τὴν", "δὲ", "τῶν", "Αἰγυπτίων", "φιλοσοφίαν", "εἶναι", "τοιαύτην", "περί", "τε", "†", "θεῶν", "†", "καὶ", "ὑπὲρ", "δικαιοσύνης", "."]),
|
||||
("⸏πόσις δ' Ἐρεχθεύς ἐστί μοι σεσωσμένος⸏", ["⸏", "πόσις", "δ'", "Ἐρεχθεύς", "ἐστί", "μοι", "σεσωσμένος", "⸏"]),
|
||||
("⸏ὔπνον ἴδωμεν⸎", ["⸏", "ὔπνον", "ἴδωμεν", "⸎"]),
|
||||
]
|
||||
# fmt: on
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text,expected_tokens", GRC_TOKEN_EXCEPTION_TESTS)
|
||||
def test_grc_tokenizer(grc_tokenizer, text, expected_tokens):
|
||||
tokens = grc_tokenizer(text)
|
||||
token_list = [token.text for token in tokens if not token.is_space]
|
||||
assert expected_tokens == token_list
|
||||
44
spacy/tests/matcher/test_levenshtein.py
Normal file
44
spacy/tests/matcher/test_levenshtein.py
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
import pytest
|
||||
from spacy.matcher import levenshtein
|
||||
|
||||
|
||||
# empty string plus 10 random ASCII, 10 random unicode, and 2 random long tests
|
||||
# from polyleven
|
||||
@pytest.mark.parametrize(
|
||||
"dist,a,b",
|
||||
[
|
||||
(0, "", ""),
|
||||
(4, "bbcb", "caba"),
|
||||
(3, "abcb", "cacc"),
|
||||
(3, "aa", "ccc"),
|
||||
(1, "cca", "ccac"),
|
||||
(1, "aba", "aa"),
|
||||
(4, "bcbb", "abac"),
|
||||
(3, "acbc", "bba"),
|
||||
(3, "cbba", "a"),
|
||||
(2, "bcc", "ba"),
|
||||
(4, "aaa", "ccbb"),
|
||||
(3, "うあい", "いいうい"),
|
||||
(2, "あううい", "うあい"),
|
||||
(3, "いういい", "うううあ"),
|
||||
(2, "うい", "あいあ"),
|
||||
(2, "いあい", "いう"),
|
||||
(1, "いい", "あいい"),
|
||||
(3, "あうあ", "いいああ"),
|
||||
(4, "いあうう", "ううああ"),
|
||||
(3, "いあいい", "ういああ"),
|
||||
(3, "いいああ", "ううあう"),
|
||||
(
|
||||
166,
|
||||
"TCTGGGCACGGATTCGTCAGATTCCATGTCCATATTTGAGGCTCTTGCAGGCAAAATTTGGGCATGTGAACTCCTTATAGTCCCCGTGC",
|
||||
"ATATGGATTGGGGGCATTCAAAGATACGGTTTCCCTTTCTTCAGTTTCGCGCGGCGCACGTCCGGGTGCGAGCCAGTTCGTCTTACTCACATTGTCGACTTCACGAATCGCGCATGATGTGCTTAGCCTGTACTTACGAACGAACTTTCGGTCCAAATACATTCTATCAACACCGAGGTATCCGTGCCACACGCCGAAGCTCGACCGTGTTCGTTGAGAGGTGGAAATGGTAAAAGATGAACATAGTC",
|
||||
),
|
||||
(
|
||||
111,
|
||||
"GGTTCGGCCGAATTCATAGAGCGTGGTAGTCGACGGTATCCCGCCTGGTAGGGGCCCCTTCTACCTAGCGGAAGTTTGTCAGTACTCTATAACACGAGGGCCTCTCACACCCTAGATCGTCCAGCCACTCGAAGATCGCAGCACCCTTACAGAAAGGCATTAATGTTTCTCCTAGCACTTGTGCAATGGTGAAGGAGTGATG",
|
||||
"CGTAACACTTCGCGCTACTGGGCTGCAACGTCTTGGGCATACATGCAAGATTATCTAATGCAAGCTTGAGCCCCGCTTGCGGAATTTCCCTAATCGGGGTCCCTTCCTGTTACGATAAGGACGCGTGCACT",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_levenshtein(dist, a, b):
|
||||
assert levenshtein(a, b) == dist
|
||||
|
|
@ -7,7 +7,7 @@ from thinc.types import Ragged
|
|||
from spacy import registry, util
|
||||
from spacy.attrs import ENT_KB_ID
|
||||
from spacy.compat import pickle
|
||||
from spacy.kb import Candidate, KnowledgeBase, get_candidates
|
||||
from spacy.kb import Candidate, InMemoryLookupKB, get_candidates, KnowledgeBase
|
||||
from spacy.lang.en import English
|
||||
from spacy.ml import load_kb
|
||||
from spacy.pipeline import EntityLinker, TrainablePipe
|
||||
|
|
@ -35,7 +35,7 @@ def assert_almost_equal(a, b):
|
|||
def test_issue4674():
|
||||
"""Test that setting entities with overlapping identifiers does not mess up IO"""
|
||||
nlp = English()
|
||||
kb = KnowledgeBase(nlp.vocab, entity_vector_length=3)
|
||||
kb = InMemoryLookupKB(nlp.vocab, entity_vector_length=3)
|
||||
vector1 = [0.9, 1.1, 1.01]
|
||||
vector2 = [1.8, 2.25, 2.01]
|
||||
with pytest.warns(UserWarning):
|
||||
|
|
@ -52,7 +52,7 @@ def test_issue4674():
|
|||
dir_path.mkdir()
|
||||
file_path = dir_path / "kb"
|
||||
kb.to_disk(str(file_path))
|
||||
kb2 = KnowledgeBase(nlp.vocab, entity_vector_length=3)
|
||||
kb2 = InMemoryLookupKB(nlp.vocab, entity_vector_length=3)
|
||||
kb2.from_disk(str(file_path))
|
||||
assert kb2.get_size_entities() == 1
|
||||
|
||||
|
|
@ -60,9 +60,9 @@ def test_issue4674():
|
|||
@pytest.mark.issue(6730)
|
||||
def test_issue6730(en_vocab):
|
||||
"""Ensure that the KB does not accept empty strings, but otherwise IO works fine."""
|
||||
from spacy.kb import KnowledgeBase
|
||||
from spacy.kb.kb_in_memory import InMemoryLookupKB
|
||||
|
||||
kb = KnowledgeBase(en_vocab, entity_vector_length=3)
|
||||
kb = InMemoryLookupKB(en_vocab, entity_vector_length=3)
|
||||
kb.add_entity(entity="1", freq=148, entity_vector=[1, 2, 3])
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
|
|
@ -128,7 +128,7 @@ def test_issue7065_b():
|
|||
|
||||
def create_kb(vocab):
|
||||
# create artificial KB
|
||||
mykb = KnowledgeBase(vocab, entity_vector_length=vector_length)
|
||||
mykb = InMemoryLookupKB(vocab, entity_vector_length=vector_length)
|
||||
mykb.add_entity(entity="Q270853", freq=12, entity_vector=[9, 1, -7])
|
||||
mykb.add_alias(
|
||||
alias="No. 8",
|
||||
|
|
@ -191,7 +191,7 @@ def test_no_entities():
|
|||
|
||||
def create_kb(vocab):
|
||||
# create artificial KB
|
||||
mykb = KnowledgeBase(vocab, entity_vector_length=vector_length)
|
||||
mykb = InMemoryLookupKB(vocab, entity_vector_length=vector_length)
|
||||
mykb.add_entity(entity="Q2146908", freq=12, entity_vector=[6, -4, 3])
|
||||
mykb.add_alias("Russ Cochran", ["Q2146908"], [0.9])
|
||||
return mykb
|
||||
|
|
@ -232,7 +232,7 @@ def test_partial_links():
|
|||
|
||||
def create_kb(vocab):
|
||||
# create artificial KB
|
||||
mykb = KnowledgeBase(vocab, entity_vector_length=vector_length)
|
||||
mykb = InMemoryLookupKB(vocab, entity_vector_length=vector_length)
|
||||
mykb.add_entity(entity="Q2146908", freq=12, entity_vector=[6, -4, 3])
|
||||
mykb.add_alias("Russ Cochran", ["Q2146908"], [0.9])
|
||||
return mykb
|
||||
|
|
@ -264,7 +264,7 @@ def test_partial_links():
|
|||
|
||||
def test_kb_valid_entities(nlp):
|
||||
"""Test the valid construction of a KB with 3 entities and two aliases"""
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=3)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=3)
|
||||
|
||||
# adding entities
|
||||
mykb.add_entity(entity="Q1", freq=19, entity_vector=[8, 4, 3])
|
||||
|
|
@ -293,7 +293,7 @@ def test_kb_valid_entities(nlp):
|
|||
|
||||
def test_kb_invalid_entities(nlp):
|
||||
"""Test the invalid construction of a KB with an alias linked to a non-existing entity"""
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
|
||||
# adding entities
|
||||
mykb.add_entity(entity="Q1", freq=19, entity_vector=[1])
|
||||
|
|
@ -309,7 +309,7 @@ def test_kb_invalid_entities(nlp):
|
|||
|
||||
def test_kb_invalid_probabilities(nlp):
|
||||
"""Test the invalid construction of a KB with wrong prior probabilities"""
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
|
||||
# adding entities
|
||||
mykb.add_entity(entity="Q1", freq=19, entity_vector=[1])
|
||||
|
|
@ -323,7 +323,7 @@ def test_kb_invalid_probabilities(nlp):
|
|||
|
||||
def test_kb_invalid_combination(nlp):
|
||||
"""Test the invalid construction of a KB with non-matching entity and probability lists"""
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
|
||||
# adding entities
|
||||
mykb.add_entity(entity="Q1", freq=19, entity_vector=[1])
|
||||
|
|
@ -339,7 +339,7 @@ def test_kb_invalid_combination(nlp):
|
|||
|
||||
def test_kb_invalid_entity_vector(nlp):
|
||||
"""Test the invalid construction of a KB with non-matching entity vector lengths"""
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=3)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=3)
|
||||
|
||||
# adding entities
|
||||
mykb.add_entity(entity="Q1", freq=19, entity_vector=[1, 2, 3])
|
||||
|
|
@ -377,7 +377,7 @@ def test_kb_initialize_empty(nlp):
|
|||
|
||||
def test_kb_serialize(nlp):
|
||||
"""Test serialization of the KB"""
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
with make_tempdir() as d:
|
||||
# normal read-write behaviour
|
||||
mykb.to_disk(d / "kb")
|
||||
|
|
@ -394,12 +394,12 @@ def test_kb_serialize(nlp):
|
|||
@pytest.mark.issue(9137)
|
||||
def test_kb_serialize_2(nlp):
|
||||
v = [5, 6, 7, 8]
|
||||
kb1 = KnowledgeBase(vocab=nlp.vocab, entity_vector_length=4)
|
||||
kb1 = InMemoryLookupKB(vocab=nlp.vocab, entity_vector_length=4)
|
||||
kb1.set_entities(["E1"], [1], [v])
|
||||
assert kb1.get_vector("E1") == v
|
||||
with make_tempdir() as d:
|
||||
kb1.to_disk(d / "kb")
|
||||
kb2 = KnowledgeBase(vocab=nlp.vocab, entity_vector_length=4)
|
||||
kb2 = InMemoryLookupKB(vocab=nlp.vocab, entity_vector_length=4)
|
||||
kb2.from_disk(d / "kb")
|
||||
assert kb2.get_vector("E1") == v
|
||||
|
||||
|
|
@ -409,7 +409,7 @@ def test_kb_set_entities(nlp):
|
|||
v = [5, 6, 7, 8]
|
||||
v1 = [1, 1, 1, 0]
|
||||
v2 = [2, 2, 2, 3]
|
||||
kb1 = KnowledgeBase(vocab=nlp.vocab, entity_vector_length=4)
|
||||
kb1 = InMemoryLookupKB(vocab=nlp.vocab, entity_vector_length=4)
|
||||
kb1.set_entities(["E0"], [1], [v])
|
||||
assert kb1.get_entity_strings() == ["E0"]
|
||||
kb1.set_entities(["E1", "E2"], [1, 9], [v1, v2])
|
||||
|
|
@ -418,7 +418,7 @@ def test_kb_set_entities(nlp):
|
|||
assert kb1.get_vector("E2") == v2
|
||||
with make_tempdir() as d:
|
||||
kb1.to_disk(d / "kb")
|
||||
kb2 = KnowledgeBase(vocab=nlp.vocab, entity_vector_length=4)
|
||||
kb2 = InMemoryLookupKB(vocab=nlp.vocab, entity_vector_length=4)
|
||||
kb2.from_disk(d / "kb")
|
||||
assert set(kb2.get_entity_strings()) == {"E1", "E2"}
|
||||
assert kb2.get_vector("E1") == v1
|
||||
|
|
@ -429,7 +429,7 @@ def test_kb_serialize_vocab(nlp):
|
|||
"""Test serialization of the KB and custom strings"""
|
||||
entity = "MyFunnyID"
|
||||
assert entity not in nlp.vocab.strings
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
assert not mykb.contains_entity(entity)
|
||||
mykb.add_entity(entity, freq=342, entity_vector=[3])
|
||||
assert mykb.contains_entity(entity)
|
||||
|
|
@ -437,14 +437,14 @@ def test_kb_serialize_vocab(nlp):
|
|||
with make_tempdir() as d:
|
||||
# normal read-write behaviour
|
||||
mykb.to_disk(d / "kb")
|
||||
mykb_new = KnowledgeBase(Vocab(), entity_vector_length=1)
|
||||
mykb_new = InMemoryLookupKB(Vocab(), entity_vector_length=1)
|
||||
mykb_new.from_disk(d / "kb")
|
||||
assert entity in mykb_new.vocab.strings
|
||||
|
||||
|
||||
def test_candidate_generation(nlp):
|
||||
"""Test correct candidate generation"""
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
doc = nlp("douglas adam Adam shrubbery")
|
||||
|
||||
douglas_ent = doc[0:1]
|
||||
|
|
@ -482,7 +482,7 @@ def test_el_pipe_configuration(nlp):
|
|||
ruler.add_patterns([pattern])
|
||||
|
||||
def create_kb(vocab):
|
||||
kb = KnowledgeBase(vocab, entity_vector_length=1)
|
||||
kb = InMemoryLookupKB(vocab, entity_vector_length=1)
|
||||
kb.add_entity(entity="Q2", freq=12, entity_vector=[2])
|
||||
kb.add_entity(entity="Q3", freq=5, entity_vector=[3])
|
||||
kb.add_alias(alias="douglas", entities=["Q2", "Q3"], probabilities=[0.8, 0.1])
|
||||
|
|
@ -501,10 +501,21 @@ def test_el_pipe_configuration(nlp):
|
|||
def get_lowercased_candidates(kb, span):
|
||||
return kb.get_alias_candidates(span.text.lower())
|
||||
|
||||
def get_lowercased_candidates_batch(kb, spans):
|
||||
return [get_lowercased_candidates(kb, span) for span in spans]
|
||||
|
||||
@registry.misc("spacy.LowercaseCandidateGenerator.v1")
|
||||
def create_candidates() -> Callable[[KnowledgeBase, "Span"], Iterable[Candidate]]:
|
||||
def create_candidates() -> Callable[
|
||||
[InMemoryLookupKB, "Span"], Iterable[Candidate]
|
||||
]:
|
||||
return get_lowercased_candidates
|
||||
|
||||
@registry.misc("spacy.LowercaseCandidateBatchGenerator.v1")
|
||||
def create_candidates_batch() -> Callable[
|
||||
[InMemoryLookupKB, Iterable["Span"]], Iterable[Iterable[Candidate]]
|
||||
]:
|
||||
return get_lowercased_candidates_batch
|
||||
|
||||
# replace the pipe with a new one with with a different candidate generator
|
||||
entity_linker = nlp.replace_pipe(
|
||||
"entity_linker",
|
||||
|
|
@ -512,6 +523,9 @@ def test_el_pipe_configuration(nlp):
|
|||
config={
|
||||
"incl_context": False,
|
||||
"get_candidates": {"@misc": "spacy.LowercaseCandidateGenerator.v1"},
|
||||
"get_candidates_batch": {
|
||||
"@misc": "spacy.LowercaseCandidateBatchGenerator.v1"
|
||||
},
|
||||
},
|
||||
)
|
||||
entity_linker.set_kb(create_kb)
|
||||
|
|
@ -533,7 +547,7 @@ def test_nel_nsents(nlp):
|
|||
|
||||
def test_vocab_serialization(nlp):
|
||||
"""Test that string information is retained across storage"""
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
|
||||
# adding entities
|
||||
mykb.add_entity(entity="Q1", freq=27, entity_vector=[1])
|
||||
|
|
@ -553,7 +567,7 @@ def test_vocab_serialization(nlp):
|
|||
|
||||
with make_tempdir() as d:
|
||||
mykb.to_disk(d / "kb")
|
||||
kb_new_vocab = KnowledgeBase(Vocab(), entity_vector_length=1)
|
||||
kb_new_vocab = InMemoryLookupKB(Vocab(), entity_vector_length=1)
|
||||
kb_new_vocab.from_disk(d / "kb")
|
||||
|
||||
candidates = kb_new_vocab.get_alias_candidates("adam")
|
||||
|
|
@ -569,7 +583,7 @@ def test_vocab_serialization(nlp):
|
|||
|
||||
def test_append_alias(nlp):
|
||||
"""Test that we can append additional alias-entity pairs"""
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
|
||||
# adding entities
|
||||
mykb.add_entity(entity="Q1", freq=27, entity_vector=[1])
|
||||
|
|
@ -600,7 +614,7 @@ def test_append_alias(nlp):
|
|||
@pytest.mark.filterwarnings("ignore:\\[W036")
|
||||
def test_append_invalid_alias(nlp):
|
||||
"""Test that append an alias will throw an error if prior probs are exceeding 1"""
|
||||
mykb = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
mykb = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
|
||||
# adding entities
|
||||
mykb.add_entity(entity="Q1", freq=27, entity_vector=[1])
|
||||
|
|
@ -622,7 +636,7 @@ def test_preserving_links_asdoc(nlp):
|
|||
vector_length = 1
|
||||
|
||||
def create_kb(vocab):
|
||||
mykb = KnowledgeBase(vocab, entity_vector_length=vector_length)
|
||||
mykb = InMemoryLookupKB(vocab, entity_vector_length=vector_length)
|
||||
# adding entities
|
||||
mykb.add_entity(entity="Q1", freq=19, entity_vector=[1])
|
||||
mykb.add_entity(entity="Q2", freq=8, entity_vector=[1])
|
||||
|
|
@ -724,7 +738,7 @@ def test_overfitting_IO():
|
|||
# create artificial KB - assign same prior weight to the two russ cochran's
|
||||
# Q2146908 (Russ Cochran): American golfer
|
||||
# Q7381115 (Russ Cochran): publisher
|
||||
mykb = KnowledgeBase(vocab, entity_vector_length=vector_length)
|
||||
mykb = InMemoryLookupKB(vocab, entity_vector_length=vector_length)
|
||||
mykb.add_entity(entity="Q2146908", freq=12, entity_vector=[6, -4, 3])
|
||||
mykb.add_entity(entity="Q7381115", freq=12, entity_vector=[9, 1, -7])
|
||||
mykb.add_alias(
|
||||
|
|
@ -806,7 +820,7 @@ def test_kb_serialization():
|
|||
kb_dir = tmp_dir / "kb"
|
||||
nlp1 = English()
|
||||
assert "Q2146908" not in nlp1.vocab.strings
|
||||
mykb = KnowledgeBase(nlp1.vocab, entity_vector_length=vector_length)
|
||||
mykb = InMemoryLookupKB(nlp1.vocab, entity_vector_length=vector_length)
|
||||
mykb.add_entity(entity="Q2146908", freq=12, entity_vector=[6, -4, 3])
|
||||
mykb.add_alias(alias="Russ Cochran", entities=["Q2146908"], probabilities=[0.8])
|
||||
assert "Q2146908" in nlp1.vocab.strings
|
||||
|
|
@ -829,7 +843,7 @@ def test_kb_serialization():
|
|||
def test_kb_pickle():
|
||||
# Test that the KB can be pickled
|
||||
nlp = English()
|
||||
kb_1 = KnowledgeBase(nlp.vocab, entity_vector_length=3)
|
||||
kb_1 = InMemoryLookupKB(nlp.vocab, entity_vector_length=3)
|
||||
kb_1.add_entity(entity="Q2146908", freq=12, entity_vector=[6, -4, 3])
|
||||
assert not kb_1.contains_alias("Russ Cochran")
|
||||
kb_1.add_alias(alias="Russ Cochran", entities=["Q2146908"], probabilities=[0.8])
|
||||
|
|
@ -843,7 +857,7 @@ def test_kb_pickle():
|
|||
def test_nel_pickle():
|
||||
# Test that a pipeline with an EL component can be pickled
|
||||
def create_kb(vocab):
|
||||
kb = KnowledgeBase(vocab, entity_vector_length=3)
|
||||
kb = InMemoryLookupKB(vocab, entity_vector_length=3)
|
||||
kb.add_entity(entity="Q2146908", freq=12, entity_vector=[6, -4, 3])
|
||||
kb.add_alias(alias="Russ Cochran", entities=["Q2146908"], probabilities=[0.8])
|
||||
return kb
|
||||
|
|
@ -865,7 +879,7 @@ def test_nel_pickle():
|
|||
def test_kb_to_bytes():
|
||||
# Test that the KB's to_bytes method works correctly
|
||||
nlp = English()
|
||||
kb_1 = KnowledgeBase(nlp.vocab, entity_vector_length=3)
|
||||
kb_1 = InMemoryLookupKB(nlp.vocab, entity_vector_length=3)
|
||||
kb_1.add_entity(entity="Q2146908", freq=12, entity_vector=[6, -4, 3])
|
||||
kb_1.add_entity(entity="Q66", freq=9, entity_vector=[1, 2, 3])
|
||||
kb_1.add_alias(alias="Russ Cochran", entities=["Q2146908"], probabilities=[0.8])
|
||||
|
|
@ -875,7 +889,7 @@ def test_kb_to_bytes():
|
|||
)
|
||||
assert kb_1.contains_alias("Russ Cochran")
|
||||
kb_bytes = kb_1.to_bytes()
|
||||
kb_2 = KnowledgeBase(nlp.vocab, entity_vector_length=3)
|
||||
kb_2 = InMemoryLookupKB(nlp.vocab, entity_vector_length=3)
|
||||
assert not kb_2.contains_alias("Russ Cochran")
|
||||
kb_2 = kb_2.from_bytes(kb_bytes)
|
||||
# check that both KBs are exactly the same
|
||||
|
|
@ -898,7 +912,7 @@ def test_kb_to_bytes():
|
|||
def test_nel_to_bytes():
|
||||
# Test that a pipeline with an EL component can be converted to bytes
|
||||
def create_kb(vocab):
|
||||
kb = KnowledgeBase(vocab, entity_vector_length=3)
|
||||
kb = InMemoryLookupKB(vocab, entity_vector_length=3)
|
||||
kb.add_entity(entity="Q2146908", freq=12, entity_vector=[6, -4, 3])
|
||||
kb.add_alias(alias="Russ Cochran", entities=["Q2146908"], probabilities=[0.8])
|
||||
return kb
|
||||
|
|
@ -988,7 +1002,7 @@ def test_legacy_architectures(name, config):
|
|||
train_examples.append(Example.from_dict(doc, annotation))
|
||||
|
||||
def create_kb(vocab):
|
||||
mykb = KnowledgeBase(vocab, entity_vector_length=vector_length)
|
||||
mykb = InMemoryLookupKB(vocab, entity_vector_length=vector_length)
|
||||
mykb.add_entity(entity="Q2146908", freq=12, entity_vector=[6, -4, 3])
|
||||
mykb.add_entity(entity="Q7381115", freq=12, entity_vector=[9, 1, -7])
|
||||
mykb.add_alias(
|
||||
|
|
@ -1055,7 +1069,7 @@ def test_no_gold_ents(patterns):
|
|||
|
||||
def create_kb(vocab):
|
||||
# create artificial KB
|
||||
mykb = KnowledgeBase(vocab, entity_vector_length=vector_length)
|
||||
mykb = InMemoryLookupKB(vocab, entity_vector_length=vector_length)
|
||||
mykb.add_entity(entity="Q613241", freq=12, entity_vector=[6, -4, 3])
|
||||
mykb.add_alias("Kirby", ["Q613241"], [0.9])
|
||||
# Placeholder
|
||||
|
|
@ -1105,7 +1119,7 @@ def test_tokenization_mismatch():
|
|||
|
||||
def create_kb(vocab):
|
||||
# create placeholder KB
|
||||
mykb = KnowledgeBase(vocab, entity_vector_length=vector_length)
|
||||
mykb = InMemoryLookupKB(vocab, entity_vector_length=vector_length)
|
||||
mykb.add_entity(entity="Q613241", freq=12, entity_vector=[6, -4, 3])
|
||||
mykb.add_alias("Kirby", ["Q613241"], [0.9])
|
||||
return mykb
|
||||
|
|
@ -1122,6 +1136,12 @@ def test_tokenization_mismatch():
|
|||
nlp.evaluate(train_examples)
|
||||
|
||||
|
||||
def test_abstract_kb_instantiation():
|
||||
"""Test whether instantiation of abstract KB base class fails."""
|
||||
with pytest.raises(TypeError):
|
||||
KnowledgeBase(None, 3)
|
||||
|
||||
|
||||
# fmt: off
|
||||
@pytest.mark.parametrize(
|
||||
"meet_threshold,config",
|
||||
|
|
@ -1152,7 +1172,7 @@ def test_threshold(meet_threshold: bool, config: Dict[str, Any]):
|
|||
|
||||
def create_kb(vocab):
|
||||
# create artificial KB
|
||||
mykb = KnowledgeBase(vocab, entity_vector_length=3)
|
||||
mykb = InMemoryLookupKB(vocab, entity_vector_length=3)
|
||||
mykb.add_entity(entity=entity_id, freq=12, entity_vector=[6, -4, 3])
|
||||
mykb.add_alias(
|
||||
alias="Mahler",
|
||||
|
|
@ -1194,7 +1214,7 @@ def test_save_activations():
|
|||
# create artificial KB - assign same prior weight to the two russ cochran's
|
||||
# Q2146908 (Russ Cochran): American golfer
|
||||
# Q7381115 (Russ Cochran): publisher
|
||||
mykb = KnowledgeBase(vocab, entity_vector_length=vector_length)
|
||||
mykb = InMemoryLookupKB(vocab, entity_vector_length=vector_length)
|
||||
mykb.add_entity(entity="Q2146908", freq=12, entity_vector=[6, -4, 3])
|
||||
mykb.add_entity(entity="Q7381115", freq=12, entity_vector=[9, 1, -7])
|
||||
mykb.add_alias(
|
||||
|
|
|
|||
|
|
@ -605,10 +605,35 @@ def test_update_with_annotates():
|
|||
assert results[component] == ""
|
||||
|
||||
|
||||
def test_load_disable_enable() -> None:
|
||||
"""
|
||||
Tests spacy.load() with dis-/enabling components.
|
||||
"""
|
||||
@pytest.mark.issue(11443)
|
||||
def test_enable_disable_conflict_with_config():
|
||||
"""Test conflict between enable/disable w.r.t. `nlp.disabled` set in the config."""
|
||||
nlp = English()
|
||||
nlp.add_pipe("tagger")
|
||||
nlp.add_pipe("senter")
|
||||
nlp.add_pipe("sentencizer")
|
||||
|
||||
with make_tempdir() as tmp_dir:
|
||||
nlp.to_disk(tmp_dir)
|
||||
# Expected to fail, as config and arguments conflict.
|
||||
with pytest.raises(ValueError):
|
||||
spacy.load(
|
||||
tmp_dir, enable=["tagger"], config={"nlp": {"disabled": ["senter"]}}
|
||||
)
|
||||
# Expected to succeed without warning due to the lack of a conflicting config option.
|
||||
spacy.load(tmp_dir, enable=["tagger"])
|
||||
# Expected to succeed with a warning, as disable=[] should override the config setting.
|
||||
with pytest.warns(UserWarning):
|
||||
spacy.load(
|
||||
tmp_dir,
|
||||
enable=["tagger"],
|
||||
disable=[],
|
||||
config={"nlp": {"disabled": ["senter"]}},
|
||||
)
|
||||
|
||||
|
||||
def test_load_disable_enable():
|
||||
"""Tests spacy.load() with dis-/enabling components."""
|
||||
|
||||
base_nlp = English()
|
||||
for pipe in ("sentencizer", "tagger", "parser"):
|
||||
|
|
|
|||
|
|
@ -230,6 +230,87 @@ def test_tok2vec_listener_callback():
|
|||
assert get_dX(Y) is not None
|
||||
|
||||
|
||||
def test_tok2vec_listener_overfitting():
|
||||
""" Test that a pipeline with a listener properly overfits, even if 'tok2vec' is in the annotating components """
|
||||
orig_config = Config().from_str(cfg_string)
|
||||
nlp = util.load_model_from_config(orig_config, auto_fill=True, validate=True)
|
||||
train_examples = []
|
||||
for t in TRAIN_DATA:
|
||||
train_examples.append(Example.from_dict(nlp.make_doc(t[0]), t[1]))
|
||||
optimizer = nlp.initialize(get_examples=lambda: train_examples)
|
||||
|
||||
for i in range(50):
|
||||
losses = {}
|
||||
nlp.update(train_examples, sgd=optimizer, losses=losses, annotates=["tok2vec"])
|
||||
assert losses["tagger"] < 0.00001
|
||||
|
||||
# test the trained model
|
||||
test_text = "I like blue eggs"
|
||||
doc = nlp(test_text)
|
||||
assert doc[0].tag_ == "N"
|
||||
assert doc[1].tag_ == "V"
|
||||
assert doc[2].tag_ == "J"
|
||||
assert doc[3].tag_ == "N"
|
||||
|
||||
# Also test the results are still the same after IO
|
||||
with make_tempdir() as tmp_dir:
|
||||
nlp.to_disk(tmp_dir)
|
||||
nlp2 = util.load_model_from_path(tmp_dir)
|
||||
doc2 = nlp2(test_text)
|
||||
assert doc2[0].tag_ == "N"
|
||||
assert doc2[1].tag_ == "V"
|
||||
assert doc2[2].tag_ == "J"
|
||||
assert doc2[3].tag_ == "N"
|
||||
|
||||
|
||||
def test_tok2vec_frozen_not_annotating():
|
||||
""" Test that a pipeline with a frozen tok2vec raises an error when the tok2vec is not annotating """
|
||||
orig_config = Config().from_str(cfg_string)
|
||||
nlp = util.load_model_from_config(orig_config, auto_fill=True, validate=True)
|
||||
train_examples = []
|
||||
for t in TRAIN_DATA:
|
||||
train_examples.append(Example.from_dict(nlp.make_doc(t[0]), t[1]))
|
||||
optimizer = nlp.initialize(get_examples=lambda: train_examples)
|
||||
|
||||
for i in range(2):
|
||||
losses = {}
|
||||
with pytest.raises(ValueError, match=r"the tok2vec embedding layer is not updated"):
|
||||
nlp.update(train_examples, sgd=optimizer, losses=losses, exclude=["tok2vec"])
|
||||
|
||||
|
||||
def test_tok2vec_frozen_overfitting():
|
||||
""" Test that a pipeline with a frozen & annotating tok2vec can still overfit """
|
||||
orig_config = Config().from_str(cfg_string)
|
||||
nlp = util.load_model_from_config(orig_config, auto_fill=True, validate=True)
|
||||
train_examples = []
|
||||
for t in TRAIN_DATA:
|
||||
train_examples.append(Example.from_dict(nlp.make_doc(t[0]), t[1]))
|
||||
optimizer = nlp.initialize(get_examples=lambda: train_examples)
|
||||
|
||||
for i in range(100):
|
||||
losses = {}
|
||||
nlp.update(train_examples, sgd=optimizer, losses=losses, exclude=["tok2vec"], annotates=["tok2vec"])
|
||||
assert losses["tagger"] < 0.0001
|
||||
|
||||
# test the trained model
|
||||
test_text = "I like blue eggs"
|
||||
doc = nlp(test_text)
|
||||
assert doc[0].tag_ == "N"
|
||||
assert doc[1].tag_ == "V"
|
||||
assert doc[2].tag_ == "J"
|
||||
assert doc[3].tag_ == "N"
|
||||
|
||||
# Also test the results are still the same after IO
|
||||
with make_tempdir() as tmp_dir:
|
||||
nlp.to_disk(tmp_dir)
|
||||
nlp2 = util.load_model_from_path(tmp_dir)
|
||||
doc2 = nlp2(test_text)
|
||||
assert doc2[0].tag_ == "N"
|
||||
assert doc2[1].tag_ == "V"
|
||||
assert doc2[2].tag_ == "J"
|
||||
assert doc2[3].tag_ == "N"
|
||||
|
||||
|
||||
def test_replace_listeners():
|
||||
orig_config = Config().from_str(cfg_string)
|
||||
nlp = util.load_model_from_config(orig_config, auto_fill=True, validate=True)
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ from unittest import TestCase
|
|||
import pytest
|
||||
import srsly
|
||||
from numpy import zeros
|
||||
from spacy.kb import KnowledgeBase, Writer
|
||||
from spacy.kb.kb_in_memory import InMemoryLookupKB, Writer
|
||||
from spacy.vectors import Vectors
|
||||
from spacy.language import Language
|
||||
from spacy.pipeline import TrainablePipe
|
||||
|
|
@ -71,7 +71,7 @@ def entity_linker():
|
|||
nlp = Language()
|
||||
|
||||
def create_kb(vocab):
|
||||
kb = KnowledgeBase(vocab, entity_vector_length=1)
|
||||
kb = InMemoryLookupKB(vocab, entity_vector_length=1)
|
||||
kb.add_entity("test", 0.0, zeros((1, 1), dtype="f"))
|
||||
return kb
|
||||
|
||||
|
|
@ -120,7 +120,7 @@ def test_writer_with_path_py35():
|
|||
|
||||
def test_save_and_load_knowledge_base():
|
||||
nlp = Language()
|
||||
kb = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
kb = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
with make_tempdir() as d:
|
||||
path = d / "kb"
|
||||
try:
|
||||
|
|
@ -129,7 +129,7 @@ def test_save_and_load_knowledge_base():
|
|||
pytest.fail(str(e))
|
||||
|
||||
try:
|
||||
kb_loaded = KnowledgeBase(nlp.vocab, entity_vector_length=1)
|
||||
kb_loaded = InMemoryLookupKB(nlp.vocab, entity_vector_length=1)
|
||||
kb_loaded.from_disk(path)
|
||||
except Exception as e:
|
||||
pytest.fail(str(e))
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ from typing import Callable
|
|||
|
||||
from spacy import util
|
||||
from spacy.util import ensure_path, registry, load_model_from_config
|
||||
from spacy.kb import KnowledgeBase
|
||||
from spacy.kb.kb_in_memory import InMemoryLookupKB
|
||||
from spacy.vocab import Vocab
|
||||
from thinc.api import Config
|
||||
|
||||
|
|
@ -22,7 +22,7 @@ def test_serialize_kb_disk(en_vocab):
|
|||
dir_path.mkdir()
|
||||
file_path = dir_path / "kb"
|
||||
kb1.to_disk(str(file_path))
|
||||
kb2 = KnowledgeBase(vocab=en_vocab, entity_vector_length=3)
|
||||
kb2 = InMemoryLookupKB(vocab=en_vocab, entity_vector_length=3)
|
||||
kb2.from_disk(str(file_path))
|
||||
|
||||
# final assertions
|
||||
|
|
@ -30,7 +30,7 @@ def test_serialize_kb_disk(en_vocab):
|
|||
|
||||
|
||||
def _get_dummy_kb(vocab):
|
||||
kb = KnowledgeBase(vocab, entity_vector_length=3)
|
||||
kb = InMemoryLookupKB(vocab, entity_vector_length=3)
|
||||
kb.add_entity(entity="Q53", freq=33, entity_vector=[0, 5, 3])
|
||||
kb.add_entity(entity="Q17", freq=2, entity_vector=[7, 1, 0])
|
||||
kb.add_entity(entity="Q007", freq=7, entity_vector=[0, 0, 7])
|
||||
|
|
@ -104,7 +104,7 @@ def test_serialize_subclassed_kb():
|
|||
custom_field = 666
|
||||
"""
|
||||
|
||||
class SubKnowledgeBase(KnowledgeBase):
|
||||
class SubInMemoryLookupKB(InMemoryLookupKB):
|
||||
def __init__(self, vocab, entity_vector_length, custom_field):
|
||||
super().__init__(vocab, entity_vector_length)
|
||||
self.custom_field = custom_field
|
||||
|
|
@ -112,9 +112,9 @@ def test_serialize_subclassed_kb():
|
|||
@registry.misc("spacy.CustomKB.v1")
|
||||
def custom_kb(
|
||||
entity_vector_length: int, custom_field: int
|
||||
) -> Callable[[Vocab], KnowledgeBase]:
|
||||
) -> Callable[[Vocab], InMemoryLookupKB]:
|
||||
def custom_kb_factory(vocab):
|
||||
kb = SubKnowledgeBase(
|
||||
kb = SubInMemoryLookupKB(
|
||||
vocab=vocab,
|
||||
entity_vector_length=entity_vector_length,
|
||||
custom_field=custom_field,
|
||||
|
|
@ -129,7 +129,7 @@ def test_serialize_subclassed_kb():
|
|||
nlp.initialize()
|
||||
|
||||
entity_linker = nlp.get_pipe("entity_linker")
|
||||
assert type(entity_linker.kb) == SubKnowledgeBase
|
||||
assert type(entity_linker.kb) == SubInMemoryLookupKB
|
||||
assert entity_linker.kb.entity_vector_length == 342
|
||||
assert entity_linker.kb.custom_field == 666
|
||||
|
||||
|
|
@ -139,6 +139,6 @@ def test_serialize_subclassed_kb():
|
|||
nlp2 = util.load_model_from_path(tmp_dir)
|
||||
entity_linker2 = nlp2.get_pipe("entity_linker")
|
||||
# After IO, the KB is the standard one
|
||||
assert type(entity_linker2.kb) == KnowledgeBase
|
||||
assert type(entity_linker2.kb) == InMemoryLookupKB
|
||||
assert entity_linker2.kb.entity_vector_length == 342
|
||||
assert not hasattr(entity_linker2.kb, "custom_field")
|
||||
|
|
|
|||
|
|
@ -404,10 +404,11 @@ def test_serialize_pipeline_disable_enable():
|
|||
assert nlp3.component_names == ["ner", "tagger"]
|
||||
with make_tempdir() as d:
|
||||
nlp3.to_disk(d)
|
||||
with pytest.warns(UserWarning):
|
||||
nlp4 = spacy.load(d, disable=["ner"])
|
||||
assert nlp4.pipe_names == []
|
||||
assert nlp4.pipe_names == ["tagger"]
|
||||
assert nlp4.component_names == ["ner", "tagger"]
|
||||
assert nlp4.disabled == ["ner", "tagger"]
|
||||
assert nlp4.disabled == ["ner"]
|
||||
with make_tempdir() as d:
|
||||
nlp.to_disk(d)
|
||||
nlp5 = spacy.load(d, exclude=["tagger"])
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ def doc(nlp):
|
|||
words = ["Sarah", "'s", "sister", "flew", "to", "Silicon", "Valley", "via", "London", "."]
|
||||
tags = ["NNP", "POS", "NN", "VBD", "IN", "NNP", "NNP", "IN", "NNP", "."]
|
||||
pos = ["PROPN", "PART", "NOUN", "VERB", "ADP", "PROPN", "PROPN", "ADP", "PROPN", "PUNCT"]
|
||||
ents = ["B-PERSON", "I-PERSON", "O", "O", "O", "B-LOC", "I-LOC", "O", "B-GPE", "O"]
|
||||
ents = ["B-PERSON", "I-PERSON", "O", "", "O", "B-LOC", "I-LOC", "O", "B-GPE", "O"]
|
||||
cats = {"TRAVEL": 1.0, "BAKING": 0.0}
|
||||
# fmt: on
|
||||
doc = Doc(nlp.vocab, words=words, tags=tags, pos=pos, ents=ents)
|
||||
|
|
@ -106,6 +106,7 @@ def test_lowercase_augmenter(nlp, doc):
|
|||
assert [(e.start, e.end, e.label) for e in eg.reference.ents] == ents
|
||||
for ref_ent, orig_ent in zip(eg.reference.ents, doc.ents):
|
||||
assert ref_ent.text == orig_ent.text.lower()
|
||||
assert [t.ent_iob for t in doc] == [t.ent_iob for t in eg.reference]
|
||||
assert [t.pos_ for t in eg.reference] == [t.pos_ for t in doc]
|
||||
|
||||
# check that augmentation works when lowercasing leads to different
|
||||
|
|
@ -166,7 +167,7 @@ def test_make_whitespace_variant(nlp):
|
|||
lemmas = ["they", "fly", "to", "New", "York", "City", ".", "\n", "then", "they", "drive", "to", "Washington", ",", "D.C."]
|
||||
heads = [1, 1, 1, 4, 5, 2, 1, 10, 10, 10, 10, 10, 11, 12, 12]
|
||||
deps = ["nsubj", "ROOT", "prep", "compound", "compound", "pobj", "punct", "dep", "advmod", "nsubj", "ROOT", "prep", "pobj", "punct", "appos"]
|
||||
ents = ["O", "O", "O", "B-GPE", "I-GPE", "I-GPE", "O", "O", "O", "O", "O", "O", "B-GPE", "O", "B-GPE"]
|
||||
ents = ["O", "", "O", "B-GPE", "I-GPE", "I-GPE", "O", "O", "O", "O", "O", "O", "B-GPE", "O", "B-GPE"]
|
||||
# fmt: on
|
||||
doc = Doc(
|
||||
nlp.vocab,
|
||||
|
|
@ -215,6 +216,8 @@ def test_make_whitespace_variant(nlp):
|
|||
assert mod_ex2.reference[j].head.i == j - 1
|
||||
# entities are well-formed
|
||||
assert len(doc.ents) == len(mod_ex.reference.ents)
|
||||
# there is one token with missing entity information
|
||||
assert any(t.ent_iob == 0 for t in mod_ex.reference)
|
||||
for ent in mod_ex.reference.ents:
|
||||
assert not ent[0].is_space
|
||||
assert not ent[-1].is_space
|
||||
|
|
|
|||
|
|
@ -73,7 +73,7 @@ class Doc:
|
|||
lemmas: Optional[List[str]] = ...,
|
||||
heads: Optional[List[int]] = ...,
|
||||
deps: Optional[List[str]] = ...,
|
||||
sent_starts: Optional[List[Union[bool, None]]] = ...,
|
||||
sent_starts: Optional[List[Union[bool, int, None]]] = ...,
|
||||
ents: Optional[List[str]] = ...,
|
||||
) -> None: ...
|
||||
@property
|
||||
|
|
|
|||
|
|
@ -217,9 +217,9 @@ cdef class Doc:
|
|||
head in the doc. Defaults to None.
|
||||
deps (Optional[List[str]]): A list of unicode strings, of the same
|
||||
length as words, to assign as token.dep. Defaults to None.
|
||||
sent_starts (Optional[List[Union[bool, None]]]): A list of values, of
|
||||
the same length as words, to assign as token.is_sent_start. Will be
|
||||
overridden by heads if heads is provided. Defaults to None.
|
||||
sent_starts (Optional[List[Union[bool, int, None]]]): A list of values,
|
||||
of the same length as words, to assign as token.is_sent_start. Will
|
||||
be overridden by heads if heads is provided. Defaults to None.
|
||||
ents (Optional[List[str]]): A list of unicode strings, of the same
|
||||
length as words, as IOB tags to assign as token.ent_iob and
|
||||
token.ent_type. Defaults to None.
|
||||
|
|
@ -286,6 +286,7 @@ cdef class Doc:
|
|||
heads = [0] * len(deps)
|
||||
if heads and not deps:
|
||||
raise ValueError(Errors.E1017)
|
||||
sent_starts = list(sent_starts) if sent_starts is not None else None
|
||||
if sent_starts is not None:
|
||||
for i in range(len(sent_starts)):
|
||||
if sent_starts[i] is True:
|
||||
|
|
@ -301,12 +302,11 @@ cdef class Doc:
|
|||
ent_iobs = None
|
||||
ent_types = None
|
||||
if ents is not None:
|
||||
ents = [ent if ent != "" else None for ent in ents]
|
||||
iob_strings = Token.iob_strings()
|
||||
# make valid IOB2 out of IOB1 or IOB2
|
||||
for i, ent in enumerate(ents):
|
||||
if ent is "":
|
||||
ents[i] = None
|
||||
elif ent is not None and not isinstance(ent, str):
|
||||
if ent is not None and not isinstance(ent, str):
|
||||
raise ValueError(Errors.E177.format(tag=ent))
|
||||
if i < len(ents) - 1:
|
||||
# OI -> OB
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ from functools import partial
|
|||
|
||||
from ..util import registry
|
||||
from .example import Example
|
||||
from .iob_utils import split_bilu_label
|
||||
from .iob_utils import split_bilu_label, _doc_to_biluo_tags_with_partial
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..language import Language # noqa: F401
|
||||
|
|
@ -62,6 +62,9 @@ def combined_augmenter(
|
|||
if orth_variants and random.random() < orth_level:
|
||||
raw_text = example.text
|
||||
orig_dict = example.to_dict()
|
||||
orig_dict["doc_annotation"]["entities"] = _doc_to_biluo_tags_with_partial(
|
||||
example.reference
|
||||
)
|
||||
variant_text, variant_token_annot = make_orth_variants(
|
||||
nlp,
|
||||
raw_text,
|
||||
|
|
@ -128,6 +131,9 @@ def lower_casing_augmenter(
|
|||
|
||||
def make_lowercase_variant(nlp: "Language", example: Example):
|
||||
example_dict = example.to_dict()
|
||||
example_dict["doc_annotation"]["entities"] = _doc_to_biluo_tags_with_partial(
|
||||
example.reference
|
||||
)
|
||||
doc = nlp.make_doc(example.text.lower())
|
||||
example_dict["token_annotation"]["ORTH"] = [t.lower_ for t in example.reference]
|
||||
return example.from_dict(doc, example_dict)
|
||||
|
|
@ -146,6 +152,9 @@ def orth_variants_augmenter(
|
|||
else:
|
||||
raw_text = example.text
|
||||
orig_dict = example.to_dict()
|
||||
orig_dict["doc_annotation"]["entities"] = _doc_to_biluo_tags_with_partial(
|
||||
example.reference
|
||||
)
|
||||
variant_text, variant_token_annot = make_orth_variants(
|
||||
nlp,
|
||||
raw_text,
|
||||
|
|
@ -248,6 +257,9 @@ def make_whitespace_variant(
|
|||
RETURNS (Example): Example with one additional space token.
|
||||
"""
|
||||
example_dict = example.to_dict()
|
||||
example_dict["doc_annotation"]["entities"] = _doc_to_biluo_tags_with_partial(
|
||||
example.reference
|
||||
)
|
||||
doc_dict = example_dict.get("doc_annotation", {})
|
||||
token_dict = example_dict.get("token_annotation", {})
|
||||
# returned unmodified if:
|
||||
|
|
|
|||
|
|
@ -60,6 +60,14 @@ def doc_to_biluo_tags(doc: Doc, missing: str = "O"):
|
|||
)
|
||||
|
||||
|
||||
def _doc_to_biluo_tags_with_partial(doc: Doc) -> List[str]:
|
||||
ents = doc_to_biluo_tags(doc, missing="-")
|
||||
for i, token in enumerate(doc):
|
||||
if token.ent_iob == 2:
|
||||
ents[i] = "O"
|
||||
return ents
|
||||
|
||||
|
||||
def offsets_to_biluo_tags(
|
||||
doc: Doc, entities: Iterable[Tuple[int, int, Union[str, int]]], missing: str = "O"
|
||||
) -> List[str]:
|
||||
|
|
|
|||
|
|
@ -67,7 +67,6 @@ LEXEME_NORM_LANGS = ["cs", "da", "de", "el", "en", "id", "lb", "mk", "pt", "ru",
|
|||
CONFIG_SECTION_ORDER = ["paths", "variables", "system", "nlp", "components", "corpora", "training", "pretraining", "initialize"]
|
||||
# fmt: on
|
||||
|
||||
|
||||
logger = logging.getLogger("spacy")
|
||||
logger_stream_handler = logging.StreamHandler()
|
||||
logger_stream_handler.setFormatter(
|
||||
|
|
@ -394,13 +393,17 @@ def get_module_path(module: ModuleType) -> Path:
|
|||
return file_path.parent
|
||||
|
||||
|
||||
# Default value for passed enable/disable values.
|
||||
_DEFAULT_EMPTY_PIPES = SimpleFrozenList()
|
||||
|
||||
|
||||
def load_model(
|
||||
name: Union[str, Path],
|
||||
*,
|
||||
vocab: Union["Vocab", bool] = True,
|
||||
disable: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
enable: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
exclude: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
disable: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
enable: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
exclude: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
config: Union[Dict[str, Any], Config] = SimpleFrozenDict(),
|
||||
) -> "Language":
|
||||
"""Load a model from a package or data path.
|
||||
|
|
@ -470,9 +473,9 @@ def load_model_from_path(
|
|||
*,
|
||||
meta: Optional[Dict[str, Any]] = None,
|
||||
vocab: Union["Vocab", bool] = True,
|
||||
disable: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
enable: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
exclude: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
disable: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
enable: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
exclude: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
config: Union[Dict[str, Any], Config] = SimpleFrozenDict(),
|
||||
) -> "Language":
|
||||
"""Load a model from a data directory path. Creates Language class with
|
||||
|
|
@ -516,9 +519,9 @@ def load_model_from_config(
|
|||
*,
|
||||
meta: Dict[str, Any] = SimpleFrozenDict(),
|
||||
vocab: Union["Vocab", bool] = True,
|
||||
disable: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
enable: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
exclude: Union[str, Iterable[str]] = SimpleFrozenList(),
|
||||
disable: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
enable: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
exclude: Union[str, Iterable[str]] = _DEFAULT_EMPTY_PIPES,
|
||||
auto_fill: bool = False,
|
||||
validate: bool = True,
|
||||
) -> "Language":
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ menu:
|
|||
- ['Text Classification', 'textcat']
|
||||
- ['Span Classification', 'spancat']
|
||||
- ['Entity Linking', 'entitylinker']
|
||||
- ['Coreference', 'coref-architectures']
|
||||
---
|
||||
|
||||
A **model architecture** is a function that wires up a
|
||||
|
|
@ -587,8 +588,8 @@ consists of either two or three subnetworks:
|
|||
run once for each batch.
|
||||
- **lower**: Construct a feature-specific vector for each `(token, feature)`
|
||||
pair. This is also run once for each batch. Constructing the state
|
||||
representation is then a matter of summing the component features and
|
||||
applying the non-linearity.
|
||||
representation is then a matter of summing the component features and applying
|
||||
the non-linearity.
|
||||
- **upper** (optional): A feed-forward network that predicts scores from the
|
||||
state representation. If not present, the output from the lower model is used
|
||||
as action scores directly.
|
||||
|
|
@ -628,8 +629,8 @@ same signature, but the `use_upper` argument was `True` by default.
|
|||
> ```
|
||||
|
||||
Build a tagger model, using a provided token-to-vector component. The tagger
|
||||
model adds a linear layer with softmax activation to predict scores given
|
||||
the token vectors.
|
||||
model adds a linear layer with softmax activation to predict scores given the
|
||||
token vectors.
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ------------------------------------------------------------------------------------------ |
|
||||
|
|
@ -920,5 +921,84 @@ A function that reads an existing `KnowledgeBase` from file.
|
|||
A function that takes as input a [`KnowledgeBase`](/api/kb) and a
|
||||
[`Span`](/api/span) object denoting a named entity, and returns a list of
|
||||
plausible [`Candidate`](/api/kb/#candidate) objects. The default
|
||||
`CandidateGenerator` uses the text of a mention to find its potential
|
||||
aliases in the `KnowledgeBase`. Note that this function is case-dependent.
|
||||
`CandidateGenerator` uses the text of a mention to find its potential aliases in
|
||||
the `KnowledgeBase`. Note that this function is case-dependent.
|
||||
|
||||
## Coreference {#coref-architectures tag="experimental"}
|
||||
|
||||
A [`CoreferenceResolver`](/api/coref) component identifies tokens that refer to
|
||||
the same entity. A [`SpanResolver`](/api/span-resolver) component infers spans
|
||||
from single tokens. Together these components can be used to reproduce
|
||||
traditional coreference models. You can also omit the `SpanResolver` if working
|
||||
with only token-level clusters is acceptable.
|
||||
|
||||
### spacy-experimental.Coref.v1 {#Coref tag="experimental"}
|
||||
|
||||
> #### Example Config
|
||||
>
|
||||
> ```ini
|
||||
>
|
||||
> [model]
|
||||
> @architectures = "spacy-experimental.Coref.v1"
|
||||
> distance_embedding_size = 20
|
||||
> dropout = 0.3
|
||||
> hidden_size = 1024
|
||||
> depth = 2
|
||||
> antecedent_limit = 50
|
||||
> antecedent_batch_size = 512
|
||||
>
|
||||
> [model.tok2vec]
|
||||
> @architectures = "spacy-transformers.TransformerListener.v1"
|
||||
> grad_factor = 1.0
|
||||
> upstream = "transformer"
|
||||
> pooling = {"@layers":"reduce_mean.v1"}
|
||||
> ```
|
||||
|
||||
The `Coref` model architecture is a Thinc `Model`.
|
||||
|
||||
| Name | Description |
|
||||
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `tok2vec` | The [`tok2vec`](#tok2vec) layer of the model. ~~Model~~ |
|
||||
| `distance_embedding_size` | A representation of the distance between candidates. ~~int~~ |
|
||||
| `dropout` | The dropout to use internally. Unlike some Thinc models, this has separate dropout for the internal PyTorch layers. ~~float~~ |
|
||||
| `hidden_size` | Size of the main internal layers. ~~int~~ |
|
||||
| `depth` | Depth of the internal network. ~~int~~ |
|
||||
| `antecedent_limit` | How many candidate antecedents to keep after rough scoring. This has a significant effect on memory usage. Typical values would be 50 to 200, or higher for very long documents. ~~int~~ |
|
||||
| `antecedent_batch_size` | Internal batch size. ~~int~~ |
|
||||
| **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ |
|
||||
|
||||
### spacy-experimental.SpanResolver.v1 {#SpanResolver tag="experimental"}
|
||||
|
||||
> #### Example Config
|
||||
>
|
||||
> ```ini
|
||||
>
|
||||
> [model]
|
||||
> @architectures = "spacy-experimental.SpanResolver.v1"
|
||||
> hidden_size = 1024
|
||||
> distance_embedding_size = 64
|
||||
> conv_channels = 4
|
||||
> window_size = 1
|
||||
> max_distance = 128
|
||||
> prefix = "coref_head_clusters"
|
||||
>
|
||||
> [model.tok2vec]
|
||||
> @architectures = "spacy-transformers.TransformerListener.v1"
|
||||
> grad_factor = 1.0
|
||||
> upstream = "transformer"
|
||||
> pooling = {"@layers":"reduce_mean.v1"}
|
||||
> ```
|
||||
|
||||
The `SpanResolver` model architecture is a Thinc `Model`. Note that
|
||||
`MentionClusters` is `List[List[Tuple[int, int]]]`.
|
||||
|
||||
| Name | Description |
|
||||
| ------------------------- | -------------------------------------------------------------------------------------------------------------------- |
|
||||
| `tok2vec` | The [`tok2vec`](#tok2vec) layer of the model. ~~Model~~ |
|
||||
| `hidden_size` | Size of the main internal layers. ~~int~~ |
|
||||
| `distance_embedding_size` | A representation of the distance between two candidates. ~~int~~ |
|
||||
| `conv_channels` | The number of channels in the internal CNN. ~~int~~ |
|
||||
| `window_size` | The number of neighboring tokens to consider in the internal CNN. `1` means consider one token on each side. ~~int~~ |
|
||||
| `max_distance` | The longest possible length of a predicted span. ~~int~~ |
|
||||
| `prefix` | The prefix that indicates spans to use for input data. ~~string~~ |
|
||||
| **CREATES** | The model using the architecture. ~~Model[List[Doc], List[MentionClusters]]~~ |
|
||||
|
|
|
|||
353
website/docs/api/coref.md
Normal file
353
website/docs/api/coref.md
Normal file
|
|
@ -0,0 +1,353 @@
|
|||
---
|
||||
title: CoreferenceResolver
|
||||
tag: class,experimental
|
||||
source: spacy-experimental/coref/coref_component.py
|
||||
teaser: 'Pipeline component for word-level coreference resolution'
|
||||
api_base_class: /api/pipe
|
||||
api_string_name: coref
|
||||
api_trainable: true
|
||||
---
|
||||
|
||||
> #### Installation
|
||||
>
|
||||
> ```bash
|
||||
> $ pip install -U spacy-experimental
|
||||
> ```
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
This component is not yet integrated into spaCy core, and is available via the
|
||||
extension package
|
||||
[`spacy-experimental`](https://github.com/explosion/spacy-experimental) starting
|
||||
in version 0.6.0. It exposes the component via
|
||||
[entry points](/usage/saving-loading/#entry-points), so if you have the package
|
||||
installed, using `factory = "experimental_coref"` in your
|
||||
[training config](/usage/training#config) or
|
||||
`nlp.add_pipe("experimental_coref")` will work out-of-the-box.
|
||||
|
||||
</Infobox>
|
||||
|
||||
A `CoreferenceResolver` component groups tokens into clusters that refer to the
|
||||
same thing. Clusters are represented as SpanGroups that start with a prefix
|
||||
(`coref_clusters` by default).
|
||||
|
||||
A `CoreferenceResolver` component can be paired with a
|
||||
[`SpanResolver`](/api/span-resolver) to expand single tokens to spans.
|
||||
|
||||
## Assigned Attributes {#assigned-attributes}
|
||||
|
||||
Predictions will be saved to `Doc.spans` as a [`SpanGroup`](/api/spangroup). The
|
||||
span key will be a prefix plus a serial number referring to the coreference
|
||||
cluster, starting from zero.
|
||||
|
||||
The span key prefix defaults to `"coref_clusters"`, but can be passed as a
|
||||
parameter.
|
||||
|
||||
| Location | Value |
|
||||
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
|
||||
| `Doc.spans[prefix + "_" + cluster_number]` | One coreference cluster, represented as single-token spans. Cluster numbers start from 1. ~~SpanGroup~~ |
|
||||
|
||||
## Config and implementation {#config}
|
||||
|
||||
The default config is defined by the pipeline component factory and describes
|
||||
how the component should be configured. You can override its settings via the
|
||||
`config` argument on [`nlp.add_pipe`](/api/language#add_pipe) or in your
|
||||
[`config.cfg` for training](/usage/training#config). See the
|
||||
[model architectures](/api/architectures#coref-architectures) documentation for
|
||||
details on the architectures and their arguments and hyperparameters.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy_experimental.coref.coref_component import DEFAULT_COREF_MODEL
|
||||
> from spacy_experimental.coref.coref_util import DEFAULT_CLUSTER_PREFIX
|
||||
> config={
|
||||
> "model": DEFAULT_COREF_MODEL,
|
||||
> "span_cluster_prefix": DEFAULT_CLUSTER_PREFIX,
|
||||
> },
|
||||
> nlp.add_pipe("experimental_coref", config=config)
|
||||
> ```
|
||||
|
||||
| Setting | Description |
|
||||
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `model` | The [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. Defaults to [Coref](/api/architectures#Coref). ~~Model~~ |
|
||||
| `span_cluster_prefix` | The prefix for the keys for clusters saved to `doc.spans`. Defaults to `coref_clusters`. ~~str~~ |
|
||||
|
||||
## CoreferenceResolver.\_\_init\_\_ {#init tag="method"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> # Construction via add_pipe with default model
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
>
|
||||
> # Construction via add_pipe with custom model
|
||||
> config = {"model": {"@architectures": "my_coref.v1"}}
|
||||
> coref = nlp.add_pipe("experimental_coref", config=config)
|
||||
>
|
||||
> # Construction from class
|
||||
> from spacy_experimental.coref.coref_component import CoreferenceResolver
|
||||
> coref = CoreferenceResolver(nlp.vocab, model)
|
||||
> ```
|
||||
|
||||
Create a new pipeline instance. In your application, you would normally use a
|
||||
shortcut for this and instantiate the component using its string name and
|
||||
[`nlp.add_pipe`](/api/language#add_pipe).
|
||||
|
||||
| Name | Description |
|
||||
| --------------------- | --------------------------------------------------------------------------------------------------- |
|
||||
| `vocab` | The shared vocabulary. ~~Vocab~~ |
|
||||
| `model` | The [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. ~~Model~~ |
|
||||
| `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ |
|
||||
| _keyword-only_ | |
|
||||
| `span_cluster_prefix` | The prefix for the key for saving clusters of spans. ~~bool~~ |
|
||||
|
||||
## CoreferenceResolver.\_\_call\_\_ {#call tag="method"}
|
||||
|
||||
Apply the pipe to one document. The document is modified in place and returned.
|
||||
This usually happens under the hood when the `nlp` object is called on a text
|
||||
and all pipeline components are applied to the `Doc` in order. Both
|
||||
[`__call__`](/api/coref#call) and [`pipe`](/api/coref#pipe) delegate to the
|
||||
[`predict`](/api/coref#predict) and
|
||||
[`set_annotations`](/api/coref#set_annotations) methods.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> doc = nlp("This is a sentence.")
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> # This usually happens under the hood
|
||||
> processed = coref(doc)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | -------------------------------- |
|
||||
| `doc` | The document to process. ~~Doc~~ |
|
||||
| **RETURNS** | The processed document. ~~Doc~~ |
|
||||
|
||||
## CoreferenceResolver.pipe {#pipe tag="method"}
|
||||
|
||||
Apply the pipe to a stream of documents. This usually happens under the hood
|
||||
when the `nlp` object is called on a text and all pipeline components are
|
||||
applied to the `Doc` in order. Both [`__call__`](/api/coref#call) and
|
||||
[`pipe`](/api/coref#pipe) delegate to the [`predict`](/api/coref#predict) and
|
||||
[`set_annotations`](/api/coref#set_annotations) methods.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> for doc in coref.pipe(docs, batch_size=50):
|
||||
> pass
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ------------------------------------------------------------- |
|
||||
| `stream` | A stream of documents. ~~Iterable[Doc]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
|
||||
| **YIELDS** | The processed documents in order. ~~Doc~~ |
|
||||
|
||||
## CoreferenceResolver.initialize {#initialize tag="method"}
|
||||
|
||||
Initialize the component for training. `get_examples` should be a function that
|
||||
returns an iterable of [`Example`](/api/example) objects. **At least one example
|
||||
should be supplied.** The data examples are used to **initialize the model** of
|
||||
the component and can either be the full training data or a representative
|
||||
sample. Initialization includes validating the network,
|
||||
[inferring missing shapes](https://thinc.ai/docs/usage-models#validation) and
|
||||
setting up the label scheme based on the data. This method is typically called
|
||||
by [`Language.initialize`](/api/language#initialize).
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> coref.initialize(lambda: examples, nlp=nlp)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `get_examples` | Function that returns gold-standard annotations in the form of [`Example`](/api/example) objects. Must contain at least one `Example`. ~~Callable[[], Iterable[Example]]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
|
||||
|
||||
## CoreferenceResolver.predict {#predict tag="method"}
|
||||
|
||||
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
|
||||
modifying them. Clusters are returned as a list of `MentionClusters`, one for
|
||||
each input `Doc`. A `MentionClusters` instance is just a list of lists of pairs
|
||||
of `int`s, where each item corresponds to a cluster, and the `int`s correspond
|
||||
to token indices.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> clusters = coref.predict([doc1, doc2])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ---------------------------------------------------------------------------- |
|
||||
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
|
||||
| **RETURNS** | The predicted coreference clusters for the `docs`. ~~List[MentionClusters]~~ |
|
||||
|
||||
## CoreferenceResolver.set_annotations {#set_annotations tag="method"}
|
||||
|
||||
Modify a batch of documents, saving coreference clusters in `Doc.spans`.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> clusters = coref.predict([doc1, doc2])
|
||||
> coref.set_annotations([doc1, doc2], clusters)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ---------- | ---------------------------------------------------------------------------- |
|
||||
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
|
||||
| `clusters` | The predicted coreference clusters for the `docs`. ~~List[MentionClusters]~~ |
|
||||
|
||||
## CoreferenceResolver.update {#update tag="method"}
|
||||
|
||||
Learn from a batch of [`Example`](/api/example) objects. Delegates to
|
||||
[`predict`](/api/coref#predict).
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> optimizer = nlp.initialize()
|
||||
> losses = coref.update(examples, sgd=optimizer)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `examples` | A batch of [`Example`](/api/example) objects to learn from. ~~Iterable[Example]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `drop` | The dropout rate. ~~float~~ |
|
||||
| `sgd` | An optimizer. Will be created via [`create_optimizer`](#create_optimizer) if not set. ~~Optional[Optimizer]~~ |
|
||||
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
|
||||
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
|
||||
|
||||
## CoreferenceResolver.create_optimizer {#create_optimizer tag="method"}
|
||||
|
||||
Create an optimizer for the pipeline component.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> optimizer = coref.create_optimizer()
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ---------------------------- |
|
||||
| **RETURNS** | The optimizer. ~~Optimizer~~ |
|
||||
|
||||
## CoreferenceResolver.use_params {#use_params tag="method, contextmanager"}
|
||||
|
||||
Modify the pipe's model, to use the given parameter values. At the end of the
|
||||
context, the original parameters are restored.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> with coref.use_params(optimizer.averages):
|
||||
> coref.to_disk("/best_model")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------- | -------------------------------------------------- |
|
||||
| `params` | The parameter values to use in the model. ~~dict~~ |
|
||||
|
||||
## CoreferenceResolver.to_disk {#to_disk tag="method"}
|
||||
|
||||
Serialize the pipe to disk.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> coref.to_disk("/path/to/coref")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
|
||||
|
||||
## CoreferenceResolver.from_disk {#from_disk tag="method"}
|
||||
|
||||
Load the pipe from disk. Modifies the object in place and returns it.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> coref.from_disk("/path/to/coref")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
|
||||
| **RETURNS** | The modified `CoreferenceResolver` object. ~~CoreferenceResolver~~ |
|
||||
|
||||
## CoreferenceResolver.to_bytes {#to_bytes tag="method"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> coref_bytes = coref.to_bytes()
|
||||
> ```
|
||||
|
||||
Serialize the pipe to a bytestring, including the `KnowledgeBase`.
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ------------------------------------------------------------------------------------------- |
|
||||
| _keyword-only_ | |
|
||||
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
|
||||
| **RETURNS** | The serialized form of the `CoreferenceResolver` object. ~~bytes~~ |
|
||||
|
||||
## CoreferenceResolver.from_bytes {#from_bytes tag="method"}
|
||||
|
||||
Load the pipe from a bytestring. Modifies the object in place and returns it.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> coref_bytes = coref.to_bytes()
|
||||
> coref = nlp.add_pipe("experimental_coref")
|
||||
> coref.from_bytes(coref_bytes)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ------------------------------------------------------------------------------------------- |
|
||||
| `bytes_data` | The data to load from. ~~bytes~~ |
|
||||
| _keyword-only_ | |
|
||||
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
|
||||
| **RETURNS** | The `CoreferenceResolver` object. ~~CoreferenceResolver~~ |
|
||||
|
||||
## Serialization fields {#serialization-fields}
|
||||
|
||||
During serialization, spaCy will export several data fields used to restore
|
||||
different aspects of the object. If needed, you can exclude them from
|
||||
serialization by passing in the string names via the `exclude` argument.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> data = coref.to_disk("/path", exclude=["vocab"])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ------- | -------------------------------------------------------------- |
|
||||
| `vocab` | The shared [`Vocab`](/api/vocab). |
|
||||
| `cfg` | The config file. You usually don't want to exclude this. |
|
||||
| `model` | The binary model data. You usually don't want to exclude this. |
|
||||
|
|
@ -32,7 +32,7 @@ Construct a `Doc` object. The most common way to get a `Doc` object is via the
|
|||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `vocab` | A storage container for lexical types. ~~Vocab~~ |
|
||||
| `words` | A list of strings or integer hash values to add to the document as words. ~~Optional[List[Union[str,int]]]~~ |
|
||||
| `spaces` | A list of boolean values indicating whether each word has a subsequent space. Must have the same length as `words`, if specified. Defaults to a sequence of `True`. ~~Optional[List[bool]]~~ |
|
||||
|
|
@ -44,7 +44,7 @@ Construct a `Doc` object. The most common way to get a `Doc` object is via the
|
|||
| `lemmas` <Tag variant="new">3</Tag> | A list of strings, of the same length as `words`, to assign as `token.lemma` for each word. Defaults to `None`. ~~Optional[List[str]]~~ |
|
||||
| `heads` <Tag variant="new">3</Tag> | A list of values, of the same length as `words`, to assign as the head for each word. Head indices are the absolute position of the head in the `Doc`. Defaults to `None`. ~~Optional[List[int]]~~ |
|
||||
| `deps` <Tag variant="new">3</Tag> | A list of strings, of the same length as `words`, to assign as `token.dep` for each word. Defaults to `None`. ~~Optional[List[str]]~~ |
|
||||
| `sent_starts` <Tag variant="new">3</Tag> | A list of values, of the same length as `words`, to assign as `token.is_sent_start`. Will be overridden by heads if `heads` is provided. Defaults to `None`. ~~Optional[List[Optional[bool]]]~~ |
|
||||
| `sent_starts` <Tag variant="new">3</Tag> | A list of values, of the same length as `words`, to assign as `token.is_sent_start`. Will be overridden by heads if `heads` is provided. Defaults to `None`. ~~Optional[List[Union[bool, int, None]]]~~ |
|
||||
| `ents` <Tag variant="new">3</Tag> | A list of strings, of the same length of `words`, to assign the token-based IOB tag. Defaults to `None`. ~~Optional[List[str]]~~ |
|
||||
|
||||
## Doc.\_\_getitem\_\_ {#getitem tag="method"}
|
||||
|
|
|
|||
|
|
@ -14,7 +14,8 @@ entities) to unique identifiers, grounding the named entities into the "real
|
|||
world". It requires a `KnowledgeBase`, as well as a function to generate
|
||||
plausible candidates from that `KnowledgeBase` given a certain textual mention,
|
||||
and a machine learning model to pick the right candidate, given the local
|
||||
context of the mention.
|
||||
context of the mention. `EntityLinker` defaults to using the
|
||||
[`InMemoryLookupKB`](/api/kb_in_memory) implementation.
|
||||
|
||||
## Assigned Attributes {#assigned-attributes}
|
||||
|
||||
|
|
@ -171,7 +172,7 @@ with the current vocab.
|
|||
>
|
||||
> ```python
|
||||
> def create_kb(vocab):
|
||||
> kb = KnowledgeBase(vocab, entity_vector_length=128)
|
||||
> kb = InMemoryLookupKB(vocab, entity_vector_length=128)
|
||||
> kb.add_entity(...)
|
||||
> kb.add_alias(...)
|
||||
> return kb
|
||||
|
|
|
|||
|
|
@ -23,11 +23,13 @@ both documents.
|
|||
> ```python
|
||||
> from spacy.tokens import Doc
|
||||
> from spacy.training import Example
|
||||
>
|
||||
> words = ["hello", "world", "!"]
|
||||
> spaces = [True, False, False]
|
||||
> predicted = Doc(nlp.vocab, words=words, spaces=spaces)
|
||||
> reference = parse_gold_doc(my_data)
|
||||
> pred_words = ["Apply", "some", "sunscreen"]
|
||||
> pred_spaces = [True, True, False]
|
||||
> gold_words = ["Apply", "some", "sun", "screen"]
|
||||
> gold_spaces = [True, True, False, False]
|
||||
> gold_tags = ["VERB", "DET", "NOUN", "NOUN"]
|
||||
> predicted = Doc(nlp.vocab, words=pred_words, spaces=pred_spaces)
|
||||
> reference = Doc(nlp.vocab, words=gold_words, spaces=gold_spaces, tags=gold_tags)
|
||||
> example = Example(predicted, reference)
|
||||
> ```
|
||||
|
||||
|
|
@ -286,10 +288,14 @@ Calculate alignment tables between two tokenizations.
|
|||
|
||||
### Alignment attributes {#alignment-attributes"}
|
||||
|
||||
Alignment attributes are managed using `AlignmentArray`, which is a
|
||||
simplified version of Thinc's [Ragged](https://thinc.ai/docs/api-types#ragged)
|
||||
type that only supports the `data` and `length` attributes.
|
||||
|
||||
| Name | Description |
|
||||
| ----- | --------------------------------------------------------------------- |
|
||||
| `x2y` | The `Ragged` object holding the alignment from `x` to `y`. ~~Ragged~~ |
|
||||
| `y2x` | The `Ragged` object holding the alignment from `y` to `x`. ~~Ragged~~ |
|
||||
| ----- | ------------------------------------------------------------------------------------- |
|
||||
| `x2y` | The `AlignmentArray` object holding the alignment from `x` to `y`. ~~AlignmentArray~~ |
|
||||
| `y2x` | The `AlignmentArray` object holding the alignment from `y` to `x`. ~~AlignmentArray~~ |
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
|
|
@ -309,10 +315,10 @@ tokenizations add up to the same string. For example, you'll be able to align
|
|||
> spacy_tokens = ["obama", "'s", "podcast"]
|
||||
> alignment = Alignment.from_strings(bert_tokens, spacy_tokens)
|
||||
> a2b = alignment.x2y
|
||||
> assert list(a2b.dataXd) == [0, 1, 1, 2]
|
||||
> assert list(a2b.data) == [0, 1, 1, 2]
|
||||
> ```
|
||||
>
|
||||
> If `a2b.dataXd[1] == a2b.dataXd[2] == 1`, that means that `A[1]` (`"'"`) and
|
||||
> If `a2b.data[1] == a2b.data[2] == 1`, that means that `A[1]` (`"'"`) and
|
||||
> `A[2]` (`"s"`) both align to `B[1]` (`"'s"`).
|
||||
|
||||
### Alignment.from_strings {#classmethod tag="function"}
|
||||
|
|
|
|||
|
|
@ -4,27 +4,45 @@ teaser:
|
|||
A storage class for entities and aliases of a specific knowledge base
|
||||
(ontology)
|
||||
tag: class
|
||||
source: spacy/kb.pyx
|
||||
source: spacy/kb/kb.pyx
|
||||
new: 2.2
|
||||
---
|
||||
|
||||
The `KnowledgeBase` object provides a method to generate
|
||||
[`Candidate`](/api/kb/#candidate) objects, which are plausible external
|
||||
The `KnowledgeBase` object is an abstract class providing a method to generate
|
||||
[`Candidate`](/api/kb#candidate) objects, which are plausible external
|
||||
identifiers given a certain textual mention. Each such `Candidate` holds
|
||||
information from the relevant KB entities, such as its frequency in text and
|
||||
possible aliases. Each entity in the knowledge base also has a pretrained entity
|
||||
vector of a fixed size.
|
||||
|
||||
Beyond that, `KnowledgeBase` classes have to implement a number of utility
|
||||
functions called by the [`EntityLinker`](/api/entitylinker) component.
|
||||
|
||||
<Infobox variant="warning">
|
||||
|
||||
This class was not abstract up to spaCy version 3.5. The `KnowledgeBase`
|
||||
implementation up to that point is available as `InMemoryLookupKB` from 3.5
|
||||
onwards.
|
||||
|
||||
</Infobox>
|
||||
|
||||
## KnowledgeBase.\_\_init\_\_ {#init tag="method"}
|
||||
|
||||
Create the knowledge base.
|
||||
`KnowledgeBase` is an abstract class and cannot be instantiated. Its child
|
||||
classes should call `__init__()` to set up some necessary attributes.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy.kb import KnowledgeBase
|
||||
> from spacy.vocab import Vocab
|
||||
>
|
||||
> class FullyImplementedKB(KnowledgeBase):
|
||||
> def __init__(self, vocab: Vocab, entity_vector_length: int):
|
||||
> super().__init__(vocab, entity_vector_length)
|
||||
> ...
|
||||
> vocab = nlp.vocab
|
||||
> kb = KnowledgeBase(vocab=vocab, entity_vector_length=64)
|
||||
> kb = FullyImplementedKB(vocab=vocab, entity_vector_length=64)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
|
|
@ -40,133 +58,66 @@ The length of the fixed-size entity vectors in the knowledge base.
|
|||
| ----------- | ------------------------------------------------ |
|
||||
| **RETURNS** | Length of the fixed-size entity vectors. ~~int~~ |
|
||||
|
||||
## KnowledgeBase.add_entity {#add_entity tag="method"}
|
||||
## KnowledgeBase.get_candidates {#get_candidates tag="method"}
|
||||
|
||||
Add an entity to the knowledge base, specifying its corpus frequency and entity
|
||||
vector, which should be of length
|
||||
[`entity_vector_length`](/api/kb#entity_vector_length).
|
||||
Given a certain textual mention as input, retrieve a list of candidate entities
|
||||
of type [`Candidate`](/api/kb#candidate).
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> kb.add_entity(entity="Q42", freq=32, entity_vector=vector1)
|
||||
> kb.add_entity(entity="Q463035", freq=111, entity_vector=vector2)
|
||||
> from spacy.lang.en import English
|
||||
> nlp = English()
|
||||
> doc = nlp("Douglas Adams wrote 'The Hitchhiker's Guide to the Galaxy'.")
|
||||
> candidates = kb.get_candidates(doc[0:2])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| --------------- | ---------------------------------------------------------- |
|
||||
| `entity` | The unique entity identifier. ~~str~~ |
|
||||
| `freq` | The frequency of the entity in a typical corpus. ~~float~~ |
|
||||
| `entity_vector` | The pretrained vector of the entity. ~~numpy.ndarray~~ |
|
||||
| ----------- | -------------------------------------------------------------------- |
|
||||
| `mention` | The textual mention or alias. ~~Span~~ |
|
||||
| **RETURNS** | An iterable of relevant `Candidate` objects. ~~Iterable[Candidate]~~ |
|
||||
|
||||
## KnowledgeBase.set_entities {#set_entities tag="method"}
|
||||
## KnowledgeBase.get_candidates_batch {#get_candidates_batch tag="method"}
|
||||
|
||||
Define the full list of entities in the knowledge base, specifying the corpus
|
||||
frequency and entity vector for each entity.
|
||||
Same as [`get_candidates()`](/api/kb#get_candidates), but for an arbitrary
|
||||
number of mentions. The [`EntityLinker`](/api/entitylinker) component will call
|
||||
`get_candidates_batch()` instead of `get_candidates()`, if the config parameter
|
||||
`candidates_batch_size` is greater or equal than 1.
|
||||
|
||||
The default implementation of `get_candidates_batch()` executes
|
||||
`get_candidates()` in a loop. We recommend implementing a more efficient way to
|
||||
retrieve candidates for multiple mentions at once, if performance is of concern
|
||||
to you.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> kb.set_entities(entity_list=["Q42", "Q463035"], freq_list=[32, 111], vector_list=[vector1, vector2])
|
||||
> from spacy.lang.en import English
|
||||
> nlp = English()
|
||||
> doc = nlp("Douglas Adams wrote 'The Hitchhiker's Guide to the Galaxy'.")
|
||||
> candidates = kb.get_candidates((doc[0:2], doc[3:]))
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ------------- | ---------------------------------------------------------------- |
|
||||
| `entity_list` | List of unique entity identifiers. ~~Iterable[Union[str, int]]~~ |
|
||||
| `freq_list` | List of entity frequencies. ~~Iterable[int]~~ |
|
||||
| `vector_list` | List of entity vectors. ~~Iterable[numpy.ndarray]~~ |
|
||||
|
||||
## KnowledgeBase.add_alias {#add_alias tag="method"}
|
||||
|
||||
Add an alias or mention to the knowledge base, specifying its potential KB
|
||||
identifiers and their prior probabilities. The entity identifiers should refer
|
||||
to entities previously added with [`add_entity`](/api/kb#add_entity) or
|
||||
[`set_entities`](/api/kb#set_entities). The sum of the prior probabilities
|
||||
should not exceed 1. Note that an empty string can not be used as alias.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> kb.add_alias(alias="Douglas", entities=["Q42", "Q463035"], probabilities=[0.6, 0.3])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| --------------- | --------------------------------------------------------------------------------- |
|
||||
| `alias` | The textual mention or alias. Can not be the empty string. ~~str~~ |
|
||||
| `entities` | The potential entities that the alias may refer to. ~~Iterable[Union[str, int]]~~ |
|
||||
| `probabilities` | The prior probabilities of each entity. ~~Iterable[float]~~ |
|
||||
|
||||
## KnowledgeBase.\_\_len\_\_ {#len tag="method"}
|
||||
|
||||
Get the total number of entities in the knowledge base.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> total_entities = len(kb)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ----------------------------------------------------- |
|
||||
| **RETURNS** | The number of entities in the knowledge base. ~~int~~ |
|
||||
|
||||
## KnowledgeBase.get_entity_strings {#get_entity_strings tag="method"}
|
||||
|
||||
Get a list of all entity IDs in the knowledge base.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> all_entities = kb.get_entity_strings()
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | --------------------------------------------------------- |
|
||||
| **RETURNS** | The list of entities in the knowledge base. ~~List[str]~~ |
|
||||
|
||||
## KnowledgeBase.get_size_aliases {#get_size_aliases tag="method"}
|
||||
|
||||
Get the total number of aliases in the knowledge base.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> total_aliases = kb.get_size_aliases()
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ---------------------------------------------------- |
|
||||
| **RETURNS** | The number of aliases in the knowledge base. ~~int~~ |
|
||||
|
||||
## KnowledgeBase.get_alias_strings {#get_alias_strings tag="method"}
|
||||
|
||||
Get a list of all aliases in the knowledge base.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> all_aliases = kb.get_alias_strings()
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | -------------------------------------------------------- |
|
||||
| **RETURNS** | The list of aliases in the knowledge base. ~~List[str]~~ |
|
||||
| ----------- | -------------------------------------------------------------------------------------------- |
|
||||
| `mentions` | The textual mention or alias. ~~Iterable[Span]~~ |
|
||||
| **RETURNS** | An iterable of iterable with relevant `Candidate` objects. ~~Iterable[Iterable[Candidate]]~~ |
|
||||
|
||||
## KnowledgeBase.get_alias_candidates {#get_alias_candidates tag="method"}
|
||||
|
||||
Given a certain textual mention as input, retrieve a list of candidate entities
|
||||
of type [`Candidate`](/api/kb/#candidate).
|
||||
<Infobox variant="warning">
|
||||
This method is _not_ available from spaCy 3.5 onwards.
|
||||
</Infobox>
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> candidates = kb.get_alias_candidates("Douglas")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ------------------------------------------------------------- |
|
||||
| `alias` | The textual mention or alias. ~~str~~ |
|
||||
| **RETURNS** | The list of relevant `Candidate` objects. ~~List[Candidate]~~ |
|
||||
From spaCy 3.5 on `KnowledgeBase` is an abstract class (with
|
||||
[`InMemoryLookupKB`](/api/kb_in_memory) being a drop-in replacement) to allow
|
||||
more flexibility in customizing knowledge bases. Some of its methods were moved
|
||||
to [`InMemoryLookupKB`](/api/kb_in_memory) during this refactoring, one of those
|
||||
being `get_alias_candidates()`. This method is now available as
|
||||
[`InMemoryLookupKB.get_alias_candidates()`](/api/kb_in_memory#get_alias_candidates).
|
||||
Note: [`InMemoryLookupKB.get_candidates()`](/api/kb_in_memory#get_candidates)
|
||||
defaults to
|
||||
[`InMemoryLookupKB.get_alias_candidates()`](/api/kb_in_memory#get_alias_candidates).
|
||||
|
||||
## KnowledgeBase.get_vector {#get_vector tag="method"}
|
||||
|
||||
|
|
@ -179,26 +130,29 @@ Given a certain entity ID, retrieve its pretrained entity vector.
|
|||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ------------------------------------ |
|
||||
| ----------- | -------------------------------------- |
|
||||
| `entity` | The entity ID. ~~str~~ |
|
||||
| **RETURNS** | The entity vector. ~~numpy.ndarray~~ |
|
||||
| **RETURNS** | The entity vector. ~~Iterable[float]~~ |
|
||||
|
||||
## KnowledgeBase.get_prior_prob {#get_prior_prob tag="method"}
|
||||
## KnowledgeBase.get_vectors {#get_vectors tag="method"}
|
||||
|
||||
Given a certain entity ID and a certain textual mention, retrieve the prior
|
||||
probability of the fact that the mention links to the entity ID.
|
||||
Same as [`get_vector()`](/api/kb#get_vector), but for an arbitrary number of
|
||||
entity IDs.
|
||||
|
||||
The default implementation of `get_vectors()` executes `get_vector()` in a loop.
|
||||
We recommend implementing a more efficient way to retrieve vectors for multiple
|
||||
entities at once, if performance is of concern to you.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> probability = kb.get_prior_prob("Q42", "Douglas")
|
||||
> vectors = kb.get_vectors(("Q42", "Q3107329"))
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ------------------------------------------------------------------------- |
|
||||
| `entity` | The entity ID. ~~str~~ |
|
||||
| `alias` | The textual mention or alias. ~~str~~ |
|
||||
| **RETURNS** | The prior probability of the `alias` referring to the `entity`. ~~float~~ |
|
||||
| ----------- | --------------------------------------------------------- |
|
||||
| `entities` | The entity IDs. ~~Iterable[str]~~ |
|
||||
| **RETURNS** | The entity vectors. ~~Iterable[Iterable[numpy.ndarray]]~~ |
|
||||
|
||||
## KnowledgeBase.to_disk {#to_disk tag="method"}
|
||||
|
||||
|
|
@ -207,12 +161,13 @@ Save the current state of the knowledge base to a directory.
|
|||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> kb.to_disk(loc)
|
||||
> kb.to_disk(path)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----- | ------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `loc` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
|
||||
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
|
||||
| `exclude` | List of components to exclude. ~~Iterable[str]~~ |
|
||||
|
||||
## KnowledgeBase.from_disk {#from_disk tag="method"}
|
||||
|
||||
|
|
@ -222,16 +177,16 @@ Restore the state of the knowledge base from a given directory. Note that the
|
|||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy.kb import KnowledgeBase
|
||||
> from spacy.vocab import Vocab
|
||||
> vocab = Vocab().from_disk("/path/to/vocab")
|
||||
> kb = KnowledgeBase(vocab=vocab, entity_vector_length=64)
|
||||
> kb = FullyImplementedKB(vocab=vocab, entity_vector_length=64)
|
||||
> kb.from_disk("/path/to/kb")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ----------------------------------------------------------------------------------------------- |
|
||||
| `loc` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
|
||||
| `exclude` | List of components to exclude. ~~Iterable[str]~~ |
|
||||
| **RETURNS** | The modified `KnowledgeBase` object. ~~KnowledgeBase~~ |
|
||||
|
||||
## Candidate {#candidate tag="class"}
|
||||
|
|
|
|||
302
website/docs/api/kb_in_memory.md
Normal file
302
website/docs/api/kb_in_memory.md
Normal file
|
|
@ -0,0 +1,302 @@
|
|||
---
|
||||
title: InMemoryLookupKB
|
||||
teaser:
|
||||
The default implementation of the KnowledgeBase interface. Stores all
|
||||
information in-memory.
|
||||
tag: class
|
||||
source: spacy/kb/kb_in_memory.pyx
|
||||
new: 3.5
|
||||
---
|
||||
|
||||
The `InMemoryLookupKB` class inherits from [`KnowledgeBase`](/api/kb) and
|
||||
implements all of its methods. It stores all KB data in-memory and generates
|
||||
[`Candidate`](/api/kb#candidate) objects by exactly matching mentions with
|
||||
entity names. It's highly optimized for both a low memory footprint and speed of
|
||||
retrieval.
|
||||
|
||||
## InMemoryLookupKB.\_\_init\_\_ {#init tag="method"}
|
||||
|
||||
Create the knowledge base.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy.kb import KnowledgeBase
|
||||
> vocab = nlp.vocab
|
||||
> kb = KnowledgeBase(vocab=vocab, entity_vector_length=64)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ---------------------- | ------------------------------------------------ |
|
||||
| `vocab` | The shared vocabulary. ~~Vocab~~ |
|
||||
| `entity_vector_length` | Length of the fixed-size entity vectors. ~~int~~ |
|
||||
|
||||
## InMemoryLookupKB.entity_vector_length {#entity_vector_length tag="property"}
|
||||
|
||||
The length of the fixed-size entity vectors in the knowledge base.
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ------------------------------------------------ |
|
||||
| **RETURNS** | Length of the fixed-size entity vectors. ~~int~~ |
|
||||
|
||||
## InMemoryLookupKB.add_entity {#add_entity tag="method"}
|
||||
|
||||
Add an entity to the knowledge base, specifying its corpus frequency and entity
|
||||
vector, which should be of length
|
||||
[`entity_vector_length`](/api/kb_in_memory#entity_vector_length).
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> kb.add_entity(entity="Q42", freq=32, entity_vector=vector1)
|
||||
> kb.add_entity(entity="Q463035", freq=111, entity_vector=vector2)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| --------------- | ---------------------------------------------------------- |
|
||||
| `entity` | The unique entity identifier. ~~str~~ |
|
||||
| `freq` | The frequency of the entity in a typical corpus. ~~float~~ |
|
||||
| `entity_vector` | The pretrained vector of the entity. ~~numpy.ndarray~~ |
|
||||
|
||||
## InMemoryLookupKB.set_entities {#set_entities tag="method"}
|
||||
|
||||
Define the full list of entities in the knowledge base, specifying the corpus
|
||||
frequency and entity vector for each entity.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> kb.set_entities(entity_list=["Q42", "Q463035"], freq_list=[32, 111], vector_list=[vector1, vector2])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ------------- | ---------------------------------------------------------------- |
|
||||
| `entity_list` | List of unique entity identifiers. ~~Iterable[Union[str, int]]~~ |
|
||||
| `freq_list` | List of entity frequencies. ~~Iterable[int]~~ |
|
||||
| `vector_list` | List of entity vectors. ~~Iterable[numpy.ndarray]~~ |
|
||||
|
||||
## InMemoryLookupKB.add_alias {#add_alias tag="method"}
|
||||
|
||||
Add an alias or mention to the knowledge base, specifying its potential KB
|
||||
identifiers and their prior probabilities. The entity identifiers should refer
|
||||
to entities previously added with [`add_entity`](/api/kb_in_memory#add_entity)
|
||||
or [`set_entities`](/api/kb_in_memory#set_entities). The sum of the prior
|
||||
probabilities should not exceed 1. Note that an empty string can not be used as
|
||||
alias.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> kb.add_alias(alias="Douglas", entities=["Q42", "Q463035"], probabilities=[0.6, 0.3])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| --------------- | --------------------------------------------------------------------------------- |
|
||||
| `alias` | The textual mention or alias. Can not be the empty string. ~~str~~ |
|
||||
| `entities` | The potential entities that the alias may refer to. ~~Iterable[Union[str, int]]~~ |
|
||||
| `probabilities` | The prior probabilities of each entity. ~~Iterable[float]~~ |
|
||||
|
||||
## InMemoryLookupKB.\_\_len\_\_ {#len tag="method"}
|
||||
|
||||
Get the total number of entities in the knowledge base.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> total_entities = len(kb)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ----------------------------------------------------- |
|
||||
| **RETURNS** | The number of entities in the knowledge base. ~~int~~ |
|
||||
|
||||
## InMemoryLookupKB.get_entity_strings {#get_entity_strings tag="method"}
|
||||
|
||||
Get a list of all entity IDs in the knowledge base.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> all_entities = kb.get_entity_strings()
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | --------------------------------------------------------- |
|
||||
| **RETURNS** | The list of entities in the knowledge base. ~~List[str]~~ |
|
||||
|
||||
## InMemoryLookupKB.get_size_aliases {#get_size_aliases tag="method"}
|
||||
|
||||
Get the total number of aliases in the knowledge base.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> total_aliases = kb.get_size_aliases()
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ---------------------------------------------------- |
|
||||
| **RETURNS** | The number of aliases in the knowledge base. ~~int~~ |
|
||||
|
||||
## InMemoryLookupKB.get_alias_strings {#get_alias_strings tag="method"}
|
||||
|
||||
Get a list of all aliases in the knowledge base.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> all_aliases = kb.get_alias_strings()
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | -------------------------------------------------------- |
|
||||
| **RETURNS** | The list of aliases in the knowledge base. ~~List[str]~~ |
|
||||
|
||||
## InMemoryLookupKB.get_candidates {#get_candidates tag="method"}
|
||||
|
||||
Given a certain textual mention as input, retrieve a list of candidate entities
|
||||
of type [`Candidate`](/api/kb#candidate). Wraps
|
||||
[`get_alias_candidates()`](/api/kb_in_memory#get_alias_candidates).
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy.lang.en import English
|
||||
> nlp = English()
|
||||
> doc = nlp("Douglas Adams wrote 'The Hitchhiker's Guide to the Galaxy'.")
|
||||
> candidates = kb.get_candidates(doc[0:2])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | -------------------------------------------------------------------- |
|
||||
| `mention` | The textual mention or alias. ~~Span~~ |
|
||||
| **RETURNS** | An iterable of relevant `Candidate` objects. ~~Iterable[Candidate]~~ |
|
||||
|
||||
## InMemoryLookupKB.get_candidates_batch {#get_candidates_batch tag="method"}
|
||||
|
||||
Same as [`get_candidates()`](/api/kb_in_memory#get_candidates), but for an
|
||||
arbitrary number of mentions. The [`EntityLinker`](/api/entitylinker) component
|
||||
will call `get_candidates_batch()` instead of `get_candidates()`, if the config
|
||||
parameter `candidates_batch_size` is greater or equal than 1.
|
||||
|
||||
The default implementation of `get_candidates_batch()` executes
|
||||
`get_candidates()` in a loop. We recommend implementing a more efficient way to
|
||||
retrieve candidates for multiple mentions at once, if performance is of concern
|
||||
to you.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy.lang.en import English
|
||||
> nlp = English()
|
||||
> doc = nlp("Douglas Adams wrote 'The Hitchhiker's Guide to the Galaxy'.")
|
||||
> candidates = kb.get_candidates((doc[0:2], doc[3:]))
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | -------------------------------------------------------------------------------------------- |
|
||||
| `mentions` | The textual mention or alias. ~~Iterable[Span]~~ |
|
||||
| **RETURNS** | An iterable of iterable with relevant `Candidate` objects. ~~Iterable[Iterable[Candidate]]~~ |
|
||||
|
||||
## InMemoryLookupKB.get_alias_candidates {#get_alias_candidates tag="method"}
|
||||
|
||||
Given a certain textual mention as input, retrieve a list of candidate entities
|
||||
of type [`Candidate`](/api/kb#candidate).
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> candidates = kb.get_alias_candidates("Douglas")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ------------------------------------------------------------- |
|
||||
| `alias` | The textual mention or alias. ~~str~~ |
|
||||
| **RETURNS** | The list of relevant `Candidate` objects. ~~List[Candidate]~~ |
|
||||
|
||||
## InMemoryLookupKB.get_vector {#get_vector tag="method"}
|
||||
|
||||
Given a certain entity ID, retrieve its pretrained entity vector.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> vector = kb.get_vector("Q42")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ------------------------------------ |
|
||||
| `entity` | The entity ID. ~~str~~ |
|
||||
| **RETURNS** | The entity vector. ~~numpy.ndarray~~ |
|
||||
|
||||
## InMemoryLookupKB.get_vectors {#get_vectors tag="method"}
|
||||
|
||||
Same as [`get_vector()`](/api/kb_in_memory#get_vector), but for an arbitrary
|
||||
number of entity IDs.
|
||||
|
||||
The default implementation of `get_vectors()` executes `get_vector()` in a loop.
|
||||
We recommend implementing a more efficient way to retrieve vectors for multiple
|
||||
entities at once, if performance is of concern to you.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> vectors = kb.get_vectors(("Q42", "Q3107329"))
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | --------------------------------------------------------- |
|
||||
| `entities` | The entity IDs. ~~Iterable[str]~~ |
|
||||
| **RETURNS** | The entity vectors. ~~Iterable[Iterable[numpy.ndarray]]~~ |
|
||||
|
||||
## InMemoryLookupKB.get_prior_prob {#get_prior_prob tag="method"}
|
||||
|
||||
Given a certain entity ID and a certain textual mention, retrieve the prior
|
||||
probability of the fact that the mention links to the entity ID.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> probability = kb.get_prior_prob("Q42", "Douglas")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ------------------------------------------------------------------------- |
|
||||
| `entity` | The entity ID. ~~str~~ |
|
||||
| `alias` | The textual mention or alias. ~~str~~ |
|
||||
| **RETURNS** | The prior probability of the `alias` referring to the `entity`. ~~float~~ |
|
||||
|
||||
## InMemoryLookupKB.to_disk {#to_disk tag="method"}
|
||||
|
||||
Save the current state of the knowledge base to a directory.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> kb.to_disk(path)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
|
||||
| `exclude` | List of components to exclude. ~~Iterable[str]~~ |
|
||||
|
||||
## InMemoryLookupKB.from_disk {#from_disk tag="method"}
|
||||
|
||||
Restore the state of the knowledge base from a given directory. Note that the
|
||||
[`Vocab`](/api/vocab) should also be the same as the one used to create the KB.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy.vocab import Vocab
|
||||
> vocab = Vocab().from_disk("/path/to/vocab")
|
||||
> kb = FullyImplementedKB(vocab=vocab, entity_vector_length=64)
|
||||
> kb.from_disk("/path/to/kb")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ----------------------------------------------------------------------------------------------- |
|
||||
| `loc` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
|
||||
| `exclude` | List of components to exclude. ~~Iterable[str]~~ |
|
||||
| **RETURNS** | The modified `KnowledgeBase` object. ~~KnowledgeBase~~ |
|
||||
|
|
@ -164,6 +164,9 @@ examples, see the
|
|||
Apply the pipeline to some text. The text can span multiple sentences, and can
|
||||
contain arbitrary whitespace. Alignment into the original string is preserved.
|
||||
|
||||
Instead of text, a `Doc` can be passed as input, in which case tokenization is
|
||||
skipped, but the rest of the pipeline is run.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
|
|
@ -173,7 +176,7 @@ contain arbitrary whitespace. Alignment into the original string is preserved.
|
|||
|
||||
| Name | Description |
|
||||
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `text` | The text to be processed. ~~str~~ |
|
||||
| `text` | The text to be processed, or a Doc. ~~Union[str, Doc]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `disable` | Names of pipeline components to [disable](/usage/processing-pipelines#disabling). ~~List[str]~~ |
|
||||
| `component_cfg` | Optional dictionary of keyword arguments for components, keyed by component names. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Any]]]~~ |
|
||||
|
|
@ -184,6 +187,9 @@ contain arbitrary whitespace. Alignment into the original string is preserved.
|
|||
Process texts as a stream, and yield `Doc` objects in order. This is usually
|
||||
more efficient than processing texts one-by-one.
|
||||
|
||||
Instead of text, a `Doc` object can be passed as input. In this case
|
||||
tokenization is skipped but the rest of the pipeline is run.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
|
|
@ -194,7 +200,7 @@ more efficient than processing texts one-by-one.
|
|||
|
||||
| Name | Description |
|
||||
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `texts` | A sequence of strings. ~~Iterable[str]~~ |
|
||||
| `texts` | A sequence of strings (or `Doc` objects). ~~Iterable[Union[str, Doc]]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `as_tuples` | If set to `True`, inputs should be a sequence of `(text, context)` tuples. Output will then be a sequence of `(doc, context)` tuples. Defaults to `False`. ~~bool~~ |
|
||||
| `batch_size` | The number of texts to buffer. ~~Optional[int]~~ |
|
||||
|
|
|
|||
|
|
@ -153,3 +153,36 @@ whole pipeline has run.
|
|||
| `attrs` | A dict of the `Doc` attributes and the values to set them to. Defaults to `{"tensor": None, "_.trf_data": None}` to clean up after `tok2vec` and `transformer` components. ~~dict~~ |
|
||||
| `silent` | If `False`, show warnings if attributes aren't found or can't be set. Defaults to `True`. ~~bool~~ |
|
||||
| **RETURNS** | The modified `Doc` with the modified attributes. ~~Doc~~ |
|
||||
|
||||
## span_cleaner {#span_cleaner tag="function,experimental"}
|
||||
|
||||
Remove `SpanGroup`s from `doc.spans` based on a key prefix. This is used to
|
||||
clean up after the [`CoreferenceResolver`](/api/coref) when it's paired with a
|
||||
[`SpanResolver`](/api/span-resolver).
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
This pipeline function is not yet integrated into spaCy core, and is available
|
||||
via the extension package
|
||||
[`spacy-experimental`](https://github.com/explosion/spacy-experimental) starting
|
||||
in version 0.6.0. It exposes the component via
|
||||
[entry points](/usage/saving-loading/#entry-points), so if you have the package
|
||||
installed, using `factory = "span_cleaner"` in your
|
||||
[training config](/usage/training#config) or `nlp.add_pipe("span_cleaner")` will
|
||||
work out-of-the-box.
|
||||
|
||||
</Infobox>
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> config = {"prefix": "coref_head_clusters"}
|
||||
> nlp.add_pipe("span_cleaner", config=config)
|
||||
> doc = nlp("text")
|
||||
> assert "coref_head_clusters_1" not in doc.spans
|
||||
> ```
|
||||
|
||||
| Setting | Description |
|
||||
| ----------- | ------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `prefix` | A prefix to check `SpanGroup` keys for. Any matching groups will be removed. Defaults to `"coref_head_clusters"`. ~~str~~ |
|
||||
| **RETURNS** | The modified `Doc` with any matching spans removed. ~~Doc~~ |
|
||||
|
|
|
|||
|
|
@ -270,3 +270,62 @@ Compute micro-PRF and per-entity PRF scores.
|
|||
| Name | Description |
|
||||
| ---------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ |
|
||||
|
||||
## score_coref_clusters {#score_coref_clusters tag="experimental"}
|
||||
|
||||
Returns LEA ([Moosavi and Strube, 2016](https://aclanthology.org/P16-1060/)) PRF
|
||||
scores for coreference clusters.
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
Note this scoring function is not yet included in spaCy core - for details, see
|
||||
the [CoreferenceResolver](/api/coref) docs.
|
||||
|
||||
</Infobox>
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> scores = score_coref_clusters(
|
||||
> examples,
|
||||
> span_cluster_prefix="coref_clusters",
|
||||
> )
|
||||
> print(scores["coref_f"])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| --------------------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `span_cluster_prefix` | The prefix used for spans representing coreference clusters. ~~str~~ |
|
||||
| **RETURNS** | A dictionary containing the scores. ~~Dict[str, Optional[float]]~~ |
|
||||
|
||||
## score_span_predictions {#score_span_predictions tag="experimental"}
|
||||
|
||||
Return accuracy for reconstructions of spans from single tokens. Only exactly
|
||||
correct predictions are counted as correct, there is no partial credit for near
|
||||
answers. Used by the [SpanResolver](/api/span-resolver).
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
Note this scoring function is not yet included in spaCy core - for details, see
|
||||
the [SpanResolver](/api/span-resolver) docs.
|
||||
|
||||
</Infobox>
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> scores = score_span_predictions(
|
||||
> examples,
|
||||
> output_prefix="coref_clusters",
|
||||
> )
|
||||
> print(scores["span_coref_clusters_accuracy"])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `output_prefix` | The prefix used for spans representing the final predicted spans. ~~str~~ |
|
||||
| **RETURNS** | A dictionary containing the scores. ~~Dict[str, Optional[float]]~~ |
|
||||
|
|
|
|||
356
website/docs/api/span-resolver.md
Normal file
356
website/docs/api/span-resolver.md
Normal file
|
|
@ -0,0 +1,356 @@
|
|||
---
|
||||
title: SpanResolver
|
||||
tag: class,experimental
|
||||
source: spacy-experimental/coref/span_resolver_component.py
|
||||
teaser: 'Pipeline component for resolving tokens into spans'
|
||||
api_base_class: /api/pipe
|
||||
api_string_name: span_resolver
|
||||
api_trainable: true
|
||||
---
|
||||
|
||||
> #### Installation
|
||||
>
|
||||
> ```bash
|
||||
> $ pip install -U spacy-experimental
|
||||
> ```
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
This component not yet integrated into spaCy core, and is available via the
|
||||
extension package
|
||||
[`spacy-experimental`](https://github.com/explosion/spacy-experimental) starting
|
||||
in version 0.6.0. It exposes the component via
|
||||
[entry points](/usage/saving-loading/#entry-points), so if you have the package
|
||||
installed, using `factory = "experimental_span_resolver"` in your
|
||||
[training config](/usage/training#config) or
|
||||
`nlp.add_pipe("experimental_span_resolver")` will work out-of-the-box.
|
||||
|
||||
</Infobox>
|
||||
|
||||
A `SpanResolver` component takes in tokens (represented as `Span` objects of
|
||||
length 1) and resolves them into `Span` objects of arbitrary length. The initial
|
||||
use case is as a post-processing step on word-level
|
||||
[coreference resolution](/api/coref). The input and output keys used to store
|
||||
`Span` objects are configurable.
|
||||
|
||||
## Assigned Attributes {#assigned-attributes}
|
||||
|
||||
Predictions will be saved to `Doc.spans` as [`SpanGroup`s](/api/spangroup).
|
||||
|
||||
Input token spans will be read in using an input prefix, by default
|
||||
`"coref_head_clusters"`, and output spans will be saved using an output prefix
|
||||
(default `"coref_clusters"`) plus a serial number starting from one. The
|
||||
prefixes are configurable.
|
||||
|
||||
| Location | Value |
|
||||
| ------------------------------------------------- | ------------------------------------------------------------------------- |
|
||||
| `Doc.spans[output_prefix + "_" + cluster_number]` | One group of predicted spans. Cluster number starts from 1. ~~SpanGroup~~ |
|
||||
|
||||
## Config and implementation {#config}
|
||||
|
||||
The default config is defined by the pipeline component factory and describes
|
||||
how the component should be configured. You can override its settings via the
|
||||
`config` argument on [`nlp.add_pipe`](/api/language#add_pipe) or in your
|
||||
[`config.cfg` for training](/usage/training#config). See the
|
||||
[model architectures](/api/architectures#coref-architectures) documentation for
|
||||
details on the architectures and their arguments and hyperparameters.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy_experimental.coref.span_resolver_component import DEFAULT_SPAN_RESOLVER_MODEL
|
||||
> from spacy_experimental.coref.coref_util import DEFAULT_CLUSTER_PREFIX, DEFAULT_CLUSTER_HEAD_PREFIX
|
||||
> config={
|
||||
> "model": DEFAULT_SPAN_RESOLVER_MODEL,
|
||||
> "input_prefix": DEFAULT_CLUSTER_HEAD_PREFIX,
|
||||
> "output_prefix": DEFAULT_CLUSTER_PREFIX,
|
||||
> },
|
||||
> nlp.add_pipe("experimental_span_resolver", config=config)
|
||||
> ```
|
||||
|
||||
| Setting | Description |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `model` | The [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. Defaults to [SpanResolver](/api/architectures#SpanResolver). ~~Model~~ |
|
||||
| `input_prefix` | The prefix to use for input `SpanGroup`s. Defaults to `coref_head_clusters`. ~~str~~ |
|
||||
| `output_prefix` | The prefix for predicted `SpanGroup`s. Defaults to `coref_clusters`. ~~str~~ |
|
||||
|
||||
## SpanResolver.\_\_init\_\_ {#init tag="method"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> # Construction via add_pipe with default model
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
>
|
||||
> # Construction via add_pipe with custom model
|
||||
> config = {"model": {"@architectures": "my_span_resolver.v1"}}
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver", config=config)
|
||||
>
|
||||
> # Construction from class
|
||||
> from spacy_experimental.coref.span_resolver_component import SpanResolver
|
||||
> span_resolver = SpanResolver(nlp.vocab, model)
|
||||
> ```
|
||||
|
||||
Create a new pipeline instance. In your application, you would normally use a
|
||||
shortcut for this and instantiate the component using its string name and
|
||||
[`nlp.add_pipe`](/api/language#add_pipe).
|
||||
|
||||
| Name | Description |
|
||||
| --------------- | --------------------------------------------------------------------------------------------------- |
|
||||
| `vocab` | The shared vocabulary. ~~Vocab~~ |
|
||||
| `model` | The [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. ~~Model~~ |
|
||||
| `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ |
|
||||
| _keyword-only_ | |
|
||||
| `input_prefix` | The prefix to use for input `SpanGroup`s. Defaults to `coref_head_clusters`. ~~str~~ |
|
||||
| `output_prefix` | The prefix for predicted `SpanGroup`s. Defaults to `coref_clusters`. ~~str~~ |
|
||||
|
||||
## SpanResolver.\_\_call\_\_ {#call tag="method"}
|
||||
|
||||
Apply the pipe to one document. The document is modified in place and returned.
|
||||
This usually happens under the hood when the `nlp` object is called on a text
|
||||
and all pipeline components are applied to the `Doc` in order. Both
|
||||
[`__call__`](#call) and [`pipe`](#pipe) delegate to the [`predict`](#predict)
|
||||
and [`set_annotations`](#set_annotations) methods.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> doc = nlp("This is a sentence.")
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> # This usually happens under the hood
|
||||
> processed = span_resolver(doc)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | -------------------------------- |
|
||||
| `doc` | The document to process. ~~Doc~~ |
|
||||
| **RETURNS** | The processed document. ~~Doc~~ |
|
||||
|
||||
## SpanResolver.pipe {#pipe tag="method"}
|
||||
|
||||
Apply the pipe to a stream of documents. This usually happens under the hood
|
||||
when the `nlp` object is called on a text and all pipeline components are
|
||||
applied to the `Doc` in order. Both [`__call__`](/api/span-resolver#call) and
|
||||
[`pipe`](/api/span-resolver#pipe) delegate to the
|
||||
[`predict`](/api/span-resolver#predict) and
|
||||
[`set_annotations`](/api/span-resolver#set_annotations) methods.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> for doc in span_resolver.pipe(docs, batch_size=50):
|
||||
> pass
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ------------------------------------------------------------- |
|
||||
| `stream` | A stream of documents. ~~Iterable[Doc]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ |
|
||||
| **YIELDS** | The processed documents in order. ~~Doc~~ |
|
||||
|
||||
## SpanResolver.initialize {#initialize tag="method"}
|
||||
|
||||
Initialize the component for training. `get_examples` should be a function that
|
||||
returns an iterable of [`Example`](/api/example) objects. **At least one example
|
||||
should be supplied.** The data examples are used to **initialize the model** of
|
||||
the component and can either be the full training data or a representative
|
||||
sample. Initialization includes validating the network,
|
||||
[inferring missing shapes](https://thinc.ai/docs/usage-models#validation) and
|
||||
setting up the label scheme based on the data. This method is typically called
|
||||
by [`Language.initialize`](/api/language#initialize).
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> span_resolver.initialize(lambda: examples, nlp=nlp)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `get_examples` | Function that returns gold-standard annotations in the form of [`Example`](/api/example) objects. Must contain at least one `Example`. ~~Callable[[], Iterable[Example]]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ |
|
||||
|
||||
## SpanResolver.predict {#predict tag="method"}
|
||||
|
||||
Apply the component's model to a batch of [`Doc`](/api/doc) objects, without
|
||||
modifying them. Predictions are returned as a list of `MentionClusters`, one for
|
||||
each input `Doc`. A `MentionClusters` instance is just a list of lists of pairs
|
||||
of `int`s, where each item corresponds to an input `SpanGroup`, and the `int`s
|
||||
correspond to token indices.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> spans = span_resolver.predict([doc1, doc2])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ------------------------------------------------------------- |
|
||||
| `docs` | The documents to predict. ~~Iterable[Doc]~~ |
|
||||
| **RETURNS** | The predicted spans for the `Doc`s. ~~List[MentionClusters]~~ |
|
||||
|
||||
## SpanResolver.set_annotations {#set_annotations tag="method"}
|
||||
|
||||
Modify a batch of documents, saving predictions using the output prefix in
|
||||
`Doc.spans`.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> spans = span_resolver.predict([doc1, doc2])
|
||||
> span_resolver.set_annotations([doc1, doc2], spans)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ------- | ------------------------------------------------------------- |
|
||||
| `docs` | The documents to modify. ~~Iterable[Doc]~~ |
|
||||
| `spans` | The predicted spans for the `docs`. ~~List[MentionClusters]~~ |
|
||||
|
||||
## SpanResolver.update {#update tag="method"}
|
||||
|
||||
Learn from a batch of [`Example`](/api/example) objects. Delegates to
|
||||
[`predict`](/api/span-resolver#predict).
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> optimizer = nlp.initialize()
|
||||
> losses = span_resolver.update(examples, sgd=optimizer)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `examples` | A batch of [`Example`](/api/example) objects to learn from. ~~Iterable[Example]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `drop` | The dropout rate. ~~float~~ |
|
||||
| `sgd` | An optimizer. Will be created via [`create_optimizer`](#create_optimizer) if not set. ~~Optional[Optimizer]~~ |
|
||||
| `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ |
|
||||
| **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ |
|
||||
|
||||
## SpanResolver.create_optimizer {#create_optimizer tag="method"}
|
||||
|
||||
Create an optimizer for the pipeline component.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> optimizer = span_resolver.create_optimizer()
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ---------------------------- |
|
||||
| **RETURNS** | The optimizer. ~~Optimizer~~ |
|
||||
|
||||
## SpanResolver.use_params {#use_params tag="method, contextmanager"}
|
||||
|
||||
Modify the pipe's model, to use the given parameter values. At the end of the
|
||||
context, the original parameters are restored.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> with span_resolver.use_params(optimizer.averages):
|
||||
> span_resolver.to_disk("/best_model")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------- | -------------------------------------------------- |
|
||||
| `params` | The parameter values to use in the model. ~~dict~~ |
|
||||
|
||||
## SpanResolver.to_disk {#to_disk tag="method"}
|
||||
|
||||
Serialize the pipe to disk.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> span_resolver.to_disk("/path/to/span_resolver")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
|
||||
|
||||
## SpanResolver.from_disk {#from_disk tag="method"}
|
||||
|
||||
Load the pipe from disk. Modifies the object in place and returns it.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> span_resolver.from_disk("/path/to/span_resolver")
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ |
|
||||
| _keyword-only_ | |
|
||||
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
|
||||
| **RETURNS** | The modified `SpanResolver` object. ~~SpanResolver~~ |
|
||||
|
||||
## SpanResolver.to_bytes {#to_bytes tag="method"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> span_resolver_bytes = span_resolver.to_bytes()
|
||||
> ```
|
||||
|
||||
Serialize the pipe to a bytestring.
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ------------------------------------------------------------------------------------------- |
|
||||
| _keyword-only_ | |
|
||||
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
|
||||
| **RETURNS** | The serialized form of the `SpanResolver` object. ~~bytes~~ |
|
||||
|
||||
## SpanResolver.from_bytes {#from_bytes tag="method"}
|
||||
|
||||
Load the pipe from a bytestring. Modifies the object in place and returns it.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> span_resolver_bytes = span_resolver.to_bytes()
|
||||
> span_resolver = nlp.add_pipe("experimental_span_resolver")
|
||||
> span_resolver.from_bytes(span_resolver_bytes)
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| -------------- | ------------------------------------------------------------------------------------------- |
|
||||
| `bytes_data` | The data to load from. ~~bytes~~ |
|
||||
| _keyword-only_ | |
|
||||
| `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ |
|
||||
| **RETURNS** | The `SpanResolver` object. ~~SpanResolver~~ |
|
||||
|
||||
## Serialization fields {#serialization-fields}
|
||||
|
||||
During serialization, spaCy will export several data fields used to restore
|
||||
different aspects of the object. If needed, you can exclude them from
|
||||
serialization by passing in the string names via the `exclude` argument.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> data = span_resolver.to_disk("/path", exclude=["vocab"])
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ------- | -------------------------------------------------------------- |
|
||||
| `vocab` | The shared [`Vocab`](/api/vocab). |
|
||||
| `cfg` | The config file. You usually don't want to exclude this. |
|
||||
| `model` | The binary model data. You usually don't want to exclude this. |
|
||||
|
|
@ -887,6 +887,27 @@ backprop passes.
|
|||
| `backprop_color` | Color identifier for backpropagation passes. Defaults to `-1`. ~~int~~ |
|
||||
| **CREATES** | A function that takes the current `nlp` and wraps forward/backprop passes in NVTX ranges. ~~Callable[[Language], Language]~~ |
|
||||
|
||||
### spacy.models_and_pipes_with_nvtx_range.v1 {#models_and_pipes_with_nvtx_range tag="registered function" new="3.4"}
|
||||
|
||||
> #### Example config
|
||||
>
|
||||
> ```ini
|
||||
> [nlp]
|
||||
> after_pipeline_creation = {"@callbacks":"spacy.models_and_pipes_with_nvtx_range.v1"}
|
||||
> ```
|
||||
|
||||
Recursively wrap both the models and methods of each pipe using
|
||||
[NVTX](https://nvidia.github.io/NVTX/) range markers. By default, the following
|
||||
methods are wrapped: `pipe`, `predict`, `set_annotations`, `update`, `rehearse`,
|
||||
`get_loss`, `initialize`, `begin_update`, `finish_update`, `update`.
|
||||
|
||||
| Name | Description |
|
||||
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `forward_color` | Color identifier for model forward passes. Defaults to `-1`. ~~int~~ |
|
||||
| `backprop_color` | Color identifier for model backpropagation passes. Defaults to `-1`. ~~int~~ |
|
||||
| `additional_pipe_functions` | Additional pipeline methods to wrap. Keys are pipeline names and values are lists of method identifiers. Defaults to `None`. ~~Optional[Dict[str, List[str]]]~~ |
|
||||
| **CREATES** | A function that takes the current `nlp` and wraps pipe models and methods in NVTX ranges. ~~Callable[[Language], Language]~~ |
|
||||
|
||||
## Training data and alignment {#gold source="spacy/training"}
|
||||
|
||||
### training.offsets_to_biluo_tags {#offsets_to_biluo_tags tag="function"}
|
||||
|
|
|
|||
|
|
@ -78,7 +78,9 @@ operates on a `Doc` and gives you access to the matched tokens **in context**.
|
|||
| Name | Description |
|
||||
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------- |
|
||||
| [`Corpus`](/api/corpus) | Class for managing annotated corpora for training and evaluation data. |
|
||||
| [`KnowledgeBase`](/api/kb) | Storage for entities and aliases of a knowledge base for entity linking. |
|
||||
| [`KnowledgeBase`](/api/kb) | Abstract base class for storage and retrieval of data for entity linking. |
|
||||
| [`InMemoryLookupKB`](/api/kb_in_memory) | Implementation of `KnowledgeBase` storing all data in memory. |
|
||||
| [`Candidate`](/api/kb#candidate) | Object associating a textual mention with a specific entity contained in a `KnowledgeBase`. |
|
||||
| [`Lookups`](/api/lookups) | Container for convenient access to large lookup tables and dictionaries. |
|
||||
| [`MorphAnalysis`](/api/morphology#morphanalysis) | A morphological analysis. |
|
||||
| [`Morphology`](/api/morphology) | Store morphological analyses and map them to and from hash values. |
|
||||
|
|
|
|||
|
|
@ -1422,9 +1422,9 @@ other_tokens = ["i", "listened", "to", "obama", "'", "s", "podcasts", "."]
|
|||
spacy_tokens = ["i", "listened", "to", "obama", "'s", "podcasts", "."]
|
||||
align = Alignment.from_strings(other_tokens, spacy_tokens)
|
||||
print(f"a -> b, lengths: {align.x2y.lengths}") # array([1, 1, 1, 1, 1, 1, 1, 1])
|
||||
print(f"a -> b, mapping: {align.x2y.dataXd}") # array([0, 1, 2, 3, 4, 4, 5, 6]) : two tokens both refer to "'s"
|
||||
print(f"a -> b, mapping: {align.x2y.data}") # array([0, 1, 2, 3, 4, 4, 5, 6]) : two tokens both refer to "'s"
|
||||
print(f"b -> a, lengths: {align.y2x.lengths}") # array([1, 1, 1, 1, 2, 1, 1]) : the token "'s" refers to two tokens
|
||||
print(f"b -> a, mappings: {align.y2x.dataXd}") # array([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
print(f"b -> a, mappings: {align.y2x.data}") # array([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
```
|
||||
|
||||
Here are some insights from the alignment information generated in the example
|
||||
|
|
@ -1433,10 +1433,10 @@ above:
|
|||
- The one-to-one mappings for the first four tokens are identical, which means
|
||||
they map to each other. This makes sense because they're also identical in the
|
||||
input: `"i"`, `"listened"`, `"to"` and `"obama"`.
|
||||
- The value of `x2y.dataXd[6]` is `5`, which means that `other_tokens[6]`
|
||||
- The value of `x2y.data[6]` is `5`, which means that `other_tokens[6]`
|
||||
(`"podcasts"`) aligns to `spacy_tokens[5]` (also `"podcasts"`).
|
||||
- `x2y.dataXd[4]` and `x2y.dataXd[5]` are both `4`, which means that both tokens
|
||||
4 and 5 of `other_tokens` (`"'"` and `"s"`) align to token 4 of `spacy_tokens`
|
||||
- `x2y.data[4]` and `x2y.data[5]` are both `4`, which means that both tokens 4
|
||||
and 5 of `other_tokens` (`"'"` and `"s"`) align to token 4 of `spacy_tokens`
|
||||
(`"'s"`).
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
|
|
|||
|
|
@ -148,6 +148,13 @@ skipped. You can also set `--force` to force re-running a command, or `--dry` to
|
|||
perform a "dry run" and see what would happen (without actually running the
|
||||
script).
|
||||
|
||||
Since spaCy v3.4.2, `spacy projects run` checks your installed dependencies to
|
||||
verify that your environment is properly set up and aligns with the project's
|
||||
`requirements.txt`, if there is one. If missing or conflicting dependencies are
|
||||
detected, a corresponding warning is displayed. If you'd like to disable the
|
||||
dependency check, set `check_requirements: false` in your project's
|
||||
`project.yml`.
|
||||
|
||||
### 4. Run a workflow {#run-workfow}
|
||||
|
||||
> #### project.yml
|
||||
|
|
@ -226,17 +233,18 @@ pipelines.
|
|||
```yaml
|
||||
%%GITHUB_PROJECTS/pipelines/tagger_parser_ud/project.yml
|
||||
```
|
||||
|
||||
> #### Tip: Overriding variables on the CLI
|
||||
>
|
||||
> If you want to override one or more variables on the CLI and are not already specifying a
|
||||
> project directory, you need to add `.` as a placeholder:
|
||||
> If you want to override one or more variables on the CLI and are not already
|
||||
> specifying a project directory, you need to add `.` as a placeholder:
|
||||
>
|
||||
> ```
|
||||
> python -m spacy project run test . --vars.foo bar
|
||||
> ```
|
||||
|
||||
| Section | Description |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `title` | An optional project title used in `--help` message and [auto-generated docs](#custom-docs). |
|
||||
| `description` | An optional project description used in [auto-generated docs](#custom-docs). |
|
||||
| `vars` | A dictionary of variables that can be referenced in paths, URLs and scripts and overriden on the CLI, just like [`config.cfg` variables](/usage/training#config-interpolation). For example, `${vars.name}` will use the value of the variable `name`. Variables need to be defined in the section `vars`, but can be a nested dict, so you're able to reference `${vars.model.name}`. |
|
||||
|
|
@ -246,6 +254,7 @@ pipelines.
|
|||
| `workflows` | A dictionary of workflow names, mapped to a list of command names, to execute in order. Workflows can be run with the [`project run`](/api/cli#project-run) command. |
|
||||
| `commands` | A list of named commands. A command can define an optional help message (shown in the CLI when the user adds `--help`) and the `script`, a list of commands to run. The `deps` and `outputs` let you define the created file the command depends on and produces, respectively. This lets spaCy determine whether a command needs to be re-run because its dependencies or outputs changed. Commands can be run as part of a workflow, or separately with the [`project run`](/api/cli#project-run) command. |
|
||||
| `spacy_version` | Optional spaCy version range like `>=3.0.0,<3.1.0` that the project is compatible with. If it's loaded with an incompatible version, an error is raised when the project is loaded. |
|
||||
| `check_requirements` <Tag variant="new">3.4.2</Tag> | A flag determining whether to verify that the installed dependencies align with the project's `requirements.txt`. Defaults to `true`. |
|
||||
|
||||
### Data assets {#data-assets}
|
||||
|
||||
|
|
@ -758,7 +767,7 @@ and [`dvc repro`](https://dvc.org/doc/command-reference/repro) to reproduce the
|
|||
workflow or individual commands.
|
||||
|
||||
```cli
|
||||
$ python -m spacy project dvc [workflow_name]
|
||||
$ python -m spacy project dvc [project_dir] [workflow_name]
|
||||
```
|
||||
|
||||
<Infobox title="Important note for multiple workflows" variant="warning">
|
||||
|
|
|
|||
|
|
@ -480,7 +480,7 @@ as-is. They are also excluded when calling
|
|||
> parse. So the evaluation results should always reflect what your pipeline will
|
||||
> produce at runtime. If you want a frozen component to run (without updating)
|
||||
> during training as well, so that downstream components can use its
|
||||
> **predictions**, you can add it to the list of
|
||||
> **predictions**, you should add it to the list of
|
||||
> [`annotating_components`](/usage/training#annotating-components).
|
||||
|
||||
```ini
|
||||
|
|
|
|||
|
|
@ -65,10 +65,10 @@ The English CNN pipelines have new word vectors:
|
|||
|
||||
| Package | Model Version | TAG | Parser LAS | NER F |
|
||||
| ----------------------------------------------- | ------------- | ---: | ---------: | ----: |
|
||||
| [`en_core_news_md`](/models/en#en_core_news_md) | v3.3.0 | 97.3 | 90.1 | 84.6 |
|
||||
| [`en_core_news_md`](/models/en#en_core_news_lg) | v3.4.0 | 97.2 | 90.3 | 85.5 |
|
||||
| [`en_core_news_lg`](/models/en#en_core_news_md) | v3.3.0 | 97.4 | 90.1 | 85.3 |
|
||||
| [`en_core_news_lg`](/models/en#en_core_news_lg) | v3.4.0 | 97.3 | 90.2 | 85.6 |
|
||||
| [`en_core_web_md`](/models/en#en_core_web_md) | v3.3.0 | 97.3 | 90.1 | 84.6 |
|
||||
| [`en_core_web_md`](/models/en#en_core_web_lg) | v3.4.0 | 97.2 | 90.3 | 85.5 |
|
||||
| [`en_core_web_lg`](/models/en#en_core_web_md) | v3.3.0 | 97.4 | 90.1 | 85.3 |
|
||||
| [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.4.0 | 97.3 | 90.2 | 85.6 |
|
||||
|
||||
## Notes about upgrading from v3.3 {#upgrading}
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,6 @@
|
|||
{ "text": "New in v3.0", "url": "/usage/v3" },
|
||||
{ "text": "New in v3.1", "url": "/usage/v3-1" },
|
||||
{ "text": "New in v3.2", "url": "/usage/v3-2" },
|
||||
{ "text": "New in v3.2", "url": "/usage/v3-2" },
|
||||
{ "text": "New in v3.3", "url": "/usage/v3-3" },
|
||||
{ "text": "New in v3.4", "url": "/usage/v3-4" }
|
||||
]
|
||||
|
|
@ -95,6 +94,7 @@
|
|||
"label": "Pipeline",
|
||||
"items": [
|
||||
{ "text": "AttributeRuler", "url": "/api/attributeruler" },
|
||||
{ "text": "CoreferenceResolver", "url": "/api/coref" },
|
||||
{ "text": "DependencyParser", "url": "/api/dependencyparser" },
|
||||
{ "text": "EditTreeLemmatizer", "url": "/api/edittreelemmatizer" },
|
||||
{ "text": "EntityLinker", "url": "/api/entitylinker" },
|
||||
|
|
@ -105,6 +105,7 @@
|
|||
{ "text": "SentenceRecognizer", "url": "/api/sentencerecognizer" },
|
||||
{ "text": "Sentencizer", "url": "/api/sentencizer" },
|
||||
{ "text": "SpanCategorizer", "url": "/api/spancategorizer" },
|
||||
{ "text": "SpanResolver", "url": "/api/span-resolver" },
|
||||
{ "text": "SpanRuler", "url": "/api/spanruler" },
|
||||
{ "text": "Tagger", "url": "/api/tagger" },
|
||||
{ "text": "TextCategorizer", "url": "/api/textcategorizer" },
|
||||
|
|
|
|||
|
|
@ -1,5 +1,62 @@
|
|||
{
|
||||
"resources": [
|
||||
{
|
||||
"id": "Zshot",
|
||||
"title": "Zshot",
|
||||
"slogan": "Zero and Few shot named entity & relationships recognition",
|
||||
"github": "ibm/zshot",
|
||||
"pip": "zshot",
|
||||
"code_example": [
|
||||
"import spacy",
|
||||
"from zshot import PipelineConfig, displacy",
|
||||
"from zshot.linker import LinkerRegen",
|
||||
"from zshot.mentions_extractor import MentionsExtractorSpacy",
|
||||
"from zshot.utils.data_models import Entity",
|
||||
"",
|
||||
"nlp = spacy.load('en_core_web_sm')",
|
||||
"# zero shot definition of entities",
|
||||
"nlp_config = PipelineConfig(",
|
||||
" mentions_extractor=MentionsExtractorSpacy(),",
|
||||
" linker=LinkerRegen(),",
|
||||
" entities=[",
|
||||
" Entity(name='Paris',",
|
||||
" description='Paris is located in northern central France, in a north-bending arc of the river Seine'),",
|
||||
" Entity(name='IBM',",
|
||||
" description='International Business Machines Corporation (IBM) is an American multinational technology corporation headquartered in Armonk, New York'),",
|
||||
" Entity(name='New York', description='New York is a city in U.S. state'),",
|
||||
" Entity(name='Florida', description='southeasternmost U.S. state'),",
|
||||
" Entity(name='American',",
|
||||
" description='American, something of, from, or related to the United States of America, commonly known as the United States or America'),",
|
||||
" Entity(name='Chemical formula',",
|
||||
" description='In chemistry, a chemical formula is a way of presenting information about the chemical proportions of atoms that constitute a particular chemical compound or molecul'),",
|
||||
" Entity(name='Acetamide',",
|
||||
" description='Acetamide (systematic name: ethanamide) is an organic compound with the formula CH3CONH2. It is the simplest amide derived from acetic acid. It finds some use as a plasticizer and as an industrial solvent.'),",
|
||||
" Entity(name='Armonk',",
|
||||
" description='Armonk is a hamlet and census-designated place (CDP) in the town of North Castle, located in Westchester County, New York, United States.'),",
|
||||
" Entity(name='Acetic Acid',",
|
||||
" description='Acetic acid, systematically named ethanoic acid, is an acidic, colourless liquid and organic compound with the chemical formula CH3COOH'),",
|
||||
" Entity(name='Industrial solvent',",
|
||||
" description='Acetamide (systematic name: ethanamide) is an organic compound with the formula CH3CONH2. It is the simplest amide derived from acetic acid. It finds some use as a plasticizer and as an industrial solvent.'),",
|
||||
" ]",
|
||||
")",
|
||||
"nlp.add_pipe('zshot', config=nlp_config, last=True)",
|
||||
"",
|
||||
"text = 'International Business Machines Corporation (IBM) is an American multinational technology corporation' \\",
|
||||
" ' headquartered in Armonk, New York, with operations in over 171 countries.'",
|
||||
"",
|
||||
"doc = nlp(text)",
|
||||
"displacy.serve(doc, style='ent')"
|
||||
],
|
||||
"thumb": "https://ibm.github.io/zshot/img/graph.png",
|
||||
"url": "https://ibm.github.io/zshot/",
|
||||
"author": "IBM Research",
|
||||
"author_links": {
|
||||
"github": "ibm",
|
||||
"twitter": "IBMResearch",
|
||||
"website": "https://research.ibm.com/labs/ireland/"
|
||||
},
|
||||
"category": ["scientific", "models", "research"]
|
||||
},
|
||||
{
|
||||
"id": "concepcy",
|
||||
"title": "concepCy",
|
||||
|
|
@ -3984,7 +4041,21 @@
|
|||
},
|
||||
"category": ["pipeline"],
|
||||
"tags": ["interpretation", "ja"]
|
||||
},
|
||||
{
|
||||
"id": "spacy-partial-tagger",
|
||||
"title": "spaCy - Partial Tagger",
|
||||
"slogan": "Sequence Tagger for Partially Annotated Dataset in spaCy",
|
||||
"description": "This is a library to build a CRF tagger with a partially annotated dataset in spaCy. You can build your own tagger only from dictionary.",
|
||||
"github": "doccano/spacy-partial-tagger",
|
||||
"pip": "spacy-partial-tagger",
|
||||
"category": ["pipeline", "training"],
|
||||
"author": "Yasufumi Taniguchi",
|
||||
"author_links": {
|
||||
"github": "yasufumy"
|
||||
}
|
||||
}
|
||||
|
||||
],
|
||||
|
||||
"categories": [
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ const DEFAULT_PLATFORM = 'x86'
|
|||
const DEFAULT_MODELS = ['en']
|
||||
const DEFAULT_OPT = 'efficiency'
|
||||
const DEFAULT_HARDWARE = 'cpu'
|
||||
const DEFAULT_CUDA = 'cuda113'
|
||||
const DEFAULT_CUDA = 'cuda-autodetect'
|
||||
const CUDA = {
|
||||
'8.0': 'cuda80',
|
||||
'9.0': 'cuda90',
|
||||
|
|
@ -17,15 +17,7 @@ const CUDA = {
|
|||
'9.2': 'cuda92',
|
||||
'10.0': 'cuda100',
|
||||
'10.1': 'cuda101',
|
||||
'10.2': 'cuda102',
|
||||
'11.0': 'cuda110',
|
||||
'11.1': 'cuda111',
|
||||
'11.2': 'cuda112',
|
||||
'11.3': 'cuda113',
|
||||
'11.4': 'cuda114',
|
||||
'11.5': 'cuda115',
|
||||
'11.6': 'cuda116',
|
||||
'11.7': 'cuda117',
|
||||
'10.2, 11.0+': 'cuda-autodetect',
|
||||
}
|
||||
const LANG_EXTRAS = ['ja'] // only for languages with models
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user