From afd7a2476d2491af864d0723bff96191ea61b429 Mon Sep 17 00:00:00 2001 From: Damian Romero <12145757+damian-romero@users.noreply.github.com> Date: Thu, 1 Dec 2022 07:06:28 -0500 Subject: [PATCH 01/44] Fix typo in vocab.md table (#11908) * Fix typo in vocab.md table Fixes explosion/spaCy/#11907 * Reformat vocab.md with Prettier --- website/docs/api/vocab.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/website/docs/api/vocab.md b/website/docs/api/vocab.md index afbd1301d..5e4de219a 100644 --- a/website/docs/api/vocab.md +++ b/website/docs/api/vocab.md @@ -308,14 +308,14 @@ Load state from a binary string. > assert type(PERSON) == int > ``` -| Name | Description | -| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `strings` | A table managing the string-to-int mapping. ~~StringStore~~ | -| `vectors` | A table associating word IDs to word vectors. ~~Vectors~~ | -| `vectors_length` | Number of dimensions for each word vector. ~~int~~ | -| `lookups` | The available lookup tables in this vocab. ~~Lookups~~ | -| `writing_system` | A dict with information about the language's writing system. ~~Dict[str, Any]~~ | -| `get_noun_chunks` 3.0 | A function that yields base noun phrases used for [`Doc.noun_chunks`](/ap/doc#noun_chunks). ~~Optional[Callable[[Union[Doc, Span], Iterator[Tuple[int, int, int]]]]]~~ | +| Name | Description | +| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `strings` | A table managing the string-to-int mapping. ~~StringStore~~ | +| `vectors` | A table associating word IDs to word vectors. ~~Vectors~~ | +| `vectors_length` | Number of dimensions for each word vector. ~~int~~ | +| `lookups` | The available lookup tables in this vocab. ~~Lookups~~ | +| `writing_system` | A dict with information about the language's writing system. ~~Dict[str, Any]~~ | +| `get_noun_chunks` 3.0 | A function that yields base noun phrases used for [`Doc.noun_chunks`](/api/doc#noun_chunks). ~~Optional[Callable[[Union[Doc, Span], Iterator[Tuple[int, int, int]]]]]~~ | ## Serialization fields {#serialization-fields} From 9cf3fa9711dfff94e88d6e137a52ebabdcceaad8 Mon Sep 17 00:00:00 2001 From: Zhangrp Date: Thu, 1 Dec 2022 20:30:27 +0800 Subject: [PATCH 02/44] Add docs for biluo_to_iob and iob_to_biluo. (#11901) * Add docs for biluo_to_iob and iob_to_biluo. * Fix typos. * Remove redundant links. --- website/docs/api/top-level.md | 48 +++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/website/docs/api/top-level.md b/website/docs/api/top-level.md index 211affa4a..26a5d42f4 100644 --- a/website/docs/api/top-level.md +++ b/website/docs/api/top-level.md @@ -1004,6 +1004,54 @@ This method was previously available as `spacy.gold.spans_from_biluo_tags`. | `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags with each tag describing one token. Each tag string will be of the form of either `""`, `"O"` or `"{action}-{label}"`, where action is one of `"B"`, `"I"`, `"L"`, `"U"`. ~~List[str]~~ | | **RETURNS** | A sequence of `Span` objects with added entity labels. ~~List[Span]~~ | +### training.biluo_to_iob {#biluo_to_iob tag="function"} + +Convert a sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags to +[IOB](/usage/linguistic-features#accessing-ner) tags. This is useful if you want +use the BILUO tags with a model that only supports IOB tags. + +> #### Example +> +> ```python +> from spacy.training import biluo_to_iob +> +> tags = ["O", "O", "B-LOC", "I-LOC", "L-LOC", "O"] +> iob_tags = biluo_to_iob(tags) +> assert iob_tags == ["O", "O", "B-LOC", "I-LOC", "I-LOC", "O"] +> ``` + +| Name | Description | +| ----------- | --------------------------------------------------------------------------------------- | +| `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ | +| **RETURNS** | A list of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ | + +### training.iob_to_biluo {#iob_to_biluo tag="function"} + +Convert a sequence of [IOB](/usage/linguistic-features#accessing-ner) tags to +[BILUO](/usage/linguistic-features#accessing-ner) tags. This is useful if you +want use the IOB tags with a model that only supports BILUO tags. + + + +This method was previously available as `spacy.gold.iob_to_biluo`. + + + +> #### Example +> +> ```python +> from spacy.training import iob_to_biluo +> +> tags = ["O", "O", "B-LOC", "I-LOC", "O"] +> biluo_tags = iob_to_biluo(tags) +> assert biluo_tags == ["O", "O", "B-LOC", "L-LOC", "O"] +> ``` + +| Name | Description | +| ----------- | ------------------------------------------------------------------------------------- | +| `tags` | A sequence of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ | +| **RETURNS** | A list of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ | + ## Utility functions {#util source="spacy/util.py"} spaCy comes with a small collection of utility functions located in From 445c670a2d537598b3d562fb7f444050164a260b Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Fri, 2 Dec 2022 09:33:52 +0100 Subject: [PATCH 03/44] Fix spancat for zero suggestions (#11860) * Add test for spancat predict with zero suggestions * Fix spancat for zero suggestions * Undo changes to extract_spans * Use .sum() as in update --- spacy/pipeline/spancat.py | 5 +++- spacy/tests/pipeline/test_spancat.py | 43 ++++++++++++++++++++++------ 2 files changed, 38 insertions(+), 10 deletions(-) diff --git a/spacy/pipeline/spancat.py b/spacy/pipeline/spancat.py index 0a84c72fd..a3388e81a 100644 --- a/spacy/pipeline/spancat.py +++ b/spacy/pipeline/spancat.py @@ -272,7 +272,10 @@ class SpanCategorizer(TrainablePipe): DOCS: https://spacy.io/api/spancategorizer#predict """ indices = self.suggester(docs, ops=self.model.ops) - scores = self.model.predict((docs, indices)) # type: ignore + if indices.lengths.sum() == 0: + scores = self.model.ops.alloc2f(0, 0) + else: + scores = self.model.predict((docs, indices)) # type: ignore return indices, scores def set_candidates( diff --git a/spacy/tests/pipeline/test_spancat.py b/spacy/tests/pipeline/test_spancat.py index 15256a763..e9db983d3 100644 --- a/spacy/tests/pipeline/test_spancat.py +++ b/spacy/tests/pipeline/test_spancat.py @@ -372,24 +372,39 @@ def test_overfitting_IO_overlapping(): def test_zero_suggestions(): - # Test with a suggester that returns 0 suggestions + # Test with a suggester that can return 0 suggestions - @registry.misc("test_zero_suggester") - def make_zero_suggester(): - def zero_suggester(docs, *, ops=None): + @registry.misc("test_mixed_zero_suggester") + def make_mixed_zero_suggester(): + def mixed_zero_suggester(docs, *, ops=None): if ops is None: ops = get_current_ops() - return Ragged( - ops.xp.zeros((0, 0), dtype="i"), ops.xp.zeros((len(docs),), dtype="i") - ) + spans = [] + lengths = [] + for doc in docs: + if len(doc) > 0 and len(doc) % 2 == 0: + spans.append((0, 1)) + lengths.append(1) + else: + lengths.append(0) + spans = ops.asarray2i(spans) + lengths_array = ops.asarray1i(lengths) + if len(spans) > 0: + output = Ragged(ops.xp.vstack(spans), lengths_array) + else: + output = Ragged(ops.xp.zeros((0, 0), dtype="i"), lengths_array) + return output - return zero_suggester + return mixed_zero_suggester fix_random_seed(0) nlp = English() spancat = nlp.add_pipe( "spancat", - config={"suggester": {"@misc": "test_zero_suggester"}, "spans_key": SPAN_KEY}, + config={ + "suggester": {"@misc": "test_mixed_zero_suggester"}, + "spans_key": SPAN_KEY, + }, ) train_examples = make_examples(nlp) optimizer = nlp.initialize(get_examples=lambda: train_examples) @@ -397,6 +412,16 @@ def test_zero_suggestions(): assert set(spancat.labels) == {"LOC", "PERSON"} nlp.update(train_examples, sgd=optimizer) + # empty doc + nlp("") + # single doc with zero suggestions + nlp("one") + # single doc with one suggestion + nlp("two two") + # batch with mixed zero/one suggestions + list(nlp.pipe(["one", "two two", "three three three", "", "four four four four"])) + # batch with no suggestions + list(nlp.pipe(["", "one", "three three three"])) def test_set_candidates(): From f9d17a644b3d037924f715c03672ada6d12e4d86 Mon Sep 17 00:00:00 2001 From: Paul O'Leary McCann Date: Fri, 2 Dec 2022 18:17:11 +0900 Subject: [PATCH 04/44] Config generation fails for GPU without transformers (#11899) If you don't have spacy-transformers installed, but try to use `init config` with the GPU flag, you'll get an error. The issue is that the `use_transformers` flag in the config is conflated with the GPU flag, and then there's an attempt to access transformers config info that may not exist. There may be a better way to do this, but this stops the error. --- spacy/cli/templates/quickstart_training.jinja | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spacy/cli/templates/quickstart_training.jinja b/spacy/cli/templates/quickstart_training.jinja index 58864883a..b961ac892 100644 --- a/spacy/cli/templates/quickstart_training.jinja +++ b/spacy/cli/templates/quickstart_training.jinja @@ -1,7 +1,7 @@ {# This is a template for training configs used for the quickstart widget in the docs and the init config command. It encodes various best practices and can help generate the best possible configuration, given a user's requirements. #} -{%- set use_transformer = hardware != "cpu" -%} +{%- set use_transformer = hardware != "cpu" and transformer_data -%} {%- set transformer = transformer_data[optimize] if use_transformer else {} -%} {%- set listener_components = ["tagger", "morphologizer", "parser", "ner", "textcat", "textcat_multilabel", "entity_linker", "spancat", "trainable_lemmatizer"] -%} [paths] From df0cb4b77be6e20a62143f5f65c3e165a4a45bcc Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 2 Dec 2022 14:49:12 +0100 Subject: [PATCH 05/44] Auto-format code with black (#11913) Co-authored-by: explosion-bot --- spacy/util.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/spacy/util.py b/spacy/util.py index cba403361..8d211a9a5 100644 --- a/spacy/util.py +++ b/spacy/util.py @@ -1643,7 +1643,9 @@ def _pipe( docs: Iterable["Doc"], proc: "PipeCallable", name: str, - default_error_handler: Callable[[str, "PipeCallable", List["Doc"], Exception], NoReturn], + default_error_handler: Callable[ + [str, "PipeCallable", List["Doc"], Exception], NoReturn + ], kwargs: Mapping[str, Any], ) -> Iterator["Doc"]: if hasattr(proc, "pipe"): From 4b2097a2713b548cba1c841fa5cb8f6f42e3e30f Mon Sep 17 00:00:00 2001 From: Sofie Van Landeghem Date: Mon, 5 Dec 2022 08:29:13 +0100 Subject: [PATCH 06/44] fix links (#11927) --- website/docs/usage/v3-4.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/usage/v3-4.md b/website/docs/usage/v3-4.md index 597fc3cc8..e10110b71 100644 --- a/website/docs/usage/v3-4.md +++ b/website/docs/usage/v3-4.md @@ -66,8 +66,8 @@ The English CNN pipelines have new word vectors: | Package | Model Version | TAG | Parser LAS | NER F | | ----------------------------------------------- | ------------- | ---: | ---------: | ----: | | [`en_core_web_md`](/models/en#en_core_web_md) | v3.3.0 | 97.3 | 90.1 | 84.6 | -| [`en_core_web_md`](/models/en#en_core_web_lg) | v3.4.0 | 97.2 | 90.3 | 85.5 | -| [`en_core_web_lg`](/models/en#en_core_web_md) | v3.3.0 | 97.4 | 90.1 | 85.3 | +| [`en_core_web_md`](/models/en#en_core_web_md) | v3.4.0 | 97.2 | 90.3 | 85.5 | +| [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.3.0 | 97.4 | 90.1 | 85.3 | | [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.4.0 | 97.3 | 90.2 | 85.6 | ## Notes about upgrading from v3.3 {#upgrading} From 5848656b5e3287d77674ce678e321eadea52f68e Mon Sep 17 00:00:00 2001 From: Paul O'Leary McCann Date: Mon, 5 Dec 2022 17:43:23 +0900 Subject: [PATCH 07/44] Switch ubuntu-latest to ubuntu-20.04 in main tests (#11928) * Switch ubuntu-latest to ubuntu-20.04 in main tests * Only use 20.04 for 3.6 --- azure-pipelines.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/azure-pipelines.yml b/azure-pipelines.yml index 9c3b92f06..0f7ea91f9 100644 --- a/azure-pipelines.yml +++ b/azure-pipelines.yml @@ -41,7 +41,7 @@ jobs: matrix: # We're only running one platform per Python version to speed up builds Python36Linux: - imageName: "ubuntu-latest" + imageName: "ubuntu-20.04" python.version: "3.6" # Python36Windows: # imageName: "windows-latest" @@ -50,7 +50,7 @@ jobs: # imageName: "macos-latest" # python.version: "3.6" # Python37Linux: - # imageName: "ubuntu-latest" + # imageName: "ubuntu-20.04" # python.version: "3.7" Python37Windows: imageName: "windows-latest" From 6f342bdd72f300cdc431d0e0f2a168c62fd2a861 Mon Sep 17 00:00:00 2001 From: Darigov Research <30328618+darigovresearch@users.noreply.github.com> Date: Mon, 5 Dec 2022 08:49:04 +0000 Subject: [PATCH 08/44] docs: Adds link to license in readme (#11924) Would resolve https://github.com/explosion/spaCy/issues/11923 if merged --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index abfc3da67..7595460fb 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ parsing, **named entity recognition**, **text classification** and more, multi-task learning with pretrained **transformers** like BERT, as well as a production-ready [**training system**](https://spacy.io/usage/training) and easy model packaging, deployment and workflow management. spaCy is commercial -open-source software, released under the MIT license. +open-source software, released under the [MIT license](https://github.com/explosion/spaCy/blob/master/LICENSE). 💫 **Version 3.4 out now!** [Check out the release notes here.](https://github.com/explosion/spaCy/releases) From 8afa8b5a7b8ee51eb42b83dabd0f3c1276369e73 Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Mon, 5 Dec 2022 10:00:00 +0100 Subject: [PATCH 09/44] Refactor kwargs in CLI msg for future wasabi compatibility (#11918) Necessary for mypy with wasabi v1+. --- spacy/cli/project/run.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spacy/cli/project/run.py b/spacy/cli/project/run.py index a109c4a5a..6dd174902 100644 --- a/spacy/cli/project/run.py +++ b/spacy/cli/project/run.py @@ -101,8 +101,8 @@ def project_run( if not (project_dir / dep).exists(): err = f"Missing dependency specified by command '{subcommand}': {dep}" err_help = "Maybe you forgot to run the 'project assets' command or a previous step?" - err_kwargs = {"exits": 1} if not dry else {} - msg.fail(err, err_help, **err_kwargs) + err_exits = 1 if not dry else None + msg.fail(err, err_help, exits=err_exits) check_spacy_commit = check_bool_env_var(ENV_VARS.PROJECT_USE_GIT_VERSION) with working_dir(project_dir) as current_dir: msg.divider(subcommand) From 1aadcfcb37ba166558688782fabbcbe3e32ea020 Mon Sep 17 00:00:00 2001 From: Ryn Daniels <397565+ryndaniels@users.noreply.github.com> Date: Mon, 5 Dec 2022 11:17:10 +0200 Subject: [PATCH 10/44] update lock-threads to v4 (#11930) --- .github/workflows/lock.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/lock.yml b/.github/workflows/lock.yml index c9833cdba..794adee85 100644 --- a/.github/workflows/lock.yml +++ b/.github/workflows/lock.yml @@ -15,11 +15,11 @@ jobs: action: runs-on: ubuntu-latest steps: - - uses: dessant/lock-threads@v3 + - uses: dessant/lock-threads@v4 with: process-only: 'issues' issue-inactive-days: '30' - issue-comment: > - This thread has been automatically locked since there - has not been any recent activity after it was closed. + issue-comment: > + This thread has been automatically locked since there + has not been any recent activity after it was closed. Please open a new issue for related bugs. From 23085ffef4bba62aff0de5993ff405cb3ff3528c Mon Sep 17 00:00:00 2001 From: Zhangrp Date: Tue, 6 Dec 2022 16:42:12 +0800 Subject: [PATCH 11/44] Fix interpolation in directory names, see #11235. (#11914) --- spacy/cli/_util.py | 8 ++++---- spacy/tests/test_cli.py | 19 +++++++++++++++++++ 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/spacy/cli/_util.py b/spacy/cli/_util.py index 7ce006108..9b97a9f19 100644 --- a/spacy/cli/_util.py +++ b/spacy/cli/_util.py @@ -158,15 +158,15 @@ def load_project_config( sys.exit(1) validate_project_version(config) validate_project_commands(config) + if interpolate: + err = f"{PROJECT_FILE} validation error" + with show_validation_error(title=err, hint_fill=False): + config = substitute_project_variables(config, overrides) # Make sure directories defined in config exist for subdir in config.get("directories", []): dir_path = path / subdir if not dir_path.exists(): dir_path.mkdir(parents=True) - if interpolate: - err = f"{PROJECT_FILE} validation error" - with show_validation_error(title=err, hint_fill=False): - config = substitute_project_variables(config, overrides) return config diff --git a/spacy/tests/test_cli.py b/spacy/tests/test_cli.py index 2e706458f..3104b49ff 100644 --- a/spacy/tests/test_cli.py +++ b/spacy/tests/test_cli.py @@ -123,6 +123,25 @@ def test_issue7055(): assert "model" in filled_cfg["components"]["ner"] +@pytest.mark.issue(11235) +def test_issue11235(): + """ + Test that the cli handles interpolation in the directory names correctly when loading project config. + """ + lang_var = "en" + variables = {"lang": lang_var} + commands = [{"name": "x", "script": ["hello ${vars.lang}"]}] + directories = ["cfg", "${vars.lang}_model"] + project = {"commands": commands, "vars": variables, "directories": directories} + with make_tempdir() as d: + srsly.write_yaml(d / "project.yml", project) + cfg = load_project_config(d) + # Check that the directories are interpolated and created correctly + assert os.path.exists(d / "cfg") + assert os.path.exists(d / f"{lang_var}_model") + assert cfg["commands"][0]["script"][0] == f"hello {lang_var}" + + def test_cli_info(): nlp = Dutch() nlp.add_pipe("textcat") From 27fac7df2e67a0dbfefd68834c14fb1f9505da49 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20de=20Kok?= Date: Wed, 7 Dec 2022 05:53:41 +0100 Subject: [PATCH 12/44] EditTreeLemmatizer: correctly add strings when initializing from labels (#11934) Strings in replacement nodes where not added to the `StringStore` when `EditTreeLemmatizer` was initialized from a set of labels. The corresponding test did not capture this because it added the strings through the examples that were passed to the initialization. This change fixes both this bug in the initialization as the 'shadowing' of the bug in the test. --- spacy/pipeline/edit_tree_lemmatizer.py | 4 +- .../pipeline/test_edit_tree_lemmatizer.py | 37 ++++++++++++++++++- 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/spacy/pipeline/edit_tree_lemmatizer.py b/spacy/pipeline/edit_tree_lemmatizer.py index 12f9b73a3..a56c9975e 100644 --- a/spacy/pipeline/edit_tree_lemmatizer.py +++ b/spacy/pipeline/edit_tree_lemmatizer.py @@ -328,9 +328,9 @@ class EditTreeLemmatizer(TrainablePipe): tree = dict(tree) if "orig" in tree: - tree["orig"] = self.vocab.strings[tree["orig"]] + tree["orig"] = self.vocab.strings.add(tree["orig"]) if "orig" in tree: - tree["subst"] = self.vocab.strings[tree["subst"]] + tree["subst"] = self.vocab.strings.add(tree["subst"]) trees.append(tree) diff --git a/spacy/tests/pipeline/test_edit_tree_lemmatizer.py b/spacy/tests/pipeline/test_edit_tree_lemmatizer.py index cf541e301..b12ca5dd4 100644 --- a/spacy/tests/pipeline/test_edit_tree_lemmatizer.py +++ b/spacy/tests/pipeline/test_edit_tree_lemmatizer.py @@ -60,10 +60,45 @@ def test_initialize_from_labels(): nlp2 = Language() lemmatizer2 = nlp2.add_pipe("trainable_lemmatizer") lemmatizer2.initialize( - get_examples=lambda: train_examples, + # We want to check that the strings in replacement nodes are + # added to the string store. Avoid that they get added through + # the examples. + get_examples=lambda: train_examples[:1], labels=lemmatizer.label_data, ) assert lemmatizer2.tree2label == {1: 0, 3: 1, 4: 2, 6: 3} + assert lemmatizer2.label_data == { + "trees": [ + {"orig": "S", "subst": "s"}, + { + "prefix_len": 1, + "suffix_len": 0, + "prefix_tree": 0, + "suffix_tree": 4294967295, + }, + {"orig": "s", "subst": ""}, + { + "prefix_len": 0, + "suffix_len": 1, + "prefix_tree": 4294967295, + "suffix_tree": 2, + }, + { + "prefix_len": 0, + "suffix_len": 0, + "prefix_tree": 4294967295, + "suffix_tree": 4294967295, + }, + {"orig": "E", "subst": "e"}, + { + "prefix_len": 1, + "suffix_len": 0, + "prefix_tree": 5, + "suffix_tree": 4294967295, + }, + ], + "labels": (1, 3, 4, 6), + } def test_no_data(): From 916191848ab7bf90e88f23401451695f61903112 Mon Sep 17 00:00:00 2001 From: Paul O'Leary McCann Date: Wed, 7 Dec 2022 18:09:04 +0900 Subject: [PATCH 13/44] Update scattertext example code (#11937) * Update scattertext example code * Remove PMI Filter Threshold --- website/meta/universe.json | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/website/meta/universe.json b/website/meta/universe.json index 97b53e9c5..8ca657561 100644 --- a/website/meta/universe.json +++ b/website/meta/universe.json @@ -1468,13 +1468,26 @@ "image": "https://jasonkessler.github.io/2012conventions0.0.2.2.png", "code_example": [ "import spacy", - "import scattertext as st", "", - "nlp = spacy.load('en')", - "corpus = st.CorpusFromPandas(convention_df,", - " category_col='party',", - " text_col='text',", - " nlp=nlp).build()" + "from scattertext import SampleCorpora, produce_scattertext_explorer", + "from scattertext import produce_scattertext_html", + "from scattertext.CorpusFromPandas import CorpusFromPandas", + "", + "nlp = spacy.load('en_core_web_sm')", + "convention_df = SampleCorpora.ConventionData2012.get_data()", + "corpus = CorpusFromPandas(convention_df,", + " category_col='party',", + " text_col='text',", + " nlp=nlp).build()", + "", + "html = produce_scattertext_html(corpus,", + " category='democrat',", + " category_name='Democratic',", + " not_category_name='Republican',", + " minimum_term_frequency=5,", + " width_in_pixels=1000)", + "open('./simple.html', 'wb').write(html.encode('utf-8'))", + "print('Open ./simple.html in Chrome or Firefox.')" ], "author": "Jason Kessler", "author_links": { From 5c3a60e8f4273aff7bd47bce01d62c8224967045 Mon Sep 17 00:00:00 2001 From: Paul O'Leary McCann Date: Wed, 7 Dec 2022 23:52:35 +0900 Subject: [PATCH 14/44] Add in errors used in the beam code that were removed at some point (#11935) I don't think there's any way to use the beam code at the moment, but as long as it's around the errors it refers to should also be present. --- spacy/errors.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/spacy/errors.py b/spacy/errors.py index e34614b0f..0e5ef91ed 100644 --- a/spacy/errors.py +++ b/spacy/errors.py @@ -345,6 +345,11 @@ class Errors(metaclass=ErrorsWithCodes): "clear the existing vectors and resize the table.") E074 = ("Error interpreting compiled match pattern: patterns are expected " "to end with the attribute {attr}. Got: {bad_attr}.") + E079 = ("Error computing states in beam: number of predicted beams " + "({pbeams}) does not equal number of gold beams ({gbeams}).") + E080 = ("Duplicate state found in beam: {key}.") + E081 = ("Error getting gradient in beam: number of histories ({n_hist}) " + "does not equal number of losses ({losses}).") E082 = ("Error deprojectivizing parse: number of heads ({n_heads}), " "projective heads ({n_proj_heads}) and labels ({n_labels}) do not " "match.") From 73919336fb1b003425373a07d41e5541dc5c3c46 Mon Sep 17 00:00:00 2001 From: Paul O'Leary McCann Date: Wed, 7 Dec 2022 23:56:03 +0900 Subject: [PATCH 15/44] Remove spacy-sentence-segmenter from Universe (#11932) --- website/meta/universe.json | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/website/meta/universe.json b/website/meta/universe.json index 8ca657561..db533c3b2 100644 --- a/website/meta/universe.json +++ b/website/meta/universe.json @@ -1023,25 +1023,6 @@ }, "category": ["pipeline"] }, - { - "id": "spacy-sentence-segmenter", - "title": "Sentence Segmenter", - "slogan": "Custom sentence segmentation for spaCy", - "code_example": [ - "from seg.newline.segmenter import NewLineSegmenter", - "import spacy", - "", - "nlseg = NewLineSegmenter()", - "nlp = spacy.load('en')", - "nlp.add_pipe(nlseg.set_sent_starts, name='sentence_segmenter', before='parser')", - "doc = nlp(my_doc_text)" - ], - "author": "tc64", - "author_links": { - "github": "tc64" - }, - "category": ["pipeline"] - }, { "id": "spacy_cld", "title": "spaCy-CLD", From 6d2ca1ab3a545491acbe058035677a263135e52a Mon Sep 17 00:00:00 2001 From: vincent d warmerdam Date: Wed, 7 Dec 2022 16:02:09 +0100 Subject: [PATCH 16/44] Update custom solutions links (#11903) * Update custom solutions Will now point to https://explosion.ai/custom-solutions * added-sidebar * added-analysis-to-readme * update-landing-page --- README.md | 2 ++ website/meta/sidebars.json | 2 +- website/meta/site.json | 2 +- website/src/widgets/landing.js | 4 ++-- 4 files changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 7595460fb..195424551 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,7 @@ open-source software, released under the [MIT license](https://github.com/explos | 🛠 **[Changelog]** | Changes and version history. | | 💝 **[Contribute]** | How to contribute to the spaCy project and code base. | | spaCy Tailored Pipelines | Get a custom spaCy pipeline, tailor-made for your NLP problem by spaCy's core developers. Streamlined, production-ready, predictable and maintainable. Start by completing our 5-minute questionnaire to tell us what you need and we'll be in touch! **[Learn more →](https://explosion.ai/spacy-tailored-pipelines)** | +| spaCy Tailored Pipelines | Bespoke advice for problem solving, strategy and analysis for applied NLP projects. Services include data strategy, code reviews, pipeline design and annotation coaching. Curious? Fill in our 5-minute questionnaire to tell us what you need and we'll be in touch! **[Learn more →](https://explosion.ai/spacy-tailored-analysis)** | [spacy 101]: https://spacy.io/usage/spacy-101 [new in v3.0]: https://spacy.io/usage/v3 @@ -59,6 +60,7 @@ open-source software, released under the [MIT license](https://github.com/explos [changelog]: https://spacy.io/usage#changelog [contribute]: https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md + ## 💬 Where to ask questions The spaCy project is maintained by the [spaCy team](https://explosion.ai/about). diff --git a/website/meta/sidebars.json b/website/meta/sidebars.json index 2d8745d77..339e4085b 100644 --- a/website/meta/sidebars.json +++ b/website/meta/sidebars.json @@ -45,7 +45,7 @@ { "text": "v2.x Documentation", "url": "https://v2.spacy.io" }, { "text": "Custom Solutions", - "url": "https://explosion.ai/spacy-tailored-pipelines" + "url": "https://explosion.ai/custom-solutions" } ] } diff --git a/website/meta/site.json b/website/meta/site.json index 360a72178..fa79d3c69 100644 --- a/website/meta/site.json +++ b/website/meta/site.json @@ -51,7 +51,7 @@ { "text": "Online Course", "url": "https://course.spacy.io" }, { "text": "Custom Solutions", - "url": "https://explosion.ai/spacy-tailored-pipelines" + "url": "https://explosion.ai/custom-solutions" } ] }, diff --git a/website/src/widgets/landing.js b/website/src/widgets/landing.js index b7ae35f6e..c3aaa8a22 100644 --- a/website/src/widgets/landing.js +++ b/website/src/widgets/landing.js @@ -105,13 +105,13 @@ const Landing = ({ data }) => { - + spaCy Tailored Pipelines From f22fc7a1138545a2a75975909b5af554e8e1d616 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 9 Dec 2022 10:15:52 +0100 Subject: [PATCH 17/44] Auto-format code with black (#11955) Co-authored-by: explosion-bot --- spacy/tests/test_cli.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spacy/tests/test_cli.py b/spacy/tests/test_cli.py index 3104b49ff..42af08749 100644 --- a/spacy/tests/test_cli.py +++ b/spacy/tests/test_cli.py @@ -140,7 +140,7 @@ def test_issue11235(): assert os.path.exists(d / "cfg") assert os.path.exists(d / f"{lang_var}_model") assert cfg["commands"][0]["script"][0] == f"hello {lang_var}" - + def test_cli_info(): nlp = Dutch() From 8c291ace0c0978e70257906438d3585022090e9f Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Mon, 12 Dec 2022 08:38:36 +0100 Subject: [PATCH 18/44] Extend to wasabi v1.1 (#11945) * Extend to wasabi v1.1 * Temporarily run mypy and tests with newest wasabi * Temporarily skip check requirements test * Revert "Temporarily skip check requirements test" This reverts commit 44f4ce20a8e8c92e8bfc8042cc68333589a96253. * Revert "Temporarily run mypy and tests with newest wasabi" This reverts commit e677a2257ced55e696cafc3a8e89eb2f7ddfc369. --- requirements.txt | 2 +- setup.cfg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/requirements.txt b/requirements.txt index 778c05e21..0440835f2 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,7 +6,7 @@ preshed>=3.0.2,<3.1.0 thinc>=8.1.0,<8.2.0 ml_datasets>=0.2.0,<0.3.0 murmurhash>=0.28.0,<1.1.0 -wasabi>=0.9.1,<1.1.0 +wasabi>=0.9.1,<1.2.0 srsly>=2.4.3,<3.0.0 catalogue>=2.0.6,<2.1.0 typer>=0.3.0,<0.8.0 diff --git a/setup.cfg b/setup.cfg index 5768c9d3e..cf6e6f84b 100644 --- a/setup.cfg +++ b/setup.cfg @@ -47,7 +47,7 @@ install_requires = cymem>=2.0.2,<2.1.0 preshed>=3.0.2,<3.1.0 thinc>=8.1.0,<8.2.0 - wasabi>=0.9.1,<1.1.0 + wasabi>=0.9.1,<1.2.0 srsly>=2.4.3,<3.0.0 catalogue>=2.0.6,<2.1.0 # Third-party dependencies From 0591e67265d7378769c0fc0df4020817f2d514ec Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Mon, 12 Dec 2022 08:45:35 +0100 Subject: [PATCH 19/44] Cast to uint64 for all array-based doc representations (#11933) * Convert all individual values explicitly to uint64 for array-based doc representations * Temporarily test with latest numpy v1.24.0rc * Remove unnecessary conversion from attr_t * Reduce number of individual casts * Convert specifically from int32 to uint64 * Revert "Temporarily test with latest numpy v1.24.0rc" This reverts commit eb0e3c5006515b9a7ff52bae59484c909b8a3f65. * Also use int32 in tests --- spacy/tests/doc/test_array.py | 4 ++-- spacy/tokens/doc.pyx | 2 ++ spacy/tokens/span.pyx | 4 ++-- spacy/training/example.pyx | 15 ++++++++------- 4 files changed, 14 insertions(+), 11 deletions(-) diff --git a/spacy/tests/doc/test_array.py b/spacy/tests/doc/test_array.py index c334cc6eb..1f2d7d999 100644 --- a/spacy/tests/doc/test_array.py +++ b/spacy/tests/doc/test_array.py @@ -123,14 +123,14 @@ def test_doc_from_array_heads_in_bounds(en_vocab): # head before start arr = doc.to_array(["HEAD"]) - arr[0] = -1 + arr[0] = numpy.int32(-1).astype(numpy.uint64) doc_from_array = Doc(en_vocab, words=words) with pytest.raises(ValueError): doc_from_array.from_array(["HEAD"], arr) # head after end arr = doc.to_array(["HEAD"]) - arr[0] = 5 + arr[0] = numpy.int32(5).astype(numpy.uint64) doc_from_array = Doc(en_vocab, words=words) with pytest.raises(ValueError): doc_from_array.from_array(["HEAD"], arr) diff --git a/spacy/tokens/doc.pyx b/spacy/tokens/doc.pyx index f2621292c..075bc4d15 100644 --- a/spacy/tokens/doc.pyx +++ b/spacy/tokens/doc.pyx @@ -359,6 +359,7 @@ cdef class Doc: for annot in annotations: if annot: if annot is heads or annot is sent_starts or annot is ent_iobs: + annot = numpy.array(annot, dtype=numpy.int32).astype(numpy.uint64) for i in range(len(words)): if attrs.ndim == 1: attrs[i] = annot[i] @@ -1558,6 +1559,7 @@ cdef class Doc: for j, (attr, annot) in enumerate(token_annotations.items()): if attr is HEAD: + annot = numpy.array(annot, dtype=numpy.int32).astype(numpy.uint64) for i in range(len(words)): array[i, j] = annot[i] elif attr is MORPH: diff --git a/spacy/tokens/span.pyx b/spacy/tokens/span.pyx index c3495f497..99a5f43bd 100644 --- a/spacy/tokens/span.pyx +++ b/spacy/tokens/span.pyx @@ -299,7 +299,7 @@ cdef class Span: for ancestor in ancestors: ancestor_i = ancestor.i - self.c.start if ancestor_i in range(length): - array[i, head_col] = ancestor_i - i + array[i, head_col] = numpy.int32(ancestor_i - i).astype(numpy.uint64) # if there is no appropriate ancestor, define a new artificial root value = array[i, head_col] @@ -307,7 +307,7 @@ cdef class Span: new_root = old_to_new_root.get(ancestor_i, None) if new_root is not None: # take the same artificial root as a previous token from the same sentence - array[i, head_col] = new_root - i + array[i, head_col] = numpy.int32(new_root - i).astype(numpy.uint64) else: # set this token as the new artificial root array[i, head_col] = 0 diff --git a/spacy/training/example.pyx b/spacy/training/example.pyx index dfd337b9e..95b0f0de9 100644 --- a/spacy/training/example.pyx +++ b/spacy/training/example.pyx @@ -443,26 +443,27 @@ def _annot2array(vocab, tok_annot, doc_annot): if key not in IDS: raise ValueError(Errors.E974.format(obj="token", key=key)) elif key in ["ORTH", "SPACY"]: - pass + continue elif key == "HEAD": attrs.append(key) - values.append([h-i if h is not None else 0 for i, h in enumerate(value)]) + row = [h-i if h is not None else 0 for i, h in enumerate(value)] elif key == "DEP": attrs.append(key) - values.append([vocab.strings.add(h) if h is not None else MISSING_DEP for h in value]) + row = [vocab.strings.add(h) if h is not None else MISSING_DEP for h in value] elif key == "SENT_START": attrs.append(key) - values.append([to_ternary_int(v) for v in value]) + row = [to_ternary_int(v) for v in value] elif key == "MORPH": attrs.append(key) - values.append([vocab.morphology.add(v) for v in value]) + row = [vocab.morphology.add(v) for v in value] else: attrs.append(key) if not all(isinstance(v, str) for v in value): types = set([type(v) for v in value]) raise TypeError(Errors.E969.format(field=key, types=types)) from None - values.append([vocab.strings.add(v) for v in value]) - array = numpy.asarray(values, dtype="uint64") + row = [vocab.strings.add(v) for v in value] + values.append([numpy.array(v, dtype=numpy.int32).astype(numpy.uint64) if v < 0 else v for v in row]) + array = numpy.array(values, dtype=numpy.uint64) return attrs, array.T From e5c7f3b0776d49c4f6aab7e02b503cdb84fb2134 Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Mon, 12 Dec 2022 10:13:10 +0100 Subject: [PATCH 20/44] CI: Install thinc-apple-ops through extra (#11963) --- .github/azure-steps.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/azure-steps.yml b/.github/azure-steps.yml index 2f77706b8..d0db75f9a 100644 --- a/.github/azure-steps.yml +++ b/.github/azure-steps.yml @@ -107,7 +107,7 @@ steps: displayName: "Run CPU tests" - script: | - python -m pip install --pre thinc-apple-ops + python -m pip install 'spacy[apple]' python -m pytest --pyargs spacy displayName: "Run CPU tests with thinc-apple-ops" condition: and(startsWith(variables['imageName'], 'macos'), eq(variables['python.version'], '3.11')) From c9d9d6847f9685c21eeec01f4b8cd053cadf8bf5 Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Thu, 15 Dec 2022 10:55:01 +0100 Subject: [PATCH 21/44] Update build constraints for python 3.11 (#11981) --- build-constraints.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/build-constraints.txt b/build-constraints.txt index 956973abf..c1e82f1b0 100644 --- a/build-constraints.txt +++ b/build-constraints.txt @@ -5,4 +5,5 @@ numpy==1.17.3; python_version=='3.8' and platform_machine!='aarch64' numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64' numpy==1.19.3; python_version=='3.9' numpy==1.21.3; python_version=='3.10' -numpy; python_version>='3.11' +numpy==1.23.2; python_version=='3.11' +numpy; python_version>='3.12' From 3a2b655a29203d1c181a2c14d230b3f9cf8dd54a Mon Sep 17 00:00:00 2001 From: cfuerbachersparks <119413757+cfuerbachersparks@users.noreply.github.com> Date: Mon, 19 Dec 2022 10:33:38 +0100 Subject: [PATCH 22/44] Update lexeme.md (#11994) Change suffix_ string to end --- website/docs/api/lexeme.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/api/lexeme.md b/website/docs/api/lexeme.md index eb76afa90..557d04cce 100644 --- a/website/docs/api/lexeme.md +++ b/website/docs/api/lexeme.md @@ -138,7 +138,7 @@ The L2 norm of the lexeme's vector representation. | `prefix` | Length-N substring from the start of the word. Defaults to `N=1`. ~~int~~ | | `prefix_` | Length-N substring from the start of the word. Defaults to `N=1`. ~~str~~ | | `suffix` | Length-N substring from the end of the word. Defaults to `N=3`. ~~int~~ | -| `suffix_` | Length-N substring from the start of the word. Defaults to `N=3`. ~~str~~ | +| `suffix_` | Length-N substring from the end of the word. Defaults to `N=3`. ~~str~~ | | `is_alpha` | Does the lexeme consist of alphabetic characters? Equivalent to `lexeme.text.isalpha()`. ~~bool~~ | | `is_ascii` | Does the lexeme consist of ASCII characters? Equivalent to `[any(ord(c) >= 128 for c in lexeme.text)]`. ~~bool~~ | | `is_digit` | Does the lexeme consist of digits? Equivalent to `lexeme.text.isdigit()`. ~~bool~~ | From 18ffe5bbd6a554920107ff48d1387df34c3f872a Mon Sep 17 00:00:00 2001 From: Jos Polfliet Date: Mon, 19 Dec 2022 16:17:49 +0100 Subject: [PATCH 23/44] Update stop_words.py (#11997) fix typo in "aangaande" --- spacy/lang/nl/stop_words.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spacy/lang/nl/stop_words.py b/spacy/lang/nl/stop_words.py index a2c6198e7..cd4fdefdf 100644 --- a/spacy/lang/nl/stop_words.py +++ b/spacy/lang/nl/stop_words.py @@ -15,7 +15,7 @@ STOP_WORDS = set( """ -aan af al alle alles allebei alleen allen als altijd ander anders andere anderen aangaangde aangezien achter achterna +aan af al alle alles allebei alleen allen als altijd ander anders andere anderen aangaande aangezien achter achterna afgelopen aldus alhoewel anderzijds ben bij bijna bijvoorbeeld behalve beide beiden beneden bent bepaald beter betere betreffende binnen binnenin boven From c223cd7a86f460f3dabb9e7369eef136a653218e Mon Sep 17 00:00:00 2001 From: kadarakos Date: Tue, 20 Dec 2022 17:11:33 +0100 Subject: [PATCH 24/44] Add apply CLI (#11376) * annotate cli first try * add batch-size and n_process * rename to apply * typing fix * handle file suffixes * walk directories * support jsonl * typing fix * remove debug * make suffix optional for walk * revert unrelated * don't warn but raise * better error message * minor touch up * Update spacy/tests/test_cli.py Co-authored-by: Adriane Boyd * Update spacy/cli/apply.py Co-authored-by: Sofie Van Landeghem * Update spacy/cli/apply.py Co-authored-by: Sofie Van Landeghem * update tests and bugfix * add force_overwrite * typo * fix adding .spacy suffix * Update spacy/cli/apply.py Co-authored-by: Sofie Van Landeghem * Update spacy/cli/apply.py Co-authored-by: Sofie Van Landeghem * Update spacy/cli/apply.py Co-authored-by: Sofie Van Landeghem * store user data and rename cmd arg * include test for user attr * rename cmd arg * better help message * documentation * prettier * black * link fix * Update spacy/cli/apply.py Co-authored-by: Paul O'Leary McCann * Update website/docs/api/cli.md Co-authored-by: Paul O'Leary McCann * Update website/docs/api/cli.md Co-authored-by: Paul O'Leary McCann * Update website/docs/api/cli.md Co-authored-by: Paul O'Leary McCann * addressing reviews * dont quit but warn * prettier Co-authored-by: Adriane Boyd Co-authored-by: Sofie Van Landeghem Co-authored-by: Paul O'Leary McCann --- spacy/cli/__init__.py | 1 + spacy/cli/_util.py | 23 +++++++ spacy/cli/apply.py | 143 ++++++++++++++++++++++++++++++++++++++++ spacy/cli/convert.py | 31 +-------- spacy/tests/test_cli.py | 78 ++++++++++++++++++++++ website/docs/api/cli.md | 35 +++++++++- 6 files changed, 280 insertions(+), 31 deletions(-) create mode 100644 spacy/cli/apply.py diff --git a/spacy/cli/__init__.py b/spacy/cli/__init__.py index aab2c8d12..aabd1cfef 100644 --- a/spacy/cli/__init__.py +++ b/spacy/cli/__init__.py @@ -16,6 +16,7 @@ from .debug_config import debug_config # noqa: F401 from .debug_model import debug_model # noqa: F401 from .debug_diff import debug_diff # noqa: F401 from .evaluate import evaluate # noqa: F401 +from .apply import apply # noqa: F401 from .convert import convert # noqa: F401 from .init_pipeline import init_pipeline_cli # noqa: F401 from .init_config import init_config, fill_config # noqa: F401 diff --git a/spacy/cli/_util.py b/spacy/cli/_util.py index 9b97a9f19..c46abffe5 100644 --- a/spacy/cli/_util.py +++ b/spacy/cli/_util.py @@ -582,6 +582,29 @@ def setup_gpu(use_gpu: int, silent=None) -> None: local_msg.info("To switch to GPU 0, use the option: --gpu-id 0") +def walk_directory(path: Path, suffix: Optional[str] = None) -> List[Path]: + if not path.is_dir(): + return [path] + paths = [path] + locs = [] + seen = set() + for path in paths: + if str(path) in seen: + continue + seen.add(str(path)) + if path.parts[-1].startswith("."): + continue + elif path.is_dir(): + paths.extend(path.iterdir()) + elif suffix is not None and not path.parts[-1].endswith(suffix): + continue + else: + locs.append(path) + # It's good to sort these, in case the ordering messes up cache. + locs.sort() + return locs + + def _format_number(number: Union[int, float], ndigits: int = 2) -> str: """Formats a number (float or int) rounding to `ndigits`, without truncating trailing 0s, as happens with `round(number, ndigits)`""" diff --git a/spacy/cli/apply.py b/spacy/cli/apply.py new file mode 100644 index 000000000..9d170bc95 --- /dev/null +++ b/spacy/cli/apply.py @@ -0,0 +1,143 @@ +import tqdm +import srsly + +from itertools import chain +from pathlib import Path +from typing import Optional, List, Iterable, cast, Union + +from wasabi import msg + +from ._util import app, Arg, Opt, setup_gpu, import_code, walk_directory + +from ..tokens import Doc, DocBin +from ..vocab import Vocab +from ..util import ensure_path, load_model + + +path_help = """Location of the documents to predict on. +Can be a single file in .spacy format or a .jsonl file. +Files with other extensions are treated as single plain text documents. +If a directory is provided it is traversed recursively to grab +all files to be processed. +The files can be a mixture of .spacy, .jsonl and text files. +If .jsonl is provided the specified field is going +to be grabbed ("text" by default).""" + +out_help = "Path to save the resulting .spacy file" +code_help = ( + "Path to Python file with additional " "code (registered functions) to be imported" +) +gold_help = "Use gold preprocessing provided in the .spacy files" +force_msg = ( + "The provided output file already exists. " + "To force overwriting the output file, set the --force or -F flag." +) + + +DocOrStrStream = Union[Iterable[str], Iterable[Doc]] + + +def _stream_docbin(path: Path, vocab: Vocab) -> Iterable[Doc]: + """ + Stream Doc objects from DocBin. + """ + docbin = DocBin().from_disk(path) + for doc in docbin.get_docs(vocab): + yield doc + + +def _stream_jsonl(path: Path, field: str) -> Iterable[str]: + """ + Stream "text" field from JSONL. If the field "text" is + not found it raises error. + """ + for entry in srsly.read_jsonl(path): + if field not in entry: + msg.fail( + f"{path} does not contain the required '{field}' field.", exits=1 + ) + else: + yield entry[field] + + +def _stream_texts(paths: Iterable[Path]) -> Iterable[str]: + """ + Yields strings from text files in paths. + """ + for path in paths: + with open(path, "r") as fin: + text = fin.read() + yield text + + +@app.command("apply") +def apply_cli( + # fmt: off + model: str = Arg(..., help="Model name or path"), + data_path: Path = Arg(..., help=path_help, exists=True), + output_file: Path = Arg(..., help=out_help, dir_okay=False), + code_path: Optional[Path] = Opt(None, "--code", "-c", help=code_help), + text_key: str = Opt("text", "--text-key", "-tk", help="Key containing text string for JSONL"), + force_overwrite: bool = Opt(False, "--force", "-F", help="Force overwriting the output file"), + use_gpu: int = Opt(-1, "--gpu-id", "-g", help="GPU ID or -1 for CPU."), + batch_size: int = Opt(1, "--batch-size", "-b", help="Batch size."), + n_process: int = Opt(1, "--n-process", "-n", help="number of processors to use.") +): + """ + Apply a trained pipeline to documents to get predictions. + Expects a loadable spaCy pipeline and path to the data, which + can be a directory or a file. + The data files can be provided in multiple formats: + 1. .spacy files + 2. .jsonl files with a specified "field" to read the text from. + 3. Files with any other extension are assumed to be containing + a single document. + DOCS: https://spacy.io/api/cli#apply + """ + data_path = ensure_path(data_path) + output_file = ensure_path(output_file) + code_path = ensure_path(code_path) + if output_file.exists() and not force_overwrite: + msg.fail(force_msg, exits=1) + if not data_path.exists(): + msg.fail(f"Couldn't find data path: {data_path}", exits=1) + import_code(code_path) + setup_gpu(use_gpu) + apply(data_path, output_file, model, text_key, batch_size, n_process) + + +def apply( + data_path: Path, + output_file: Path, + model: str, + json_field: str, + batch_size: int, + n_process: int, +): + docbin = DocBin(store_user_data=True) + paths = walk_directory(data_path) + if len(paths) == 0: + docbin.to_disk(output_file) + msg.warn("Did not find data to process," + f" {data_path} seems to be an empty directory.") + return + nlp = load_model(model) + msg.good(f"Loaded model {model}") + vocab = nlp.vocab + streams: List[DocOrStrStream] = [] + text_files = [] + for path in paths: + if path.suffix == ".spacy": + streams.append(_stream_docbin(path, vocab)) + elif path.suffix == ".jsonl": + streams.append(_stream_jsonl(path, json_field)) + else: + text_files.append(path) + if len(text_files) > 0: + streams.append(_stream_texts(text_files)) + datagen = cast(DocOrStrStream, chain(*streams)) + for doc in tqdm.tqdm(nlp.pipe(datagen, batch_size=batch_size, n_process=n_process)): + docbin.add(doc) + if output_file.suffix == "": + output_file = output_file.with_suffix(".spacy") + docbin.to_disk(output_file) diff --git a/spacy/cli/convert.py b/spacy/cli/convert.py index 04eb7078f..7f365ae2c 100644 --- a/spacy/cli/convert.py +++ b/spacy/cli/convert.py @@ -1,4 +1,4 @@ -from typing import Callable, Iterable, Mapping, Optional, Any, List, Union +from typing import Callable, Iterable, Mapping, Optional, Any, Union from enum import Enum from pathlib import Path from wasabi import Printer @@ -7,7 +7,7 @@ import re import sys import itertools -from ._util import app, Arg, Opt +from ._util import app, Arg, Opt, walk_directory from ..training import docs_to_json from ..tokens import Doc, DocBin from ..training.converters import iob_to_docs, conll_ner_to_docs, json_to_docs @@ -189,33 +189,6 @@ def autodetect_ner_format(input_data: str) -> Optional[str]: return None -def walk_directory(path: Path, converter: str) -> List[Path]: - if not path.is_dir(): - return [path] - paths = [path] - locs = [] - seen = set() - for path in paths: - if str(path) in seen: - continue - seen.add(str(path)) - if path.parts[-1].startswith("."): - continue - elif path.is_dir(): - paths.extend(path.iterdir()) - elif converter == "json" and not path.parts[-1].endswith("json"): - continue - elif converter == "conll" and not path.parts[-1].endswith("conll"): - continue - elif converter == "iob" and not path.parts[-1].endswith("iob"): - continue - else: - locs.append(path) - # It's good to sort these, in case the ordering messes up cache. - locs.sort() - return locs - - def verify_cli_args( msg: Printer, input_path: Path, diff --git a/spacy/tests/test_cli.py b/spacy/tests/test_cli.py index 42af08749..c6768a3fd 100644 --- a/spacy/tests/test_cli.py +++ b/spacy/tests/test_cli.py @@ -5,6 +5,7 @@ from typing import Tuple, List, Dict, Any import pkg_resources import time +import spacy import numpy import pytest import srsly @@ -32,6 +33,7 @@ from spacy.cli.package import _is_permitted_package_name from spacy.cli.project.remote_storage import RemoteStorage from spacy.cli.project.run import _check_requirements from spacy.cli.validate import get_model_pkgs +from spacy.cli.apply import apply from spacy.cli.find_threshold import find_threshold from spacy.lang.en import English from spacy.lang.nl import Dutch @@ -885,6 +887,82 @@ def test_span_length_freq_dist_output_must_be_correct(): assert list(span_freqs.keys()) == [3, 1, 4, 5, 2] +def test_applycli_empty_dir(): + with make_tempdir() as data_path: + output = data_path / "test.spacy" + apply(data_path, output, "blank:en", "text", 1, 1) + + +def test_applycli_docbin(): + with make_tempdir() as data_path: + output = data_path / "testout.spacy" + nlp = spacy.blank("en") + doc = nlp("testing apply cli.") + # test empty DocBin case + docbin = DocBin() + docbin.to_disk(data_path / "testin.spacy") + apply(data_path, output, "blank:en", "text", 1, 1) + docbin.add(doc) + docbin.to_disk(data_path / "testin.spacy") + apply(data_path, output, "blank:en", "text", 1, 1) + + +def test_applycli_jsonl(): + with make_tempdir() as data_path: + output = data_path / "testout.spacy" + data = [{"field": "Testing apply cli.", "key": 234}] + data2 = [{"field": "234"}] + srsly.write_jsonl(data_path / "test.jsonl", data) + apply(data_path, output, "blank:en", "field", 1, 1) + srsly.write_jsonl(data_path / "test2.jsonl", data2) + apply(data_path, output, "blank:en", "field", 1, 1) + + +def test_applycli_txt(): + with make_tempdir() as data_path: + output = data_path / "testout.spacy" + with open(data_path / "test.foo", "w") as ftest: + ftest.write("Testing apply cli.") + apply(data_path, output, "blank:en", "text", 1, 1) + + +def test_applycli_mixed(): + with make_tempdir() as data_path: + output = data_path / "testout.spacy" + text = "Testing apply cli" + nlp = spacy.blank("en") + doc = nlp(text) + jsonl_data = [{"text": text}] + srsly.write_jsonl(data_path / "test.jsonl", jsonl_data) + docbin = DocBin() + docbin.add(doc) + docbin.to_disk(data_path / "testin.spacy") + with open(data_path / "test.txt", "w") as ftest: + ftest.write(text) + apply(data_path, output, "blank:en", "text", 1, 1) + # Check whether it worked + result = list(DocBin().from_disk(output).get_docs(nlp.vocab)) + assert len(result) == 3 + for doc in result: + assert doc.text == text + + +def test_applycli_user_data(): + Doc.set_extension("ext", default=0) + val = ("ext", 0) + with make_tempdir() as data_path: + output = data_path / "testout.spacy" + nlp = spacy.blank("en") + doc = nlp("testing apply cli.") + doc._.ext = val + docbin = DocBin(store_user_data=True) + docbin.add(doc) + docbin.to_disk(data_path / "testin.spacy") + apply(data_path, output, "blank:en", "", 1, 1) + result = list(DocBin().from_disk(output).get_docs(nlp.vocab)) + assert result[0]._.ext == val + + def test_local_remote_storage(): with make_tempdir() as d: filename = "a.txt" diff --git a/website/docs/api/cli.md b/website/docs/api/cli.md index 8823a3bd8..275e37ee0 100644 --- a/website/docs/api/cli.md +++ b/website/docs/api/cli.md @@ -12,6 +12,7 @@ menu: - ['train', 'train'] - ['pretrain', 'pretrain'] - ['evaluate', 'evaluate'] + - ['apply', 'apply'] - ['find-threshold', 'find-threshold'] - ['assemble', 'assemble'] - ['package', 'package'] @@ -474,7 +475,7 @@ report span characteristics such as the average span length and the span (or span boundary) distinctiveness. The distinctiveness measure shows how different the tokens are with respect to the rest of the corpus using the KL-divergence of the token distributions. To learn more, you can check out Papay et al.'s work on -[*Dissecting Span Identification Tasks with Performance Prediction* (EMNLP 2020)](https://aclanthology.org/2020.emnlp-main.396/). +[_Dissecting Span Identification Tasks with Performance Prediction_ (EMNLP 2020)](https://aclanthology.org/2020.emnlp-main.396/). @@ -1162,6 +1163,37 @@ $ python -m spacy evaluate [model] [data_path] [--output] [--code] [--gold-prepr | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | Training results and optional metrics and visualizations. | +## apply {#apply new="3.5" tag="command"} + +Applies a trained pipeline to data and stores the resulting annotated documents +in a `DocBin`. The input can be a single file or a directory. The recognized +input formats are: + +1. `.spacy` +2. `.jsonl` containing a user specified `text_key` +3. Files with any other extension are assumed to be plain text files containing + a single document. + +When a directory is provided it is traversed recursively to collect all files. + +```cli +$ python -m spacy apply [model] [data-path] [output-file] [--code] [--text-key] [--force-overwrite] [--gpu-id] [--batch-size] [--n-process] +``` + +| Name | Description | +| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `model` | Pipeline to apply to the data. Can be a package or a path to a data directory. ~~str (positional)~~ | +| `data_path` | Location of data to be evaluated in spaCy's [binary format](/api/data-formats#training), jsonl, or plain text. ~~Path (positional)~~ | +| `output-file`, `-o` | Output `DocBin` path. ~~str (positional)~~ | +| `--code`, `-c` 3 | Path to Python file with additional code to be imported. Allows [registering custom functions](/usage/training#custom-functions) for new architectures. ~~Optional[Path] \(option)~~ | +| `--text-key`, `-tk` | The key for `.jsonl` files to use to grab the texts from. Defaults to `text`. ~~Optional[str] \(option)~~ | +| `--force-overwrite`, `-F` | If the provided `output-file` already exists, then force `apply` to overwrite it. If this is `False` (default) then quits with a warning instead. ~~bool (flag)~~ | +| `--gpu-id`, `-g` | GPU to use, if any. Defaults to `-1` for CPU. ~~int (option)~~ | +| `--batch-size`, `-b` | Batch size to use for prediction. Defaults to `1`. ~~int (option)~~ | +| `--n-process`, `-n` | Number of processes to use for prediction. Defaults to `1`. ~~int (option)~~ | +| `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | +| **CREATES** | A `DocBin` with the annotations from the `model` for all the files found in `data-path`. | + ## find-threshold {#find-threshold new="3.5" tag="command"} Runs prediction trials for a trained model with varying tresholds to maximize @@ -1187,7 +1219,6 @@ be provided. > $ python -m spacy find-threshold my_nlp data.spacy spancat threshold spans_sc_f > ``` - | Name | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `model` | Pipeline to evaluate. Can be a package or a path to a data directory. ~~str (positional)~~ | From eef3d950b4266ab9545143de8070456ce7967950 Mon Sep 17 00:00:00 2001 From: Raphael Mitsch Date: Wed, 21 Dec 2022 18:54:27 +0100 Subject: [PATCH 25/44] Fix `SpanGroup` and `Span` typing (#12009) * Correct Span.label, Span.kb_id types. Fix SpanGroup.__iter__(). * Extend test. * Rename test. Fix typo. * Add comment. * Fix types for Span.label, Span.kb_id, Span.char_span(). * Update spacy/tests/doc/test_span_group.py Co-authored-by: Adriane Boyd * Update docs. * Fix typo. * Update spacy/tokens/span_group.pyx Co-authored-by: Adriane Boyd Co-authored-by: Adriane Boyd --- spacy/tests/doc/test_span_group.py | 15 ++++++++++++++- spacy/tokens/span.pyi | 4 ++-- spacy/tokens/span_group.pyi | 1 + spacy/tokens/span_group.pyx | 10 ++++++++++ website/docs/api/spangroup.md | 17 +++++++++++++++++ 5 files changed, 44 insertions(+), 3 deletions(-) diff --git a/spacy/tests/doc/test_span_group.py b/spacy/tests/doc/test_span_group.py index 8c70a83e1..818569c64 100644 --- a/spacy/tests/doc/test_span_group.py +++ b/spacy/tests/doc/test_span_group.py @@ -1,7 +1,10 @@ +from typing import List + import pytest from random import Random from spacy.matcher import Matcher -from spacy.tokens import Span, SpanGroup +from spacy.tokens import Span, SpanGroup, Doc +from spacy.util import filter_spans @pytest.fixture @@ -240,3 +243,13 @@ def test_span_group_extend(doc): def test_span_group_dealloc(span_group): with pytest.raises(AttributeError): print(span_group.doc) + + +@pytest.mark.issue(11975) +def test_span_group_typing(doc: Doc): + """Tests whether typing of `SpanGroup` as `Iterable[Span]`-like object is accepted by mypy.""" + span_group: SpanGroup = doc.spans["SPANS"] + spans: List[Span] = list(span_group) + for i, span in enumerate(span_group): + assert span == span_group[i] == spans[i] + filter_spans(span_group) diff --git a/spacy/tokens/span.pyi b/spacy/tokens/span.pyi index 0a6f306a6..9986a90e6 100644 --- a/spacy/tokens/span.pyi +++ b/spacy/tokens/span.pyi @@ -95,8 +95,8 @@ class Span: self, start_idx: int, end_idx: int, - label: int = ..., - kb_id: int = ..., + label: Union[int, str] = ..., + kb_id: Union[int, str] = ..., vector: Optional[Floats1d] = ..., ) -> Span: ... @property diff --git a/spacy/tokens/span_group.pyi b/spacy/tokens/span_group.pyi index 21cd124ab..0b4aa83aa 100644 --- a/spacy/tokens/span_group.pyi +++ b/spacy/tokens/span_group.pyi @@ -18,6 +18,7 @@ class SpanGroup: def doc(self) -> Doc: ... @property def has_overlap(self) -> bool: ... + def __iter__(self): ... def __len__(self) -> int: ... def append(self, span: Span) -> None: ... def extend(self, spans: Iterable[Span]) -> None: ... diff --git a/spacy/tokens/span_group.pyx b/spacy/tokens/span_group.pyx index 1aa3c0bc8..608dda283 100644 --- a/spacy/tokens/span_group.pyx +++ b/spacy/tokens/span_group.pyx @@ -158,6 +158,16 @@ cdef class SpanGroup: return self._concat(other) return NotImplemented + def __iter__(self): + """ + Iterate over the spans in this SpanGroup. + YIELDS (Span): A span in this SpanGroup. + + DOCS: https://spacy.io/api/spangroup#iter + """ + for i in range(self.c.size()): + yield self[i] + def append(self, Span span): """Add a span to the group. The span must refer to the same Doc object as the span group. diff --git a/website/docs/api/spangroup.md b/website/docs/api/spangroup.md index 2d1cf73c4..bd9659acb 100644 --- a/website/docs/api/spangroup.md +++ b/website/docs/api/spangroup.md @@ -202,6 +202,23 @@ already present in the current span group. | `other` | The span group or spans to append. ~~Union[SpanGroup, Iterable[Span]]~~ | | **RETURNS** | The span group. ~~SpanGroup~~ | +## SpanGroup.\_\_iter\_\_ {#iter tag="method" new="3.5"} + +Iterate over the spans in this span group. + +> #### Example +> +> ```python +> doc = nlp("Their goi ng home") +> doc.spans["errors"] = [doc[0:1], doc[1:3]] +> for error_span in doc.spans["errors"]: +> print(error_span) +> ``` + +| Name | Description | +| ---------- | ----------------------------------- | +| **YIELDS** | A span in this span group. ~~Span~~ | + ## SpanGroup.append {#append tag="method"} Add a [`Span`](/api/span) object to the group. The span must refer to the same From 64d2d27c5dbf8e5657187975d2c9627f30e108a2 Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Thu, 22 Dec 2022 10:53:16 +0100 Subject: [PATCH 26/44] Add classifier for python 3.11 (#12013) --- setup.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/setup.cfg b/setup.cfg index cf6e6f84b..d290d706c 100644 --- a/setup.cfg +++ b/setup.cfg @@ -22,6 +22,7 @@ classifiers = Programming Language :: Python :: 3.8 Programming Language :: Python :: 3.9 Programming Language :: Python :: 3.10 + Programming Language :: Python :: 3.11 Topic :: Scientific/Engineering project_urls = Release notes = https://github.com/explosion/spaCy/releases From 90896504a5dba1babac04a2b88662179409ae006 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 23 Dec 2022 12:44:07 +0100 Subject: [PATCH 27/44] Auto-format code with black (#12019) Co-authored-by: explosion-bot --- spacy/cli/apply.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/spacy/cli/apply.py b/spacy/cli/apply.py index 9d170bc95..f0df4e757 100644 --- a/spacy/cli/apply.py +++ b/spacy/cli/apply.py @@ -53,9 +53,7 @@ def _stream_jsonl(path: Path, field: str) -> Iterable[str]: """ for entry in srsly.read_jsonl(path): if field not in entry: - msg.fail( - f"{path} does not contain the required '{field}' field.", exits=1 - ) + msg.fail(f"{path} does not contain the required '{field}' field.", exits=1) else: yield entry[field] @@ -118,8 +116,10 @@ def apply( paths = walk_directory(data_path) if len(paths) == 0: docbin.to_disk(output_file) - msg.warn("Did not find data to process," - f" {data_path} seems to be an empty directory.") + msg.warn( + "Did not find data to process," + f" {data_path} seems to be an empty directory." + ) return nlp = load_model(model) msg.good(f"Loaded model {model}") From aa2b471a6e289d1c1bb51558df779ae028671225 Mon Sep 17 00:00:00 2001 From: Madeesh Kannan Date: Fri, 23 Dec 2022 15:21:44 +0100 Subject: [PATCH 28/44] New console logger with expanded progress tracking (#11972) * Add `ConsoleLogger.v3` This addition expands the progress bar feature to count up the training/distillation steps to either the next evaluation pass or the maximum number of steps. * Rename progress bar types * Add defaults to docs Minor fixes * Move comment * Minor punctuation fixes * Explicitly check for `None` when validating progress bar type Co-authored-by: Paul O'Leary McCann --- spacy/errors.py | 1 + spacy/training/loggers.py | 48 ++++++++++++++++++++++++++++++++--- website/docs/api/top-level.md | 34 ++++++++++++++++++++----- 3 files changed, 74 insertions(+), 9 deletions(-) diff --git a/spacy/errors.py b/spacy/errors.py index 0e5ef91ed..cd9281e91 100644 --- a/spacy/errors.py +++ b/spacy/errors.py @@ -962,6 +962,7 @@ class Errors(metaclass=ErrorsWithCodes): E1046 = ("{cls_name} is an abstract class and cannot be instantiated. If you are looking for spaCy's default " "knowledge base, use `InMemoryLookupKB`.") E1047 = ("`find_threshold()` only supports components with a `scorer` attribute.") + E1048 = ("Got '{unexpected}' as console progress bar type, but expected one of the following: {expected}") # Deprecated model shortcuts, only used in errors and warnings diff --git a/spacy/training/loggers.py b/spacy/training/loggers.py index 408ea7140..7de31822e 100644 --- a/spacy/training/loggers.py +++ b/spacy/training/loggers.py @@ -26,6 +26,8 @@ def setup_table( return final_cols, final_widths, ["r" for _ in final_widths] +# We cannot rename this method as it's directly imported +# and used by external packages such as spacy-loggers. @registry.loggers("spacy.ConsoleLogger.v2") def console_logger( progress_bar: bool = False, @@ -33,7 +35,27 @@ def console_logger( output_file: Optional[Union[str, Path]] = None, ): """The ConsoleLogger.v2 prints out training logs in the console and/or saves them to a jsonl file. - progress_bar (bool): Whether the logger should print the progress bar. + progress_bar (bool): Whether the logger should print a progress bar tracking the steps till the next evaluation pass. + console_output (bool): Whether the logger should print the logs on the console. + output_file (Optional[Union[str, Path]]): The file to save the training logs to. + """ + return console_logger_v3( + progress_bar=None if progress_bar is False else "eval", + console_output=console_output, + output_file=output_file, + ) + + +@registry.loggers("spacy.ConsoleLogger.v3") +def console_logger_v3( + progress_bar: Optional[str] = None, + console_output: bool = True, + output_file: Optional[Union[str, Path]] = None, +): + """The ConsoleLogger.v3 prints out training logs in the console and/or saves them to a jsonl file. + progress_bar (Optional[str]): Type of progress bar to show in the console. Allowed values: + train - Tracks the number of steps from the beginning of training until the full training run is complete (training.max_steps is reached). + eval - Tracks the number of steps between the previous and next evaluation (training.eval_frequency is reached). console_output (bool): Whether the logger should print the logs on the console. output_file (Optional[Union[str, Path]]): The file to save the training logs to. """ @@ -70,6 +92,7 @@ def console_logger( for name, proc in nlp.pipeline if hasattr(proc, "is_trainable") and proc.is_trainable ] + max_steps = nlp.config["training"]["max_steps"] eval_frequency = nlp.config["training"]["eval_frequency"] score_weights = nlp.config["training"]["score_weights"] score_cols = [col for col, value in score_weights.items() if value is not None] @@ -84,6 +107,13 @@ def console_logger( write(msg.row(table_header, widths=table_widths, spacing=spacing)) write(msg.row(["-" * width for width in table_widths], spacing=spacing)) progress = None + expected_progress_types = ("train", "eval") + if progress_bar is not None and progress_bar not in expected_progress_types: + raise ValueError( + Errors.E1048.format( + unexpected=progress_bar, expected=expected_progress_types + ) + ) def log_step(info: Optional[Dict[str, Any]]) -> None: nonlocal progress @@ -141,11 +171,23 @@ def console_logger( ) ) if progress_bar: + if progress_bar == "train": + total = max_steps + desc = f"Last Eval Epoch: {info['epoch']}" + initial = info["step"] + else: + total = eval_frequency + desc = f"Epoch {info['epoch']+1}" + initial = 0 # Set disable=None, so that it disables on non-TTY progress = tqdm.tqdm( - total=eval_frequency, disable=None, leave=False, file=stderr + total=total, + disable=None, + leave=False, + file=stderr, + initial=initial, ) - progress.set_description(f"Epoch {info['epoch']+1}") + progress.set_description(desc) def finalize() -> None: if output_stream: diff --git a/website/docs/api/top-level.md b/website/docs/api/top-level.md index 26a5d42f4..883c5e3b9 100644 --- a/website/docs/api/top-level.md +++ b/website/docs/api/top-level.md @@ -513,7 +513,7 @@ a [Weights & Biases](https://www.wandb.com/) dashboard. Instead of using one of the built-in loggers, you can [implement your own](/usage/training#custom-logging). -#### spacy.ConsoleLogger.v2 {#ConsoleLogger tag="registered function"} +#### spacy.ConsoleLogger.v2 {tag="registered function"} > #### Example config > @@ -564,11 +564,33 @@ start decreasing across epochs. -| Name | Description | -| ---------------- | --------------------------------------------------------------------- | -| `progress_bar` | Whether the logger should print the progress bar ~~bool~~ | -| `console_output` | Whether the logger should print the logs on the console. ~~bool~~ | -| `output_file` | The file to save the training logs to. ~~Optional[Union[str, Path]]~~ | +| Name | Description | +| ---------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| `progress_bar` | Whether the logger should print a progress bar tracking the steps till the next evaluation pass (default: `False`). ~~bool~~ | +| `console_output` | Whether the logger should print the logs in the console (default: `True`). ~~bool~~ | +| `output_file` | The file to save the training logs to (default: `None`). ~~Optional[Union[str, Path]]~~ | + +#### spacy.ConsoleLogger.v3 {#ConsoleLogger tag="registered function"} + +> #### Example config +> +> ```ini +> [training.logger] +> @loggers = "spacy.ConsoleLogger.v3" +> progress_bar = "all_steps" +> console_output = true +> output_file = "training_log.jsonl" +> ``` + +Writes the results of a training step to the console in a tabular format and +optionally saves them to a `jsonl` file. + +| Name | Description | +| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `progress_bar` | Type of progress bar to show in the console: `"train"`, `"eval"` or `None`. | +| | The bar tracks the number of steps until `training.max_steps` and `training.eval_frequency` are reached respectively (default: `None`). ~~Optional[str]~~ | +| `console_output` | Whether the logger should print the logs in the console (default: `True`). ~~bool~~ | +| `output_file` | The file to save the training logs to (default: `None`). ~~Optional[Union[str, Path]]~~ | ## Readers {#readers} From 933b54ac798a7d64f9cde4d85b55556e84e44bd6 Mon Sep 17 00:00:00 2001 From: kadarakos Date: Mon, 26 Dec 2022 13:26:35 +0100 Subject: [PATCH 29/44] typo fix (#11995) --- spacy/pipeline/span_ruler.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spacy/pipeline/span_ruler.py b/spacy/pipeline/span_ruler.py index 807a4ffe5..0e7e9ebf7 100644 --- a/spacy/pipeline/span_ruler.py +++ b/spacy/pipeline/span_ruler.py @@ -170,7 +170,7 @@ def prioritize_existing_ents_filter( @registry.misc("spacy.prioritize_existing_ents_filter.v1") -def make_preverse_existing_ents_filter(): +def make_preserve_existing_ents_filter(): return prioritize_existing_ents_filter From ef9e504eacc806162666c964bd00d152fc15f9e3 Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Thu, 29 Dec 2022 14:01:08 +0100 Subject: [PATCH 30/44] Rename modified textcat scorer to v2 (#11971) As a follow-up to #11696, rename the modified scorer to v2 and move the v1 scorer to `spacy-legacy`. --- requirements.txt | 2 +- setup.cfg | 2 +- spacy/pipeline/textcat.py | 4 ++-- spacy/tests/pipeline/test_textcat.py | 17 +++++++++++++++++ 4 files changed, 21 insertions(+), 4 deletions(-) diff --git a/requirements.txt b/requirements.txt index 0440835f2..5bc1c8684 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ # Our libraries -spacy-legacy>=3.0.10,<3.1.0 +spacy-legacy>=3.0.11,<3.1.0 spacy-loggers>=1.0.0,<2.0.0 cymem>=2.0.2,<2.1.0 preshed>=3.0.2,<3.1.0 diff --git a/setup.cfg b/setup.cfg index d290d706c..cee8c0c33 100644 --- a/setup.cfg +++ b/setup.cfg @@ -42,7 +42,7 @@ setup_requires = thinc>=8.1.0,<8.2.0 install_requires = # Our libraries - spacy-legacy>=3.0.10,<3.1.0 + spacy-legacy>=3.0.11,<3.1.0 spacy-loggers>=1.0.0,<2.0.0 murmurhash>=0.28.0,<1.1.0 cymem>=2.0.2,<2.1.0 diff --git a/spacy/pipeline/textcat.py b/spacy/pipeline/textcat.py index 65121114d..650a01949 100644 --- a/spacy/pipeline/textcat.py +++ b/spacy/pipeline/textcat.py @@ -74,7 +74,7 @@ subword_features = true default_config={ "threshold": 0.0, "model": DEFAULT_SINGLE_TEXTCAT_MODEL, - "scorer": {"@scorers": "spacy.textcat_scorer.v1"}, + "scorer": {"@scorers": "spacy.textcat_scorer.v2"}, }, default_score_weights={ "cats_score": 1.0, @@ -117,7 +117,7 @@ def textcat_score(examples: Iterable[Example], **kwargs) -> Dict[str, Any]: ) -@registry.scorers("spacy.textcat_scorer.v1") +@registry.scorers("spacy.textcat_scorer.v2") def make_textcat_scorer(): return textcat_score diff --git a/spacy/tests/pipeline/test_textcat.py b/spacy/tests/pipeline/test_textcat.py index 155ce99a2..eafe4c128 100644 --- a/spacy/tests/pipeline/test_textcat.py +++ b/spacy/tests/pipeline/test_textcat.py @@ -895,3 +895,20 @@ def test_textcat_multi_threshold(): scores = nlp.evaluate(train_examples, scorer_cfg={"threshold": 0}) assert scores["cats_f_per_type"]["POSITIVE"]["r"] == 1.0 + + +@pytest.mark.parametrize("component_name,scorer", [("textcat", "spacy.textcat_scorer.v1")]) +def test_textcat_legacy_scorers(component_name, scorer): + """Check that legacy scorers are registered and produce the expected score + keys.""" + nlp = English() + nlp.add_pipe(component_name, config={"scorer": {"@scorers": scorer}}) + + train_examples = [] + for text, annotations in TRAIN_DATA_SINGLE_LABEL: + train_examples.append(Example.from_dict(nlp.make_doc(text), annotations)) + nlp.initialize(get_examples=lambda: train_examples) + + # score the model (it's not actually trained but that doesn't matter) + scores = nlp.evaluate(train_examples) + assert 0 <= scores["cats_score"] <= 1 From abb0ab109d33d2deaa6155a61fad649a25472f9c Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 2 Jan 2023 11:59:57 +0100 Subject: [PATCH 31/44] Auto-format code with black (#12035) Co-authored-by: explosion-bot --- spacy/tests/pipeline/test_textcat.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/spacy/tests/pipeline/test_textcat.py b/spacy/tests/pipeline/test_textcat.py index eafe4c128..048586cec 100644 --- a/spacy/tests/pipeline/test_textcat.py +++ b/spacy/tests/pipeline/test_textcat.py @@ -897,7 +897,9 @@ def test_textcat_multi_threshold(): assert scores["cats_f_per_type"]["POSITIVE"]["r"] == 1.0 -@pytest.mark.parametrize("component_name,scorer", [("textcat", "spacy.textcat_scorer.v1")]) +@pytest.mark.parametrize( + "component_name,scorer", [("textcat", "spacy.textcat_scorer.v1")] +) def test_textcat_legacy_scorers(component_name, scorer): """Check that legacy scorers are registered and produce the expected score keys.""" From 31c1beba787446059de58a1478e6aec197fd0bbb Mon Sep 17 00:00:00 2001 From: Wannaphong Phatthiyaphaibun Date: Tue, 3 Jan 2023 15:03:59 +0700 Subject: [PATCH 32/44] Add spacy-pythainlp (#12038) * Add spacy-pythainlp * Move submission to right section * Minor cleanup * Remove extra list call * Update universe.json Co-authored-by: Paul O'Leary McCann --- website/meta/universe.json | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/website/meta/universe.json b/website/meta/universe.json index db533c3b2..99d121507 100644 --- a/website/meta/universe.json +++ b/website/meta/universe.json @@ -4062,6 +4062,33 @@ "author_links": { "github": "yasufumy" } + }, + { + "id": "spacy-pythainlp", + "title": "spaCy-PyThaiNLP", + "slogan": "PyThaiNLP for spaCy", + "description": "This package wraps the PyThaiNLP library to add support for Thai to spaCy.", + "github": "PyThaiNLP/spaCy-PyThaiNLP", + "code_example": [ + "import spacy", + "import spacy_pythainlp.core", + "", + "nlp = spacy.blank('th')", + "nlp.add_pipe('pythainlp')", + "doc = nlp('ผมเป็นคนไทย แต่มะลิอยากไปโรงเรียนส่วนผมจะไปไหน ผมอยากไปเที่ยว')", + "", + "print(list(doc.sents))", + "# output: [ผมเป็นคนไทย แต่มะลิอยากไปโรงเรียนส่วนผมจะไปไหน , ผมอยากไปเที่ยว]" + ], + "code_language": "python", + "author": "Wannaphong Phatthiyaphaibun", + "author_links": { + "twitter": "@wannaphong_p", + "github": "wannaphong", + "website": "https://iam.wannaphong.com/" + }, + "category": ["pipeline", "research"], + "tags": ["Thai"] } ], From dbd829f0ed2dba3eb6eb5b59b18396ed38e326b9 Mon Sep 17 00:00:00 2001 From: Paul O'Leary McCann Date: Wed, 4 Jan 2023 12:51:40 +0900 Subject: [PATCH 33/44] Fix inconsistency in displaCy docs about page option (#12047) * Fix inconsistency in displaCy docs about page option The `page` option, which wraps the output SVG in HTML, is true by default for `serve` but not for `render`. The `render` docs were wrong though, so this updates them. * Update the same statement in more docs A few renderers used the same language --- spacy/displacy/__init__.py | 2 +- spacy/displacy/render.py | 4 ++-- website/docs/api/top-level.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/spacy/displacy/__init__.py b/spacy/displacy/__init__.py index bc32001d7..2f2058b8e 100644 --- a/spacy/displacy/__init__.py +++ b/spacy/displacy/__init__.py @@ -36,7 +36,7 @@ def render( jupyter (bool): Override Jupyter auto-detection. options (dict): Visualiser-specific options, e.g. colors. manual (bool): Don't parse `Doc` and instead expect a dict/list of dicts. - RETURNS (str): Rendered HTML markup. + RETURNS (str): Rendered SVG or HTML markup. DOCS: https://spacy.io/api/top-level#displacy.render USAGE: https://spacy.io/usage/visualizers diff --git a/spacy/displacy/render.py b/spacy/displacy/render.py index 50dc3466c..f74222dc2 100644 --- a/spacy/displacy/render.py +++ b/spacy/displacy/render.py @@ -94,7 +94,7 @@ class SpanRenderer: parsed (list): Dependency parses to render. page (bool): Render parses wrapped as full HTML page. minify (bool): Minify HTML markup. - RETURNS (str): Rendered HTML markup. + RETURNS (str): Rendered SVG or HTML markup. """ rendered = [] for i, p in enumerate(parsed): @@ -510,7 +510,7 @@ class EntityRenderer: parsed (list): Dependency parses to render. page (bool): Render parses wrapped as full HTML page. minify (bool): Minify HTML markup. - RETURNS (str): Rendered HTML markup. + RETURNS (str): Rendered SVG or HTML markup. """ rendered = [] for i, p in enumerate(parsed): diff --git a/website/docs/api/top-level.md b/website/docs/api/top-level.md index 883c5e3b9..6a63e07da 100644 --- a/website/docs/api/top-level.md +++ b/website/docs/api/top-level.md @@ -266,7 +266,7 @@ Render a dependency parse tree or named entity visualization. | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `docs` | Document(s) or span(s) to visualize. ~~Union[Iterable[Union[Doc, Span, dict]], Doc, Span, dict]~~ | | `style` | Visualization style, `"dep"`, `"ent"` or `"span"` 3.3. Defaults to `"dep"`. ~~str~~ | -| `page` | Render markup as full HTML page. Defaults to `True`. ~~bool~~ | +| `page` | Render markup as full HTML page. Defaults to `False`. ~~bool~~ | | `minify` | Minify HTML markup. Defaults to `False`. ~~bool~~ | | `options` | [Visualizer-specific options](#displacy_options), e.g. colors. ~~Dict[str, Any]~~ | | `manual` | Don't parse `Doc` and instead expect a dict or list of dicts. [See here](/usage/visualizers#manual-usage) for formats and examples. Defaults to `False`. ~~bool~~ | From 7f6c638c3acd732c0b52a45a2b3ad0388cd1ae66 Mon Sep 17 00:00:00 2001 From: Sofie Van Landeghem Date: Thu, 5 Jan 2023 10:21:00 +0100 Subject: [PATCH 34/44] fix processing of "auto" in convert (#12050) * fix processing of "auto" in walk_directory * add check for None * move AUTO check to convert and fix verification of args * add specific CLI test with CliRunner * cleanup * more cleanup * update docstring --- spacy/cli/_util.py | 4 ++++ spacy/cli/convert.py | 26 ++++++++++++++++---------- spacy/tests/test_cli.py | 26 +++++++++++++++++++++++++- spacy/tests/test_cli_app.py | 33 +++++++++++++++++++++++++++++++++ 4 files changed, 78 insertions(+), 11 deletions(-) create mode 100644 spacy/tests/test_cli_app.py diff --git a/spacy/cli/_util.py b/spacy/cli/_util.py index c46abffe5..0f4e9f599 100644 --- a/spacy/cli/_util.py +++ b/spacy/cli/_util.py @@ -583,6 +583,10 @@ def setup_gpu(use_gpu: int, silent=None) -> None: def walk_directory(path: Path, suffix: Optional[str] = None) -> List[Path]: + """Given a directory and a suffix, recursively find all files matching the suffix. + Directories or files with names beginning with a . are ignored, but hidden flags on + filesystems are not checked. + When provided with a suffix `None`, there is no suffix-based filtering.""" if not path.is_dir(): return [path] paths = [path] diff --git a/spacy/cli/convert.py b/spacy/cli/convert.py index 7f365ae2c..68d454b3e 100644 --- a/spacy/cli/convert.py +++ b/spacy/cli/convert.py @@ -28,6 +28,8 @@ CONVERTERS: Mapping[str, Callable[..., Iterable[Doc]]] = { "json": json_to_docs, } +AUTO = "auto" + # File types that can be written to stdout FILE_TYPES_STDOUT = ("json",) @@ -49,7 +51,7 @@ def convert_cli( model: Optional[str] = Opt(None, "--model", "--base", "-b", help="Trained spaCy pipeline for sentence segmentation to use as base (for --seg-sents)"), morphology: bool = Opt(False, "--morphology", "-m", help="Enable appending morphology to tags"), merge_subtokens: bool = Opt(False, "--merge-subtokens", "-T", help="Merge CoNLL-U subtokens"), - converter: str = Opt("auto", "--converter", "-c", help=f"Converter: {tuple(CONVERTERS.keys())}"), + converter: str = Opt(AUTO, "--converter", "-c", help=f"Converter: {tuple(CONVERTERS.keys())}"), ner_map: Optional[Path] = Opt(None, "--ner-map", "-nm", help="NER tag mapping (as JSON-encoded dict of entity types)", exists=True), lang: Optional[str] = Opt(None, "--lang", "-l", help="Language (if tokenizer required)"), concatenate: bool = Opt(None, "--concatenate", "-C", help="Concatenate output to a single file"), @@ -70,8 +72,8 @@ def convert_cli( output_dir: Union[str, Path] = "-" if output_dir == Path("-") else output_dir silent = output_dir == "-" msg = Printer(no_print=silent) - verify_cli_args(msg, input_path, output_dir, file_type.value, converter, ner_map) converter = _get_converter(msg, converter, input_path) + verify_cli_args(msg, input_path, output_dir, file_type.value, converter, ner_map) convert( input_path, output_dir, @@ -100,7 +102,7 @@ def convert( model: Optional[str] = None, morphology: bool = False, merge_subtokens: bool = False, - converter: str = "auto", + converter: str, ner_map: Optional[Path] = None, lang: Optional[str] = None, concatenate: bool = False, @@ -212,18 +214,22 @@ def verify_cli_args( input_locs = walk_directory(input_path, converter) if len(input_locs) == 0: msg.fail("No input files in directory", input_path, exits=1) - file_types = list(set([loc.suffix[1:] for loc in input_locs])) - if converter == "auto" and len(file_types) >= 2: - file_types_str = ",".join(file_types) - msg.fail("All input files must be same type", file_types_str, exits=1) - if converter != "auto" and converter not in CONVERTERS: + if converter not in CONVERTERS: msg.fail(f"Can't find converter for {converter}", exits=1) def _get_converter(msg, converter, input_path: Path): if input_path.is_dir(): - input_path = walk_directory(input_path, converter)[0] - if converter == "auto": + if converter == AUTO: + input_locs = walk_directory(input_path, suffix=None) + file_types = list(set([loc.suffix[1:] for loc in input_locs])) + if len(file_types) >= 2: + file_types_str = ",".join(file_types) + msg.fail("All input files must be same type", file_types_str, exits=1) + input_path = input_locs[0] + else: + input_path = walk_directory(input_path, suffix=converter)[0] + if converter == AUTO: converter = input_path.suffix[1:] if converter == "ner" or converter == "iob": with input_path.open(encoding="utf8") as file_: diff --git a/spacy/tests/test_cli.py b/spacy/tests/test_cli.py index c6768a3fd..c88e20de2 100644 --- a/spacy/tests/test_cli.py +++ b/spacy/tests/test_cli.py @@ -4,6 +4,7 @@ from collections import Counter from typing import Tuple, List, Dict, Any import pkg_resources import time +from pathlib import Path import spacy import numpy @@ -15,7 +16,7 @@ from thinc.api import Config, ConfigValidationError from spacy import about from spacy.cli import info -from spacy.cli._util import is_subpath_of, load_project_config +from spacy.cli._util import is_subpath_of, load_project_config, walk_directory from spacy.cli._util import parse_config_overrides, string_to_list from spacy.cli._util import substitute_project_variables from spacy.cli._util import validate_project_commands @@ -1185,3 +1186,26 @@ def test_upload_download_local_file(): download_file(remote_file, local_file) with local_file.open(mode="r") as file_: assert file_.read() == content + + +def test_walk_directory(): + with make_tempdir() as d: + files = [ + "data1.iob", + "data2.iob", + "data3.json", + "data4.conll", + "data5.conll", + "data6.conll", + "data7.txt", + ] + + for f in files: + Path(d / f).touch() + + assert (len(walk_directory(d))) == 7 + assert (len(walk_directory(d, suffix=None))) == 7 + assert (len(walk_directory(d, suffix="json"))) == 1 + assert (len(walk_directory(d, suffix="iob"))) == 2 + assert (len(walk_directory(d, suffix="conll"))) == 3 + assert (len(walk_directory(d, suffix="pdf"))) == 0 diff --git a/spacy/tests/test_cli_app.py b/spacy/tests/test_cli_app.py new file mode 100644 index 000000000..873a3ff66 --- /dev/null +++ b/spacy/tests/test_cli_app.py @@ -0,0 +1,33 @@ +import os +from pathlib import Path +from typer.testing import CliRunner + +from spacy.cli._util import app +from .util import make_tempdir + + +def test_convert_auto(): + with make_tempdir() as d_in, make_tempdir() as d_out: + for f in ["data1.iob", "data2.iob", "data3.iob"]: + Path(d_in / f).touch() + + # ensure that "automatic" suffix detection works + result = CliRunner().invoke(app, ["convert", str(d_in), str(d_out)]) + assert "Generated output file" in result.stdout + out_files = os.listdir(d_out) + assert len(out_files) == 3 + assert "data1.spacy" in out_files + assert "data2.spacy" in out_files + assert "data3.spacy" in out_files + + +def test_convert_auto_conflict(): + with make_tempdir() as d_in, make_tempdir() as d_out: + for f in ["data1.iob", "data2.iob", "data3.json"]: + Path(d_in / f).touch() + + # ensure that "automatic" suffix detection warns when there are different file types + result = CliRunner().invoke(app, ["convert", str(d_in), str(d_out)]) + assert "All input files must be same type" in result.stdout + out_files = os.listdir(d_out) + assert len(out_files) == 0 From f1dcdefc8abb21345680b79e9d538f06cf62bca0 Mon Sep 17 00:00:00 2001 From: Madeesh Kannan Date: Thu, 5 Jan 2023 11:46:04 +0100 Subject: [PATCH 35/44] Add version tag to `before_update` config key (#12059) --- website/docs/api/data-formats.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/api/data-formats.md b/website/docs/api/data-formats.md index 768844cf3..420e827a0 100644 --- a/website/docs/api/data-formats.md +++ b/website/docs/api/data-formats.md @@ -186,7 +186,7 @@ process that are used when you run [`spacy train`](/api/cli#train). | `accumulate_gradient` | Whether to divide the batch up into substeps. Defaults to `1`. ~~int~~ | | `batcher` | Callable that takes an iterator of [`Doc`](/api/doc) objects and yields batches of `Doc`s. Defaults to [`batch_by_words`](/api/top-level#batch_by_words). ~~Callable[[Iterator[Doc], Iterator[List[Doc]]]]~~ | | `before_to_disk` | Optional callback to modify `nlp` object right before it is saved to disk during and after training. Can be used to remove or reset config values or disable components. Defaults to `null`. ~~Optional[Callable[[Language], Language]]~~ | -| `before_update` | Optional callback that is invoked at the start of each training step with the `nlp` object and a `Dict` containing the following entries: `step`, `epoch`. Can be used to make deferred changes to components. Defaults to `null`. ~~Optional[Callable[[Language, Dict[str, Any]], None]]~~ | +| `before_update` 3.5 | Optional callback that is invoked at the start of each training step with the `nlp` object and a `Dict` containing the following entries: `step`, `epoch`. Can be used to make deferred changes to components. Defaults to `null`. ~~Optional[Callable[[Language, Dict[str, Any]], None]]~~ | | `dev_corpus` | Dot notation of the config location defining the dev corpus. Defaults to `corpora.dev`. ~~str~~ | | `dropout` | The dropout rate. Defaults to `0.1`. ~~float~~ | | `eval_frequency` | How often to evaluate during training (steps). Defaults to `200`. ~~int~~ | From 6d03b04901e95a71747a7e1ef0b00bc87bb2c807 Mon Sep 17 00:00:00 2001 From: Sofie Van Landeghem Date: Mon, 9 Jan 2023 11:43:48 +0100 Subject: [PATCH 36/44] Improve score_cats for use with multiple textcat components (#11820) * add test for running evaluate on an nlp pipeline with two distinct textcat components * cleanup * merge dicts instead of overwrite * don't add more labels to the given set * Revert "merge dicts instead of overwrite" This reverts commit 89bee0ed7798389e6de882a0234e6075fbdaf331. * Switch tests to separate scorer keys rather than merged dicts * Revert unrelated edits * Switch textcat scorers to v2 * formatting Co-authored-by: Adriane Boyd --- spacy/pipeline/textcat_multilabel.py | 4 +- spacy/scorer.py | 6 +- spacy/tests/pipeline/test_textcat.py | 6 +- spacy/tests/test_language.py | 107 +++++++++++++++++++++++++++ 4 files changed, 116 insertions(+), 7 deletions(-) diff --git a/spacy/pipeline/textcat_multilabel.py b/spacy/pipeline/textcat_multilabel.py index 328cee723..41c0e2f63 100644 --- a/spacy/pipeline/textcat_multilabel.py +++ b/spacy/pipeline/textcat_multilabel.py @@ -74,7 +74,7 @@ subword_features = true default_config={ "threshold": 0.5, "model": DEFAULT_MULTI_TEXTCAT_MODEL, - "scorer": {"@scorers": "spacy.textcat_multilabel_scorer.v1"}, + "scorer": {"@scorers": "spacy.textcat_multilabel_scorer.v2"}, }, default_score_weights={ "cats_score": 1.0, @@ -120,7 +120,7 @@ def textcat_multilabel_score(examples: Iterable[Example], **kwargs) -> Dict[str, ) -@registry.scorers("spacy.textcat_multilabel_scorer.v1") +@registry.scorers("spacy.textcat_multilabel_scorer.v2") def make_textcat_multilabel_scorer(): return textcat_multilabel_score diff --git a/spacy/scorer.py b/spacy/scorer.py index 16fc303a0..d8c383ab8 100644 --- a/spacy/scorer.py +++ b/spacy/scorer.py @@ -476,14 +476,12 @@ class Scorer: f_per_type = {label: PRFScore() for label in labels} auc_per_type = {label: ROCAUCScore() for label in labels} labels = set(labels) - if labels: - for eg in examples: - labels.update(eg.predicted.cats.keys()) - labels.update(eg.reference.cats.keys()) for example in examples: # Through this loop, None in the gold_cats indicates missing label. pred_cats = getter(example.predicted, attr) + pred_cats = {k: v for k, v in pred_cats.items() if k in labels} gold_cats = getter(example.reference, attr) + gold_cats = {k: v for k, v in gold_cats.items() if k in labels} for label in labels: pred_score = pred_cats.get(label, 0.0) diff --git a/spacy/tests/pipeline/test_textcat.py b/spacy/tests/pipeline/test_textcat.py index 048586cec..d042f3445 100644 --- a/spacy/tests/pipeline/test_textcat.py +++ b/spacy/tests/pipeline/test_textcat.py @@ -898,7 +898,11 @@ def test_textcat_multi_threshold(): @pytest.mark.parametrize( - "component_name,scorer", [("textcat", "spacy.textcat_scorer.v1")] + "component_name,scorer", + [ + ("textcat", "spacy.textcat_scorer.v1"), + ("textcat_multilabel", "spacy.textcat_multilabel_scorer.v1"), + ], ) def test_textcat_legacy_scorers(component_name, scorer): """Check that legacy scorers are registered and produce the expected score diff --git a/spacy/tests/test_language.py b/spacy/tests/test_language.py index 03a98d32f..03790eb86 100644 --- a/spacy/tests/test_language.py +++ b/spacy/tests/test_language.py @@ -3,6 +3,7 @@ import logging from unittest import mock import pytest from spacy.language import Language +from spacy.scorer import Scorer from spacy.tokens import Doc, Span from spacy.vocab import Vocab from spacy.training import Example @@ -126,6 +127,112 @@ def test_evaluate_no_pipe(nlp): nlp.evaluate([Example.from_dict(doc, annots)]) +def test_evaluate_textcat_multilabel(en_vocab): + """Test that evaluate works with a multilabel textcat pipe.""" + nlp = Language(en_vocab) + textcat_multilabel = nlp.add_pipe("textcat_multilabel") + for label in ("FEATURE", "REQUEST", "BUG", "QUESTION"): + textcat_multilabel.add_label(label) + nlp.initialize() + + annots = {"cats": {"FEATURE": 1.0, "QUESTION": 1.0}} + doc = nlp.make_doc("hello world") + example = Example.from_dict(doc, annots) + scores = nlp.evaluate([example]) + labels = nlp.get_pipe("textcat_multilabel").labels + for label in labels: + assert scores["cats_f_per_type"].get(label) is not None + for key in example.reference.cats.keys(): + if key not in labels: + assert scores["cats_f_per_type"].get(key) is None + + +def test_evaluate_multiple_textcat_final(en_vocab): + """Test that evaluate evaluates the final textcat component in a pipeline + with more than one textcat or textcat_multilabel.""" + nlp = Language(en_vocab) + textcat = nlp.add_pipe("textcat") + for label in ("POSITIVE", "NEGATIVE"): + textcat.add_label(label) + textcat_multilabel = nlp.add_pipe("textcat_multilabel") + for label in ("FEATURE", "REQUEST", "BUG", "QUESTION"): + textcat_multilabel.add_label(label) + nlp.initialize() + + annots = { + "cats": { + "POSITIVE": 1.0, + "NEGATIVE": 0.0, + "FEATURE": 1.0, + "QUESTION": 1.0, + "POSITIVE": 1.0, + "NEGATIVE": 0.0, + } + } + doc = nlp.make_doc("hello world") + example = Example.from_dict(doc, annots) + scores = nlp.evaluate([example]) + # get the labels from the final pipe + labels = nlp.get_pipe(nlp.pipe_names[-1]).labels + for label in labels: + assert scores["cats_f_per_type"].get(label) is not None + for key in example.reference.cats.keys(): + if key not in labels: + assert scores["cats_f_per_type"].get(key) is None + + +def test_evaluate_multiple_textcat_separate(en_vocab): + """Test that evaluate can evaluate multiple textcat components separately + with custom scorers.""" + + def custom_textcat_score(examples, **kwargs): + scores = Scorer.score_cats( + examples, + "cats", + multi_label=False, + **kwargs, + ) + return {f"custom_{k}": v for k, v in scores.items()} + + @spacy.registry.scorers("test_custom_textcat_scorer") + def make_custom_textcat_scorer(): + return custom_textcat_score + + nlp = Language(en_vocab) + textcat = nlp.add_pipe( + "textcat", + config={"scorer": {"@scorers": "test_custom_textcat_scorer"}}, + ) + for label in ("POSITIVE", "NEGATIVE"): + textcat.add_label(label) + textcat_multilabel = nlp.add_pipe("textcat_multilabel") + for label in ("FEATURE", "REQUEST", "BUG", "QUESTION"): + textcat_multilabel.add_label(label) + nlp.initialize() + + annots = { + "cats": { + "POSITIVE": 1.0, + "NEGATIVE": 0.0, + "FEATURE": 1.0, + "QUESTION": 1.0, + "POSITIVE": 1.0, + "NEGATIVE": 0.0, + } + } + doc = nlp.make_doc("hello world") + example = Example.from_dict(doc, annots) + scores = nlp.evaluate([example]) + # check custom scores for the textcat pipe + assert "custom_cats_f_per_type" in scores + labels = nlp.get_pipe("textcat").labels + assert set(scores["custom_cats_f_per_type"].keys()) == set(labels) + # check default scores for the textcat_multilabel pipe + assert "cats_f_per_type" in scores + labels = nlp.get_pipe("textcat_multilabel").labels + assert set(scores["cats_f_per_type"].keys()) == set(labels) + + def vector_modification_pipe(doc): doc.vector += 1 return doc From eb8bb35c13a5f59826761065e4eeccee69d4c5a7 Mon Sep 17 00:00:00 2001 From: Zhangrp Date: Tue, 10 Jan 2023 14:52:57 +0800 Subject: [PATCH 37/44] improve ux for displacy when the serve port is in use (#11948) * check port in use and add itself * check port in use and add itself * Auto switch to nearest available port. * Use bind to check port instead of connect_ex. * Reformat. * Add auto_select_port argument. * update docs for displacy.serve * Update spacy/errors.py Co-authored-by: Paul O'Leary McCann * Update website/docs/api/top-level.md Co-authored-by: Paul O'Leary McCann * Update spacy/errors.py Co-authored-by: Paul O'Leary McCann * Add test using multiprocessing * fix argument name * Increase sleep times Want to rule this out as a cause of test failure * Don't terminate a process that isn't alive * Refactor port finding logic This moves all the port logic into its own util function, which can be tested without having to background a server directly. * Use with for the server This ensures the server is closed correctly. * Pass in the host when checking port availability * Shorten argument name * Update error codes following merge * Add types for arguments, specify docstrings. * Add typing for arguments with default value. * Update docstring to match spaCy format. * Update docstring to match spaCy format. * Fix docs Arg name changed from `auto_select_port` to just `auto_select`. * Revert "Fix docs" This reverts commit 356966fe849660c0c08b670c6aee1aa2af05c1c1. Co-authored-by: zhiiw <1302593554@qq.com> Co-authored-by: Paul O'Leary McCann Co-authored-by: Raphael Mitsch --- spacy/displacy/__init__.py | 9 ++++++- spacy/errors.py | 5 ++++ spacy/tests/test_misc.py | 15 ++++++++++- spacy/util.py | 48 +++++++++++++++++++++++++++++++++++ website/docs/api/top-level.md | 21 +++++++-------- 5 files changed, 86 insertions(+), 12 deletions(-) diff --git a/spacy/displacy/__init__.py b/spacy/displacy/__init__.py index 2f2058b8e..a3cfd96dd 100644 --- a/spacy/displacy/__init__.py +++ b/spacy/displacy/__init__.py @@ -11,6 +11,7 @@ from .render import DependencyRenderer, EntityRenderer, SpanRenderer from ..tokens import Doc, Span from ..errors import Errors, Warnings from ..util import is_in_jupyter +from ..util import find_available_port _html = {} @@ -82,6 +83,7 @@ def serve( manual: bool = False, port: int = 5000, host: str = "0.0.0.0", + auto_select_port: bool = False, ) -> None: """Serve displaCy visualisation. @@ -93,15 +95,20 @@ def serve( manual (bool): Don't parse `Doc` and instead expect a dict/list of dicts. port (int): Port to serve visualisation. host (str): Host to serve visualisation. + auto_select_port (bool): Automatically select a port if the specified port is in use. DOCS: https://spacy.io/api/top-level#displacy.serve USAGE: https://spacy.io/usage/visualizers """ from wsgiref import simple_server + port = find_available_port(port, host, auto_select_port) + if is_in_jupyter(): warnings.warn(Warnings.W011) - render(docs, style=style, page=page, minify=minify, options=options, manual=manual) + render( + docs, style=style, page=page, minify=minify, options=options, manual=manual + ) httpd = simple_server.make_server(host, port, app) print(f"\nUsing the '{style}' visualizer") print(f"Serving on http://{host}:{port} ...\n") diff --git a/spacy/errors.py b/spacy/errors.py index cd9281e91..498df0320 100644 --- a/spacy/errors.py +++ b/spacy/errors.py @@ -214,6 +214,7 @@ class Warnings(metaclass=ErrorsWithCodes): "is a Cython extension type.") W123 = ("Argument `enable` with value {enable} does not contain all values specified in the config option " "`enabled` ({enabled}). Be aware that this might affect other components in your pipeline.") + W124 = ("{host}:{port} is already in use, using the nearest available port {serve_port} as an alternative.") class Errors(metaclass=ErrorsWithCodes): @@ -963,6 +964,10 @@ class Errors(metaclass=ErrorsWithCodes): "knowledge base, use `InMemoryLookupKB`.") E1047 = ("`find_threshold()` only supports components with a `scorer` attribute.") E1048 = ("Got '{unexpected}' as console progress bar type, but expected one of the following: {expected}") + E1049 = ("No available port found for displaCy on host {host}. Please specify an available port " + "with `displacy.serve(doc, port)`") + E1050 = ("Port {port} is already in use. Please specify an available port with `displacy.serve(doc, port)` " + "or use `auto_switch_port=True` to pick an available port automatically.") # Deprecated model shortcuts, only used in errors and warnings diff --git a/spacy/tests/test_misc.py b/spacy/tests/test_misc.py index 1c9b045ac..618f17334 100644 --- a/spacy/tests/test_misc.py +++ b/spacy/tests/test_misc.py @@ -8,7 +8,7 @@ from spacy import prefer_gpu, require_gpu, require_cpu from spacy.ml._precomputable_affine import PrecomputableAffine from spacy.ml._precomputable_affine import _backprop_precomputable_affine_padding from spacy.util import dot_to_object, SimpleFrozenList, import_file -from spacy.util import to_ternary_int +from spacy.util import to_ternary_int, find_available_port from thinc.api import Config, Optimizer, ConfigValidationError from thinc.api import get_current_ops, set_current_ops, NumpyOps, CupyOps, MPSOps from thinc.compat import has_cupy_gpu, has_torch_mps_gpu @@ -434,3 +434,16 @@ def test_to_ternary_int(): assert to_ternary_int(-10) == -1 assert to_ternary_int("string") == -1 assert to_ternary_int([0, "string"]) == -1 + + +def test_find_available_port(): + host = "0.0.0.0" + port = 5000 + assert find_available_port(port, host) == port, "Port 5000 isn't free" + + from wsgiref.simple_server import make_server, demo_app + + with make_server(host, port, demo_app) as httpd: + with pytest.warns(UserWarning, match="already in use"): + found_port = find_available_port(port, host, auto_select=True) + assert found_port == port + 1, "Didn't find next port" diff --git a/spacy/util.py b/spacy/util.py index 8d211a9a5..8bf8fb1b0 100644 --- a/spacy/util.py +++ b/spacy/util.py @@ -31,6 +31,7 @@ import shlex import inspect import pkgutil import logging +import socket try: import cupy.random @@ -1736,3 +1737,50 @@ def all_equal(iterable): (or if the input is an empty sequence), False otherwise.""" g = itertools.groupby(iterable) return next(g, True) and not next(g, False) + + +def _is_port_in_use(port: int, host: str = "localhost") -> bool: + """Check if 'host:port' is in use. Return True if it is, False otherwise. + + port (int): the port to check + host (str): the host to check (default "localhost") + RETURNS (bool): Whether 'host:port' is in use. + """ + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + try: + s.bind((host, port)) + return False + except socket.error: + return True + finally: + s.close() + + +def find_available_port(start: int, host: str, auto_select: bool = False) -> int: + """Given a starting port and a host, handle finding a port. + + If `auto_select` is False, a busy port will raise an error. + + If `auto_select` is True, the next free higher port will be used. + + start (int): the port to start looking from + host (str): the host to find a port on + auto_select (bool): whether to automatically select a new port if the given port is busy (default False) + RETURNS (int): The port to use. + """ + if not _is_port_in_use(start, host): + return start + + port = start + if not auto_select: + raise ValueError(Errors.E1050.format(port=port)) + + while _is_port_in_use(port, host) and port < 65535: + port += 1 + + if port == 65535 and _is_port_in_use(port, host): + raise ValueError(Errors.E1049.format(host=host)) + + # if we get here, the port changed + warnings.warn(Warnings.W124.format(host=host, port=start, serve_port=port)) + return port diff --git a/website/docs/api/top-level.md b/website/docs/api/top-level.md index 6a63e07da..9d3e463d8 100644 --- a/website/docs/api/top-level.md +++ b/website/docs/api/top-level.md @@ -237,16 +237,17 @@ browser. Will run a simple web server. > displacy.serve([doc1, doc2], style="dep") > ``` -| Name | Description | -| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `docs` | Document(s) or span(s) to visualize. ~~Union[Iterable[Union[Doc, Span]], Doc, Span]~~ | -| `style` | Visualization style, `"dep"`, `"ent"` or `"span"` 3.3. Defaults to `"dep"`. ~~str~~ | -| `page` | Render markup as full HTML page. Defaults to `True`. ~~bool~~ | -| `minify` | Minify HTML markup. Defaults to `False`. ~~bool~~ | -| `options` | [Visualizer-specific options](#displacy_options), e.g. colors. ~~Dict[str, Any]~~ | -| `manual` | Don't parse `Doc` and instead expect a dict or list of dicts. [See here](/usage/visualizers#manual-usage) for formats and examples. Defaults to `False`. ~~bool~~ | -| `port` | Port to serve visualization. Defaults to `5000`. ~~int~~ | -| `host` | Host to serve visualization. Defaults to `"0.0.0.0"`. ~~str~~ | +| Name | Description | +| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `docs` | Document(s) or span(s) to visualize. ~~Union[Iterable[Union[Doc, Span]], Doc, Span]~~ | +| `style` | Visualization style, `"dep"`, `"ent"` or `"span"` 3.3. Defaults to `"dep"`. ~~str~~ | +| `page` | Render markup as full HTML page. Defaults to `True`. ~~bool~~ | +| `minify` | Minify HTML markup. Defaults to `False`. ~~bool~~ | +| `options` | [Visualizer-specific options](#displacy_options), e.g. colors. ~~Dict[str, Any]~~ | +| `manual` | Don't parse `Doc` and instead expect a dict or list of dicts. [See here](/usage/visualizers#manual-usage) for formats and examples. Defaults to `False`. ~~bool~~ | +| `port` | Port to serve visualization. Defaults to `5000`. ~~int~~ | +| `host` | Host to serve visualization. Defaults to `"0.0.0.0"`. ~~str~~ | +| `auto_select_port` | If `True`, automatically switch to a different port if the specified port is already in use. Defaults to `False`. ~~bool~~ | ### displacy.render {#displacy.render tag="method" new="2"} From 19650ebb52222cf2bc3402b7c74f68f3a9f0a4e3 Mon Sep 17 00:00:00 2001 From: Kevin Humphreys Date: Tue, 10 Jan 2023 01:36:17 -0800 Subject: [PATCH 38/44] Enable fuzzy text matching in Matcher (#11359) * enable fuzzy matching * add fuzzy param to EntityMatcher * include rapidfuzz_capi not yet used * fix type * add FUZZY predicate * add fuzzy attribute list * fix type properly * tidying * remove unnecessary dependency * handle fuzzy sets * simplify fuzzy sets * case fix * switch to FUZZYn predicates use Levenshtein distance. remove fuzzy param. remove rapidfuzz_capi. * revert changes added for fuzzy param * switch to polyleven (Python package) * enable fuzzy matching * add fuzzy param to EntityMatcher * include rapidfuzz_capi not yet used * fix type * add FUZZY predicate * add fuzzy attribute list * fix type properly * tidying * remove unnecessary dependency * handle fuzzy sets * simplify fuzzy sets * case fix * switch to FUZZYn predicates use Levenshtein distance. remove fuzzy param. remove rapidfuzz_capi. * revert changes added for fuzzy param * switch to polyleven (Python package) * fuzzy match only on oov tokens * remove polyleven * exclude whitespace tokens * don't allow more edits than characters * fix min distance * reinstate FUZZY operator with length-based distance function * handle sets inside regex operator * remove is_oov check * attempt build fix no mypy failure locally * re-attempt build fix * don't overwrite fuzzy param value * move fuzzy_match to its own Python module to allow patching * move fuzzy_match back inside Matcher simplify logic and add tests * Format tests * Parametrize fuzzyn tests * Parametrize and merge fuzzy+set tests * Format * Move fuzzy_match to a standalone method * Change regex kwarg type to bool * Add types for fuzzy_match - Refactor variable names - Add test for symmetrical behavior * Parametrize fuzzyn+set tests * Minor refactoring for fuzz/fuzzy * Make fuzzy_match a Matcher kwarg * Update type for _default_fuzzy_match * don't overwrite function param * Rename to fuzzy_compare * Update fuzzy_compare default argument declarations * allow fuzzy_compare override from EntityRuler * define new Matcher keyword arg * fix type definition * Implement fuzzy_compare config option for EntityRuler and SpanRuler * Rename _default_fuzzy_compare to fuzzy_compare, remove from reexported objects * Use simpler fuzzy_compare algorithm * Update types * Increase minimum to 2 in fuzzy_compare to allow one transposition * Fix predicate keys and matching for SetPredicate with FUZZY and REGEX * Add FUZZY6..9 * Add initial docs * Increase default fuzzy to rounded 30% of pattern length * Update docs for fuzzy_compare in components * Update EntityRuler and SpanRuler API docs * Rename EntityRuler and SpanRuler setting to matcher_fuzzy_compare To having naming similar to `phrase_matcher_attr`, rename `fuzzy_compare` setting for `EntityRuler` and `SpanRuler` to `matcher_fuzzy_compare. Organize next to `phrase_matcher_attr` in docs. * Fix schema aliases Co-authored-by: Sofie Van Landeghem * Fix typo Co-authored-by: Sofie Van Landeghem * Add FUZZY6-9 operators and update tests * Parameterize test over greedy Co-authored-by: Sofie Van Landeghem * Fix type for fuzzy_compare to remove Optional * Rename to spacy.levenshtein_compare.v1, move to spacy.matcher.levenshtein * Update docs following levenshtein_compare renaming Co-authored-by: Adriane Boyd Co-authored-by: Sofie Van Landeghem --- spacy/matcher/levenshtein.pyx | 17 +++ spacy/matcher/matcher.pxd | 1 + spacy/matcher/matcher.pyi | 3 +- spacy/matcher/matcher.pyx | 170 ++++++++++++++++----- spacy/pipeline/entityruler.py | 24 ++- spacy/pipeline/span_ruler.py | 18 ++- spacy/schemas.py | 12 +- spacy/tests/matcher/test_levenshtein.py | 29 ++++ spacy/tests/matcher/test_matcher_api.py | 173 ++++++++++++++++++++++ spacy/tests/pipeline/test_entity_ruler.py | 37 +++++ website/docs/api/entityruler.md | 53 +++---- website/docs/api/matcher.md | 31 ++-- website/docs/api/spanruler.md | 48 +++--- website/docs/usage/rule-based-matching.md | 40 +++++ 14 files changed, 554 insertions(+), 102 deletions(-) diff --git a/spacy/matcher/levenshtein.pyx b/spacy/matcher/levenshtein.pyx index 8463d913d..0e8cd26da 100644 --- a/spacy/matcher/levenshtein.pyx +++ b/spacy/matcher/levenshtein.pyx @@ -4,6 +4,8 @@ from libc.stdint cimport int64_t from typing import Optional +from ..util import registry + cdef extern from "polyleven.c": int64_t polyleven(PyObject *o1, PyObject *o2, int64_t k) @@ -13,3 +15,18 @@ cpdef int64_t levenshtein(a: str, b: str, k: Optional[int] = None): if k is None: k = -1 return polyleven(a, b, k) + + +cpdef bint levenshtein_compare(input_text: str, pattern_text: str, fuzzy: int = -1): + if fuzzy >= 0: + max_edits = fuzzy + else: + # allow at least two edits (to allow at least one transposition) and up + # to 20% of the pattern string length + max_edits = max(2, round(0.3 * len(pattern_text))) + return levenshtein(input_text, pattern_text, max_edits) <= max_edits + + +@registry.misc("spacy.levenshtein_compare.v1") +def make_levenshtein_compare(): + return levenshtein_compare diff --git a/spacy/matcher/matcher.pxd b/spacy/matcher/matcher.pxd index 455f978cc..51854d562 100644 --- a/spacy/matcher/matcher.pxd +++ b/spacy/matcher/matcher.pxd @@ -77,3 +77,4 @@ cdef class Matcher: cdef public object _extensions cdef public object _extra_predicates cdef public object _seen_attrs + cdef public object _fuzzy_compare diff --git a/spacy/matcher/matcher.pyi b/spacy/matcher/matcher.pyi index 390629ff8..77ea7b7a6 100644 --- a/spacy/matcher/matcher.pyi +++ b/spacy/matcher/matcher.pyi @@ -5,7 +5,8 @@ from ..vocab import Vocab from ..tokens import Doc, Span class Matcher: - def __init__(self, vocab: Vocab, validate: bool = ...) -> None: ... + def __init__(self, vocab: Vocab, validate: bool = ..., + fuzzy_compare: Callable[[str, str, int], bool] = ...) -> None: ... def __reduce__(self) -> Any: ... def __len__(self) -> int: ... def __contains__(self, key: str) -> bool: ... diff --git a/spacy/matcher/matcher.pyx b/spacy/matcher/matcher.pyx index c4a057ca0..ea1b4b66b 100644 --- a/spacy/matcher/matcher.pyx +++ b/spacy/matcher/matcher.pyx @@ -1,4 +1,4 @@ -# cython: infer_types=True, profile=True +# cython: binding=True, infer_types=True, profile=True from typing import List, Iterable from libcpp.vector cimport vector @@ -20,10 +20,12 @@ from ..tokens.token cimport Token from ..tokens.morphanalysis cimport MorphAnalysis from ..attrs cimport ID, attr_id_t, NULL_ATTR, ORTH, POS, TAG, DEP, LEMMA, MORPH, ENT_IOB +from .levenshtein import levenshtein_compare from ..schemas import validate_token_pattern from ..errors import Errors, MatchPatternError, Warnings from ..strings import get_string_id from ..attrs import IDS +from ..util import registry DEF PADDING = 5 @@ -36,11 +38,13 @@ cdef class Matcher: USAGE: https://spacy.io/usage/rule-based-matching """ - def __init__(self, vocab, validate=True): + def __init__(self, vocab, validate=True, *, fuzzy_compare=levenshtein_compare): """Create the Matcher. vocab (Vocab): The vocabulary object, which must be shared with the - documents the matcher will operate on. + validate (bool): Validate all patterns added to this matcher. + fuzzy_compare (Callable[[str, str, int], bool]): The comparison method + for the FUZZY operators. """ self._extra_predicates = [] self._patterns = {} @@ -51,9 +55,10 @@ cdef class Matcher: self.vocab = vocab self.mem = Pool() self.validate = validate + self._fuzzy_compare = fuzzy_compare def __reduce__(self): - data = (self.vocab, self._patterns, self._callbacks) + data = (self.vocab, self._patterns, self._callbacks, self.validate, self._fuzzy_compare) return (unpickle_matcher, data, None, None) def __len__(self): @@ -128,7 +133,7 @@ cdef class Matcher: for pattern in patterns: try: specs = _preprocess_pattern(pattern, self.vocab, - self._extensions, self._extra_predicates) + self._extensions, self._extra_predicates, self._fuzzy_compare) self.patterns.push_back(init_pattern(self.mem, key, specs)) for spec in specs: for attr, _ in spec[1]: @@ -326,8 +331,8 @@ cdef class Matcher: return key -def unpickle_matcher(vocab, patterns, callbacks): - matcher = Matcher(vocab) +def unpickle_matcher(vocab, patterns, callbacks, validate, fuzzy_compare): + matcher = Matcher(vocab, validate=validate, fuzzy_compare=fuzzy_compare) for key, pattern in patterns.items(): callback = callbacks.get(key, None) matcher.add(key, pattern, on_match=callback) @@ -754,7 +759,7 @@ cdef attr_t get_ent_id(const TokenPatternC* pattern) nogil: return id_attr.value -def _preprocess_pattern(token_specs, vocab, extensions_table, extra_predicates): +def _preprocess_pattern(token_specs, vocab, extensions_table, extra_predicates, fuzzy_compare): """This function interprets the pattern, converting the various bits of syntactic sugar before we compile it into a struct with init_pattern. @@ -781,7 +786,7 @@ def _preprocess_pattern(token_specs, vocab, extensions_table, extra_predicates): ops = _get_operators(spec) attr_values = _get_attr_values(spec, string_store) extensions = _get_extensions(spec, string_store, extensions_table) - predicates = _get_extra_predicates(spec, extra_predicates, vocab) + predicates = _get_extra_predicates(spec, extra_predicates, vocab, fuzzy_compare) for op in ops: tokens.append((op, list(attr_values), list(extensions), list(predicates), token_idx)) return tokens @@ -826,16 +831,45 @@ def _get_attr_values(spec, string_store): # These predicate helper classes are used to match the REGEX, IN, >= etc # extensions to the matcher introduced in #3173. +class _FuzzyPredicate: + operators = ("FUZZY", "FUZZY1", "FUZZY2", "FUZZY3", "FUZZY4", "FUZZY5", + "FUZZY6", "FUZZY7", "FUZZY8", "FUZZY9") + + def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None, + regex=False, fuzzy=None, fuzzy_compare=None): + self.i = i + self.attr = attr + self.value = value + self.predicate = predicate + self.is_extension = is_extension + if self.predicate not in self.operators: + raise ValueError(Errors.E126.format(good=self.operators, bad=self.predicate)) + fuzz = self.predicate[len("FUZZY"):] # number after prefix + self.fuzzy = int(fuzz) if fuzz else -1 + self.fuzzy_compare = fuzzy_compare + self.key = (self.attr, self.fuzzy, self.predicate, srsly.json_dumps(value, sort_keys=True)) + + def __call__(self, Token token): + if self.is_extension: + value = token._.get(self.attr) + else: + value = token.vocab.strings[get_token_attr_for_matcher(token.c, self.attr)] + if self.value == value: + return True + return self.fuzzy_compare(value, self.value, self.fuzzy) + + class _RegexPredicate: operators = ("REGEX",) - def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None): + def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None, + regex=False, fuzzy=None, fuzzy_compare=None): self.i = i self.attr = attr self.value = re.compile(value) self.predicate = predicate self.is_extension = is_extension - self.key = (attr, self.predicate, srsly.json_dumps(value, sort_keys=True)) + self.key = (self.attr, self.predicate, srsly.json_dumps(value, sort_keys=True)) if self.predicate not in self.operators: raise ValueError(Errors.E126.format(good=self.operators, bad=self.predicate)) @@ -850,18 +884,28 @@ class _RegexPredicate: class _SetPredicate: operators = ("IN", "NOT_IN", "IS_SUBSET", "IS_SUPERSET", "INTERSECTS") - def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None): + def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None, + regex=False, fuzzy=None, fuzzy_compare=None): self.i = i self.attr = attr self.vocab = vocab + self.regex = regex + self.fuzzy = fuzzy + self.fuzzy_compare = fuzzy_compare if self.attr == MORPH: # normalize morph strings self.value = set(self.vocab.morphology.add(v) for v in value) else: - self.value = set(get_string_id(v) for v in value) + if self.regex: + self.value = set(re.compile(v) for v in value) + elif self.fuzzy is not None: + # add to string store + self.value = set(self.vocab.strings.add(v) for v in value) + else: + self.value = set(get_string_id(v) for v in value) self.predicate = predicate self.is_extension = is_extension - self.key = (attr, self.predicate, srsly.json_dumps(value, sort_keys=True)) + self.key = (self.attr, self.regex, self.fuzzy, self.predicate, srsly.json_dumps(value, sort_keys=True)) if self.predicate not in self.operators: raise ValueError(Errors.E126.format(good=self.operators, bad=self.predicate)) @@ -889,9 +933,29 @@ class _SetPredicate: return False if self.predicate == "IN": - return value in self.value + if self.regex: + value = self.vocab.strings[value] + return any(bool(v.search(value)) for v in self.value) + elif self.fuzzy is not None: + value = self.vocab.strings[value] + return any(self.fuzzy_compare(value, self.vocab.strings[v], self.fuzzy) + for v in self.value) + elif value in self.value: + return True + else: + return False elif self.predicate == "NOT_IN": - return value not in self.value + if self.regex: + value = self.vocab.strings[value] + return not any(bool(v.search(value)) for v in self.value) + elif self.fuzzy is not None: + value = self.vocab.strings[value] + return not any(self.fuzzy_compare(value, self.vocab.strings[v], self.fuzzy) + for v in self.value) + elif value in self.value: + return False + else: + return True elif self.predicate == "IS_SUBSET": return value <= self.value elif self.predicate == "IS_SUPERSET": @@ -906,13 +970,14 @@ class _SetPredicate: class _ComparisonPredicate: operators = ("==", "!=", ">=", "<=", ">", "<") - def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None): + def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None, + regex=False, fuzzy=None, fuzzy_compare=None): self.i = i self.attr = attr self.value = value self.predicate = predicate self.is_extension = is_extension - self.key = (attr, self.predicate, srsly.json_dumps(value, sort_keys=True)) + self.key = (self.attr, self.predicate, srsly.json_dumps(value, sort_keys=True)) if self.predicate not in self.operators: raise ValueError(Errors.E126.format(good=self.operators, bad=self.predicate)) @@ -935,7 +1000,7 @@ class _ComparisonPredicate: return value < self.value -def _get_extra_predicates(spec, extra_predicates, vocab): +def _get_extra_predicates(spec, extra_predicates, vocab, fuzzy_compare): predicate_types = { "REGEX": _RegexPredicate, "IN": _SetPredicate, @@ -949,6 +1014,16 @@ def _get_extra_predicates(spec, extra_predicates, vocab): "<=": _ComparisonPredicate, ">": _ComparisonPredicate, "<": _ComparisonPredicate, + "FUZZY": _FuzzyPredicate, + "FUZZY1": _FuzzyPredicate, + "FUZZY2": _FuzzyPredicate, + "FUZZY3": _FuzzyPredicate, + "FUZZY4": _FuzzyPredicate, + "FUZZY5": _FuzzyPredicate, + "FUZZY6": _FuzzyPredicate, + "FUZZY7": _FuzzyPredicate, + "FUZZY8": _FuzzyPredicate, + "FUZZY9": _FuzzyPredicate, } seen_predicates = {pred.key: pred.i for pred in extra_predicates} output = [] @@ -966,22 +1041,47 @@ def _get_extra_predicates(spec, extra_predicates, vocab): attr = "ORTH" attr = IDS.get(attr.upper()) if isinstance(value, dict): - processed = False - value_with_upper_keys = {k.upper(): v for k, v in value.items()} - for type_, cls in predicate_types.items(): - if type_ in value_with_upper_keys: - predicate = cls(len(extra_predicates), attr, value_with_upper_keys[type_], type_, vocab=vocab) - # Don't create a redundant predicates. - # This helps with efficiency, as we're caching the results. - if predicate.key in seen_predicates: - output.append(seen_predicates[predicate.key]) - else: - extra_predicates.append(predicate) - output.append(predicate.i) - seen_predicates[predicate.key] = predicate.i - processed = True - if not processed: - warnings.warn(Warnings.W035.format(pattern=value)) + output.extend(_get_extra_predicates_dict(attr, value, vocab, predicate_types, + extra_predicates, seen_predicates, fuzzy_compare=fuzzy_compare)) + return output + + +def _get_extra_predicates_dict(attr, value_dict, vocab, predicate_types, + extra_predicates, seen_predicates, regex=False, fuzzy=None, fuzzy_compare=None): + output = [] + for type_, value in value_dict.items(): + type_ = type_.upper() + cls = predicate_types.get(type_) + if cls is None: + warnings.warn(Warnings.W035.format(pattern=value_dict)) + # ignore unrecognized predicate type + continue + elif cls == _RegexPredicate: + if isinstance(value, dict): + # add predicates inside regex operator + output.extend(_get_extra_predicates_dict(attr, value, vocab, predicate_types, + extra_predicates, seen_predicates, + regex=True)) + continue + elif cls == _FuzzyPredicate: + if isinstance(value, dict): + # add predicates inside fuzzy operator + fuzz = type_[len("FUZZY"):] # number after prefix + fuzzy_val = int(fuzz) if fuzz else -1 + output.extend(_get_extra_predicates_dict(attr, value, vocab, predicate_types, + extra_predicates, seen_predicates, + fuzzy=fuzzy_val, fuzzy_compare=fuzzy_compare)) + continue + predicate = cls(len(extra_predicates), attr, value, type_, vocab=vocab, + regex=regex, fuzzy=fuzzy, fuzzy_compare=fuzzy_compare) + # Don't create redundant predicates. + # This helps with efficiency, as we're caching the results. + if predicate.key in seen_predicates: + output.append(seen_predicates[predicate.key]) + else: + extra_predicates.append(predicate) + output.append(predicate.i) + seen_predicates[predicate.key] = predicate.i return output diff --git a/spacy/pipeline/entityruler.py b/spacy/pipeline/entityruler.py index 8154a077d..6a3755533 100644 --- a/spacy/pipeline/entityruler.py +++ b/spacy/pipeline/entityruler.py @@ -11,6 +11,7 @@ from ..errors import Errors, Warnings from ..util import ensure_path, to_disk, from_disk, SimpleFrozenList, registry from ..tokens import Doc, Span from ..matcher import Matcher, PhraseMatcher +from ..matcher.levenshtein import levenshtein_compare from ..scorer import get_ner_prf @@ -23,6 +24,7 @@ PatternType = Dict[str, Union[str, List[Dict[str, Any]]]] assigns=["doc.ents", "token.ent_type", "token.ent_iob"], default_config={ "phrase_matcher_attr": None, + "matcher_fuzzy_compare": {"@misc": "spacy.levenshtein_compare.v1"}, "validate": False, "overwrite_ents": False, "ent_id_sep": DEFAULT_ENT_ID_SEP, @@ -39,6 +41,7 @@ def make_entity_ruler( nlp: Language, name: str, phrase_matcher_attr: Optional[Union[int, str]], + matcher_fuzzy_compare: Callable, validate: bool, overwrite_ents: bool, ent_id_sep: str, @@ -48,6 +51,7 @@ def make_entity_ruler( nlp, name, phrase_matcher_attr=phrase_matcher_attr, + matcher_fuzzy_compare=matcher_fuzzy_compare, validate=validate, overwrite_ents=overwrite_ents, ent_id_sep=ent_id_sep, @@ -81,6 +85,7 @@ class EntityRuler(Pipe): name: str = "entity_ruler", *, phrase_matcher_attr: Optional[Union[int, str]] = None, + matcher_fuzzy_compare: Callable = levenshtein_compare, validate: bool = False, overwrite_ents: bool = False, ent_id_sep: str = DEFAULT_ENT_ID_SEP, @@ -99,7 +104,10 @@ class EntityRuler(Pipe): added. Used to disable the current entity ruler while creating phrase patterns with the nlp object. phrase_matcher_attr (int / str): Token attribute to match on, passed - to the internal PhraseMatcher as `attr` + to the internal PhraseMatcher as `attr`. + matcher_fuzzy_compare (Callable): The fuzzy comparison method for the + internal Matcher. Defaults to + spacy.matcher.levenshtein.levenshtein_compare. validate (bool): Whether patterns should be validated, passed to Matcher and PhraseMatcher as `validate` patterns (iterable): Optional patterns to load in. @@ -117,7 +125,10 @@ class EntityRuler(Pipe): self.token_patterns = defaultdict(list) # type: ignore self.phrase_patterns = defaultdict(list) # type: ignore self._validate = validate - self.matcher = Matcher(nlp.vocab, validate=validate) + self.matcher_fuzzy_compare = matcher_fuzzy_compare + self.matcher = Matcher( + nlp.vocab, validate=validate, fuzzy_compare=self.matcher_fuzzy_compare + ) self.phrase_matcher_attr = phrase_matcher_attr self.phrase_matcher = PhraseMatcher( nlp.vocab, attr=self.phrase_matcher_attr, validate=validate @@ -337,7 +348,11 @@ class EntityRuler(Pipe): self.token_patterns = defaultdict(list) self.phrase_patterns = defaultdict(list) self._ent_ids = defaultdict(tuple) - self.matcher = Matcher(self.nlp.vocab, validate=self._validate) + self.matcher = Matcher( + self.nlp.vocab, + validate=self._validate, + fuzzy_compare=self.matcher_fuzzy_compare, + ) self.phrase_matcher = PhraseMatcher( self.nlp.vocab, attr=self.phrase_matcher_attr, validate=self._validate ) @@ -431,7 +446,8 @@ class EntityRuler(Pipe): self.overwrite = cfg.get("overwrite", False) self.phrase_matcher_attr = cfg.get("phrase_matcher_attr", None) self.phrase_matcher = PhraseMatcher( - self.nlp.vocab, attr=self.phrase_matcher_attr + self.nlp.vocab, + attr=self.phrase_matcher_attr, ) self.ent_id_sep = cfg.get("ent_id_sep", DEFAULT_ENT_ID_SEP) else: diff --git a/spacy/pipeline/span_ruler.py b/spacy/pipeline/span_ruler.py index 0e7e9ebf7..b0669c0ef 100644 --- a/spacy/pipeline/span_ruler.py +++ b/spacy/pipeline/span_ruler.py @@ -13,6 +13,7 @@ from ..util import ensure_path, SimpleFrozenList, registry from ..tokens import Doc, Span from ..scorer import Scorer from ..matcher import Matcher, PhraseMatcher +from ..matcher.levenshtein import levenshtein_compare from .. import util PatternType = Dict[str, Union[str, List[Dict[str, Any]]]] @@ -28,6 +29,7 @@ DEFAULT_SPANS_KEY = "ruler" "overwrite_ents": False, "scorer": {"@scorers": "spacy.entity_ruler_scorer.v1"}, "ent_id_sep": "__unused__", + "matcher_fuzzy_compare": {"@misc": "spacy.levenshtein_compare.v1"}, }, default_score_weights={ "ents_f": 1.0, @@ -40,6 +42,7 @@ def make_entity_ruler( nlp: Language, name: str, phrase_matcher_attr: Optional[Union[int, str]], + matcher_fuzzy_compare: Callable, validate: bool, overwrite_ents: bool, scorer: Optional[Callable], @@ -57,6 +60,7 @@ def make_entity_ruler( annotate_ents=True, ents_filter=ents_filter, phrase_matcher_attr=phrase_matcher_attr, + matcher_fuzzy_compare=matcher_fuzzy_compare, validate=validate, overwrite=False, scorer=scorer, @@ -72,6 +76,7 @@ def make_entity_ruler( "annotate_ents": False, "ents_filter": {"@misc": "spacy.first_longest_spans_filter.v1"}, "phrase_matcher_attr": None, + "matcher_fuzzy_compare": {"@misc": "spacy.levenshtein_compare.v1"}, "validate": False, "overwrite": True, "scorer": { @@ -94,6 +99,7 @@ def make_span_ruler( annotate_ents: bool, ents_filter: Callable[[Iterable[Span], Iterable[Span]], Iterable[Span]], phrase_matcher_attr: Optional[Union[int, str]], + matcher_fuzzy_compare: Callable, validate: bool, overwrite: bool, scorer: Optional[Callable], @@ -106,6 +112,7 @@ def make_span_ruler( annotate_ents=annotate_ents, ents_filter=ents_filter, phrase_matcher_attr=phrase_matcher_attr, + matcher_fuzzy_compare=matcher_fuzzy_compare, validate=validate, overwrite=overwrite, scorer=scorer, @@ -216,6 +223,7 @@ class SpanRuler(Pipe): [Iterable[Span], Iterable[Span]], Iterable[Span] ] = util.filter_chain_spans, phrase_matcher_attr: Optional[Union[int, str]] = None, + matcher_fuzzy_compare: Callable = levenshtein_compare, validate: bool = False, overwrite: bool = False, scorer: Optional[Callable] = partial( @@ -246,6 +254,9 @@ class SpanRuler(Pipe): phrase_matcher_attr (Optional[Union[int, str]]): Token attribute to match on, passed to the internal PhraseMatcher as `attr`. Defaults to `None`. + matcher_fuzzy_compare (Callable): The fuzzy comparison method for the + internal Matcher. Defaults to + spacy.matcher.levenshtein.levenshtein_compare. validate (bool): Whether patterns should be validated, passed to Matcher and PhraseMatcher as `validate`. overwrite (bool): Whether to remove any existing spans under this spans @@ -266,6 +277,7 @@ class SpanRuler(Pipe): self.spans_filter = spans_filter self.ents_filter = ents_filter self.scorer = scorer + self.matcher_fuzzy_compare = matcher_fuzzy_compare self._match_label_id_map: Dict[int, Dict[str, str]] = {} self.clear() @@ -451,7 +463,11 @@ class SpanRuler(Pipe): DOCS: https://spacy.io/api/spanruler#clear """ self._patterns: List[PatternType] = [] - self.matcher: Matcher = Matcher(self.nlp.vocab, validate=self.validate) + self.matcher: Matcher = Matcher( + self.nlp.vocab, + validate=self.validate, + fuzzy_compare=self.matcher_fuzzy_compare, + ) self.phrase_matcher: PhraseMatcher = PhraseMatcher( self.nlp.vocab, attr=self.phrase_matcher_attr, diff --git a/spacy/schemas.py b/spacy/schemas.py index e48fe1702..3675c12dd 100644 --- a/spacy/schemas.py +++ b/spacy/schemas.py @@ -156,12 +156,22 @@ def validate_token_pattern(obj: list) -> List[str]: class TokenPatternString(BaseModel): - REGEX: Optional[StrictStr] = Field(None, alias="regex") + REGEX: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="regex") IN: Optional[List[StrictStr]] = Field(None, alias="in") NOT_IN: Optional[List[StrictStr]] = Field(None, alias="not_in") IS_SUBSET: Optional[List[StrictStr]] = Field(None, alias="is_subset") IS_SUPERSET: Optional[List[StrictStr]] = Field(None, alias="is_superset") INTERSECTS: Optional[List[StrictStr]] = Field(None, alias="intersects") + FUZZY: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="fuzzy") + FUZZY1: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="fuzzy1") + FUZZY2: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="fuzzy2") + FUZZY3: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="fuzzy3") + FUZZY4: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="fuzzy4") + FUZZY5: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="fuzzy5") + FUZZY6: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="fuzzy6") + FUZZY7: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="fuzzy7") + FUZZY8: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="fuzzy8") + FUZZY9: Optional[Union[StrictStr, "TokenPatternString"]] = Field(None, alias="fuzzy9") class Config: extra = "forbid" diff --git a/spacy/tests/matcher/test_levenshtein.py b/spacy/tests/matcher/test_levenshtein.py index d30e36132..5afb7e1fc 100644 --- a/spacy/tests/matcher/test_levenshtein.py +++ b/spacy/tests/matcher/test_levenshtein.py @@ -1,5 +1,6 @@ import pytest from spacy.matcher import levenshtein +from spacy.matcher.levenshtein import levenshtein_compare # empty string plus 10 random ASCII, 10 random unicode, and 2 random long tests @@ -42,3 +43,31 @@ from spacy.matcher import levenshtein ) def test_levenshtein(dist, a, b): assert levenshtein(a, b) == dist + + +@pytest.mark.parametrize( + "a,b,fuzzy,expected", + [ + ("a", "a", 1, True), + ("a", "a", 0, True), + ("a", "a", -1, True), + ("a", "ab", 1, True), + ("a", "ab", 0, False), + ("a", "ab", -1, True), + ("ab", "ac", 1, True), + ("ab", "ac", -1, True), + ("abc", "cde", 4, True), + ("abc", "cde", -1, False), + ("abcdef", "cdefgh", 4, True), + ("abcdef", "cdefgh", 3, False), + ("abcdef", "cdefgh", -1, False), # default (2 for length 6) + ("abcdefgh", "cdefghijk", 5, True), + ("abcdefgh", "cdefghijk", 4, False), + ("abcdefgh", "cdefghijk", -1, False), # default (2) + ("abcdefgh", "cdefghijkl", 6, True), + ("abcdefgh", "cdefghijkl", 5, False), + ("abcdefgh", "cdefghijkl", -1, False), # default (2) + ], +) +def test_levenshtein_compare(a, b, fuzzy, expected): + assert levenshtein_compare(a, b, fuzzy) == expected diff --git a/spacy/tests/matcher/test_matcher_api.py b/spacy/tests/matcher/test_matcher_api.py index ac905eeb4..09ab6c7dc 100644 --- a/spacy/tests/matcher/test_matcher_api.py +++ b/spacy/tests/matcher/test_matcher_api.py @@ -118,6 +118,155 @@ def test_matcher_match_multi(matcher): ] +@pytest.mark.parametrize( + "rules,match_locs", + [ + ( + { + "GoogleNow": [[{"ORTH": {"FUZZY": "Google"}}, {"ORTH": "Now"}]], + }, + [(2, 4)], + ), + ( + { + "Java": [[{"LOWER": {"FUZZY": "java"}}]], + }, + [(5, 6)], + ), + ( + { + "JS": [[{"ORTH": {"FUZZY": "JavaScript"}}]], + "GoogleNow": [[{"ORTH": {"FUZZY": "Google"}}, {"ORTH": "Now"}]], + "Java": [[{"LOWER": {"FUZZY": "java"}}]], + }, + [(2, 4), (5, 6), (8, 9)], + ), + # only the second pattern matches (check that predicate keys used for + # caching don't collide) + ( + { + "A": [[{"ORTH": {"FUZZY": "Javascripts"}}]], + "B": [[{"ORTH": {"FUZZY5": "Javascripts"}}]], + }, + [(8, 9)], + ), + ], +) +def test_matcher_match_fuzzy(en_vocab, rules, match_locs): + words = ["They", "like", "Goggle", "Now", "and", "Jav", "but", "not", "JvvaScrpt"] + doc = Doc(en_vocab, words=words) + + matcher = Matcher(en_vocab) + for key, patterns in rules.items(): + matcher.add(key, patterns) + assert match_locs == [(start, end) for m_id, start, end in matcher(doc)] + + +@pytest.mark.parametrize("set_op", ["IN", "NOT_IN"]) +def test_matcher_match_fuzzy_set_op_longest(en_vocab, set_op): + rules = { + "GoogleNow": [[{"ORTH": {"FUZZY": {set_op: ["Google", "Now"]}}, "OP": "+"}]] + } + matcher = Matcher(en_vocab) + for key, patterns in rules.items(): + matcher.add(key, patterns, greedy="LONGEST") + + words = ["They", "like", "Goggle", "Noo"] + doc = Doc(en_vocab, words=words) + assert len(matcher(doc)) == 1 + + +def test_matcher_match_fuzzy_set_multiple(en_vocab): + rules = { + "GoogleNow": [ + [ + { + "ORTH": {"FUZZY": {"IN": ["Google", "Now"]}, "NOT_IN": ["Goggle"]}, + "OP": "+", + } + ] + ] + } + matcher = Matcher(en_vocab) + for key, patterns in rules.items(): + matcher.add(key, patterns, greedy="LONGEST") + + words = ["They", "like", "Goggle", "Noo"] + doc = Doc(matcher.vocab, words=words) + assert matcher(doc) == [ + (doc.vocab.strings["GoogleNow"], 3, 4), + ] + + +@pytest.mark.parametrize("fuzzyn", range(1, 10)) +def test_matcher_match_fuzzyn_all_insertions(en_vocab, fuzzyn): + matcher = Matcher(en_vocab) + matcher.add("GoogleNow", [[{"ORTH": {f"FUZZY{fuzzyn}": "GoogleNow"}}]]) + # words with increasing edit distance + words = ["GoogleNow" + "a" * i for i in range(0, 10)] + doc = Doc(en_vocab, words) + assert len(matcher(doc)) == fuzzyn + 1 + + +@pytest.mark.parametrize("fuzzyn", range(1, 6)) +def test_matcher_match_fuzzyn_various_edits(en_vocab, fuzzyn): + matcher = Matcher(en_vocab) + matcher.add("GoogleNow", [[{"ORTH": {f"FUZZY{fuzzyn}": "GoogleNow"}}]]) + # words with increasing edit distance of different edit types + words = [ + "GoogleNow", + "GoogleNuw", + "GoogleNuew", + "GoogleNoweee", + "GiggleNuw3", + "gouggle5New", + ] + doc = Doc(en_vocab, words) + assert len(matcher(doc)) == fuzzyn + 1 + + +@pytest.mark.parametrize("greedy", ["FIRST", "LONGEST"]) +@pytest.mark.parametrize("set_op", ["IN", "NOT_IN"]) +def test_matcher_match_fuzzyn_set_op_longest(en_vocab, greedy, set_op): + rules = { + "GoogleNow": [[{"ORTH": {"FUZZY2": {set_op: ["Google", "Now"]}}, "OP": "+"}]] + } + matcher = Matcher(en_vocab) + for key, patterns in rules.items(): + matcher.add(key, patterns, greedy=greedy) + + words = ["They", "like", "Goggle", "Noo"] + doc = Doc(matcher.vocab, words=words) + spans = matcher(doc, as_spans=True) + assert len(spans) == 1 + if set_op == "IN": + assert spans[0].text == "Goggle Noo" + else: + assert spans[0].text == "They like" + + +def test_matcher_match_fuzzyn_set_multiple(en_vocab): + rules = { + "GoogleNow": [ + [ + { + "ORTH": {"FUZZY1": {"IN": ["Google", "Now"]}, "NOT_IN": ["Goggle"]}, + "OP": "+", + } + ] + ] + } + matcher = Matcher(en_vocab) + for key, patterns in rules.items(): + matcher.add(key, patterns, greedy="LONGEST") + + words = ["They", "like", "Goggle", "Noo"] + doc = Doc(matcher.vocab, words=words) + assert matcher(doc) == [ + (doc.vocab.strings["GoogleNow"], 3, 4), + ] + + def test_matcher_empty_dict(en_vocab): """Test matcher allows empty token specs, meaning match on any token.""" matcher = Matcher(en_vocab) @@ -437,6 +586,30 @@ def test_matcher_regex(en_vocab): assert len(matches) == 0 +def test_matcher_regex_set_in(en_vocab): + matcher = Matcher(en_vocab) + pattern = [{"ORTH": {"REGEX": {"IN": [r"(?:a)", r"(?:an)"]}}}] + matcher.add("A_OR_AN", [pattern]) + doc = Doc(en_vocab, words=["an", "a", "hi"]) + matches = matcher(doc) + assert len(matches) == 2 + doc = Doc(en_vocab, words=["bye"]) + matches = matcher(doc) + assert len(matches) == 0 + + +def test_matcher_regex_set_not_in(en_vocab): + matcher = Matcher(en_vocab) + pattern = [{"ORTH": {"REGEX": {"NOT_IN": [r"(?:a)", r"(?:an)"]}}}] + matcher.add("A_OR_AN", [pattern]) + doc = Doc(en_vocab, words=["an", "a", "hi"]) + matches = matcher(doc) + assert len(matches) == 1 + doc = Doc(en_vocab, words=["bye"]) + matches = matcher(doc) + assert len(matches) == 1 + + def test_matcher_regex_shape(en_vocab): matcher = Matcher(en_vocab) pattern = [{"SHAPE": {"REGEX": r"^[^x]+$"}}] diff --git a/spacy/tests/pipeline/test_entity_ruler.py b/spacy/tests/pipeline/test_entity_ruler.py index 6851e2a7c..417f930cb 100644 --- a/spacy/tests/pipeline/test_entity_ruler.py +++ b/spacy/tests/pipeline/test_entity_ruler.py @@ -382,6 +382,43 @@ def test_entity_ruler_overlapping_spans(nlp, entity_ruler_factory): assert doc.ents[0].label_ == "FOOBAR" +@pytest.mark.parametrize("entity_ruler_factory", ENTITY_RULERS) +def test_entity_ruler_fuzzy_pipe(nlp, entity_ruler_factory): + ruler = nlp.add_pipe(entity_ruler_factory, name="entity_ruler") + patterns = [{"label": "HELLO", "pattern": [{"LOWER": {"FUZZY": "hello"}}]}] + ruler.add_patterns(patterns) + doc = nlp("helloo") + assert len(doc.ents) == 1 + assert doc.ents[0].label_ == "HELLO" + + +@pytest.mark.parametrize("entity_ruler_factory", ENTITY_RULERS) +def test_entity_ruler_fuzzy(nlp, entity_ruler_factory): + ruler = nlp.add_pipe(entity_ruler_factory, name="entity_ruler") + patterns = [{"label": "HELLO", "pattern": [{"LOWER": {"FUZZY": "hello"}}]}] + ruler.add_patterns(patterns) + doc = nlp("helloo") + assert len(doc.ents) == 1 + assert doc.ents[0].label_ == "HELLO" + + +@pytest.mark.parametrize("entity_ruler_factory", ENTITY_RULERS) +def test_entity_ruler_fuzzy_disabled(nlp, entity_ruler_factory): + @registry.misc("test_fuzzy_compare_disabled") + def make_test_fuzzy_compare_disabled(): + return lambda x, y, z: False + + ruler = nlp.add_pipe( + entity_ruler_factory, + name="entity_ruler", + config={"matcher_fuzzy_compare": {"@misc": "test_fuzzy_compare_disabled"}}, + ) + patterns = [{"label": "HELLO", "pattern": [{"LOWER": {"FUZZY": "hello"}}]}] + ruler.add_patterns(patterns) + doc = nlp("helloo") + assert len(doc.ents) == 0 + + @pytest.mark.parametrize("n_process", [1, 2]) @pytest.mark.parametrize("entity_ruler_factory", ENTITY_RULERS) def test_entity_ruler_multiprocessing(nlp, n_process, entity_ruler_factory): diff --git a/website/docs/api/entityruler.md b/website/docs/api/entityruler.md index c2ba33f01..f15c648ff 100644 --- a/website/docs/api/entityruler.md +++ b/website/docs/api/entityruler.md @@ -55,13 +55,14 @@ how the component should be configured. You can override its settings via the > nlp.add_pipe("entity_ruler", config=config) > ``` -| Setting | Description | -| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `phrase_matcher_attr` | Optional attribute name match on for the internal [`PhraseMatcher`](/api/phrasematcher), e.g. `LOWER` to match on the lowercase token text. Defaults to `None`. ~~Optional[Union[int, str]]~~ | -| `validate` | Whether patterns should be validated (passed to the `Matcher` and `PhraseMatcher`). Defaults to `False`. ~~bool~~ | -| `overwrite_ents` | If existing entities are present, e.g. entities added by the model, overwrite them by matches if necessary. Defaults to `False`. ~~bool~~ | -| `ent_id_sep` | Separator used internally for entity IDs. Defaults to `"\|\|"`. ~~str~~ | -| `scorer` | The scoring method. Defaults to [`spacy.scorer.get_ner_prf`](/api/scorer#get_ner_prf). ~~Optional[Callable]~~ | +| Setting | Description | +| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `phrase_matcher_attr` | Optional attribute name match on for the internal [`PhraseMatcher`](/api/phrasematcher), e.g. `LOWER` to match on the lowercase token text. Defaults to `None`. ~~Optional[Union[int, str]]~~ | +| `matcher_fuzzy_compare` 3.5 | The fuzzy comparison method, passed on to the internal `Matcher`. Defaults to `spacy.matcher.levenshtein.levenshtein_compare`. ~~Callable~~ | +| `validate` | Whether patterns should be validated (passed to the `Matcher` and `PhraseMatcher`). Defaults to `False`. ~~bool~~ | +| `overwrite_ents` | If existing entities are present, e.g. entities added by the model, overwrite them by matches if necessary. Defaults to `False`. ~~bool~~ | +| `ent_id_sep` | Separator used internally for entity IDs. Defaults to `"\|\|"`. ~~str~~ | +| `scorer` | The scoring method. Defaults to [`spacy.scorer.get_ner_prf`](/api/scorer#get_ner_prf). ~~Optional[Callable]~~ | ```python %%GITHUB_SPACY/spacy/pipeline/entityruler.py @@ -85,23 +86,25 @@ be a token pattern (list) or a phrase pattern (string). For example: > ruler = EntityRuler(nlp, overwrite_ents=True) > ``` -| Name | Description | -| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `nlp` | The shared nlp object to pass the vocab to the matchers and process phrase patterns. ~~Language~~ | -| `name` 3 | Instance name of the current pipeline component. Typically passed in automatically from the factory when the component is added. Used to disable the current entity ruler while creating phrase patterns with the nlp object. ~~str~~ | -| _keyword-only_ | | -| `phrase_matcher_attr` | Optional attribute name match on for the internal [`PhraseMatcher`](/api/phrasematcher), e.g. `LOWER` to match on the lowercase token text. Defaults to `None`. ~~Optional[Union[int, str]]~~ | -| `validate` | Whether patterns should be validated, passed to Matcher and PhraseMatcher as `validate`. Defaults to `False`. ~~bool~~ | -| `overwrite_ents` | If existing entities are present, e.g. entities added by the model, overwrite them by matches if necessary. Defaults to `False`. ~~bool~~ | -| `ent_id_sep` | Separator used internally for entity IDs. Defaults to `"\|\|"`. ~~str~~ | -| `patterns` | Optional patterns to load in on initialization. ~~Optional[List[Dict[str, Union[str, List[dict]]]]]~~ | +| Name | Description | +| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `nlp` | The shared nlp object to pass the vocab to the matchers and process phrase patterns. ~~Language~~ | +| `name` 3 | Instance name of the current pipeline component. Typically passed in automatically from the factory when the component is added. Used to disable the current entity ruler while creating phrase patterns with the nlp object. ~~str~~ | +| _keyword-only_ | | +| `phrase_matcher_attr` | Optional attribute name match on for the internal [`PhraseMatcher`](/api/phrasematcher), e.g. `LOWER` to match on the lowercase token text. Defaults to `None`. ~~Optional[Union[int, str]]~~ | +| `matcher_fuzzy_compare` 3.5 | The fuzzy comparison method, passed on to the internal `Matcher`. Defaults to `spacy.matcher.levenshtein.levenshtein_compare`. ~~Callable~~ | +| `validate` | Whether patterns should be validated, passed to Matcher and PhraseMatcher as `validate`. Defaults to `False`. ~~bool~~ | +| `overwrite_ents` | If existing entities are present, e.g. entities added by the model, overwrite them by matches if necessary. Defaults to `False`. ~~bool~~ | +| `ent_id_sep` | Separator used internally for entity IDs. Defaults to `"\|\|"`. ~~str~~ | +| `patterns` | Optional patterns to load in on initialization. ~~Optional[List[Dict[str, Union[str, List[dict]]]]]~~ | +| `scorer` | The scoring method. Defaults to [`spacy.scorer.get_ner_prf`](/api/scorer#get_ner_prf). ~~Optional[Callable]~~ | ## EntityRuler.initialize {#initialize tag="method" new="3"} Initialize the component with data and used before training to load in rules -from a [pattern file](/usage/rule-based-matching/#entityruler-files). This method -is typically called by [`Language.initialize`](/api/language#initialize) and -lets you customize arguments it receives via the +from a [pattern file](/usage/rule-based-matching/#entityruler-files). This +method is typically called by [`Language.initialize`](/api/language#initialize) +and lets you customize arguments it receives via the [`[initialize.components]`](/api/data-formats#config-initialize) block in the config. @@ -210,10 +213,10 @@ of dicts) or a phrase pattern (string). For more details, see the usage guide on | ---------- | ---------------------------------------------------------------- | | `patterns` | The patterns to add. ~~List[Dict[str, Union[str, List[dict]]]]~~ | - ## EntityRuler.remove {#remove tag="method" new="3.2.1"} -Remove a pattern by its ID from the entity ruler. A `ValueError` is raised if the ID does not exist. +Remove a pattern by its ID from the entity ruler. A `ValueError` is raised if +the ID does not exist. > #### Example > @@ -224,9 +227,9 @@ Remove a pattern by its ID from the entity ruler. A `ValueError` is raised if th > ruler.remove("apple") > ``` -| Name | Description | -| ---------- | ---------------------------------------------------------------- | -| `id` | The ID of the pattern rule. ~~str~~ | +| Name | Description | +| ---- | ----------------------------------- | +| `id` | The ID of the pattern rule. ~~str~~ | ## EntityRuler.to_disk {#to_disk tag="method"} diff --git a/website/docs/api/matcher.md b/website/docs/api/matcher.md index cd7bfa070..bd5f6ac24 100644 --- a/website/docs/api/matcher.md +++ b/website/docs/api/matcher.md @@ -86,14 +86,20 @@ it compares to another value. > ] > ``` -| Attribute | Description | -| -------------------------- | -------------------------------------------------------------------------------------------------------- | -| `IN` | Attribute value is member of a list. ~~Any~~ | -| `NOT_IN` | Attribute value is _not_ member of a list. ~~Any~~ | -| `IS_SUBSET` | Attribute value (for `MORPH` or custom list attributes) is a subset of a list. ~~Any~~ | -| `IS_SUPERSET` | Attribute value (for `MORPH` or custom list attributes) is a superset of a list. ~~Any~~ | -| `INTERSECTS` | Attribute value (for `MORPH` or custom list attribute) has a non-empty intersection with a list. ~~Any~~ | -| `==`, `>=`, `<=`, `>`, `<` | Attribute value is equal, greater or equal, smaller or equal, greater or smaller. ~~Union[int, float]~~ | +| Attribute | Description | +| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `REGEX` | Attribute value matches the regular expression at any position in the string. ~~Any~~ | +| `FUZZY` | Attribute value matches if the `fuzzy_compare` method matches for `(value, pattern, -1)`. The default method allows a Levenshtein edit distance of at least 2 and up to 30% of the pattern string length. ~~Any~~ | +| `FUZZY1`, `FUZZY2`, ... `FUZZY9` | Attribute value matches if the `fuzzy_compare` method matches for `(value, pattern, N)`. The default method allows a Levenshtein edit distance of at most N (1-9). ~~Any~~ | +| `IN` | Attribute value is member of a list. ~~Any~~ | +| `NOT_IN` | Attribute value is _not_ member of a list. ~~Any~~ | +| `IS_SUBSET` | Attribute value (for `MORPH` or custom list attributes) is a subset of a list. ~~Any~~ | +| `IS_SUPERSET` | Attribute value (for `MORPH` or custom list attributes) is a superset of a list. ~~Any~~ | +| `INTERSECTS` | Attribute value (for `MORPH` or custom list attribute) has a non-empty intersection with a list. ~~Any~~ | +| `==`, `>=`, `<=`, `>`, `<` | Attribute value is equal, greater or equal, smaller or equal, greater or smaller. ~~Union[int, float]~~ | + +As of spaCy v3.5, `REGEX` and `FUZZY` can be used in combination with `IN` and +`NOT_IN`. ## Matcher.\_\_init\_\_ {#init tag="method"} @@ -109,10 +115,11 @@ string where an integer is expected) or unexpected property names. > matcher = Matcher(nlp.vocab) > ``` -| Name | Description | -| ---------- | ----------------------------------------------------------------------------------------------------- | -| `vocab` | The vocabulary object, which must be shared with the documents the matcher will operate on. ~~Vocab~~ | -| `validate` | Validate all patterns added to this matcher. ~~bool~~ | +| Name | Description | +| --------------- | ----------------------------------------------------------------------------------------------------- | +| `vocab` | The vocabulary object, which must be shared with the documents the matcher will operate on. ~~Vocab~~ | +| `validate` | Validate all patterns added to this matcher. ~~bool~~ | +| `fuzzy_compare` | The comparison method used for the `FUZZY` operators. ~~Callable[[str, str, int], bool]~~ | ## Matcher.\_\_call\_\_ {#call tag="method"} diff --git a/website/docs/api/spanruler.md b/website/docs/api/spanruler.md index b573f7c58..31f04ccf9 100644 --- a/website/docs/api/spanruler.md +++ b/website/docs/api/spanruler.md @@ -46,16 +46,17 @@ how the component should be configured. You can override its settings via the > nlp.add_pipe("span_ruler", config=config) > ``` -| Setting | Description | -| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `spans_key` | The spans key to save the spans under. If `None`, no spans are saved. Defaults to `"ruler"`. ~~Optional[str]~~ | -| `spans_filter` | The optional method to filter spans before they are assigned to doc.spans. Defaults to `None`. ~~Optional[Callable[[Iterable[Span], Iterable[Span]], List[Span]]]~~ | -| `annotate_ents` | Whether to save spans to doc.ents. Defaults to `False`. ~~bool~~ | -| `ents_filter` | The method to filter spans before they are assigned to doc.ents. Defaults to `util.filter_chain_spans`. ~~Callable[[Iterable[Span], Iterable[Span]], List[Span]]~~ | -| `phrase_matcher_attr` | Token attribute to match on, passed to the internal PhraseMatcher as `attr`. Defaults to `None`. ~~Optional[Union[int, str]]~~ | -| `validate` | Whether patterns should be validated, passed to Matcher and PhraseMatcher as `validate`. Defaults to `False`. ~~bool~~ | -| `overwrite` | Whether to remove any existing spans under `Doc.spans[spans key]` if `spans_key` is set, or to remove any ents under `Doc.ents` if `annotate_ents` is set. Defaults to `True`. ~~bool~~ | -| `scorer` | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for `Doc.spans[spans_key]` with overlapping spans allowed. ~~Optional[Callable]~~ | +| Setting | Description | +| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `spans_key` | The spans key to save the spans under. If `None`, no spans are saved. Defaults to `"ruler"`. ~~Optional[str]~~ | +| `spans_filter` | The optional method to filter spans before they are assigned to doc.spans. Defaults to `None`. ~~Optional[Callable[[Iterable[Span], Iterable[Span]], List[Span]]]~~ | +| `annotate_ents` | Whether to save spans to doc.ents. Defaults to `False`. ~~bool~~ | +| `ents_filter` | The method to filter spans before they are assigned to doc.ents. Defaults to `util.filter_chain_spans`. ~~Callable[[Iterable[Span], Iterable[Span]], List[Span]]~~ | +| `phrase_matcher_attr` | Token attribute to match on, passed to the internal `PhraseMatcher` as `attr`. Defaults to `None`. ~~Optional[Union[int, str]]~~ | +| `matcher_fuzzy_compare` 3.5 | The fuzzy comparison method, passed on to the internal `Matcher`. Defaults to `spacy.matcher.levenshtein.levenshtein_compare`. ~~Callable~~ | +| `validate` | Whether patterns should be validated, passed to `Matcher` and `PhraseMatcher` as `validate`. Defaults to `False`. ~~bool~~ | +| `overwrite` | Whether to remove any existing spans under `Doc.spans[spans key]` if `spans_key` is set, or to remove any ents under `Doc.ents` if `annotate_ents` is set. Defaults to `True`. ~~bool~~ | +| `scorer` | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for `Doc.spans[spans_key]` with overlapping spans allowed. ~~Optional[Callable]~~ | ```python %%GITHUB_SPACY/spacy/pipeline/span_ruler.py @@ -79,19 +80,20 @@ token pattern (list) or a phrase pattern (string). For example: > ruler = SpanRuler(nlp, overwrite=True) > ``` -| Name | Description | -| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `nlp` | The shared nlp object to pass the vocab to the matchers and process phrase patterns. ~~Language~~ | -| `name` | Instance name of the current pipeline component. Typically passed in automatically from the factory when the component is added. Used to disable the current span ruler while creating phrase patterns with the nlp object. ~~str~~ | -| _keyword-only_ | | -| `spans_key` | The spans key to save the spans under. If `None`, no spans are saved. Defaults to `"ruler"`. ~~Optional[str]~~ | -| `spans_filter` | The optional method to filter spans before they are assigned to doc.spans. Defaults to `None`. ~~Optional[Callable[[Iterable[Span], Iterable[Span]], List[Span]]]~~ | -| `annotate_ents` | Whether to save spans to doc.ents. Defaults to `False`. ~~bool~~ | -| `ents_filter` | The method to filter spans before they are assigned to doc.ents. Defaults to `util.filter_chain_spans`. ~~Callable[[Iterable[Span], Iterable[Span]], List[Span]]~~ | -| `phrase_matcher_attr` | Token attribute to match on, passed to the internal PhraseMatcher as `attr`. Defaults to `None`. ~~Optional[Union[int, str]]~~ | -| `validate` | Whether patterns should be validated, passed to Matcher and PhraseMatcher as `validate`. Defaults to `False`. ~~bool~~ | -| `overwrite` | Whether to remove any existing spans under `Doc.spans[spans key]` if `spans_key` is set, or to remove any ents under `Doc.ents` if `annotate_ents` is set. Defaults to `True`. ~~bool~~ | -| `scorer` | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for `Doc.spans[spans_key]` with overlapping spans allowed. ~~Optional[Callable]~~ | +| Name | Description | +| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `nlp` | The shared nlp object to pass the vocab to the matchers and process phrase patterns. ~~Language~~ | +| `name` | Instance name of the current pipeline component. Typically passed in automatically from the factory when the component is added. Used to disable the current span ruler while creating phrase patterns with the nlp object. ~~str~~ | +| _keyword-only_ | | +| `spans_key` | The spans key to save the spans under. If `None`, no spans are saved. Defaults to `"ruler"`. ~~Optional[str]~~ | +| `spans_filter` | The optional method to filter spans before they are assigned to doc.spans. Defaults to `None`. ~~Optional[Callable[[Iterable[Span], Iterable[Span]], List[Span]]]~~ | +| `annotate_ents` | Whether to save spans to doc.ents. Defaults to `False`. ~~bool~~ | +| `ents_filter` | The method to filter spans before they are assigned to doc.ents. Defaults to `util.filter_chain_spans`. ~~Callable[[Iterable[Span], Iterable[Span]], List[Span]]~~ | +| `phrase_matcher_attr` | Token attribute to match on, passed to the internal PhraseMatcher as `attr`. Defaults to `None`. ~~Optional[Union[int, str]]~~ | +| `matcher_fuzzy_compare` 3.5 | The fuzzy comparison method, passed on to the internal `Matcher`. Defaults to `spacy.matcher.levenshtein.levenshtein_compare`. ~~Callable~~ | +| `validate` | Whether patterns should be validated, passed to Matcher and PhraseMatcher as `validate`. Defaults to `False`. ~~bool~~ | +| `overwrite` | Whether to remove any existing spans under `Doc.spans[spans key]` if `spans_key` is set, or to remove any ents under `Doc.ents` if `annotate_ents` is set. Defaults to `True`. ~~bool~~ | +| `scorer` | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for `Doc.spans[spans_key]` with overlapping spans allowed. ~~Optional[Callable]~~ | ## SpanRuler.initialize {#initialize tag="method"} diff --git a/website/docs/usage/rule-based-matching.md b/website/docs/usage/rule-based-matching.md index ad8ea27f3..3e15fca36 100644 --- a/website/docs/usage/rule-based-matching.md +++ b/website/docs/usage/rule-based-matching.md @@ -364,6 +364,46 @@ else: +#### Fuzzy matching {#fuzzy new="3.5"} + +Fuzzy matching allows you to match tokens with alternate spellings, typos, etc. +without specifying every possible variant. + +```python +# Matches "favourite", "favorites", "gavorite", "theatre", "theatr", ... +pattern = [{"TEXT": {"FUZZY": "favorite"}}, + {"TEXT": {"FUZZY": "theater"}}] +``` + +The `FUZZY` attribute allows fuzzy matches for any attribute string value, +including custom attributes. Just like `REGEX`, it always needs to be applied to +an attribute like `TEXT` or `LOWER`. By default `FUZZY` allows a Levenshtein +edit distance of at least 2 and up to 30% of the pattern string length. Using +the more specific attributes `FUZZY1`..`FUZZY9` you can specify the maximum +allowed edit distance directly. + +```python +# Match lowercase with fuzzy matching (allows 2 edits) +pattern = [{"LOWER": {"FUZZY": "definitely"}}] + +# Match custom attribute values with fuzzy matching (allows 2 edits) +pattern = [{"_": {"country": {"FUZZY": "Kyrgyzstan"}}}] + +# Match with exact Levenshtein edit distance limits (allows 3 edits) +pattern = [{"_": {"country": {"FUZZY3": "Kyrgyzstan"}}}] +``` + +#### Regex and fuzzy matching with lists {#regex-fuzzy-lists new="3.5"} + +Starting in spaCy v3.5, both `REGEX` and `FUZZY` can be combined with the +attributes `IN` and `NOT_IN`: + +```python +pattern = [{"TEXT": {"FUZZY": {"IN": ["awesome", "cool", "wonderful"]}}}] + +pattern = [{"TEXT": {"REGEX": {"NOT_IN": ["^awe(some)?$", "^wonder(ful)?"]}}}] +``` + --- #### Operators and quantifiers {#quantifiers} From 9e0322de1abfb21c4d87d1e58a9ef886f5e20603 Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Wed, 11 Jan 2023 08:01:47 +0100 Subject: [PATCH 39/44] Restore v2 token_acc score implementation (#12073) In the v3 scorer refactoring, `token_acc` was implemented incorrectly. It should use `precision` instead of `fscore` for the measure of correctly aligned tokens / number of predicted tokens. Fix the docs to reflect that the measure uses the number of predicted tokens rather than the number of gold tokens. --- spacy/scorer.py | 2 +- spacy/tests/test_scorer.py | 2 +- website/docs/api/scorer.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/spacy/scorer.py b/spacy/scorer.py index d8c383ab8..de4f52be6 100644 --- a/spacy/scorer.py +++ b/spacy/scorer.py @@ -174,7 +174,7 @@ class Scorer: prf_score.score_set(pred_spans, gold_spans) if len(acc_score) > 0: return { - "token_acc": acc_score.fscore, + "token_acc": acc_score.precision, "token_p": prf_score.precision, "token_r": prf_score.recall, "token_f": prf_score.fscore, diff --git a/spacy/tests/test_scorer.py b/spacy/tests/test_scorer.py index b903f1669..dbb47b423 100644 --- a/spacy/tests/test_scorer.py +++ b/spacy/tests/test_scorer.py @@ -110,7 +110,7 @@ def test_tokenization(sented_doc): ) example.predicted[1].is_sent_start = False scores = scorer.score([example]) - assert scores["token_acc"] == approx(0.66666666) + assert scores["token_acc"] == 0.5 assert scores["token_p"] == 0.5 assert scores["token_r"] == approx(0.33333333) assert scores["token_f"] == 0.4 diff --git a/website/docs/api/scorer.md b/website/docs/api/scorer.md index 9ef36e6fc..86e61da1e 100644 --- a/website/docs/api/scorer.md +++ b/website/docs/api/scorer.md @@ -76,7 +76,7 @@ core pipeline components, the individual score names start with the `Token` or Scores the tokenization: -- `token_acc`: number of correct tokens / number of gold tokens +- `token_acc`: number of correct tokens / number of predicted tokens - `token_p`, `token_r`, `token_f`: precision, recall and F-score for token character spans From e0168ccce940251351711ac0196d8560cb77547e Mon Sep 17 00:00:00 2001 From: Adriane Boyd Date: Wed, 11 Jan 2023 13:54:58 +0100 Subject: [PATCH 40/44] Allow spacy-transformers v1.2.x in transformers extra (#12092) --- setup.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.cfg b/setup.cfg index cee8c0c33..79dff9e30 100644 --- a/setup.cfg +++ b/setup.cfg @@ -74,7 +74,7 @@ console_scripts = lookups = spacy_lookups_data>=1.0.3,<1.1.0 transformers = - spacy_transformers>=1.1.2,<1.2.0 + spacy_transformers>=1.1.2,<1.3.0 ray = spacy_ray>=0.1.0,<1.0.0 cuda = From 554df9ef20184bf439495873a7454ec8f28cf94e Mon Sep 17 00:00:00 2001 From: Sofie Van Landeghem Date: Wed, 11 Jan 2023 17:30:07 +0100 Subject: [PATCH 41/44] Website migration from Gatsby to Next (#12058) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Rename all MDX file to `.mdx` * Lock current node version (#11885) * Apply Prettier (#11996) * Minor website fixes (#11974) [ci skip] * fix table * Migrate to Next WEB-17 (#12005) * Initial commit * Run `npx create-next-app@13 next-blog` * Install MDX packages Following: https://github.com/vercel/next.js/blob/77b5f79a4dff453abb62346bf75b14d859539b81/packages/next-mdx/readme.md * Add MDX to Next * Allow Next to handle `.md` and `.mdx` files. * Add VSCode extension recommendation * Disabled TypeScript strict mode for now * Add prettier * Apply Prettier to all files * Make sure to use correct Node version * Add basic implementation for `MDXRemote` * Add experimental Rust MDX parser * Add `/public` * Add SASS support * Remove default pages and styling * Convert to module This allows to use `import/export` syntax * Add import for custom components * Add ability to load plugins * Extract function This will make the next commit easier to read * Allow to handle directories for page creation * Refactoring * Allow to parse subfolders for pages * Extract logic * Redirect `index.mdx` to parent directory * Disabled ESLint during builds * Disabled typescript during build * Remove Gatsby from `README.md` * Rephrase Docker part of `README.md` * Update project structure in `README.md` * Move and rename plugins * Update plugin for wrapping sections * Add dependencies for plugin * Use plugin * Rename wrapper type * Simplify unnessary adding of id to sections The slugified section ids are useless, because they can not be referenced anywhere anyway. The navigation only works if the section has the same id as the heading. * Add plugin for custom attributes on Markdown elements * Add plugin to readd support for tables * Add plugin to fix problem with wrapped images For more details see this issue: https://github.com/mdx-js/mdx/issues/1798 * Add necessary meta data to pages * Install necessary dependencies * Remove outdated MDX handling * Remove reliance on `InlineList` * Use existing Remark components * Remove unallowed heading Before `h1` components where not overwritten and would never have worked and they aren't used anywhere either. * Add missing components to MDX * Add correct styling * Fix broken list * Fix broken CSS classes * Implement layout * Fix links * Fix broken images * Fix pattern image * Fix heading attributes * Rename heading attribute `new` was causing some weird issue, so renaming it to `version` * Update comment syntax in MDX * Merge imports * Fix markdown rendering inside components * Add model pages * Simplify anchors * Fix default value for theme * Add Universe index page * Add Universe categories * Add Universe projects * Fix Next problem with copy Next complains when the server renders something different then the client, therfor we move the differing logic to `useEffect` * Fix improper component nesting Next doesn't allow block elements inside a `

` * Replace landing page MDX with page component * Remove inlined iframe content * Remove ability to inline HTML content in iFrames * Remove MDX imports * Fix problem with image inside link in MDX * Escape character for MDX * Fix unescaped characters in MDX * Fix headings with logo * Allow to export static HTML pages * Add prebuild script This command is automatically run by Next * Replace `svg-loader` with `react-inlinesvg` `svg-loader` is no longer maintained * Fix ESLint `react-hooks/exhaustive-deps` * Fix dropdowns * Change code language from `cli` to `bash` * Remove unnessary language `none` * Fix invalid code language `markdown_` with an underscore was used to basically turn of syntax highlighting, but using unknown languages know throws an error. * Enable code blocks plugin * Readd `InlineCode` component MDX2 removed the `inlineCode` component > The special component name `inlineCode` was removed, we recommend to use `pre` for the block version of code, and code for both the block and inline versions Source: https://mdxjs.com/migrating/v2/#update-mdx-content * Remove unused code * Extract function to own file * Fix code syntax highlighting * Update syntax for code block meta data * Remove unused prop * Fix internal link recognition There is a problem with regex between Node and browser, and since Next runs the component on both, this create an error. `Prop `rel` did not match. Server: "null" Client: "noopener nofollow noreferrer"` This simplifies the implementation and fixes the above error. * Replace `react-helmet` with `next/head` * Fix `className` problem for JSX component * Fix broken bold markdown * Convert file to `.mjs` to be used by Node process * Add plugin to replace strings * Fix custom table row styling * Fix problem with `span` inside inline `code` React doesn't allow a `span` inside an inline `code` element and throws an error in dev mode. * Add `_document` to be able to customize `` and `` * Add `lang="en"` * Store Netlify settings in file This way we don't need to update via Netlify UI, which can be tricky if changing build settings. * Add sitemap * Add Smartypants * Add PWA support * Add `manifest.webmanifest` * Fix bug with anchor links after reloading There was no need for the previous implementation, since the browser handles this nativly. Additional the manual scrolling into view was actually broken, because the heading would disappear behind the menu bar. * Rename custom event I was googeling for ages to find out what kind of event `inview` is, only to figure out it was a custom event with a name that sounds pretty much like a native one. 🫠 * Fix missing comment syntax highlighting * Refactor Quickstart component The previous implementation was hidding the irrelevant lines via data-props and dynamically generated CSS. This created problems with Next and was also hard to follow. CSS was used to do what React is supposed to handle. The new implementation simplfy filters the list of children (React elements) via their props. * Fix syntax highlighting for Training Quickstart * Unify code rendering * Improve error logging in Juniper * Fix Juniper component * Automatically generate "Read Next" link * Add Plausible * Use recent DocSearch component and adjust styling * Fix images * Turn of image optimization > Image Optimization using Next.js' default loader is not compatible with `next export`. We currently deploy to Netlify via `next export` * Dont build pages starting with `_` * Remove unused files * Add Next plugin to Netlify * Fix button layout MDX automatically adds `p` tags around text on a new line and Prettier wants to put the text on a new line. Hacking with JSX string. * Add 404 page * Apply Prettier * Update Prettier for `package.json` Next sometimes wants to patch `package-lock.json`. The old Prettier setting indended with 4 spaces, but Next always indends with 2 spaces. Since `npm install` automatically uses the indendation from `package.json` for `package-lock.json` and to avoid the format switching back and forth, both files are now set to 2 spaces. * Apply Next patch to `package-lock.json` When starting the dev server Next would warn `warn - Found lockfile missing swc dependencies, patching...` and update the `package-lock.json`. These are the patched changes. * fix link Co-authored-by: Sofie Van Landeghem * small backslash fixes * adjust to new style Co-authored-by: Marcus Blättermann --- website/.eslintrc.json | 3 + website/.gitignore | 44 + website/.nvmrc | 1 + website/.prettierignore | 1 + website/.prettierrc | 5 +- website/.vscode/extensions.json | 8 + website/README.md | 28 +- website/UNIVERSE.md | 78 +- .../{architectures.md => architectures.mdx} | 78 +- .../{attributeruler.md => attributeruler.mdx} | 30 +- .../api/{attributes.md => attributes.mdx} | 7 +- website/docs/api/{cli.md => cli.mdx} | 177 +- website/docs/api/{coref.md => coref.mdx} | 32 +- website/docs/api/{corpus.md => corpus.mdx} | 17 +- .../{cython-classes.md => cython-classes.mdx} | 32 +- .../{cython-structs.md => cython-structs.mdx} | 22 +- website/docs/api/{cython.md => cython.mdx} | 4 +- .../api/{data-formats.md => data-formats.mdx} | 44 +- ...ndencymatcher.md => dependencymatcher.mdx} | 21 +- ...pendencyparser.md => dependencyparser.mdx} | 42 +- website/docs/api/{doc.md => doc.mdx} | 72 +- website/docs/api/{docbin.md => docbin.mdx} | 23 +- ...eelemmatizer.md => edittreelemmatizer.mdx} | 40 +- .../api/{entitylinker.md => entitylinker.mdx} | 36 +- ...tityrecognizer.md => entityrecognizer.mdx} | 42 +- .../api/{entityruler.md => entityruler.mdx} | 36 +- website/docs/api/{example.md => example.mdx} | 40 +- website/docs/api/{index.md => index.mdx} | 2 - website/docs/api/{kb.md => kb.mdx} | 28 +- .../api/{kb_in_memory.md => kb_in_memory.mdx} | 36 +- .../docs/api/{language.md => language.mdx} | 88 +- website/docs/api/{legacy.md => legacy.mdx} | 32 +- .../api/{lemmatizer.md => lemmatizer.mdx} | 32 +- website/docs/api/{lexeme.md => lexeme.mdx} | 16 +- website/docs/api/{lookups.md => lookups.mdx} | 40 +- website/docs/api/{matcher.md => matcher.mdx} | 16 +- .../{morphologizer.md => morphologizer.mdx} | 42 +- .../api/{morphology.md => morphology.mdx} | 40 +- .../{phrasematcher.md => phrasematcher.mdx} | 14 +- website/docs/api/{pipe.md => pipe.mdx} | 50 +- ...ne-functions.md => pipeline-functions.mdx} | 12 +- website/docs/api/{scorer.md => scorer.mdx} | 24 +- ...cerecognizer.md => sentencerecognizer.mdx} | 38 +- .../api/{sentencizer.md => sentencizer.mdx} | 23 +- .../{span-resolver.md => span-resolver.mdx} | 32 +- website/docs/api/{span.md => span.mdx} | 56 +- ...spancategorizer.md => spancategorizer.mdx} | 50 +- .../docs/api/{spangroup.md => spangroup.mdx} | 33 +- .../docs/api/{spanruler.md => spanruler.mdx} | 40 +- .../api/{stringstore.md => stringstore.mdx} | 24 +- website/docs/api/{tagger.md => tagger.mdx} | 42 +- ...textcategorizer.md => textcategorizer.mdx} | 46 +- website/docs/api/{tok2vec.md => tok2vec.mdx} | 32 +- website/docs/api/{token.md => token.mdx} | 48 +- .../docs/api/{tokenizer.md => tokenizer.mdx} | 28 +- .../docs/api/{top-level.md => top-level.mdx} | 177 +- .../api/{transformer.md => transformer.mdx} | 54 +- website/docs/api/{vectors.md => vectors.mdx} | 50 +- website/docs/api/{vocab.md => vocab.mdx} | 36 +- website/docs/images/displacy-dep-founded.html | 58 - website/docs/images/displacy-ent-custom.html | 33 - website/docs/images/displacy-ent-snek.html | 26 - website/docs/images/displacy-ent1.html | 37 - website/docs/images/displacy-ent2.html | 39 - website/docs/images/displacy-long2.html | 84 - website/docs/images/displacy-span-custom.html | 31 - website/docs/images/displacy-span.html | 41 - website/docs/index.md | 6 - website/docs/models/{index.md => index.mdx} | 22 +- .../docs/{styleguide.md => styleguide.mdx} | 165 +- .../{_architecture.md => _architecture.mdx} | 12 +- .../{_language-data.md => _language-data.mdx} | 0 ..._named-entities.md => _named-entities.mdx} | 20 +- .../101/{_pipelines.md => _pipelines.mdx} | 4 +- .../usage/101/{_pos-deps.md => _pos-deps.mdx} | 12 +- .../{_serialization.md => _serialization.mdx} | 0 .../{_tokenization.md => _tokenization.mdx} | 5 +- .../usage/101/{_training.md => _training.mdx} | 8 +- ...-similarity.md => _vectors-similarity.mdx} | 18 +- ...marks-models.md => _benchmarks-models.mdx} | 10 +- ...formers.md => embeddings-transformers.mdx} | 99 +- .../{facts-figures.md => facts-figures.mdx} | 24 +- website/docs/usage/{index.md => index.mdx} | 61 +- ...hitectures.md => layers-architectures.mdx} | 138 +- ...ic-features.md => linguistic-features.mdx} | 249 +- website/docs/usage/{models.md => models.mdx} | 72 +- ...-pipelines.md => processing-pipelines.mdx} | 133 +- .../docs/usage/{projects.md => projects.mdx} | 219 +- ...ed-matching.md => rule-based-matching.mdx} | 200 +- .../{saving-loading.md => saving-loading.mdx} | 87 +- .../usage/{spacy-101.md => spacy-101.mdx} | 102 +- .../docs/usage/{training.md => training.mdx} | 256 +- website/docs/usage/{v2-1.md => v2-1.mdx} | 24 +- website/docs/usage/{v2-2.md => v2-2.mdx} | 17 +- website/docs/usage/{v2-3.md => v2-3.mdx} | 12 +- website/docs/usage/{v2.md => v2.mdx} | 44 +- website/docs/usage/{v3-1.md => v3-1.mdx} | 48 +- website/docs/usage/{v3-2.md => v3-2.mdx} | 32 +- website/docs/usage/{v3-3.md => v3-3.mdx} | 28 +- website/docs/usage/{v3-4.md => v3-4.mdx} | 22 +- website/docs/usage/{v3.md => v3.mdx} | 123 +- .../usage/{visualizers.md => visualizers.mdx} | 115 +- website/gatsby-browser.js | 50 - website/gatsby-config.js | 200 - website/gatsby-node.js | 290 - website/meta/dynamicMeta.mjs | 14 + website/meta/languageSorted.tsx | 5 + website/meta/languages.json | 152 +- website/meta/recordLanguages.tsx | 7 + website/meta/recordSections.tsx | 5 + website/meta/recordUniverse.tsx | 9 + website/meta/sidebarFlat.tsx | 5 + website/meta/site.json | 1 - website/meta/universe.json | 62 +- website/netlify.toml | 18 + website/next-sitemap.config.mjs | 10 + website/next.config.mjs | 38 + website/package-lock.json | 50124 +++++++--------- website/package.json | 142 +- website/pages/404.js | 32 + website/pages/[...listPathPage].tsx | 150 + website/pages/_app.tsx | 33 + website/pages/_document.tsx | 13 + .../widgets/landing.js => pages/index.tsx} | 198 +- website/pages/models/[slug].tsx | 66 + website/pages/universe/category/[slug].tsx | 43 + website/pages/universe/index.tsx | 17 + website/pages/universe/project/[slug].tsx | 41 + website/plugins/getProps.mjs | 39 + website/plugins/index.mjs | 20 + .../remarkCodeBlocks.mjs} | 45 +- website/plugins/remarkCustomAttrs.mjs | 38 + website/plugins/remarkFindAndReplace.mjs | 42 + .../remarkWrapSections.mjs} | 19 +- website/public/favicon.ico | Bin 0 -> 25931 bytes website/public/icons/icon-192x192.png | Bin 0 -> 12396 bytes website/public/icons/icon-256x256.png | Bin 0 -> 11554 bytes website/public/icons/icon-384x384.png | Bin 0 -> 28487 bytes website/public/icons/icon-512x512.png | Bin 0 -> 24892 bytes .../{docs => public}/images/architecture.svg | 0 .../images/cli_init_fill-config_diff.jpg | Bin website/{docs => public}/images/course.jpg | Bin .../images/dep-match-diagram.svg | 0 .../images/displacy-compact.svg | 0 .../images/displacy-custom-parser.svg | 0 .../public/images/displacy-dep-founded.html | 155 + .../public/images/displacy-ent-custom.html | 80 + website/public/images/displacy-ent-snek.html | 59 + website/public/images/displacy-ent1.html | 84 + website/public/images/displacy-ent2.html | 86 + .../images/displacy-long.html | 8 +- website/public/images/displacy-long2.html | 212 + .../images/displacy-model-rules.svg | 0 .../images/displacy-model-rules2.svg | 0 .../images/displacy-small.svg | 0 .../public/images/displacy-span-custom.html | 84 + website/public/images/displacy-span.html | 123 + website/{docs => public}/images/displacy.svg | 0 .../images/displacy_jupyter.jpg | Bin .../images/huggingface_hub.jpg | Bin website/{docs => public}/images/lifecycle.svg | 0 .../{docs => public}/images/matcher-demo.jpg | Bin .../images/pipeline-design.svg | 0 website/{docs => public}/images/pipeline.svg | 0 .../images/pipeline_transformer.svg | 0 website/{docs => public}/images/prodigy.jpg | Bin .../images/prodigy_overview.jpg | Bin .../images/prodigy_spans-manual.jpg | Bin .../images/prodigy_train_curve.jpg | Bin .../images/project_document.jpg | Bin website/{docs => public}/images/projects.png | Bin website/{docs => public}/images/projects.svg | 0 website/{docs => public}/images/sense2vec.jpg | Bin website/{docs => public}/images/spacy-ray.svg | 0 .../images/spacy-streamlit.png | Bin .../images/spacy-tailored-pipelines_wide.png | Bin .../{docs => public}/images/thinc_mypy.jpg | Bin .../images/tok2vec-listener.svg | 0 website/{docs => public}/images/tok2vec.svg | 0 .../{docs => public}/images/tokenization.svg | 0 .../images/trainable_component.svg | 0 website/{docs => public}/images/training.svg | 0 .../images/vocab_stringstore.svg | 0 website/{docs => public}/images/wandb1.jpg | Bin website/{docs => public}/images/wandb2.jpg | Bin website/public/manifest.webmanifest | 31 + website/public/vercel.svg | 4 + website/runtime.txt | 2 +- website/setup/setup.sh | 2 +- website/src/components/accordion.js | 4 +- website/src/components/card.js | 1 + website/src/components/code.js | 436 +- website/src/components/copy.js | 8 +- website/src/components/dropdown.js | 5 +- website/src/components/embed.js | 37 +- website/src/components/footer.js | 122 +- website/src/components/github.js | 20 +- website/src/components/icon.js | 98 +- website/src/components/infobox.js | 2 +- website/src/components/juniper.js | 181 +- website/src/components/landing.js | 57 +- website/src/components/link.js | 33 +- website/src/components/list.js | 6 +- website/src/components/main.js | 12 +- website/src/components/navigation.js | 9 +- website/src/components/quickstart.js | 133 +- website/src/components/search.js | 41 +- website/src/components/section.js | 2 +- website/src/components/seo.js | 173 +- website/src/components/sidebar.js | 38 +- website/src/components/table.js | 8 +- website/src/components/title.js | 6 +- website/src/components/typography.js | 16 +- website/src/components/util.js | 39 +- website/src/html.js | 43 - website/src/pages/404.js | 49 - website/src/plugins/remark-custom-attrs.js | 52 - website/src/remark.js | 113 + website/src/styles/aside.module.sass | 7 +- website/src/styles/code.module.sass | 2 +- website/src/styles/embed.module.sass | 13 +- website/src/styles/grid.module.sass | 3 +- website/src/styles/layout.sass | 76 +- website/src/styles/quickstart.module.sass | 7 +- website/src/styles/search.module.sass | 58 - website/src/styles/search.sass | 27 + website/src/styles/sidebar.module.sass | 9 +- website/src/templates/docs.js | 205 +- website/src/templates/index.js | 157 +- website/src/templates/mdx-renderer.js | 21 - website/src/templates/models.js | 123 +- website/src/templates/universe.js | 123 +- website/src/widgets/changelog.js | 14 +- website/src/widgets/features.js | 125 +- website/src/widgets/integration.js | 36 +- website/src/widgets/languages.js | 122 +- website/src/widgets/quickstart-install.js | 397 +- website/src/widgets/quickstart-models.js | 125 +- website/src/widgets/quickstart-training.js | 106 +- website/src/widgets/styleguide.js | 15 +- website/tsconfig.json | 20 + 241 files changed, 26957 insertions(+), 34416 deletions(-) create mode 100644 website/.eslintrc.json create mode 100644 website/.gitignore create mode 100644 website/.nvmrc create mode 100644 website/.prettierignore create mode 100644 website/.vscode/extensions.json rename website/docs/api/{architectures.md => architectures.mdx} (96%) rename website/docs/api/{attributeruler.md => attributeruler.mdx} (94%) rename website/docs/api/{attributes.md => attributes.mdx} (98%) rename website/docs/api/{cli.md => cli.mdx} (97%) rename website/docs/api/{coref.md => coref.mdx} (94%) rename website/docs/api/{corpus.md => corpus.mdx} (96%) rename website/docs/api/{cython-classes.md => cython-classes.mdx} (91%) rename website/docs/api/{cython-structs.md => cython-structs.mdx} (94%) rename website/docs/api/{cython.md => cython.mdx} (99%) rename website/docs/api/{data-formats.md => data-formats.mdx} (98%) rename website/docs/api/{dependencymatcher.md => dependencymatcher.mdx} (96%) rename website/docs/api/{dependencyparser.md => dependencyparser.mdx} (95%) rename website/docs/api/{doc.md => doc.mdx} (95%) rename website/docs/api/{docbin.md => docbin.mdx} (93%) rename website/docs/api/{edittreelemmatizer.md => edittreelemmatizer.mdx} (95%) rename website/docs/api/{entitylinker.md => entitylinker.mdx} (96%) rename website/docs/api/{entityrecognizer.md => entityrecognizer.mdx} (95%) rename website/docs/api/{entityruler.md => entityruler.mdx} (94%) rename website/docs/api/{example.md => example.mdx} (92%) rename website/docs/api/{index.md => index.mdx} (58%) rename website/docs/api/{kb.md => kb.mdx} (92%) rename website/docs/api/{kb_in_memory.md => kb_in_memory.mdx} (90%) rename website/docs/api/{language.md => language.mdx} (96%) rename website/docs/api/{legacy.md => legacy.mdx} (95%) rename website/docs/api/{lemmatizer.md => lemmatizer.mdx} (95%) rename website/docs/api/{lexeme.md => lexeme.mdx} (97%) rename website/docs/api/{lookups.md => lookups.mdx} (89%) rename website/docs/api/{matcher.md => matcher.mdx} (97%) rename website/docs/api/{morphologizer.md => morphologizer.mdx} (95%) rename website/docs/api/{morphology.md => morphology.mdx} (89%) rename website/docs/api/{phrasematcher.md => phrasematcher.mdx} (96%) rename website/docs/api/{pipe.md => pipe.mdx} (93%) rename website/docs/api/{pipeline-functions.md => pipeline-functions.mdx} (95%) rename website/docs/api/{scorer.md => scorer.mdx} (96%) rename website/docs/api/{sentencerecognizer.md => sentencerecognizer.mdx} (94%) rename website/docs/api/{sentencizer.md => sentencizer.mdx} (94%) rename website/docs/api/{span-resolver.md => span-resolver.mdx} (94%) rename website/docs/api/{span.md => span.mdx} (93%) rename website/docs/api/{spancategorizer.md => spancategorizer.mdx} (94%) rename website/docs/api/{spangroup.md => spangroup.mdx} (92%) rename website/docs/api/{spanruler.md => spanruler.mdx} (94%) rename website/docs/api/{stringstore.md => stringstore.mdx} (89%) rename website/docs/api/{tagger.md => tagger.mdx} (95%) rename website/docs/api/{textcategorizer.md => textcategorizer.mdx} (94%) rename website/docs/api/{tok2vec.md => tok2vec.mdx} (94%) rename website/docs/api/{token.md => token.mdx} (96%) rename website/docs/api/{tokenizer.md => tokenizer.mdx} (95%) rename website/docs/api/{top-level.md => top-level.mdx} (93%) rename website/docs/api/{transformer.md => transformer.mdx} (95%) rename website/docs/api/{vectors.md => vectors.mdx} (94%) rename website/docs/api/{vocab.md => vocab.mdx} (94%) delete mode 100644 website/docs/images/displacy-dep-founded.html delete mode 100644 website/docs/images/displacy-ent-custom.html delete mode 100644 website/docs/images/displacy-ent-snek.html delete mode 100644 website/docs/images/displacy-ent1.html delete mode 100644 website/docs/images/displacy-ent2.html delete mode 100644 website/docs/images/displacy-long2.html delete mode 100644 website/docs/images/displacy-span-custom.html delete mode 100644 website/docs/images/displacy-span.html delete mode 100644 website/docs/index.md rename website/docs/models/{index.md => index.mdx} (95%) rename website/docs/{styleguide.md => styleguide.mdx} (86%) rename website/docs/usage/101/{_architecture.md => _architecture.mdx} (96%) rename website/docs/usage/101/{_language-data.md => _language-data.mdx} (100%) rename website/docs/usage/101/{_named-entities.md => _named-entities.mdx} (75%) rename website/docs/usage/101/{_pipelines.md => _pipelines.mdx} (98%) rename website/docs/usage/101/{_pos-deps.md => _pos-deps.mdx} (92%) rename website/docs/usage/101/{_serialization.md => _serialization.mdx} (100%) rename website/docs/usage/101/{_tokenization.md => _tokenization.mdx} (95%) rename website/docs/usage/101/{_training.md => _training.mdx} (91%) rename website/docs/usage/101/{_vectors-similarity.md => _vectors-similarity.mdx} (96%) rename website/docs/usage/{_benchmarks-models.md => _benchmarks-models.mdx} (86%) rename website/docs/usage/{embeddings-transformers.md => embeddings-transformers.mdx} (94%) rename website/docs/usage/{facts-figures.md => facts-figures.mdx} (92%) rename website/docs/usage/{index.md => index.mdx} (93%) rename website/docs/usage/{layers-architectures.md => layers-architectures.mdx} (91%) rename website/docs/usage/{linguistic-features.md => linguistic-features.mdx} (94%) rename website/docs/usage/{models.md => models.mdx} (93%) rename website/docs/usage/{processing-pipelines.md => processing-pipelines.mdx} (96%) rename website/docs/usage/{projects.md => projects.mdx} (92%) rename website/docs/usage/{rule-based-matching.md => rule-based-matching.mdx} (95%) rename website/docs/usage/{saving-loading.md => saving-loading.mdx} (95%) rename website/docs/usage/{spacy-101.md => spacy-101.mdx} (91%) rename website/docs/usage/{training.md => training.mdx} (91%) rename website/docs/usage/{v2-1.md => v2-1.mdx} (94%) rename website/docs/usage/{v2-2.md => v2-2.mdx} (97%) rename website/docs/usage/{v2-3.md => v2-3.mdx} (98%) rename website/docs/usage/{v2.md => v2.mdx} (95%) rename website/docs/usage/{v3-1.md => v3-1.mdx} (91%) rename website/docs/usage/{v3-2.md => v3-2.mdx} (92%) rename website/docs/usage/{v3-3.md => v3-3.mdx} (95%) rename website/docs/usage/{v3-4.md => v3-4.mdx} (90%) rename website/docs/usage/{v3.md => v3.mdx} (95%) rename website/docs/usage/{visualizers.md => visualizers.mdx} (87%) delete mode 100644 website/gatsby-browser.js delete mode 100644 website/gatsby-config.js delete mode 100644 website/gatsby-node.js create mode 100644 website/meta/dynamicMeta.mjs create mode 100644 website/meta/languageSorted.tsx create mode 100644 website/meta/recordLanguages.tsx create mode 100644 website/meta/recordSections.tsx create mode 100644 website/meta/recordUniverse.tsx create mode 100644 website/meta/sidebarFlat.tsx create mode 100644 website/netlify.toml create mode 100644 website/next-sitemap.config.mjs create mode 100644 website/next.config.mjs create mode 100644 website/pages/404.js create mode 100644 website/pages/[...listPathPage].tsx create mode 100644 website/pages/_app.tsx create mode 100644 website/pages/_document.tsx rename website/{src/widgets/landing.js => pages/index.tsx} (62%) create mode 100644 website/pages/models/[slug].tsx create mode 100644 website/pages/universe/category/[slug].tsx create mode 100644 website/pages/universe/index.tsx create mode 100644 website/pages/universe/project/[slug].tsx create mode 100644 website/plugins/getProps.mjs create mode 100644 website/plugins/index.mjs rename website/{src/plugins/remark-code-blocks.js => plugins/remarkCodeBlocks.mjs} (67%) create mode 100644 website/plugins/remarkCustomAttrs.mjs create mode 100644 website/plugins/remarkFindAndReplace.mjs rename website/{src/plugins/remark-wrap-section.js => plugins/remarkWrapSections.mjs} (80%) create mode 100644 website/public/favicon.ico create mode 100644 website/public/icons/icon-192x192.png create mode 100644 website/public/icons/icon-256x256.png create mode 100644 website/public/icons/icon-384x384.png create mode 100644 website/public/icons/icon-512x512.png rename website/{docs => public}/images/architecture.svg (100%) rename website/{docs => public}/images/cli_init_fill-config_diff.jpg (100%) rename website/{docs => public}/images/course.jpg (100%) rename website/{docs => public}/images/dep-match-diagram.svg (100%) rename website/{docs => public}/images/displacy-compact.svg (100%) rename website/{docs => public}/images/displacy-custom-parser.svg (100%) create mode 100644 website/public/images/displacy-dep-founded.html create mode 100644 website/public/images/displacy-ent-custom.html create mode 100644 website/public/images/displacy-ent-snek.html create mode 100644 website/public/images/displacy-ent1.html create mode 100644 website/public/images/displacy-ent2.html rename website/{docs => public}/images/displacy-long.html (98%) create mode 100644 website/public/images/displacy-long2.html rename website/{docs => public}/images/displacy-model-rules.svg (100%) rename website/{docs => public}/images/displacy-model-rules2.svg (100%) rename website/{docs => public}/images/displacy-small.svg (100%) create mode 100644 website/public/images/displacy-span-custom.html create mode 100644 website/public/images/displacy-span.html rename website/{docs => public}/images/displacy.svg (100%) rename website/{docs => public}/images/displacy_jupyter.jpg (100%) rename website/{docs => public}/images/huggingface_hub.jpg (100%) rename website/{docs => public}/images/lifecycle.svg (100%) rename website/{docs => public}/images/matcher-demo.jpg (100%) rename website/{docs => public}/images/pipeline-design.svg (100%) rename website/{docs => public}/images/pipeline.svg (100%) rename website/{docs => public}/images/pipeline_transformer.svg (100%) rename website/{docs => public}/images/prodigy.jpg (100%) rename website/{docs => public}/images/prodigy_overview.jpg (100%) rename website/{docs => public}/images/prodigy_spans-manual.jpg (100%) rename website/{docs => public}/images/prodigy_train_curve.jpg (100%) rename website/{docs => public}/images/project_document.jpg (100%) rename website/{docs => public}/images/projects.png (100%) rename website/{docs => public}/images/projects.svg (100%) rename website/{docs => public}/images/sense2vec.jpg (100%) rename website/{docs => public}/images/spacy-ray.svg (100%) rename website/{docs => public}/images/spacy-streamlit.png (100%) rename website/{docs => public}/images/spacy-tailored-pipelines_wide.png (100%) rename website/{docs => public}/images/thinc_mypy.jpg (100%) rename website/{docs => public}/images/tok2vec-listener.svg (100%) rename website/{docs => public}/images/tok2vec.svg (100%) rename website/{docs => public}/images/tokenization.svg (100%) rename website/{docs => public}/images/trainable_component.svg (100%) rename website/{docs => public}/images/training.svg (100%) rename website/{docs => public}/images/vocab_stringstore.svg (100%) rename website/{docs => public}/images/wandb1.jpg (100%) rename website/{docs => public}/images/wandb2.jpg (100%) create mode 100644 website/public/manifest.webmanifest create mode 100644 website/public/vercel.svg delete mode 100644 website/src/html.js delete mode 100644 website/src/pages/404.js delete mode 100644 website/src/plugins/remark-custom-attrs.js create mode 100644 website/src/remark.js delete mode 100644 website/src/styles/search.module.sass create mode 100644 website/src/styles/search.sass delete mode 100644 website/src/templates/mdx-renderer.js create mode 100644 website/tsconfig.json diff --git a/website/.eslintrc.json b/website/.eslintrc.json new file mode 100644 index 000000000..1c2aa65d7 --- /dev/null +++ b/website/.eslintrc.json @@ -0,0 +1,3 @@ +{ + "extends": "next/core-web-vitals" +} diff --git a/website/.gitignore b/website/.gitignore new file mode 100644 index 000000000..70ef99fa5 --- /dev/null +++ b/website/.gitignore @@ -0,0 +1,44 @@ +# See https://help.github.com/articles/ignoring-files/ for more about ignoring files. + +# dependencies +/node_modules +/.pnp +.pnp.js + +# testing +/coverage + +# next.js +/.next/ +/out/ + +# production +/build + +# misc +.DS_Store +*.pem + +# debug +npm-debug.log* +yarn-debug.log* +yarn-error.log* +.pnpm-debug.log* + +# local env files +.env*.local + +# vercel +.vercel + +# typescript +*.tsbuildinfo +next-env.d.ts + +!.vscode/extensions.json +!public + +public/robots.txt +public/sitemap* +public/sw.js* +public/workbox* \ No newline at end of file diff --git a/website/.nvmrc b/website/.nvmrc new file mode 100644 index 000000000..3c032078a --- /dev/null +++ b/website/.nvmrc @@ -0,0 +1 @@ +18 diff --git a/website/.prettierignore b/website/.prettierignore new file mode 100644 index 000000000..d0d878e40 --- /dev/null +++ b/website/.prettierignore @@ -0,0 +1 @@ +.next \ No newline at end of file diff --git a/website/.prettierrc b/website/.prettierrc index 7555c734a..03904b1c4 100644 --- a/website/.prettierrc +++ b/website/.prettierrc @@ -20,12 +20,11 @@ } }, { - "files": "*.md", + "files": ["package.json", "package-lock.json"], "options": { "tabWidth": 2, "printWidth": 80, - "proseWrap": "always", - "htmlWhitespaceSensitivity": "strict" + "proseWrap": "always" } }, { diff --git a/website/.vscode/extensions.json b/website/.vscode/extensions.json new file mode 100644 index 000000000..4b533827a --- /dev/null +++ b/website/.vscode/extensions.json @@ -0,0 +1,8 @@ +{ + "recommendations": [ + "dbaeumer.vscode-eslint", + "unifiedjs.vscode-mdx", + "esbenp.prettier-vscode", + "syler.sass-indented" + ] +} diff --git a/website/README.md b/website/README.md index 890a48ef9..e9d7aec26 100644 --- a/website/README.md +++ b/website/README.md @@ -7,17 +7,16 @@ The styleguide for the spaCy website is available at ## Setup and installation -Before running the setup, make sure your versions of -[Node](https://nodejs.org/en/) and [npm](https://www.npmjs.com/) are up to date. -Node v10.15 or later is required. - ```bash # Clone the repository git clone https://github.com/explosion/spaCy cd spaCy/website -# Install Gatsby's command-line tool -npm install --global gatsby-cli +# Switch to the correct Node version +# +# If you don't have NVM and don't want to use it, you can manually switch to the Node version +# stated in /.nvmrc and skip this step +nvm use # Install the dependencies npm install @@ -36,8 +35,7 @@ file in the root defines the settings used in this codebase. ## Building & developing the site with Docker -Sometimes it's hard to get a local environment working due to rapid updates to -node dependencies, so it may be easier to use docker for building the docs. +While it shouldn't be necessary and is not recommended you can run this site in a Docker container. If you'd like to do this, **be sure you do _not_ include your local `node_modules` folder**, since there are some dependencies that need to be built @@ -76,12 +74,14 @@ bit of time. ```yaml ├── docs # the actual markdown content ├── meta # JSON-formatted site metadata +| ├── dynamicMeta.js # At build time generated meta data | ├── languages.json # supported languages and statistical models | ├── sidebars.json # sidebar navigations for different sections | ├── site.json # general site metadata | ├── type-annotations.json # Type annotations | └── universe.json # data for the spaCy universe section -├── public # compiled site +├── pages # Next router pages +├── public # static images and other assets ├── setup # Jinja setup ├── src # source | ├── components # React components @@ -96,9 +96,11 @@ bit of time. | | └── universe.js # layout templates for universe | └── widgets # non-reusable components with content, e.g. changelog ├── .eslintrc.json # ESLint config file +├── .nvmrc # NVM config file +| # (to support "nvm use" to switch to correct Node version) +| ├── .prettierrc # Prettier config file -├── gatsby-browser.js # browser-specific hooks for Gatsby -├── gatsby-config.js # Gatsby configuration -├── gatsby-node.js # Node-specific hooks for Gatsby -└── package.json # package settings and dependencies +├── next.config.mjs # Next config file +├── package.json # package settings and dependencies +└── tsconfig.json # TypeScript config file ``` diff --git a/website/UNIVERSE.md b/website/UNIVERSE.md index 770bbde13..ac4e2e684 100644 --- a/website/UNIVERSE.md +++ b/website/UNIVERSE.md @@ -2,42 +2,52 @@ # spaCy Universe -The [spaCy Universe](https://spacy.io/universe) collects the many great resources developed with or for spaCy. It -includes standalone packages, plugins, extensions, educational materials, -operational utilities and bindings for other languages. +The [spaCy Universe](https://spacy.io/universe) collects the many great +resources developed with or for spaCy. It includes standalone packages, plugins, +extensions, educational materials, operational utilities and bindings for other +languages. If you have a project that you want the spaCy community to make use of, you can suggest it by submitting a pull request to this repository. The Universe database is open-source and collected in a simple JSON file. Looking for inspiration for your own spaCy plugin or extension? Check out the -[`project ideas`](https://github.com/explosion/spaCy/discussions?discussions_q=category%3A%22New+Features+%26+Project+Ideas%22) +[`project ideas`](https://github.com/explosion/spaCy/discussions?discussions_q=category%3A%22New+Features+%26+Project+Ideas%22) discussion forum. ## Checklist ### Projects -✅ Libraries and packages should be **open-source** (with a user-friendly license) and at least somewhat **documented** (e.g. a simple `README` with usage instructions). +✅ Libraries and packages should be **open-source** (with a user-friendly +license) and at least somewhat **documented** (e.g. a simple `README` with usage +instructions). -✅ We're happy to include work in progress and prereleases, but we'd like to keep the emphasis on projects that should be useful to the community **right away**. +✅ We're happy to include work in progress and prereleases, but we'd like to +keep the emphasis on projects that should be useful to the community **right +away**. ✅ Demos and visualizers should be available via a **public URL**. ### Educational Materials -✅ Books should be **available for purchase or download** (not just pre-order). Ebooks and self-published books are fine, too, if they include enough substantial content. +✅ Books should be **available for purchase or download** (not just pre-order). +Ebooks and self-published books are fine, too, if they include enough +substantial content. -✅ The `"url"` of book entries should either point to the publisher's website or a reseller of your choice (ideally one that ships worldwide or as close as possible). +✅ The `"url"` of book entries should either point to the publisher's website or +a reseller of your choice (ideally one that ships worldwide or as close as +possible). -✅ If an online course is only available behind a paywall, it should at least have a **free excerpt** or chapter available, so users know what to expect. +✅ If an online course is only available behind a paywall, it should at least +have a **free excerpt** or chapter available, so users know what to expect. ## JSON format -To add a project, fork this repository, edit the [`universe.json`](meta/universe.json) -and add an object of the following format to the list of `"resources"`. Before -you submit your pull request, make sure to use a linter to verify that your -markup is correct. +To add a project, fork this repository, edit the +[`universe.json`](meta/universe.json) and add an object of the following format +to the list of `"resources"`. Before you submit your pull request, make sure to +use a linter to verify that your markup is correct. ```json { @@ -69,26 +79,26 @@ markup is correct. } ``` -| Field | Type | Description | -| --- | --- | --- | -| `id` | string | Unique ID of the project. | -| `title` | string | Project title. If not set, the `id` will be used as the display title. | -| `slogan` | string | A short description of the project. Displayed in the overview and under the title. | -| `description` | string | A longer description of the project. Markdown is allowed, but should be limited to basic formatting like bold, italics, code or links. | -| `github` | string | Associated GitHub repo in the format `user/repo`. Will be displayed as a link and used for release, license and star badges. | -| `pip` | string | Package name on pip. If available, the installation command will be displayed. | -| `cran` | string | For R packages: package name on CRAN. If available, the installation command will be displayed. | -| `code_example` | array | Short example that shows how to use the project. Formatted as an array with one string per line. | -| `code_language` | string | Defaults to `'python'`. Optional code language used for syntax highlighting with [Prism](http://prismjs.com/). | -| `url` | string | Optional project link to display as button. | -| `thumb` | string | Optional URL to project thumbnail to display in overview and project header. Recommended size is 100x100px. | -| `image` | string | Optional URL to project image to display with description. | -| `author` | string | Name(s) of project author(s). | -| `author_links` | object | Usernames and links to display as icons to author info. Currently supports `twitter` and `github` usernames, as well as `website` link. | -| `category` | list | One or more categories to assign to project. Must be one of the available options. | -| `tags` | list | Still experimental and not used for filtering: one or more tags to assign to project. | +| Field | Type | Description | +| --------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------- | +| `id` | string | Unique ID of the project. | +| `title` | string | Project title. If not set, the `id` will be used as the display title. | +| `slogan` | string | A short description of the project. Displayed in the overview and under the title. | +| `description` | string | A longer description of the project. Markdown is allowed, but should be limited to basic formatting like bold, italics, code or links. | +| `github` | string | Associated GitHub repo in the format `user/repo`. Will be displayed as a link and used for release, license and star badges. | +| `pip` | string | Package name on pip. If available, the installation command will be displayed. | +| `cran` | string | For R packages: package name on CRAN. If available, the installation command will be displayed. | +| `code_example` | array | Short example that shows how to use the project. Formatted as an array with one string per line. | +| `code_language` | string | Defaults to `'python'`. Optional code language used for syntax highlighting with [Prism](http://prismjs.com/). | +| `url` | string | Optional project link to display as button. | +| `thumb` | string | Optional URL to project thumbnail to display in overview and project header. Recommended size is 100x100px. | +| `image` | string | Optional URL to project image to display with description. | +| `author` | string | Name(s) of project author(s). | +| `author_links` | object | Usernames and links to display as icons to author info. Currently supports `twitter` and `github` usernames, as well as `website` link. | +| `category` | list | One or more categories to assign to project. Must be one of the available options. | +| `tags` | list | Still experimental and not used for filtering: one or more tags to assign to project. | To separate them from the projects, educational materials also specify -`"type": "education`. Books can also set a `"cover"` field containing a URL -to a cover image. If available, it's used in the overview and displayed on -the individual book page. +`"type": "education`. Books can also set a `"cover"` field containing a URL to a +cover image. If available, it's used in the overview and displayed on the +individual book page. diff --git a/website/docs/api/architectures.md b/website/docs/api/architectures.mdx similarity index 96% rename from website/docs/api/architectures.md rename to website/docs/api/architectures.mdx index 4c5447f75..2a1bc4380 100644 --- a/website/docs/api/architectures.md +++ b/website/docs/api/architectures.mdx @@ -26,9 +26,9 @@ part of the [training config](/usage/training#custom-functions). Also see the usage documentation on [layers and model architectures](/usage/layers-architectures). -## Tok2Vec architectures {#tok2vec-arch source="spacy/ml/models/tok2vec.py"} +## Tok2Vec architectures {id="tok2vec-arch",source="spacy/ml/models/tok2vec.py"} -### spacy.Tok2Vec.v2 {#Tok2Vec} +### spacy.Tok2Vec.v2 {id="Tok2Vec"} > #### Example config > @@ -56,7 +56,7 @@ blog post for background. | `encode` | Encode context into the embeddings, using an architecture such as a CNN, BiLSTM or transformer. For example, [MaxoutWindowEncoder](/api/architectures#MaxoutWindowEncoder). ~~Model[List[Floats2d], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | -### spacy.HashEmbedCNN.v2 {#HashEmbedCNN} +### spacy.HashEmbedCNN.v2 {id="HashEmbedCNN"} > #### Example Config > @@ -89,7 +89,7 @@ consisting of a CNN and a layer-normalized maxout activation function. | `pretrained_vectors` | Whether to also use static vectors. ~~bool~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | -### spacy.Tok2VecListener.v1 {#Tok2VecListener} +### spacy.Tok2VecListener.v1 {id="Tok2VecListener"} > #### Example config > @@ -139,7 +139,7 @@ the `Tok2Vec` component. | `upstream` | A string to identify the "upstream" `Tok2Vec` component to communicate with. By default, the upstream name is the wildcard string `"*"`, but you could also specify the name of the `Tok2Vec` component. You'll almost never have multiple upstream `Tok2Vec` components, so the wildcard string will almost always be fine. ~~str~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | -### spacy.MultiHashEmbed.v2 {#MultiHashEmbed} +### spacy.MultiHashEmbed.v2 {id="MultiHashEmbed"} > #### Example config > @@ -170,7 +170,7 @@ updated). | `include_static_vectors` | Whether to also use static word vectors. Requires a vectors table to be loaded in the [`Doc`](/api/doc) objects' vocab. ~~bool~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | -### spacy.CharacterEmbed.v2 {#CharacterEmbed} +### spacy.CharacterEmbed.v2 {id="CharacterEmbed"} > #### Example config > @@ -207,7 +207,7 @@ network to construct a single vector to represent the information. | `nC` | The number of UTF-8 bytes to embed per word. Recommended values are between `3` and `8`, although it may depend on the length of words in the language. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | -### spacy.MaxoutWindowEncoder.v2 {#MaxoutWindowEncoder} +### spacy.MaxoutWindowEncoder.v2 {id="MaxoutWindowEncoder"} > #### Example config > @@ -231,7 +231,7 @@ and residual connections. | `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ | -### spacy.MishWindowEncoder.v2 {#MishWindowEncoder} +### spacy.MishWindowEncoder.v2 {id="MishWindowEncoder"} > #### Example config > @@ -254,7 +254,7 @@ and residual connections. | `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ | -### spacy.TorchBiLSTMEncoder.v1 {#TorchBiLSTMEncoder} +### spacy.TorchBiLSTMEncoder.v1 {id="TorchBiLSTMEncoder"} > #### Example config > @@ -276,7 +276,7 @@ Encode context using bidirectional LSTM layers. Requires | `dropout` | Creates a Dropout layer on the outputs of each LSTM layer except the last layer. Set to 0.0 to disable this functionality. ~~float~~ | | **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ | -### spacy.StaticVectors.v2 {#StaticVectors} +### spacy.StaticVectors.v2 {id="StaticVectors"} > #### Example config > @@ -306,7 +306,7 @@ mapped to a zero vector. See the documentation on | `key_attr` | Defaults to `"ORTH"`. ~~str~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Ragged]~~ | -### spacy.FeatureExtractor.v1 {#FeatureExtractor} +### spacy.FeatureExtractor.v1 {id="FeatureExtractor"} > #### Example config > @@ -324,7 +324,7 @@ of feature names to extract, which should refer to token attributes. | `columns` | The token attributes to extract. ~~List[Union[int, str]]~~ | | **CREATES** | The created feature extraction layer. ~~Model[List[Doc], List[Ints2d]]~~ | -## Transformer architectures {#transformers source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/architectures.py"} +## Transformer architectures {id="transformers",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/architectures.py"} The following architectures are provided by the package [`spacy-transformers`](https://github.com/explosion/spacy-transformers). See the @@ -341,7 +341,7 @@ for details and system requirements. -### spacy-transformers.TransformerModel.v3 {#TransformerModel} +### spacy-transformers.TransformerModel.v3 {id="TransformerModel"} > #### Example Config > @@ -390,7 +390,7 @@ in other components, see | | | -Mixed-precision support is currently an experimental feature. + Mixed-precision support is currently an experimental feature. @@ -404,7 +404,7 @@ The other arguments are shared between all versions. -### spacy-transformers.TransformerListener.v1 {#TransformerListener} +### spacy-transformers.TransformerListener.v1 {id="TransformerListener"} > #### Example Config > @@ -434,7 +434,7 @@ a single token vector given zero or more wordpiece vectors. | `upstream` | A string to identify the "upstream" `Transformer` component to communicate with. By default, the upstream name is the wildcard string `"*"`, but you could also specify the name of the `Transformer` component. You'll almost never have multiple upstream `Transformer` components, so the wildcard string will almost always be fine. ~~str~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | -### spacy-transformers.Tok2VecTransformer.v3 {#Tok2VecTransformer} +### spacy-transformers.Tok2VecTransformer.v3 {id="Tok2VecTransformer"} > #### Example Config > @@ -467,7 +467,7 @@ one component. | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | -Mixed-precision support is currently an experimental feature. + Mixed-precision support is currently an experimental feature. @@ -481,7 +481,7 @@ The other arguments are shared between all versions. -## Pretraining architectures {#pretrain source="spacy/ml/models/multi_task.py"} +## Pretraining architectures {id="pretrain",source="spacy/ml/models/multi_task.py"} The spacy `pretrain` command lets you initialize a `Tok2Vec` layer in your pipeline with information from raw text. To this end, additional layers are @@ -494,7 +494,7 @@ BERT. For more information, see the section on [pretraining](/usage/embeddings-transformers#pretraining). -### spacy.PretrainVectors.v1 {#pretrain_vectors} +### spacy.PretrainVectors.v1 {id="pretrain_vectors"} > #### Example config > @@ -525,7 +525,7 @@ vectors. | `loss` | The loss function can be either "cosine" or "L2". We typically recommend to use "cosine". ~~~str~~ | | **CREATES** | A callable function that can create the Model, given the `vocab` of the pipeline and the `tok2vec` layer to pretrain. ~~Callable[[Vocab, Model], Model]~~ | -### spacy.PretrainCharacters.v1 {#pretrain_chars} +### spacy.PretrainCharacters.v1 {id="pretrain_chars"} > #### Example config > @@ -551,9 +551,9 @@ for a Tok2Vec layer. | `n_characters` | The window of characters - e.g. if `n_characters = 2`, the model will try to predict the first two and last two characters of the word. ~~int~~ | | **CREATES** | A callable function that can create the Model, given the `vocab` of the pipeline and the `tok2vec` layer to pretrain. ~~Callable[[Vocab, Model], Model]~~ | -## Parser & NER architectures {#parser} +## Parser & NER architectures {id="parser"} -### spacy.TransitionBasedParser.v2 {#TransitionBasedParser source="spacy/ml/models/parser.py"} +### spacy.TransitionBasedParser.v2 {id="TransitionBasedParser",source="spacy/ml/models/parser.py"} > #### Example Config > @@ -612,9 +612,9 @@ same signature, but the `use_upper` argument was `True` by default. -## Tagging architectures {#tagger source="spacy/ml/models/tagger.py"} +## Tagging architectures {id="tagger",source="spacy/ml/models/tagger.py"} -### spacy.Tagger.v2 {#Tagger} +### spacy.Tagger.v2 {id="Tagger"} > #### Example Config > @@ -648,7 +648,7 @@ The other arguments are shared between all versions. -## Text classification architectures {#textcat source="spacy/ml/models/textcat.py"} +## Text classification architectures {id="textcat",source="spacy/ml/models/textcat.py"} A text classification architecture needs to take a [`Doc`](/api/doc) as input, and produce a score for each potential label class. Textcat challenges can be @@ -672,7 +672,7 @@ single-label use-cases where `exclusive_classes = true`, while the -### spacy.TextCatEnsemble.v2 {#TextCatEnsemble} +### spacy.TextCatEnsemble.v2 {id="TextCatEnsemble"} > #### Example Config > @@ -737,7 +737,7 @@ but used an internal `tok2vec` instead of taking it as argument: -### spacy.TextCatCNN.v2 {#TextCatCNN} +### spacy.TextCatCNN.v2 {id="TextCatCNN"} > #### Example Config > @@ -777,7 +777,7 @@ after training. -### spacy.TextCatBOW.v2 {#TextCatBOW} +### spacy.TextCatBOW.v2 {id="TextCatBOW"} > #### Example Config > @@ -809,9 +809,9 @@ after training. -## Span classification architectures {#spancat source="spacy/ml/models/spancat.py"} +## Span classification architectures {id="spancat",source="spacy/ml/models/spancat.py"} -### spacy.SpanCategorizer.v1 {#SpanCategorizer} +### spacy.SpanCategorizer.v1 {id="SpanCategorizer"} > #### Example Config > @@ -848,7 +848,7 @@ single vector, and a scorer model to map the vectors to probabilities. | `scorer` | The scorer model. ~~Model[Floats2d, Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[Tuple[List[Doc], Ragged], Floats2d]~~ | -### spacy.mean_max_reducer.v1 {#mean_max_reducer} +### spacy.mean_max_reducer.v1 {id="mean_max_reducer"} Reduce sequences by concatenating their mean and max pooled vectors, and then combine the concatenated vectors with a hidden layer. @@ -857,7 +857,7 @@ combine the concatenated vectors with a hidden layer. | ------------- | ------------------------------------- | | `hidden_size` | The size of the hidden layer. ~~int~~ | -## Entity linking architectures {#entitylinker source="spacy/ml/models/entity_linker.py"} +## Entity linking architectures {id="entitylinker",source="spacy/ml/models/entity_linker.py"} An [`EntityLinker`](/api/entitylinker) component disambiguates textual mentions (tagged as named entities) to unique identifiers, grounding the named entities @@ -870,7 +870,7 @@ into the "real world". This requires 3 main components: - A machine learning [`Model`](https://thinc.ai/docs/api-model) that picks the most plausible ID from the set of candidates. -### spacy.EntityLinker.v2 {#EntityLinker} +### spacy.EntityLinker.v2 {id="EntityLinker"} > #### Example Config > @@ -899,7 +899,7 @@ The `EntityLinker` model architecture is a Thinc `Model` with a | `nO` | Output dimension, determined by the length of the vectors encoding each entity in the KB. If the `nO` dimension is not set, the entity linking component will set it when `initialize` is called. ~~Optional[int]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | -### spacy.EmptyKB.v1 {#EmptyKB} +### spacy.EmptyKB.v1 {id="EmptyKB"} A function that creates an empty `KnowledgeBase` from a [`Vocab`](/api/vocab) instance. This is the default when a new entity linker component is created. @@ -908,7 +908,7 @@ instance. This is the default when a new entity linker component is created. | ---------------------- | ----------------------------------------------------------------------------------- | | `entity_vector_length` | The length of the vectors encoding each entity in the KB. Defaults to `64`. ~~int~~ | -### spacy.KBFromFile.v1 {#KBFromFile} +### spacy.KBFromFile.v1 {id="KBFromFile"} A function that reads an existing `KnowledgeBase` from file. @@ -916,7 +916,7 @@ A function that reads an existing `KnowledgeBase` from file. | --------- | -------------------------------------------------------- | | `kb_path` | The location of the KB that was stored to file. ~~Path~~ | -### spacy.CandidateGenerator.v1 {#CandidateGenerator} +### spacy.CandidateGenerator.v1 {id="CandidateGenerator"} A function that takes as input a [`KnowledgeBase`](/api/kb) and a [`Span`](/api/span) object denoting a named entity, and returns a list of @@ -924,7 +924,7 @@ plausible [`Candidate`](/api/kb/#candidate) objects. The default `CandidateGenerator` uses the text of a mention to find its potential aliases in the `KnowledgeBase`. Note that this function is case-dependent. -## Coreference {#coref-architectures tag="experimental"} +## Coreference {id="coref-architectures",tag="experimental"} A [`CoreferenceResolver`](/api/coref) component identifies tokens that refer to the same entity. A [`SpanResolver`](/api/span-resolver) component infers spans @@ -932,7 +932,7 @@ from single tokens. Together these components can be used to reproduce traditional coreference models. You can also omit the `SpanResolver` if working with only token-level clusters is acceptable. -### spacy-experimental.Coref.v1 {#Coref tag="experimental"} +### spacy-experimental.Coref.v1 {id="Coref",tag="experimental"} > #### Example Config > @@ -967,7 +967,7 @@ The `Coref` model architecture is a Thinc `Model`. | `antecedent_batch_size` | Internal batch size. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | -### spacy-experimental.SpanResolver.v1 {#SpanResolver tag="experimental"} +### spacy-experimental.SpanResolver.v1 {id="SpanResolver",tag="experimental"} > #### Example Config > diff --git a/website/docs/api/attributeruler.md b/website/docs/api/attributeruler.mdx similarity index 94% rename from website/docs/api/attributeruler.md rename to website/docs/api/attributeruler.mdx index 965bffbcc..c18319187 100644 --- a/website/docs/api/attributeruler.md +++ b/website/docs/api/attributeruler.mdx @@ -2,7 +2,7 @@ title: AttributeRuler tag: class source: spacy/pipeline/attributeruler.py -new: 3 +version: 3 teaser: 'Pipeline component for rule-based token attribute assignment' api_string_name: attribute_ruler api_trainable: false @@ -15,7 +15,7 @@ between attributes such as mapping fine-grained POS tags to coarse-grained POS tags. See the [usage guide](/usage/linguistic-features/#mappings-exceptions) for examples. -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -37,7 +37,7 @@ how the component should be configured. You can override its settings via the %%GITHUB_SPACY/spacy/pipeline/attributeruler.py ``` -## AttributeRuler.\_\_init\_\_ {#init tag="method"} +## AttributeRuler.\_\_init\_\_ {id="init",tag="method"} Initialize the attribute ruler. @@ -56,7 +56,7 @@ Initialize the attribute ruler. | `validate` | Whether patterns should be validated (passed to the [`Matcher`](/api/matcher#init)). Defaults to `False`. ~~bool~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attributes `"tag`", `"pos"`, `"morph"` and `"lemma"` and [`Scorer.score_token_attr_per_feat`](/api/scorer#score_token_attr_per_feat) for the attribute `"morph"`. ~~Optional[Callable]~~ | -## AttributeRuler.\_\_call\_\_ {#call tag="method"} +## AttributeRuler.\_\_call\_\_ {id="call",tag="method"} Apply the attribute ruler to a `Doc`, setting token attributes for tokens matched by the provided patterns. @@ -66,7 +66,7 @@ matched by the provided patterns. | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## AttributeRuler.add {#add tag="method"} +## AttributeRuler.add {id="add",tag="method"} Add patterns to the attribute ruler. The patterns are a list of `Matcher` patterns and the attributes are a dict of attributes to set on the matched @@ -89,7 +89,7 @@ may be negative to index from the end of the span. | `attrs` | The attributes to assign to the target token in the matched span. ~~Dict[str, Any]~~ | | `index` | The index of the token in the matched span to modify. May be negative to index from the end of the span. Defaults to `0`. ~~int~~ | -## AttributeRuler.add_patterns {#add_patterns tag="method"} +## AttributeRuler.add_patterns {id="add_patterns",tag="method"} > #### Example > @@ -116,7 +116,7 @@ keys `"patterns"`, `"attrs"` and `"index"`, which match the arguments of | ---------- | -------------------------------------------------------------------------- | | `patterns` | The patterns to add. ~~Iterable[Dict[str, Union[List[dict], dict, int]]]~~ | -## AttributeRuler.patterns {#patterns tag="property"} +## AttributeRuler.patterns {id="patterns",tag="property"} Get all patterns that have been added to the attribute ruler in the `patterns_dict` format accepted by @@ -126,7 +126,7 @@ Get all patterns that have been added to the attribute ruler in the | ----------- | -------------------------------------------------------------------------------------------- | | **RETURNS** | The patterns added to the attribute ruler. ~~List[Dict[str, Union[List[dict], dict, int]]]~~ | -## AttributeRuler.initialize {#initialize tag="method"} +## AttributeRuler.initialize {id="initialize",tag="method"} Initialize the component with data and used before training to load in rules from a file. This method is typically called by @@ -160,7 +160,7 @@ config. | `tag_map` | The tag map that maps fine-grained tags to coarse-grained tags and morphological features. Defaults to `None`. ~~Optional[Dict[str, Dict[Union[int, str], Union[int, str]]]]~~ | | `morph_rules` | The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Dict[Union[int, str], Union[int, str]]]]]~~ | -## AttributeRuler.load_from_tag_map {#load_from_tag_map tag="method"} +## AttributeRuler.load_from_tag_map {id="load_from_tag_map",tag="method"} Load attribute ruler patterns from a tag map. @@ -168,7 +168,7 @@ Load attribute ruler patterns from a tag map. | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `tag_map` | The tag map that maps fine-grained tags to coarse-grained tags and morphological features. ~~Dict[str, Dict[Union[int, str], Union[int, str]]]~~ | -## AttributeRuler.load_from_morph_rules {#load_from_morph_rules tag="method"} +## AttributeRuler.load_from_morph_rules {id="load_from_morph_rules",tag="method"} Load attribute ruler patterns from morph rules. @@ -176,7 +176,7 @@ Load attribute ruler patterns from morph rules. | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `morph_rules` | The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. ~~Dict[str, Dict[str, Dict[Union[int, str], Union[int, str]]]]~~ | -## AttributeRuler.to_disk {#to_disk tag="method"} +## AttributeRuler.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -193,7 +193,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## AttributeRuler.from_disk {#from_disk tag="method"} +## AttributeRuler.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -211,7 +211,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `AttributeRuler` object. ~~AttributeRuler~~ | -## AttributeRuler.to_bytes {#to_bytes tag="method"} +## AttributeRuler.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -228,7 +228,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `AttributeRuler` object. ~~bytes~~ | -## AttributeRuler.from_bytes {#from_bytes tag="method"} +## AttributeRuler.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -247,7 +247,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `AttributeRuler` object. ~~AttributeRuler~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/attributes.md b/website/docs/api/attributes.mdx similarity index 98% rename from website/docs/api/attributes.md rename to website/docs/api/attributes.mdx index adacd3898..3142b741d 100644 --- a/website/docs/api/attributes.md +++ b/website/docs/api/attributes.mdx @@ -41,10 +41,9 @@ from string attribute names to internal attribute IDs is stored in The corresponding [`Token` object attributes](/api/token#attributes) can be accessed using the same names in lowercase, e.g. `token.orth` or `token.length`. -For attributes that represent string values, the internal integer ID is -accessed as `Token.attr`, e.g. `token.dep`, while the string value can be -retrieved by appending `_` as in `token.dep_`. - +For attributes that represent string values, the internal integer ID is accessed +as `Token.attr`, e.g. `token.dep`, while the string value can be retrieved by +appending `_` as in `token.dep_`. | Attribute | Description | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/website/docs/api/cli.md b/website/docs/api/cli.mdx similarity index 97% rename from website/docs/api/cli.md rename to website/docs/api/cli.mdx index 275e37ee0..8b84a02ff 100644 --- a/website/docs/api/cli.md +++ b/website/docs/api/cli.mdx @@ -26,7 +26,7 @@ a list of available commands, you can type `python -m spacy --help`. You can also add the `--help` flag to any command or subcommand to see the description, available arguments and usage. -## download {#download tag="command"} +## download {id="download",tag="command"} Download [trained pipelines](/usage/models) for spaCy. The downloader finds the best-matching compatible version and uses `pip install` to download the Python @@ -44,7 +44,7 @@ pipeline name to be specified with its version (e.g. `en_core_web_sm-3.0.0`). > will also allow you to add it as a versioned package dependency to your > project. -```cli +```bash $ python -m spacy download [model] [--direct] [--sdist] [pip_args] ``` @@ -57,24 +57,24 @@ $ python -m spacy download [model] [--direct] [--sdist] [pip_args] | pip args | Additional installation options to be passed to `pip install` when installing the pipeline package. For example, `--user` to install to the user home directory or `--no-deps` to not install package dependencies. ~~Any (option/flag)~~ | | **CREATES** | The installed pipeline package in your `site-packages` directory. | -## info {#info tag="command"} +## info {id="info",tag="command"} Print information about your spaCy installation, trained pipelines and local setup, and generate [Markdown](https://en.wikipedia.org/wiki/Markdown)-formatted markup to copy-paste into [GitHub issues](https://github.com/explosion/spaCy/issues). -```cli +```bash $ python -m spacy info [--markdown] [--silent] [--exclude] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy info en_core_web_lg --markdown > ``` -```cli +```bash $ python -m spacy info [model] [--markdown] [--silent] [--exclude] ``` @@ -88,7 +88,7 @@ $ python -m spacy info [model] [--markdown] [--silent] [--exclude] | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **PRINTS** | Information about your spaCy installation. | -## validate {#validate new="2" tag="command"} +## validate {id="validate",version="2",tag="command"} Find all trained pipeline packages installed in the current environment and check whether they are compatible with the currently installed version of spaCy. @@ -103,7 +103,7 @@ compatible versions and command for updating are shown. > suite, to ensure all packages are up to date before proceeding. If > incompatible packages are found, it will return `1`. -```cli +```bash $ python -m spacy validate ``` @@ -111,12 +111,12 @@ $ python -m spacy validate | ---------- | -------------------------------------------------------------------- | | **PRINTS** | Details about the compatibility of your installed pipeline packages. | -## init {#init new="3"} +## init {id="init",version="3"} The `spacy init` CLI includes helpful commands for initializing training config files and pipeline directories. -### init config {#init-config new="3" tag="command"} +### init config {id="init-config",version="3",tag="command"} Initialize and save a [`config.cfg` file](/usage/training#config) using the **recommended settings** for your use case. It works just like the @@ -128,11 +128,11 @@ customize those settings in your config file later. > #### Example > -> ```cli +> ```bash > $ python -m spacy init config config.cfg --lang en --pipeline ner,textcat --optimize accuracy > ``` -```cli +```bash $ python -m spacy init config [output_file] [--lang] [--pipeline] [--optimize] [--gpu] [--pretraining] [--force] ``` @@ -148,7 +148,7 @@ $ python -m spacy init config [output_file] [--lang] [--pipeline] [--optimize] [ | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | The config file for training. | -### init fill-config {#init-fill-config new="3"} +### init fill-config {id="init-fill-config",version="3"} Auto-fill a partial [.cfg file](/usage/training#config) with **all default values**, e.g. a config generated with the @@ -162,15 +162,15 @@ validation error with more details. > #### Example > -> ```cli +> ```bash > $ python -m spacy init fill-config base.cfg config.cfg --diff > ``` > > #### Example diff > -> ![Screenshot of visual diff in terminal](../images/cli_init_fill-config_diff.jpg) +> ![Screenshot of visual diff in terminal](/images/cli_init_fill-config_diff.jpg) -```cli +```bash $ python -m spacy init fill-config [base_path] [output_file] [--diff] ``` @@ -184,7 +184,7 @@ $ python -m spacy init fill-config [base_path] [output_file] [--diff] | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | Complete and auto-filled config file for training. | -### init vectors {#init-vectors new="3" tag="command"} +### init vectors {id="init-vectors",version="3",tag="command"} Convert [word vectors](/usage/linguistic-features#vectors-similarity) for use with spaCy. Will export an `nlp` object that you can use in the @@ -199,7 +199,7 @@ This functionality was previously available as part of the command `init-model`. -```cli +```bash $ python -m spacy init vectors [lang] [vectors_loc] [output_dir] [--prune] [--truncate] [--name] [--verbose] ``` @@ -216,7 +216,7 @@ $ python -m spacy init vectors [lang] [vectors_loc] [output_dir] [--prune] [--tr | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | A spaCy pipeline directory containing the vocab and vectors. | -### init labels {#init-labels new="3" tag="command"} +### init labels {id="init-labels",version="3",tag="command"} Generate JSON files for the labels in the data. This helps speed up the training process, since spaCy won't have to preprocess the data to extract the labels. @@ -234,7 +234,7 @@ After generating the labels, you can provide them to components that accept a > path = "corpus/labels/ner.json > ``` -```cli +```bash $ python -m spacy init labels [config_path] [output_path] [--code] [--verbose] [--gpu-id] [overrides] ``` @@ -249,7 +249,7 @@ $ python -m spacy init labels [config_path] [output_path] [--code] [--verbose] [ | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ | | **CREATES** | The label files. | -## convert {#convert tag="command"} +## convert {id="convert",tag="command"} Convert files into spaCy's [binary training data format](/api/data-formats#binary-training), a serialized @@ -257,7 +257,7 @@ Convert files into spaCy's management functions. The converter can be specified on the command line, or chosen based on the file extension of the input file. -```cli +```bash $ python -m spacy convert [input_file] [output_dir] [--converter] [--file-type] [--n-sents] [--seg-sents] [--base] [--morphology] [--merge-subtokens] [--ner-map] [--lang] ``` @@ -278,7 +278,7 @@ $ python -m spacy convert [input_file] [output_dir] [--converter] [--file-type] | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | Binary [`DocBin`](/api/docbin) training data that can be used with [`spacy train`](/api/cli#train). | -### Converters {#converters} +### Converters {id="converters"} | ID | Description | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -288,12 +288,12 @@ $ python -m spacy convert [input_file] [output_dir] [--converter] [--file-type] | `ner` / `conll` | NER with IOB/IOB2/BILUO tags, one token per line with columns separated by whitespace. The first column is the token and the final column is the NER tag. Sentences are separated by blank lines and documents are separated by the line `-DOCSTART- -X- O O`. Supports CoNLL 2003 NER format. See [sample data](%%GITHUB_SPACY/extra/example_data/ner_example_data). | | `iob` | NER with IOB/IOB2/BILUO tags, one sentence per line with tokens separated by whitespace and annotation separated by `\|`, either `word\|B-ENT`or`word\|POS\|B-ENT`. See [sample data](%%GITHUB_SPACY/extra/example_data/ner_example_data). | -## debug {#debug new="3"} +## debug {id="debug",version="3"} The `spacy debug` CLI includes helpful commands for debugging and profiling your configs, data and implementations. -### debug config {#debug-config new="3" tag="command"} +### debug config {id="debug-config",version="3",tag="command"} Debug a [`config.cfg` file](/usage/training#config) and show validation errors. The command will create all objects in the tree and validate them. Note that @@ -303,13 +303,13 @@ errors at once and some issues are only shown once previous errors have been fixed. To auto-fill a partial config and save the result, you can use the [`init fill-config`](/api/cli#init-fill-config) command. -```cli +```bash $ python -m spacy debug config [config_path] [--code] [--show-functions] [--show-variables] [overrides] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy debug config config.cfg > ``` @@ -333,7 +333,7 @@ python -m spacy init fill-config tmp/starter-config_invalid.cfg tmp/starter-conf -```cli +```bash $ python -m spacy debug config ./config.cfg --show-functions --show-variables ``` @@ -453,7 +453,7 @@ File /path/to/thinc/thinc/schedules.py (line 91) | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ | | **PRINTS** | Config validation errors, if available. | -### debug data {#debug-data tag="command"} +### debug data {id="debug-data",tag="command"} Analyze, debug and validate your training and development data. Get useful stats, and find problems like invalid entity annotations, cyclic dependencies, @@ -479,13 +479,13 @@ the token distributions. To learn more, you can check out Papay et al.'s work on -```cli +```bash $ python -m spacy debug data [config_path] [--code] [--ignore-warnings] [--verbose] [--no-format] [overrides] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy debug data ./config.cfg > ``` @@ -639,7 +639,7 @@ will not be available. | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ | | **PRINTS** | Debugging information. | -### debug diff-config {#debug-diff tag="command"} +### debug diff-config {id="debug-diff",tag="command"} Show a diff of a config file with respect to spaCy's defaults or another config file. If additional settings were used in the creation of the config file, then @@ -647,13 +647,13 @@ you must supply these as extra parameters to the command when comparing to the default settings. The generated diff can also be used when posting to the discussion forum to provide more information for the maintainers. -```cli +```bash $ python -m spacy debug diff-config [config_path] [--compare-to] [--optimize] [--gpu] [--pretraining] [--markdown] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy debug diff-config ./config.cfg > ``` @@ -868,7 +868,7 @@ after_init = null | `markdown`, `-md` | Generate Markdown for Github issues. Defaults to `False`. ~~bool (flag)~~ | | **PRINTS** | Diff between the two config files. | -### debug profile {#debug-profile tag="command"} +### debug profile {id="debug-profile",tag="command"} Profile which functions take the most time in a spaCy pipeline. Input should be formatted as one JSON object per line with a key `"text"`. It can either be @@ -882,7 +882,7 @@ The `profile` command is now available as a subcommand of `spacy debug`. -```cli +```bash $ python -m spacy debug profile [model] [inputs] [--n-texts] ``` @@ -894,12 +894,12 @@ $ python -m spacy debug profile [model] [inputs] [--n-texts] | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **PRINTS** | Profiling information for the pipeline. | -### debug model {#debug-model new="3" tag="command"} +### debug model {id="debug-model",version="3",tag="command"} Debug a Thinc [`Model`](https://thinc.ai/docs/api-model) by running it on a sample text and checking how it updates its internal weights and parameters. -```cli +```bash $ python -m spacy debug model [config_path] [component] [--layers] [--dimensions] [--parameters] [--gradients] [--attributes] [--print-step0] [--print-step1] [--print-step2] [--print-step3] [--gpu-id] ``` @@ -910,7 +910,7 @@ model ("Step 0"), which helps us to understand the internal structure of the Neural Network, and to focus on specific layers that we want to inspect further (see next example). -```cli +```bash $ python -m spacy debug model ./config.cfg tagger -P0 ``` @@ -956,7 +956,7 @@ an all-zero matrix determined by the `nO` and `nI` dimensions. After a first training step (Step 2), this matrix has clearly updated its values through the training feedback loop. -```cli +```bash $ python -m spacy debug model ./config.cfg tagger -l "5,15" -DIM -PAR -P0 -P1 -P2 ``` @@ -1017,7 +1017,7 @@ $ python -m spacy debug model ./config.cfg tagger -l "5,15" -DIM -PAR -P0 -P1 -P | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ | | **PRINTS** | Debugging information. | -## train {#train tag="command"} +## train {id="train",tag="command"} Train a pipeline. Expects data in spaCy's [binary format](/api/data-formats#training) and a @@ -1043,11 +1043,11 @@ in the section `[paths]`. > #### Example > -> ```cli +> ```bash > $ python -m spacy train config.cfg --output ./output --paths.train ./train --paths.dev ./dev > ``` -```cli +```bash $ python -m spacy train [config_path] [--output] [--code] [--verbose] [--gpu-id] [overrides] ``` @@ -1062,7 +1062,7 @@ $ python -m spacy train [config_path] [--output] [--code] [--verbose] [--gpu-id] | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.train ./train.spacy`. ~~Any (option/flag)~~ | | **CREATES** | The final trained pipeline and the best trained pipeline. | -### Calling the training function from Python {#train-function new="3.2"} +### Calling the training function from Python {id="train-function",version="3.2"} The training CLI exposes a `train` helper function that lets you run the training just like `spacy train`. Usually it's easier to use the command line @@ -1085,7 +1085,7 @@ directly, but if you need to kick off training from code this is how to do it. | `use_gpu` | Which GPU to use. Defaults to -1 for no GPU. ~~int~~ | | `overrides` | Values to override config settings. ~~Dict[str, Any]~~ | -## pretrain {#pretrain new="2.1" tag="command,experimental"} +## pretrain {id="pretrain",version="2.1",tag="command,experimental"} Pretrain the "token to vector" ([`Tok2vec`](/api/tok2vec)) layer of pipeline components on raw text, using an approximate language-modeling objective. @@ -1113,11 +1113,11 @@ auto-generated by setting `--pretraining` on > #### Example > -> ```cli +> ```bash > $ python -m spacy pretrain config.cfg ./output_pretrain --paths.raw_text ./data.jsonl > ``` -```cli +```bash $ python -m spacy pretrain [config_path] [output_dir] [--code] [--resume-path] [--epoch-resume] [--gpu-id] [overrides] ``` @@ -1133,7 +1133,7 @@ $ python -m spacy pretrain [config_path] [output_dir] [--code] [--resume-path] [ | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--training.dropout 0.2`. ~~Any (option/flag)~~ | | **CREATES** | The pretrained weights that can be used to initialize `spacy train`. | -## evaluate {#evaluate new="2" tag="command"} +## evaluate {id="evaluate",version="2",tag="command"} Evaluate a trained pipeline. Expects a loadable spaCy pipeline (package name or path) and evaluation data in the @@ -1146,7 +1146,7 @@ skew. To render a sample of dependency parses in a HTML file using the [displaCy visualizations](/usage/visualizers), set as output directory as the `--displacy-path` argument. -```cli +```bash $ python -m spacy evaluate [model] [data_path] [--output] [--code] [--gold-preproc] [--gpu-id] [--displacy-path] [--displacy-limit] ``` @@ -1163,7 +1163,7 @@ $ python -m spacy evaluate [model] [data_path] [--output] [--code] [--gold-prepr | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | Training results and optional metrics and visualizations. | -## apply {#apply new="3.5" tag="command"} +## apply {id="apply", version="3.5", tag="command"} Applies a trained pipeline to data and stores the resulting annotated documents in a `DocBin`. The input can be a single file or a directory. The recognized @@ -1194,7 +1194,8 @@ $ python -m spacy apply [model] [data-path] [output-file] [--code] [--text-key] | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | A `DocBin` with the annotations from the `model` for all the files found in `data-path`. | -## find-threshold {#find-threshold new="3.5" tag="command"} + +## find-threshold {id="find-threshold",version="3.5",tag="command"} Runs prediction trials for a trained model with varying tresholds to maximize the specified metric. The search space for the threshold is traversed linearly @@ -1209,12 +1210,12 @@ be provided. > #### Examples > -> ```cli +> ```bash > # For textcat_multilabel: > $ python -m spacy find-threshold my_nlp data.spacy textcat_multilabel threshold cats_macro_f > ``` > -> ```cli +> ```bash > # For spancat: > $ python -m spacy find-threshold my_nlp data.spacy spancat threshold spans_sc_f > ``` @@ -1233,7 +1234,7 @@ be provided. | `--silent`, `-V`, `-VV` | GPU to use, if any. Defaults to `-1` for CPU. ~~int (option)~~ | | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | -## assemble {#assemble tag="command"} +## assemble {id="assemble",tag="command"} Assemble a pipeline from a config file without additional training. Expects a [config file](/api/data-formats#config) with all settings and hyperparameters. @@ -1243,11 +1244,11 @@ config. > #### Example > -> ```cli +> ```bash > $ python -m spacy assemble config.cfg ./output > ``` -```cli +```bash $ python -m spacy assemble [config_path] [output_dir] [--code] [--verbose] [overrides] ``` @@ -1261,7 +1262,7 @@ $ python -m spacy assemble [config_path] [output_dir] [--code] [--verbose] [over | overrides | Config parameters to override. Should be options starting with `--` that correspond to the config section and value to override, e.g. `--paths.data ./data`. ~~Any (option/flag)~~ | | **CREATES** | The final assembled pipeline. | -## package {#package tag="command"} +## package {id="package",tag="command"} Generate an installable [Python package](/usage/training#models-generating) from an existing pipeline data directory. All data files are copied over. If @@ -1287,13 +1288,13 @@ the sdist and wheel by setting `--build sdist,wheel`. -```cli +```bash $ python -m spacy package [input_dir] [output_dir] [--code] [--meta-path] [--create-meta] [--build] [--name] [--version] [--force] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy package /input /output > $ cd /output/en_pipeline-0.0.0 > $ pip install dist/en_pipeline-0.0.0.tar.gz @@ -1313,13 +1314,13 @@ $ python -m spacy package [input_dir] [output_dir] [--code] [--meta-path] [--cre | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | A Python package containing the spaCy pipeline. | -## project {#project new="3"} +## project {id="project",version="3"} The `spacy project` CLI includes subcommands for working with [spaCy projects](/usage/projects), end-to-end workflows for building and deploying custom spaCy pipelines. -### project clone {#project-clone tag="command"} +### project clone {id="project-clone",tag="command"} Clone a project template from a Git repository. Calls into `git` under the hood and can use the sparse checkout feature if available, so you're only downloading @@ -1328,19 +1329,19 @@ what you need. By default, spaCy's can provide any other repo (public or private) that you have access to using the `--repo` option. -```cli +```bash $ python -m spacy project clone [name] [dest] [--repo] [--branch] [--sparse] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy project clone pipelines/ner_wikiner > ``` > > Clone from custom repo: > -> ```cli +> ```bash > $ python -m spacy project clone template --repo https://github.com/your_org/your_repo > ``` @@ -1354,7 +1355,7 @@ $ python -m spacy project clone [name] [dest] [--repo] [--branch] [--sparse] | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | The cloned [project directory](/usage/projects#project-files). | -### project assets {#project-assets tag="command"} +### project assets {id="project-assets",tag="command"} Fetch project assets like datasets and pretrained weights. Assets are defined in the `assets` section of the [`project.yml`](/usage/projects#project-yml). If a @@ -1365,13 +1366,13 @@ considered "private" and you have to take care of putting them into the destination directory yourself. If a local path is provided, the asset is copied into the current project. -```cli +```bash $ python -m spacy project assets [project_dir] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy project assets [--sparse] > ``` @@ -1382,7 +1383,7 @@ $ python -m spacy project assets [project_dir] | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | Downloaded or copied assets defined in the `project.yml`. | -### project run {#project-run tag="command"} +### project run {id="project-run",tag="command"} Run a named command or workflow defined in the [`project.yml`](/usage/projects#project-yml). If a workflow name is specified, @@ -1391,13 +1392,13 @@ all commands in the workflow are run, in order. If commands define re-run if state has changed. For example, if the input dataset changes, a preprocessing command that depends on those files will be re-run. -```cli +```bash $ python -m spacy project run [subcommand] [project_dir] [--force] [--dry] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy project run train > ``` @@ -1410,7 +1411,7 @@ $ python -m spacy project run [subcommand] [project_dir] [--force] [--dry] | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **EXECUTES** | The command defined in the `project.yml`. | -### project push {#project-push tag="command"} +### project push {id="project-push",tag="command"} Upload all available files or directories listed as in the `outputs` section of commands to a remote storage. Outputs are archived and compressed prior to @@ -1430,13 +1431,13 @@ remote storages, so you can use any protocol that `Pathy` supports, including filesystem, although you may need to install extra dependencies to use certain protocols. -```cli +```bash $ python -m spacy project push [remote] [project_dir] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy project push my_bucket > ``` > @@ -1453,7 +1454,7 @@ $ python -m spacy project push [remote] [project_dir] | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **UPLOADS** | All project outputs that exist and are not already stored in the remote. | -### project pull {#project-pull tag="command"} +### project pull {id="project-pull",tag="command"} Download all files or directories listed as `outputs` for commands, unless they are not already present locally. When searching for files in the remote, `pull` @@ -1475,13 +1476,13 @@ remote storages, so you can use any protocol that `Pathy` supports, including filesystem, although you may need to install extra dependencies to use certain protocols. -```cli +```bash $ python -m spacy project pull [remote] [project_dir] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy project pull my_bucket > ``` > @@ -1498,7 +1499,7 @@ $ python -m spacy project pull [remote] [project_dir] | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **DOWNLOADS** | All project outputs that do not exist locally and can be found in the remote. | -### project document {#project-document tag="command"} +### project document {id="project-document",tag="command"} Auto-generate a pretty Markdown-formatted `README` for your project, based on its [`project.yml`](/usage/projects#project-yml). Will create sections that @@ -1507,13 +1508,13 @@ content will be placed between two hidden markers, so you can add your own custom content before or after the auto-generated documentation. When you re-run the `project document` command, only the auto-generated part is replaced. -```cli +```bash $ python -m spacy project document [project_dir] [--output] [--no-emoji] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy project document --output README.md > ``` @@ -1522,7 +1523,7 @@ $ python -m spacy project document [project_dir] [--output] [--no-emoji] For more examples, see the templates in our [`projects`](https://github.com/explosion/projects) repo. -![Screenshot of auto-generated Markdown Readme](../images/project_document.jpg) +![Screenshot of auto-generated Markdown Readme](/images/project_document.jpg) @@ -1533,7 +1534,7 @@ For more examples, see the templates in our | `--no-emoji`, `-NE` | Don't use emoji in the titles. ~~bool (flag)~~ | | **CREATES** | The Markdown-formatted project documentation. | -### project dvc {#project-dvc tag="command"} +### project dvc {id="project-dvc",tag="command"} Auto-generate [Data Version Control](https://dvc.org) (DVC) config file. Calls [`dvc run`](https://dvc.org/doc/command-reference/run) with `--no-exec` under @@ -1553,13 +1554,13 @@ You'll also need to add the assets you want to track with -```cli +```bash $ python -m spacy project dvc [project_dir] [workflow] [--force] [--verbose] [--quiet] ``` > #### Example > -> ```cli +> ```bash > $ git init > $ dvc init > $ python -m spacy project dvc all @@ -1575,14 +1576,14 @@ $ python -m spacy project dvc [project_dir] [workflow] [--force] [--verbose] [-- | `--help`, `-h` | Show help message and available arguments. ~~bool (flag)~~ | | **CREATES** | A `dvc.yaml` file in the project directory, based on the steps defined in the given workflow. | -## huggingface-hub {#huggingface-hub new="3.1"} +## huggingface-hub {id="huggingface-hub",version="3.1"} The `spacy huggingface-cli` CLI includes commands for uploading your trained spaCy pipelines to the [Hugging Face Hub](https://huggingface.co/). > #### Installation > -> ```cli +> ```bash > $ pip install spacy-huggingface-hub > $ huggingface-cli login > ``` @@ -1596,19 +1597,19 @@ package installed. Installing the package will automatically add the -### huggingface-hub push {#huggingface-hub-push tag="command"} +### huggingface-hub push {id="huggingface-hub-push",tag="command"} Push a spaCy pipeline to the Hugging Face Hub. Expects a `.whl` file packaged with [`spacy package`](/api/cli#package) and `--build wheel`. For more details, see the spaCy project [integration](/usage/projects#huggingface_hub). -```cli +```bash $ python -m spacy huggingface-hub push [whl_path] [--org] [--msg] [--local-repo] [--verbose] ``` > #### Example > -> ```cli +> ```bash > $ python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl > ``` diff --git a/website/docs/api/coref.md b/website/docs/api/coref.mdx similarity index 94% rename from website/docs/api/coref.md rename to website/docs/api/coref.mdx index 8f54422d6..8647f35d1 100644 --- a/website/docs/api/coref.md +++ b/website/docs/api/coref.mdx @@ -34,7 +34,7 @@ same thing. Clusters are represented as SpanGroups that start with a prefix A `CoreferenceResolver` component can be paired with a [`SpanResolver`](/api/span-resolver) to expand single tokens to spans. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Predictions will be saved to `Doc.spans` as a [`SpanGroup`](/api/spangroup). The span key will be a prefix plus a serial number referring to the coreference @@ -47,7 +47,7 @@ parameter. | ------------------------------------------ | ------------------------------------------------------------------------------------------------------- | | `Doc.spans[prefix + "_" + cluster_number]` | One coreference cluster, represented as single-token spans. Cluster numbers start from 1. ~~SpanGroup~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -73,7 +73,7 @@ details on the architectures and their arguments and hyperparameters. | `model` | The [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. Defaults to [Coref](/api/architectures#Coref). ~~Model~~ | | `span_cluster_prefix` | The prefix for the keys for clusters saved to `doc.spans`. Defaults to `coref_clusters`. ~~str~~ | -## CoreferenceResolver.\_\_init\_\_ {#init tag="method"} +## CoreferenceResolver.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -102,7 +102,7 @@ shortcut for this and instantiate the component using its string name and | _keyword-only_ | | | `span_cluster_prefix` | The prefix for the key for saving clusters of spans. ~~bool~~ | -## CoreferenceResolver.\_\_call\_\_ {#call tag="method"} +## CoreferenceResolver.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -125,7 +125,7 @@ and all pipeline components are applied to the `Doc` in order. Both | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## CoreferenceResolver.pipe {#pipe tag="method"} +## CoreferenceResolver.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -148,7 +148,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/coref#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## CoreferenceResolver.initialize {#initialize tag="method"} +## CoreferenceResolver.initialize {id="initialize",tag="method"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -172,7 +172,7 @@ by [`Language.initialize`](/api/language#initialize). | _keyword-only_ | | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | -## CoreferenceResolver.predict {#predict tag="method"} +## CoreferenceResolver.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects, without modifying them. Clusters are returned as a list of `MentionClusters`, one for @@ -192,7 +192,7 @@ to token indices. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The predicted coreference clusters for the `docs`. ~~List[MentionClusters]~~ | -## CoreferenceResolver.set_annotations {#set_annotations tag="method"} +## CoreferenceResolver.set_annotations {id="set_annotations",tag="method"} Modify a batch of documents, saving coreference clusters in `Doc.spans`. @@ -209,7 +209,7 @@ Modify a batch of documents, saving coreference clusters in `Doc.spans`. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `clusters` | The predicted coreference clusters for the `docs`. ~~List[MentionClusters]~~ | -## CoreferenceResolver.update {#update tag="method"} +## CoreferenceResolver.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects. Delegates to [`predict`](/api/coref#predict). @@ -231,7 +231,7 @@ Learn from a batch of [`Example`](/api/example) objects. Delegates to | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## CoreferenceResolver.create_optimizer {#create_optimizer tag="method"} +## CoreferenceResolver.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -246,7 +246,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## CoreferenceResolver.use_params {#use_params tag="method, contextmanager"} +## CoreferenceResolver.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored. @@ -263,7 +263,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## CoreferenceResolver.to_disk {#to_disk tag="method"} +## CoreferenceResolver.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -280,7 +280,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## CoreferenceResolver.from_disk {#from_disk tag="method"} +## CoreferenceResolver.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -298,7 +298,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `CoreferenceResolver` object. ~~CoreferenceResolver~~ | -## CoreferenceResolver.to_bytes {#to_bytes tag="method"} +## CoreferenceResolver.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -315,7 +315,7 @@ Serialize the pipe to a bytestring, including the `KnowledgeBase`. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `CoreferenceResolver` object. ~~bytes~~ | -## CoreferenceResolver.from_bytes {#from_bytes tag="method"} +## CoreferenceResolver.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -334,7 +334,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `CoreferenceResolver` object. ~~CoreferenceResolver~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/corpus.md b/website/docs/api/corpus.mdx similarity index 96% rename from website/docs/api/corpus.md rename to website/docs/api/corpus.mdx index 88c4befd7..c58723e82 100644 --- a/website/docs/api/corpus.md +++ b/website/docs/api/corpus.mdx @@ -3,7 +3,7 @@ title: Corpus teaser: An annotated corpus tag: class source: spacy/training/corpus.py -new: 3 +version: 3 --- This class manages annotated corpora and can be used for training and @@ -13,7 +13,7 @@ customize the data loading during training, you can register your own see the usage guide on [data utilities](/usage/training#data) for more details and examples. -## Config and implementation {#config} +## Config and implementation {id="config"} `spacy.Corpus.v1` is a registered function that creates a `Corpus` of training or evaluation data. It takes the same arguments as the `Corpus` class and @@ -49,7 +49,7 @@ streaming. %%GITHUB_SPACY/spacy/training/corpus.py ``` -## Corpus.\_\_init\_\_ {#init tag="method"} +## Corpus.\_\_init\_\_ {id="init",tag="method"} Create a `Corpus` for iterating [Example](/api/example) objects from a file or directory of [`.spacy` data files](/api/data-formats#binary-training). The @@ -81,7 +81,7 @@ train/test skew. | `augmenter` | Optional data augmentation callback. ~~Callable[[Language, Example], Iterable[Example]]~~ | | `shuffle` | Whether to shuffle the examples. Defaults to `False`. ~~bool~~ | -## Corpus.\_\_call\_\_ {#call tag="method"} +## Corpus.\_\_call\_\_ {id="call",tag="method"} Yield examples from the data. @@ -101,7 +101,7 @@ Yield examples from the data. | `nlp` | The current `nlp` object. ~~Language~~ | | **YIELDS** | The examples. ~~Example~~ | -## JsonlCorpus {#jsonlcorpus tag="class"} +## JsonlCorpus {id="jsonlcorpus",tag="class"} Iterate Doc objects from a file or directory of JSONL (newline-delimited JSON) formatted raw text files. Can be used to read the raw text corpus for language @@ -120,14 +120,13 @@ file. > srsly.write_jsonl("/path/to/text.jsonl", data) > ``` -```json -### Example +```json {title="Example"} {"text": "Can I ask where you work now and what you do, and if you enjoy it?"} {"text": "They may just pull out of the Seattle market completely, at least until they have autonomous vehicles."} {"text": "My cynical view on this is that it will never be free to the public. Reason: what would be the draw of joining the military? Right now their selling point is free Healthcare and Education. Ironically both are run horribly and most, that I've talked to, come out wishing they never went in."} ``` -### JsonlCorpus.\_\init\_\_ {#jsonlcorpus tag="method"} +### JsonlCorpus.\_\_init\_\_ {id="jsonlcorpus",tag="method"} Initialize the reader. @@ -157,7 +156,7 @@ Initialize the reader. | `max_length` | Maximum document length (in tokens). Longer documents will be skipped. Defaults to `0`, which indicates no limit. ~~int~~ | | `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ | -### JsonlCorpus.\_\_call\_\_ {#jsonlcorpus-call tag="method"} +### JsonlCorpus.\_\_call\_\_ {id="jsonlcorpus-call",tag="method"} Yield examples from the data. diff --git a/website/docs/api/cython-classes.md b/website/docs/api/cython-classes.mdx similarity index 91% rename from website/docs/api/cython-classes.md rename to website/docs/api/cython-classes.mdx index a4ecf294a..ce7c03940 100644 --- a/website/docs/api/cython-classes.md +++ b/website/docs/api/cython-classes.mdx @@ -9,7 +9,7 @@ menu: - ['StringStore', 'stringstore'] --- -## Doc {#doc tag="cdef class" source="spacy/tokens/doc.pxd"} +## Doc {id="doc",tag="cdef class",source="spacy/tokens/doc.pxd"} The `Doc` object holds an array of [`TokenC`](/api/cython-structs#tokenc) structs. @@ -21,7 +21,7 @@ accessed from Python. For the Python documentation, see [`Doc`](/api/doc). -### Attributes {#doc_attributes} +### Attributes {id="doc_attributes"} | Name | Description | | ------------ | -------------------------------------------------------------------------------------------------------- | @@ -31,7 +31,7 @@ accessed from Python. For the Python documentation, see [`Doc`](/api/doc). | `length` | The number of tokens in the document. ~~int~~ | | `max_length` | The underlying size of the `Doc.c` array. ~~int~~ | -### Doc.push_back {#doc_push_back tag="method"} +### Doc.push_back {id="doc_push_back",tag="method"} Append a token to the `Doc`. The token can be provided as a [`LexemeC`](/api/cython-structs#lexemec) or @@ -55,7 +55,7 @@ Append a token to the `Doc`. The token can be provided as a | `lex_or_tok` | The word to append to the `Doc`. ~~LexemeOrToken~~ | | `has_space` | Whether the word has trailing whitespace. ~~bint~~ | -## Token {#token tag="cdef class" source="spacy/tokens/token.pxd"} +## Token {id="token",tag="cdef class",source="spacy/tokens/token.pxd"} A Cython class providing access and methods for a [`TokenC`](/api/cython-structs#tokenc) struct. Note that the `Token` object does @@ -68,7 +68,7 @@ accessed from Python. For the Python documentation, see [`Token`](/api/token). -### Attributes {#token_attributes} +### Attributes {id="token_attributes"} | Name | Description | | ------- | -------------------------------------------------------------------------- | @@ -77,7 +77,7 @@ accessed from Python. For the Python documentation, see [`Token`](/api/token). | `i` | The offset of the token within the document. ~~int~~ | | `doc` | The parent document. ~~Doc~~ | -### Token.cinit {#token_cinit tag="method"} +### Token.cinit {id="token_cinit",tag="method"} Create a `Token` object from a `TokenC*` pointer. @@ -94,7 +94,7 @@ Create a `Token` object from a `TokenC*` pointer. | `offset` | The offset of the token within the document. ~~int~~ | | `doc` | The parent document. ~~int~~ | -## Span {#span tag="cdef class" source="spacy/tokens/span.pxd"} +## Span {id="span",tag="cdef class",source="spacy/tokens/span.pxd"} A Cython class providing access and methods for a slice of a `Doc` object. @@ -105,7 +105,7 @@ accessed from Python. For the Python documentation, see [`Span`](/api/span). -### Attributes {#span_attributes} +### Attributes {id="span_attributes"} | Name | Description | | ------------ | ----------------------------------------------------------------------------- | @@ -116,7 +116,7 @@ accessed from Python. For the Python documentation, see [`Span`](/api/span). | `end_char` | The index of the last character of the span. ~~int~~ | | `label` | A label to attach to the span, e.g. for named entities. ~~attr_t (uint64_t)~~ | -## Lexeme {#lexeme tag="cdef class" source="spacy/lexeme.pxd"} +## Lexeme {id="lexeme",tag="cdef class",source="spacy/lexeme.pxd"} A Cython class providing access and methods for an entry in the vocabulary. @@ -127,7 +127,7 @@ accessed from Python. For the Python documentation, see [`Lexeme`](/api/lexeme). -### Attributes {#lexeme_attributes} +### Attributes {id="lexeme_attributes"} | Name | Description | | ------- | ----------------------------------------------------------------------------- | @@ -135,7 +135,7 @@ accessed from Python. For the Python documentation, see [`Lexeme`](/api/lexeme). | `vocab` | A reference to the shared `Vocab` object. ~~Vocab~~ | | `orth` | ID of the verbatim text content. ~~attr_t (uint64_t)~~ | -## Vocab {#vocab tag="cdef class" source="spacy/vocab.pxd"} +## Vocab {id="vocab",tag="cdef class",source="spacy/vocab.pxd"} A Cython class providing access and methods for a vocabulary and other data shared across a language. @@ -147,7 +147,7 @@ accessed from Python. For the Python documentation, see [`Vocab`](/api/vocab). -### Attributes {#vocab_attributes} +### Attributes {id="vocab_attributes"} | Name | Description | | --------- | ---------------------------------------------------------------------------------------------------------- | @@ -155,7 +155,7 @@ accessed from Python. For the Python documentation, see [`Vocab`](/api/vocab). | `strings` | A `StringStore` that maps string to hash values and vice versa. ~~StringStore~~ | | `length` | The number of entries in the vocabulary. ~~int~~ | -### Vocab.get {#vocab_get tag="method"} +### Vocab.get {id="vocab_get",tag="method"} Retrieve a [`LexemeC*`](/api/cython-structs#lexemec) pointer from the vocabulary. @@ -172,7 +172,7 @@ vocabulary. | `string` | The string of the word to look up. ~~str~~ | | **RETURNS** | The lexeme in the vocabulary. ~~const LexemeC\*~~ | -### Vocab.get_by_orth {#vocab_get_by_orth tag="method"} +### Vocab.get_by_orth {id="vocab_get_by_orth",tag="method"} Retrieve a [`LexemeC*`](/api/cython-structs#lexemec) pointer from the vocabulary. @@ -189,7 +189,7 @@ vocabulary. | `orth` | ID of the verbatim text content. ~~attr_t (uint64_t)~~ | | **RETURNS** | The lexeme in the vocabulary. ~~const LexemeC\*~~ | -## StringStore {#stringstore tag="cdef class" source="spacy/strings.pxd"} +## StringStore {id="stringstore",tag="cdef class",source="spacy/strings.pxd"} A lookup table to retrieve strings by 64-bit hashes. @@ -201,7 +201,7 @@ accessed from Python. For the Python documentation, see -### Attributes {#stringstore_attributes} +### Attributes {id="stringstore_attributes"} | Name | Description | | ------ | ---------------------------------------------------------------------------------------------------------------- | diff --git a/website/docs/api/cython-structs.md b/website/docs/api/cython-structs.mdx similarity index 94% rename from website/docs/api/cython-structs.md rename to website/docs/api/cython-structs.mdx index 4c8514b64..106a27e90 100644 --- a/website/docs/api/cython-structs.md +++ b/website/docs/api/cython-structs.mdx @@ -7,7 +7,7 @@ menu: - ['LexemeC', 'lexemec'] --- -## TokenC {#tokenc tag="C struct" source="spacy/structs.pxd"} +## TokenC {id="tokenc",tag="C struct",source="spacy/structs.pxd"} Cython data container for the `Token` object. @@ -39,7 +39,7 @@ Cython data container for the `Token` object. | `ent_type` | Named entity type. ~~attr_t (uint64_t)~~ | | `ent_id` | ID of the entity the token is an instance of, if any. Currently not used, but potentially for coreference resolution. ~~attr_t (uint64_t)~~ | -### Token.get_struct_attr {#token_get_struct_attr tag="staticmethod, nogil" source="spacy/tokens/token.pxd"} +### Token.get_struct_attr {id="token_get_struct_attr",tag="staticmethod, nogil",source="spacy/tokens/token.pxd"} Get the value of an attribute from the `TokenC` struct by attribute ID. @@ -58,7 +58,7 @@ Get the value of an attribute from the `TokenC` struct by attribute ID. | `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ | | **RETURNS** | The value of the attribute. ~~attr_t (uint64_t)~~ | -### Token.set_struct_attr {#token_set_struct_attr tag="staticmethod, nogil" source="spacy/tokens/token.pxd"} +### Token.set_struct_attr {id="token_set_struct_attr",tag="staticmethod, nogil",source="spacy/tokens/token.pxd"} Set the value of an attribute of the `TokenC` struct by attribute ID. @@ -78,7 +78,7 @@ Set the value of an attribute of the `TokenC` struct by attribute ID. | `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ | | `value` | The value to set. ~~attr_t (uint64_t)~~ | -### token_by_start {#token_by_start tag="function" source="spacy/tokens/doc.pxd"} +### token_by_start {id="token_by_start",tag="function",source="spacy/tokens/doc.pxd"} Find a token in a `TokenC*` array by the offset of its first character. @@ -100,7 +100,7 @@ Find a token in a `TokenC*` array by the offset of its first character. | `start_char` | The start index to search for. ~~int~~ | | **RETURNS** | The index of the token in the array or `-1` if not found. ~~int~~ | -### token_by_end {#token_by_end tag="function" source="spacy/tokens/doc.pxd"} +### token_by_end {id="token_by_end",tag="function",source="spacy/tokens/doc.pxd"} Find a token in a `TokenC*` array by the offset of its final character. @@ -122,7 +122,7 @@ Find a token in a `TokenC*` array by the offset of its final character. | `end_char` | The end index to search for. ~~int~~ | | **RETURNS** | The index of the token in the array or `-1` if not found. ~~int~~ | -### set_children_from_heads {#set_children_from_heads tag="function" source="spacy/tokens/doc.pxd"} +### set_children_from_heads {id="set_children_from_heads",tag="function",source="spacy/tokens/doc.pxd"} Set attributes that allow lookup of syntactic children on a `TokenC*` array. This function must be called after making changes to the `TokenC.head` @@ -148,7 +148,7 @@ attribute, in order to make the parse tree navigation consistent. | `tokens` | A `TokenC*` array. ~~const TokenC\*~~ | | `length` | The number of tokens in the array. ~~int~~ | -## LexemeC {#lexemec tag="C struct" source="spacy/structs.pxd"} +## LexemeC {id="lexemec",tag="C struct",source="spacy/structs.pxd"} Struct holding information about a lexical type. `LexemeC` structs are usually owned by the `Vocab`, and accessed through a read-only pointer on the `TokenC` @@ -172,7 +172,7 @@ struct. | `prefix` | Length-N substring from the start of the lexeme. Defaults to `N=1`. ~~attr_t (uint64_t)~~ | | `suffix` | Length-N substring from the end of the lexeme. Defaults to `N=3`. ~~attr_t (uint64_t)~~ | -### Lexeme.get_struct_attr {#lexeme_get_struct_attr tag="staticmethod, nogil" source="spacy/lexeme.pxd"} +### Lexeme.get_struct_attr {id="lexeme_get_struct_attr",tag="staticmethod, nogil",source="spacy/lexeme.pxd"} Get the value of an attribute from the `LexemeC` struct by attribute ID. @@ -192,7 +192,7 @@ Get the value of an attribute from the `LexemeC` struct by attribute ID. | `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ | | **RETURNS** | The value of the attribute. ~~attr_t (uint64_t)~~ | -### Lexeme.set_struct_attr {#lexeme_set_struct_attr tag="staticmethod, nogil" source="spacy/lexeme.pxd"} +### Lexeme.set_struct_attr {id="lexeme_set_struct_attr",tag="staticmethod, nogil",source="spacy/lexeme.pxd"} Set the value of an attribute of the `LexemeC` struct by attribute ID. @@ -212,7 +212,7 @@ Set the value of an attribute of the `LexemeC` struct by attribute ID. | `feat_name` | The ID of the attribute to look up. The attributes are enumerated in `spacy.typedefs`. ~~attr_id_t~~ | | `value` | The value to set. ~~attr_t (uint64_t)~~ | -### Lexeme.c_check_flag {#lexeme_c_check_flag tag="staticmethod, nogil" source="spacy/lexeme.pxd"} +### Lexeme.c_check_flag {id="lexeme_c_check_flag",tag="staticmethod, nogil",source="spacy/lexeme.pxd"} Check the value of a binary flag attribute. @@ -232,7 +232,7 @@ Check the value of a binary flag attribute. | `flag_id` | The ID of the flag to look up. The flag IDs are enumerated in `spacy.typedefs`. ~~attr_id_t~~ | | **RETURNS** | The boolean value of the flag. ~~bint~~ | -### Lexeme.c_set_flag {#lexeme_c_set_flag tag="staticmethod, nogil" source="spacy/lexeme.pxd"} +### Lexeme.c_set_flag {id="lexeme_c_set_flag",tag="staticmethod, nogil",source="spacy/lexeme.pxd"} Set the value of a binary flag attribute. diff --git a/website/docs/api/cython.md b/website/docs/api/cython.mdx similarity index 99% rename from website/docs/api/cython.md rename to website/docs/api/cython.mdx index 16b11cead..764ff10f4 100644 --- a/website/docs/api/cython.md +++ b/website/docs/api/cython.mdx @@ -6,7 +6,7 @@ menu: - ['Conventions', 'conventions'] --- -## Overview {#overview hidden="true"} +## Overview {id="overview",hidden="true"} > #### What's Cython? > @@ -37,7 +37,7 @@ class holds a [`LexemeC`](/api/cython-structs#lexemec) struct, at `Lexeme.c`. This lets you shed the Python container, and pass a pointer to the underlying data into C-level functions. -## Conventions {#conventions} +## Conventions {id="conventions"} spaCy's core data structures are implemented as [Cython](http://cython.org/) `cdef` classes. Memory is managed through the diff --git a/website/docs/api/data-formats.md b/website/docs/api/data-formats.mdx similarity index 98% rename from website/docs/api/data-formats.md rename to website/docs/api/data-formats.mdx index 420e827a0..c9d88f87c 100644 --- a/website/docs/api/data-formats.md +++ b/website/docs/api/data-formats.mdx @@ -14,7 +14,7 @@ vocabulary data. For an overview of label schemes used by the models, see the [models directory](/models). Each trained pipeline documents the label schemes used in its components, depending on the data it was trained on. -## Training config {#config new="3"} +## Training config {id="config",version="3"} Config files define the training process and pipeline and can be passed to [`spacy train`](/api/cli#train). They use @@ -52,7 +52,7 @@ your config and check that it's valid, you can run the -### nlp {#config-nlp tag="section"} +### nlp {id="config-nlp",tag="section"} > #### Example > @@ -83,7 +83,7 @@ Defines the `nlp` object, its tokenizer and | `tokenizer` | The tokenizer to use. Defaults to [`Tokenizer`](/api/tokenizer). ~~Callable[[str], Doc]~~ | | `batch_size` | Default batch size for [`Language.pipe`](/api/language#pipe) and [`Language.evaluate`](/api/language#evaluate). ~~int~~ | -### components {#config-components tag="section"} +### components {id="config-components",tag="section"} > #### Example > @@ -106,7 +106,7 @@ function to use to create component) or a `source` (name of path of trained pipeline to copy components from). See the docs on [defining pipeline components](/usage/training#config-components) for details. -### paths, system {#config-variables tag="variables"} +### paths, system {id="config-variables",tag="variables"} These sections define variables that can be referenced across the other sections as variables. For example `${paths.train}` uses the value of `train` defined in @@ -116,11 +116,11 @@ need paths, you can define them here. All config values can also be [`spacy train`](/api/cli#train), which is especially relevant for data paths that you don't want to hard-code in your config file. -```cli +```bash $ python -m spacy train config.cfg --paths.train ./corpus/train.spacy ``` -### corpora {#config-corpora tag="section"} +### corpora {id="config-corpora",tag="section"} > #### Example > @@ -176,7 +176,7 @@ single corpus once and then divide it up into `train` and `dev` partitions. | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `corpora` | A dictionary keyed by string names, mapped to corpus functions that receive the current `nlp` object and return an iterator of [`Example`](/api/example) objects. ~~Dict[str, Callable[[Language], Iterator[Example]]]~~ | -### training {#config-training tag="section"} +### training {id="config-training",tag="section"} This section defines settings and controls for the training and evaluation process that are used when you run [`spacy train`](/api/cli#train). @@ -202,7 +202,7 @@ process that are used when you run [`spacy train`](/api/cli#train). | `seed` | The random seed. Defaults to variable `${system.seed}`. ~~int~~ | | `train_corpus` | Dot notation of the config location defining the train corpus. Defaults to `corpora.train`. ~~str~~ | -### pretraining {#config-pretraining tag="section,optional"} +### pretraining {id="config-pretraining",tag="section,optional"} This section is optional and defines settings and controls for [language model pretraining](/usage/embeddings-transformers#pretraining). It's @@ -220,7 +220,7 @@ used when you run [`spacy pretrain`](/api/cli#pretrain). | `component` | Component name to identify the layer with the model to pretrain. Defaults to `"tok2vec"`. ~~str~~ | | `layer` | The specific layer of the model to pretrain. If empty, the whole model will be used. ~~str~~ | -### initialize {#config-initialize tag="section"} +### initialize {id="config-initialize",tag="section"} This config block lets you define resources for **initializing the pipeline**. It's used by [`Language.initialize`](/api/language#initialize) and typically @@ -255,9 +255,9 @@ Also see the usage guides on the | `vectors` | Name or path of pipeline containing pretrained word vectors to use, e.g. created with [`init vectors`](/api/cli#init-vectors). Defaults to `null`. ~~Optional[str]~~ | | `vocab_data` | Path to JSONL-formatted [vocabulary file](/api/data-formats#vocab-jsonl) to initialize vocabulary. ~~Optional[str]~~ | -## Training data {#training} +## Training data {id="training"} -### Binary training format {#binary-training new="3"} +### Binary training format {id="binary-training",version="3"} > #### Example > @@ -288,7 +288,7 @@ Note that while this is the format used to save training data, you do not have to understand the internal details to use it or create training data. See the section on [preparing training data](/usage/training#training-data). -### JSON training format {#json-input tag="deprecated"} +### JSON training format {id="json-input",tag="deprecated"} @@ -300,7 +300,7 @@ objects to JSON, you can now serialize them directly using the [`spacy convert`](/api/cli) lets you convert your JSON data to the new `.spacy` format: -```cli +```bash $ python -m spacy convert ./data.json . ``` @@ -317,8 +317,7 @@ $ python -m spacy convert ./data.json . > [`offsets_to_biluo_tags`](/api/top-level#offsets_to_biluo_tags) function can > help you convert entity offsets to the right format. -```python -### Example structure +```python {title="Example structure"} [{ "id": int, # ID of the document within the corpus "paragraphs": [{ # list of paragraphs in the corpus @@ -357,7 +356,7 @@ https://github.com/explosion/spaCy/blob/v2.3.x/examples/training/training-data.j -### Annotation format for creating training examples {#dict-input} +### Annotation format for creating training examples {id="dict-input"} An [`Example`](/api/example) object holds the information for one training instance. It stores two [`Doc`](/api/doc) objects: one for holding the @@ -436,8 +435,7 @@ file to keep track of your settings and hyperparameters and your own -```python -### Examples +```python {title="Examples"} # Training data for a part-of-speech tagger doc = Doc(vocab, words=["I", "like", "stuff"]) gold_dict = {"tags": ["NOUN", "VERB", "NOUN"]} @@ -466,7 +464,7 @@ gold_dict = {"entities": [(0, 12, "PERSON")], example = Example.from_dict(doc, gold_dict) ``` -## Lexical data for vocabulary {#vocab-jsonl new="2"} +## Lexical data for vocabulary {id="vocab-jsonl",version="2"} This data file can be provided via the `vocab_data` setting in the `[initialize]` block of the training config to pre-define the lexical data to @@ -483,13 +481,11 @@ spaCy's [`Lexeme`](/api/lexeme#attributes) object. > vocab_data = "/path/to/vocab-data.jsonl" > ``` -```python -### First line +```python {title="First line"} {"lang": "en", "settings": {"oov_prob": -20.502029418945312}} ``` -```python -### Entry structure +```python {title="Entry structure"} { "orth": string, # the word text "id": int, # can correspond to row in vectors table @@ -526,7 +522,7 @@ Here's an example of the 20 most frequent lexemes in the English training data: %%GITHUB_SPACY/extra/example_data/vocab-data.jsonl ``` -## Pipeline meta {#meta} +## Pipeline meta {id="meta"} The pipeline meta is available as the file `meta.json` and exported automatically when you save an `nlp` object to disk. Its contents are available diff --git a/website/docs/api/dependencymatcher.md b/website/docs/api/dependencymatcher.mdx similarity index 96% rename from website/docs/api/dependencymatcher.md rename to website/docs/api/dependencymatcher.mdx index cae4221bf..390034a6c 100644 --- a/website/docs/api/dependencymatcher.md +++ b/website/docs/api/dependencymatcher.mdx @@ -2,7 +2,7 @@ title: DependencyMatcher teaser: Match subtrees within a dependency parse tag: class -new: 3 +version: 3 source: spacy/matcher/dependencymatcher.pyx --- @@ -14,7 +14,7 @@ It requires a pretrained [`DependencyParser`](/api/parser) or other component that sets the `Token.dep` and `Token.head` attributes. See the [usage guide](/usage/rule-based-matching#dependencymatcher) for examples. -## Pattern format {#patterns} +## Pattern format {id="patterns"} > ```python > ### Example @@ -62,7 +62,7 @@ of relations, see the usage guide on -### Operators {#operators} +### Operators {id="operators"} The following operators are supported by the `DependencyMatcher`, most of which come directly from @@ -87,8 +87,7 @@ come directly from | `A <++ B` | `B` is a right parent of `A`, i.e. `A` is a child of `B` and `A.i < B.i` _(not in Semgrex)_. | | `A <-- B` | `B` is a left parent of `A`, i.e. `A` is a child of `B` and `A.i > B.i` _(not in Semgrex)_. | - -## DependencyMatcher.\_\_init\_\_ {#init tag="method"} +## DependencyMatcher.\_\_init\_\_ {id="init",tag="method"} Create a `DependencyMatcher`. @@ -105,7 +104,7 @@ Create a `DependencyMatcher`. | _keyword-only_ | | | `validate` | Validate all patterns added to this matcher. ~~bool~~ | -## DependencyMatcher.\_\call\_\_ {#call tag="method"} +## DependencyMatcher.\_\_call\_\_ {id="call",tag="method"} Find all tokens matching the supplied patterns on the `Doc` or `Span`. @@ -127,7 +126,7 @@ Find all tokens matching the supplied patterns on the `Doc` or `Span`. | `doclike` | The `Doc` or `Span` to match over. ~~Union[Doc, Span]~~ | | **RETURNS** | A list of `(match_id, token_ids)` tuples, describing the matches. The `match_id` is the ID of the match pattern and `token_ids` is a list of token indices matched by the pattern, where the position of each token in the list corresponds to the position of the node specification in the pattern. ~~List[Tuple[int, List[int]]]~~ | -## DependencyMatcher.\_\_len\_\_ {#len tag="method"} +## DependencyMatcher.\_\_len\_\_ {id="len",tag="method"} Get the number of rules added to the dependency matcher. Note that this only returns the number of rules (identical with the number of IDs), not the number @@ -148,7 +147,7 @@ of individual patterns. | ----------- | ---------------------------- | | **RETURNS** | The number of rules. ~~int~~ | -## DependencyMatcher.\_\_contains\_\_ {#contains tag="method"} +## DependencyMatcher.\_\_contains\_\_ {id="contains",tag="method"} Check whether the matcher contains rules for a match ID. @@ -166,7 +165,7 @@ Check whether the matcher contains rules for a match ID. | `key` | The match ID. ~~str~~ | | **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ | -## DependencyMatcher.add {#add tag="method"} +## DependencyMatcher.add {id="add",tag="method"} Add a rule to the matcher, consisting of an ID key, one or more patterns, and an optional callback function to act on the matches. The callback function will @@ -191,7 +190,7 @@ will be overwritten. | _keyword-only_ | | | `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[DependencyMatcher, Doc, int, List[Tuple], Any]]~~ | -## DependencyMatcher.get {#get tag="method"} +## DependencyMatcher.get {id="get",tag="method"} Retrieve the pattern stored for a key. Returns the rule as an `(on_match, patterns)` tuple containing the callback and available patterns. @@ -208,7 +207,7 @@ Retrieve the pattern stored for a key. Returns the rule as an | `key` | The ID of the match rule. ~~str~~ | | **RETURNS** | The rule, as an `(on_match, patterns)` tuple. ~~Tuple[Optional[Callable], List[List[Union[Dict, Tuple]]]]~~ | -## DependencyMatcher.remove {#remove tag="method"} +## DependencyMatcher.remove {id="remove",tag="method"} Remove a rule from the dependency matcher. A `KeyError` is raised if the match ID does not exist. diff --git a/website/docs/api/dependencyparser.md b/website/docs/api/dependencyparser.mdx similarity index 95% rename from website/docs/api/dependencyparser.md rename to website/docs/api/dependencyparser.mdx index 27e315592..a6bc48cdf 100644 --- a/website/docs/api/dependencyparser.md +++ b/website/docs/api/dependencyparser.mdx @@ -25,7 +25,7 @@ current state. The weights are updated such that the scores assigned to the set of optimal actions is increased, while scores assigned to other actions are decreased. Note that more than one action may be optimal for a given state. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Dependency predictions are assigned to the `Token.dep` and `Token.head` fields. Beside the dependencies themselves, the parser decides sentence boundaries, @@ -39,7 +39,7 @@ which are saved in `Token.is_sent_start` and accessible via `Doc.sents`. | `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. After the parser runs this will be `True` or `False` for all tokens. ~~bool~~ | | `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -74,7 +74,7 @@ architectures and their arguments and hyperparameters. %%GITHUB_SPACY/spacy/pipeline/dep_parser.pyx ``` -## DependencyParser.\_\_init\_\_ {#init tag="method"} +## DependencyParser.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -107,7 +107,7 @@ shortcut for this and instantiate the component using its string name and | `min_action_freq` | The minimum frequency of labelled actions to retain. Rarer labelled actions have their label backed-off to "dep". While this primarily affects the label accuracy, it can also affect the attachment structure, as the labels are used to represent the pseudo-projectivity transformation. ~~int~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_deps`](/api/scorer#score_deps) for the attribute `"dep"` ignoring the labels `p` and `punct` and [`Scorer.score_spans`](/api/scorer/#score_spans) for the attribute `"sents"`. ~~Optional[Callable]~~ | -## DependencyParser.\_\_call\_\_ {#call tag="method"} +## DependencyParser.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -131,7 +131,7 @@ and all pipeline components are applied to the `Doc` in order. Both | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## DependencyParser.pipe {#pipe tag="method"} +## DependencyParser.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -155,7 +155,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/dependencyparser#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## DependencyParser.initialize {#initialize tag="method" new="3"} +## DependencyParser.initialize {id="initialize",tag="method",version="3"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -198,7 +198,7 @@ This method was previously called `begin_training`. | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Dict[str, Dict[str, int]]]~~ | -## DependencyParser.predict {#predict tag="method"} +## DependencyParser.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects, without modifying them. @@ -215,7 +215,7 @@ modifying them. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | A helper class for the parse state (internal). ~~StateClass~~ | -## DependencyParser.set_annotations {#set_annotations tag="method"} +## DependencyParser.set_annotations {id="set_annotations",tag="method"} Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. @@ -232,7 +232,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `scores` | The scores to set, produced by `DependencyParser.predict`. Returns an internal helper class for the parse state. ~~List[StateClass]~~ | -## DependencyParser.update {#update tag="method"} +## DependencyParser.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects, updating the pipe's model. Delegates to [`predict`](/api/dependencyparser#predict) and @@ -255,7 +255,7 @@ model. Delegates to [`predict`](/api/dependencyparser#predict) and | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## DependencyParser.get_loss {#get_loss tag="method"} +## DependencyParser.get_loss {id="get_loss",tag="method"} Find the loss and gradient of loss for the batch of documents and their predicted scores. @@ -274,7 +274,7 @@ predicted scores. | `scores` | Scores representing the model's predictions. ~~StateClass~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | -## DependencyParser.create_optimizer {#create_optimizer tag="method"} +## DependencyParser.create_optimizer {id="create_optimizer",tag="method"} Create an [`Optimizer`](https://thinc.ai/docs/api-optimizers) for the pipeline component. @@ -290,7 +290,7 @@ component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## DependencyParser.use_params {#use_params tag="method, contextmanager"} +## DependencyParser.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored. @@ -307,7 +307,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## DependencyParser.add_label {#add_label tag="method"} +## DependencyParser.add_label {id="add_label",tag="method"} Add a new label to the pipe. Note that you don't have to call this method if you provide a **representative data sample** to the [`initialize`](#initialize) @@ -327,7 +327,7 @@ to the model, and the output dimension will be | `label` | The label to add. ~~str~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | -## DependencyParser.set_output {#set_output tag="method"} +## DependencyParser.set_output {id="set_output",tag="method"} Change the output dimension of the component's model by calling the model's attribute `resize_output`. This is a function that takes the original model and @@ -346,7 +346,7 @@ forgetting" problem. | ---- | --------------------------------- | | `nO` | The new output dimension. ~~int~~ | -## DependencyParser.to_disk {#to_disk tag="method"} +## DependencyParser.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -363,7 +363,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## DependencyParser.from_disk {#from_disk tag="method"} +## DependencyParser.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -381,7 +381,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `DependencyParser` object. ~~DependencyParser~~ | -## DependencyParser.to_bytes {#to_bytes tag="method"} +## DependencyParser.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -398,7 +398,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `DependencyParser` object. ~~bytes~~ | -## DependencyParser.from_bytes {#from_bytes tag="method"} +## DependencyParser.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -417,7 +417,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `DependencyParser` object. ~~DependencyParser~~ | -## DependencyParser.labels {#labels tag="property"} +## DependencyParser.labels {id="labels",tag="property"} The labels currently added to the component. @@ -432,7 +432,7 @@ The labels currently added to the component. | ----------- | ------------------------------------------------------ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | -## DependencyParser.label_data {#label_data tag="property" new="3"} +## DependencyParser.label_data {id="label_data",tag="property",version="3"} The labels currently added to the component and their internal meta information. This is the data generated by [`init labels`](/api/cli#init-labels) and used by @@ -450,7 +450,7 @@ the model with a pre-defined label set. | ----------- | ------------------------------------------------------------------------------- | | **RETURNS** | The label data added to the component. ~~Dict[str, Dict[str, Dict[str, int]]]~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/doc.md b/website/docs/api/doc.mdx similarity index 95% rename from website/docs/api/doc.md rename to website/docs/api/doc.mdx index 090489d83..a5f3de6be 100644 --- a/website/docs/api/doc.md +++ b/website/docs/api/doc.mdx @@ -12,7 +12,7 @@ compressed binary strings. The `Doc` object holds an array of [`Span`](/api/span) objects are views of this array, i.e. they don't own the data themselves. -## Doc.\_\_init\_\_ {#init tag="method"} +## Doc.\_\_init\_\_ {id="init",tag="method"} Construct a `Doc` object. The most common way to get a `Doc` object is via the `nlp` object. @@ -47,7 +47,7 @@ Construct a `Doc` object. The most common way to get a `Doc` object is via the | `sent_starts` 3 | A list of values, of the same length as `words`, to assign as `token.is_sent_start`. Will be overridden by heads if `heads` is provided. Defaults to `None`. ~~Optional[List[Union[bool, int, None]]]~~ | | `ents` 3 | A list of strings, of the same length of `words`, to assign the token-based IOB tag. Defaults to `None`. ~~Optional[List[str]]~~ | -## Doc.\_\_getitem\_\_ {#getitem tag="method"} +## Doc.\_\_getitem\_\_ {id="getitem",tag="method"} Get a [`Token`](/api/token) object at position `i`, where `i` is an integer. Negative indexing is supported, and follows the usual Python semantics, i.e. @@ -80,7 +80,7 @@ semantics. | `start_end` | The slice of the document to get. ~~Tuple[int, int]~~ | | **RETURNS** | The span at `doc[start:end]`. ~~Span~~ | -## Doc.\_\_iter\_\_ {#iter tag="method"} +## Doc.\_\_iter\_\_ {id="iter",tag="method"} Iterate over `Token` objects, from which the annotations can be easily accessed. @@ -100,7 +100,7 @@ underlying C data directly from Cython. | ---------- | --------------------------- | | **YIELDS** | A `Token` object. ~~Token~~ | -## Doc.\_\_len\_\_ {#len tag="method"} +## Doc.\_\_len\_\_ {id="len",tag="method"} Get the number of tokens in the document. @@ -115,7 +115,7 @@ Get the number of tokens in the document. | ----------- | --------------------------------------------- | | **RETURNS** | The number of tokens in the document. ~~int~~ | -## Doc.set_extension {#set_extension tag="classmethod" new="2"} +## Doc.set_extension {id="set_extension",tag="classmethod",version="2"} Define a custom attribute on the `Doc` which becomes available via `Doc._`. For details, see the documentation on @@ -140,7 +140,7 @@ details, see the documentation on | `setter` | Setter function that takes the `Doc` and a value, and modifies the object. Is called when the user writes to the `Doc._` attribute. ~~Optional[Callable[[Doc, Any], None]]~~ | | `force` | Force overwriting existing attribute. ~~bool~~ | -## Doc.get_extension {#get_extension tag="classmethod" new="2"} +## Doc.get_extension {id="get_extension",tag="classmethod",version="2"} Look up a previously registered extension by name. Returns a 4-tuple `(default, method, getter, setter)` if the extension is registered. Raises a @@ -160,7 +160,7 @@ Look up a previously registered extension by name. Returns a 4-tuple | `name` | Name of the extension. ~~str~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | -## Doc.has_extension {#has_extension tag="classmethod" new="2"} +## Doc.has_extension {id="has_extension",tag="classmethod",version="2"} Check whether an extension has been registered on the `Doc` class. @@ -177,7 +177,7 @@ Check whether an extension has been registered on the `Doc` class. | `name` | Name of the extension to check. ~~str~~ | | **RETURNS** | Whether the extension has been registered. ~~bool~~ | -## Doc.remove_extension {#remove_extension tag="classmethod" new="2.0.12"} +## Doc.remove_extension {id="remove_extension",tag="classmethod",version="2.0.12"} Remove a previously registered extension. @@ -195,7 +195,7 @@ Remove a previously registered extension. | `name` | Name of the extension. ~~str~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | -## Doc.char_span {#char_span tag="method" new="2"} +## Doc.char_span {id="char_span",tag="method",version="2"} Create a `Span` object from the slice `doc.text[start_idx:end_idx]`. Returns `None` if the character indices don't map to a valid span using the default @@ -219,7 +219,7 @@ alignment mode `"strict". | `alignment_mode` | How character indices snap to token boundaries. Options: `"strict"` (no snapping), `"contract"` (span of all tokens completely within the character span), `"expand"` (span of all tokens at least partially covered by the character span). Defaults to `"strict"`. ~~str~~ | | **RETURNS** | The newly constructed object or `None`. ~~Optional[Span]~~ | -## Doc.set_ents {#set_ents tag="method" new="3"} +## Doc.set_ents {id="set_ents",tag="method",version="3"} Set the named entities in the document. @@ -243,7 +243,7 @@ Set the named entities in the document. | `outside` | Spans outside of entities (O in IOB). ~~Optional[List[Span]]~~ | | `default` | How to set entity annotation for tokens outside of any provided spans. Options: `"blocked"`, `"missing"`, `"outside"` and `"unmodified"` (preserve current state). Defaults to `"outside"`. ~~str~~ | -## Doc.similarity {#similarity tag="method" model="vectors"} +## Doc.similarity {id="similarity",tag="method",model="vectors"} Make a semantic similarity estimate. The default estimate is cosine similarity using an average of word vectors. @@ -263,7 +263,7 @@ using an average of word vectors. | `other` | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ | | **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ | -## Doc.count_by {#count_by tag="method"} +## Doc.count_by {id="count_by",tag="method"} Count the frequencies of a given attribute. Produces a dict of `{attr (int): count (ints)}` frequencies, keyed by the values of the given @@ -284,7 +284,7 @@ attribute ID. | `attr_id` | The attribute ID. ~~int~~ | | **RETURNS** | A dictionary mapping attributes to integer counts. ~~Dict[int, int]~~ | -## Doc.get_lca_matrix {#get_lca_matrix tag="method"} +## Doc.get_lca_matrix {id="get_lca_matrix",tag="method"} Calculates the lowest common ancestor matrix for a given `Doc`. Returns LCA matrix containing the integer index of the ancestor, or `-1` if no common @@ -302,7 +302,7 @@ ancestor is found, e.g. if span excludes a necessary ancestor. | ----------- | -------------------------------------------------------------------------------------- | | **RETURNS** | The lowest common ancestor matrix of the `Doc`. ~~numpy.ndarray[ndim=2, dtype=int32]~~ | -## Doc.has_annotation {#has_annotation tag="method"} +## Doc.has_annotation {id="has_annotation",tag="method"} Check whether the doc contains annotation on a [`Token` attribute](/api/token#attributes). @@ -327,7 +327,7 @@ doc = nlp("This is a text") | `require_complete` | Whether to check that the attribute is set on every token in the doc. Defaults to `False`. ~~bool~~ | | **RETURNS** | Whether specified annotation is present in the doc. ~~bool~~ | -## Doc.to_array {#to_array tag="method"} +## Doc.to_array {id="to_array",tag="method"} Export given token attributes to a numpy `ndarray`. If `attr_ids` is a sequence of `M` attributes, the output array will be of shape `(N, M)`, where `N` is the @@ -355,7 +355,7 @@ Returns a 2D array with one row per token and one column per attribute (when | `attr_ids` | A list of attributes (int IDs or string names) or a single attribute (int ID or string name). ~~Union[int, str, List[Union[int, str]]]~~ | | **RETURNS** | The exported attributes as a numpy array. ~~Union[numpy.ndarray[ndim=2, dtype=uint64], numpy.ndarray[ndim=1, dtype=uint64]]~~ | -## Doc.from_array {#from_array tag="method"} +## Doc.from_array {id="from_array",tag="method"} Load attributes from a numpy array. Write to a `Doc` object, from an `(M, N)` array of attributes. @@ -379,7 +379,7 @@ array of attributes. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `Doc` itself. ~~Doc~~ | -## Doc.from_docs {#from_docs tag="staticmethod" new="3"} +## Doc.from_docs {id="from_docs",tag="staticmethod",version="3"} Concatenate multiple `Doc` objects to form a new one. Raises an error if the `Doc` objects do not all share the same `Vocab`. @@ -408,7 +408,7 @@ Concatenate multiple `Doc` objects to form a new one. Raises an error if the | `exclude` 3.3 | String names of Doc attributes to exclude. Supported: `spans`, `tensor`, `user_data`. ~~Iterable[str]~~ | | **RETURNS** | The new `Doc` object that is containing the other docs or `None`, if `docs` is empty or `None`. ~~Optional[Doc]~~ | -## Doc.to_disk {#to_disk tag="method" new="2"} +## Doc.to_disk {id="to_disk",tag="method",version="2"} Save the current state to a directory. @@ -424,7 +424,7 @@ Save the current state to a directory. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## Doc.from_disk {#from_disk tag="method" new="2"} +## Doc.from_disk {id="from_disk",tag="method",version="2"} Loads state from a directory. Modifies the object in place and returns it. @@ -443,7 +443,7 @@ Loads state from a directory. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `Doc` object. ~~Doc~~ | -## Doc.to_bytes {#to_bytes tag="method"} +## Doc.to_bytes {id="to_bytes",tag="method"} Serialize, i.e. export the document contents to a binary string. @@ -460,7 +460,7 @@ Serialize, i.e. export the document contents to a binary string. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | A losslessly serialized copy of the `Doc`, including all annotations. ~~bytes~~ | -## Doc.from_bytes {#from_bytes tag="method"} +## Doc.from_bytes {id="from_bytes",tag="method"} Deserialize, i.e. import the document contents from a binary string. @@ -481,7 +481,7 @@ Deserialize, i.e. import the document contents from a binary string. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `Doc` object. ~~Doc~~ | -## Doc.to_json {#to_json tag="method"} +## Doc.to_json {id="to_json",tag="method"} Serializes a document to JSON. Note that this is format differs from the deprecated [`JSON training format`](/api/data-formats#json-input). @@ -498,7 +498,7 @@ deprecated [`JSON training format`](/api/data-formats#json-input). | `underscore` | Optional list of string names of custom `Doc` attributes. Attribute values need to be JSON-serializable. Values will be added to an `"_"` key in the data, e.g. `"_": {"foo": "bar"}`. ~~Optional[List[str]]~~ | | **RETURNS** | The data in JSON format. ~~Dict[str, Any]~~ | -## Doc.from_json {#from_json tag="method" new="3.3.1"} +## Doc.from_json {id="from_json",tag="method",version="3.3.1"} Deserializes a document from JSON, i.e. generates a document from the provided JSON data as generated by [`Doc.to_json()`](/api/doc#to_json). @@ -520,7 +520,7 @@ JSON data as generated by [`Doc.to_json()`](/api/doc#to_json). | `validate` | Whether to validate the JSON input against the expected schema for detailed debugging. Defaults to `False`. ~~bool~~ | | **RETURNS** | A `Doc` corresponding to the provided JSON. ~~Doc~~ | -## Doc.retokenize {#retokenize tag="contextmanager" new="2.1"} +## Doc.retokenize {id="retokenize",tag="contextmanager",version="2.1"} Context manager to handle retokenization of the `Doc`. Modifications to the `Doc`'s tokenization are stored, and then made all at once when the context @@ -540,7 +540,7 @@ invalidated, although they may accidentally continue to work. | ----------- | -------------------------------- | | **RETURNS** | The retokenizer. ~~Retokenizer~~ | -### Retokenizer.merge {#retokenizer.merge tag="method"} +### Retokenizer.merge {id="retokenizer.merge",tag="method"} Mark a span for merging. The `attrs` will be applied to the resulting token (if they're context-dependent token attributes like `LEMMA` or `DEP`) or to the @@ -563,7 +563,7 @@ values. | `span` | The span to merge. ~~Span~~ | | `attrs` | Attributes to set on the merged token. ~~Dict[Union[str, int], Any]~~ | -### Retokenizer.split {#retokenizer.split tag="method"} +### Retokenizer.split {id="retokenizer.split",tag="method"} Mark a token for splitting, into the specified `orths`. The `heads` are required to specify how the new subtokens should be integrated into the dependency tree. @@ -599,7 +599,7 @@ underlying lexeme (if they're context-independent lexical attributes like | `heads` | List of `token` or `(token, subtoken)` tuples specifying the tokens to attach the newly split subtokens to. ~~List[Union[Token, Tuple[Token, int]]]~~ | | `attrs` | Attributes to set on all split tokens. Attribute names mapped to list of per-token attribute values. ~~Dict[Union[str, int], List[Any]]~~ | -## Doc.ents {#ents tag="property" model="NER"} +## Doc.ents {id="ents",tag="property",model="NER"} The named entities in the document. Returns a tuple of named entity `Span` objects, if the entity recognizer has been applied. @@ -617,7 +617,7 @@ objects, if the entity recognizer has been applied. | ----------- | ---------------------------------------------------------------- | | **RETURNS** | Entities in the document, one `Span` per entity. ~~Tuple[Span]~~ | -## Doc.spans {#spans tag="property"} +## Doc.spans {id="spans",tag="property"} A dictionary of named span groups, to store and access additional span annotations. You can write to it by assigning a list of [`Span`](/api/span) @@ -634,7 +634,7 @@ objects or a [`SpanGroup`](/api/spangroup) to a given key. | ----------- | ------------------------------------------------------------------ | | **RETURNS** | The span groups assigned to the document. ~~Dict[str, SpanGroup]~~ | -## Doc.cats {#cats tag="property" model="text classifier"} +## Doc.cats {id="cats",tag="property",model="text classifier"} Maps a label to a score for categories applied to the document. Typically set by the [`TextCategorizer`](/api/textcategorizer). @@ -650,7 +650,7 @@ the [`TextCategorizer`](/api/textcategorizer). | ----------- | ---------------------------------------------------------- | | **RETURNS** | The text categories mapped to scores. ~~Dict[str, float]~~ | -## Doc.noun_chunks {#noun_chunks tag="property" model="parser"} +## Doc.noun_chunks {id="noun_chunks",tag="property",model="parser"} Iterate over the base noun phrases in the document. Yields base noun-phrase `Span` objects, if the document has been syntactically parsed. A base noun @@ -677,7 +677,7 @@ implemented for the given language, a `NotImplementedError` is raised. | ---------- | ------------------------------------- | | **YIELDS** | Noun chunks in the document. ~~Span~~ | -## Doc.sents {#sents tag="property" model="sentences"} +## Doc.sents {id="sents",tag="property",model="sentences"} Iterate over the sentences in the document. Sentence spans have no label. @@ -699,7 +699,7 @@ will raise an error otherwise. | ---------- | ----------------------------------- | | **YIELDS** | Sentences in the document. ~~Span~~ | -## Doc.has_vector {#has_vector tag="property" model="vectors"} +## Doc.has_vector {id="has_vector",tag="property",model="vectors"} A boolean value indicating whether a word vector is associated with the object. @@ -714,7 +714,7 @@ A boolean value indicating whether a word vector is associated with the object. | ----------- | --------------------------------------------------------- | | **RETURNS** | Whether the document has a vector data attached. ~~bool~~ | -## Doc.vector {#vector tag="property" model="vectors"} +## Doc.vector {id="vector",tag="property",model="vectors"} A real-valued meaning representation. Defaults to an average of the token vectors. @@ -731,7 +731,7 @@ vectors. | ----------- | -------------------------------------------------------------------------------------------------- | | **RETURNS** | A 1-dimensional array representing the document's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | -## Doc.vector_norm {#vector_norm tag="property" model="vectors"} +## Doc.vector_norm {id="vector_norm",tag="property",model="vectors"} The L2 norm of the document's vector representation. @@ -749,7 +749,7 @@ The L2 norm of the document's vector representation. | ----------- | --------------------------------------------------- | | **RETURNS** | The L2 norm of the vector representation. ~~float~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | @@ -768,7 +768,7 @@ The L2 norm of the document's vector representation. | `has_unknown_spaces` | Whether the document was constructed without known spacing between tokens (typically when created from gold tokenization). ~~bool~~ | | `_` | User space for adding custom [attribute extensions](/usage/processing-pipelines#custom-components-attributes). ~~Underscore~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/docbin.md b/website/docs/api/docbin.mdx similarity index 93% rename from website/docs/api/docbin.md rename to website/docs/api/docbin.mdx index b1d1798ba..b5cf29df7 100644 --- a/website/docs/api/docbin.md +++ b/website/docs/api/docbin.mdx @@ -1,7 +1,7 @@ --- title: DocBin tag: class -new: 2.2 +version: 2.2 teaser: Pack Doc objects for binary serialization source: spacy/tokens/_serialize.py --- @@ -15,8 +15,7 @@ notable downside to this format is that you can't easily extract just one document from the `DocBin`. The serialization format is gzipped msgpack, where the msgpack object has the following structure: -```python -### msgpack object structure +```python {title="msgpack object structure"} { "version": str, # DocBin version number "attrs": List[uint64], # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE] @@ -33,7 +32,7 @@ object. This means the storage is more efficient if you pack more documents together, because you have less duplication in the strings. For usage examples, see the docs on [serializing `Doc` objects](/usage/saving-loading#docs). -## DocBin.\_\_init\_\_ {#init tag="method"} +## DocBin.\_\_init\_\_ {id="init",tag="method"} Create a `DocBin` object to hold serialized annotations. @@ -50,7 +49,7 @@ Create a `DocBin` object to hold serialized annotations. | `store_user_data` | Whether to write the `Doc.user_data` and the values of custom extension attributes to file/bytes. Defaults to `False`. ~~bool~~ | | `docs` | `Doc` objects to add on initialization. ~~Iterable[Doc]~~ | -## DocBin.\_\len\_\_ {#len tag="method"} +## DocBin.\_\_len\_\_ {id="len",tag="method"} Get the number of `Doc` objects that were added to the `DocBin`. @@ -67,7 +66,7 @@ Get the number of `Doc` objects that were added to the `DocBin`. | ----------- | --------------------------------------------------- | | **RETURNS** | The number of `Doc`s added to the `DocBin`. ~~int~~ | -## DocBin.add {#add tag="method"} +## DocBin.add {id="add",tag="method"} Add a `Doc`'s annotations to the `DocBin` for serialization. @@ -83,7 +82,7 @@ Add a `Doc`'s annotations to the `DocBin` for serialization. | -------- | -------------------------------- | | `doc` | The `Doc` object to add. ~~Doc~~ | -## DocBin.get_docs {#get_docs tag="method"} +## DocBin.get_docs {id="get_docs",tag="method"} Recover `Doc` objects from the annotations, using the given vocab. @@ -98,7 +97,7 @@ Recover `Doc` objects from the annotations, using the given vocab. | `vocab` | The shared vocab. ~~Vocab~~ | | **YIELDS** | The `Doc` objects. ~~Doc~~ | -## DocBin.merge {#merge tag="method"} +## DocBin.merge {id="merge",tag="method"} Extend the annotations of this `DocBin` with the annotations from another. Will raise an error if the pre-defined `attrs` of the two `DocBin`s don't match. @@ -118,7 +117,7 @@ raise an error if the pre-defined `attrs` of the two `DocBin`s don't match. | -------- | ------------------------------------------------------ | | `other` | The `DocBin` to merge into the current bin. ~~DocBin~~ | -## DocBin.to_bytes {#to_bytes tag="method"} +## DocBin.to_bytes {id="to_bytes",tag="method"} Serialize the `DocBin`'s annotations to a bytestring. @@ -134,7 +133,7 @@ Serialize the `DocBin`'s annotations to a bytestring. | ----------- | ---------------------------------- | | **RETURNS** | The serialized `DocBin`. ~~bytes~~ | -## DocBin.from_bytes {#from_bytes tag="method"} +## DocBin.from_bytes {id="from_bytes",tag="method"} Deserialize the `DocBin`'s annotations from a bytestring. @@ -150,7 +149,7 @@ Deserialize the `DocBin`'s annotations from a bytestring. | `bytes_data` | The data to load from. ~~bytes~~ | | **RETURNS** | The loaded `DocBin`. ~~DocBin~~ | -## DocBin.to_disk {#to_disk tag="method" new="3"} +## DocBin.to_disk {id="to_disk",tag="method",version="3"} Save the serialized `DocBin` to a file. Typically uses the `.spacy` extension and the result can be used as the input data for @@ -168,7 +167,7 @@ and the result can be used as the input data for | -------- | -------------------------------------------------------------------------- | | `path` | The file path, typically with the `.spacy` extension. ~~Union[str, Path]~~ | -## DocBin.from_disk {#from_disk tag="method" new="3"} +## DocBin.from_disk {id="from_disk",tag="method",version="3"} Load a serialized `DocBin` from a file. Typically uses the `.spacy` extension. diff --git a/website/docs/api/edittreelemmatizer.md b/website/docs/api/edittreelemmatizer.mdx similarity index 95% rename from website/docs/api/edittreelemmatizer.md rename to website/docs/api/edittreelemmatizer.mdx index 63e4bf910..82967482c 100644 --- a/website/docs/api/edittreelemmatizer.md +++ b/website/docs/api/edittreelemmatizer.mdx @@ -2,7 +2,7 @@ title: EditTreeLemmatizer tag: class source: spacy/pipeline/edit_tree_lemmatizer.py -new: 3.3 +version: 3.3 teaser: 'Pipeline component for lemmatization' api_base_class: /api/pipe api_string_name: trainable_lemmatizer @@ -18,7 +18,7 @@ and construction method used by this lemmatizer were proposed in For a lookup and rule-based lemmatizer, see [`Lemmatizer`](/api/lemmatizer). -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Predictions are assigned to `Token.lemma`. @@ -27,7 +27,7 @@ Predictions are assigned to `Token.lemma`. | `Token.lemma` | The lemma (hash). ~~int~~ | | `Token.lemma_` | The lemma. ~~str~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -57,7 +57,7 @@ architectures and their arguments and hyperparameters. %%GITHUB_SPACY/spacy/pipeline/edit_tree_lemmatizer.py ``` -## EditTreeLemmatizer.\_\_init\_\_ {#init tag="method"} +## EditTreeLemmatizer.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -90,7 +90,7 @@ shortcut for this and instantiate the component using its string name and | `top_k` | The number of most probable edit trees to try before resorting to `backoff`. Defaults to `1`. ~~int~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attribute `"lemma"`. ~~Optional[Callable]~~ | -## EditTreeLemmatizer.\_\_call\_\_ {#call tag="method"} +## EditTreeLemmatizer.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -114,7 +114,7 @@ and all pipeline components are applied to the `Doc` in order. Both | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## EditTreeLemmatizer.pipe {#pipe tag="method"} +## EditTreeLemmatizer.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -138,7 +138,7 @@ and [`pipe`](/api/edittreelemmatizer#pipe) delegate to the | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## EditTreeLemmatizer.initialize {#initialize tag="method" new="3"} +## EditTreeLemmatizer.initialize {id="initialize",tag="method",version="3"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -175,7 +175,7 @@ config. | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ | -## EditTreeLemmatizer.predict {#predict tag="method"} +## EditTreeLemmatizer.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects, without modifying them. @@ -192,7 +192,7 @@ modifying them. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The model's prediction for each document. | -## EditTreeLemmatizer.set_annotations {#set_annotations tag="method"} +## EditTreeLemmatizer.set_annotations {id="set_annotations",tag="method"} Modify a batch of [`Doc`](/api/doc) objects, using pre-computed tree identifiers. @@ -210,7 +210,7 @@ identifiers. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `tree_ids` | The identifiers of the edit trees to apply, produced by `EditTreeLemmatizer.predict`. | -## EditTreeLemmatizer.update {#update tag="method"} +## EditTreeLemmatizer.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects containing the predictions and gold-standard annotations, and update the component's model. @@ -234,7 +234,7 @@ Delegates to [`predict`](/api/edittreelemmatizer#predict) and | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## EditTreeLemmatizer.get_loss {#get_loss tag="method"} +## EditTreeLemmatizer.get_loss {id="get_loss",tag="method"} Find the loss and gradient of loss for the batch of documents and their predicted scores. @@ -253,7 +253,7 @@ predicted scores. | `scores` | Scores representing the model's predictions. | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | -## EditTreeLemmatizer.create_optimizer {#create_optimizer tag="method"} +## EditTreeLemmatizer.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -268,7 +268,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## EditTreeLemmatizer.use_params {#use_params tag="method, contextmanager"} +## EditTreeLemmatizer.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored. @@ -285,7 +285,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## EditTreeLemmatizer.to_disk {#to_disk tag="method"} +## EditTreeLemmatizer.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -302,7 +302,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## EditTreeLemmatizer.from_disk {#from_disk tag="method"} +## EditTreeLemmatizer.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -320,7 +320,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `EditTreeLemmatizer` object. ~~EditTreeLemmatizer~~ | -## EditTreeLemmatizer.to_bytes {#to_bytes tag="method"} +## EditTreeLemmatizer.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -337,7 +337,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `EditTreeLemmatizer` object. ~~bytes~~ | -## EditTreeLemmatizer.from_bytes {#from_bytes tag="method"} +## EditTreeLemmatizer.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -356,7 +356,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `EditTreeLemmatizer` object. ~~EditTreeLemmatizer~~ | -## EditTreeLemmatizer.labels {#labels tag="property"} +## EditTreeLemmatizer.labels {id="labels",tag="property"} The labels currently added to the component. @@ -371,7 +371,7 @@ identifiers of edit trees. | ----------- | ------------------------------------------------------ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | -## EditTreeLemmatizer.label_data {#label_data tag="property" new="3"} +## EditTreeLemmatizer.label_data {id="label_data",tag="property",version="3"} The labels currently added to the component and their internal meta information. This is the data generated by [`init labels`](/api/cli#init-labels) and used by @@ -389,7 +389,7 @@ initialize the model with a pre-defined label set. | ----------- | ---------------------------------------------------------- | | **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/entitylinker.md b/website/docs/api/entitylinker.mdx similarity index 96% rename from website/docs/api/entitylinker.md rename to website/docs/api/entitylinker.mdx index 40ec8afb5..5c30d252e 100644 --- a/website/docs/api/entitylinker.md +++ b/website/docs/api/entitylinker.mdx @@ -2,7 +2,7 @@ title: EntityLinker tag: class source: spacy/pipeline/entity_linker.py -new: 2.2 +version: 2.2 teaser: 'Pipeline component for named entity linking and disambiguation' api_base_class: /api/pipe api_string_name: entity_linker @@ -17,7 +17,7 @@ and a machine learning model to pick the right candidate, given the local context of the mention. `EntityLinker` defaults to using the [`InMemoryLookupKB`](/api/kb_in_memory) implementation. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Predictions, in the form of knowledge base IDs, will be assigned to `Token.ent_kb_id_`. @@ -27,7 +27,7 @@ Predictions, in the form of knowledge base IDs, will be assigned to | `Token.ent_kb_id` | Knowledge base ID (hash). ~~int~~ | | `Token.ent_kb_id_` | Knowledge base ID. ~~str~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -71,7 +71,7 @@ architectures and their arguments and hyperparameters. %%GITHUB_SPACY/spacy/pipeline/entity_linker.py ``` -## EntityLinker.\_\_init\_\_ {#init tag="method"} +## EntityLinker.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -114,7 +114,7 @@ custom knowledge base, you should either call | `scorer` 3.2 | The scoring method. Defaults to [`Scorer.score_links`](/api/scorer#score_links). ~~Optional[Callable]~~ | | `threshold` 3.4 | Confidence threshold for entity predictions. The default of `None` implies that all predictions are accepted, otherwise those with a score beneath the treshold are discarded. If there are no predictions with scores above the threshold, the linked entity is `NIL`. ~~Optional[float]~~ | -## EntityLinker.\_\_call\_\_ {#call tag="method"} +## EntityLinker.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -137,7 +137,7 @@ delegate to the [`predict`](/api/entitylinker#predict) and | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## EntityLinker.pipe {#pipe tag="method"} +## EntityLinker.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -161,7 +161,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/entitylinker#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## EntityLinker.set_kb {#set_kb tag="method" new="3"} +## EntityLinker.set_kb {id="set_kb",tag="method",version="3"} The `kb_loader` should be a function that takes a `Vocab` instance and creates the `KnowledgeBase`, ensuring that the strings of the knowledge base are synced @@ -183,7 +183,7 @@ with the current vocab. | ----------- | ---------------------------------------------------------------------------------------------------------------- | | `kb_loader` | Function that creates a [`KnowledgeBase`](/api/kb) from a `Vocab` instance. ~~Callable[[Vocab], KnowledgeBase]~~ | -## EntityLinker.initialize {#initialize tag="method" new="3"} +## EntityLinker.initialize {id="initialize",tag="method",version="3"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -219,7 +219,7 @@ This method was previously called `begin_training`. | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `kb_loader` | Function that creates a [`KnowledgeBase`](/api/kb) from a `Vocab` instance. ~~Callable[[Vocab], KnowledgeBase]~~ | -## EntityLinker.predict {#predict tag="method"} +## EntityLinker.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects, without modifying them. Returns the KB IDs for each entity in each doc, including `NIL` @@ -237,7 +237,7 @@ if there is no prediction. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The predicted KB identifiers for the entities in the `docs`. ~~List[str]~~ | -## EntityLinker.set_annotations {#set_annotations tag="method"} +## EntityLinker.set_annotations {id="set_annotations",tag="method"} Modify a batch of documents, using pre-computed entity IDs for a list of named entities. @@ -255,7 +255,7 @@ entities. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `kb_ids` | The knowledge base identifiers for the entities in the docs, predicted by `EntityLinker.predict`. ~~List[str]~~ | -## EntityLinker.update {#update tag="method"} +## EntityLinker.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects, updating both the pipe's entity linking model and context encoder. Delegates to @@ -278,7 +278,7 @@ pipe's entity linking model and context encoder. Delegates to | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## EntityLinker.create_optimizer {#create_optimizer tag="method"} +## EntityLinker.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -293,7 +293,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## EntityLinker.use_params {#use_params tag="method, contextmanager"} +## EntityLinker.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored. @@ -310,7 +310,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## EntityLinker.to_disk {#to_disk tag="method"} +## EntityLinker.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -327,7 +327,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## EntityLinker.from_disk {#from_disk tag="method"} +## EntityLinker.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -345,7 +345,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `EntityLinker` object. ~~EntityLinker~~ | -## EntityLinker.to_bytes {#to_bytes tag="method"} +## EntityLinker.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -362,7 +362,7 @@ Serialize the pipe to a bytestring, including the `KnowledgeBase`. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `EntityLinker` object. ~~bytes~~ | -## EntityLinker.from_bytes {#from_bytes tag="method"} +## EntityLinker.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -381,7 +381,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `EntityLinker` object. ~~EntityLinker~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/entityrecognizer.md b/website/docs/api/entityrecognizer.mdx similarity index 95% rename from website/docs/api/entityrecognizer.md rename to website/docs/api/entityrecognizer.mdx index a535e8316..c80406a5b 100644 --- a/website/docs/api/entityrecognizer.md +++ b/website/docs/api/entityrecognizer.mdx @@ -20,7 +20,7 @@ your entities will be close to their initial tokens. If your entities are long and characterized by tokens in their middle, the component will likely not be a good fit for your task. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Predictions will be saved to `Doc.ents` as a tuple. Each label will also be reflected to each underlying token, where it is saved in the `Token.ent_type` @@ -38,7 +38,7 @@ non-overlapping, or an error will be thrown. | `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ | | `Token.ent_type_` | The label part of the named entity tag. ~~str~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -72,7 +72,7 @@ architectures and their arguments and hyperparameters. %%GITHUB_SPACY/spacy/pipeline/ner.pyx ``` -## EntityRecognizer.\_\_init\_\_ {#init tag="method"} +## EntityRecognizer.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -103,7 +103,7 @@ shortcut for this and instantiate the component using its string name and | `update_with_oracle_cut_size` | During training, cut long sequences into shorter segments by creating intermediate states based on the gold-standard history. The model is not very sensitive to this parameter, so you usually won't need to change it. Defaults to `100`. ~~int~~ | | `incorrect_spans_key` | Identifies spans that are known to be incorrect entity annotations. The incorrect entity annotations can be stored in the span group in [`Doc.spans`](/api/doc#spans), under this key. Defaults to `None`. ~~Optional[str]~~ | -## EntityRecognizer.\_\_call\_\_ {#call tag="method"} +## EntityRecognizer.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -127,7 +127,7 @@ and all pipeline components are applied to the `Doc` in order. Both | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## EntityRecognizer.pipe {#pipe tag="method"} +## EntityRecognizer.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -151,7 +151,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/entityrecognizer#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## EntityRecognizer.initialize {#initialize tag="method" new="3"} +## EntityRecognizer.initialize {id="initialize",tag="method",version="3"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -194,7 +194,7 @@ This method was previously called `begin_training`. | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Dict[str, Dict[str, int]]]~~ | -## EntityRecognizer.predict {#predict tag="method"} +## EntityRecognizer.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects, without modifying them. @@ -211,7 +211,7 @@ modifying them. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | A helper class for the parse state (internal). ~~StateClass~~ | -## EntityRecognizer.set_annotations {#set_annotations tag="method"} +## EntityRecognizer.set_annotations {id="set_annotations",tag="method"} Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. @@ -228,7 +228,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `scores` | The scores to set, produced by `EntityRecognizer.predict`. Returns an internal helper class for the parse state. ~~List[StateClass]~~ | -## EntityRecognizer.update {#update tag="method"} +## EntityRecognizer.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects, updating the pipe's model. Delegates to [`predict`](/api/entityrecognizer#predict) and @@ -251,7 +251,7 @@ model. Delegates to [`predict`](/api/entityrecognizer#predict) and | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## EntityRecognizer.get_loss {#get_loss tag="method"} +## EntityRecognizer.get_loss {id="get_loss",tag="method"} Find the loss and gradient of loss for the batch of documents and their predicted scores. @@ -270,7 +270,7 @@ predicted scores. | `scores` | Scores representing the model's predictions. ~~StateClass~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | -## EntityRecognizer.create_optimizer {#create_optimizer tag="method"} +## EntityRecognizer.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -285,7 +285,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## EntityRecognizer.use_params {#use_params tag="method, contextmanager"} +## EntityRecognizer.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored. @@ -302,7 +302,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## EntityRecognizer.add_label {#add_label tag="method"} +## EntityRecognizer.add_label {id="add_label",tag="method"} Add a new label to the pipe. Note that you don't have to call this method if you provide a **representative data sample** to the [`initialize`](#initialize) @@ -322,7 +322,7 @@ to the model, and the output dimension will be | `label` | The label to add. ~~str~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | -## EntityRecognizer.set_output {#set_output tag="method"} +## EntityRecognizer.set_output {id="set_output",tag="method"} Change the output dimension of the component's model by calling the model's attribute `resize_output`. This is a function that takes the original model and @@ -341,7 +341,7 @@ forgetting" problem. | ---- | --------------------------------- | | `nO` | The new output dimension. ~~int~~ | -## EntityRecognizer.to_disk {#to_disk tag="method"} +## EntityRecognizer.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -358,7 +358,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## EntityRecognizer.from_disk {#from_disk tag="method"} +## EntityRecognizer.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -376,7 +376,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `EntityRecognizer` object. ~~EntityRecognizer~~ | -## EntityRecognizer.to_bytes {#to_bytes tag="method"} +## EntityRecognizer.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -393,7 +393,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `EntityRecognizer` object. ~~bytes~~ | -## EntityRecognizer.from_bytes {#from_bytes tag="method"} +## EntityRecognizer.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -412,7 +412,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `EntityRecognizer` object. ~~EntityRecognizer~~ | -## EntityRecognizer.labels {#labels tag="property"} +## EntityRecognizer.labels {id="labels",tag="property"} The labels currently added to the component. @@ -427,7 +427,7 @@ The labels currently added to the component. | ----------- | ------------------------------------------------------ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | -## EntityRecognizer.label_data {#label_data tag="property" new="3"} +## EntityRecognizer.label_data {id="label_data",tag="property",version="3"} The labels currently added to the component and their internal meta information. This is the data generated by [`init labels`](/api/cli#init-labels) and used by @@ -445,7 +445,7 @@ the model with a pre-defined label set. | ----------- | ------------------------------------------------------------------------------- | | **RETURNS** | The label data added to the component. ~~Dict[str, Dict[str, Dict[str, int]]]~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/entityruler.md b/website/docs/api/entityruler.mdx similarity index 94% rename from website/docs/api/entityruler.md rename to website/docs/api/entityruler.mdx index f15c648ff..27624398e 100644 --- a/website/docs/api/entityruler.md +++ b/website/docs/api/entityruler.mdx @@ -2,7 +2,7 @@ title: EntityRuler tag: class source: spacy/pipeline/entityruler.py -new: 2.1 +version: 2.1 teaser: 'Pipeline component for rule-based named entity recognition' api_string_name: entity_ruler api_trainable: false @@ -15,7 +15,7 @@ used on its own to implement a purely rule-based entity recognition system. For usage examples, see the docs on [rule-based entity recognition](/usage/rule-based-matching#entityruler). -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} This component assigns predictions basically the same way as the [`EntityRecognizer`](/api/entityrecognizer). @@ -36,7 +36,7 @@ non-overlapping, or an error will be thrown. | `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ | | `Token.ent_type_` | The label part of the named entity tag. ~~str~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -68,7 +68,7 @@ how the component should be configured. You can override its settings via the %%GITHUB_SPACY/spacy/pipeline/entityruler.py ``` -## EntityRuler.\_\_init\_\_ {#init tag="method"} +## EntityRuler.\_\_init\_\_ {id="init",tag="method"} Initialize the entity ruler. If patterns are supplied here, they need to be a list of dictionaries with a `"label"` and `"pattern"` key. A pattern can either @@ -99,7 +99,7 @@ be a token pattern (list) or a phrase pattern (string). For example: | `patterns` | Optional patterns to load in on initialization. ~~Optional[List[Dict[str, Union[str, List[dict]]]]]~~ | | `scorer` | The scoring method. Defaults to [`spacy.scorer.get_ner_prf`](/api/scorer#get_ner_prf). ~~Optional[Callable]~~ | -## EntityRuler.initialize {#initialize tag="method" new="3"} +## EntityRuler.initialize {id="initialize",tag="method",version="3"} Initialize the component with data and used before training to load in rules from a [pattern file](/usage/rule-based-matching/#entityruler-files). This @@ -131,7 +131,7 @@ config. | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `patterns` | The list of patterns. Defaults to `None`. ~~Optional[Sequence[Dict[str, Union[str, List[Dict[str, Any]]]]]]~~ | -## EntityRuler.\_\len\_\_ {#len tag="method"} +## EntityRuler.\_\_len\_\_ {id="len",tag="method"} The number of all patterns added to the entity ruler. @@ -148,7 +148,7 @@ The number of all patterns added to the entity ruler. | ----------- | ------------------------------- | | **RETURNS** | The number of patterns. ~~int~~ | -## EntityRuler.\_\_contains\_\_ {#contains tag="method"} +## EntityRuler.\_\_contains\_\_ {id="contains",tag="method"} Whether a label is present in the patterns. @@ -166,7 +166,7 @@ Whether a label is present in the patterns. | `label` | The label to check. ~~str~~ | | **RETURNS** | Whether the entity ruler contains the label. ~~bool~~ | -## EntityRuler.\_\_call\_\_ {#call tag="method"} +## EntityRuler.\_\_call\_\_ {id="call",tag="method"} Find matches in the `Doc` and add them to the `doc.ents`. Typically, this happens automatically after the component has been added to the pipeline using @@ -192,7 +192,7 @@ is chosen. | `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ | | **RETURNS** | The modified `Doc` with added entities, if available. ~~Doc~~ | -## EntityRuler.add_patterns {#add_patterns tag="method"} +## EntityRuler.add_patterns {id="add_patterns",tag="method"} Add patterns to the entity ruler. A pattern can either be a token pattern (list of dicts) or a phrase pattern (string). For more details, see the usage guide on @@ -213,7 +213,7 @@ of dicts) or a phrase pattern (string). For more details, see the usage guide on | ---------- | ---------------------------------------------------------------- | | `patterns` | The patterns to add. ~~List[Dict[str, Union[str, List[dict]]]]~~ | -## EntityRuler.remove {#remove tag="method" new="3.2.1"} +## EntityRuler.remove {id="remove",tag="method",version="3.2.1"} Remove a pattern by its ID from the entity ruler. A `ValueError` is raised if the ID does not exist. @@ -231,7 +231,7 @@ the ID does not exist. | ---- | ----------------------------------- | | `id` | The ID of the pattern rule. ~~str~~ | -## EntityRuler.to_disk {#to_disk tag="method"} +## EntityRuler.to_disk {id="to_disk",tag="method"} Save the entity ruler patterns to a directory. The patterns will be saved as newline-delimited JSON (JSONL). If a file with the suffix `.jsonl` is provided, @@ -250,7 +250,7 @@ only the patterns are saved as JSONL. If a directory name is provided, a | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `path` | A path to a JSONL file or directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | -## EntityRuler.from_disk {#from_disk tag="method"} +## EntityRuler.from_disk {id="from_disk",tag="method"} Load the entity ruler from a path. Expects either a file containing newline-delimited JSON (JSONL) with one entry per line, or a directory @@ -270,7 +270,7 @@ configuration. | `path` | A path to a JSONL file or directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | **RETURNS** | The modified `EntityRuler` object. ~~EntityRuler~~ | -## EntityRuler.to_bytes {#to_bytes tag="method"} +## EntityRuler.to_bytes {id="to_bytes",tag="method"} Serialize the entity ruler patterns to a bytestring. @@ -285,7 +285,7 @@ Serialize the entity ruler patterns to a bytestring. | ----------- | ---------------------------------- | | **RETURNS** | The serialized patterns. ~~bytes~~ | -## EntityRuler.from_bytes {#from_bytes tag="method"} +## EntityRuler.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -302,7 +302,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `bytes_data` | The bytestring to load. ~~bytes~~ | | **RETURNS** | The modified `EntityRuler` object. ~~EntityRuler~~ | -## EntityRuler.labels {#labels tag="property"} +## EntityRuler.labels {id="labels",tag="property"} All labels present in the match patterns. @@ -310,7 +310,7 @@ All labels present in the match patterns. | ----------- | -------------------------------------- | | **RETURNS** | The string labels. ~~Tuple[str, ...]~~ | -## EntityRuler.ent_ids {#ent_ids tag="property" new="2.2.2"} +## EntityRuler.ent_ids {id="ent_ids",tag="property",version="2.2.2"} All entity IDs present in the `id` properties of the match patterns. @@ -318,7 +318,7 @@ All entity IDs present in the `id` properties of the match patterns. | ----------- | ----------------------------------- | | **RETURNS** | The string IDs. ~~Tuple[str, ...]~~ | -## EntityRuler.patterns {#patterns tag="property"} +## EntityRuler.patterns {id="patterns",tag="property"} Get all patterns that were added to the entity ruler. @@ -326,7 +326,7 @@ Get all patterns that were added to the entity ruler. | ----------- | ---------------------------------------------------------------------------------------- | | **RETURNS** | The original patterns, one dictionary per pattern. ~~List[Dict[str, Union[str, dict]]]~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | ----------------- | --------------------------------------------------------------------------------------------------------------------- | diff --git a/website/docs/api/example.md b/website/docs/api/example.mdx similarity index 92% rename from website/docs/api/example.md rename to website/docs/api/example.mdx index 63768d58f..a29d5a7e0 100644 --- a/website/docs/api/example.md +++ b/website/docs/api/example.mdx @@ -3,7 +3,7 @@ title: Example teaser: A training instance tag: class source: spacy/training/example.pyx -new: 3.0 +version: 3.0 --- An `Example` holds the information for one training instance. It stores two @@ -12,7 +12,7 @@ holding the predictions of the pipeline. An [`Alignment`](/api/example#alignment-object) object stores the alignment between these two documents, as they can differ in tokenization. -## Example.\_\_init\_\_ {#init tag="method"} +## Example.\_\_init\_\_ {id="init",tag="method"} Construct an `Example` object from the `predicted` document and the `reference` document. If `alignment` is `None`, it will be initialized from the words in @@ -40,7 +40,7 @@ both documents. | _keyword-only_ | | | `alignment` | An object holding the alignment between the tokens of the `predicted` and `reference` documents. ~~Optional[Alignment]~~ | -## Example.from_dict {#from_dict tag="classmethod"} +## Example.from_dict {id="from_dict",tag="classmethod"} Construct an `Example` object from the `predicted` document and the reference annotations provided as a dictionary. For more details on the required format, @@ -64,7 +64,7 @@ see the [training format documentation](/api/data-formats#dict-input). | `example_dict` | The gold-standard annotations as a dictionary. Cannot be `None`. ~~Dict[str, Any]~~ | | **RETURNS** | The newly constructed object. ~~Example~~ | -## Example.text {#text tag="property"} +## Example.text {id="text",tag="property"} The text of the `predicted` document in this `Example`. @@ -78,7 +78,7 @@ The text of the `predicted` document in this `Example`. | ----------- | --------------------------------------------- | | **RETURNS** | The text of the `predicted` document. ~~str~~ | -## Example.predicted {#predicted tag="property"} +## Example.predicted {id="predicted",tag="property"} The `Doc` holding the predictions. Occasionally also referred to as `example.x`. @@ -94,7 +94,7 @@ The `Doc` holding the predictions. Occasionally also referred to as `example.x`. | ----------- | ------------------------------------------------------ | | **RETURNS** | The document containing (partial) predictions. ~~Doc~~ | -## Example.reference {#reference tag="property"} +## Example.reference {id="reference",tag="property"} The `Doc` holding the gold-standard annotations. Occasionally also referred to as `example.y`. @@ -111,7 +111,7 @@ as `example.y`. | ----------- | ---------------------------------------------------------- | | **RETURNS** | The document containing gold-standard annotations. ~~Doc~~ | -## Example.alignment {#alignment tag="property"} +## Example.alignment {id="alignment",tag="property"} The [`Alignment`](/api/example#alignment-object) object mapping the tokens of the `predicted` document to those of the `reference` document. @@ -131,7 +131,7 @@ the `predicted` document to those of the `reference` document. | ----------- | ---------------------------------------------------------------- | | **RETURNS** | The document containing gold-standard annotations. ~~Alignment~~ | -## Example.get_aligned {#get_aligned tag="method"} +## Example.get_aligned {id="get_aligned",tag="method"} Get the aligned view of a certain token attribute, denoted by its int ID or string name. @@ -152,7 +152,7 @@ string name. | `as_string` | Whether or not to return the list of values as strings. Defaults to `False`. ~~bool~~ | | **RETURNS** | List of integer values, or string values if `as_string` is `True`. ~~Union[List[int], List[str]]~~ | -## Example.get_aligned_parse {#get_aligned_parse tag="method"} +## Example.get_aligned_parse {id="get_aligned_parse",tag="method"} Get the aligned view of the dependency parse. If `projectivize` is set to `True`, non-projective dependency trees are made projective through the @@ -172,7 +172,7 @@ Pseudo-Projective Dependency Parsing algorithm by Nivre and Nilsson (2005). | `projectivize` | Whether or not to projectivize the dependency trees. Defaults to `True`. ~~bool~~ | | **RETURNS** | List of integer values, or string values if `as_string` is `True`. ~~Union[List[int], List[str]]~~ | -## Example.get_aligned_ner {#get_aligned_ner tag="method"} +## Example.get_aligned_ner {id="get_aligned_ner",tag="method"} Get the aligned view of the NER [BILUO](/usage/linguistic-features#accessing-ner) tags. @@ -193,7 +193,7 @@ Get the aligned view of the NER | ----------- | ------------------------------------------------------------------------------------------------- | | **RETURNS** | List of BILUO values, denoting whether tokens are part of an NER annotation or not. ~~List[str]~~ | -## Example.get_aligned_spans_y2x {#get_aligned_spans_y2x tag="method"} +## Example.get_aligned_spans_y2x {id="get_aligned_spans_y2x",tag="method"} Get the aligned view of any set of [`Span`](/api/span) objects defined over [`Example.reference`](/api/example#reference). The resulting span indices will @@ -219,7 +219,7 @@ align to the tokenization in [`Example.predicted`](/api/example#predicted). | `allow_overlap` | Whether the resulting `Span` objects may overlap or not. Set to `False` by default. ~~bool~~ | | **RETURNS** | `Span` objects aligned to the tokenization of `predicted`. ~~List[Span]~~ | -## Example.get_aligned_spans_x2y {#get_aligned_spans_x2y tag="method"} +## Example.get_aligned_spans_x2y {id="get_aligned_spans_x2y",tag="method"} Get the aligned view of any set of [`Span`](/api/span) objects defined over [`Example.predicted`](/api/example#predicted). The resulting span indices will @@ -247,7 +247,7 @@ against the original gold-standard annotation. | `allow_overlap` | Whether the resulting `Span` objects may overlap or not. Set to `False` by default. ~~bool~~ | | **RETURNS** | `Span` objects aligned to the tokenization of `reference`. ~~List[Span]~~ | -## Example.to_dict {#to_dict tag="method"} +## Example.to_dict {id="to_dict",tag="method"} Return a [dictionary representation](/api/data-formats#dict-input) of the reference annotation contained in this `Example`. @@ -262,7 +262,7 @@ reference annotation contained in this `Example`. | ----------- | ------------------------------------------------------------------------- | | **RETURNS** | Dictionary representation of the reference annotation. ~~Dict[str, Any]~~ | -## Example.split_sents {#split_sents tag="method"} +## Example.split_sents {id="split_sents",tag="method"} Split one `Example` into multiple `Example` objects, one for each sentence. @@ -282,15 +282,15 @@ Split one `Example` into multiple `Example` objects, one for each sentence. | ----------- | ---------------------------------------------------------------------------- | | **RETURNS** | List of `Example` objects, one for each original sentence. ~~List[Example]~~ | -## Alignment {#alignment-object new="3"} +## Alignment {id="alignment-object",version="3"} Calculate alignment tables between two tokenizations. -### Alignment attributes {#alignment-attributes"} +### Alignment attributes {id="alignment-attributes"} -Alignment attributes are managed using `AlignmentArray`, which is a -simplified version of Thinc's [Ragged](https://thinc.ai/docs/api-types#ragged) -type that only supports the `data` and `length` attributes. +Alignment attributes are managed using `AlignmentArray`, which is a simplified +version of Thinc's [Ragged](https://thinc.ai/docs/api-types#ragged) type that +only supports the `data` and `length` attributes. | Name | Description | | ----- | ------------------------------------------------------------------------------------- | @@ -321,7 +321,7 @@ tokenizations add up to the same string. For example, you'll be able to align > If `a2b.data[1] == a2b.data[2] == 1`, that means that `A[1]` (`"'"`) and > `A[2]` (`"s"`) both align to `B[1]` (`"'s"`). -### Alignment.from_strings {#classmethod tag="function"} +### Alignment.from_strings {id="classmethod",tag="function"} | Name | Description | | ----------- | ------------------------------------------------------------- | diff --git a/website/docs/api/index.md b/website/docs/api/index.mdx similarity index 58% rename from website/docs/api/index.md rename to website/docs/api/index.mdx index a9dc408f6..6c6e1fff4 100644 --- a/website/docs/api/index.md +++ b/website/docs/api/index.mdx @@ -3,6 +3,4 @@ title: Library Architecture next: /api/architectures --- -import Architecture101 from 'usage/101/\_architecture.md' - diff --git a/website/docs/api/kb.md b/website/docs/api/kb.mdx similarity index 92% rename from website/docs/api/kb.md rename to website/docs/api/kb.mdx index b217a1678..887b7fe97 100644 --- a/website/docs/api/kb.md +++ b/website/docs/api/kb.mdx @@ -5,7 +5,7 @@ teaser: (ontology) tag: class source: spacy/kb/kb.pyx -new: 2.2 +version: 2.2 --- The `KnowledgeBase` object is an abstract class providing a method to generate @@ -26,7 +26,7 @@ onwards. -## KnowledgeBase.\_\_init\_\_ {#init tag="method"} +## KnowledgeBase.\_\_init\_\_ {id="init",tag="method"} `KnowledgeBase` is an abstract class and cannot be instantiated. Its child classes should call `__init__()` to set up some necessary attributes. @@ -50,7 +50,7 @@ classes should call `__init__()` to set up some necessary attributes. | `vocab` | The shared vocabulary. ~~Vocab~~ | | `entity_vector_length` | Length of the fixed-size entity vectors. ~~int~~ | -## KnowledgeBase.entity_vector_length {#entity_vector_length tag="property"} +## KnowledgeBase.entity_vector_length {id="entity_vector_length",tag="property"} The length of the fixed-size entity vectors in the knowledge base. @@ -58,7 +58,7 @@ The length of the fixed-size entity vectors in the knowledge base. | ----------- | ------------------------------------------------ | | **RETURNS** | Length of the fixed-size entity vectors. ~~int~~ | -## KnowledgeBase.get_candidates {#get_candidates tag="method"} +## KnowledgeBase.get_candidates {id="get_candidates",tag="method"} Given a certain textual mention as input, retrieve a list of candidate entities of type [`Candidate`](/api/kb#candidate). @@ -77,7 +77,7 @@ of type [`Candidate`](/api/kb#candidate). | `mention` | The textual mention or alias. ~~Span~~ | | **RETURNS** | An iterable of relevant `Candidate` objects. ~~Iterable[Candidate]~~ | -## KnowledgeBase.get_candidates_batch {#get_candidates_batch tag="method"} +## KnowledgeBase.get_candidates_batch {id="get_candidates_batch",tag="method"} Same as [`get_candidates()`](/api/kb#get_candidates), but for an arbitrary number of mentions. The [`EntityLinker`](/api/entitylinker) component will call @@ -103,10 +103,10 @@ to you. | `mentions` | The textual mention or alias. ~~Iterable[Span]~~ | | **RETURNS** | An iterable of iterable with relevant `Candidate` objects. ~~Iterable[Iterable[Candidate]]~~ | -## KnowledgeBase.get_alias_candidates {#get_alias_candidates tag="method"} +## KnowledgeBase.get_alias_candidates {id="get_alias_candidates",tag="method"} -This method is _not_ available from spaCy 3.5 onwards. + This method is _not_ available from spaCy 3.5 onwards. From spaCy 3.5 on `KnowledgeBase` is an abstract class (with @@ -119,7 +119,7 @@ Note: [`InMemoryLookupKB.get_candidates()`](/api/kb_in_memory#get_candidates) defaults to [`InMemoryLookupKB.get_alias_candidates()`](/api/kb_in_memory#get_alias_candidates). -## KnowledgeBase.get_vector {#get_vector tag="method"} +## KnowledgeBase.get_vector {id="get_vector",tag="method"} Given a certain entity ID, retrieve its pretrained entity vector. @@ -134,7 +134,7 @@ Given a certain entity ID, retrieve its pretrained entity vector. | `entity` | The entity ID. ~~str~~ | | **RETURNS** | The entity vector. ~~Iterable[float]~~ | -## KnowledgeBase.get_vectors {#get_vectors tag="method"} +## KnowledgeBase.get_vectors {id="get_vectors",tag="method"} Same as [`get_vector()`](/api/kb#get_vector), but for an arbitrary number of entity IDs. @@ -154,7 +154,7 @@ entities at once, if performance is of concern to you. | `entities` | The entity IDs. ~~Iterable[str]~~ | | **RETURNS** | The entity vectors. ~~Iterable[Iterable[numpy.ndarray]]~~ | -## KnowledgeBase.to_disk {#to_disk tag="method"} +## KnowledgeBase.to_disk {id="to_disk",tag="method"} Save the current state of the knowledge base to a directory. @@ -169,7 +169,7 @@ Save the current state of the knowledge base to a directory. | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `exclude` | List of components to exclude. ~~Iterable[str]~~ | -## KnowledgeBase.from_disk {#from_disk tag="method"} +## KnowledgeBase.from_disk {id="from_disk",tag="method"} Restore the state of the knowledge base from a given directory. Note that the [`Vocab`](/api/vocab) should also be the same as the one used to create the KB. @@ -189,7 +189,7 @@ Restore the state of the knowledge base from a given directory. Note that the | `exclude` | List of components to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `KnowledgeBase` object. ~~KnowledgeBase~~ | -## Candidate {#candidate tag="class"} +## Candidate {id="candidate",tag="class"} A `Candidate` object refers to a textual mention (alias) that may or may not be resolved to a specific entity from a `KnowledgeBase`. This will be used as input @@ -197,7 +197,7 @@ for the entity linking algorithm which will disambiguate the various candidates to the correct one. Each candidate `(alias, entity)` pair is assigned to a certain prior probability. -### Candidate.\_\_init\_\_ {#candidate-init tag="method"} +### Candidate.\_\_init\_\_ {id="candidate-init",tag="method"} Construct a `Candidate` object. Usually this constructor is not called directly, but instead these objects are returned by the `get_candidates` method of the @@ -218,7 +218,7 @@ but instead these objects are returned by the `get_candidates` method of the | `alias_hash` | The hash of the textual mention or alias. ~~int~~ | | `prior_prob` | The prior probability of the `alias` referring to the `entity`. ~~float~~ | -## Candidate attributes {#candidate-attributes} +## Candidate attributes {id="candidate-attributes"} | Name | Description | | --------------- | ------------------------------------------------------------------------ | diff --git a/website/docs/api/kb_in_memory.md b/website/docs/api/kb_in_memory.mdx similarity index 90% rename from website/docs/api/kb_in_memory.md rename to website/docs/api/kb_in_memory.mdx index 9e3279e6a..e85b63c45 100644 --- a/website/docs/api/kb_in_memory.md +++ b/website/docs/api/kb_in_memory.mdx @@ -5,7 +5,7 @@ teaser: information in-memory. tag: class source: spacy/kb/kb_in_memory.pyx -new: 3.5 +version: 3.5 --- The `InMemoryLookupKB` class inherits from [`KnowledgeBase`](/api/kb) and @@ -14,7 +14,7 @@ implements all of its methods. It stores all KB data in-memory and generates entity names. It's highly optimized for both a low memory footprint and speed of retrieval. -## InMemoryLookupKB.\_\_init\_\_ {#init tag="method"} +## InMemoryLookupKB.\_\_init\_\_ {id="init",tag="method"} Create the knowledge base. @@ -31,7 +31,7 @@ Create the knowledge base. | `vocab` | The shared vocabulary. ~~Vocab~~ | | `entity_vector_length` | Length of the fixed-size entity vectors. ~~int~~ | -## InMemoryLookupKB.entity_vector_length {#entity_vector_length tag="property"} +## InMemoryLookupKB.entity_vector_length {id="entity_vector_length",tag="property"} The length of the fixed-size entity vectors in the knowledge base. @@ -39,7 +39,7 @@ The length of the fixed-size entity vectors in the knowledge base. | ----------- | ------------------------------------------------ | | **RETURNS** | Length of the fixed-size entity vectors. ~~int~~ | -## InMemoryLookupKB.add_entity {#add_entity tag="method"} +## InMemoryLookupKB.add_entity {id="add_entity",tag="method"} Add an entity to the knowledge base, specifying its corpus frequency and entity vector, which should be of length @@ -58,7 +58,7 @@ vector, which should be of length | `freq` | The frequency of the entity in a typical corpus. ~~float~~ | | `entity_vector` | The pretrained vector of the entity. ~~numpy.ndarray~~ | -## InMemoryLookupKB.set_entities {#set_entities tag="method"} +## InMemoryLookupKB.set_entities {id="set_entities",tag="method"} Define the full list of entities in the knowledge base, specifying the corpus frequency and entity vector for each entity. @@ -75,7 +75,7 @@ frequency and entity vector for each entity. | `freq_list` | List of entity frequencies. ~~Iterable[int]~~ | | `vector_list` | List of entity vectors. ~~Iterable[numpy.ndarray]~~ | -## InMemoryLookupKB.add_alias {#add_alias tag="method"} +## InMemoryLookupKB.add_alias {id="add_alias",tag="method"} Add an alias or mention to the knowledge base, specifying its potential KB identifiers and their prior probabilities. The entity identifiers should refer @@ -96,7 +96,7 @@ alias. | `entities` | The potential entities that the alias may refer to. ~~Iterable[Union[str, int]]~~ | | `probabilities` | The prior probabilities of each entity. ~~Iterable[float]~~ | -## InMemoryLookupKB.\_\_len\_\_ {#len tag="method"} +## InMemoryLookupKB.\_\_len\_\_ {id="len",tag="method"} Get the total number of entities in the knowledge base. @@ -110,7 +110,7 @@ Get the total number of entities in the knowledge base. | ----------- | ----------------------------------------------------- | | **RETURNS** | The number of entities in the knowledge base. ~~int~~ | -## InMemoryLookupKB.get_entity_strings {#get_entity_strings tag="method"} +## InMemoryLookupKB.get_entity_strings {id="get_entity_strings",tag="method"} Get a list of all entity IDs in the knowledge base. @@ -124,7 +124,7 @@ Get a list of all entity IDs in the knowledge base. | ----------- | --------------------------------------------------------- | | **RETURNS** | The list of entities in the knowledge base. ~~List[str]~~ | -## InMemoryLookupKB.get_size_aliases {#get_size_aliases tag="method"} +## InMemoryLookupKB.get_size_aliases {id="get_size_aliases",tag="method"} Get the total number of aliases in the knowledge base. @@ -138,7 +138,7 @@ Get the total number of aliases in the knowledge base. | ----------- | ---------------------------------------------------- | | **RETURNS** | The number of aliases in the knowledge base. ~~int~~ | -## InMemoryLookupKB.get_alias_strings {#get_alias_strings tag="method"} +## InMemoryLookupKB.get_alias_strings {id="get_alias_strings",tag="method"} Get a list of all aliases in the knowledge base. @@ -152,7 +152,7 @@ Get a list of all aliases in the knowledge base. | ----------- | -------------------------------------------------------- | | **RETURNS** | The list of aliases in the knowledge base. ~~List[str]~~ | -## InMemoryLookupKB.get_candidates {#get_candidates tag="method"} +## InMemoryLookupKB.get_candidates {id="get_candidates",tag="method"} Given a certain textual mention as input, retrieve a list of candidate entities of type [`Candidate`](/api/kb#candidate). Wraps @@ -172,7 +172,7 @@ of type [`Candidate`](/api/kb#candidate). Wraps | `mention` | The textual mention or alias. ~~Span~~ | | **RETURNS** | An iterable of relevant `Candidate` objects. ~~Iterable[Candidate]~~ | -## InMemoryLookupKB.get_candidates_batch {#get_candidates_batch tag="method"} +## InMemoryLookupKB.get_candidates_batch {id="get_candidates_batch",tag="method"} Same as [`get_candidates()`](/api/kb_in_memory#get_candidates), but for an arbitrary number of mentions. The [`EntityLinker`](/api/entitylinker) component @@ -198,7 +198,7 @@ to you. | `mentions` | The textual mention or alias. ~~Iterable[Span]~~ | | **RETURNS** | An iterable of iterable with relevant `Candidate` objects. ~~Iterable[Iterable[Candidate]]~~ | -## InMemoryLookupKB.get_alias_candidates {#get_alias_candidates tag="method"} +## InMemoryLookupKB.get_alias_candidates {id="get_alias_candidates",tag="method"} Given a certain textual mention as input, retrieve a list of candidate entities of type [`Candidate`](/api/kb#candidate). @@ -214,7 +214,7 @@ of type [`Candidate`](/api/kb#candidate). | `alias` | The textual mention or alias. ~~str~~ | | **RETURNS** | The list of relevant `Candidate` objects. ~~List[Candidate]~~ | -## InMemoryLookupKB.get_vector {#get_vector tag="method"} +## InMemoryLookupKB.get_vector {id="get_vector",tag="method"} Given a certain entity ID, retrieve its pretrained entity vector. @@ -229,7 +229,7 @@ Given a certain entity ID, retrieve its pretrained entity vector. | `entity` | The entity ID. ~~str~~ | | **RETURNS** | The entity vector. ~~numpy.ndarray~~ | -## InMemoryLookupKB.get_vectors {#get_vectors tag="method"} +## InMemoryLookupKB.get_vectors {id="get_vectors",tag="method"} Same as [`get_vector()`](/api/kb_in_memory#get_vector), but for an arbitrary number of entity IDs. @@ -249,7 +249,7 @@ entities at once, if performance is of concern to you. | `entities` | The entity IDs. ~~Iterable[str]~~ | | **RETURNS** | The entity vectors. ~~Iterable[Iterable[numpy.ndarray]]~~ | -## InMemoryLookupKB.get_prior_prob {#get_prior_prob tag="method"} +## InMemoryLookupKB.get_prior_prob {id="get_prior_prob",tag="method"} Given a certain entity ID and a certain textual mention, retrieve the prior probability of the fact that the mention links to the entity ID. @@ -266,7 +266,7 @@ probability of the fact that the mention links to the entity ID. | `alias` | The textual mention or alias. ~~str~~ | | **RETURNS** | The prior probability of the `alias` referring to the `entity`. ~~float~~ | -## InMemoryLookupKB.to_disk {#to_disk tag="method"} +## InMemoryLookupKB.to_disk {id="to_disk",tag="method"} Save the current state of the knowledge base to a directory. @@ -281,7 +281,7 @@ Save the current state of the knowledge base to a directory. | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | `exclude` | List of components to exclude. ~~Iterable[str]~~ | -## InMemoryLookupKB.from_disk {#from_disk tag="method"} +## InMemoryLookupKB.from_disk {id="from_disk",tag="method"} Restore the state of the knowledge base from a given directory. Note that the [`Vocab`](/api/vocab) should also be the same as the one used to create the KB. diff --git a/website/docs/api/language.md b/website/docs/api/language.mdx similarity index 96% rename from website/docs/api/language.md rename to website/docs/api/language.mdx index ad0ac2a46..93ddd79a2 100644 --- a/website/docs/api/language.md +++ b/website/docs/api/language.mdx @@ -15,7 +15,7 @@ the tagger or parser that are called on a document in order. You can also add your own processing pipeline components that take a `Doc` object, modify it and return it. -## Language.\_\_init\_\_ {#init tag="method"} +## Language.\_\_init\_\_ {id="init",tag="method"} Initialize a `Language` object. Note that the `meta` is only used for meta information in [`Language.meta`](/api/language#meta) and not to configure the @@ -44,7 +44,7 @@ information in [`Language.meta`](/api/language#meta) and not to configure the | `create_tokenizer` | Optional function that receives the `nlp` object and returns a tokenizer. ~~Callable[[Language], Callable[[str], Doc]]~~ | | `batch_size` | Default batch size for [`pipe`](#pipe) and [`evaluate`](#evaluate). Defaults to `1000`. ~~int~~ | -## Language.from_config {#from_config tag="classmethod" new="3"} +## Language.from_config {id="from_config",tag="classmethod",version="3"} Create a `Language` object from a loaded config. Will set up the tokenizer and language data, add pipeline components based on the pipeline and add pipeline @@ -76,7 +76,7 @@ spaCy loads a model under the hood based on its | `validate` | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ | | **RETURNS** | The initialized object. ~~Language~~ | -## Language.component {#component tag="classmethod" new="3"} +## Language.component {id="component",tag="classmethod",version="3"} Register a custom pipeline component under a given name. This allows initializing the component by name using @@ -112,7 +112,7 @@ decorator. For more details and examples, see the | `retokenizes` | Whether the component changes tokenization. Used for [pipe analysis](/usage/processing-pipelines#analysis). ~~bool~~ | | `func` | Optional function if not used as a decorator. ~~Optional[Callable[[Doc], Doc]]~~ | -## Language.factory {#factory tag="classmethod"} +## Language.factory {id="factory",tag="classmethod"} Register a custom pipeline component factory under a given name. This allows initializing the component by name using @@ -159,7 +159,7 @@ examples, see the | `default_score_weights` | The scores to report during training, and their default weight towards the final score used to select the best model. Weights should sum to `1.0` per component and will be combined and normalized for the whole pipeline. If a weight is set to `None`, the score will not be logged or weighted. ~~Dict[str, Optional[float]]~~ | | `func` | Optional function if not used as a decorator. ~~Optional[Callable[[...], Callable[[Doc], Doc]]]~~ | -## Language.\_\_call\_\_ {#call tag="method"} +## Language.\_\_call\_\_ {id="call",tag="method"} Apply the pipeline to some text. The text can span multiple sentences, and can contain arbitrary whitespace. Alignment into the original string is preserved. @@ -182,7 +182,7 @@ skipped, but the rest of the pipeline is run. | `component_cfg` | Optional dictionary of keyword arguments for components, keyed by component names. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Any]]]~~ | | **RETURNS** | A container for accessing the annotations. ~~Doc~~ | -## Language.pipe {#pipe tag="method"} +## Language.pipe {id="pipe",tag="method"} Process texts as a stream, and yield `Doc` objects in order. This is usually more efficient than processing texts one-by-one. @@ -209,7 +209,7 @@ tokenization is skipped but the rest of the pipeline is run. | `n_process` | Number of processors to use. Defaults to `1`. ~~int~~ | | **YIELDS** | Documents in the order of the original text. ~~Doc~~ | -## Language.set_error_handler {#set_error_handler tag="method" new="3"} +## Language.set_error_handler {id="set_error_handler",tag="method",version="3"} Define a callback that will be invoked when an error is thrown during processing of one or more documents. Specifically, this function will call @@ -231,7 +231,7 @@ being processed, and the original error. | --------------- | -------------------------------------------------------------------------------------------------------------- | | `error_handler` | A function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ | -## Language.initialize {#initialize tag="method" new="3"} +## Language.initialize {id="initialize",tag="method",version="3"} Initialize the pipeline for training and return an [`Optimizer`](https://thinc.ai/docs/api-optimizers). Under the hood, it uses the @@ -282,7 +282,7 @@ objects. | `sgd` | An optimizer. Will be created via [`create_optimizer`](#create_optimizer) if not set. ~~Optional[Optimizer]~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## Language.resume_training {#resume_training tag="method,experimental" new="3"} +## Language.resume_training {id="resume_training",tag="method,experimental",version="3"} Continue training a trained pipeline. Create and return an optimizer, and initialize "rehearsal" for any pipeline component that has a `rehearse` method. @@ -304,7 +304,7 @@ a batch of [Example](/api/example) objects. | `sgd` | An optimizer. Will be created via [`create_optimizer`](#create_optimizer) if not set. ~~Optional[Optimizer]~~ | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## Language.update {#update tag="method"} +## Language.update {id="update",tag="method"} Update the models in the pipeline. @@ -342,7 +342,7 @@ and custom registered functions if needed. See the | `component_cfg` | Optional dictionary of keyword arguments for components, keyed by component names. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Any]]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## Language.rehearse {#rehearse tag="method,experimental" new="3"} +## Language.rehearse {id="rehearse",tag="method,experimental",version="3"} Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the current model to make predictions similar to an initial model, to try to address @@ -364,7 +364,7 @@ the "catastrophic forgetting" problem. This feature is experimental. | `losses` | Dictionary to update with the loss, keyed by pipeline component. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## Language.evaluate {#evaluate tag="method"} +## Language.evaluate {id="evaluate",tag="method"} Evaluate a pipeline's components. @@ -392,7 +392,7 @@ objects instead of tuples of `Doc` and `GoldParse` objects. | `scorer_cfg` | Optional dictionary of keyword arguments for the `Scorer`. Defaults to `None`. ~~Optional[Dict[str, Any]]~~ | | **RETURNS** | A dictionary of evaluation scores. ~~Dict[str, Union[float, Dict[str, float]]]~~ | -## Language.use_params {#use_params tag="contextmanager, method"} +## Language.use_params {id="use_params",tag="contextmanager, method"} Replace weights of models in the pipeline with those provided in the params dictionary. Can be used as a context manager, in which case, models go back to @@ -409,7 +409,7 @@ their original weights after the block. | -------- | ------------------------------------------------------ | | `params` | A dictionary of parameters keyed by model ID. ~~dict~~ | -## Language.add_pipe {#add_pipe tag="method" new="2"} +## Language.add_pipe {id="add_pipe",tag="method",version="2"} Add a component to the processing pipeline. Expects a name that maps to a component factory registered using @@ -458,7 +458,7 @@ component, adds it to the pipeline and returns it. | `validate` 3 | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ | | **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ | -## Language.create_pipe {#create_pipe tag="method" new="2"} +## Language.create_pipe {id="create_pipe",tag="method",version="2"} Create a pipeline component from a factory. @@ -487,7 +487,7 @@ To create a component and add it to the pipeline, you should always use | `validate` 3 | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ | | **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ | -## Language.has_factory {#has_factory tag="classmethod" new="3"} +## Language.has_factory {id="has_factory",tag="classmethod",version="3"} Check whether a factory name is registered on the `Language` class or subclass. Will check for @@ -514,7 +514,7 @@ the `Language` base class, available to all subclasses. | `name` | Name of the pipeline factory to check. ~~str~~ | | **RETURNS** | Whether a factory of that name is registered on the class. ~~bool~~ | -## Language.has_pipe {#has_pipe tag="method" new="2"} +## Language.has_pipe {id="has_pipe",tag="method",version="2"} Check whether a component is present in the pipeline. Equivalent to `name in nlp.pipe_names`. @@ -536,7 +536,7 @@ Check whether a component is present in the pipeline. Equivalent to | `name` | Name of the pipeline component to check. ~~str~~ | | **RETURNS** | Whether a component of that name exists in the pipeline. ~~bool~~ | -## Language.get_pipe {#get_pipe tag="method" new="2"} +## Language.get_pipe {id="get_pipe",tag="method",version="2"} Get a pipeline component for a given component name. @@ -552,7 +552,7 @@ Get a pipeline component for a given component name. | `name` | Name of the pipeline component to get. ~~str~~ | | **RETURNS** | The pipeline component. ~~Callable[[Doc], Doc]~~ | -## Language.replace_pipe {#replace_pipe tag="method" new="2"} +## Language.replace_pipe {id="replace_pipe",tag="method",version="2"} Replace a component in the pipeline and return the new component. @@ -580,7 +580,7 @@ and instead expects the **name of a component factory** registered using | `validate` 3 | Whether to validate the component config and arguments against the types expected by the factory. Defaults to `True`. ~~bool~~ | | **RETURNS** | The new pipeline component. ~~Callable[[Doc], Doc]~~ | -## Language.rename_pipe {#rename_pipe tag="method" new="2"} +## Language.rename_pipe {id="rename_pipe",tag="method",version="2"} Rename a component in the pipeline. Useful to create custom names for pre-defined and pre-loaded components. To change the default name of a component @@ -598,7 +598,7 @@ added to the pipeline, you can also use the `name` argument on | `old_name` | Name of the component to rename. ~~str~~ | | `new_name` | New name of the component. ~~str~~ | -## Language.remove_pipe {#remove_pipe tag="method" new="2"} +## Language.remove_pipe {id="remove_pipe",tag="method",version="2"} Remove a component from the pipeline. Returns the removed component name and component function. @@ -615,7 +615,7 @@ component function. | `name` | Name of the component to remove. ~~str~~ | | **RETURNS** | A `(name, component)` tuple of the removed component. ~~Tuple[str, Callable[[Doc], Doc]]~~ | -## Language.disable_pipe {#disable_pipe tag="method" new="3"} +## Language.disable_pipe {id="disable_pipe",tag="method",version="3"} Temporarily disable a pipeline component so it's not run as part of the pipeline. Disabled components are listed in @@ -641,7 +641,7 @@ does nothing. | ------ | ----------------------------------------- | | `name` | Name of the component to disable. ~~str~~ | -## Language.enable_pipe {#enable_pipe tag="method" new="3"} +## Language.enable_pipe {id="enable_pipe",tag="method",version="3"} Enable a previously disabled component (e.g. via [`Language.disable_pipes`](/api/language#disable_pipes)) so it's run as part of @@ -663,7 +663,7 @@ already enabled, this method does nothing. | ------ | ---------------------------------------- | | `name` | Name of the component to enable. ~~str~~ | -## Language.select_pipes {#select_pipes tag="contextmanager, method" new="3"} +## Language.select_pipes {id="select_pipes",tag="contextmanager, method",version="3"} Disable one or more pipeline components. If used as a context manager, the pipeline will be restored to the initial state at the end of the block. @@ -706,7 +706,7 @@ As of spaCy v3.0, the `disable_pipes` method has been renamed to `select_pipes`: | `enable` | Name(s) of pipeline component(s) that will not be disabled. ~~Optional[Union[str, Iterable[str]]]~~ | | **RETURNS** | The disabled pipes that can be restored by calling the object's `.restore()` method. ~~DisabledPipes~~ | -## Language.get_factory_meta {#get_factory_meta tag="classmethod" new="3"} +## Language.get_factory_meta {id="get_factory_meta",tag="classmethod",version="3"} Get the factory meta information for a given pipeline component name. Expects the name of the component **factory**. The factory meta is an instance of the @@ -728,7 +728,7 @@ information about the component and its default provided by the | `name` | The factory name. ~~str~~ | | **RETURNS** | The factory meta. ~~FactoryMeta~~ | -## Language.get_pipe_meta {#get_pipe_meta tag="method" new="3"} +## Language.get_pipe_meta {id="get_pipe_meta",tag="method",version="3"} Get the factory meta information for a given pipeline component name. Expects the name of the component **instance** in the pipeline. The factory meta is an @@ -751,7 +751,7 @@ contains the information about the component and its default provided by the | `name` | The pipeline component name. ~~str~~ | | **RETURNS** | The factory meta. ~~FactoryMeta~~ | -## Language.analyze_pipes {#analyze_pipes tag="method" new="3"} +## Language.analyze_pipes {id="analyze_pipes",tag="method",version="3"} Analyze the current pipeline components and show a summary of the attributes they assign and require, and the scores they set. The data is based on the @@ -780,8 +780,7 @@ doesn't, the pipeline analysis won't catch that. -```json -### Structured +```json {title="Structured"} { "summary": { "tagger": { @@ -799,7 +798,12 @@ doesn't, the pipeline analysis won't catch that. }, "problems": { "tagger": [], - "entity_linker": ["doc.ents", "doc.sents", "token.ent_iob", "token.ent_type"] + "entity_linker": [ + "doc.ents", + "doc.sents", + "token.ent_iob", + "token.ent_type" + ] }, "attrs": { "token.ent_iob": { "assigns": [], "requires": ["entity_linker"] }, @@ -840,7 +844,7 @@ token.ent_iob, token.ent_type | `pretty` | Pretty-print the results as a table. Defaults to `False`. ~~bool~~ | | **RETURNS** | Dictionary containing the pipe analysis, keyed by `"summary"` (component meta by pipe), `"problems"` (attribute names by pipe) and `"attrs"` (pipes that assign and require an attribute, keyed by attribute). ~~Optional[Dict[str, Any]]~~ | -## Language.replace_listeners {#replace_listeners tag="method" new="3"} +## Language.replace_listeners {id="replace_listeners",tag="method",version="3"} Find [listener layers](/usage/embeddings-transformers#embedding-layers) (connecting to a shared token-to-vector embedding component) of a given pipeline @@ -885,7 +889,7 @@ when loading a config with | `pipe_name` | Name of pipeline component to replace listeners for. ~~str~~ | | `listeners` | The paths to the listeners, relative to the component config, e.g. `["model.tok2vec"]`. Typically, implementations will only connect to one tok2vec component, `model.tok2vec`, but in theory, custom models can use multiple listeners. The value here can either be an empty list to not replace any listeners, or a _complete_ list of the paths to all listener layers used by the model that should be replaced.~~Iterable[str]~~ | -## Language.meta {#meta tag="property"} +## Language.meta {id="meta",tag="property"} Meta data for the `Language` class, including name, version, data sources, license, author information and more. If a trained pipeline is loaded, this @@ -911,7 +915,7 @@ information is expressed in the [`config.cfg`](/api/data-formats#config). | ----------- | --------------------------------- | | **RETURNS** | The meta data. ~~Dict[str, Any]~~ | -## Language.config {#config tag="property" new="3"} +## Language.config {id="config",tag="property",version="3"} Export a trainable [`config.cfg`](/api/data-formats#config) for the current `nlp` object. Includes the current pipeline, all configs used to create the @@ -932,7 +936,7 @@ subclass of the built-in `dict`. It supports the additional methods `to_disk` | ----------- | ---------------------- | | **RETURNS** | The config. ~~Config~~ | -## Language.to_disk {#to_disk tag="method" new="2"} +## Language.to_disk {id="to_disk",tag="method",version="2"} Save the current state to a directory. Under the hood, this method delegates to the `to_disk` methods of the individual pipeline components, if available. This @@ -951,7 +955,7 @@ will be saved to disk. | _keyword-only_ | | | `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## Language.from_disk {#from_disk tag="method" new="2"} +## Language.from_disk {id="from_disk",tag="method",version="2"} Loads state from a directory, including all data that was saved with the `Language` object. Modifies the object in place and returns it. @@ -984,7 +988,7 @@ you want to load a serialized pipeline from a directory, you should use | `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `Language` object. ~~Language~~ | -## Language.to_bytes {#to_bytes tag="method"} +## Language.to_bytes {id="to_bytes",tag="method"} Serialize the current state to a binary string. @@ -1000,7 +1004,7 @@ Serialize the current state to a binary string. | `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~iterable~~ | | **RETURNS** | The serialized form of the `Language` object. ~~bytes~~ | -## Language.from_bytes {#from_bytes tag="method"} +## Language.from_bytes {id="from_bytes",tag="method"} Load state from a binary string. Note that this method is commonly used via the subclasses like `English` or `German` to make language-specific functionality @@ -1028,7 +1032,7 @@ details. | `exclude` | Names of pipeline components or [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `Language` object. ~~Language~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | @@ -1046,7 +1050,7 @@ details. | `disabled` 3 | Names of components that are currently disabled and don't run as part of the pipeline. ~~List[str]~~ | | `path` | Path to the pipeline data directory, if a pipeline is loaded from a path or package. Otherwise `None`. ~~Optional[Path]~~ | -## Class attributes {#class-attributes} +## Class attributes {id="class-attributes"} | Name | Description | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -1054,7 +1058,7 @@ details. | `lang` | [IETF language tag](https://www.w3.org/International/articles/language-tags/), such as 'en' for English. ~~str~~ | | `default_config` | Base [config](/usage/training#config) to use for [Language.config](/api/language#config). Defaults to [`default_config.cfg`](%%GITHUB_SPACY/spacy/default_config.cfg). ~~Config~~ | -## Defaults {#defaults} +## Defaults {id="defaults"} The following attributes can be set on the `Language.Defaults` class to customize the default language data: @@ -1097,7 +1101,7 @@ customize the default language data: | `writing_system` | Information about the language's writing system, available via `Vocab.writing_system`. Defaults to: `{"direction": "ltr", "has_case": True, "has_letters": True}.`.
**Example:** [`zh/__init__.py`](%%GITHUB_SPACY/spacy/lang/zh/__init__.py) ~~Dict[str, Any]~~ | | `config` | Default [config](/usage/training#config) added to `nlp.config`. This can include references to custom tokenizers or lemmatizers.
**Example:** [`zh/__init__.py`](%%GITHUB_SPACY/spacy/lang/zh/__init__.py) ~~Config~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from @@ -1117,7 +1121,7 @@ serialization by passing in the string names via the `exclude` argument. | `meta` | The meta data, available as [`Language.meta`](/api/language#meta). | | ... | String names of pipeline components, e.g. `"ner"`. | -## FactoryMeta {#factorymeta new="3" tag="dataclass"} +## FactoryMeta {id="factorymeta",version="3",tag="dataclass"} The `FactoryMeta` contains the information about the component and its default provided by the [`@Language.component`](/api/language#component) or diff --git a/website/docs/api/legacy.md b/website/docs/api/legacy.mdx similarity index 95% rename from website/docs/api/legacy.md rename to website/docs/api/legacy.mdx index d9167c76f..ea6d3a899 100644 --- a/website/docs/api/legacy.md +++ b/website/docs/api/legacy.mdx @@ -12,11 +12,11 @@ functions that may still be used in projects. You can find the detailed documentation of each such legacy function on this page. -## Architectures {#architectures} +## Architectures {id="architectures"} These functions are available from `@spacy.registry.architectures`. -### spacy.Tok2Vec.v1 {#Tok2Vec_v1} +### spacy.Tok2Vec.v1 {id="Tok2Vec_v1"} The `spacy.Tok2Vec.v1` architecture was expecting an `encode` model of type `Model[Floats2D, Floats2D]` such as `spacy.MaxoutWindowEncoder.v1` or @@ -48,7 +48,7 @@ blog post for background. | `encode` | Encode context into the embeddings, using an architecture such as a CNN, BiLSTM or transformer. For example, [MaxoutWindowEncoder.v1](/api/legacy#MaxoutWindowEncoder_v1). ~~Model[Floats2d, Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | -### spacy.MaxoutWindowEncoder.v1 {#MaxoutWindowEncoder_v1} +### spacy.MaxoutWindowEncoder.v1 {id="MaxoutWindowEncoder_v1"} The `spacy.MaxoutWindowEncoder.v1` architecture was producing a model of type `Model[Floats2D, Floats2D]`. Since `spacy.MaxoutWindowEncoder.v2`, this has been @@ -76,7 +76,7 @@ and residual connections. | `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[Floats2d, Floats2d]~~ | -### spacy.MishWindowEncoder.v1 {#MishWindowEncoder_v1} +### spacy.MishWindowEncoder.v1 {id="MishWindowEncoder_v1"} The `spacy.MishWindowEncoder.v1` architecture was producing a model of type `Model[Floats2D, Floats2D]`. Since `spacy.MishWindowEncoder.v2`, this has been @@ -103,24 +103,24 @@ and residual connections. | `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[Floats2d, Floats2d]~~ | -### spacy.HashEmbedCNN.v1 {#HashEmbedCNN_v1} +### spacy.HashEmbedCNN.v1 {id="HashEmbedCNN_v1"} Identical to [`spacy.HashEmbedCNN.v2`](/api/architectures#HashEmbedCNN) except using [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are included. -### spacy.MultiHashEmbed.v1 {#MultiHashEmbed_v1} +### spacy.MultiHashEmbed.v1 {id="MultiHashEmbed_v1"} Identical to [`spacy.MultiHashEmbed.v2`](/api/architectures#MultiHashEmbed) except with [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are included. -### spacy.CharacterEmbed.v1 {#CharacterEmbed_v1} +### spacy.CharacterEmbed.v1 {id="CharacterEmbed_v1"} Identical to [`spacy.CharacterEmbed.v2`](/api/architectures#CharacterEmbed) except using [`spacy.StaticVectors.v1`](#StaticVectors_v1) if vectors are included. -### spacy.TextCatEnsemble.v1 {#TextCatEnsemble_v1} +### spacy.TextCatEnsemble.v1 {id="TextCatEnsemble_v1"} The `spacy.TextCatEnsemble.v1` architecture built an internal `tok2vec` and `linear_model`. Since `spacy.TextCatEnsemble.v2`, this has been refactored so @@ -158,7 +158,7 @@ network has an internal CNN Tok2Vec layer and uses attention. | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | -### spacy.TextCatCNN.v1 {#TextCatCNN_v1} +### spacy.TextCatCNN.v1 {id="TextCatCNN_v1"} Since `spacy.TextCatCNN.v2`, this architecture has become resizable, which means that you can add labels to a previously trained textcat. `TextCatCNN` v1 did not @@ -194,7 +194,7 @@ architecture is usually less accurate than the ensemble, but runs faster. | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | -### spacy.TextCatBOW.v1 {#TextCatBOW_v1} +### spacy.TextCatBOW.v1 {id="TextCatBOW_v1"} Since `spacy.TextCatBOW.v2`, this architecture has become resizable, which means that you can add labels to a previously trained textcat. `TextCatBOW` v1 did not @@ -222,17 +222,17 @@ the others, but may not be as accurate, especially if texts are short. | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | -### spacy.TransitionBasedParser.v1 {#TransitionBasedParser_v1} +### spacy.TransitionBasedParser.v1 {id="TransitionBasedParser_v1"} Identical to [`spacy.TransitionBasedParser.v2`](/api/architectures#TransitionBasedParser) except the `use_upper` was set to `True` by default. -## Layers {#layers} +## Layers {id="layers"} These functions are available from `@spacy.registry.layers`. -### spacy.StaticVectors.v1 {#StaticVectors_v1} +### spacy.StaticVectors.v1 {id="StaticVectors_v1"} Identical to [`spacy.StaticVectors.v2`](/api/architectures#StaticVectors) except for the handling of tokens without vectors. @@ -246,11 +246,11 @@ added to an existing vectors table. See more details in -## Loggers {#loggers} +## Loggers {id="loggers"} These functions are available from `@spacy.registry.loggers`. -### spacy.ConsoleLogger.v1 {#ConsoleLogger_v1} +### spacy.ConsoleLogger.v1 {id="ConsoleLogger_v1"} > #### Example config > @@ -264,7 +264,7 @@ Writes the results of a training step to the console in a tabular format. -```cli +```bash $ python -m spacy train config.cfg ``` diff --git a/website/docs/api/lemmatizer.md b/website/docs/api/lemmatizer.mdx similarity index 95% rename from website/docs/api/lemmatizer.md rename to website/docs/api/lemmatizer.mdx index 905096338..f6657dbf4 100644 --- a/website/docs/api/lemmatizer.md +++ b/website/docs/api/lemmatizer.mdx @@ -2,7 +2,7 @@ title: Lemmatizer tag: class source: spacy/pipeline/lemmatizer.py -new: 3 +version: 3 teaser: 'Pipeline component for lemmatization' api_string_name: lemmatizer api_trainable: false @@ -32,7 +32,7 @@ available in the pipeline and runs _before_ the lemmatizer. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Lemmas generated by rules or predicted will be saved to `Token.lemma`. @@ -94,7 +94,7 @@ libraries (`pymorphy3`). %%GITHUB_SPACY/spacy/pipeline/lemmatizer.py ``` -## Lemmatizer.\_\_init\_\_ {#init tag="method"} +## Lemmatizer.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -120,7 +120,7 @@ shortcut for this and instantiate the component using its string name and | mode | The lemmatizer mode, e.g. `"lookup"` or `"rule"`. Defaults to `"lookup"`. ~~str~~ | | overwrite | Whether to overwrite existing lemmas. ~~bool~~ | -## Lemmatizer.\_\_call\_\_ {#call tag="method"} +## Lemmatizer.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -140,7 +140,7 @@ and all pipeline components are applied to the `Doc` in order. | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## Lemmatizer.pipe {#pipe tag="method"} +## Lemmatizer.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -161,7 +161,7 @@ applied to the `Doc` in order. | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## Lemmatizer.initialize {#initialize tag="method"} +## Lemmatizer.initialize {id="initialize",tag="method"} Initialize the lemmatizer and load any data resources. This method is typically called by [`Language.initialize`](/api/language#initialize) and lets you @@ -192,7 +192,7 @@ training. At runtime, all data is loaded from disk. | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `lookups` | The lookups object containing the tables such as `"lemma_rules"`, `"lemma_index"`, `"lemma_exc"` and `"lemma_lookup"`. If `None`, default tables are loaded from [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data). Defaults to `None`. ~~Optional[Lookups]~~ | -## Lemmatizer.lookup_lemmatize {#lookup_lemmatize tag="method"} +## Lemmatizer.lookup_lemmatize {id="lookup_lemmatize",tag="method"} Lemmatize a token using a lookup-based approach. If no lemma is found, the original string is returned. @@ -202,7 +202,7 @@ original string is returned. | `token` | The token to lemmatize. ~~Token~~ | | **RETURNS** | A list containing one or more lemmas. ~~List[str]~~ | -## Lemmatizer.rule_lemmatize {#rule_lemmatize tag="method"} +## Lemmatizer.rule_lemmatize {id="rule_lemmatize",tag="method"} Lemmatize a token using a rule-based approach. Typically relies on POS tags. @@ -211,7 +211,7 @@ Lemmatize a token using a rule-based approach. Typically relies on POS tags. | `token` | The token to lemmatize. ~~Token~~ | | **RETURNS** | A list containing one or more lemmas. ~~List[str]~~ | -## Lemmatizer.is_base_form {#is_base_form tag="method"} +## Lemmatizer.is_base_form {id="is_base_form",tag="method"} Check whether we're dealing with an uninflected paradigm, so we can avoid lemmatization entirely. @@ -221,7 +221,7 @@ lemmatization entirely. | `token` | The token to analyze. ~~Token~~ | | **RETURNS** | Whether the token's attributes (e.g., part-of-speech tag, morphological features) describe a base form. ~~bool~~ | -## Lemmatizer.get_lookups_config {#get_lookups_config tag="classmethod"} +## Lemmatizer.get_lookups_config {id="get_lookups_config",tag="classmethod"} Returns the lookups configuration settings for a given mode for use in [`Lemmatizer.load_lookups`](/api/lemmatizer#load_lookups). @@ -231,7 +231,7 @@ Returns the lookups configuration settings for a given mode for use in | `mode` | The lemmatizer mode. ~~str~~ | | **RETURNS** | The required table names and the optional table names. ~~Tuple[List[str], List[str]]~~ | -## Lemmatizer.to_disk {#to_disk tag="method"} +## Lemmatizer.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -248,7 +248,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## Lemmatizer.from_disk {#from_disk tag="method"} +## Lemmatizer.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -266,7 +266,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `Lemmatizer` object. ~~Lemmatizer~~ | -## Lemmatizer.to_bytes {#to_bytes tag="method"} +## Lemmatizer.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -283,7 +283,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `Lemmatizer` object. ~~bytes~~ | -## Lemmatizer.from_bytes {#from_bytes tag="method"} +## Lemmatizer.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -302,7 +302,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `Lemmatizer` object. ~~Lemmatizer~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | --------- | ------------------------------------------- | @@ -310,7 +310,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `lookups` | The lookups object. ~~Lookups~~ | | `mode` | The lemmatizer mode. ~~str~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/lexeme.md b/website/docs/api/lexeme.mdx similarity index 97% rename from website/docs/api/lexeme.md rename to website/docs/api/lexeme.mdx index 557d04cce..539f502f0 100644 --- a/website/docs/api/lexeme.md +++ b/website/docs/api/lexeme.mdx @@ -9,7 +9,7 @@ A `Lexeme` has no string context – it's a word type, as opposed to a word toke It therefore has no part-of-speech tag, dependency parse, or lemma (if lemmatization depends on the part-of-speech tag). -## Lexeme.\_\_init\_\_ {#init tag="method"} +## Lexeme.\_\_init\_\_ {id="init",tag="method"} Create a `Lexeme` object. @@ -18,7 +18,7 @@ Create a `Lexeme` object. | `vocab` | The parent vocabulary. ~~Vocab~~ | | `orth` | The orth id of the lexeme. ~~int~~ | -## Lexeme.set_flag {#set_flag tag="method"} +## Lexeme.set_flag {id="set_flag",tag="method"} Change the value of a boolean flag. @@ -34,7 +34,7 @@ Change the value of a boolean flag. | `flag_id` | The attribute ID of the flag to set. ~~int~~ | | `value` | The new value of the flag. ~~bool~~ | -## Lexeme.check_flag {#check_flag tag="method"} +## Lexeme.check_flag {id="check_flag",tag="method"} Check the value of a boolean flag. @@ -51,7 +51,7 @@ Check the value of a boolean flag. | `flag_id` | The attribute ID of the flag to query. ~~int~~ | | **RETURNS** | The value of the flag. ~~bool~~ | -## Lexeme.similarity {#similarity tag="method" model="vectors"} +## Lexeme.similarity {id="similarity",tag="method",model="vectors"} Compute a semantic similarity estimate. Defaults to cosine over vectors. @@ -70,7 +70,7 @@ Compute a semantic similarity estimate. Defaults to cosine over vectors. | other | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ | | **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ | -## Lexeme.has_vector {#has_vector tag="property" model="vectors"} +## Lexeme.has_vector {id="has_vector",tag="property",model="vectors"} A boolean value indicating whether a word vector is associated with the lexeme. @@ -85,7 +85,7 @@ A boolean value indicating whether a word vector is associated with the lexeme. | ----------- | ------------------------------------------------------- | | **RETURNS** | Whether the lexeme has a vector data attached. ~~bool~~ | -## Lexeme.vector {#vector tag="property" model="vectors"} +## Lexeme.vector {id="vector",tag="property",model="vectors"} A real-valued meaning representation. @@ -101,7 +101,7 @@ A real-valued meaning representation. | ----------- | ------------------------------------------------------------------------------------------------ | | **RETURNS** | A 1-dimensional array representing the lexeme's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | -## Lexeme.vector_norm {#vector_norm tag="property" model="vectors"} +## Lexeme.vector_norm {id="vector_norm",tag="property",model="vectors"} The L2 norm of the lexeme's vector representation. @@ -119,7 +119,7 @@ The L2 norm of the lexeme's vector representation. | ----------- | --------------------------------------------------- | | **RETURNS** | The L2 norm of the vector representation. ~~float~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/website/docs/api/lookups.md b/website/docs/api/lookups.mdx similarity index 89% rename from website/docs/api/lookups.md rename to website/docs/api/lookups.mdx index 9565e478f..71a857c60 100644 --- a/website/docs/api/lookups.md +++ b/website/docs/api/lookups.mdx @@ -3,7 +3,7 @@ title: Lookups teaser: A container for large lookup tables and dictionaries tag: class source: spacy/lookups.py -new: 2.2 +version: 2.2 --- This class allows convenient access to large lookup tables and dictionaries, @@ -13,7 +13,7 @@ can be accessed before the pipeline components are applied (e.g. in the tokenizer and lemmatizer), as well as within the pipeline components via `doc.vocab.lookups`. -## Lookups.\_\_init\_\_ {#init tag="method"} +## Lookups.\_\_init\_\_ {id="init",tag="method"} Create a `Lookups` object. @@ -24,7 +24,7 @@ Create a `Lookups` object. > lookups = Lookups() > ``` -## Lookups.\_\_len\_\_ {#len tag="method"} +## Lookups.\_\_len\_\_ {id="len",tag="method"} Get the current number of tables in the lookups. @@ -39,7 +39,7 @@ Get the current number of tables in the lookups. | ----------- | -------------------------------------------- | | **RETURNS** | The number of tables in the lookups. ~~int~~ | -## Lookups.\_\contains\_\_ {#contains tag="method"} +## Lookups.\_\_contains\_\_ {id="contains",tag="method"} Check if the lookups contain a table of a given name. Delegates to [`Lookups.has_table`](/api/lookups#has_table). @@ -57,7 +57,7 @@ Check if the lookups contain a table of a given name. Delegates to | `name` | Name of the table. ~~str~~ | | **RETURNS** | Whether a table of that name is in the lookups. ~~bool~~ | -## Lookups.tables {#tables tag="property"} +## Lookups.tables {id="tables",tag="property"} Get the names of all tables in the lookups. @@ -73,7 +73,7 @@ Get the names of all tables in the lookups. | ----------- | ------------------------------------------------- | | **RETURNS** | Names of the tables in the lookups. ~~List[str]~~ | -## Lookups.add_table {#add_table tag="method"} +## Lookups.add_table {id="add_table",tag="method"} Add a new table with optional data to the lookups. Raises an error if the table exists. @@ -91,7 +91,7 @@ exists. | `data` | Optional data to add to the table. ~~dict~~ | | **RETURNS** | The newly added table. ~~Table~~ | -## Lookups.get_table {#get_table tag="method"} +## Lookups.get_table {id="get_table",tag="method"} Get a table from the lookups. Raises an error if the table doesn't exist. @@ -109,7 +109,7 @@ Get a table from the lookups. Raises an error if the table doesn't exist. | `name` | Name of the table. ~~str~~ | | **RETURNS** | The table. ~~Table~~ | -## Lookups.remove_table {#remove_table tag="method"} +## Lookups.remove_table {id="remove_table",tag="method"} Remove a table from the lookups. Raises an error if the table doesn't exist. @@ -127,7 +127,7 @@ Remove a table from the lookups. Raises an error if the table doesn't exist. | `name` | Name of the table to remove. ~~str~~ | | **RETURNS** | The removed table. ~~Table~~ | -## Lookups.has_table {#has_table tag="method"} +## Lookups.has_table {id="has_table",tag="method"} Check if the lookups contain a table of a given name. Equivalent to [`Lookups.__contains__`](/api/lookups#contains). @@ -145,7 +145,7 @@ Check if the lookups contain a table of a given name. Equivalent to | `name` | Name of the table. ~~str~~ | | **RETURNS** | Whether a table of that name is in the lookups. ~~bool~~ | -## Lookups.to_bytes {#to_bytes tag="method"} +## Lookups.to_bytes {id="to_bytes",tag="method"} Serialize the lookups to a bytestring. @@ -159,7 +159,7 @@ Serialize the lookups to a bytestring. | ----------- | --------------------------------- | | **RETURNS** | The serialized lookups. ~~bytes~~ | -## Lookups.from_bytes {#from_bytes tag="method"} +## Lookups.from_bytes {id="from_bytes",tag="method"} Load the lookups from a bytestring. @@ -176,7 +176,7 @@ Load the lookups from a bytestring. | `bytes_data` | The data to load from. ~~bytes~~ | | **RETURNS** | The loaded lookups. ~~Lookups~~ | -## Lookups.to_disk {#to_disk tag="method"} +## Lookups.to_disk {id="to_disk",tag="method"} Save the lookups to a directory as `lookups.bin`. Expects a path to a directory, which will be created if it doesn't exist. @@ -191,7 +191,7 @@ which will be created if it doesn't exist. | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | -## Lookups.from_disk {#from_disk tag="method"} +## Lookups.from_disk {id="from_disk",tag="method"} Load lookups from a directory containing a `lookups.bin`. Will skip loading if the file doesn't exist. @@ -209,7 +209,7 @@ the file doesn't exist. | `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | **RETURNS** | The loaded lookups. ~~Lookups~~ | -## Table {#table tag="class, ordererddict"} +## Table {id="table",tag="class, ordererddict"} A table in the lookups. Subclass of `OrderedDict` that implements a slightly more consistent and unified API and includes a Bloom filter to speed up missed @@ -218,7 +218,7 @@ lookups. Supports **all other methods and attributes** of `OrderedDict` / accept both integers and strings (which will be hashed before being added to the table). -### Table.\_\_init\_\_ {#table.init tag="method"} +### Table.\_\_init\_\_ {id="table.init",tag="method"} Initialize a new table. @@ -236,7 +236,7 @@ Initialize a new table. | ------ | ------------------------------------------ | | `name` | Optional table name for reference. ~~str~~ | -### Table.from_dict {#table.from_dict tag="classmethod"} +### Table.from_dict {id="table.from_dict",tag="classmethod"} Initialize a new table from a dict. @@ -254,7 +254,7 @@ Initialize a new table from a dict. | `name` | Optional table name for reference. ~~str~~ | | **RETURNS** | The newly constructed object. ~~Table~~ | -### Table.set {#table.set tag="method"} +### Table.set {id="table.set",tag="method"} Set a new key / value pair. String keys will be hashed. Same as `table[key] = value`. @@ -273,7 +273,7 @@ Set a new key / value pair. String keys will be hashed. Same as | `key` | The key. ~~Union[str, int]~~ | | `value` | The value. | -### Table.to_bytes {#table.to_bytes tag="method"} +### Table.to_bytes {id="table.to_bytes",tag="method"} Serialize the table to a bytestring. @@ -287,7 +287,7 @@ Serialize the table to a bytestring. | ----------- | ------------------------------- | | **RETURNS** | The serialized table. ~~bytes~~ | -### Table.from_bytes {#table.from_bytes tag="method"} +### Table.from_bytes {id="table.from_bytes",tag="method"} Load a table from a bytestring. @@ -304,7 +304,7 @@ Load a table from a bytestring. | `bytes_data` | The data to load. ~~bytes~~ | | **RETURNS** | The loaded table. ~~Table~~ | -### Attributes {#table-attributes} +### Attributes {id="table-attributes"} | Name | Description | | -------------- | ------------------------------------------------------------- | diff --git a/website/docs/api/matcher.md b/website/docs/api/matcher.mdx similarity index 97% rename from website/docs/api/matcher.md rename to website/docs/api/matcher.mdx index bd5f6ac24..c66579da8 100644 --- a/website/docs/api/matcher.md +++ b/website/docs/api/matcher.mdx @@ -13,7 +13,7 @@ tokens in context. For in-depth examples and workflows for combining rules and statistical models, see the [usage guide](/usage/rule-based-matching) on rule-based matching. -## Pattern format {#patterns} +## Pattern format {id="patterns"} > ```json > ### Example @@ -101,7 +101,7 @@ it compares to another value. As of spaCy v3.5, `REGEX` and `FUZZY` can be used in combination with `IN` and `NOT_IN`. -## Matcher.\_\_init\_\_ {#init tag="method"} +## Matcher.\_\_init\_\_ {id="init",tag="method"} Create the rule-based `Matcher`. If `validate=True` is set, all patterns added to the matcher will be validated against a JSON schema and a `MatchPatternError` @@ -121,7 +121,7 @@ string where an integer is expected) or unexpected property names. | `validate` | Validate all patterns added to this matcher. ~~bool~~ | | `fuzzy_compare` | The comparison method used for the `FUZZY` operators. ~~Callable[[str, str, int], bool]~~ | -## Matcher.\_\_call\_\_ {#call tag="method"} +## Matcher.\_\_call\_\_ {id="call",tag="method"} Find all token sequences matching the supplied patterns on the `Doc` or `Span`. @@ -150,7 +150,7 @@ the match. | `with_alignments` 3.0.6 | Return match alignment information as part of the match tuple as `List[int]` with the same length as the matched span. Each entry denotes the corresponding index of the token in the pattern. If `as_spans` is set to `True`, this setting is ignored. Defaults to `False`. ~~bool~~ | | **RETURNS** | A list of `(match_id, start, end)` tuples, describing the matches. A match tuple describes a span `doc[start:end`]. The `match_id` is the ID of the added match pattern. If `as_spans` is set to `True`, a list of `Span` objects is returned instead. ~~Union[List[Tuple[int, int, int]], List[Span]]~~ | -## Matcher.\_\_len\_\_ {#len tag="method" new="2"} +## Matcher.\_\_len\_\_ {id="len",tag="method",version="2"} Get the number of rules added to the matcher. Note that this only returns the number of rules (identical with the number of IDs), not the number of individual @@ -169,7 +169,7 @@ patterns. | ----------- | ---------------------------- | | **RETURNS** | The number of rules. ~~int~~ | -## Matcher.\_\_contains\_\_ {#contains tag="method" new="2"} +## Matcher.\_\_contains\_\_ {id="contains",tag="method",version="2"} Check whether the matcher contains rules for a match ID. @@ -187,7 +187,7 @@ Check whether the matcher contains rules for a match ID. | `key` | The match ID. ~~str~~ | | **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ | -## Matcher.add {#add tag="method" new="2"} +## Matcher.add {id="add",tag="method",version="2"} Add a rule to the matcher, consisting of an ID key, one or more patterns, and an optional callback function to act on the matches. The callback function will @@ -233,7 +233,7 @@ patterns = [[{"TEXT": "Google"}, {"TEXT": "Now"}], [{"TEXT": "GoogleNow"}]] | `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[Matcher, Doc, int, List[tuple], Any]]~~ | | `greedy` 3 | Optional filter for greedy matches. Can either be `"FIRST"` or `"LONGEST"`. ~~Optional[str]~~ | -## Matcher.remove {#remove tag="method" new="2"} +## Matcher.remove {id="remove",tag="method",version="2"} Remove a rule from the matcher. A `KeyError` is raised if the match ID does not exist. @@ -251,7 +251,7 @@ exist. | ----- | --------------------------------- | | `key` | The ID of the match rule. ~~str~~ | -## Matcher.get {#get tag="method" new="2"} +## Matcher.get {id="get",tag="method",version="2"} Retrieve the pattern stored for a key. Returns the rule as an `(on_match, patterns)` tuple containing the callback and available patterns. diff --git a/website/docs/api/morphologizer.md b/website/docs/api/morphologizer.mdx similarity index 95% rename from website/docs/api/morphologizer.md rename to website/docs/api/morphologizer.mdx index f874e8bea..f097f2ae3 100644 --- a/website/docs/api/morphologizer.md +++ b/website/docs/api/morphologizer.mdx @@ -2,7 +2,7 @@ title: Morphologizer tag: class source: spacy/pipeline/morphologizer.pyx -new: 3 +version: 3 teaser: 'Pipeline component for predicting morphological features' api_base_class: /api/tagger api_string_name: morphologizer @@ -15,7 +15,7 @@ coarse-grained POS tags following the Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) annotation guidelines. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Predictions are saved to `Token.morph` and `Token.pos`. @@ -25,7 +25,7 @@ Predictions are saved to `Token.morph` and `Token.pos`. | `Token.pos_` | The UPOS part of speech. ~~str~~ | | `Token.morph` | Morphological features. ~~MorphAnalysis~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -53,7 +53,7 @@ architectures and their arguments and hyperparameters. %%GITHUB_SPACY/spacy/pipeline/morphologizer.pyx ``` -## Morphologizer.\_\_init\_\_ {#init tag="method"} +## Morphologizer.\_\_init\_\_ {id="init",tag="method"} Create a new pipeline instance. In your application, you would normally use a shortcut for this and instantiate the component using its string name and @@ -97,7 +97,7 @@ annotation `C=E|X=Y`): | `extend` 3.2 | Whether existing feature types (whose values may or may not be overwritten depending on `overwrite`) are preserved. Defaults to `False`. ~~bool~~ | | `scorer` 3.2 | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attributes `"pos"` and `"morph"` and [`Scorer.score_token_attr_per_feat`](/api/scorer#score_token_attr_per_feat) for the attribute `"morph"`. ~~Optional[Callable]~~ | -## Morphologizer.\_\_call\_\_ {#call tag="method"} +## Morphologizer.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -120,7 +120,7 @@ delegate to the [`predict`](/api/morphologizer#predict) and | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## Morphologizer.pipe {#pipe tag="method"} +## Morphologizer.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -144,7 +144,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/morphologizer#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## Morphologizer.initialize {#initialize tag="method"} +## Morphologizer.initialize {id="initialize",tag="method"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -181,7 +181,7 @@ config. | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[dict]~~ | -## Morphologizer.predict {#predict tag="method"} +## Morphologizer.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects, without modifying them. @@ -198,7 +198,7 @@ modifying them. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The model's prediction for each document. | -## Morphologizer.set_annotations {#set_annotations tag="method"} +## Morphologizer.set_annotations {id="set_annotations",tag="method"} Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. @@ -215,7 +215,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `scores` | The scores to set, produced by `Morphologizer.predict`. | -## Morphologizer.update {#update tag="method"} +## Morphologizer.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects containing the predictions and gold-standard annotations, and update the component's model. @@ -239,7 +239,7 @@ Delegates to [`predict`](/api/morphologizer#predict) and | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## Morphologizer.get_loss {#get_loss tag="method"} +## Morphologizer.get_loss {id="get_loss",tag="method"} Find the loss and gradient of loss for the batch of documents and their predicted scores. @@ -258,7 +258,7 @@ predicted scores. | `scores` | Scores representing the model's predictions. | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | -## Morphologizer.create_optimizer {#create_optimizer tag="method"} +## Morphologizer.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -273,7 +273,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## Morphologizer.use_params {#use_params tag="method, contextmanager"} +## Morphologizer.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored. @@ -290,7 +290,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## Morphologizer.add_label {#add_label tag="method"} +## Morphologizer.add_label {id="add_label",tag="method"} Add a new label to the pipe. If the `Morphologizer` should set annotations for both `pos` and `morph`, the label should include the UPOS as the feature `POS`. @@ -313,7 +313,7 @@ will be automatically added to the model, and the output dimension will be | `label` | The label to add. ~~str~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | -## Morphologizer.to_disk {#to_disk tag="method"} +## Morphologizer.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -330,7 +330,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## Morphologizer.from_disk {#from_disk tag="method"} +## Morphologizer.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -348,7 +348,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `Morphologizer` object. ~~Morphologizer~~ | -## Morphologizer.to_bytes {#to_bytes tag="method"} +## Morphologizer.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -365,7 +365,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `Morphologizer` object. ~~bytes~~ | -## Morphologizer.from_bytes {#from_bytes tag="method"} +## Morphologizer.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -384,7 +384,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `Morphologizer` object. ~~Morphologizer~~ | -## Morphologizer.labels {#labels tag="property"} +## Morphologizer.labels {id="labels",tag="property"} The labels currently added to the component in the Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) @@ -403,7 +403,7 @@ coarse-grained POS as the feature `POS`. | ----------- | ------------------------------------------------------ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | -## Morphologizer.label_data {#label_data tag="property" new="3"} +## Morphologizer.label_data {id="label_data",tag="property",version="3"} The labels currently added to the component and their internal meta information. This is the data generated by [`init labels`](/api/cli#init-labels) and used by @@ -421,7 +421,7 @@ model with a pre-defined label set. | ----------- | ----------------------------------------------- | | **RETURNS** | The label data added to the component. ~~dict~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/morphology.md b/website/docs/api/morphology.mdx similarity index 89% rename from website/docs/api/morphology.md rename to website/docs/api/morphology.mdx index 20fcd1a40..68d80b814 100644 --- a/website/docs/api/morphology.md +++ b/website/docs/api/morphology.mdx @@ -10,7 +10,7 @@ morphological analysis, so queries of morphological attributes are delegated to this class. See [`MorphAnalysis`](/api/morphology#morphanalysis) for the container storing a single morphological analysis. -## Morphology.\_\_init\_\_ {#init tag="method"} +## Morphology.\_\_init\_\_ {id="init",tag="method"} Create a `Morphology` object. @@ -26,7 +26,7 @@ Create a `Morphology` object. | --------- | --------------------------------- | | `strings` | The string store. ~~StringStore~~ | -## Morphology.add {#add tag="method"} +## Morphology.add {id="add",tag="method"} Insert a morphological analysis in the morphology table, if not already present. The morphological analysis may be provided in the Universal Dependencies @@ -46,7 +46,7 @@ new analysis. | ---------- | ------------------------------------------------ | | `features` | The morphological features. ~~Union[Dict, str]~~ | -## Morphology.get {#get tag="method"} +## Morphology.get {id="get",tag="method"} > #### Example > @@ -64,7 +64,7 @@ string for the hash of the morphological analysis. | ------- | ----------------------------------------------- | | `morph` | The hash of the morphological analysis. ~~int~~ | -## Morphology.feats_to_dict {#feats_to_dict tag="staticmethod"} +## Morphology.feats_to_dict {id="feats_to_dict",tag="staticmethod"} Convert a string [FEATS](https://universaldependencies.org/format.html#morphological-annotation) @@ -84,7 +84,7 @@ tag map. | `feats` | The morphological features in Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ | | **RETURNS** | The morphological features as a dictionary. ~~Dict[str, str]~~ | -## Morphology.dict_to_feats {#dict_to_feats tag="staticmethod"} +## Morphology.dict_to_feats {id="dict_to_feats",tag="staticmethod"} Convert a dictionary of features and values to a string [FEATS](https://universaldependencies.org/format.html#morphological-annotation) @@ -103,19 +103,19 @@ representation. | `feats_dict` | The morphological features as a dictionary. ~~Dict[str, str]~~ | | **RETURNS** | The morphological features in Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} -| Name | Description | -| ------------- | ------------------------------------------------------------------------------------------------------------------------------ | -| `FEATURE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) feature separator. Default is `|`. ~~str~~ | -| `FIELD_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) field separator. Default is `=`. ~~str~~ | -| `VALUE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) value separator. Default is `,`. ~~str~~ | +| Name | Description | +| ------------- | ------------------------------------------------------------------------------------------------------------------------------- | +| `FEATURE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) feature separator. Default is `\|`. ~~str~~ | +| `FIELD_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) field separator. Default is `=`. ~~str~~ | +| `VALUE_SEP` | The [FEATS](https://universaldependencies.org/format.html#morphological-annotation) value separator. Default is `,`. ~~str~~ | -## MorphAnalysis {#morphanalysis tag="class" source="spacy/tokens/morphanalysis.pyx"} +## MorphAnalysis {id="morphanalysis",tag="class",source="spacy/tokens/morphanalysis.pyx"} Stores a single morphological analysis. -### MorphAnalysis.\_\_init\_\_ {#morphanalysis-init tag="method"} +### MorphAnalysis.\_\_init\_\_ {id="morphanalysis-init",tag="method"} Initialize a MorphAnalysis object from a Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) @@ -135,7 +135,7 @@ string or a dictionary of morphological features. | `vocab` | The vocab. ~~Vocab~~ | | `features` | The morphological features. ~~Union[Dict[str, str], str]~~ | -### MorphAnalysis.\_\_contains\_\_ {#morphanalysis-contains tag="method"} +### MorphAnalysis.\_\_contains\_\_ {id="morphanalysis-contains",tag="method"} Whether a feature/value pair is in the analysis. @@ -151,7 +151,7 @@ Whether a feature/value pair is in the analysis. | ----------- | --------------------------------------------- | | **RETURNS** | A feature/value pair in the analysis. ~~str~~ | -### MorphAnalysis.\_\_iter\_\_ {#morphanalysis-iter tag="method"} +### MorphAnalysis.\_\_iter\_\_ {id="morphanalysis-iter",tag="method"} Iterate over the feature/value pairs in the analysis. @@ -167,7 +167,7 @@ Iterate over the feature/value pairs in the analysis. | ---------- | --------------------------------------------- | | **YIELDS** | A feature/value pair in the analysis. ~~str~~ | -### MorphAnalysis.\_\_len\_\_ {#morphanalysis-len tag="method"} +### MorphAnalysis.\_\_len\_\_ {id="morphanalysis-len",tag="method"} Returns the number of features in the analysis. @@ -183,7 +183,7 @@ Returns the number of features in the analysis. | ----------- | ----------------------------------------------- | | **RETURNS** | The number of features in the analysis. ~~int~~ | -### MorphAnalysis.\_\_str\_\_ {#morphanalysis-str tag="method"} +### MorphAnalysis.\_\_str\_\_ {id="morphanalysis-str",tag="method"} Returns the morphological analysis in the Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) @@ -201,7 +201,7 @@ string format. | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **RETURNS** | The analysis in the Universal Dependencies [FEATS](https://universaldependencies.org/format.html#morphological-annotation) format. ~~str~~ | -### MorphAnalysis.get {#morphanalysis-get tag="method"} +### MorphAnalysis.get {id="morphanalysis-get",tag="method"} Retrieve values for a feature by field. @@ -218,7 +218,7 @@ Retrieve values for a feature by field. | `field` | The field to retrieve. ~~str~~ | | **RETURNS** | A list of the individual features. ~~List[str]~~ | -### MorphAnalysis.to_dict {#morphanalysis-to_dict tag="method"} +### MorphAnalysis.to_dict {id="morphanalysis-to_dict",tag="method"} Produce a dict representation of the analysis, in the same format as the tag map. @@ -235,7 +235,7 @@ map. | ----------- | ----------------------------------------------------------- | | **RETURNS** | The dict representation of the analysis. ~~Dict[str, str]~~ | -### MorphAnalysis.from_id {#morphanalysis-from_id tag="classmethod"} +### MorphAnalysis.from_id {id="morphanalysis-from_id",tag="classmethod"} Create a morphological analysis from a given hash ID. diff --git a/website/docs/api/phrasematcher.md b/website/docs/api/phrasematcher.mdx similarity index 96% rename from website/docs/api/phrasematcher.md rename to website/docs/api/phrasematcher.mdx index cd419ae5c..14ccefb77 100644 --- a/website/docs/api/phrasematcher.md +++ b/website/docs/api/phrasematcher.mdx @@ -3,7 +3,7 @@ title: PhraseMatcher teaser: Match sequences of tokens, based on documents tag: class source: spacy/matcher/phrasematcher.pyx -new: 2 +version: 2 --- The `PhraseMatcher` lets you efficiently match large terminology lists. While @@ -12,7 +12,7 @@ descriptions, the `PhraseMatcher` accepts match patterns in the form of `Doc` objects. See the [usage guide](/usage/rule-based-matching#phrasematcher) for examples. -## PhraseMatcher.\_\_init\_\_ {#init tag="method"} +## PhraseMatcher.\_\_init\_\_ {id="init",tag="method"} Create the rule-based `PhraseMatcher`. Setting a different `attr` to match on will change the token attributes that will be compared to determine a match. By @@ -42,7 +42,7 @@ be shown. | `attr` | The token attribute to match on. Defaults to `ORTH`, i.e. the verbatim token text. ~~Union[int, str]~~ | | `validate` | Validate patterns added to the matcher. ~~bool~~ | -## PhraseMatcher.\_\_call\_\_ {#call tag="method"} +## PhraseMatcher.\_\_call\_\_ {id="call",tag="method"} Find all token sequences matching the supplied patterns on the `Doc` or `Span`. @@ -76,7 +76,7 @@ match_id_string = nlp.vocab.strings[match_id] -## PhraseMatcher.\_\_len\_\_ {#len tag="method"} +## PhraseMatcher.\_\_len\_\_ {id="len",tag="method"} Get the number of rules added to the matcher. Note that this only returns the number of rules (identical with the number of IDs), not the number of individual @@ -95,7 +95,7 @@ patterns. | ----------- | ---------------------------- | | **RETURNS** | The number of rules. ~~int~~ | -## PhraseMatcher.\_\_contains\_\_ {#contains tag="method"} +## PhraseMatcher.\_\_contains\_\_ {id="contains",tag="method"} Check whether the matcher contains rules for a match ID. @@ -113,7 +113,7 @@ Check whether the matcher contains rules for a match ID. | `key` | The match ID. ~~str~~ | | **RETURNS** | Whether the matcher contains rules for this match ID. ~~bool~~ | -## PhraseMatcher.add {#add tag="method"} +## PhraseMatcher.add {id="add",tag="method"} Add a rule to the matcher, consisting of an ID key, one or more patterns, and a callback function to act on the matches. The callback function will receive the @@ -155,7 +155,7 @@ patterns = [nlp("health care reform"), nlp("healthcare reform")] | _keyword-only_ | | | `on_match` | Callback function to act on matches. Takes the arguments `matcher`, `doc`, `i` and `matches`. ~~Optional[Callable[[Matcher, Doc, int, List[tuple], Any]]~~ | -## PhraseMatcher.remove {#remove tag="method" new="2.2"} +## PhraseMatcher.remove {id="remove",tag="method",version="2.2"} Remove a rule from the matcher by match ID. A `KeyError` is raised if the key does not exist. diff --git a/website/docs/api/pipe.md b/website/docs/api/pipe.mdx similarity index 93% rename from website/docs/api/pipe.md rename to website/docs/api/pipe.mdx index 263942e3e..c2777edf0 100644 --- a/website/docs/api/pipe.md +++ b/website/docs/api/pipe.mdx @@ -12,7 +12,7 @@ spaCy pipeline. See the docs on [writing trainable components](/usage/processing-pipelines#trainable-components) for how to use the `TrainablePipe` base class to implement custom components. - +{/* TODO: Pipe vs TrainablePipe, check methods below (all renamed to TrainablePipe for now) */} > #### Why is it implemented in Cython? > @@ -27,7 +27,7 @@ for how to use the `TrainablePipe` base class to implement custom components. %%GITHUB_SPACY/spacy/pipeline/trainable_pipe.pyx ``` -## TrainablePipe.\_\_init\_\_ {#init tag="method"} +## TrainablePipe.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -54,7 +54,7 @@ shortcut for this and instantiate the component using its string name and | `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ | | `**cfg` | Additional config parameters and settings. Will be available as the dictionary `cfg` and is serialized with the component. | -## TrainablePipe.\_\_call\_\_ {#call tag="method"} +## TrainablePipe.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -77,7 +77,7 @@ and all pipeline components are applied to the `Doc` in order. Both | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## TrainablePipe.pipe {#pipe tag="method"} +## TrainablePipe.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -100,7 +100,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/pipe#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## TrainablePipe.set_error_handler {#set_error_handler tag="method" new="3"} +## TrainablePipe.set_error_handler {id="set_error_handler",tag="method",version="3"} Define a callback that will be invoked when an error is thrown during processing of one or more documents with either [`__call__`](/api/pipe#call) or @@ -122,7 +122,7 @@ processed, and the original error. | --------------- | -------------------------------------------------------------------------------------------------------------- | | `error_handler` | A function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ | -## TrainablePipe.get_error_handler {#get_error_handler tag="method" new="3"} +## TrainablePipe.get_error_handler {id="get_error_handler",tag="method",version="3"} Retrieve the callback that performs error handling for this component's [`__call__`](/api/pipe#call) and [`pipe`](/api/pipe#pipe) methods. If no custom @@ -141,7 +141,7 @@ returned that simply reraises the exception. | ----------- | ---------------------------------------------------------------------------------------------------------------- | | **RETURNS** | The function that performs custom error handling. ~~Callable[[str, Callable[[Doc], Doc], List[Doc], Exception]~~ | -## TrainablePipe.initialize {#initialize tag="method" new="3"} +## TrainablePipe.initialize {id="initialize",tag="method",version="3"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. The data examples are @@ -171,7 +171,7 @@ This method was previously called `begin_training`. | _keyword-only_ | | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | -## TrainablePipe.predict {#predict tag="method"} +## TrainablePipe.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects, without modifying them. @@ -194,7 +194,7 @@ This method needs to be overwritten with your own custom `predict` method. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The model's prediction for each document. | -## TrainablePipe.set_annotations {#set_annotations tag="method"} +## TrainablePipe.set_annotations {id="set_annotations",tag="method"} Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. @@ -218,7 +218,7 @@ method. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `scores` | The scores to set, produced by `Tagger.predict`. | -## TrainablePipe.update {#update tag="method"} +## TrainablePipe.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects containing the predictions and gold-standard annotations, and update the component's model. @@ -240,7 +240,7 @@ predictions and gold-standard annotations, and update the component's model. | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## TrainablePipe.rehearse {#rehearse tag="method,experimental" new="3"} +## TrainablePipe.rehearse {id="rehearse",tag="method,experimental",version="3"} Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the current model to make predictions similar to an initial model, to try to address @@ -262,7 +262,7 @@ the "catastrophic forgetting" problem. This feature is experimental. | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## TrainablePipe.get_loss {#get_loss tag="method"} +## TrainablePipe.get_loss {id="get_loss",tag="method"} Find the loss and gradient of loss for the batch of documents and their predicted scores. @@ -287,7 +287,7 @@ This method needs to be overwritten with your own custom `get_loss` method. | `scores` | Scores representing the model's predictions. | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | -## TrainablePipe.score {#score tag="method" new="3"} +## TrainablePipe.score {id="score",tag="method",version="3"} Score a batch of examples. @@ -304,7 +304,7 @@ Score a batch of examples. | `\*\*kwargs` | Any additional settings to pass on to the scorer. ~~Any~~ | | **RETURNS** | The scores, e.g. produced by the [`Scorer`](/api/scorer). ~~Dict[str, Union[float, Dict[str, float]]]~~ | -## TrainablePipe.create_optimizer {#create_optimizer tag="method"} +## TrainablePipe.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. Defaults to [`Adam`](https://thinc.ai/docs/api-optimizers#adam) with default settings. @@ -320,7 +320,7 @@ Create an optimizer for the pipeline component. Defaults to | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## TrainablePipe.use_params {#use_params tag="method, contextmanager"} +## TrainablePipe.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored. @@ -337,7 +337,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## TrainablePipe.finish_update {#finish_update tag="method"} +## TrainablePipe.finish_update {id="finish_update",tag="method"} Update parameters using the current parameter gradients. Defaults to calling [`self.model.finish_update`](https://thinc.ai/docs/api-model#finish_update). @@ -355,7 +355,7 @@ Update parameters using the current parameter gradients. Defaults to calling | ----- | ------------------------------------- | | `sgd` | An optimizer. ~~Optional[Optimizer]~~ | -## TrainablePipe.add_label {#add_label tag="method"} +## TrainablePipe.add_label {id="add_label",tag="method"} > #### Example > @@ -390,7 +390,7 @@ case, all labels found in the sample will be automatically added to the model, and the output dimension will be [inferred](/usage/layers-architectures#thinc-shape-inference) automatically. -## TrainablePipe.is_resizable {#is_resizable tag="property"} +## TrainablePipe.is_resizable {id="is_resizable",tag="property"} > #### Example > @@ -421,7 +421,7 @@ as an attribute to the component's model. | ----------- | ---------------------------------------------------------------------------------------------- | | **RETURNS** | Whether or not the output dimension of the model can be changed after initialization. ~~bool~~ | -## TrainablePipe.set_output {#set_output tag="method"} +## TrainablePipe.set_output {id="set_output",tag="method"} Change the output dimension of the component's model. If the component is not [resizable](#is_resizable), this method will raise a `NotImplementedError`. If a @@ -441,7 +441,7 @@ care should be taken to avoid the "catastrophic forgetting" problem. | ---- | --------------------------------- | | `nO` | The new output dimension. ~~int~~ | -## TrainablePipe.to_disk {#to_disk tag="method"} +## TrainablePipe.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -458,7 +458,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## TrainablePipe.from_disk {#from_disk tag="method"} +## TrainablePipe.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -476,7 +476,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified pipe. ~~TrainablePipe~~ | -## TrainablePipe.to_bytes {#to_bytes tag="method"} +## TrainablePipe.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -493,7 +493,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the pipe. ~~bytes~~ | -## TrainablePipe.from_bytes {#from_bytes tag="method"} +## TrainablePipe.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -512,7 +512,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The pipe. ~~TrainablePipe~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | ------- | --------------------------------------------------------------------------------------------------------------------------------- | @@ -521,7 +521,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `name` | The name of the component instance in the pipeline. Can be used in the losses. ~~str~~ | | `cfg` | Keyword arguments passed to [`TrainablePipe.__init__`](/api/pipe#init). Will be serialized with the component. ~~Dict[str, Any]~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/pipeline-functions.md b/website/docs/api/pipeline-functions.mdx similarity index 95% rename from website/docs/api/pipeline-functions.md rename to website/docs/api/pipeline-functions.mdx index 070292782..545ace2f2 100644 --- a/website/docs/api/pipeline-functions.md +++ b/website/docs/api/pipeline-functions.mdx @@ -10,7 +10,7 @@ menu: - ['doc_cleaner', 'doc_cleaner'] --- -## merge_noun_chunks {#merge_noun_chunks tag="function"} +## merge_noun_chunks {id="merge_noun_chunks",tag="function"} Merge noun chunks into a single token. Also available via the string name `"merge_noun_chunks"`. @@ -40,7 +40,7 @@ all other components. | `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ | | **RETURNS** | The modified `Doc` with merged noun chunks. ~~Doc~~ | -## merge_entities {#merge_entities tag="function"} +## merge_entities {id="merge_entities",tag="function"} Merge named entities into a single token. Also available via the string name `"merge_entities"`. @@ -70,7 +70,7 @@ components to the end of the pipeline and after all other components. | `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ | | **RETURNS** | The modified `Doc` with merged entities. ~~Doc~~ | -## merge_subtokens {#merge_subtokens tag="function" new="2.1"} +## merge_subtokens {id="merge_subtokens",tag="function",version="2.1"} Merge subtokens into a single token. Also available via the string name `"merge_subtokens"`. As of v2.1, the parser is able to predict "subtokens" that @@ -110,7 +110,7 @@ end of the pipeline and after all other components. | `label` | The subtoken dependency label. Defaults to `"subtok"`. ~~str~~ | | **RETURNS** | The modified `Doc` with merged subtokens. ~~Doc~~ | -## token_splitter {#token_splitter tag="function" new="3.0"} +## token_splitter {id="token_splitter",tag="function",version="3.0"} Split tokens longer than a minimum length into shorter tokens. Intended for use with transformer pipelines where long spaCy tokens lead to input text that @@ -132,7 +132,7 @@ exceed the transformer model max length. | `split_length` | The length of the split tokens. Defaults to `5`. ~~int~~ | | **RETURNS** | The modified `Doc` with the split tokens. ~~Doc~~ | -## doc_cleaner {#doc_cleaner tag="function" new="3.2.1"} +## doc_cleaner {id="doc_cleaner",tag="function",version="3.2.1"} Clean up `Doc` attributes. Intended for use at the end of pipelines with `tok2vec` or `transformer` pipeline components that store tensors and other @@ -154,7 +154,7 @@ whole pipeline has run. | `silent` | If `False`, show warnings if attributes aren't found or can't be set. Defaults to `True`. ~~bool~~ | | **RETURNS** | The modified `Doc` with the modified attributes. ~~Doc~~ | -## span_cleaner {#span_cleaner tag="function,experimental"} +## span_cleaner {id="span_cleaner",tag="function,experimental"} Remove `SpanGroup`s from `doc.spans` based on a key prefix. This is used to clean up after the [`CoreferenceResolver`](/api/coref) when it's paired with a diff --git a/website/docs/api/scorer.md b/website/docs/api/scorer.mdx similarity index 96% rename from website/docs/api/scorer.md rename to website/docs/api/scorer.mdx index 86e61da1e..6f0c95f6f 100644 --- a/website/docs/api/scorer.md +++ b/website/docs/api/scorer.mdx @@ -10,7 +10,7 @@ The `Scorer` computes evaluation scores. It's typically created by provides a number of evaluation methods for evaluating [`Token`](/api/token) and [`Doc`](/api/doc) attributes. -## Scorer.\_\_init\_\_ {#init tag="method"} +## Scorer.\_\_init\_\_ {id="init",tag="method"} Create a new `Scorer`. @@ -35,7 +35,7 @@ Create a new `Scorer`. | _keyword-only_ | | | `\*\*kwargs` | Any additional settings to pass on to the individual scoring methods. ~~Any~~ | -## Scorer.score {#score tag="method"} +## Scorer.score {id="score",tag="method"} Calculate the scores for a list of [`Example`](/api/example) objects using the scoring methods provided by the components in the pipeline. @@ -72,7 +72,7 @@ core pipeline components, the individual score names start with the `Token` or | `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ | | **RETURNS** | A dictionary of scores. ~~Dict[str, Union[float, Dict[str, float]]]~~ | -## Scorer.score_tokenization {#score_tokenization tag="staticmethod" new="3"} +## Scorer.score_tokenization {id="score_tokenization",tag="staticmethod",version="3"} Scores the tokenization: @@ -93,7 +93,7 @@ Docs with `has_unknown_spaces` are skipped during scoring. | `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ | | **RETURNS** | `Dict` | A dictionary containing the scores `token_acc`, `token_p`, `token_r`, `token_f`. ~~Dict[str, float]]~~ | -## Scorer.score_token_attr {#score_token_attr tag="staticmethod" new="3"} +## Scorer.score_token_attr {id="score_token_attr",tag="staticmethod",version="3"} Scores a single token attribute. Tokens with missing values in the reference doc are skipped during scoring. @@ -114,7 +114,7 @@ are skipped during scoring. | `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ | | **RETURNS** | A dictionary containing the score `{attr}_acc`. ~~Dict[str, float]~~ | -## Scorer.score_token_attr_per_feat {#score_token_attr_per_feat tag="staticmethod" new="3"} +## Scorer.score_token_attr_per_feat {id="score_token_attr_per_feat",tag="staticmethod",version="3"} Scores a single token attribute per feature for a token attribute in the Universal Dependencies @@ -138,7 +138,7 @@ scoring. | `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ | | **RETURNS** | A dictionary containing the micro PRF scores under the key `{attr}_micro_p/r/f` and the per-feature PRF scores under `{attr}_per_feat`. ~~Dict[str, Dict[str, float]]~~ | -## Scorer.score_spans {#score_spans tag="staticmethod" new="3"} +## Scorer.score_spans {id="score_spans",tag="staticmethod",version="3"} Returns PRF scores for labeled or unlabeled spans. @@ -160,7 +160,7 @@ Returns PRF scores for labeled or unlabeled spans. | `allow_overlap` | Defaults to `False`. Whether or not to allow overlapping spans. If set to `False`, the alignment will automatically resolve conflicts. ~~bool~~ | | **RETURNS** | A dictionary containing the PRF scores under the keys `{attr}_p`, `{attr}_r`, `{attr}_f` and the per-type PRF scores under `{attr}_per_type`. ~~Dict[str, Union[float, Dict[str, float]]]~~ | -## Scorer.score_deps {#score_deps tag="staticmethod" new="3"} +## Scorer.score_deps {id="score_deps",tag="staticmethod",version="3"} Calculate the UAS, LAS, and LAS per type scores for dependency parses. Tokens with missing values for the `attr` (typically `dep`) are skipped during scoring. @@ -194,7 +194,7 @@ with missing values for the `attr` (typically `dep`) are skipped during scoring. | `missing_values` | Attribute values to treat as missing annotation in the reference annotation. Defaults to `{0, None, ""}`. ~~Set[Any]~~ | | **RETURNS** | A dictionary containing the scores: `{attr}_uas`, `{attr}_las`, and `{attr}_las_per_type`. ~~Dict[str, Union[float, Dict[str, float]]]~~ | -## Scorer.score_cats {#score_cats tag="staticmethod" new="3"} +## Scorer.score_cats {id="score_cats",tag="staticmethod",version="3"} Calculate PRF and ROC AUC scores for a doc-level attribute that is a dict containing scores for each label like `Doc.cats`. The returned dictionary @@ -241,7 +241,7 @@ The reported `{attr}_score` depends on the classification properties: | `threshold` | Cutoff to consider a prediction "positive". Defaults to `0.5` for multi-label, and `0.0` (i.e. whatever's highest scoring) otherwise. ~~float~~ | | **RETURNS** | A dictionary containing the scores, with inapplicable scores as `None`. ~~Dict[str, Optional[float]]~~ | -## Scorer.score_links {#score_links tag="staticmethod" new="3"} +## Scorer.score_links {id="score_links",tag="staticmethod",version="3"} Returns PRF for predicted links on the entity level. To disentangle the performance of the NEL from the NER, this method only evaluates NEL links for @@ -264,7 +264,7 @@ entities that overlap between the gold reference and the predictions. | `negative_labels` | The string values that refer to no annotation (e.g. "NIL"). ~~Iterable[str]~~ | | **RETURNS** | A dictionary containing the scores. ~~Dict[str, Optional[float]]~~ | -## get_ner_prf {#get_ner_prf new="3"} +## get_ner_prf {id="get_ner_prf",version="3"} Compute micro-PRF and per-entity PRF scores. @@ -272,7 +272,7 @@ Compute micro-PRF and per-entity PRF scores. | ---------- | ------------------------------------------------------------------------------------------------------------------- | | `examples` | The `Example` objects holding both the predictions and the correct gold-standard annotations. ~~Iterable[Example]~~ | -## score_coref_clusters {#score_coref_clusters tag="experimental"} +## score_coref_clusters {id="score_coref_clusters",tag="experimental"} Returns LEA ([Moosavi and Strube, 2016](https://aclanthology.org/P16-1060/)) PRF scores for coreference clusters. @@ -301,7 +301,7 @@ the [CoreferenceResolver](/api/coref) docs. | `span_cluster_prefix` | The prefix used for spans representing coreference clusters. ~~str~~ | | **RETURNS** | A dictionary containing the scores. ~~Dict[str, Optional[float]]~~ | -## score_span_predictions {#score_span_predictions tag="experimental"} +## score_span_predictions {id="score_span_predictions",tag="experimental"} Return accuracy for reconstructions of spans from single tokens. Only exactly correct predictions are counted as correct, there is no partial credit for near diff --git a/website/docs/api/sentencerecognizer.md b/website/docs/api/sentencerecognizer.mdx similarity index 94% rename from website/docs/api/sentencerecognizer.md rename to website/docs/api/sentencerecognizer.mdx index 2f50350ae..5435399f9 100644 --- a/website/docs/api/sentencerecognizer.md +++ b/website/docs/api/sentencerecognizer.mdx @@ -2,7 +2,7 @@ title: SentenceRecognizer tag: class source: spacy/pipeline/senter.pyx -new: 3 +version: 3 teaser: 'Pipeline component for sentence segmentation' api_base_class: /api/tagger api_string_name: senter @@ -12,7 +12,7 @@ api_trainable: true A trainable pipeline component for sentence segmentation. For a simpler, rule-based strategy, see the [`Sentencizer`](/api/sentencizer). -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Predicted values will be assigned to `Token.is_sent_start`. The resulting sentences can be accessed using `Doc.sents`. @@ -22,7 +22,7 @@ sentences can be accessed using `Doc.sents`. | `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. This will be either `True` or `False` for all tokens. ~~bool~~ | | `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -49,7 +49,7 @@ architectures and their arguments and hyperparameters. %%GITHUB_SPACY/spacy/pipeline/senter.pyx ``` -## SentenceRecognizer.\_\_init\_\_ {#init tag="method"} +## SentenceRecognizer.\_\_init\_\_ {id="init",tag="method"} Initialize the sentence recognizer. @@ -81,7 +81,7 @@ shortcut for this and instantiate the component using its string name and | `overwrite` 3.2 | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ | | `scorer` 3.2 | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"`. ~~Optional[Callable]~~ | -## SentenceRecognizer.\_\_call\_\_ {#call tag="method"} +## SentenceRecognizer.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -105,7 +105,7 @@ and all pipeline components are applied to the `Doc` in order. Both | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## SentenceRecognizer.pipe {#pipe tag="method"} +## SentenceRecognizer.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -129,7 +129,7 @@ and [`pipe`](/api/sentencerecognizer#pipe) delegate to the | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## SentenceRecognizer.initialize {#initialize tag="method"} +## SentenceRecognizer.initialize {id="initialize",tag="method"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -153,7 +153,7 @@ by [`Language.initialize`](/api/language#initialize). | _keyword-only_ | | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | -## SentenceRecognizer.predict {#predict tag="method"} +## SentenceRecognizer.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects, without modifying them. @@ -170,7 +170,7 @@ modifying them. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The model's prediction for each document. | -## SentenceRecognizer.set_annotations {#set_annotations tag="method"} +## SentenceRecognizer.set_annotations {id="set_annotations",tag="method"} Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. @@ -187,7 +187,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `scores` | The scores to set, produced by `SentenceRecognizer.predict`. | -## SentenceRecognizer.update {#update tag="method"} +## SentenceRecognizer.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects containing the predictions and gold-standard annotations, and update the component's model. @@ -211,7 +211,7 @@ Delegates to [`predict`](/api/sentencerecognizer#predict) and | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## SentenceRecognizer.rehearse {#rehearse tag="method,experimental" new="3"} +## SentenceRecognizer.rehearse {id="rehearse",tag="method,experimental",version="3"} Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the current model to make predictions similar to an initial model to try to address @@ -234,7 +234,7 @@ the "catastrophic forgetting" problem. This feature is experimental. | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## SentenceRecognizer.get_loss {#get_loss tag="method"} +## SentenceRecognizer.get_loss {id="get_loss",tag="method"} Find the loss and gradient of loss for the batch of documents and their predicted scores. @@ -253,7 +253,7 @@ predicted scores. | `scores` | Scores representing the model's predictions. | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | -## SentenceRecognizer.create_optimizer {#create_optimizer tag="method"} +## SentenceRecognizer.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -268,7 +268,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## SentenceRecognizer.use_params {#use_params tag="method, contextmanager"} +## SentenceRecognizer.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored. @@ -285,7 +285,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## SentenceRecognizer.to_disk {#to_disk tag="method"} +## SentenceRecognizer.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -302,7 +302,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## SentenceRecognizer.from_disk {#from_disk tag="method"} +## SentenceRecognizer.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -320,7 +320,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `SentenceRecognizer` object. ~~SentenceRecognizer~~ | -## SentenceRecognizer.to_bytes {#to_bytes tag="method"} +## SentenceRecognizer.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -337,7 +337,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `SentenceRecognizer` object. ~~bytes~~ | -## SentenceRecognizer.from_bytes {#from_bytes tag="method"} +## SentenceRecognizer.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -356,7 +356,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `SentenceRecognizer` object. ~~SentenceRecognizer~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/sentencizer.md b/website/docs/api/sentencizer.mdx similarity index 94% rename from website/docs/api/sentencizer.md rename to website/docs/api/sentencizer.mdx index b75c7a2f1..9fb5ea71f 100644 --- a/website/docs/api/sentencizer.md +++ b/website/docs/api/sentencizer.mdx @@ -13,7 +13,7 @@ performed by the [`DependencyParser`](/api/dependencyparser), so the `Sentencizer` lets you implement a simpler, rule-based strategy that doesn't require a statistical model to be loaded. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Calculated values will be assigned to `Token.is_sent_start`. The resulting sentences can be accessed using `Doc.sents`. @@ -23,7 +23,7 @@ sentences can be accessed using `Doc.sents`. | `Token.is_sent_start` | A boolean value indicating whether the token starts a sentence. This will be either `True` or `False` for all tokens. ~~bool~~ | | `Doc.sents` | An iterator over sentences in the `Doc`, determined by `Token.is_sent_start` values. ~~Iterator[Span]~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -39,7 +39,7 @@ how the component should be configured. You can override its settings via the | Setting | Description | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `punct_chars` | Optional custom list of punctuation characters that mark sentence ends. See below for defaults if not set. Defaults to `None`. ~~Optional[List[str]]~~ | `None` | +| `punct_chars` | Optional custom list of punctuation characters that mark sentence ends. See below for defaults if not set. Defaults to `None`. ~~Optional[List[str]]~~ | | `overwrite` 3.2 | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ | | `scorer` 3.2 | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"` ~~Optional[Callable]~~ | @@ -47,7 +47,7 @@ how the component should be configured. You can override its settings via the %%GITHUB_SPACY/spacy/pipeline/sentencizer.pyx ``` -## Sentencizer.\_\_init\_\_ {#init tag="method"} +## Sentencizer.\_\_init\_\_ {id="init",tag="method"} Initialize the sentencizer. @@ -69,8 +69,7 @@ Initialize the sentencizer. | `overwrite` 3.2 | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ | | `scorer` 3.2 | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for the attribute `"sents"` ~~Optional[Callable]~~ | -```python -### punct_chars defaults +```python {title="punct_chars defaults"} ['!', '.', '?', '։', '؟', '۔', '܀', '܁', '܂', '߹', '।', '॥', '၊', '။', '።', '፧', '፨', '᙮', '᜵', '᜶', '᠃', '᠉', '᥄', '᥅', '᪨', '᪩', '᪪', '᪫', '᭚', '᭛', '᭞', '᭟', '᰻', '᰼', '᱾', '᱿', '‼', '‽', '⁇', '⁈', '⁉', @@ -83,7 +82,7 @@ Initialize the sentencizer. '𑪜', '𑱁', '𑱂', '𖩮', '𖩯', '𖫵', '𖬷', '𖬸', '𖭄', '𛲟', '𝪈', '｡', '。'] ``` -## Sentencizer.\_\_call\_\_ {#call tag="method"} +## Sentencizer.\_\_call\_\_ {id="call",tag="method"} Apply the sentencizer on a `Doc`. Typically, this happens automatically after the component has been added to the pipeline using @@ -105,7 +104,7 @@ the component has been added to the pipeline using | `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ | | **RETURNS** | The modified `Doc` with added sentence boundaries. ~~Doc~~ | -## Sentencizer.pipe {#pipe tag="method"} +## Sentencizer.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -126,7 +125,7 @@ applied to the `Doc` in order. | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## Sentencizer.to_disk {#to_disk tag="method"} +## Sentencizer.to_disk {id="to_disk",tag="method"} Save the sentencizer settings (punctuation characters) to a directory. Will create a file `sentencizer.json`. This also happens automatically when you save @@ -144,7 +143,7 @@ an `nlp` object with a sentencizer added to its pipeline. | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | `path` | A path to a JSON file, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | -## Sentencizer.from_disk {#from_disk tag="method"} +## Sentencizer.from_disk {id="from_disk",tag="method"} Load the sentencizer settings from a file. Expects a JSON file. This also happens automatically when you load an `nlp` object or model with a sentencizer @@ -162,7 +161,7 @@ added to its pipeline. | `path` | A path to a JSON file. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | **RETURNS** | The modified `Sentencizer` object. ~~Sentencizer~~ | -## Sentencizer.to_bytes {#to_bytes tag="method"} +## Sentencizer.to_bytes {id="to_bytes",tag="method"} Serialize the sentencizer settings to a bytestring. @@ -178,7 +177,7 @@ Serialize the sentencizer settings to a bytestring. | ----------- | ------------------------------ | | **RETURNS** | The serialized data. ~~bytes~~ | -## Sentencizer.from_bytes {#from_bytes tag="method"} +## Sentencizer.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. diff --git a/website/docs/api/span-resolver.md b/website/docs/api/span-resolver.mdx similarity index 94% rename from website/docs/api/span-resolver.md rename to website/docs/api/span-resolver.mdx index 3e992cd03..f061d8df3 100644 --- a/website/docs/api/span-resolver.md +++ b/website/docs/api/span-resolver.mdx @@ -33,7 +33,7 @@ use case is as a post-processing step on word-level [coreference resolution](/api/coref). The input and output keys used to store `Span` objects are configurable. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Predictions will be saved to `Doc.spans` as [`SpanGroup`s](/api/spangroup). @@ -46,7 +46,7 @@ prefixes are configurable. | ------------------------------------------------- | ------------------------------------------------------------------------- | | `Doc.spans[output_prefix + "_" + cluster_number]` | One group of predicted spans. Cluster number starts from 1. ~~SpanGroup~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -74,7 +74,7 @@ details on the architectures and their arguments and hyperparameters. | `input_prefix` | The prefix to use for input `SpanGroup`s. Defaults to `coref_head_clusters`. ~~str~~ | | `output_prefix` | The prefix for predicted `SpanGroup`s. Defaults to `coref_clusters`. ~~str~~ | -## SpanResolver.\_\_init\_\_ {#init tag="method"} +## SpanResolver.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -104,7 +104,7 @@ shortcut for this and instantiate the component using its string name and | `input_prefix` | The prefix to use for input `SpanGroup`s. Defaults to `coref_head_clusters`. ~~str~~ | | `output_prefix` | The prefix for predicted `SpanGroup`s. Defaults to `coref_clusters`. ~~str~~ | -## SpanResolver.\_\_call\_\_ {#call tag="method"} +## SpanResolver.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -126,7 +126,7 @@ and [`set_annotations`](#set_annotations) methods. | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## SpanResolver.pipe {#pipe tag="method"} +## SpanResolver.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -150,7 +150,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/span-resolver#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## SpanResolver.initialize {#initialize tag="method"} +## SpanResolver.initialize {id="initialize",tag="method"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -174,7 +174,7 @@ by [`Language.initialize`](/api/language#initialize). | _keyword-only_ | | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | -## SpanResolver.predict {#predict tag="method"} +## SpanResolver.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects, without modifying them. Predictions are returned as a list of `MentionClusters`, one for @@ -194,7 +194,7 @@ correspond to token indices. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The predicted spans for the `Doc`s. ~~List[MentionClusters]~~ | -## SpanResolver.set_annotations {#set_annotations tag="method"} +## SpanResolver.set_annotations {id="set_annotations",tag="method"} Modify a batch of documents, saving predictions using the output prefix in `Doc.spans`. @@ -212,7 +212,7 @@ Modify a batch of documents, saving predictions using the output prefix in | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `spans` | The predicted spans for the `docs`. ~~List[MentionClusters]~~ | -## SpanResolver.update {#update tag="method"} +## SpanResolver.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects. Delegates to [`predict`](/api/span-resolver#predict). @@ -234,7 +234,7 @@ Learn from a batch of [`Example`](/api/example) objects. Delegates to | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## SpanResolver.create_optimizer {#create_optimizer tag="method"} +## SpanResolver.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -249,7 +249,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## SpanResolver.use_params {#use_params tag="method, contextmanager"} +## SpanResolver.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored. @@ -266,7 +266,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## SpanResolver.to_disk {#to_disk tag="method"} +## SpanResolver.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -283,7 +283,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## SpanResolver.from_disk {#from_disk tag="method"} +## SpanResolver.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -301,7 +301,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `SpanResolver` object. ~~SpanResolver~~ | -## SpanResolver.to_bytes {#to_bytes tag="method"} +## SpanResolver.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -318,7 +318,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `SpanResolver` object. ~~bytes~~ | -## SpanResolver.from_bytes {#from_bytes tag="method"} +## SpanResolver.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -337,7 +337,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `SpanResolver` object. ~~SpanResolver~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/span.md b/website/docs/api/span.mdx similarity index 93% rename from website/docs/api/span.md rename to website/docs/api/span.mdx index 69bbe8db1..bd7794edc 100644 --- a/website/docs/api/span.md +++ b/website/docs/api/span.mdx @@ -6,7 +6,7 @@ source: spacy/tokens/span.pyx A slice from a [`Doc`](/api/doc) object. -## Span.\_\_init\_\_ {#init tag="method"} +## Span.\_\_init\_\_ {id="init",tag="method"} Create a `Span` object from the slice `doc[start : end]`. @@ -29,7 +29,7 @@ Create a `Span` object from the slice `doc[start : end]`. | `kb_id` | A knowledge base ID to attach to the span, e.g. for named entities. ~~Union[str, int]~~ | | `span_id` | An ID to associate with the span. ~~Union[str, int]~~ | -## Span.\_\_getitem\_\_ {#getitem tag="method"} +## Span.\_\_getitem\_\_ {id="getitem",tag="method"} Get a `Token` object. @@ -61,7 +61,7 @@ Get a `Span` object. | `start_end` | The slice of the span to get. ~~Tuple[int, int]~~ | | **RETURNS** | The span at `span[start : end]`. ~~Span~~ | -## Span.\_\_iter\_\_ {#iter tag="method"} +## Span.\_\_iter\_\_ {id="iter",tag="method"} Iterate over `Token` objects. @@ -77,7 +77,7 @@ Iterate over `Token` objects. | ---------- | --------------------------- | | **YIELDS** | A `Token` object. ~~Token~~ | -## Span.\_\_len\_\_ {#len tag="method"} +## Span.\_\_len\_\_ {id="len",tag="method"} Get the number of tokens in the span. @@ -93,7 +93,7 @@ Get the number of tokens in the span. | ----------- | ----------------------------------------- | | **RETURNS** | The number of tokens in the span. ~~int~~ | -## Span.set_extension {#set_extension tag="classmethod" new="2"} +## Span.set_extension {id="set_extension",tag="classmethod",version="2"} Define a custom attribute on the `Span` which becomes available via `Span._`. For details, see the documentation on @@ -118,7 +118,7 @@ For details, see the documentation on | `setter` | Setter function that takes the `Span` and a value, and modifies the object. Is called when the user writes to the `Span._` attribute. ~~Optional[Callable[[Span, Any], None]]~~ | | `force` | Force overwriting existing attribute. ~~bool~~ | -## Span.get_extension {#get_extension tag="classmethod" new="2"} +## Span.get_extension {id="get_extension",tag="classmethod",version="2"} Look up a previously registered extension by name. Returns a 4-tuple `(default, method, getter, setter)` if the extension is registered. Raises a @@ -138,7 +138,7 @@ Look up a previously registered extension by name. Returns a 4-tuple | `name` | Name of the extension. ~~str~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | -## Span.has_extension {#has_extension tag="classmethod" new="2"} +## Span.has_extension {id="has_extension",tag="classmethod",version="2"} Check whether an extension has been registered on the `Span` class. @@ -155,7 +155,7 @@ Check whether an extension has been registered on the `Span` class. | `name` | Name of the extension to check. ~~str~~ | | **RETURNS** | Whether the extension has been registered. ~~bool~~ | -## Span.remove_extension {#remove_extension tag="classmethod" new="2.0.12"} +## Span.remove_extension {id="remove_extension",tag="classmethod",version="2.0.12"} Remove a previously registered extension. @@ -173,7 +173,7 @@ Remove a previously registered extension. | `name` | Name of the extension. ~~str~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | -## Span.char_span {#char_span tag="method" new="2.2.4"} +## Span.char_span {id="char_span",tag="method",version="2.2.4"} Create a `Span` object from the slice `span.text[start:end]`. Returns `None` if the character indices don't map to a valid span. @@ -195,7 +195,7 @@ the character indices don't map to a valid span. | `vector` | A meaning representation of the span. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | | **RETURNS** | The newly constructed object or `None`. ~~Optional[Span]~~ | -## Span.similarity {#similarity tag="method" model="vectors"} +## Span.similarity {id="similarity",tag="method",model="vectors"} Make a semantic similarity estimate. The default estimate is cosine similarity using an average of word vectors. @@ -216,7 +216,7 @@ using an average of word vectors. | `other` | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ | | **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ | -## Span.get_lca_matrix {#get_lca_matrix tag="method"} +## Span.get_lca_matrix {id="get_lca_matrix",tag="method"} Calculates the lowest common ancestor matrix for a given `Span`. Returns LCA matrix containing the integer index of the ancestor, or `-1` if no common @@ -235,7 +235,7 @@ ancestor is found, e.g. if span excludes a necessary ancestor. | ----------- | --------------------------------------------------------------------------------------- | | **RETURNS** | The lowest common ancestor matrix of the `Span`. ~~numpy.ndarray[ndim=2, dtype=int32]~~ | -## Span.to_array {#to_array tag="method" new="2"} +## Span.to_array {id="to_array",tag="method",version="2"} Given a list of `M` attribute IDs, export the tokens to a numpy `ndarray` of shape `(N, M)`, where `N` is the length of the document. The values will be @@ -256,7 +256,7 @@ shape `(N, M)`, where `N` is the length of the document. The values will be | `attr_ids` | A list of attributes (int IDs or string names) or a single attribute (int ID or string name). ~~Union[int, str, List[Union[int, str]]]~~ | | **RETURNS** | The exported attributes as a numpy array. ~~Union[numpy.ndarray[ndim=2, dtype=uint64], numpy.ndarray[ndim=1, dtype=uint64]]~~ | -## Span.ents {#ents tag="property" new="2.0.13" model="ner"} +## Span.ents {id="ents",tag="property",version="2.0.13",model="ner"} The named entities that fall completely within the span. Returns a tuple of `Span` objects. @@ -276,7 +276,7 @@ The named entities that fall completely within the span. Returns a tuple of | ----------- | ----------------------------------------------------------------- | | **RETURNS** | Entities in the span, one `Span` per entity. ~~Tuple[Span, ...]~~ | -## Span.noun_chunks {#noun_chunks tag="property" model="parser"} +## Span.noun_chunks {id="noun_chunks",tag="property",model="parser"} Iterate over the base noun phrases in the span. Yields base noun-phrase `Span` objects, if the document has been syntactically parsed. A base noun phrase, or @@ -302,7 +302,7 @@ raised. | ---------- | --------------------------------- | | **YIELDS** | Noun chunks in the span. ~~Span~~ | -## Span.as_doc {#as_doc tag="method"} +## Span.as_doc {id="as_doc",tag="method"} Create a new `Doc` object corresponding to the `Span`, with a copy of the data. @@ -326,7 +326,7 @@ time. | `array` | Precomputed array version of the original doc as generated by [`Doc.to_array`](/api/doc#to_array). ~~numpy.ndarray~~ | | **RETURNS** | A `Doc` object of the `Span`'s content. ~~Doc~~ | -## Span.root {#root tag="property" model="parser"} +## Span.root {id="root",tag="property",model="parser"} The token with the shortest path to the root of the sentence (or the root itself). If multiple tokens are equally high in the tree, the first token is @@ -347,7 +347,7 @@ taken. | ----------- | ------------------------- | | **RETURNS** | The root token. ~~Token~~ | -## Span.conjuncts {#conjuncts tag="property" model="parser"} +## Span.conjuncts {id="conjuncts",tag="property",model="parser"} A tuple of tokens coordinated to `span.root`. @@ -363,7 +363,7 @@ A tuple of tokens coordinated to `span.root`. | ----------- | --------------------------------------------- | | **RETURNS** | The coordinated tokens. ~~Tuple[Token, ...]~~ | -## Span.lefts {#lefts tag="property" model="parser"} +## Span.lefts {id="lefts",tag="property",model="parser"} Tokens that are to the left of the span, whose heads are within the span. @@ -379,7 +379,7 @@ Tokens that are to the left of the span, whose heads are within the span. | ---------- | ---------------------------------------------- | | **YIELDS** | A left-child of a token of the span. ~~Token~~ | -## Span.rights {#rights tag="property" model="parser"} +## Span.rights {id="rights",tag="property",model="parser"} Tokens that are to the right of the span, whose heads are within the span. @@ -395,7 +395,7 @@ Tokens that are to the right of the span, whose heads are within the span. | ---------- | ----------------------------------------------- | | **YIELDS** | A right-child of a token of the span. ~~Token~~ | -## Span.n_lefts {#n_lefts tag="property" model="parser"} +## Span.n_lefts {id="n_lefts",tag="property",model="parser"} The number of tokens that are to the left of the span, whose heads are within the span. @@ -411,7 +411,7 @@ the span. | ----------- | ---------------------------------------- | | **RETURNS** | The number of left-child tokens. ~~int~~ | -## Span.n_rights {#n_rights tag="property" model="parser"} +## Span.n_rights {id="n_rights",tag="property",model="parser"} The number of tokens that are to the right of the span, whose heads are within the span. @@ -427,7 +427,7 @@ the span. | ----------- | ----------------------------------------- | | **RETURNS** | The number of right-child tokens. ~~int~~ | -## Span.subtree {#subtree tag="property" model="parser"} +## Span.subtree {id="subtree",tag="property",model="parser"} Tokens within the span and tokens which descend from them. @@ -443,7 +443,7 @@ Tokens within the span and tokens which descend from them. | ---------- | ----------------------------------------------------------- | | **YIELDS** | A token within the span, or a descendant from it. ~~Token~~ | -## Span.has_vector {#has_vector tag="property" model="vectors"} +## Span.has_vector {id="has_vector",tag="property",model="vectors"} A boolean value indicating whether a word vector is associated with the object. @@ -458,7 +458,7 @@ A boolean value indicating whether a word vector is associated with the object. | ----------- | ----------------------------------------------------- | | **RETURNS** | Whether the span has a vector data attached. ~~bool~~ | -## Span.vector {#vector tag="property" model="vectors"} +## Span.vector {id="vector",tag="property",model="vectors"} A real-valued meaning representation. Defaults to an average of the token vectors. @@ -475,7 +475,7 @@ vectors. | ----------- | ----------------------------------------------------------------------------------------------- | | **RETURNS** | A 1-dimensional array representing the span's vector. ~~`numpy.ndarray[ndim=1, dtype=float32]~~ | -## Span.vector_norm {#vector_norm tag="property" model="vectors"} +## Span.vector_norm {id="vector_norm",tag="property",model="vectors"} The L2 norm of the span's vector representation. @@ -492,7 +492,7 @@ The L2 norm of the span's vector representation. | ----------- | --------------------------------------------------- | | **RETURNS** | The L2 norm of the vector representation. ~~float~~ | -## Span.sent {#sent tag="property" model="sentences"} +## Span.sent {id="sent",tag="property",model="sentences"} The sentence span that this span is a part of. This property is only available when [sentence boundaries](/usage/linguistic-features#sbd) have been set on the @@ -520,7 +520,7 @@ sent = doc[sent.start : max(sent.end, span.end)] | ----------- | ------------------------------------------------------- | | **RETURNS** | The sentence span that this span is a part of. ~~Span~~ | -## Span.sents {#sents tag="property" model="sentences" new="3.2.1"} +## Span.sents {id="sents",tag="property",model="sentences",version="3.2.1"} Returns a generator over the sentences the span belongs to. This property is only available when [sentence boundaries](/usage/linguistic-features#sbd) have @@ -542,7 +542,7 @@ overlaps with will be returned. | ----------- | -------------------------------------------------------------------------- | | **RETURNS** | A generator yielding sentences this `Span` is a part of ~~Iterable[Span]~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | -------------- | ----------------------------------------------------------------------------------------------------------------------------- | diff --git a/website/docs/api/spancategorizer.md b/website/docs/api/spancategorizer.mdx similarity index 94% rename from website/docs/api/spancategorizer.md rename to website/docs/api/spancategorizer.mdx index 58a06bcf5..f39c0aff9 100644 --- a/website/docs/api/spancategorizer.md +++ b/website/docs/api/spancategorizer.mdx @@ -2,7 +2,7 @@ title: SpanCategorizer tag: class,experimental source: spacy/pipeline/spancat.py -new: 3.1 +version: 3.1 teaser: 'Pipeline component for labeling potentially overlapping spans of text' api_base_class: /api/pipe api_string_name: spancat @@ -16,7 +16,7 @@ that predicts zero or more labels for each candidate. Predicted spans will be saved in a [`SpanGroup`](/api/spangroup) on the doc. Individual span scores can be found in `spangroup.attrs["scores"]`. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Predictions will be saved to `Doc.spans[spans_key]` as a [`SpanGroup`](/api/spangroup). The scores for the spans in the `SpanGroup` will @@ -29,7 +29,7 @@ be saved in `SpanGroup.attrs["scores"]`. | `Doc.spans[spans_key]` | The annotated spans. ~~SpanGroup~~ | | `Doc.spans[spans_key].attrs["scores"]` | The score for each span in the `SpanGroup`. ~~Floats1d~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -65,7 +65,7 @@ architectures and their arguments and hyperparameters. %%GITHUB_SPACY/spacy/pipeline/spancat.py ``` -## SpanCategorizer.\_\_init\_\_ {#init tag="method"} +## SpanCategorizer.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -97,7 +97,7 @@ shortcut for this and instantiate the component using its string name and | `threshold` | Minimum probability to consider a prediction positive. Spans with a positive prediction will be saved on the Doc. Defaults to `0.5`. ~~float~~ | | `max_positive` | Maximum number of labels to consider positive per span. Defaults to `None`, indicating no limit. ~~Optional[int]~~ | -## SpanCategorizer.\_\_call\_\_ {#call tag="method"} +## SpanCategorizer.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -120,7 +120,7 @@ delegate to the [`predict`](/api/spancategorizer#predict) and | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## SpanCategorizer.pipe {#pipe tag="method"} +## SpanCategorizer.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -144,7 +144,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/spancategorizer#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## SpanCategorizer.initialize {#initialize tag="method"} +## SpanCategorizer.initialize {id="initialize",tag="method"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -181,7 +181,7 @@ config. | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ | -## SpanCategorizer.predict {#predict tag="method"} +## SpanCategorizer.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects without modifying them. @@ -198,7 +198,7 @@ modifying them. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The model's prediction for each document. | -## SpanCategorizer.set_annotations {#set_annotations tag="method"} +## SpanCategorizer.set_annotations {id="set_annotations",tag="method"} Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores. @@ -215,7 +215,7 @@ Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `scores` | The scores to set, produced by `SpanCategorizer.predict`. | -## SpanCategorizer.update {#update tag="method"} +## SpanCategorizer.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects containing the predictions and gold-standard annotations, and update the component's model. @@ -239,7 +239,7 @@ Delegates to [`predict`](/api/spancategorizer#predict) and | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## SpanCategorizer.set_candidates {#set_candidates tag="method", new="3.3"} +## SpanCategorizer.set_candidates {id="set_candidates",tag="method", version="3.3"} Use the suggester to add a list of [`Span`](/api/span) candidates to a list of [`Doc`](/api/doc) objects. This method is intended to be used for debugging @@ -257,7 +257,7 @@ purposes. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `candidates_key` | Key of the Doc.spans dict to save the candidate spans under. ~~str~~ | -## SpanCategorizer.get_loss {#get_loss tag="method"} +## SpanCategorizer.get_loss {id="get_loss",tag="method"} Find the loss and gradient of loss for the batch of documents and their predicted scores. @@ -276,7 +276,7 @@ predicted scores. | `spans_scores` | Scores representing the model's predictions. ~~Tuple[Ragged, Floats2d]~~ | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | -## SpanCategorizer.create_optimizer {#create_optimizer tag="method"} +## SpanCategorizer.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -291,7 +291,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## SpanCategorizer.use_params {#use_params tag="method, contextmanager"} +## SpanCategorizer.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model to use the given parameter values. @@ -307,7 +307,7 @@ Modify the pipe's model to use the given parameter values. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## SpanCategorizer.add_label {#add_label tag="method"} +## SpanCategorizer.add_label {id="add_label",tag="method"} Add a new label to the pipe. Raises an error if the output dimension is already set, or if the model has already been fully [initialized](#initialize). Note @@ -329,7 +329,7 @@ automatically. | `label` | The label to add. ~~str~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | -## SpanCategorizer.to_disk {#to_disk tag="method"} +## SpanCategorizer.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -346,7 +346,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## SpanCategorizer.from_disk {#from_disk tag="method"} +## SpanCategorizer.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -364,7 +364,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `SpanCategorizer` object. ~~SpanCategorizer~~ | -## SpanCategorizer.to_bytes {#to_bytes tag="method"} +## SpanCategorizer.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -381,7 +381,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `SpanCategorizer` object. ~~bytes~~ | -## SpanCategorizer.from_bytes {#from_bytes tag="method"} +## SpanCategorizer.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -400,7 +400,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `SpanCategorizer` object. ~~SpanCategorizer~~ | -## SpanCategorizer.labels {#labels tag="property"} +## SpanCategorizer.labels {id="labels",tag="property"} The labels currently added to the component. @@ -415,7 +415,7 @@ The labels currently added to the component. | ----------- | ------------------------------------------------------ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | -## SpanCategorizer.label_data {#label_data tag="property"} +## SpanCategorizer.label_data {id="label_data",tag="property"} The labels currently added to the component and their internal meta information. This is the data generated by [`init labels`](/api/cli#init-labels) and used by @@ -433,7 +433,7 @@ the model with a pre-defined label set. | ----------- | ---------------------------------------------------------- | | **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from @@ -451,9 +451,9 @@ serialization by passing in the string names via the `exclude` argument. | `cfg` | The config file. You usually don't want to exclude this. | | `model` | The binary model data. You usually don't want to exclude this. | -## Suggesters {#suggesters tag="registered functions" source="spacy/pipeline/spancat.py"} +## Suggesters {id="suggesters",tag="registered functions",source="spacy/pipeline/spancat.py"} -### spacy.ngram_suggester.v1 {#ngram_suggester} +### spacy.ngram_suggester.v1 {id="ngram_suggester"} > #### Example Config > @@ -471,7 +471,7 @@ integers. The array has two columns, indicating the start and end position. | `sizes` | The phrase lengths to suggest. For example, `[1, 2]` will suggest phrases consisting of 1 or 2 tokens. ~~List[int]~~ | | **CREATES** | The suggester function. ~~Callable[[Iterable[Doc], Optional[Ops]], Ragged]~~ | -### spacy.ngram_range_suggester.v1 {#ngram_range_suggester} +### spacy.ngram_range_suggester.v1 {id="ngram_range_suggester"} > #### Example Config > diff --git a/website/docs/api/spangroup.md b/website/docs/api/spangroup.mdx similarity index 92% rename from website/docs/api/spangroup.md rename to website/docs/api/spangroup.mdx index bd9659acb..cd0accb6a 100644 --- a/website/docs/api/spangroup.md +++ b/website/docs/api/spangroup.mdx @@ -2,7 +2,7 @@ title: SpanGroup tag: class source: spacy/tokens/span_group.pyx -new: 3 +version: 3 --- A group of arbitrary, potentially overlapping [`Span`](/api/span) objects that @@ -13,7 +13,7 @@ into a `SpanGroup` object for you automatically on assignment. `SpanGroup` objects behave similar to `list`s, so you can append `Span` objects to them or access a member at a given index. -## SpanGroup.\_\_init\_\_ {#init tag="method"} +## SpanGroup.\_\_init\_\_ {id="init",tag="method"} Create a `SpanGroup`. @@ -42,7 +42,7 @@ Create a `SpanGroup`. | `attrs` | Optional JSON-serializable attributes to attach to the span group. ~~Dict[str, Any]~~ | | `spans` | The spans to add to the span group. ~~Iterable[Span]~~ | -## SpanGroup.doc {#doc tag="property"} +## SpanGroup.doc {id="doc",tag="property"} The [`Doc`](/api/doc) object the span group is referring to. @@ -68,7 +68,7 @@ the scope of your function. | ----------- | ------------------------------- | | **RETURNS** | The reference document. ~~Doc~~ | -## SpanGroup.has_overlap {#has_overlap tag="property"} +## SpanGroup.has_overlap {id="has_overlap",tag="property"} Check whether the span group contains overlapping spans. @@ -86,7 +86,7 @@ Check whether the span group contains overlapping spans. | ----------- | -------------------------------------------------- | | **RETURNS** | Whether the span group contains overlaps. ~~bool~~ | -## SpanGroup.\_\_len\_\_ {#len tag="method"} +## SpanGroup.\_\_len\_\_ {id="len",tag="method"} Get the number of spans in the group. @@ -102,7 +102,7 @@ Get the number of spans in the group. | ----------- | ----------------------------------------- | | **RETURNS** | The number of spans in the group. ~~int~~ | -## SpanGroup.\_\_getitem\_\_ {#getitem tag="method"} +## SpanGroup.\_\_getitem\_\_ {id="getitem",tag="method"} Get a span from the group. Note that a copy of the span is returned, so if any changes are made to this span, they are not reflected in the corresponding @@ -125,7 +125,7 @@ changes to be reflected in the span group. | `i` | The item index. ~~int~~ | | **RETURNS** | The span at the given index. ~~Span~~ | -## SpanGroup.\_\_setitem\_\_ {#setitem tag="method", new="3.3"} +## SpanGroup.\_\_setitem\_\_ {id="setitem",tag="method", version="3.3"} Set a span in the span group. @@ -144,7 +144,7 @@ Set a span in the span group. | `i` | The item index. ~~int~~ | | `span` | The new value. ~~Span~~ | -## SpanGroup.\_\_delitem\_\_ {#delitem tag="method", new="3.3"} +## SpanGroup.\_\_delitem\_\_ {id="delitem",tag="method", version="3.3"} Delete a span from the span group. @@ -161,7 +161,7 @@ Delete a span from the span group. | ---- | ----------------------- | | `i` | The item index. ~~int~~ | -## SpanGroup.\_\_add\_\_ {#add tag="method", new="3.3"} +## SpanGroup.\_\_add\_\_ {id="add",tag="method", version="3.3"} Concatenate the current span group with another span group and return the result in a new span group. Any `attrs` from the first span group will have precedence @@ -182,7 +182,7 @@ over `attrs` in the second. | `other` | The span group or spans to concatenate. ~~Union[SpanGroup, Iterable[Span]]~~ | | **RETURNS** | The new span group. ~~SpanGroup~~ | -## SpanGroup.\_\_iadd\_\_ {#iadd tag="method", new="3.3"} +## SpanGroup.\_\_iadd\_\_ {id="iadd",tag="method", version="3.3"} Append an iterable of spans or the content of a span group to the current span group. Any `attrs` in the other span group will be added for keys that are not @@ -202,7 +202,7 @@ already present in the current span group. | `other` | The span group or spans to append. ~~Union[SpanGroup, Iterable[Span]]~~ | | **RETURNS** | The span group. ~~SpanGroup~~ | -## SpanGroup.\_\_iter\_\_ {#iter tag="method" new="3.5"} +## SpanGroup.\_\_iter\_\_ {id="iter",tag="method",version="3.5"} Iterate over the spans in this span group. @@ -219,7 +219,8 @@ Iterate over the spans in this span group. | ---------- | ----------------------------------- | | **YIELDS** | A span in this span group. ~~Span~~ | -## SpanGroup.append {#append tag="method"} + +## SpanGroup.append {id="append",tag="method"} Add a [`Span`](/api/span) object to the group. The span must refer to the same [`Doc`](/api/doc) object as the span group. @@ -237,7 +238,7 @@ Add a [`Span`](/api/span) object to the group. The span must refer to the same | ------ | ---------------------------- | | `span` | The span to append. ~~Span~~ | -## SpanGroup.extend {#extend tag="method"} +## SpanGroup.extend {id="extend",tag="method"} Add multiple [`Span`](/api/span) objects or contents of another `SpanGroup` to the group. All spans must refer to the same [`Doc`](/api/doc) object as the span @@ -258,7 +259,7 @@ group. | ------- | -------------------------------------------------------- | | `spans` | The spans to add. ~~Union[SpanGroup, Iterable["Span"]]~~ | -## SpanGroup.copy {#copy tag="method", new="3.3"} +## SpanGroup.copy {id="copy",tag="method", version="3.3"} Return a copy of the span group. @@ -277,7 +278,7 @@ Return a copy of the span group. | `doc` | The document to which the copy is bound. Defaults to `None` for the current doc. ~~Optional[Doc]~~ | | **RETURNS** | A copy of the `SpanGroup` object. ~~SpanGroup~~ | -## SpanGroup.to_bytes {#to_bytes tag="method"} +## SpanGroup.to_bytes {id="to_bytes",tag="method"} Serialize the span group to a bytestring. @@ -293,7 +294,7 @@ Serialize the span group to a bytestring. | ----------- | ------------------------------------- | | **RETURNS** | The serialized `SpanGroup`. ~~bytes~~ | -## SpanGroup.from_bytes {#from_bytes tag="method"} +## SpanGroup.from_bytes {id="from_bytes",tag="method"} Load the span group from a bytestring. Modifies the object in place and returns it. diff --git a/website/docs/api/spanruler.md b/website/docs/api/spanruler.mdx similarity index 94% rename from website/docs/api/spanruler.md rename to website/docs/api/spanruler.mdx index 31f04ccf9..d2d41f620 100644 --- a/website/docs/api/spanruler.md +++ b/website/docs/api/spanruler.mdx @@ -2,7 +2,7 @@ title: SpanRuler tag: class source: spacy/pipeline/span_ruler.py -new: 3.3 +version: 3.3 teaser: 'Pipeline component for rule-based span and named entity recognition' api_string_name: span_ruler api_trainable: false @@ -13,7 +13,7 @@ The span ruler lets you add spans to [`Doc.spans`](/api/doc#spans) and/or usage examples, see the docs on [rule-based span matching](/usage/rule-based-matching#spanruler). -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Matches will be saved to `Doc.spans[spans_key]` as a [`SpanGroup`](/api/spangroup) and/or to `Doc.ents`, where the annotation is @@ -28,7 +28,7 @@ saved in the `Token.ent_type` and `Token.ent_iob` fields. | `Token.ent_type` | The label part of the named entity tag (hash). ~~int~~ | | `Token.ent_type_` | The label part of the named entity tag. ~~str~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -62,7 +62,7 @@ how the component should be configured. You can override its settings via the %%GITHUB_SPACY/spacy/pipeline/span_ruler.py ``` -## SpanRuler.\_\_init\_\_ {#init tag="method"} +## SpanRuler.\_\_init\_\_ {id="init",tag="method"} Initialize the span ruler. If patterns are supplied here, they need to be a list of dictionaries with a `"label"` and `"pattern"` key. A pattern can either be a @@ -95,7 +95,7 @@ token pattern (list) or a phrase pattern (string). For example: | `overwrite` | Whether to remove any existing spans under `Doc.spans[spans key]` if `spans_key` is set, or to remove any ents under `Doc.ents` if `annotate_ents` is set. Defaults to `True`. ~~bool~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_spans`](/api/scorer#score_spans) for `Doc.spans[spans_key]` with overlapping spans allowed. ~~Optional[Callable]~~ | -## SpanRuler.initialize {#initialize tag="method"} +## SpanRuler.initialize {id="initialize",tag="method"} Initialize the component with data and used before training to load in rules from a [pattern file](/usage/rule-based-matching/#spanruler-files). This method @@ -127,7 +127,7 @@ config. Any existing patterns are removed on initialization. | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `patterns` | The list of patterns. Defaults to `None`. ~~Optional[Sequence[Dict[str, Union[str, List[Dict[str, Any]]]]]]~~ | -## SpanRuler.\_\len\_\_ {#len tag="method"} +## SpanRuler.\_\_len\_\_ {id="len",tag="method"} The number of all patterns added to the span ruler. @@ -144,7 +144,7 @@ The number of all patterns added to the span ruler. | ----------- | ------------------------------- | | **RETURNS** | The number of patterns. ~~int~~ | -## SpanRuler.\_\_contains\_\_ {#contains tag="method"} +## SpanRuler.\_\_contains\_\_ {id="contains",tag="method"} Whether a label is present in the patterns. @@ -162,7 +162,7 @@ Whether a label is present in the patterns. | `label` | The label to check. ~~str~~ | | **RETURNS** | Whether the span ruler contains the label. ~~bool~~ | -## SpanRuler.\_\_call\_\_ {#call tag="method"} +## SpanRuler.\_\_call\_\_ {id="call",tag="method"} Find matches in the `Doc` and add them to `doc.spans[span_key]` and/or `doc.ents`. Typically, this happens automatically after the component has been @@ -186,7 +186,7 @@ will be removed. | `doc` | The `Doc` object to process, e.g. the `Doc` in the pipeline. ~~Doc~~ | | **RETURNS** | The modified `Doc` with added spans/entities. ~~Doc~~ | -## SpanRuler.add_patterns {#add_patterns tag="method"} +## SpanRuler.add_patterns {id="add_patterns",tag="method"} Add patterns to the span ruler. A pattern can either be a token pattern (list of dicts) or a phrase pattern (string). For more details, see the usage guide on @@ -207,7 +207,7 @@ dicts) or a phrase pattern (string). For more details, see the usage guide on | ---------- | ---------------------------------------------------------------- | | `patterns` | The patterns to add. ~~List[Dict[str, Union[str, List[dict]]]]~~ | -## SpanRuler.remove {#remove tag="method"} +## SpanRuler.remove {id="remove",tag="method"} Remove patterns by label from the span ruler. A `ValueError` is raised if the label does not exist in any patterns. @@ -225,7 +225,7 @@ label does not exist in any patterns. | ------- | -------------------------------------- | | `label` | The label of the pattern rule. ~~str~~ | -## SpanRuler.remove_by_id {#remove_by_id tag="method"} +## SpanRuler.remove_by_id {id="remove_by_id",tag="method"} Remove patterns by ID from the span ruler. A `ValueError` is raised if the ID does not exist in any patterns. @@ -243,7 +243,7 @@ does not exist in any patterns. | ------------ | ----------------------------------- | | `pattern_id` | The ID of the pattern rule. ~~str~~ | -## SpanRuler.clear {#clear tag="method"} +## SpanRuler.clear {id="clear",tag="method"} Remove all patterns the span ruler. @@ -256,7 +256,7 @@ Remove all patterns the span ruler. > ruler.clear() > ``` -## SpanRuler.to_disk {#to_disk tag="method"} +## SpanRuler.to_disk {id="to_disk",tag="method"} Save the span ruler patterns to a directory. The patterns will be saved as newline-delimited JSON (JSONL). @@ -272,7 +272,7 @@ newline-delimited JSON (JSONL). | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | -## SpanRuler.from_disk {#from_disk tag="method"} +## SpanRuler.from_disk {id="from_disk",tag="method"} Load the span ruler from a path. @@ -288,7 +288,7 @@ Load the span ruler from a path. | `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | **RETURNS** | The modified `SpanRuler` object. ~~SpanRuler~~ | -## SpanRuler.to_bytes {#to_bytes tag="method"} +## SpanRuler.to_bytes {id="to_bytes",tag="method"} Serialize the span ruler to a bytestring. @@ -303,7 +303,7 @@ Serialize the span ruler to a bytestring. | ----------- | ---------------------------------- | | **RETURNS** | The serialized patterns. ~~bytes~~ | -## SpanRuler.from_bytes {#from_bytes tag="method"} +## SpanRuler.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -320,7 +320,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `bytes_data` | The bytestring to load. ~~bytes~~ | | **RETURNS** | The modified `SpanRuler` object. ~~SpanRuler~~ | -## SpanRuler.labels {#labels tag="property"} +## SpanRuler.labels {id="labels",tag="property"} All labels present in the match patterns. @@ -328,7 +328,7 @@ All labels present in the match patterns. | ----------- | -------------------------------------- | | **RETURNS** | The string labels. ~~Tuple[str, ...]~~ | -## SpanRuler.ids {#ids tag="property"} +## SpanRuler.ids {id="ids",tag="property"} All IDs present in the `id` property of the match patterns. @@ -336,7 +336,7 @@ All IDs present in the `id` property of the match patterns. | ----------- | ----------------------------------- | | **RETURNS** | The string IDs. ~~Tuple[str, ...]~~ | -## SpanRuler.patterns {#patterns tag="property"} +## SpanRuler.patterns {id="patterns",tag="property"} All patterns that were added to the span ruler. @@ -344,7 +344,7 @@ All patterns that were added to the span ruler. | ----------- | ---------------------------------------------------------------------------------------- | | **RETURNS** | The original patterns, one dictionary per pattern. ~~List[Dict[str, Union[str, dict]]]~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | ---------------- | -------------------------------------------------------------------------------- | diff --git a/website/docs/api/stringstore.md b/website/docs/api/stringstore.mdx similarity index 89% rename from website/docs/api/stringstore.md rename to website/docs/api/stringstore.mdx index cd414b1f0..47d3715c1 100644 --- a/website/docs/api/stringstore.md +++ b/website/docs/api/stringstore.mdx @@ -8,7 +8,7 @@ Look up strings by 64-bit hashes. As of v2.0, spaCy uses hash values instead of integer IDs. This ensures that strings always map to the same ID, even from different `StringStores`. -## StringStore.\_\_init\_\_ {#init tag="method"} +## StringStore.\_\_init\_\_ {id="init",tag="method"} Create the `StringStore`. @@ -23,7 +23,7 @@ Create the `StringStore`. | --------- | ---------------------------------------------------------------------- | | `strings` | A sequence of strings to add to the store. ~~Optional[Iterable[str]]~~ | -## StringStore.\_\_len\_\_ {#len tag="method"} +## StringStore.\_\_len\_\_ {id="len",tag="method"} Get the number of strings in the store. @@ -38,7 +38,7 @@ Get the number of strings in the store. | ----------- | ------------------------------------------- | | **RETURNS** | The number of strings in the store. ~~int~~ | -## StringStore.\_\_getitem\_\_ {#getitem tag="method"} +## StringStore.\_\_getitem\_\_ {id="getitem",tag="method"} Retrieve a string from a given hash, or vice versa. @@ -56,7 +56,7 @@ Retrieve a string from a given hash, or vice versa. | `string_or_id` | The value to encode. ~~Union[bytes, str, int]~~ | | **RETURNS** | The value to be retrieved. ~~Union[str, int]~~ | -## StringStore.\_\_contains\_\_ {#contains tag="method"} +## StringStore.\_\_contains\_\_ {id="contains",tag="method"} Check whether a string is in the store. @@ -73,7 +73,7 @@ Check whether a string is in the store. | `string` | The string to check. ~~str~~ | | **RETURNS** | Whether the store contains the string. ~~bool~~ | -## StringStore.\_\_iter\_\_ {#iter tag="method"} +## StringStore.\_\_iter\_\_ {id="iter",tag="method"} Iterate over the strings in the store, in order. Note that a newly initialized store will always include an empty string `""` at position `0`. @@ -90,7 +90,7 @@ store will always include an empty string `""` at position `0`. | ---------- | ------------------------------ | | **YIELDS** | A string in the store. ~~str~~ | -## StringStore.add {#add tag="method" new="2"} +## StringStore.add {id="add",tag="method",version="2"} Add a string to the `StringStore`. @@ -110,7 +110,7 @@ Add a string to the `StringStore`. | `string` | The string to add. ~~str~~ | | **RETURNS** | The string's hash value. ~~int~~ | -## StringStore.to_disk {#to_disk tag="method" new="2"} +## StringStore.to_disk {id="to_disk",tag="method",version="2"} Save the current state to a directory. @@ -124,7 +124,7 @@ Save the current state to a directory. | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | -## StringStore.from_disk {#from_disk tag="method" new="2"} +## StringStore.from_disk {id="from_disk",tag="method",version="2"} Loads state from a directory. Modifies the object in place and returns it. @@ -140,7 +140,7 @@ Loads state from a directory. Modifies the object in place and returns it. | `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | **RETURNS** | The modified `StringStore` object. ~~StringStore~~ | -## StringStore.to_bytes {#to_bytes tag="method"} +## StringStore.to_bytes {id="to_bytes",tag="method"} Serialize the current state to a binary string. @@ -154,7 +154,7 @@ Serialize the current state to a binary string. | ----------- | ---------------------------------------------------------- | | **RETURNS** | The serialized form of the `StringStore` object. ~~bytes~~ | -## StringStore.from_bytes {#from_bytes tag="method"} +## StringStore.from_bytes {id="from_bytes",tag="method"} Load state from a binary string. @@ -171,9 +171,9 @@ Load state from a binary string. | `bytes_data` | The data to load from. ~~bytes~~ | | **RETURNS** | The `StringStore` object. ~~StringStore~~ | -## Utilities {#util} +## Utilities {id="util"} -### strings.hash_string {#hash_string tag="function"} +### strings.hash_string {id="hash_string",tag="function"} Get a 64-bit hash for a given string. diff --git a/website/docs/api/tagger.md b/website/docs/api/tagger.mdx similarity index 95% rename from website/docs/api/tagger.md rename to website/docs/api/tagger.mdx index 90a49b197..ee38de81c 100644 --- a/website/docs/api/tagger.md +++ b/website/docs/api/tagger.mdx @@ -14,7 +14,7 @@ part-of-speech tag set. In the pre-trained pipelines, the tag schemas vary by language; see the [individual model pages](/models) for details. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Predictions are assigned to `Token.tag`. @@ -23,7 +23,7 @@ Predictions are assigned to `Token.tag`. | `Token.tag` | The part of speech (hash). ~~int~~ | | `Token.tag_` | The part of speech. ~~str~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -51,7 +51,7 @@ architectures and their arguments and hyperparameters. %%GITHUB_SPACY/spacy/pipeline/tagger.pyx ``` -## Tagger.\_\_init\_\_ {#init tag="method"} +## Tagger.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -81,7 +81,7 @@ shortcut for this and instantiate the component using its string name and | `overwrite` 3.2 | Whether existing annotation is overwritten. Defaults to `False`. ~~bool~~ | | `scorer` 3.2 | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attribute `"tag"`. ~~Optional[Callable]~~ | -## Tagger.\_\_call\_\_ {#call tag="method"} +## Tagger.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -104,7 +104,7 @@ and all pipeline components are applied to the `Doc` in order. Both | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## Tagger.pipe {#pipe tag="method"} +## Tagger.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -127,7 +127,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/tagger#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## Tagger.initialize {#initialize tag="method" new="3"} +## Tagger.initialize {id="initialize",tag="method",version="3"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -170,7 +170,7 @@ This method was previously called `begin_training`. | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ | -## Tagger.predict {#predict tag="method"} +## Tagger.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects, without modifying them. @@ -187,7 +187,7 @@ modifying them. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The model's prediction for each document. | -## Tagger.set_annotations {#set_annotations tag="method"} +## Tagger.set_annotations {id="set_annotations",tag="method"} Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. @@ -204,7 +204,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `scores` | The scores to set, produced by `Tagger.predict`. | -## Tagger.update {#update tag="method"} +## Tagger.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects containing the predictions and gold-standard annotations, and update the component's model. @@ -228,7 +228,7 @@ Delegates to [`predict`](/api/tagger#predict) and | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## Tagger.rehearse {#rehearse tag="method,experimental" new="3"} +## Tagger.rehearse {id="rehearse",tag="method,experimental",version="3"} Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the current model to make predictions similar to an initial model, to try to address @@ -251,7 +251,7 @@ the "catastrophic forgetting" problem. This feature is experimental. | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## Tagger.get_loss {#get_loss tag="method"} +## Tagger.get_loss {id="get_loss",tag="method"} Find the loss and gradient of loss for the batch of documents and their predicted scores. @@ -270,7 +270,7 @@ predicted scores. | `scores` | Scores representing the model's predictions. | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | -## Tagger.create_optimizer {#create_optimizer tag="method"} +## Tagger.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -285,7 +285,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## Tagger.use_params {#use_params tag="method, contextmanager"} +## Tagger.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model, to use the given parameter values. At the end of the context, the original parameters are restored. @@ -302,7 +302,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## Tagger.add_label {#add_label tag="method"} +## Tagger.add_label {id="add_label",tag="method"} Add a new label to the pipe. Raises an error if the output dimension is already set, or if the model has already been fully [initialized](#initialize). Note @@ -324,7 +324,7 @@ automatically. | `label` | The label to add. ~~str~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | -## Tagger.to_disk {#to_disk tag="method"} +## Tagger.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -341,7 +341,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## Tagger.from_disk {#from_disk tag="method"} +## Tagger.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -359,7 +359,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `Tagger` object. ~~Tagger~~ | -## Tagger.to_bytes {#to_bytes tag="method"} +## Tagger.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -376,7 +376,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `Tagger` object. ~~bytes~~ | -## Tagger.from_bytes {#from_bytes tag="method"} +## Tagger.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -395,7 +395,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `Tagger` object. ~~Tagger~~ | -## Tagger.labels {#labels tag="property"} +## Tagger.labels {id="labels",tag="property"} The labels currently added to the component. @@ -410,7 +410,7 @@ The labels currently added to the component. | ----------- | ------------------------------------------------------ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | -## Tagger.label_data {#label_data tag="property" new="3"} +## Tagger.label_data {id="label_data",tag="property",version="3"} The labels currently added to the component and their internal meta information. This is the data generated by [`init labels`](/api/cli#init-labels) and used by @@ -428,7 +428,7 @@ pre-defined label set. | ----------- | ---------------------------------------------------------- | | **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/textcategorizer.md b/website/docs/api/textcategorizer.mdx similarity index 94% rename from website/docs/api/textcategorizer.md rename to website/docs/api/textcategorizer.mdx index f5f8706ec..a259b7b3c 100644 --- a/website/docs/api/textcategorizer.md +++ b/website/docs/api/textcategorizer.mdx @@ -2,7 +2,7 @@ title: TextCategorizer tag: class source: spacy/pipeline/textcat.py -new: 2 +version: 2 teaser: 'Pipeline component for text classification' api_base_class: /api/pipe api_string_name: textcat @@ -29,7 +29,7 @@ only. -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} Predictions will be saved to `doc.cats` as a dictionary, where the key is the name of the category and the value is a score between 0 and 1 (inclusive). For @@ -49,7 +49,7 @@ supported. | ---------- | ------------------------------------- | | `Doc.cats` | Category scores. ~~Dict[str, float]~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -93,7 +93,7 @@ architectures and their arguments and hyperparameters. %%GITHUB_SPACY/spacy/pipeline/textcat_multilabel.py ``` -## TextCategorizer.\_\_init\_\_ {#init tag="method"} +## TextCategorizer.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -125,7 +125,7 @@ shortcut for this and instantiate the component using its string name and | `threshold` | Cutoff to consider a prediction "positive", relevant for `textcat_multilabel` when calculating accuracy scores. ~~float~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_cats`](/api/scorer#score_cats) for the attribute `"cats"`. ~~Optional[Callable]~~ | -## TextCategorizer.\_\_call\_\_ {#call tag="method"} +## TextCategorizer.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -148,7 +148,7 @@ delegate to the [`predict`](/api/textcategorizer#predict) and | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## TextCategorizer.pipe {#pipe tag="method"} +## TextCategorizer.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -172,7 +172,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/textcategorizer#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## TextCategorizer.initialize {#initialize tag="method" new="3"} +## TextCategorizer.initialize {id="initialize",tag="method",version="3"} Initialize the component for training. `get_examples` should be a function that returns an iterable of [`Example`](/api/example) objects. **At least one example @@ -217,7 +217,7 @@ This method was previously called `begin_training`. | `labels` | The label information to add to the component, as provided by the [`label_data`](#label_data) property after initialization. To generate a reusable JSON file from your data, you should run the [`init labels`](/api/cli#init-labels) command. If no labels are provided, the `get_examples` callback is used to extract the labels from the data, which may be a lot slower. ~~Optional[Iterable[str]]~~ | | `positive_label` | The positive label for a binary task with exclusive classes, `None` otherwise and by default. This parameter is only used during scoring. It is not available when using the `textcat_multilabel` component. ~~Optional[str]~~ | -## TextCategorizer.predict {#predict tag="method"} +## TextCategorizer.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects without modifying them. @@ -234,7 +234,7 @@ modifying them. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The model's prediction for each document. | -## TextCategorizer.set_annotations {#set_annotations tag="method"} +## TextCategorizer.set_annotations {id="set_annotations",tag="method"} Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores. @@ -251,7 +251,7 @@ Modify a batch of [`Doc`](/api/doc) objects using pre-computed scores. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `scores` | The scores to set, produced by `TextCategorizer.predict`. | -## TextCategorizer.update {#update tag="method"} +## TextCategorizer.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects containing the predictions and gold-standard annotations, and update the component's model. @@ -275,7 +275,7 @@ Delegates to [`predict`](/api/textcategorizer#predict) and | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## TextCategorizer.rehearse {#rehearse tag="method,experimental" new="3"} +## TextCategorizer.rehearse {id="rehearse",tag="method,experimental",version="3"} Perform a "rehearsal" update from a batch of data. Rehearsal updates teach the current model to make predictions similar to an initial model to try to address @@ -298,7 +298,7 @@ the "catastrophic forgetting" problem. This feature is experimental. | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## TextCategorizer.get_loss {#get_loss tag="method"} +## TextCategorizer.get_loss {id="get_loss",tag="method"} Find the loss and gradient of loss for the batch of documents and their predicted scores. @@ -317,7 +317,7 @@ predicted scores. | `scores` | Scores representing the model's predictions. | | **RETURNS** | The loss and the gradient, i.e. `(loss, gradient)`. ~~Tuple[float, float]~~ | -## TextCategorizer.score {#score tag="method" new="3"} +## TextCategorizer.score {id="score",tag="method",version="3"} Score a batch of examples. @@ -333,7 +333,7 @@ Score a batch of examples. | _keyword-only_ | | | **RETURNS** | The scores, produced by [`Scorer.score_cats`](/api/scorer#score_cats). ~~Dict[str, Union[float, Dict[str, float]]]~~ | -## TextCategorizer.create_optimizer {#create_optimizer tag="method"} +## TextCategorizer.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -348,7 +348,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## TextCategorizer.use_params {#use_params tag="method, contextmanager"} +## TextCategorizer.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model to use the given parameter values. @@ -364,7 +364,7 @@ Modify the pipe's model to use the given parameter values. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## TextCategorizer.add_label {#add_label tag="method"} +## TextCategorizer.add_label {id="add_label",tag="method"} Add a new label to the pipe. Raises an error if the output dimension is already set, or if the model has already been fully [initialized](#initialize). Note @@ -386,7 +386,7 @@ automatically. | `label` | The label to add. ~~str~~ | | **RETURNS** | `0` if the label is already present, otherwise `1`. ~~int~~ | -## TextCategorizer.to_disk {#to_disk tag="method"} +## TextCategorizer.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -403,7 +403,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## TextCategorizer.from_disk {#from_disk tag="method"} +## TextCategorizer.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -421,7 +421,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `TextCategorizer` object. ~~TextCategorizer~~ | -## TextCategorizer.to_bytes {#to_bytes tag="method"} +## TextCategorizer.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -438,7 +438,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `TextCategorizer` object. ~~bytes~~ | -## TextCategorizer.from_bytes {#from_bytes tag="method"} +## TextCategorizer.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -457,7 +457,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `TextCategorizer` object. ~~TextCategorizer~~ | -## TextCategorizer.labels {#labels tag="property"} +## TextCategorizer.labels {id="labels",tag="property"} The labels currently added to the component. @@ -472,7 +472,7 @@ The labels currently added to the component. | ----------- | ------------------------------------------------------ | | **RETURNS** | The labels added to the component. ~~Tuple[str, ...]~~ | -## TextCategorizer.label_data {#label_data tag="property" new="3"} +## TextCategorizer.label_data {id="label_data",tag="property",version="3"} The labels currently added to the component and their internal meta information. This is the data generated by [`init labels`](/api/cli#init-labels) and used by @@ -490,7 +490,7 @@ the model with a pre-defined label set. | ----------- | ---------------------------------------------------------- | | **RETURNS** | The label data added to the component. ~~Tuple[str, ...]~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/tok2vec.md b/website/docs/api/tok2vec.mdx similarity index 94% rename from website/docs/api/tok2vec.md rename to website/docs/api/tok2vec.mdx index 2dcb1a013..a1bb1265e 100644 --- a/website/docs/api/tok2vec.md +++ b/website/docs/api/tok2vec.mdx @@ -1,7 +1,7 @@ --- title: Tok2Vec source: spacy/pipeline/tok2vec.py -new: 3 +version: 3 teaser: null api_base_class: /api/pipe api_string_name: tok2vec @@ -23,7 +23,7 @@ components can backpropagate to the shared weights. This implementation is used because it allows us to avoid relying on object identity within the models to achieve the parameter sharing. -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -48,7 +48,7 @@ architectures and their arguments and hyperparameters. %%GITHUB_SPACY/spacy/pipeline/tok2vec.py ``` -## Tok2Vec.\_\_init\_\_ {#init tag="method"} +## Tok2Vec.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -75,7 +75,7 @@ shortcut for this and instantiate the component using its string name and | `model` | The Thinc [`Model`](https://thinc.ai/docs/api-model) powering the pipeline component. ~~Model[List[Doc], List[Floats2d]~~ | | `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ | -## Tok2Vec.\_\_call\_\_ {#call tag="method"} +## Tok2Vec.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document and add context-sensitive embeddings to the `Doc.tensor` attribute, allowing them to be used as features by downstream @@ -100,7 +100,7 @@ pipeline components are applied to the `Doc` in order. Both | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## Tok2Vec.pipe {#pipe tag="method"} +## Tok2Vec.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -123,7 +123,7 @@ and [`set_annotations`](/api/tok2vec#set_annotations) methods. | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## Tok2Vec.initialize {#initialize tag="method"} +## Tok2Vec.initialize {id="initialize",tag="method"} Initialize the component for training and return an [`Optimizer`](https://thinc.ai/docs/api-optimizers). `get_examples` should be a @@ -148,7 +148,7 @@ by [`Language.initialize`](/api/language#initialize). | _keyword-only_ | | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | -## Tok2Vec.predict {#predict tag="method"} +## Tok2Vec.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects without modifying them. @@ -165,7 +165,7 @@ modifying them. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The model's prediction for each document. | -## Tok2Vec.set_annotations {#set_annotations tag="method"} +## Tok2Vec.set_annotations {id="set_annotations",tag="method"} Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. @@ -182,7 +182,7 @@ Modify a batch of [`Doc`](/api/doc) objects, using pre-computed scores. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `scores` | The scores to set, produced by `Tok2Vec.predict`. | -## Tok2Vec.update {#update tag="method"} +## Tok2Vec.update {id="update",tag="method"} Learn from a batch of [`Example`](/api/example) objects containing the predictions and gold-standard annotations, and update the component's model. @@ -205,7 +205,7 @@ Delegates to [`predict`](/api/tok2vec#predict). | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## Tok2Vec.create_optimizer {#create_optimizer tag="method"} +## Tok2Vec.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -220,7 +220,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## Tok2Vec.use_params {#use_params tag="method, contextmanager"} +## Tok2Vec.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model to use the given parameter values. At the end of the context, the original parameters are restored. @@ -237,7 +237,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## Tok2Vec.to_disk {#to_disk tag="method"} +## Tok2Vec.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -254,7 +254,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## Tok2Vec.from_disk {#from_disk tag="method"} +## Tok2Vec.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -272,7 +272,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `Tok2Vec` object. ~~Tok2Vec~~ | -## Tok2Vec.to_bytes {#to_bytes tag="method"} +## Tok2Vec.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -289,7 +289,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `Tok2Vec` object. ~~bytes~~ | -## Tok2Vec.from_bytes {#from_bytes tag="method"} +## Tok2Vec.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -308,7 +308,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `Tok2Vec` object. ~~Tok2Vec~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/token.md b/website/docs/api/token.mdx similarity index 96% rename from website/docs/api/token.md rename to website/docs/api/token.mdx index 89bd77447..63ee1080b 100644 --- a/website/docs/api/token.md +++ b/website/docs/api/token.mdx @@ -5,7 +5,7 @@ tag: class source: spacy/tokens/token.pyx --- -## Token.\_\_init\_\_ {#init tag="method"} +## Token.\_\_init\_\_ {id="init",tag="method"} Construct a `Token` object. @@ -23,7 +23,7 @@ Construct a `Token` object. | `doc` | The parent document. ~~Doc~~ | | `offset` | The index of the token within the document. ~~int~~ | -## Token.\_\_len\_\_ {#len tag="method"} +## Token.\_\_len\_\_ {id="len",tag="method"} The number of unicode characters in the token, i.e. `token.text`. @@ -39,7 +39,7 @@ The number of unicode characters in the token, i.e. `token.text`. | ----------- | ------------------------------------------------------ | | **RETURNS** | The number of unicode characters in the token. ~~int~~ | -## Token.set_extension {#set_extension tag="classmethod" new="2"} +## Token.set_extension {id="set_extension",tag="classmethod",version="2"} Define a custom attribute on the `Token` which becomes available via `Token._`. For details, see the documentation on @@ -64,7 +64,7 @@ For details, see the documentation on | `setter` | Setter function that takes the `Token` and a value, and modifies the object. Is called when the user writes to the `Token._` attribute. ~~Optional[Callable[[Token, Any], None]]~~ | | `force` | Force overwriting existing attribute. ~~bool~~ | -## Token.get_extension {#get_extension tag="classmethod" new="2"} +## Token.get_extension {id="get_extension",tag="classmethod",version="2"} Look up a previously registered extension by name. Returns a 4-tuple `(default, method, getter, setter)` if the extension is registered. Raises a @@ -84,7 +84,7 @@ Look up a previously registered extension by name. Returns a 4-tuple | `name` | Name of the extension. ~~str~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | -## Token.has_extension {#has_extension tag="classmethod" new="2"} +## Token.has_extension {id="has_extension",tag="classmethod",version="2"} Check whether an extension has been registered on the `Token` class. @@ -101,7 +101,7 @@ Check whether an extension has been registered on the `Token` class. | `name` | Name of the extension to check. ~~str~~ | | **RETURNS** | Whether the extension has been registered. ~~bool~~ | -## Token.remove_extension {#remove_extension tag="classmethod" new=""2.0.11""} +## Token.remove_extension {id="remove_extension",tag="classmethod",version="2.0.11"} Remove a previously registered extension. @@ -119,7 +119,7 @@ Remove a previously registered extension. | `name` | Name of the extension. ~~str~~ | | **RETURNS** | A `(default, method, getter, setter)` tuple of the removed extension. ~~Tuple[Optional[Any], Optional[Callable], Optional[Callable], Optional[Callable]]~~ | -## Token.check_flag {#check_flag tag="method"} +## Token.check_flag {id="check_flag",tag="method"} Check the value of a boolean flag. @@ -137,7 +137,7 @@ Check the value of a boolean flag. | `flag_id` | The attribute ID of the flag to check. ~~int~~ | | **RETURNS** | Whether the flag is set. ~~bool~~ | -## Token.similarity {#similarity tag="method" model="vectors"} +## Token.similarity {id="similarity",tag="method",model="vectors"} Compute a semantic similarity estimate. Defaults to cosine over vectors. @@ -155,7 +155,7 @@ Compute a semantic similarity estimate. Defaults to cosine over vectors. | other | The object to compare with. By default, accepts `Doc`, `Span`, `Token` and `Lexeme` objects. ~~Union[Doc, Span, Token, Lexeme]~~ | | **RETURNS** | A scalar similarity score. Higher is more similar. ~~float~~ | -## Token.nbor {#nbor tag="method"} +## Token.nbor {id="nbor",tag="method"} Get a neighboring token. @@ -172,7 +172,7 @@ Get a neighboring token. | `i` | The relative position of the token to get. Defaults to `1`. ~~int~~ | | **RETURNS** | The token at position `self.doc[self.i+i]`. ~~Token~~ | -## Token.set_morph {#set_morph tag="method"} +## Token.set_morph {id="set_morph",tag="method"} Set the morphological analysis from a UD FEATS string, hash value of a UD FEATS string, features dict or `MorphAnalysis`. The value `None` can be used to reset @@ -191,7 +191,7 @@ the morph to an unset state. | -------- | --------------------------------------------------------------------------------- | | features | The morphological features to set. ~~Union[int, dict, str, MorphAnalysis, None]~~ | -## Token.has_morph {#has_morph tag="method"} +## Token.has_morph {id="has_morph",tag="method"} Check whether the token has annotated morph information. Return `False` when the morph annotation is unset/missing. @@ -200,7 +200,7 @@ morph annotation is unset/missing. | ----------- | --------------------------------------------- | | **RETURNS** | Whether the morph annotation is set. ~~bool~~ | -## Token.is_ancestor {#is_ancestor tag="method" model="parser"} +## Token.is_ancestor {id="is_ancestor",tag="method",model="parser"} Check whether this token is a parent, grandparent, etc. of another in the dependency tree. @@ -219,7 +219,7 @@ dependency tree. | descendant | Another token. ~~Token~~ | | **RETURNS** | Whether this token is the ancestor of the descendant. ~~bool~~ | -## Token.ancestors {#ancestors tag="property" model="parser"} +## Token.ancestors {id="ancestors",tag="property",model="parser"} A sequence of the token's syntactic ancestors (parents, grandparents, etc). @@ -237,7 +237,7 @@ A sequence of the token's syntactic ancestors (parents, grandparents, etc). | ---------- | ------------------------------------------------------------------------------- | | **YIELDS** | A sequence of ancestor tokens such that `ancestor.is_ancestor(self)`. ~~Token~~ | -## Token.conjuncts {#conjuncts tag="property" model="parser"} +## Token.conjuncts {id="conjuncts",tag="property",model="parser"} A tuple of coordinated tokens, not including the token itself. @@ -253,7 +253,7 @@ A tuple of coordinated tokens, not including the token itself. | ----------- | --------------------------------------------- | | **RETURNS** | The coordinated tokens. ~~Tuple[Token, ...]~~ | -## Token.children {#children tag="property" model="parser"} +## Token.children {id="children",tag="property",model="parser"} A sequence of the token's immediate syntactic children. @@ -269,7 +269,7 @@ A sequence of the token's immediate syntactic children. | ---------- | ------------------------------------------------------- | | **YIELDS** | A child token such that `child.head == self`. ~~Token~~ | -## Token.lefts {#lefts tag="property" model="parser"} +## Token.lefts {id="lefts",tag="property",model="parser"} The leftward immediate children of the word in the syntactic dependency parse. @@ -285,7 +285,7 @@ The leftward immediate children of the word in the syntactic dependency parse. | ---------- | ------------------------------------ | | **YIELDS** | A left-child of the token. ~~Token~~ | -## Token.rights {#rights tag="property" model="parser"} +## Token.rights {id="rights",tag="property",model="parser"} The rightward immediate children of the word in the syntactic dependency parse. @@ -301,7 +301,7 @@ The rightward immediate children of the word in the syntactic dependency parse. | ---------- | ------------------------------------- | | **YIELDS** | A right-child of the token. ~~Token~~ | -## Token.n_lefts {#n_lefts tag="property" model="parser"} +## Token.n_lefts {id="n_lefts",tag="property",model="parser"} The number of leftward immediate children of the word in the syntactic dependency parse. @@ -317,7 +317,7 @@ dependency parse. | ----------- | ---------------------------------------- | | **RETURNS** | The number of left-child tokens. ~~int~~ | -## Token.n_rights {#n_rights tag="property" model="parser"} +## Token.n_rights {id="n_rights",tag="property",model="parser"} The number of rightward immediate children of the word in the syntactic dependency parse. @@ -333,7 +333,7 @@ dependency parse. | ----------- | ----------------------------------------- | | **RETURNS** | The number of right-child tokens. ~~int~~ | -## Token.subtree {#subtree tag="property" model="parser"} +## Token.subtree {id="subtree",tag="property",model="parser"} A sequence containing the token and all the token's syntactic descendants. @@ -349,7 +349,7 @@ A sequence containing the token and all the token's syntactic descendants. | ---------- | ------------------------------------------------------------------------------------ | | **YIELDS** | A descendant token such that `self.is_ancestor(token)` or `token == self`. ~~Token~~ | -## Token.has_vector {#has_vector tag="property" model="vectors"} +## Token.has_vector {id="has_vector",tag="property",model="vectors"} A boolean value indicating whether a word vector is associated with the token. @@ -365,7 +365,7 @@ A boolean value indicating whether a word vector is associated with the token. | ----------- | ------------------------------------------------------ | | **RETURNS** | Whether the token has a vector data attached. ~~bool~~ | -## Token.vector {#vector tag="property" model="vectors"} +## Token.vector {id="vector",tag="property",model="vectors"} A real-valued meaning representation. @@ -382,7 +382,7 @@ A real-valued meaning representation. | ----------- | ----------------------------------------------------------------------------------------------- | | **RETURNS** | A 1-dimensional array representing the token's vector. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | -## Token.vector_norm {#vector_norm tag="property" model="vectors"} +## Token.vector_norm {id="vector_norm",tag="property",model="vectors"} The L2 norm of the token's vector representation. @@ -401,7 +401,7 @@ The L2 norm of the token's vector representation. | ----------- | --------------------------------------------------- | | **RETURNS** | The L2 norm of the vector representation. ~~float~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/website/docs/api/tokenizer.md b/website/docs/api/tokenizer.mdx similarity index 95% rename from website/docs/api/tokenizer.md rename to website/docs/api/tokenizer.mdx index 6eb7e8024..0a579ab4c 100644 --- a/website/docs/api/tokenizer.md +++ b/website/docs/api/tokenizer.mdx @@ -20,7 +20,7 @@ The tokenizer is typically created automatically when a like punctuation and special case rules from the [`Language.Defaults`](/api/language#defaults) provided by the language subclass. -## Tokenizer.\_\_init\_\_ {#init tag="method"} +## Tokenizer.\_\_init\_\_ {id="init",tag="method"} Create a `Tokenizer` to create `Doc` objects given unicode text. For examples of how to construct a custom tokenizer with different tokenization rules, see the @@ -55,7 +55,7 @@ how to construct a custom tokenizer with different tokenization rules, see the | `url_match` | A function matching the signature of `re.compile(string).match` to find token matches after considering prefixes and suffixes. ~~Optional[Callable[[str], Optional[Match]]]~~ | | `faster_heuristics` 3.3.0 | Whether to restrict the final `Matcher`-based pass for rules to those containing affixes or space. Defaults to `True`. ~~bool~~ | -## Tokenizer.\_\_call\_\_ {#call tag="method"} +## Tokenizer.\_\_call\_\_ {id="call",tag="method"} Tokenize a string. @@ -71,7 +71,7 @@ Tokenize a string. | `string` | The string to tokenize. ~~str~~ | | **RETURNS** | A container for linguistic annotations. ~~Doc~~ | -## Tokenizer.pipe {#pipe tag="method"} +## Tokenizer.pipe {id="pipe",tag="method"} Tokenize a stream of texts. @@ -89,7 +89,7 @@ Tokenize a stream of texts. | `batch_size` | The number of texts to accumulate in an internal buffer. Defaults to `1000`. ~~int~~ | | **YIELDS** | The tokenized `Doc` objects, in order. ~~Doc~~ | -## Tokenizer.find_infix {#find_infix tag="method"} +## Tokenizer.find_infix {id="find_infix",tag="method"} Find internal split points of the string. @@ -98,7 +98,7 @@ Find internal split points of the string. | `string` | The string to split. ~~str~~ | | **RETURNS** | A list of `re.MatchObject` objects that have `.start()` and `.end()` methods, denoting the placement of internal segment separators, e.g. hyphens. ~~List[Match]~~ | -## Tokenizer.find_prefix {#find_prefix tag="method"} +## Tokenizer.find_prefix {id="find_prefix",tag="method"} Find the length of a prefix that should be segmented from the string, or `None` if no prefix rules match. @@ -108,7 +108,7 @@ if no prefix rules match. | `string` | The string to segment. ~~str~~ | | **RETURNS** | The length of the prefix if present, otherwise `None`. ~~Optional[int]~~ | -## Tokenizer.find_suffix {#find_suffix tag="method"} +## Tokenizer.find_suffix {id="find_suffix",tag="method"} Find the length of a suffix that should be segmented from the string, or `None` if no suffix rules match. @@ -118,7 +118,7 @@ if no suffix rules match. | `string` | The string to segment. ~~str~~ | | **RETURNS** | The length of the suffix if present, otherwise `None`. ~~Optional[int]~~ | -## Tokenizer.add_special_case {#add_special_case tag="method"} +## Tokenizer.add_special_case {id="add_special_case",tag="method"} Add a special-case tokenization rule. This mechanism is also used to add custom tokenizer exceptions to the language data. See the usage guide on the @@ -139,7 +139,7 @@ details and examples. | `string` | The string to specially tokenize. ~~str~~ | | `token_attrs` | A sequence of dicts, where each dict describes a token and its attributes. The `ORTH` fields of the attributes must exactly match the string when they are concatenated. ~~Iterable[Dict[int, str]]~~ | -## Tokenizer.explain {#explain tag="method"} +## Tokenizer.explain {id="explain",tag="method"} Tokenize a string with a slow debugging tokenizer that provides information about which tokenizer rule or pattern was matched for each token. The tokens @@ -158,7 +158,7 @@ produced are identical to `Tokenizer.__call__` except for whitespace tokens. | `string` | The string to tokenize with the debugging tokenizer. ~~str~~ | | **RETURNS** | A list of `(pattern_string, token_string)` tuples. ~~List[Tuple[str, str]]~~ | -## Tokenizer.to_disk {#to_disk tag="method"} +## Tokenizer.to_disk {id="to_disk",tag="method"} Serialize the tokenizer to disk. @@ -175,7 +175,7 @@ Serialize the tokenizer to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## Tokenizer.from_disk {#from_disk tag="method"} +## Tokenizer.from_disk {id="from_disk",tag="method"} Load the tokenizer from disk. Modifies the object in place and returns it. @@ -193,7 +193,7 @@ Load the tokenizer from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `Tokenizer` object. ~~Tokenizer~~ | -## Tokenizer.to_bytes {#to_bytes tag="method"} +## Tokenizer.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -210,7 +210,7 @@ Serialize the tokenizer to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `Tokenizer` object. ~~bytes~~ | -## Tokenizer.from_bytes {#from_bytes tag="method"} +## Tokenizer.from_bytes {id="from_bytes",tag="method"} Load the tokenizer from a bytestring. Modifies the object in place and returns it. @@ -230,7 +230,7 @@ it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `Tokenizer` object. ~~Tokenizer~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -241,7 +241,7 @@ it. | `token_match` | A function matching the signature of `re.compile(string).match` to find token matches. Returns an `re.MatchObject` or `None`. ~~Optional[Callable[[str], Optional[Match]]]~~ | | `rules` | A dictionary of tokenizer exceptions and special cases. ~~Optional[Dict[str, List[Dict[int, str]]]]~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/api/top-level.md b/website/docs/api/top-level.mdx similarity index 93% rename from website/docs/api/top-level.md rename to website/docs/api/top-level.mdx index 9d3e463d8..a222cfa8f 100644 --- a/website/docs/api/top-level.md +++ b/website/docs/api/top-level.mdx @@ -13,9 +13,9 @@ menu: - ['Utility Functions', 'util'] --- -## spaCy {#spacy hidden="true"} +## spaCy {id="spacy",hidden="true"} -### spacy.load {#spacy.load tag="function"} +### spacy.load {id="spacy.load",tag="function"} Load a pipeline using the name of an installed [package](/usage/saving-loading#models), a string path or a `Path`-like object. @@ -61,8 +61,7 @@ Essentially, `spacy.load()` is a convenience wrapper that reads the pipeline's information to construct a `Language` object, loads in the model data and weights, and returns it. -```python -### Abstract example +```python {title="Abstract example"} cls = spacy.util.get_lang_class(lang) # 1. Get Language class, e.g. English nlp = cls() # 2. Initialize it for name in pipeline: @@ -70,7 +69,7 @@ for name in pipeline: nlp.from_disk(data_path) # 4. Load in the binary data ``` -### spacy.blank {#spacy.blank tag="function" new="2"} +### spacy.blank {id="spacy.blank",tag="function",version="2"} Create a blank pipeline of a given language class. This function is the twin of `spacy.load()`. @@ -91,7 +90,7 @@ Create a blank pipeline of a given language class. This function is the twin of | `meta` | Optional meta overrides for [`nlp.meta`](/api/language#meta). ~~Dict[str, Any]~~ | | **RETURNS** | An empty `Language` object of the appropriate subclass. ~~Language~~ | -### spacy.info {#spacy.info tag="function"} +### spacy.info {id="spacy.info",tag="function"} The same as the [`info` command](/api/cli#info). Pretty-print information about your installation, installed pipelines and local setup from within spaCy. @@ -111,7 +110,7 @@ your installation, installed pipelines and local setup from within spaCy. | `markdown` | Print information as Markdown. ~~bool~~ | | `silent` | Don't print anything, just return. ~~bool~~ | -### spacy.explain {#spacy.explain tag="function"} +### spacy.explain {id="spacy.explain",tag="function"} Get a description for a given POS tag, dependency label or entity type. For a list of available terms, see [`glossary.py`](%%GITHUB_SPACY/spacy/glossary.py). @@ -134,7 +133,7 @@ list of available terms, see [`glossary.py`](%%GITHUB_SPACY/spacy/glossary.py). | `term` | Term to explain. ~~str~~ | | **RETURNS** | The explanation, or `None` if not found in the glossary. ~~Optional[str]~~ | -### spacy.prefer_gpu {#spacy.prefer_gpu tag="function" new="2.0.14"} +### spacy.prefer_gpu {id="spacy.prefer_gpu",tag="function",version="2.0.14"} Allocate data and perform operations on [GPU](/usage/#gpu), if available. If data has already been allocated on CPU, it will not be moved. Ideally, this @@ -162,7 +161,7 @@ ensure that the model is loaded on the correct device. See | `gpu_id` | Device index to select. Defaults to `0`. ~~int~~ | | **RETURNS** | Whether the GPU was activated. ~~bool~~ | -### spacy.require_gpu {#spacy.require_gpu tag="function" new="2.0.14"} +### spacy.require_gpu {id="spacy.require_gpu",tag="function",version="2.0.14"} Allocate data and perform operations on [GPU](/usage/#gpu). Will raise an error if no GPU is available. If data has already been allocated on CPU, it will not @@ -190,7 +189,7 @@ ensure that the model is loaded on the correct device. See | `gpu_id` | Device index to select. Defaults to `0`. ~~int~~ | | **RETURNS** | `True` ~~bool~~ | -### spacy.require_cpu {#spacy.require_cpu tag="function" new="3.0.0"} +### spacy.require_cpu {id="spacy.require_cpu",tag="function",version="3.0.0"} Allocate data and perform operations on CPU. If data has already been allocated on GPU, it will not be moved. Ideally, this function should be called right @@ -216,12 +215,12 @@ ensure that the model is loaded on the correct device. See | ----------- | --------------- | | **RETURNS** | `True` ~~bool~~ | -## displaCy {#displacy source="spacy/displacy"} +## displaCy {id="displacy",source="spacy/displacy"} As of v2.0, spaCy comes with a built-in visualization suite. For more info and examples, see the usage guide on [visualizing spaCy](/usage/visualizers). -### displacy.serve {#displacy.serve tag="method" new="2"} +### displacy.serve {id="displacy.serve",tag="method",version="2"} Serve a dependency parse tree or named entity visualization to view it in your browser. Will run a simple web server. @@ -249,7 +248,7 @@ browser. Will run a simple web server. | `host` | Host to serve visualization. Defaults to `"0.0.0.0"`. ~~str~~ | | `auto_select_port` | If `True`, automatically switch to a different port if the specified port is already in use. Defaults to `False`. ~~bool~~ | -### displacy.render {#displacy.render tag="method" new="2"} +### displacy.render {id="displacy.render",tag="method",version="2"} Render a dependency parse tree or named entity visualization. @@ -274,7 +273,7 @@ Render a dependency parse tree or named entity visualization. | `jupyter` | Explicitly enable or disable "[Jupyter](http://jupyter.org/) mode" to return markup ready to be rendered in a notebook. Detected automatically if `None` (default). ~~Optional[bool]~~ | | **RETURNS** | The rendered HTML markup. ~~str~~ | -### displacy.parse_deps {#displacy.parse_deps tag="method" new="2"} +### displacy.parse_deps {id="displacy.parse_deps",tag="method",version="2"} Generate dependency parse in `{'words': [], 'arcs': []}` format. For use with the `manual=True` argument in `displacy.render`. @@ -296,7 +295,7 @@ the `manual=True` argument in `displacy.render`. | `options` | Dependency parse specific visualisation options. ~~Dict[str, Any]~~ | | **RETURNS** | Generated dependency parse keyed by words and arcs. ~~dict~~ | -### displacy.parse_ents {#displacy.parse_ents tag="method" new="2"} +### displacy.parse_ents {id="displacy.parse_ents",tag="method",version="2"} Generate named entities in `[{start: i, end: i, label: 'label'}]` format. For use with the `manual=True` argument in `displacy.render`. @@ -318,7 +317,7 @@ use with the `manual=True` argument in `displacy.render`. | `options` | NER-specific visualisation options. ~~Dict[str, Any]~~ | | **RETURNS** | Generated entities keyed by text (original text) and ents. ~~dict~~ | -### displacy.parse_spans {#displacy.parse_spans tag="method" new="2"} +### displacy.parse_spans {id="displacy.parse_spans",tag="method",version="2"} Generate spans in `[{start_token: i, end_token: i, label: 'label'}]` format. For use with the `manual=True` argument in `displacy.render`. @@ -341,12 +340,12 @@ use with the `manual=True` argument in `displacy.render`. | `options` | Span-specific visualisation options. ~~Dict[str, Any]~~ | | **RETURNS** | Generated entities keyed by text (original text) and ents. ~~dict~~ | -### Visualizer options {#displacy_options} +### Visualizer options {id="displacy_options"} The `options` argument lets you specify additional settings for each visualizer. If a setting is not present in the options, the default value will be used. -#### Dependency Visualizer options {#options-dep} +#### Dependency Visualizer options {id="options-dep"} > #### Example > @@ -372,7 +371,7 @@ If a setting is not present in the options, the default value will be used. | `word_spacing` | Vertical spacing between words and arcs in px. Defaults to `45`. ~~int~~ | | `distance` | Distance between words in px. Defaults to `175` in regular mode and `150` in compact mode. ~~int~~ | -#### Named Entity Visualizer options {#displacy_options-ent} +#### Named Entity Visualizer options {id="displacy_options-ent"} > #### Example > @@ -389,7 +388,7 @@ If a setting is not present in the options, the default value will be used. | `template` | Optional template to overwrite the HTML used to render entity spans. Should be a format string and can use `{bg}`, `{text}` and `{label}`. See [`templates.py`](%%GITHUB_SPACY/spacy/displacy/templates.py) for examples. ~~Optional[str]~~ | | `kb_url_template` 3.2.1 | Optional template to construct the KB url for the entity to link to. Expects a python f-string format with single field to fill in. ~~Optional[str]~~ | -#### Span Visualizer options {#displacy_options-span} +#### Span Visualizer options {id="displacy_options-span"} > #### Example > @@ -420,7 +419,7 @@ span. If you wish to link an entity to their URL then consider using the should redirect you to their Wikidata page, in this case `https://www.wikidata.org/wiki/Q95`. -## registry {#registry source="spacy/util.py" new="3"} +## registry {id="registry",source="spacy/util.py",version="3"} spaCy's function registry extends [Thinc's `registry`](https://thinc.ai/docs/api-config#registry) and allows you @@ -470,7 +469,7 @@ factories. | `scorers` | Registry for functions that create scoring methods for user with the [`Scorer`](/api/scorer). Scoring methods are called with `Iterable[Example]` and arbitrary `\*\*kwargs` and return scores as `Dict[str, Any]`. | | `tokenizers` | Registry for tokenizer factories. Registered functions should return a callback that receives the `nlp` object and returns a [`Tokenizer`](/api/tokenizer) or a custom callable. | -### spacy-transformers registry {#registry-transformers} +### spacy-transformers registry {id="registry-transformers"} The following registries are added by the [`spacy-transformers`](https://github.com/explosion/spacy-transformers) package. @@ -495,7 +494,7 @@ See the [`Transformer`](/api/transformer) API reference and | [`span_getters`](/api/transformer#span_getters) | Registry for functions that take a batch of `Doc` objects and return a list of `Span` objects to process by the transformer, e.g. sentences. | | [`annotation_setters`](/api/transformer#annotation_setters) | Registry for functions that create annotation setters. Annotation setters are functions that take a batch of `Doc` objects and a [`FullTransformerBatch`](/api/transformer#fulltransformerbatch) and can set additional annotations on the `Doc`. | -## Loggers {#loggers source="spacy/training/loggers.py" new="3"} +## Loggers {id="loggers",source="spacy/training/loggers.py",version="3"} A logger records the training results. When a logger is created, two functions are returned: one for logging the information for each training step, and a @@ -531,7 +530,7 @@ saves them to a `jsonl` file. -```cli +```bash $ python -m spacy train config.cfg ``` @@ -571,7 +570,7 @@ start decreasing across epochs. | `console_output` | Whether the logger should print the logs in the console (default: `True`). ~~bool~~ | | `output_file` | The file to save the training logs to (default: `None`). ~~Optional[Union[str, Path]]~~ | -#### spacy.ConsoleLogger.v3 {#ConsoleLogger tag="registered function"} +#### spacy.ConsoleLogger.v3 {id="ConsoleLogger",tag="registered function"} > #### Example config > @@ -593,9 +592,9 @@ optionally saves them to a `jsonl` file. | `console_output` | Whether the logger should print the logs in the console (default: `True`). ~~bool~~ | | `output_file` | The file to save the training logs to (default: `None`). ~~Optional[Union[str, Path]]~~ | -## Readers {#readers} +## Readers {id="readers"} -### File readers {#file-readers source="github.com/explosion/srsly" new="3"} +### File readers {id="file-readers",source="github.com/explosion/srsly",version="3"} The following file readers are provided by our serialization library [`srsly`](https://github.com/explosion/srsly). All registered functions take one @@ -625,7 +624,7 @@ blocks that are **not executed at runtime** – for example, in `[training]` and -#### spacy.read_labels.v1 {#read_labels tag="registered function"} +#### spacy.read_labels.v1 {id="read_labels",tag="registered function"} Read a JSON-formatted labels file generated with [`init labels`](/api/cli#init-labels). Typically used in the @@ -651,7 +650,7 @@ label sets. | `require` | Whether to require the file to exist. If set to `False` and the labels file doesn't exist, the loader will return `None` and the `initialize` method will extract the labels from the data. Defaults to `False`. ~~bool~~ | | **CREATES** | The list of labels. ~~List[str]~~ | -### Corpus readers {#corpus-readers source="spacy/training/corpus.py" new="3"} +### Corpus readers {id="corpus-readers",source="spacy/training/corpus.py",version="3"} Corpus readers are registered functions that load data and return a function that takes the current `nlp` object and yields [`Example`](/api/example) objects @@ -661,7 +660,7 @@ with your own registered function in the [`@readers` registry](/api/top-level#registry) to customize the data loading and streaming. -#### spacy.Corpus.v1 {#corpus tag="registered function"} +#### spacy.Corpus.v1 {id="corpus",tag="registered function"} The `Corpus` reader manages annotated corpora and can be used for training and development datasets in the [DocBin](/api/docbin) (`.spacy`) format. Also see @@ -690,7 +689,7 @@ the [`Corpus`](/api/corpus) class. | `augmenter` | Apply some simply data augmentation, where we replace tokens with variations. This is especially useful for punctuation and case replacement, to help generalize beyond corpora that don't have smart-quotes, or only have smart quotes, etc. Defaults to `None`. ~~Optional[Callable]~~ | | **CREATES** | The corpus reader. ~~Corpus~~ | -#### spacy.JsonlCorpus.v1 {#jsonlcorpus tag="registered function"} +#### spacy.JsonlCorpus.v1 {id="jsonlcorpus",tag="registered function"} Create [`Example`](/api/example) objects from a JSONL (newline-delimited JSON) file of texts keyed by `"text"`. Can be used to read the raw text corpus for @@ -719,7 +718,7 @@ JSONL file. Also see the [`JsonlCorpus`](/api/corpus#jsonlcorpus) class. | `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ | | **CREATES** | The corpus reader. ~~JsonlCorpus~~ | -## Batchers {#batchers source="spacy/training/batchers.py" new="3"} +## Batchers {id="batchers",source="spacy/training/batchers.py",version="3"} A data batcher implements a batching strategy that essentially turns a stream of items into a stream of batches, with each batch consisting of one item or a list @@ -733,7 +732,7 @@ Instead of using one of the built-in batchers listed here, you can also [implement your own](/usage/training#custom-code-readers-batchers), which may or may not use a custom schedule. -### spacy.batch_by_words.v1 {#batch_by_words tag="registered function"} +### spacy.batch_by_words.v1 {id="batch_by_words",tag="registered function"} Create minibatches of roughly a given number of words. If any examples are longer than the specified batch length, they will appear in a batch by @@ -761,7 +760,7 @@ themselves, or be discarded if `discard_oversize` is set to `True`. The argument | `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ | | **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ | -### spacy.batch_by_sequence.v1 {#batch_by_sequence tag="registered function"} +### spacy.batch_by_sequence.v1 {id="batch_by_sequence",tag="registered function"} > #### Example config > @@ -780,7 +779,7 @@ Create a batcher that creates batches of the specified size. | `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ | | **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ | -### spacy.batch_by_padded.v1 {#batch_by_padded tag="registered function"} +### spacy.batch_by_padded.v1 {id="batch_by_padded",tag="registered function"} > #### Example config > @@ -806,7 +805,7 @@ sequences in the batch. | `get_length` | Optional function that receives a sequence item and returns its length. Defaults to the built-in `len()` if not set. ~~Optional[Callable[[Any], int]]~~ | | **CREATES** | The batcher that takes an iterable of items and returns batches. ~~Callable[[Iterable[Any]], Iterable[List[Any]]]~~ | -## Augmenters {#augmenters source="spacy/training/augment.py" new="3"} +## Augmenters {id="augmenters",source="spacy/training/augment.py",version="3"} Data augmentation is the process of applying small modifications to the training data. It can be especially useful for punctuation and case replacement – for @@ -815,7 +814,7 @@ variations using regular quotes, or to make the model less sensitive to capitalization by including a mix of capitalized and lowercase examples. See the [usage guide](/usage/training#data-augmentation) for details and examples. -### spacy.orth_variants.v1 {#orth_variants tag="registered function"} +### spacy.orth_variants.v1 {id="orth_variants",tag="registered function"} > #### Example config > @@ -842,7 +841,7 @@ beyond corpora that don't have smart quotes, or only have smart quotes etc. | `orth_variants` | A dictionary containing the single and paired orth variants. Typically loaded from a JSON file. See [`en_orth_variants.json`](https://github.com/explosion/spacy-lookups-data/blob/master/spacy_lookups_data/data/en_orth_variants.json) for an example. ~~Dict[str, Dict[List[Union[str, List[str]]]]]~~ | | **CREATES** | A function that takes the current `nlp` object and an [`Example`](/api/example) and yields augmented `Example` objects. ~~Callable[[Language, Example], Iterator[Example]]~~ | -### spacy.lower_case.v1 {#lower_case tag="registered function"} +### spacy.lower_case.v1 {id="lower_case",tag="registered function"} > #### Example config > @@ -861,12 +860,12 @@ useful for making the model less sensitive to capitalization. | `level` | The percentage of texts that will be augmented. ~~float~~ | | **CREATES** | A function that takes the current `nlp` object and an [`Example`](/api/example) and yields augmented `Example` objects. ~~Callable[[Language, Example], Iterator[Example]]~~ | -## Callbacks {#callbacks source="spacy/training/callbacks.py" new="3"} +## Callbacks {id="callbacks",source="spacy/training/callbacks.py",version="3"} The config supports [callbacks](/usage/training#custom-code-nlp-callbacks) at several points in the lifecycle that can be used modify the `nlp` object. -### spacy.copy_from_base_model.v1 {#copy_from_base_model tag="registered function"} +### spacy.copy_from_base_model.v1 {id="copy_from_base_model",tag="registered function"} > #### Example config > @@ -890,7 +889,7 @@ from the specified model. Intended for use in `[initialize.before_init]`. | `vocab` | The pipeline to copy the vocab from. The vocab includes the lookups and vectors. Defaults to `None`. ~~Optional[str]~~ | | **CREATES** | A function that takes the current `nlp` object and modifies its `tokenizer` and `vocab`. ~~Callable[[Language], None]~~ | -### spacy.models_with_nvtx_range.v1 {#models_with_nvtx_range tag="registered function"} +### spacy.models_with_nvtx_range.v1 {id="models_with_nvtx_range",tag="registered function"} > #### Example config > @@ -910,7 +909,7 @@ backprop passes. | `backprop_color` | Color identifier for backpropagation passes. Defaults to `-1`. ~~int~~ | | **CREATES** | A function that takes the current `nlp` and wraps forward/backprop passes in NVTX ranges. ~~Callable[[Language], Language]~~ | -### spacy.models_and_pipes_with_nvtx_range.v1 {#models_and_pipes_with_nvtx_range tag="registered function" new="3.4"} +### spacy.models_and_pipes_with_nvtx_range.v1 {id="models_and_pipes_with_nvtx_range",tag="registered function",version="3.4"} > #### Example config > @@ -931,9 +930,9 @@ methods are wrapped: `pipe`, `predict`, `set_annotations`, `update`, `rehearse`, | `additional_pipe_functions` | Additional pipeline methods to wrap. Keys are pipeline names and values are lists of method identifiers. Defaults to `None`. ~~Optional[Dict[str, List[str]]]~~ | | **CREATES** | A function that takes the current `nlp` and wraps pipe models and methods in NVTX ranges. ~~Callable[[Language], Language]~~ | -## Training data and alignment {#gold source="spacy/training"} +## Training data and alignment {id="gold",source="spacy/training"} -### training.offsets_to_biluo_tags {#offsets_to_biluo_tags tag="function"} +### training.offsets_to_biluo_tags {id="offsets_to_biluo_tags",tag="function"} Encode labelled spans into per-token tags, using the [BILUO scheme](/usage/linguistic-features#accessing-ner) (Begin, In, Last, Unit, @@ -970,7 +969,7 @@ This method was previously available as `spacy.gold.biluo_tags_from_offsets`. | `missing` | The label used for missing values, e.g. if tokenization doesn't align with the entity offsets. Defaults to `"O"`. ~~str~~ | | **RETURNS** | A list of strings, describing the [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ | -### training.biluo_tags_to_offsets {#biluo_tags_to_offsets tag="function"} +### training.biluo_tags_to_offsets {id="biluo_tags_to_offsets",tag="function"} Encode per-token tags following the [BILUO scheme](/usage/linguistic-features#accessing-ner) into entity offsets. @@ -998,7 +997,7 @@ This method was previously available as `spacy.gold.offsets_from_biluo_tags`. | `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags with each tag describing one token. Each tag string will be of the form of either `""`, `"O"` or `"{action}-{label}"`, where action is one of `"B"`, `"I"`, `"L"`, `"U"`. ~~List[str]~~ | | **RETURNS** | A sequence of `(start, end, label)` triples. `start` and `end` will be character-offset integers denoting the slice into the original string. ~~List[Tuple[int, int, str]]~~ | -### training.biluo_tags_to_spans {#biluo_tags_to_spans tag="function" new="2.1"} +### training.biluo_tags_to_spans {id="biluo_tags_to_spans",tag="function",version="2.1"} Encode per-token tags following the [BILUO scheme](/usage/linguistic-features#accessing-ner) into @@ -1027,7 +1026,7 @@ This method was previously available as `spacy.gold.spans_from_biluo_tags`. | `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags with each tag describing one token. Each tag string will be of the form of either `""`, `"O"` or `"{action}-{label}"`, where action is one of `"B"`, `"I"`, `"L"`, `"U"`. ~~List[str]~~ | | **RETURNS** | A sequence of `Span` objects with added entity labels. ~~List[Span]~~ | -### training.biluo_to_iob {#biluo_to_iob tag="function"} +### training.biluo_to_iob {id="biluo_to_iob",tag="function"} Convert a sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags to [IOB](/usage/linguistic-features#accessing-ner) tags. This is useful if you want @@ -1048,7 +1047,7 @@ use the BILUO tags with a model that only supports IOB tags. | `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ | | **RETURNS** | A list of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ | -### training.iob_to_biluo {#iob_to_biluo tag="function"} +### training.iob_to_biluo {id="iob_to_biluo",tag="function"} Convert a sequence of [IOB](/usage/linguistic-features#accessing-ner) tags to [BILUO](/usage/linguistic-features#accessing-ner) tags. This is useful if you @@ -1075,7 +1074,55 @@ This method was previously available as `spacy.gold.iob_to_biluo`. | `tags` | A sequence of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ | | **RETURNS** | A list of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ | -## Utility functions {#util source="spacy/util.py"} +### training.biluo_to_iob {id="biluo_to_iob",tag="function"} + +Convert a sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags to +[IOB](/usage/linguistic-features#accessing-ner) tags. This is useful if you want +use the BILUO tags with a model that only supports IOB tags. + +> #### Example +> +> ```python +> from spacy.training import biluo_to_iob +> +> tags = ["O", "O", "B-LOC", "I-LOC", "L-LOC", "O"] +> iob_tags = biluo_to_iob(tags) +> assert iob_tags == ["O", "O", "B-LOC", "I-LOC", "I-LOC", "O"] +> ``` + +| Name | Description | +| ----------- | --------------------------------------------------------------------------------------- | +| `tags` | A sequence of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ | +| **RETURNS** | A list of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ | + +### training.iob_to_biluo {id="iob_to_biluo",tag="function"} + +Convert a sequence of [IOB](/usage/linguistic-features#accessing-ner) tags to +[BILUO](/usage/linguistic-features#accessing-ner) tags. This is useful if you +want use the IOB tags with a model that only supports BILUO tags. + + + +This method was previously available as `spacy.gold.iob_to_biluo`. + + + +> #### Example +> +> ```python +> from spacy.training import iob_to_biluo +> +> tags = ["O", "O", "B-LOC", "I-LOC", "O"] +> biluo_tags = iob_to_biluo(tags) +> assert biluo_tags == ["O", "O", "B-LOC", "L-LOC", "O"] +> ``` + +| Name | Description | +| ----------- | ------------------------------------------------------------------------------------- | +| `tags` | A sequence of [IOB](/usage/linguistic-features#accessing-ner) tags. ~~Iterable[str]~~ | +| **RETURNS** | A list of [BILUO](/usage/linguistic-features#accessing-ner) tags. ~~List[str]~~ | + +## Utility functions {id="util",source="spacy/util.py"} spaCy comes with a small collection of utility functions located in [`spacy/util.py`](%%GITHUB_SPACY/spacy/util.py). Because utility functions are @@ -1085,7 +1132,7 @@ use and we'll try to ensure backwards compatibility. However, we recommend having additional tests in place if your application depends on any of spaCy's utilities. -### util.get_lang_class {#util.get_lang_class tag="function"} +### util.get_lang_class {id="util.get_lang_class",tag="function"} Import and load a `Language` class. Allows lazy-loading [language data](/usage/linguistic-features#language-data) and importing @@ -1106,7 +1153,7 @@ custom language class, you can register it using the | `lang` | Two-letter language code, e.g. `"en"`. ~~str~~ | | **RETURNS** | The respective subclass. ~~Language~~ | -### util.lang_class_is_loaded {#util.lang_class_is_loaded tag="function" new="2.1"} +### util.lang_class_is_loaded {id="util.lang_class_is_loaded",tag="function",version="2.1"} Check whether a `Language` subclass is already loaded. `Language` subclasses are loaded lazily to avoid expensive setup code associated with the language data. @@ -1124,7 +1171,7 @@ loaded lazily to avoid expensive setup code associated with the language data. | `name` | Two-letter language code, e.g. `"en"`. ~~str~~ | | **RETURNS** | Whether the class has been loaded. ~~bool~~ | -### util.load_model {#util.load_model tag="function" new="2"} +### util.load_model {id="util.load_model",tag="function",version="2"} Load a pipeline from a package or data path. If called with a string name, spaCy will assume the pipeline is a Python package and import and call its `load()` @@ -1152,7 +1199,7 @@ and create a `Language` object. The model data will then be loaded in via | `config` 3 | Config overrides as nested dict or flat dict keyed by section values in dot notation, e.g. `"nlp.pipeline"`. ~~Union[Dict[str, Any], Config]~~ | | **RETURNS** | `Language` class with the loaded pipeline. ~~Language~~ | -### util.load_model_from_init_py {#util.load_model_from_init_py tag="function" new="2"} +### util.load_model_from_init_py {id="util.load_model_from_init_py",tag="function",version="2"} A helper function to use in the `load()` method of a pipeline package's [`__init__.py`](https://github.com/explosion/spacy-models/tree/master/template/model/xx_model_name/__init__.py). @@ -1177,7 +1224,7 @@ A helper function to use in the `load()` method of a pipeline package's | `config` 3 | Config overrides as nested dict or flat dict keyed by section values in dot notation, e.g. `"nlp.pipeline"`. ~~Union[Dict[str, Any], Config]~~ | | **RETURNS** | `Language` class with the loaded pipeline. ~~Language~~ | -### util.load_config {#util.load_config tag="function" new="3"} +### util.load_config {id="util.load_config",tag="function",version="3"} Load a pipeline's [`config.cfg`](/api/data-formats#config) from a file path. The config typically includes details about the components and how they're created, @@ -1197,7 +1244,7 @@ as well as all training settings and hyperparameters. | `interpolate` | Whether to interpolate the config and replace variables like `${paths.train}` with their values. Defaults to `False`. ~~bool~~ | | **RETURNS** | The pipeline's config. ~~Config~~ | -### util.load_meta {#util.load_meta tag="function" new="3"} +### util.load_meta {id="util.load_meta",tag="function",version="3"} Get a pipeline's [`meta.json`](/api/data-formats#meta) from a file path and validate its contents. The meta typically includes details about author, @@ -1214,7 +1261,7 @@ licensing, data sources and version. | `path` | Path to the pipeline's `meta.json`. ~~Union[str, Path]~~ | | **RETURNS** | The pipeline's meta data. ~~Dict[str, Any]~~ | -### util.get_installed_models {#util.get_installed_models tag="function" new="3"} +### util.get_installed_models {id="util.get_installed_models",tag="function",version="3"} List all pipeline packages installed in the current environment. This will include any spaCy pipeline that was packaged with @@ -1232,7 +1279,7 @@ object. | ----------- | ------------------------------------------------------------------------------------- | | **RETURNS** | The string names of the pipelines installed in the current environment. ~~List[str]~~ | -### util.is_package {#util.is_package tag="function"} +### util.is_package {id="util.is_package",tag="function"} Check if string maps to a package installed via pip. Mainly used to validate [pipeline packages](/usage/models). @@ -1249,7 +1296,7 @@ Check if string maps to a package installed via pip. Mainly used to validate | `name` | Name of package. ~~str~~ | | **RETURNS** | `True` if installed package, `False` if not. ~~bool~~ | -### util.get_package_path {#util.get_package_path tag="function" new="2"} +### util.get_package_path {id="util.get_package_path",tag="function",version="2"} Get path to an installed package. Mainly used to resolve the location of [pipeline packages](/usage/models). Currently imports the package to find its @@ -1267,7 +1314,7 @@ path. | `package_name` | Name of installed package. ~~str~~ | | **RETURNS** | Path to pipeline package directory. ~~Path~~ | -### util.is_in_jupyter {#util.is_in_jupyter tag="function" new="2"} +### util.is_in_jupyter {id="util.is_in_jupyter",tag="function",version="2"} Check if user is running spaCy from a [Jupyter](https://jupyter.org) notebook by detecting the IPython kernel. Mainly used for the @@ -1286,7 +1333,7 @@ detecting the IPython kernel. Mainly used for the | ----------- | ---------------------------------------------- | | **RETURNS** | `True` if in Jupyter, `False` if not. ~~bool~~ | -### util.compile_prefix_regex {#util.compile_prefix_regex tag="function"} +### util.compile_prefix_regex {id="util.compile_prefix_regex",tag="function"} Compile a sequence of prefix rules into a regex object. @@ -1303,7 +1350,7 @@ Compile a sequence of prefix rules into a regex object. | `entries` | The prefix rules, e.g. [`lang.punctuation.TOKENIZER_PREFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ | | **RETURNS** | The regex object to be used for [`Tokenizer.prefix_search`](/api/tokenizer#attributes). ~~Pattern~~ | -### util.compile_suffix_regex {#util.compile_suffix_regex tag="function"} +### util.compile_suffix_regex {id="util.compile_suffix_regex",tag="function"} Compile a sequence of suffix rules into a regex object. @@ -1320,7 +1367,7 @@ Compile a sequence of suffix rules into a regex object. | `entries` | The suffix rules, e.g. [`lang.punctuation.TOKENIZER_SUFFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ | | **RETURNS** | The regex object to be used for [`Tokenizer.suffix_search`](/api/tokenizer#attributes). ~~Pattern~~ | -### util.compile_infix_regex {#util.compile_infix_regex tag="function"} +### util.compile_infix_regex {id="util.compile_infix_regex",tag="function"} Compile a sequence of infix rules into a regex object. @@ -1337,7 +1384,7 @@ Compile a sequence of infix rules into a regex object. | `entries` | The infix rules, e.g. [`lang.punctuation.TOKENIZER_INFIXES`](%%GITHUB_SPACY/spacy/lang/punctuation.py). ~~Iterable[Union[str, Pattern]]~~ | | **RETURNS** | The regex object to be used for [`Tokenizer.infix_finditer`](/api/tokenizer#attributes). ~~Pattern~~ | -### util.minibatch {#util.minibatch tag="function" new="2"} +### util.minibatch {id="util.minibatch",tag="function",version="2"} Iterate over batches of items. `size` may be an iterator, so that batch-size can vary on each step. @@ -1356,7 +1403,7 @@ vary on each step. | `size` | The batch size(s). ~~Union[int, Sequence[int]]~~ | | **YIELDS** | The batches. | -### util.filter_spans {#util.filter_spans tag="function" new="2.1.4"} +### util.filter_spans {id="util.filter_spans",tag="function",version="2.1.4"} Filter a sequence of [`Span`](/api/span) objects and remove duplicates or overlaps. Useful for creating named entities (where one token can only be part @@ -1377,7 +1424,7 @@ of one entity) or when merging spans with | `spans` | The spans to filter. ~~Iterable[Span]~~ | | **RETURNS** | The filtered spans. ~~List[Span]~~ | -### util.get_words_and_spaces {#get_words_and_spaces tag="function" new="3"} +### util.get_words_and_spaces {id="get_words_and_spaces",tag="function",version="3"} Given a list of words and a text, reconstruct the original tokens and return a list of words and spaces that can be used to create a [`Doc`](/api/doc#init). diff --git a/website/docs/api/transformer.md b/website/docs/api/transformer.mdx similarity index 95% rename from website/docs/api/transformer.md rename to website/docs/api/transformer.mdx index e747ad383..ad8ecce54 100644 --- a/website/docs/api/transformer.md +++ b/website/docs/api/transformer.mdx @@ -3,7 +3,7 @@ title: Transformer teaser: Pipeline component for multi-task learning with transformer models tag: class source: github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py -new: 3 +version: 3 api_base_class: /api/pipe api_string_name: transformer --- @@ -44,7 +44,7 @@ package also adds the function registries [`@span_getters`](#span_getters) and functions. For more details, see the [usage documentation](/usage/embeddings-transformers). -## Assigned Attributes {#assigned-attributes} +## Assigned Attributes {id="assigned-attributes"} The component sets the following [custom extension attribute](/usage/processing-pipeline#custom-components-attributes): @@ -53,7 +53,7 @@ The component sets the following | ---------------- | ------------------------------------------------------------------------ | | `Doc._.trf_data` | Transformer tokens and outputs for the `Doc` object. ~~TransformerData~~ | -## Config and implementation {#config} +## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the @@ -81,7 +81,7 @@ on the transformer architectures and their arguments and hyperparameters. https://github.com/explosion/spacy-transformers/blob/master/spacy_transformers/pipeline_component.py ``` -## Transformer.\_\_init\_\_ {#init tag="method"} +## Transformer.\_\_init\_\_ {id="init",tag="method"} > #### Example > @@ -124,7 +124,7 @@ component using its string name and [`nlp.add_pipe`](/api/language#create_pipe). | `name` | String name of the component instance. Used to add entries to the `losses` during training. ~~str~~ | | `max_batch_items` | Maximum size of a padded batch. Defaults to `128*32`. ~~int~~ | -## Transformer.\_\_call\_\_ {#call tag="method"} +## Transformer.\_\_call\_\_ {id="call",tag="method"} Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the `nlp` object is called on a text @@ -147,7 +147,7 @@ to the [`predict`](/api/transformer#predict) and | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | -## Transformer.pipe {#pipe tag="method"} +## Transformer.pipe {id="pipe",tag="method"} Apply the pipe to a stream of documents. This usually happens under the hood when the `nlp` object is called on a text and all pipeline components are @@ -171,7 +171,7 @@ applied to the `Doc` in order. Both [`__call__`](/api/transformer#call) and | `batch_size` | The number of documents to buffer. Defaults to `128`. ~~int~~ | | **YIELDS** | The processed documents in order. ~~Doc~~ | -## Transformer.initialize {#initialize tag="method"} +## Transformer.initialize {id="initialize",tag="method"} Initialize the component for training and return an [`Optimizer`](https://thinc.ai/docs/api-optimizers). `get_examples` should be a @@ -196,7 +196,7 @@ by [`Language.initialize`](/api/language#initialize). | _keyword-only_ | | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | -## Transformer.predict {#predict tag="method"} +## Transformer.predict {id="predict",tag="method"} Apply the component's model to a batch of [`Doc`](/api/doc) objects without modifying them. @@ -213,7 +213,7 @@ modifying them. | `docs` | The documents to predict. ~~Iterable[Doc]~~ | | **RETURNS** | The model's prediction for each document. | -## Transformer.set_annotations {#set_annotations tag="method"} +## Transformer.set_annotations {id="set_annotations",tag="method"} Assign the extracted features to the `Doc` objects. By default, the [`TransformerData`](/api/transformer#transformerdata) object is written to the @@ -233,7 +233,7 @@ callback is then called, if provided. | `docs` | The documents to modify. ~~Iterable[Doc]~~ | | `scores` | The scores to set, produced by `Transformer.predict`. | -## Transformer.update {#update tag="method"} +## Transformer.update {id="update",tag="method"} Prepare for an update to the transformer. Like the [`Tok2Vec`](/api/tok2vec) component, the `Transformer` component is unusual in that it does not receive @@ -266,7 +266,7 @@ and call the optimizer, while the others simply increment the gradients. | `losses` | Optional record of the loss during training. Updated using the component name as the key. ~~Optional[Dict[str, float]]~~ | | **RETURNS** | The updated `losses` dictionary. ~~Dict[str, float]~~ | -## Transformer.create_optimizer {#create_optimizer tag="method"} +## Transformer.create_optimizer {id="create_optimizer",tag="method"} Create an optimizer for the pipeline component. @@ -281,7 +281,7 @@ Create an optimizer for the pipeline component. | ----------- | ---------------------------- | | **RETURNS** | The optimizer. ~~Optimizer~~ | -## Transformer.use_params {#use_params tag="method, contextmanager"} +## Transformer.use_params {id="use_params",tag="method, contextmanager"} Modify the pipe's model to use the given parameter values. At the end of the context, the original parameters are restored. @@ -298,7 +298,7 @@ context, the original parameters are restored. | -------- | -------------------------------------------------- | | `params` | The parameter values to use in the model. ~~dict~~ | -## Transformer.to_disk {#to_disk tag="method"} +## Transformer.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. @@ -315,7 +315,7 @@ Serialize the pipe to disk. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## Transformer.from_disk {#from_disk tag="method"} +## Transformer.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. @@ -333,7 +333,7 @@ Load the pipe from disk. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `Transformer` object. ~~Transformer~~ | -## Transformer.to_bytes {#to_bytes tag="method"} +## Transformer.to_bytes {id="to_bytes",tag="method"} > #### Example > @@ -350,7 +350,7 @@ Serialize the pipe to a bytestring. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `Transformer` object. ~~bytes~~ | -## Transformer.from_bytes {#from_bytes tag="method"} +## Transformer.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. @@ -369,7 +369,7 @@ Load the pipe from a bytestring. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `Transformer` object. ~~Transformer~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from @@ -387,7 +387,7 @@ serialization by passing in the string names via the `exclude` argument. | `cfg` | The config file. You usually don't want to exclude this. | | `model` | The binary model data. You usually don't want to exclude this. | -## TransformerData {#transformerdata tag="dataclass"} +## TransformerData {id="transformerdata",tag="dataclass"} Transformer tokens and outputs for one `Doc` object. The transformer models return tensors that refer to a whole padded batch of documents. These tensors @@ -405,7 +405,7 @@ by this class. Instances of this class are typically assigned to the | `align` | Alignment from the `Doc`'s tokenization to the wordpieces. This is a ragged array, where `align.lengths[i]` indicates the number of wordpiece tokens that token `i` aligns against. The actual indices are provided at `align[i].dataXd`. ~~Ragged~~ | | `width` | The width of the last hidden layer. ~~int~~ | -### TransformerData.empty {#transformerdata-emoty tag="classmethod"} +### TransformerData.empty {id="transformerdata-emoty",tag="classmethod"} Create an empty `TransformerData` container. @@ -425,7 +425,7 @@ model. -## FullTransformerBatch {#fulltransformerbatch tag="dataclass"} +## FullTransformerBatch {id="fulltransformerbatch",tag="dataclass"} Holds a batch of input and output objects for a transformer model. The data can then be split to a list of [`TransformerData`](/api/transformer#transformerdata) @@ -440,7 +440,7 @@ objects to associate the outputs to each [`Doc`](/api/doc) in the batch. | `align` | Alignment from the spaCy tokenization to the wordpieces. This is a ragged array, where `align.lengths[i]` indicates the number of wordpiece tokens that token `i` aligns against. The actual indices are provided at `align[i].dataXd`. ~~Ragged~~ | | `doc_data` | The outputs, split per `Doc` object. ~~List[TransformerData]~~ | -### FullTransformerBatch.unsplit_by_doc {#fulltransformerbatch-unsplit_by_doc tag="method"} +### FullTransformerBatch.unsplit_by_doc {id="fulltransformerbatch-unsplit_by_doc",tag="method"} Return a new `FullTransformerBatch` from a split batch of activations, using the current object's spans, tokens and alignment. This is used during the backward @@ -452,7 +452,7 @@ model. | `arrays` | The split batch of activations. ~~List[List[Floats3d]]~~ | | **RETURNS** | The transformer batch. ~~FullTransformerBatch~~ | -### FullTransformerBatch.split_by_doc {#fulltransformerbatch-split_by_doc tag="method"} +### FullTransformerBatch.split_by_doc {id="fulltransformerbatch-split_by_doc",tag="method"} Split a `TransformerData` object that represents a batch into a list with one `TransformerData` per `Doc`. @@ -468,7 +468,7 @@ In `spacy-transformers` v1.0, the model output is stored in -## Span getters {#span_getters source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/span_getters.py"} +## Span getters {id="span_getters",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/span_getters.py"} Span getters are functions that take a batch of [`Doc`](/api/doc) objects and return a lists of [`Span`](/api/span) objects for each doc to be processed by @@ -498,7 +498,7 @@ using the `@spacy.registry.span_getters` decorator. | `docs` | A batch of `Doc` objects. ~~Iterable[Doc]~~ | | **RETURNS** | The spans to process by the transformer. ~~List[List[Span]]~~ | -### doc_spans.v1 {#doc_spans tag="registered function"} +### doc_spans.v1 {id="doc_spans",tag="registered function"} > #### Example config > @@ -511,7 +511,7 @@ Create a span getter that uses the whole document as its spans. This is the best approach if your [`Doc`](/api/doc) objects already refer to relatively short texts. -### sent_spans.v1 {#sent_spans tag="registered function"} +### sent_spans.v1 {id="sent_spans",tag="registered function"} > #### Example config > @@ -531,7 +531,7 @@ To set sentence boundaries with the `sentencizer` during training, add a [`[training.annotating_components]`](/usage/training#annotating-components) to have it set the sentence boundaries before the `transformer` component runs. -### strided_spans.v1 {#strided_spans tag="registered function"} +### strided_spans.v1 {id="strided_spans",tag="registered function"} > #### Example config > @@ -553,7 +553,7 @@ right context. | `window` | The window size. ~~int~~ | | `stride` | The stride size. ~~int~~ | -## Annotation setters {#annotation_setters tag="registered functions" source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/annotation_setters.py"} +## Annotation setters {id="annotation_setters",tag="registered functions",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/annotation_setters.py"} Annotation setters are functions that take a batch of `Doc` objects and a [`FullTransformerBatch`](/api/transformer#fulltransformerbatch) and can set diff --git a/website/docs/api/vectors.md b/website/docs/api/vectors.mdx similarity index 94% rename from website/docs/api/vectors.md rename to website/docs/api/vectors.mdx index d4702b592..d6033c096 100644 --- a/website/docs/api/vectors.md +++ b/website/docs/api/vectors.mdx @@ -3,7 +3,7 @@ title: Vectors teaser: Store, save and load word vectors tag: class source: spacy/vectors.pyx -new: 2 +version: 2 --- Vectors data is kept in the `Vectors.data` attribute, which should be an @@ -25,7 +25,7 @@ As of spaCy v3.2, `Vectors` supports two types of vector tables: the sum of one or more rows as determined by the settings related to character ngrams and the hash table. -## Vectors.\_\_init\_\_ {#init tag="method"} +## Vectors.\_\_init\_\_ {id="init",tag="method"} Create a new vector store. With the default mode, you can set the vector values and keys directly on initialization, or supply a `shape` keyword argument to @@ -61,7 +61,7 @@ modified later. | `bow` 3.2 | The floret BOW string (default: `"<"`). ~~str~~ | | `eow` 3.2 | The floret EOW string (default: `">"`). ~~str~~ | -## Vectors.\_\_getitem\_\_ {#getitem tag="method"} +## Vectors.\_\_getitem\_\_ {id="getitem",tag="method"} Get a vector by key. If the key is not found in the table, a `KeyError` is raised. @@ -79,7 +79,7 @@ raised. | `key` | The key to get the vector for. ~~Union[int, str]~~ | | **RETURNS** | The vector for the key. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | -## Vectors.\_\_setitem\_\_ {#setitem tag="method"} +## Vectors.\_\_setitem\_\_ {id="setitem",tag="method"} Set a vector for the given key. Not supported for `floret` mode. @@ -96,7 +96,7 @@ Set a vector for the given key. Not supported for `floret` mode. | `key` | The key to set the vector for. ~~int~~ | | `vector` | The vector to set. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | -## Vectors.\_\_iter\_\_ {#iter tag="method"} +## Vectors.\_\_iter\_\_ {id="iter",tag="method"} Iterate over the keys in the table. In `floret` mode, the keys table is not used. @@ -112,7 +112,7 @@ used. | ---------- | --------------------------- | | **YIELDS** | A key in the table. ~~int~~ | -## Vectors.\_\_len\_\_ {#len tag="method"} +## Vectors.\_\_len\_\_ {id="len",tag="method"} Return the number of vectors in the table. @@ -127,7 +127,7 @@ Return the number of vectors in the table. | ----------- | ------------------------------------------- | | **RETURNS** | The number of vectors in the table. ~~int~~ | -## Vectors.\_\_contains\_\_ {#contains tag="method"} +## Vectors.\_\_contains\_\_ {id="contains",tag="method"} Check whether a key has been mapped to a vector entry in the table. In `floret` mode, returns `True` for all keys. @@ -145,7 +145,7 @@ mode, returns `True` for all keys. | `key` | The key to check. ~~int~~ | | **RETURNS** | Whether the key has a vector entry. ~~bool~~ | -## Vectors.add {#add tag="method"} +## Vectors.add {id="add",tag="method"} Add a key to the table, optionally setting a vector value as well. Keys can be mapped to an existing vector by setting `row`, or a new vector can be added. Not @@ -168,7 +168,7 @@ supported for `floret` mode. | `row` | An optional row number of a vector to map the key to. ~~int~~ | | **RETURNS** | The row the vector was added to. ~~int~~ | -## Vectors.resize {#resize tag="method"} +## Vectors.resize {id="resize",tag="method"} Resize the underlying vectors array. If `inplace=True`, the memory is reallocated. This may cause other references to the data to become invalid, so @@ -189,7 +189,7 @@ for `floret` mode. | `inplace` | Reallocate the memory. ~~bool~~ | | **RETURNS** | The removed items as a list of `(key, row)` tuples. ~~List[Tuple[int, int]]~~ | -## Vectors.keys {#keys tag="method"} +## Vectors.keys {id="keys",tag="method"} A sequence of the keys in the table. In `floret` mode, the keys table is not used. @@ -205,7 +205,7 @@ used. | ----------- | --------------------------- | | **RETURNS** | The keys. ~~Iterable[int]~~ | -## Vectors.values {#values tag="method"} +## Vectors.values {id="values",tag="method"} Iterate over vectors that have been assigned to at least one key. Note that some vectors may be unassigned, so the number of vectors returned may be less than @@ -222,7 +222,7 @@ the length of the vectors table. In `floret` mode, the keys table is not used. | ---------- | --------------------------------------------------------------- | | **YIELDS** | A vector in the table. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | -## Vectors.items {#items tag="method"} +## Vectors.items {id="items",tag="method"} Iterate over `(key, vector)` pairs, in order. In `floret` mode, the keys table is empty. @@ -238,7 +238,7 @@ is empty. | ---------- | ------------------------------------------------------------------------------------- | | **YIELDS** | `(key, vector)` pairs, in order. ~~Tuple[int, numpy.ndarray[ndim=1, dtype=float32]]~~ | -## Vectors.find {#find tag="method"} +## Vectors.find {id="find",tag="method"} Look up one or more keys by row, or vice versa. Not supported for `floret` mode. @@ -260,7 +260,7 @@ Look up one or more keys by row, or vice versa. Not supported for `floret` mode. | `rows` | Find the keys that point to the rows. Returns `numpy.ndarray`. ~~Iterable[int]~~ | | **RETURNS** | The requested key, keys, row or rows. ~~Union[int, numpy.ndarray[ndim=1, dtype=float32]]~~ | -## Vectors.shape {#shape tag="property"} +## Vectors.shape {id="shape",tag="property"} Get `(rows, dims)` tuples of number of rows and number of dimensions in the vector table. @@ -279,7 +279,7 @@ vector table. | ----------- | ------------------------------------------ | | **RETURNS** | A `(rows, dims)` pair. ~~Tuple[int, int]~~ | -## Vectors.size {#size tag="property"} +## Vectors.size {id="size",tag="property"} The vector size, i.e. `rows * dims`. @@ -294,7 +294,7 @@ The vector size, i.e. `rows * dims`. | ----------- | ------------------------ | | **RETURNS** | The vector size. ~~int~~ | -## Vectors.is_full {#is_full tag="property"} +## Vectors.is_full {id="is_full",tag="property"} Whether the vectors table is full and has no slots are available for new keys. If a table is full, it can be resized using @@ -313,7 +313,7 @@ full and cannot be resized. | ----------- | ------------------------------------------- | | **RETURNS** | Whether the vectors table is full. ~~bool~~ | -## Vectors.n_keys {#n_keys tag="property"} +## Vectors.n_keys {id="n_keys",tag="property"} Get the number of keys in the table. Note that this is the number of _all_ keys, not just unique vectors. If several keys are mapped to the same vectors, they @@ -331,7 +331,7 @@ will be counted individually. In `floret` mode, the keys table is not used. | ----------- | ----------------------------------------------------------------------------- | | **RETURNS** | The number of all keys in the table. Returns `-1` for floret vectors. ~~int~~ | -## Vectors.most_similar {#most_similar tag="method"} +## Vectors.most_similar {id="most_similar",tag="method"} For each of the given vectors, find the `n` most similar entries to it by cosine. Queries are by vector. Results are returned as a @@ -356,7 +356,7 @@ supported for `floret` mode. | `sort` | Whether to sort the entries returned by score. Defaults to `True`. ~~bool~~ | | **RETURNS** | The most similar entries as a `(keys, best_rows, scores)` tuple. ~~Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]~~ | -## Vectors.get_batch {#get_batch tag="method" new="3.2"} +## Vectors.get_batch {id="get_batch",tag="method",version="3.2"} Get the vectors for the provided keys efficiently as a batch. @@ -371,7 +371,7 @@ Get the vectors for the provided keys efficiently as a batch. | ------ | --------------------------------------- | | `keys` | The keys. ~~Iterable[Union[int, str]]~~ | -## Vectors.to_ops {#to_ops tag="method"} +## Vectors.to_ops {id="to_ops",tag="method"} Change the embedding matrix to use different Thinc ops. @@ -388,7 +388,7 @@ Change the embedding matrix to use different Thinc ops. | ----- | -------------------------------------------------------- | | `ops` | The Thinc ops to switch the embedding matrix to. ~~Ops~~ | -## Vectors.to_disk {#to_disk tag="method"} +## Vectors.to_disk {id="to_disk",tag="method"} Save the current state to a directory. @@ -403,7 +403,7 @@ Save the current state to a directory. | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | -## Vectors.from_disk {#from_disk tag="method"} +## Vectors.from_disk {id="from_disk",tag="method"} Loads state from a directory. Modifies the object in place and returns it. @@ -419,7 +419,7 @@ Loads state from a directory. Modifies the object in place and returns it. | `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | **RETURNS** | The modified `Vectors` object. ~~Vectors~~ | -## Vectors.to_bytes {#to_bytes tag="method"} +## Vectors.to_bytes {id="to_bytes",tag="method"} Serialize the current state to a binary string. @@ -433,7 +433,7 @@ Serialize the current state to a binary string. | ----------- | ------------------------------------------------------ | | **RETURNS** | The serialized form of the `Vectors` object. ~~bytes~~ | -## Vectors.from_bytes {#from_bytes tag="method"} +## Vectors.from_bytes {id="from_bytes",tag="method"} Load state from a binary string. @@ -451,7 +451,7 @@ Load state from a binary string. | `data` | The data to load from. ~~bytes~~ | | **RETURNS** | The `Vectors` object. ~~Vectors~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} | Name | Description | | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/website/docs/api/vocab.md b/website/docs/api/vocab.mdx similarity index 94% rename from website/docs/api/vocab.md rename to website/docs/api/vocab.mdx index 5e4de219a..131e4ce0a 100644 --- a/website/docs/api/vocab.md +++ b/website/docs/api/vocab.mdx @@ -10,7 +10,7 @@ The `Vocab` object provides a lookup table that allows you to access [`StringStore`](/api/stringstore). It also owns underlying C-data that is shared between `Doc` objects. -## Vocab.\_\_init\_\_ {#init tag="method"} +## Vocab.\_\_init\_\_ {id="init",tag="method"} Create the vocabulary. @@ -31,7 +31,7 @@ Create the vocabulary. | `writing_system` | A dictionary describing the language's writing system. Typically provided by [`Language.Defaults`](/api/language#defaults). ~~Dict[str, Any]~~ | | `get_noun_chunks` | A function that yields base noun phrases used for [`Doc.noun_chunks`](/api/doc#noun_chunks). ~~Optional[Callable[[Union[Doc, Span], Iterator[Tuple[int, int, int]]]]]~~ | -## Vocab.\_\_len\_\_ {#len tag="method"} +## Vocab.\_\_len\_\_ {id="len",tag="method"} Get the current number of lexemes in the vocabulary. @@ -46,7 +46,7 @@ Get the current number of lexemes in the vocabulary. | ----------- | ------------------------------------------------ | | **RETURNS** | The number of lexemes in the vocabulary. ~~int~~ | -## Vocab.\_\_getitem\_\_ {#getitem tag="method"} +## Vocab.\_\_getitem\_\_ {id="getitem",tag="method"} Retrieve a lexeme, given an int ID or a string. If a previously unseen string is given, a new lexeme is created and stored. @@ -63,7 +63,7 @@ given, a new lexeme is created and stored. | `id_or_string` | The hash value of a word, or its string. ~~Union[int, str]~~ | | **RETURNS** | The lexeme indicated by the given ID. ~~Lexeme~~ | -## Vocab.\_\_iter\_\_ {#iter tag="method"} +## Vocab.\_\_iter\_\_ {id="iter",tag="method"} Iterate over the lexemes in the vocabulary. @@ -77,7 +77,7 @@ Iterate over the lexemes in the vocabulary. | ---------- | -------------------------------------- | | **YIELDS** | An entry in the vocabulary. ~~Lexeme~~ | -## Vocab.\_\_contains\_\_ {#contains tag="method"} +## Vocab.\_\_contains\_\_ {id="contains",tag="method"} Check whether the string has an entry in the vocabulary. To get the ID for a given string, you need to look it up in @@ -97,7 +97,7 @@ given string, you need to look it up in | `string` | The ID string. ~~str~~ | | **RETURNS** | Whether the string has an entry in the vocabulary. ~~bool~~ | -## Vocab.add_flag {#add_flag tag="method"} +## Vocab.add_flag {id="add_flag",tag="method"} Set a new boolean flag to words in the vocabulary. The `flag_getter` function will be called over the words currently in the vocab, and then applied to new @@ -122,7 +122,7 @@ using `token.check_flag(flag_id)`. | `flag_id` | An integer between `1` and `63` (inclusive), specifying the bit at which the flag will be stored. If `-1`, the lowest available bit will be chosen. ~~int~~ | | **RETURNS** | The integer ID by which the flag value can be checked. ~~int~~ | -## Vocab.reset_vectors {#reset_vectors tag="method" new="2"} +## Vocab.reset_vectors {id="reset_vectors",tag="method",version="2"} Drop the current vector table. Because all vectors must be the same width, you have to call this to change the size of the vectors. Only one of the `width` and @@ -140,7 +140,7 @@ have to call this to change the size of the vectors. Only one of the `width` and | `width` | The new width. ~~int~~ | | `shape` | The new shape. ~~int~~ | -## Vocab.prune_vectors {#prune_vectors tag="method" new="2"} +## Vocab.prune_vectors {id="prune_vectors",tag="method",version="2"} Reduce the current vector table to `nr_row` unique entries. Words mapped to the discarded vectors will be remapped to the closest vector among those remaining. @@ -165,7 +165,7 @@ cosines are calculated in minibatches to reduce memory usage. | `batch_size` | Batch of vectors for calculating the similarities. Larger batch sizes might be faster, while temporarily requiring more memory. ~~int~~ | | **RETURNS** | A dictionary keyed by removed words mapped to `(string, score)` tuples, where `string` is the entry the removed word was mapped to, and `score` the similarity score between the two words. ~~Dict[str, Tuple[str, float]]~~ | -## Vocab.deduplicate_vectors {#deduplicate_vectors tag="method" new="3.3"} +## Vocab.deduplicate_vectors {id="deduplicate_vectors",tag="method",version="3.3"} > #### Example > @@ -176,7 +176,7 @@ cosines are calculated in minibatches to reduce memory usage. Remove any duplicate rows from the current vector table, maintaining the mappings for all words in the vectors. -## Vocab.get_vector {#get_vector tag="method" new="2"} +## Vocab.get_vector {id="get_vector",tag="method",version="2"} Retrieve a vector for a word in the vocabulary. Words can be looked up by string or hash value. If the current vectors do not contain an entry for the word, a @@ -194,7 +194,7 @@ or hash value. If the current vectors do not contain an entry for the word, a | `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ | | **RETURNS** | A word vector. Size and shape are determined by the `Vocab.vectors` instance. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | -## Vocab.set_vector {#set_vector tag="method" new="2"} +## Vocab.set_vector {id="set_vector",tag="method",version="2"} Set a vector for a word in the vocabulary. Words can be referenced by string or hash value. @@ -210,7 +210,7 @@ hash value. | `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ | | `vector` | The vector to set. ~~numpy.ndarray[ndim=1, dtype=float32]~~ | -## Vocab.has_vector {#has_vector tag="method" new="2"} +## Vocab.has_vector {id="has_vector",tag="method",version="2"} Check whether a word has a vector. Returns `False` if no vectors are loaded. Words can be looked up by string or hash value. @@ -227,7 +227,7 @@ Words can be looked up by string or hash value. | `orth` | The hash value of a word, or its unicode string. ~~Union[int, str]~~ | | **RETURNS** | Whether the word has a vector. ~~bool~~ | -## Vocab.to_disk {#to_disk tag="method" new="2"} +## Vocab.to_disk {id="to_disk",tag="method",version="2"} Save the current state to a directory. @@ -243,7 +243,7 @@ Save the current state to a directory. | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | -## Vocab.from_disk {#from_disk tag="method" new="2"} +## Vocab.from_disk {id="from_disk",tag="method",version="2"} Loads state from a directory. Modifies the object in place and returns it. @@ -261,7 +261,7 @@ Loads state from a directory. Modifies the object in place and returns it. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `Vocab` object. ~~Vocab~~ | -## Vocab.to_bytes {#to_bytes tag="method"} +## Vocab.to_bytes {id="to_bytes",tag="method"} Serialize the current state to a binary string. @@ -277,7 +277,7 @@ Serialize the current state to a binary string. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `Vocab` object. ~~Vocab~~ | -## Vocab.from_bytes {#from_bytes tag="method"} +## Vocab.from_bytes {id="from_bytes",tag="method"} Load state from a binary string. @@ -297,7 +297,7 @@ Load state from a binary string. | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `Vocab` object. ~~Vocab~~ | -## Attributes {#attributes} +## Attributes {id="attributes"} > #### Example > @@ -317,7 +317,7 @@ Load state from a binary string. | `writing_system` | A dict with information about the language's writing system. ~~Dict[str, Any]~~ | | `get_noun_chunks` 3.0 | A function that yields base noun phrases used for [`Doc.noun_chunks`](/api/doc#noun_chunks). ~~Optional[Callable[[Union[Doc, Span], Iterator[Tuple[int, int, int]]]]]~~ | -## Serialization fields {#serialization-fields} +## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from diff --git a/website/docs/images/displacy-dep-founded.html b/website/docs/images/displacy-dep-founded.html deleted file mode 100644 index e22984ee1..000000000 --- a/website/docs/images/displacy-dep-founded.html +++ /dev/null @@ -1,58 +0,0 @@ - - - Smith - - - - - founded - - - - - a - - - - - healthcare - - - - - company - - - - - - - nsubj - - - - - - - - det - - - - - - - - compound - - - - - - - - dobj - - - - diff --git a/website/docs/images/displacy-ent-custom.html b/website/docs/images/displacy-ent-custom.html deleted file mode 100644 index 709c6f631..000000000 --- a/website/docs/images/displacy-ent-custom.html +++ /dev/null @@ -1,33 +0,0 @@ -

But - Google - ORGis starting from behind. The company made a late push into hardware, and - Apple - ORG’s Siri, available on iPhones, and - Amazon - ORG’s Alexa software, which runs on its Echo and Dot devices, have clear leads in consumer - adoption.
diff --git a/website/docs/images/displacy-ent-snek.html b/website/docs/images/displacy-ent-snek.html deleted file mode 100644 index c8b416d8d..000000000 --- a/website/docs/images/displacy-ent-snek.html +++ /dev/null @@ -1,26 +0,0 @@ -
- 🌱🌿 - 🐍 - SNEK - ____ 🌳🌲 ____ - 👨‍🌾 - HUMAN - 🏘️ -
diff --git a/website/docs/images/displacy-ent1.html b/website/docs/images/displacy-ent1.html deleted file mode 100644 index 708df8093..000000000 --- a/website/docs/images/displacy-ent1.html +++ /dev/null @@ -1,37 +0,0 @@ -
- - Apple - ORG - - is looking at buying - - U.K. - GPE - - startup for - - $1 billion - MONEY - -
diff --git a/website/docs/images/displacy-ent2.html b/website/docs/images/displacy-ent2.html deleted file mode 100644 index 5e1833ca0..000000000 --- a/website/docs/images/displacy-ent2.html +++ /dev/null @@ -1,39 +0,0 @@ -
- When - - Sebastian Thrun - PERSON - - started working on self-driving cars at - - Google - ORG - - in - - 2007 - DATE - - , few people outside of the company took him seriously. -
diff --git a/website/docs/images/displacy-long2.html b/website/docs/images/displacy-long2.html deleted file mode 100644 index abe18c42a..000000000 --- a/website/docs/images/displacy-long2.html +++ /dev/null @@ -1,84 +0,0 @@ - - - Autonomous - ADJ - - - - cars - NOUN - - - - shift - VERB - - - - insurance - NOUN - - - - liability - NOUN - - - - toward - ADP - - - - manufacturers - NOUN - - - - - - amod - - - - - - - - nsubj - - - - - - - - compound - - - - - - - - dobj - - - - - - - - prep - - - - - - - - pobj - - - - diff --git a/website/docs/images/displacy-span-custom.html b/website/docs/images/displacy-span-custom.html deleted file mode 100644 index 97dd3b140..000000000 --- a/website/docs/images/displacy-span-custom.html +++ /dev/null @@ -1,31 +0,0 @@ -
- Welcome to the - - Bank - - - - - BANK - - - - - of - - - - - China - - - - - . -
\ No newline at end of file diff --git a/website/docs/images/displacy-span.html b/website/docs/images/displacy-span.html deleted file mode 100644 index 9bbc6403c..000000000 --- a/website/docs/images/displacy-span.html +++ /dev/null @@ -1,41 +0,0 @@ -
- Welcome to the - - Bank - - - - - ORG - - - - - of - - - - - - China - - - - - - - GPE - - - - . -
\ No newline at end of file diff --git a/website/docs/index.md b/website/docs/index.md deleted file mode 100644 index 48e487d08..000000000 --- a/website/docs/index.md +++ /dev/null @@ -1,6 +0,0 @@ ---- ---- - -import Landing from 'widgets/landing.js' - - diff --git a/website/docs/models/index.md b/website/docs/models/index.mdx similarity index 95% rename from website/docs/models/index.md rename to website/docs/models/index.mdx index 203555651..371e4460f 100644 --- a/website/docs/models/index.md +++ b/website/docs/models/index.mdx @@ -7,7 +7,7 @@ menu: - ['Pipeline Design', 'design'] --- - +{/* TODO: include interactive demo */} ### Quickstart {hidden="true"} @@ -16,11 +16,9 @@ menu: > For more details on how to use trained pipelines with spaCy, see the > [usage guide](/usage/models). -import QuickstartModels from 'widgets/quickstart-models.js' - -## Package naming conventions {#conventions} +## Package naming conventions {id="conventions"} In general, spaCy expects all pipeline packages to follow the naming convention of `[lang]\_[name]`. For spaCy's pipelines, we also chose to divide the name @@ -45,7 +43,7 @@ For example, [`en_core_web_sm`](/models/en#en_core_web_sm) is a small English pipeline trained on written web text (blogs, news, comments), that includes vocabulary, syntax and entities. -### Package versioning {#model-versioning} +### Package versioning {id="model-versioning"} Additionally, the pipeline package versioning reflects both the compatibility with spaCy, as well as the model version. A package version `a.b.c` translates @@ -62,7 +60,7 @@ For a detailed compatibility overview, see the This is also the source of spaCy's internal compatibility check, performed when you run the [`download`](/api/cli#download) command. -## Trained pipeline design {#design} +## Trained pipeline design {id="design"} The spaCy v3 trained pipelines are designed to be efficient and configurable. For example, multiple components can share a common "token-to-vector" model and @@ -89,9 +87,9 @@ Main changes from spaCy v2 models: - The lemmatizer tables and processing move from the vocab and tagger to a separate `lemmatizer` component. -### CNN/CPU pipeline design {#design-cnn} +### CNN/CPU pipeline design {id="design-cnn"} -![Components and their dependencies in the CNN pipelines](../images/pipeline-design.svg) +![Components and their dependencies in the CNN pipelines](/images/pipeline-design.svg) In the `sm`/`md`/`lg` models: @@ -132,13 +130,13 @@ vector keys for default vectors. - [`Vectors.most_similar`](/api/vectors#most_similar) is not supported because there's no fixed list of vectors to compare your vectors to. -### Transformer pipeline design {#design-trf} +### Transformer pipeline design {id="design-trf"} In the transformer (`trf`) models, the `tagger`, `parser` and `ner` (if present) all listen to the `transformer` component. The `attribute_ruler` and `lemmatizer` have the same configuration as in the CNN models. -### Modifying the default pipeline {#design-modify} +### Modifying the default pipeline {id="design-modify"} For faster processing, you may only want to run a subset of the components in a trained pipeline. The `disable` and `exclude` arguments to @@ -189,8 +187,8 @@ than the rule-based `sentencizer`. #### Switch from trainable lemmatizer to default lemmatizer -Since v3.3, a number of pipelines use a trainable lemmatizer. You can check whether -the lemmatizer is trainable: +Since v3.3, a number of pipelines use a trainable lemmatizer. You can check +whether the lemmatizer is trainable: ```python nlp = spacy.load("de_core_web_sm") diff --git a/website/docs/styleguide.md b/website/docs/styleguide.mdx similarity index 86% rename from website/docs/styleguide.md rename to website/docs/styleguide.mdx index 47bca1ed4..276137aab 100644 --- a/website/docs/styleguide.md +++ b/website/docs/styleguide.mdx @@ -42,9 +42,7 @@ enough, JSX components can be used. > For more details on editing the site locally, see the installation > instructions and markdown reference below. -## Logo {#logo source="website/src/images/logo.svg"} - -import { Logos } from 'widgets/styleguide' +## Logo {id="logo",source="website/src/images/logo.svg"} If you would like to use the spaCy logo on your site, please get in touch and ask us first. However, if you want to show support and tell others that your @@ -53,9 +51,7 @@ project is using spaCy, you can grab one of our -## Colors {#colors} - -import { Colors, Patterns } from 'widgets/styleguide' +## Colors {id="colors"} @@ -63,17 +59,16 @@ import { Colors, Patterns } from 'widgets/styleguide' -## Typography {#typography} - -import { H1, H2, H3, H4, H5, Label, InlineList, Comment } from -'components/typography' +## Typography {id="typography"} > #### Markdown > -> ```markdown_ +> ```markdown > ## Headline 2 -> ## Headline 2 {#some_id} -> ## Headline 2 {#some_id tag="method"} +> +> ## Headline 2 {id="some_id"} +> +> ## Headline 2 {id="some_id" tag="method"} > ``` > > #### JSX @@ -101,12 +96,11 @@ in the sidebar menu.
-

Headline 1

-

Headline 2

-

Headline 3

-

Headline 4

-
Headline 5
- +

Headline 2

+

Headline 3

+

Headline 4

+
Headline 5
+
--- @@ -116,16 +110,16 @@ example, to add a tag for the documented type or mark features that have been introduced in a specific version or require statistical models to be loaded. Tags are also available as standalone `` components. -| Argument | Example | Result | -| -------- | -------------------------- | ----------------------------------------- | -| `tag` | `{tag="method"}` | method | -| `new` | `{new="3"}` | 3 | -| `model` | `{model="tagger, parser"}` | tagger, parser | -| `hidden` | `{hidden="true"}` | | +| Argument | Example | Result | +| --------- | -------------------------- | ----------------------------------------- | +| `tag` | `{tag="method"}` | method | +| `version` | `{version="3"}` | 3 | +| `model` | `{model="tagger, parser"}` | tagger, parser | +| `hidden` | `{hidden="true"}` | | -## Elements {#elements} +## Elements {id="elements"} -### Links {#links} +### Links {id="links"} > #### Markdown > @@ -147,9 +141,7 @@ Special link styles are used depending on the link URL. - [I am a link to a model](/models/en#en_core_web_sm) - [I am a link to GitHub](https://github.com/explosion/spaCy) -### Abbreviations {#abbr} - -import { Abbr } from 'components/typography' +### Abbreviations {id="abbr"} > #### JSX > @@ -161,13 +153,11 @@ Some text with an abbreviation. On small screens, I collapse and the explanation text is displayed next to the abbreviation. -### Tags {#tags} - -import Tag from 'components/tag' +### Tags {id="tags"} > ```jsx > method -> 4 +> 4 > tagger, parser > ``` @@ -180,16 +170,13 @@ new anymore. Setting `variant="model"` takes a description of model capabilities and can be used to mark features that require a respective model to be installed. - +

+ method + 4 + tagger, parser +

-method 4 tagger, -parser - - - -### Buttons {#buttons} - -import Button from 'components/button' +### Buttons {id="buttons"} > ```jsx > @@ -200,21 +187,29 @@ Link buttons come in two variants, `primary` and `secondary` and two sizes, with an optional `large` size modifier. Since they're mostly used as enhanced links, the buttons are implemented as styled links instead of native button elements. - - +

+ -
+{' '} - - + +

+ +

+ + +{' '} + + +

## Components -### Table {#table} +### Table {id="table"} > #### Markdown > -> ```markdown_ +> ```markdown > | Header 1 | Header 2 | > | -------- | -------- | > | Column 1 | Column 2 | @@ -248,7 +243,7 @@ be italicized: > #### Markdown > -> ```markdown_ +> ```markdown > | Header 1 | Header 2 | Header 3 | > | -------- | -------- | -------- | > | Column 1 | Column 2 | Column 3 | @@ -262,11 +257,11 @@ be italicized: | _Hello_ | | | | Column 1 | Column 2 | Column 3 | -### Type Annotations {#type-annotations} +### Type Annotations {id="type-annotations"} > #### Markdown > -> ```markdown_ +> ```markdown > ~~Model[List[Doc], Floats2d]~~ > ``` > @@ -295,9 +290,9 @@ always be the **last element** in the row. > #### Markdown > -> ```markdown_ -> | Header 1 | Header 2 | -> | -------- | ----------------------- | +> ```markdown +> | Header 1 | Header 2 | +> | -------- | ---------------------- | > | Column 1 | Column 2 ~~List[Doc]~~ | > ``` @@ -307,11 +302,11 @@ always be the **last element** in the row. | `model` | The Thinc [`Model`](https://thinc.ai/docs/api-model) wrapping the transformer. ~~Model[List[Doc], FullTransformerBatch]~~ | | `set_extra_annotations` | Function that takes a batch of `Doc` objects and transformer outputs and can set additional annotations on the `Doc`. ~~Callable[[List[Doc], FullTransformerBatch], None]~~ | -### List {#list} +### List {id="list"} > #### Markdown > -> ```markdown_ +> ```markdown > 1. One > 2. Two > ``` @@ -338,12 +333,13 @@ automatically. 3. Lorem ipsum dolor 4. consectetur adipiscing elit -### Aside {#aside} +### Aside {id="aside"} > #### Markdown > -> ```markdown_ +> ```markdown > > #### Aside title +> > > > This is aside text. > ``` > @@ -363,11 +359,11 @@ To make them easier to use in Markdown, paragraphs formatted as blockquotes will turn into asides by default. Level 4 headlines (with a leading `####`) will become aside titles. -### Code Block {#code-block} +### Code Block {id="code-block"} > #### Markdown > -> ````markdown_ +> ````markdown > ```python > ### This is a title > import spacy @@ -388,8 +384,7 @@ to raw text with no highlighting. An optional label can be added as the first line with the prefix `####` (Python-like) and `///` (JavaScript-like). the indented block as plain text and preserve whitespace. -```python -### Using spaCy +```python {title="Using spaCy"} import spacy nlp = spacy.load("en_core_web_sm") doc = nlp("This is a sentence.") @@ -403,7 +398,7 @@ adding `{highlight="..."}` to the headline. Acceptable ranges are spans like > #### Markdown > -> ````markdown_ +> ````markdown > ```python > ### This is a title {highlight="1-2"} > import spacy @@ -411,8 +406,7 @@ adding `{highlight="..."}` to the headline. Acceptable ranges are spans like > ``` > ```` -```python -### Using the matcher {highlight="5-7"} +```python {title="Using the matcher",highlight="5-7"} import spacy from spacy.matcher import Matcher @@ -431,7 +425,7 @@ interactive widget defaults to a regular code block. > #### Markdown > -> ````markdown_ +> ````markdown > ```python > ### {executable="true"} > import spacy @@ -439,8 +433,7 @@ interactive widget defaults to a regular code block. > ``` > ```` -```python -### {executable="true"} +```python {executable="true"} import spacy nlp = spacy.load("en_core_web_sm") doc = nlp("This is a sentence.") @@ -454,7 +447,7 @@ original file is shown at the top of the widget. > #### Markdown > -> ````markdown_ +> ````markdown > ```python > https://github.com/... > ``` @@ -470,9 +463,7 @@ original file is shown at the top of the widget. https://github.com/explosion/spaCy/tree/master/spacy/language.py ``` -### Infobox {#infobox} - -import Infobox from 'components/infobox' +### Infobox {id="infobox"} > #### JSX > @@ -508,9 +499,7 @@ blocks. -### Accordion {#accordion} - -import Accordion from 'components/accordion' +### Accordion {id="accordion"} > #### JSX > @@ -537,9 +526,9 @@ sit amet dignissim justo congue. -## Markdown reference {#markdown} +## Markdown reference {id="markdown"} -All page content and page meta lives in the `.md` files in the `/docs` +All page content and page meta lives in the `.mdx` files in the `/docs` directory. The frontmatter block at the top of each file defines the page title and other settings like the sidebar menu. @@ -548,7 +537,7 @@ and other settings like the sidebar menu. title: Page title --- -## Headline starting a section {#some_id} +## Headline starting a section {id="some_id"} This is a regular paragraph with a [link](https://spacy.io) and **bold text**. @@ -562,8 +551,7 @@ This is a regular paragraph with a [link](https://spacy.io) and **bold text**. | -------- | -------- | | Column 1 | Column 2 | -```python -### Code block title {highlight="2-3"} +```python {title="Code block title",highlight="2-3"} import spacy nlp = spacy.load("en_core_web_sm") doc = nlp("Hello world") @@ -585,7 +573,7 @@ In addition to the native markdown elements, you can use the components [abbr]: https://spacy.io/styleguide#abbr [tag]: https://spacy.io/styleguide#tag -## Editorial {#editorial} +## Editorial {id="editorial"} - "spaCy" should always be spelled with a lowercase "s" and a capital "C", unless it specifically refers to the Python package or Python import `spacy` @@ -609,21 +597,16 @@ In addition to the native markdown elements, you can use the components - ❌ The [`Span`](/api/span) and [`Token`](/api/token) objects are views of a [`Doc`](/api/doc). [`Span.as_doc`](/api/span#as_doc) creates a [`Doc`](/api/doc) object from a [`Span`](/api/span). - -* Other things we format as code are: references to trained pipeline packages +- Other things we format as code are: references to trained pipeline packages like `en_core_web_sm` or file names like `code.py` or `meta.json`. - - ✅ After training, the `config.cfg` is saved to disk. - -* [Type annotations](#type-annotations) are a special type of code formatting, +- [Type annotations](#type-annotations) are a special type of code formatting, expressed by wrapping the text in `~~` instead of backticks. The result looks like this: ~~List[Doc]~~. All references to known types will be linked automatically. - - ✅ The model has the input type ~~List[Doc]~~ and it outputs a ~~List[Array2d]~~. - -* We try to keep links meaningful but short. +- We try to keep links meaningful but short. - ✅ For details, see the usage guide on [training with custom code](/usage/training#custom-code). - ❌ For details, see diff --git a/website/docs/usage/101/_architecture.md b/website/docs/usage/101/_architecture.mdx similarity index 96% rename from website/docs/usage/101/_architecture.md rename to website/docs/usage/101/_architecture.mdx index 4ebca2756..5727c6921 100644 --- a/website/docs/usage/101/_architecture.md +++ b/website/docs/usage/101/_architecture.mdx @@ -14,9 +14,9 @@ of the pipeline. The `Language` object coordinates these components. It takes raw text and sends it through the pipeline, returning an **annotated document**. It also orchestrates training and serialization. -![Library architecture](../../images/architecture.svg) +![Library architecture {{w:1080, h:1254}}](/images/architecture.svg) -### Container objects {#architecture-containers} +### Container objects {id="architecture-containers"} | Name | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -29,7 +29,7 @@ It also orchestrates training and serialization. | [`SpanGroup`](/api/spangroup) | A named collection of spans belonging to a `Doc`. | | [`Token`](/api/token) | An individual token — i.e. a word, punctuation symbol, whitespace, etc. | -### Processing pipeline {#architecture-pipeline} +### Processing pipeline {id="architecture-pipeline"} The processing pipeline consists of one or more **pipeline components** that are called on the `Doc` in order. The tokenizer runs before the components. Pipeline @@ -39,7 +39,7 @@ rule-based modifications to the `Doc`. spaCy provides a range of built-in components for different language processing tasks and also allows adding [custom components](/usage/processing-pipelines#custom-components). -![The processing pipeline](../../images/pipeline.svg) +![The processing pipeline](/images/pipeline.svg) | Name | Description | | ----------------------------------------------- | ------------------------------------------------------------------------------------------- | @@ -61,7 +61,7 @@ components for different language processing tasks and also allows adding | [`Transformer`](/api/transformer) | Use a transformer model and set its outputs. | | [Other functions](/api/pipeline-functions) | Automatically apply something to the `Doc`, e.g. to merge spans of tokens. | -### Matchers {#architecture-matchers} +### Matchers {id="architecture-matchers"} Matchers help you find and extract information from [`Doc`](/api/doc) objects based on match patterns describing the sequences you're looking for. A matcher @@ -73,7 +73,7 @@ operates on a `Doc` and gives you access to the matched tokens **in context**. | [`Matcher`](/api/matcher) | Match sequences of tokens, based on pattern rules, similar to regular expressions. | | [`PhraseMatcher`](/api/phrasematcher) | Match sequences of tokens based on phrases. | -### Other classes {#architecture-other} +### Other classes {id="architecture-other"} | Name | Description | | ------------------------------------------------ | -------------------------------------------------------------------------------------------------- | diff --git a/website/docs/usage/101/_language-data.md b/website/docs/usage/101/_language-data.mdx similarity index 100% rename from website/docs/usage/101/_language-data.md rename to website/docs/usage/101/_language-data.mdx diff --git a/website/docs/usage/101/_named-entities.md b/website/docs/usage/101/_named-entities.mdx similarity index 75% rename from website/docs/usage/101/_named-entities.md rename to website/docs/usage/101/_named-entities.mdx index 2abc45cbd..9ae4134d8 100644 --- a/website/docs/usage/101/_named-entities.md +++ b/website/docs/usage/101/_named-entities.mdx @@ -1,14 +1,13 @@ A named entity is a "real-world object" that's assigned a name – for example, a person, a country, a product or a book title. spaCy can **recognize various -types of named entities in a document, by asking the model for a -prediction**. Because models are statistical and strongly depend on the -examples they were trained on, this doesn't always work _perfectly_ and might -need some tuning later, depending on your use case. +types of named entities in a document, by asking the model for a prediction**. +Because models are statistical and strongly depend on the examples they were +trained on, this doesn't always work _perfectly_ and might need some tuning +later, depending on your use case. Named entities are available as the `ents` property of a `Doc`: -```python -### {executable="true"} +```python {executable="true"} import spacy nlp = spacy.load("en_core_web_sm") @@ -32,7 +31,8 @@ for ent in doc.ents: Using spaCy's built-in [displaCy visualizer](/usage/visualizers), here's what our example sentence and its named entities look like: -import DisplaCyEntHtml from 'images/displacy-ent1.html'; import { Iframe } from -'components/embed' - -