* enable fuzzy matching * add fuzzy param to EntityMatcher * include rapidfuzz_capi not yet used * fix type * add FUZZY predicate * add fuzzy attribute list * fix type properly * tidying * remove unnecessary dependency * handle fuzzy sets * simplify fuzzy sets * case fix * switch to FUZZYn predicates use Levenshtein distance. remove fuzzy param. remove rapidfuzz_capi. * revert changes added for fuzzy param * switch to polyleven (Python package) * enable fuzzy matching * add fuzzy param to EntityMatcher * include rapidfuzz_capi not yet used * fix type * add FUZZY predicate * add fuzzy attribute list * fix type properly * tidying * remove unnecessary dependency * handle fuzzy sets * simplify fuzzy sets * case fix * switch to FUZZYn predicates use Levenshtein distance. remove fuzzy param. remove rapidfuzz_capi. * revert changes added for fuzzy param * switch to polyleven (Python package) * fuzzy match only on oov tokens * remove polyleven * exclude whitespace tokens * don't allow more edits than characters * fix min distance * reinstate FUZZY operator with length-based distance function * handle sets inside regex operator * remove is_oov check * attempt build fix no mypy failure locally * re-attempt build fix * don't overwrite fuzzy param value * move fuzzy_match to its own Python module to allow patching * move fuzzy_match back inside Matcher simplify logic and add tests * Format tests * Parametrize fuzzyn tests * Parametrize and merge fuzzy+set tests * Format * Move fuzzy_match to a standalone method * Change regex kwarg type to bool * Add types for fuzzy_match - Refactor variable names - Add test for symmetrical behavior * Parametrize fuzzyn+set tests * Minor refactoring for fuzz/fuzzy * Make fuzzy_match a Matcher kwarg * Update type for _default_fuzzy_match * don't overwrite function param * Rename to fuzzy_compare * Update fuzzy_compare default argument declarations * allow fuzzy_compare override from EntityRuler * define new Matcher keyword arg * fix type definition * Implement fuzzy_compare config option for EntityRuler and SpanRuler * Rename _default_fuzzy_compare to fuzzy_compare, remove from reexported objects * Use simpler fuzzy_compare algorithm * Update types * Increase minimum to 2 in fuzzy_compare to allow one transposition * Fix predicate keys and matching for SetPredicate with FUZZY and REGEX * Add FUZZY6..9 * Add initial docs * Increase default fuzzy to rounded 30% of pattern length * Update docs for fuzzy_compare in components * Update EntityRuler and SpanRuler API docs * Rename EntityRuler and SpanRuler setting to matcher_fuzzy_compare To having naming similar to `phrase_matcher_attr`, rename `fuzzy_compare` setting for `EntityRuler` and `SpanRuler` to `matcher_fuzzy_compare. Organize next to `phrase_matcher_attr` in docs. * Fix schema aliases Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> * Fix typo Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> * Add FUZZY6-9 operators and update tests * Parameterize test over greedy Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com> * Fix type for fuzzy_compare to remove Optional * Rename to spacy.levenshtein_compare.v1, move to spacy.matcher.levenshtein * Update docs following levenshtein_compare renaming Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com> Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
18 KiB
title | tag | source | new | teaser | api_string_name | api_trainable |
---|---|---|---|---|---|---|
EntityRuler | class | spacy/pipeline/entityruler.py | 2.1 | Pipeline component for rule-based named entity recognition | entity_ruler | false |
The entity ruler lets you add spans to the Doc.ents
using
token-based rules or exact phrase matches. It can be combined with the
statistical EntityRecognizer
to boost accuracy, or
used on its own to implement a purely rule-based entity recognition system. For
usage examples, see the docs on
rule-based entity recognition.
Assigned Attributes
This component assigns predictions basically the same way as the
EntityRecognizer
.
Predictions can be accessed under Doc.ents
as a tuple. Each label will also be
reflected in each underlying token, where it is saved in the Token.ent_type
and Token.ent_iob
fields. Note that by definition each token can only have one
label.
When setting Doc.ents
to create training data, all the spans must be valid and
non-overlapping, or an error will be thrown.
Location | Value |
---|---|
Doc.ents |
The annotated spans. |
Token.ent_iob |
An enum encoding of the IOB part of the named entity tag. |
Token.ent_iob_ |
The IOB part of the named entity tag. |
Token.ent_type |
The label part of the named entity tag (hash). |
Token.ent_type_ |
The label part of the named entity tag. |
Config and implementation
The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
config
argument on nlp.add_pipe
or in your
config.cfg
for training.
Example
config = { "phrase_matcher_attr": None, "validate": True, "overwrite_ents": False, "ent_id_sep": "||", } nlp.add_pipe("entity_ruler", config=config)
Setting | Description |
---|---|
phrase_matcher_attr |
Optional attribute name match on for the internal PhraseMatcher , e.g. LOWER to match on the lowercase token text. Defaults to None . |
matcher_fuzzy_compare 3.5 |
The fuzzy comparison method, passed on to the internal Matcher . Defaults to spacy.matcher.levenshtein.levenshtein_compare . |
validate |
Whether patterns should be validated (passed to the Matcher and PhraseMatcher ). Defaults to False . |
overwrite_ents |
If existing entities are present, e.g. entities added by the model, overwrite them by matches if necessary. Defaults to False . |
ent_id_sep |
Separator used internally for entity IDs. Defaults to "||" . |
scorer |
The scoring method. Defaults to spacy.scorer.get_ner_prf . |
%%GITHUB_SPACY/spacy/pipeline/entityruler.py
EntityRuler.__init__
Initialize the entity ruler. If patterns are supplied here, they need to be a
list of dictionaries with a "label"
and "pattern"
key. A pattern can either
be a token pattern (list) or a phrase pattern (string). For example:
{"label": "ORG", "pattern": "Apple"}
.
Example
# Construction via add_pipe ruler = nlp.add_pipe("entity_ruler") # Construction from class from spacy.pipeline import EntityRuler ruler = EntityRuler(nlp, overwrite_ents=True)
Name | Description |
---|---|
nlp |
The shared nlp object to pass the vocab to the matchers and process phrase patterns. |
name 3 |
Instance name of the current pipeline component. Typically passed in automatically from the factory when the component is added. Used to disable the current entity ruler while creating phrase patterns with the nlp object. |
keyword-only | |
phrase_matcher_attr |
Optional attribute name match on for the internal PhraseMatcher , e.g. LOWER to match on the lowercase token text. Defaults to None . |
matcher_fuzzy_compare 3.5 |
The fuzzy comparison method, passed on to the internal Matcher . Defaults to spacy.matcher.levenshtein.levenshtein_compare . |
validate |
Whether patterns should be validated, passed to Matcher and PhraseMatcher as validate . Defaults to False . |
overwrite_ents |
If existing entities are present, e.g. entities added by the model, overwrite them by matches if necessary. Defaults to False . |
ent_id_sep |
Separator used internally for entity IDs. Defaults to "||" . |
patterns |
Optional patterns to load in on initialization. |
scorer |
The scoring method. Defaults to spacy.scorer.get_ner_prf . |
EntityRuler.initialize
Initialize the component with data and used before training to load in rules
from a pattern file. This
method is typically called by Language.initialize
and lets you customize arguments it receives via the
[initialize.components]
block in the
config.
Example
entity_ruler = nlp.add_pipe("entity_ruler") entity_ruler.initialize(lambda: [], nlp=nlp, patterns=patterns)
### config.cfg [initialize.components.entity_ruler] [initialize.components.entity_ruler.patterns] @readers = "srsly.read_jsonl.v1" path = "corpus/entity_ruler_patterns.jsonl
Name | Description |
---|---|
get_examples |
Function that returns gold-standard annotations in the form of Example objects. Not used by the EntityRuler . |
keyword-only | |
nlp |
The current nlp object. Defaults to None . |
patterns |
The list of patterns. Defaults to None . |
EntityRuler._\len__
The number of all patterns added to the entity ruler.
Example
ruler = nlp.add_pipe("entity_ruler") assert len(ruler) == 0 ruler.add_patterns([{"label": "ORG", "pattern": "Apple"}]) assert len(ruler) == 1
Name | Description |
---|---|
RETURNS | The number of patterns. |
EntityRuler.__contains__
Whether a label is present in the patterns.
Example
ruler = nlp.add_pipe("entity_ruler") ruler.add_patterns([{"label": "ORG", "pattern": "Apple"}]) assert "ORG" in ruler assert not "PERSON" in ruler
Name | Description |
---|---|
label |
The label to check. |
RETURNS | Whether the entity ruler contains the label. |
EntityRuler.__call__
Find matches in the Doc
and add them to the doc.ents
. Typically, this
happens automatically after the component has been added to the pipeline using
nlp.add_pipe
. If the entity ruler was initialized
with overwrite_ents=True
, existing entities will be replaced if they overlap
with the matches. When matches overlap in a Doc, the entity ruler prioritizes
longer patterns over shorter, and if equal the match occuring first in the Doc
is chosen.
Example
ruler = nlp.add_pipe("entity_ruler") ruler.add_patterns([{"label": "ORG", "pattern": "Apple"}]) doc = nlp("A text about Apple.") ents = [(ent.text, ent.label_) for ent in doc.ents] assert ents == [("Apple", "ORG")]
Name | Description |
---|---|
doc |
The Doc object to process, e.g. the Doc in the pipeline. |
RETURNS | The modified Doc with added entities, if available. |
EntityRuler.add_patterns
Add patterns to the entity ruler. A pattern can either be a token pattern (list of dicts) or a phrase pattern (string). For more details, see the usage guide on rule-based matching.
Example
patterns = [ {"label": "ORG", "pattern": "Apple"}, {"label": "GPE", "pattern": [{"lower": "san"}, {"lower": "francisco"}]} ] ruler = nlp.add_pipe("entity_ruler") ruler.add_patterns(patterns)
Name | Description |
---|---|
patterns |
The patterns to add. |
EntityRuler.remove
Remove a pattern by its ID from the entity ruler. A ValueError
is raised if
the ID does not exist.
Example
patterns = [{"label": "ORG", "pattern": "Apple", "id": "apple"}] ruler = nlp.add_pipe("entity_ruler") ruler.add_patterns(patterns) ruler.remove("apple")
Name | Description |
---|---|
id |
The ID of the pattern rule. |
EntityRuler.to_disk
Save the entity ruler patterns to a directory. The patterns will be saved as
newline-delimited JSON (JSONL). If a file with the suffix .jsonl
is provided,
only the patterns are saved as JSONL. If a directory name is provided, a
patterns.jsonl
and cfg
file with the component configuration is exported.
Example
ruler = nlp.add_pipe("entity_ruler") ruler.to_disk("/path/to/patterns.jsonl") # saves patterns only ruler.to_disk("/path/to/entity_ruler") # saves patterns and config
Name | Description |
---|---|
path |
A path to a JSONL file or directory, which will be created if it doesn't exist. Paths may be either strings or Path -like objects. |
EntityRuler.from_disk
Load the entity ruler from a path. Expects either a file containing
newline-delimited JSON (JSONL) with one entry per line, or a directory
containing a patterns.jsonl
file and a cfg
file with the component
configuration.
Example
ruler = nlp.add_pipe("entity_ruler") ruler.from_disk("/path/to/patterns.jsonl") # loads patterns only ruler.from_disk("/path/to/entity_ruler") # loads patterns and config
Name | Description |
---|---|
path |
A path to a JSONL file or directory. Paths may be either strings or Path -like objects. |
RETURNS | The modified EntityRuler object. |
EntityRuler.to_bytes
Serialize the entity ruler patterns to a bytestring.
Example
ruler = nlp.add_pipe("entity_ruler") ruler_bytes = ruler.to_bytes()
Name | Description |
---|---|
RETURNS | The serialized patterns. |
EntityRuler.from_bytes
Load the pipe from a bytestring. Modifies the object in place and returns it.
Example
ruler_bytes = ruler.to_bytes() ruler = nlp.add_pipe("entity_ruler") ruler.from_bytes(ruler_bytes)
Name | Description |
---|---|
bytes_data |
The bytestring to load. |
RETURNS | The modified EntityRuler object. |
EntityRuler.labels
All labels present in the match patterns.
Name | Description |
---|---|
RETURNS | The string labels. |
EntityRuler.ent_ids
All entity IDs present in the id
properties of the match patterns.
Name | Description |
---|---|
RETURNS | The string IDs. |
EntityRuler.patterns
Get all patterns that were added to the entity ruler.
Name | Description |
---|---|
RETURNS | The original patterns, one dictionary per pattern. |
Attributes
Name | Description |
---|---|
matcher |
The underlying matcher used to process token patterns. |
phrase_matcher |
The underlying phrase matcher used to process phrase patterns. |
token_patterns |
The token patterns present in the entity ruler, keyed by label. |
phrase_patterns |
The phrase patterns present in the entity ruler, keyed by label. |