spaCy/spacy/pipeline
Kevin Humphreys 19650ebb52
Enable fuzzy text matching in Matcher (#11359)
* enable fuzzy matching

* add fuzzy param to EntityMatcher

* include rapidfuzz_capi

not yet used

* fix type

* add FUZZY predicate

* add fuzzy attribute list

* fix type properly

* tidying

* remove unnecessary dependency

* handle fuzzy sets

* simplify fuzzy sets

* case fix

* switch to FUZZYn predicates

use Levenshtein distance.
remove fuzzy param.
remove rapidfuzz_capi.

* revert changes added for fuzzy param

* switch to polyleven

(Python package)

* enable fuzzy matching

* add fuzzy param to EntityMatcher

* include rapidfuzz_capi

not yet used

* fix type

* add FUZZY predicate

* add fuzzy attribute list

* fix type properly

* tidying

* remove unnecessary dependency

* handle fuzzy sets

* simplify fuzzy sets

* case fix

* switch to FUZZYn predicates

use Levenshtein distance.
remove fuzzy param.
remove rapidfuzz_capi.

* revert changes added for fuzzy param

* switch to polyleven

(Python package)

* fuzzy match only on oov tokens

* remove polyleven

* exclude whitespace tokens

* don't allow more edits than characters

* fix min distance

* reinstate FUZZY operator

with length-based distance function

* handle sets inside regex operator

* remove is_oov check

* attempt build fix

no mypy failure locally

* re-attempt build fix

* don't overwrite fuzzy param value

* move fuzzy_match

to its own Python module to allow patching

* move fuzzy_match back inside Matcher

simplify logic and add tests

* Format tests

* Parametrize fuzzyn tests

* Parametrize and merge fuzzy+set tests

* Format

* Move fuzzy_match to a standalone method

* Change regex kwarg type to bool

* Add types for fuzzy_match

- Refactor variable names
- Add test for symmetrical behavior

* Parametrize fuzzyn+set tests

* Minor refactoring for fuzz/fuzzy

* Make fuzzy_match a Matcher kwarg

* Update type for _default_fuzzy_match

* don't overwrite function param

* Rename to fuzzy_compare

* Update fuzzy_compare default argument declarations

* allow fuzzy_compare override from EntityRuler

* define new Matcher keyword arg

* fix type definition

* Implement fuzzy_compare config option for EntityRuler and SpanRuler

* Rename _default_fuzzy_compare to fuzzy_compare, remove from reexported objects

* Use simpler fuzzy_compare algorithm

* Update types

* Increase minimum to 2 in fuzzy_compare to allow one transposition

* Fix predicate keys and matching for SetPredicate with FUZZY and REGEX

* Add FUZZY6..9

* Add initial docs

* Increase default fuzzy to rounded 30% of pattern length

* Update docs for fuzzy_compare in components

* Update EntityRuler and SpanRuler API docs

* Rename EntityRuler and SpanRuler setting to matcher_fuzzy_compare

To having naming similar to `phrase_matcher_attr`, rename
`fuzzy_compare` setting for `EntityRuler` and `SpanRuler` to
`matcher_fuzzy_compare. Organize next to `phrase_matcher_attr` in docs.

* Fix schema aliases

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Fix typo

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Add FUZZY6-9 operators and update tests

* Parameterize test over greedy

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

* Fix type for fuzzy_compare to remove Optional

* Rename to spacy.levenshtein_compare.v1, move to spacy.matcher.levenshtein

* Update docs following levenshtein_compare renaming

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
2023-01-10 10:36:17 +01:00
..
_edit_tree_internals Refactor error messages to remove hardcoded strings (#10729) 2022-05-02 13:38:46 +02:00
_parser_internals account for NER labels with a hyphen in the name (#10960) 2022-06-17 20:02:37 +01:00
legacy Refactor KB for easier customization (#11268) 2022-09-08 10:38:07 +02:00
__init__.py Add SpanRuler component (#9880) 2022-06-02 13:12:53 +02:00
attributeruler.py Merge remote-tracking branch 'upstream/master' into chore/update-develop-from-master-v3.2-1 2021-10-26 11:53:50 +02:00
dep_parser.pyx account for NER labels with a hyphen in the name (#10960) 2022-06-17 20:02:37 +01:00
edit_tree_lemmatizer.py EditTreeLemmatizer: correctly add strings when initializing from labels (#11934) 2022-12-07 13:53:41 +09:00
entity_linker.py Refactor KB for easier customization (#11268) 2022-09-08 10:38:07 +02:00
entityruler.py Enable fuzzy text matching in Matcher (#11359) 2023-01-10 10:36:17 +01:00
functions.py Add doc_cleaner component (#9659) 2021-11-23 15:33:33 +01:00
lemmatizer.py Merge remote-tracking branch 'upstream/master' into chore/update-develop-from-master-v3.2-1 2021-10-26 11:53:50 +02:00
morphologizer.pyx Tagger: use unnormalized probabilities for inference (#10197) 2022-03-15 14:15:31 +01:00
multitask.pyx Replace negative rows with 0 in StaticVectors (#7674) 2021-04-22 18:04:15 +10:00
ner.pyx account for NER labels with a hyphen in the name (#10960) 2022-06-17 20:02:37 +01:00
pipe.pxd TrainablePipe (#6213) 2020-10-08 21:33:49 +02:00
pipe.pyi Add Pipe.hide_labels to omit labels from pipeline meta (#10175) 2022-02-05 17:59:24 +01:00
pipe.pyx Fix config validation failures caused by NVTX pipeline wrappers (#11460) 2022-09-12 14:55:41 +02:00
sentencizer.pyx Add overwrite settings for more components (#9050) 2021-09-30 15:35:55 +02:00
senter.pyx Tagger: use unnormalized probabilities for inference (#10197) 2022-03-15 14:15:31 +01:00
span_ruler.py Enable fuzzy text matching in Matcher (#11359) 2023-01-10 10:36:17 +01:00
spancat.py Fix spancat for zero suggestions (#11860) 2022-12-02 09:33:52 +01:00
tagger.pyx Tagger: use unnormalized probabilities for inference (#10197) 2022-03-15 14:15:31 +01:00
textcat_multilabel.py Improve score_cats for use with multiple textcat components (#11820) 2023-01-09 11:43:48 +01:00
textcat.py Rename modified textcat scorer to v2 (#11971) 2022-12-29 14:01:08 +01:00
tok2vec.py Prevent tok2vec to broadcast to listeners when predicting (#11385) 2022-09-12 15:36:48 +02:00
trainable_pipe.pxd Refactor scoring methods to use registered functions (#8766) 2021-08-10 15:13:39 +02:00
trainable_pipe.pyx Fix config validation failures caused by NVTX pipeline wrappers (#11460) 2022-09-12 14:55:41 +02:00
transition_parser.pxd Parser: use C saxpy/sgemm provided by the Ops implementation (#10773) 2022-05-27 11:20:52 +02:00
transition_parser.pyx precompute_hiddens/Parser: do not look up CPU ops (3.4) (#11069) 2022-07-05 10:53:42 +02:00