spaCy/spacy/pipeline
Adriane Boyd fa79a0db9f
Add AttributeRuler for token attribute exceptions (#5842)
* Add AttributeRuler for token attribute exceptions

Add the `AttributeRuler` to handle exceptions for token-level
attributes. The `AttributeRuler` uses `Matcher` patterns to identify
target spans and applies the specified attributes to the token at the
provided index in the matched span. A negative index can be used to
index from the end of the matched span. The retokenizer is used to
"merge" the individual tokens and assign them the provided attributes.

Helper functions can import existing tag maps and morph rules to the
corresponding `Matcher` patterns.

There is an additional minor bug fix for `MORPH` attributes in the
retokenizer to correctly normalize the values and to handle `MORPH`
alongside `_` in an attrs dict.

* Fix default name

* Update name in error message

* Extend AttributeRuler functionality

* Add option to initialize with a dict of AttributeRuler patterns

* Instead of silently discarding overlapping matches (the default
behavior for the retokenizer if only the attrs differ), split the
matches into disjoint sets and retokenize each set separately. This
allows, for instance, one pattern to set the POS and another pattern to
set the lemma. (If two matches modify the same attribute, it looks like
the attrs are applied in the order they were added, but it may not be
deterministic?)

* Improve types

* Sort spans before processing

* Fix index boundaries in Span

* Refactor retokenizer to separate attrs methods

Add top-level `normalize_token_attrs` and `set_token_attrs` methods.

* Update AttributeRuler to use refactored methods

Update `AttributeRuler` to replace use of full retokenizer with only the
relevant methods for normalizing and setting attributes for a single
token.

* Update spacy/pipeline/attributeruler.py

Co-authored-by: Ines Montani <ines@ines.io>

* Make API more similar to EntityRuler

* Add `AttributeRuler.add_patterns` to add patterns from a list of dicts
* Return list of dicts as property `AttributeRuler.patterns`

* Make attrs_unnormed private

* Add test loading patterns from assets

* Revert "Fix index boundaries in Span"

This reverts commit 8f8a5c3386.

* Add Span index boundary checks (#5861)

* Add Span index boundary checks

* Return Span-specific IndexError in all cases

* Simplify and fix if/else

Co-authored-by: Ines Montani <ines@ines.io>
2020-08-04 17:02:39 +02:00
..
_parser_internals The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
__init__.py Add AttributeRuler for token attribute exceptions (#5842) 2020-08-04 17:02:39 +02:00
attributeruler.py Add AttributeRuler for token attribute exceptions (#5842) 2020-08-04 17:02:39 +02:00
dep_parser.pyx Fix default arguments in DependencyParser.score 2020-07-31 10:55:44 +02:00
entity_linker.py Default empty KB in EL component (#5872) 2020-08-04 14:34:09 +02:00
entityruler.py Update docs and consistency 2020-07-29 15:14:07 +02:00
functions.py Option for returning only greedy matches (#5771) 2020-07-29 11:04:43 +02:00
morphologizer.pyx Merge pull request #5834 from explosion/feature/vectors 2020-07-29 18:49:26 +02:00
multitask.pyx The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
ner.pyx The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
nn_parser.pyx The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
pipe.pxd The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
pipe.pyx The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
sentencizer.pyx The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
senter.pyx The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
simple_ner.py The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
tagger.pyx Merge pull request #5834 from explosion/feature/vectors 2020-07-29 18:49:26 +02:00
textcat.py The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
tok2vec.py The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
transition_parser.pxd The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00
transition_parser.pyx The Parser is now a Pipe (2) (#5844) 2020-07-30 23:30:54 +02:00