mirror of
https://github.com/explosion/spaCy.git
synced 2024-12-28 02:46:35 +03:00
63f5951f8b
Add the `AttributeRuler` to handle exceptions for token-level attributes. The `AttributeRuler` uses `Matcher` patterns to identify target spans and applies the specified attributes to the token at the provided index in the matched span. A negative index can be used to index from the end of the matched span. The retokenizer is used to "merge" the individual tokens and assign them the provided attributes. Helper functions can import existing tag maps and morph rules to the corresponding `Matcher` patterns. There is an additional minor bug fix for `MORPH` attributes in the retokenizer to correctly normalize the values and to handle `MORPH` alongside `_` in an attrs dict. |
||
|---|---|---|
| .. | ||
| __init__.pxd | ||
| __init__.py | ||
| _retokenize.pyx | ||
| _serialize.py | ||
| doc.pxd | ||
| doc.pyx | ||
| morphanalysis.pxd | ||
| morphanalysis.pyx | ||
| span.pxd | ||
| span.pyx | ||
| token.pxd | ||
| token.pyx | ||
| underscore.py | ||