spaCy/spacy/tokens
Adriane Boyd bf9096437e
Set default lemmas in retokenizer (#6667)
Instead of unsetting lemmas on retokenized tokens, set the default
lemmas to:

* merge: concatenate any existing lemmas with `SPACY` preserved
* split: use the new `ORTH` values if lemmas were previously set,
  otherwise leave unset
2021-01-06 12:29:44 +08:00
..
__init__.pxd * Break up tokens.pyx into tokens/doc.pyx, tokens/token.pyx, tokens/spans.pyx 2015-07-13 20:20:58 +02:00
__init__.py Modify morphology to support arbitrary features (#4932) 2020-01-23 22:01:54 +01:00
_retokenize.pyx Set default lemmas in retokenizer (#6667) 2021-01-06 12:29:44 +08:00
_serialize.py Add error message if DocBin zlib decompress fails (#6394) 2020-11-27 14:39:49 +08:00
doc.pxd Refactor Docs.is_ flags (#6044) 2020-09-17 00:14:01 +02:00
doc.pyx Prevent 0-length mem alloc (#6653) 2021-01-06 12:50:17 +11:00
morphanalysis.pxd Modify morphology to support arbitrary features (#4932) 2020-01-23 22:01:54 +01:00
morphanalysis.pyx Minor refactor for Morphology and MorphAnalysis (#5804) 2020-07-24 09:28:06 +02:00
span.pxd Remove Span._recalculate_indices 2020-10-09 14:42:51 +02:00
span.pyx Remove Span._recalculate_indices 2020-10-09 14:42:51 +02:00
token.pxd Tidy up compiler flags and imports (#5071) 2020-03-02 11:48:10 +01:00
token.pyx Also accept MorphAnalysis in set_morph 2020-10-02 08:33:43 +02:00
underscore.py Remove object subclassing 2020-07-12 14:03:23 +02:00