explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-12-25 01:16:28 +03:00
Code Issues Packages Projects Releases Wiki Activity
e77220e3ae
spaCy/spacy
History
Matthew Honnibal 80b94313b6 💫 Fix interaction of lemmatizer and tokenizer exceptions (#3388) 
Closes #2203. Closes #3268.

Lemmas set from outside the `Morphology` class were being overwritten. The result was especially confusing when deserialising, as it meant some lemmas could change when storing and retrieving a `Doc` object.

This PR applies two fixes:

1) When we go to set the lemma in the `Morphology` class, first check whether a lemma is already set. If so, don't overwrite.
2) When we load with `doc.from_array()`, take care to apply the `TAG` field first. This allows other fields to overwrite the `TAG` implied properties, if they're provided explicitly (e.g. the `LEMMA`).

## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I have submitted the spaCy Contributor Agreement.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
2019-03-11 01:31:21 +01:00
..
cli Only link model after download if shortcut link (#3378) 2019-03-10 13:02:24 +01:00
data Make spacy/data a package 2017-03-18 20:04:22 +01:00
displacy Tidy up and improve docs and docstrings (#3370) 2019-03-08 11:42:26 +01:00
lang Fix English tag map 2019-03-11 01:06:02 +01:00
matcher Tidy up and improve docs and docstrings (#3370) 2019-03-08 11:42:26 +01:00
pipeline 💫 Make serialization methods consistent (#3385) 2019-03-10 19:16:45 +01:00
syntax 💫 Make serialization methods consistent (#3385) 2019-03-10 19:16:45 +01:00
tests 💫 Fix interaction of lemmatizer and tokenizer exceptions (#3388) 2019-03-11 01:31:21 +01:00
tokens 💫 Fix interaction of lemmatizer and tokenizer exceptions (#3388) 2019-03-11 01:31:21 +01:00
__init__.pxd * Seems to be working after refactor. Need to wire up more POS tag features, and wire up save/load of POS tags. 2014-10-24 02:23:42 +11:00
__init__.py Tidy up and format remaining files 2018-11-30 17:43:08 +01:00
__main__.py 💫 New JSON helpers, training data internals & CLI rewrite (#2932) 2018-11-30 20:16:14 +01:00
_align.pyx Improve alignment around quotes 2018-08-16 01:04:34 +02:00
_ml.py Auto-format [ci skip] 2019-03-10 19:22:59 +01:00
about.py Set version to v2.1.0a10 2019-02-27 12:26:13 +01:00
attrs.pxd Fix LANG symbol 2018-02-17 18:10:50 +01:00
attrs.pyx Revert "Merge branch 'develop' of https://github.com/explosion/spaCy into develop" 2018-03-27 19:23:02 +02:00
compat.py Tidy up and improve docs and docstrings (#3370) 2019-03-08 11:42:26 +01:00
errors.py 💫 Make serialization methods consistent (#3385) 2019-03-10 19:16:45 +01:00
glossary.py 💫 Tidy up and auto-format .py files (#2983) 2018-11-30 17:03:03 +01:00
gold.pxd Add support for sent_start to GoldParse 2017-08-25 20:03:14 -05:00
gold.pyx 💫 Improve converters and training data file formats (#3374) 2019-03-08 23:15:23 +01:00
language.py 💫 Allow passing of config parameters to specific pipeline components (#3386) 2019-03-10 23:36:47 +01:00
lemmatizer.py Tidy up and improve docs and docstrings (#3370) 2019-03-08 11:42:26 +01:00
lexeme.pxd 💫 Support lexical attributes in retokenizer attrs (closes #2390) (#3325) 2019-02-24 21:13:51 +01:00
lexeme.pyx Tidy up and improve docs and docstrings (#3370) 2019-03-08 11:42:26 +01:00
morphology.pxd Revert "Merge branch 'develop' of https://github.com/explosion/spaCy into develop" 2018-03-27 19:23:02 +02:00
morphology.pyx 💫 Fix interaction of lemmatizer and tokenizer exceptions (#3388) 2019-03-11 01:31:21 +01:00
parts_of_speech.pxd Add support for Universal Dependencies v2.0 2017-03-03 13:17:34 +01:00
parts_of_speech.pyx Tidy up rest 2017-10-27 21:07:59 +02:00
scorer.py 💫 Tidy up and auto-format .py files (#2983) 2018-11-30 17:03:03 +01:00
strings.pxd Try to fix StringStore clean up (see #1506) 2017-11-11 03:11:27 +03:00
strings.pyx 💫 Make serialization methods consistent (#3385) 2019-03-10 19:16:45 +01:00
structs.pxd Make NORM a token attribute (#3029) 2018-12-08 10:49:10 +01:00
symbols.pxd Revert "Merge branch 'develop' of https://github.com/explosion/spaCy into develop" 2018-03-27 19:23:02 +02:00
symbols.pyx Revert "Merge branch 'develop' of https://github.com/explosion/spaCy into develop" 2018-03-27 19:23:02 +02:00
tokenizer.pxd Disable tokenizer cache for special-cases. Fixes #1250 2017-10-24 16:08:05 +02:00
tokenizer.pyx 💫 Make serialization methods consistent (#3385) 2019-03-10 19:16:45 +01:00
typedefs.pxd Work on changing StringStore to return hashes. 2017-05-28 12:36:27 +02:00
typedefs.pyx Tidy up rest 2017-10-27 21:07:59 +02:00
util.py 💫 Make serialization methods consistent (#3385) 2019-03-10 19:16:45 +01:00
vectors.pyx 💫 Make serialization methods consistent (#3385) 2019-03-10 19:16:45 +01:00
vocab.pxd 💫 Small efficiency fixes to tokenizer (#2587) 2018-07-24 23:35:54 +02:00
vocab.pyx 💫 Make serialization methods consistent (#3385) 2019-03-10 19:16:45 +01:00