Rule-based French Lemmatizer (#2818)
<!--- Provide a general summary of your changes in the title. -->
## Description
<!--- Use this section to describe your changes. If your changes required
testing, include information about the testing environment and the tests you
ran. If your test fixes a bug reported in an issue, don't forget to include the
issue number. If your PR is still a work in progress, that's totally fine – just
include a note to let us know. -->
Add a rule-based French Lemmatizer following the english one and the excellent PR for [greek language optimizations](https://github.com/explosion/spaCy/pull/2558) to adapt the Lemmatizer class.
### Types of change
<!-- What type of change does your PR cover? Is it a bug fix, an enhancement
or new feature, or a change to the documentation? -->
- Lemma dictionary used can be found [here](http://infolingu.univ-mlv.fr/DonneesLinguistiques/Dictionnaires/telechargement.html), I used the XML version.
- Add several files containing exhaustive list of words for each part of speech
- Add some lemma rules
- Add POS that are not checked in the standard Lemmatizer, i.e PRON, DET, ADV and AUX
- Modify the Lemmatizer class to check in lookup table as a last resort if POS not mentionned
- Modify the lemmatize function to check in lookup table as a last resort
- Init files are updated so the model can support all the functionalities mentioned above
- Add words to tokenizer_exceptions_list.py in respect to regex used in tokenizer_exceptions.py
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [X] I have submitted the spaCy Contributor Agreement.
- [X] I ran the tests, and all new and existing tests passed.
- [X] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-10-13 17:38:21 +03:00
|
|
|
# coding: utf8
|
|
|
|
|
from __future__ import unicode_literals
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ADJECTIVE_RULES = [
|
|
|
|
|
["s", ""],
|
|
|
|
|
["e", ""],
|
|
|
|
|
["es", ""]
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
NOUN_RULES = [
|
2018-12-07 00:42:18 +03:00
|
|
|
["a", "a"],
|
|
|
|
|
["à", "à"],
|
|
|
|
|
["â", "â"],
|
|
|
|
|
["b", "b"],
|
|
|
|
|
["c", "c"],
|
|
|
|
|
["ç", "ç"],
|
|
|
|
|
["d", "d"],
|
|
|
|
|
["e", "e"],
|
|
|
|
|
["é", "é"],
|
|
|
|
|
["è", "è"],
|
|
|
|
|
["ê", "ê"],
|
|
|
|
|
["ë", "ë"],
|
|
|
|
|
["f", "f"],
|
|
|
|
|
["g", "g"],
|
|
|
|
|
["h", "h"],
|
|
|
|
|
["i", "i"],
|
|
|
|
|
["î", "î"],
|
|
|
|
|
["ï", "ï"],
|
|
|
|
|
["j", "j"],
|
|
|
|
|
["k", "k"],
|
|
|
|
|
["l", "l"],
|
|
|
|
|
["m", "m"],
|
|
|
|
|
["n", "n"],
|
|
|
|
|
["o", "o"],
|
|
|
|
|
["ô", "ö"],
|
|
|
|
|
["ö", "ö"],
|
|
|
|
|
["p", "p"],
|
|
|
|
|
["q", "q"],
|
|
|
|
|
["r", "r"],
|
|
|
|
|
["t", "t"],
|
|
|
|
|
["u", "u"],
|
|
|
|
|
["û", "û"],
|
|
|
|
|
["v", "v"],
|
|
|
|
|
["w", "w"],
|
|
|
|
|
["y", "y"],
|
|
|
|
|
["z", "z"],
|
|
|
|
|
|
|
|
|
|
["as", "a"],
|
|
|
|
|
["aux", "au"],
|
|
|
|
|
["cs", "c"],
|
|
|
|
|
["chs", "ch"],
|
|
|
|
|
["ds", "d"],
|
|
|
|
|
["és", "é"],
|
|
|
|
|
["es", "e"],
|
|
|
|
|
["eux", "eu"],
|
|
|
|
|
["fs", "f"],
|
|
|
|
|
["gs", "g"],
|
|
|
|
|
["hs", "h"],
|
|
|
|
|
["is", "i"],
|
|
|
|
|
["ïs", "ï"],
|
|
|
|
|
["js", "j"],
|
|
|
|
|
["ks", "k"],
|
|
|
|
|
["ls", "l"],
|
|
|
|
|
["ms", "m"],
|
|
|
|
|
["ns", "n"],
|
|
|
|
|
["oux", "ou"],
|
|
|
|
|
["os", "o"],
|
|
|
|
|
["ps", "p"],
|
|
|
|
|
["qs", "q"],
|
|
|
|
|
["rs", "r"],
|
|
|
|
|
["ses", "se"],
|
|
|
|
|
["se", "se"],
|
|
|
|
|
["ts", "t"],
|
|
|
|
|
["us", "u"],
|
|
|
|
|
["vs", "v"],
|
|
|
|
|
["ws", "w"],
|
|
|
|
|
["ys", "y"],
|
|
|
|
|
|
|
|
|
|
["nt(e", "nt"],
|
|
|
|
|
["nt(e)", "nt"],
|
|
|
|
|
["al(e", "ale"],
|
|
|
|
|
["é(", "é"],
|
|
|
|
|
["é(e", "é"],
|
|
|
|
|
["é.e", "é"],
|
|
|
|
|
["el(le", "el"],
|
|
|
|
|
["eurs(rices", "eur"],
|
|
|
|
|
["eur(rice", "eur"],
|
|
|
|
|
["eux(se", "eux"],
|
|
|
|
|
["ial(e","ial"],
|
|
|
|
|
["er(ère", "er"],
|
|
|
|
|
["eur(se", "eur"],
|
|
|
|
|
["teur(trice", "teur"],
|
|
|
|
|
["teurs(trices", "teur"]
|
Rule-based French Lemmatizer (#2818)
<!--- Provide a general summary of your changes in the title. -->
## Description
<!--- Use this section to describe your changes. If your changes required
testing, include information about the testing environment and the tests you
ran. If your test fixes a bug reported in an issue, don't forget to include the
issue number. If your PR is still a work in progress, that's totally fine – just
include a note to let us know. -->
Add a rule-based French Lemmatizer following the english one and the excellent PR for [greek language optimizations](https://github.com/explosion/spaCy/pull/2558) to adapt the Lemmatizer class.
### Types of change
<!-- What type of change does your PR cover? Is it a bug fix, an enhancement
or new feature, or a change to the documentation? -->
- Lemma dictionary used can be found [here](http://infolingu.univ-mlv.fr/DonneesLinguistiques/Dictionnaires/telechargement.html), I used the XML version.
- Add several files containing exhaustive list of words for each part of speech
- Add some lemma rules
- Add POS that are not checked in the standard Lemmatizer, i.e PRON, DET, ADV and AUX
- Modify the Lemmatizer class to check in lookup table as a last resort if POS not mentionned
- Modify the lemmatize function to check in lookup table as a last resort
- Init files are updated so the model can support all the functionalities mentioned above
- Add words to tokenizer_exceptions_list.py in respect to regex used in tokenizer_exceptions.py
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [X] I have submitted the spaCy Contributor Agreement.
- [X] I ran the tests, and all new and existing tests passed.
- [X] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-10-13 17:38:21 +03:00
|
|
|
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
VERB_RULES = [
|
|
|
|
|
["é", "er"],
|
|
|
|
|
["és", "er"],
|
|
|
|
|
["ée", "er"],
|
|
|
|
|
["ées", "er"],
|
|
|
|
|
["é", "er"],
|
|
|
|
|
["es", "er"],
|
|
|
|
|
["ons", "er"],
|
|
|
|
|
["ez", "er"],
|
|
|
|
|
["ent", "er"],
|
|
|
|
|
["ais", "er"],
|
|
|
|
|
["ait", "er"],
|
|
|
|
|
["ions", "er"],
|
|
|
|
|
["iez", "er"],
|
|
|
|
|
["aient", "er"],
|
|
|
|
|
["ai", "er"],
|
|
|
|
|
["as", "er"],
|
|
|
|
|
["a", "er"],
|
|
|
|
|
["âmes", "er"],
|
|
|
|
|
["âtes", "er"],
|
|
|
|
|
["èrent", "er"],
|
|
|
|
|
["erai", "er"],
|
|
|
|
|
["eras", "er"],
|
|
|
|
|
["era", "er"],
|
|
|
|
|
["erons", "er"],
|
|
|
|
|
["erez", "er"],
|
|
|
|
|
["eront", "er"],
|
|
|
|
|
["erais", "er"],
|
|
|
|
|
["erait", "er"],
|
|
|
|
|
["erions", "er"],
|
|
|
|
|
["eriez", "er"],
|
|
|
|
|
["eraient", "er"],
|
|
|
|
|
["asse", "er"],
|
|
|
|
|
["asses", "er"],
|
|
|
|
|
["ât", "er"],
|
|
|
|
|
["assions", "er"],
|
|
|
|
|
["assiez", "er"],
|
|
|
|
|
["assent", "er"],
|
2018-12-06 17:49:28 +03:00
|
|
|
["ant", "er"],
|
|
|
|
|
["ante", "er"],
|
|
|
|
|
["ants", "er"],
|
|
|
|
|
["antes", "er"],
|
|
|
|
|
|
|
|
|
|
["u(er", "u"],
|
|
|
|
|
["és(ées", "er"],
|
|
|
|
|
["é()e", "er"],
|
2018-12-07 00:42:18 +03:00
|
|
|
["é()", "er"]
|
Rule-based French Lemmatizer (#2818)
<!--- Provide a general summary of your changes in the title. -->
## Description
<!--- Use this section to describe your changes. If your changes required
testing, include information about the testing environment and the tests you
ran. If your test fixes a bug reported in an issue, don't forget to include the
issue number. If your PR is still a work in progress, that's totally fine – just
include a note to let us know. -->
Add a rule-based French Lemmatizer following the english one and the excellent PR for [greek language optimizations](https://github.com/explosion/spaCy/pull/2558) to adapt the Lemmatizer class.
### Types of change
<!-- What type of change does your PR cover? Is it a bug fix, an enhancement
or new feature, or a change to the documentation? -->
- Lemma dictionary used can be found [here](http://infolingu.univ-mlv.fr/DonneesLinguistiques/Dictionnaires/telechargement.html), I used the XML version.
- Add several files containing exhaustive list of words for each part of speech
- Add some lemma rules
- Add POS that are not checked in the standard Lemmatizer, i.e PRON, DET, ADV and AUX
- Modify the Lemmatizer class to check in lookup table as a last resort if POS not mentionned
- Modify the lemmatize function to check in lookup table as a last resort
- Init files are updated so the model can support all the functionalities mentioned above
- Add words to tokenizer_exceptions_list.py in respect to regex used in tokenizer_exceptions.py
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [X] I have submitted the spaCy Contributor Agreement.
- [X] I ran the tests, and all new and existing tests passed.
- [X] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-10-13 17:38:21 +03:00
|
|
|
]
|
