mirror of
https://github.com/explosion/spaCy.git
synced 2024-09-23 12:29:18 +03:00
36514b5762
<!--- Provide a general summary of your changes in the title. --> ## Description <!--- Use this section to describe your changes. If your changes required testing, include information about the testing environment and the tests you ran. If your test fixes a bug reported in an issue, don't forget to include the issue number. If your PR is still a work in progress, that's totally fine – just include a note to let us know. --> Add a rule-based French Lemmatizer following the english one and the excellent PR for [greek language optimizations](https://github.com/explosion/spaCy/pull/2558) to adapt the Lemmatizer class. ### Types of change <!-- What type of change does your PR cover? Is it a bug fix, an enhancement or new feature, or a change to the documentation? --> - Lemma dictionary used can be found [here](http://infolingu.univ-mlv.fr/DonneesLinguistiques/Dictionnaires/telechargement.html), I used the XML version. - Add several files containing exhaustive list of words for each part of speech - Add some lemma rules - Add POS that are not checked in the standard Lemmatizer, i.e PRON, DET, ADV and AUX - Modify the Lemmatizer class to check in lookup table as a last resort if POS not mentionned - Modify the lemmatize function to check in lookup table as a last resort - Init files are updated so the model can support all the functionalities mentioned above - Add words to tokenizer_exceptions_list.py in respect to regex used in tokenizer_exceptions.py ## Checklist <!--- Before you submit the PR, go over this checklist and make sure you can tick off all the boxes. [] -> [x] --> - [X] I have submitted the spaCy Contributor Agreement. - [X] I ran the tests, and all new and existing tests passed. - [X] My changes don't require a change to the documentation, or if they do, I've added all required information.
57 lines
942 B
Python
57 lines
942 B
Python
# coding: utf8
|
|
from __future__ import unicode_literals
|
|
|
|
|
|
ADJECTIVE_RULES = [
|
|
["s", ""],
|
|
["e", ""],
|
|
["es", ""]
|
|
]
|
|
|
|
|
|
NOUN_RULES = [
|
|
["s", ""]
|
|
]
|
|
|
|
|
|
VERB_RULES = [
|
|
["é", "er"],
|
|
["és", "er"],
|
|
["ée", "er"],
|
|
["ées", "er"],
|
|
["é", "er"],
|
|
["es", "er"],
|
|
["ons", "er"],
|
|
["ez", "er"],
|
|
["ent", "er"],
|
|
["ais", "er"],
|
|
["ait", "er"],
|
|
["ions", "er"],
|
|
["iez", "er"],
|
|
["aient", "er"],
|
|
["ai", "er"],
|
|
["as", "er"],
|
|
["a", "er"],
|
|
["âmes", "er"],
|
|
["âtes", "er"],
|
|
["èrent", "er"],
|
|
["erai", "er"],
|
|
["eras", "er"],
|
|
["era", "er"],
|
|
["erons", "er"],
|
|
["erez", "er"],
|
|
["eront", "er"],
|
|
["erais", "er"],
|
|
["erait", "er"],
|
|
["erions", "er"],
|
|
["eriez", "er"],
|
|
["eraient", "er"],
|
|
["asse", "er"],
|
|
["asses", "er"],
|
|
["ât", "er"],
|
|
["assions", "er"],
|
|
["assiez", "er"],
|
|
["assent", "er"],
|
|
["ant", "er"]
|
|
]
|