explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-12-27 10:26:35 +03:00
Code Issues Packages Projects Releases Wiki Activity
1b2d66f98e
spaCy/spacy/lang/ga/tokenizer_exceptions.py

72 lines
1.8 KiB
Python
Raw Normal View History

isort all the things
2023-06-26 12:41:03 +03:00
from ...symbols import NORM, ORTH
Tidy up and move noun_chunks, token_match, url_match
2020-07-22 23:18:46 +03:00
from ...util import update_exc
isort all the things
2023-06-26 12:41:03 +03:00
from ..tokenizer_exceptions import BASE_EXCEPTIONS
attempt a port from #1147
2017-06-26 23:24:55 +03:00
_exc = {
Remove POS, TAG and LEMMA from tokenizer exceptions
2020-07-23 00:09:01 +03:00
"'acha'n": [{ORTH: "'ach", NORM: "gach"}, {ORTH: "a'n", NORM: "aon"}],
"dem'": [{ORTH: "de", NORM: "de"}, {ORTH: "m'", NORM: "mo"}],
"ded'": [{ORTH: "de", NORM: "de"}, {ORTH: "d'", NORM: "do"}],
"lem'": [{ORTH: "le", NORM: "le"}, {ORTH: "m'", NORM: "mo"}],
"led'": [{ORTH: "le", NORM: "le"}, {ORTH: "d'", NORM: "do"}],
Add an Irish lemmatiser, based on BuNaMo (#9102) * add tréis/théis * remove previous contents, add demutate/unponc * fmt off/on wrapping * type hints * IrishLemmatizer (sic) * Use spacy-lookups-data>=1.0.3 * Minor bug fixes, refactoring for IrishLemmatizer * Fix return type for ADP list lookups * Fix and refactor lookup table lookups for missing/string/list * Remove unused variables * skip lookup of verbal substantives and adjectives; just demutate * Fix morph checks API details * Add types and format * Move helper methods into lemmatizer Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
2021-09-30 15:18:47 +03:00
"théis": [{ORTH: "th", NORM: "tar"}, {ORTH: "éis", NORM: "éis"}],
"tréis": [{ORTH: "tr", NORM: "tar"}, {ORTH: "éis", NORM: "éis"}],
attempt a port from #1147
2017-06-26 23:24:55 +03:00
}
for exc_data in [
Remove POS, TAG and LEMMA from tokenizer exceptions
2020-07-23 00:09:01 +03:00
{ORTH: "'gus", NORM: "agus"},
{ORTH: "'ach", NORM: "gach"},
{ORTH: "ao'", NORM: "aon"},
{ORTH: "'niar", NORM: "aniar"},
{ORTH: "'níos", NORM: "aníos"},
{ORTH: "'ndiu", NORM: "inniu"},
{ORTH: "'nocht", NORM: "anocht"},
{ORTH: "m'"},
{ORTH: "Aib."},
{ORTH: "Ath."},
{ORTH: "Beal."},
{ORTH: "a.C.n."},
{ORTH: "m.sh."},
{ORTH: "M.F."},
{ORTH: "M.Fómh."},
{ORTH: "D.F."},
{ORTH: "D.Fómh."},
{ORTH: "r.C."},
{ORTH: "R.C."},
{ORTH: "r.Ch."},
{ORTH: "r.Chr."},
{ORTH: "R.Ch."},
{ORTH: "R.Chr."},
{ORTH: "⁊rl."},
{ORTH: "srl."},
{ORTH: "Co."},
{ORTH: "Ean."},
{ORTH: "Feab."},
{ORTH: "gCo."},
{ORTH: ".i."},
{ORTH: "B'"},
{ORTH: "b'"},
{ORTH: "lch."},
{ORTH: "Lch."},
{ORTH: "lgh."},
{ORTH: "Lgh."},
{ORTH: "Lún."},
{ORTH: "Már."},
{ORTH: "Meith."},
{ORTH: "Noll."},
{ORTH: "Samh."},
{ORTH: "tAth."},
{ORTH: "tUas."},
{ORTH: "teo."},
{ORTH: "Teo."},
{ORTH: "Uas."},
{ORTH: "uimh."},
{ORTH: "Uimh."},
💫 Tidy up and auto-format .py files (#2983) <!--- Provide a general summary of your changes in the title. --> ## Description - [x] Use [`black`](https://github.com/ambv/black) to auto-format all `.py` files. - [x] Update flake8 config to exclude very large files (lemmatization tables etc.) - [x] Update code to be compatible with flake8 rules - [x] Fix various small bugs, inconsistencies and messy stuff in the language data - [x] Update docs to explain new code style (`black`, `flake8`, when to use `# fmt: off` and `# fmt: on` and what `# noqa` means) Once #2932 is merged, which auto-formats and tidies up the CLI, we'll be able to run `flake8 spacy` actually get meaningful results. At the moment, the code style and linting isn't applied automatically, but I'm hoping that the new [GitHub Actions](https://github.com/features/actions) will let us auto-format pull requests and post comments with relevant linting information. ### Types of change enhancement, code style ## Checklist <!--- Before you submit the PR, go over this checklist and make sure you can tick off all the boxes. [] -> [x] --> - [x] I have submitted the spaCy Contributor Agreement. - [x] I ran the tests, and all new and existing tests passed. - [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-11-30 19:03:03 +03:00
]:
Remove comma that caused list to wrap in tuple! Also removed extra dict wrappings for performance (we used to have them in there, but they should only really exist if copying the dict is absolutely necessary)
2017-10-31 22:13:16 +03:00
_exc[exc_data[ORTH]] = [exc_data]
attempt a port from #1147
2017-06-26 23:24:55 +03:00
💫 Tidy up and auto-format .py files (#2983) <!--- Provide a general summary of your changes in the title. --> ## Description - [x] Use [`black`](https://github.com/ambv/black) to auto-format all `.py` files. - [x] Update flake8 config to exclude very large files (lemmatization tables etc.) - [x] Update code to be compatible with flake8 rules - [x] Fix various small bugs, inconsistencies and messy stuff in the language data - [x] Update docs to explain new code style (`black`, `flake8`, when to use `# fmt: off` and `# fmt: on` and what `# noqa` means) Once #2932 is merged, which auto-formats and tidies up the CLI, we'll be able to run `flake8 spacy` actually get meaningful results. At the moment, the code style and linting isn't applied automatically, but I'm hoping that the new [GitHub Actions](https://github.com/features/actions) will let us auto-format pull requests and post comments with relevant linting information. ### Types of change enhancement, code style ## Checklist <!--- Before you submit the PR, go over this checklist and make sure you can tick off all the boxes. [] -> [x] --> - [x] I have submitted the spaCy Contributor Agreement. - [x] I ran the tests, and all new and existing tests passed. - [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-11-30 19:03:03 +03:00
for orth in ["d'", "D'"]:
attempt a port from #1147
2017-06-26 23:24:55 +03:00
_exc[orth] = [{ORTH: orth}]
Tidy up and move noun_chunks, token_match, url_match
2020-07-22 23:18:46 +03:00
TOKENIZER_EXCEPTIONS = update_exc(BASE_EXCEPTIONS, _exc)
Reference in New Issue Copy Permalink