explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-11-11 20:28:20 +03:00
Code Issues Packages Projects Releases Wiki Activity
67652bcbb5
spaCy/spacy/lang/ga/tokenizer_exceptions.py

71 lines
1.7 KiB
Python
Raw Normal View History

Tidy up and move noun_chunks, token_match, url_match
2020-07-22 23:18:46 +03:00
from ..tokenizer_exceptions import BASE_EXCEPTIONS
Remove POS, TAG and LEMMA from tokenizer exceptions
2020-07-23 00:09:01 +03:00
from ...symbols import ORTH, NORM
Tidy up and move noun_chunks, token_match, url_match
2020-07-22 23:18:46 +03:00
from ...util import update_exc
attempt a port from #1147
2017-06-26 23:24:55 +03:00
_exc = {
Remove POS, TAG and LEMMA from tokenizer exceptions
2020-07-23 00:09:01 +03:00
"'acha'n": [{ORTH: "'ach", NORM: "gach"}, {ORTH: "a'n", NORM: "aon"}],
"dem'": [{ORTH: "de", NORM: "de"}, {ORTH: "m'", NORM: "mo"}],
"ded'": [{ORTH: "de", NORM: "de"}, {ORTH: "d'", NORM: "do"}],
"lem'": [{ORTH: "le", NORM: "le"}, {ORTH: "m'", NORM: "mo"}],
"led'": [{ORTH: "le", NORM: "le"}, {ORTH: "d'", NORM: "do"}],
attempt a port from #1147
2017-06-26 23:24:55 +03:00
}
for exc_data in [
Remove POS, TAG and LEMMA from tokenizer exceptions
2020-07-23 00:09:01 +03:00
{ORTH: "'gus", NORM: "agus"},
{ORTH: "'ach", NORM: "gach"},
{ORTH: "ao'", NORM: "aon"},
{ORTH: "'niar", NORM: "aniar"},
{ORTH: "'níos", NORM: "aníos"},
{ORTH: "'ndiu", NORM: "inniu"},
{ORTH: "'nocht", NORM: "anocht"},
{ORTH: "m'"},
{ORTH: "Aib."},
{ORTH: "Ath."},
{ORTH: "Beal."},
{ORTH: "a.C.n."},
{ORTH: "m.sh."},
{ORTH: "M.F."},
{ORTH: "M.Fómh."},
{ORTH: "D.F."},
{ORTH: "D.Fómh."},
{ORTH: "r.C."},
{ORTH: "R.C."},
{ORTH: "r.Ch."},
{ORTH: "r.Chr."},
{ORTH: "R.Ch."},
{ORTH: "R.Chr."},
{ORTH: "⁊rl."},
{ORTH: "srl."},
{ORTH: "Co."},
{ORTH: "Ean."},
{ORTH: "Feab."},
{ORTH: "gCo."},
{ORTH: ".i."},
{ORTH: "B'"},
{ORTH: "b'"},
{ORTH: "lch."},
{ORTH: "Lch."},
{ORTH: "lgh."},
{ORTH: "Lgh."},
{ORTH: "Lún."},
{ORTH: "Már."},
{ORTH: "Meith."},
{ORTH: "Noll."},
{ORTH: "Samh."},
{ORTH: "tAth."},
{ORTH: "tUas."},
{ORTH: "teo."},
{ORTH: "Teo."},
{ORTH: "Uas."},
{ORTH: "uimh."},
{ORTH: "Uimh."},
💫 Tidy up and auto-format .py files (#2983) <!--- Provide a general summary of your changes in the title. --> ## Description - [x] Use [`black`](https://github.com/ambv/black) to auto-format all `.py` files. - [x] Update flake8 config to exclude very large files (lemmatization tables etc.) - [x] Update code to be compatible with flake8 rules - [x] Fix various small bugs, inconsistencies and messy stuff in the language data - [x] Update docs to explain new code style (`black`, `flake8`, when to use `# fmt: off` and `# fmt: on` and what `# noqa` means) Once #2932 is merged, which auto-formats and tidies up the CLI, we'll be able to run `flake8 spacy` actually get meaningful results. At the moment, the code style and linting isn't applied automatically, but I'm hoping that the new [GitHub Actions](https://github.com/features/actions) will let us auto-format pull requests and post comments with relevant linting information. ### Types of change enhancement, code style ## Checklist <!--- Before you submit the PR, go over this checklist and make sure you can tick off all the boxes. [] -> [x] --> - [x] I have submitted the spaCy Contributor Agreement. - [x] I ran the tests, and all new and existing tests passed. - [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-11-30 19:03:03 +03:00
]:
Remove comma that caused list to wrap in tuple! Also removed extra dict wrappings for performance (we used to have them in there, but they should only really exist if copying the dict is absolutely necessary)
2017-10-31 22:13:16 +03:00
_exc[exc_data[ORTH]] = [exc_data]
attempt a port from #1147
2017-06-26 23:24:55 +03:00
💫 Tidy up and auto-format .py files (#2983) <!--- Provide a general summary of your changes in the title. --> ## Description - [x] Use [`black`](https://github.com/ambv/black) to auto-format all `.py` files. - [x] Update flake8 config to exclude very large files (lemmatization tables etc.) - [x] Update code to be compatible with flake8 rules - [x] Fix various small bugs, inconsistencies and messy stuff in the language data - [x] Update docs to explain new code style (`black`, `flake8`, when to use `# fmt: off` and `# fmt: on` and what `# noqa` means) Once #2932 is merged, which auto-formats and tidies up the CLI, we'll be able to run `flake8 spacy` actually get meaningful results. At the moment, the code style and linting isn't applied automatically, but I'm hoping that the new [GitHub Actions](https://github.com/features/actions) will let us auto-format pull requests and post comments with relevant linting information. ### Types of change enhancement, code style ## Checklist <!--- Before you submit the PR, go over this checklist and make sure you can tick off all the boxes. [] -> [x] --> - [x] I have submitted the spaCy Contributor Agreement. - [x] I ran the tests, and all new and existing tests passed. - [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-11-30 19:03:03 +03:00
for orth in ["d'", "D'"]:
attempt a port from #1147
2017-06-26 23:24:55 +03:00
_exc[orth] = [{ORTH: orth}]
Tidy up and move noun_chunks, token_match, url_match
2020-07-22 23:18:46 +03:00
TOKENIZER_EXCEPTIONS = update_exc(BASE_EXCEPTIONS, _exc)
Reference in New Issue Copy Permalink