2017-03-12 15:53:46 +03:00
|
|
|
|
# coding: utf8
|
2016-12-18 17:35:36 +03:00
|
|
|
|
from __future__ import unicode_literals
|
|
|
|
|
|
|
|
|
|
|
2017-03-12 15:53:46 +03:00
|
|
|
|
# Stop words
|
💫 Tidy up and auto-format .py files (#2983)
<!--- Provide a general summary of your changes in the title. -->
## Description
- [x] Use [`black`](https://github.com/ambv/black) to auto-format all `.py` files.
- [x] Update flake8 config to exclude very large files (lemmatization tables etc.)
- [x] Update code to be compatible with flake8 rules
- [x] Fix various small bugs, inconsistencies and messy stuff in the language data
- [x] Update docs to explain new code style (`black`, `flake8`, when to use `# fmt: off` and `# fmt: on` and what `# noqa` means)
Once #2932 is merged, which auto-formats and tidies up the CLI, we'll be able to run `flake8 spacy` actually get meaningful results.
At the moment, the code style and linting isn't applied automatically, but I'm hoping that the new [GitHub Actions](https://github.com/features/actions) will let us auto-format pull requests and post comments with relevant linting information.
### Types of change
enhancement, code style
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I have submitted the spaCy Contributor Agreement.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-11-30 19:03:03 +03:00
|
|
|
|
STOP_WORDS = set(
|
|
|
|
|
"""
|
2016-12-18 17:35:36 +03:00
|
|
|
|
a about above across after afterwards again against all almost alone along
|
|
|
|
|
already also although always am among amongst amount an and another any anyhow
|
|
|
|
|
anyone anything anyway anywhere are around as at
|
|
|
|
|
|
|
|
|
|
back be became because become becomes becoming been before beforehand behind
|
|
|
|
|
being below beside besides between beyond both bottom but by
|
|
|
|
|
|
|
|
|
|
call can cannot ca could
|
|
|
|
|
|
|
|
|
|
did do does doing done down due during
|
|
|
|
|
|
2017-07-03 16:41:59 +03:00
|
|
|
|
each eight either eleven else elsewhere empty enough even ever every
|
2016-12-18 17:35:36 +03:00
|
|
|
|
everyone everything everywhere except
|
|
|
|
|
|
|
|
|
|
few fifteen fifty first five for former formerly forty four from front full
|
|
|
|
|
further
|
|
|
|
|
|
|
|
|
|
get give go
|
|
|
|
|
|
|
|
|
|
had has have he hence her here hereafter hereby herein hereupon hers herself
|
|
|
|
|
him himself his how however hundred
|
|
|
|
|
|
2017-07-03 16:41:59 +03:00
|
|
|
|
i if in indeed into is it its itself
|
2016-12-18 17:35:36 +03:00
|
|
|
|
|
|
|
|
|
keep
|
|
|
|
|
|
|
|
|
|
last latter latterly least less
|
|
|
|
|
|
|
|
|
|
just
|
|
|
|
|
|
|
|
|
|
made make many may me meanwhile might mine more moreover most mostly move much
|
|
|
|
|
must my myself
|
|
|
|
|
|
|
|
|
|
name namely neither never nevertheless next nine no nobody none noone nor not
|
2019-04-09 12:40:19 +03:00
|
|
|
|
nothing now nowhere
|
2016-12-18 17:35:36 +03:00
|
|
|
|
|
|
|
|
|
of off often on once one only onto or other others otherwise our ours ourselves
|
|
|
|
|
out over own
|
|
|
|
|
|
|
|
|
|
part per perhaps please put
|
|
|
|
|
|
|
|
|
|
quite
|
|
|
|
|
|
|
|
|
|
rather re really regarding
|
|
|
|
|
|
|
|
|
|
same say see seem seemed seeming seems serious several she should show side
|
|
|
|
|
since six sixty so some somehow someone something sometime sometimes somewhere
|
|
|
|
|
still such
|
|
|
|
|
|
|
|
|
|
take ten than that the their them themselves then thence there thereafter
|
|
|
|
|
thereby therefore therein thereupon these they third this those though three
|
|
|
|
|
through throughout thru thus to together too top toward towards twelve twenty
|
|
|
|
|
two
|
|
|
|
|
|
|
|
|
|
under until up unless upon us used using
|
|
|
|
|
|
|
|
|
|
various very very via was we well were what whatever when whence whenever where
|
|
|
|
|
whereafter whereas whereby wherein whereupon wherever whether which while
|
|
|
|
|
whither who whoever whole whom whose why will with within without would
|
|
|
|
|
|
|
|
|
|
yet you your yours yourself yourselves
|
💫 Tidy up and auto-format .py files (#2983)
<!--- Provide a general summary of your changes in the title. -->
## Description
- [x] Use [`black`](https://github.com/ambv/black) to auto-format all `.py` files.
- [x] Update flake8 config to exclude very large files (lemmatization tables etc.)
- [x] Update code to be compatible with flake8 rules
- [x] Fix various small bugs, inconsistencies and messy stuff in the language data
- [x] Update docs to explain new code style (`black`, `flake8`, when to use `# fmt: off` and `# fmt: on` and what `# noqa` means)
Once #2932 is merged, which auto-formats and tidies up the CLI, we'll be able to run `flake8 spacy` actually get meaningful results.
At the moment, the code style and linting isn't applied automatically, but I'm hoping that the new [GitHub Actions](https://github.com/features/actions) will let us auto-format pull requests and post comments with relevant linting information.
### Types of change
enhancement, code style
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I have submitted the spaCy Contributor Agreement.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-11-30 19:03:03 +03:00
|
|
|
|
""".split()
|
|
|
|
|
)
|
2019-04-02 14:24:59 +03:00
|
|
|
|
|
2019-04-03 14:50:33 +03:00
|
|
|
|
contractions = ["n't", "'d", "'ll", "'m", "'re", "'s", "'ve"]
|
|
|
|
|
STOP_WORDS.update(contractions)
|
|
|
|
|
|
|
|
|
|
for apostrophe in ["‘", "’"]:
|
|
|
|
|
for stopword in contractions:
|
|
|
|
|
STOP_WORDS.add(stopword.replace("'", apostrophe))
|