* make merge more efficient
* fix offsets
* merge works with relative indices
* remove printing
* Add the SCA
* fix SCA date
* more cythonize _retokenize.pyx
* more cythonize _retokenize.pyx
* fix only declaration in _retokenize.pyx
* switch back to absolute head
* switch back to absolute head
* fix comment
* merge from origin repo
* Added RONEC to spaCy Universe
* Added contributor file
* Corrected date from .github/contributors/avramandrei.md
* Convert tabs to spaces
* Remove duplicate keys
Can only have one GitHub link unfortunately
* Also add models category
* Adjust ID
This is used to generate the URL, so a simpler string is better
* customizable template for entities display, allowing to pass additional parameters along each entity
* contributor agreement
* simpler naming for the additional parameters given to the span entities renderer
Co-Authored-By: Ines Montani <ines@ines.io>
* change of default parameter, as suggested
Co-Authored-By: Ines Montani <ines@ines.io>
* Check whether two entities overlap
- biluo_gold_biluo_overlap now throw exception when entities passed in have overlaps
- added unit test
* SCA agreement
* pytest file for issue4104 established
* edited default lookup english lemmatizer for spun; fixes issue 4102
* eliminated parameterization and sorted dictionary dependnency in issue 4104 test
* added contributor agreement
* Add entry for Blackstone in universe.json
Add an entry for the Blackstone project. Checked JSON is valid.
* Create ICLRandD.md
* Fix indentation (tabs to spaces)
It looks like during validation, the JSON file automatically changed spaces to tabs. This caused the diff to show *everything* as changed, which is obviously not true. This hopefully fixes that.
* Try to fix formatting for diff
* Fix diff
Co-authored-by: Ines Montani <ines@ines.io>
* Update gold corpus code to properly ingest a directory of jsonlines files
In response to: https://github.com/explosion/spaCy/issues/3975
* Update spacy/gold.pyx
Co-Authored-By: Ines Montani <ines@ines.io>
* Typo fix for AllenAI url
Changed incorrect home page url for AllenAI from appenai.org to allenai.org
* Sign contributor agreement
* Change date format
* Perserve flags in EntityRuler
The EntityRuler (explosion/spaCy#3526) does not preserve
overwrite flags (or `ent_id_sep`) when serialized. This
commit adds support for serialization/deserialization preserving
overwrite and ent_id_sep flags.
* add signed contributor agreement
* flake8 cleanup
mostly blank line issues.
* mark test from the issue as needing a model
The test from the issue needs some language model for serialization
but the test wasn't originally marked correctly.
* remove unneeded model loading
The model didn't need to be loaded, and I replaced it with
a change that doesn't require it (using existings fixtures)
* change tempdir handling to be compatible with python 2.7
* Adds code to handle item saved before this change.
This code chanes how the save files are handled and how the bytes
are stored as well. This code adds check to dispatch correctly
if it encounters bytes or files saved in the old format (and tests
for those cases).
* use util function for tempdir management
Updated after PR comments: this code now uses the make_tempdir function from util
instead of doing it by hand.
* Norwegian fix
Add support for alternative past tense verb form (vaska).
* Norwegian months
Add all Norwegian months to tokenizer excpetions.
* More Norwegian abbreviations
Add more Norwegian abbreviations to tokenizer_exceptions.
* Contributor agreement khellan
Add signed contributor agreement for khellan (Knut O. Hellan).
* initial LT lang support
* Added more stopwords. Started setting up some basic test environment (not complete)
* Initial morph rules for LT lang
* Closes#1 Adds tokenizer exceptions for Lithuanian
* Closes#5 Punctuation rules. Closes#6 Lexical Attributes
* test: add native examples to basic tests
* feat: add tag map for lt lang
* fix: remove undefined tag attribute 'Definite'
* feat: add lemmatizer for lt lang
* refactor: add new instances to lt lang morph rules; use tags from tag map
* refactor: add morph rules to lt lang defaults
* refactor: only keep nouns, verbs, adverbs and adjectives in lt lang lemmatizer lookup
* refactor: add capitalized words to lt lang lemmatizer
* refactor: add more num words to lt lang lex attrs
* refactor: update lt lang stop word set
* refactor: add new instances to lt lang tokenizer exceptions
* refactor: remove comments form lt lang init file
* refactor: use function instead of lambda in lt lex lang getter
* refactor: remove conversion to dict in lt init when dict is already provided
* chore: rename lt 'test_basic' to 'test_text'
* feat: add more lt text tests
* feat: add lemmatizer tests
* refactor: remove unused imports, add newline to end of file
* chore: add contributor agreement
* chore: change 'en' to 'lt' in lt example description
* fix: add missing encoding info
* style: add newline to end of file
* refactor: use python2 compatible syntax
* style: reformat code using black
* Changed learning rate by its param name.
I've been searching for a while how the parameter learning rate was named, with `beta1` and `beta2` its easy as they are marked as code, but learning rate wasn't. I think writing the actual parameter name would be helpful.
* Signing SCA
* Update tokenizer.md for construction example
Self contained example. You should really say what nlp is so that the example will work as is
* Update CONTRIBUTOR_AGREEMENT.md
* Restore contributor agreement
* Adjust construction examples
* Adding support for entity_id in EntityRuler pipeline component
* Adding Spacy Contributor aggreement
* Updating EntityRuler to use string.format instead of f strings
* Update Entity Ruler to support an 'id' attribute per pattern that explicitly identifies an entity.
* Fixing tests
* Remove custom extension entity_id and use built in ent_id token attribute.
* Changing entity_id to ent_id for consistent naming
* entity_ids => ent_ids
* Removing kb, cleaning up tests, making util functions private, use rsplit instead of split
* Adding Marathi language details and folder to it
* Adding few changes and running tests
* Adding few changes and running tests
* Update __init__.py
mh -> mr
* Rename spacy/lang/mh/__init__.py to spacy/lang/mr/__init__.py
* mh -> mr
* Request to add Holmes to spaCy Universe
Dear spaCy team, I would be grateful if you would consider my Python library Holmes for inclusion in the spaCy Universe. Holmes transforms the syntactic structures delivered by spaCy into semantic structures that, together with various other techniques including ontological matching and word embeddings, serve as the basis for information extraction. Holmes supports several use cases including chatbot, structured search, topic matching and supervised document classification. I had the basic idea for Holmes around 15 years ago and now spaCy has made it possible to build an implementation that is stable and fast enough to actually be of use - thank you! At present Holmes supports English and German (I am based in Munich) but could easily be extended to support any other language with a spaCy model.
* Added
<!--- Provide a general summary of your changes in the title. -->
## Description
<!--- Use this section to describe your changes. If your changes required
testing, include information about the testing environment and the tests you
ran. If your test fixes a bug reported in an issue, don't forget to include the
issue number. If your PR is still a work in progress, that's totally fine – just
include a note to let us know. -->
### Types of change
<!-- What type of change does your PR cover? Is it a bug fix, an enhancement
or new feature, or a change to the documentation? -->
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I have submitted the spaCy Contributor Agreement.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [ ] I have submitted the spaCy Contributor Agreement.
- [ ] I ran the tests, and all new and existing tests passed.
- [ ] My changes don't require a change to the documentation, or if they do, I've added all required information.
Co-authored-by: Ines Montani <ines@ines.io>
<!--- Provide a general summary of your changes in the title. -->
## Description
<!--- Use this section to describe your changes. If your changes required
testing, include information about the testing environment and the tests you
ran. If your test fixes a bug reported in an issue, don't forget to include the
issue number. If your PR is still a work in progress, that's totally fine – just
include a note to let us know. -->
Fix a bug in the test of JapaneseTokenizer.
This PR may require @polm's review.
### Types of change
<!-- What type of change does your PR cover? Is it a bug fix, an enhancement
or new feature, or a change to the documentation? -->
Bug fix
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I have submitted the spaCy Contributor Agreement.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
* fix(util): fix decaying function output
* fix(util): better test and adhere to code standards
* fix(util): correct variable name, pytestify test, update website text
* merging conllu/conll and conllubio scripts
* tabs to spaces
* removing conllubio2json from converters/__init__.py
* Move not-really-CLI tests to misc
* Add converter test using no-ud data
* Fix test I broke
* removing include_biluo parameter
* fixing read_conllx
* remove include_biluo from convert.py
* Improved stop words list
* Removed some wrong stop words form list
* Improved stop words list
* Removed some wrong stop words form list
* Improved Polish Tokenizer (#38)
* Add tests for polish tokenizer
* Add polish tokenizer exceptions
* Don't split any words containing hyphens
* Fix test case with wrong model answer
* Remove commented out line of code until better solution is found
* Add source srx' license
* Rename exception_list.py to match spaCy conventionality
* Add a brief explanation of where the exception list comes from
* Add newline after reach exception
* Rename COPYING.txt to LICENSE
* Delete old files
* Add header to the license
* Agreements signed
* Stanisław Giziński agreement
* Krzysztof Kowalczyk - signed agreement
* Mateusz Olko agreement
* Add DoomCoder's contributor agreement
* Improve like number checking in polish lang
* like num tests added
* all from SI system added
* Final licence and removed splitting exceptions
* Added polish stop words to LEX_ATTRA
* Add encoding info to pl tokenizer exceptions
* Added the same punctuation rules as danish language.
* Added abbreviations and also the possibility to have capitalized abbreviations on some. Added a few specific cases too
* Added test for long texts in swedish
* Added morph rules, infixes and suffixes to __init__.py for swedish
* Added some tests for prefixes, infixes and suffixes
* Added tests for lemma
* Renamed files to follow convention
* [sv] Removed ambigious abbreviations
* Added more tests for tokenizer exceptions
* Added test for problem with punctuation in issue #2578
* Contributor agreement
* Removed faulty lemmatization of 'jag' ('I') as it was lemmatized to 'jaga' ('hunt')
Tamil language support to spaCy
Description
Hereby, creating new PR to add support for Tamil language in spaCy
added stop words, examples and numerical attributes
<--Working on other language data-->
Types of change
Enhancement
Checklist
[ x] I have submitted the spaCy Contributor Agreement.
[x ] I ran the tests, and all new and existing tests passed.
[ x] My changes don't require a change to the documentation, or if they do, I've added all required information.
* Update call to `mkdir()` to create the parents
- Update the call to `output_dir.mkdir()` to also create the parents if needed
* don't automatically create parents but fail fast if cannot create directory
* add signed contributors agreement for retnuh
* adding adverbs and irregular cases for empty words
* adding adverbs and irregular cases for empty words
* adding adverbs and irregular cases for empty words
* updating contributor agreement for amperinet
* modifying French lookup that contained wrong lemmas
* correcting wrong line breaks on hyphen
* adding contributor agreement for amperinet@
* correcting a typo
* Test on #2396: bug in Doc.get_lca_matrix()
* reimplementation of Doc.get_lca_matrix(), (closes#2396)
* reimplement Span.get_lca_matrix(), and call it from Doc.get_lca_matrix()
* tests Span.get_lca_matrix() as well as Doc.get_lca_matrix()
* implement _get_lca_matrix as a helper function in doc.pyx; call it from Doc.get_lca_matrix and Span.get_lca_matrix
* use memory view instead of np.ndarray in _get_lca_matrix (faster)
* fix bug when calling Span.get_lca_matrix; return lca matrix as np.array instead of memoryview
* cleaner conditional, add comment
* Test on #2396: bug in Doc.get_lca_matrix()
* reimplementation of Doc.get_lca_matrix(), (closes#2396)
* reimplement Span.get_lca_matrix(), and call it from Doc.get_lca_matrix()
* tests Span.get_lca_matrix() as well as Doc.get_lca_matrix()
* implement _get_lca_matrix as a helper function in doc.pyx; call it from Doc.get_lca_matrix and Span.get_lca_matrix
* use memory view instead of np.ndarray in _get_lca_matrix (faster)
* fix bug when calling Span.get_lca_matrix; return lca matrix as np.array instead of memoryview
* cleaner conditional, add comment
* issue #3012: add test
* add contributor aggreement
* Make test work without models and fix typos
ten.pos_ instead of ten.orth_ and comparison against "10" instead of integer 10
I have added alpha support for the Tagalog language from the Philippines. It is the basis for the country's national language Filipino. I have heavily based the format to the EN and ES languages.
I have provided several words in the lemmatizer lookup table, added stop words from a source, translated numeric words to its Tagalog counterpart, added some tokenizer exceptions, and kept the tag map the same as the English language.
While the alpha language passed the preliminary testing that you provided, I think it needs more data to be useful for most cases.
* Added alpha support for Tagalog language
* Edited contributor template
* Included SCA; Reverted templates
* Fixed SCA template
* Fixed changes in SCA template
* Upadate Unidic link for latest version in document
This patch improves #3017 . The link for Unidic was old version one, so will the lates version.
* Add contributor agreement
* Use more specific link for unidic-cwj
* updating rules and vocabulary for French lemmatization of verbs
* updating the file with French auxiliary verb
* updating rules and vocabulary for French lemmatization of verbs
* adding contributor agreement for amperinet
* adding rules for words with inclusive parentheses wrongly tokenized
* Updated wordforms for Norwegian lemmatizer
Upload of updated lists of wordforms for the Norwegian lemmatizer (nouns, verbs, adverbs, adjectives and lookup).
* Add spaCy contributor agreement for user beatesi
* Updated wordforms for Norwegian lemmatizer
* additional unit test for new entr word not in other lists
* bugfix - unit test works
* use _latin_lower instead of alpha_lower for french
* revert back to ALPHA_LOWER (following the code for languages)
* contributor agreement
* fixes symbolic link on py3 and windows
during setup of spacy using command
python -m spacy link en_core_web_sm en
closes#2948
* Update spacy/compat.py
Co-Authored-By: cicorias <cicorias@users.noreply.github.com>
Some documentation improvements
## Description
1. Fixed the dead URL to joblib
2. Fixed Stack Overflow brand name (with space)
### Types of change
Documentation
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I have submitted the spaCy Contributor Agreement.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
* Correct some grammatical inaccuracies in lang\ru\examples.py; filled Contributor Agreement
* Correct some grammatical inaccuracies in lang\ru\examples.py
* Move contributor agreement to separate file
* Fix error
ValueError: cannot resize an array that references or is referenced
by another array in this way. Use the resize function
* added spaCy Contributor Agreement
* adding e-KTP in tokenizer exceptions list
* add exception token
* removing lines with containing space as it won't matter since we use .split() method in the end, added new tokens in exception
* add tokenizer exceptions list
* combining base_norms with norm_exceptions
* adding norm_exception
* fix double key in lemmatizer
* remove unused import on punctuation.py
* reformat stop_words to reduce number of lines, improve readibility
* updating tokenizer exception
* implement is_currency for lang/id
* adding orth_first_upper in tokenizer_exceptions
* update the norm_exception list
* remove bunch of abbreviations
* adding contributors file
