* Override language defaults for null token and URL match
When the serialized `token_match` or `url_match` is `None`, override the
language defaults to preserve `None` on deserialization.
* Fix fixtures in tests
* Update stop_words.py
Added three aditional stopwords: "a" and "o" that means "the", and "e" that means "and"
* Create cristianasp.md
* zero edit to push CI
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
* add syntax iterators for danish
* add test noun chunks for danish syntax iterators
* add contributor agreement
* update da syntax iterators to remove nested chunks
* add tests for da noun chunks
* Fix test
* add missing import
* fix example
* Prevent overlapping noun chunks
Prevent overlapping noun chunks by tracking the end index of the
previous noun chunk span.
Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
* Add Amharic to space
* clean up
* Add some PRON_LEMMA
* add Tigrinya support
* remove text_noun_chunks
* Tigrinya Support
* added some more details for ti
* fix unit test
* add amharic char range
* changes from review
* amharic and tigrinya share same unicode block
* get rid of _amharic/_tigrinya in char_classes
Co-authored-by: Josiah Solomon <jsolomon@meteorcomm.com>
* Remove blis version constraints
After updating the blis sdist in v0.7.4, remove python version
constraints for blis build and install dependencies.
* Install sdist with --prefer-binary for python 3.5
* Fix duplicate sdist install steps
* Fix sdist install step types
* Fix blis pins in requirements.txt
* Remove wheel hack for python 3.5 from CI
Remove the non-working `--use-chars` option from the train CLI. The
implementation of the option across component types and the CLI settings
could be fixed, but the `CharacterEmbed` model does not work on GPU in
v2 so it's better to remove it.
* Fix blis build dependencies
* Add blis with python_version constraints to pyproject.toml
* Add blis to setup_requires
* Remove --only-binary from CI
* Reduce number of builds to speed up CI
* Add hack to install wheel for python 3.5 in linux
* Remove os spec from CI
* Remove detailed numpy build constraints
* Remove detailed numpy build constraints from `pyproject.toml` because
it is too difficult to maintain for many architectures
* These constraints are more a reflection of what is available on
pypi as binary wheels rather than any real build requirements that
it is necessary for users to follow when building from source
* Users building their own binary packages will need to enforce the
constraints that make sense in their environments, e.g., the `conda`
compatible numpy pins
* Keep the build constraints in `build-constraints.txt` for use with our
builds
* Our builds with wheelwright are built against the earliest
compatible binary versions of numpy on pypi
* These constraints are documented within the distribution
* Revert "Remove os spec from CI"
This reverts commit 7489476688.
* Only set NORM on Token in retokenizer
Instead of setting `NORM` on both the token and lexeme, set `NORM` only
on the token.
The retokenizer tries to set all possible attributes with
`Token/Lexeme.set_struct_attr` so that it doesn't have to enumerate
which attributes are available for each. `NORM` is the only attribute
that's stored on both and for most cases it doesn't make sense to set
the global norms based on a individual retokenization. For lexeme-only
attributes like `IS_STOP` there's no way to avoid the global side
effects, but I think that `NORM` would be better only on the token.
* Fix test
