Remove corpus-specific tag maps from the language data for languages
without custom tokenizers. For languages with custom word segmenters
that also provide tags (Japanese and Korean), the tag maps for the
custom tokenizers are kept as the default.
The default tag maps for languages without custom tokenizers are now the
default tag map from `lang/tag_map/py`, UPOS -> UPOS.
* Add morph to morphology in Doc.from_array
Add morphological analyses to morphology table in `Doc.from_array`.
* Use separate vocab in DocBin roundtrip test
* adding debug-model to print the internals for debugging purposes
* expend debug-model script with 4 stages: before, init, train, predict
* avoid enforcing to have a seed in the train script
* small fixes
* Update project CLI hashes, directories, skipping
* Improve clone success message
* Remove unused context args
* Move project-specific utils to project utils
The hashing/checksum functions may not end up being general-purpose functions and are more designed for the projects, so they shouldn't live in spacy.util
* Improve run help and add workflows
* Add note re: directory checksum speed
* Fix cloning from subdirectories and output messages
* Remove hard-coded dirs
* add keyword separator for update functions and drop unused "state"
* few more Example tests and various small fixes
* consistently return losses after update call
* eliminate unused tensors field across pipe components
* fix name
* fix arg name
* Add initial reproducibility tests
* failing test for default_text_classifier (WIP)
* track trouble to underlying tok2vec layer
* add regression test for Issue 5551
* tests go green with https://github.com/explosion/thinc/pull/359
* update test
* adding fixed seeds to HashEmbed layers, seems to fix the reproducility issue
Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
* Make project command a submodule
* Update with WIP
* Add helper for joining commands
* Update docstrins, formatting and types
* Update assets and add support for copying local files
* Fix type
* Update success messages
* Fix get_loss for None alignments in senter
When converting the `sent_start` values back to `SentenceRecognizer`
labels, handle `None` alignments.
* Handle SENT_START as -1
Handle SENT_START as -1 (or -1 converted to uint64) by treating any
values other than 1 the same as 0 in `SentenceRecognizer.get_loss`.
