* Extending debug-data with dependency checks, etc.
* Modify debug-data to load with GoldCorpus to iterate over .json/.jsonl
files within directories
* Add GoldCorpus iterator train_docs_without_preprocessing to load
original train docs without shuffling and projectivizing
* Report number of misaligned tokens
* Add more dependency checks and messages
* Update spacy/cli/debug_data.py
Co-Authored-By: Ines Montani <ines@ines.io>
* Fixed conflict
* Move counts to _compile_gold()
* Move all dependency nonproj/sent/head/cycle counting to
_compile_gold()
* Unclobber previous merges
* Update variable names
* Update more variable names, fix misspelling
* Don't clobber loading error messages
* Only warn about misaligned tokens if present
* Check whether two entities overlap
- biluo_gold_biluo_overlap now throw exception when entities passed in have overlaps
- added unit test
* SCA agreement
Provide the tokens in the cycle and the first 50 tokens from document in
the error message so it's easier to track down the location of the cycle
in the data.
Addresses feature request in #3698.
* pytest file for issue4104 established
* edited default lookup english lemmatizer for spun; fixes issue 4102
* eliminated parameterization and sorted dictionary dependnency in issue 4104 test
* added contributor agreement
* document token ent_kb_id
* document span kb_id
* update pipeline documentation
* prior and context weights as bool's instead
* entitylinker api documentation
* drop for both models
* finish entitylinker documentation
* small fixes
* documentation for KB
* candidate documentation
* links to api pages in code
* small fix
* frequency examples as counts for consistency
* consistent documentation about tensors returned by predict
* add entity linking to usage 101
* add entity linking infobox and KB section to 101
* entity-linking in linguistic features
* small typo corrections
* training example and docs for entity_linker
* predefined nlp and kb
* revert back to similarity encodings for simplicity (for now)
* set prior probabilities to 0 when excluded
* code clean up
* bugfix: deleting kb ID from tokens when entities were removed
* refactor train el example to use either model or vocab
* pretrain_kb example for example kb generation
* add to training docs for KB + EL example scripts
* small fixes
* error numbering
* ensure the language of vocab and nlp stay consistent across serialization
* equality with =
* avoid conflict in errors file
* add error 151
* final adjustements to the train scripts - consistency
* update of goldparse documentation
* small corrections
* push commit
* turn kb_creator into CLI script (wip)
* proper parameters for training entity vectors
* wikidata pipeline split up into two executable scripts
* remove context_width
* move wikidata scripts in bin directory, remove old dummy script
* refine KB script with logs and preprocessing options
* small edits
* small improvements to logging of EL CLI script
* Update gold corpus code to properly ingest a directory of jsonlines files
In response to: https://github.com/explosion/spaCy/issues/3975
* Update spacy/gold.pyx
Co-Authored-By: Ines Montani <ines@ines.io>
* Improve NER per type scoring
* include all gold labels in per type scoring, not only when recall > 0
* improve efficiency of per type scoring
* Create Scorer tests, initially with NER tests
* move regression test #3968 (per type NER scoring) to Scorer tests
* add new test for per type NER scoring with imperfect P/R/F and per
type P/R/F including a case where R == 0.0
* Improve error message when model.from_bytes() dies
When Thinc's model.from_bytes() is called with a mismatched model, often
we get a particularly ungraceful error,
e.g. "AttributeError: FunctionLayer has no attribute G"
This is because we're trying to load the parameters for something like
a LayerNorm layer, and the model architecture has some other layer there
instead. This is obviously terrible, especially since the error *type*
is wrong.
I've changed it to raise a ValueError. The error message is still
probably a bit terse, but it's hard to be sure exactly what's gone
wrong.
* Update spacy/pipeline/pipes.pyx
* Update spacy/pipeline/pipes.pyx
* Update spacy/pipeline/pipes.pyx
* Update spacy/syntax/nn_parser.pyx
* Update spacy/syntax/nn_parser.pyx
* Update spacy/pipeline/pipes.pyx
Co-Authored-By: Matthew Honnibal <honnibal+gh@gmail.com>
* Update spacy/pipeline/pipes.pyx
Co-Authored-By: Matthew Honnibal <honnibal+gh@gmail.com>
Co-authored-by: Ines Montani <ines@ines.io>
* failing unit test for issue 3962
* attempt to fix Issue #3962
* create artificial unit test example
* using length instead of self.length
* sp
* reformat with black
* find better ancestor within span and use generic 'dep'
* attach to span.root if there is no appropriate ancestor
* comment span text
* clean up ancestor code
* reconstruct dep tree to keep same number of sentences
