mirror of
https://github.com/explosion/spaCy.git
synced 2024-12-27 10:26:35 +03:00
b5d999e510
* Add doc.cats to spacy.gold at the paragraph level
Support `doc.cats` as `"cats": [{"label": string, "value": number}]` in
the spacy JSON training format at the paragraph level.
* `spacy.gold.docs_to_json()` writes `docs.cats`
* `GoldCorpus` reads in cats in each `GoldParse`
* Update instances of gold_tuples to handle cats
Update iteration over gold_tuples / gold_parses to handle addition of
cats at the paragraph level.
* Add textcat to train CLI
* Add textcat options to train CLI
* Add textcat labels in `TextCategorizer.begin_training()`
* Add textcat evaluation to `Scorer`:
* For binary exclusive classes with provided label: F1 for label
* For 2+ exclusive classes: F1 macro average
* For multilabel (not exclusive): ROC AUC macro average (currently
relying on sklearn)
* Provide user info on textcat evaluation settings, potential
incompatibilities
* Provide pipeline to Scorer in `Language.evaluate` for textcat config
* Customize train CLI output to include only metrics relevant to current
pipeline
* Add textcat evaluation to evaluate CLI
* Fix handling of unset arguments and config params
Fix handling of unset arguments and model confiug parameters in Scorer
initialization.
* Temporarily add sklearn requirement
* Remove sklearn version number
* Improve Scorer handling of models without textcats
* Fixing Scorer handling of models without textcats
* Update Scorer output for python 2.7
* Modify inf in Scorer for python 2.7
* Auto-format
Also make small adjustments to make auto-formatting with black easier and produce nicer results
* Move error message to Errors
* Update documentation
* Add cats to annotation JSON format [ci skip]
* Fix tpl flag and docs [ci skip]
* Switch to internal roc_auc_score
Switch to internal `roc_auc_score()` adapted from scikit-learn.
* Add AUCROCScore tests and improve errors/warnings
* Add tests for AUCROCScore and roc_auc_score
* Add missing error for only positive/negative values
* Remove unnecessary warnings and errors
* Make reduced roc_auc_score functions private
Because most of the checks and warnings have been stripped for the
internal functions and access is only intended through `ROCAUCScore`,
make the functions for roc_auc_score adapted from scikit-learn private.
* Check that data corresponds with multilabel flag
Check that the training instances correspond with the multilabel flag,
adding the multilabel flag if required.
* Add textcat score to early stopping check
* Add more checks to debug-data for textcat
* Add example training data for textcat
* Add more checks to textcat train CLI
* Check configuration when extending base model
* Fix typos
* Update textcat example data
* Provide licensing details and licenses for data
* Remove two labels with no positive instances from jigsaw-toxic-comment
data.
Co-authored-by: Ines Montani <ines@ines.io>
35 lines
1.0 KiB
Markdown
35 lines
1.0 KiB
Markdown
## Examples of textcat training data
|
|
|
|
spacy JSON training files were generated from JSONL with:
|
|
|
|
```
|
|
python textcatjsonl_to_trainjson.py -m en file.jsonl .
|
|
```
|
|
|
|
`cooking.json` is an example with mutually-exclusive classes with two labels:
|
|
|
|
* `baking`
|
|
* `not_baking`
|
|
|
|
`jigsaw-toxic-comment.json` is an example with multiple labels per instance:
|
|
|
|
* `insult`
|
|
* `obscene`
|
|
* `severe_toxic`
|
|
* `toxic`
|
|
|
|
### Data Sources
|
|
|
|
* `cooking.jsonl`: https://cooking.stackexchange.com. The meta IDs link to the
|
|
original question as `https://cooking.stackexchange.com/questions/ID`, e.g.,
|
|
`https://cooking.stackexchange.com/questions/2` for the first instance.
|
|
* `jigsaw-toxic-comment.jsonl`: [Jigsaw Toxic Comments Classification
|
|
Challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge)
|
|
|
|
### Data Licenses
|
|
|
|
* `cooking.jsonl`: CC BY-SA 4.0 ([`CC_BY-SA-4.0.txt`](CC_BY-SA-4.0.txt))
|
|
* `jigsaw-toxic-comment.jsonl`:
|
|
* text: CC BY-SA 3.0 ([`CC_BY-SA-3.0.txt`](CC_BY-SA-3.0.txt))
|
|
* annotation: CC0 ([`CC0.txt`](CC0.txt))
|