spaCy/spacy/cli/schemas/training.json

147 lines
3.6 KiB
JSON
Raw Normal View History

💫 New JSON helpers, training data internals & CLI rewrite (#2932) * Support nowrap setting in util.prints * Tidy up and fix whitespace * Simplify script and use read_jsonl helper * Add JSON schemas (see #2928) * Deprecate Doc.print_tree Will be replaced with Doc.to_json, which will produce a unified format * Add Doc.to_json() method (see #2928) Converts Doc objects to JSON using the same unified format as the training data. Method also supports serializing selected custom attributes in the doc._. space. * Remove outdated test * Add write_json and write_jsonl helpers * WIP: Update spacy train * Tidy up spacy train * WIP: Use wasabi for formatting * Add GoldParse helpers for JSON format * WIP: add debug-data command * Fix typo * Add missing import * Update wasabi pin * Add missing import * 💫 Refactor CLI (#2943) To be merged into #2932. ## Description - [x] refactor CLI To use [`wasabi`](https://github.com/ines/wasabi) - [x] use [`black`](https://github.com/ambv/black) for auto-formatting - [x] add `flake8` config - [x] move all messy UD-related scripts to `cli.ud` - [x] make converters function that take the opened file and return the converted data (instead of having them handle the IO) ### Types of change enhancement ## Checklist <!--- Before you submit the PR, go over this checklist and make sure you can tick off all the boxes. [] -> [x] --> - [x] I have submitted the spaCy Contributor Agreement. - [x] I ran the tests, and all new and existing tests passed. - [x] My changes don't require a change to the documentation, or if they do, I've added all required information. * Update wasabi pin * Delete old test * Update errors * Fix typo * Tidy up and format remaining code * Fix formatting * Improve formatting of messages * Auto-format remaining code * Add tok2vec stuff to spacy.train * Fix typo * Update wasabi pin * Fix path checks for when train() is called as function * Reformat and tidy up pretrain script * Update argument annotations * Raise error if model language doesn't match lang * Document new train command
2018-11-30 22:16:14 +03:00
{
"$schema": "http://json-schema.org/draft-06/schema",
"title": "Training data for spaCy models",
"type": "array",
"items": {
"type": "object",
"properties": {
"text": {
"title": "The text of the training example",
"type": "string",
"minLength": 1
},
"ents": {
"title": "Named entity spans in the text",
"type": "array",
"items": {
"type": "object",
"properties": {
"start": {
"title": "Start character offset of the span",
"type": "integer",
"minimum": 0
},
"end": {
"title": "End character offset of the span",
"type": "integer",
"minimum": 0
},
"label": {
"title": "Entity label",
"type": "string",
"minLength": 1,
"pattern": "^[A-Z0-9]*$"
}
},
"required": [
"start",
"end",
"label"
]
}
},
"sents": {
"title": "Sentence spans in the text",
"type": "array",
"items": {
"type": "object",
"properties": {
"start": {
"title": "Start character offset of the span",
"type": "integer",
"minimum": 0
},
"end": {
"title": "End character offset of the span",
"type": "integer",
"minimum": 0
}
},
"required": [
"start",
"end"
]
}
},
"cats": {
"title": "Text categories for the text classifier",
"type": "object",
"patternProperties": {
"*": {
"title": "A text category",
"oneOf": [
{
"type": "boolean"
},
{
"type": "number",
"minimum": 0
}
]
}
},
"propertyNames": {
"pattern": "^[A-Z0-9]*$",
"minLength": 1
}
},
"tokens": {
"title": "The tokens in the text",
"type": "array",
"items": {
"type": "object",
"minProperties": 1,
"properties": {
"id": {
"title": "Token ID, usually token index",
"type": "integer",
"minimum": 0
},
"start": {
"title": "Start character offset of the token",
"type": "integer",
"minimum": 0
},
"end": {
"title": "End character offset of the token",
"type": "integer",
"minimum": 0
},
"pos": {
"title": "Coarse-grained part-of-speech tag",
"type": "string",
"minLength": 1
},
"tag": {
"title": "Fine-grained part-of-speech tag",
"type": "string",
"minLength": 1
},
"dep": {
"title": "Dependency label",
"type": "string",
"minLength": 1
},
"head": {
"title": "Index of the token's head",
"type": "integer",
"minimum": 0
}
},
"required": [
"start",
"end"
]
}
},
"_": {
"title": "Custom user space",
"type": "object"
}
},
"required": [
"text"
]
}
}