mirror of
https://github.com/explosion/spaCy.git
synced 2024-12-28 10:56:31 +03:00
37c7c85a86
* Support nowrap setting in util.prints * Tidy up and fix whitespace * Simplify script and use read_jsonl helper * Add JSON schemas (see #2928) * Deprecate Doc.print_tree Will be replaced with Doc.to_json, which will produce a unified format * Add Doc.to_json() method (see #2928) Converts Doc objects to JSON using the same unified format as the training data. Method also supports serializing selected custom attributes in the doc._. space. * Remove outdated test * Add write_json and write_jsonl helpers * WIP: Update spacy train * Tidy up spacy train * WIP: Use wasabi for formatting * Add GoldParse helpers for JSON format * WIP: add debug-data command * Fix typo * Add missing import * Update wasabi pin * Add missing import * 💫 Refactor CLI (#2943) To be merged into #2932. ## Description - [x] refactor CLI To use [`wasabi`](https://github.com/ines/wasabi) - [x] use [`black`](https://github.com/ambv/black) for auto-formatting - [x] add `flake8` config - [x] move all messy UD-related scripts to `cli.ud` - [x] make converters function that take the opened file and return the converted data (instead of having them handle the IO) ### Types of change enhancement ## Checklist <!--- Before you submit the PR, go over this checklist and make sure you can tick off all the boxes. [] -> [x] --> - [x] I have submitted the spaCy Contributor Agreement. - [x] I ran the tests, and all new and existing tests passed. - [x] My changes don't require a change to the documentation, or if they do, I've added all required information. * Update wasabi pin * Delete old test * Update errors * Fix typo * Tidy up and format remaining code * Fix formatting * Improve formatting of messages * Auto-format remaining code * Add tok2vec stuff to spacy.train * Fix typo * Update wasabi pin * Fix path checks for when train() is called as function * Reformat and tidy up pretrain script * Update argument annotations * Raise error if model language doesn't match lang * Document new train command
52 lines
1.5 KiB
Python
52 lines
1.5 KiB
Python
# coding: utf-8
|
|
from __future__ import unicode_literals
|
|
|
|
from pathlib import Path
|
|
from jsonschema import Draft4Validator
|
|
|
|
from ...errors import Errors
|
|
from ...util import read_json
|
|
|
|
|
|
SCHEMAS = {}
|
|
|
|
|
|
def get_schema(name):
|
|
"""Get the JSON schema for a given name. Looks for a .json file in
|
|
spacy.cli.schemas, validates the schema and raises ValueError if not found.
|
|
|
|
EXAMPLE:
|
|
>>> schema = get_schema('training')
|
|
|
|
name (unicode): The name of the schema.
|
|
RETURNS (dict): The JSON schema.
|
|
"""
|
|
if name not in SCHEMAS:
|
|
schema_path = Path(__file__).parent / "{}.json".format(name)
|
|
if not schema_path.exists():
|
|
raise ValueError(Errors.E104.format(name=name))
|
|
schema = read_json(schema_path)
|
|
# TODO: replace with (stable) Draft6Validator, if available
|
|
validator = Draft4Validator(schema)
|
|
validator.check_schema(schema)
|
|
SCHEMAS[name] = schema
|
|
return SCHEMAS[name]
|
|
|
|
|
|
def validate_json(data, schema):
|
|
"""Validate data against a given JSON schema (see https://json-schema.org).
|
|
|
|
data: JSON-serializable data to validate.
|
|
schema (dict): The JSON schema.
|
|
RETURNS (list): A list of error messages, if available.
|
|
"""
|
|
validator = Draft4Validator(schema)
|
|
errors = []
|
|
for err in sorted(validator.iter_errors(data), key=lambda e: e.path):
|
|
if err.path:
|
|
err_path = "[{}]".format(" -> ".join([str(p) for p in err.path]))
|
|
else:
|
|
err_path = ""
|
|
errors.append(err.message + " " + err_path)
|
|
return errors
|