mirror of
https://github.com/explosion/spaCy.git
synced 2024-11-14 05:37:03 +03:00
37c7c85a86
* Support nowrap setting in util.prints * Tidy up and fix whitespace * Simplify script and use read_jsonl helper * Add JSON schemas (see #2928) * Deprecate Doc.print_tree Will be replaced with Doc.to_json, which will produce a unified format * Add Doc.to_json() method (see #2928) Converts Doc objects to JSON using the same unified format as the training data. Method also supports serializing selected custom attributes in the doc._. space. * Remove outdated test * Add write_json and write_jsonl helpers * WIP: Update spacy train * Tidy up spacy train * WIP: Use wasabi for formatting * Add GoldParse helpers for JSON format * WIP: add debug-data command * Fix typo * Add missing import * Update wasabi pin * Add missing import * 💫 Refactor CLI (#2943) To be merged into #2932. ## Description - [x] refactor CLI To use [`wasabi`](https://github.com/ines/wasabi) - [x] use [`black`](https://github.com/ambv/black) for auto-formatting - [x] add `flake8` config - [x] move all messy UD-related scripts to `cli.ud` - [x] make converters function that take the opened file and return the converted data (instead of having them handle the IO) ### Types of change enhancement ## Checklist <!--- Before you submit the PR, go over this checklist and make sure you can tick off all the boxes. [] -> [x] --> - [x] I have submitted the spaCy Contributor Agreement. - [x] I ran the tests, and all new and existing tests passed. - [x] My changes don't require a change to the documentation, or if they do, I've added all required information. * Update wasabi pin * Delete old test * Update errors * Fix typo * Tidy up and format remaining code * Fix formatting * Improve formatting of messages * Auto-format remaining code * Add tok2vec stuff to spacy.train * Fix typo * Update wasabi pin * Fix path checks for when train() is called as function * Reformat and tidy up pretrain script * Update argument annotations * Raise error if model language doesn't match lang * Document new train command
46 lines
1.3 KiB
Python
46 lines
1.3 KiB
Python
# coding: utf8
|
|
from __future__ import unicode_literals
|
|
|
|
from cytoolz import partition_all
|
|
|
|
from ...gold import iob_to_biluo
|
|
|
|
|
|
def iob2json(input_data, n_sents=10, *args, **kwargs):
|
|
"""
|
|
Convert IOB files into JSON format for use with train cli.
|
|
"""
|
|
docs = []
|
|
for group in partition_all(n_sents, docs):
|
|
group = list(group)
|
|
first = group.pop(0)
|
|
to_extend = first["paragraphs"][0]["sentences"]
|
|
for sent in group[1:]:
|
|
to_extend.extend(sent["paragraphs"][0]["sentences"])
|
|
docs.append(first)
|
|
return docs
|
|
|
|
|
|
def read_iob(raw_sents):
|
|
sentences = []
|
|
for line in raw_sents:
|
|
if not line.strip():
|
|
continue
|
|
tokens = [t.split("|") for t in line.split()]
|
|
if len(tokens[0]) == 3:
|
|
words, pos, iob = zip(*tokens)
|
|
else:
|
|
words, iob = zip(*tokens)
|
|
pos = ["-"] * len(words)
|
|
biluo = iob_to_biluo(iob)
|
|
sentences.append(
|
|
[
|
|
{"orth": w, "tag": p, "ner": ent}
|
|
for (w, p, ent) in zip(words, pos, biluo)
|
|
]
|
|
)
|
|
sentences = [{"tokens": sent} for sent in sentences]
|
|
paragraphs = [{"sentences": [sent]} for sent in sentences]
|
|
docs = [{"id": 0, "paragraphs": [para]} for para in paragraphs]
|
|
return docs
|