spaCy/examples/training/ner_example_data/ner-token-per-line.iob

67 lines
527 B
Plaintext
Raw Normal View History

Updates/bugfixes for NER/IOB converters (#4186) * Updates/bugfixes for NER/IOB converters * Converter formats `ner` and `iob` use autodetect to choose a converter if possible * `iob2json` is reverted to handle sentence-per-line data like `word1|pos1|ent1 word2|pos2|ent2` * Fix bug in `merge_sentences()` so the second sentence in each batch isn't skipped * `conll_ner2json` is made more general so it can handle more formats with whitespace-separated columns * Supports all formats where the first column is the token and the final column is the IOB tag; if present, the second column is the POS tag * As in CoNLL 2003 NER, blank lines separate sentences, `-DOCSTART- -X- O O` separates documents * Add option for segmenting sentences (new flag `-s`) * Parser-based sentence segmentation with a provided model, otherwise with sentencizer (new option `-b` to specify model) * Can group sentences into documents with `n_sents` as long as sentence segmentation is available * Only applies automatic segmentation when there are no existing delimiters in the data * Provide info about settings applied during conversion with warnings and suggestions if settings conflict or might not be not optimal. * Add tests for common formats * Add '(default)' back to docs for -c auto * Add document count back to output * Revert changes to converter output message * Use explicit tabs in convert CLI test data * Adjust/add messages for n_sents=1 default * Add sample NER data to training examples * Update README * Add links in docs to example NER data * Define msg within converters
2019-08-29 13:04:01 +03:00
When O
Sebastian B-PERSON
Thrun I-PERSON
started O
working O
on O
self O
- O
driving O
cars O
at O
Google B-ORG
in O
2007 B-DATE
, O
few O
people O
outside O
of O
the O
company O
took O
him O
seriously O
. O
“ O
I O
can O
tell O
you O
very O
senior O
CEOs O
of O
major O
American B-NORP
car O
companies O
would O
shake O
my O
hand O
and O
turn O
away O
because O
I O
was O
nt O
worth O
talking O
to O
, O
” O
said O
Thrun B-PERSON
, O
in O
an O
interview O
with O
Recode B-ORG
earlier B-DATE
this I-DATE
week I-DATE
. O