spaCy/examples/training/ner_example_data/ner-token-per-line.iob

When	O
Sebastian	B-PERSON
Thrun	I-PERSON
started	O
working	O
on	O
self	O
-	O
driving	O
cars	O
at	O
Google	B-ORG
in	O
2007	B-DATE
,	O
few	O
people	O
outside	O
of	O
the	O
company	O
took	O
him	O
seriously	O
.	O
“	O
I	O
can	O
tell	O
you	O
very	O
senior	O
CEOs	O
of	O
major	O
American	B-NORP
car	O
companies	O
would	O
shake	O
my	O
hand	O
and	O
turn	O
away	O
because	O
I	O
was	O
n’t	O
worth	O
talking	O
to	O
,	O
”	O
said	O
Thrun	B-PERSON
,	O
in	O
an	O
interview	O
with	O
Recode	B-ORG
earlier	B-DATE
this	I-DATE
week	I-DATE
.	O
-												Updates/bugfixes for NER/IOB converters (#4186)

* Updates/bugfixes for NER/IOB converters

* Converter formats `ner` and `iob` use autodetect to choose a converter if
  possible

* `iob2json` is reverted to handle sentence-per-line data like
  `word1|pos1|ent1 word2|pos2|ent2`

  * Fix bug in `merge_sentences()` so the second sentence in each batch isn't
    skipped

* `conll_ner2json` is made more general so it can handle more formats with
  whitespace-separated columns

  * Supports all formats where the first column is the token and the final
    column is the IOB tag; if present, the second column is the POS tag

  * As in CoNLL 2003 NER, blank lines separate sentences, `-DOCSTART- -X- O O`
    separates documents

  * Add option for segmenting sentences (new flag `-s`)

  * Parser-based sentence segmentation with a provided model, otherwise with
    sentencizer (new option `-b` to specify model)

  * Can group sentences into documents with `n_sents` as long as sentence
    segmentation is available

  * Only applies automatic segmentation when there are no existing delimiters
    in the data

* Provide info about settings applied during conversion with warnings and
  suggestions if settings conflict or might not be not optimal.

* Add tests for common formats

* Add '(default)' back to docs for -c auto

* Add document count back to output

* Revert changes to converter output message

* Use explicit tabs in convert CLI test data

* Adjust/add messages for n_sents=1 default

* Add sample NER data to training examples

* Update README

* Add links in docs to example NER data

* Define msg within converters

											
										
										
											2019-08-29 13:04:01 +03:00
+								When	O
 								Sebastian	B-PERSON
 								Thrun	I-PERSON
 								started	O
 								working	O
 								on	O
 								self	O
 								-	O
 								driving	O
 								cars	O
 								at	O
 								Google	B-ORG
 								in	O
 	B-DATE
 								,	O
 								few	O
 								people	O
 								outside	O
 								of	O
 								the	O
 								company	O
 								took	O
 								him	O
 								seriously	O
 								.	O
 								“	O
 								I	O
 								can	O
 								tell	O
 								you	O
 								very	O
 								senior	O
 								CEOs	O
 								of	O
 								major	O
 								American	B-NORP
 								car	O
 								companies	O
 								would	O
 								shake	O
 								my	O
 								hand	O
 								and	O
 								turn	O
 								away	O
 								because	O
 								I	O
 								was	O
 								n’t	O
 								worth	O
 								talking	O
 								to	O
 								,	O
 								”	O
 								said	O
 								Thrun	B-PERSON
 								,	O
 								in	O
 								an	O
 								interview	O
 								with	O
 								Recode	B-ORG
 								earlier	B-DATE
 								this	I-DATE
 								week	I-DATE
 								.	O