mirror of
https://github.com/explosion/spaCy.git
synced 2024-09-21 03:19:13 +03:00
8d69874afb
* Add `spacy.PlainTextCorpusReader.v1` This is a corpus reader that reads plain text corpora with the following format: - UTF-8 encoding - One line per document. - Blank lines are ignored. It is useful for applications where we deal with very large corpora, such as distillation, and don't want to deal with the space overhead of serialized formats. Additionally, many large corpora already use such a text format, keeping the necessary preprocessing to a minimum. * Update spacy/training/corpus.py Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com> * docs: add version to `PlainTextCorpus` * Add docstring to registry function * Add plain text corpus tests * Only strip newline/carriage return * Add return type _string_to_tmp_file helper * Use a temporary directory in place of file name Different OS auto delete/sharing semantics are just wonky. * This will be new in 3.5.1 (rather than 4) * Test improvements from code review Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
243 lines
12 KiB
Plaintext
243 lines
12 KiB
Plaintext
---
|
|
title: Corpus
|
|
teaser: An annotated corpus
|
|
tag: class
|
|
source: spacy/training/corpus.py
|
|
version: 3
|
|
---
|
|
|
|
This class manages annotated corpora and can be used for training and
|
|
development datasets in the [`DocBin`](/api/docbin) (`.spacy`) format. To
|
|
customize the data loading during training, you can register your own
|
|
[data readers and batchers](/usage/training#custom-code-readers-batchers). Also
|
|
see the usage guide on [data utilities](/usage/training#data) for more details
|
|
and examples.
|
|
|
|
## Config and implementation {id="config"}
|
|
|
|
`spacy.Corpus.v1` is a registered function that creates a `Corpus` of training
|
|
or evaluation data. It takes the same arguments as the `Corpus` class and
|
|
returns a callable that yields [`Example`](/api/example) objects. You can
|
|
replace it with your own registered function in the
|
|
[`@readers` registry](/api/top-level#registry) to customize the data loading and
|
|
streaming.
|
|
|
|
> #### Example config
|
|
>
|
|
> ```ini
|
|
> [paths]
|
|
> train = "corpus/train.spacy"
|
|
>
|
|
> [corpora.train]
|
|
> @readers = "spacy.Corpus.v1"
|
|
> path = ${paths.train}
|
|
> gold_preproc = false
|
|
> max_length = 0
|
|
> limit = 0
|
|
> augmenter = null
|
|
> ```
|
|
|
|
| Name | Description |
|
|
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `path` | The directory or filename to read from. Expects data in spaCy's binary [`.spacy` format](/api/data-formats#binary-training). ~~Path~~ |
|
|
| `gold_preproc` | Whether to set up the Example object with gold-standard sentences and tokens for the predictions. See [`Corpus`](/api/corpus#init) for details. ~~bool~~ |
|
|
| `max_length` | Maximum document length. Longer documents will be split into sentences, if sentence boundaries are available. Defaults to `0` for no limit. ~~int~~ |
|
|
| `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ |
|
|
| `augmenter` | Apply some simply data augmentation, where we replace tokens with variations. This is especially useful for punctuation and case replacement, to help generalize beyond corpora that don't have smart-quotes, or only have smart quotes, etc. Defaults to `None`. ~~Optional[Callable]~~ |
|
|
|
|
```python
|
|
%%GITHUB_SPACY/spacy/training/corpus.py
|
|
```
|
|
|
|
## Corpus.\_\_init\_\_ {id="init",tag="method"}
|
|
|
|
Create a `Corpus` for iterating [Example](/api/example) objects from a file or
|
|
directory of [`.spacy` data files](/api/data-formats#binary-training). The
|
|
`gold_preproc` setting lets you specify whether to set up the `Example` object
|
|
with gold-standard sentences and tokens for the predictions. Gold preprocessing
|
|
helps the annotations align to the tokenization, and may result in sequences of
|
|
more consistent length. However, it may reduce runtime accuracy due to
|
|
train/test skew.
|
|
|
|
> #### Example
|
|
>
|
|
> ```python
|
|
> from spacy.training import Corpus
|
|
>
|
|
> # With a single file
|
|
> corpus = Corpus("./data/train.spacy")
|
|
>
|
|
> # With a directory
|
|
> corpus = Corpus("./data", limit=10)
|
|
> ```
|
|
|
|
| Name | Description |
|
|
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `path` | The directory or filename to read from. ~~Union[str, Path]~~ |
|
|
| _keyword-only_ | |
|
|
| `gold_preproc` | Whether to set up the Example object with gold-standard sentences and tokens for the predictions. Defaults to `False`. ~~bool~~ |
|
|
| `max_length` | Maximum document length. Longer documents will be split into sentences, if sentence boundaries are available. Defaults to `0` for no limit. ~~int~~ |
|
|
| `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ |
|
|
| `augmenter` | Optional data augmentation callback. ~~Callable[[Language, Example], Iterable[Example]]~~ |
|
|
| `shuffle` | Whether to shuffle the examples. Defaults to `False`. ~~bool~~ |
|
|
|
|
## Corpus.\_\_call\_\_ {id="call",tag="method"}
|
|
|
|
Yield examples from the data.
|
|
|
|
> #### Example
|
|
>
|
|
> ```python
|
|
> from spacy.training import Corpus
|
|
> import spacy
|
|
>
|
|
> corpus = Corpus("./train.spacy")
|
|
> nlp = spacy.blank("en")
|
|
> train_data = corpus(nlp)
|
|
> ```
|
|
|
|
| Name | Description |
|
|
| ---------- | -------------------------------------- |
|
|
| `nlp` | The current `nlp` object. ~~Language~~ |
|
|
| **YIELDS** | The examples. ~~Example~~ |
|
|
|
|
## JsonlCorpus {id="jsonlcorpus",tag="class"}
|
|
|
|
Iterate Doc objects from a file or directory of JSONL (newline-delimited JSON)
|
|
formatted raw text files. Can be used to read the raw text corpus for language
|
|
model [pretraining](/usage/embeddings-transformers#pretraining) from a JSONL
|
|
file.
|
|
|
|
> #### Tip: Writing JSONL
|
|
>
|
|
> Our utility library [`srsly`](https://github.com/explosion/srsly) provides a
|
|
> handy `write_jsonl` helper that takes a file path and list of dictionaries and
|
|
> writes out JSONL-formatted data.
|
|
>
|
|
> ```python
|
|
> import srsly
|
|
> data = [{"text": "Some text"}, {"text": "More..."}]
|
|
> srsly.write_jsonl("/path/to/text.jsonl", data)
|
|
> ```
|
|
|
|
```json {title="Example"}
|
|
{"text": "Can I ask where you work now and what you do, and if you enjoy it?"}
|
|
{"text": "They may just pull out of the Seattle market completely, at least until they have autonomous vehicles."}
|
|
{"text": "My cynical view on this is that it will never be free to the public. Reason: what would be the draw of joining the military? Right now their selling point is free Healthcare and Education. Ironically both are run horribly and most, that I've talked to, come out wishing they never went in."}
|
|
```
|
|
|
|
### JsonlCorpus.\_\_init\_\_ {id="jsonlcorpus",tag="method"}
|
|
|
|
Initialize the reader.
|
|
|
|
> #### Example
|
|
>
|
|
> ```python
|
|
> from spacy.training import JsonlCorpus
|
|
>
|
|
> corpus = JsonlCorpus("./data/texts.jsonl")
|
|
> ```
|
|
>
|
|
> ```ini
|
|
> ### Example config
|
|
> [corpora.pretrain]
|
|
> @readers = "spacy.JsonlCorpus.v1"
|
|
> path = "corpus/raw_text.jsonl"
|
|
> min_length = 0
|
|
> max_length = 0
|
|
> limit = 0
|
|
> ```
|
|
|
|
| Name | Description |
|
|
| -------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `path` | The directory or filename to read from. Expects newline-delimited JSON with a key `"text"` for each record. ~~Union[str, Path]~~ |
|
|
| _keyword-only_ | |
|
|
| `min_length` | Minimum document length (in tokens). Shorter documents will be skipped. Defaults to `0`, which indicates no limit. ~~int~~ |
|
|
| `max_length` | Maximum document length (in tokens). Longer documents will be skipped. Defaults to `0`, which indicates no limit. ~~int~~ |
|
|
| `limit` | Limit corpus to a subset of examples, e.g. for debugging. Defaults to `0` for no limit. ~~int~~ |
|
|
|
|
### JsonlCorpus.\_\_call\_\_ {id="jsonlcorpus-call",tag="method"}
|
|
|
|
Yield examples from the data.
|
|
|
|
> #### Example
|
|
>
|
|
> ```python
|
|
> from spacy.training import JsonlCorpus
|
|
> import spacy
|
|
>
|
|
> corpus = JsonlCorpus("./texts.jsonl")
|
|
> nlp = spacy.blank("en")
|
|
> data = corpus(nlp)
|
|
> ```
|
|
|
|
| Name | Description |
|
|
| ---------- | -------------------------------------- |
|
|
| `nlp` | The current `nlp` object. ~~Language~~ |
|
|
| **YIELDS** | The examples. ~~Example~~ |
|
|
|
|
## PlainTextCorpus {id="plaintextcorpus",tag="class",version="3.5.1"}
|
|
|
|
Iterate over documents from a plain text file. Can be used to read the raw text
|
|
corpus for language model
|
|
[pretraining](/usage/embeddings-transformers#pretraining). The expected file
|
|
format is:
|
|
|
|
- UTF-8 encoding
|
|
- One document per line
|
|
- Blank lines are ignored.
|
|
|
|
```text {title="Example"}
|
|
Can I ask where you work now and what you do, and if you enjoy it?
|
|
They may just pull out of the Seattle market completely, at least until they have autonomous vehicles.
|
|
My cynical view on this is that it will never be free to the public. Reason: what would be the draw of joining the military? Right now their selling point is free Healthcare and Education. Ironically both are run horribly and most, that I've talked to, come out wishing they never went in.
|
|
```
|
|
|
|
### PlainTextCorpus.\_\_init\_\_ {id="plaintextcorpus-init",tag="method"}
|
|
|
|
Initialize the reader.
|
|
|
|
> #### Example
|
|
>
|
|
> ```python
|
|
> from spacy.training import PlainTextCorpus
|
|
>
|
|
> corpus = PlainTextCorpus("./data/docs.txt")
|
|
> ```
|
|
>
|
|
> ```ini
|
|
> ### Example config
|
|
> [corpora.pretrain]
|
|
> @readers = "spacy.PlainTextCorpus.v1"
|
|
> path = "corpus/raw_text.txt"
|
|
> min_length = 0
|
|
> max_length = 0
|
|
> ```
|
|
|
|
| Name | Description |
|
|
| -------------- | -------------------------------------------------------------------------------------------------------------------------- |
|
|
| `path` | The directory or filename to read from. Expects newline-delimited documents in UTF8 format. ~~Union[str, Path]~~ |
|
|
| _keyword-only_ | |
|
|
| `min_length` | Minimum document length (in tokens). Shorter documents will be skipped. Defaults to `0`, which indicates no limit. ~~int~~ |
|
|
| `max_length` | Maximum document length (in tokens). Longer documents will be skipped. Defaults to `0`, which indicates no limit. ~~int~~ |
|
|
|
|
### PlainTextCorpus.\_\_call\_\_ {id="plaintextcorpus-call",tag="method"}
|
|
|
|
Yield examples from the data.
|
|
|
|
> #### Example
|
|
>
|
|
> ```python
|
|
> from spacy.training import PlainTextCorpus
|
|
> import spacy
|
|
>
|
|
> corpus = PlainTextCorpus("./docs.txt")
|
|
> nlp = spacy.blank("en")
|
|
> data = corpus(nlp)
|
|
> ```
|
|
|
|
| Name | Description |
|
|
| ---------- | -------------------------------------- |
|
|
| `nlp` | The current `nlp` object. ~~Language~~ |
|
|
| **YIELDS** | The examples. ~~Example~~ |
|