mirror of
https://github.com/explosion/spaCy.git
synced 2024-09-21 03:19:13 +03:00
e48a09df4e
* OrigAnnot class instead of gold.orig_annot list of zipped tuples * from_orig to replace from_annot_tuples * rename to RawAnnot * some unit tests for GoldParse creation and internal format * removing orig_annot and switching to lists instead of tuple * rewriting tuples to use RawAnnot (+ debug statements, WIP) * fix pop() changing the data * small fixes * pop-append fixes * return RawAnnot for existing GoldParse to have uniform interface * clean up imports * fix merge_sents * add unit test for 4402 with new structure (not working yet) * introduce DocAnnot * typo fixes * add unit test for merge_sents * rename from_orig to from_raw * fixing unit tests * fix nn parser * read_annots to produce text, doc_annot pairs * _make_golds fix * rename golds_to_gold_annots * small fixes * fix encoding * have golds_to_gold_annots use DocAnnot * missed a spot * merge_sents as function in DocAnnot * allow specifying only part of the token-level annotations * refactor with Example class + underlying dicts * pipeline components to work with Example objects (wip) * input checking * fix yielding * fix calls to update * small fixes * fix scorer unit test with new format * fix kwargs order * fixes for ud and conllu scripts * fix reading data for conllu script * add in proper errors (not fixed numbering yet to avoid merge conflicts) * fixing few more small bugs * fix EL script |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| entity_linker_evaluation.py | ||
| kb_creator.py | ||
| README.md | ||
| train_descriptions.py | ||
| wiki_io.py | ||
| wiki_namespaces.py | ||
| wikidata_pretrain_kb.py | ||
| wikidata_processor.py | ||
| wikidata_train_entity_linker.py | ||
| wikipedia_processor.py | ||
Entity Linking with Wikipedia and Wikidata
Step 1: Create a Knowledge Base (KB) and training data
Run wikipedia_pretrain_kb.py
- This takes as input the locations of a Wikipedia and a Wikidata dump, and produces a KB directory + training file
- WikiData: get
latest-all.json.bz2from https://dumps.wikimedia.org/wikidatawiki/entities/ - Wikipedia: get
enwiki-latest-pages-articles-multistream.xml.bz2from https://dumps.wikimedia.org/enwiki/latest/ (or for any other language)
- WikiData: get
- You can set the filtering parameters for KB construction:
max_per_alias: (max) number of candidate entities in the KB per alias/synonymmin_freq: threshold of number of times an entity should occur in the corpus to be included in the KBmin_pair: threshold of number of times an entity+alias combination should occur in the corpus to be included in the KB
- Further parameters to set:
descriptions_from_wikipedia: whether to parse descriptions from Wikipedia (True) or Wikidata (False)entity_vector_length: length of the pre-trained entity description vectorslang: language for which to fetch Wikidata information (as the dump contains all languages)
Quick testing and rerunning:
- When trying out the pipeline for a quick test, set
limit_prior,limit_trainand/orlimit_wdto read only parts of the dumps instead of everything. - If you only want to (re)run certain parts of the pipeline, just remove the corresponding files and they will be recalculated or reparsed.
Step 2: Train an Entity Linking model
Run wikidata_train_entity_linker.py
- This takes the KB directory produced by Step 1, and trains an Entity Linking model
- You can set the learning parameters for the EL training:
epochs: number of training iterationsdropout: dropout ratelr: learning ratel2: L2 regularization
- Specify the number of training and dev testing entities with
train_instanddev_instrespectively - Further parameters to set:
labels_discard: NER label types to discard during training