svlandeg
6435458d51
simplify expression
2020-09-23 12:12:38 +02:00
svlandeg
20b0ec5dcf
avoid logging performance of frozen components
2020-09-23 10:37:12 +02:00
Ines Montani
e863b3dc14
Merge pull request #6092 from adrianeboyd/bugfix/load-vocab-lookups-2
2020-09-19 12:33:38 +02:00
Sofie Van Landeghem
39872de1f6
Introducing the gpu_allocator ( #6091 )
...
* rename 'use_pytorch_for_gpu_memory' to 'gpu_allocator'
* --code instead of --code-path
* update documentation
* avoid querying the "system" section directly
* add explanation of gpu_allocator to TF/PyTorch section in docs
* fix typo
* fix typo 2
* use set_gpu_allocator from thinc 8.0.0a34
* default null instead of empty string
2020-09-19 01:17:02 +02:00
Adriane Boyd
eed4b785f5
Load vocab lookups tables at beginning of training
...
Similar to how vectors are handled, move the vocab lookups to be loaded
at the start of training rather than when the vocab is initialized,
since the vocab doesn't have access to the full config when it's
created.
The option moves from `nlp.load_vocab_data` to `training.lookups`.
Typically these tables will come from `spacy-lookups-data`, but any
`Lookups` object can be provided.
The loading from `spacy-lookups-data` is now strict, so configs for each
language should specify the exact tables required. This also makes it
easier to control whether the larger clusters and probs tables are
included.
To load `lexeme_norm` from `spacy-lookups-data`:
```
[training.lookups]
@misc = "spacy.LoadLookupsData.v1"
lang = ${nlp.lang}
tables = ["lexeme_norm"]
```
2020-09-18 15:59:16 +02:00
svlandeg
427dbecdd6
cleanup and formatting
2020-09-17 11:48:04 +02:00
svlandeg
0c35885751
generalize corpora, dot notation for dev and train corpus
2020-09-17 11:38:59 +02:00
svlandeg
51fa929f47
rewrite train_corpus to corpus.train in config
2020-09-15 21:58:04 +02:00
Sofie Van Landeghem
3216a33149
positive_label config for textcat ( #6062 )
...
* hook up positive_label in textcat
* unit tests
* documentation
* formatting
* tests
* fix typo
* move verify_config to after begin_training
* revert accidential commit
2020-09-14 17:08:00 +02:00
Sofie Van Landeghem
8e7557656f
Renaming gold & annotation_setter ( #6042 )
...
* version bump to 3.0.0a16
* rename "gold" folder to "training"
* rename 'annotation_setter' to 'set_extra_annotations'
* formatting
2020-09-09 10:31:03 +02:00
Matthew Honnibal
ba5f4c9b32
Add words and seconds to train info
2020-09-08 15:24:47 +02:00
Ines Montani
ab1bb421ed
Update docs links in codebase
2020-09-04 12:58:50 +02:00
Ines Montani
b5a0657fd6
"model" terminology consistency in docs
2020-09-03 13:13:03 +02:00
Matthew Honnibal
122cb02001
Fix averages
2020-09-02 19:37:43 +02:00
Matthew Honnibal
ec660e3131
Fix use_pytorch_for_gpu_memory
2020-09-01 00:41:38 +02:00
Matthw Honnibal
c38298b8fa
Merge branch 'develop' of https://github.com/explosion/spaCy into develop
2020-08-31 19:55:55 +02:00
Matthw Honnibal
fe298fa50a
Shuffle on first epoch of train
2020-08-31 19:55:22 +02:00
svlandeg
13ee742fb4
example of custom logger
2020-08-31 14:24:41 +02:00
svlandeg
5230529de2
add loggers registry & logger docs sections
2020-08-28 21:44:04 +02:00
Ines Montani
a5fff1df51
Remove outdated non-empty output dir warning [ci skip]
2020-08-26 15:45:51 +02:00
Sofie Van Landeghem
79d460e3a2
Weights & Biases logger for train CLI ( #5971 )
...
* quick test as part of train script
* train_logger in config, default ConsoleLogger in loggers catalogue
* entitiy typo
* add wandb_logger
* cleanup
* Update spacy/cli/train_logger.py
Co-authored-by: Ines Montani <ines@ines.io>
* move loggers to gold.loggers
Co-authored-by: Ines Montani <ines@ines.io>
2020-08-26 15:24:33 +02:00
Matthew Honnibal
8038b87f04
Various small tweaks to project CLI ( #5965 )
...
* Fix up/download of http and local paths
* Support git_sparse_checkout for assets
* Fix scorer
* Handle already-present directories for git assets
* Improve convert command
* Fix support for existant files in git assets
* Support branches in git sparse checkout
* Format
* Fix git assets
* Document git block in assets
* Fix test
* Fix test
* Revert "Fix test"
This reverts commit cf3097260f964d636e27
2020-08-25 00:30:52 +02:00
Ines Montani
0e7f99da58
Fix handling of optional [pretraining] block ( #5954 )
...
* Fix handling of optional [pretraining] block
* Remote pretraining from default config
* Fix test
* Add schema option for empty pretrain block
2020-08-24 15:56:03 +02:00
Matthew Honnibal
160a855246
Format
2020-08-23 21:15:12 +02:00
Matthew Honnibal
fe1cf7e124
Allow score_weights to list extra scores
2020-08-23 18:31:30 +02:00
svlandeg
a8acedd4ba
example of custom reader and batcher
2020-08-18 19:15:16 +02:00
Sofie Van Landeghem
688e77562b
Train CLI script fixes ( #5931 )
...
* fix dash replacement in overrides arguments
* perform interpolation on training config
* make sure only .spacy files are read
2020-08-18 16:06:37 +02:00
Ines Montani
45f13cbf64
Merge pull request #5916 from explosion/feature/new-thinc-config
2020-08-16 15:24:12 +02:00
Ines Montani
8128e5eb35
Replace lexeme_norm warning with logging
2020-08-14 15:00:52 +02:00
Ines Montani
37814b608d
Remove env_opt and simplfy default Optimizer
2020-08-14 14:59:54 +02:00
Ines Montani
ab1d165bba
Pass optimizer defined in config to resume/begin_training
...
Otherwise, this would create a default optimizer, which isn't what we want?
2020-08-14 14:59:22 +02:00
Ines Montani
67cc39af7f
Update Thinc and include section order
2020-08-14 14:06:22 +02:00
Ines Montani
88b0a96801
Update for new Thinc and adjust config
2020-08-13 17:38:30 +02:00
Ines Montani
950832f087
Tidy up pipes ( #5906 )
...
* Tidy up pipes
* Fix init, defaults and raise custom errors
* Update docs
* Update docs [ci skip]
* Apply suggestions from code review
Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
* Tidy up error handling and validation, fix consistency
* Simplify get_examples check
* Remove unused import [ci skip]
Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
2020-08-11 23:29:31 +02:00
Ines Montani
56c17973aa
Use "raise ... from" in custom errors for better tracebacks
2020-08-05 23:53:21 +02:00
Ines Montani
5cc0d89fad
Simplify config overrides in CLI and deserialization ( #5880 )
2020-08-05 23:35:09 +02:00
Matthew Honnibal
50c0e49741
Fix train CLI
2020-08-05 15:40:47 +02:00
Ines Montani
b795f02fbd
Allow adding pipeline components from source model ( #5857 )
...
* Allow adding pipeline components from source model
* Config: name -> component
* Improve error messages
* Fix error and test
* Add frozen components and exclude logic
* Remove exclude from Language.evaluate
* Init sourced components with current vocab
* Fix error codes
2020-08-04 23:39:19 +02:00
Matthew Honnibal
ecb3c4e8f4
Create corpus iterator and batcher from registry during training ( #5865 )
...
* Move batchers into their own module (and registry)
* Update CLI
* Update Corpus and batcher
* Update tests
* Update one config
* Merge 'evaluation' block back under [training]
* Import batchers in gold __init__
* Fix batchers
* Update config
* Update schema
* Update util
* Don't assume train and dev are actually paths
* Update onto-joint config
* Fix missing import
* Format
* Format
* Update spacy/gold/corpus.py
Co-authored-by: Ines Montani <ines@ines.io>
* Fix name
* Update default config
* Fix get_length option in batchers
* Update test
* Add comment
* Pass path into Corpus
* Update docstring
* Update schema and configs
* Update config
* Fix test
* Fix paths
* Fix print
* Fix create_train_batches
* [training.read_train] -> [training.train_corpus]
* Update onto-joint config
Co-authored-by: Ines Montani <ines@ines.io>
2020-08-04 15:09:37 +02:00
Ines Montani
934447a611
Merge pull request #5855 from svlandeg/fix/cli-debug
2020-08-03 13:09:20 +02:00
Ines Montani
4c055f0aa7
Add init CLI and init config ( #5854 )
...
* Add init CLI and init config draft
* Improve config validation
* Auto-format
* Don't export anything in debug config
* Update docs
2020-08-02 15:18:30 +02:00
svlandeg
08d3c36c20
bugfix in train CLI
2020-07-31 15:03:43 +02:00
Matthew Honnibal
2af741d7e3
Fix train arg
2020-07-29 14:56:01 +02:00
Matthew Honnibal
2aff3c4b5a
Load vectors in 'spacy train'
2020-07-29 14:00:13 +02:00
Adriane Boyd
191a12d75f
Fix score_weights typo in train CLI ( #5835 )
2020-07-29 11:04:12 +02:00
Adriane Boyd
0cddb0dbe9
Move timing into Language.evaluate ( #5836 )
...
Move timing into `Language.evaluate` so that only the processing is
timing, not processing + scoring. `Language.evaluate` returns
`scores["speed"]` as words per second, which should be identical to how
the speed was added to the scores previously. Also add the speed to the
evaluate CLI output.
2020-07-29 11:02:31 +02:00
Ines Montani
0094cb0d04
Remove scores list from config and document
2020-07-28 11:22:24 +02:00
Adriane Boyd
8bb0507777
Add and update score methods and score weights
...
Add and update `score` methods, provided `scores`, and default weights
`default_score_weights` for pipeline components.
* `scores` provides all top-level keys returned by `score` (merely informative, similar to `assigns`).
* `default_score_weights` provides the default weights for a default config.
* The keys from `default_score_weights` determine which values will be
shown in the `spacy train` output, so keys with weight `0.0` will be
displayed but not counted toward the overall score.
2020-07-27 14:44:53 +02:00
Ines Montani
e92df281ce
Tidy up, autoformat, add types
2020-07-25 15:01:15 +02:00
Ines Montani
cdbd6ba912
Merge pull request #5798 from explosion/feature/language-data-config
2020-07-25 13:34:49 +02:00
