Commit Graph

7652 Commits

Author SHA1 Message Date
Sofie Van Landeghem
688e77562b
Train CLI script fixes (#5931)
* fix dash replacement in overrides arguments

* perform interpolation on training config

* make sure only .spacy files are read
2020-08-18 16:06:37 +02:00
Ines Montani
82f0e20318 Update docs and consistency [ci skip] 2020-08-18 14:39:40 +02:00
svlandeg
10e67b400c output_file required, spacy-transformers prefered instead of required 2020-08-18 13:38:43 +02:00
Ines Montani
1c3bcfb488 Update docs and util consistency 2020-08-18 01:22:59 +02:00
Ines Montani
990c6b4c32 Update docs and CLI [ci skip] 2020-08-17 21:38:20 +02:00
Ines Montani
3ae5e02f4f Update docs, types and API consistency 2020-08-17 16:45:24 +02:00
Matthew Honnibal
a95a36ce2a Set version to v3.0.0a7 2020-08-16 15:51:05 +02:00
Ines Montani
6ae83bde0c Fix CLI consistency [ci skip] 2020-08-16 15:46:29 +02:00
Ines Montani
45f13cbf64
Merge pull request #5916 from explosion/feature/new-thinc-config 2020-08-16 15:24:12 +02:00
Ines Montani
34bda91695 Show warnings if there's nothing to auto-fill 2020-08-16 14:19:43 +02:00
Ines Montani
dd5804d499 Update type hints 2020-08-16 14:19:33 +02:00
Ines Montani
a570c304df Update quickstart, template and docs 2020-08-15 14:50:29 +02:00
Ines Montani
3272a63430
Merge pull request #5920 from explosion/fix/logging-warning-various 2020-08-15 14:41:15 +02:00
Ines Montani
fdcde9b0bf Add init fill-config 2020-08-14 16:49:26 +02:00
Matthew Honnibal
9ebf39fb5f Relax test 2020-08-14 16:31:09 +02:00
Ines Montani
8128e5eb35 Replace lexeme_norm warning with logging 2020-08-14 15:00:52 +02:00
Ines Montani
37814b608d Remove env_opt and simplfy default Optimizer 2020-08-14 14:59:54 +02:00
Ines Montani
ab1d165bba Pass optimizer defined in config to resume/begin_training
Otherwise, this would create a default optimizer, which isn't what we want?
2020-08-14 14:59:22 +02:00
Ines Montani
e4d0990857 Only receive from listener if listener exists 2020-08-14 14:58:48 +02:00
Ines Montani
cef97e4b63 Fix path check 2020-08-14 14:58:18 +02:00
Ines Montani
db2dbc8e59 Remove unused warning 2020-08-14 14:58:03 +02:00
Ines Montani
67cc39af7f Update Thinc and include section order 2020-08-14 14:06:22 +02:00
Ines Montani
88b0a96801 Update for new Thinc and adjust config 2020-08-13 17:38:30 +02:00
Adam Bittlingmayer
7b33b2854f
Add Armenian sentence-final verchaket, Greek question mark and Arabic question mark to default punct (#5910)
* Add Armenian sentence-final verchaket

* Add Greek and Arabic question marks, and contributor agreement

* Check box
2020-08-12 15:36:14 +02:00
graue70
49e690bde1
Fix typos in comments (#5904)
* Fix typo in comment

* Fix typo

* Add spaCy Contributor Agreement
2020-08-12 15:35:25 +02:00
graue70
ba84371ab0
Use init parameter (#5909) 2020-08-11 23:41:58 +02:00
Ines Montani
950832f087
Tidy up pipes (#5906)
* Tidy up pipes

* Fix init, defaults and raise custom errors

* Update docs

* Update docs [ci skip]

* Apply suggestions from code review

Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>

* Tidy up error handling and validation, fix consistency

* Simplify get_examples check

* Remove unused import [ci skip]

Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
2020-08-11 23:29:31 +02:00
Ines Montani
f79e4c094d Remove generic type
Seems to cause error on Python 3.8 with Cython?
2020-08-10 17:24:30 +02:00
Ines Montani
c099f6eece Add Token.lex 2020-08-10 16:43:52 +02:00
Ines Montani
933a7cf8d1 Fix Lexeme.from_ptr 2020-08-10 16:43:37 +02:00
Ines Montani
64f2f84098 Update docstrings and docs [ci skip] 2020-08-10 13:45:22 +02:00
Ines Montani
a4b448eec4 Remove unused compiler flag 2020-08-10 13:13:18 +02:00
Ines Montani
3eaeb73342 Tidy up and auto-format 2020-08-09 22:36:23 +02:00
Ines Montani
d5c78c7a34 Update docs and fix consistency 2020-08-09 22:31:52 +02:00
Ines Montani
7c6854d8d4 Fix missing imports 2020-08-09 22:28:29 +02:00
Matthew Honnibal
0fc13b2f14 Set version to v3.0.0a6 2020-08-09 21:53:32 +02:00
Ines Montani
a15c5fb191 Update docstrings and docs 2020-08-09 16:10:48 +02:00
Ines Montani
8d2baa153d Update tokenizer docs and add test 2020-08-09 15:24:01 +02:00
Matthew Honnibal
134d933d67 Add docstring for entity linker factory 2020-08-09 15:19:28 +02:00
Matthew Honnibal
992ee1c02f Update tagger docstring 2020-08-09 15:09:31 +02:00
Matthew Honnibal
ebf9a7acbf Add textcat docstring 2020-08-09 15:07:09 +02:00
Matthew Honnibal
8a13f510d6 Update tests 2020-08-09 15:01:16 +02:00
Matthew Honnibal
bbd8acd4bf Add docstrings for parser and NER. Simplify some arguments 2020-08-09 14:46:13 +02:00
Matthew Honnibal
39a3d64c01 Add docstrings for Tok2Vec component 2020-08-09 00:48:03 +02:00
Ines Montani
fd20f84927
Merge pull request #5895 from explosion/docs/batchers
Draft docstrings for batchers
2020-08-07 20:07:10 +02:00
Matthew Honnibal
f5c4e0b751 Add docstrings for batchers 2020-08-07 18:51:02 +02:00
Ines Montani
fe29ceec9e Merge branch 'develop' into docs/model-docstrings 2020-08-07 18:42:01 +02:00
Ines Montani
3a193eb8f1 Fix imports, types and default configs 2020-08-07 18:40:54 +02:00
Matthew Honnibal
b1d83fc13e Fix imports 2020-08-07 16:55:54 +02:00
Matthew Honnibal
473504d837 Format 2020-08-07 16:49:00 +02:00
Matthew Honnibal
234c52a91e Add tok2vec docstrings 2020-08-07 16:48:48 +02:00
Matthew Honnibal
547bc8a82b Add docstring notes 2020-08-07 16:17:34 +02:00
Ines Montani
6f3649923c
Merge pull request #5893 from explosion/feature/validate-arg 2020-08-07 15:47:20 +02:00
Adriane Boyd
e962784531
Add Lemmatizer and simplify related components (#5848)
* Add Lemmatizer and simplify related components

* Add `Lemmatizer` pipe with `lookup` and `rule` modes using the
`Lookups` tables.
* Reduce `Tagger` to a simple tagger that sets `Token.tag` (no pos or lemma)
* Reduce `Morphology` to only keep track of morph tags (no tag map, lemmatizer,
or morph rules)
* Remove lemmatizer from `Vocab`
* Adjust many many tests

Differences:

* No default lookup lemmas
* No special treatment of TAG in `from_array` and similar required
* Easier to modify labels in a `Tagger`
* No extra strings added from morphology / tag map

* Fix test

* Initial fix for Lemmatizer config/serialization

* Adjust init test to be more generic

* Adjust init test to force empty Lookups

* Add simple cache to rule-based lemmatizer

* Convert language-specific lemmatizers

Convert language-specific lemmatizers to component lemmatizers. Remove
previous lemmatizer class.

* Fix French and Polish lemmatizers

* Remove outdated UPOS conversions

* Update Russian lemmatizer init in tests

* Add minimal init/run tests for custom lemmatizers

* Add option to overwrite existing lemmas

* Update mode setting, lookup loading, and caching

* Make `mode` an immutable property
* Only enforce strict `load_lookups` for known supported modes
* Move caching into individual `_lemmatize` methods

* Implement strict when lang is not found in lookups

* Fix tables/lookups in make_lemmatizer

* Reallow provided lookups and allow for stricter checks

* Add lookups asset to all Lemmatizer pipe tests

* Rename lookups in lemmatizer init test

* Clean up merge

* Refactor lookup table loading

* Add helper from `load_lemmatizer_lookups` that loads required and
optional lookups tables based on settings provided by a config.

Additional slight refactor of lookups:

* Add `Lookups.set_table` to set a table from a provided `Table`
* Reorder class definitions to be able to specify type as `Table`

* Move registry assets into test methods

* Refactor lookups tables config

Use class methods within `Lemmatizer` to provide the config for
particular modes and to load the lookups from a config.

* Add pipe and score to lemmatizer

* Simplify Tagger.score

* Add missing import

* Clean up imports and auto-format

* Remove unused kwarg

* Tidy up and auto-format

* Update docstrings for Lemmatizer

Update docstrings for Lemmatizer.

Additionally modify `is_base_form` API to take `Token` instead of
individual features.

* Update docstrings

* Remove tag map values from Tagger.add_label

* Update API docs

* Fix relative link in Lemmatizer API docs
2020-08-07 15:27:13 +02:00
Matthew Honnibal
da6e59519e Add docstrings for simple_ner 2020-08-07 15:09:49 +02:00
Matthew Honnibal
7ef8a64df9 Add docstring for parser 2020-08-07 14:59:34 +02:00
Ines Montani
fc9a4fe827 Update attribute ruler 2020-08-07 14:43:55 +02:00
Ines Montani
a8404c3517 validation -> validate 2020-08-07 14:43:47 +02:00
Ines Montani
1d01d89b79 Update CLI docs and evaluate command [ci skip] 2020-08-07 14:40:58 +02:00
Ines Montani
ef2c67cca5
Add DocBin to/from_disk methods and update docs (#5892)
* Add DocBin to/from_disk methods and update docs

* Use DocBin.from_disk in Corpus
2020-08-07 14:30:59 +02:00
Ines Montani
4ca08c6d5d
Merge pull request #5891 from adrianeboyd/docs/attribute-ruler-api
Add AttributeRuler API docs
2020-08-07 13:55:12 +02:00
Adriane Boyd
b8d0c23857 Add AttributeRuler API docs
With additional minor updates to AttributeRuler docstrings.
2020-08-07 12:43:23 +02:00
svlandeg
b17db0e994 Merge remote-tracking branch 'upstream/develop' into feature/el-docs
# Conflicts:
#	website/docs/usage/training.md
2020-08-06 19:48:52 +02:00
Adriane Boyd
06c3a5e048
Add pipe to AttributeRuler (#5889) 2020-08-06 19:43:09 +02:00
Ines Montani
9b7f198390 Fix format 2020-08-06 19:30:53 +02:00
Ines Montani
3c4389110d Remove unused imports 2020-08-06 19:30:47 +02:00
Matthew Honnibal
d4525816ef
Be less choosy about reporting textcat scores (#5879)
* Set textcat scores more consistently

* Refactor textcat scores

* Fixes to scorer

* Add comments

* Add threshold

* Rename just 'f' to micro_f in textcat scorer

* Fix textcat score for two-class

* Fix syntax

* Fix textcat score

* Fix docstring
2020-08-06 16:24:13 +02:00
svlandeg
0b4d1e1bc4 'debug data' instead of 'debug-data' 2020-08-06 15:47:31 +02:00
svlandeg
881e3f8fd0 add docbin explanation and example 2020-08-06 15:29:44 +02:00
Adriane Boyd
5e683a6e46
Fix return values for per feat score (#5885)
* Fix return values for per feat score

Convert `PRFScore` to dict as other per type scores.

* Update tests accordingly
2020-08-06 15:14:47 +02:00
Ines Montani
913d21f0a3
Merge pull request #5882 from explosion/feature/raise-from
Use "raise ... from" in custom errors for better tracebacks
2020-08-06 00:35:26 +02:00
Ines Montani
06e80d95cd
Sync develop with nightly docs state (#5883)
Co-authored-by: svlandeg <sofie.vanlandeghem@gmail.com>
2020-08-06 00:28:14 +02:00
Ines Montani
d92954ac1d
Merge pull request #5881 from explosion/feature/better-error-model-shortcuts 2020-08-06 00:13:35 +02:00
Ines Montani
56c17973aa Use "raise ... from" in custom errors for better tracebacks 2020-08-05 23:53:21 +02:00
Ines Montani
5cc0d89fad
Simplify config overrides in CLI and deserialization (#5880) 2020-08-05 23:35:09 +02:00
Ines Montani
0881455a5d Update error message 2020-08-05 23:15:05 +02:00
Ines Montani
2a1fa86a0d Add better error for failed model shortcut loading 2020-08-05 23:10:29 +02:00
Ines Montani
c675746ca2 Update docstrings and types 2020-08-05 20:29:46 +02:00
Ines Montani
823e533dc1
Add config callbacks for modifying nlp object before and after init (#5866)
* WIP: Concept for modifying nlp object before and after init

* Make callbacks return nlp object

Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>

* Raise if callbacks don't return correct type

* Rename, update types, add after_pipeline_creation

Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
2020-08-05 19:47:54 +02:00
Ines Montani
586d695775 Merge branch 'develop' of https://github.com/explosion/spaCy into develop 2020-08-05 16:01:11 +02:00
Ines Montani
e68459296d Tidy up and auto-format 2020-08-05 16:00:59 +02:00
Matthew Honnibal
50c0e49741 Fix train CLI 2020-08-05 15:40:47 +02:00
Matthew Honnibal
b9df4d6116 Fix textcat.begin_training if vectors set 2020-08-05 15:40:36 +02:00
Adriane Boyd
4193402c47
Add warning when Matcher subpattern is discarded (#5873)
* Add a warning when a subpattern is not processed and discarded

* Normalize subpattern attribute/operator keys to upper case like
top-level attributes
2020-08-05 14:56:14 +02:00
Adriane Boyd
af125875cf
Update SimpleNER (#5878)
* Fix `get_loss` to use NER annotation
* Add labels as part of cfg
* Add simple overfitting test
2020-08-05 14:43:29 +02:00
Sofie Van Landeghem
b88c5c701a
Bugfix in nlp.replace_pipe (#5875)
* bugfix and unit test

* merge two conditions
2020-08-05 09:30:58 +02:00
Ines Montani
b795f02fbd
Allow adding pipeline components from source model (#5857)
* Allow adding pipeline components from source model

* Config: name -> component

* Improve error messages

* Fix error and test

* Add frozen components and exclude logic

* Remove exclude from Language.evaluate

* Init sourced components with current vocab

* Fix error codes
2020-08-04 23:39:19 +02:00
Sofie Van Landeghem
34873c4911
Example Dict format consistency (#5858)
* consistently use upper-case IDS in token_annotation format and for get_aligned

* remove ID from to_dict (not used in from_dict either)

* fix test

Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
2020-08-04 22:22:26 +02:00
Adriane Boyd
fa79a0db9f
Add AttributeRuler for token attribute exceptions (#5842)
* Add AttributeRuler for token attribute exceptions

Add the `AttributeRuler` to handle exceptions for token-level
attributes. The `AttributeRuler` uses `Matcher` patterns to identify
target spans and applies the specified attributes to the token at the
provided index in the matched span. A negative index can be used to
index from the end of the matched span. The retokenizer is used to
"merge" the individual tokens and assign them the provided attributes.

Helper functions can import existing tag maps and morph rules to the
corresponding `Matcher` patterns.

There is an additional minor bug fix for `MORPH` attributes in the
retokenizer to correctly normalize the values and to handle `MORPH`
alongside `_` in an attrs dict.

* Fix default name

* Update name in error message

* Extend AttributeRuler functionality

* Add option to initialize with a dict of AttributeRuler patterns

* Instead of silently discarding overlapping matches (the default
behavior for the retokenizer if only the attrs differ), split the
matches into disjoint sets and retokenize each set separately. This
allows, for instance, one pattern to set the POS and another pattern to
set the lemma. (If two matches modify the same attribute, it looks like
the attrs are applied in the order they were added, but it may not be
deterministic?)

* Improve types

* Sort spans before processing

* Fix index boundaries in Span

* Refactor retokenizer to separate attrs methods

Add top-level `normalize_token_attrs` and `set_token_attrs` methods.

* Update AttributeRuler to use refactored methods

Update `AttributeRuler` to replace use of full retokenizer with only the
relevant methods for normalizing and setting attributes for a single
token.

* Update spacy/pipeline/attributeruler.py

Co-authored-by: Ines Montani <ines@ines.io>

* Make API more similar to EntityRuler

* Add `AttributeRuler.add_patterns` to add patterns from a list of dicts
* Return list of dicts as property `AttributeRuler.patterns`

* Make attrs_unnormed private

* Add test loading patterns from assets

* Revert "Fix index boundaries in Span"

This reverts commit 8f8a5c3386.

* Add Span index boundary checks (#5861)

* Add Span index boundary checks

* Return Span-specific IndexError in all cases

* Simplify and fix if/else

Co-authored-by: Ines Montani <ines@ines.io>
2020-08-04 17:02:39 +02:00
Sofie Van Landeghem
492d1ec5de
Prevent alignment when texts don't match (#5867)
* remove empty gold.pyx

* add alignment unit test (to be used in docs)

* ensure that Alignment is only used on equal texts

* additional test using example.alignment

* formatting

Co-authored-by: Matthew Honnibal <honnibal+gh@gmail.com>
2020-08-04 16:29:18 +02:00
Matthew Honnibal
ecb3c4e8f4
Create corpus iterator and batcher from registry during training (#5865)
* Move batchers into their own module (and registry)

* Update CLI

* Update Corpus and batcher

* Update tests

* Update one config

* Merge 'evaluation' block back under [training]

* Import batchers in gold __init__

* Fix batchers

* Update config

* Update schema

* Update util

* Don't assume train and dev are actually paths

* Update onto-joint config

* Fix missing import

* Format

* Format

* Update spacy/gold/corpus.py

Co-authored-by: Ines Montani <ines@ines.io>

* Fix name

* Update default config

* Fix get_length option in batchers

* Update test

* Add comment

* Pass path into Corpus

* Update docstring

* Update schema and configs

* Update config

* Fix test

* Fix paths

* Fix print

* Fix create_train_batches

* [training.read_train] -> [training.train_corpus]

* Update onto-joint config

Co-authored-by: Ines Montani <ines@ines.io>
2020-08-04 15:09:37 +02:00
Sofie Van Landeghem
82347110f5
Default empty KB in EL component (#5872)
* EL field documentation

* documentation consistent with docs

* default empty KB, initialize vocab separately

* formatting

* add test for changing the default entity vector length

* update comment
2020-08-04 14:34:09 +02:00
Adriane Boyd
b7e3018d97
Recalculate alignment if tokenization differs (#5868)
* Recalculate alignment if tokenization differs

* Refactor cached alignment data
2020-08-04 14:31:32 +02:00
Adriane Boyd
c62fd878a3
Allow Doc.char_span to snap to token boundaries (#5849)
* Allow Doc.char_span to snap to token boundaries

Add a `mode` option to allow `Doc.char_span` to snap to token
boundaries. The `mode` options:

* `strict`: character offsets must match token boundaries (default, same as
before)
* `inside`: all tokens completely within the character span
* `outside`: all tokens at least partially covered by the character span

Add a new helper function `token_by_char` that returns the token
corresponding to a character position in the text. Update
`token_by_start` and `token_by_end` to use `token_by_char` for more
efficient searching.

* Remove unused import

* Rename mode to alignment_mode

Rename `mode` to `alignment_mode` with the options
`strict`/`contract`/`expand`. Any unrecognized modes are silently
converted to `strict`.
2020-08-04 13:36:32 +02:00
Adriane Boyd
b841248589
Add Span index boundary checks (#5861)
* Add Span index boundary checks

* Return Span-specific IndexError in all cases

* Simplify and fix if/else
2020-08-04 13:35:25 +02:00
Adriane Boyd
cd59979ab4
Fix span boundary handling in Spanish noun_chunks (#5860) 2020-08-03 13:53:15 +02:00
Ines Montani
934447a611
Merge pull request #5855 from svlandeg/fix/cli-debug 2020-08-03 13:09:20 +02:00
Ines Montani
4c055f0aa7
Add init CLI and init config (#5854)
* Add init CLI and init config draft

* Improve config validation

* Auto-format

* Don't export anything in debug config

* Update docs
2020-08-02 15:18:30 +02:00
svlandeg
6f4e46ee93 Merge remote-tracking branch 'upstream/develop' into fix/cli-debug
# Conflicts:
#	pyproject.toml
#	requirements.txt
#	setup.cfg
2020-08-01 18:38:59 +02:00
Ines Montani
b40f44419b Simplify pipe analysis
- remove unused code
- don't print by default
- integrate attrs info into analysis output
2020-08-01 13:40:06 +02:00
Ines Montani
b68c53858c Remove global 2020-07-31 18:37:58 +02:00
Ines Montani
30a76fcf6f Integrate and simplify pipe analysis 2020-07-31 18:34:35 +02:00
svlandeg
9b719dfb1a use divider inbetween steps 2020-07-31 18:06:48 +02:00
svlandeg
51ffc4a166 rename pipe_name to component 2020-07-31 17:58:55 +02:00
svlandeg
878327d38e printing final predictions by default to False 2020-07-31 17:36:32 +02:00
Ines Montani
2d955fbf98 Fix linting [ci skip] 2020-07-31 17:05:28 +02:00
Ines Montani
e9e8fa2466 Update docs and types 2020-07-31 17:02:54 +02:00
svlandeg
cc2f58a1b0 use data_validation context manager 2020-07-31 16:49:42 +02:00
Adriane Boyd
ac14ce7c30
Prefer earlier spans in EntityRuler (#5843)
Similar to #4414, update the sorting in EntityRuler to prefer the first
span in overlapping spans.
2020-07-31 16:09:32 +02:00
svlandeg
5fa3235d06 set DATA_VALIDATION to False for debug_model (upgrade thinc) 2020-07-31 15:21:01 +02:00
svlandeg
08d3c36c20 bugfix in train CLI 2020-07-31 15:03:43 +02:00
Adriane Boyd
9b509aa87f Move Language.evaluate scorer config to new arg
Move `Language.evaluate` scorer config from `component_cfg` to separate
argument `scorer_cfg`.
2020-07-31 11:05:16 +02:00
Adriane Boyd
901801b33b Fix default arguments in DependencyParser.score 2020-07-31 10:55:44 +02:00
Adriane Boyd
9d79916792 Merge branch 'develop' into feature/scorer-adjustments 2020-07-31 10:48:14 +02:00
Sofie Van Landeghem
ca491722ad
The Parser is now a Pipe (2) (#5844)
* moving syntax folder to _parser_internals

* moving nn_parser and transition_system

* move nn_parser and transition_system out of internals folder

* moving nn_parser code into transition_system file

* rename transition_system to transition_parser

* moving parser_model and _state to ml

* move _state back to internals

* The Parser now inherits from Pipe!

* small code fixes

* removing unnecessary imports

* remove link_vectors_to_models

* transition_system to internals folder

* little bit more cleanup

* newlines
2020-07-30 23:30:54 +02:00
svlandeg
0b23594953 pipe_name instead of section in debug_model 2020-07-30 20:06:28 +02:00
Rahul Gupta
f76fae0e8d
English: adds ordinal numbers (#5830) 2020-07-29 20:22:47 +02:00
Ines Montani
7a21775cd0
Merge pull request #5834 from explosion/feature/vectors 2020-07-29 18:49:26 +02:00
Gustavo Zadrozny Leyendecker
90b958fd01
Fix on EntityRendered to support break lines (after last entity) (closes #5838) 2020-07-29 18:48:39 +02:00
Ines Montani
b0f57a0cac Update docs and consistency 2020-07-29 15:14:07 +02:00
Matthew Honnibal
a2d573c039 Merge branch 'feature/vectors' of https://github.com/explosion/spaCy into feature/vectors 2020-07-29 14:56:27 +02:00
Matthew Honnibal
2af741d7e3 Fix train arg 2020-07-29 14:56:01 +02:00
Matthew Honnibal
c27309f839
Merge branch 'develop' into feature/vectors 2020-07-29 14:54:10 +02:00
Ines Montani
62266fb828 Fix broken type annotation 2020-07-29 14:49:49 +02:00
Matthew Honnibal
142b58be92 Fix import 2020-07-29 14:45:09 +02:00
Matthew Honnibal
c99a653070 Adjust textcat model 2020-07-29 14:38:15 +02:00
Matthew Honnibal
9e1b11dd81 Update vectors in textcat 2020-07-29 14:35:36 +02:00
Matthew Honnibal
105cf29967 Fix DocBin 2020-07-29 14:23:13 +02:00
Ines Montani
ff0bc05da8 Fix docstrings [ci skip] 2020-07-29 14:09:37 +02:00
Ines Montani
6e2623d3f8 Fix docstring [ci skip] 2020-07-29 14:08:05 +02:00
Ines Montani
8d56260d92 Fix docstrings [ci skip] 2020-07-29 14:07:13 +02:00
Ines Montani
80b18124d2 Fix docstring [ci skip] 2020-07-29 14:03:35 +02:00
Matthew Honnibal
f0cf4a2dca Update tests 2020-07-29 14:01:14 +02:00
Matthew Honnibal
07b47eaac8 Update tok2vec layer 2020-07-29 14:01:13 +02:00
Matthew Honnibal
5ae8628571 Fix CharacterEmbed layer 2020-07-29 14:01:13 +02:00
Matthew Honnibal
97d3651574 Fix stray link_vectors_to_models call 2020-07-29 14:01:13 +02:00
Matthew Honnibal
c7d1ece3eb Update tests 2020-07-29 14:01:13 +02:00
Matthew Honnibal
00de30bcc2 Update CharacterEmbed function 2020-07-29 14:01:12 +02:00
Matthew Honnibal
6a6b09bd32 Update morphologizer model 2020-07-29 14:01:12 +02:00
Matthew Honnibal
20e9098e3f Update tests 2020-07-29 14:01:12 +02:00
Matthew Honnibal
c35d6282fc Add previous HashEmbedCNN tok2vec to make transition easier 2020-07-29 14:01:12 +02:00
Matthew Honnibal
1784c95827 Clean up link_vectors_to_models unused stuff 2020-07-29 14:01:11 +02:00
Matthew Honnibal
0c17ea4c85 Format 2020-07-29 14:00:13 +02:00
Matthew Honnibal
2aff3c4b5a Load vectors in 'spacy train' 2020-07-29 14:00:13 +02:00
Matthew Honnibal
7852a68a75 Fix load_vectors_into_model function 2020-07-29 14:00:13 +02:00
Matthew Honnibal
7299419fe4 Dont load vectors in Language.from_config 2020-07-29 14:00:12 +02:00
Matthew Honnibal
30dd96c540 Load vectors in Language.from_config 2020-07-29 14:00:12 +02:00
Matthew Honnibal
df95e2af64 Add load_vectors_into_model util 2020-07-29 14:00:12 +02:00
Matthew Honnibal
475d7c1c7c Fix StaticVectors class 2020-07-29 14:00:11 +02:00
Matthew Honnibal
44d350dc94 Use spaCy's StaticVectors 2020-07-29 14:00:11 +02:00
Matthew Honnibal
acc64e138a Add import 2020-07-29 14:00:11 +02:00
Matthew Honnibal
9987ea9e4d Fix Tok2Vec begin_training 2020-07-29 14:00:10 +02:00
Matthew Honnibal
099e9331c5 Fix tok2vec 2020-07-29 14:00:10 +02:00
Matthew Honnibal
fe0cdcd461 Fixes 2020-07-29 14:00:09 +02:00
Matthew Honnibal
123f8b832d Refactor Tok2Vec model 2020-07-29 14:00:09 +02:00
Matthew Honnibal
c6b4f63c7c Remove obsolete function 2020-07-29 14:00:09 +02:00
Matthew Honnibal
9cc7262224 Draft StaticVectors layer 2020-07-29 14:00:09 +02:00
Matthew Honnibal
cb9654e98c WIP on new StaticVectors 2020-07-29 14:00:09 +02:00
Ines Montani
e257e66ab9 Merge branch 'develop' of https://github.com/explosion/spaCy into develop 2020-07-29 11:36:45 +02:00
Ines Montani
e0ffe36e79 Update docstrings, docs and types 2020-07-29 11:36:42 +02:00
Sofie Van Landeghem
40c995b1be
Option for returning only greedy matches (#5771)
* add "greedy" option for match pattern

* distinction between greedy FIRST or LONGEST

* check for proper values, throw custom warning otherwise

* unxfail one more test

* add comment in docstring

* add test that LONGEST also prefers first match if equal length

* use c arrays for more efficient processing

* rename 'greediness' to 'greedy'
2020-07-29 11:04:43 +02:00
Adriane Boyd
191a12d75f
Fix score_weights typo in train CLI (#5835) 2020-07-29 11:04:12 +02:00
Adriane Boyd
0cddb0dbe9
Move timing into Language.evaluate (#5836)
Move timing into `Language.evaluate` so that only the processing is
timing, not processing + scoring. `Language.evaluate` returns
`scores["speed"]` as words per second, which should be identical to how
the speed was added to the scores previously. Also add the speed to the
evaluate CLI output.
2020-07-29 11:02:31 +02:00
Adriane Boyd
c689ae8f0a Fix types in Scorer 2020-07-29 10:40:30 +02:00
oculusrepairo
03ab518f28
Update examples.py (#5820)
* Update examples.py

adding factual sentences to the list

* Add missing comma separators

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
2020-07-29 10:28:56 +02:00
Ines Montani
7adffc5361 Remove unused schema 2020-07-28 23:12:47 +02:00
Ines Montani
e5d9eaf79c Tidy up docstrings and arguments 2020-07-28 23:12:42 +02:00
Ines Montani
ac24adec73 Small adjustments to Scorer and docs 2020-07-28 21:39:42 +02:00
Ines Montani
2c7a32cf12 Remove unused methods 2020-07-28 16:50:02 +02:00
Ines Montani
ba22111ff4 Move error to Errors 2020-07-28 16:24:14 +02:00
Ines Montani
2748249217 Re-add meta["pipeline"] for now 2020-07-28 16:14:23 +02:00
Ines Montani
b83ead5bf5
Merge pull request #5824 from svlandeg/fix/textcat-v3 2020-07-28 15:04:25 +02:00
Ines Montani
06a97a8766 Support --opt=value format in CLI config overrides 2020-07-28 13:43:15 +02:00
Ines Montani
ae4d8a6ffd Update docstrings, docs and pipe consistency 2020-07-28 13:37:31 +02:00
Ines Montani
0094cb0d04 Remove scores list from config and document 2020-07-28 11:22:24 +02:00
graue70
b97dbab998
Fix typo in unit tests (#5823) 2020-07-27 20:18:48 +02:00
Ines Montani
894e20c466 Merge branch 'develop' into feature/component-scores 2020-07-27 18:14:39 +02:00
Ines Montani
d8b519c23c API docs, docstrings and argument consistency 2020-07-27 18:11:45 +02:00
svlandeg
85b2dcfd67 cleanup 2020-07-27 17:54:44 +02:00
svlandeg
61068e0fb1 util function dot_to_object and corresponding unit test 2020-07-27 17:50:12 +02:00
Ines Montani
10b84e1e27 Add flag to toggle sdist creation on package [ci skip] 2020-07-27 16:52:23 +02:00
Adriane Boyd
34c92dfe63 Add missing Scorer imports 2020-07-27 15:08:51 +02:00
Adriane Boyd
8bb0507777 Add and update score methods and score weights
Add and update `score` methods, provided `scores`, and default weights
`default_score_weights` for pipeline components.

* `scores` provides all top-level keys returned by `score` (merely informative, similar to `assigns`).
* `default_score_weights` provides the default weights for a default config.
* The keys from `default_score_weights` determine which values will be
shown in the `spacy train` output, so keys with weight `0.0` will be
displayed but not counted toward the overall score.
2020-07-27 14:44:53 +02:00
Adriane Boyd
baf19fd652 Update cats scoring to provide overall score
* Provide top-level score as `attr_score`
* Provide a description of the score as `attr_score_desc`
* Provide all potential scores keys, setting unused keys to `None`
* Update CLI evaluate accordingly
2020-07-27 12:26:10 +02:00
Adriane Boyd
f8cf378be9 Combine weights from multiple components
Combine weights from multiple components for the same score.
2020-07-27 10:21:31 +02:00
Ines Montani
3d56a3f286 Make more args keyword-only 2020-07-27 00:27:53 +02:00
Matthew Honnibal
80271ac0ba Update default config 2020-07-26 15:27:39 +02:00
Ines Montani
ed61fb10fc Rename default textcat arch to TextCatEnsemble 2020-07-26 15:11:43 +02:00
Ines Montani
53d37da29a Make sure @factories is removed from config 2020-07-26 15:11:24 +02:00
Ines Montani
4060c2d5a6 Fix test 2020-07-26 13:40:19 +02:00
Ines Montani
2470486543 Allow pipeline components to set default scores and weights 2020-07-26 13:18:43 +02:00
Ines Montani
787d066e22 Remove pipes.pyx
Probably accidentally re-added in a merge?
2020-07-26 13:08:52 +02:00
Matthew Honnibal
520d25cb50
Add smart_open dependency to fetch project assets (#5812)
* Use smart_open for project assets

* Fix assets.py

* Update pyproject.toml
2020-07-26 12:15:00 +02:00
Ines Montani
e92df281ce Tidy up, autoformat, add types 2020-07-25 15:01:15 +02:00
Matthew Honnibal
71242327b2 Set version to v3.0.0a5 2020-07-25 14:06:01 +02:00
Ines Montani
cdbd6ba912
Merge pull request #5798 from explosion/feature/language-data-config 2020-07-25 13:34:49 +02:00
Ines Montani
49f27a2a7b Tidy up [ci skip] 2020-07-25 13:00:49 +02:00
Ines Montani
4a0a692875 Add missing lex_attr_getters (resolves #5806 ) 2020-07-25 12:55:18 +02:00
Adriane Boyd
2bcceb80c4
Refactor the Scorer to improve flexibility (#5731)
* Refactor the Scorer to improve flexibility

Refactor the `Scorer` to improve flexibility for arbitrary pipeline
components.

* Individual pipeline components provide their own `evaluate` methods
that score a list of `Example`s and return a dictionary of scores
* `Scorer` is initialized either:
  * with a provided pipeline containing components to be scored
  * with a default pipeline containing the built-in statistical
    components (senter, tagger, morphologizer, parser, ner)
* `Scorer.score` evaluates a list of `Example`s and returns a dictionary
of scores referring to the scores provided by the components in the
pipeline

Significant differences:

* `tags_acc` is renamed to `tag_acc` to be consistent with `token_acc`
and the new `morph_acc`, `pos_acc`, and `lemma_acc`
* Scoring is no longer cumulative: `Scorer.score` scores a list of
examples rather than a single example and does not retain any state
about previously scored examples
* PRF values in the returned scores are no longer multiplied by 100

* Add kwargs to Morphologizer.evaluate

* Create generalized scoring methods in Scorer

* Generalized static scoring methods are added to `Scorer`
  * Methods require an attribute (either on Token or Doc) that is
used to key the returned scores

Naming differences:

* `uas`, `las`, and `las_per_type` in the scores dict are renamed to
`dep_uas`, `dep_las`, and `dep_las_per_type`

Scoring differences:

* `Doc.sents` is now scored as spans rather than on sentence-initial
token positions so that `Doc.sents` and `Doc.ents` can be scored with
the same method (this lowers scores since a single incorrect sentence
start results in two incorrect spans)

* Simplify / extend hasattr check for eval method

* Add hasattr check to tokenizer scoring
* Simplify to hasattr check for component scoring

* Reset Example alignment if docs are set

Reset the Example alignment if either doc is set in case the
tokenization has changed.

* Add PRF tokenization scoring for tokens as spans

Add PRF scores for tokens as character spans. The scores are:

* token_acc: # correct tokens / # gold tokens
* token_p/r/f: PRF for (token.idx, token.idx + len(token))

* Add docstring to Scorer.score_tokenization

* Rename component.evaluate() to component.score()

* Update Scorer API docs

* Update scoring for positive_label in textcat

* Fix TextCategorizer.score kwargs

* Update Language.evaluate docs

* Update score names in default config
2020-07-25 12:53:02 +02:00
Ines Montani
c003d26b94 Tidy up 2020-07-25 12:21:37 +02:00