1
1
mirror of https://github.com/explosion/spaCy.git synced 2025-01-25 00:34:20 +03:00
Commit Graph

1 Commits

Author SHA1 Message Date
Matthew Honnibal
bb911e5f4e Fix : 'subtok' label being added even if learn_tokens=False ()
* Prevent subtok label if not learning tokens

The parser introduces the subtok label to mark tokens that should be
merged during post-processing. Previously this happened even if we did
not have the --learn-tokens flag set. This patch passes the config
through to the parser, to prevent the problem.

* Make merge_subtokens a parser post-process if learn_subtokens

* Fix train script

* Add test for 3830: subtok problem

* Fix handlign of non-subtok in parser training
2019-08-23 17:54:00 +02:00