Adriane Boyd bf0cdae8d4 Add token_splitter component (#6726) ... * Add long_token_splitter component Add a `long_token_splitter` component for use with transformer pipelines. This component splits up long tokens like URLs into smaller tokens. This is particularly relevant for pretrained pipelines with `strided_spans`, since the user can't change the length of the span `window` and may not wish to preprocess the input texts. The `long_token_splitter` splits tokens that are at least `long_token_length` tokens long into smaller tokens of `split_length` size. Notes: * Since this is intended for use as the first component in a pipeline, the token splitter does not try to preserve any token annotation. * API docs to come when the API is stable. * Adjust API, add test * Fix name in factory		2021-01-17 19:54:41 +08:00
..
cli	Tidy up and auto-format	2021-01-15 11:57:36 +11:00
displacy	Refactor Docs.is_ flags (#6044)	2020-09-17 00:14:01 +02:00
lang	Remove unused is_base_form for mk lemmatizer (#6743)	2021-01-17 09:41:35 +01:00
matcher	Add SPACY as a Matcher attribute (#6463)	2020-11-30 09:34:50 +08:00
ml	Tidy up and auto-format	2021-01-15 11:57:36 +11:00
pipeline	Add token_splitter component (#6726)	2021-01-17 19:54:41 +08:00
tests	Add token_splitter component (#6726)	2021-01-17 19:54:41 +08:00
tokens	Handle unset token.morph in Morphologizer (#6704)	2021-01-15 17:20:10 +01:00
training	Questionable fix for parser training bug with misaligned sentences (#6694)	2021-01-15 14:18:24 +01:00
__init__.pxd	* Seems to be working after refactor. Need to wire up more POS tag features, and wire up save/load of POS tags.	2014-10-24 02:23:42 +11:00
__init__.py	require_cpu functionality (#6336)	2020-12-08 14:42:40 +08:00
__main__.py	Tidy up	2020-06-22 00:45:40 +02:00
about.py	Set version to v2.3.5	2020-12-10 13:32:10 +01:00
attrs.pxd	Merge branch 'develop' into master-tmp	2020-05-21 18:39:06 +02:00
attrs.pyx	Merge branch 'develop' into master-tmp	2020-05-21 18:39:06 +02:00
compat.py	Use Literal type for nr_feature_tokens	2020-09-23 16:00:03 +02:00
default_config_pretraining.cfg	pretrain architectures (#6451)	2020-12-08 14:41:03 +08:00
default_config.cfg	Add initialize.before_init and after_init callbacks	2021-01-12 13:07:44 +01:00
errors.py	Validate seed and gpu_allocator manually	2021-01-14 16:57:57 +01:00
glossary.py	unicode -> str consistency	2020-05-24 17:20:58 +02:00
kb.pxd	Revert added_strings change (#6236)	2020-10-10 18:55:07 +02:00
kb.pyx	Revert added_strings change (#6236)	2020-10-10 18:55:07 +02:00
language.py	Add all strings from source models (#6736)	2021-01-16 12:26:15 +11:00
lexeme.pxd	Fix Lexeme.from_ptr	2020-08-10 16:43:37 +02:00
lexeme.pyx	Update docs links in codebase	2020-09-04 12:58:50 +02:00
lookups.py	Always serialize lookups and vectors to disk	2020-10-05 09:40:20 +02:00
morphology.pxd	Add Lemmatizer and simplify related components (#5848)	2020-08-07 15:27:13 +02:00
morphology.pyx	Prevent 0-length mem alloc (#6653)	2021-01-06 12:50:17 +11:00
parts_of_speech.pxd	Add support for Universal Dependencies v2.0	2017-03-03 13:17:34 +01:00
parts_of_speech.pyx	Drop Python 2.7 and 3.5 (#4828)	2019-12-22 01:53:56 +01:00
pipe_analysis.py	Tidy up and auto-format	2020-09-29 21:39:28 +02:00
schemas.py	Add initialize.before_init and after_init callbacks	2021-01-12 13:07:44 +01:00
scorer.py	multi-label textcat component (#6474)	2021-01-06 13:07:14 +11:00
strings.pxd	Remove 'cleanup' of strings (#6007)	2020-09-01 16:12:15 +02:00
strings.pyx	Update docs links in codebase	2020-09-04 12:58:50 +02:00
structs.pxd	Add SpanGroup and Graph container types to represent arbitrary annotations (#6696)	2021-01-14 17:30:41 +11:00
symbols.pxd	introduce token.has_head and refer to MISSING_DEP_ (WIP)	2021-01-12 17:17:06 +01:00
symbols.pyx	introduce token.has_head and refer to MISSING_DEP_ (WIP)	2021-01-12 17:17:06 +01:00
tokenizer.pxd	Simplify specials and cache checks (#6012)	2020-09-03 09:42:49 +02:00
tokenizer.pyx	Merge remote-tracking branch 'upstream/master' into chore/update-develop-from-master-rc3	2021-01-14 11:49:58 +01:00
typedefs.pxd	Merge remote-tracking branch 'upstream/master' into chore/update-develop-from-master	2020-11-25 11:49:34 +01:00
typedefs.pyx	Tidy up rest	2017-10-27 21:07:59 +02:00
util.py	Tidy up and auto-format	2021-01-05 13:41:53 +11:00
vectors.pyx	Update docs links in codebase	2020-09-04 12:58:50 +02:00
vocab.pxd	Merge remote-tracking branch 'upstream/master' into chore/update-develop-from-master	2020-11-25 11:49:34 +01:00
vocab.pyx	Prevent 0-length mem alloc (#6653)	2021-01-06 12:50:17 +11:00