adrianeboyd 2c876eb672 Add tokenizer explain() debugging method (#4596) ... * Expose tokenizer rules as a property Expose the tokenizer rules property in the same way as the other core properties. (The cache resetting is overkill, but consistent with `from_bytes` for now.) Add tests and update Tokenizer API docs. * Update Hungarian punctuation to remove empty string Update Hungarian punctuation definitions so that `_units` does not match an empty string. * Use _load_special_tokenization consistently Use `_load_special_tokenization()` and have it to handle `None` checks. * Fix precedence of `token_match` vs. special cases Remove `token_match` check from `_split_affixes()` so that special cases have precedence over `token_match`. `token_match` is checked only before infixes are split. * Add `make_debug_doc()` to the Tokenizer Add `make_debug_doc()` to the Tokenizer as a working implementation of the pseudo-code in the docs. Add a test (marked as slow) that checks that `nlp.tokenizer()` and `nlp.tokenizer.make_debug_doc()` return the same non-whitespace tokens for all languages that have `examples.sentences` that can be imported. * Update tokenization usage docs Update pseudo-code and algorithm description to correspond to `nlp.tokenizer.make_debug_doc()` with example debugging usage. Add more examples for customizing tokenizers while preserving the existing defaults. Minor edits / clarifications. * Revert "Update Hungarian punctuation to remove empty string" This reverts commit f0a577f7a5. * Rework `make_debug_doc()` as `explain()` Rework `make_debug_doc()` as `explain()`, which returns a list of `(pattern_string, token_string)` tuples rather than a non-standard `Doc`. Update docs and tests accordingly, leaving the visualization for future work. * Handle cases with bad tokenizer patterns Detect when tokenizer patterns match empty prefixes and suffixes so that `explain()` does not hang on bad patterns. * Remove unused displacy image * Add tokenizer.explain() to usage docs		2019-11-20 13:07:25 +01:00
..
__init__.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_customized_tokenizer.py	Add tokenizer explain() debugging method (#4596)	2019-11-20 13:07:25 +01:00
test_exceptions.py	Move lookup tables out of the core library (#4346)	2019-10-01 00:01:27 +02:00
test_indices.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_noun_chunks.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_parser.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_prefix_suffix_infix.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_punct.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_sbd.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_tagger.py	Revert #4334	2019-09-29 17:32:12 +02:00
test_text.py	Revert #4334	2019-09-29 17:32:12 +02:00