Adriane Boyd 0e7f94b247 Update Tokenizer.explain with special matches (#7749) ... * Update Tokenizer.explain with special matches Update `Tokenizer.explain` and the pseudo-code in the docs to include the processing of special cases that contain affixes or whitespace. * Handle optional settings in explain * Add test for special matches in explain Add test for `Tokenizer.explain` for special cases containing affixes.		2021-04-19 19:08:20 +10:00
..
__init__.py	Revert #4334	2019-09-29 17:32:12 +02:00
sun.txt	Revert #4334	2019-09-29 17:32:12 +02:00
test_exceptions.py	Tidy up and auto-format	2020-02-18 15:38:18 +01:00
test_explain.py	Update Tokenizer.explain with special matches (#7749)	2021-04-19 19:08:20 +10:00
test_naughty_strings.py	Merge branch 'develop' into master-tmp	2020-09-04 13:15:36 +02:00
test_tokenizer.py	Use special matcher for exceptions with spaces (#6668)	2021-01-06 12:05:10 +08:00
test_urls.py	Merge branch 'develop' into master-tmp	2020-06-20 15:52:00 +02:00
test_whitespace.py	Merge branch 'develop' into master-tmp	2020-09-04 13:15:36 +02:00