Sofie 46dfe773e1 Replacing regex library with re to increase tokenization speed (#3218) ... * replace unicode categories with raw list of code points * simplifying ranges * fixing variable length quotes * removing redundant regular expression * small cleanup of regexp notations * quotes and alpha as ranges instead of alterations * removed most regexp dependencies and features * exponential backtracking - unit tests * rewrote expression with pathological backtracking * disabling double hyphen tests for now * test additional variants of repeating punctuation * remove regex and redundant backslashes from load_reddit script * small typo fixes * disable double punctuation test for russian * clean up old comments * format block code * final cleanup * naming consistency * french strings as unicode for python 2 support * french regular expression case insensitive		2019-02-01 18:05:22 +11:00
..
__init__.py	Remove imports in /lang/__init__.py	2017-05-08 23:58:07 +02:00
test_customized_tokenizer.py	Replacing regex library with re to increase tokenization speed (#3218)	2019-02-01 18:05:22 +11:00
test_exceptions.py	Replacing regex library with re to increase tokenization speed (#3218)	2019-02-01 18:05:22 +11:00
test_indices.py	💫 Refactor test suite (#2568)	2018-07-24 23:38:44 +02:00
test_noun_chunks.py	Tidy up and format remaining files	2018-11-30 17:43:08 +01:00
test_parser.py	💫 Tidy up and auto-format tests (#2967)	2018-11-27 01:09:36 +01:00
test_prefix_suffix_infix.py	💫 Tidy up and auto-format tests (#2967)	2018-11-27 01:09:36 +01:00
test_punct.py	Replacing regex library with re to increase tokenization speed (#3218)	2019-02-01 18:05:22 +11:00
test_sbd.py	💫 Tidy up and auto-format tests (#2967)	2018-11-27 01:09:36 +01:00
test_tagger.py	💫 Tidy up and auto-format tests (#2967)	2018-11-27 01:09:36 +01:00
test_text.py	💫 Tidy up and auto-format tests (#2967)	2018-11-27 01:09:36 +01:00