ines a8e58e04ef Add symbols class to punctuation rules to handle emoji (see #1088) ... Currently doesn't work for Hungarian, because of conflicts with the custom punctuation rules. Also doesn't take multi-character emoji like 👩🏽‍💻 into account.		2017-05-27 17:57:10 +02:00
..
__init__.py	add __init__.py to empty package dirs	2016-03-14 11:28:03 +01:00
sun.txt	Move text file back to tokenizer tests directory	2017-01-12 02:10:23 +01:00
test_exceptions.py	Add symbols class to punctuation rules to handle emoji (see #1088)	2017-05-27 17:57:10 +02:00
test_tokenizer.py	Remove unused utf8open util and replace os.path with ensure_path	2017-04-16 20:37:45 +02:00
test_urls.py	Fix xpassing URL test case	2017-04-07 17:36:05 +02:00
test_whitespace.py	Move non-English-specific tests back to general tokenizer tests	2017-01-05 18:09:29 +01:00