spaCy/tests/tokenizer/test_only_punct.py

from __future__ import unicode_literals


def test_only_pre1(en_tokenizer):
    assert len(en_tokenizer("(")) == 1


def test_only_pre2(en_tokenizer):
    assert len(en_tokenizer("((")) == 2
* Refactor tokenization, enable cache, and ensure we look up specials correctly even when there's confusing punctuation surrounding the token. 2014-09-16 20:01:46 +04:00			`from __future__ import unicode_literals`


* Set up tokenizer/ tests properly, using a session-scoped fixture to avoid long load/unload times. Tokenizer tests now complete in 20 seconds. 2015-06-07 18:24:49 +03:00			`def test_only_pre1(en_tokenizer):`
			`assert len(en_tokenizer("(")) == 1`
Tweak line spacing 2015-04-19 22:39:18 +03:00
* Refactor tokenization, enable cache, and ensure we look up specials correctly even when there's confusing punctuation surrounding the token. 2014-09-16 20:01:46 +04:00
* Set up tokenizer/ tests properly, using a session-scoped fixture to avoid long load/unload times. Tokenizer tests now complete in 20 seconds. 2015-06-07 18:24:49 +03:00			`def test_only_pre2(en_tokenizer):`
			`assert len(en_tokenizer("((")) == 2`