spaCy/tests/tokenizer/test_infix.py

from __future__ import unicode_literals

import pytest

def test_hyphen(en_tokenizer):
    tokens = en_tokenizer('best-known')
    assert len(tokens) == 3


def test_period(en_tokenizer):
    tokens = en_tokenizer('best.Known')
    assert len(tokens) == 3
    tokens = en_tokenizer('zombo.com')
    assert len(tokens) == 1
* Test hyphenation etc 2014-10-14 13:26:16 +04:00			`from __future__ import unicode_literals`

			`import pytest`

* Set up tokenizer/ tests properly, using a session-scoped fixture to avoid long load/unload times. Tokenizer tests now complete in 20 seconds. 2015-06-07 18:24:49 +03:00			`def test_hyphen(en_tokenizer):`
			`tokens = en_tokenizer('best-known')`
* Restore hyphenation test to test_infix 2015-06-06 06:57:36 +03:00			`assert len(tokens) == 3`
* Test hyphenation etc 2014-10-14 13:26:16 +04:00

* Set up tokenizer/ tests properly, using a session-scoped fixture to avoid long load/unload times. Tokenizer tests now complete in 20 seconds. 2015-06-07 18:24:49 +03:00			`def test_period(en_tokenizer):`
			`tokens = en_tokenizer('best.Known')`
* Test hyphenation etc 2014-10-14 13:26:16 +04:00			`assert len(tokens) == 3`
* Set up tokenizer/ tests properly, using a session-scoped fixture to avoid long load/unload times. Tokenizer tests now complete in 20 seconds. 2015-06-07 18:24:49 +03:00			`tokens = en_tokenizer('zombo.com')`
* Test hyphenation etc 2014-10-14 13:26:16 +04:00			`assert len(tokens) == 1`