spaCy/spacy/tests/tokenizer/test_tokens_from_list.py

from __future__ import unicode_literals
import pytest


def test1(en_tokenizer):
    words = ['JAPAN', 'GET', 'LUCKY']
    tokens = en_tokenizer.tokens_from_list(words)
    assert len(tokens) == 3
    assert tokens[0].orth_ == 'JAPAN'
* Commit outstanding tests 2014-11-12 15:24:32 +03:00			`from __future__ import unicode_literals`
* Tests passing except for morphology/lemmatization stuff 2014-12-23 03:40:32 +03:00			`import pytest`
* Commit outstanding tests 2014-11-12 15:24:32 +03:00

* Set up tokenizer/ tests properly, using a session-scoped fixture to avoid long load/unload times. Tokenizer tests now complete in 20 seconds. 2015-06-07 18:24:49 +03:00			`def test1(en_tokenizer):`
* Commit outstanding tests 2014-11-12 15:24:32 +03:00			`words = ['JAPAN', 'GET', 'LUCKY']`
* Set up tokenizer/ tests properly, using a session-scoped fixture to avoid long load/unload times. Tokenizer tests now complete in 20 seconds. 2015-06-07 18:24:49 +03:00			`tokens = en_tokenizer.tokens_from_list(words)`
* Commit outstanding tests 2014-11-12 15:24:32 +03:00			`assert len(tokens) == 3`
* Upd tests for new meaning of 'string' 2015-01-23 23:22:30 +03:00			`assert tokens[0].orth_ == 'JAPAN'`