5 Commits

Author	SHA1	Message	Date
Chris DuBois	dac8fe7bdb	Add __reduce__ to Tokenizer so that English pickles. ... - Add tests to test_pickle and test_tokenizer that save to tempfiles.	2015-10-23 22:24:03 -07:00
Matthew Honnibal	bd4f5f89cb	* Add note about failed tokenization	2015-06-08 16:17:07 +02:00
Matthew Honnibal	7195c07897	* Add explicit tokenizer test	2015-06-07 21:29:17 +02:00
Matthew Honnibal	877abb0e5b	* Set up tokenizer/ tests properly, using a session-scoped fixture to avoid long load/unload times. Tokenizer tests now complete in 20 seconds.	2015-06-07 17:24:49 +02:00
Matthew Honnibal	d37dca72dd	* Reorganize tests	2015-06-07 16:49:46 +02:00