14 Commits

Author	SHA1	Message	Date
Matthew Honnibal	01469b0888	* Refactor spacy so that chunks return arrays of lexemes, so that there is properly one lexeme per word.	2014-08-18 19:14:00 +02:00
Matthew Honnibal	b555e2dc5d	* Add hash tests	2014-08-02 21:58:31 +01:00
Matthew Honnibal	6319ff0f22	* Add length property	2014-08-02 21:26:44 +01:00
Matthew Honnibal	e494494d80	* Add tests for group_by	2014-07-23 17:36:12 +01:00
Matthew Honnibal	bc6c1f6156	* Add test for open apostrophe bug	2014-07-07 23:24:20 +02:00
Matthew Honnibal	e60b958b7d	* Add test to check how well we match ptb tokenizer. Needs more text.	2014-07-07 05:11:31 +02:00
Matthew Honnibal	2c431f9fdc	* Upd tokenization test	2014-07-07 05:11:04 +02:00
Matthew Honnibal	25849fc926	* Generalize tokenization rules to capitals	2014-07-07 05:07:21 +02:00
Matthew Honnibal	e4263a241a	* Tests passing for reorganized version	2014-07-07 04:23:46 +02:00
Matthew Honnibal	12f8a0e3c2	* Tests passing for reorganized version	2014-07-07 04:23:20 +02:00
Matthew Honnibal	a62c38e1ef	* Working tokenization. en doesn't match PTB perfectly. Need to reorganize before adding more schemes.	2014-07-07 01:15:59 +02:00
Matthew Honnibal	4e79446dc2	* Reading in tokenization rules correctly. Passing tests.	2014-07-07 00:02:55 +02:00
Matthew Honnibal	9bef797afe	* Rejigged tests. Working possessives, but no other contractions	2014-07-06 20:02:00 +02:00
Matthew Honnibal	556f6a18ca	* Initial commit. Tests passing for punctuation handling. Need contractions, file transport, tokenize function, etc.	2014-07-05 20:51:42 +02:00