4 Commits

Author	SHA1	Message	Date
Matthew Honnibal	5233f110c4	* Adding PTB3 tokenizer back in, so can understand how much boilerplate is in the docs for multiple tokenizers	2014-08-29 02:30:27 +02:00
Matthew Honnibal	9815c7649e	* Refactor around Word objects, adapting tests. Tests passing, except for string views.	2014-08-23 19:55:06 +02:00
Matthew Honnibal	e289896603	* Fix ptb3 module	2014-08-22 16:36:17 +02:00
Matthew Honnibal	89d6faa9c9	* Move en_ptb to ptb3	2014-08-22 04:24:05 +02:00