2 Commits

Author	SHA1	Message	Date
Matthew Honnibal	e5ad35787c	WIP on adding split-token actions to parser ... This patch starts getting the StateC object ready to split tokens. The split function is implemented by pushing indices into the buffer that indicate an out-of-length token. Still todo: * Update the oracles * Update GoldParseC * Interpret the parse once it's complete * Add retokenizer.split() method	2018-03-31 20:05:27 +02:00
Matthew Honnibal	e0375132bd	Add state tests, esp. for split function	2018-03-30 13:25:46 +02:00