11168 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Matthew Honnibal	f11c8e22eb	* Remove happax stuff	2014-08-02 22:11:28 +01:00
Matthew Honnibal	6050209f65	* Update gitignore	2014-08-02 22:06:30 +01:00
Matthew Honnibal	b555e2dc5d	* Add hash tests	2014-08-02 21:58:31 +01:00
Matthew Honnibal	d6e07aa922	* Switch to 32bit hash for strings	2014-08-02 21:51:52 +01:00
Matthew Honnibal	365a2af756	* Restore happax. commit uncommited work	2014-08-02 21:27:03 +01:00
Matthew Honnibal	6319ff0f22	* Add length property	2014-08-02 21:26:44 +01:00
Matthew Honnibal	18fb76b2c4	* Removed happax. Not sure if good idea.	2014-08-02 20:53:35 +01:00
Matthew Honnibal	edd38a84b1	* Removing happax stuff. Added length	2014-08-02 20:45:12 +01:00
Matthew Honnibal	fc7c10d7f8	* Ugly but seemingly working fix to the token memory leak	2014-08-01 09:43:19 +01:00
Matthew Honnibal	c7bb6b329c	* Don't free clobbered lexemes, as they might be part of a tail	2014-08-01 08:22:38 +01:00
Matthew Honnibal	c48214460e	* Free lexemes clobbered as happaxes	2014-08-01 07:40:20 +01:00
Matthew Honnibal	5b6457e80e	* Free lexemes clobbered as happaxes	2014-08-01 07:37:50 +01:00
Matthew Honnibal	d8cb2288ce	* Roll back to using murmurhash2 for now	2014-08-01 07:28:47 +01:00
Matthew Honnibal	d4b8bc07ce	* Use FixedTable to control index size	2014-08-01 07:27:48 +01:00
Matthew Honnibal	f39211b2b1	* Add FixedTable for hashing	2014-08-01 07:27:21 +01:00
Matthew Honnibal	a44e15f623	* Hack around lack of distribution features for now.	2014-07-31 18:24:51 +01:00
Matthew Honnibal	4cb88c940b	* Fix memory leak in tokenizer, caused by having a fixed vocab.	2014-07-31 18:19:38 +01:00
Matthew Honnibal	5b81ee716f	* Use a sparse_hash_map to store happax vocab items, with a max size.	2014-07-31 17:40:43 +01:00
Matthew Honnibal	a235804730	* Fix setup.py	2014-07-31 02:03:53 +01:00
Matthew Honnibal	5461399924	* Fix setup.py	2014-07-31 02:03:10 +01:00
Matthew Honnibal	e5da104bc7	* Remove ext files	2014-07-25 16:48:26 +01:00
Matthew Honnibal	b9016c4633	* Switch to using sparsehash and murmurhash libraries out of pip	2014-07-25 15:47:27 +01:00
Matthew Honnibal	3f7cbb93e0	* Upd requirements	2014-07-23 17:50:34 +01:00
Matthew Honnibal	d2a151ec75	* Push to master	2014-07-23 17:39:02 +01:00
Matthew Honnibal	150cf6dd3b	* Merge string views feature branch	2014-07-23 17:37:48 +01:00
Matthew Honnibal	e80ec5525f	* Delete docs attempt	2014-07-23 17:36:37 +01:00
Matthew Honnibal	e494494d80	* Add tests for group_by	2014-07-23 17:36:12 +01:00
Matthew Honnibal	a895fe5ddb	* Upd from spacy	2014-07-23 17:35:18 +01:00
Matthew Honnibal	87bf205b82	* Fix open apostrophe bug	2014-07-07 23:26:01 +02:00
Matthew Honnibal	bc6c1f6156	* Add test for open apostrophe bug	2014-07-07 23:24:20 +02:00
Matthew Honnibal	571808a274	Group-by seems to be working	2014-07-07 20:27:02 +02:00
Matthew Honnibal	80b36f9f27	* 710k words per second for counts	2014-07-07 19:12:19 +02:00
Matthew Honnibal	057c21969b	* Refactor for string view features. Working on setting up flags and enums.	2014-07-07 16:58:48 +02:00
Matthew Honnibal	9fd085bf90	Merge branch 'feature/group_by' into develop	2014-07-07 12:51:40 +02:00
Matthew Honnibal	1c5ab3b49a	* Add tokens module to setup	2014-07-07 12:51:23 +02:00
Matthew Honnibal	f1bcbd4c4e	* Reorganized code to accomodate Tokens class. Need string views before group_by and count_by can be done well.	2014-07-07 12:47:21 +02:00
Matthew Honnibal	6da8010577	Merge branch 'feature/scale' into develop	2014-07-07 08:15:54 +02:00
Matthew Honnibal	6668e44961	* Whitespace	2014-07-07 08:15:44 +02:00
Matthew Honnibal	0074ae2fc0	* Switch to dynamically allocating array, based on the document length	2014-07-07 08:05:29 +02:00
Matthew Honnibal	0575f16ade	* Upd requirements	2014-07-07 07:37:29 +02:00
Matthew Honnibal	ff1869ff07	* Fixed major efficiency problem, from not quite grokking pass by reference in cython c++	2014-07-07 07:36:43 +02:00
Matthew Honnibal	aaae66114c	Merge branch 'feature/contractions' into develop	2014-07-07 05:11:43 +02:00
Matthew Honnibal	e60b958b7d	* Add test to check how well we match ptb tokenizer. Needs more text.	2014-07-07 05:11:31 +02:00
Matthew Honnibal	2c431f9fdc	* Upd tokenization test	2014-07-07 05:11:04 +02:00
Matthew Honnibal	0c76143b72	* Give value for assert	2014-07-07 05:10:46 +02:00
Matthew Honnibal	648d1fe3ed	* Compile en_ptb	2014-07-07 05:10:28 +02:00
Matthew Honnibal	60a2dac13c	* Upd gitignore	2014-07-07 05:10:19 +02:00
Matthew Honnibal	e244739dfe	* Fix ptb tokenization	2014-07-07 05:10:09 +02:00
Matthew Honnibal	cc3971ce5c	* Fix error in tokenization rules	2014-07-07 05:09:34 +02:00
Matthew Honnibal	997551241f	* Upd ptb tokenization rules	2014-07-07 05:09:22 +02:00