explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-12-25 17:36:30 +03:00
Code Issues Packages Projects Releases Wiki Activity
86 Commits 70 Branches 177 Tags 279 MiB
3379d7a571
Commit Graph

86 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Matthew Honnibal
3379d7a571 * Reforming data model for lexemes 2014-08-19 02:40:37 +02:00
Matthew Honnibal
e091b6a241 * Base master on temp branch 2014-08-18 23:29:21 +02:00
Matthew Honnibal
85df22c379 * Remove murmurhash from requirements 2014-08-18 23:26:20 +02:00
Matthew Honnibal
ab9b0daabf * Whitespace 2014-08-18 23:21:49 +02:00
Matthew Honnibal
a2047fa5aa * Add 's suffix to tokenization table 2014-08-18 23:21:37 +02:00
Matthew Honnibal
1b71cbfe28 * Roll back to using unicode, and never Py_UNICODE. No dependence on murmurhash either. 2014-08-18 20:48:48 +02:00
Matthew Honnibal
bbf9a2c944 * Working version that uses arrays for chunks, which should be more memory efficient 2014-08-18 20:23:54 +02:00
Matthew Honnibal
8d3f6082be * Working version, adding improvements 2014-08-18 19:59:59 +02:00
Matthew Honnibal
01469b0888 * Refactor spacy so that chunks return arrays of lexemes, so that there is properly one lexeme per word. 2014-08-18 19:14:00 +02:00
Matthew Honnibal
b94c9b72c9 * WordTree in use. Need to reform the way chunks are handled. Should be properly one Lexeme per word, with split points being the things that are cached. 2014-08-16 20:10:22 +02:00
Matthew Honnibal
34b68a18ab * Progress to getting WordTree working. Tests pass, but so far it's slower. 2014-08-16 19:59:38 +02:00
Matthew Honnibal
865cacfaf7 * Remove dependence on murmurhash 2014-08-16 17:37:09 +02:00
Matthew Honnibal
515d41d325 * Restore string saving to spacy 2014-08-16 16:09:24 +02:00
Matthew Honnibal
36073b89fe * Restore unicode, work on improving string storage. 2014-08-16 14:35:34 +02:00
Matthew Honnibal
a225ca5b0d * Refactoring tokenizer 2014-08-16 03:22:03 +02:00
Matthew Honnibal
213a440ffc * Add string decode and encode helpers to string_tools 2014-08-15 23:57:27 +02:00
Matthew Honnibal
7fd9b2f1f8 * Add murmurhash to setup while we figure out cython includes 2014-08-15 23:56:57 +02:00
Matthew Honnibal
3ada25b92d * Shifting to WordTree instead of dense_hash_map for storage. 2014-08-15 23:06:46 +02:00
Matthew Honnibal
f11c8e22eb * Remove happax stuff 2014-08-02 22:11:28 +01:00
Matthew Honnibal
6050209f65 * Update gitignore 2014-08-02 22:06:30 +01:00
Matthew Honnibal
b555e2dc5d * Add hash tests 2014-08-02 21:58:31 +01:00
Matthew Honnibal
d6e07aa922 * Switch to 32bit hash for strings 2014-08-02 21:51:52 +01:00
Matthew Honnibal
365a2af756 * Restore happax. commit uncommited work 2014-08-02 21:27:03 +01:00
Matthew Honnibal
6319ff0f22 * Add length property 2014-08-02 21:26:44 +01:00
Matthew Honnibal
18fb76b2c4 * Removed happax. Not sure if good idea. 2014-08-02 20:53:35 +01:00
Matthew Honnibal
edd38a84b1 * Removing happax stuff. Added length 2014-08-02 20:45:12 +01:00
Matthew Honnibal
fc7c10d7f8 * Ugly but seemingly working fix to the token memory leak 2014-08-01 09:43:19 +01:00
Matthew Honnibal
c7bb6b329c * Don't free clobbered lexemes, as they might be part of a tail 2014-08-01 08:22:38 +01:00
Matthew Honnibal
c48214460e * Free lexemes clobbered as happaxes 2014-08-01 07:40:20 +01:00
Matthew Honnibal
5b6457e80e * Free lexemes clobbered as happaxes 2014-08-01 07:37:50 +01:00
Matthew Honnibal
d8cb2288ce * Roll back to using murmurhash2 for now 2014-08-01 07:28:47 +01:00
Matthew Honnibal
d4b8bc07ce * Use FixedTable to control index size 2014-08-01 07:27:48 +01:00
Matthew Honnibal
f39211b2b1 * Add FixedTable for hashing 2014-08-01 07:27:21 +01:00
Matthew Honnibal
a44e15f623 * Hack around lack of distribution features for now. 2014-07-31 18:24:51 +01:00
Matthew Honnibal
4cb88c940b * Fix memory leak in tokenizer, caused by having a fixed vocab. 2014-07-31 18:19:38 +01:00
Matthew Honnibal
5b81ee716f * Use a sparse_hash_map to store happax vocab items, with a max size. 2014-07-31 17:40:43 +01:00
Matthew Honnibal
a235804730 * Fix setup.py 2014-07-31 02:03:53 +01:00
Matthew Honnibal
5461399924 * Fix setup.py 2014-07-31 02:03:10 +01:00
Matthew Honnibal
e5da104bc7 * Remove ext files 2014-07-25 16:48:26 +01:00
Matthew Honnibal
b9016c4633 * Switch to using sparsehash and murmurhash libraries out of pip 2014-07-25 15:47:27 +01:00
Matthew Honnibal
3f7cbb93e0 * Upd requirements 2014-07-23 17:50:34 +01:00
Matthew Honnibal
d2a151ec75 * Push to master 2014-07-23 17:39:02 +01:00
Matthew Honnibal
150cf6dd3b * Merge string views feature branch 2014-07-23 17:37:48 +01:00
Matthew Honnibal
e80ec5525f * Delete docs attempt 2014-07-23 17:36:37 +01:00
Matthew Honnibal
e494494d80 * Add tests for group_by 2014-07-23 17:36:12 +01:00
Matthew Honnibal
a895fe5ddb * Upd from spacy 2014-07-23 17:35:18 +01:00
Matthew Honnibal
87bf205b82 * Fix open apostrophe bug 2014-07-07 23:26:01 +02:00
Matthew Honnibal
bc6c1f6156 * Add test for open apostrophe bug 2014-07-07 23:24:20 +02:00
Matthew Honnibal
571808a274 Group-by seems to be working 2014-07-07 20:27:02 +02:00
Matthew Honnibal
80b36f9f27 * 710k words per second for counts 2014-07-07 19:12:19 +02:00