explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-12-25 09:26:27 +03:00
Code Issues Packages Projects Releases Wiki Activity
15,624 Commits 70 Branches 177 Tags 279 MiB
2e52479eec
Commit Graph

15624 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Matthew Honnibal
515d41d325 * Restore string saving to spacy 2014-08-16 16:09:24 +02:00
Matthew Honnibal
36073b89fe * Restore unicode, work on improving string storage. 2014-08-16 14:35:34 +02:00
Matthew Honnibal
a225ca5b0d * Refactoring tokenizer 2014-08-16 03:22:03 +02:00
Matthew Honnibal
213a440ffc * Add string decode and encode helpers to string_tools 2014-08-15 23:57:27 +02:00
Matthew Honnibal
7fd9b2f1f8 * Add murmurhash to setup while we figure out cython includes 2014-08-15 23:56:57 +02:00
Matthew Honnibal
3ada25b92d * Shifting to WordTree instead of dense_hash_map for storage. 2014-08-15 23:06:46 +02:00
Matthew Honnibal
f11c8e22eb * Remove happax stuff 2014-08-02 22:11:28 +01:00
Matthew Honnibal
6050209f65 * Update gitignore 2014-08-02 22:06:30 +01:00
Matthew Honnibal
b555e2dc5d * Add hash tests 2014-08-02 21:58:31 +01:00
Matthew Honnibal
d6e07aa922 * Switch to 32bit hash for strings 2014-08-02 21:51:52 +01:00
Matthew Honnibal
365a2af756 * Restore happax. commit uncommited work 2014-08-02 21:27:03 +01:00
Matthew Honnibal
6319ff0f22 * Add length property 2014-08-02 21:26:44 +01:00
Matthew Honnibal
18fb76b2c4 * Removed happax. Not sure if good idea. 2014-08-02 20:53:35 +01:00
Matthew Honnibal
edd38a84b1 * Removing happax stuff. Added length 2014-08-02 20:45:12 +01:00
Matthew Honnibal
fc7c10d7f8 * Ugly but seemingly working fix to the token memory leak 2014-08-01 09:43:19 +01:00
Matthew Honnibal
c7bb6b329c * Don't free clobbered lexemes, as they might be part of a tail 2014-08-01 08:22:38 +01:00
Matthew Honnibal
c48214460e * Free lexemes clobbered as happaxes 2014-08-01 07:40:20 +01:00
Matthew Honnibal
5b6457e80e * Free lexemes clobbered as happaxes 2014-08-01 07:37:50 +01:00
Matthew Honnibal
d8cb2288ce * Roll back to using murmurhash2 for now 2014-08-01 07:28:47 +01:00
Matthew Honnibal
d4b8bc07ce * Use FixedTable to control index size 2014-08-01 07:27:48 +01:00
Matthew Honnibal
f39211b2b1 * Add FixedTable for hashing 2014-08-01 07:27:21 +01:00
Matthew Honnibal
a44e15f623 * Hack around lack of distribution features for now. 2014-07-31 18:24:51 +01:00
Matthew Honnibal
4cb88c940b * Fix memory leak in tokenizer, caused by having a fixed vocab. 2014-07-31 18:19:38 +01:00
Matthew Honnibal
5b81ee716f * Use a sparse_hash_map to store happax vocab items, with a max size. 2014-07-31 17:40:43 +01:00
Matthew Honnibal
a235804730 * Fix setup.py 2014-07-31 02:03:53 +01:00
Matthew Honnibal
5461399924 * Fix setup.py 2014-07-31 02:03:10 +01:00
Matthew Honnibal
e5da104bc7 * Remove ext files 2014-07-25 16:48:26 +01:00
Matthew Honnibal
b9016c4633 * Switch to using sparsehash and murmurhash libraries out of pip 2014-07-25 15:47:27 +01:00
Matthew Honnibal
3f7cbb93e0 * Upd requirements 2014-07-23 17:50:34 +01:00
Matthew Honnibal
d2a151ec75 * Push to master 2014-07-23 17:39:02 +01:00
Matthew Honnibal
150cf6dd3b * Merge string views feature branch 2014-07-23 17:37:48 +01:00
Matthew Honnibal
e80ec5525f * Delete docs attempt 2014-07-23 17:36:37 +01:00
Matthew Honnibal
e494494d80 * Add tests for group_by 2014-07-23 17:36:12 +01:00
Matthew Honnibal
a895fe5ddb * Upd from spacy 2014-07-23 17:35:18 +01:00
Matthew Honnibal
87bf205b82 * Fix open apostrophe bug 2014-07-07 23:26:01 +02:00
Matthew Honnibal
bc6c1f6156 * Add test for open apostrophe bug 2014-07-07 23:24:20 +02:00
Matthew Honnibal
571808a274 Group-by seems to be working 2014-07-07 20:27:02 +02:00
Matthew Honnibal
80b36f9f27 * 710k words per second for counts 2014-07-07 19:12:19 +02:00
Matthew Honnibal
057c21969b * Refactor for string view features. Working on setting up flags and enums. 2014-07-07 16:58:48 +02:00
Matthew Honnibal
9fd085bf90 Merge branch 'feature/group_by' into develop 2014-07-07 12:51:40 +02:00
Matthew Honnibal
1c5ab3b49a * Add tokens module to setup 2014-07-07 12:51:23 +02:00
Matthew Honnibal
f1bcbd4c4e * Reorganized code to accomodate Tokens class. Need string views before group_by and count_by can be done well. 2014-07-07 12:47:21 +02:00
Matthew Honnibal
6da8010577 Merge branch 'feature/scale' into develop 2014-07-07 08:15:54 +02:00
Matthew Honnibal
6668e44961 * Whitespace 2014-07-07 08:15:44 +02:00
Matthew Honnibal
0074ae2fc0 * Switch to dynamically allocating array, based on the document length 2014-07-07 08:05:29 +02:00
Matthew Honnibal
0575f16ade * Upd requirements 2014-07-07 07:37:29 +02:00
Matthew Honnibal
ff1869ff07 * Fixed major efficiency problem, from not quite grokking pass by reference in cython c++ 2014-07-07 07:36:43 +02:00
Matthew Honnibal
aaae66114c Merge branch 'feature/contractions' into develop 2014-07-07 05:11:43 +02:00
Matthew Honnibal
e60b958b7d * Add test to check how well we match ptb tokenizer. Needs more text. 2014-07-07 05:11:31 +02:00
Matthew Honnibal
2c431f9fdc * Upd tokenization test 2014-07-07 05:11:04 +02:00