explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-12-25 17:36:30 +03:00
Code Issues Packages Projects Releases Wiki Activity
960 Commits 70 Branches 177 Tags 279 MiB
e775e05313
Commit Graph

960 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Matthew Honnibal
fc7c10d7f8 * Ugly but seemingly working fix to the token memory leak 2014-08-01 09:43:19 +01:00
Matthew Honnibal
c7bb6b329c * Don't free clobbered lexemes, as they might be part of a tail 2014-08-01 08:22:38 +01:00
Matthew Honnibal
c48214460e * Free lexemes clobbered as happaxes 2014-08-01 07:40:20 +01:00
Matthew Honnibal
5b6457e80e * Free lexemes clobbered as happaxes 2014-08-01 07:37:50 +01:00
Matthew Honnibal
d8cb2288ce * Roll back to using murmurhash2 for now 2014-08-01 07:28:47 +01:00
Matthew Honnibal
d4b8bc07ce * Use FixedTable to control index size 2014-08-01 07:27:48 +01:00
Matthew Honnibal
f39211b2b1 * Add FixedTable for hashing 2014-08-01 07:27:21 +01:00
Matthew Honnibal
a44e15f623 * Hack around lack of distribution features for now. 2014-07-31 18:24:51 +01:00
Matthew Honnibal
4cb88c940b * Fix memory leak in tokenizer, caused by having a fixed vocab. 2014-07-31 18:19:38 +01:00
Matthew Honnibal
5b81ee716f * Use a sparse_hash_map to store happax vocab items, with a max size. 2014-07-31 17:40:43 +01:00
Matthew Honnibal
a235804730 * Fix setup.py 2014-07-31 02:03:53 +01:00
Matthew Honnibal
5461399924 * Fix setup.py 2014-07-31 02:03:10 +01:00
Matthew Honnibal
e5da104bc7 * Remove ext files 2014-07-25 16:48:26 +01:00
Matthew Honnibal
b9016c4633 * Switch to using sparsehash and murmurhash libraries out of pip 2014-07-25 15:47:27 +01:00
Matthew Honnibal
3f7cbb93e0 * Upd requirements 2014-07-23 17:50:34 +01:00
Matthew Honnibal
d2a151ec75 * Push to master 2014-07-23 17:39:02 +01:00
Matthew Honnibal
150cf6dd3b * Merge string views feature branch 2014-07-23 17:37:48 +01:00
Matthew Honnibal
e80ec5525f * Delete docs attempt 2014-07-23 17:36:37 +01:00
Matthew Honnibal
e494494d80 * Add tests for group_by 2014-07-23 17:36:12 +01:00
Matthew Honnibal
a895fe5ddb * Upd from spacy 2014-07-23 17:35:18 +01:00
Matthew Honnibal
87bf205b82 * Fix open apostrophe bug 2014-07-07 23:26:01 +02:00
Matthew Honnibal
bc6c1f6156 * Add test for open apostrophe bug 2014-07-07 23:24:20 +02:00
Matthew Honnibal
571808a274 Group-by seems to be working 2014-07-07 20:27:02 +02:00
Matthew Honnibal
80b36f9f27 * 710k words per second for counts 2014-07-07 19:12:19 +02:00
Matthew Honnibal
057c21969b * Refactor for string view features. Working on setting up flags and enums. 2014-07-07 16:58:48 +02:00
Matthew Honnibal
9fd085bf90 Merge branch 'feature/group_by' into develop 2014-07-07 12:51:40 +02:00
Matthew Honnibal
1c5ab3b49a * Add tokens module to setup 2014-07-07 12:51:23 +02:00
Matthew Honnibal
f1bcbd4c4e * Reorganized code to accomodate Tokens class. Need string views before group_by and count_by can be done well. 2014-07-07 12:47:21 +02:00
Matthew Honnibal
6da8010577 Merge branch 'feature/scale' into develop 2014-07-07 08:15:54 +02:00
Matthew Honnibal
6668e44961 * Whitespace 2014-07-07 08:15:44 +02:00
Matthew Honnibal
0074ae2fc0 * Switch to dynamically allocating array, based on the document length 2014-07-07 08:05:29 +02:00
Matthew Honnibal
0575f16ade * Upd requirements 2014-07-07 07:37:29 +02:00
Matthew Honnibal
ff1869ff07 * Fixed major efficiency problem, from not quite grokking pass by reference in cython c++ 2014-07-07 07:36:43 +02:00
Matthew Honnibal
aaae66114c Merge branch 'feature/contractions' into develop 2014-07-07 05:11:43 +02:00
Matthew Honnibal
e60b958b7d * Add test to check how well we match ptb tokenizer. Needs more text. 2014-07-07 05:11:31 +02:00
Matthew Honnibal
2c431f9fdc * Upd tokenization test 2014-07-07 05:11:04 +02:00
Matthew Honnibal
0c76143b72 * Give value for assert 2014-07-07 05:10:46 +02:00
Matthew Honnibal
648d1fe3ed * Compile en_ptb 2014-07-07 05:10:28 +02:00
Matthew Honnibal
60a2dac13c * Upd gitignore 2014-07-07 05:10:19 +02:00
Matthew Honnibal
e244739dfe * Fix ptb tokenization 2014-07-07 05:10:09 +02:00
Matthew Honnibal
cc3971ce5c * Fix error in tokenization rules 2014-07-07 05:09:34 +02:00
Matthew Honnibal
997551241f * Upd ptb tokenization rules 2014-07-07 05:09:22 +02:00
Matthew Honnibal
dc20500920 * Remove cpp files 2014-07-07 05:09:05 +02:00
Matthew Honnibal
25849fc926 * Generalize tokenization rules to capitals 2014-07-07 05:07:21 +02:00
Matthew Honnibal
df0458001d * Begin work on full PTB-compatible English tokenization 2014-07-07 04:29:24 +02:00
Matthew Honnibal
0c1be7effe * Compile string_tools module 2014-07-07 04:24:00 +02:00
Matthew Honnibal
e4263a241a * Tests passing for reorganized version 2014-07-07 04:23:46 +02:00
Matthew Honnibal
12f8a0e3c2 * Tests passing for reorganized version 2014-07-07 04:23:20 +02:00
Matthew Honnibal
d5bef02c72 * Reorganized, moving language-independent stuff to spacy. The functions in spacy ask for the dictionaries and split function on input, but the language-specific modules are curried versions that use the globals 2014-07-07 04:21:06 +02:00
Matthew Honnibal
a62c38e1ef * Working tokenization. en doesn't match PTB perfectly. Need to reorganize before adding more schemes. 2014-07-07 01:15:59 +02:00