Matthew Honnibal
|
0c507bd80a
|
* Fix tokenizer
|
2015-07-22 14:10:30 +02:00 |
|
Matthew Honnibal
|
c86dbe4944
|
* Update English.save_models for new Packer save/load stuff
|
2015-07-22 13:40:23 +02:00 |
|
Matthew Honnibal
|
bf77bcd6b9
|
* Add comment explaining hash_string
|
2015-07-22 13:39:42 +02:00 |
|
Matthew Honnibal
|
815bda201d
|
* Remove UniStr struct
|
2015-07-22 13:39:17 +02:00 |
|
Matthew Honnibal
|
2fc66e3723
|
* Use Py_UNICODE in tokenizer for now, while sort out Py_UCS4 stuff
|
2015-07-22 13:38:45 +02:00 |
|
Matthew Honnibal
|
4d61239eac
|
* Reorganize the serialization functions on Doc
|
2015-07-22 04:53:01 +02:00 |
|
Matthew Honnibal
|
109106a949
|
* Replace UniStr, using unicode objects instead
|
2015-07-22 04:52:05 +02:00 |
|
Matthew Honnibal
|
424854028f
|
* Fix decode_int32
|
2015-07-21 20:09:59 +00:00 |
|
Matthew Honnibal
|
304d0e2633
|
* Use decode_int32 in _orth_decode
|
2015-07-21 20:40:55 +02:00 |
|
Matthew Honnibal
|
9cfa59ec33
|
* Optimistically try orth encoding, with char as a back-off
|
2015-07-21 20:22:45 +02:00 |
|
Matthew Honnibal
|
c8b89e37a5
|
* Bug fix to faster huffman decoding
|
2015-07-21 20:05:53 +02:00 |
|
Matthew Honnibal
|
b166d1d2a2
|
* Use encode32 and decode32
|
2015-07-21 19:59:06 +02:00 |
|
Matthew Honnibal
|
c6cd0ddce8
|
* Add faster encode_int32 and decode_int32 methods
|
2015-07-21 19:58:45 +02:00 |
|
Matthew Honnibal
|
dd60594f41
|
* Fix double encoding error in strings.pyx
|
2015-07-20 13:52:56 +02:00 |
|
Matthew Honnibal
|
06639dc497
|
* Add length cap to word shape feature
|
2015-07-20 12:06:59 +02:00 |
|
Matthew Honnibal
|
128b6d9714
|
* Move Utf8Str struct to strings module, as that's the only place it's relevant
|
2015-07-20 12:06:41 +02:00 |
|
Matthew Honnibal
|
01a97b90f3
|
* Fix header for string store
|
2015-07-20 12:06:10 +02:00 |
|
Matthew Honnibal
|
52d538ea42
|
* Fix short string optimization in strings.pyx. StringStore tests now all pass.
|
2015-07-20 12:05:23 +02:00 |
|
Matthew Honnibal
|
09a3055630
|
* Work on short string optimization in Utf8Str
|
2015-07-20 11:26:46 +02:00 |
|
Matthew Honnibal
|
bb0ba1f0cd
|
* Improve serialization speed
|
2015-07-20 03:27:59 +02:00 |
|
Matthew Honnibal
|
8743a8c084
|
* Update Doc serialization for new Packer interface
|
2015-07-20 01:38:04 +02:00 |
|
Matthew Honnibal
|
1f7170e0e1
|
* Reinstate the fixed vocabulary --- words are only added to the lexicon in init_model, after that we create LexemeC structs with the Pool given to us.
|
2015-07-20 01:37:34 +02:00 |
|
Matthew Honnibal
|
5a7d060d9c
|
* Switch between the orth and char codecs depending on which is shorter for that message. Mostly orth is shorter, except if there are OOV words.
|
2015-07-20 01:36:22 +02:00 |
|
Matthew Honnibal
|
5a042ee0d3
|
* Add function to predict number of bits needed to encode message
|
2015-07-20 01:35:11 +02:00 |
|
Matthew Honnibal
|
b89b489bb4
|
* Implement both character and orth encoding in Packer, so that we can decide which to use per-text
|
2015-07-19 22:39:45 +02:00 |
|
Matthew Honnibal
|
ae78c9e3ce
|
* Implement character-based codec, so that we can do word/char backoff
|
2015-07-19 22:03:39 +02:00 |
|
Matthew Honnibal
|
cd1d047cb8
|
* Delete out-dated HuffmanCodec comment
|
2015-07-19 18:28:14 +02:00 |
|
Matthew Honnibal
|
b8086067d5
|
* Build Huffman codec from unsorted inputs
|
2015-07-19 17:58:44 +02:00 |
|
Matthew Honnibal
|
317cbbc015
|
* Serialization round trip now working with decent API, but with rough spots in the organisation and requiring vocabulary to be fixed ahead of time.
|
2015-07-19 15:18:17 +02:00 |
|
Matthew Honnibal
|
6b13e7227c
|
* Remove duplicate get_lex_attr method from doc.pyx
|
2015-07-18 22:46:07 +02:00 |
|
Matthew Honnibal
|
e49c7f1478
|
* Update oov check in tokenizer
|
2015-07-18 22:45:28 +02:00 |
|
Matthew Honnibal
|
cfd842769e
|
* Allow infix tokens to be variable length
|
2015-07-18 22:45:00 +02:00 |
|
Matthew Honnibal
|
5b4c78bbb2
|
* Use an AttributeCodec based on orth for words. Still no oov handling mechanism.
|
2015-07-18 22:43:18 +02:00 |
|
Matthew Honnibal
|
82d84b0f2b
|
* Index lexemes by orth, instead of a lexemes vector. Breaks the mechanism for deciding not to own LexemeC structs during parsing. Need to reinstate this.
|
2015-07-18 22:42:15 +02:00 |
|
Matthew Honnibal
|
4dddc8a69b
|
* Fix type declarations for attr_t. Remove unused id_t.
|
2015-07-18 22:39:57 +02:00 |
|
Matthew Honnibal
|
ced59ab9ea
|
* Make minor efficiency improvement in Doc.__iter__
|
2015-07-18 04:10:53 +02:00 |
|
Matthew Honnibal
|
cd91914dd8
|
* Fix hard-coded length
|
2015-07-18 04:09:56 +02:00 |
|
Matthew Honnibal
|
b1d74ce60d
|
* Remove unused joint.pyx and joint.pxd files
|
2015-07-17 23:31:44 +02:00 |
|
Matthew Honnibal
|
c27514512b
|
* Remove cruft ner/ directory
|
2015-07-17 23:24:32 +02:00 |
|
Matthew Honnibal
|
f8d6d319f4
|
* Remove cruft module
|
2015-07-17 23:23:05 +02:00 |
|
Matthew Honnibal
|
fb0a641a2d
|
* Don't release the gil around Parser.parse. Does this indicate thread problems?
|
2015-07-17 23:07:37 +02:00 |
|
Matthew Honnibal
|
e29daea85f
|
* Fix bint/int typing problem in TransitionSystem. In C++ bint* means bool*, but in C it means int*. So, type-casting to bint* is unsafe.
|
2015-07-17 22:37:24 +02:00 |
|
Matthew Honnibal
|
cf0c788892
|
* Tests passing on round-trip pack/unpack on basic example
|
2015-07-17 21:20:48 +02:00 |
|
Matthew Honnibal
|
44f39a876f
|
* Add a blank attrs.pyx
|
2015-07-17 16:40:42 +02:00 |
|
Matthew Honnibal
|
c2c83120d4
|
* Remove codec property from Vocab
|
2015-07-17 16:40:11 +02:00 |
|
Matthew Honnibal
|
dfdf19f6a9
|
* Draft a from_orth method for Doc
|
2015-07-17 16:39:54 +02:00 |
|
Matthew Honnibal
|
9e3f17051b
|
* Move to ORTH instead of ID for encoding lexemes. Basic tests of the codec wrappers now passing
|
2015-07-17 16:38:29 +02:00 |
|
Matthew Honnibal
|
15ff739996
|
* Fix passing of ID attribute in string store
|
2015-07-17 14:49:42 +02:00 |
|
Matthew Honnibal
|
95e57c2780
|
* Remove unnecessary key and id properties from Utf8String.
|
2015-07-17 01:40:18 +02:00 |
|
Matthew Honnibal
|
234c7e440a
|
* Add spacy/serialize/__init__ files
|
2015-07-17 01:37:33 +02:00 |
|