36 Commits

Author	SHA1	Message	Date
Matthew Honnibal	936e6246aa	Fix Issue #459 -- failed to deserialize empty doc.	2016-10-23 16:31:05 +02:00
Matthew Honnibal	49c117960c	Fix bug where huffman codec died if given empty freqs dict.	2016-10-23 16:28:05 +02:00
Matthew Honnibal	ea4ff94cde	* Whitespace	2016-01-29 03:59:22 +01:00
Matthew Honnibal	1ce5d5602d	* Rename Doc.data to Doc.c	2015-11-04 00:17:13 +11:00
Matthew Honnibal	25a8774f42	* Fix regression in packer	2015-07-27 21:53:38 +02:00
Matthew Honnibal	1601e488ee	* Fix bug in decoding non-ascii characters	2015-07-27 21:43:58 +02:00
Matthew Honnibal	6a95409cd2	* Fix type on bits	2015-07-27 21:16:49 +02:00
Matthew Honnibal	45460f505c	* Fix data type on read32 in BitArray	2015-07-27 21:12:13 +02:00
Matthew Honnibal	af6ed18f2a	* Ensure we don't use orth_encode on OOV words.	2015-07-27 02:12:01 +02:00
Matthew Honnibal	d62eb34b76	* More Py 2/3 compatibility in bit strings	2015-07-24 04:52:06 +02:00
Matthew Honnibal	c4ff410fdb	* Fix bytes problems for Python3	2015-07-24 03:48:23 +02:00
Matthew Honnibal	1ab25e4dad	* Fix python3 type error	2015-07-24 02:45:34 +02:00
Matthew Honnibal	f35ff173b0	* Fix bits.pyx unicode error	2015-07-23 20:37:57 +02:00
Matthew Honnibal	1406e24327	* Fix unicode error for Python3	2015-07-23 19:36:21 +02:00
Matthew Honnibal	1f31d96bf9	* Fix Packer API, so that it reads and writes bytes strings, instead of BitArray. Docs are always byte aligned anyway.	2015-07-23 01:13:02 +02:00
Matthew Honnibal	815bda201d	* Remove UniStr struct	2015-07-22 13:39:17 +02:00
Matthew Honnibal	424854028f	* Fix decode_int32	2015-07-21 20:09:59 +00:00
Matthew Honnibal	304d0e2633	* Use decode_int32 in _orth_decode	2015-07-21 20:40:55 +02:00
Matthew Honnibal	9cfa59ec33	* Optimistically try orth encoding, with char as a back-off	2015-07-21 20:22:45 +02:00
Matthew Honnibal	c8b89e37a5	* Bug fix to faster huffman decoding	2015-07-21 20:05:53 +02:00
Matthew Honnibal	b166d1d2a2	* Use encode32 and decode32	2015-07-21 19:59:06 +02:00
Matthew Honnibal	c6cd0ddce8	* Add faster encode_int32 and decode_int32 methods	2015-07-21 19:58:45 +02:00
Matthew Honnibal	bb0ba1f0cd	* Improve serialization speed	2015-07-20 03:27:59 +02:00
Matthew Honnibal	5a7d060d9c	* Switch between the orth and char codecs depending on which is shorter for that message. Mostly orth is shorter, except if there are OOV words.	2015-07-20 01:36:22 +02:00
Matthew Honnibal	5a042ee0d3	* Add function to predict number of bits needed to encode message	2015-07-20 01:35:11 +02:00
Matthew Honnibal	b89b489bb4	* Implement both character and orth encoding in Packer, so that we can decide which to use per-text	2015-07-19 22:39:45 +02:00
Matthew Honnibal	ae78c9e3ce	* Implement character-based codec, so that we can do word/char backoff	2015-07-19 22:03:39 +02:00
Matthew Honnibal	cd1d047cb8	* Delete out-dated HuffmanCodec comment	2015-07-19 18:28:14 +02:00
Matthew Honnibal	b8086067d5	* Build Huffman codec from unsorted inputs	2015-07-19 17:58:44 +02:00
Matthew Honnibal	317cbbc015	* Serialization round trip now working with decent API, but with rough spots in the organisation and requiring vocabulary to be fixed ahead of time.	2015-07-19 15:18:17 +02:00
Matthew Honnibal	5b4c78bbb2	* Use an AttributeCodec based on orth for words. Still no oov handling mechanism.	2015-07-18 22:43:18 +02:00
Matthew Honnibal	cd91914dd8	* Fix hard-coded length	2015-07-18 04:09:56 +02:00
Matthew Honnibal	cf0c788892	* Tests passing on round-trip pack/unpack on basic example	2015-07-17 21:20:48 +02:00
Matthew Honnibal	9e3f17051b	* Move to ORTH instead of ID for encoding lexemes. Basic tests of the codec wrappers now passing	2015-07-17 16:38:29 +02:00
Matthew Honnibal	234c7e440a	* Add spacy/serialize/__init__ files	2015-07-17 01:37:33 +02:00
Matthew Honnibal	db9dfd2e23	* Major refactor of serialization. Nearly complete now.	2015-07-17 01:27:54 +02:00