spaCy/spacy/strings.pxd

from cymem.cymem cimport Pool
from preshed.maps cimport PreshMap
from murmurhash.mrmr cimport hash64
from .typedefs cimport attr_t

from libc.stdint cimport int64_t

from .typedefs cimport hash_t

cpdef hash_t hash_string(unicode string) except 0


ctypedef union Utf8Str:
    unsigned char[8] s
    unsigned char* p


cdef class StringStore:
    cdef Pool mem
    cdef Utf8Str* c
    cdef int64_t size

    cdef public PreshMap _map
    cdef int64_t _resize_at

    cdef const Utf8Str* intern(self, unicode py_string) except NULL
    cdef const Utf8Str* _intern_utf8(self, char* utf8_string, int length) except NULL
* Move StringStore class to its own file 2014-12-19 22:42:01 +03:00			`from cymem.cymem cimport Pool`
			`from preshed.maps cimport PreshMap`
* Move murmurhash to .pxd in strings file 2014-12-19 23:41:08 +03:00			`from murmurhash.mrmr cimport hash64`
* Fix type declarations for attr_t. Remove unused id_t. 2015-07-18 23:39:57 +03:00			`from .typedefs cimport attr_t`

			`from libc.stdint cimport int64_t`
* Move StringStore class to its own file 2014-12-19 22:42:01 +03:00
* Tmp commit. Refactoring to create a Python Lexeme class. 2015-01-12 02:26:22 +03:00			`from .typedefs cimport hash_t`

			`cpdef hash_t hash_string(unicode string) except 0`
* Move murmurhash to .pxd in strings file 2014-12-19 23:41:08 +03:00

* Fix header for string store 2015-07-20 13:06:10 +03:00			`ctypedef union Utf8Str:`
			`unsigned char[8] s`
			`unsigned char* p`


* Move StringStore class to its own file 2014-12-19 22:42:01 +03:00			`cdef class StringStore:`
			`cdef Pool mem`
* Fix header for string store 2015-07-20 13:06:10 +03:00			`cdef Utf8Str* c`
* Fix type declarations for attr_t. Remove unused id_t. 2015-07-18 23:39:57 +03:00			`cdef int64_t size`
* Move StringStore class to its own file 2014-12-19 22:42:01 +03:00
* Use unicode in StringStore.intern, instead of unreliably casting to bytes. 2015-11-05 14:32:19 +03:00			`cdef public PreshMap _map`
			`cdef int64_t _resize_at`
* Move StringStore class to its own file 2014-12-19 22:42:01 +03:00
Revert "Work on Issue #285: intern strings into document-specific pools, to address streaming data memory growth. StringStore.__getitem__ now raises KeyError when it can't find the string. Use StringStore.intern() to get the old behaviour. Still need to hunt down all uses of StringStore.__getitem__ in library and do testing, but logic looks good." This reverts commit 8423e8627f0c9e54ab026d5c51701e9e0a9c38d2. 2016-09-30 21:20:22 +03:00			`cdef const Utf8Str* intern(self, unicode py_string) except NULL`
			`cdef const Utf8Str* _intern_utf8(self, char* utf8_string, int length) except NULL`