spaCy/spacy/lexeme.pxd

from .typedefs cimport hash_t, utf8_t, flag_t, id_t, len_t, tag_t

from .utf8string cimport StringStore
from libc.stdint cimport uint16_t

cpdef flag_t OOV_DIST_FLAGS

# Flags
cpdef enum:
    IS_ALPHA
    IS_ASCII
    IS_DIGIT
    IS_LOWER
    IS_PUNCT
    IS_SPACE
    IS_TITLE
    IS_UPPER

    LIKE_URL
    LIKE_NUMBER

    OFT_LOWER
    OFT_TITLE
    OFT_UPPER

    IN_MALES
    IN_FEMALES
    IN_SURNAMES
    IN_PLACES
    IN_GAMES
    IN_CELEBS
    IN_NAMES


cdef struct Lexeme:
    flag_t flags
   
    id_t id
    id_t sic
    id_t norm
    id_t shape
    id_t asciied
    id_t prefix
    id_t suffix

    float prob
    
    len_t length
    tag_t cluster
    tag_t postype
    tag_t supersense


cdef Lexeme EMPTY_LEXEME

cpdef Lexeme init(id_t i, unicode string, hash_t hashed,
                  StringStore store, dict props) except *
 

cdef inline bint check_flag(Lexeme* lexeme, size_t flag_id) nogil:
    return lexeme.flags & (1 << flag_id)
* Fiddle with data types on Lexeme, to compress them to a much smaller size. 2014-10-30 07:42:15 +03:00			`from .typedefs cimport hash_t, utf8_t, flag_t, id_t, len_t, tag_t`
* Restoring Lexeme-as-struct 2014-09-10 22:41:37 +04:00
* Large refactor, particularly to Python API 2014-10-23 17:59:17 +04:00			`from .utf8string cimport StringStore`
* Fiddle with data types on Lexeme, to compress them to a much smaller size. 2014-10-30 07:42:15 +03:00			`from libc.stdint cimport uint16_t`
* Revising data model of lexeme. Compiles. 2014-10-09 12:53:30 +04:00
* Large refactor, particularly to Python API 2014-10-23 17:59:17 +04:00			`cpdef flag_t OOV_DIST_FLAGS`
* Revising data model of lexeme. Compiles. 2014-10-09 12:53:30 +04:00
* Large refactor, particularly to Python API 2014-10-23 17:59:17 +04:00			`# Flags`
			`cpdef enum:`
			`IS_ALPHA`
			`IS_ASCII`
			`IS_DIGIT`
			`IS_LOWER`
			`IS_PUNCT`
			`IS_SPACE`
			`IS_TITLE`
			`IS_UPPER`
* Revising data model of lexeme. Compiles. 2014-10-09 12:53:30 +04:00
* Add LIKE_URL and LIKE_NUMBER flag features 2014-11-02 05:19:05 +03:00			`LIKE_URL`
			`LIKE_NUMBER`

* Large refactor, particularly to Python API 2014-10-23 17:59:17 +04:00			`OFT_LOWER`
			`OFT_TITLE`
			`OFT_UPPER`
* Revising data model of lexeme. Compiles. 2014-10-09 12:53:30 +04:00
* Remove vocab10k field, and add flags for gazetteers 2014-11-02 16:13:51 +03:00			`IN_MALES`
			`IN_FEMALES`
			`IN_SURNAMES`
			`IN_PLACES`
			`IN_GAMES`
			`IN_CELEBS`
			`IN_NAMES`

* Revising data model of lexeme. Compiles. 2014-10-09 12:53:30 +04:00
* Large refactor, particularly to Python API 2014-10-23 17:59:17 +04:00			`cdef struct Lexeme:`
* Fiddle with data types on Lexeme, to compress them to a much smaller size. 2014-10-30 07:42:15 +03:00			`flag_t flags`
* Rewriting Lexeme serialization. 2014-10-29 15:19:38 +03:00
* Restore id attribute to lexeme, and rename pos field to postype, to store clustered tag dictionaries 2014-10-31 09:43:00 +03:00			`id_t id`
* Fiddle with data types on Lexeme, to compress them to a much smaller size. 2014-10-30 07:42:15 +03:00			`id_t sic`
			`id_t norm`
			`id_t shape`
			`id_t asciied`
			`id_t prefix`
			`id_t suffix`
* Slight cleaning of tokenizer code 2014-10-10 12:17:22 +04:00
* Large refactor, particularly to Python API 2014-10-23 17:59:17 +04:00			`float prob`
* Fiddle with data types on Lexeme, to compress them to a much smaller size. 2014-10-30 07:42:15 +03:00
			`len_t length`
			`tag_t cluster`
* Restore id attribute to lexeme, and rename pos field to postype, to store clustered tag dictionaries 2014-10-31 09:43:00 +03:00			`tag_t postype`
* Fiddle with data types on Lexeme, to compress them to a much smaller size. 2014-10-30 07:42:15 +03:00			`tag_t supersense`

* Remove the feature array stuff from Tokens class, and replace vector with array-based implementation, with padding. 2014-10-22 18:57:59 +04:00
* Large refactor, particularly to Python API 2014-10-23 17:59:17 +04:00			`cdef Lexeme EMPTY_LEXEME`
* Restoring Lexeme-as-struct 2014-09-10 22:41:37 +04:00
* Restore id attribute to lexeme, and rename pos field to postype, to store clustered tag dictionaries 2014-10-31 09:43:00 +03:00			`cpdef Lexeme init(id_t i, unicode string, hash_t hashed,`
* Rewriting Lexeme serialization. 2014-10-29 15:19:38 +03:00			`StringStore store, dict props) except *`

* Add serialize/deserialize functions for lexeme, transport to/from python dict. 2014-10-09 07:10:46 +04:00
* Large refactor, particularly to Python API 2014-10-23 17:59:17 +04:00			`cdef inline bint check_flag(Lexeme* lexeme, size_t flag_id) nogil:`
			`return lexeme.flags & (1 << flag_id)`