explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2025-01-15 12:06:25 +03:00
Code Issues Packages Projects Releases Wiki Activity
e7e7c942c7
spaCy/spacy/tokenizer.pxd

46 lines
1.7 KiB
Cython
Raw Normal View History

* Move tokenizer to its own file, and refactor
2014-12-19 23:29:16 +03:00
from libcpp.vector cimport vector
from preshed.maps cimport PreshMap
from cymem.cymem cimport Pool
from .typedefs cimport hash_t
* Hack out morphology stuff from tokenizer, while morphology being reimplemented.
2015-08-26 20:20:11 +03:00
from .structs cimport LexemeC, TokenC
* Move tokenizer to its own file, and refactor
2014-12-19 23:29:16 +03:00
from .strings cimport StringStore
* Break up tokens.pyx into tokens/doc.pyx, tokens/token.pyx, tokens/spans.pyx
2015-07-13 21:20:58 +03:00
from .tokens.doc cimport Doc
* More work on language-generic parsing
2015-08-28 03:02:33 +03:00
from .vocab cimport Vocab, LexemesOrTokens, _Cached
* Move tokenizer to its own file, and refactor
2014-12-19 23:29:16 +03:00
cdef class Tokenizer:
cdef Pool mem
cdef PreshMap _cache
cdef PreshMap _specials
cpdef readonly Vocab vocab
Flush tokenizer cache when necessary (#4258) Flush tokenizer cache when affixes, token_match, or special cases are modified. Fixes #4238, same issue as in #1250.
2019-09-08 21:52:46 +03:00
cdef object _token_match
cdef object _prefix_search
cdef object _suffix_search
cdef object _infix_finditer
Add __reduce__ to Tokenizer so that English pickles. - Add tests to test_pickle and test_tokenizer that save to tempfiles.
2015-10-24 08:18:47 +03:00
cdef object _rules
Generalize handling of tokenizer special cases Handle tokenizer special cases more generally by using the Matcher internally to match special cases after the affix/token_match tokenization is complete. Instead of only matching special cases while processing balanced or nearly balanced prefixes and suffixes, this recognizes special cases in a wider range of contexts: * Allows arbitrary numbers of prefixes/affixes around special cases * Allows special cases separated by infixes Existing tests/settings that couldn't be preserved as before: * The emoticon '")' is no longer a supported special case * The emoticon ':)' in "example:)" is a false positive again When merged with #4258 (or the relevant cache bugfix), the affix and token_match properties should be modified to flush and reload all special cases to use the updated internal tokenization with the Matcher.
2019-09-08 21:29:59 +03:00
cdef object _special_matcher
Reload special cases when necessary Reload special cases when affixes or token_match are modified. Skip reloading during initialization.
2019-09-08 23:40:08 +03:00
cdef int _property_init_count
* Move tokenizer to its own file, and refactor
2014-12-19 23:29:16 +03:00
* Rename Tokens to Doc
2015-07-08 19:53:00 +03:00
cpdef Doc tokens_from_list(self, list strings)
* Move tokenizer to its own file, and refactor
2014-12-19 23:29:16 +03:00
Use special Matcher only for cases with affixes * Reinsert specials cache checks during normal tokenization for special cases as much as possible * Additionally include specials cache checks while splitting on infixes * Since the special Matcher needs consistent affix-only tokenization for the special cases themselves, introduce the argument `with_special_cases` in order to do tokenization with or without specials cache checks * After normal tokenization, postprocess with special cases Matcher for special cases containing affixes
2019-09-16 15:16:30 +03:00
cdef Doc _tokenize_affixes(self, unicode string, bint with_special_cases)
Generalize handling of tokenizer special cases Handle tokenizer special cases more generally by using the Matcher internally to match special cases after the affix/token_match tokenization is complete. Instead of only matching special cases while processing balanced or nearly balanced prefixes and suffixes, this recognizes special cases in a wider range of contexts: * Allows arbitrary numbers of prefixes/affixes around special cases * Allows special cases separated by infixes Existing tests/settings that couldn't be preserved as before: * The emoticon '")' is no longer a supported special case * The emoticon ':)' in "example:)" is a false positive again When merged with #4258 (or the relevant cache bugfix), the affix and token_match properties should be modified to flush and reload all special cases to use the updated internal tokenization with the Matcher.
2019-09-08 21:29:59 +03:00
cdef int _apply_special_cases(self, Doc doc)
* Refactor tokenizer, to set the 'spacy' field on TokenC instead of passing a string
2015-07-13 22:46:02 +03:00
cdef int _try_cache(self, hash_t key, Doc tokens) except -1
Use special Matcher only for cases with affixes * Reinsert specials cache checks during normal tokenization for special cases as much as possible * Additionally include specials cache checks while splitting on infixes * Since the special Matcher needs consistent affix-only tokenization for the special cases themselves, introduce the argument `with_special_cases` in order to do tokenization with or without specials cache checks * After normal tokenization, postprocess with special cases Matcher for special cases containing affixes
2019-09-16 15:16:30 +03:00
cdef int _try_specials(self, hash_t key, Doc tokens,
int* has_special) except -1
cdef int _tokenize(self, Doc tokens, unicode span, hash_t key,
int* has_special, bint with_special_cases) except -1
cdef unicode _split_affixes(self, Pool mem, unicode string,
vector[LexemeC*] *prefixes,
vector[LexemeC*] *suffixes, int* has_special,
bint with_special_cases)
* Replace UniStr, using unicode objects instead
2015-07-22 05:49:39 +03:00
cdef int _attach_tokens(self, Doc tokens, unicode string,
Use special Matcher only for cases with affixes * Reinsert specials cache checks during normal tokenization for special cases as much as possible * Additionally include specials cache checks while splitting on infixes * Since the special Matcher needs consistent affix-only tokenization for the special cases themselves, introduce the argument `with_special_cases` in order to do tokenization with or without specials cache checks * After normal tokenization, postprocess with special cases Matcher for special cases containing affixes
2019-09-16 15:16:30 +03:00
vector[LexemeC*] *prefixes,
vector[LexemeC*] *suffixes, int* has_special,
bint with_special_cases) except -1
Generalize handling of tokenizer special cases Handle tokenizer special cases more generally by using the Matcher internally to match special cases after the affix/token_match tokenization is complete. Instead of only matching special cases while processing balanced or nearly balanced prefixes and suffixes, this recognizes special cases in a wider range of contexts: * Allows arbitrary numbers of prefixes/affixes around special cases * Allows special cases separated by infixes Existing tests/settings that couldn't be preserved as before: * The emoticon '")' is no longer a supported special case * The emoticon ':)' in "example:)" is a false positive again When merged with #4258 (or the relevant cache bugfix), the affix and token_match properties should be modified to flush and reload all special cases to use the updated internal tokenization with the Matcher.
2019-09-08 21:29:59 +03:00
cdef int _save_cached(self, const TokenC* tokens, hash_t key,
Use special Matcher only for cases with affixes * Reinsert specials cache checks during normal tokenization for special cases as much as possible * Additionally include specials cache checks while splitting on infixes * Since the special Matcher needs consistent affix-only tokenization for the special cases themselves, introduce the argument `with_special_cases` in order to do tokenization with or without specials cache checks * After normal tokenization, postprocess with special cases Matcher for special cases containing affixes
2019-09-16 15:16:30 +03:00
int* has_special, int n) except -1
Reference in New Issue Copy Permalink