diff --git a/spacy/tokenizer.pyx b/spacy/tokenizer.pyx index f8df13610..4a148b356 100644 --- a/spacy/tokenizer.pyx +++ b/spacy/tokenizer.pyx @@ -45,10 +45,12 @@ cdef class Tokenizer: `re.compile(string).search` to match suffixes. `infix_finditer` (callable): A function matching the signature of `re.compile(string).finditer` to find infixes. - token_match (callable): A boolean function matching strings to be + token_match (callable): A function matching the signature of + `re.compile(string).match`, for matching strings to be recognized as tokens. - url_match (callable): A boolean function matching strings to be - recognized as tokens after considering prefixes and suffixes. + url_match (callable): A function matching the signature of + `re.compile(string).match`, for matching strings to be + recognized as urls. EXAMPLE: >>> tokenizer = Tokenizer(nlp.vocab)