explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-09-23 04:19:11 +03:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into spacy.io

Browse Source
This commit is contained in:
Ines Montani 2019-08-01 14:28:52 +02:00
commit 80f6e7feaa
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
website/docs/api/tokenizer.md
View File

@ -5,7 +5,7 @@ tag: class
source: spacy/tokenizer.pyx source: spacy/tokenizer.pyx
--- ---
Segment text, and create `Doc` objects with the discovered segment boundaries. Segment text, and create `Doc` objects with the discovered segment boundaries. For a deeper understanding, see the docs on [how spaCy's tokenizer works](/usage/linguistic-features#how-tokenizer-works).
## Tokenizer.\_\_init\_\_ {#init tag="method"} ## Tokenizer.\_\_init\_\_ {#init tag="method"}
@ -109,7 +109,7 @@ if no suffix rules match.
Add a special-case tokenization rule. This mechanism is also used to add custom Add a special-case tokenization rule. This mechanism is also used to add custom
tokenizer exceptions to the language data. See the usage guide on tokenizer exceptions to the language data. See the usage guide on
[adding languages](/usage/adding-languages#tokenizer-exceptions) for more [adding languages](/usage/adding-languages#tokenizer-exceptions) and [linguistic features](/usage/linguistic-features#special-cases) for more
details and examples. details and examples.
> #### Example > #### Example