explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2025-04-12 21:24:14 +03:00
Code Issues Packages Projects Releases Wiki Activity

Fix in BertTokenizer docs (#12955)

Browse Source 
* fix BertWordPieceTokenizer constructor call

* fix

* Update website/docs/usage/linguistic-features.mdx

---------

Co-authored-by: Adriane Boyd <adrianeboyd@gmail.com>
This commit is contained in:
Sofie Van Landeghem 2023-09-13 13:21:58 +02:00 committed by Adriane Boyd
parent fb59288d1c
commit 63f7df8a1c
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
website/docs/usage/linguistic-features.mdx
View File

@ -1299,9 +1299,9 @@ correct type.
```python {title="functions.py",highlight="1"}
@spacy.registry.tokenizers("bert_word_piece_tokenizer")
def create_whitespace_tokenizer(vocab_file: str, lowercase: bool):
def create_bert_tokenizer(vocab_file: str, lowercase: bool):
def create_tokenizer(nlp):
return BertWordPieceTokenizer(nlp.vocab, vocab_file, lowercase)
return BertTokenizer(nlp.vocab, vocab_file, lowercase)
return create_tokenizer
```