explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2025-07-27 08:29:51 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix BertWordPieceTokenizer constructor call

Browse Source
This commit is contained in:
svlandeg 2023-09-05 13:24:17 +02:00
parent 6d1f6d9a23
commit 3bc564c28e
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
website/docs/usage/linguistic-features.mdx
View File

@ -1301,7 +1301,7 @@ correct type.
@spacy.registry.tokenizers("bert_word_piece_tokenizer") @spacy.registry.tokenizers("bert_word_piece_tokenizer")
def create_whitespace_tokenizer(vocab_file: str, lowercase: bool): def create_whitespace_tokenizer(vocab_file: str, lowercase: bool):
def create_tokenizer(nlp): def create_tokenizer(nlp):
return BertWordPieceTokenizer(nlp.vocab, vocab_file, lowercase) return BertWordPieceTokenizer(vocab_file, lowercase=lowercase)
return create_tokenizer return create_tokenizer
``` ```