Commit Graph

10 Commits

Author SHA1 Message Date
Ines Montani
615dba9d99 Fix tokenizer exceptions 2021-01-27 22:11:42 +11:00
Ines Montani
e3f8be9a94 Update language data 2021-01-27 13:29:22 +11:00
muratjumashev
53abf759ad Fix punctuation 2021-01-24 20:54:22 +06:00
muratjumashev
2a2646362b Fix language subclass 2021-01-23 22:00:50 +06:00
muratjumashev
e30bbf5432 Add examples 2021-01-23 21:49:08 +06:00
muratjumashev
2f385385a9 Remove comment 2021-01-23 21:36:28 +06:00
muratjumashev
d53724ba1d Add lex_attrs 2021-01-23 21:35:25 +06:00
muratjumashev
4418ec2eee Add punctuation 2021-01-23 21:31:31 +06:00
muratjumashev
101d265778 Add stopwords 2021-01-23 21:25:28 +06:00
muratjumashev
28d06ab860 Add tokenizer_exceptions 2021-01-22 23:08:41 +06:00