Ines Montani
cb57b35bb8
Also include lowercase norm exceptions
2018-10-13 15:37:30 +02:00
JKhakpour
74a30d883c
Add Persian(Farsi) language support ( #2797 )
2018-10-13 15:31:49 +02:00
Marina Lysyuk
b76fe08308
Correcting lang/ru/examples.py ( #2845 )
...
* Correct some grammatical inaccuracies in lang\ru\examples.py; filled Contributor Agreement
* Correct some grammatical inaccuracies in lang\ru\examples.py
* Move contributor agreement to separate file
2018-10-13 15:19:43 +02:00
Ines Montani
ea20b72c08
💫 Make like_num work for prefixed numbers ( #2808 )
...
* Only split + prefix if not numbers
* Make like_num work for prefixed numbers
* Add test for like_num
2018-10-01 10:49:14 +02:00
Filipe Caixeta
6c498f9ff4
Update Portuguese Language ( #2790 )
...
* Add words to portuguese language _num_words
* Add words to portuguese language _num_words
* Portuguese - Add/remove stopwords, fix tokenizer, add currency symbols
* Extended punctuation and norm_exceptions in the Portuguese language
2018-09-29 09:51:45 +02:00
Ines Montani
1f1bab9264
Remove unused import
2018-09-27 16:41:37 +02:00
Ines Montani
71cdbeada7
Revert "Also include lowercase norm exceptions"
...
This reverts commit 70f4e8adf3
.
2018-09-27 12:29:25 +02:00
Ines Montani
5e0dfb34fa
Merge branch 'master' of https://github.com/explosion/spaCy
2018-09-26 11:13:58 +02:00
Ines Montani
70f4e8adf3
Also include lowercase norm exceptions
2018-09-25 12:22:02 +02:00
Keshan
9a016d17c2
Adding basic support for Sinhala language. ( #2788 )
...
* adding Sinhala language package, stop words, examples and lex_attrs.
* Adding contributor agreement
* Updating contributor agreement
2018-09-25 12:18:25 +02:00
mauryaland
68b3c544d5
Adding French hyphenated first name ( #2786 )
2018-09-21 10:38:13 +02:00
Andrew Ongko
81564cc4e8
Update Indonesian model ( #2752 )
...
* adding e-KTP in tokenizer exceptions list
* add exception token
* removing lines with containing space as it won't matter since we use .split() method in the end, added new tokens in exception
* add tokenizer exceptions list
* combining base_norms with norm_exceptions
* adding norm_exception
* fix double key in lemmatizer
* remove unused import on punctuation.py
* reformat stop_words to reduce number of lines, improve readibility
* updating tokenizer exception
* implement is_currency for lang/id
* adding orth_first_upper in tokenizer_exceptions
* update the norm_exception list
* remove bunch of abbreviations
* adding contributors file
2018-09-14 12:30:32 +02:00
Filipe Caixeta
fe515085f3
Add words to portuguese language _num_words ( #2759 )
...
* Add words to portuguese language _num_words
* Add words to portuguese language _num_words
2018-09-14 12:30:16 +02:00
tyburam
476472d181
Lex _attrs for polish language ( #2750 )
...
* Signed spaCy contributor agreement
* Added polish version of english lex_attrs
2018-09-10 11:53:57 +02:00
Sainath Adapa
77139bc03c
Basic support for Telugu language ( #2751 )
2018-09-10 11:53:18 +02:00
Aniruddha Adhikary
4530ddcc51
update bengali token rules for hyphen and digits ( #2731 )
2018-09-05 21:49:00 +02:00
Matthew Honnibal
598dbf1ce0
Fix character-based tokenization for Japanese
2018-08-27 01:51:38 +02:00
Matthew Honnibal
4336397ecb
Update develop from master
2018-08-14 03:04:28 +02:00
Ioannis Daras
fe94e696d3
Optimize Greek language support ( #2658 )
2018-08-14 02:31:32 +02:00
Aashish Gangwani
6eebfc7bf4
Added numbers to ../lang/hi/lex_attrs.py ( #2629 )
...
I have added numbers in hindi lex_attrs.py file according to Indian numbering system(https://en.wikipedia.org/wiki/Indian_numbering_system ) and here are there english translations:
'शून्य' => zero
'एक' => one
'दो' => two
'तीन' => three
'चार' => four
'पांच' => five
'छह' => six
'सात'=>seven
'आठ' => eight
'नौ' => nine
'दस' => ten
'ग्यारह' => eleven
'बारह' => twelve
'तेरह' => thirteen
'चौदह' => fourteen
'पंद्रह' => fifteen
'सोलह'=> sixteen
'सत्रह' => seventeen
'अठारह' => eighteen
'उन्नीस' => nineteen
'बीस' => twenty
'तीस' => thirty
'चालीस' => forty
'पचास' => fifty
'साठ' => sixty
'सत्तर' => seventy
'अस्सी' => eighty
'नब्बे' => ninety
'सौ' => hundred
'हज़ार' => thousand
'लाख' => hundred thousand
'करोड़' => ten million
'अरब' => billion
'खरब' => hundred billion
<!--- Provide a general summary of your changes in the title. -->
## Description
<!--- Use this section to describe your changes. If your changes required
testing, include information about the testing environment and the tests you
ran. If your test fixes a bug reported in an issue, don't forget to include the
issue number. If your PR is still a work in progress, that's totally fine – just
include a note to let us know. -->
### Types of change
<!-- What type of change does your PR cover? Is it a bug fix, an enhancement
or new feature, or a change to the documentation? -->
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I have submitted the spaCy Contributor Agreement.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-08-08 16:06:11 +02:00
Emil Stenström
3834f4146d
Add abbreviations from UD_Swedish-Talbanken ( #2613 )
...
* Add abbreviations from UD_Swedish-Talbanken
* Add contributor agreement.
2018-08-07 13:53:17 +02:00
Xiaoquan Kong
87fa847e6e
Fix Chinese language related bugs ( #2634 )
2018-08-07 11:26:31 +02:00
Emil Stenström
1914c488d3
Swedish: Exceptions for single letter words ending sentence ( #2615 )
...
* Exceptions for single letter words ending sentence
Sentences ending in "i." (as in "... peka i."), "m." (as in "...än 2000 m."), should be tokenized as two separate tokens.
* Add test
2018-08-05 14:14:30 +02:00
Dmitry Bruhanov
07d0cc9de7
Update examples.py ( #2597 )
2018-07-25 22:20:24 +02:00
Matthew Honnibal
66983d8412
Port BenDerPan's Chinese changes to v2 (finally) ( #2591 )
...
* add template files for Chinese
* add template files for Chinese, and test directory .
2018-07-25 02:47:23 +02:00
ines
f2e3e039b7
Update French stop words ( resolves #2540 )
2018-07-24 23:41:51 +02:00
Ines Montani
a43ad114c2
Fix typo [ci skip]
2018-07-24 18:45:40 +02:00
Dmitry Bruhanov
27160b1516
added some widespread written jargon & dialectizms ( #2584 )
...
This jargon is not offencive but emotionally colored as funny due to its deviation from the norm for various reasons: immitating a dialect, deliberately wrong spelling emphasizing its low colloquial nature, obsolete form, foreign borrowing with native flections, etc.
Dmitry Briukhanov, Linguist & Pythonist
2018-07-24 18:44:29 +02:00
ines
3c30d1763c
Merge branch 'master' into develop
2018-07-21 15:34:18 +02:00
katarkor
5ca853bee0
changed tag_map, morph_rules, lemmatizer for Norwegian ( #2565 )
...
* changed tag_map, morph_rules, lemmatizer for Norwegian
* Move unicode declaration up
Hopefully fixes test failure on Python 2
* Update CONTRIBUTOR_AGREEMENT.md
* Move unicode declarations
Hopefully fixes test this time
* Revert "Merge remote-tracking branch 'origin/patch-1'"
This reverts commit f5ccd5dd0d
, reversing
changes made to dd07e180ea
.
* Update contributor agreement [ci skip]
2018-07-19 19:38:24 +02:00
ines
02aefe7cc0
Merge branch 'master' into develop
2018-07-18 18:52:59 +02:00
Ioannis Daras
6ed18412d0
Greek language optimizations ( #2558 )
...
* Greek language optimizations
* Add encoding on files containing greek words
* Add encoding on files containing greek words
2018-07-18 18:51:38 +02:00
ines
80e7485630
Merge branch 'master' into develop
2018-07-18 17:28:47 +02:00
Paul O'Leary McCann
61ef0739b8
Add Japanese stop words. ( #2549 )
...
List created by taking the 2000 top words from a Wikipedia dump and
removing everything that wasn't hiragana.
Tried going through kanji words and deciding what to keep but there were
too many obvious non-stopwords (東京 was in the top 500) and many other
words where it wasn't clear if they should be included or not.
2018-07-17 10:12:48 +02:00
Tero K
f35980f865
Enhancement/lang fi examples ( #2547 )
...
* Added a file with examples in finnish
* added contributor agreement
2018-07-15 09:50:27 +02:00
Paul O'Leary McCann
1987f3f784
Add Japanese lemmas ( #2543 )
...
This info was already available from Mecab, forgot to add it before.
2018-07-13 10:55:14 +02:00
ines
3a321e79ac
Merge branch 'master' into develop
2018-07-10 13:49:08 +02:00
Eleni170
6042723535
Add support for Greek language ( #2535 )
...
* Add contributor agreement
* Support for Greek language
* Fix missing el_tokenizer
2018-07-10 13:48:38 +02:00
Stefan Schweter
3dfc7f86be
lemmatizer: correct lemma for Rang ( #2537 )
...
<!--- Provide a general summary of your changes in the title. -->
## Description
This PR corrects the German lemma form for the word "Rang". Initially, the lemma form was "ringen", which is not correct, because it refers to the verb ("ringen") and not to the noun ("Rang").
### Types of change
The lemma form for "Rang" is corrected to "Rang", see also the [Duden](https://www.duden.de/rechtschreibung/Rang ) entry.
## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I have submitted the spaCy Contributor Agreement.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
2018-07-10 13:11:19 +02:00
ines
fd6207426a
Merge branch 'master' into develop
2018-07-09 18:05:10 +02:00
Duygu Altinok
00b9a58558
German lemmatizer additions ( #2529 )
...
* lemma of was-> was
* added new pairs issue @2486
* added article tests
2018-07-09 11:10:15 +02:00
ines
63666af328
Merge branch 'master' into develop
2018-07-04 14:52:25 +02:00
Muhammad Irfan
f33c703066
Add Urdu Language Support ( #2430 )
...
* added Urdu language support.
* added Urdu language tests.
* modified conftest.py for Urdu language support.
* added spacy contributor agreement.
2018-06-22 11:14:03 +02:00
himkt
14d9007efd
fix wrong indexing ( #2416 )
...
* fix wrong indexing
* add agreement
2018-06-19 10:20:57 +02:00
Aliia E
428bae66b5
Add Tatar Language Support ( #2444 )
...
* add Tatar lang support
* add Tatar letters
* add Tatar tests
* sign contributor agreement
* sign contributor agreement [x]
* remove comments from Language class
* remove all template comments
2018-06-19 10:17:53 +02:00
ines
778e5f4da3
Merge branch 'master' into develop
2018-06-11 00:38:04 +02:00
Nour Shalabi
a169b79092
Additions to Arabic stop words. ( #2422 )
...
* Additions to Arabic stop words.
* Create nourshalabi.md
2018-06-08 02:33:23 +02:00
ines
330c039106
Merge branch 'master' into develop
2018-05-26 18:30:52 +02:00
Aristo Rinjuang
432ede04af
adding more words and rephrasing ( #2351 )
...
* adding more words and rephrasing
* adding a contributor
* tokenizer bugs solved
2018-05-24 11:40:57 +02:00
Jani Monoses
ec62cadf4c
Updates to Romanian support ( #2354 )
...
* Add back Romanian in conftest
* Romanian lex_attr
* More tokenizer exceptions for Romanian
* Add tests for some Romanian tokenizer exceptions
2018-05-24 11:40:00 +02:00
Matthew Honnibal
546dd99cdf
Merge master into develop -- mostly Arabic and website
2018-05-15 18:14:28 +02:00
Tahar Zanouda
00417794d3
Add Arabic language ( #2314 )
...
* added support for Arabic lang
* added Arabic language support
* updated conftest
2018-05-15 00:27:19 +02:00
Jani Monoses
0e08e49e87
Lemmatizer ro ( #2319 )
...
* Add Romanian lemmatizer lookup table.
Adapted from http://www.lexiconista.com/datasets/lemmatization/
by replacing cedillas with commas (ș and ț).
The original dataset is licensed under the Open Database License.
* Fix one blatant issue in the Romanian lemmatizer
* Romanian examples file
* Add ro_tokenizer in conftest
* Add Romanian lemmatizer test
2018-05-12 15:20:04 +02:00
Jani Monoses
42b34832e4
Update Romanian stopword list ( #2316 )
...
* Contributor agreement for janimo
* Update Romanian stopword list
Include the correct spellings of all the words already in the repo
that are using cedillas (ş and ţ) instead of commas (ș and ț).
Add another unrelated spelling fix.
See https://github.com/stopwords-iso/stopwords-ro/pull/1 and
https://github.com/stopwords-iso/stopwords-ro/pull/2
2018-05-10 12:16:56 +02:00
Lucas Abbade
be7fdc59d1
Update lex_attrs.py ( #2307 )
...
* Update lex_attrs.py
Fixed spelling mistakes of some numbers (according to Brazilian Portuguese).
* Update lex_attrs.py
As requested, I've included the correct spelling for both Brazilian Portuguese and Portuguese Portuguese.
I will advise however, that the two are separated in the future. Brazilian Portuguese is a very different language from the original one, although most of the writing is unified, the way people talk in both countries is radically different. Keeping both languages as one may lead to bigger issues in the future, especially when it comes to spell checking.
2018-05-09 20:49:31 +02:00
mauryaland
5368ba028a
Update stop_words.py for French language ( #2310 )
...
* Add contraction forms of some common stopwords
All the stopwords added contain the apostrophe" ' "or " ’ ".
* Adds contributor agreement mauryaland
* Update mauryaland.md
2018-05-09 12:04:38 +02:00
Paul O'Leary McCann
bd72fbf09c
Port Japanese mecab tokenizer from v1 ( #2036 )
...
* Port Japanese mecab tokenizer from v1
This brings the Mecab-based Japanese tokenization introduced in #1246 to
spaCy v2. There isn't a JapaneseTagger implementation yet, but POS tag
information from Mecab is stored in a token extension. A tag map is also
included.
As a reminder, Mecab is required because Universal Dependencies are
based on Unidic tags, and Janome doesn't support Unidic.
Things to check:
1. Is this the right way to use a token extension?
2. What's the right way to implement a JapaneseTagger? The approach in
#1246 relied on `tag_from_strings` which is just gone now. I guess the
best thing is to just try training spaCy's default Tagger?
-POLM
* Add tagging/make_doc and tests
2018-05-03 18:38:26 +02:00
Matthew Honnibal
2c4a6d66fa
Merge master into develop. Big merge, many conflicts -- need to review
2018-04-29 14:49:26 +02:00
Robin Linderborg
1f9904ef12
fixes #2238 ( #2241 )
...
* Remove erroneous lemma lookup år > åra in Swedish
* Add contributors agreement
* Add contrib agreement to correct directory
* Revert change to CONTRIBUTOR_AGREEMENT
2018-04-28 14:55:22 +02:00
Robin Linderborg
d01f503b54
Remove incorrect lemma lookup gäng->gänga ( #2252 )
...
* Remove incorrect lemma lookup gäng->gänga
In modern Swedish, "gäng" is mostly associated with "gang" or "group of people". The removed lemma lookup lemmatized it to the verb "thread".
* Add contrib agreement to correct directory
* Revert change to CONTRIBUTOR_AGREEMENT
2018-04-28 14:54:41 +02:00
ines
686225eadd
Fix Spanish noun_chunks ( resolves #2210 )
...
Make sure 'NP' label is added to StringStore and move noun_bounds helper into a closure to allow reusing label sets
2018-04-18 18:44:01 -04:00
Jens Dahl Møllerhøj
e5055e3cf6
Add Danish lemmatizer ( #2184 )
...
* add danish lemmatizer
* fill contributor agreement
2018-04-07 19:07:28 +02:00
Matthew Honnibal
21047bde52
Fix syntax error in italian lemmatizer
2018-04-03 23:13:22 +02:00
Viet Trung Tran
ea2af94cd9
Add support for Vietnamese in spaCy by leveraging Pyvi, an external Vietnamese tokenizer ( #2155 )
...
* support for Vietnamese
* Contributor Agreement for adding Vietnamese support on spaCy
2018-03-29 12:19:51 +02:00
ines
11c4735ccf
Fix issue in Italian lemmatizer data ( resolves #2050 )
2018-03-27 23:55:22 +02:00
Matthew Honnibal
1f7229f40f
Revert "Merge branch 'develop' of https://github.com/explosion/spaCy into develop"
...
This reverts commit c9ba3d3c2d
, reversing
changes made to 92c26a35d4
.
2018-03-27 19:23:02 +02:00
Ines Montani
68226109f4
Merge pull request #2142 from jimregan/polish-more-tokens
...
more exceptions
2018-03-24 19:06:44 +01:00
Matthew Honnibal
0d3bf0d4eb
Merge branch 'master' of https://github.com/explosion/spaCy
2018-03-24 17:31:49 +01:00
dejanmarich
ccd1c04c63
Update stop_words.py
...
Added more words
2018-03-24 17:31:24 +01:00
ines
f1446b0257
Port over Turkish changes
2018-03-24 17:31:07 +01:00
DuyguA
cd604878a4
quick typo fix
2018-03-24 17:26:35 +01:00
Jim O'Regan
efe037e8be
more exceptions
2018-03-24 00:05:27 +00:00
alldefector
f4e5904fc2
Fix Spanish noun_chunks failure caused by typo
2018-03-14 17:03:17 +01:00
Ines Montani
14e7e0f12a
Merge pull request #2000 from jimregan/polish-tag-map
...
Polish tag map
2018-02-18 19:05:58 +01:00
Matthew Honnibal
eb3040ce46
Merge pull request #1891 from fucking-signup/master
...
Fix issue #1889
2018-02-18 13:47:47 +01:00
4altinok
94fb0b75e3
code for is_currency
2018-02-11 18:51:32 +01:00
Ines Montani
0954e15dda
Merge pull request #1913 from ohenrik/nb_syntax_iterator
...
Norwegian Language (nb) - Added french syntax iterator with explanation
2018-02-06 04:59:07 +01:00
Ole Henrik Skogstrøm
251a7805fe
Copied French syntax iterator to simplify future changes
2018-02-05 14:45:05 +01:00
ines
f1d3deffac
Add Russian example sentences (see #1107 )
2018-02-01 20:09:40 +01:00
Ole Henrik Skogstrøm
e40465487c
Added french syntax iterator with explenation
2018-01-30 15:44:29 +01:00
Matthew Honnibal
cb7110c22e
Merge pull request #1882 from ohenrik/nb_lemma_and_tag_map
...
Add norwegian bokmål ('nb') lemmatizer and tag_map
2018-01-29 18:18:50 +01:00
Ali Zarezade
bb6bd3d8ae
add persian language
2018-01-27 13:27:26 +03:30
Ali Zarezade
d195675db5
add persian language
2018-01-27 13:21:38 +03:30
Kit
4b42267ba3
Fix issue #1889
2018-01-25 23:17:22 +01:00
Ole Henrik Skogstrøm
8e2c9f2475
Cleaned up nb tag_map comments
2018-01-25 11:09:28 +01:00
Ole Henrik Skogstrøm
1107e89fcf
Updated doc string on nb tag_map module
2018-01-25 11:08:28 +01:00
Ole Henrik Skogstrøm
4058a7d579
Fix æøå characters in lemmatizer
2018-01-24 14:03:14 +01:00
Ole Henrik Skogstrøm
42248f423f
Updated tag map
2018-01-24 13:50:33 +01:00
Ole Henrik Skogstrøm
74b430b49a
Correct Lemmatizer
2018-01-24 13:26:33 +01:00
Ole Henrik Skogstrøm
b9b3a40c78
Add norwegian lemmatizer and tag_map
2018-01-24 12:28:29 +01:00
Ali Zarezade
42349471bc
add ٪ as punctuation
2018-01-23 18:11:33 +03:30
Ali Zarezade
2bda582135
Add Persian character and symbols
...
Add Persian characters and the following:
- ٪ used instead of %
- ؟ used instead of ?
- ﷼ used instead of $
- ، used instead of ,
- ؛ used instead of ;
2018-01-23 13:20:36 +03:30
Kit
701e7cc6aa
Rename variable to keep code consistent
2018-01-08 03:38:44 +01:00
Kit
ed0db95183
Find lowercased forms of ordinal words, where possible
2018-01-08 03:28:50 +01:00
Kit
9bc524982e
Find lowercased forms of numeric words
2018-01-08 03:25:08 +01:00
Kevin Humphreys
7918fa4ef9
handle would've
2018-01-03 12:25:48 -08:00
zqhZY
f27859fa99
add ChineseDefaults class for pickling
2017-12-28 17:13:58 +08:00
Søren Lind Kristiansen
bef735aef7
Fix Danish abbreviation 'm.h.t.'
2017-12-21 09:24:31 +01:00
Ines Montani
a3dd167d7f
Merge branch 'master' into da_ud_tokenization
2017-12-20 21:05:34 +00:00
Ines Montani
97f100f69f
Merge pull request #1742 from kimfalk/master
...
Two corrections in the da lan.
2017-12-20 21:02:00 +00:00
Ines Montani
d682a8803e
Merge pull request #1672 from cbilgili/master
...
Adds Turkish Lemmatization
2017-12-20 21:01:00 +00:00
Benjamin Peterson
9452134cd1
remove no-break spaces from Hindi example ( fixes #1750 )
2017-12-20 11:35:30 -08:00
Søren Lind Kristiansen
7a2f2f6f94
Fix formatting.
2017-12-20 18:37:37 +01:00
Søren Lind Kristiansen
15d13efafd
Tune Danish tokenizer to more closely match tokenization in Universal Dependencies.
2017-12-20 17:36:52 +01:00
Kim FalkJørgensen
648dc60755
Remove the incorrect exception 'm.h.t'
2017-12-20 10:02:39 +01:00
Kim FalkJørgensen
9c9f4ef84a
Fixing a translation error in examples.py
...
Adding an exception in the tokenizer_exceptions.py
2017-12-19 15:26:50 +01:00
ines
22dc744b48
Fix check for '@' in like_url (see #1715 )
2017-12-16 13:48:43 +01:00
Ines Montani
6455b574fc
Check for email address first
2017-12-12 10:25:13 +01:00
Bri-Will
d77361d76c
Update lex_attrs.py. Fix like_url from matching on e-mail
2017-12-11 14:13:28 -08:00
Matthew Honnibal
2ab0f2d186
Merge pull request #1664 from jimregan/italian-lemmatizer
...
BOM in Italian lemmatiser
2017-12-06 11:09:04 +01:00
Matthew Honnibal
3f247119d3
Merge pull request #1668 from sorenlind/da_morph
...
Add more Danish morph rules and clean up existing ones
2017-12-06 11:08:09 +01:00
ines
f2ea6d4713
Add Dutch example sentences (see #1107 )
2017-12-01 23:36:05 +01:00
Canbey Bilgili
abe098b255
Adds Turkish Lemmatization
2017-12-01 17:04:32 +03:00
Søren Lind Kristiansen
d86b537a38
Enable morph rules for Danish
2017-11-30 15:58:02 +01:00
Søren Lind Kristiansen
13a988adc3
Remove 'Number[psor]'
2017-11-30 15:55:04 +01:00
Søren Lind Kristiansen
dd6fde18a9
Add more Danish morph rules and clean up existing ones
2017-11-30 11:17:19 +01:00
Vadim Mazaev
4ba7ddf651
Bugfixies
2017-11-30 12:29:38 +03:00
Jim O'Regan
c3e6cee17a
use inan in polimorf tagset conversion
2017-11-29 23:15:47 +00:00
Jim O'Regan
b32575e78c
imports
2017-11-29 23:03:41 +00:00
Jim O'Regan
3696ce6a7b
add UD mapping
2017-11-29 22:59:19 +00:00
Matthew Honnibal
f9ed9ea529
Merge pull request #1624 from GreenRiverRUS/russian
...
Add support for Russian
2017-11-29 23:10:01 +01:00
Jim O'Regan
076a6fc60a
symbols
2017-11-29 20:11:20 +00:00
Jim O'Regan
834ba3c69a
(semi generated) Polimorf mapping
2017-11-29 20:08:24 +00:00
Jim O'Regan
ba6a23fd11
BOM in Italian lemmatiser
2017-11-29 17:40:07 +00:00
Ines Montani
9052643e2c
Merge pull request #1653 from sorenlind/da_example_typo
...
Fix typo
2017-11-27 14:47:42 +00:00
Søren Lind Kristiansen
5fe58b885b
Fix typo
2017-11-27 15:36:18 +01:00
Ines Montani
d52b1ab245
Add unicode_literals (hopefully fixes test failure on Python 2)
2017-11-27 15:16:54 +01:00
Søren Lind Kristiansen
0ffd27b0f6
Add several Danish alternative spellings
2017-11-27 13:35:41 +01:00
Vadim Mazaev
cacd859dcd
Added tag map, fixed tests fails, added more exceptions
2017-11-26 20:54:48 +03:00
Søren Lind Kristiansen
ef03e9ea53
Remove unused import.
2017-11-25 13:04:02 +01:00
Søren Lind Kristiansen
6aa241bcec
Add day of month tokenizer exceptions for Danish.
2017-11-24 15:03:24 +01:00
Søren Lind Kristiansen
0c276ed020
Add weekday abbreviations and remove abiguous month abbreviations for Danish.
2017-11-24 14:43:29 +01:00
Søren Lind Kristiansen
056547e989
Add multiple tokenizer exceptions for Danish.
2017-11-24 11:51:26 +01:00
Søren Lind Kristiansen
ac8116510d
Fix tokenization of 'i.' for Danish.
2017-11-24 11:16:53 +01:00
Vadim Mazaev
81314f8659
Fixed tokenizer: added char classes; added first lemmatizer and
...
tokenizer tests
2017-11-21 22:23:59 +03:00
Vadim Mazaev
52ee1f9bf9
Updated Russian Language, added lemmatizer, norm exceptions and lex
...
attrs
2017-11-21 11:44:46 +03:00
Vadim Mazaev
a0739a06d4
Returned russian support from v1.10 branch
2017-11-17 17:06:15 +03:00
ines
c9d72de0fb
Add dummy serialization methods for Japanese and missing lang getter ( resolves #1557 )
2017-11-15 12:44:02 +01:00
Mathias Deschamps
c0691b2ab4
Add tokenizer exceptions for ing verbs
...
Extend list of tokenizing exceptions introduced in 123810b
2017-11-13 17:46:05 +01:00
Mathias Deschamps
288298ead9
Add norm exception for ing verbs
...
Some ing verbs are sometimes written in or in'. Make the NORM form correct
2017-11-13 17:46:05 +01:00
Abhinav Sharma
59f5740ede
improved upon the list of included stop_words
2017-11-13 17:13:49 +05:30
ines
123810b6de
Add "lovin'" to tokenizer exceptions (see #1248 )
2017-11-09 17:09:30 +01:00
Ines Montani
42b241ccd0
Update language code in usage example in comment
2017-11-08 11:36:38 +01:00
Abhinav Sharma
84edade82d
Create examples.py
...
Populated the file with the translations of English example sentences
2017-11-08 13:23:08 +05:30
ines
bcf42b8846
Fix typo
2017-11-08 01:06:37 +01:00
ines
acb9bdb852
Fix PRON_LEMMA imports
2017-11-06 17:41:53 +01:00
ines
baa231745c
Fix Dutch tag map
2017-11-05 21:41:50 +01:00
ines
507ecb67af
Fix Spanish tag map
2017-11-05 19:23:34 +01:00
ines
975e1042ff
Fix Italian tag map
2017-11-05 18:34:09 +01:00
ines
6b2d6e4937
Fix Portuguese tag map
2017-11-05 18:31:00 +01:00
ines
fa2687fded
Fix Dutch tag map
2017-11-05 17:57:59 +01:00
ines
fb8990d916
Fix Spanish tag map
2017-11-05 17:48:46 +01:00
ines
9d13288f73
Fix French tag map
2017-11-05 17:47:59 +01:00
ines
54579805c5
Fix French tag map
2017-11-05 17:44:05 +01:00
Matthew Honnibal
0d4bd6414e
Fix Italian tag map
2017-11-05 14:11:03 +01:00
ines
ef597622a6
Add Portuguese tag map
2017-11-05 13:58:34 +01:00
ines
793c62dfda
Add Dutch tag map
2017-11-05 13:48:07 +01:00
ines
f7485a09c8
Fix Italian tag map
2017-11-05 13:12:58 +01:00
ines
3cef901834
Add tag map for French and Italian
2017-11-04 23:32:51 +01:00
ines
6c15aafebd
Fix formatting
2017-11-04 23:07:02 +01:00
ines
9baab241b4
Add skeleton language data for Turkish
2017-11-02 16:32:24 +01:00
ines
c6fea3e5f6
Add Romanian and Croatian skeletons (experimental)
...
Add language data templates to make it easier for others to contribute to the language support
2017-11-01 23:04:28 +01:00
ines
18c859500b
Add missing imports
2017-11-01 23:02:51 +01:00
ines
819e30a26e
Tidy up tokenizer exceptions
2017-11-01 23:02:45 +01:00
ines
9659391944
Update deprecated methods and add warnings
2017-11-01 16:49:42 +01:00
Ines Montani
d11659463b
Merge pull request #1152 from jimregan/develop-irish
...
[WIP] attempt a port from #1147
2017-11-01 00:23:43 +01:00
ines
7e424a1804
Don't copy exception dicts if not necessary and tidy up
2017-10-31 21:05:29 +01:00
Ines Montani
06c25a8882
Remove comma that caused list to wrap in tuple!
...
Also removed extra dict wrappings for performance (we used to have them in there, but they should only really exist if copying the dict is absolutely necessary)
2017-10-31 20:13:16 +01:00
Ines Montani
147448b65b
Add missing symbols
2017-10-31 19:34:45 +01:00
Ines Montani
9b0de9fb43
Fix import of symbols (now nested one level lower)
2017-10-31 19:17:58 +01:00
Jim O'Regan
41dd29e48e
merge
2017-10-31 14:07:45 +00:00
Ines Montani
090bd00369
Merge pull request #1464 from mayukh18/develop_bengali_pronouns
...
added the bengali pronouns for v2.0
2017-10-25 21:55:25 +02:00
mayukh18
1bc07758fa
added few bengali pronouns
2017-10-25 22:24:40 +05:30
Ines Montani
d3bf488e16
Merge pull request #1171 from mollerhoj/support-danish
...
Improve basic support for Danish
2017-10-24 20:29:57 +02:00
Matthew Honnibal
66766c1454
Restore SP tag to English tag_map, until models migrate
2017-10-24 17:05:00 +02:00
ines
c55db0a4a1
Add example sentences for Japanese and Chinese (see #1107 )
2017-10-24 13:02:24 +02:00
ines
66f8f9d4a0
Fix Japanese tokenizer
...
JapaneseTokenizer now returns a Doc, not individual words
2017-10-24 13:02:19 +02:00
Ines Montani
facf77e541
Merge branch 'develop' into support-danish
2017-10-24 11:53:19 +02:00
Matthew Honnibal
49895fbef6
Rename 'SP' special tag to '_SP'
...
Renaming the tag with an underscore lets us add it to the tag map
without worrying that we'll change the sequence of tags, which throws
off the tag-to-ID mapping. For instance, if we inserted a 'SP' tag,
the "VERB" tag is pushed to a different class ID, and the model is all
messed up.
2017-10-20 14:01:12 +02:00
Ines Montani
f0d577e460
Merge pull request #1425 from explosion/feature/hindi-tokenizer
...
💫 Basic Hindi tokenization support
2017-10-18 13:34:52 +02:00
Matthew Honnibal
839de87ca9
Make lambda func a named function, for pickling
2017-10-17 18:21:20 +02:00
Matthew Honnibal
9ce7d6af87
Make lex attr functions top-level functions, to promote pickling
2017-10-17 18:19:18 +02:00
Ines Montani
aab299c8ae
Merge pull request #1429 from vishnunekkanti/develop
...
fix syntax error in zh
2017-10-17 14:45:02 +02:00
ines
485c4f6df5
Add Hungarian examples (see #1107 )
2017-10-17 02:37:45 +02:00
Vishnu Kumar Nekkanti
d3c54cf39a
fixed SyntaxError while checking for jieba
2017-10-16 18:51:33 +05:30
ines
266e7180a7
Add Language class, stop words and basic stemmer that sets NORM
2017-10-14 14:59:52 +02:00
ines
e85e1d571b
Update base punctuation
2017-10-14 14:59:23 +02:00
ines
9d6c8eaa49
Update base norm exceptions with more unicode characters
...
e.g. unicode variations of punctuation used in Chinese
2017-10-14 14:58:52 +02:00
ines
38c756fd85
Port over changes from #1287
2017-10-14 13:16:21 +02:00
ines
612224c10d
Port over changes from #1157
2017-10-14 13:11:39 +02:00
ines
a4d974d97b
Port over URL pattern changes from #1411
2017-10-14 12:58:07 +02:00
ines
09aed58140
Port over changes from #1333 and add comments
2017-10-14 12:52:59 +02:00
ines
8ce6f96180
Don't make copies of language data components
2017-10-11 15:34:55 +02:00
ines
417d45f5d0
Add lemmatizer data as variable on language data
...
Don't create lookup lemmatizer within Language class and just pass in
the data so it can be set on Token creation
2017-10-11 02:24:58 +02:00
ines
0c2343d73a
Tidy up language data
2017-10-11 02:22:49 +02:00
Matthew Honnibal
8143618497
Set prefix length back to 1
2017-10-10 19:32:54 +02:00
Matthew Honnibal
dce8afb9cf
Set prefix length to 3
2017-10-09 21:55:55 -05:00
Ines Montani
959c46eabe
Merge pull request #1365 from wannaphongcom/develop
...
Add Thai language for spaCy v2
2017-09-26 23:43:05 +02:00
Wannaphong Phatthiyaphaibun
3d5046c499
fix import in th
2017-09-26 22:41:20 +07:00
Wannaphong Phatthiyaphaibun
a63f790b8c
fix thai tag_map
2017-09-26 22:28:57 +07:00