Merge branch 'master' into develop

.github/contributors/tjkemp.md

@@ -0,0 +1,106 @@
+# spaCy contributor agreement
+
+This spaCy Contributor Agreement (**"SCA"**) is based on the
+[Oracle Contributor Agreement](http://www.oracle.com/technetwork/oca-405177.pdf).
+The SCA applies to any contribution that you make to any product or project
+managed by us (the **"project"**), and sets out the intellectual property rights
+you grant to us in the contributed materials. The term **"us"** shall mean
+[ExplosionAI UG (haftungsbeschränkt)](https://explosion.ai/legal). The term
+**"you"** shall mean the person or entity identified below.
+
+If you agree to be bound by these terms, fill in the information requested
+below and include the filled-in version with your first pull request, under the
+folder [`.github/contributors/`](/.github/contributors/). The name of the file
+should be your GitHub username, with the extension `.md`. For example, the user
+example_user would create the file `.github/contributors/example_user.md`.
+
+Read this agreement carefully before signing. These terms and conditions
+constitute a binding legal agreement.
+
+## Contributor Agreement
+
+1. The term "contribution" or "contributed materials" means any source code,
+object code, patch, tool, sample, graphic, specification, manual,
+documentation, or any other material posted or submitted by you to the project.
+
+2. With respect to any worldwide copyrights, or copyright applications and
+registrations, in your contribution:
+
+    * you hereby assign to us joint ownership, and to the extent that such
+    assignment is or becomes invalid, ineffective or unenforceable, you hereby
+    grant to us a perpetual, irrevocable, non-exclusive, worldwide, no-charge,
+    royalty-free, unrestricted license to exercise all rights under those
+    copyrights. This includes, at our option, the right to sublicense these same
+    rights to third parties through multiple levels of sublicensees or other
+    licensing arrangements;
+
+    * you agree that each of us can do all things in relation to your
+    contribution as if each of us were the sole owners, and if one of us makes
+    a derivative work of your contribution, the one who makes the derivative
+    work (or has it made will be the sole owner of that derivative work;
+
+    * you agree that you will not assert any moral rights in your contribution
+    against us, our licensees or transferees;
+
+    * you agree that we may register a copyright in your contribution and
+    exercise all ownership rights associated with it; and
+
+    * you agree that neither of us has any duty to consult with, obtain the
+    consent of, pay or render an accounting to the other for any use or
+    distribution of your contribution.
+
+3. With respect to any patents you own, or that you can license without payment
+to any third party, you hereby grant to us a perpetual, irrevocable,
+non-exclusive, worldwide, no-charge, royalty-free license to:
+
+    * make, have made, use, sell, offer to sell, import, and otherwise transfer
+    your contribution in whole or in part, alone or in combination with or
+    included in any product, work or materials arising out of the project to
+    which your contribution was submitted, and
+
+    * at our option, to sublicense these same rights to third parties through
+    multiple levels of sublicensees or other licensing arrangements.
+
+4. Except as set out above, you keep all right, title, and interest in your
+contribution. The rights that you grant to us under these terms are effective
+on the date you first submitted a contribution to us, even if your submission
+took place before the date you sign these terms.
+
+5. You covenant, represent, warrant and agree that:
+
+    * Each contribution that you submit is and shall be an original work of
+    authorship and you can legally grant the rights set out in this SCA;
+
+    * to the best of your knowledge, each contribution will not violate any
+    third party's copyrights, trademarks, patents, or other intellectual
+    property rights; and
+
+    * each contribution shall be in compliance with U.S. export control laws and
+    other applicable export and import laws. You agree to notify us if you
+    become aware of any circumstance which would make any of the foregoing
+    representations inaccurate in any respect. We may publicly disclose your
+    participation in the project, including the fact that you have signed the SCA.
+
+6. This SCA is governed by the laws of the State of California and applicable
+U.S. Federal law. Any choice of law rules will not apply.
+
+7. Please place an “x” on one of the applicable statement below. Please do NOT
+mark both statements:
+
+    * [ ] I am signing on behalf of myself as an individual and no other person
+    or entity, including my employer, has or will have rights with respect to my
+    contributions.
+
+    * [ ] I am signing on behalf of my employer or a legal entity and I have the
+    actual authority to contractually bind that entity.
+
+## Contributor Details
+
+| Field                          | Entry                |
+|------------------------------- | -------------------- |
+| Name                           | Tero Kemppi          |
+| Company name (if applicable)   |                      |
+| Title or role (if applicable)  |                      |
+| Date                           | 14-07-2018           |
+| GitHub username                | tjkemp               |
+| Website (optional)             |                      |

.github/contributors/x-ji.md

@@ -0,0 +1,106 @@
+# spaCy contributor agreement
+
+This spaCy Contributor Agreement (**"SCA"**) is based on the
+[Oracle Contributor Agreement](http://www.oracle.com/technetwork/oca-405177.pdf).
+The SCA applies to any contribution that you make to any product or project
+managed by us (the **"project"**), and sets out the intellectual property rights
+you grant to us in the contributed materials. The term **"us"** shall mean
+[ExplosionAI UG (haftungsbeschränkt)](https://explosion.ai/legal). The term
+**"you"** shall mean the person or entity identified below.
+
+If you agree to be bound by these terms, fill in the information requested
+below and include the filled-in version with your first pull request, under the
+folder [`.github/contributors/`](/.github/contributors/). The name of the file
+should be your GitHub username, with the extension `.md`. For example, the user
+example_user would create the file `.github/contributors/example_user.md`.
+
+Read this agreement carefully before signing. These terms and conditions
+constitute a binding legal agreement.
+
+## Contributor Agreement
+
+1. The term "contribution" or "contributed materials" means any source code,
+object code, patch, tool, sample, graphic, specification, manual,
+documentation, or any other material posted or submitted by you to the project.
+
+2. With respect to any worldwide copyrights, or copyright applications and
+registrations, in your contribution:
+
+    * you hereby assign to us joint ownership, and to the extent that such
+    assignment is or becomes invalid, ineffective or unenforceable, you hereby
+    grant to us a perpetual, irrevocable, non-exclusive, worldwide, no-charge,
+    royalty-free, unrestricted license to exercise all rights under those
+    copyrights. This includes, at our option, the right to sublicense these same
+    rights to third parties through multiple levels of sublicensees or other
+    licensing arrangements;
+
+    * you agree that each of us can do all things in relation to your
+    contribution as if each of us were the sole owners, and if one of us makes
+    a derivative work of your contribution, the one who makes the derivative
+    work (or has it made will be the sole owner of that derivative work;
+
+    * you agree that you will not assert any moral rights in your contribution
+    against us, our licensees or transferees;
+
+    * you agree that we may register a copyright in your contribution and
+    exercise all ownership rights associated with it; and
+
+    * you agree that neither of us has any duty to consult with, obtain the
+    consent of, pay or render an accounting to the other for any use or
+    distribution of your contribution.
+
+3. With respect to any patents you own, or that you can license without payment
+to any third party, you hereby grant to us a perpetual, irrevocable,
+non-exclusive, worldwide, no-charge, royalty-free license to:
+
+    * make, have made, use, sell, offer to sell, import, and otherwise transfer
+    your contribution in whole or in part, alone or in combination with or
+    included in any product, work or materials arising out of the project to
+    which your contribution was submitted, and
+
+    * at our option, to sublicense these same rights to third parties through
+    multiple levels of sublicensees or other licensing arrangements.
+
+4. Except as set out above, you keep all right, title, and interest in your
+contribution. The rights that you grant to us under these terms are effective
+on the date you first submitted a contribution to us, even if your submission
+took place before the date you sign these terms.
+
+5. You covenant, represent, warrant and agree that:
+
+    * Each contribution that you submit is and shall be an original work of
+    authorship and you can legally grant the rights set out in this SCA;
+
+    * to the best of your knowledge, each contribution will not violate any
+    third party's copyrights, trademarks, patents, or other intellectual
+    property rights; and
+
+    * each contribution shall be in compliance with U.S. export control laws and
+    other applicable export and import laws. You agree to notify us if you
+    become aware of any circumstance which would make any of the foregoing
+    representations inaccurate in any respect. We may publicly disclose your
+    participation in the project, including the fact that you have signed the SCA.
+
+6. This SCA is governed by the laws of the State of California and applicable
+U.S. Federal law. Any choice of law rules will not apply.
+
+7. Please place an “x” on one of the applicable statement below. Please do NOT
+mark both statements:
+
+    * [x] I am signing on behalf of myself as an individual and no other person
+    or entity, including my employer, has or will have rights with respect to my
+    contributions.
+
+    * [ ] I am signing on behalf of my employer or a legal entity and I have the
+    actual authority to contractually bind that entity.
+
+## Contributor Details
+
+| Field                         | Entry        |
+|-------------------------------|--------------|
+| Name                          | Xiang Ji     |
+| Company name (if applicable)  |              |
+| Title or role (if applicable) |              |
+| Date                          | 07 July 2018 |
+| GitHub username               | x-ji         |
+| Website (optional)            |              |

CONTRIBUTING.md

@@ -138,11 +138,11 @@ files, a compiler, [pip](https://pip.pypa.io/en/latest/installing/),
 [git](https://git-scm.com) installed. The compiler is usually the trickiest part.
 
 ```
-python -m pip install -U pip venv
+python -m pip install -U pip
 git clone https://github.com/explosion/spaCy
 cd spaCy
 
-venv .env
+python -m venv .env
 source .env/bin/activate
 export PYTHONPATH=`pwd`
 pip install -r requirements.txt

README.rst

@@ -133,7 +133,7 @@ environment to avoid modifying system state:
 
 .. code:: bash
 
-    venv .env
+    python -m venv .env
     source .env/bin/activate
     pip install spacy
 
@@ -255,12 +255,12 @@ details.
 
 .. code:: bash
 
-    # make sure you are using recent pip/virtualenv versions
-    python -m pip install -U pip venv
+    # make sure you are using the latest pip
+    python -m pip install -U pip
     git clone https://github.com/explosion/spaCy
     cd spaCy
 
-    venv .env
+    python -m venv .env
     source .env/bin/activate
     export PYTHONPATH=`pwd`
     pip install -r requirements.txt

spacy/lang/fi/examples.py

@@ -8,8 +8,11 @@ Example sentences to test spaCy and its language models.
 """
 
 sentences = [
-    "Apple harkitsee ostavansa startup-yrityksen UK:sta 1 miljardilla dollarilla.",
-    "Itseajavat autot siirtävät vakuutusriskin valmistajille.",
-    "San Francisco harkitsee jakelurobottien kieltämistä jalkakäytävillä.",
-    "Lontoo on iso kaupunki Iso-Britanniassa."
+    "Itseajavat autot siirtävät vakuutusvastuun autojen valmistajille",
+    "San Francisco harkitsee toimitusrobottien liikkumisen kieltämistä jalkakäytävillä",
+    "Lontoo on suuri kaupunki Yhdistyneessä Kuningaskunnassa.",
+    "Missä sinä olet?",
+    "Mikä on Yhdysvaltojen pääkaupunki?",
+    "Kuka on Suomen presidentti?",
+    "Milloin Sauli Niinistö on syntynyt?"
 ]

spacy/lang/ja/__init__.py

@@ -82,6 +82,7 @@ class JapaneseTokenizer(object):
         for token, dtoken in zip(doc, dtokens):
             token._.mecab_tag = dtoken.pos
             token.tag_ = resolve_pos(dtoken)
+            token.lemma_ = dtoken.lemma
         return doc
 
     # add dummy methods for to_bytes, from_bytes, to_disk and from_disk to

spacy/lang/ja/stop_words.py

@@ -0,0 +1,49 @@
+# coding: utf8
+from __future__ import unicode_literals
+
+# This list was created by taking the top 2000 words from a Wikipedia dump and
+# filtering out everything that wasn't hiragana. ー (one) was also added.
+# Considered keeping some non-hiragana words but too many place names were
+# present.
+STOP_WORDS = set("""
+あ あっ あまり あり ある あるいは あれ
+い いい いう いく いずれ いっ いつ いる いわ
+うち
+え
+お おい おけ および おら おり
+か かけ かつ かつて かなり から が
+き きっかけ
+くる くん
+こ こう ここ こと この これ ご ごと
+さ さらに さん
+し しか しかし しまう しまっ しよう
+す すぐ すべて する ず
+せ せい せる
+そう そこ そして その それ それぞれ
+た たい ただし たち ため たら たり だ だけ だっ
+ち ちゃん
+つ つい つけ つつ
+て で でき できる です
+と とき ところ とっ とも どう
+な ない なお なかっ ながら なく なけれ なし なっ など なら なり なる
+に にて
+ぬ
+ね
+の のち のみ
+は はじめ ば
+ひと
+ぶり
+へ べき
+ほか ほとんど ほど ほぼ
+ま ます また まで まま
+み
+も もう もっ もと もの
+や やっ
+よ よう よく よっ より よる よれ
+ら らしい られ られる
+る
+れ れる
+を
+ん
+一
+""".split())

spacy/tests/conftest.py

@@ -49,6 +49,11 @@ def RU(request):
     pymorphy = pytest.importorskip('pymorphy2')
     return util.get_lang_class('ru')()
 
+@pytest.fixture()
+def JA(request):
+    mecab = pytest.importorskip("MeCab")
+    return util.get_lang_class('ja')()
+
 
 #@pytest.fixture(params=_languages)
 #def tokenizer(request):

spacy/tests/lang/ja/test_lemma.py

@@ -0,0 +1,18 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import pytest
+
+LEMMAS = (
+        ('新しく', '新しい'),
+        ('赤く', '赤い'),
+        ('すごく', '凄い'),
+        ('いただきました', '頂く'),
+        ('なった', '成る'))
+
+@pytest.mark.parametrize('word,lemma', LEMMAS)
+def test_japanese_lemmas(JA, word, lemma):
+    test_lemma = JA(word)[0].lemma_
+    assert test_lemma == lemma
+
+

website/usage/_install/_instructions.jade

@@ -23,7 +23,7 @@ p
     |  virtual environment to avoid modifying system state:
 
 +code(false, "bash").
-    venv .env
+    python -m venv .env
     source .env/bin/activate
     pip install spacy
 
@@ -121,11 +121,11 @@ p
     |  #[a(href="#source-windows") Windows] for details.
 
 +code(false, "bash").
-    python -m pip install -U pip venv              # update pip & virtualenv
+    python -m pip install -U pip                   # update pip
     git clone #{gh("spaCy")}   # clone spaCy
     cd spaCy                                       # navigate into directory
 
-    venv .env                                      # create environment in .env
+    python -m venv .env                            # create environment in .env
     source .env/bin/activate                       # activate virtual environment
     export PYTHONPATH=`pwd`                        # set Python path to spaCy directory
     pip install -r requirements.txt                # install all requirements

website/usage/_install/_quickstart.jade

@@ -6,7 +6,7 @@
     +qs({config: 'venv', python: 2}) python -m pip install -U virtualenv
     +qs({config: 'venv', python: 3}) python -m pip install -U venv
     +qs({config: 'venv', python: 2}) virtualenv .env
-    +qs({config: 'venv', python: 3}) venv .env
+    +qs({config: 'venv', python: 3}) python -m venv .env
     +qs({config: 'venv', os: 'mac'}) source .env/bin/activate
     +qs({config: 'venv', os: 'linux'}) source .env/bin/activate
     +qs({config: 'venv', os: 'windows'}) .env\Scripts\activate