diff --git a/README.rst b/README.rst index 8d53e8548..0ca3ade29 100644 --- a/README.rst +++ b/README.rst @@ -1,10 +1,10 @@ spaCy: Industrial-strength NLP ****************************** -spaCy is a library for advanced natural language processing in Python and -Cython. spaCy is built on the very latest research, but it isn't researchware. -It was designed from day one to be used in real products. spaCy currently supports -English and German, as well as tokenization for Chinese, Spanish, Italian, French, +spaCy is a library for advanced natural language processing in Python and +Cython. spaCy is built on the very latest research, but it isn't researchware. +It was designed from day one to be used in real products. spaCy currently supports +English and German, as well as tokenization for Chinese, Spanish, Italian, French, Portuguese, Dutch, Swedish, Finnish, Hungarian and Bengali. It's commercial open-source software, released under the MIT license. @@ -13,23 +13,23 @@ software, released under the MIT license. .. image:: https://img.shields.io/travis/explosion/spaCy/master.svg?style=flat-square :target: https://travis-ci.org/explosion/spaCy :alt: Build Status - + .. image:: https://img.shields.io/github/release/explosion/spacy.svg?style=flat-square - :target: https://github.com/explosion/spaCy/releases + :target: https://github.com/explosion/spaCy/releases :alt: Current Release Version - + .. image:: https://img.shields.io/pypi/v/spacy.svg?style=flat-square :target: https://pypi.python.org/pypi/spacy :alt: pypi Version - + .. image:: https://anaconda.org/conda-forge/spacy/badges/version.svg :target: https://anaconda.org/conda-forge/spacy :alt: conda Version - + .. image:: https://img.shields.io/badge/gitter-join%20chat%20%E2%86%92-09a3d5.svg?style=flat-square :target: https://gitter.im/explosion/spaCy :alt: spaCy on Gitter - + .. image:: https://img.shields.io/twitter/follow/spacy_io.svg?style=social&label=Follow :target: https://twitter.com/spacy_io :alt: spaCy on Twitter @@ -55,7 +55,7 @@ software, released under the MIT license. +---------------------------+------------------------------------------------------------------------------------------------------------+ | **Bug reports**     | `GitHub Issue tracker `_                                     | +---------------------------+------------------------------------------------------------------------------------------------------------+ -| **Usage questions**   | `StackOverflow `_, `Reddit usergroup                     | +| **Usage questions**   | `StackOverflow `_, `Reddit usergroup                     | | | `_, `Gitter chat `_ | +---------------------------+------------------------------------------------------------------------------------------------------------+ | **General discussion** | `Reddit usergroup `_, | @@ -104,100 +104,143 @@ Supports Install spaCy ============= -spaCy is compatible with 64-bit CPython 2.6+/3.3+ and runs on Unix/Linux, OS X -and Windows. Source packages are available via -`pip `_. Please make sure that -you have a working build enviroment set up. See notes on Ubuntu, macOS/OS X and Windows -for details. +spaCy is compatible with **64-bit CPython 2.6+/3.3+** and runs on **Unix/Linux**, +**macOS/OS X** and **Windows**. The latest spaCy releases are available over +`pip `_ (source packages only) and +`conda `_. Installation requires a working +build environment. See notes on Ubuntu, macOS/OS X and Windows for details. pip --- -When using pip it is generally recommended to install packages in a virtualenv to -avoid modifying system state: +Using pip, spaCy releases are currently only available as source packages. .. code:: bash - pip install spacy + pip install -U spacy -Python packaging is awkward at the best of times, and it's particularly tricky with -C extensions, built via Cython, requiring large data files. So, please report issues -as you encounter them. +When using pip it is generally recommended to install packages in a ``virtualenv`` +to avoid modifying system state: + +.. code:: bash + + virtualenv .env + source .env/bin/activate + pip install spacy conda ----- -If you're using conda, you can install spaCy via ``conda-forge``: +Thanks to our great community, we've finally re-added conda support. You can now +install spaCy via ``conda-forge``: .. code:: bash   conda config --add channels conda-forge   conda install spacy - + For the feedstock including the build recipe and configuration, check out `this repository `_. -Thanks to our great community, we've finally re-added conda support — improvements -and pull requests to the recipe and setup are always appreciated. +Improvements and pull requests to the recipe and setup are always appreciated. -Install model -============= +Download models +=============== -After installation you need to download a language model. Currently only models for -English and German, named ``en`` and ``de``, are available. +After installation you need to download a language model. Models for English +(``en``) and German (``de``) are available. .. code:: bash python -m spacy.en.download all python -m spacy.de.download all -The download command fetches about 1 GB of data which it installs +The download command fetches about 1 GB of data which it installs within the ``spacy`` package directory. -Upgrading spaCy -=============== - -To upgrade spaCy to the latest release: - -pip ---- - -.. code:: bash - - pip install -U spacy - -Sometimes new releases require a new language model. Then you will have to upgrade to -a new model, too. You can also force re-downloading and installing a new language model: +Sometimes new releases require a new language model. Then you will have to +upgrade to a new model, too. You can also force re-downloading and installing a +new language model: .. code:: bash python -m spacy.en.download --force +Download model to custom location +--------------------------------- + +You can specify where ``spacy.en.download`` and ``spacy.de.download`` download +the language model to using the ``--data-path`` or ``-d`` argument: + +.. code:: bash + + python -m spacy.en.download all --data-path /some/dir + +If you choose to download to a custom location, you will need to tell spaCy where to load the model +from in order to use it. You can do this either by calling ``spacy.util.set_data_path()`` before +calling ``spacy.load()``, or by passing a ``path`` argument to the ``spacy.en.English`` or +``spacy.de.German`` constructors. + +Download models manually +------------------------ + +As of v1.6, the models and word vectors are also available as direct downloads +from GitHub, attached to the `releases `_ +as ``.tar.gz`` archives. + +To install the models manually, first find the default data path. You can use +``spacy.util.get_data_path()`` to find the directory where spaCy will look for +its models, or change the default data path with ``spacy.util.set_data_path()``. +Then simply unpack the archive and place the contained folder in that directory. +You can now load the models via ``spacy.load()``. + Compile from source =================== -The other way to install spaCy is to clone its GitHub repository and build it from +The other way to install spaCy is to clone its +`GitHub repository `_ and build it from source. That is the common way if you want to make changes to the code base. - -You'll need to make sure that you have a development enviroment consisting of a -Python distribution including header files, a compiler, pip, virtualenv and git -installed. The compiler part is the trickiest. How to do that depends on your -system. See notes on Ubuntu, OS X and Windows for details. +You'll need to make sure that you have a development enviroment consisting of a +Python distribution including header files, a compiler, +`pip `__, `virtualenv `_ +and `git `_ installed. The compiler part is the trickiest. +How to do that depends on your system. See notes on Ubuntu, OS X and Windows for +details. .. code:: bash # make sure you are using recent pip/virtualenv versions python -m pip install -U pip virtualenv - - # find git install instructions at https://git-scm.com/downloads - git clone https://github.com/explosion/spaCy.git - + git clone #{gh("spaCy")} cd spaCy - virtualenv .env && source .env/bin/activate + + virtualenv .env + source .env/bin/activate pip install -r requirements.txt pip install -e . - -Compared to regular install via pip `requirements.txt `_ -additionally installs developer dependencies such as cython. + +Compared to regular install via pip `requirements.txt `_ +additionally installs developer dependencies such as Cython. + +Instead of the above verbose commands, you can also use the following +`Fabric `_ commands: + ++---------------+--------------------------------------------------------------+ +| ``fab env`` | Create ``virtualenv`` and delete previous one, if it exists. | ++---------------+--------------------------------------------------------------+ +| ``fab make`` | Compile the source. | ++---------------+--------------------------------------------------------------+ +| ``fab clean`` | Remove compiled objects, including the generated C++. | ++---------------+--------------------------------------------------------------+ +| ``fab test`` | Run basic tests, aborting after first failure. | ++---------------+--------------------------------------------------------------+ + +All commands assume that your ``virtualenv`` is located in a directory ``.env``. +If you're using a different directory, you can change it via the environment +variable ``VENV_DIR``, for example: + +.. code:: bash + + VENV_DIR=".custom-env" fab clean make Ubuntu ------ @@ -211,54 +254,38 @@ Install system-level dependencies via ``apt-get``: macOS / OS X ------------ -Install a recent version of `XCode `_, -including the so-called "Command Line Tools". macOS and OS X ship with Python +Install a recent version of `XCode `_, +including the so-called "Command Line Tools". macOS and OS X ship with Python and git preinstalled. Windows ------- Install a version of `Visual Studio Express `_ -or higher that matches the version that was used to compile your Python -interpreter. For official distributions these are VS 2008 (Python 2.7), +or higher that matches the version that was used to compile your Python +interpreter. For official distributions these are VS 2008 (Python 2.7), VS 2010 (Python 3.4) and VS 2015 (Python 3.5). Run tests ========= -spaCy comes with an extensive test suite. First, find out where spaCy is -installed: +spaCy comes with an `extensive test suite `_. First, find out where +spaCy is installed: .. code:: bash - + python -c "import os; import spacy; print(os.path.dirname(spacy.__file__))" -Then run ``pytest`` on that directory. The flags ``--vectors``, ``--slow`` +Then run ``pytest`` on that directory. The flags ``--vectors``, ``--slow`` and ``--model`` are optional and enable additional tests: .. code:: bash - + # make sure you are using recent pytest version python -m pip install -U pytest python -m pytest --vectors --model --slow -Download model to custom location -================================= - -You can specify where ``spacy.en.download`` and ``spacy.de.download`` download the language model -to using the ``--data-path`` or ``-d`` argument: - -.. code:: bash - - python -m spacy.en.download all --data-path /some/dir - - -If you choose to download to a custom location, you will need to tell spaCy where to load the model -from in order to use it. You can do this either by calling ``spacy.util.set_data_path()`` before -calling ``spacy.load()``, or by passing a ``path`` argument to the ``spacy.en.English`` or -``spacy.de.German`` constructors. - Changelog ========= @@ -473,10 +500,10 @@ Thanks to `@daylen `_, `@RahulKulhari `_: *German!* ------------------------------------------------------------------------------------------- -spaCy finally supports another language, in addition to English. We're lucky -to have Wolfgang Seeker on the team, and the new German model is just the -beginning. Now that there are multiple languages, you should consider loading -spaCy via the ``load()`` function. This function also makes it easier to load extra +spaCy finally supports another language, in addition to English. We're lucky +to have Wolfgang Seeker on the team, and the new German model is just the +beginning. Now that there are multiple languages, you should consider loading +spaCy via the ``load()`` function. This function also makes it easier to load extra word vector data for English: .. code:: python @@ -484,25 +511,25 @@ word vector data for English: import spacy en_nlp = spacy.load('en', vectors='en_glove_cc_300_1m_vectors') de_nlp = spacy.load('de') - -To support use of the load function, there are also two new helper functions: -``spacy.get_lang_class`` and ``spacy.set_lang_class``. Once the German model is + +To support use of the load function, there are also two new helper functions: +``spacy.get_lang_class`` and ``spacy.set_lang_class``. Once the German model is loaded, you can use it just like the English model: .. code:: python doc = nlp(u'''Wikipedia ist ein Projekt zum Aufbau einer Enzyklopädie aus freien Inhalten, zu dem du mit deinem Wissen beitragen kannst. Seit Mai 2001 sind 1.936.257 Artikel in deutscher Sprache entstanden.''') - + for sent in doc.sents: print(sent.root.text, sent.root.n_lefts, sent.root.n_rights) - + # (u'ist', 1, 2) # (u'sind', 1, 3) - -The German model provides tokenization, POS tagging, sentence boundary detection, -syntactic dependency parsing, recognition of organisation, location and person -entities, and word vector representations trained on a mix of open subtitles and -Wikipedia data. It doesn't yet provide lemmatisation or morphological analysis, + +The German model provides tokenization, POS tagging, sentence boundary detection, +syntactic dependency parsing, recognition of organisation, location and person +entities, and word vector representations trained on a mix of open subtitles and +Wikipedia data. It doesn't yet provide lemmatisation or morphological analysis, and it doesn't yet recognise numeric entities such as numbers and dates. **Bugfixes** @@ -518,7 +545,7 @@ and it doesn't yet recognise numeric entities such as numbers and dates. 2016-03-08 `v0.100.6 `_: *Add support for GloVe vectors* ----------------------------------------------------------------------------------------------------------------- -This release offers improved support for replacing the word vectors used by spaCy. +This release offers improved support for replacing the word vectors used by spaCy. To install Stanford's GloVe vectors, trained on the Common Crawl, just run: .. code:: bash @@ -527,8 +554,8 @@ To install Stanford's GloVe vectors, trained on the Common Crawl, just run: To reduce memory usage and loading time, we've trimmed the vocabulary down to 1m entries. -This release also integrates all the code necessary for German parsing. A German model -will be released shortly. To assist in multi-lingual processing, we've added a ``load()`` +This release also integrates all the code necessary for German parsing. A German model +will be released shortly. To assist in multi-lingual processing, we've added a ``load()`` function. To load the English model with the GloVe vectors: .. code:: python diff --git a/website/docs/usage/index.jade b/website/docs/usage/index.jade index f7a957ad9..479635e4b 100644 --- a/website/docs/usage/index.jade +++ b/website/docs/usage/index.jade @@ -66,7 +66,9 @@ p # Check whether the model was successfully installed python -c "import spacy; spacy.load('en'); print('OK')" -p The download command fetches about 1 GB of data which it installs within the #[code spacy] package directory. +p + | The download command fetches about 1 GB of data which it + | installs within the #[code spacy] package directory. +h(3, "custom-location") Download model to custom location @@ -144,7 +146,7 @@ p +row +cell #[code fab clean] - +cell Remove compiled objects including the generated C++. + +cell Remove compiled objects, including the generated C++. +row +cell #[code fab test] @@ -153,7 +155,7 @@ p p | All commands assume that your #[code virtualenv] is located in a | directory #[code .env]. If you're using a different directory, you can - | change it via environment the variable #[code VENV_DIR], for example: + | change it via the environment variable #[code VENV_DIR], for example: +code(false, "bash"). VENV_DIR=".custom-env" fab clean make