From e77ed953f4a032d05329102fe4afc42a89f7f630 Mon Sep 17 00:00:00 2001 From: Matthew Honnibal Date: Sun, 4 Jun 2017 12:03:22 +0200 Subject: [PATCH 1/7] Update GPU instructions --- website/docs/usage/index.jade | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/website/docs/usage/index.jade b/website/docs/usage/index.jade index d3deaa17e..3f940cbb1 100644 --- a/website/docs/usage/index.jade +++ b/website/docs/usage/index.jade @@ -96,27 +96,15 @@ p | #[+a("http://chainer.org") Chainer]'s CuPy module, which provides | a NumPy-compatible interface for GPU arrays. -+aside("Why is this so complicated?") - | Installing Chainer when no GPU is available currently causes an - | error. We therefore do not specify Chainer as a dependency. However, - | CuPy will be split out into - | #[+a("https://www.slideshare.net/beam2d/chainer-v2-alpha/7") its own package] - | in Chainer v2.0. We'll have a smoother installation process for this - | in an upcoming version. - p | First, install follows the normal CUDA installation procedure. Next, set | your environment variables so that the installation will be able to find - | CUDA. Next, install Chainer, and check that CuPy can be imported - | correctly. Finally, install spaCy. + | CUDA. Finally, install spaCy. +code(false, "bash"). export CUDA_HOME=/usr/local/cuda-8.0 # Or wherever your CUDA is export PATH=$PATH:$CUDA_HOME/bin - pip install chainer - python -c "import cupy; assert cupy" # Check it installed - pip install spacy python -c "import thinc.neural.gpu_ops" # Check the GPU ops were built From 64ca5123bbf1ec0bba8df6ac3174a945f1fbc035 Mon Sep 17 00:00:00 2001 From: Matthew Honnibal Date: Sun, 4 Jun 2017 13:09:19 +0200 Subject: [PATCH 2/7] Add Architecture 101 blurb --- website/docs/usage/_spacy-101/_architecture.jade | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 website/docs/usage/_spacy-101/_architecture.jade diff --git a/website/docs/usage/_spacy-101/_architecture.jade b/website/docs/usage/_spacy-101/_architecture.jade new file mode 100644 index 000000000..46ab11a41 --- /dev/null +++ b/website/docs/usage/_spacy-101/_architecture.jade @@ -0,0 +1,15 @@ +//- 💫 DOCS > USAGE > SPACY 101 > ARCHITECTURE + +p + | The central data structures in spaCy are the #[code Doc] and the #[code Vocab]. + | The #[code doc] object owns the sequence of tokens and all their annotations. + | the #[code vocab] owns a set of look-up tables that make common information + | available across documents. By centralising strings, word vectors and lexical + | attributes, we avoid storing multiple copies of this data. This saves memory, and + | ensures there's a single source of truth. Text annotations are also designed to + | allow a single source of truth: the #[code Doc] object owns the data, and + | #[code Span] and #[code Token] are views that point into it. The #[code Doc] + | object is constructed by the #[code Tokenizer], and then modified in-place by + | the components of the pipeline. The #[code Language] object coordinates these + | components. It takes raw text and sends it through the pipeline, returning + | an annotated document. It also orchestrates training and serialisation. From aca53b95e1f4c822dd8a38605304d47eafa90c29 Mon Sep 17 00:00:00 2001 From: Matthew Honnibal Date: Sun, 4 Jun 2017 13:10:06 +0200 Subject: [PATCH 3/7] Link architecture blurb --- website/docs/usage/spacy-101.jade | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/usage/spacy-101.jade b/website/docs/usage/spacy-101.jade index 4f2642af0..50769cc4f 100644 --- a/website/docs/usage/spacy-101.jade +++ b/website/docs/usage/spacy-101.jade @@ -268,7 +268,7 @@ include _spacy-101/_language-data +h(2, "architecture") Architecture -+under-construction +include _spacy-101/_architecture.jade +image include ../../assets/img/docs/architecture.svg From f2c4a9f690bfbce42b94f980623449a1538f202d Mon Sep 17 00:00:00 2001 From: Matthew Honnibal Date: Sun, 4 Jun 2017 13:10:27 +0200 Subject: [PATCH 4/7] Edits to spacy-101 page --- website/docs/usage/spacy-101.jade | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/website/docs/usage/spacy-101.jade b/website/docs/usage/spacy-101.jade index 50769cc4f..629e5b12f 100644 --- a/website/docs/usage/spacy-101.jade +++ b/website/docs/usage/spacy-101.jade @@ -65,13 +65,15 @@ p | not designed specifically for chat bots, and only provides the | underlying text processing capabilities. +item #[strong spaCy is not research software]. - | It's is built on the latest research, but unlike - | #[+a("https://github./nltk/nltk") NLTK], which is intended for - | teaching and research, spaCy follows a more opinionated approach and - | focuses on production usage. Its aim is to provide you with the best - | possible general-purpose solution for text processing and machine learning - | with text input – but this also means that there's only one implementation - | of each component. + | It's is built on the latest research, but it's designed to get + | things done. This leads to fairly different design decisions than + | #[+a("https://github./nltk/nltk") NLTK] + | or #[+a("https://stanfordnlp.github.io/CorenlP") CoreNLP], which were + | created as platforms for teaching and research. The main difference + | is that spaCy is integrated and opinionated. We try to avoid asking + | the user to choose between multiple algorithms that deliver equivalent + | functionality. Keeping our menu small lets us deliver generally better + | performance and developer experience. +item #[strong spaCy is not a company]. | It's an open-source library. Our company publishing spaCy and other | software is called #[+a(COMPANY_URL, true) Explosion AI]. @@ -79,7 +81,7 @@ p +h(2, "features") Features p - | Across the documentations, you'll come across mentions of spaCy's + | Across the documentation, you'll come across mentions of spaCy's | features and capabilities. Some of them refer to linguistic concepts, | while others are related to more general machine learning functionality. @@ -171,7 +173,9 @@ p p | Even though a #[code Doc] is processed – e.g. split into individual words | and annotated – it still holds #[strong all information of the original text], - | like whitespace characters. This way, you'll never lose any information + | like whitespace characters. You can always get the offset of a token into the + | original string, or reconstruct the original by joining the tokens and their + | trailing whitespace. This way, you'll never lose any information | when processing text with spaCy. +h(3, "annotations-token") Tokenization From 1d6377218a92c19f403ff63f0cbcdbdfe4b9f6a6 Mon Sep 17 00:00:00 2001 From: ines Date: Sun, 4 Jun 2017 13:28:58 +0200 Subject: [PATCH 5/7] Update architecture blurb and move other info --- .../docs/usage/_spacy-101/_architecture.jade | 92 ++++++++++++++++--- website/docs/usage/spacy-101.jade | 62 ------------- 2 files changed, 80 insertions(+), 74 deletions(-) diff --git a/website/docs/usage/_spacy-101/_architecture.jade b/website/docs/usage/_spacy-101/_architecture.jade index 46ab11a41..4905171e7 100644 --- a/website/docs/usage/_spacy-101/_architecture.jade +++ b/website/docs/usage/_spacy-101/_architecture.jade @@ -1,15 +1,83 @@ //- 💫 DOCS > USAGE > SPACY 101 > ARCHITECTURE p - | The central data structures in spaCy are the #[code Doc] and the #[code Vocab]. - | The #[code doc] object owns the sequence of tokens and all their annotations. - | the #[code vocab] owns a set of look-up tables that make common information - | available across documents. By centralising strings, word vectors and lexical - | attributes, we avoid storing multiple copies of this data. This saves memory, and - | ensures there's a single source of truth. Text annotations are also designed to - | allow a single source of truth: the #[code Doc] object owns the data, and - | #[code Span] and #[code Token] are views that point into it. The #[code Doc] - | object is constructed by the #[code Tokenizer], and then modified in-place by - | the components of the pipeline. The #[code Language] object coordinates these - | components. It takes raw text and sends it through the pipeline, returning - | an annotated document. It also orchestrates training and serialisation. + | The central data structures in spaCy are the #[code Doc] and the + | #[code Vocab]. The #[code Doc] object owns the + | #[strong sequence of tokens] and all their annotations. The #[code Vocab] + | object owns a set of #[strong look-up tables] that make common + | information available across documents. By centralising strings, word + | vectors and lexical attributes, we avoid storing multiple copies of this + | data. This saves memory, and ensures there's a + | #[strong single source of truth]. + +p + | Text annotations are also designed to allow a single source of truth: the + | #[code Doc] object owns the data, and #[code Span] and #[code Token] are + | #[strong views that point into it]. The #[code Doc] object is constructed + | by the #[code Tokenizer], and then #[strong modified in place] by the + | components of the pipeline. The #[code Language] object coordinates these + | components. It takes raw text and sends it through the pipeline, + | returning an #[strong annotated document]. It also orchestrates training + | and serialization. + ++image + include ../../../assets/img/docs/architecture.svg + .u-text-right + +button("/assets/img/docs/architecture.svg", false, "secondary").u-text-tag View large graphic + ++table(["Name", "Description"]) + +row + +cell #[+api("language") #[code Language]] + +cell + | A text-processing pipeline. Usually you'll load this once per + | process as #[code nlp] and pass the instance around your application. + + +row + +cell #[+api("doc") #[code Doc]] + +cell A container for accessing linguistic annotations. + + +row + +cell #[+api("span") #[code Span]] + +cell A slice from a #[code Doc] object. + + +row + +cell #[+api("token") #[code Token]] + +cell + | An individual token — i.e. a word, punctuation symbol, whitespace, + | etc. + + +row + +cell #[+api("lexeme") #[code Lexeme]] + +cell + | An entry in the vocabulary. It's a word type with no context, as + | opposed to a word token. It therefore has no part-of-speech tag, + | dependency parse etc. + + +row + +cell #[+api("vocab") #[code Vocab]] + +cell + | A lookup table for the vocabulary that allows you to access + | #[code Lexeme] objects. + + +row + +cell #[code Morphology] + +cell + | Assign linguistic features like lemmas, noun case, verb tense etc. + | based on the word and its part-of-speech tag. + + +row + +cell #[+api("stringstore") #[code StringStore]] + +cell Map strings to and from hash values. + + +row + +row + +cell #[+api("tokenizer") #[code Tokenizer]] + +cell + | Segment text, and create #[code Doc] objects with the discovered + | segment boundaries. + + +row + +cell #[+api("matcher") #[code Matcher]] + +cell + | Match sequences of tokens, based on pattern rules, similar to + | regular expressions. diff --git a/website/docs/usage/spacy-101.jade b/website/docs/usage/spacy-101.jade index 629e5b12f..67d3a83fe 100644 --- a/website/docs/usage/spacy-101.jade +++ b/website/docs/usage/spacy-101.jade @@ -274,68 +274,6 @@ include _spacy-101/_language-data include _spacy-101/_architecture.jade -+image - include ../../assets/img/docs/architecture.svg - .u-text-right - +button("/assets/img/docs/architecture.svg", false, "secondary").u-text-tag View large graphic - -+table(["Name", "Description"]) - +row - +cell #[+api("language") #[code Language]] - +cell - | A text-processing pipeline. Usually you'll load this once per - | process as #[code nlp] and pass the instance around your application. - - +row - +cell #[+api("doc") #[code Doc]] - +cell A container for accessing linguistic annotations. - - +row - +cell #[+api("span") #[code Span]] - +cell A slice from a #[code Doc] object. - - +row - +cell #[+api("token") #[code Token]] - +cell - | An individual token — i.e. a word, punctuation symbol, whitespace, - | etc. - - +row - +cell #[+api("lexeme") #[code Lexeme]] - +cell - | An entry in the vocabulary. It's a word type with no context, as - | opposed to a word token. It therefore has no part-of-speech tag, - | dependency parse etc. - - +row - +cell #[+api("vocab") #[code Vocab]] - +cell - | A lookup table for the vocabulary that allows you to access - | #[code Lexeme] objects. - - +row - +cell #[code Morphology] - +cell - | Assign linguistic features like lemmas, noun case, verb tense etc. - | based on the word and its part-of-speech tag. - - +row - +cell #[+api("stringstore") #[code StringStore]] - +cell Map strings to and from hash values. - - +row - +row - +cell #[+api("tokenizer") #[code Tokenizer]] - +cell - | Segment text, and create #[code Doc] objects with the discovered - | segment boundaries. - - +row - +cell #[+api("matcher") #[code Matcher]] - +cell - | Match sequences of tokens, based on pattern rules, similar to - | regular expressions. - +h(3, "architecture-pipeline") Pipeline components +table(["Name", "Description"]) From 22dd18c3644119c5885f520beb102dc45af71e6d Mon Sep 17 00:00:00 2001 From: ines Date: Sun, 4 Jun 2017 13:29:13 +0200 Subject: [PATCH 6/7] Remove redundant CPU commands --- website/docs/usage/index.jade | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/website/docs/usage/index.jade b/website/docs/usage/index.jade index 3f940cbb1..817b08ba9 100644 --- a/website/docs/usage/index.jade +++ b/website/docs/usage/index.jade @@ -21,13 +21,8 @@ p +qs({config: 'venv', os: 'linux'}) source .env/bin/activate +qs({config: 'venv', os: 'windows'}) .env\Scripts\activate - +qs({config: 'gpu', os: 'mac'}) export CUDA_HOME=/usr/local/cuda-8.0 - +qs({config: 'gpu', os: 'mac'}) export PATH=$PATH:$CUDA_HOME/bin - +qs({config: 'gpu', os: 'linux'}) export CUDA_HOME=/usr/local/cuda-8.0 - +qs({config: 'gpu', os: 'linux'}) export PATH=$PATH:$CUDA_HOME/bin - +qs({config: 'gpu', package: 'pip'}) pip install -U chainer - +qs({config: 'gpu', package: 'source'}) pip install -U chainer - +qs({config: 'gpu', package: 'conda'}) conda install -c anaconda chainer + +qs({config: 'gpu', os: 'mac'}) export PATH=$PATH:/usr/local/cuda-8.0/bin + +qs({config: 'gpu', os: 'linux'}) export PATH=$PATH:/usr/local/cuda-8.0/bin +qs({package: 'pip'}) pip install -U spacy +qs({package: 'conda'}) conda install -c conda-forge spacy From 809903dcad447c234531e3c0d0abd5e71f299347 Mon Sep 17 00:00:00 2001 From: ines Date: Sun, 4 Jun 2017 13:29:20 +0200 Subject: [PATCH 7/7] Fix link and update wording --- website/docs/usage/spacy-101.jade | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/usage/spacy-101.jade b/website/docs/usage/spacy-101.jade index 67d3a83fe..5b7908651 100644 --- a/website/docs/usage/spacy-101.jade +++ b/website/docs/usage/spacy-101.jade @@ -68,11 +68,11 @@ p | It's is built on the latest research, but it's designed to get | things done. This leads to fairly different design decisions than | #[+a("https://github./nltk/nltk") NLTK] - | or #[+a("https://stanfordnlp.github.io/CorenlP") CoreNLP], which were - | created as platforms for teaching and research. The main difference - | is that spaCy is integrated and opinionated. We try to avoid asking + | or #[+a("https://stanfordnlp.github.io/CoreNLP/") CoreNLP], which were + | created as platforms for teaching and research. The main difference + | is that spaCy is integrated and opinionated. spaCy tries to avoid asking | the user to choose between multiple algorithms that deliver equivalent - | functionality. Keeping our menu small lets us deliver generally better + | functionality. Keeping the menu small lets spaCy deliver generally better | performance and developer experience. +item #[strong spaCy is not a company]. | It's an open-source library. Our company publishing spaCy and other