Lexeme Features
===============

A lexeme is an entry in the lexicon --- the vocabulary --- for a word, punctuation
symbol, whitespace unit, etc.  Lexemes come with lots of pre-computed information,
that help you write good feature functions.  Features are integer-valued where
possible --- instead of strings, spaCy refers to strings by consecutive ID numbers,
which you can use to look up the string values if necessary.

String features
---------------

+---------+-------------------------------------------------------------------+
| SIC     | The word as it appeared in the sentence, unaltered.               |
+---------+-------------------------------------------------------------------+
| NORM    | For frequent words, case normalization is applied.                |
|         | Otherwise, back-off to SHAPE.                                     |
+---------+-------------------------------------------------------------------+
| SHAPE   | Remap the characters of the word as follows:                      |
|         |                                                                   |
|         | a-z --> x, A-Z --> X, 0-9 --> d, ,.;:"'?!$- --> self, other --> \*|
|         |                                                                   |
|         | Trim sequences of length 3+ to 3, e.g                             |
|         |                                                                   |
|         | apples --> xxx, Apples --> Xxxx, app9LES@ --> xxx9XXX*            |
+---------+-------------------------------------------------------------------+
| ASCIIED | Use unidecode.unidecode(sic) to approximate the word using the    |
|         | ascii characters.                                                 |
+---------+-------------------------------------------------------------------+
| PREFIX  | sic_unicode_string[:1]                                            |
+---------+-------------------------------------------------------------------+
| SUFFIX  | sic_unicode_string[-3:]                                           |
+---------+-------------------------------------------------------------------+


Integer features
----------------

+--------------+--------------------------------------------------------------+
| LENGTH       |  Length of the string, in unicode                            |
+--------------+--------------------------------------------------------------+
| CLUSTER      | Brown cluster                                                |
+--------------+--------------------------------------------------------------+
| POS_TYPE     | K-means cluster of word's tag affinities                     |
+--------------+--------------------------------------------------------------+
| SENSE_TYPE   | K-means cluster of word's sense affinities                   |
+--------------+--------------------------------------------------------------+

Boolean features
----------------

+-------------+--------------------------------------------------------------+
| IS_ALPHA    | The result of sic.isalpha()                                  |
+-------------+--------------------------------------------------------------+
| IS_ASCII    | Check whether all the word's characters are ascii characters |
+-------------+--------------------------------------------------------------+
| IS_DIGIT    | The result of sic.isdigit()                                  |
+-------------+--------------------------------------------------------------+
| IS_LOWER    | The result of sic.islower()                                  |
+-------------+--------------------------------------------------------------+
| IS_PUNCT    | Check whether all characters are in the class TODO           |
+-------------+--------------------------------------------------------------+
| IS_SPACE    | The result of sic.isspace()                                  |
+-------------+--------------------------------------------------------------+
| IS_TITLE    | The result of sic.istitle()                                  |
+-------------+--------------------------------------------------------------+
| IS_UPPER    | The result of sic.isupper()                                  |
+-------------+--------------------------------------------------------------+
| LIKE_URL    | Check whether the string looks like it could be a URL.  Aims |
|             | for low false negative rate.                                 |
+-------------+--------------------------------------------------------------+
| LIKE_NUMBER | Check whether the string looks like it could be a numeric    |
|             | entity, e.g. 10,000 10th .10 . Skews for low false negative  |
|             | rate.                                                        |
+-------------+--------------------------------------------------------------+
| IN_LIST     | Facility for loading arbitrary run-time word lists?          |
+-------------+--------------------------------------------------------------+