* Add emoticons to tokenization

2024-12-25 17:36:30 +03:00 · 2014-11-01 15:14:46 +11:00 · 2014-11-01 15:14:46 +11:00 · 11e42fd070
commit 11e42fd070
parent 39743323ea
1 changed files with 56 additions and 0 deletions
--- a/data/en/tokenization
+++ b/data/en/tokenization
@ -100,3 +100,59 @@ U.N.    U.N.
 Ms. Ms.
 Mr. Mr.
 P.  P.
+:)  :)
+<3  <3
+;)  ;)
+(:  (:
+:(  :(
+-_- -_-
+♥ ♥
+=)  =)
+:/  :/
+:>  :>
+;-) ;-)
+:Y  :Y
+:P  :P
+:-P :-P
+:3  :3
+=3  =3
+xD  xD
+^_^ ^_^
+=]  =]
+=D  =D
+<333    <333
+:)) :))
+:0  :0
+-__-    -__-
+♥♥♥ ♥♥♥
+♥♡♥ ♥♡♥
+ಠ◡ಠ ಠ◡ಠ
+⌣́_⌣̀)    ⌣́_⌣̀)
+☺   ☺
+☹ ☹
+¯\_(ツ)_/¯  ¯\_(ツ)_/¯
+xDD xDD
+o_o o_o
+o_O o_O
+\m/ \m/
+V_V V_V
+=[[ =[[
+<33 <33
+;p  ;p
+;D  ;D
+;-p ;-p
+;(  ;(
+:p  :p
+:]  :]
+:O  :O
+:-\ :-\
+:-/ :-/
+:-) :-)
+:(((    :(((
+:(( :((
+:') :')
+(^_^)   (^_^)
+(=  (=
+o.O o.O
+")  ")
+....    ....