* Upd ptb tokenization rules

data/en_ptb/tokenization

@@ -4,90 +4,101 @@
 #*---* ---
 #*'s  's
 
-ain't   are not
-aren't  are not
-can't   can not
-could've    could have
-couldn't    could not
-couldn't've could not have
-didn't  did not
-doesn't does not
-don't   do not
-hadn't  had not
-hadn't've   had not have
-hasn't  has not
-haven't have not
-he'd    he would
-he'd've he would have
-he'll   he will
+cannot  can not
+d'ye    d' ye
+gimme   gim me
+gonna   gon na
+lemme   lem me
+more'n  more 'n
+'tis    't is
+'twas   't was
+wanna   wan na
+whaddya wha dd ya
+whatcha wha t cha
+ain't   ai n't
+aren't  are n't
+can't   can n't
+could've    could 've
+couldn't    could n't
+couldn't've could n't 've
+didn't  did n't
+doesn't does n't
+don't   do n't
+hadn't  had n't
+hadn't've   had n't 've
+hasn't  has n't
+haven't have n't
+he'd    he 'd
+he'd've he 'd 've
+he'll   he 'll
 he's    he 's
-how'd   he would
-how'll  he will
+how'd   he 'd
+how'll  he 'll
 how's   how 's
-I'd I would
-I'd've  I would have
-I'll    I will
-I'm I am
-I've    I have
-isn't   is not
-it'd    it would
-it'd've it would have
-it'll   it will
+I'd I 'd
+I'd've  I 'd 've
+I'll    I 'll
+I'm I 'm
+I've    I 've
+isn't   is n't
+it'd    it 'd
+it'd've it 'd 've
+it'll   it 'll
 it's    it 's
 let's   let 's
-mightn't    might not
-mightn't've might not have
-might've    might have
-mustn't must not
-must've must have
-needn't need not
-not've  not have
-shan't  shall not
-she'd   she would
-she'd've    she would have
-she'll  she will
+mightn't    might n't
+mightn't've might n't 've
+might've    might 've
+mustn't must n't
+must've must 've
+needn't need n't
+not've  not h've
+shan't  sha n't
+she'd   she 'd
+she'd've    she 'd 've
+she'll  she 'll
 she's   she 's
-should've   should have
-shouldn't   should not
-shouldn't've    should not have
+should've   should 've
+shouldn't   should n't
+shouldn't've    should n't 've
 that's  that 's
-there'd there would
-there'd've  there would have
-there's there is
-they'd  there would
-they'd've   they would have
-they'll they will
-they're they are
-they've they have
-wasn't  was not
-we'd    we would
-we'd've we would have
-we'll   we will
-we're   we are
-we've   we have
-weren't were not
-what'll what will
-what're what are
+there'd there 'd
+there'd've  there 'd 've
+there's there 's
+they'd  there 'd
+they'd've   they 'd 've
+they'll they 'll
+they're they 're
+they've they 've
+wasn't  was n't
+we'd    we 'd
+we'd've we 'd h've
+we'll   we 'll
+we're   we 're
+we've   we h've
+weren't were n't
+what'll what 'll
+what're what 're
 what's  what 's
-what've what have
+what've what 've
 when's  when 's
-where'd where would
+where'd where 'd
 where's where 's
-where've    where have
-who'd   who would
-who'll  who will
-who're  who are
+where've    where 've
+who'd   who 'd
+who'll  who 'll
+who're  who 're
 who's   who 's
-who've  who have
-why'll  who will
-why're  why are
-why's   why is
-won't   will not
-would've    would have
-wouldn't    would not
-wouldn't've would not have
-you'd   you would
-you'd've    you would have
-you'll  you will
-you're  you are
-you've  you have
+who've  who 've
+why'll  why 'll
+why're  why 're
+why's   why 's
+won't   will n't
+would've    would 've
+wouldn't    would n't
+wouldn't've would n't 've
+you'd   you 'd
+you'd've    you 'd 've
+you'll  you 'll
+you're  you 're
+you've  you 've