explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-12-25 01:16:28 +03:00
Code Issues Packages Projects Releases Wiki Activity

* Upd ptb tokenization rules

Browse Source
This commit is contained in:
Matthew Honnibal 2014-07-07 05:09:22 +02:00
parent dc20500920
commit 997551241f
1 changed files with 88 additions and 77 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

165
data/en_ptb/tokenization
View File

@ -4,90 +4,101 @@
#*---* --- #*---* ---
#*'s 's #*'s 's
ain't are not cannot can not
aren't are not d'ye d' ye
can't can not gimme gim me
could've could have gonna gon na
couldn't could not lemme lem me
couldn't've could not have more'n more 'n
didn't did not 'tis 't is
doesn't does not 'twas 't was
don't do not wanna wan na
hadn't had not whaddya wha dd ya
hadn't've had not have whatcha wha t cha
hasn't has not ain't ai n't
haven't have not aren't are n't
he'd he would can't can n't
he'd've he would have could've could 've
he'll he will couldn't could n't
couldn't've could n't 've
didn't did n't
doesn't does n't
don't do n't
hadn't had n't
hadn't've had n't 've
hasn't has n't
haven't have n't
he'd he 'd
he'd've he 'd 've
he'll he 'll
he's he 's he's he 's
how'd he would how'd he 'd
how'll he will how'll he 'll
how's how 's how's how 's
I'd I would I'd I 'd
I'd've I would have I'd've I 'd 've
I'll I will I'll I 'll
I'm I am I'm I 'm
I've I have I've I 've
isn't is not isn't is n't
it'd it would it'd it 'd
it'd've it would have it'd've it 'd 've
it'll it will it'll it 'll
it's it 's it's it 's
let's let 's let's let 's
mightn't might not mightn't might n't
mightn't've might not have mightn't've might n't 've
might've might have might've might 've
mustn't must not mustn't must n't
must've must have must've must 've
needn't need not needn't need n't
not've not have not've not h've
shan't shall not shan't sha n't
she'd she would she'd she 'd
she'd've she would have she'd've she 'd 've
she'll she will she'll she 'll
she's she 's she's she 's
should've should have should've should 've
shouldn't should not shouldn't should n't
shouldn't've should not have shouldn't've should n't 've
that's that 's that's that 's
there'd there would there'd there 'd
there'd've there would have there'd've there 'd 've
there's there is there's there 's
they'd there would they'd there 'd
they'd've they would have they'd've they 'd 've
they'll they will they'll they 'll
they're they are they're they 're
they've they have they've they 've
wasn't was not wasn't was n't
we'd we would we'd we 'd
we'd've we would have we'd've we 'd h've
we'll we will we'll we 'll
we're we are we're we 're
we've we have we've we h've
weren't were not weren't were n't
what'll what will what'll what 'll
what're what are what're what 're
what's what 's what's what 's
what've what have what've what 've
when's when 's when's when 's
where'd where would where'd where 'd
where's where 's where's where 's
where've where have where've where 've
who'd who would who'd who 'd
who'll who will who'll who 'll
who're who are who're who 're
who's who 's who's who 's
who've who have who've who 've
why'll who will why'll why 'll
why're why are why're why 're
why's why is why's why 's
won't will not won't will n't
would've would have would've would 've
wouldn't would not wouldn't would n't
wouldn't've would not have wouldn't've would n't 've
you'd you would you'd you 'd
you'd've you would have you'd've you 'd 've
you'll you will you'll you 'll
you're you are you're you 're
you've you have you've you 've