explosion/spaCy
1
1
Fork 0
mirror of https://github.com/explosion/spaCy.git synced 2024-09-21 11:29:13 +03:00
Code Issues Packages Projects Releases Wiki Activity

* Revise tokenization rules to match PTB. Rules are pretty messy around periods, need better support for these.

Browse Source
This commit is contained in:
Matthew Honnibal 2014-12-07 22:04:47 +11:00
parent 5fe5e6e66b
commit 677e111ee7
2 changed files with 109 additions and 78 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
data/en/prefix
View File

@ -11,3 +11,8 @@ $
'
``
`
#
US$
C$
A$
a-

182
data/en/tokenization
View File

@ -6,99 +6,100 @@
's 's
'S 'S
ain't are not
aren't are not
can't can not
ain't ai n't
aren't are n't
can't ca n't
cannot can not
could've could have
couldn't could not
couldn't've could not have
didn't did not
doesn't does not
don't do not
hadn't had not
hadn't've had not have
hasn't has not
haven't have not
he'd he would
he'd've he would have
he'll he will
could've could 've
couldn't could n't
couldn't've could n't 've
didn't did n't
doesn't does n't
don't do n't
hadn't had n't
hadn't've had n't 've
hasn't has n't
haven't have n't
he'd he 'd
he'd've he 'd 've
he'll he 'll
he's he 's
how'd he would
how'll he will
how'd he 'd
how'll he 'll
how's how 's
I'd I would
I'd've I would have
I'll I will
I'm I am
I'ma I will
I've I have
isn't is not
it'd it would
it'd've it would have
it'll it will
I'd I 'd
I'd've I 'd 've
I'll I 'll
I'm I 'm
I'ma I 'ma
I've I 've
isn't is n't
it'd it 'd
it'd've it 'd 've
it'll it 'll
it's it 's
let's let 's
mightn't might not
mightn't've might not have
might've might have
mustn't must not
must've must have
needn't need not
not've not have
shan't shall not
she'd she would
she'd've she would have
mightn't might n't
mightn't've might n't 've
might've might 've
mustn't must n't
must've must 've
needn't need n't
not've not 've
shan't sha n't
she'd she 'd
she'd've she 'd 've
she'll she will
she's she 's
should've should have
shouldn't should not
shouldn't've should not have
should've should 've
shouldn't should n't
shouldn't've should n't 've
that's that 's
there'd there would
there'd've there would have
there's there is
they'd there would
they'd've they would have
they'll they will
they're they are
they've they have
wasn't was not
we'd we would
we'd've we would have
we'll we will
we're we are
we've we have
weren't were not
what'll what will
what're what are
there'd there 'd
there'd've there 'd 've
there's there 's
they'd there 'd
they'd've they 'd 've
they'll they 'll
they're they 're
they've they 've
wasn't was n't
we'd we 'd
we'd've we 'd 've
we'll we 'll
we're we 're
we've we 've
weren't were n't
what'll what 'll
what're what 're
what's what 's
what've what have
what've what 've
when's when 's
where'd where would
where'd where 'd
where's where 's
where've where have
who'd who would
who'll who will
who're who are
where've where 've
who'd who 'd
who'll who 'll
who're who 're
who's who 's
who've who have
why'll who will
why're why are
who've who 've
why'll why 'll
why're why 're
why's why 's
won't will not
would've would have
wouldn't would not
wouldn't've would not have
you'd you would
you'd've you would have
you'll you will
you're you are
you've you have
'em them
'ol old
won't wo n't
would've would 've
wouldn't would n't
wouldn't've would n't 've
you'd you 'd
you'd've you 'd 've
you'll you 'll
you're you 're
you've you 've
'em 'em
'ol 'ol
10km 10 km
U.S. U.S.
U.K. U.K.
non-U.S. non-U.S.
U.N. U.N.
Co. Co.
@ -115,7 +116,12 @@ A.G. A.G.
Rep. Rep.
Ms. Ms.
Mr. Mr.
Mrs. Mrs.
a.m. a.m.
Sen. Sen.
INC. INC.
CO. CO.
COS. COS.
p.m. p.m.
Nos. Nos.
a.k.a. a.k.a.
@ -127,6 +133,7 @@ E. E.
F. F.
G. G.
H. H.
I. I.
J. J.
K. K.
L. L.
@ -205,6 +212,9 @@ Wash. Wash.
W.Va. W.Va.
Wis. Wis.
Wyo. Wyo.
L.A. L.A.
R.H. R.H.
Gov. Gov.
'' ''
:) :)
<3 <3
@ -262,3 +272,19 @@ V_V V_V
o.O o.O
") ")
.... ....
a- a -
Messrs. Messrs.
No. No.
vs. vs.
Gen. Gen.
Cos. Cos.
L.J. L.J.
D.T. D.T.
Prof. Prof.
Bros. Bros.
J.C. J.C.
Neb. Neb.
Adm. Adm.
U.S.S.R. U.S.S.R.
Rev. Rev.
H.F. H.F.