Matthew Honnibal
|
8b5de4a411
|
* Add word / tag / label sets, for use in neural net
|
2015-06-28 05:46:53 +02:00 |
|
Matthew Honnibal
|
cfcbd8d256
|
* Fix punctuation eval in scorer.py
|
2015-06-28 01:31:39 +02:00 |
|
Matthew Honnibal
|
ed40a8380e
|
* Remove hard-coding of vector lengths
|
2015-06-27 04:18:47 +02:00 |
|
Matthew Honnibal
|
ebe630cc8d
|
* Enable more features for NN
|
2015-06-27 04:17:29 +02:00 |
|
Matthew Honnibal
|
f8bb43475e
|
* Bridge to Theano working. Very disorganised. Using thinc adb60aba966ed2
|
2015-06-27 02:39:18 +02:00 |
|
Matthew Honnibal
|
2fe98b8a9a
|
* Prepare for new models to be plugged in by using Example class
|
2015-06-26 13:51:39 +02:00 |
|
Matthew Honnibal
|
6896455884
|
* Rejig parser interface to use new thinc.api.Example class, in prep of theano model. Comment out beam search
|
2015-06-26 06:25:36 +02:00 |
|
Matthew Honnibal
|
b266a63f2c
|
* Inc version of downloadble data
|
2015-06-24 04:53:08 +02:00 |
|
Matthew Honnibal
|
02b171ee67
|
* Bug fixes to edge calculation
|
2015-06-24 04:28:02 +02:00 |
|
Matthew Honnibal
|
a4e9bdf4c1
|
* Work on a theano-driven model for the parser
|
2015-06-24 01:02:40 +02:00 |
|
Matthew Honnibal
|
7f9384f53c
|
* Remove deprecated _state module
|
2015-06-23 17:28:24 +02:00 |
|
Matthew Honnibal
|
6dbe182491
|
* Fix merge conflicts
|
2015-06-23 17:28:00 +02:00 |
|
Matthew Honnibal
|
579735a095
|
* Remove import of _state module
|
2015-06-23 17:25:08 +02:00 |
|
Matthew Honnibal
|
88f55d136b
|
* Remove deprecated _state module
|
2015-06-23 17:19:51 +02:00 |
|
Matthew Honnibal
|
9ab9dd2bf7
|
* Clean up unused orig_arc_eager and tree_arc_eager modules, which were only added for EMNLP experiments
|
2015-06-23 17:17:33 +02:00 |
|
Matthew Honnibal
|
7ebfe4b983
|
* Fixes to edge features
|
2015-06-23 16:32:54 +02:00 |
|
Matthew Honnibal
|
7b125f5a86
|
* Fixes to edge features
|
2015-06-23 16:31:01 +02:00 |
|
Matthew Honnibal
|
8d4bbacfc5
|
* Fix edge navigation in Token objects
|
2015-06-23 16:07:34 +02:00 |
|
Matthew Honnibal
|
35c290bee4
|
* Fix edge features
|
2015-06-23 15:50:56 +02:00 |
|
Matthew Honnibal
|
221e2e485f
|
* Assign 'ROOT' as label, not 'root'
|
2015-06-23 15:09:54 +02:00 |
|
Matthew Honnibal
|
a7bf7b0626
|
* Rename sent_start to sent_end, to reflect its new usage in the Break transition
|
2015-06-23 05:39:43 +02:00 |
|
Matthew Honnibal
|
ee3e56f27b
|
* Fix bounds checking on entities
|
2015-06-23 04:35:08 +02:00 |
|
Matthew Honnibal
|
43ef5ddea5
|
* Ensure root albel is spelled ROOT, for backwards compatibility
|
2015-06-23 04:14:03 +02:00 |
|
Matthew Honnibal
|
065c2e1d2d
|
* Add some bounds checking around state arrays
|
2015-06-23 04:13:09 +02:00 |
|
Matthew Honnibal
|
89ae218b75
|
* Add import to tokens.pyx from weird Cython compiler issue with casting from memory views
|
2015-06-23 03:04:34 +02:00 |
|
Matthew Honnibal
|
f01b3d043e
|
* Add padding to arrays in stateclass. May be papering over a deeper bug.
|
2015-06-23 03:03:41 +02:00 |
|
Matthew Honnibal
|
5e94b5d581
|
* Have Tokens return proper numpy arrays, not Cython views.
|
2015-06-23 00:07:34 +02:00 |
|
Matthew Honnibal
|
69507bc729
|
* Re-enable Break transition in arc_eager.pyx
|
2015-06-23 00:03:30 +02:00 |
|
Matthew Honnibal
|
cc579ed429
|
* Add __len__ function to StringStore
|
2015-06-23 00:02:50 +02:00 |
|
Matthew Honnibal
|
46fb24e9fd
|
* Add cycle-checking code in gold.pyx
|
2015-06-23 00:02:22 +02:00 |
|
Matthew Honnibal
|
60d26243e3
|
* Fix head alignment in read_conll.parse, which was causing corrupt parses when strip_bad_periods=True. A similar problem may apply to other data readers.
|
2015-06-18 16:35:27 +02:00 |
|
Matthew Honnibal
|
f868175e43
|
* Whitespace
|
2015-06-16 23:37:46 +02:00 |
|
Matthew Honnibal
|
ab110be125
|
* Remove debugging in parser.pyx
|
2015-06-16 23:37:25 +02:00 |
|
Matthew Honnibal
|
9b13d11ab3
|
* Fix handling of entities in StateClass
|
2015-06-16 23:35:21 +02:00 |
|
Matthew Honnibal
|
c40a2c661c
|
* Add tree_arc_eager
|
2015-06-15 08:23:24 +02:00 |
|
Matthew Honnibal
|
5da5cf7084
|
* Add some more features for S1/S0
|
2015-06-15 04:07:13 +02:00 |
|
Matthew Honnibal
|
8156a01bca
|
* Fix root label for orig_arc_eager
|
2015-06-15 02:54:55 +02:00 |
|
Matthew Honnibal
|
21930ede15
|
* Switch toggle on USE_ROOT_ARC_SEGMENT
|
2015-06-15 02:54:32 +02:00 |
|
Matthew Honnibal
|
38a6afa484
|
* Make possibly dubious correction to the unshift oracle
|
2015-06-15 02:50:00 +02:00 |
|
Matthew Honnibal
|
f66228f253
|
* Add some more features, esp for labels
|
2015-06-14 21:18:02 +02:00 |
|
Matthew Honnibal
|
3da8e0f317
|
* Add orig_arc_eager
|
2015-06-14 20:31:44 +02:00 |
|
Matthew Honnibal
|
ea8a103007
|
* Fix import of TransitionSystem in parser.pyx
|
2015-06-14 19:01:26 +02:00 |
|
Matthew Honnibal
|
e0984ca139
|
* Fix valency features in StateClass
|
2015-06-14 17:50:26 +02:00 |
|
Matthew Honnibal
|
e50ac1a47f
|
* Add verbose printing to scorer
|
2015-06-14 17:45:50 +02:00 |
|
Matthew Honnibal
|
763cbd23d5
|
* Upd stateclass.print_state
|
2015-06-14 17:44:29 +02:00 |
|
Matthew Honnibal
|
bdd07bf000
|
* Fix Break oracle, but disable the Break transition for now, while we finalize the gold-standard experiments
|
2015-06-14 17:44:03 +02:00 |
|
Matthew Honnibal
|
399f15fbdf
|
* Add flag to toggle handling of multi-root inputs without the Break transition. Clear up now unused best_valid stuff.
|
2015-06-14 00:28:37 +02:00 |
|
Matthew Honnibal
|
75289b4761
|
* Don't refuse to parse single token sentences, incase some transition system needs them, e.g. single word entity. Instead fix error in _init_state.
|
2015-06-13 22:55:55 +02:00 |
|
Matthew Honnibal
|
77d7e79c7e
|
* Fix r/l and distance features.
|
2015-06-12 13:06:15 +02:00 |
|
Matthew Honnibal
|
b643cb3d5c
|
* Allow training documents to be filtered in gold.pyx
|
2015-06-12 02:42:08 +02:00 |
|
Matthew Honnibal
|
15e177d7a1
|
* Fixes to unshift/fast-forward strategy. Getting 91.55 greedy on NW dev, gold preproc
|
2015-06-12 01:50:23 +02:00 |
|
Matthew Honnibal
|
afd77a529b
|
* Prepare for break transition, with fast-forwarding. 86.5 on 1k nw gold preproc
|
2015-06-10 14:08:30 +02:00 |
|
Matthew Honnibal
|
495f528709
|
* Add support for sentence breaks in stateclass
|
2015-06-10 12:34:28 +02:00 |
|
Matthew Honnibal
|
b7b18c279d
|
* Fix Reduce oracle. Getting 86.35
|
2015-06-10 11:33:39 +02:00 |
|
Matthew Honnibal
|
bb09b5d91a
|
* Fix shifted bit vector in stateclass --- should reflect whether the word has been *unshifted*.
|
2015-06-10 11:33:09 +02:00 |
|
Matthew Honnibal
|
aa9625f688
|
* Do non-monotonic Unshift. Every word can be shifted at most 1 time. When the Reduce move is used, if S0 has no head, we put the word back on the buffer. Gets 86.4 on nw 1k with gold pre-proc. Break transition not yet implemented for this.
|
2015-06-10 10:15:56 +02:00 |
|
Matthew Honnibal
|
7bf6b7de3e
|
* Add unshift action to StateClass, and track which moves have been shifted
|
2015-06-10 10:13:03 +02:00 |
|
Matthew Honnibal
|
f7c8069e65
|
* Fix bug in distance feature
|
2015-06-10 10:12:17 +02:00 |
|
Matthew Honnibal
|
abd07c067a
|
* Inline B and S methods on stateclass
|
2015-06-10 07:22:33 +02:00 |
|
Matthew Honnibal
|
e2f9a80713
|
* Remove old _state imports
|
2015-06-10 07:09:17 +02:00 |
|
Matthew Honnibal
|
e9aaecc619
|
* Remove from_struct method from StateClass
|
2015-06-10 06:58:27 +02:00 |
|
Matthew Honnibal
|
18cc326dc0
|
* Bug fixes to ner.pyx
|
2015-06-10 06:57:41 +02:00 |
|
Matthew Honnibal
|
e5570c9700
|
* Set nogil for oracle functions
|
2015-06-10 06:56:56 +02:00 |
|
Matthew Honnibal
|
4575e7a60f
|
* Fix beam search with new StateClass
|
2015-06-10 06:33:39 +02:00 |
|
Matthew Honnibal
|
04b1cd9b8c
|
* Greedy parsing working with new StateClass. Beam parsing broken
|
2015-06-10 04:20:23 +02:00 |
|
Matthew Honnibal
|
6a94b64eca
|
* Remove State* from parser.pyx entirely, switching over to StateClass. Beam parsing still untested.
|
2015-06-10 02:03:38 +02:00 |
|
Matthew Honnibal
|
f14a1526aa
|
* Remove version of fill_context that takes State*
|
2015-06-10 01:39:07 +02:00 |
|
Matthew Honnibal
|
d68c686ec1
|
* Move StateClass into interface of transition functions
|
2015-06-10 01:35:28 +02:00 |
|
Matthew Honnibal
|
4b98b3e9c8
|
* Cost functions now take StateClass argument, instead of State*.
|
2015-06-10 00:40:43 +02:00 |
|
Matthew Honnibal
|
e0cf61f591
|
* Move StateClass into the interface for is_valid
|
2015-06-09 23:23:28 +02:00 |
|
Matthew Honnibal
|
0895d454fb
|
* Prepare to switch to using state class, instead of state struct
|
2015-06-09 21:20:14 +02:00 |
|
Matthew Honnibal
|
2b9629ed62
|
* Begin adding stateclass to ArcEager
|
2015-06-09 01:41:09 +02:00 |
|
Matthew Honnibal
|
ba10fd8af5
|
* Add StateClass, to replace/refactor the mess in _state
|
2015-06-09 01:39:54 +02:00 |
|
Matthew Honnibal
|
c7e3dfc1dc
|
* Don't automatically push words when stack is empty, as it messes up beam parsing. Add hash method to beam state.
|
2015-06-08 14:49:04 +02:00 |
|
Matthew Honnibal
|
00a0dfcb59
|
* Avoid shipping the spacy.munge package
|
2015-06-08 00:54:13 +02:00 |
|
Matthew Honnibal
|
7d265a9c62
|
* Revert to wget in spacy.en.download
|
2015-06-08 00:48:56 +02:00 |
|
Matthew Honnibal
|
a8fc5f1285
|
* Fix munge/read_ner
|
2015-06-08 00:35:04 +02:00 |
|
Matthew Honnibal
|
1515862861
|
* Fix download.py
|
2015-06-08 00:08:05 +02:00 |
|
Matthew Honnibal
|
7e9e8f654a
|
* Use urllib in spacy.en.download
|
2015-06-07 23:51:38 +02:00 |
|
Matthew Honnibal
|
80cff41a9c
|
* Upd download.py
|
2015-06-07 19:13:28 +02:00 |
|
Matthew Honnibal
|
6e2564239d
|
* Bug fixes to beam parser. Search still broken on non-gold sentences
|
2015-06-07 19:12:59 +02:00 |
|
Matthew Honnibal
|
1ec4e6fc95
|
* Don't score whitespace tokens
|
2015-06-07 19:10:32 +02:00 |
|
Matthew Honnibal
|
731e5f1e46
|
* Add get() function in spacy/syntax/Config
|
2015-06-07 19:09:15 +02:00 |
|
Matthew Honnibal
|
8f142c1838
|
* Refactor transition system oracles, to split out move and label cost. Preparing to add Unshift move. Will exclude non-monotonic.
|
2015-06-07 03:21:29 +02:00 |
|
Matthew Honnibal
|
89b8775887
|
* Fix output from _min_edit_path when inputs match.
|
2015-06-06 05:58:53 +02:00 |
|
Matthew Honnibal
|
98cfd84123
|
* Remove hyphenation from main tokenizer loop: do it in infix.txt instead. This lets emoticons work
|
2015-06-06 05:57:03 +02:00 |
|
Matthew Honnibal
|
1fee7ade61
|
* Tweak to ner
|
2015-06-05 23:48:43 +02:00 |
|
Matthew Honnibal
|
33e70b167f
|
* Remove dead code from ner.pyx
|
2015-06-05 17:12:47 +02:00 |
|
Matthew Honnibal
|
88ac5c6e98
|
* Send beam_width < 0 to greedy parser
|
2015-06-05 17:12:06 +02:00 |
|
Matthew Honnibal
|
0114e7600d
|
* Fix NER oracle
|
2015-06-05 17:11:26 +02:00 |
|
Matthew Honnibal
|
c04e6ebca6
|
* Allow user to load different sized vectors.
|
2015-06-05 16:26:39 +02:00 |
|
Matthew Honnibal
|
6bf35cecc3
|
* Refactor transition system to use classes with staticmethods.
|
2015-06-05 02:27:17 +02:00 |
|
Matthew Honnibal
|
36a34d544b
|
* Refactoring arc_eager, grouping oracle functions into transitions
|
2015-06-04 22:43:03 +02:00 |
|
Matthew Honnibal
|
4433396005
|
* Impove efficiency of dynamic oracle, making beam training faster
|
2015-06-04 21:15:14 +02:00 |
|
Matthew Honnibal
|
079dad28a7
|
* Update for faster beam training
|
2015-06-04 19:32:32 +02:00 |
|
Matthew Honnibal
|
f8843906ad
|
Merge branch 'constituency'
Add beam parsing and training from JSON files, with Levenshtein alignment.
|
2015-06-03 06:07:24 +02:00 |
|
Matthew Honnibal
|
ae653b850a
|
* Remove unused import from gold.pyx
|
2015-06-03 06:07:15 +02:00 |
|
Matthew Honnibal
|
a2627b6102
|
* Fix bug in refactored init_transition
|
2015-06-03 06:01:26 +02:00 |
|
Matthew Honnibal
|
dd0867645d
|
* Remove stray const from State header
|
2015-06-03 00:10:04 +02:00 |
|
Matthew Honnibal
|
6c47b10a6e
|
* Make optimization to children_in_buffer: stop searching when we would cross a bracket.
|
2015-06-02 21:05:24 +02:00 |
|
Matthew Honnibal
|
a513ec500f
|
* Have oracle functions take a struct instead of a Python object
|
2015-06-02 20:01:06 +02:00 |
|
Matthew Honnibal
|
d1b55310a1
|
* Refactor _advance_beam function
|
2015-06-02 18:38:41 +02:00 |
|
Matthew Honnibal
|
0786d9b3c7
|
* Refactor TransitionSystem, adding set_valid method
|
2015-06-02 18:38:07 +02:00 |
|
Matthew Honnibal
|
bd82a49994
|
* Add set_scores method to Model
|
2015-06-02 18:37:10 +02:00 |
|
Matthew Honnibal
|
a3964957f6
|
* Add profiling for _state.pyx
|
2015-06-02 18:36:27 +02:00 |
|
Matthew Honnibal
|
e822df0867
|
* Fix bugs in new greedy/beam parser
|
2015-06-02 02:01:33 +02:00 |
|
Matthew Honnibal
|
66dfa95847
|
* Revise greedy_parse/beam_parse ownership goof
|
2015-06-02 01:34:19 +02:00 |
|
Matthew Honnibal
|
75658b2ed3
|
* Remove use of new beam.loss property, to maintain compatibility with older versions of thinc for now.
|
2015-06-02 00:57:09 +02:00 |
|
Matthew Honnibal
|
7c29362d60
|
* Rename parser class in parser.pxd, now that beam parsing is supported
|
2015-06-02 00:53:49 +02:00 |
|
Matthew Honnibal
|
58d5ac0944
|
* Add beam search capabilities to Parser. Rename GreedyParser to Parser.
|
2015-06-02 00:28:02 +02:00 |
|
Matthew Honnibal
|
62424e6c76
|
* Remove unused regularize argument from _ml.Model
|
2015-06-02 00:27:07 +02:00 |
|
Matthew Honnibal
|
adeb57cb1e
|
* Fix long line
|
2015-06-01 23:07:00 +02:00 |
|
Matthew Honnibal
|
e09a08bd00
|
* Add copy_state function
|
2015-06-01 23:06:30 +02:00 |
|
Matthew Honnibal
|
c7876aa8b6
|
* Add get_valid method
|
2015-06-01 23:06:00 +02:00 |
|
Matthew Honnibal
|
d82f9d958d
|
* Remove regularization cruft from _ml, move score from .pxd file to .pyx
|
2015-05-31 18:48:05 +02:00 |
|
Matthew Honnibal
|
5e99ff94c8
|
* Edits to arc eager oracle. Couldn't figure out how the non-monotonic lines made sense. They seem covered by children_in_stack
|
2015-05-31 15:14:37 +02:00 |
|
Matthew Honnibal
|
6c5632b71c
|
* Roll back proposed change to Break transition while investigate effect
|
2015-05-31 06:49:52 +02:00 |
|
Matthew Honnibal
|
6bba793df3
|
* Disable the Zipf-reweighting thing while investigate effect
|
2015-05-31 06:48:43 +02:00 |
|
Matthew Honnibal
|
e77940565d
|
* Add length cap to distance feature
|
2015-05-31 05:25:30 +02:00 |
|
Matthew Honnibal
|
fd596351ba
|
* Fix valency features
|
2015-05-31 05:24:33 +02:00 |
|
Matthew Honnibal
|
87d6551d19
|
* Allow gold parse to cut non-projective arcs
|
2015-05-31 01:11:56 +02:00 |
|
Matthew Honnibal
|
c4f0914b4e
|
* Fix POS tag evaluation in scorer.py: do evaluate punctuation tags
|
2015-05-30 18:24:32 +02:00 |
|
Matthew Honnibal
|
9e39a206da
|
* Fix efficiency of JSON reading, by using ujson instead of stream
|
2015-05-30 17:54:52 +02:00 |
|
Matthew Honnibal
|
76300bbb1b
|
* Use updated JSON format, with sentences below paragraphs. Allows use of gold preprocessing flag.
|
2015-05-30 01:25:46 +02:00 |
|
Matthew Honnibal
|
b76bbbd12c
|
* Read json files recursively from a directory, instead of requiring a single .json file
|
2015-05-29 03:52:55 +02:00 |
|
Matthew Honnibal
|
8f31d3b864
|
* Relax constraint on Break transition for non-monotonic parsing.
|
2015-05-28 23:39:52 +02:00 |
|
Matthew Honnibal
|
6b2e5c4b8a
|
* Avoid NER scoring for sentences with some missing NER values.
|
2015-05-28 22:39:08 +02:00 |
|
Matthew Honnibal
|
d25d31442d
|
* Hackishly support broken NER annotations. Should fix this.
|
2015-05-27 19:14:31 +02:00 |
|
Matthew Honnibal
|
7a2725bca4
|
* Read input json in a streaming way
|
2015-05-27 19:13:11 +02:00 |
|
Matthew Honnibal
|
6a1c91675e
|
* Add file to read ENAMEX ner data
|
2015-05-27 17:36:23 +02:00 |
|
Matthew Honnibal
|
732fa7709a
|
* Edits to align_raw script, for use in prepare_treebank
|
2015-05-27 04:23:31 +02:00 |
|
Matthew Honnibal
|
4010b9b6d9
|
* Pass parameter for regularization in parser.pyx
|
2015-05-27 03:18:50 +02:00 |
|
Matthew Honnibal
|
4c6058baa7
|
* Fix evaluation of NER in scorer.py
|
2015-05-27 03:18:16 +02:00 |
|
Matthew Honnibal
|
6016ee83a6
|
* Fix reading of NER in gold.pyx
|
2015-05-27 03:17:50 +02:00 |
|
Matthew Honnibal
|
04bda8648d
|
* Pass parameter for regularization to model
|
2015-05-27 03:16:58 +02:00 |
|
Matthew Honnibal
|
f69fe6a635
|
* Fix heads problem in read_conll
|
2015-05-27 01:14:54 +02:00 |
|
Matthew Honnibal
|
0eec1d12af
|
* Add comment about zipf reweighting
|
2015-05-27 01:14:07 +02:00 |
|
Matthew Honnibal
|
4d37b66c55
|
* Make Zipf regularization a bit more efficient
|
2015-05-27 01:12:50 +02:00 |
|
Matthew Honnibal
|
7fc24821bc
|
* Experiment with Zipfian corruptions when calculating prediction
|
2015-05-26 22:17:15 +02:00 |
|
Matthew Honnibal
|
eba7b34f66
|
* Add flag to disable loading of word vectors
|
2015-05-25 01:02:42 +02:00 |
|
Matthew Honnibal
|
3593babd35
|
* Add functions for Levenshtein distance alignment
|
2015-05-24 21:50:48 +02:00 |
|
Matthew Honnibal
|
744f06abf5
|
* Add script to read OntoNotes source documents
|
2015-05-24 21:49:58 +02:00 |
|
Matthew Honnibal
|
fc75210941
|
* Move spacy.syntax.conll to spacy.gold
|
2015-05-24 21:35:02 +02:00 |
|
Matthew Honnibal
|
765b61cac4
|
* Update spacy.scorer, to use P/R/F to support tokenization errors
|
2015-05-24 20:07:18 +02:00 |
|
Matthew Honnibal
|
efe7a7d7d6
|
* Clean unused functions from spacy.syntax.conll
|
2015-05-24 20:06:46 +02:00 |
|
Matthew Honnibal
|
78487f3e66
|
* Update parser oracle for missing heads
|
2015-05-24 20:05:58 +02:00 |
|
Matthew Honnibal
|
1044a13413
|
* Begin refactoring scorer to use recall over gold dependencies
|
2015-05-24 17:40:15 +02:00 |
|
Matthew Honnibal
|
acd1245ad4
|
* Remove cruft from conll.pyx --- unused stuff about evlauation, which now lives in spacy.scorer
|
2015-05-24 17:35:49 +02:00 |
|
Matthew Honnibal
|
20f1d868a3
|
* Tmp commit. Working on whole document parsing
|
2015-05-24 02:49:56 +02:00 |
|
Matthew Honnibal
|
f2ee9c4feb
|
* Comment out constituency parsing stuff, so that code compiles
|
2015-05-20 16:55:05 +02:00 |
|