Commit Graph

119 Commits

Author SHA1 Message Date
Miroslav Stampar
5f2bb88037 Some code refactoring 2017-04-18 15:48:05 +02:00
Miroslav Stampar
0e206da7c0 Minor patches (pydiatra) 2017-04-14 13:08:51 +02:00
Miroslav Stampar
ec0c103952 Bug fix (reported privately) 2017-02-15 10:30:29 +01:00
Miroslav Stampar
55272f7a3b New version preparation 2017-01-02 14:19:18 +01:00
Miroslav Stampar
17c556a63d Minor patches (and one bug from ML) 2016-12-20 09:53:44 +01:00
Miroslav Stampar
bb6e8fd4ce Minor bug fix (reported privately via email) 2016-12-15 16:09:09 +01:00
Miroslav Stampar
26d4dec5fb Minor refactoring 2016-05-31 13:02:26 +02:00
Miroslav Stampar
d0d676ccce Update of copyright string 2016-01-06 00:06:12 +01:00
Miroslav Stampar
193f8190c4 Adding new warning message 2015-11-07 23:30:24 +01:00
Miroslav Stampar
95ce5a4a09 Fixes #1444 2015-10-05 16:33:10 +02:00
Miroslav Stampar
16f8e4c8ba Removing unused imports 2015-07-12 12:25:02 +02:00
Miroslav Stampar
dbfa8f1cfc Fix for a bug reported by the user (conf.scheme/conf.hostname/conf.port were None in multiple targets mode) 2015-04-14 11:05:17 +02:00
Miroslav Stampar
0e4800f73c Changing default answer for sitemap checking to N 2015-04-14 09:30:01 +02:00
Miroslav Stampar
1e7f2d6da2 Implements #1215 2015-04-06 22:07:22 +02:00
Miroslav Stampar
25b23750e8 Bug fix for crawling over non-80 port 2015-03-12 11:49:52 +01:00
Miroslav Stampar
9f4a32ca2b Automatically checking for sitemap existence in case of --crawl 2015-01-20 10:03:35 +01:00
Miroslav Stampar
45bdefd29b Update of copyright 2015-01-06 15:02:16 +01:00
Miroslav Stampar
605b126758 Patch for an Issue #976 2014-11-26 13:38:21 +01:00
Miroslav Stampar
1a8b58fca6 Minor update 2014-11-20 16:42:06 +01:00
Miroslav Stampar
f8a8cbf9a6 Storing crawling results to a temporary file (for eventual further processing) 2014-11-20 16:29:17 +01:00
Miroslav Stampar
34aed7cde0 Bug fix (now it's possible to use multiple parsed requests without mixing associated headers) 2014-10-22 13:49:29 +02:00
Miroslav Stampar
4e3a4eb0ff Added a prompt for choosing a number of threads when in crawling mode 2014-10-10 12:09:08 +02:00
Bernardo Damele
43a4e85749 updated copyright 2014-01-13 17:24:49 +00:00
stamparm
2bfdac5ebc Minor update for crawler 2013-04-30 18:32:46 +02:00
stamparm
ebe8ee3500 Fix for crawler and redirection case 2013-04-30 18:08:26 +02:00
stamparm
3c110b3620 Minor bug fix 2013-04-30 16:40:16 +02:00
stamparm
8c9da95343 Style and consistency update (url -> URL) 2013-04-09 11:48:42 +02:00
stamparm
3948b527dd Update for an Issue #429 2013-04-09 11:36:33 +02:00
stamparm
91054099aa Minor style update 2013-04-09 10:42:58 +02:00
Bernardo Damele
4b9d8ed673 reverted a previous commit as not all distributions create a link file /usr/bin/python2 to the Python interpreter 2013-02-14 11:32:17 +00:00
Bernardo Damele
a67ef4117f make sure to use Python 2 interpreter when default system Python is version 3 2013-02-14 11:25:04 +00:00
Bernardo Damele
a43202f3c0 updated copyright 2013-01-18 14:07:51 +00:00
Miroslav Stampar
ca3d35a878 Some PEP8 related style cleaning 2013-01-10 13:18:44 +01:00
Miroslav Stampar
bf5544903b Minor style update 2013-01-09 16:10:26 +01:00
Miroslav Stampar
3d4f381ab5 Patch for an Issue #169 2013-01-09 15:22:21 +01:00
Miroslav Stampar
cb91729913 Fix for an Issue #324 (crawling when HTML is not well-formed) 2012-12-27 20:55:37 +01:00
Miroslav Stampar
712cf4e4db Fix for an Issue #316 2012-12-20 20:55:59 +01:00
Miroslav Stampar
c2c4601d6e Minor restyling 2012-12-20 11:06:52 +01:00
Miroslav Stampar
974407396e Doing some more style updating (capitalization of exception classes; using _ is enough for private members - __ is used in Python specific methods) 2012-12-06 14:14:19 +01:00
Miroslav Stampar
baccbd6f48 Implementation for an Issue #283 2012-12-06 11:57:57 +01:00
Miroslav Stampar
b6650add46 Introducing 'new style classes' (idea from Pull request #284) 2012-12-06 10:42:53 +01:00
Miroslav Stampar
2de52927f3 Code refactoring (epecially Google search code) 2012-10-30 18:38:10 +01:00
Miroslav Stampar
87ecf205cb More work for Issue #66 2012-07-14 17:01:04 +02:00
Bernardo Damele
162da75a04 modified homepage address 2012-07-12 18:38:03 +01:00
jekil
c39e5a85ba Removed $id$ tags 2012-06-27 20:56:43 +02:00
Miroslav Stampar
6c4bd84d18 minor fix (turning back the functionality of kb.suppressResumeInfo) 2012-06-25 16:19:51 +00:00
Miroslav Stampar
d2dd47fb23 some more refactoring 2012-06-14 13:52:56 +00:00
Miroslav Stampar
b3bd4144f5 removing of unused imports together with some general code refactoring 2012-02-22 10:40:11 +00:00
Miroslav Stampar
95f89ab63a updating copyright date 2012-01-11 14:59:46 +00:00
Miroslav Stampar
29f502fe29 some refactoring 2011-12-28 16:27:17 +00:00
Miroslav Stampar
bdc724cb46 minor bug fix 2011-12-20 10:34:28 +00:00
Miroslav Stampar
ef987c6954 adding compatibility support for using --crawl and --forms together 2011-10-29 09:32:20 +00:00
Miroslav Stampar
f5e45bf113 quick fix for a bug reported by jovon.itwaru@gmail.com 2011-07-11 08:54:39 +00:00
Bernardo Damele
aedcf8c8d7 Changed homepage address 2011-07-07 20:10:03 +00:00
Miroslav Stampar
e00cf81f7e minor update 2011-06-24 19:50:13 +00:00
Miroslav Stampar
e9286ddd5b fix for a bug reported by g@brindi.si (UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
47: ordinal not in range(128))
2011-06-24 19:24:11 +00:00
Miroslav Stampar
eaa2a4202f changing to: --crawl=CRAWLDEPTH 2011-06-24 05:40:03 +00:00
Miroslav Stampar
dfc02d8c3c sorry Bernardo, i hope your mobile is turned off :))) 2011-06-20 22:47:24 +00:00
Miroslav Stampar
2a4a284a29 crawler fix (skip binary files) 2011-06-20 22:41:38 +00:00
Miroslav Stampar
20bb1a685b really minor update 2011-06-20 21:57:53 +00:00
Miroslav Stampar
812cd2f19b minor update 2011-06-20 21:47:03 +00:00
Miroslav Stampar
e8ac7414f2 bug fix 2011-06-20 21:36:15 +00:00
Miroslav Stampar
d6062e8fc9 minor fix for crawler and far less message overlaps in future 2011-06-20 21:18:12 +00:00
Miroslav Stampar
8968c708a0 minor update 2011-06-20 14:27:24 +00:00
Miroslav Stampar
17fac6f67f minor update 2011-06-20 13:53:39 +00:00
Miroslav Stampar
4d1fa5596b added support for --scope in --crawl mode 2011-06-20 12:37:51 +00:00
Miroslav Stampar
42746cc706 bug fix 2011-06-20 12:18:46 +00:00
Miroslav Stampar
cda39ca350 minor update 2011-06-20 11:46:23 +00:00
Miroslav Stampar
07e2c72943 adding Beautifulsoup (BSD) into extras; adding --crawl to options 2011-06-20 11:32:30 +00:00