aboutsummaryrefslogtreecommitdiff
path: root/src/phinde/Crawler.php
AgeCommit message (Collapse)Author
2016-11-10add log classChristian Weiske
2016-11-07Big patch merging crawling+indexing into one command, new json document ↵Christian Weiske
structure
2016-08-30tell why crawler stopsChristian Weiske
2016-08-30Add crawlBlacklist configuration optionChristian Weiske
Resolves: #7
2016-08-30Fix noticeChristian Weiske
2016-08-30Option to disable linked URL indexingChristian Weiske
Resolves: #2
2016-08-29Send If-Modified-Since header on crawling and indexingChristian Weiske
2016-02-11debug option for crawlerChristian Weiske
2016-02-10remove debug statementChristian Weiske
2016-02-10crawler supports "nofollow" nowChristian Weiske
2016-02-10send accept header during crawlChristian Weiske
2016-02-10rework crawler; add atom link extractionChristian Weiske