aboutsummaryrefslogtreecommitdiff
path: root/src/phinde/Crawler.php
AgeCommit message (Expand)Author
2016-08-30tell why crawler stopsChristian Weiske
2016-08-30Add crawlBlacklist configuration optionChristian Weiske
2016-08-30Fix noticeChristian Weiske
2016-08-30Option to disable linked URL indexingChristian Weiske
2016-08-29Send If-Modified-Since header on crawling and indexingChristian Weiske
2016-02-11debug option for crawlerChristian Weiske
2016-02-10remove debug statementChristian Weiske
2016-02-10crawler supports "nofollow" nowChristian Weiske
2016-02-10send accept header during crawlChristian Weiske
2016-02-10rework crawler; add atom link extractionChristian Weiske