Add crawlBlacklist configuration option
[phinde.git] / src / phinde / Crawler.php
2016-08-30 Christian WeiskeAdd crawlBlacklist configuration option
2016-08-30 Christian WeiskeFix notice
2016-08-30 Christian WeiskeOption to disable linked URL indexing
2016-08-29 Christian WeiskeSend If-Modified-Since header on crawling and indexing
2016-02-11 Christian Weiskedebug option for crawler
2016-02-10 Christian Weiskeremove debug statement
2016-02-10 Christian Weiskecrawler supports "nofollow" now
2016-02-10 Christian Weiskesend accept header during crawl
2016-02-10 Christian Weiskerework crawler; add atom link extraction