2016-09-01 | Christian Weiske | remove anchor from source URLs | tree | commitdiff |
2016-02-11 | Christian Weiske | support base href | tree | commitdiff |
2016-02-11 | Christian Weiske | sanitize title better | tree | commitdiff |
2016-02-11 | Christian Weiske | use correct meta robots attribute | tree | commitdiff |
2016-02-10 | Christian Weiske | crawler supports "nofollow" now | tree | commitdiff |
2016-02-10 | Christian Weiske | rework crawler; add atom link extraction | tree | commitdiff |