phinde.git
19 months agoDo not use STDOUT and STDERR constants master github/master
Christian Weiske [Fri, 21 Apr 2023 19:40:20 +0000 (21:40 +0200)]
Do not use STDOUT and STDERR constants

.. they do not exist in webserver context

19 months agoMention gearadmin
Christian Weiske [Sun, 2 Apr 2023 17:03:00 +0000 (19:03 +0200)]
Mention gearadmin

19 months agoFix string array access in php 8
Christian Weiske [Sun, 2 Apr 2023 16:57:18 +0000 (18:57 +0200)]
Fix string array access in php 8

19 months agoUpgrade to pear/http2 to support php 8.2
Christian Weiske [Sun, 2 Apr 2023 16:16:00 +0000 (18:16 +0200)]
Upgrade to pear/http2 to support php 8.2

20 months agoAdd required php extensions to composer.json
Christian Weiske [Mon, 6 Mar 2023 20:19:50 +0000 (21:19 +0100)]
Add required php extensions to composer.json

20 months agoUpdate dependencies to latest version
Christian Weiske [Mon, 6 Mar 2023 19:46:06 +0000 (20:46 +0100)]
Update dependencies to latest version

$ php -v
PHP 8.2.2 (cli) (built: Feb  7 2023 11:27:52) (NTS)

$ composer update
Lock file operations: 5 installs, 28 updates, 6 removals
  - Removing phpdocumentor/reflection-common (2.0.0)
  - Removing phpdocumentor/reflection-docblock (5.1.0)
  - Removing phpdocumentor/type-resolver (1.1.0)
  - Removing phpspec/prophecy (v1.10.3)
  - Removing phpunit/php-token-stream (4.0.0)
  - Removing webmozart/assert (1.7.0)
  - Upgrading doctrine/instantiator (1.3.0 => 2.0.0)
  - Upgrading myclabs/deep-copy (1.9.5 => 1.11.0)
  - Locking nikic/php-parser (v4.15.4)
  - Upgrading pear/console_commandline (v1.2.2 => v1.2.4)
  - Upgrading pear/http_request2 (v2.3.0 => v2.5.1)
  - Upgrading pear/pear_exception (v1.0.0 => v1.0.2)
  - Upgrading phar-io/manifest (1.0.3 => 2.0.3)
  - Upgrading phar-io/version (2.0.1 => 3.2.1)
  - Upgrading phpunit/php-code-coverage (8.0.1 => 9.2.26)
  - Upgrading phpunit/php-file-iterator (3.0.0 => 3.0.6)
  - Upgrading phpunit/php-invoker (3.0.0 => 3.1.1)
  - Upgrading phpunit/php-text-template (2.0.0 => 2.0.4)
  - Upgrading phpunit/php-timer (3.0.0 => 5.0.3)
  - Upgrading phpunit/phpunit (9.0.1 => 9.6.4)
  - Locking sebastian/cli-parser (1.0.1)
  - Locking sebastian/code-unit (1.0.8)
  - Upgrading sebastian/code-unit-reverse-lookup (2.0.0 => 2.0.3)
  - Upgrading sebastian/comparator (4.0.0 => 4.0.8)
  - Locking sebastian/complexity (2.0.2)
  - Upgrading sebastian/diff (4.0.0 => 4.0.4)
  - Upgrading sebastian/environment (5.0.1 => 5.1.5)
  - Upgrading sebastian/exporter (4.0.0 => 4.0.5)
  - Upgrading sebastian/global-state (4.0.0 => 5.0.5)
  - Locking sebastian/lines-of-code (1.0.3)
  - Upgrading sebastian/object-enumerator (4.0.0 => 4.0.4)
  - Upgrading sebastian/object-reflector (2.0.0 => 2.0.4)
  - Upgrading sebastian/recursion-context (4.0.0 => 4.0.5)
  - Upgrading sebastian/resource-operations (3.0.0 => 3.0.3)
  - Upgrading sebastian/type (2.0.0 => 3.2.1)
  - Upgrading sebastian/version (3.0.0 => 3.0.2)
  - Upgrading symfony/polyfill-ctype (v1.10.0 => v1.27.0)
  - Upgrading theseer/tokenizer (1.1.3 => 1.2.1)
  - Upgrading twig/twig (v1.35.4 => v1.44.7)

4 years agoWrite errors to stderr
Christian Weiske [Fri, 17 Apr 2020 12:03:55 +0000 (14:03 +0200)]
Write errors to stderr

4 years agoSupport location redirect changes
Christian Weiske [Sun, 8 Mar 2020 22:54:16 +0000 (23:54 +0100)]
Support location redirect changes

4 years agoFix accept header in crawler
Christian Weiske [Sun, 8 Mar 2020 22:37:31 +0000 (23:37 +0100)]
Fix accept header in crawler

4 years agoSupport subscriptions to redirect URLs
Christian Weiske [Sat, 7 Mar 2020 21:26:59 +0000 (22:26 +0100)]
Support subscriptions to redirect URLs

Resolves: https://github.com/cweiske/phinde/issues/37

4 years agoSupport multiple hub links for WebSub
Christian Weiske [Sat, 7 Mar 2020 20:42:45 +0000 (21:42 +0100)]
Support multiple hub links for WebSub

4 years agolink websub on status page
Christian Weiske [Sat, 7 Mar 2020 20:03:41 +0000 (21:03 +0100)]
link websub on status page

4 years agoemacs dumb-jump configuration
Christian Weiske [Sat, 7 Mar 2020 17:48:58 +0000 (18:48 +0100)]
emacs dumb-jump configuration

4 years agoUse own HttpRequest class so we get get our own user agent
Christian Weiske [Sat, 7 Mar 2020 17:48:43 +0000 (18:48 +0100)]
Use own HttpRequest class so we get get our own user agent

4 years agoTalk about WebSub
Christian Weiske [Fri, 6 Mar 2020 17:39:59 +0000 (18:39 +0100)]
Talk about WebSub

4 years agoAdd cli tool to unsubscribe from a topic
Christian Weiske [Fri, 6 Mar 2020 17:37:30 +0000 (18:37 +0100)]
Add cli tool to unsubscribe from a topic

4 years agoStore the hub that was used for Websub subscription
Christian Weiske [Fri, 6 Mar 2020 17:33:42 +0000 (18:33 +0100)]
Store the hub that was used for Websub subscription

4 years agoSupport web linking in rss feed (atom:link)
Christian Weiske [Thu, 5 Mar 2020 20:34:40 +0000 (21:34 +0100)]
Support web linking in rss feed (atom:link)

4 years agoAdd atom and rss feed link url extraction
Christian Weiske [Thu, 5 Mar 2020 20:26:57 +0000 (21:26 +0100)]
Add atom and rss feed link url extraction

4 years agoAdd unit test preparation
Christian Weiske [Thu, 5 Mar 2020 20:26:36 +0000 (21:26 +0100)]
Add unit test preparation

4 years agoInstall phpunit
Christian Weiske [Thu, 5 Mar 2020 20:26:10 +0000 (21:26 +0100)]
Install phpunit

4 years agoDocument subscribing to websub topics
Christian Weiske [Tue, 3 Mar 2020 16:23:15 +0000 (17:23 +0100)]
Document subscribing to websub topics

4 years agoInstall composer package pear/http2 for websub
Christian Weiske [Tue, 3 Mar 2020 16:02:01 +0000 (17:02 +0100)]
Install composer package pear/http2 for websub

4 years agoDo not break when searching for slashes
Christian Weiske [Sat, 29 Feb 2020 21:24:32 +0000 (22:24 +0100)]
Do not break when searching for slashes

Resolves: https://github.com/cweiske/phinde/issues/36

4 years agoAdd URL rewrites/replacements
Christian Weiske [Sat, 29 Feb 2020 21:08:30 +0000 (22:08 +0100)]
Add URL rewrites/replacements

4 years agoDocument MySQL setup for websub
Christian Weiske [Sat, 29 Feb 2020 20:23:50 +0000 (21:23 +0100)]
Document MySQL setup for websub

5 years agomention debian 9 gearman problems
Christian Weiske [Tue, 8 Jan 2019 20:50:02 +0000 (21:50 +0100)]
mention debian 9 gearman problems

5 years agoDo not throw exception but send 400
Christian Weiske [Tue, 8 Jan 2019 19:38:47 +0000 (20:38 +0100)]
Do not throw exception but send 400

5 years agoDo not allow pages > 100
Christian Weiske [Tue, 8 Jan 2019 19:35:34 +0000 (20:35 +0100)]
Do not allow pages > 100

5 years agocomposer dependency installation
Christian Weiske [Sat, 15 Dec 2018 23:10:00 +0000 (00:10 +0100)]
composer dependency installation

6 years agobetter mobile view
Christian Weiske [Mon, 15 Oct 2018 20:53:20 +0000 (22:53 +0200)]
better mobile view

6 years agotalk about gearman
Christian Weiske [Mon, 15 Oct 2018 20:53:07 +0000 (22:53 +0200)]
talk about gearman

6 years agomention twig
Christian Weiske [Mon, 15 Oct 2018 19:40:23 +0000 (21:40 +0200)]
mention twig

7 years agoFix date searches
Christian Weiske [Wed, 26 Jul 2017 13:14:46 +0000 (15:14 +0200)]
Fix date searches

Resolves: https://github.com/cweiske/phinde/issues/25

7 years agoKeep filter when clicking pager links
Christian Weiske [Wed, 26 Jul 2017 13:03:58 +0000 (15:03 +0200)]
Keep filter when clicking pager links

Resolves: https://github.com/cweiske/phinde/issues/33

7 years agoSetup instructions
Christian Weiske [Wed, 31 May 2017 20:18:06 +0000 (22:18 +0200)]
Setup instructions

7 years agotell how to delete a domain
Christian Weiske [Sun, 21 May 2017 08:44:08 +0000 (10:44 +0200)]
tell how to delete a domain

7 years agoFix color for open tasks
Christian Weiske [Fri, 21 Apr 2017 16:07:52 +0000 (18:07 +0200)]
Fix color for open tasks

7 years agoRenew subscriptions that got no response.
Christian Weiske [Fri, 21 Apr 2017 15:48:14 +0000 (17:48 +0200)]
Renew subscriptions that got no response.

May happen if there were network issues or the server phinde is running on
was shut down directly after the subscription request was sent.

Resolves: https://github.com/cweiske/phinde/issues/28

7 years agoAdd colors to status page to make it easy to spot problems
Christian Weiske [Fri, 21 Apr 2017 15:15:53 +0000 (17:15 +0200)]
Add colors to status page to make it easy to spot problems

Resolves: https://github.com/cweiske/phinde/issues/30

7 years agodocument systemd service setup
Christian Weiske [Thu, 20 Apr 2017 21:47:19 +0000 (23:47 +0200)]
document systemd service setup

7 years agoadd log file support v0.2.1
Christian Weiske [Sat, 3 Dec 2016 21:32:19 +0000 (22:32 +0100)]
add log file support

7 years agoDo not crash status page when gearman worker is not registered
Christian Weiske [Sat, 3 Dec 2016 21:32:08 +0000 (22:32 +0100)]
Do not crash status page when gearman worker is not registered

7 years agoRemove URL check from process.php. Checking is done in Crawler already
Christian Weiske [Sat, 3 Dec 2016 13:28:21 +0000 (14:28 +0100)]
Remove URL check from process.php. Checking is done in Crawler already

7 years agoblacklist config option is not used
Christian Weiske [Sat, 3 Dec 2016 13:18:53 +0000 (14:18 +0100)]
blacklist config option is not used

7 years agobaseurl config option
Christian Weiske [Sat, 3 Dec 2016 12:15:48 +0000 (13:15 +0100)]
baseurl config option

7 years agoscript to renew websub subscriptions v0.2.0
Christian Weiske [Fri, 25 Nov 2016 06:54:49 +0000 (07:54 +0100)]
script to renew websub subscriptions

7 years agoshow subscriptions on status page
Christian Weiske [Thu, 24 Nov 2016 22:11:52 +0000 (23:11 +0100)]
show subscriptions on status page

7 years agohelp text in sidebar
Christian Weiske [Thu, 24 Nov 2016 21:38:56 +0000 (22:38 +0100)]
help text in sidebar

7 years agoautofocus input field if there is no query
Christian Weiske [Thu, 24 Nov 2016 21:24:17 +0000 (22:24 +0100)]
autofocus input field if there is no query

7 years agomake search bar visible on status page
Christian Weiske [Thu, 24 Nov 2016 21:20:43 +0000 (22:20 +0100)]
make search bar visible on status page

7 years agolink status page
Christian Weiske [Thu, 24 Nov 2016 21:20:33 +0000 (22:20 +0100)]
link status page

7 years agowebsub subcriptions work
Christian Weiske [Thu, 24 Nov 2016 21:09:28 +0000 (22:09 +0100)]
websub subcriptions work

8 years agoConfiguration for default sort order
Christian Weiske [Thu, 17 Nov 2016 17:21:14 +0000 (18:21 +0100)]
Configuration for default sort order

8 years agonew pager
Christian Weiske [Wed, 16 Nov 2016 10:14:23 +0000 (11:14 +0100)]
new pager

8 years agoformat document number on status page
Christian Weiske [Fri, 11 Nov 2016 20:26:34 +0000 (21:26 +0100)]
format document number on status page

8 years agoimprove status page
Christian Weiske [Fri, 11 Nov 2016 20:13:56 +0000 (21:13 +0100)]
improve status page

8 years agostatus page
Christian Weiske [Fri, 11 Nov 2016 19:54:12 +0000 (20:54 +0100)]
status page

8 years agoadd log class
Christian Weiske [Thu, 10 Nov 2016 19:52:35 +0000 (20:52 +0100)]
add log class

8 years agopager: move next and prev links to the outside for easier clicking
Christian Weiske [Thu, 10 Nov 2016 14:22:05 +0000 (15:22 +0100)]
pager: move next and prev links to the outside for easier clicking

8 years agoadd command to shut down a worker
Christian Weiske [Thu, 10 Nov 2016 14:13:51 +0000 (15:13 +0100)]
add command to shut down a worker

8 years agoproperly handle noindex pages
Christian Weiske [Wed, 9 Nov 2016 20:46:05 +0000 (21:46 +0100)]
properly handle noindex pages

8 years agoBig patch merging crawling+indexing into one command, new json document structure
Christian Weiske [Mon, 7 Nov 2016 20:41:36 +0000 (21:41 +0100)]
Big patch merging crawling+indexing into one command, new json document structure

8 years agosetup: check json before dropping current index
Christian Weiske [Sun, 6 Nov 2016 16:16:15 +0000 (17:16 +0100)]
setup: check json before dropping current index

8 years agoMake title configurable
Christian Weiske [Fri, 2 Sep 2016 16:05:00 +0000 (18:05 +0200)]
Make title configurable

Resolves: #11

8 years agoLink github
Christian Weiske [Fri, 2 Sep 2016 16:04:30 +0000 (18:04 +0200)]
Link github

8 years agoSupport multiple "nick:" terms in search field
Christian Weiske [Fri, 2 Sep 2016 16:01:58 +0000 (18:01 +0200)]
Support multiple "nick:" terms in search field

Resolves: #17

8 years agoperformance debug timer
Christian Weiske [Fri, 2 Sep 2016 15:54:15 +0000 (17:54 +0200)]
performance debug timer

8 years agoFix chat log links
Christian Weiske [Fri, 2 Sep 2016 13:20:17 +0000 (15:20 +0200)]
Fix chat log links

Resolves: #16

8 years agomassively improve crawl speed by ditching "exists" queries
Christian Weiske [Fri, 2 Sep 2016 09:01:28 +0000 (11:01 +0200)]
massively improve crawl speed by ditching "exists" queries

8 years agomicro optimization for "exists" ES queries
Christian Weiske [Thu, 1 Sep 2016 18:36:23 +0000 (20:36 +0200)]
micro optimization for "exists" ES queries

8 years agoMake search result hit template configurable, add chat template
Christian Weiske [Thu, 1 Sep 2016 06:11:44 +0000 (08:11 +0200)]
Make search result hit template configurable, add chat template

Resolves: #9

8 years agoAlways show text, make text extract size configurable.
Christian Weiske [Thu, 1 Sep 2016 05:47:49 +0000 (07:47 +0200)]
Always show text, make text extract size configurable.

Resolves: #8

8 years agoremove anchor from source URLs
Christian Weiske [Thu, 1 Sep 2016 05:38:08 +0000 (07:38 +0200)]
remove anchor from source URLs

8 years agotell why crawler stops
Christian Weiske [Tue, 30 Aug 2016 19:37:50 +0000 (21:37 +0200)]
tell why crawler stops

8 years agoAdd crawlBlacklist configuration option
Christian Weiske [Tue, 30 Aug 2016 11:35:05 +0000 (13:35 +0200)]
Add crawlBlacklist configuration option

Resolves: #7

8 years agoAllow worker instances of multiple projects in parallel
Christian Weiske [Tue, 30 Aug 2016 11:10:03 +0000 (13:10 +0200)]
Allow worker instances of multiple projects in parallel

Change "queuePrefix" configuration in each project

Resolves: #5

8 years agoFix notice
Christian Weiske [Tue, 30 Aug 2016 11:05:14 +0000 (13:05 +0200)]
Fix notice

8 years agoMake phinde-worker configurable; allow queue selection
Christian Weiske [Tue, 30 Aug 2016 11:03:26 +0000 (13:03 +0200)]
Make phinde-worker configurable; allow queue selection

Resolves #6

8 years agoOption to disable linked URL indexing
Christian Weiske [Tue, 30 Aug 2016 06:13:33 +0000 (08:13 +0200)]
Option to disable linked URL indexing

Resolves: #2

8 years agoAdd support for modification date queries: "before:", "after:" and "date:"
Christian Weiske [Tue, 30 Aug 2016 06:05:00 +0000 (08:05 +0200)]
Add support for modification date queries: "before:", "after:" and "date:"

Resolves: #4

8 years agoSupport "nick:cweiske" search syntax as alias for "author.name"
Christian Weiske [Tue, 30 Aug 2016 05:36:34 +0000 (07:36 +0200)]
Support "nick:cweiske" search syntax as alias for "author.name"

Resolves: #3

8 years agoRespect <meta name="robots" content="noindex"/>
Christian Weiske [Mon, 29 Aug 2016 20:59:16 +0000 (22:59 +0200)]
Respect <meta name="robots" content="noindex"/>

Fixes: #1
8 years agoSend If-Modified-Since header on crawling and indexing
Christian Weiske [Mon, 29 Aug 2016 18:30:45 +0000 (20:30 +0200)]
Send If-Modified-Since header on crawling and indexing

8 years agoadd LICENSE file
Christian Weiske [Thu, 26 May 2016 13:20:23 +0000 (15:20 +0200)]
add LICENSE file

8 years agowip pubsubhubbub
Christian Weiske [Thu, 31 Mar 2016 18:46:01 +0000 (20:46 +0200)]
wip pubsubhubbub

8 years agoopensearch paging
Christian Weiske [Fri, 12 Feb 2016 16:04:42 +0000 (17:04 +0100)]
opensearch paging

8 years agotrim query string
Christian Weiske [Fri, 12 Feb 2016 06:43:25 +0000 (07:43 +0100)]
trim query string

8 years agoopensearch support v0.1.0
Christian Weiske [Thu, 11 Feb 2016 21:43:34 +0000 (22:43 +0100)]
opensearch support

8 years agosupport base href
Christian Weiske [Thu, 11 Feb 2016 19:02:30 +0000 (20:02 +0100)]
support base href

8 years agosanitize title better
Christian Weiske [Thu, 11 Feb 2016 16:37:12 +0000 (17:37 +0100)]
sanitize title better

8 years agouse correct meta robots attribute
Christian Weiske [Thu, 11 Feb 2016 16:00:58 +0000 (17:00 +0100)]
use correct meta robots attribute

8 years agodebug option for crawler
Christian Weiske [Thu, 11 Feb 2016 07:43:01 +0000 (08:43 +0100)]
debug option for crawler

8 years agoadd date sorting
Christian Weiske [Wed, 10 Feb 2016 21:02:11 +0000 (22:02 +0100)]
add date sorting

8 years agoremove debug statement
Christian Weiske [Wed, 10 Feb 2016 20:15:35 +0000 (21:15 +0100)]
remove debug statement

8 years agocrawler supports "nofollow" now
Christian Weiske [Wed, 10 Feb 2016 16:26:15 +0000 (17:26 +0100)]
crawler supports "nofollow" now

8 years agosend accept header during crawl
Christian Weiske [Wed, 10 Feb 2016 16:09:56 +0000 (17:09 +0100)]
send accept header during crawl

8 years agosome styling, noindex for search result pages
Christian Weiske [Wed, 10 Feb 2016 14:14:34 +0000 (15:14 +0100)]
some styling, noindex for search result pages

8 years agorework crawler; add atom link extraction
Christian Weiske [Wed, 10 Feb 2016 13:56:20 +0000 (14:56 +0100)]
rework crawler; add atom link extraction

8 years agoabout section readme
Christian Weiske [Sat, 6 Feb 2016 19:27:58 +0000 (20:27 +0100)]
about section readme