Feature: Implement endpoint or push notifications for source health checks

[FoxxMD]

Based on #65

[ ] Implement a healthcheck endpoint to monitor status (running/idle) of sources/clients
[ ] Integrate discord webhook or something like pusher webhook to allow notifications when health changes for sources/clients

[Karsten-Yan]

The webhook would probably also support REST notification services like gotify or notify.sh, correct? If so, that would be awesome, looking forward to that feature!

[FoxxMD]

@iluvatyr @Karsten-Yan

I've implemented webhooks for Gotify/Ntfy as well as a health check endpoint. Preview in develop

Webhooks can be added to config.json using webhooks property (see schema for examples and structure)

{
  "sources": [...],
  "clients": [...],
  "webhooks": [
    {
      "type": "gotify",
      "url": "..."
    }
  ],
}

For now, webhooks notifications only push for:

• polling started
• polling retry
• polling stopped on error
• scrobble client scrobble failure

Health endpoint can be accessed at http://YourMsURL/health

• returns 200 when everything is working or 500 if anything is not
• the plain url (/health) aggregates status of all clients/sources -- so any failing client/source will make status return 500
• use query params type or name to restrict client/sources aggregated IE /health?type=spotify or /health?name=MyMaloja
• on 500 response returns a JSON payload with "messages" that describes the issue
  • for any clients/sources that require authentication /health will return 500 if they are not authenticated
  • for sources that poll (spotify, yt music, subsonic) /health will 500 if they are not polling

[FoxxMD]

Implemented in v0.4.1. Docs here

If there are any other events you think webhooks should notify on let me know.

[iluvatyr]

This is one of the best things! Thanks for this. Now it can be monitored both using uptime-kuma /uptimerobot via /health endpoint and also via gotify. Awesome! I already tested it and it works flawlessly for me.

I only suggest adding something to send a testmessage to test if the message is coming correctly, either by pressing "Ping gotify/ntfy" in webui, via accessing a URL (/notifytest or else) or via terminal command from within the docker container.

Also it would be great to be able to choose, which notifications should be used. Someone may for example not want to be notified for "polling started", or some only want to be notified for "polling failed". Although you making it possible to set priorities is already making this not a big problem.

[FoxxMD]

adding something to send a testmessage

I'll think about it...you can already verify your notification servers are at least reachable by checking the logs at startup. MS attempts to contact the servers on known version/health endpoints and will log an error or success based on that.

2023-02-28T12:43:23-05:00 verbose: [Gotify - Config 0] Initialized. Found Server version 2.2.4
2023-02-28T12:43:23-05:00 info   : [Ntfy - Config 1  ] Initialized. Found Ntfy server

2023-02-28T12:45:29-05:00 error  : [Gotify - Config 0] Failed to contact server | Error: connect ECONNREFUSED 192.168.0.100:8076
2023-02-28T12:45:32-05:00 error  : [Ntfy - Config 1  ] Failed to contact Ntfy server | Error: connect EHOSTUNREACH 192.168.0.100:9991

Will also think on filtering notification types.