Setting up monitoring/alerts

[victorb]

Acceptance Criteria

• Can view jenkins monitoring dashboards
• Can deploy changes to monitoring by reading the documentation
• Will get alerts if certain metrics goes under the threshold

Tasks

• checkout Prometheus integration for jenkins and apply if possible
• Connect to existing Prometheus deployment
• Setup dashboard
• Setup alerts

Dependencies

Depends on #1

[ghost]

Remember what this means?

• Can check out the monitoring dashboards

Can view the monitoring dashboard? Or: can run the dashboard locally, in an automated way?

[victorb]

@lgierth my intention was "can view the monitoring dashboard".

Thinking about it now, we should probably have two tasks, one for being able to run monitoring locally and one for having it working in production. What you think?

[ghost]

Yeah let that running-locally be separate: ipfs/infra#52

[victorb]

Ok, in that case, this task would depend on us running jenkins deployed somewhere before you can start working on this task. Correct?

[ghost]

Nah I can get started with the local jenkins

[ghost]

#11 enables the scraping endpoint for prometheus, and the dashboard is here: http://metrics.ipfs.team/dashboard/db/jenkins?from=now-1h&to=now -- I simply imported the one from grafana.net and I'm not very convinced by it.

The dashboard will need tuning when there's actual jobs to monitor :) And that's also when we can start setting alert conditions.

Prometheus is currently set to scrape [fce3:5702:8051:3e65:3a36:1299:c458:1470]:8090/prometheus and we can change that to what comes out of #8.

[ghost]

I'm wrapping this up with ipfs/infra#235 which makes all provsn units systemd-compatible, so that we now also get host metrics (cpu, ram, io).

The dashboard should start showing numbers tomorrow when @victorbjelkholm brings jenkins back up (I broke it). Over the rest of the sprint we'll tune the dashboard and add alerts as we see fit.

[ghost]

Splitting off tuning and alerting to #31.

[victorb]

Moved infrastructure for jenkins and monitoring is not setup yet. Reopening this in the meantime.

[victorb]

Yay, jenkins monitoring dashboard is back online!