feat: add loki integration

[amandahla]

Applicable spec:

Overview

This PR adds loki integration. Collects metrics from workload and the charm. Also provides alert rules in case of errors.

Rationale

Collect and monitor logs.

Juju Events Changes

Module Changes

Library Changes

lib/charms/observability_libs/

Checklist

• The charm style guide was applied
• The contributing guide was applied
• The changes are compliant with ISD054 - Managing Charm Complexity
• The documentation is generated using src-docs
• The documentation for charmhub is updated
• The PR is tagged with appropriate label (urgent, trivial, complex)

[github-actions]

Test coverage for e7dc373

Name           Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------
src/charm.py     154     11     22      6    90%   158-160, 179, 183, 205, 209, 227, 254, 259, 284
----------------------------------------------------------
TOTAL            154     11     22      6    90%

Static code analysis report

Run started:2024-10-17 17:29:32.702930

Test results:
  No issues identified.

Code scanned:
  Total lines of code: 683
  Total lines skipped (#nosec): 0
  Total potential issues skipped due to specifically being disabled (e.g., #nosec BXXX): 0

Run metrics:
  Total issues (by severity):
  	Undefined: 0
  	Low: 0
  	Medium: 0
  	High: 0
  Total issues (by confidence):
  	Undefined: 0
  	Low: 0
  	Medium: 0
  	High: 0
Files skipped (0):

[yanksyoon]

LGTM! questions for my own understandings only :D

[yanksyoon]

Btw, would you have integration tests for loki logs? Perhaps something like https://github.com/canonical/jenkins-k8s-operator/blob/main/tests/integration/test_cos.py#L60 might suffice :D

[amandahla]

Btw, would you have integration tests for loki logs? Perhaps something like https://github.com/canonical/jenkins-k8s-operator/blob/main/tests/integration/test_cos.py#L60 might suffice :D

Good point, thanks for sharing, I'll add one in a further PR.
Update: ISD-2574 store created.

[cbartz]

@amandahla Since alerts should always be actionable, is it expected that there is an action for every logged error? Might it be worth raising the threshold a bit (e.g. 5 errors per minute, which would be > 0.083)? Same for the NGINX alert below.

[amandahla]

Since I have no idea how Maubot behaves in a production environment with high usage, I decided to be extra cautious. I don't have a strong opinion, so we could start with a 5m rate and see what happens.