[DPE-4617] Add support for tracing with tempo_k8s

[shayancanonical]

Issue

We would like to be able to instrument our charm and generate traces to discover inefficiencies in charm code execution. The observability team has developed a charm for tempo that we would like to integrate with.

Solution

Integrate with the tempo charm

Considerations

1. We must use COS lite with edge until it is release to stable with some critical changes
2. We are sending traces with HTTP until some rough edges are rounded with HTTPS

[taurus-forever]

Tnx!

[marceloneppel]

Hi, @shayancanonical! Most of the errors in the async replication tests should be fixed by #526. It was not moved to review yet because another issue happened today when running on CI.

Also, the update in the lib/charms/observability_libs/v1/kubernetes_service_patch.py library is breaking the async replication tests due to canonical/observability-libs#101.

To merge this PR, you can revert the lib/charms/observability_libs/v1/kubernetes_service_patch.py update. After a few retries, the async replication tests should pass.

[shayancanonical]

thanks for helping investigate the root cause of the failing test @marceloneppel. i will momentarily revert the charm lib and update this PR

[codecov]

Codecov Report

Attention: Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 70.23%. Comparing base (212f2c8) to head (bbf9a36).

Files	Patch %	Lines
src/charm.py	75.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #497      +/-   ##
==========================================
+ Coverage   70.19%   70.23%   +0.03%     
==========================================
  Files          10       10              
  Lines        2775     2785      +10     
  Branches      516      518       +2     
==========================================
+ Hits         1948     1956       +8     
- Misses        729      731       +2     
  Partials       98       98              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.