Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(operator): Use safe bearer token authentication to scrape operator metrics #12164

Merged
merged 15 commits into from
Mar 14, 2024

Conversation

periklis
Copy link
Collaborator

@periklis periklis commented Mar 8, 2024

What this PR does / why we need it:
In OpenShift clusters we have the option to scrape operator metrics either via cluster-monitoring (default case) or user-workload-monitoring (managed clusters, where users track operator metrics themselves). Until now the service monitor for scraping operator metrics was only compatible with cluster-monitoring that allows using bearerTokenFile and tlsConfig.caFile. Both are not allowed when scraping with user-workload-monitoring. The Prometheus Operator in user-workload-monitoring is configured with ArbitraryFSAccessThroughSMsConfig.Deny: true which in turn disallows the prometheus binary to access it's own serviceaccount token to scrape metrics.

Which issue(s) this PR fixes:
Fixes LOG-5165, Replaces #11680

Special notes for your reviewer:
The changeset below introduces a set of new manifests to make an explicit distinction which serviceaccount is used by the Loki Operator itself as well as which is used by prometheus to access metrics only, i.e.

  1. The serviceaccount loki-operator-controller-manager is introduced to be used only by the Loki Operator manager container. This account is bound to RBAC listed in each supported bundle ClusterServiceVersion.
  2. The serviceaccount loki-operator-controller-manager-metrics-reader is introduced along with a secret that holds a long-lived API token and the service CA certificate. The token is referenced in the ServiceMonitor in authorization.credentials replacing bearerTokenFile. The certificate is referenced in the ServiceMonitor in tlsConfig.ca replacing tlsConfig.caFile. Also it is used by Prometheus to scrape metrics from the Loki Operator manager container only through the kube-rbac-proxy sidecar. This serviceaccount is assigned in a ClusterRoleBinding namely loki-operator-controller-manager-read-metrics to get access to the Non-Resoure-URL get/metrics.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
    • If the change is worth mentioning in the release notes, add add-to-release-notes label
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@periklis periklis self-assigned this Mar 8, 2024
@periklis periklis requested review from xperimental and a team as code owners March 8, 2024 10:12
@pull-request-size pull-request-size bot added size/L and removed size/M labels Mar 11, 2024
Copy link
Collaborator

@JoaoBraveCoding JoaoBraveCoding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a code POV lgtm (didn't get to test it on a cluster). This is mainly for use cases where we deploy the LokiOperator in non openshift- namespaces correct?

@periklis
Copy link
Collaborator Author

From a code POV lgtm (didn't get to test it on a cluster). This is mainly for use cases where we deploy the LokiOperator in non openshift- namespaces correct?

No this when we install on openshift-operators-redhat but the cluster admin is monitoring this namespace with user-workload-monitoring instead of cluster-monitoring. This happens on managed clusters where OLM operators are considered as user-workloads.

@periklis periklis changed the title fix(operator): Use service-ca provided cert/key/ca for operator sm fix(operator): Use safe bearer token authentication to scrape operator metrics Mar 12, 2024
Copy link
Collaborator

@xperimental xperimental left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me this needed a change to work (see comment).

@periklis periklis enabled auto-merge (squash) March 14, 2024 18:26
@periklis periklis merged commit 862d0fb into grafana:main Mar 14, 2024
18 checks passed
periklis added a commit to periklis/loki that referenced this pull request Mar 14, 2024
periklis added a commit to periklis/loki that referenced this pull request Mar 14, 2024
periklis added a commit to periklis/loki that referenced this pull request Mar 14, 2024
openshift-merge-bot bot added a commit to openshift/loki that referenced this pull request Mar 15, 2024
openshift-merge-bot bot added a commit to openshift/loki that referenced this pull request Mar 15, 2024
openshift-merge-bot bot added a commit to openshift/loki that referenced this pull request Mar 15, 2024
rhnasc pushed a commit to inloco/loki that referenced this pull request Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants