Helm - Loki: Introduce runtimeConfig #8179

nervo · 2023-01-17T14:43:24Z

This is completely inspired by similar value available in loki-distributed via grafana/helm-charts#1827, which is itself largely inspired by similar value in mimir-distributed helm chart.

The idea is to have a dedicated configmap to handle runtime_config, alway present even if empty. This way, having it seperately from main config, components pods are guaranteed not to restart for any changes.

runtimeConfig:
  overrides:
    tenant1:
      ingestion_rate_mb: 10
      max_streams_per_user: 100000
      max_chunks_per_query: 100000
    tenant2:
      max_streams_per_user: 1000000
      max_chunks_per_query: 1000000

  multi_kv_config:
      mirror-enabled: false
      primary: consul

Fixes #7918

Some questionable details:

Like in loki-distributed/mimir-distributed, runtimeConfig is available at values root (.Values.runtimeConfig). I feel a bit uncomfortable with this, i would have preferred .Values.loki.runtimeConfig, but having kind-of compatibility with loki-distributed/mimir-distributed may be more desirable.
mimir-distributed uses /data for mounting data and /var/mimir for mounting runtime config. As loki uses /var/loki for mounting its data, i've decided to stay in /var directory and uses /var/loki-runtime for its runtime config.

Checklist

Reviewed the CONTRIBUTING.md guide (required)
Documentation added
Tests updated
CHANGELOG.md updated
Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

CLAassistant · 2023-01-17T14:43:30Z

All committers have signed the CLA.

grafanabot · 2023-01-17T14:47:41Z

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2023-01-17T15:06:18Z

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2023-01-18T08:21:22Z

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

salvacorts

Awesome work @nervo, thanks for putting this together. It's very useful!

I've left a couple of comments and questions.

salvacorts · 2023-01-18T10:58:00Z

production/helm/loki/CHANGELOG.md

@@ -21,6 +21,9 @@ Entries should include a reference to the pull request that introduced the chang
 - [CHANGE] **BREKAING** Rename and change format of `enterprise.provisioner.tenants`. Property has been renamed to `enterprise.provisioner.additionalTenants`, and is now an array of objects rather than string. Each object must contain a `name` and a `secretNamespace` field, where `name` is the name of the tenant and `secretNamespace` is the namespace to create the secret with the tenant's read and write token.
 - [CHANGE] **BREAKING** Change the structure of `monitoring.selfMonitoring.tenant` from a string to an object. The new object must have a `name` and a `secretNamespace` field, where `name` is the name of the self-monitoring tenant and `secretNamespace` is the namespace to create an additional secret with the tenant's token. A secret will still also be created in the release namespace as it's needed by the Loki canary.
 - [CHANGE] **BREAKING** Remove ability to create self-monitoring resources in different namespaces (with the exception of dashboard configmaps).
+## 3.10.1


I think this should be 4.0.1 (as the change on Chart.yaml) and be added before 4.0.

In fact, looking at the proposed changes in production/helm/loki/Chart.yaml and production/helm/loki/README.md, I'm not sure if those align with the release process for the Loki HELM charts.

@trevorwhitney may have more context on this, but, shouldn't we have a Next section here with the changes to be released with the next version of the charts? Otherwise, any new PR to these charts will result in a new chart version.

Ooops :) Nice catch, sorry.

Fyi, in my company, we use https://keepachangelog.com/en/1.0.0/ . It recommends a ## [Unreleased] section at the top of the changelog, comparable to the Nextsection you suggest.

production/helm/loki/templates/write/statefulset-write.yaml

production/helm/loki/values.yaml

production/helm/loki/templates/runtime-configmap.yaml

production/helm/loki/templates/tokengen/job-tokengen.yaml

production/helm/loki/templates/single-binary/statefulset.yaml

production/helm/loki/templates/read/statefulset-read.yaml

production/helm/loki/templates/read/deployment-read.yaml

production/helm/loki/templates/backend/statefulset-backend.yaml

jeschkies

Thanks for your great work. Please address our comments out or should be good to go.

production/helm/loki/values.yaml

grafanabot · 2023-01-18T13:31:20Z

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2023-01-18T13:37:06Z

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2023-01-18T20:30:12Z

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2023-01-18T20:35:33Z

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

nervo · 2023-01-18T20:46:00Z

comments addressed, pr rebased, docs fixed, tests ok !

trevorwhitney

This looks great, thanks for the contribution. You'll need to rebase and bump the version, and I think it's better to bump the minor version instead of the patch as this is new functionality. Thanks, LGTM!

trevorwhitney · 2023-01-18T22:32:32Z

production/helm/loki/Chart.yaml

@@ -4,7 +4,7 @@ name: loki
 description: Helm chart for Grafana Loki in simple, scalable mode
 type: application
 appVersion: 2.7.0
-version: 4.1.0
+version: 4.1.1


I think we can just to a minor version for this. New functionality, so not really a patch IMO

trevorwhitney · 2023-01-18T22:34:20Z

production/helm/loki/templates/runtime-configmap.yaml

@@ -0,0 +1,9 @@
+apiVersion: v1


I agree with @jeschkies, keep a ConfigMap for now, and I'd like to see us implement a configuration as talked about in #7358 to specify ConfigMap vs Secret for both this and the regular config at a later point.

trevorwhitney · 2023-01-18T22:37:12Z

production/helm/loki/values.yaml

@@ -206,6 +209,9 @@ loki:
    max_cache_freshness_per_query: 10m
    split_queries_by_interval: 15m

+  # -- Provides a reloadable runtime configuration file for some specific configuration
+  runtimeConfig: {}


I agree with this decision to make this loki.runtimeConfig rather than top level 👍

trevorwhitney · 2023-01-18T22:42:41Z

I took the liberty of rebasing and bumping the version, hope you don't mind 😄

grafanabot · 2023-01-18T22:46:41Z

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

trevorwhitney · 2023-01-18T22:59:35Z

@jeschkies this looks good to go, just needs an approval from you.

trallnag · 2023-01-25T16:02:43Z

Why is the changelog pointing to 4.1.1 while chart version is 4.4.0?

trevorwhitney · 2023-01-25T23:47:35Z

@trallnag good catch, looks likes a bit of cleanup is needed.

nervo requested a review from a team as a code owner January 17, 2023 14:43

pull-request-size bot added the size/M label Jan 17, 2023

github-actions bot added the area/helm label Jan 17, 2023

nervo changed the title ~~[Helm - Loki] Introduce runtimeConfig~~ Helm - Loki: Introduce runtimeConfig Jan 17, 2023

nervo force-pushed the helm-loki/runtimeConfig branch from b5be135 to 16b2b65 Compare January 17, 2023 15:01

github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Jan 17, 2023

nervo force-pushed the helm-loki/runtimeConfig branch from 16b2b65 to a460467 Compare January 18, 2023 08:15

salvacorts self-requested a review January 18, 2023 11:17

salvacorts reviewed Jan 18, 2023

View reviewed changes

jeschkies requested changes Jan 18, 2023

View reviewed changes

production/helm/loki/values.yaml Outdated Show resolved Hide resolved

nervo force-pushed the helm-loki/runtimeConfig branch from a460467 to b6eab91 Compare January 18, 2023 13:25

nervo force-pushed the helm-loki/runtimeConfig branch from b6eab91 to 1385533 Compare January 18, 2023 13:32

nervo force-pushed the helm-loki/runtimeConfig branch 3 times, most recently from 5b4443b to ad707b4 Compare January 18, 2023 20:25

nervo force-pushed the helm-loki/runtimeConfig branch from ad707b4 to 077ad8f Compare January 18, 2023 20:30

trevorwhitney approved these changes Jan 18, 2023

View reviewed changes

[Helm - Loki] Introduce runtimeConfig

fc645ff

trevorwhitney force-pushed the helm-loki/runtimeConfig branch from 077ad8f to fc645ff Compare January 18, 2023 22:42

jeschkies approved these changes Jan 19, 2023

View reviewed changes

jeschkies merged commit 7aa5967 into grafana:main Jan 19, 2023

nervo deleted the helm-loki/runtimeConfig branch January 19, 2023 07:32

mraboosk pushed a commit to mraboosk/loki that referenced this pull request Oct 7, 2024

Helm - Loki: Introduce runtimeConfig (grafana#8179)

1b133ec

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Helm - Loki: Introduce runtimeConfig #8179

Helm - Loki: Introduce runtimeConfig #8179

nervo commented Jan 17, 2023 •

edited

Loading

CLAassistant commented Jan 17, 2023 •

edited

Loading

grafanabot commented Jan 17, 2023

grafanabot commented Jan 17, 2023

grafanabot commented Jan 18, 2023

salvacorts left a comment

salvacorts Jan 18, 2023

salvacorts Jan 18, 2023

nervo Jan 18, 2023

jeschkies left a comment

grafanabot commented Jan 18, 2023

grafanabot commented Jan 18, 2023

grafanabot commented Jan 18, 2023

grafanabot commented Jan 18, 2023

nervo commented Jan 18, 2023

trevorwhitney left a comment

trevorwhitney Jan 18, 2023

trevorwhitney Jan 18, 2023

trevorwhitney Jan 18, 2023

trevorwhitney commented Jan 18, 2023

grafanabot commented Jan 18, 2023

trevorwhitney commented Jan 18, 2023

trallnag commented Jan 25, 2023

trevorwhitney commented Jan 25, 2023

Helm - Loki: Introduce runtimeConfig #8179

Helm - Loki: Introduce runtimeConfig #8179

Conversation

nervo commented Jan 17, 2023 • edited Loading

CLAassistant commented Jan 17, 2023 • edited Loading

grafanabot commented Jan 17, 2023

grafanabot commented Jan 17, 2023

grafanabot commented Jan 18, 2023

salvacorts left a comment

Choose a reason for hiding this comment

salvacorts Jan 18, 2023

Choose a reason for hiding this comment

salvacorts Jan 18, 2023

Choose a reason for hiding this comment

nervo Jan 18, 2023

Choose a reason for hiding this comment

jeschkies left a comment

Choose a reason for hiding this comment

grafanabot commented Jan 18, 2023

grafanabot commented Jan 18, 2023

grafanabot commented Jan 18, 2023

grafanabot commented Jan 18, 2023

nervo commented Jan 18, 2023

trevorwhitney left a comment

Choose a reason for hiding this comment

trevorwhitney Jan 18, 2023

Choose a reason for hiding this comment

trevorwhitney Jan 18, 2023

Choose a reason for hiding this comment

trevorwhitney Jan 18, 2023

Choose a reason for hiding this comment

trevorwhitney commented Jan 18, 2023

grafanabot commented Jan 18, 2023

trevorwhitney commented Jan 18, 2023

trallnag commented Jan 25, 2023

trevorwhitney commented Jan 25, 2023

nervo commented Jan 17, 2023 •

edited

Loading

CLAassistant commented Jan 17, 2023 •

edited

Loading