operator: Add support for built-in-cert-rotation for all internal lokistack encryption #7064

periklis · 2022-09-06T09:44:31Z

What this PR does / why we need it:
The following PR adds support for built-in cert-rotation for all internal LokiStack data-in-transit encryption. This ensures that users enabling the feature gate HTTPEncryption and GRPCEncryption none of the running components except the lokistack-gateway can be access with a TLS client certificate signed by the operator-managed CA.

In particular the lokistack-gateway requires from users enabling the HTTPEncryption feature gate to provide:

A custom CA bundle that consists of the list of singning certificates as a ConfigMap with a single key service-ca.crt
A custom Cert/Key pair that represents the serving certificate/key pair as a Secret with keys tls.crt and `tls.key

This distinction is in place because the lokistack-gateway represents the public available API of the LokiStack. In general public APIs might be secured by a different PKI with custom validity, etc. The internal communication though between the lokistack-gateway and the upstream Loki services distributor and query-frontend is maintained by the built-in cert rotation provide PKI.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
Requires:

Checklist

Documentation added
Tests updated
Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

grafanabot · 2022-09-06T09:49:17Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-06T11:11:07Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-06T17:36:53Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-07T11:31:32Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-07T11:43:44Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-07T13:06:03Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-07T16:40:35Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

xperimental

Just a few small things for now.

operator/hack/addons_kind_certs.yaml

operator/internal/certrotation/build.go

operator/internal/certrotation/signer_test.go

operator/internal/handlers/internal/certificates/options_test.go

operator/internal/handlers/lokistack_rotate_certs.go

operator/internal/manifests/gateway.go

operator/cmd/loki-broker/main.go

grafanabot · 2022-09-07T16:59:54Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-09T11:20:58Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-09T12:11:24Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
-               loki	-0.6%

grafanabot · 2022-09-09T12:28:29Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-12T12:38:38Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-12T12:48:17Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

Red-GV

Just a few comments/questions on my side.

operator/internal/certrotation/cert_creator.go

operator/internal/certrotation/signer.go

grafanabot · 2022-09-14T16:38:00Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-15T17:36:34Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

operator/cmd/loki-broker/main.go

operator/controllers/loki/internal/lokistack/certrotation_discovery.go

operator/controllers/loki/certrotation_controller.go

operator/internal/certrotation/build.go

operator/internal/certrotation/cabundle.go

operator/internal/certrotation/options.go

grafanabot · 2022-09-16T17:21:10Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-10-26T14:57:47Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-10-26T15:20:15Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-10-26T16:07:24Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-10-26T16:20:43Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

operator/apis/config/v1/projectconfig_types.go

operator/internal/certrotation/signer.go

operator/internal/certrotation/rotation.go

xperimental · 2022-10-26T15:19:05Z

operator/internal/certrotation/rotation.go

+	// Are we at 80% of validity?
+	at80Percent := notAfter.Add(-validity / 5)
+	if time.Now().After(at80Percent) {
+		return fmt.Sprintf("past its latest possible time %v", at80Percent)


Do we need this check if we have the refresh-time? Currently you can only set a refresh-time "lower than 80%" for it to be effective. My assumption is that for very-long certificate validities you'd want to go past 80% with the refresh as opposed to already renewing "earlier" with shorter validity. Don't know if this additional 80% check will make the refresh-time setting intransparent.

operator/internal/handlers/internal/certificates/options.go

operator/controllers/loki/certrotation_controller_test.go

operator/internal/certrotation/target.go

operator/internal/handlers/lokistack_check_cert_expiry.go

operator/internal/handlers/lokistack_rotate_certs.go

operator/internal/certrotation/cabundle.go

grafanabot · 2022-10-27T09:56:51Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0.1%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

operator/internal/certrotation/build.go

operator/internal/certrotation/signer_test.go

grafanabot · 2022-10-27T11:38:22Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
- querier/queryrange	-0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

xperimental · 2022-11-01T17:35:09Z

operator/internal/manifests/openshift/service_ca.go

 // serviceCA configmap. This configmap is used to configure
 // the gateway and components to verify TLS certificates.
-func BuildServiceCAConfigMap(opts Options) *corev1.ConfigMap {
+func BuildGatewayCAConfigMap(opts Options) *corev1.ConfigMap {


Why BuildGatewayCAConfigMap? This seems to be the ca-bundle used for every service.

…ounts.go Co-authored-by: Robert Jacob <xperimental@solidproject.de>

grafanabot · 2022-11-02T09:10:06Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-11-02T13:14:31Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-11-02T13:31:44Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

-           ingester	-0.1%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

xperimental

Looks good to me. There are two small comments still unresolved, have a look, but I don't think they are show-stoppers.

…istack encryption (grafana#7064)

periklis requested a review from xperimental September 6, 2022 09:44

periklis requested a review from a team as a code owner September 6, 2022 09:44

periklis self-assigned this Sep 6, 2022

pull-request-size bot added the size/XXL label Sep 6, 2022

github-actions bot added the sig/operator label Sep 6, 2022

periklis force-pushed the mtls-support branch from 6b37875 to 8b5a00a Compare September 6, 2022 11:06

periklis force-pushed the mtls-support branch 3 times, most recently from 54d183f to 558c334 Compare September 6, 2022 17:31

xperimental reviewed Sep 7, 2022

View reviewed changes

periklis force-pushed the mtls-support branch from d643654 to dd378a9 Compare September 12, 2022 12:33

Red-GV reviewed Sep 13, 2022

View reviewed changes

operator/internal/certrotation/cert_creator.go Outdated Show resolved Hide resolved

operator/internal/certrotation/cert_creator.go Outdated Show resolved Hide resolved

operator/internal/certrotation/signer.go Outdated Show resolved Hide resolved

periklis force-pushed the mtls-support branch from fce2459 to 0a8491f Compare September 15, 2022 17:31

xperimental reviewed Sep 16, 2022

View reviewed changes

Eliminate multiple time.ParseDuration for rotation options

c1a71c8

Use cert refresh for requeue

d56d8d3

periklis added 2 commits October 26, 2022 18:08

Rename RotationOptions to Rotation

dabb1e4

Naming improvements

3264457

xperimental reviewed Oct 26, 2022

View reviewed changes

Red-GV reviewed Oct 26, 2022

View reviewed changes

operator/internal/certrotation/cabundle.go Show resolved Hide resolved

periklis added 2 commits October 27, 2022 09:33

Apply code review suggestions

68134bc

Naming improvements

102fc70

xperimental reviewed Oct 27, 2022

View reviewed changes

Apply code review suggestions

9b4b1cd

xperimental reviewed Nov 1, 2022

View reviewed changes

Update operator/internal/handlers/internal/serviceaccounts/serviceacc…

06dc940

…ounts.go Co-authored-by: Robert Jacob <xperimental@solidproject.de>

Fix upgrade from serviceCA on OpenShift

36b44fc

Fix upgrade servicemonitors

eb71085

xperimental approved these changes Nov 3, 2022

View reviewed changes

periklis merged commit abf08e2 into grafana:main Nov 3, 2022

periklis mentioned this pull request Nov 4, 2022

operator: Fix lookup CA Bundle options on fresh install #7596

Merged

5 tasks

lxwzy pushed a commit to lxwzy/loki that referenced this pull request Nov 7, 2022

operator: Add support for built-in-cert-rotation for all internal lok…

7b3a178

…istack encryption (grafana#7064)

changhyuni pushed a commit to changhyuni/loki that referenced this pull request Nov 8, 2022

operator: Add support for built-in-cert-rotation for all internal lok…

27b8d8d

…istack encryption (grafana#7064)

Abuelodelanada pushed a commit to canonical/loki that referenced this pull request Dec 1, 2022

operator: Add support for built-in-cert-rotation for all internal lok…

e301e48

…istack encryption (grafana#7064)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

operator: Add support for built-in-cert-rotation for all internal lokistack encryption #7064

operator: Add support for built-in-cert-rotation for all internal lokistack encryption #7064

periklis commented Sep 6, 2022 •

edited

Loading

grafanabot commented Sep 6, 2022

grafanabot commented Sep 6, 2022

grafanabot commented Sep 6, 2022

grafanabot commented Sep 7, 2022

grafanabot commented Sep 7, 2022

grafanabot commented Sep 7, 2022

grafanabot commented Sep 7, 2022

xperimental left a comment

grafanabot commented Sep 7, 2022

grafanabot commented Sep 9, 2022

grafanabot commented Sep 9, 2022

grafanabot commented Sep 9, 2022

grafanabot commented Sep 12, 2022

grafanabot commented Sep 12, 2022

Red-GV left a comment

grafanabot commented Sep 14, 2022

grafanabot commented Sep 15, 2022

grafanabot commented Sep 16, 2022

grafanabot commented Oct 26, 2022

grafanabot commented Oct 26, 2022

grafanabot commented Oct 26, 2022

grafanabot commented Oct 26, 2022

xperimental Oct 26, 2022

grafanabot commented Oct 27, 2022

grafanabot commented Oct 27, 2022

xperimental Nov 1, 2022

grafanabot commented Nov 2, 2022

grafanabot commented Nov 2, 2022

grafanabot commented Nov 2, 2022

xperimental left a comment

operator: Add support for built-in-cert-rotation for all internal lokistack encryption #7064

operator: Add support for built-in-cert-rotation for all internal lokistack encryption #7064

Conversation

periklis commented Sep 6, 2022 • edited Loading

grafanabot commented Sep 6, 2022

grafanabot commented Sep 6, 2022

grafanabot commented Sep 6, 2022

grafanabot commented Sep 7, 2022

grafanabot commented Sep 7, 2022

grafanabot commented Sep 7, 2022

grafanabot commented Sep 7, 2022

xperimental left a comment

Choose a reason for hiding this comment

grafanabot commented Sep 7, 2022

grafanabot commented Sep 9, 2022

grafanabot commented Sep 9, 2022

grafanabot commented Sep 9, 2022

grafanabot commented Sep 12, 2022

grafanabot commented Sep 12, 2022

Red-GV left a comment

Choose a reason for hiding this comment

grafanabot commented Sep 14, 2022

grafanabot commented Sep 15, 2022

grafanabot commented Sep 16, 2022

grafanabot commented Oct 26, 2022

grafanabot commented Oct 26, 2022

grafanabot commented Oct 26, 2022

grafanabot commented Oct 26, 2022

xperimental Oct 26, 2022

Choose a reason for hiding this comment

grafanabot commented Oct 27, 2022

grafanabot commented Oct 27, 2022

xperimental Nov 1, 2022

Choose a reason for hiding this comment

grafanabot commented Nov 2, 2022

grafanabot commented Nov 2, 2022

grafanabot commented Nov 2, 2022

xperimental left a comment

Choose a reason for hiding this comment

periklis commented Sep 6, 2022 •

edited

Loading