Dynamically secure RouteGroups using Kubernetes TLS Secrets #2814

rickhlx · 2024-01-01T19:18:22Z

Updating RouteGroup CRD from Add optional TLS spec for RouteGroups szuecs/routegroup-client#16
Refactor to use single function to add a cert to registry on both RouteGroups and Ingress
Both internal and external RouteGroups can terminate TLS with Kubernetes secret

dataclients/kubernetes/routegroup.go

szuecs · 2024-01-02T17:51:24Z

dataclients/kubernetes/routegroup.go

+// and adds the TLS secret to the registry if a match is found.
+func (r *routeGroups) addRouteGroupTLS(ctx *routeGroupContext, tls *definitions.RouteTLSSpec) {
+	// Host in the tls section need to explicitly match the host in the RouteGroup
+	hostlist := compareStringList(tls.Hosts, ctx.routeGroup.Spec.UniqueHosts())


I think what we want here is to check if tls.Hosts is a subset of ctx.routeGroup.Spec.UniqueHosts().
compareStringList returns all items from tls.Hosts that are also part of ctx.routeGroup.Spec.UniqueHosts().
So it reduces the hostlist.

Did you do this by intention and I miss something?

We only want to add certs which have a match from from tls.Hosts and the routeGroup hosts, so reducing the list is desired. If there are no matches we log and return, but if there are matches, we use the reduced list to add the matching tls certs to the registry.

We could log the non-matching hosts in tls.hosts to help debugging a miss configuration. What do you think?

Yes logging makes sense.
I wonder in case of an update that is applied wrong, what we want to have as behavior. For example a timeline like this:

rg foo has tls.hosts foo.example, bar.example and there is a referenced secret that has only these two names foo.example, bar.example as hosts in SAN

rg was updated to add baz.example and remove bar.example in tls.hosts

The current behavior would drop bar.example.

Should this better stop updating it, such that bar.example is not replaced by baz.example ?!

I am thinking of migrations, but can not come up with an idea how this could break less with either case, so from my side we can also just log it.

@AlexanderYastrebov wdyt?

Maybe this should be a part of validator and invalid routegroup should be rejected. We already use validator in webhook and dataclient.

The same is for Ingress - we plan to add ingress validator to dataclient, see #2757

Is this something we could handle in a separate PR to keep the scope limited? Happy to work on getting in validator for both Ingress and RouteGroup.

Lets have it like it is now (log mismatch) and add tls validation for ingress and routegroup as a future improvement (makes sense after we land #2757).
@MustafaSaber FYI

dataclients/kubernetes/routegroup.go

dataclients/kubernetes/testdata/ingressV1/tls/tls-no-secret.yaml

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

TLS secrets is optional in the IngressV1 spec, using similar config for RouteGroups. Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

szuecs · 2024-01-03T15:24:40Z

This is the error:

 go vet ./...
# github.com/zalando/skipper/dataclients/kubernetes
Error: dataclients/kubernetes/ingressv1.go:318:23: call of addTLSCertToRegistry copies lock value: github.com/zalando/skipper/secrets/certregistry.CertRegistry contains sync.Mutex
Error: dataclients/kubernetes/kube.go:601:30: addTLSCertToRegistry passes lock by value: github.com/zalando/skipper/secrets/certregistry.CertRegistry contains sync.Mutex
Error: dataclients/kubernetes/routegroup.go:515:23: call of addTLSCertToRegistry copies lock value: github.com/zalando/skipper/secrets/certregistry.CertRegistry contains sync.Mutex
make: *** [Makefile:127: vet] Error 1
Error: Process completed with exit code 2.

Just pass certregistry.CertRegistry as pointer in ingress and routegroup case

dataclients/kubernetes/ingressv1.go

dataclients/kubernetes/kube.go

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

dataclients/kubernetes/kube.go

dataclients/kubernetes/ingressv1.go

AlexanderYastrebov · 2024-01-06T02:16:29Z

dataclients/kubernetes/routegroup.go

+// and adds the TLS secret to the registry if a match is found.
+func (r *routeGroups) addRouteGroupTLS(ctx *routeGroupContext, tls *definitions.RouteTLSSpec) {
+	// Host in the tls section need to explicitly match the host in the RouteGroup
+	hostlist := compareStringList(tls.Hosts, ctx.routeGroup.Spec.UniqueHosts())


Maybe this should be a part of validator and invalid routegroup should be rejected. We already use validator in webhook and dataclient.

dataclients/kubernetes/testdata/ingressV1/tls/tls-host-mismatch.log

dataclients/kubernetes/testdata/ingressV1/tls/tls-host-mismatch.yaml

dataclients/kubernetes/testdata/ingressV1/tls/tls-missing-host.log

AlexanderYastrebov · 2024-01-06T02:20:48Z

dataclients/kubernetes/routegroup.go

+// and adds the TLS secret to the registry if a match is found.
+func (r *routeGroups) addRouteGroupTLS(ctx *routeGroupContext, tls *definitions.RouteTLSSpec) {
+	// Host in the tls section need to explicitly match the host in the RouteGroup
+	hostlist := compareStringList(tls.Hosts, ctx.routeGroup.Spec.UniqueHosts())


The same is for Ingress - we plan to add ingress validator to dataclient, see #2757

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

dataclients/kubernetes/routegroup.go

dataclients/kubernetes/testdata/routegroups/tls/tls-missing-host.log

AlexanderYastrebov · 2024-01-08T13:33:32Z

Please add missing newlines at the end of test fixtures #2814 (comment) and fix log level #2814 (comment)

LGTM but I think we should land szuecs/routegroup-client#16 first.

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

szuecs · 2024-01-08T14:50:56Z

LGTM but I think we should land szuecs/routegroup-client#16 first.

Why?
I think first data type change then client change.
We don't use the client here.

AlexanderYastrebov · 2024-01-08T15:02:02Z

Why?
I think first data type change then client change.
We don't use the client here.

But we generate CRD there. I think we should settle all changes in the client, then copy CRD here, then we can land client PR.

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

AlexanderYastrebov · 2024-01-09T02:06:06Z

👍

szuecs · 2024-01-09T10:46:54Z

👍

szuecs · 2024-01-09T10:47:03Z

thanks for your work!

szuecs · 2024-01-09T16:09:52Z

I tagged the merged PR commit with https://github.com/szuecs/routegroup-client/releases/tag/v0.23.0

See changes * szuecs/routegroup-client#31 * szuecs/routegroup-client#32 It restores a link to the origin removed by #2814 to make it clear that CRD is the copy. Signed-off-by: Alexander Yastrebov <alexander.yastrebov@zalando.de>

rickhlx commented Jan 1, 2024

View reviewed changes

dataclients/kubernetes/routegroup.go Show resolved Hide resolved

rickhlx force-pushed the routegroup-tls branch from 57c00c5 to 8059893 Compare January 1, 2024 19:25

szuecs mentioned this pull request Jan 2, 2024

Add optional TLS spec for RouteGroups szuecs/routegroup-client#16

Merged

szuecs added the major moderate risk, for example new API, small filter changes that have no risk like refactoring or logs label Jan 2, 2024

szuecs reviewed Jan 2, 2024

View reviewed changes

dataclients/kubernetes/routegroup.go Outdated Show resolved Hide resolved

szuecs reviewed Jan 2, 2024

View reviewed changes

dataclients/kubernetes/routegroup.go Outdated Show resolved Hide resolved

szuecs reviewed Jan 2, 2024

View reviewed changes

dataclients/kubernetes/routegroup.go Outdated Show resolved Hide resolved

szuecs reviewed Jan 2, 2024

View reviewed changes

dataclients/kubernetes/testdata/ingressV1/tls/tls-no-secret.yaml Outdated Show resolved Hide resolved

rickhlx added 14 commits January 3, 2024 10:49

Add RouteGroup tests for securing TLS using secrets

93f2431

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Add TLS spec for RouteGroups

c71636b

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Load TLS certs to registry from RouteGroup TLS definition

c358be5

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Add description to addRouteGroupTLS function

4553fe7

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Refactor using function to add TLS cert to registry

5c5bca5

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

add fixtures for when no tls secret is defined

5e99692

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Skip adding cert to registry when no secret is defined

99ac3bf

TLS secrets is optional in the IngressV1 spec, using similar config for RouteGroups. Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Update RouteGroup CRD with TLS spec

4aca418

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Fix comments for new functions and typos

f175dd0

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Use plain value instead of pointer for secretID

bccbca3

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Use Debug log level when no secrets defined in TLS

2b529a3

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Log hosts in tls and hosts in route or ingress when no match

ded3e16

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Add logging for non matching tls and ingress/routegroup

04e8061

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Remove unneeded deployment definition from tests

0daaddf

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

rickhlx force-pushed the routegroup-tls branch from 72b728d to 0daaddf Compare January 3, 2024 10:50

Update fixture tests with log checks

73aac01

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

rickhlx force-pushed the routegroup-tls branch from f385e7d to 73aac01 Compare January 3, 2024 12:10

Remove all unused Deployments from Ingress fixtures

090e10b

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

szuecs reviewed Jan 3, 2024

View reviewed changes

dataclients/kubernetes/ingressv1.go Outdated Show resolved Hide resolved

szuecs reviewed Jan 3, 2024

View reviewed changes

dataclients/kubernetes/kube.go Outdated Show resolved Hide resolved

Fix how cert registry is passed to addTLSCertToRegistry

5795c51

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

AlexanderYastrebov reviewed Jan 6, 2024

View reviewed changes

rickhlx added 2 commits January 6, 2024 16:04

Add missing newlines for ingress tls fixtures

9bf60a0

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Add local variable for ingress hosts in addSpecIngressTLSV1

31348fc

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

AlexanderYastrebov reviewed Jan 8, 2024

View reviewed changes

dataclients/kubernetes/routegroup.go Show resolved Hide resolved

AlexanderYastrebov reviewed Jan 8, 2024

View reviewed changes

dataclients/kubernetes/routegroup.go Outdated Show resolved Hide resolved

AlexanderYastrebov reviewed Jan 8, 2024

View reviewed changes

dataclients/kubernetes/testdata/routegroups/tls/tls-missing-host.log Outdated Show resolved Hide resolved

rickhlx added 3 commits January 8, 2024 08:40

Log Error when no TLS hosts match Ingress and RouteGroups hosts

d9671ac

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Add missing newlines to fixture log files

2ceba75

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Update fixture for tests for error log

0e82071

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

Update with latest version of routegroup_crd

6750482

Signed-off-by: Ricardo Herrera <rickhl@outlook.com>

rickhlx force-pushed the routegroup-tls branch from 1dc62ce to 6750482 Compare January 8, 2024 18:45

szuecs merged commit cec514c into zalando:master Jan 9, 2024
10 checks passed

rickhlx deleted the routegroup-tls branch January 9, 2024 11:11

AlexanderYastrebov mentioned this pull request Mar 11, 2024

crd: update RouteGroup CRD #2982

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamically secure RouteGroups using Kubernetes TLS Secrets #2814

Dynamically secure RouteGroups using Kubernetes TLS Secrets #2814

rickhlx commented Jan 1, 2024

szuecs Jan 2, 2024

rickhlx Jan 3, 2024

szuecs Jan 3, 2024

AlexanderYastrebov Jan 6, 2024

AlexanderYastrebov Jan 6, 2024

rickhlx Jan 8, 2024

AlexanderYastrebov Jan 8, 2024

szuecs commented Jan 3, 2024 •

edited

Loading

AlexanderYastrebov Jan 6, 2024

AlexanderYastrebov Jan 6, 2024

AlexanderYastrebov commented Jan 8, 2024

szuecs commented Jan 8, 2024

AlexanderYastrebov commented Jan 8, 2024

AlexanderYastrebov commented Jan 9, 2024

szuecs commented Jan 9, 2024

szuecs commented Jan 9, 2024

szuecs commented Jan 9, 2024

Dynamically secure RouteGroups using Kubernetes TLS Secrets #2814

Dynamically secure RouteGroups using Kubernetes TLS Secrets #2814

Conversation

rickhlx commented Jan 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szuecs commented Jan 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexanderYastrebov commented Jan 8, 2024

szuecs commented Jan 8, 2024

AlexanderYastrebov commented Jan 8, 2024

AlexanderYastrebov commented Jan 9, 2024

szuecs commented Jan 9, 2024

szuecs commented Jan 9, 2024

szuecs commented Jan 9, 2024

szuecs commented Jan 3, 2024 •

edited

Loading