Configuration of the "aaa" cluster at GCP

[bartsmykla]

When I was working on moving the perf-dash into the aaa cluster I faced few issues and want to document everything for others, so please help me understand it better.

Let me get you through my thought process, and if anything here is wrong, please comment on that.

1. We are provisioning our cluster with terraform as I can find here: https://github.com/kubernetes/k8s.io/tree/master/infra/gcp/clusters/kubernetes-public/aaa
2. After the [https://github.com/Move perf-dash to community owned infra #549#issuecomment-596019678](@thockin's comment) when I want to deploy a project with it's own subdomain I'm doing so by deploying service with type: NodePort (like in gcsweb and ingress with annotation with the name of static ip address, so I need to add the static address in the first place.
3. Related to that I don't see any terraform code which describes IP addresses so I assume they are being added manually. Is there a reason behind it? Can we do it via terraform instead? (I'm happy to work on this).
4. So after adding the static IP, we can create a PR and point A record with our subdomain to just created IP address at https://github.com/kubernetes/k8s.io/blob/master/dns/zone-configs/k8s.io._0_base.yaml.
5. We also have to update ClusterIssuers (prod, staging) with just created subdomain
6. When it'll be ready we can finally deployour manifests (as in gcsweb example)

@munnerz do you see any problems with my foughts process in relation to cert-manager here?
/assign @munnerz

@BenTheElder @thockin is this process correct? Will it be enought to point a subdomain to a project?
/assign @BenTheElder @thockin

@thockin @dims @cblecker do you think we could put the configuration of static IP addresses into our terraform configuration files?
/assign @dims @cblecker

I want to improve our documentation which I see as a really important for our contributors (new and the current ones) to understand our infra processes better, so I'm very grateful for any help :-)

/assign

[thockin]

ingress with annotation with the name of static ip address, so I need to add the static address in the first place.

The pattern so far has been to create an Ingress without a static IP, let it get allocated and then manually mark it static. In reality, it's not really required to be static, but it's an old habit :)

The main reason NOT to TF it is that (so far) TF is for the cluster, not the things in the cluster.

We also have to update ClusterIssuers

Oh, I didn't realize that. That's...obnoxious. @munnerz is there any way to make this more self-service? Like "allow any domain name asserting under k8s.io or kubernetes.io" and let DNS working or not-working be the deciding factor?

[munnerz]

Oh, I didn't realize that. That's...obnoxious. @munnerz is there any way to make this more self-service? Like "allow any domain name asserting under k8s.io or kubernetes.io" and let DNS working or not-working be the deciding factor?

This is already the case - there is a dnsZones selector in place that matches on all domains under k8s.io and kubernetes.io:

k8s.io/cert-manager/letsencrypt-prod.yaml

Lines 26 to 32 in fe6fbdd

	- selector:
	dnsZones:
	- kubernetes.io
	- k8s.io
	http01:
	ingress:
	name: k8s-io

There is also an explicit exception for gcsweb just above in that file:

k8s.io/cert-manager/letsencrypt-prod.yaml

Lines 14 to 22 in fe6fbdd

	# Make a special exception for gcsweb - it has its own Ingress resource, so
	# define a dedicated solver for it.
	- selector:
	dnsNames:
	- gcsweb.k8s.io
	- gcsweb.kubernetes.io
	http01:
	ingress:
	name: gcsweb

That said, if you create a new Ingress resource (i.e. GCLB) to serve traffic under (i.e. you are not pointing the new domain at the k8s-io Ingress resource), you will need to either:

1. add a new entry, similar to what gcsweb has, to point the 'http01 solver' to use the correct Ingress resource to solve the challenge

• OR -

2. Add the acme.cert-manager.io/http01-override-ingress-name annotation to your Certificate resource to 'override' the configured ingress.name used for solving challenges on a per-certificate basis: https://github.com/jetstack/cert-manager/blob/7a40669fcce22e09ddc3d3ad58078c86e0260357/pkg/apis/acme/v1alpha2/types.go#L20-L25

Hope that helps 😄

[munnerz]

Like "allow any domain name asserting under k8s.io or kubernetes.io" and let DNS working or not-working be the deciding factor?

We need to know which Ingress resource should be updated in order to 'solve' the HTTP01 challenge, so there's no totally magic way to do this - the above annotation is probably the closest thing to that though 😄

[bartsmykla]

Great! Thank you @munnerz! What is the reason gcsweb is treated differently in that case?

[munnerz]

Because gcsweb is exposed under a different IP address with a different Ingress resource (named gcsweb). The redirector service is what is exposed under the k8s-io ingress. If you add a new Ingress which uses yet another new IP, you'll also need to inform cert-manager that it is that ingress resource that should be modified with 'challenge routes' in order to solve challenges.

[munnerz]

The additional config step is a product of the ingress-gce ingress controller mapping a single front-end IP to a single Ingress resource. This means that we have to edit the 'existing' Ingress resource in order to solve challenges.

With ingress controllers like ingress-nginx, which 'flatten' Ingress configuration into a single nginx config, we are able to simply create new Ingress resources that only contain the HTTP01 challenge solving routes, and save this awkward extra step.

[thockin]

To clarify - that annotation and the edit to the cert-manager config achieve the same result? De-centralized is what we want...

…

On Tue, Mar 10, 2020 at 5:01 AM James Munnelly ***@***.***> wrote: The additional config step is a product of the ingress-gce ingress controller mapping a single front-end IP to a single Ingress resource. This means that we *have* to edit the 'existing' Ingress resource in order to solve challenges. With ingress controllers like ingress-nginx, which 'flatten' Ingress configuration into a single nginx config, we are able to simply create *new* Ingress resources that *only* contain the HTTP01 challenge solving routes, and save this awkward extra step. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#643?email_source=notifications&email_token=ABKWAVETAVAS7S46RMBBFQ3RGYT3LA5CNFSM4LDNGVIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOLEBKY#issuecomment-597049515>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVBSSTPQ3KM44G7PSILRGYT3LANCNFSM4LDNGVIA> .

[thockin]

Ping @munnerz

…

On Tue, Mar 10, 2020 at 5:58 PM Tim Hockin ***@***.***> wrote: To clarify - that annotation and the edit to the cert-manager config achieve the same result? De-centralized is what we want... On Tue, Mar 10, 2020 at 5:01 AM James Munnelly ***@***.***> wrote: > > The additional config step is a product of the ingress-gce ingress controller mapping a single front-end IP to a single Ingress resource. This means that we have to edit the 'existing' Ingress resource in order to solve challenges. > > With ingress controllers like ingress-nginx, which 'flatten' Ingress configuration into a single nginx config, we are able to simply create new Ingress resources that only contain the HTTP01 challenge solving routes, and save this awkward extra step. > > — > You are receiving this because you were assigned. > Reply to this email directly, view it on GitHub, or unsubscribe.

[munnerz]

So if we add a new solver entry:

 # matches all Certificate resources
- selector: {}
  http01: 
    ingress:
      name: "anything" # this value will be overridden by the 'http01-override-ingress-name' annotation

and then add acme.cert-manager.io/http01-override-ingress-name: "name-of-ingress-in-current-namespace" to the Certificate resource, we achieve what we after. It effectively applied a per-Certificate override to the clusterissuer.spec.solvers[].http01.ingress.name.

We could also continue to make the 'solver' match on dnsZones too if we want to make it required that any new zone being added has to require a change to our ClusterIssuer (which required admins to approve) (i.e. still keep the selector containing dnsZones: ["k8s.io", "kubernetes.io"]), e.g.:

 # matches all Certificate resources
- selector:
    dnsZones:
    - k8s.io
    - kubernetes.io
  http01: 
    ingress:
      name: "anything" # this value will be overridden by the 'http01-override-ingress-name' annotation

I'm happy to put a PR together if you want :)

[thockin]

That sounds like what we want, I think. PR would be welcome.

…

On Tue, Mar 17, 2020 at 6:33 AM James Munnelly ***@***.***> wrote: So if we add a new solver entry: # matches all Certificate resources - selector: {} http01: ingress: name: "anything" # this value will be overridden by the 'http01-override-ingress-name' annotation and then add acme.cert-manager.io/http01-override-ingress-name: "name-of-ingress-in-current-namespace" to the *Certificate* resource, we achieve what we after. It effectively applied a per-Certificate override to the clusterissuer.spec.solvers[].http01.ingress.name. We could also continue to make the 'solver' match on dnsZones too if we want to make it required that any new zone being added has to require a change to our ClusterIssuer (which required admins to approve) (i.e. still keep the selector containing dnsZones: ["k8s.io", "kubernetes.io"]), e.g.: # matches all Certificate resources - selector: dnsZones: - k8s.io - kubernetes.io http01: ingress: name: "anything" # this value will be overridden by the 'http01-override-ingress-name' annotation I'm happy to put a PR together if you want :) — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#643 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVGHRWMTE2HHITNTLADRH533JANCNFSM4LDNGVIA> .

[spiffxp]

/close
Discussed during today's meeting, will close and make sure docs include the reasoning for why certs are annotated the way they are

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Discussed during today's meeting, will close and make sure docs include the reasoning for why certs are annotated the way they are

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.