Add Kubeapps setup for headless (API-only mode)

[castelblanque]

Description of the change

Adds a new chart flag ingress.apiOnly.enabled that allows to deploy Kubeapps with an ingress setup that only allows API calls to:

• /apis using http
• / using grpc

When the mode is enabled, the default ingress used so far is not deployed.
Instead deploys two ingresses:

• kubeapps ingress with path / and protocol grpc
• kubeapps-http-api ingress with path /apis

There is a requirement and enforcement message to disable the dashboard, as / path will not be available to serve it.

Benefits

• Kubeapps can be used in API-only mode:

  • REST calls can still be made
  • Also GRPC calls can be made, e.g.

  export APIHOST=<INGRESS_HOSTNAME>:<INGRESS_PORT>
  grpcurl $APIHOST kubeappsapis.core.plugins.v1alpha1.PluginsService.GetConfiguredPlugins

• It might also be beneficial for development, as there won't be the need to do port-forward to use GRPC.

• Does not change default behavior

Possible drawbacks

This forces the user to disable the dashboard, as it will be unusable when the API mode is on. We could remove the requirement, but users might complain when trying to access dashboard.

Applicable issues

• related to Introduce multicluster proxy mode for Kubeapps APIs #5468

[netlify]

✅ Deploy Preview for kubeapps-dev canceled.

Name	Link
🔨 Latest commit	dff2572
🔍 Latest deploy log	https://app.netlify.com/sites/kubeapps-dev/deploys/6358e679364a3d000855dde4

[absoludity]

Adds a new chart flag ingress.apiOnly.enabled

Should we keep any change to the chart like this to a featureFlags.newMulticluster.apiOnly and similar for now, so that people don't start using them and expect them not to change etc?

When the mode is enabled, the default ingress used so far is not deployed. Instead deploys two ingresses:

* `kubeapps` ingress with path `/` and protocol `grpc`

* `kubeapps-http-api` ingress with path `/apis`

I'm not sure, but do these need to be two separate ingresses rather than one ingress with multiple rules? (not sure).

There is a requirement and enforcement message to disable the dashboard, as / path will not be available to serve it.

I think the strategy here should be thought through or discussed a bit more. Here are a few questions just looking at the outline here:

1. How will we enforce TLS (as we do for the current inter-cluster multicluster communication) so that credentials are not sent in clear-text to the workload clusters (remember, you've not got oauth2-proxy in front on the workload clusters - I assume).
2. Will we be able to set this up in a dev environment (not a reason against it, but something we'll want to be able to do, but could be tricky if multiple ingresses are required - not sure). Maybe you're already doing this and testing it?

Generally, I'd recommend:

1. keeping this PR and Add Kubeapps API proxy mode #5529 in draft mode until you've got an example of the proxying working (without the header in the other PR). Sample instructions (or a Makefile target) would be handy to try it out etc.
2. only then, after review, we land these two PRs but clearly behind a featureFlag option as mentioned earlier, so it's clear this is not yet functionality to be used generally,

so that you can then continue developing and exploring behind that feature flag, without (1) affecting existing functionality and (2) accidentally releasing functionality before it's ready?

[ppbaena]

+1. Totally agree with @absoludity comment.

[castelblanque]

Should we keep any change to the chart like this to a featureFlags.newMulticluster.apiOnly and similar for now

This change is caused by multicluster, but it doesn't have to be always associated to it necessarily. If someone wants to deploy Kubeapps in API mode only, unrelated to multicluster, this is the way.
Is it ok if I use featureFlags.apiOnly.enabled ?

I'm not sure, but do these need to be two separate ingresses rather than one ingress with multiple rules? (not sure).

Yes, for Nginx we need two separate ingresses. In order to correctly handle GRPC requests, Nginx requires the annotation nginx.ingress.kubernetes.io/backend-protocol: GRPC which in theory will not work for http/s. It adds a bunch of GRPC-related confguration in the generated Nginx conf file.

How will we enforce TLS

We are dialing the connection with TLS transport, so I assume we have server-side TLS. See here. We are also supporting custom CAs.
Do you think we need mutual TLS in this case?

Will we be able to set this up in a dev environment

Why not? Ingress in second cluster is only accessed internally through kind network from the first cluster, so it can go to 443 as well. This is, the 443 opened port on the second cluster.
For the ingress, I will add the changes for developers to install Nginx in the additional cluster with a make goal.

keeping this PR and #5529 in draft mode until you've got an example of the proxying working

As mentioned above, this PR comes first and is totally independent from the multicluster feature. It just sets up Kubeapps in API only mode. There might be use cases that only want to invoke some APIs on Kubeapps. After enabling the flag, only GRPC and REST can be used, no UI. Why blocking this PR?

[absoludity]

This change is caused by multicluster, but it doesn't have to be always associated to it necessarily. If someone wants to deploy Kubeapps in API mode only, unrelated to multicluster, this is the way. Is it ok if I use featureFlags.apiOnly.enabled ?

Yeah, sure - as long as it's under featureFlags for now, I'm happy :)

Yes, for Nginx we need two separate ingresses. In order to correctly handle GRPC requests, Nginx requires the annotation nginx.ingress.kubernetes.io/backend-protocol: GRPC which in theory will not work for http/s. It adds a bunch of GRPC-related confguration in the generated Nginx conf file.

Cool - I'll be interested to hear how the nginx grpc config works. I thought from memory it required http 2 only or some slight difference which might make it tricky - hopefully not (there may be more details on the issue where we investigated running using envoy as a proxy, as I tried a few things with nginx for that too, from memory).

How will we enforce TLS

We are dialing the connection with TLS transport, so I assume we have server-side TLS. See here. We are also supporting custom CAs.

Yes - I just meant that with the current setup (where Kubeapps api server communicates with the k8s api server of the other clusters), we already have TLS setup and so just need to include the CAcert in our clusters config. That configuration part will be similar, but setting it up will mean we now have to create the certs or have cert-manager installed on the other clusters as well as some ingress support etc. All do-able, just quite complex to setup.

Do you think we need mutual TLS in this case?

No - I mean, I don't see a need for kubeapps-apis on the worker to need to identify the client, unless we want to ensure no one else can use the API. I'd leave that for the future, if at all, personally.

keeping this PR and #5529 in draft mode until you've got an example of the proxying working

As mentioned above, this PR comes first and is totally independent from the multicluster feature. It just sets up Kubeapps in API only mode. There might be use cases that only want to invoke some APIs on Kubeapps. After enabling the flag, only GRPC and REST can be used, no UI. Why blocking this PR?

If you've tested this PR by installing it in a cluster with nginx-ingress and gRPC setup and verified that you can use gRPCurl or similar to ensure it can be used the way you intend - great, I just thought it'd be easier to do that by verifying it works with our other PR, but I don't mind if you verify the external gRPC requests independently and land it. What I'd be keen to avoid is landing things on the assumption that it'll work before we've actually tested it in the way it's intended to work, that's all.

[castelblanque]

as long as it's under featureFlags for now, I'm happy :)

Flag is moved from ingress to featureFlags.

I'll be interested to hear how the nginx grpc config works

Based on the annotation nginx.ingress.kubernetes.io/backend-protocol: GRPC, the controller generates specific configuration, e.g. many grpc_set_header commands and a grpc_pass grpc://upstream_balancer; one.

You can see an example of how it works in Create the Kubernetes Ingress resource for the gRPC app, setting up an Nginx ingress for GRPC is pretty straight forward.

I thought from memory it required http 2 only or some slight difference which might make it tricky

I don't think Nginx is the one introducing the requirement for http2. As it is mentioned here: By design, the gRPC protocol cannot be transported over HTTP/1.x. The gRPC protocol mandates HTTP/2 in order to take advantage of the multiplexing and streaming features of an HTTP/2 connection..
As consumers for the GRPC ingress will be GRPC clients, I don't think there will be problems with protocols unless at the transport level.

we already have TLS setup and so just need to include the CAcert in our clusters config.

That belongs to the other PR, I believe. This one does not touch clusters config. It just sets an ingress setup for Kubeapps APIs only. I like to think about this forgetting about multiclustering. This feature could live without multicluster.
The only TLS certificate we need here is like for any other ingress.

Do you think we need mutual TLS in this case?

No - I mean, I don't see a need for kubeapps-apis on the worker to need to identify the client, unless we want to ensure no one else can use the API. I'd leave that for the future, if at all, personally.

Agree, no need for mutual TLS.

If you've tested this PR by installing it in a cluster with nginx-ingress and gRPC setup and verified that you can use gRPCurl or similar to ensure it can be used the way you intend

Of course I have verified it. It works just as any other ingress. Added a make devel/additional-nginx to help us developers with deploying Nginx in the additional cluster.

You can try yourself with:

1. Create a multicluster setup

   make multi-cluster-kind

2. Install Nginx in the additional cluster

   make devel/additional-nginx

3. Prepare installation of Kubeapps in the additional cluster

   kubectl --kubeconfig ~/.kube/kind-config-kubeapps-additional create ns kubeapps
   # Secrets setup (TLS, PostgreSQL)
   mkcert -key-file ./devel/cluster-key.pem -cert-file ./devel/cluster-cert.pem kubeapps-additional-control-plane
   
   kubectl --kubeconfig ~/.kube/kind-config-kubeapps-additional -n kubeapps create secret tls ingress-tls \
     --key ./devel/cluster-key.pem \
     --cert ./devel/cluster-cert.pem || true
   
   kubectl --kubeconfig ~/.kube/kind-config-kubeapps-additional -n kubeapps create secret generic postgresql-db \
   	--from-literal=postgres-postgres-password=dev-only-fake-password \
   	--from-literal=postgres-password=dev-only-fake-password

4. Install Kubeapps with the following values

   featureFlags:
     apiOnly:
       enabled: true
   dashboard:
     enabled: false
   ingress:
     hostname: kubeapps-additional-control-plane
     extraTls:
       - hosts:
           - kubeapps-additional-control-plane
         secretName: ingress-tls

   helm --kubeconfig ~/.kube/kind-config-kubeapps-additional upgrade --install kubeapps --namespace kubeapps --create-namespace ./chart/kubeapps \
   --values ./site/content/docs/latest/reference/manifests/kubeapps-local-dev-values.yaml \
   --values <VALUES_FILE>

Then the installation can be checked from a pod in the other cluster:

1. Run a temporary grpcurl pod.

   kubectl --kubeconfig ~/.kube/kind-config-kubeapps run tmp-grpcurl --restart=Never --rm -i --tty --image networld/grpcurl

2. From another terminal, copy the CA certificate file into the pod

   kubectl --kubeconfig ~/.kube/kind-config-kubeapps cp "$(mkcert --CAROOT)"/rootCa.pem tmp-grpcurl:/tmp/ca-auth.pem

3. Do the test from the pod

   # GRPC calls
   ./grpcurl -cacert /tmp/ca-auth.pem kubeapps-additional-control-plane:443 kubeappsapis.core.plugins.v1alpha1.PluginsService.GetConfiguredPlugins
   
   # REST calls
   curl --cacert /tmp/ca-auth.pem https://kubeapps-additional-control-plane/apis/core/plugins/v1alpha1/configured-plugins

[absoludity]

Perfect - thanks for the instructions for verifying it, that's what I was missing (or any indication that that had been done - glad it's obvious to you that it would have been, but it's not always the case).

I don't think Nginx is the one introducing the requirement for http2. As it is mentioned here: By design, the gRPC protocol cannot be transported over HTTP/1.x. The gRPC protocol mandates HTTP/2 in order to take advantage of the multiplexing and streaming features of an HTTP/2 connection.. As consumers for the GRPC ingress will be GRPC clients, I don't think there will be problems with protocols unless at the transport level.

Great. Yes the issue I'm remembering was related to gRPC-Web I think, so should be fine.

we already have TLS setup and so just need to include the CAcert in our clusters config.

That belongs to the other PR, I believe. This one does not touch clusters config. It just sets an ingress setup for Kubeapps APIs only. I like to think about this forgetting about multiclustering. This feature could live without multicluster. The only TLS certificate we need here is like for any other ingress.

Sorry - miscommunication: that sentence wasn't meaning that you need to add the CAcert to the clusters config in this PR, but rather that that part will remain similar to what it is today, but the extra manual certs will be extra complexity, that's all:

That configuration part will be similar, but setting it up will mean we now have to create the certs or have cert-manager installed on the other clusters as well as some ingress support etc. All do-able, just quite complex to setup.

Anyway, +1 for this one, thanks for the details of the setup and testing.

4. Install Kubeapps with the following values
```yaml
featureFlags:
apiOnly:
enabled: true
dashboard:
enabled: false

I think we currently have enabled on each service, but seems here it'd be better not to have that option on dashboard and instead update whether the dashboard is enabled as a function of the apiOnly flag only? Fine either way, just a thought.

[absoludity]

Great, thanks Rafa!

[absoludity]

Great - thanks for catching that even if we don't have a single value flag. Do we need a similar check for the condition on line 1, in that, does it make sense for people to deploy with apiOnly.enabled: true while ingress itself is disabled?

[castelblanque]

This feature is basically about Ingresses. Initially was under .ingress chart value, and it is documented with Enable ingress for API operations only, but I'll add a check for that.

[absoludity]

Just wondering why this one is configurable - won't our own clients need this specific annotation, even if others are optional/configurable? I expected this one to be hard-coded into the annotations of the ingress. Ah, or perhaps you're just separating that dependency here (ie. other clients may want other annotations). Related question, why does the default in the README show up as {} ?

[castelblanque]

you're just separating that dependency here

Exactly. There is no trace of Nginx in the templates of ingresses. It is only through annotations that it is customized, hence this value.

[castelblanque]

why does the default in the README show up as {} ?

No idea, it is generated by the readme generator, so changing it manually is not an option :S

[absoludity]

Yes, we do that in our cluster-kind target which installs ingress-nginx:

kubeapps/script/makefiles/cluster-kind.mk

Lines 28 to 31 in e571216

	kubectl wait --kubeconfig=${CLUSTER_CONFIG} --namespace ingress-nginx \
	--for=condition=ready pod \
	--selector=app.kubernetes.io/component=controller \
	--timeout=120s