[bug] hostname mismatch error when using a helm repository URL that redirects to another URL in a cluster behind a proxy using self-signed certs

[guruprasadc6]

Describe the Bug

We have a Kubernetes cluster behind a mitmproxy running in transparent mode that uses self-signed certs.
Created a HelmRepository CR of version source.toolkit.fluxcd.io/v1beta2 and a secret to refer to the proxy certificate.

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: bitnamitest
  namespace: default
spec:
  secretRef:
    name: https-ca-credentials
  interval: 10m0s
  provider: generic
  timeout: 60s
  url: https://charts.bitnami.com/bitnami

apiVersion: v1
kind: Secret
metadata:
  name: https-ca-credentials
  namespace: default
type: Opaque
data:
  caFile: <certificate data>

The URL https://charts.bitnami.com/bitnami/index.yaml redirects to https://repo.vmware.com/bitnami-files/index.yaml.
The source controller fails to reconcile the HelmRepository with the error failed to fetch Helm repository index: failed to cache index to temporary file: failed to fetch https://charts.bitnami.com/bitnami/index.yaml : 502 Bad Gateway
On the mitmproxy, we are seeing the error Certificate verify failed: hostname mismatch

When we dug deeper we saw on line(in the latest version of the code)

source-controller/internal/helm/getter/getter.go

Line 102 in af854cf

tlsConf.ServerName = u.Hostname()

the tlsConfig.ServerName is populated with the hostname of the helm repo URL. For the redirect request(https://repo.vmware.com) the source controller uses the same tlsConfig with ServerName set to the original repo URL hostname (https://charts.bitnami.com), because of which the hostname mismatch error is happening.

Source Controller Version
1.1.0 (latest)

[darkowlzz]

Hi, I'm assuming that the certificate that you're setting here is of the proxy server and not of the upstream chart index host. We don't have support for proxy configuration in HelmRepository API. The recommended way at present is to set the proxy configuration in the environment variables, refer https://fluxcd.io/flux/installation/configuration/proxy-setting/#using-https-proxy-for-egress-traffic.
The ServerName being set in the TLS configuration is the same as what upstream helm does, refer https://github.com/helm/helm/blob/v3.12.3/pkg/getter/httpgetter.go#L136 for setting SNI. If you don't explicitly provide TLS configuration in the HelmRepository spec, this code won't run and no explicit cert will be configured in the http client. The http client would use the container's default certs from /etc/ssl/certs to establish TLS connection, which already contains Amazon and DigiCert root certs that are needed for Bitnami and VMware, respectively. To bypass this, you can set skip verify TLS in helm CLI but that's not an option in Flux HelmRepository API yet. There's this issue #957 to track its support. Since the upstream blocker got resolved recently, I'd expect it to be implemented soon for HelmRepository.

Since bitnami started publishing OCI helm charts, just as an experiment, you can try pulling an OCI helm chart using the OCIRepository resource, which has a .spec.insecure field. Something like

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: bitnami-common
  namespace: default
spec:
  interval: 1m
  url: oci://registry-1.docker.io/bitnamicharts/common
  ref:
    tag: 2.9.1
  insecure: true

And then check if the object is ready in its status. Any failure would also be reported on the object status.

Although this won't allow you to install the helm chart, if this works for you, HelmRepository OCI with insecure support will also work for you whenever that gets implemented.

Hope this helps.