Loki gateway does not deploy on RKE2

[mjnagel]

Environment

Device and OS: ec2 instance, amazon linux 2023
App version: 0.6.2
Kubernetes distro being used: RKE2 1.27.x

Steps to reproduce

1. Attempt to deploy the example uds-core bundle on an RKE2 cluster.
2. Loki gateway fails to become healthy with the error:

   nginx: [emerg] host not found in resolver "kube-dns.kube-system.svc.cluster.local."
   

Expected result

Happy deploy!

Actual Result

The loki gateway is configured to point at the DNS service in kube-system. The default svc name it expects is kube-dns and RKE2 has a different name (rke2-coredns-rke2-coredns).

Severity/Priority

Medium? Completely blocks deployment after loki if the dns service is not named kube-dns, but with bundle overrides it could be changed.

Additional Context

While this could be done with a variable it would be annoying to have to pass in an environment specific value like this. Ideally a standard service could be created that maps to the DNS pods in cluster already?

[jbrewer3]

Dropping override example here for those hitting issues.

- name: uds-core
    repository: oci://ghcr.io/defenseunicorns/packages/uds/core
    ref: 0.10.0-upstream
    overrides:
      loki:
        loki:
          values:
            # Override default dns service name for Loki Gateway
            - path: "global.dnsService"
              value: "rke2-coredns-rke2-coredns"

[mjnagel]

Dropping a comment update here - it wouldn't fix the "automagic" nature of this but we could expose this Loki DNS value as a zarf helm override (defenseunicorns/zarf#2403).

[mjnagel]

Switching this to enhancement issue. I would like to tackle this by adding a standardized DNS service that we manage as part of the Loki uds-config chart. In reviewing this issue I noted a few things:

• Our generated DNS network policy assumes a standard remoteSelector, namespace, and port - we have not to my knowledge seen any issues with this
• kube-prometheus-stack sets up a standard DNS metrics service to monitor on that uses the same label by default

The path forward here:

• Create a service in kube-system named uds-loki-dns (or similar, just identify the purpose + ownership clearly via labels/name)
• Service should be a template in the uds-loki-config chart since its usage is currently attached to that
• Loki's default values should be updated to modify global.dnsService to point to that service

This should be a relatively seamless change, and for existing users overriding this they would be able to drop their override. Marking this as a good-first-issue, happy to converse with anyone who wants to pick this up.