[BUG] k3s proxied downstream cluster does not work on v1.24.4+k3s1 but does work on v1.24.6+k3s1

[slickwarren]

Rancher Server Setup

• Rancher version: v2.6-head (e139976)
• Installation option (Docker install/Helm Chart): docker
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): n/a
• Proxy/Cert Details:n/a

Information about the Cluster

• Kubernetes version: v1.24.4+k3s1
• Cluster Type (Local/Downstream): downstream
  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): custom , proxied cluster

User Information

• What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)
  • If custom, define the set of permissions: tested with standard user, cluster owner and as admin

Describe the bug

k3s proxied downstream cluster does not work on v1.24.4+k3s1 but does work on v1.24.6+k3s1

To Reproduce

• deploy proxy for use in the downstream clsuter
• in rancher, deploy a custom cluster and specify HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables and use the latest k3s version available
• connect a node to the custom cluster that is able to reach the proxy you created in an earlier step.

Result
node makes an initial connection, but never gets to an active state.

Expected Result

node should be able to register with rancher from behind the proxy

Screenshots

Additional context

• found when testing: [Backport v2.6] [BUG] rke2/k3s proxied custom cluster unable to finish rancher-system-agent with Main process exited, code=exited, status=2/INVALIDARGUMENT #39068
• does not happen on 1.24.6+k3sr1

rancher-system-agent logs:

level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"
rancher-system-agent[2790]: time="2022-10-12T18:09:47Z" level=info msg="[82a072010f1716ef0f3c866533fcbef07010c7ce199cc7c51a6d16b7d20dce0a_0:stdout]: Name Location Size Created"
rancher-system-agent[2790]: time="2022-10-12T18:09:47Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"

k3s logs just keep spamming this message:

k3s[2936]: I1012 18:50:27.610159    2936 scope.go:110] "RemoveContainer" containerID="c9617aee67a6aae3a26ef124208a83bfa2dbc9029b0dbb0b570389a4eb9000e0"
k3s[2936]: I1012 18:50:36.190489    2936 scope.go:110] "RemoveContainer" containerID="c9617aee67a6aae3a26ef124208a83bfa2dbc9029b0dbb0b570389a4eb9000e0"
k3s[2936]: I1012 18:50:36.191032    2936 scope.go:110] "RemoveContainer" containerID="704d2ab8b106ef5fb27b04e75341bfcc47a258dbb15649986c51a00f4db90a22"
k3s[2936]: E1012 18:50:36.191476    2936 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cluster-register\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=cluster-register pod=cattle-cluster-agent-6f55b75c86-ppcq6_cattle-system(94520d8a-72e0-427f-b45b-b6842edeeebc)\"" pod="cattle-system/cattle-cluster-agent-6f55b75c86-ppcq6" podUID=94520d8a-72e0-427f-b45b-b6842edeeebc
k3s[2936]: I1012 18:50:50.609819    2936 scope.go:110] "RemoveContainer" containerID="704d2ab8b106ef5fb27b04e75341bfcc47a258dbb15649986c51a00f4db90a22"

[Sahota1225]

proxy is broken and a fix will be available with a later version

[Sahota1225]

@slickwarren can you please test this with 1.24.9 and see if the issue still exist?

[snasovich]

@slickwarren , moving it "To Test" to check the above. If it's not reproducible on 1.24.9 (the latest to be in 2.7.x), let's close it. We're not going to fix 1.24.4 retroactively.

[slickwarren]

confirmed, this is working on v1.24.8+k3s1. will close this issue