New to Flux, k8s -- getting fluxctl Error: No pod found in namespace "flux" using the following selectors

[crb-66]

I'm new to K8S an Flux and have recently installed both as part of a dev platform. No fluxctl commands are working for me, all are returning:

$ fluxctl --k8s-fwd-ns=flux list-images
Error: No pod found in namespace "flux" using the following selectors:
app=flux
name in (flux,fluxd,weave-flux-agent)

Make sure Flux is running in namespace "flux".
If Flux is running in another different namespace, please supply it to --k8s-fwd-ns.
Run 'fluxctl list-images --help' for usage.

However I can list the pods in the 'flux' namespace with kubectl and clearly see one with the 'app=flux' label:

$ kubectl get pods -n flux --show-labels
NAME READY STATUS RESTARTS AGE LABELS
flux-XXXXXXXXXX-xxxxx 1/1 Running 0 3h54m app=flux,pod-template-hash=XXXXXXXXXX,release=flux

I've tried fluxctl versions 1.23.1 and 1.24.1.

Other users running the same fluxctl version against the same cluster do -NOT- get the same error and fluxctl is working for them.

[kingdonb]

Based on a quick read of the source I found that --k8s-fwd-ns works by port forwarding and sending a connection to the Flux daemon in the flux deployment's pod. If you cannot port forward then fluxctl will not work.

This is the "Flux API" of Flux v1. In Flux v2, there is no API at least not in the sense of RPC that you reach by an ip and port. So in Flux v2, there is no port forward necessary, you create API Resources in K8s as you would create any other resources (likely via GitOps), there's no server or availability, only Kubernetes control plane. (This is probably not interesting to you.)

How does your user get created, and can you compare your permissions to the permission of someone whose account is working acceptably?

$ kubectl auth can-i create pods/portforward
yes

I am not super familiar with these RBAC roles and how this works but here is an example I found that may help:

https://medium.com/@ManagedKube/kubernetes-rbac-giving-permissions-for-logging-and-port-forwarding-882694c91927

Please consider upgrading to Flux v2! The fluxctl CLI is for Flux v1, which is in maintenance mode and has many disadvantages compared to the latest Flux.

[kingdonb]

If there is a particular feature in fluxctl that is keeping you on Flux v1, please make sure to file an / comment on an issue somewhere in the Flux v2 orbit of repositories, wherever it makes the most sense. We are very interested in feedback from Flux v1 users who have found any reasons not to upgrade.

Flux v2 should be at feature parity since quite some time (around 0.8 or 0.9?) -- the experience keeps improving, a roadmap covers many ways in which the latest version of Flux matches and improves on the legacy Flux v1: https://fluxcd.io/roadmap/

The biggest reason to upgrade is that Flux v1 in maintenance mode means eventually it won't be supported anymore:

• Flux v1 is in maintenance mode #3320

Hoping that with your feedback and help, we can bring everyone along when that horizon arrives! We also offer migration assistance, including workshops that are sponsored by the FluxCD org and in no way tied to any company or service contract. If you are interested in upgrading but maybe just need a little help or encouragement, please let us know that too.

[crb-66]

@kingdonb kubectl auth can-i create pods/portforward returns 'yes' for me. Also my permissions are identical to the other user for whom fluxctl commands work -- we're both AWS Power Users. Also I have successfully set up port forwarding for other purposes in the same K8S cluster.

I understand v1.x is deprecated and we have plans to upgrade to v2, but for organizational reasons it probably won't happen for ~6 mos. or so. Appreciate the update on v2 and feature parity.

[kingdonb]

That's odd. I'm not sure what else could get in the way. The Flux v1 API is basically not authenticated (it assumes the cluster CNI or VPC address of each pod is private, which is generally safe as an assumption but not the same as authenticating.)

The only other thing that I can think of which might be in the way is an oddly defined kubeconfig?

You might have set a value for FLUX_FORWARD_NAMESPACE or FLUX_FORWARD_LABELS – I'm not sure if that would take precedence over the command-line flag --k8s-fwd-ns or --k8s-fwd-labels but I wouldn't think so.

I almost suspect it's a problem on your local workstation, since none of the other ideas I listed above sound likely to me. Maybe there is already a conflicting service listening / something blocking you from opening a local listener port-forward? I'm reading the code trying to understand portforwarder.ListenPort and where the listener port is assigned. It looks like it picks a free port dynamically, so it is unlikely that you have managed to have a conflict, unless there is something blocking ALL listeners from listening.

So you are authorized by kube API, but can you port-forward successfully in general using kubectl and reach services on those ports from your workstation?

(It honestly doesn't even seem like it gets that far, with Error: No pod found in namespace "flux" using the following selectors)

Do you have any other environment variables set that might be overriding your expected kubeconfig or fluxctl settings?

[kingdonb]

Haven't heard back in a while, closing. Feel free to reopen if needed.

[challamani]

does fluxctl works for flux v2 ? I'm encountering some sort of problem.

fluxctl list-workloads --k8s-fwd-ns=flux-system --namespace=hello 

Error: No pod found in namespace "flux-system" using the following selectors:
app=flux
name in (flux,fluxd,weave-flux-agent)

Make sure Flux is running in namespace "flux-system".
If Flux is running in another different namespace, please supply it to --k8s-fwd-ns.

[pjbgf]

@java-manikanta for Flux v2 you should use the flux CLI instead. Here's how to install it: https://fluxcd.io/docs/installation/#install-the-flux-cli

[challamani]

Thank you @pjbgf - it helped