-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nomad node status doesn't return in use CSI volumes #17923
Comments
Hi @the-nando 👋 I think your analysis is correct, and we're missing the But for you get an empty result is a bit strange 🤔 I don't think that's the problem, but have you tried calling the API with a management token just to see if the problem is not related to permissions? |
Hi @lgfa29, I'm using a management token with the curl. If I try without I get a #!/usr/bin/env bash
total=0
while read -r NODE; do
node_id=${NODE% *}
volumes_count=$(nomad operator api -X GET '/v1/volumes?type=csi&node_id='${node_id}'&namespace=*' < /dev/null | jq length)
[[ "${volumes_count}" -gt 0 ]] && echo "Found ${volumes_count} volumes on ${NODE}"
total=$((total+volumes_count))
done < <(nomad operator api -X GET '/v1/nodes' | jq -r '.[] | "\(.ID) \(.Name)"' )
echo "Total: ${total}" Sample output, with NOMAD_ADDR pointing to the active leader:
Would you have the possibility to test this? I'm looking at setting up a local environment but due to the CSI driver / controller requirements it's a bit involved. |
That's very strange 🤔 Are your allocations stable? Looking at the code I see that only volumes for allocations that are currently running are returned: nomad/nomad/state/state_store.go Lines 2568 to 2572 in e53955b
|
The allocations are stable and I get the same results in different federated regions and environments. I don't actively use that API endpoint but I'll see if I can find why the odd results, thanks so far! |
Would it be possible for you to provide us with a full list of volume results per node instead of an aggregate count? |
Hey @lgfa29 I've just opened a support request and shared the logs there :) |
Nomad version
Issue
A call to
nomad node status -verbose <node_id>
for a node with an allocation using a CSI volume running on it, shows only the header for the CSI Volumes:Volume status output:
Reproduction steps
nomad node status -verbose <node_id>
Expected Result
nomad node status -verbose
shows all CSI volumes in use on the node.Actual Result
No volumes are displayed for the node.
I believe one of the problems is that the call to
/v1/volumes
doesn't setnamespace=*
as query params, but there seems to be more to it as setting it still doesn't return the volumes.I also get an empty list if I hit the API directly with a curl:
The text was updated successfully, but these errors were encountered: