-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad send wrong target_path to CSI plugin during read_only volume unpublish #9259
Comments
Even if I remove |
If it helps: My plugin does not support volume staging, it just uses publish/unpublish |
@apollo13 can you share the plugin job file and the Nomad debug logs for the client around the time of mounting? |
The plugin job:
debug logs (stripped the status/leader query):
Btw one last time (I hope I keep remembering it then). Those logs are generated via |
Ok, I think I figured out what's happening here and it's definitely not just your plugin. In 0.12.2 (in PR #8580 ) we made a change to fix a bug during node drains. The workflow changed slightly so that the client sends that unpublish RPC to the server and then it fires the node unpublish back to the client once it's figured out whether is needs to do anything with a controller. But there's a bug in the logic where we've effectively erased the "ro" information that's available for the claim when we send it up. It shouldn't be a huge lift to fix so I'll make sure to get that into the next 1.0 beta release. Interestingly, earlier today I was doing some end-to-end testing for #9239 and was finding unexpected behaviors during unpublish that aren't getting picked up by our E2E suite. So I think I have the test case all ready to go! |
Thank you for the investigation, this is great to hear. Between the issues I run into and possible bugs in the CSI plugin it is often hard to determine for me what is at fault :) |
Understood on that. Thanks for trying this all out -- having someone using Nomad who's working on a custom plugin is a great way for us to find lots of issues in the CSI beta! |
The fix in #9323 will ship in 1.0-beta3 later this week. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
0.12.5
Issue
I could not find anything in the 1.0 changelog, so I am reporting this here. I have my own CSI plugin which gets a volume registered:
Now when I use the volume in a job like this:
I get the following logs (from my plugin):
As you can see in the logs
NodePublishVolume
got called with a patch containingro-file-system-multi-node-multi-writer
whileNodeUnpublishVolume
gets called withrw-file-system-multi-node-multi-writer
(note the ro/rw asymmetry there). Now I do realize that my job file might be buggy (specifyingread_only
twice, once forvolumes
and once forvolume_mount
), but I think nomad should still send the path it send for mounting also during unmount.The text was updated successfully, but these errors were encountered: