-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feedback on Nomad CSI #7
Comments
Thanks for taking a look @scaleoutsean!
|
That's helpful, thank you.
I went with that assumption (and a few others for other steps, which made troubleshooting harder due to several assumptions at once) and I tried to leave connAuth empty, but later I also configured connection authentication as I was trying to see if the lack of it was causing my problems (it seemed it wasn't). Which brings me to this part of the page you linked above:
That seems to refer only to values in YAML files and doesn't apply to the secret string in the connAuth file (sample here) because that's part of a template file used for configuration and not an argument passed to beegfs-client binary. But as I just discovered, that's not the case: surrounding values in plugin.nomad's docker template (connAuth, connUseRDMA) with double quotes seems to have helped and now the plugin works. So that may be a bug in reference plugin.nomad file. And indeed, volume.hcl (from the example) indicates that quotes should be used. I still can't create a volume (getting |
Interesting! This commit added plugin.nomad on 08/05/21, which was tested in a Nomad environment we had set up at the time. plugin.nomad hasn't substantially change since then. This commit changed the YAML parser the driver uses on 11/20/21. Both csi-beegfs-config.yaml and csi-beegfs-connauth.yaml are parsed in the same way, so the warning applies to both. When that commit went in, we updated all YAML files in the project, but it looks like we missed the Nomad templates (as you found). As far as I understand (and remember) only values that might otherwise be interpreted as non-strings must be quoted. In the PR you submitted, it makes sense to me that "true" should be quoted. In my mind, this is what was causing the issue. I do not think "1.1.1.1" or "secret1" need to be quoted, and I would be curious whether the driver would run without. |
You're right, it works with only the boolean value surrounded by quotes. But:
The details from bullet two are in another file not changed in my PR, so if you want to selectively surround boolean values with quotes that's fine, the PR allows "minor edits from maintainers" but if you make edits I would suggest to update the other example and that note (can be done in a separate PR by you) as well to minimize confusion for users. Since I'm already editing this comment, I will add that yesterday I was thinking about making a suggestion to move /deploy and /examples into /docs and have everything under /docs publish-able to GH pages or elsewhere. I didn't mention that because it's not a big pain point now, but if additional changes are made to the docs, we could use the opportunity to do that. |
Any group of characters beginning with an alphabetic or numeric character in a YAML file is interpreted as a string unless it belongs to a special group (like integer, boolean, time, etc.). Additionally, any value can be forced to be interpreted as a string using double quotes. My intention with that beegfs-client.conf comment was to remind users to force special values in the config section to strings (because the Kubernetes YAML parser, for whatever reason, will error out instead of unmarshalling true to "true" in a map[string]string). I didn't intend it to mean, quote everything, but quotes around everything certainly doesn't hurt! |
We'd like to incorporate your PR into the upcoming 1.2.2 release. Our current process does not allow us to merge PRs directly on GitHub, but we can pull the commits in, test them in our infrastructure (which currently doesn't include a Nomad deployment, so it's just a formality), and include them when the release goes live. It'd be best if we did that on a fully working example though. Hopefully we can get to the bottom of the remaining issue. To that end, if there is additional commands or output you can share, I'd be happy to try and help troubleshoot. |
Do you mean for the next step (volume create)? Sure, I haven't been able to figure that one out.
There's no log file about the config file being deleted, so I assume it should be there if it was created but that doesn't seem to be the case. I haven't looked at the source to see if deletions are logged as well - if they're not, maybe it'd be good to add that so that we know if the file was possibly created but quickly removed. The volume workflow from this repo uses
I've tried several different things, no luck. I can't see anything important in BeeGFS Mgmt server logs - I assume that's because that config map can't be loaded so nothing gets sent its way when |
Another feedback - it's a little hard to tell when plugin status is good by looking at the CLI output ( In this particular case volume doesn't get created so we know there's something wrong, but until that point it seems hard to tell. For example, I can enter SysMgmtHost IP as 1.1.1.1 (edit: I mean in plugin.nomad) and the plugin will show as healthy in However, when I enter 192.168.1.195 (own IP of the Nomad Server / Nomad Client / BeeGFS Client), then the plugin does show in the Web UI. That may be due to its monolithic nature, but it also indicates that those health checks may not be reliable. I wonder if a check command of some sort (maybe even just manually executed curl commands) could be used to test whether a temp volume can be created and deleted. |
For the "another feedback", it sounds like things are working as expected here. The configuration options provided to the driver in csi-beegfs-config.yaml csi-beegfs-connauth.yaml are used as needed. If a volume is created that references the file system with sysMgmtdHost 1.1.1.1, the driver will use the configuration associated with that file system. If no volume is ever created referencing the file system with sysMgmtdHost 1.1.1.1, the driver never uses that configuration, and simply having that configuration does not constitute an error. Of course, the reverse is also true (and this is one of the main reasons the driver was designed to work this way). You can specify absolutely no configuration and call out the sysMgmtdHos of no specific file systems and still create volumes referencing arbitrary file systems. As long as those file systems don't NEED special configuration, the driver can still mount them. |
The volume creation issues are difficult to troubleshoot without a working Nomad environment, but I can at least give you a bit of context. The driver writes client configuration files to the configured directory directly (e.g. When the driver executes beegfs-ctl, it uses chroot to make this execution happen in the host namespace. This allows it to use the beegfs-ctl utility already installed on the host (instead of shipping one). Note that this is even more important for the node services NodeStageVolume command, as mount calls must reference a client configuration file in the host's namespace. We take care in both Kubernetes and Nomad to ensure that the driver and the host both see the client configuration files as having the same path. That way, both the driver and the host can refer to them at that path. In Kubernetes, we accomplish this with a bind mount:
There is an implicit assumption in plugin.nomad that this That being said, the logs tell a different story. Since there is no error on directory creation, we can safely assume that the driver creates the configuration directory in its mount namespace. Since there is a failure on the part of beegfs-ctl to load the map file, I suspect that the host has a different view of what is or isn't contained in the same directory. This could be the result of some Nomad change since the version we tested with or an overlooked detail. (That "cleaning up path" message on log line 6 is an indicator that the client.conf file is blown away after the failure, so I wouldn't expect to be able to find it after the fact.) |
That's a good argument in favor of current approach. It also lets us configure CSI before storage is ready and leave the configuration in place during storage maintenance or downtime. But I still wonder if at least some warning i(if not outright error status) should be emitted to alert the user can tell that plugin cannot access SysMgmt host. While in a large cluster there may always be some worker(s) that can't access SysMgmt IP, it would be useful to know that before the problem bubbles up to applications.
I'm also willing to consider I may have made some incorrect assumptions (as I mentioned, I'm not 100% sure I didn't, so I'm open to rechecking other details or offering additional information about my environment). I looked the the bind thing and also this note which made me leave beegfs-client.conf in place, although my Ansible scripts deployed beegfs-client-b1.conf (b1 is hostname of SysMgmt worker). But I copied that file to beegfs-client.conf for BeeGFS CSI to load, so these have identical content. Distroless-based containers don't have any shell, otherwise it'd be easy to get in and find out what's wrong. I had to implement a proven technique that I used in the 90's whenever Windows 3.1 had issues finding correct DLL files... $ sudo mkdir -p /opt/nomad/data/client/csi/monolith/beegfs-plugin0/192.168.1.191_mnt_beegfs_dyn_VOLUME__NAME/
$ sudo cp /etc/beegfs/beegfs-client.conf /opt/nomad/data/client/csi/monolith/beegfs-plugin0/192.168.1.191_mnt_beegfs_dyn_VOLUME__NAME/ Then
This isn't how it's supposed to work, but I may be able to continue testing other things. Note that Access Mode and other stuff in
|
If you could run the plugin.nomad, then capture a |
Sure!
|
I am actively reworking our Nomad support now (though only in my spare time for the moment). I have a Nomad cluster up and have started to work through the issues you experienced. As best as I can tell, there HAVE been changes to the way Nomad handles CSI paths since our original implementation, and, as I guessed above, the unique need for our driver and the host to agree on the full path to configuration files is causing problems. I fixed your initial CreateVolume issue with a new bind mount (/opt/nomad/client/csi/monolith/beegfs-plugin0:/opt/nomad/client/csi/monolith/beegfs-plugin0) that allows the controller service (running in the container) and beegfs-ctl/mount (outside the container) to agree on the location of beegfs-client.conf. There were some additional code changes required to make this workable, so I don't recommend trying it with a v1.2.2 container. The node service issue you ran into is a bigger challenge. Nomad is now bind mounting /opt/nomad/client/csi/monolith/beegfs-plugin0:/local/csi automatically for CSI drivers and providing staging_target_paths in NodeStateVolume like /local/csi/.... (For what it's worth, Kubernetes provides absolute staging_target_paths like /var/lib/kubelet/plugins/beegfs.csi.netapp.com, which are much easier for us to deal with). Many drivers don't care, as all userspace utilities run inside the driver container and have a synchronized view of the file system. However, we choose not to package beegfs-ctl and the core utilities in our container. We're talking through fixes, but we may need something like an additional command line argument that helps the driver modify the staging_target_paths it is provided before attempting to mount. |
We have identified a twofold path to get Nomad working again.
|
The changes we introduced in v1.3.0 addressed the known Nomad issues. Additional cleanup is being done in v1.4.0. I'm going to go ahead and close this one for now, but please feel free to open a followup issue with additional feedback if/when it makes sense. |
"--endpoint=${CSI_ENDPOINT}"
so maybe that will simplify things somewhat). This is what I get no matter how I try to change the paths. I'll try to take another look at this another day.node-id
(seen here) seems to benodeid
in more recent config examples by HashiCorp staff.It's not entirely clear from the instructions if driver has to be deployed on a host (worker) with a functioning BeeGFS client or not. For example,
connAuth:
in plugin.nomad: if BeeGFS cluster isn't using connAuth, can that section be left out from the plugin job? Or will using the plugin break BeeGFS client if that's left out (or if it's not left out)? I know I'll find out after I get there, but it should be clear before because many may be concerned if plugin can interfere their BeeGFS clientbeegfs-dkms-client uninstalls beegfs-client (v7.3.0) which is dangerous. If that package isn't necessary, let's not suggest to install it
The text was updated successfully, but these errors were encountered: