-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CSI: EBS plugin requires topology feature #10891
Comments
Hi @m1keil and thanks for the report. I'll need a little time to dig into this; after which i'll come back to you. |
Cheers. If there are any logs or other debug info I can provide let me know. |
I can confirm this is still the case and its more about the csi plugin itself than nomad:
This places 1 controller on each availability zone with a plugin id of Then when i create a volume:
Then create the volume:
The plugin that takes this request is on
The aws-csi-ebs-plugin (https://github.com/kubernetes-sigs/aws-ebs-csi-driver) does not have a parameter exposed to set the availability zone when creating a volume request, nor does the controller have an argument to limit itself to a particular AZ
Nomad does nothing with the struct that contains |
Hi @m1keil! Yeah, as @mrproper has noted this is a behavior specific to the AWS EBS plugin. We have a note about this kind of thing (which calls out that plugin specifically) in the
|
This shouldn't be closed. the ebs plugin already allows for locking a a storage class to particular availability zones: https://cluster-api-aws.sigs.k8s.io/topics/external-cloud-provider-with-ebs-csi-driver.html |
@tgross I'm aware of the documentation (I think I'm the reason it was added :))
The setup I have follows the directions as instructed and yet the volume isn't being created where I execute the create command. |
Hm... @m1keil can you provide the output of |
What az am I on?
Agent CSI info:
Plugin job (nodes):
Plugin job (controller):
Lets create the volume:
Show volume
|
Ok, so the controllers certainly seem to line up on the right AZs:
I know our E2E tests don't run into this issue but they're also running an older version of the plugin ( Is the plugin fingerprinting its availability zone correctly? What does the controller plugin for an AZ say in its logs (both stdout and stderr) when it comes up? |
Yes, fingerprinting seems to be fine. Logs:
|
Now I've upgraded to the latest v1.4.0 controller and ran the create command again. Now logs show this:
but this is being run on a node located in As a final test, I went and shut down the controller in This seems to be an identical issue and indeed related to topology support. |
Thanks @m1keil. Looks like that's the solution then. It hasn't been on our immediate roadmap but if it blocks a common usage like this that's something we'll want to bump up in priority. |
I'm actively working on #7669 which will resolve this issue. |
Resolved by #12129, which will ship in Nomad 1.3.0 |
Thanks, I'm planning to upgrade and test once this is released. Will report back. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.1.1 (7feec97c04de4f8afff54ca9e56d66a61dfbfeb3)
Operating system and Environment details
AWS VPC with 3 subnets, each in a separate availability zone. We run CSI EBS controller on every AZ (3 in total). Every node plugin job is configured to run against the suitable controller in the same AZ.
Issue
When using
nomad volume create
, the volume is not created in the same availability zone that the plugin is running at.Reproduction steps
Assuming our plugin ids are:
aws-ebs0-eu-west-1a
,aws-ebs0-eu-west-1b
andaws-ebs0-eu-west-1c
, save the following config involume.hcl
and executenomad create volume volume.hcl
.Expected Result
Volume created in the eu-west-1c availability zone.
Actual Result
Volume constantly being created in the eu-west-1a availability zone. I'm not sure why it's always created by that controller.
The text was updated successfully, but these errors were encountered: