-
Notifications
You must be signed in to change notification settings - Fork 95
compose yml with datastore selection #1315
Comments
@nbaillie Thanks for reporting issue. Not able to specify |
@nbaillie The following is a workaround:
May I know why you need to specify the datastore name explicitly? By default volumes are created on the same datastore as VM. Thanks! |
Thanks for the workaround, and I'm happy to help catch this. We have the VMs Primary Volume on a vVol datastore and want to have the Docker volumes not on vVol as our storage layer has better support and features for non-vVol datastore additionally we are looking at using different performance tiers of storage for certain volumes, Logs,db,archives |
this is also necessary if your worker nodes are distributed over several datastores.. If you don't specify the datastore and the container fails over to a worker that is stored in another datastore, the container can't have access to its persistent data.. |
Tracking issue with docker/compose #4886 |
docker/compose #4886 was closed. Filed a new issue with moby/moby #33529. |
Is it possible to have the datastore as a driver_opt as well? Just an idea, and it means we don't have to worry about special characters in the volume name. |
@madjam002 Thanks for your suggestion. We are trying to figure out the best approach to fix this issue and having the datastore as a driver_opt will be one approach we will evaluate. |
@madjam002 @nbaillie - I believe we already support default_datastore configuration (with a few caveats), in experimental multitenancy support and related admin To define default datastore (and many other things like access control or size quotas) you would need to init the configuration, and then configure the desired default-datastore for vmgroup Assuming the desired name is vsanDatastore (can be any shared DS) , the commands on ESX look as follows:
Note that at any moment you can use
CAVEATS for config on a shared datastore:
Or, you can use local configuration:
(or you can configure it once and then copy No conflicts here, but unfortunately the local config won't survive ESX reboot, you'd need to
|
We have provided workaround to customers. To fix this issue completely, we may need to fix docker code. Pushing to next release for now. |
@shaominchen Is there any documentation on the workaround that is available? Cheers |
@madjam002 If you can pre-create the volume, you can follow Liping's comments above: #1315 (comment) If you don't want to pre-create the volume, you can follow Mark's comments above: #1315 (comment) We have filed the issue to Docker: moby/moby#33529. If this issue cannot be addressed from Docker, we will provide a document for above workarounds. Hope this helps. |
@shaominchen Unless I'm missing something, this doesn't allow you to create volumes on various different datastores all from the same compose file? So it's not 100% a workaround. For example, take a container that needs both a volume on an SSD backed datastore and a slower magnetic datastore. |
@madjam002 , if you need to create volumes from different datastores from the same compose file, then setting the default_datastore for a vmgroup does not help here. Can we ask why the pre-create volumes does not work for you? I filed a moby issue moby/moby#33529 and has some discussion with docker engineer there. They asked why using the external volume in compose file (which means you pre-create the volume) does not work. Is there any specific reason that using external volume in compose file not work for you? You can also update your input in that moby issue. Thanks! |
@lipingxue Pre-creating volumes kind of defeats the purpose of using the Compose file format in the first place - defining all of the resources which should be spun up in one place. But it is a workaround for now I guess. |
It is a temporary band aid only, indeed. @lipingxue - what can we do to accelerate the actual solution ? |
moby/moby#33529 is closed and another issue docker/cli#274 in docker/cli repo is created to track this. I have created a PR docker/cli#306 for this. |
This has been fixed through docker/cli#306, and will be available on docker 17.08.0. |
@lipingxue We need to add test when Docker 17.08.0 is available. Do you want to file a new issue or reopen this for tracking purpose? |
#1666 is filed to track add test for this. |
I am confused as to why this issue would be closed. I had to literally search all the CLOSED issues to find this reference to this issue and it is absolutely a bug! When compose supports but then vomits and cannot simply parse the fully supported default naming format that the fully supported vsphere volume driver uses, and that docker itself uses and itself displays when issuing a "docker volume ls" - this should be made a much higher priority to resolve. If the darn "@" is "optional" (and I question through my tests if it really even is), and the other half of the docker tools vomits and throws errors when that "optional" half of the name is used........my 30yrs as a dev says perhaps you either fix the half to recognize the "@" 'cause it's here to stay.......or buh-bye Mr. "@" we can't support that with half our code, so we're going to remove that since it's lingering in the "optional" world right now, anyway. :) My testing: I have upgraded docker to: 17.09.0-ce-rc3 ...and after an hour of attempts at every known syntax including referencing an external name in my compose yml files, I cannot simply add an external vmware vsphere volume with that "@" and run compose without error, so this should really be re-opened until someone can confirm via tests that it has been addressed or clearly document a valid workaround. |
I wanted to post my current (I have tested several) compose files that seems to match the posted "workaround" from above. I just want ANY way right now to create a VMDK-backed vSphere volume for persistence (which works today), and then actually USE it in Docker.
The above posted config actually gives me this error:
If I change it to this (the official docs show "true" as a requirement for "external"):
Then I get this error (where line 63, column 11 is the beginning of the "name:" line):
And if I change the file to:
Because I have also seen this "workaround" typed out as if the volume param needed to NOT match the actual vsphere volume (before the @) and then use the "name:" to map it back to the real external volume. But, it results in the 1st error:
And finally, to make this whole thing THAT much more absurd, if I simply move the "name:" line unindented as shown in newer examples:
I get a completely different error:
The sections all follow the +2 indentation rule, and if I simply remove the last volume section and replace the service volume to point to a normal container persistent volume on the host, it builds fine but this issue has rendered the entire vsphere volume driver broken and useless if you cannot actually USE the darn vsphere volumes! Can someone please copy my example here and indicate exactly what needs to change in syntax to supposedly make this work as a workaround? Also, if I am already on 17.09rc3 I am not clear if this was supposedly fixed in 17.08 why it's even an issue at all and either way do we know for certain what build this parsing/schema issue WILL be finally fixed? Thanks! |
Please do not close issue before the fix is verified (unless you close as won't fix or not an issue) . //CC @lipingxue @tusharnt |
Thanks so much @msterin! I swear I have tried every combination now (including the OP's original approach of using standard syntax with full volume name including the "@") before and after the upgrade to 17.09rc3 but just to be safe and re-config it for you, I wanted to carefully run through once each step and provide all the output in case it is somehow anything on my end (and if not - help you guys on your end). So first, the full output on the versions of this test:
I then verified the compose version:
I set my file to this syntax which follows the latest doc's syntax to simply indicate "true" for volume external and assumes that now "@" is legal in volume:
And then just to be safe, did a FULL REBOOT, brought it back up and very unfortunately, like the OP, still do receive the same error:
Certainly not trying to hijack the OP's thread if I have some underlying version issue I have missed then I apologize - but ultimately I AM trying to do the same thing as the OP (and others) and receiving the same errors and hopefully my testing can help everyone with this issue EDIT: Figured out the file version error I was getting when switching - was due to venv setup - but the OP's issue does still seem to be an issue unless I've missed something else. EDIT: So I had some pollution from venv causing a couple of issues including the original error above that the OP posted of.....however now being pretty darn sure that latest 1.16 is reading the file (because I CAN change the file version to 2.3+), I get a totally different error of:
Which sure seems directly like there is still in fact SOME kind of parsing+schema issue present. I know you guys have the automated tests set up so unless I find any additional issues on my end, I'll await what is hopefully just some direction/confirmation of everything required (minimum compose file version #, syntax, etc.) to support the full vSphere volume name. |
Unbelievable. So I am on about hour number twelve of trial and error and I can definitively say that I have found a workaround and some definitive answers. Hopefully you Docker guys can still detail if/when this is/was fixed and does/will not require this workaround but hopefully this testing I have done will further assist you in pinpointing the root cause of this issue #1315. In my test case, one issue was if you see my posts above, I was using "common.yml" with "extends:". This should NOT be this big of an issue but these findings show just how deep this parsing issue muse be and if you consider the merging of "extends" layers, how out of control this becomes (hence my all day wasted on the MASSIVE number of test combinations). It is also in my case, the reason the above posted workaround was not working. Scenario 1:Attempting the workaround for (still) inability to include "@" in vsphere volumes due to above regex parsing & schema issue #1315. Configurationdocker-compose.yml:
common.yml:
ResultFailure with following error:
As you can see, it IS declared. In fact, I've spent hours RE-declaring it and as shown in my posts above, the friggin parser DOES read the typed section, because it happily complains and throws 100 different errors if you attempt different indentations or keywords like "name:"......but will NEVER work. Scenario 2:Attempting the workaround for (still) inability to include "@" in vsphere volumes due to above regex parsing & schema issue #1315. Configurationdocker-compose.yml:
common.yml:
ResultSUCCESS! To demonstrate how problematic this parsing seems to be, and why this literally burned an entire day of my life testing the various combinations - if we have well proven from all the posts above that an INVALID ((or so the parser says)) value in the "key" portion in the "volume:" declaration section will throw a terminating error, then logic would dictate that a VALID declaration and/or values are also being parsed and utilized further. Scenario 3 shows for some odd reason, that is not the case. Scenario 3:Demonstrate that a INVALID value in an EXTEND yml in VOLUME section despite being parsed for errors, does NOT appear to use any correctly parsed volume data. Configurationdocker-compose.yml:
common.yml:
ResultSUCCESS! What?!? Of note, it shows that some routines on the back-end after the parser does a questionable job at validation, then get very selective and literally IGNORE certain values (note above stating the external as "FALSE", and using a totally INVALID, non-existent driver name) - as long as it has found them in the root docker-compose.yml. However, all historical behavior matches the documentation that these values should be MERGED and clearly in this case of volume declaration - they are not. Well, then there is no way all the usage could be in the extend and the declaration could be ONLY in the root docker-compose.yml, where it is not even actually referred to once, right? Wrong: Scenario 4:Test whether the volume declaration section for vsphere volumes can be missing in the very extend file in which they are referenced, and placed in the root where is it not being actively referenced, unlike any of the other declaration sections. Configurationdocker-compose.yml:
common.yml:
ResultSUCCESS! What?!? Okay another unexpected result. Now, suddenly parsing that extend file that continually threw parsing errors reading the "required" volume declaration, now is suddenly okay that it is totally GONE. For volume declaration, it appears as long as it is present in the root docker-compose.yml, it can be completely void in the extend yml in which all the declared volumes are used. Note the upshot - if you simply MOVE the WORKING declaration section from docker-compose.yml into common.yml - IT RESULTS IN ERROR. Move it back - things work. Current workaround for using vsphere volume names containing the "@datastore" from my testing:
|
@CollinChaffin Thanks for your feedback and detailed test. According to this release notes, the fix docker/cli#306 has been included in build 17.09.0-ce. I will try to verify with this new ce build. |
I have verified the with docker 17.09.0-ce-rc3, the fix to docker/cli#306 has been included in this build and it works as expected.
Please see the following example:
Deploy a stack using the above yaml file:
After stack deploy, volume "postgres_vol@sharedVmfs-1" has been created by vSphere driver as expected.
Wait for a while, check the service
|
@CollinChaffin, we don't need the workaround now as I have verified the fix above. Could you try it at your side and let me know if you have any question? Thanks! |
@lipingxue - thanks for validating. It looks like the main damage was from closing the issue too early and referring the wrong version (above: @CollinChaffin - sorry about the headache, please let us know if 17.09 (and yaml 3.4) finally works as expected on your setup |
@nbaillie @CollinChaffin Could you verify that this has issue has been fixed at your side with docker 17.09 and yaml 3.4? Thanks! |
@nbaillie @CollinChaffin Please reopen if you are still facing any issue. |
Hi Andrew,
I cannot see your update in that issue since that issue has been closed.
Could you reopen that issue and put your update here? I will take a look.
Regards,
Liping
…On Mon, Oct 30, 2017 at 5:12 PM, Andrew Savinykh ***@***.***> wrote:
I'm certainly still facing the issue. I have:
***@***.***:~/nexus-test$ docker --version
Docker version 17.10.0-ce, build f4ffd25
***@***.***:~/nexus-test$ docker-compose --version
docker-compose version 1.17.0-rc1, build a0f95af
***@***.***:~/nexus-test$
and:
***@***.***:~/nexus-test$ cat docker-compose.yml
version: '3.4'
volumes:
***@***.***:
external: true
driver: vsphere
services:
Nexus:
image: sonatype/nexus3
stdin_open: true
volumes:
- ***@***.***:/nexus-data
tty: true
And I'm getting:
***@***.***:~/nexus-test$ docker-compose up
ERROR: The Compose file './docker-compose.yml' is invalid because:
volumes value ***@***.***' does not match any of the regexes: u'^[a-zA-Z0-9._-]+$'
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1315 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEiMM2xrIiATbM_37W9ygVwzU5UU-ZnAks5sxmXvgaJpZM4NtM-B>
.
|
@lipingxue, thank you for your response. You cannot see my update not because the issue is closed but because I deleted it. I deleted it because I realized that I was doing something silly. |
Hi Andrew,
OK. Glad to hear your issue is resolved.
…On Tue, Oct 31, 2017 at 12:20 PM, Andrew Savinykh ***@***.***> wrote:
@lipingxue <https://github.com/lipingxue>, thank you for your response.
You cannot see my update not because the issue is closed but because I
deleted it. I deleted it because I realized that I was doing something
silly.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1315 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEiMM8pJVoTuMszEM_wcOHba4xnHWdvMks5sx3MIgaJpZM4NtM-B>
.
|
im ok with the creation of volumes like the bellow:
docker volume create --driver=vsphere --name=MyTestVol@VMWARE-DOCKER -o size=1gb
But how can i add the @VMWARE-DOCKER component to the yml for stack deployment?
i have tried Like.....
volumes:
db:
driver: vsphere
driver_opts:
size: 120Gb
datastore: VMWARE-DOCKER
and.....
volumes:
db@VMWARE-DOCKER:
driver: vsphere
driver_opts:
size: 120Gb
datastore: VMWARE-DOCKER
and some other variations.
The text was updated successfully, but these errors were encountered: