VolumeDriver.Mount: error mounting <volume>: exit status 1

[jgraeve]

I have 3 centos 7 nodes with gluster installed

[root@ds01 ~]# gluster volume info
Volume Name: gv0
Type: Replicate
Volume ID: 98f1c6e3-8fa0-4a2b-93d5-95b64435eb04
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ds01:/data/brick1/gv0
Brick2: ds02:/data/brick1/gv0
Brick3: ds03:/data/brick1/gv0
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off

The gv0 gluster volume is working fine. When I mount it and create files they are replicated to ds01,ds02,ds03

But when I create a docker volume with the glusterfs plugin and try to mount it in a docker instance I get
VolumeDriver.Mount: error mounting : exit status 1

All help appreciated!

versions used:
docker version 19.03.12
glusterfs 7.6
centos 7
Linux ds01 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[root@ds01 ~]# docker plugin install --alias glusterfs trajano/glusterfs-volume-plugin --grant-all-permissions --disable
latest: Pulling from trajano/glusterfs-volume-plugin
a392984758fa: Download complete
Digest: sha256:f054d37e71aca82426c12c0d4a555861e1225969df2df6c45641303b4cf18a95
Status: Downloaded newer image for trajano/glusterfs-volume-plugin:latest
Installed plugin trajano/glusterfs-volume-plugin
[root@ds01 ~]# docker plugin set glusterfs SERVERS=ds01,ds02,ds03
[root@ds01 ~]# docker plugin enable glusterfs
glusterfs
[root@ds01 ~]# docker plugin list
ID NAME DESCRIPTION ENABLED
e6ee819ceeca glusterfs:latest GlusterFS plugin for Docker true
[root@ds01 ~]# docker plugin inspect glusterfs
[
{
"Config": {
"Args": {
"Description": "",
"Name": "",
"Settable": null,
"Value": null
},
"Description": "GlusterFS plugin for Docker",
"DockerVersion": "18.03.0-ce",
"Documentation": "https://github.com/trajano/docker-volume-plugins/",
"Entrypoint": [
"/glusterfs-volume-plugin"
],
"Env": [
{
"Description": "",
"Name": "SERVERS",
"Settable": [
"value"
],
"Value": ""
}
],
"Interface": {
"Socket": "gfs.sock",
"Types": [
"docker.volumedriver/1.0"
]
},
"IpcHost": false,
"Linux": {
"AllowAllDevices": false,
"Capabilities": [
"CAP_SYS_ADMIN"
],
"Devices": [
{
"Description": "",
"Name": "",
"Path": "/dev/fuse",
"Settable": null
}
]
},
"Mounts": null,
"Network": {
"Type": "host"
},
"PidHost": false,
"PropagatedMount": "/var/lib/docker-volumes",
"User": {},
"WorkDir": "",
"rootfs": {
"diff_ids": [
"sha256:9e8240f5b99231266ccb3260422fa333d6f72b3d7f6c77042d775ffc0e89b9ba"
],
"type": "layers"
}
},
"Enabled": true,
"Id": "e6ee819ceeca73104da53ae7cc9b16150ddef6536f0dfbf7bcc282f8915b6887",
"Name": "glusterfs:latest",
"PluginReference": "docker.io/trajano/glusterfs-volume-plugin:latest",
"Settings": {
"Args": [],
"Devices": [
{
"Description": "",
"Name": "",
"Path": "/dev/fuse",
"Settable": null
}
],
"Env": [
"SERVERS=ds01,ds02,ds03"
],
"Mounts": []
}
}
]
[root@ds01 ~]# docker volume create -d glusterfs gv2
gv2
[root@ds01 ~]# docker run -it -v gv2:/mnt alpine
docker: Error response from daemon: VolumeDriver.Mount: error mounting gv2: exit status 1.
See 'docker run --help'.

Jul 10 14:57:29 ds01 dockerd: time="2020-07-10T14:57:28+02:00" level=error msg="Entering go-plugins-helpers capabilitiesPath" plugin=e6ee819ceeca73104da53ae7cc9b16150ddef6536f0dfbf7bcc282f8915b6887
Jul 10 14:57:29 ds01 dockerd: time="2020-07-10T14:57:29+02:00" level=error msg="Entering go-plugins-helpers getPath" plugin=e6ee819ceeca73104da53ae7cc9b16150ddef6536f0dfbf7bcc282f8915b6887
Jul 10 14:57:29 ds01 dockerd: time="2020-07-10T14:57:29+02:00" level=error msg="Entering go-plugins-helpers capabilitiesPath" plugin=e6ee819ceeca73104da53ae7cc9b16150ddef6536f0dfbf7bcc282f8915b6887
Jul 10 14:57:29 ds01 dockerd: time="2020-07-10T14:57:29+02:00" level=error msg="Entering go-plugins-helpers getPath" plugin=e6ee819ceeca73104da53ae7cc9b16150ddef6536f0dfbf7bcc282f8915b6887
Jul 10 14:57:29 ds01 dockerd: time="2020-07-10T14:57:29+02:00" level=error msg="Entering go-plugins-helpers capabilitiesPath" plugin=e6ee819ceeca73104da53ae7cc9b16150ddef6536f0dfbf7bcc282f8915b6887
Jul 10 14:57:29 ds01 dockerd: time="2020-07-10T14:57:29+02:00" level=error msg="Entering go-plugins-helpers mountPath" plugin=e6ee819ceeca73104da53ae7cc9b16150ddef6536f0dfbf7bcc282f8915b6887
Jul 10 14:57:29 ds01 dockerd: time="2020-07-10T14:57:29+02:00" level=error msg="[-s ds01 -s ds02 -s ds03 --volfile-id=gv2 /var/lib/docker-volumes/de3b2d20f02f69bf4fd0112157e8ac2a259e820a06610b906d20693bb091241b]" plugin=e6ee819ceeca73104da53ae7cc9b16150ddef6536f0dfbf7bcc282f8915b6887
Jul 10 14:57:29 ds01 dockerd: time="2020-07-10T14:57:29+02:00" level=info msg="Command output: " plugin=e6ee819ceeca73104da53ae7cc9b16150ddef6536f0dfbf7bcc282f8915b6887
Jul 10 14:57:29 ds01 dockerd: time="2020-07-10T14:57:29.059610710+02:00" level=error msg="Handler for POST /v1.40/containers/create returned error: VolumeDriver.Mount: error mounting gv2: exit status 1"

[goodwolf]

The latest tag is the old version, install the plugin with the tag v2.0.3

docker plugin install --alias glusterfs trajano/glusterfs-volume-plugin:v2.0.3 --grant-all-permissions --disable

[pmorch]

I had the exact same problem, and @goodwolf's answer fixed it.

It is a problem that latest (the default) is older than 2.0.3 and that latest doesn't work. @jgraeve , did this fix your problem too. (Or have you moved on to other things and can't test it? That is a valid real-life answer too...)

[wiryonolau]

It doesn't work even when using 2.0.3, same error message

[crmillner]

I think that instead of:
docker volume create -d glusterfs gv2

it should be:
docker volume create -d glusterfs gv0

I think that the volume name must match the gluster volume name. At least that seemed to work for me when I got this error.

[wiryonolau]

So we need to create the volume name in the gluster first if we want to use specific name ?
If we have multiple stack it will be bad to use same volume name

I think mounting glusterfs first to the os then use local-persist docker driver is more flexible.