docker engine fails to recognize managed plugin driver after uninstalling DEB package #32635

shuklanirdesh82 · 2017-04-14T22:29:36Z

Description

I was having docker-volume-vshere plugin, called as vDVS, using package based installation so far. Now the plugin is having support of Docker Managed plugin system, so uninstalled vDVS plugin (dpkg -P docker-volume-vsphere) and installed managed plugin by pulling from docker store (using docker plugin install --grant-all-permissions --alias vsphere vmware/docker-volume-vsphere:latest).

Following issues I have observed:

docker volume ls takes more time to complete/respond
volume creation request fails
docker volume ls shows volume multiple entries for the same volume name

docker volume ls pause for ~30 seconds and shows the output (shows volumes created previously).
Further volume creation fails with following error.

root@sc-rdops-vm18-dhcp-57-89:~# docker volume create -d vsphere vol5
Error response from daemon: create vol5: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Create: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory

Sometimes, docker volume ls shows volume entries twice one for package based installation driver and another for managed plugin based installation driver ( see received result area for more detail)

Steps to reproduce the issue:
Pre-reqs: VMWare ESX with a Docker host VM inside. (Ubuntu 16.04 Mini) and docker-volume-vsphere VIB installed.

See https://github.com/vmware/docker-volume-vsphere/ for more info

Downloads: https://github.com/vmware/docker-volume-vsphere/releases/tag/0.13

Install vDVS using package installation DEB on ubuntu vm

1. wget https://bintray.com/vmware/vDVS/download_file?file_path=docker-volume-vsphere_0.13.15d313a_amd64.deb
2. dpkg -i download_file\?file_path\=docker-volume-vsphere_0.13.15d313a_amd64.deb

Create volume using driver called vsphere and make sure volume is created successfully

docker volume create -d vsphere vol4
root@sc-rdops-vm18-dhcp-57-89:~# docker volume ls
DRIVER              VOLUME NAME
vsphere             vol4@TestDatastore

Uninstall vDVS deb package

dpkg -r docker-volume-vsphere

Install vDVS managed plugin

docker plugin install --grant-all-permissions --alias vsphere vmware/docker-volume-vsphere:latest

Invoke docker plugin ls to validate managed plugin installation
Invoke volume creation as shown below and observed an error

root@sc-rdops-vm18-dhcp-57-89:~# docker volume create -d vsphere vol5
Error response from daemon: create vol5: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Create: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory

Describe the results you received:

root@sc-rdops-vm18-dhcp-57-89:~# docker plugin ls
ID                  NAME                DESCRIPTION                           ENABLED
3e1fbe5c9aa3        vsphere:latest      VMWare vSphere Docker Volume plugin   true
root@sc-rdops-vm18-dhcp-57-89:~# docker volume create -d vsphere vol5
Error response from daemon: create vol5: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Create: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory
root@sc-rdops-vm18-dhcp-57-89:~# docker volume create -d vsphere:latest vol5
vol5
root@sc-rdops-vm18-dhcp-57-89:~# docker volume ls
DRIVER              VOLUME NAME
vsphere             vol4   <<<<<<<<<<<<<<<<<<< notice two entries
vsphere:latest      vol4@TestDatastore
vsphere:latest      vol5@TestDatastore

docker.service log

Apr 14 21:46:33 sc-rdops-vm18-dhcp-57-89 dockerd[6922]: time="2017-04-14T21:46:33.327462375Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vsphere.sock/VolumeDriver.Get: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Get: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory, retrying in 1s"
Apr 14 21:46:34 sc-rdops-vm18-dhcp-57-89 dockerd[6922]: time="2017-04-14T21:46:34.329087624Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vsphere.sock/VolumeDriver.Get: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Get: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory, retrying in 2s"
Apr 14 21:46:36 sc-rdops-vm18-dhcp-57-89 dockerd[6922]: time="2017-04-14T21:46:36.329938956Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vsphere.sock/VolumeDriver.Get: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Get: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory, retrying in 4s"
Apr 14 21:46:40 sc-rdops-vm18-dhcp-57-89 dockerd[6922]: time="2017-04-14T21:46:40.330874529Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vsphere.sock/VolumeDriver.Get: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Get: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory, retrying in 8s"
Apr 14 21:46:48 sc-rdops-vm18-dhcp-57-89 dockerd[6922]: time="2017-04-14T21:46:48.332211437Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vsphere.sock/VolumeDriver.Create: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Create: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory, retrying in 1s"
Apr 14 21:46:49 sc-rdops-vm18-dhcp-57-89 dockerd[6922]: time="2017-04-14T21:46:49.333514812Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vsphere.sock/VolumeDriver.Create: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Create: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory, retrying in 2s"
Apr 14 21:46:51 sc-rdops-vm18-dhcp-57-89 dockerd[6922]: time="2017-04-14T21:46:51.334540395Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vsphere.sock/VolumeDriver.Create: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Create: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory, retrying in 4s"
Apr 14 21:46:55 sc-rdops-vm18-dhcp-57-89 dockerd[6922]: time="2017-04-14T21:46:55.335566859Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vsphere.sock/VolumeDriver.Create: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Create: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory, retrying in 8s"
Apr 14 21:47:03 sc-rdops-vm18-dhcp-57-89 dockerd[6922]: time="2017-04-14T21:47:03.336726902Z" level=error msg="Handler for POST /v1.26/volumes/create returned error: create vol5: Post http://%2Frun%2Fdocker%2Fplugins%2Fvsphere.sock/VolumeDriver.Create: dial unix /run/docker/plugins/vsphere.sock: connect: no such file or directory"

Describe the results you expected:
Moving to managed plugin based installation should be transparent; either docker service should be restarted automatically as part of docker plugin install or call out explicitly if any extra steps needs to be done.

As you can see, the plugin is installed using --alias keyword, so further volume creation should not be failed.

Additional information you deem important (e.g. issue happens only occasionally):

Note: After restarting docker service, while invoking above command again works fine and docker volume ls also responds quickly.

Output of docker version:

root@sc-rdops-vm18-dhcp-57-89:~# docker version
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:02:43 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:02:43 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

root@sc-rdops-vm18-dhcp-57-89:~# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.03.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 2
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local vsphere
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-66-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 992.3 MiB
Name: sc-rdops-vm18-dhcp-57-89
ID: QI6A:4FBR:BMQ2:S6T3:QNAL:P7IB:3UBU:5UVR:THMQ:C4T4:XSXL:FELA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: shuklanirdesh
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-66-generic x86_64)
VMWare ESXi 6.0U2 GA release

/CC @msterin @pdhamdhere

The text was updated successfully, but these errors were encountered:

thaJeztah · 2017-04-14T23:14:32Z

/cc @anusha-ragunathan @cpuguy83

anusha-ragunathan · 2017-04-17T19:55:42Z

Just because user does a dpkg -r docker-volume-vsphere doesnt make docker engine remove pluginv1 (in this case "vsphere" plugin) from its inventory. Docker-engine discovery of pluginv1 happens during startup and if the plugin is missing (i.e not responsive to API calls over the socket), then it will indeed result in behavior such as docker plugin ls being slow.

Also, user is creating an pluginv2 alias which conflicts with the pluginv1. This again results in expected pluginv2 discovery issues and get resolved with referencing the plugin with name:tag which is specific to pluginv2.

To transition between pluginv1 and pluginv2, users will need to restart engine.

shuklanirdesh82 · 2017-04-17T20:37:55Z

Thanks @anusha-ragunathan !

Just because user does a dpkg -r docker-volume-vsphere doesnt make docker engine remove pluginv1 (in this case "vsphere" plugin) from its inventory.

While using package installation (using deb/rpm), there is no issue at all.

Volume creation failure or docker volume ls being slowness only happens while using plugin based installation.

To transition between pluginv1 and pluginv2, users will need to restart engine.

Shouldn't docker plugin install takes care about docker engine restart?

anusha-ragunathan · 2017-04-18T00:11:00Z

No docker API will have that kind of privilege.

msterin · 2017-04-18T15:32:36Z

When the v1 (unmanaged) plugin is removed, the .Sock file (and json if it was around) is removed. Methinks it's a good hint to remove the old driver from internal tracking without restart. At any rate, it's a doc update for us so it does not feel like a hi priority issue,

GordonTheTurtle added area/plugins version/17.03 labels Apr 14, 2017

shuklanirdesh82 mentioned this issue Apr 15, 2017

Refreshing Upgrade plugin instruction and FAQ for upgrading to 0.13 managed plugin [CI SKIP] vmware-archive/vsphere-storage-for-docker#1169

Merged

anusha-ragunathan closed this as completed Apr 18, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker engine fails to recognize managed plugin driver after uninstalling DEB package #32635

docker engine fails to recognize managed plugin driver after uninstalling DEB package #32635

shuklanirdesh82 commented Apr 14, 2017 •

edited

Loading

thaJeztah commented Apr 14, 2017

anusha-ragunathan commented Apr 17, 2017

shuklanirdesh82 commented Apr 17, 2017 •

edited

Loading

anusha-ragunathan commented Apr 18, 2017

msterin commented Apr 18, 2017

docker engine fails to recognize managed plugin driver after uninstalling DEB package #32635

docker engine fails to recognize managed plugin driver after uninstalling DEB package #32635

Comments

shuklanirdesh82 commented Apr 14, 2017 • edited Loading

thaJeztah commented Apr 14, 2017

anusha-ragunathan commented Apr 17, 2017

shuklanirdesh82 commented Apr 17, 2017 • edited Loading

anusha-ragunathan commented Apr 18, 2017

msterin commented Apr 18, 2017

shuklanirdesh82 commented Apr 14, 2017 •

edited

Loading

shuklanirdesh82 commented Apr 17, 2017 •

edited

Loading