[Multi-ESX] volume metadata file got deleted while removing volume from VM(placed on Esx2) which is in use by another VM(placed on Esx1)

[shuklanirdesh82]

Volume metadata file got deleted while removing volume from VM(placed on Esx2) which is in use by another VM(placed on Esx1); remove volume request deletes vmfd (metadata file) and volume properties (Filesystem, Disk Format, Created By, Created Date) got reset. Admin_CLI looses some important information that leads no information and bad experience to user.

Test bed setup:

1. 2 ESX (E1 & E2) accessing a shared datastore SD1
2. 2 VMs (VM1 & VM2):

• VM1 is created on SD1 (registered at E1)
• VM2 is created on SD1 (registered at E2)

Steps to reproduce (command + output for the reference)

1. create volume and validates its creation
2. mount volume from VM1 and validates its attachment state (leave it mounted)
3. remove the volume from VM2 (another vm from different ESX) and expects an error message saying volume is in use

Expectation: expects an error message saying volume is in use and no change in docker volume inspect & admin_cli volume ls

Current behavior: remove volume request deletes vmfd (metadata file) and volume properties (Filesystem, Disk Format, Created By, Created Date) got reset as shown below

# /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume  Datastore     VMGroup   Capacity  Used  Filesystem  Policy  Disk Format  Attached-to  Access      Attach-as               Created By  Created Date  
------  ------------  --------  --------  ----  ----------  ------  -----------  -----------  ----------  ----------------------  ----------  ------------  
vol2    sharedVmfs-0  _DEFAULT  100MB     15MB  N/A         N/A     N/A          detached     read-write  independent_persistent  N/A         N/A           

command + output for the reference:

1. create volume and validates its creation

on VM1: 

root@photon-xVmYMbyTn [ ~ ]#docker volume create -d vsphere vol2
vol2
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      vol2@sharedVmfs-0

on VM2:

root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      vol2@sharedVmfs-0

on E1:
[root@sc2-rdops-vm05-dhcp-160-17:/vmfs/volumes/58f90ff7-dc27eee6-2501-020035a84afe/dockvols] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume  Datastore     VMGroup   Capacity  Used  Filesystem  Policy  Disk Format  Attached-to  Access      Attach-as               Created By   Created Date              
------  ------------  --------  --------  ----  ----------  ------  -----------  -----------  ----------  ----------------------  -----------  ------------------------  
vol2    sharedVmfs-0  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached     read-write  independent_persistent  VM_17_SVMFS  Tue Apr 25 19:26:55 2017  

[root@sc2-rdops-vm05-dhcp-160-17:/vmfs/volumes/58f90ff7-dc27eee6-2501-020035a84afe/dockvols] ls -la /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111/
total 46104
drwxr-xr-x    1 root     root           980 Apr 25 19:29 .
drwxr-xr-x    1 root     root           420 Apr 25 19:12 ..
-rw-------    1 root     root          4096 Apr 25 19:29 vol2-2204c79fe109d3ae.vmfd
-rw-------    1 root     root     104857600 Apr 25 19:30 vol2-flat.vmdk
-rw-------    1 root     root           554 Apr 25 19:29 vol2.vmdk

2. mount volume from VM1 and validates its attachment state (leave it mounted)

on VM1:

root@photon-xVmYMbyTn [ ~ ]# docker run --rm -it -v vol2@sharedVmfs-0:/vol1 --name cnt_abc11 busybox
/ #     <<<<<<< leave it as mounted

on VM2:

root@photon-xVmYMbyTn [ ~ ]# docker volume inspect vol2@sharedVmfs-0
[
    {
        "Driver": "vsphere:latest",
        "Labels": null,
        "Mountpoint": "/mnt/vmdk/vol2@sharedVmfs-0",
        "Name": "vol2@sharedVmfs-0",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "attached to VM": "VM_17_SVMFS",
            "capacity": {
                "allocated": "15MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Tue Apr 25 19:26:55 2017",
            "created by VM": "VM_17_SVMFS",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "attached"
        }
    }
]

on E1:

[root@sc2-rdops-vm05-dhcp-160-17:/vmfs/volumes/58f90ff7-dc27eee6-2501-020035a84afe/dockvols] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume  Datastore     VMGroup   Capacity  Used  Filesystem  Policy  Disk Format  Attached-to  Access      Attach-as               Created By   Created Date              
------  ------------  --------  --------  ----  ----------  ------  -----------  -----------  ----------  ----------------------  -----------  ------------------------  
vol2    sharedVmfs-0  _DEFAULT  100MB     15MB  ext4        N/A     thin         VM_17_SVMFS  read-write  independent_persistent  VM_17_SVMFS  Tue Apr 25 19:26:55 2017  

3. remove the volume from VM2 (another vm from different ESX) and expects an error message saying volume is in use

on VM2:

root@photon-xVmYMbyTn [ ~ ]# docker volume rm vol2@sharedVmfs-0
Error response from daemon: Error while removing volume vol2@sharedVmfs-0: remove vol2@sharedVmfs-0: VolumeDriver.Remove: Failed to remove volume: Error caused by file /vmfs/volumes/58f90ff7-dc27eee6-2501-020035a84afe/dockvols/11111111-1111-1111-1111-111111111111/vol2.vmdk

on E1:

[root@sc2-rdops-vm05-dhcp-160-17:/vmfs/volumes/58f90ff7-dc27eee6-2501-020035a84afe/dockvols] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume  Datastore     VMGroup   Capacity  Used  Filesystem  Policy  Disk Format  Attached-to  Access      Attach-as               Created By  Created Date  
------  ------------  --------  --------  ----  ----------  ------  -----------  -----------  ----------  ----------------------  ----------  ------------  
vol2    sharedVmfs-0  _DEFAULT  100MB     15MB  N/A         N/A     N/A          detached     read-write  independent_persistent  N/A         N/A           
[root@sc2-rdops-vm05-dhcp-160-17:/vmfs/volumes/58f90ff7-dc27eee6-2501-020035a84afe/dockvols] ls -la /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111/
total 46096
drwxr-xr-x    1 root     root           840 Apr 25 19:32 .
drwxr-xr-x    1 root     root           420 Apr 25 19:12 ..
-rw-------    1 root     root     104857600 Apr 25 19:30 vol2-flat.vmdk
-rw-------    1 root     root           488 Apr 25 19:32 vol2.vmdk

vm_1_docker.txt
vm_2_docker.txt
vmdk_ops.txt

[shuklanirdesh82]

docker volume rm requests come from any VMs registered on E1 are not running into this issue, it is specific to multi-ESX setup.

output from another VM placed on E1:

root@photon-xVmYMbyTn [ ~ ]# docker volume rm vol2@sharedVmfs-0
Error response from daemon: Error while removing volume vol2@sharedVmfs-0: remove vol2@sharedVmfs-0: VolumeDriver.Remove: Failed to remove volume /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111/vol2.vmdk, in use by VM uuid = 564d81f7-d886-7c9c-66f9-27c4d05875ba.
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      vol2@sharedVmfs-0
root@photon-xVmYMbyTn [ ~ ]# docker volume inspect vol2@sharedVmfs-0
[
    {
        "Driver": "vsphere:latest",
        "Labels": null,
        "Mountpoint": "/mnt/vmdk/vol2@sharedVmfs-0",
        "Name": "vol2@sharedVmfs-0",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "attached to VM": "VM_17_SVMFS",
            "capacity": {
                "allocated": "15MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Tue Apr 25 19:26:55 2017",
            "created by VM": "VM_17_SVMFS",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "attached"
        }
    }
]

[govint]

This is an issue on the esx disk libraries and already fixed for the next up coming release of ESX. If needed we can request a backport to the versions supported for the plugin. .

[tusharnt]

@shuklanirdesh82 to add this to known issues.
@govint to follow up with ESX folks to backport the fix to patch releases.

[msterin]

@govint writes:

This is an issue on the esx disk libraries and already fixed for the next up coming release of ESX. If needed we can request a backport to the versions supported for the plugin.

What exactly is the "the issue in the ESX disklib" ? Until it's backported (and backport deployed) we at least need to understand the specifics, and the way to fix it in our code if needed.

Reopening until the details are added.

[tusharnt]

Cross ESX multi tenancy feature is experimental.So pushing this issue to the back burner for now.