You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Note - this is closed on the Redhat BZ but I have reproduced the same issue on 4.5.4 today.
Affected version
IMPORTANT!: oVirt prior to 4.5 reached end of life.
If you have an issue on an older version please
upgrade to latest stable release before opening an issue.
oVirt Engine: 4.5.4-1.el8
Host OS Variant: Rocky Linux release 8.7 (Green Obsidian)
VDSM version: 4.50.3.4-1.el8
Description of problem:
VM has 2 disks and base and snapshot (leaf/top) image.
When trying to delete the snapshot, it fails with Drive image file could not be found', 'code': 13.
VM images are in Gluster FS storage domain.
ON the ovirt-engine logs, fails with a large NEP [I1].
On the host VDSM logs, fails with the same [I2].
The VM XML shows to have the correct the base and leaf (top) images [I3].
Also when checking the lsof, the same base and leaf images are open and used by the VM [I3].
[I1] Failure on the engine side.
~~~
2020-06-10 07:47:44,653-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] START, MergeVDSCommand(HostName = HOST-virt1, MergeVDSCommandParameters:{hostId='4653ba39-cb4d-4c35-9caa-0fa38b583792', vmId='e246d259-b386-4ef8-a047-909a97be6c89', storagePoolId='941567b0-8a4e-11e1-aff8-777f93db9152', storageDomainId='1f48f887-dd49-4363-9e5c-603c007a9baf', imageGroupId='eb12f2bc-b98d-4c3a-9d75-c31184b22a1d', imageId='6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', baseImageId='75e0a259-d0bd-47ff-ad6a-8f292968e99d', topImageId='6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', bandwidth='0'}), log id: 37125211
2020-06-10 07:47:44,656-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Failed in 'MergeVDS' method
2020-06-10 07:47:44,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM HOST-virt1 command MergeVDS failed: Drive image file could not be found
2020-06-10 07:47:44,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand' return value 'StatusOnlyReturn [status=Status [code=13, message=Drive image file could not be found]]'
2020-06-10 07:47:44,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] HostName = HOST-virt1
2020-06-10 07:47:44,663-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Command 'MergeVDSCommand(HostName = HOST-virt1, MergeVDSCommandParameters:{hostId='4653ba39-cb4d-4c35-9caa-0fa38b583792', vmId='e246d259-b386-4ef8-a047-909a97be6c89', storagePoolId='941567b0-8a4e-11e1-aff8-777f93db9152', storageDomainId='1f48f887-dd49-4363-9e5c-603c007a9baf', imageGroupId='eb12f2bc-b98d-4c3a-9d75-c31184b22a1d', imageId='6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', baseImageId='75e0a259-d0bd-47ff-ad6a-8f292968e99d', topImageId='6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Drive image file could not be found, code = 13
2020-06-10 07:47:44,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] FINISH, MergeVDSCommand, return: , log id: 37125211
2020-06-10 07:47:44,666-04 INFO [org.ovirt.engine.core.bll.MergeCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Running command: MergeCommand internal: true. Entities affected : ID: 1f48f887-dd49-4363-9e5c-603c007a9baf Type: Storage
2020-06-10 07:47:44,667-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] START, MergeVDSCommand(HostName = HOST-virt1, MergeVDSCommandParameters:{hostId='4653ba39-cb4d-4c35-9caa-0fa38b583792', vmId='e246d259-b386-4ef8-a047-909a97be6c89', storagePoolId='941567b0-8a4e-11e1-aff8-777f93db9152', storageDomainId='1f48f887-dd49-4363-9e5c-603c007a9baf', imageGroupId='708bec02-c56e-49e6-880f-b4341ceb865f', imageId='c9d23c2e-f84d-4e06-856c-71f6739f79ab', baseImageId='2cf64b4e-a034-46bc-8755-5cac3d33b5e1', topImageId='c9d23c2e-f84d-4e06-856c-71f6739f79ab', bandwidth='0'}), log id: 31a2170f
2020-06-10 07:47:44,670-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Failed in 'MergeVDS' method
2020-06-10 07:47:44,663-04 ERROR [org.ovirt.engine.core.bll.MergeCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Drive image file could not be found, code = 13 (Failed with error imageErr and code 13)
at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:118) [bll.jar:]
at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2112) [bll.jar:]
at org.ovirt.engine.core.bll.MergeCommand.executeCommand(MergeCommand.java:45) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1326) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2001) [bll.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:]
2020-06-10 07:47:44,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand' return value 'StatusOnlyReturn [status=Status [code=13, message=Drive image file could not be found]]'
2020-06-10 07:47:44,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] HostName = HOST-virt1
2020-06-10 07:47:44,672-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Command 'MergeVDSCommand(HostName = HOST-virt1, MergeVDSCommandParameters:{hostId='4653ba39-cb4d-4c35-9caa-0fa38b583792', vmId='e246d259-b386-4ef8-a047-909a97be6c89', storagePoolId='941567b0-8a4e-11e1-aff8-777f93db9152', storageDomainId='1f48f887-dd49-4363-9e5c-603c007a9baf', imageGroupId='708bec02-c56e-49e6-880f-b4341ceb865f', imageId='c9d23c2e-f84d-4e06-856c-71f6739f79ab', baseImageId='2cf64b4e-a034-46bc-8755-5cac3d33b5e1', topImageId='c9d23c2e-f84d-4e06-856c-71f6739f79ab', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Drive image file could not be found, code = 13
2020-06-10 07:47:44,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] FINISH, MergeVDSCommand, return: , log id: 31a2170f
2020-06-10 07:47:44,672-04 ERROR [org.ovirt.engine.core.bll.MergeCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Drive image file could not be found, code = 13 (Failed with error imageErr and code 13)
at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:118) [bll.jar:]
at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2112) [bll.jar:]
at org.ovirt.engine.core.bll.MergeCommand.executeCommand(MergeCommand.java:45) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1326) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2001) [bll.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:]
~~~
[I2] Failure on the VDSM side.
~~~
2020-06-10 07:47:44,656-0400 INFO (jsonrpc/6) [api.virt] START merge(drive={u'imageID': u'eb12f2bc-b98d-4c3a-9d75-c31184b22a1d', u'volumeID': u'6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', u'domainID': u'1f48f887-dd49-4363-9e5c-603c007a9baf', u'poolID': u'941567b0-8a4e-11e1-aff8-777f93db9152'}, baseVolUUID=u'75e0a259-d0bd-47ff-ad6a-8f292968e99d', topVolUUID=u'6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', bandwidth=u'0', jobUUID=u'509b885b-6ba6-4bb3-81c8-c5975ead489e') from=::ffff:XXX.XXX.XX.XXX,XXXXX, flow_id=5b06be20-1408-4892-a6a4-2ffe6366bebb, vmId=e246d259-b386-4ef8-a047-909a97be6c89 (api:48)
2020-06-10 07:47:44,657-0400 INFO (jsonrpc/6) [api.virt] FINISH merge return={'status': {'message': 'Drive image file could not be found', 'code': 13}} from=::ffff:XXX.XXX.XX.XXX,XXXXX, flow_id=5b06be20-1408-4892-a6a4-2ffe6366bebb, vmId=e246d259-b386-4ef8-a047-909a97be6c89 (api:54)
2020-06-10 07:47:44,657-0400 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.merge failed (error 13) in 0.00 seconds (__init__:312)
2020-06-10 07:47:44,671-0400 INFO (jsonrpc/5) [api.virt] START merge(drive={u'imageID': u'708bec02-c56e-49e6-880f-b4341ceb865f', u'volumeID': u'c9d23c2e-f84d-4e06-856c-71f6739f79ab', u'domainID': u'1f48f887-dd49-4363-9e5c-603c007a9baf', u'poolID': u'941567b0-8a4e-11e1-aff8-777f93db9152'}, baseVolUUID=u'2cf64b4e-a034-46bc-8755-5cac3d33b5e1', topVolUUID=u'c9d23c2e-f84d-4e06-856c-71f6739f79ab', bandwidth=u'0', jobUUID=u'1087ae45-141b-4683-8d8f-463687e4bcc9') from=::ffff:XXX.XXX.XX.XXX,XXXXX, flow_id=5b06be20-1408-4892-a6a4-2ffe6366bebb, vmId=e246d259-b386-4ef8-a047-909a97be6c89 (api:48)
2020-06-10 07:47:44,671-0400 INFO (jsonrpc/5) [api.virt] FINISH merge return={'status': {'message': 'Drive image file could not be found', 'code': 13}} from=::ffff:XXX.XXX.XX.XXX,XXXXX, flow_id=5b06be20-1408-4892-a6a4-2ffe6366bebb, vmId=e246d259-b386-4ef8-a047-909a97be6c89 (api:54)
2020-06-10 07:47:44,671-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.merge failed (error 13) in 0.00 seconds (__init__:312)
The text was updated successfully, but these errors were encountered:
same things occur to me as well, but with different setup. all of our cluster hosts using CentOS Stream 8 and using NFS & iSCSI backend for storage domain. The second try we delete the snapshot it will completed successully, but If that VM (the one that having failed snapshot deletion and the snapshot are deleted in second try) is turned off, it cant be turned on again with illegal disk error. maybe any workaround for this issue ? because it's very critical for us.
Note - this is closed on the Redhat BZ but I have reproduced the same issue on 4.5.4 today.
Affected version
Description of problem:
VM has 2 disks and base and snapshot (leaf/top) image.
When trying to delete the snapshot, it fails with Drive image file could not be found', 'code': 13.
VM images are in Gluster FS storage domain.
ON the ovirt-engine logs, fails with a large NEP [I1].
On the host VDSM logs, fails with the same [I2].
The VM XML shows to have the correct the base and leaf (top) images [I3].
Also when checking the lsof, the same base and leaf images are open and used by the VM [I3].
[I1] Failure on the engine side.
[I2] Failure on the VDSM side.
The text was updated successfully, but these errors were encountered: