Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1849193 - Live merge fails with Drive image file could not be found', 'code': 13 #827

Open
alexcrow1974 opened this issue Mar 29, 2023 · 1 comment
Open

Bug 1849193 - Live merge fails with Drive image file could not be found', 'code': 13 #827

alexcrow1974 opened this issue Mar 29, 2023 · 1 comment
Labels
bug

Comments

@alexcrow1974
Copy link

Note - this is closed on the Redhat BZ but I have reproduced the same issue on 4.5.4 today.

Affected version

IMPORTANT!: oVirt prior to 4.5 reached end of life.
If you have an issue on an older version please
upgrade to latest stable release before opening an issue.

  • oVirt Engine: 4.5.4-1.el8
  • Host OS Variant: Rocky Linux release 8.7 (Green Obsidian)
  • VDSM version: 4.50.3.4-1.el8

Description of problem:

VM has 2 disks and base and snapshot (leaf/top) image.
When trying to delete the snapshot, it fails with Drive image file could not be found', 'code': 13.
VM images are in Gluster FS storage domain.

ON the ovirt-engine logs, fails with a large NEP [I1].
On the host VDSM logs, fails with the same [I2].
The VM XML shows to have the correct the base and leaf (top) images [I3].
Also when checking the lsof, the same base and leaf images are open and used by the VM [I3].

[I1] Failure on the engine side.

~~~
2020-06-10 07:47:44,653-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] START, MergeVDSCommand(HostName = HOST-virt1, MergeVDSCommandParameters:{hostId='4653ba39-cb4d-4c35-9caa-0fa38b583792', vmId='e246d259-b386-4ef8-a047-909a97be6c89', storagePoolId='941567b0-8a4e-11e1-aff8-777f93db9152', storageDomainId='1f48f887-dd49-4363-9e5c-603c007a9baf', imageGroupId='eb12f2bc-b98d-4c3a-9d75-c31184b22a1d', imageId='6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', baseImageId='75e0a259-d0bd-47ff-ad6a-8f292968e99d', topImageId='6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', bandwidth='0'}), log id: 37125211
2020-06-10 07:47:44,656-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Failed in 'MergeVDS' method
2020-06-10 07:47:44,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM HOST-virt1 command MergeVDS failed: Drive image file could not be found
2020-06-10 07:47:44,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand' return value 'StatusOnlyReturn [status=Status [code=13, message=Drive image file could not be found]]'
2020-06-10 07:47:44,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] HostName = HOST-virt1
2020-06-10 07:47:44,663-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Command 'MergeVDSCommand(HostName = HOST-virt1, MergeVDSCommandParameters:{hostId='4653ba39-cb4d-4c35-9caa-0fa38b583792', vmId='e246d259-b386-4ef8-a047-909a97be6c89', storagePoolId='941567b0-8a4e-11e1-aff8-777f93db9152', storageDomainId='1f48f887-dd49-4363-9e5c-603c007a9baf', imageGroupId='eb12f2bc-b98d-4c3a-9d75-c31184b22a1d', imageId='6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', baseImageId='75e0a259-d0bd-47ff-ad6a-8f292968e99d', topImageId='6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Drive image file could not be found, code = 13
2020-06-10 07:47:44,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] FINISH, MergeVDSCommand, return: , log id: 37125211
2020-06-10 07:47:44,666-04 INFO  [org.ovirt.engine.core.bll.MergeCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Running command: MergeCommand internal: true. Entities affected :  ID: 1f48f887-dd49-4363-9e5c-603c007a9baf Type: Storage
2020-06-10 07:47:44,667-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] START, MergeVDSCommand(HostName = HOST-virt1, MergeVDSCommandParameters:{hostId='4653ba39-cb4d-4c35-9caa-0fa38b583792', vmId='e246d259-b386-4ef8-a047-909a97be6c89', storagePoolId='941567b0-8a4e-11e1-aff8-777f93db9152', storageDomainId='1f48f887-dd49-4363-9e5c-603c007a9baf', imageGroupId='708bec02-c56e-49e6-880f-b4341ceb865f', imageId='c9d23c2e-f84d-4e06-856c-71f6739f79ab', baseImageId='2cf64b4e-a034-46bc-8755-5cac3d33b5e1', topImageId='c9d23c2e-f84d-4e06-856c-71f6739f79ab', bandwidth='0'}), log id: 31a2170f
2020-06-10 07:47:44,670-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Failed in 'MergeVDS' method
2020-06-10 07:47:44,663-04 ERROR [org.ovirt.engine.core.bll.MergeCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Drive image file could not be found, code = 13 (Failed with error imageErr and code 13)
        at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:118) [bll.jar:]
        at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2112) [bll.jar:]
        at org.ovirt.engine.core.bll.MergeCommand.executeCommand(MergeCommand.java:45) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1326) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2001) [bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:]


2020-06-10 07:47:44,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand' return value 'StatusOnlyReturn [status=Status [code=13, message=Drive image file could not be found]]'
2020-06-10 07:47:44,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] HostName = HOST-virt1
2020-06-10 07:47:44,672-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Command 'MergeVDSCommand(HostName = HOST-virt1, MergeVDSCommandParameters:{hostId='4653ba39-cb4d-4c35-9caa-0fa38b583792', vmId='e246d259-b386-4ef8-a047-909a97be6c89', storagePoolId='941567b0-8a4e-11e1-aff8-777f93db9152', storageDomainId='1f48f887-dd49-4363-9e5c-603c007a9baf', imageGroupId='708bec02-c56e-49e6-880f-b4341ceb865f', imageId='c9d23c2e-f84d-4e06-856c-71f6739f79ab', baseImageId='2cf64b4e-a034-46bc-8755-5cac3d33b5e1', topImageId='c9d23c2e-f84d-4e06-856c-71f6739f79ab', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Drive image file could not be found, code = 13
2020-06-10 07:47:44,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] FINISH, MergeVDSCommand, return: , log id: 31a2170f
2020-06-10 07:47:44,672-04 ERROR [org.ovirt.engine.core.bll.MergeCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-9) [5b06be20-1408-4892-a6a4-2ffe6366bebb] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Drive image file could not be found, code = 13 (Failed with error imageErr and code 13)
        at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:118) [bll.jar:]
        at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2112) [bll.jar:]
        at org.ovirt.engine.core.bll.MergeCommand.executeCommand(MergeCommand.java:45) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1326) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2001) [bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:]
~~~

[I2] Failure on the VDSM side.

~~~
2020-06-10 07:47:44,656-0400 INFO  (jsonrpc/6) [api.virt] START merge(drive={u'imageID': u'eb12f2bc-b98d-4c3a-9d75-c31184b22a1d', u'volumeID': u'6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', u'domainID': u'1f48f887-dd49-4363-9e5c-603c007a9baf', u'poolID': u'941567b0-8a4e-11e1-aff8-777f93db9152'}, baseVolUUID=u'75e0a259-d0bd-47ff-ad6a-8f292968e99d', topVolUUID=u'6ea11a8a-3ef1-46e9-bc2d-5ad528b0a389', bandwidth=u'0', jobUUID=u'509b885b-6ba6-4bb3-81c8-c5975ead489e') from=::ffff:XXX.XXX.XX.XXX,XXXXX, flow_id=5b06be20-1408-4892-a6a4-2ffe6366bebb, vmId=e246d259-b386-4ef8-a047-909a97be6c89 (api:48)
2020-06-10 07:47:44,657-0400 INFO  (jsonrpc/6) [api.virt] FINISH merge return={'status': {'message': 'Drive image file could not be found', 'code': 13}} from=::ffff:XXX.XXX.XX.XXX,XXXXX, flow_id=5b06be20-1408-4892-a6a4-2ffe6366bebb, vmId=e246d259-b386-4ef8-a047-909a97be6c89 (api:54)
2020-06-10 07:47:44,657-0400 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.merge failed (error 13) in 0.00 seconds (__init__:312)

2020-06-10 07:47:44,671-0400 INFO  (jsonrpc/5) [api.virt] START merge(drive={u'imageID': u'708bec02-c56e-49e6-880f-b4341ceb865f', u'volumeID': u'c9d23c2e-f84d-4e06-856c-71f6739f79ab', u'domainID': u'1f48f887-dd49-4363-9e5c-603c007a9baf', u'poolID': u'941567b0-8a4e-11e1-aff8-777f93db9152'}, baseVolUUID=u'2cf64b4e-a034-46bc-8755-5cac3d33b5e1', topVolUUID=u'c9d23c2e-f84d-4e06-856c-71f6739f79ab', bandwidth=u'0', jobUUID=u'1087ae45-141b-4683-8d8f-463687e4bcc9') from=::ffff:XXX.XXX.XX.XXX,XXXXX, flow_id=5b06be20-1408-4892-a6a4-2ffe6366bebb, vmId=e246d259-b386-4ef8-a047-909a97be6c89 (api:48)
2020-06-10 07:47:44,671-0400 INFO  (jsonrpc/5) [api.virt] FINISH merge return={'status': {'message': 'Drive image file could not be found', 'code': 13}} from=::ffff:XXX.XXX.XX.XXX,XXXXX, flow_id=5b06be20-1408-4892-a6a4-2ffe6366bebb, vmId=e246d259-b386-4ef8-a047-909a97be6c89 (api:54)
2020-06-10 07:47:44,671-0400 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.merge failed (error 13) in 0.00 seconds (__init__:312)
@harajukakei
Copy link

same things occur to me as well, but with different setup. all of our cluster hosts using CentOS Stream 8 and using NFS & iSCSI backend for storage domain. The second try we delete the snapshot it will completed successully, but If that VM (the one that having failed snapshot deletion and the snapshot are deleted in second try) is turned off, it cant be turned on again with illegal disk error. maybe any workaround for this issue ? because it's very critical for us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alexcrow1974 @harajukakei