-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Attribute error during timeout cancelling transfer #208
Comments
@jsattelb Thanks for reporting! do you want to try to send a simple PR? This looks like trivial error. Adding a test to cover this case would be nice, I think we already have tests in this area. |
The bug was introduced by 03e2f60 renaming TicketCancelTimeout to TransferCancelTimeout without replacing all usages of the old name. |
After changing TicketCancelTimeout to TransferCancelTimeout I still get an Error when backing up a VM (incremental), but now it's a conflict Error: command RemoveImageTicketVDS failed: Image daemon request failed: 'status=409, reason=Conflict, error=Timeout cancelling transfer b1d9c8fd-ed54-4935-968f-200d8cde3c97\n' This is the /var/log/ovirt-imageio/daemon.log af the transfer on the Node:
The problem startet with the last update of our backup software (Veeam), and they say this error does not occure on RHV. |
I still receive multiple status=409 conflict messages throughout the backup process in the oVirt UI, but the backups seem to complete successfully. I haven't seen the backups stuck in a "finalizing" state for nearly three weeks. The response is used to inform the process that there is still an active connection (e.g., backup) in progress and to try again later. That is my understanding of it, at least from acf69ee. I also receive the same backtrace referenced in the debug log when the backup request no longer exists. It's not reported in the oVirt UI, so I ignored it for now. |
What we see in the log (with the fix) is no error on imageio side. If there is an issue it is engine or in the client code.
This means that there active client connections, probably blcoked in recv() on the client socket, waiting for the next request. This means that the client did not close all connections to the image before ending the transfer - this is a client error.
This means that the client closed the connection without reading the entire response. I'm not sure how this happens if the backup is successful, but this is again client errror.
I don't know what is the client code accessing image, but I guess it does this:
What the client should do is:
With this the ticket will removed immediately and the transfer will finish very quickly. The error in imageio should be fix but it is not the root cause. Latest RHV contains the fix that introduced the AttributeError, but maybe Veeam did not test with an older RHV version? |
@jsattelb Regarding RHV - the issue was introduced in imageio 2.4.1 released in the last oVirt 4.4.z version. oVirt 4.5.x is called RHV 4.4 SPy - I'm not sure about the exact numbers. However imageio version is exactly the same in oVirt and RHV. You can ask Veeam which imageio version was tested. |
@nirs Thanks for the detailed analysis and explanation!
Does that mean that latest offical RHV (4.4 SP1 BU3?) has TransferCancelTimeout in tickets.py and latest oVirt (4.5.5) tickets.py has TicketCancelTimeout? Veeam Support told me that 4.4 SP1 is required, but 4.4 SP1 without Batch Update is quite old, so if this was fixed / changed in 4.4 SP1 Batch Update 2 or 3, I think the Veeam System Requirements are incorrect and they tested only with a recent Version. @jsattelb I had stucking and failed backups with first backups after the update to 12.1, but after some days those are gone, without the fix, but they produce this errors and need more time as with 12.0, perhaps a result of the incorrect sequenze they use now. |
I compared ovirt_imageio/_internal/handlers/tickets.py in |
I compiled and downgraded a Node to: |
Since this issue exists since ovirt-imageio 2.4.1 getting "500 Internal Server Error" is not surprising. |
Do you know if: |
I think what you find in ftp.redhat.com is the official package. But more important, ovirt-imageio from github contains exactly the same code you get in oVirt rpms or RHV rpms. There is no special code added in RHV. |
Recently our backup appliance kept receiving a "status=500" from RemoveImageTicketVDS. I believe the actual call should be a status=409, conflict. Changing the reference to TicketCancelTimeout to TransferCancelTimeout in _internal/handlers/tickets.py seems to produce the desired event.
oVirt Event Log (before):
oVirt Event Log (after):
File: /var/log/ovirt-imageio/daemon.log (before)
File: /var/log/ovirt-imageio/daemon.log (after)
The text was updated successfully, but these errors were encountered: