Shotgrid: reviving archived assets is broken #122

kalisp · 2024-08-13T12:02:01Z

Changelog Description

This PR supports recreation of children Tasks in AYON if deleted Asset is revived in Shotgrid.

Additional info

I don't like this solution too much, open for comments.

Main issue is that revival of assets on SG sends these events (in order) - attribute_change for CHILD task, attribute_change for Asset, revival_entity.
If we are processing first attribute_change, it wants to update non existing Task under non existing Asset.
Right now this throws new MissingParentError exception, that could be handled specifically.

It fails processing event, it creates new source event on shotgrid.event (there is no way how to create new event on processing topic, imho), which gets put on the end of the queue.
Then asset gets recreated, then this new event gets reprocessed (only once), this time hopefully it finds parent asset in AYON, so it finishes successfully (or fails forever).

Testing notes:

create asset and task(s) in AYON
let them be synchronized to SG (easiest to run in this repo service_tools/manage.ps services`)
delete asset in AYON Editor
let it delete asset in SG
go to trash (top right your profile, Trash at the bottom, Asset)
check if asset and tasks under it got recreated in AYON

When asset with tasks is revived in SG, first attribute_change of child task is sent, then attribute_change for parent asset and then only entity_revival. attribute_change of task is changed to create task. attribute_change of asset is skipped, only entity_revival should be processed.

This could be raised if parent (eg. asset) is not yet recreated in AYON. It recreates event, puts it at the end of the queue(failing is not enough) and tries it once more, hopefully after Asset gets already created.

Added filtering only on regular events (starting with 'Leeched'), ignores `Recreated` events

ynbot · 2024-08-13T12:02:19Z

Task linked: AY-5570 shotgrid: reviving archived assets is broken

MilaKudr

This PR works for me.

… enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken

services/leecher/leecher/listener.py

iLLiCiTiT · 2024-08-15T10:29:09Z

services/processor/processor/processor.py

@@ -244,8 +246,47 @@ def start_processing(self):
                            self,
                            payload,
                        )
-
+                    except MissingParentError:


Why we can't recreate parents instead of retriggering the event? Processor should not trigger events, it will add the event to the end of queue which might lead to huge issues as the task might change meanwhile, but you'll reapply changes from older event.

If this https://github.com/ynput/ayon-shotgrid/pull/122/files#r1718231580 happens because of this, then that is another reason against.

BTW is possible missing parent can happen even if it was not "reactivated"? Can it happen if the parents couldn't be synced (e.g. invalid chars of some other issue)?

iLLiCiTiT · 2024-08-16T09:23:41Z

I am really against re-submitting leeched events in processor, it should not touch leeched events at all. At the first time of processing the event there might be other events changing or removing the same entity, so moving the event to the end of queue is a huge risk. There might be a real reason why parent was not synced, and re-submitting of the event can cause infinite loop of the event. It should be handled at the moment when we find out the parents are missing.

There also might be other services listening to leeched events, and re-submitting the events can cause their failure.

I'm not responsible for this addon, but for me this is no way.

Removed weird filtering on description, replaced with subtopic which is currently picked up too. Creates simpler event.

services/shotgrid_common/ayon_shotgrid_hub/__init__.py

services/leecher/leecher/listener.py

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

iLLiCiTiT · 2024-09-10T10:14:07Z

@MilaKudr could you re-test this please?

BigRoy

Cosmetics

services/shotgrid_common/ayon_shotgrid_hub/update_from_shotgrid.py

services/shotgrid_common/ayon_shotgrid_hub/__init__.py

services/processor/processor/processor.py

Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>

MilaKudr · 2024-09-10T15:23:49Z

It is not working for me at the moment, after deleting the task in Ayon I get error messages like:
"2024-09-10 17:18:18.531 | update_from_ayon | remove_sg_entity_from_ayon_event | WARNING: Entity 'a82667b96f8711ef8ccdccd9acc131df' does not have a path to remove from Shotgrid.

… enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken

…iving-archived-assets-is-broken' into enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken

kalisp · 2024-09-27T09:27:00Z

It is not working for me at the moment, after deleting the task in Ayon I get error messages like: "2024-09-10 17:18:18.531 | update_from_ayon | remove_sg_entity_from_ayon_event | WARNING: Entity 'a82667b96f8711ef8ccdccd9acc131df' does not have a path to remove from Shotgrid.

That was sorted out.

Currently its waiting for budget as there is a pushback for current implementation.

It shouldn't modify/create new messages, but re-create full folder structure if necessary. That is not trivial because of current code structure.

Now it tries to add parents in AYON instead of creating new event at the end of the queue.

kalisp · 2024-10-14T15:35:41Z

I tried to rework logic without creating new event just by creating missing parents in the loop. Not most elegant code there though :/.

services/shotgrid_common/utils.py

services/shotgrid_common/ayon_shotgrid_hub/__init__.py

services/shotgrid_common/ayon_shotgrid_hub/update_from_ayon.py

Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>

… enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken

…iving-archived-assets-is-broken' into enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken

retirement_date might come when retiring in SG, but will have filled new_value

Removal of tasks should be working.

MilaKudr

It works for assets and tasks

kalisp added 3 commits August 13, 2024 13:47

Added MissingParentError to processor

5d9213f

This could be raised if parent (eg. asset) is not yet recreated in AYON. It recreates event, puts it at the end of the queue(failing is not enough) and tries it once more, hopefully after Asset gets already created.

Updated query of last event

6e70e33

Added filtering only on regular events (starting with 'Leeched'), ignores `Recreated` events

kalisp requested review from jakubjezek001, iLLiCiTiT and MilaKudr August 13, 2024 12:02

kalisp assigned jakubjezek001 Aug 13, 2024

MilaKudr approved these changes Aug 13, 2024

View reviewed changes

Merge branch 'develop' of https://github.com/ynput/ayon-shotgrid into…

c4e8015

… enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken

iLLiCiTiT reviewed Aug 15, 2024

View reviewed changes

services/leecher/leecher/listener.py Outdated Show resolved Hide resolved

iLLiCiTiT reviewed Aug 15, 2024

View reviewed changes

services/leecher/leecher/listener.py Outdated Show resolved Hide resolved

iLLiCiTiT reviewed Aug 15, 2024

View reviewed changes

Cleaned up logic

26c9e53

Removed weird filtering on description, replaced with subtopic which is currently picked up too. Creates simpler event.

iLLiCiTiT reviewed Aug 16, 2024

View reviewed changes

services/shotgrid_common/ayon_shotgrid_hub/__init__.py Outdated Show resolved Hide resolved

iLLiCiTiT reviewed Aug 16, 2024

View reviewed changes

services/leecher/leecher/listener.py Outdated Show resolved Hide resolved

kalisp and others added 2 commits August 19, 2024 10:54

Formatting

5163b29

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

Formatting change

3fcac8f

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

tatiana-ynput requested review from iLLiCiTiT and MilaKudr September 6, 2024 12:39

BigRoy reviewed Sep 10, 2024

View reviewed changes

kalisp and others added 4 commits September 10, 2024 15:04

Cosmetics fix

8fd3c90

Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>

Cosmetics fix

05e2339

Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>

Fix typo

a08c442

Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>

Fix typo

638105c

Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>

kalisp added 2 commits September 11, 2024 11:58

Merge branch 'develop' of https://github.com/ynput/ayon-shotgrid into…

4b65500

… enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken

Merge remote-tracking branch 'origin/enhancement/AY-5570_shotgrid-rev…

5b857ec

…iving-archived-assets-is-broken' into enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken

Fix typo

a674780

jakubjezek001 assigned kalisp and unassigned jakubjezek001 Sep 26, 2024

kalisp marked this pull request as draft September 27, 2024 09:27

BigRoy requested review from BigRoy and removed request for BigRoy October 1, 2024 21:43

kalisp added 3 commits October 10, 2024 17:40

Merge develop

c449b04

Refactor yank out create_new_ayon_entity

b6985bc

Reworked revival from MissingParentError to parent creation

4c2f5f5

Now it tries to add parents in AYON instead of creating new event at the end of the queue.

kalisp marked this pull request as ready for review October 14, 2024 15:34

tatiana-ynput added the sponsored This is directly sponsored by a client or community member label Oct 15, 2024

BigRoy reviewed Oct 17, 2024

View reviewed changes

jakubjezek001 added the type: bug Something isn't working label Oct 24, 2024

kalisp and others added 9 commits November 12, 2024 13:00

Formatting change

a0eaeaa

Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>

Merge branch 'develop' of https://github.com/ynput/ayon-shotgrid into…

ba4849c

… enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken

Merge remote-tracking branch 'origin/enhancement/AY-5570_shotgrid-rev…

fe9d09f

…iving-archived-assets-is-broken' into enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken

Updated docstring

2ff9b42

Updated logging

0da0e9e

Return path for logging

af426a9

Update logging

af870c9

Fix logic to change only real Task revival

759bd0a

retirement_date might come when retiring in SG, but will have filled new_value

Removed filtering on tasks

72f1393

Removal of tasks should be working.

kalisp requested a review from BigRoy November 12, 2024 15:14

MilaKudr approved these changes Nov 12, 2024

View reviewed changes

Fix ruff

1281ba1

kalisp merged commit 8e3b0db into develop Nov 13, 2024
1 check passed

kalisp deleted the enhancement/AY-5570_shotgrid-reviving-archived-assets-is-broken branch November 13, 2024 10:17

jakubjezek001 mentioned this pull request Dec 4, 2024

Services: Task removal/retiring is not synchronizing #106

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shotgrid: reviving archived assets is broken #122

Shotgrid: reviving archived assets is broken #122

kalisp commented Aug 13, 2024

ynbot commented Aug 13, 2024

MilaKudr left a comment

iLLiCiTiT Aug 15, 2024 •

edited

Loading

iLLiCiTiT commented Aug 16, 2024

iLLiCiTiT commented Sep 10, 2024

BigRoy left a comment

MilaKudr commented Sep 10, 2024

kalisp commented Sep 27, 2024

kalisp commented Oct 14, 2024

MilaKudr left a comment

Shotgrid: reviving archived assets is broken #122

Shotgrid: reviving archived assets is broken #122

Conversation

kalisp commented Aug 13, 2024

Changelog Description

Additional info

Testing notes:

ynbot commented Aug 13, 2024

MilaKudr left a comment

Choose a reason for hiding this comment

iLLiCiTiT Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

iLLiCiTiT commented Aug 16, 2024

iLLiCiTiT commented Sep 10, 2024

BigRoy left a comment

Choose a reason for hiding this comment

MilaKudr commented Sep 10, 2024

kalisp commented Sep 27, 2024

kalisp commented Oct 14, 2024

MilaKudr left a comment

Choose a reason for hiding this comment

iLLiCiTiT Aug 15, 2024 •

edited

Loading