Ansible-runner: list artifacts one by one

[mnecas]

The artifacts created by ansible-runner have the id prefix, which we used to gather the events.
We were listing new artifacts by counting the number of previously processed artifacts and checking the new artifacts from that number as a prefix id. This approach causes multiple issues with async execution where the artifacts are not created in order but there could be a bit of time delay between them. For example, artifact 4 was created before 3.

The new approach fixes this issue by waiting for each file with an id prefix until the playbook finishes or times out.

[ahadas]

The artifacts created by ansible-runner have the id prefix, which we used to gather the events. We were listing new artifacts by counting the number of previously processed artifacts and checking the new artifacts from that number as a prefix id. This approach causes multiple issues with async execution where the artifacts are not created in order but there could be a bit of time delay between them. For example, artifact 4 was created before 3.

The new approach fixes this issue by waiting for each file with an id prefix until the playbook finishes or times out.

so IIUC this means that we might have missed an artifact before and now we'll get all artifacts in order - how does it solve the problem that we get the same output more than once?

[michalskrivanek]

so IIUC this means that we might have missed an artifact before and now we'll get all artifacts in order

we didn't miss it, we just processed it out of order and sometimes twice

• how does it solve the problem that we get the same output more than once?

now we won't process events twice so we won't have same output twice

[mnecas]

/ost

[ahadas]

so IIUC this means that we might have missed an artifact before and now we'll get all artifacts in order

we didn't miss it, we just processed it out of order and sometimes twice

would you be so kind to explain how we may end up in this state (of processing something twice)? is it because the lastEventId was not set to the latest one in the sorted list of events?

I have quite a few comments on the code itself, but it would help if I'll better understand the root cause which is the more important part here first

[mnecas]

so IIUC this means that we might have missed an artifact before and now we'll get all artifacts in order

we didn't miss it, we just processed it out of order and sometimes twice

would you be so kind to explain how we may end up in this state (of processing something twice)? is it because the lastEventId was not set to the latest one in the sorted list of events?

When running an async task in ansible runner it generates the log after it is finished. The code before had a counter of already processed files and the counter was used as an ID prefix of the log files. So if we would have a playbook that would create normal 3 logs and the async task would be in the middle the logs would look like 1 3-> 1 2 3. So the 3 would be processed 2x.

I have quite a few comments on the code itself, but it would help if I'll better understand the root cause which is the more important part here first

If you comments to the patch itself please do comment anything, if you mean the ansible runner in engine, please checkout my refactoring patch #261

[ahadas]

so IIUC this means that we might have missed an artifact before and now we'll get all artifacts in order

we didn't miss it, we just processed it out of order and sometimes twice

would you be so kind to explain how we may end up in this state (of processing something twice)? is it because the lastEventId was not set to the latest one in the sorted list of events?

When running an async task in ansible runner it generates the log after it is finished. The code before had a counter of already processed files and the counter was used as an ID prefix of the log files. So if we would have a playbook that would create normal 3 logs and the async task would be in the middle the logs would look like 1 3-> 1 2 3. So the 3 would be processed 2x.

ah ok, I thought that in this case you do lastEventId = 3 and therefore miss '2'
where is the code that sets the lastEventId to the count of events that were already processed?

I have quite a few comments on the code itself, but it would help if I'll better understand the root cause which is the more important part here first

If you comments to the patch itself please do comment anything, if you mean the ansible runner in engine, please checkout my refactoring patch #261

I see that you did one of the things I wanted to suggest there, but not everything - some relates to this patch
I'll comment soon

[ahadas]

/ost