fix(job_attachments): fix output syncing when using identically named local File System Locations across different OS #295
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What was the problem/requirement? (What/Why)
We have identified a bug related to output syncing when using identically named local File System Locations across Windows submitter and Linux workers.
The issue occurs in the following scenario:
However, our current implementation (see this PR) skips files outside the session directory during output syncing. As a result, the output files generated on the Linux worker are skipped because they are outside the session directory.
What was the solution? (How)
we updated the output syncing logic, adding the line
session_dir = local_root
when a storage profile path is available. We use thissession_dir
to filter out any files resolved to a location outside of that directory. If storage profile path is available, we can consider thesession_dir
to be the storage profile path.What is the impact of this change?
Ensures that output files generated under the identically named FSL path are properly synced back to S3.
How was this change tested?
Manual E2E testing on both Linux and Windows, and made sure generated output are synced correctly.
Was this change documented?
No.
Is this a breaking change?
No.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.