Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(job_attachments): fix output syncing when using identically named local File System Locations across different OS #295

Merged

Conversation

gahyusuh
Copy link
Contributor

What was the problem/requirement? (What/Why)

We have identified a bug related to output syncing when using identically named local File System Locations across Windows submitter and Linux workers.

The issue occurs in the following scenario:

  1. A queue has storage profiles attached with identically named File System Locations (e.g., FSL1) of the LOCAL type.
  2. A job bundle is submitted on Windows, and the bundle is located under the FSL1 path.
  3. The job runs on a Linux worker. In this case, the input assets are downloaded to the Linux worker's FSL1 as the root path, and the output files are generated under the same root path.

However, our current implementation (see this PR) skips files outside the session directory during output syncing. As a result, the output files generated on the Linux worker are skipped because they are outside the session directory.

What was the solution? (How)

we updated the output syncing logic, adding the line session_dir = local_root when a storage profile path is available. We use this session_dir to filter out any files resolved to a location outside of that directory. If storage profile path is available, we can consider the session_dir to be the storage profile path.

What is the impact of this change?

Ensures that output files generated under the identically named FSL path are properly synced back to S3.

How was this change tested?

Manual E2E testing on both Linux and Windows, and made sure generated output are synced correctly.

Was this change documented?

No.

Is this a breaking change?

No.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@gahyusuh gahyusuh force-pushed the gahyusuh/fix_output_sync_with_fsl branch from 7836e66 to f67d0a1 Compare April 11, 2024 19:40
@gahyusuh gahyusuh marked this pull request as ready for review April 11, 2024 19:40
@gahyusuh gahyusuh requested a review from a team as a code owner April 11, 2024 19:40
marofke
marofke previously approved these changes Apr 11, 2024
Copy link
Contributor

@marofke marofke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this so quickly!

… local File System Locations across different OS

Signed-off-by: Gahyun Suh <132245153+gahyusuh@users.noreply.github.com>
@gahyusuh gahyusuh merged commit 7fcf845 into aws-deadline:mainline Apr 11, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants