Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explore ways to share workdir across resources #3

Open
soichih opened this issue Sep 28, 2017 · 0 comments
Open

Explore ways to share workdir across resources #3

soichih opened this issue Sep 28, 2017 · 0 comments

Comments

@soichih
Copy link
Contributor

soichih commented Sep 28, 2017

Bigred2 / Karst / Carbonate all shares the same file system. There is no point of copying / duplicating workdirs across these resources.

I've been trying to find a way to make inter-resource rsyncing of task directories that happens to be the identical file system to work with no side effects. The most recent attempt to set -K(keep-dirlinks) option on rsync would fail for symlinks inside another directory probably because the directory gets renamed(?) while being processed. --inplace should workaround this, but it didn't.

Instead of relying on rsync, I think I should add some kind of check to see if src/dest workdir is actually the same file system by comparing the inode #, or do something along that line and don't even bother running rsync.

If this works, then user can configure the same workdir for all 3 clusters (BR2/Karst/Cargonate) to reduce disk usage and unnecessary syncing of input data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant