Fix recursive symlinks when uploading to sky storage

[romilbhardwaj]

Closes #814 by using --no-follow-symlinks flag for aws s3 sync. Sky Storage now ignores all symlinks (like regular file_mounts, I believe).

Tested:

• ./tests/run_smoke_tests.sh test_file_mounts
• pytest test_smoke.py::TestStorageWithCredentials

[concretevitamin]

Thanks!

Sky Storage now ignores all symlinks (like regular file_mounts, I believe).

By ignore, do you mean we now copy symlinks as-is, without following? Please also verify regular file_mounts do this (preferably in a smoke test, using_file_mounts!).

Relatedly: can we also update all relevant docs to reflect this change?

[michaelzhiluo]

For regular filemounts, I think there was a prior PR #431 that does not upload symlinks. Think this is just a nit pertaining to Sky Storage.

[romilbhardwaj]

Thanks!

Sky Storage now ignores all symlinks (like regular file_mounts, I believe).

By ignore, do you mean we now copy symlinks as-is, without following? Please also verify regular file_mounts do this (preferably in a smoke test, using_file_mounts!).

Relatedly: can we also update all relevant docs to reflect this change?

Sorry I should have clarified:

• With Sky Storage - symlinks are not copied.
• With file_mounts - symlinks are copied, but the links are broken. Their targets need to be added as a separate file_mount and the links must be fixed by the user.

I have added test cases for both using_file_mounts and TestStorageWithCredentials in smoke tests. Docs also updated.

(Commits will be pushed after the repo compression is done)

[concretevitamin]

Thanks @romilbhardwaj - some comments.