[Storage] Add .skyignore support

[yika-luo]

If you have a .skyignore file in your sky working directory, sky will exclude the listed files when uploading your work dir to the sky clusters, and sky will NOT use any of your .gitignore files. However, if you don't have a .skyignore, sky will fallback to use your .gitignore. In other words, sky will never use both .skyignore and .gitignore.

Example .skyignore:

# Files that match pattern under ONE directory
/hello.py
/*.txt
/dir
/dir/*.txt

# Files that match pattern under ALL directories
*.txt
hello.py

Please do NOT use patterns like ./*.txt because these expressions do not behave consistently across the APIs.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Tested sky launch on a simple GCP .yaml and .skyignore, ssh into sky cluster to make sure both workdir and mounted dir include the correct list of files
• Tested sky jobs launch on the same .yaml and .skyignore, check the GCS buckets to make sure both workdir and mounted buckets include the correct list of files

[Michaelvll]

Thanks for fixing this @yika-luo! It is very useful to our users. The PR looks mostly good to me. To expose this option to users, let's add some instructions in our doc as well (can be another PR): https://skypilot.readthedocs.io/en/latest/examples/syncing-code-artifacts.html

[Michaelvll]

Thanks for the updates @yika-luo! It mostly looks good to me now.

[Michaelvll]

Just wondering if this will be too specific, e.g., does test.log match test.log in all subdir; does test*.log match all sub dir? Wondering if we should add **/ to all lines, except the ones start with /
(the newly proposed version might be very slow in a large folder with many files/folders)

Suggested change

	if line.startswith('*.'):
	line = '**/' + line
	if line.startswith('*.'):
	line = '**/' + line

[yika-luo]

test.log and test*.log should be just matching in current dir. That being said, I should remove this logic and requires users to do **/* if they want to match ALL directories. I also added in doc saying users should avoid doing *.txt or ./*.txt

[Michaelvll]

Thanks for adding this support @yika-luo! LGTM with the minor comment of /* vs /.