Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Allow credential files with the same basename. #2042

Merged
merged 1 commit into from
Jun 6, 2023

Conversation

concretevitamin
Copy link
Member

@concretevitamin concretevitamin commented Jun 6, 2023

With OCI supported and enabled, I encountered the following error on every sky launch:

...
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/backends/backend_utils.py", line 975, in write_cluster_config
    _optimize_file_mounts(tmp_yaml_path)
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/backends/backend_utils.py", line 228, in _optimize_file_mounts
    f'Duplicated src basename: {src_basename}; mounts: {file_mounts}')
AssertionError: Duplicated src basename: config; mounts: {'~/.sky/sky_ray.yml': '/Users/zongheng/.sky/generated/sky-0fe1-zongheng.yml.tmp', '~/.sky/wheels/4ae6677e0222ae99e0d7c13323ab1f16': '/var/folders/8f/56gzvwkd3n3293xjlrztr6600000gp/T/4ae6677e0222ae99e0d7c13323ab1f16', '~/.aws/credentials': '~/.aws/credentials', '~/.azure/azureProfile.json': '~/.azure/azureProfile.json', '~/.azure/clouds.config': '~/.azure/clouds.config', '~/.azure/config': '~/.azure/config', '~/.azure/msal_token_cache.json': '~/.azure/msal_token_cache.json', '~/.config/gcloud/credentials.db': '~/.config/gcloud/credentials.db', '~/.config/gcloud/application_default_credentials.json': '~/.config/gcloud/application_default_credentials.json', '~/.config/gcloud/access_tokens.db': '~/.config/gcloud/access_tokens.db', '~/.config/gcloud/configurations': '~/.config/gcloud/configurations', '~/.config/gcloud/legacy_credentials': '~/.config/gcloud/legacy_credentials', '~/.config/gcloud/active_config': '~/.config/gcloud/active_config', '~/.sky/.sky_gcp_config_default': '~/.sky/.sky_gcp_config_default', '~/.lambda_cloud/lambda_keys': '~/.lambda_cloud/lambda_keys', '~/.ibm/credentials.yaml': '~/.ibm/credentials.yaml', '~/.oci/config': '~/.oci/config', '~/.oci/oracleidentitycloudservice_gengming.chen-06-06-05-01.pem': '~/.oci/oracleidentitycloudservice_gengming.chen-06-06-05-01.pem', '~/.ssh/sky-key.pub': '~/.ssh/sky-key.pub'}

The root cause is because both ~/.oci/config and ~/.azure/config exist locally, and our previous file_mounts optimization couldn't handle basename conflicts.

Fixed by using uuids as basename when doing the file_mounts optimization.

Tested (run the relevant ones):

  • Any manual or new tests for this PR (please specify below)
  • All smoke tests: pytest tests/test_smoke.py: running
  • Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
  • Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

Copy link
Collaborator

@Michaelvll Michaelvll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this @concretevitamin! LGTM.

@concretevitamin
Copy link
Member Author

Smoke tests passed*, merging.

*Re-tried pytest tests/test_smoke.py::test_large_job_queue once. The first time it failed at

[large_job_queue] Failed.
[large_job_queue] Reason: s=$(sky queue t-large-job-queue-3917-e7); echo "$s"; echo; echo; echo "$s" | grep -v grep | grep PENDING | wc -l | grep 43

with the grep count being incorrect. Is it expected?

@concretevitamin concretevitamin merged commit fbd1990 into master Jun 6, 2023
@concretevitamin concretevitamin deleted the credential-base branch June 6, 2023 23:02
@Michaelvll Michaelvll mentioned this pull request Jun 19, 2023
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants