[Core] Use original ray path instead of using python dir for ray path

[Michaelvll]

Fixes #3335

The issue was caused by the ray package in our kubernetes default image being installed in ~/.local/bin instead of the same dir as python /opt/conda/bin which is likely due to some write permission issue. We now use the original ray path by saving the executable path in a file.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • Reproduceable code in [core] sky launch fails on k8s with bash: /opt/conda/bin/ray: No such file or directory #3335
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[romilbhardwaj]

Thanks @Michaelvll! I can confirm it works with the base image (sky launch -c myclus --cloud kubernetes -- echo hi). However, using a custom image fails:

sky launch -c myclus --image-id docker:ubuntu:latest --cloud kubernetes -- echo hi

...
D 03-20 08:16:45 provisioner.py:585]   File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/provision/instance_setup.py", line 223, in _setup_node
D 03-20 08:16:45 provisioner.py:585]     raise RuntimeError(
D 03-20 08:16:45 provisioner.py:585] RuntimeError: Failed to run setup commands on an instance. (exit code 1). Error: ===== stdout =====
D 03-20 08:16:45 provisioner.py:585]
D 03-20 08:16:45 provisioner.py:585] WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
D 03-20 08:16:45 provisioner.py:585]
D 03-20 08:16:45 provisioner.py:585] Reading package lists...
D 03-20 08:16:45 provisioner.py:585] Building dependency tree...
D 03-20 08:16:45 provisioner.py:585] Reading state information...
D 03-20 08:16:45 provisioner.py:585] gcc is already the newest version (4:11.2.0-1ubuntu1).
D 03-20 08:16:45 provisioner.py:585] patch is already the newest version (2.7.6-7build2).
D 03-20 08:16:45 provisioner.py:585] pciutils is already the newest version (1:3.7.0-6).
D 03-20 08:16:45 provisioner.py:585] fuse is already the newest version (2.9.9-5ubuntu3).
D 03-20 08:16:45 provisioner.py:585] curl is already the newest version (7.81.0-1ubuntu1.15).
D 03-20 08:16:45 provisioner.py:585] rsync is already the newest version (3.2.7-0ubuntu0.22.04.2).
D 03-20 08:16:45 provisioner.py:585] 0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
D 03-20 08:16:45 provisioner.py:585] File ‘Miniconda3-Linux-x86_64.sh’ already there; not retrieving.
D 03-20 08:16:45 provisioner.py:585] bash: conda: command not found
D 03-20 08:16:45 provisioner.py:585] /usr/bin/python3: No module named pip
D 03-20 08:16:45 provisioner.py:585] bash: status: command not found
D 03-20 08:16:45 provisioner.py:585] /usr/bin/python3: No module named pip
D 03-20 08:16:45 provisioner.py:585] /usr/bin/python3: No module named pip
D 03-20 08:16:45 provisioner.py:585] /usr/bin/python3: No module named pip
D 03-20 08:16:45 provisioner.py:585] /usr/bin/python3: No module named pip

Perhaps we also need to setup paths for pip?

EDIT - I was running sky local up on my apple silicon machine, which has issues with custom images if the image isn't built for arm64. I tested on GKE and using custom image also works fine. LGTM!

[concretevitamin]

Not sure I understand the problem & fix. Q: why did this issue show up in K8s but not on cloud VM tests?

[Michaelvll]

Not sure I understand the problem & fix. Q: why did this issue show up in K8s but not on cloud VM tests?

Just updated the PR description: "The issue was caused by the ray package in our kubernetes default image being installed in ~/.local/bin instead of the same dir as python /opt/conda/bin which is likely due to some write permission issue."