[spark][bugfix] Fix Ray on Spark running on layered virtualenv python…

… environment (ray-project#32996)

Supposing we create a virtualenv python environment based on the original python environment,
and Ray is installed in the original python environment,
and we run setup_ray_cluster on the virtualenv python environment,
then ray_exec_path = os.path.join(os.path.dirname(sys.executable), "ray") will get a invalid path,
because the ray script is installed in the original python environment.

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
Signed-off-by: Jack He <jackhe2345@gmail.com>

python/ray/util/spark/start_ray_node.py

@@ -42,7 +42,7 @@
 
     temp_dir = os.path.normpath(temp_dir)
 
-    ray_exec_path = os.path.join(os.path.dirname(sys.executable), "ray")
+    ray_cli_cmd = "ray"
 
     lock_file = temp_dir + ".lock"
     lock_fd = os.open(lock_file, os.O_RDWR | os.O_CREAT | os.O_TRUNC)
@@ -51,7 +51,7 @@
     # same temp directory, adding a shared lock representing current ray node is
     # using the temp directory.
     fcntl.flock(lock_fd, fcntl.LOCK_SH)
-    process = subprocess.Popen([ray_exec_path, "start", *arg_list], text=True)
+    process = subprocess.Popen([ray_cli_cmd, "start", *arg_list], text=True)
 
     def try_clean_temp_dir_at_exit():
         try: