[Dashboard] Fix dashboard when multiple jobs for single job submission

[nikitavemuri]

Why are these changes needed?

The job submission code currently prevents a single job submission id from being reused across jobs. In the case when there are multiple drivers in a single job submission, only the latest driver should be returned for various observability APIs (

ray/dashboard/modules/job/utils.py

Line 152 in e9109e6

Only the last driver of a submission job is returned.

), however the current code picks any of the drivers causing the API response to not be stable. This PR sorts the job list from GCS so the latest driver is consistently picked.

Related issue number

Closes #46265

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(