[Tune] Sync trial artifacts to cloud

[justinvyu]

Why are these changes needed?

When a remote upload directory is specified, artifacts logged to the trial directory of remote workers are not uploaded. This PR enables artifact syncing to happen in a similar fashion as checkpoint syncing. Whenever a checkpoint is uploaded, also push artifacts to the cloud. Whenever a trial is restored from the cloud, also pull artifacts. When trials complete/pause, upload artifacts.

See user issues:

• [Bug] Tune doesn't sync additional files back to head #22445
• [tune] Make tfevents files smaller  #24737
• https://discuss.ray.io/t/cannot-find-checkpoint-when-gpus-per-trial-0/8824/7

What is an artifact?

Artifacts include everything contained in the trial level directory that is logged by the trainable itself and is not a trial checkpoint.

D = driver syncing
C = checkpoint syncing from trainable
A = artifact syncing from trainable

# Head node
experiment_dir/
	experiment-state.json [D]
	basic-variant-state.json [D]
	callback-state.pkl [D]
	tuner.pkl [D]
	trial_0/
		artifact.pkl [A]
		checkpoint_00000/ [C]
		progress.csv [D]
		results.json [D]
		params.json [D]
		params.pkl [D]
	trial_1/
		progress.csv [D]
		results.json [D]
		params.json [D]
		params.pkl [D]

# Worker node
experiment_dir/
	trial_1/
		artifact.pkl [A]
		checkpoint_00000/ [C]

More Implementation Details

Where do we perform artifact uploading?

• Paused/stopped trial (Trainable.stop)
• Checkpointing trial (Trainable.save)

Where do we perform artifact downloading?

• Trainable.restore
• Only download artifacts for a trial if we’re loading it from a checkpoint. (This handles resume_unfinished, resume_errored cases.)

What about artifacts created by the driver? Who is responsible for uploading these?

• For now: just go with excluding the default logger artifacts, and do double-uploading for user defined callback artifacts.
• Follow-up PR:
  • Callbacks should define what artifacts they will write to (folders, files)
  • Trainables should exclude all these driver callbacks from their upload. This is to prevent double-uploading of artifacts for workers on the head node.
  • For user-defined callbacks, they will need to also supply this, but if they don’t, we’ll do some double-uploading, which doesn’t break anything – just inefficient.
• In the future:
  • We can move the responsibility of the logger artifacts to the trainable. Driver shouldn’t upload anything in the trial level directories.
  • Requires more refactoring of callbacks. Too wide of a scope for this current PR.

Warn users if artifact syncing takes a long time. Tell them to consider reducing the number of saved artifacts, or disable artifact syncing.

What about Train workers that live on different nodes? This will be a follow-up PR.

Known Limitations of this PR

1. sync_artifacts=False will only disable artifacts from being uploaded from worker nodes. The driver node will still upload any artifacts that exist in its trial dirs. See [AIR][Tune] Make SyncConfig(sync_artifacts=False) work fully (for all syncing methods) #32783.
2. Restoring and continuing to append to an artifact isn't supported currently, and this is highlighted in the documentation. See [AIR][Tune] Enable trial artifacts to be restored from cloud and updated correctly #32782.

Related issue number

Closes #30071

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[richardliaw]

We don't want to recommend this generically? I think you want to say specifically that you will want to do this only when you need to append, but for example if the file is to be rewritten (like a checkpoint) it's probably ok right?

[justinvyu]

Actually, even if it's rewritten, as long as the filename is the same, it'll keep getting overwritten by the driver's old copy of the artifact. So, unique files are kind of needed for restore behavior to work.