[Spot] Event callback for adding notification

[Michaelvll]

A user would like to have a customizable event callback for the spot jobs, so that they can implement some notification when the spot job status change.

Contributing one idea that would enable us to implement some custom notifications when spot jobs are killed / gone into error states: can we add some event_callback configuration to a job’s yaml config? For now this could be restricted to managed spot jobs and the spot controller could be responsible for doing this.

Example:

event_callback:
  # some env variable will denote what the event is about, e.g. some change of status
  # custom user code can trigger notification / write logs etc
  send_slack_message $SKYPILOT_JOB_ID $JOB_STATUS

[MaoZiming]

For sending notifications (via Slack or Email), does SkyPilot have an existing mechanism? If not we need to register SkyPilot for Slack API and collect user info (slack id or email).

[Michaelvll]

For sending notifications (via Slack or Email), does SkyPilot have an existing mechanism? If not we need to register SkyPilot for Slack API and collect user info (slack id or email).

We don't have such a mechanism in SkyPilot. As the first step, we probably don't need to support such functionality out-of-the-box. Instead, we can just support the callback field, so that the user can provide their own tool/script to do the message sending.

[MaoZiming]

Thanks. Maybe something like:

event_callback:
  callback_script: /path/to/script $JOB_ID $JOB_STATUS

The controller could launch /path/to/script/ as a subprocess with input JOB_ID and JOB_STATUS.