Skip to content
This repository has been archived by the owner on Jan 16, 2024. It is now read-only.

feat: Kill spot instance requests #14

Merged
merged 1 commit into from
Jul 27, 2023
Merged

Conversation

spalladino
Copy link
Contributor

Cancels spot instance requests at the end of spot_run_script to prevent them from counting against quota. We have been getting the following error in our CI:

An error occurred (MaxSpotInstanceCountExceeded) when calling the RequestSpotInstances operation: Max spot instance count exceeded

And this answer hints at spot requests not closed. So let's see if this fixes it.

@ludamad
Copy link
Collaborator

ludamad commented Jul 27, 2023

Not sure if it applies but a trap is something to consider, it's similar to a BASH finally block (normal execution or a sudden error will both run it)

#!/bin/bash
function on_exit {
    if [ -f "sir-$COMMIT_HASH:$JOB_NAME.txt" ]; then
        SIR=$(cat sir-$COMMIT_HASH:$JOB_NAME.txt)
        echo "Cancelling spot instance request $SIR"
        aws ec2 cancel-spot-instance-requests --spot-instance-request-ids $SIR > /dev/null
    fi
}
trap on_exit EXIT

@ludamad ludamad merged commit 819a987 into master Jul 27, 2023
ludamad added a commit to AztecProtocol/aztec-packages that referenced this pull request Jul 27, 2023
@spalladino spalladino deleted the feat/kill-spot-requests branch July 27, 2023 21:52
@spalladino
Copy link
Contributor Author

I like the idea of using a trap, you should've just pushed it on top of this!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants