Add re-submission of tasks during spot interruption disconnects #485
Conversation
blanked
changed the title
|
Ah ok there are some initialization errors in the tests that I have written that didn't occur when I ran the tests locally. I'll take a look at them soon.. |
| } | ||
|
|
||
| @DataBoundConstructor | ||
| public EC2SpotSlave(String name, String spotInstanceRequestId, String templateDescription, String remoteFS, int numExecutors, Mode mode, String initScript, String tmpDir, String labelString, List<? extends NodeProperty<?>> nodeProperties, String remoteAdmin, String jvmopts, String idleTerminationMinutes, List<EC2Tag> tags, String cloudName, int launchTimeout, AMITypeData amiType, ConnectionStrategy connectionStrategy, int maxTotalUses) | ||
| public EC2SpotSlave(String name, String spotInstanceRequestId, String templateDescription, String remoteFS, int numExecutors, Mode mode, String initScript, String tmpDir, String labelString, List<? extends NodeProperty<?>> nodeProperties, String remoteAdmin, String jvmopts, String idleTerminationMinutes, List<EC2Tag> tags, String cloudName, int launchTimeout, AMITypeData amiType, ConnectionStrategy connectionStrategy, int maxTotalUses, boolean restartSpotInterruption) |
There was a problem hiding this comment.
Please don't modify an existing constructor, create a new one and deprecate the older one
There was a problem hiding this comment.
added the new constructor and added @Deprecated annotation to the old constructor
|
|
||
| @RunWith(PowerMockRunner.class) |
There was a problem hiding this comment.
Adding
@PowerMockIgnore({"javax.crypto.*", "org.hamcrest.*", "javax.net.ssl.*", "com.sun.org.apache.xerces.*", "javax.xml.*", "org.xml.*"})
as seen AmazonEC2CloudUnitTest.java and EC2StepTest.java in might resolve the test failure
There was a problem hiding this comment.
oh hey I actually forgot about this PR... let me try that, thanks!
This PR adds a new feature - re-submission of tasks for agents that are disconnected due to spot interruption event in AWS.
Description
Whenever an agent is disconnected, there are checks to determine if it is an unexpected disconnect and if the disconnection is a spot interruption event. If the answer is yes to both, the tasks that were running on the agent will be re-submitted to the queue.
Motivation
Builds may fail due to spot instances being terminated. This PR can help to reduce the number of build failures for spot interruption events.
Notes
This may or may not prevent build failures. There doesn't seem to be any documentation on how tasks can be resubmitted. This PR is inspired by another Jenkins plugin that has the suggested behaviour implemented - https://github.com/jenkinsci/ec2-fleet-plugin/blob/master/src/main/java/com/amazon/jenkins/ec2fleet/EC2FleetAutoResubmitComputerLauncher.java