[JENKIN-57795] Orphan nodes logic #448
Conversation
…nodes even when its unable to provision more nodes due to hitting the instance cap
|
The test seems to be failing |
|
The tests were passing for me locally. I've added some more ignores as per a suggestion online to see if that clears up the failing PowerMock test. |
| @@ -645,6 +627,21 @@ else if (jenkinsInstance.isTerminating()) { | |||
| } | |||
| } | |||
|
|
|||
| private void attachSlavesToJenkins(Jenkins jenkins, List<EC2AbstractSlave> slaves, SlaveTemplate t) throws IOException { | |||
There was a problem hiding this comment.
| private void attachSlavesToJenkins(Jenkins jenkins, List<EC2AbstractSlave> slaves, SlaveTemplate t) throws IOException { | |
| private static void attachSlavesToJenkins(Jenkins jenkins, List<EC2AbstractSlave> slaves, SlaveTemplate t) throws IOException { |
| import java.io.PrintStream; | ||
| import java.net.URL; | ||
| import java.nio.charset.StandardCharsets; | ||
| import java.util.*; |
There was a problem hiding this comment.
Could you not use star imports?
| String subnetId = chooseSubnetId(); | ||
| String subnetId; | ||
| if (rotateSubnet) { | ||
| subnetId = chooseSubnetId(); |
There was a problem hiding this comment.
can choose subnet id handle this specific?
| List<String> groupIds = null; | ||
| if (!getSecurityGroupSet().isEmpty()) { | ||
| groupIds = getEc2SecurityGroups(ec2); |
There was a problem hiding this comment.
| List<String> groupIds = null; | |
| if (!getSecurityGroupSet().isEmpty()) { | |
| groupIds = getEc2SecurityGroups(ec2); | |
| if (!getSecurityGroupSet().isEmpty()) { | |
| List<String> groupIds = getEc2SecurityGroups(ec2); |
| HashMap<RunInstancesRequest, List<Filter>> ret = new HashMap<>(); | ||
| ret.put(riRequest, diFilters); | ||
| return ret; |
There was a problem hiding this comment.
| HashMap<RunInstancesRequest, List<Filter>> ret = new HashMap<>(); | |
| ret.put(riRequest, diFilters); | |
| return ret; | |
| return Collections.singletonMap(riRequest, diFilters); |
…ubnet rotation to chooseSubnetId()
| tagList.add(tagSpecification.clone().withResourceType(ResourceType.Volume)); | ||
| riRequest.setTagSpecifications(tagList); | ||
|
|
||
| return (HashMap<RunInstancesRequest, List<Filter>>) Collections.singletonMap(riRequest, diFilters); |
There was a problem hiding this comment.
The cast should not be required
There was a problem hiding this comment.
Without the cast, there's a type check complaint. I've removed using Collections.singletonMap and just reverted it to using the HashMap with a single entry.
|
@res0nance there seems to be some upper bound dependencies checks failing. Can you explain how those can be fixed? They seem to be coming from other dependencies that this project doesn't manage? |
|
Looks like this was caused by a pipeline library issue (It's failing on master as well), I'll work on fixing it |
|
@res0nance Wondering if there's been any update on this? It seems to be failing with a different error now. |
|
I've fixed the CI on our part it looks like an infra issue right now |
|
I wrote this piece of groovy script to remove orphaned disconnected agents. Specifally seeing EC2 agent on Linux that will stay connected for a good while and than suddenly get stuck on import jenkins.model.Jenkins
import hudson.plugins.ec2.EC2AbstractSlave
println "Cleaning up offline EC2 slaves..."
Jenkins.instance.slaves.each { node ->
if(!(node in EC2AbstractSlave)) return
def computer = node.getComputer()
if (computer && !computer.isOffline()) return
def logs = computer.getLog()
if (!logs) return
if (!"Connection terminated".equals(logs.readLines().last())) return
println "Deleting ${node.name}"
computer.doDoDelete()
}
null |
|
link to see current commit author: https://github.com/jenkinsci/ec2-plugin/commit/fc6528041fa2d6958a265ba96ef73ab7c8d1165d.patch @pyieh Or look into fixing your git user.email config 😅 to get proper credit for your commits 🏆 |
|
@jetersen yeah I realized I was pushing to my personal repo using my work email. Honestly I'm not that concerned with "proper credit". It does show under my name as the committer, albeit not tied to my Github account so no worries. |
|
@res0nance if everything looks good, can we get this merged? |
|
@res0nance Wanted to follow up and see if we can get this merged. Thank you |
|
@fcojfernandez Can I get my PR merged? @res0nance approved it a while back, but it hasn't been merged yet. |
|
A big thank-you to pyieh for fixing this annoying bug. I see that the patch has been merged into master, but there hasn't been an official release yet. Does anybody know when it will be released? |
|
Hello Folks, any update on the release with this fix? |
|
This has been released since |
|
Today I tried 1.53 version and the issue is not resolved. However, Jenkins identified the node and the node details screen shows "Launch Agent" Button. But the agent is not running. Please note, this is a windows agent. |
@jetersen how did you use this script? As an init script or a scheduled job running in a very short interval? Since the official fix is not working, I am looking to implement something like this. Thank you |
|
As a pipeline job with a cron schedule that would curl the script console and use a jenkins token from credentials |
|
Thanks @jetersen for the quick reply. |
Addresses issue of orphan nodes not being re-attached when the plugin has already hit an instance cap. Because the orphan re-attachment logic was tied to the EC2Cloud.provision() logic, if the instance cap was hit, then provisioning wasn't even attempted and the orphan re-attachment isn't triggered.
Add functionality to EC2Cloud to attempt to re-attach orphan/stopped nodes even when its unable to provision more nodes due to hitting the instance cap.