Hetzner Cloud Plugin creates more servers than instance cap

[palacsint]

Jenkins and plugins versions report

Environment

Jenkins: 2.452.2 OS: Linux - 5.4.0-186-generic Java: 11.0.23 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

hetzner-cloud:84.v8acf5510fd35

What Operating System are you using (both controller, and any agents involved in the problem)?

Ubuntu 22.04

Reproduction steps

1. Configure Hetzner cloud plugin with Instance cap of 2.
2. Click "Build Now" 11 times. (We have five on-prem build nodes.)

Expected Results

The plugin creates two server instances.

Actual Results

The plugin creates six server instances.

Anything else?

Jenkins log:

Aug 26, 2024 5:00:06 PM FINE cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
provision(cloud=hetzner-java-slaves,label=java&&docker,excessWorkload=6)
Aug 26, 2024 5:00:07 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
Creating new agent with 1 executors, have 0 running VMs
Aug 26, 2024 5:00:07 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
Creating new agent with 1 executors, have 0 running VMs
Aug 26, 2024 5:00:07 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
Creating new agent with 1 executors, have 0 running VMs
Aug 26, 2024 5:00:08 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
Creating new agent with 1 executors, have 0 running VMs
Aug 26, 2024 5:00:08 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloudResourceManager searchResourceByLabelExpression
Trying to find single resource for label expression 'app=jenkins-java-slave'
Aug 26, 2024 5:00:08 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
Creating new agent with 1 executors, have 0 running VMs
Aug 26, 2024 5:00:09 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
Creating new agent with 1 executors, have 0 running VMs
Aug 26, 2024 5:00:11 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloudResourceManager searchResourceByLabelExpression
Trying to find single resource for label expression 'app=jenkins-java-slave'
Aug 26, 2024 5:00:14 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloudResourceManager searchResourceByLabelExpression
Trying to find single resource for label expression 'app=jenkins-java-slave'
Aug 26, 2024 5:00:16 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloudResourceManager searchResourceByLabelExpression
Trying to find single resource for label expression 'app=jenkins-java-slave'
Aug 26, 2024 5:00:18 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloudResourceManager searchResourceByLabelExpression
Trying to find single resource for label expression 'app=jenkins-java-slave'
Aug 26, 2024 5:00:20 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloudResourceManager searchResourceByLabelExpression
Trying to find single resource for label expression 'app=jenkins-java-slave'
Aug 26, 2024 5:00:26 PM FINE cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
provision(cloud=hetzner-java-slaves,label=java&&docker,excessWorkload=1)
Aug 26, 2024 5:00:27 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
Creating new agent with 1 executors, have 5 running VMs
Aug 26, 2024 5:00:27 PM WARNING cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
Cloud capacity reached (2). Has 5 VMs running, but want 1 more executors
Aug 26, 2024 5:00:36 PM FINE cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
provision(cloud=hetzner-java-slaves,label=java&&docker,excessWorkload=1)
Aug 26, 2024 5:00:37 PM INFO cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
Creating new agent with 1 executors, have 5 running VMs
Aug 26, 2024 5:00:37 PM WARNING cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision
Cloud capacity reached (2). Has 5 VMs running, but want 1 more executors
Aug 26, 2024 5:00:46 PM FINE cloud.dnation.jenkins.plugins.hetzner.HetznerCloud provision

Are you interested in contributing a fix?

No response

[rkosegi]

Hi, thank you for a bug report.
Unfortunately, I'm not aware of any fix for this situation. Thing is, that there is window of opportunity for this to happen (there is delay in Hetzner cloud between the point when you ask for new VM and the point where VM is reported as running).

So if you have scenario where you expect burst usage, then maybe use agent with more executors - then only 1 VM is allocated, capable of handling multiple builds.
Note that idle VMs are deleted regularly to reduce unnecessary cost

[palacsint]

Thank you for the prompt response!

Unfortunately, we cannot increase the number of executors on a VM, as we need to keep builds (and tests) completely isolated.

Additionally, we have other build jobs that run regularly and trigger numerous other builds. If we move these to the Hetzner Cloud, I'm concerned that this could lead to the creation of more VMs than our instance cap allows whenever the cron job schedules them (which happens several times daily). This could potentially result in significant additional costs, or at the very least, make it really difficult to manage and reason about.

I suggest mentioning this behavior in the documentation. It could help others avoid confusion and it might also save hours of debugging for others.

The current documentation says this:

You can place the upward limit to the number of agents that Jenkins may launch from this cloud. This is useful for avoiding surprises in the billing statement.

Having this issue is quite surprising for us after this documentation.

Our current workaround is to limit the available primary IPs to 2 in the Hetzner Cloud Console. This currently ensures that the plugin receives an error message if it tries to create more than two instances.