Temporary name resolution failures on plugin BOM builds

[MarkEWaite]

Service(s)

ci.jenkins.io

Summary

The plugin bill of materials generates a new release when a maintainer (like me) runs the build on the master branch. Unfortunately, the master branch has failed the last few builds with an agent message "Temporary failure in name resolution"

apache/airflow#22319 (comment) suggests that we may need to configure the kubedns component with more replicas or we may need to configure it to autoscale when heavily loaded. I don't know how to do that, but the Kubernetes documentation at https://kubernetes.io/docs/tasks/administer-cluster/dns-horizontal-autoscaling/ may be helpful

Reproduction steps

1. Open https://ci.jenkins.io/job/Tools/job/bom/job/master/1668/ or https://ci.jenkins.io/job/Tools/job/bom/job/master/1666/ and confirm that the job failed due to a temporary name resolution failure

[lemeurherve]

I've manually restarted a build with withArtifactCachingProxy(false) to unblock it until I have the time to check this error, also noted by @dduportal in #3481 (comment)

[lemeurherve]

I've removed the artifact caching proxy label from this issue as it doesn't seem related since the restarted build failed with the same error without it:

[ERROR] Unresolveable build extension: Plugin org.jenkins-ci.tools:maven-hpi-plugin:3.41 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.jenkins-ci.tools:maven-hpi-plugin:jar:3.41: The following artifacts could not be resolved: org.jenkins-ci.tools:maven-hpi-plugin:pom:3.41 (absent): Could not transfer artifact org.jenkins-ci.tools:maven-hpi-plugin:pom:3.41 from/to repo.jenkins-ci.org (https://repo.jenkins-ci.org/public/): repo.jenkins-ci.org: Temporary failure in name resolution: Unknown host repo.jenkins-ci.org: Temporary failure in name resolution -> [Help 2]

[dduportal]

@lemeurherve can you check the logs of the core-dns application on the cluster cik8s ? Also check its metrics in datadog for this cluster.

Looks like the cluster's internal DNS is not robust

[dduportal]

As compensating measures:

• chore(cik8s,doks): switch to official Datadog Docker image without custom checks kubernetes-management#3897 removed the "custom datadog probes" which were killing the CoreDNS service on cik8s
• chore: remove custom Datadog HTTP checks from eks-public cluster kubernetes-management#3920 does the same for eks-public (ACP for AWS)

[dduportal]

The builkds https://ci.jenkins.io/job/Tools/job/bom/job/master/1683/ and https://ci.jenkins.io/job/Tools/job/bom/job/master/1678/ ran without DNS errors: we can close this issue.