Jenkins flake on build - timeout to external endpoint

[stevekuznetsov]

Seeing failures in the initial build phase here and here:

Step 0 : FROM centos:centos7
 ---> ce20c473cd8a
Step 1 : MAINTAINER Scott Dodson <sdodson@redhat.com>
 ---> Using cache
 ---> 5a9835f89d67
Step 2 : ADD https://copr.fedoraproject.org/coprs/maxamillion/origin-next/repo/epel-7/maxamillion-origin-next-epel-7.repo /etc/yum.repos.d/


Stderr from the command:

Get https://copr.fedoraproject.org/coprs/maxamillion/origin-next/repo/epel-7/maxamillion-origin-next-epel-7.repo: dial tcp 209.132.184.54:443: no route to host
!!! Error in hack/build-images.sh:62
    'docker build -t $1:latest $2' exited with status 1
Call stack:
    1: hack/build-images.sh:62 image(...)
    2: hack/build-images.sh:68 main(...)

Between the flakes on this ping and those on the registry endpoints for integration tests (#5559), I'm wondering if something is wrong with the networking on the devenv ami, or if all of our external dependencies are just spotty this week.

[stevekuznetsov]

Seen a lot of these just now. Manual curl for https://copr.fedoraproject.org/coprs/maxamillion/origin-next/repo/epel-7/maxamillion-origin-next-epel-7.repo succeeds - more evidence that something is wrong with the networking in the devenv ami

[bparees]

This is not an openshift build, it's a docker build of the base openshift images (eg the openvswitch image and the node image). Taking a shot in the dark and passing it to @sdodson in case he has any copr knowledge.

[bparees]

@danmcp also i recall we had an AMI networking issue a month or so back, did you actually find out what package caused that problem or it just went away after you rolled back and hasn't come back since?

[danmcp]

@danmcp also i recall we had an AMI networking issue a month or so back, did you actually find out what package caused that problem or it just went away after you rolled back and hasn't come back since?

@bparees The latter.

[sdodson]

This is not an openshift build, it's a docker build of the base openshift images (eg the openvswitch image and the node image). Taking a shot in the dark and passing it to @sdodson in case he has any copr knowledge.

Fedora infrastructure did have some outages last week which included copr, I can't find a historic calendar of outages to correlate the events though.

Really this package never changes, we could create a local mirror on ci.openshift.redhat.com if that's something we've done before.

[stevekuznetsov]

There were many problems with it last night @sdodson but as I was seeing them come in on Jenkins I also had a curl loop running locally without any issues hitting the endpoint.

[0xmichalis]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/8134/console

[sdodson]

@smarterclayton copr occasionally goes MIA during fedora maintenance outages (it's back now). We could mirror it locally on the jenkins host and use that instead, really all we're using from that repo is the openvswitch package right now.

[smarterclayton]

Closing because this isn't really actionable without undue effort.