Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Infrastructure Flake on Jenkins when building openshift/node image: [Errno -1] repomd.xml does not match metalink for epel #8571

Closed
stevekuznetsov opened this issue Apr 20, 2016 · 67 comments · May be fixed by openshift/vagrant-openshift#497
Closed

Infrastructure Flake on Jenkins when building openshift/node image: [Errno -1] repomd.xml does not match metalink for epel #8571

stevekuznetsov opened this issue Apr 20, 2016 · 67 comments · May be fixed by openshift/vagrant-openshift#497
Assignees
@stevekuznetsov
Labels
area/infrastructure area/tests kind/test-flake Categorizes issue or PR as related to test flakes. priority/P1
Milestone
3.6.0

Comments

@stevekuznetsov
Copy link
Contributor

When building the openshift/node docker image:

--- openshift/node ---
Sending build context to Docker daemon 18.94 kB
Sending build context to Docker daemon 18.94 kB

Step 1 : FROM openshift/origin
 ---> 6e39b07a70f9
Step 2 : MAINTAINER Devan Goodwin <dgoodwin@redhat.com>
 ---> Running in 73d875dde0c7
 ---> 4837e7ea6aff
Removing intermediate container 73d875dde0c7
Step 3 : ADD https://copr.fedoraproject.org/coprs/maxamillion/origin-next/repo/epel-7/maxamillion-origin-next-epel-7.repo /etc/yum.repos.d/

 ---> 14cab38052e9
Removing intermediate container 9eb5cda1db50
Step 4 : RUN INSTALL_PKGS="libmnl libnetfilter_conntrack openvswitch     libnfnetlink iptables iproute bridge-utils procps-ng ethtool socat openssl     binutils xz kmod-libs kmod sysvinit-tools device-mapper-libs dbus     ceph-common iscsi-initiator-utils" &&     yum install -y $INSTALL_PKGS &&     rpm -V $INSTALL_PKGS &&     yum clean all
 ---> Running in 5dfb4fa924e4
Loaded plugins: fastestmirror, ovl
http://mirror.cc.columbia.edu/pub/linux/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
...
...


 One of the configured repositories failed (Extra Packages for Enterprise Linux 7 - x86_64),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable epel

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=epel.skip_if_unavailable=true

failure: repodata/repomd.xml from epel: [Errno 256] No more mirrors to try.

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/208/console
@stevekuznetsov stevekuznetsov added priority/P3 area/tests kind/test-flake Categorizes issue or PR as related to test flakes. priority/P2 and removed priority/P3 labels Apr 20, 2016
@stevekuznetsov stevekuznetsov mentioned this issue Apr 20, 2016
Merged
@0xmichalis 0xmichalis mentioned this issue May 9, 2016
Merged
@cgwalters
Copy link
Member

See also https://lists.centos.org/pipermail/ci-users/2016-April/thread.html

This was referenced May 9, 2016
Merged
Merged
@stevekuznetsov
Copy link
Contributor Author

This has gotten worse ...

http://s3-mirror-us-east-1.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.es.its.nyu.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://download-i2.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.cc.columbia.edu/pub/linux/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.cs.pitt.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://lug.mtu.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.oss.ou.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://ftp.osuosl.org/pub/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://linux.mirrors.es.net/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://kdeforge2.unl.edu/mirrors/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.cs.princeton.edu/pub/mirrors/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.unl.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirrors.rit.edu/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://mirror.umd.edu/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://archive.linux.duke.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.math.princeton.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.us.leaseweb.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.sfo12.us.leaseweb.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://fedora-epel.mirror.lstn.net/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://mirrors.kernel.org/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.datto.com/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://mirrors.xmission.com/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.cogentco.com/pub/linux/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirrors.syringanetworks.net/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.nexcess.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.metrocast.net/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://reflector.westga.edu/repos/Fedora-EPEL/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.symnds.com/distributions/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirrors.tummy.com/pub/fedora.redhat.com/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://epel.wallawalla.edu/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.prgmr.com/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://muug.ca/mirror/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://fedora.westmancom.com/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://mirror.cpsc.ucalgary.ca/mirror/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.


 One of the configured repositories failed (Extra Packages for Enterprise Linux 7 - x86_64),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable epel

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=epel.skip_if_unavailable=true

failure: repodata/repomd.xml from epel: [Errno 256] No more mirrors to try.
http://s3-mirror-us-east-1.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.es.its.nyu.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://download-i2.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.cc.columbia.edu/pub/linux/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.cs.pitt.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://lug.mtu.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.oss.ou.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://ftp.osuosl.org/pub/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://linux.mirrors.es.net/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://kdeforge2.unl.edu/mirrors/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.cs.princeton.edu/pub/mirrors/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.unl.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirrors.rit.edu/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://mirror.umd.edu/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://archive.linux.duke.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.math.princeton.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.us.leaseweb.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.sfo12.us.leaseweb.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://fedora-epel.mirror.lstn.net/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://mirrors.kernel.org/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.datto.com/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://mirrors.xmission.com/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.cogentco.com/pub/linux/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirrors.syringanetworks.net/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.nexcess.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.metrocast.net/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://reflector.westga.edu/repos/Fedora-EPEL/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.symnds.com/distributions/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirrors.tummy.com/pub/fedora.redhat.com/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://epel.wallawalla.edu/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.prgmr.com/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://muug.ca/mirror/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://fedora.westmancom.com/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://mirror.cpsc.ucalgary.ca/mirror/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel

@tdawson did you say this was just EPEL mirrors being out of sync with the source? Can we mitigate failures like this in CI?

@stevekuznetsov stevekuznetsov mentioned this issue Jun 20, 2016
Merged
@stevekuznetsov
Copy link
Contributor Author

Upgrading this since we're seeing quite a lot of it.

@danmcp @tdawson is there any possible way for us to latch onto a stable EPEL mirror from RH so our CI doesn't suffer needlessly? Can we cache the RPMs we're actually using somewhere?

@danmcp
Copy link

danmcp commented Jun 21, 2016

Mirrors are certainly possible. @tdawson Do you have any particular recommendations or advice on the best approach?

@stevekuznetsov stevekuznetsov mentioned this issue Jun 21, 2016
Merged
@sdodson
Copy link
Member

sdodson commented Jun 21, 2016

@stevekuznetsov Can you try adding a yum clean all prior to attempting to use the repos?

@cgwalters
Copy link
Member

It has nothing to do with the yum caches - due to the way the repo is managed updates will always be racy. It would be a lot less racy if 2 copies of the RPM data were cached and the RPMs were sync'd before the repodata.

@tdawson
Copy link
Member

tdawson commented Jun 21, 2016

I'd say use openshifts mirrors, they have epel mirrored and are only updated once a day, so you won't get a weird update halfway through your build.

Instead of installing epel-release, do the following

RUN wget -O /etc/yum.repos.d/epel7.repo https://mirror.openshift.com/mirror/epel/epel7.repo

@0xmichalis 0xmichalis mentioned this issue Jun 22, 2016
Merged
@stevekuznetsov
Copy link
Contributor Author

We can address this by switching to the mirror.openshfit.com EPEL mirror for AMI builds, etc, but we will need oc ex dockerbuild capability to inject secrets (sslclientcert/key) for the mirror into our Docker builds, where we see the majority of these flakes.

This will be blocked on @smarterclayton implementing that secrets feature.

@tdawson
Copy link
Member

tdawson commented Jun 22, 2016 •
edited

You do not need the certs/key for that area https://mirror.openshift.com/mirror/epel/

Since there is nothing private in that repo, you do not need a certificate to access it.

If you are getting failures when you are accessing them, please let me know.

@stevekuznetsov
Copy link
Contributor Author

@tdawson ah, incredible. I assumed I would need them due to some other repos on the domain needing them.

@smarterclayton since our internal mirror doesn't need the secrets, can you see anything wrong with simply disabling the upstream EPEL repo in the RHEL base image that I think we use for all of our builds?

@fabianofranz fabianofranz mentioned this issue Mar 13, 2017
Merged
@sosiouxme sosiouxme mentioned this issue Apr 26, 2017
Merged
@soltysh
Copy link
Member

soltysh commented May 2, 2017

Slightly different, but still this:

Failure talking to yum: failure: repodata/repomd.xml from centos-paas-sig-openshift-origin13-rpms: [Errno 256] No more mirrors to try.
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin13/repodata/repomd.xml: [Errno 14] curl#52 - \"Empty reply from server\""

in https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_install/1590/console

@soltysh soltysh mentioned this issue May 2, 2017
Merged
@soltysh soltysh mentioned this issue May 17, 2017
Merged
@stevekuznetsov stevekuznetsov mentioned this issue Jul 5, 2017
Closed
@tbielawa tbielawa mentioned this issue Jul 17, 2017
Merged
@kwoodson kwoodson mentioned this issue Aug 1, 2017
Merged
This was referenced Aug 7, 2017
Merged
Merged
@miminar miminar mentioned this issue Aug 8, 2017
Merged
This was referenced Aug 8, 2017
Merged
Merged
This was referenced Aug 9, 2017
Merged
Merged
This was referenced Aug 9, 2017
Merged
Merged
@dmage dmage mentioned this issue Aug 10, 2017
Merged
@kwoodson kwoodson mentioned this issue Aug 10, 2017
Merged
@juanvallejo juanvallejo mentioned this issue Aug 10, 2017
Merged
This was referenced Aug 11, 2017
Merged
Merged
Merged
@juanvallejo juanvallejo mentioned this issue Aug 11, 2017
Merged
@kwoodson kwoodson mentioned this issue Aug 15, 2017
Merged
@juanvallejo juanvallejo mentioned this issue Sep 13, 2017
Merged
@coreydaley coreydaley mentioned this issue Jan 4, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stevekuznetsov stevekuznetsov

Labels
area/infrastructure area/tests kind/test-flake Categorizes issue or PR as related to test flakes. priority/P1
Projects
None yet
Milestone
3.6.0
Development

Successfully merging a pull request may close this issue.

Add setup-mirrors command mfojtik/vagrant-openshift