Mitigate dockerhub changes rolling out Nov 1st #19477

spiffxp · 2020-10-07T14:16:07Z

Creating this as a placeholder based on discussion in Slack and yesterday's SIG Testing meeting. I'll sketch in what I remember but to defer to @BenTheElder for a plan

Dockerhub is going to rate-limit pulls starting Nov 1st. See https://www.docker.com/pricing/resource-consumption-updates

A pull request is up to two GET requests to the registry URL path ‘/v2//manifests/’. [...] Note that all pull requests, including ones for images you already have, are counted by this method. This is the trade-off for not counting individual layers.

Rate limits for Docker image pulls are based on the account type of the user requesting the image - not the account type of the image’s owner.

Because we are limiting on manifest requests, the number of layers (blob requests) related to a pull is unlimited at this time.

Please see https://docs.docker.com/registry/recipes/mirror/ to run docker/distribution as a mirror

Pull limit is:

anonymous (rate limited per IP) - 100 per 6h (~16/h)
authenticated - 200 per 6h (~33/h)
paid plans - unlimited

Ideas:

invest in a pool of paid dockerhub accounts that can be distributed to all CI nodes and all e2e clusters under test
pull through cache
audit our tests and image builds, and reduce or remove our dependence on any dockerhub-hosted images
steer people toward k8s.gcr.io and reduce friction

We will likely need to fan out audit/change jobs to all SIGs / subprojects.

I think to start with we should ensure merge-blocking kubernetes/kubernetes jobs are safe, since they represent significant CI volume

spiffxp · 2020-10-07T14:20:27Z

/assign @BenTheElder
/sig testing
/priority important-soon
We should survey soon and evaluate priority based on that.

spiffxp · 2020-10-07T14:21:27Z

/area images
/area jobs
for changing jobs
/area prow
for the idea of a pull-through cache

aojea · 2020-10-07T14:33:28Z

If all the images used are here

https://github.com/kubernetes/kubernetes/blob/e99df0e5a75eb6e86123b56d53e9b7ca0fd00419/test/utils/image/manifest.go#L203-L245

There are only 8 images, it seems easy to move them to gcr.io

        configs[BusyBox] = Config{dockerLibraryRegistry, "busybox", "1.29"}
        configs[GlusterDynamicProvisioner] = Config{dockerGluster, "glusterdynamic-provisioner", "v1.0"}
        configs[Httpd] = Config{dockerLibraryRegistry, "httpd", "2.4.38-alpine"}
        configs[HttpdNew] = Config{dockerLibraryRegistry, "httpd", "2.4.39-alpine"}
        configs[Nginx] = Config{dockerLibraryRegistry, "nginx", "1.14-alpine"}
        configs[NginxNew] = Config{dockerLibraryRegistry, "nginx", "1.15-alpine"}
        configs[Perl] = Config{dockerLibraryRegistry, "perl", "5.26"}
        configs[Redis] = Config{dockerLibraryRegistry, "redis", "5.0.5-alpine"}

spiffxp · 2020-10-07T15:27:40Z

@aojea yeah we should probably start there, and mirror these images used as part of e2e tests into k8s.gcr.io (FYI @dims @thockin). I would probably put them in the same repo we're putting other e2e test images

I took a swag at images pulled by kublets during a run of https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-default. It's shell, and I haven't verified whether the specific log line(s) this pulls out represent dockerhub's definition of a pull, or whether it's just the first "real" pull that counts.

$ gsutil cp -r gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/1313838937554292736/artifacts e2e-gce-default
$ ag pulled | sed -e 's/.*message=.*image \\"\([^ ]*\)\\".*/\1/' | grep -v 'rejected\|recycler' | sort | uniq -c | sort -rg | grep -v gcr.io
 488 docker.io/library/busybox:1.29
 143 docker.io/library/httpd:2.4.38-alpine
  69 docker.io/library/nginx:1.14-alpine
  16 docker.io/library/httpd:2.4.39-alpine
   1 docker.io/gluster/glusterdynamic-provisioner:v1.0

spiffxp · 2020-10-07T17:01:58Z

Here's what kubelet pulls for each node in the k8s-infra-prow-build cluster.

$ gcloud logging read 
  'resource.type="k8s_pod" jsonPayload.message=~"Successfully pulled image"'
  --project=k8s-infra-prow-build \
  --freshness=7d \
  --format="value(jsonPayload.message)" \
  | tee k8s-infra-prow-build-node-pulls.txt
$ <k8s-infra-prow-build-node-pulls.txt grep -v gcr.io | cut -d\  -f4 | sort -rg | uniq -c
2094 "alpine:3.6" # averages to 74.7 per 6h

This doesn't catch pulls that are done as part of jobs that run on nodes, still looking...

spiffxp · 2020-10-13T17:54:25Z

Looks like we could stand to mirror golang, node, python and alpine too.

$ cd ~/w/kubernetes/test-infra/config/jobs
$ ag image: | sed -e 's/.*image: //g' | sort | uniq -c | sort | grep -v gcr.io
  39 golang:1.13
   7 node:12
   7 ${dind_image}
   6 golang:1.15
   5 golang:1.14
   4 python:3.7
   4 python:3.6
   4 golang:1.12
   4 alpine
   3 golang:1.15.2
   3 golang:1.15.0
   3 golang:1.11.5
   2 {{e2e_image}}
   2 golang:1.13.10
   2 golang:1.12.5
   1 quay.io/kubespray/kubespray:v2.13.0
   1 quay.io/kubermatic/yamllint:0.1
   1 quay.io/k8s/release-tools:latest
   1 quay.io/k8s/release-tools-centos:latest
   1 python:3.8
   1 python:3.5
   1 python:3
   1 python:2.7
   1 praqma/linkchecker:v9.3.1-154-g22449abb-10
   1 pouchcontainer/pouchlinter:v0.1.2
   1 kubernetes/kops/build-grid.py:270:    if kops_image:
   1 golangci/golangci-lint:v1.29.0
   1 golangci/golangci-lint:v1.26
   1 golangci/golangci-lint:v1.21.0
   1 golangci/golangci-lint:latest
   1 golang:1.14.4
   1 golang:1.14.1
   1 golang:1.11
   1 docker.io/haskell:8.6.5
   1 cypress/base:12.1.0

My concern is we probably don't want to end up paying to serve these much more commonly used images to non-kubernetes projects.

spiffxp · 2020-10-13T17:55:39Z

These might be available on mirror.gcr.io https://cloud.google.com/container-registry/docs/pulling-cached-images

BenTheElder · 2020-10-14T20:48:07Z

Note that the most popular images are in mirror.gcr.io which our CI nodes and our clusters should be generally using*

* not kind, and possibly not docker in docker. the latter is easy to move to this if it's not already.

Something like glusterdynamic-provisioner is most likely not in it.

ameukam · 2020-10-14T22:46:08Z

@spiffxp @claudiubelu Turns out busybox and golang are in mirror.gcr.io:

 gcloud container images list --repository=mirror.gcr.io/library                                                                                                                                                                                                                                                                            
NAME
mirror.gcr.io/library/alpine
mirror.gcr.io/library/bash
mirror.gcr.io/library/buildpack-deps
mirror.gcr.io/library/busybox
mirror.gcr.io/library/centos
mirror.gcr.io/library/chronograf
mirror.gcr.io/library/consul
mirror.gcr.io/library/couchdb
mirror.gcr.io/library/debian
mirror.gcr.io/library/docker
mirror.gcr.io/library/elasticsearch
mirror.gcr.io/library/flink
mirror.gcr.io/library/ghost
mirror.gcr.io/library/golang
mirror.gcr.io/library/haproxy
mirror.gcr.io/library/hello-world
mirror.gcr.io/library/httpd
mirror.gcr.io/library/kong
mirror.gcr.io/library/mariadb
mirror.gcr.io/library/matomo
mirror.gcr.io/library/maven
mirror.gcr.io/library/memcached
mirror.gcr.io/library/mongo
mirror.gcr.io/library/mongo-express
mirror.gcr.io/library/mysql
mirror.gcr.io/library/nginx
mirror.gcr.io/library/node
mirror.gcr.io/library/openjdk
mirror.gcr.io/library/percona
mirror.gcr.io/library/perl
mirror.gcr.io/library/php
mirror.gcr.io/library/postgres
mirror.gcr.io/library/python
mirror.gcr.io/library/rabbitmq
mirror.gcr.io/library/redis
mirror.gcr.io/library/ruby
mirror.gcr.io/library/solr
mirror.gcr.io/library/sonarqube
mirror.gcr.io/library/telegraf
mirror.gcr.io/library/traefik
mirror.gcr.io/library/ubuntu
mirror.gcr.io/library/vault
mirror.gcr.io/library/wordpress
mirror.gcr.io/library/zookeeper

gcloud container images list-tags mirror.gcr.io/library/busybox                                                                                                                                                                                                                                                                              
DIGEST        TAGS         TIMESTAMP
c9249fdf5613  latest       2020-10-14T12:07:34
2ca5e69e244d               2020-09-09T03:38:02
fd4a8673d034  1.31,1.31.1  2020-06-02T23:19:57
dd97a3fe6d72  1.31.0       2019-09-04T21:20:16
e004c2cc521c  1.29         2018-12-26T09:20:43

claudiubelu · 2020-10-15T19:37:10Z

@spiffxp @claudiubelu Turns out busybox and golang are in mirror.gcr.io:

Interesting. But we actually have a new type of issue I didn't think about at the meeting: support for multiple architecture types. And it seems like I'm right:

docker run --rm mplatform/mquery mirror.gcr.io/library/busybox:1.29
Unable to find image 'mplatform/mquery:latest' locally
latest: Pulling from mplatform/mquery
db6020507de3: Pull complete
f11a2bcbeb86: Pull complete
Digest: sha256:e15189e3d6fbcee8a6ad2ef04c1ec80420ab0fdcf0d70408c0e914af80dfb107
Status: Downloaded newer image for mplatform/mquery:latest
Image: mirror.gcr.io/library/busybox:1.29
 * Manifest List: No
 * Supports: amd64/linux

mirror.gcr.io/library/busybox:1.29 is not a manfiest list, unlike the dockerhub counterpart. It's just a linux/amd64 image. This means that we'll still have problems for people trying to test other architecture types. I know that someone is interested in running s390x. @rajaskakodkar

aojea · 2020-10-15T22:02:12Z

support for multiple architecture types

naive question, do you think that those CIs in anothers architectures can hit dockerhub limits?

claudiubelu · 2020-10-16T14:26:23Z

support for multiple architecture types

naive question, do you think that those CIs in anothers architectures can hit dockerhub limits?

Good question, actually.

Doing a grep in test-infra/config/jobs, I can see:

arm64: 3 periodic jobs with an 8h interval: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/kops/kops-periodics-misc.yaml
arm, arm64, ppc64le, s390x presubmits: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/service-catalog/service-catalog-presubmits.yaml , but those jobs are only building images, and their base is already hosted on gcr.io.

Then, there are these boards:

https://testgrid.k8s.io/conformance-arm
https://testgrid.k8s.io/conformance-ppc64le
https://testgrid.k8s.io/conformance-s390x
https://testgrid.k8s.io/sig-node-arm64
https://testgrid.k8s.io/sig-node-ppc64le

The conformance-* boards have periodic jobs. Not sure if sig-node-* jobs are periodic, but they're all running conformance tests.

But from the docker images you've listed, not all of them are commonly used:

Redis: was being used for the guestbook app test, but we've switched to agnhost since. I think we can simply remove that image.
Perl: only used once in kubernetes/test/e2e_node, in a feature-specific, disruptive test, which means that it's rarely used.
GlusterDynamicProvisioner: only used in the test: [sig-storage] Dynamic Provisioning [k8s.io] GlusterDynamicProvisioner should create and delete persistent volumes [fast], which only runs on gke.

That leaves us with 5 images that are being used in most conformance runs: busybox, nginx, nginx-new, httpd, httpd-new. It also depends on how many nodes are in cluster: if there are 2 nodes in the cluster, the images will be pulled twice (we use 2 nodes for Windows test runs, for example).

After that, we'd have to take a look at all the image building jobs. There are quite a few of them unfortunately: https://cs.k8s.io/?q=BASEIMAGE&i=nope&files=&repos= . Typically, those are postsubmit jobs and they don't run that often, but I'm suspecting that there's a higher chance for multiple jobs to run at the end of a cycle.

We could switch to gcr.io base images?

spiffxp · 2020-10-19T19:57:16Z

After that, we'd have to take a look at all the image building jobs

These all end up invoking GCB builds which is where the image pulls would happen. I am less concerned about these, mostly because I doubt they run at a volume that would cause any rate limiting even if they all theoretically ran on the same instance.

I am more concerned about the long tail of jobs on prow's build cluster causing a specific node to get rate-limited.

claudiubelu · 2020-10-20T14:57:26Z

After that, we'd have to take a look at all the image building jobs

These all end up invoking GCB builds which is where the image pulls would happen. I am less concerned about these, mostly because I doubt they run at a volume that would cause any rate limiting even if they all theoretically ran on the same instance.

I am more concerned about the long tail of jobs on prow's build cluster causing a specific node to get rate-limited.

That makes sense, that should be less concerning for us then. Although, GCB could have the same issues since we're not the only ones building images through GCB.

I've read the dockerhub FAQ, and I saw this:

Will Docker offer dedicated plans for open source projects?
Yes, as part of Docker’s commitment to the open source community, we will be announcing
the availability of new open source plans. To apply for an open source plan, complete our
application at: https://www.docker.com/community/open-source/application.

This should apply to us too, right? If so, have we applied yet?

In any case, we can configure the tests to use the mirror.gcr.io images by default (which are only for linux/amd64), and configure the other architecture type jobs to use dockerhub for now. The tests can be configured to use different registries by having a KUBE_TEST_REPO_LIST env variable [1], which will contain a yaml file specifying which registries to use. [2]

[1]

test-infra/kubetest/aksengine.go

Line 1257 in 882eb3f

if err := os.Setenv("KUBE_TEST_REPO_LIST", downloadPath); err != nil {

[2]

test-infra/config/jobs/kubernetes-sigs/sig-windows/sig-windows-config.yaml

Line 15 in b14c489

    
           value: https://raw.githubusercontent.com/kubernetes-sigs/windows-testing/master/images/image-repo-list

The FAQ also mentions a pull-through cache mirror registry. However, it doesn't mention how it behaves with manifest lists. My first guess is that it doesn't handle manifest lists, and just pulls / caches the images for the platform it's on. This would mean that if we would try this option, we'd have different mirrors, one for earch architecture type we're currently testing.

spiffxp · 2020-10-22T18:58:29Z

We should not be referencing mirror.gcr.io directly but configuring docker and containerd to do so https://cloud.google.com/container-registry/docs/pulling-cached-images#configure

k8s-infra-prow-build nodes have containerd setup to use it:

spiffxp@gke-prow-build-pool4-2020082817590115-6b0f3325-c0r9:~$ cat /etc/containerd/config.toml 
# ...snip...
[plugins.cri.registry.mirrors."docker.io"]
  endpoint = ["https://mirror.gcr.io","https://registry-1.docker.io"]

as does k8s-prow-builds (aka the 'default' build cluster)

bentheelder@gke-prow-default-pool-cf4891d4-0178:~$ cat /etc/containerd/config.toml 
# ... snip ...
[plugins.cri.registry.mirrors."docker.io"]
  endpoint = ["https://mirror.gcr.io","https://registry-1.docker.io"]

it is unclear to me whether this is picked up by any pods that try to explicitly run docker commands

spiffxp · 2020-10-22T19:05:20Z

Clusters stood up using kube-up.sh also have this enabled by default: https://github.com/kubernetes/kubernetes/blob/ededd08ba131b727e60f663bd7217fffaaccd448/cluster/gce/config-default.sh#L163-L164

and while under test:
https://github.com/kubernetes/kubernetes/blob/ededd08ba131b727e60f663bd7217fffaaccd448/cluster/gce/config-test.sh#L175-L176

which causes mirror.gcr.io to be set as a registry mirror url here:
https://github.com/kubernetes/kubernetes/blob/ededd08ba131b727e60f663bd7217fffaaccd448/cluster/gce/util.sh#L313-L315

which is then setup as the registry mirror here: https://github.com/kubernetes/kubernetes/blob/ededd08ba131b727e60f663bd7217fffaaccd448/cluster/gce/gci/configure-helper.sh#L1470-L1475

dims · 2020-10-22T21:43:25Z

we are pausing enforcement of the changes to image retention until mid 2021
https://www.docker.com/blog/docker-hub-image-retention-policy-delayed-and-subscription-updates/

spiffxp · 2020-10-22T21:55:30Z

Image retention isn't the concern here, but good to know. They're still planning to move forward with pull rate-limits

ddebroy · 2020-10-26T06:45:05Z

@claudiubelu mentioned this issue in the weekly SIG-Windows meeting last week. I work closely with the Docker Hub team and wanted to call out a few things here:

As @spiffxp correctly mentions above, the note mentioned above applies only to image retention. There are no change in plans with enforcement of pull rate-limits.
As mentioned here, the pull usage will be tracked per IP address. Assuming each node in a test clusters set up by PROW gets their own public IP (i.e. there is no NAT involved), do we expect some of the individual nodes to indeed exceed the published limit for anonymous pulls - 100 pulls / 6 hr?
If the gcr/mirror options do not work out due to lack of support for manifest lists (or some other reason) and the tests do continue to depend on DockerHub, a reliable approach (that won't run into any published limits) would require [i] ability of the test nodes to authenticate (docker login) with a DockerHub user ID and [ii] have the DockerHub user IDs exempted from the pull limits of authenticated users (200 pulls / 6 hr - rather than 100 pulls / 6 hr) through one of the following:
a) Subscribe each of the DockerHub IDs to an individual Pro Plan on Docker Hub ($60/year/DockerHub user ID).
b) Create a DockerHub Team and make the DockerHub IDs members of the Team (total: $300/year for first 5 DockerHub users and $7/mo for each additional DockerHub user).
c) Work with Docker Inc. to get an exception for the DockerHub Team in (b) so that members of the team are exempted from the pull rate-limit.

BenTheElder · 2020-10-27T21:07:22Z

Note that the GCE e2e clusters (created during testing) are generally using ephemeral IPs so we have no guarantee that a previous user wasn't performing many pulls.

I looked and I think the build clusters at least aren't NATed and the VMs are relatively long-lived.

There's also places other than our CI to consider, e.g. third party CI. We may not want to continue using images from dockerhub while mitigating for ourselves if others have to work around it too, long term. (vs. images on e.g. k8s.gcr.io where users are not limited)

aledbf · 2020-11-02T23:23:39Z

Per https://www.docker.com/blog/expanded-support-for-open-source-software-projects/

We got great feedback from our extensive user base, and adjusted our policies to delay the policies on image retention until mid-2021.

For the approved, non-commercial, open source projects, we are thrilled to announce that we will suspend data pull rate restrictions, where no egress restrictions will apply to any Docker users pulling images from the approved OSS namespaces.

Does this mean this issue could not be a problem anymore?
(for things like kind for instance?)

howardjohn · 2020-11-02T23:55:17Z

The OSS projects agreement has some strings attached:

Joint Marketing Programs

While the publisher retains the Open Source project status, the Publisher agrees to -

Become a Docker public reference for press releases, blogs, webinars, etc

Create joint blogs, webinars and other marketing content

Create explicit links to their Docker Hub repos, with no ‘wrapping’ or hiding sources of their images

Document that Docker Engine or Docker Desktop are required to run their whitelisted images

Give Docker full attribution

source: I applied (a non-kubernetes project) and got an email

BenTheElder · 2020-11-03T00:49:35Z

Huh... thanks @howardjohn

Become a Docker public reference for press releases, blogs, webinars, etc

Sounds reasonable?

Create joint blogs, webinars and other marketing content

I think in our case that might require steering's input ...

Create explicit links to their Docker Hub repos, with no ‘wrapping’ or hiding sources of their images

Seems reasonable.

Document that Docker Engine or Docker Desktop are required to run their whitelisted images

... versus podman, cri-o, containerd etc.? 😕

Give Docker full attribution

Seems unclear. 🤔

dims · 2020-11-03T00:52:23Z

/me wearing my own cap (not steering)

LOL. nice try docker!

spiffxp · 2020-11-03T18:42:03Z

@howardjohn Yeah I am waiting on an answer of whether the benefit we get for this cost is unlimited pulls of other images by our CI. We don't publish to dockerhub, and my read of the benefits was unlimited pulls of our images for other users.

@aledbf if every image we happen to pull from dockerhub has applied and been exempted from rate limiting, that may lessen the impact

mkvitnitsky · 2020-11-04T19:54:00Z

Docker has removed the bottom 2 bullets on @howardjohn's post from all public communications, and replaced it with -

Include information about using Docker, wherever it is appropriate on your website and in your documentation.

I think that this is a better description of the kind of support that we are looking for...

@spiffxp , I have your response on my to do list, and will try to get back to you before the end of today.

spiffxp · 2020-12-03T17:19:40Z

We've gotten reports from downstream users who don't use mirroring in their CI that kubernetes e2e tests are occasionally hitting rate limits. Opened kubernetes/kubernetes#97027 to cover moving those images/tests to a community-owned registry

fejta-bot · 2021-03-03T18:14:21Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

spiffxp · 2021-03-03T19:23:54Z

/remove-lifecycle stale

kubernetes/kubernetes#97027 has probably addressed the bulk of this for kubernetes/kubernetes

We can enforce that jobs should be using k8s-staging or k8s.gcr.io images

Beyond that I think any piecemeal followup for specific sub projects should considered out of scope for this, unless anyone has suggestions

fejta-bot · 2021-06-01T20:52:28Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

BenTheElder · 2021-06-18T03:18:34Z

I think we've only seen very minor issues upstream in Kubernetes, unclear that we need to prioritize anything further here.

For downstream concerns, we will finish migrating any images used by e2e.test in kubernetes.

fejta-bot · 2021-07-18T04:15:36Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

spiffxp · 2021-07-19T13:54:03Z

/remove-lifecycle rotten
/close
Agree that anything further can be opened up as followup issues

k8s-ci-robot · 2021-07-19T13:54:10Z

@spiffxp: Closing this issue.

In response to this:

/remove-lifecycle rotten
/close
Agree that anything further can be opened up as followup issues

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

spiffxp · 2021-08-11T15:29:34Z

/milestone v1.21

k8s-ci-robot assigned BenTheElder Oct 7, 2020

k8s-ci-robot added sig/testing Categorizes an issue or PR as relevant to SIG Testing. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Oct 7, 2020

k8s-ci-robot added area/images area/jobs area/prow Issues or PRs related to prow labels Oct 7, 2020

claudiubelu mentioned this issue Oct 14, 2020

test images: Mirrors dockerhub images to staging kubernetes/kubernetes#95567

Merged

BenTheElder mentioned this issue Oct 22, 2020

Docker Hub's new policy for limiting image pulls kubernetes/kubernetes#94018

Closed

claudiubelu mentioned this issue Oct 22, 2020

use mirror.gcr.io instead of dockerhub images kubernetes/kubernetes#95732

Closed

spiffxp mentioned this issue Oct 30, 2020

Configure kubekins dockerd to use mirror.gcr.io #19780

Merged

BenTheElder mentioned this issue Nov 2, 2020

Move dockerhub kindest/node to a non rate limited registry kubernetes-sigs/kind#1895

Closed

sjenning mentioned this issue Nov 6, 2020

openshift-e2e-tests: override docker.io registries for test images openshift/release#13460

Merged

antoninbas mentioned this issue Nov 20, 2020

[CI] Add Windows and Linux CI scripts for remote Jenkins in Antrea repo antrea-io/antrea#1462

Merged

This was referenced Nov 20, 2020

Investigate if we're impacted by docker hub rate limiting kubernetes-csi/csi-release-tools#115

Closed

Move storage e2es off of busybox kubernetes/kubernetes#96769

Closed

justaugustus mentioned this issue Nov 20, 2020

Establish a staging repository for mirroring Docker Hub images kubernetes/k8s.io#1441

Merged

justaugustus mentioned this issue Dec 2, 2020

[Umbrella] Considerations around a mirror registry kubernetes/sig-release#1369

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 3, 2021

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 3, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 1, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 18, 2021

k8s-ci-robot closed this as completed Jul 19, 2021

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jul 19, 2021

k8s-ci-robot added this to the v1.21 milestone Aug 11, 2021

Mitigate dockerhub changes rolling out Nov 1st #19477

Mitigate dockerhub changes rolling out Nov 1st #19477

Comments

spiffxp commented Oct 7, 2020 • edited Loading

spiffxp commented Oct 7, 2020

spiffxp commented Oct 7, 2020

aojea commented Oct 7, 2020 • edited Loading

spiffxp commented Oct 7, 2020

spiffxp commented Oct 7, 2020 • edited Loading

spiffxp commented Oct 13, 2020

spiffxp commented Oct 13, 2020

BenTheElder commented Oct 14, 2020

ameukam commented Oct 14, 2020

claudiubelu commented Oct 15, 2020

aojea commented Oct 15, 2020 • edited Loading

claudiubelu commented Oct 16, 2020

spiffxp commented Oct 19, 2020

claudiubelu commented Oct 20, 2020 • edited Loading

spiffxp commented Oct 22, 2020 • edited Loading

spiffxp commented Oct 22, 2020

dims commented Oct 22, 2020

spiffxp commented Oct 22, 2020

ddebroy commented Oct 26, 2020

BenTheElder commented Oct 27, 2020

aledbf commented Nov 2, 2020 • edited Loading

howardjohn commented Nov 2, 2020

BenTheElder commented Nov 3, 2020

dims commented Nov 3, 2020

spiffxp commented Nov 3, 2020

mkvitnitsky commented Nov 4, 2020

spiffxp commented Dec 3, 2020

fejta-bot commented Mar 3, 2021

spiffxp commented Mar 3, 2021

fejta-bot commented Jun 1, 2021

BenTheElder commented Jun 18, 2021

fejta-bot commented Jul 18, 2021

spiffxp commented Jul 19, 2021

k8s-ci-robot commented Jul 19, 2021

spiffxp commented Aug 11, 2021

spiffxp commented Oct 7, 2020 •

edited

Loading

aojea commented Oct 7, 2020 •

edited

Loading

spiffxp commented Oct 7, 2020 •

edited

Loading

aojea commented Oct 15, 2020 •

edited

Loading

claudiubelu commented Oct 20, 2020 •

edited

Loading

spiffxp commented Oct 22, 2020 •

edited

Loading

aledbf commented Nov 2, 2020 •

edited

Loading