KEP-3000: Image Promotion and Distribution Policy #3079

hh · 2021-12-07T10:49:08Z

One-line PR description: KEP-3000: Artifact Promotion and Distribution Policy
Issue link: [KEP] Sharing the load of K8s Artifacts distribution between vendors to reduce the cost burden on the Kubernetes community infrastructure #3055

hh · 2021-12-07T10:56:18Z

ameukam · 2021-12-07T10:58:20Z

keps/sig-architecture/3000-artifact-distribution/README.md

+
+### Non-Goals
+
+Anything related to creation of artifacts, bom, digital signatures.


artifacts

The title of the KEP implies artifacts are concerned by this KEP. We need more clarification about which type of artifacts are non-goals.

keps/sig-architecture/3000-artifact-distribution/kep.yaml

ameukam · 2021-12-07T15:35:51Z

keps/sig-architecture/3000-artifact-distribution/README.md

+Then the promotion process occurs
+```
+
+#### Cloud Customer - Installing K8s via kubeadm


Is it possible to consider non-cloud users ?

The greatest percentage of cost comes from cloud customers, so we will focus on them for this KEP

justaugustus

(Mentioned on Slack first)

@hh -- I've left a few tweaks (which will also fix the presubmits) here: https://github.com/justaugustus/enhancements/tree/MST-3000

hh · 2021-12-07T22:00:50Z

/assign @spiffxp

eddiezane · 2021-12-08T20:53:54Z

/cc

thockin

I know there aare more questions I am forgetting to ask

thockin · 2021-12-08T23:30:06Z

keps/sig-release/3000-artifact-distribution/README.md

+
+### Goals
+
+A policy and procedure for use by SIG Release to promote container images and release binaries to multiple registries and mirrors.


I think we should break this into 2 major phases:

Container images

Other artifacts

We may even want to break it to 2 KEPs so we can "finish" one.

Good thought! This KEP just focuses on container images now.

thockin · 2021-12-08T23:34:33Z

keps/sig-release/3000-artifact-distribution/README.md

+
+## Design Details
+
+### Artifact Promotion


Let's document and consider two main approaches:

Push. Each mirror provider gives us a mechanism and credentials to push container images. As we promote images from staging to prod, we push to all mirrors. We need to consider credential security and rotation.

Pull. We publish a log (git repo?) of image changes and mirrors are expected to sync changes in a reasonable period of time (99p @ 10 mins?).

Opted into a push based mechanism, where sig-k8s-infra manages the content of the buckets

thockin · 2021-12-08T23:45:38Z

keps/sig-release/3000-artifact-distribution/README.md

+
+### Artifact Distribution
+
+#### Policy


We need to detail how to on-board a mirror. E.g.:

provide some guarantees of service (e.g. a contract with CNCF) and point of contact

provide an emergency contact in case of outage

provide a mapping of client IPs to mirror (maybe through a git repo)

Then we can add the mirror and have the front-end server start redirecting traffic.

We will want to periodically healthcheck each mirror (e.g. pull a random blob, measure latency). If HC fails, remove mirror until it passes N times. We need a site or something indicating which mirrors are healthy, maybe stats.

We will want to log all redirects and set up a PII-anonymizing process so we can publish some aggregated information about how much traffic is going to each mirror, top images globally, etc.

provide some guarantees of service (e.g. a contract with CNCF) and point of contact

~~Most of the managed services for Container registries have SLAs. We need to agree about minimum level of SLA.~~ We now plan to usage object storage services

Going off of what @ameukam says here, we are working closely with the providers who consume the most to bring up infra that we manage.

thockin · 2021-12-08T23:46:41Z

keps/sig-release/3000-artifact-distribution/README.md

+
+#### Policy
+
+#### Process


We will need to detail how we turn up the new DNS name and redirector and how we plan to convert users of old GCR name into the new name.

Members of sig-k8s-infra have made PRs against projects like kops and Kubernetes to change the defaults.
There's also e2e testing through changing the domain for various jobs running in Prow.

thockin · 2021-12-09T00:28:57Z

keps/sig-release/3000-artifact-distribution/README.md

+
+Anything related to creation of artifacts, bom, digital signatures.
+
+## Proposal


We should also document that we're explicitly OK with a model where the management of the mirror is opaque to us as long as the other criteria are met.

ameukam · 2021-12-13T19:15:45Z

keps/sig-release/3000-artifact-distribution/kep.yaml

+approvers:
+  - "@ameukam"
+  - "@justaugustus"


Suggested change

approvers:

- "@ameukam"

- "@justaugustus"

approvers:

- "@ameukam"

- "@dims"

- "@justaugustus"

- "@saschagrunert"

updated in 6bfecc8

ameukam · 2021-12-23T00:43:52Z

keps/sig-release/3000-artifact-distribution/README.md

+#### Cloud Customer - Installing K8s via kubeadm
+
+```feature
+As a CLOUD end-user


We should not stick the user story to a cloud environment. We don't want to break the existing way of consuming those container images produced.

Since the most of the spend is only cloud users, this should be fine to focus on for this KEP

Pulled out User Stories for this merge

ameukam · 2021-12-23T00:46:08Z

keps/sig-release/3000-artifact-distribution/README.md

+
+Given some compute resources at CLOUD
+When I use kubeadm to deploy Kubernetes
+Then I will be redirected to a local CLOUD registry


we should be more clear about the meaning of "local":

Suggested change

Then I will be redirected to a local CLOUD registry

Then I will be redirected to the closest network endpoint of the registry

resolved in f960d6f

ameukam · 2021-12-23T00:56:39Z

keps/sig-release/3000-artifact-distribution/README.md

+
+### How much is this going to save us?
+
+![Cost of K8s Artifact hosting - Data Studio Graphs](https://i.imgur.com/LAn4UIE.png)


We should provide a link to the full report and not just a screenshot for better transparency.

Update to show AWS involvement without exposing company specific usage patterns more context, in 7030358.

ameukam · 2021-12-23T01:00:09Z

keps/sig-release/3000-artifact-distribution/README.md

+
+## Infrastructure Needed
+
+It would be good to request some donations for some larger providers, including one in China, via [Cloud Native Credits program](https://www.cncf.io/credits/).


We should provide a list of what exactly is needed in terms of infrastructure to ensure this KEP go to implementable.

keps/sig-release/3000-artifact-distribution/README.md

source: Ben's doc (kubernetes-sigs/oci-proxy/cmd/archeio/docs/request-handling.md)

Co-Authored-By: Riaan Kleinhans <riaan@ii.coop>

linux-foundation-easycla · 2022-05-11T03:55:36Z

The committers listed above are authorized under a signed CLA.

✅ login: hh / name: Hippie Hacker (1919dc5, f15fdd6)
✅ login: justaugustus / name: Stephen Augustus (he/him) (467d84a, bfdee87)
✅ login: BobyMCbobs / name: Caleb Woodbine (300cc1d, b5ffc45, 6e68a6c, 35d575b, 6a1b9bf, f960d6f, 0e5fafb, f074245, 4a77099, 2abfc73, 6bfecc8, 7030358)

hh · 2022-05-11T04:03:48Z

(Mentioned on Slack first)

@hh -- I've left a few tweaks (which will also fix the presubmits) here: https://github.com/justaugustus/enhancements/tree/MST-3000

Thanks!

dims · 2022-05-11T11:25:29Z

@justaugustus @saschagrunert this is ready (all conversations resolved). Please take a look and approve if appropriate.

saschagrunert

/approve

dims · 2022-05-11T20:11:51Z

Let's merge and iterate thanks @BobyMCbobs

/approve
/lgtm

k8s-ci-robot · 2022-05-11T20:12:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, hh, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-release/OWNERS~~ [dims,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

https://github.com/kubernetes/k8s.io/wiki/New-Registry-url-for-Kubernetes-(registry.k8s.io) kubernetes/enhancements#3079

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 7, 2021

k8s-ci-robot requested review from derekwaynecarr and johnbelamaric December 7, 2021 10:49

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. labels Dec 7, 2021

k8s-ci-robot assigned dims and justaugustus Dec 7, 2021

ameukam reviewed Dec 7, 2021

View reviewed changes

justaugustus reviewed Dec 7, 2021

View reviewed changes

keps/sig-architecture/3000-artifact-distribution/kep.yaml Outdated Show resolved Hide resolved

ameukam reviewed Dec 7, 2021

View reviewed changes

keps/sig-architecture/3000-artifact-distribution/kep.yaml Outdated Show resolved Hide resolved

ameukam reviewed Dec 7, 2021

View reviewed changes

justaugustus requested changes Dec 7, 2021

View reviewed changes

hh force-pushed the MST-3000 branch from 3f1060d to 4b61e5a Compare December 7, 2021 21:42

k8s-ci-robot added the sig/release Categorizes an issue or PR as relevant to SIG Release. label Dec 7, 2021

hh force-pushed the MST-3000 branch from 4b61e5a to 1331872 Compare December 7, 2021 21:49

k8s-ci-robot assigned spiffxp Dec 7, 2021

k8s-ci-robot requested a review from eddiezane December 8, 2021 20:53

thockin reviewed Dec 8, 2021

View reviewed changes

thockin self-assigned this Dec 9, 2021

thockin reviewed Dec 9, 2021

View reviewed changes

ameukam reviewed Dec 13, 2021

View reviewed changes

ameukam reviewed Dec 23, 2021

View reviewed changes

keps/sig-release/3000-artifact-distribution/README.md Outdated Show resolved Hide resolved

BobyMCbobs added 4 commits April 28, 2022 16:26

Update readme

f960d6f

Update kep.yaml

0e5fafb

Add request handling

f074245

source: Ben's doc (kubernetes-sigs/oci-proxy/cmd/archeio/docs/request-handling.md)

Fix spelling and update goal

4a77099

dims mentioned this pull request May 10, 2022

Move from k8s.gcr.io to registry.k8s.io kubernetes/kubernetes#109938

Merged

BobyMCbobs and others added 3 commits May 11, 2022 08:41

Fix toc

2abfc73

Update approvers

6bfecc8

Update cost saving analysis section notes

7030358

Co-Authored-By: Riaan Kleinhans <riaan@ii.coop>

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 11, 2022

Removing un-necessary sections

f15fdd6

hh force-pushed the MST-3000 branch from c7d8af1 to f15fdd6 Compare May 11, 2022 03:59

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 11, 2022

update ToC

4b14ce8

saschagrunert approved these changes May 11, 2022

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 11, 2022

k8s-ci-robot assigned dims May 11, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 11, 2022

k8s-ci-robot merged commit 35d3a8d into kubernetes:master May 11, 2022

k8s-ci-robot added this to the v1.25 milestone May 11, 2022

yselkowitz added a commit to yselkowitz/csi-driver-host-path that referenced this pull request Jun 2, 2022

Move from k8s.gcr.io to registry.k8s.io

d4682ce

https://github.com/kubernetes/k8s.io/wiki/New-Registry-url-for-Kubernetes-(registry.k8s.io) kubernetes/enhancements#3079

This was referenced Jun 2, 2022

Move from k8s.gcr.io to registry.k8s.io kubernetes-csi/csi-driver-host-path#357

Merged

Move from k8s.gcr.io to registry.k8s.io kubernetes-csi/csi-release-tools#194

Merged

yselkowitz added a commit to yselkowitz/csi-driver-host-path that referenced this pull request Jun 3, 2022

Move from k8s.gcr.io to registry.k8s.io

bfdb3e0

https://github.com/kubernetes/k8s.io/wiki/New-Registry-url-for-Kubernetes-(registry.k8s.io) kubernetes/enhancements#3079

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-3000: Image Promotion and Distribution Policy #3079

KEP-3000: Image Promotion and Distribution Policy #3079

hh commented Dec 7, 2021

hh commented Dec 7, 2021

ameukam Dec 7, 2021 •

edited

Loading

ameukam Dec 7, 2021

BobyMCbobs May 10, 2022

justaugustus left a comment

hh commented Dec 7, 2021

eddiezane commented Dec 8, 2021

thockin left a comment

thockin Dec 8, 2021

BobyMCbobs May 10, 2022

thockin Dec 8, 2021

BobyMCbobs May 10, 2022

thockin Dec 8, 2021

ameukam Dec 23, 2021 •

edited

Loading

BobyMCbobs May 10, 2022

thockin Dec 8, 2021

BobyMCbobs May 10, 2022

thockin Dec 9, 2021

ameukam Dec 13, 2021

BobyMCbobs May 10, 2022

ameukam Dec 23, 2021

BobyMCbobs May 10, 2022

hh May 11, 2022

ameukam Dec 23, 2021

BobyMCbobs May 10, 2022

ameukam Dec 23, 2021

BobyMCbobs May 10, 2022

ameukam Dec 23, 2021

linux-foundation-easycla bot commented May 11, 2022 •

edited

Loading

hh commented May 11, 2022

dims commented May 11, 2022

saschagrunert left a comment

dims commented May 11, 2022

k8s-ci-robot commented May 11, 2022


		### Non-Goals

		Anything related to creation of artifacts, bom, digital signatures.


		### Goals

		A policy and procedure for use by SIG Release to promote container images and release binaries to multiple registries and mirrors.


		Anything related to creation of artifacts, bom, digital signatures.

		## Proposal

	Then I will be redirected to a local CLOUD registry
	Then I will be redirected to the closest network endpoint of the registry


		### How much is this going to save us?

		![Cost of K8s Artifact hosting - Data Studio Graphs](https://i.imgur.com/LAn4UIE.png)


		## Infrastructure Needed

		It would be good to request some donations for some larger providers, including one in China, via [Cloud Native Credits program](https://www.cncf.io/credits/).


		#### Policy

		#### Process

KEP-3000: Image Promotion and Distribution Policy #3079

KEP-3000: Image Promotion and Distribution Policy #3079

Conversation

hh commented Dec 7, 2021

hh commented Dec 7, 2021

ameukam Dec 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justaugustus left a comment

Choose a reason for hiding this comment

hh commented Dec 7, 2021

eddiezane commented Dec 8, 2021

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ameukam Dec 23, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linux-foundation-easycla bot commented May 11, 2022 • edited Loading

hh commented May 11, 2022

dims commented May 11, 2022

saschagrunert left a comment

Choose a reason for hiding this comment

dims commented May 11, 2022

k8s-ci-robot commented May 11, 2022

ameukam Dec 7, 2021 •

edited

Loading

ameukam Dec 23, 2021 •

edited

Loading

linux-foundation-easycla bot commented May 11, 2022 •

edited

Loading