Add k8s-release project and buckets for ci

[spiffxp]

We need a home for community-owned buckets that are equivalent to:

• gs://kubernetes-release-dev (hosts periodic builds of kubernetes)
• gs://kuberentes-release-pull (hosts pr builds of kubernetes, eg: from e2e test jobs)

I'm suggesting we create a k8s-release project, and set it up
identically to the k8s-release-test-prod project currently in use by
release-engineering.

Then setup two equivalent buckets:

• gs://k8s-release-dev
• gs://k8s-release-pull

And give prow write access to these buckets. I suspect prow will need
admin access, but let's find out.

ref: #846 (comment)

[spiffxp]

/cc @justaugustus @tpepper
for input from @kubernetes/release-engineering

/cc @thockin
FYI

[justaugustus]

Matches what Aaron and I chatted about.
/lgtm
/approve

/hold for any other reviewers; lift when ready

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justaugustus, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[thockin]

Why kubernetes-release-dev (and I assume you mean the 3 regionals) and kubernetes-release-pull and not kubernetes-release (and regionals)?

Looking at stats, kubernetes-release is the next-highest cost bucket.

I think there's a way to literally have the bucket moved between projects - would that be a better path?

I know @justinsb started thinking about billing breakdown per-bucket. Unfortunately, GCP's normal billing breakdown stops at a project+SKU, but we can enable finer-grained logs. We should decide if it makes sense to lump ANYTHING together, or if we're better off with a project per bucket (or set of common-purpose buckets). Puke.

In other words, what if we just moved them all into k8s-artifacts-prod (or into a new k8s-release project)? That would mean that all external tools remain functional.

BTW - I have no idea where kubernetes-release-pull lives. Sigh.

[spiffxp]

Why kubernetes-release-dev (and I assume you mean the 3 regionals) and kubernetes-release-pull and not kubernetes-release (and regionals)?

I'm not aiming at (monetary) cost right now, but opportunity cost. We're blocked on moving some of the CI jobs out of google.com because gs://kuberetes-release-dev (technically its host project google-containers) won't allow non-google.com accounts to be added to IAM (ref: #846).

Moving CI related builds into community owned bucket(s) means we can move all release-blocking/merge-blocking jobs into community owned infra, and will empower more people to help out.

I think there's a way to literally have the bucket moved between projects - would that be a better path?

Discussed offline. There does not appear to be such a way, unfortunately. So migrating piecemeal or stop-the-world-and-sync are our options.

Unfortunately, GCP's normal billing breakdown stops at a project+SKU, but we can enable finer-grained logs.

The "stops at a project+SKU" is why I'm proposing something (k8s-release) instead of k8s-artifacts-prod. The dividing line in my head is k8s-artifacts-prod is for final/static release artifacts for every single kubernetes subproject; k8s-release is for all the intermediary stuff (per-pr, postsubmits, nightlies, etc) related to kubernetes

BTW - I have no idea where kubernetes-release-pull lives.

It's part of the kubernetes-jenkins project. That's going to be a fun one to migrate away from. But we don't have to do that today.

[thockin]

Why kubernetes-release-dev (and I assume you mean the 3 regionals) and kubernetes-release-pull and not kubernetes-release (and regionals)?

I'm not aiming at (monetary) cost right now, but opportunity cost. We're blocked on moving some of the CI jobs out of google.com because gs://kuberetes-release-dev (technically its host project google-containers) won't allow non-google.com accounts to be added to IAM (ref: #846).

Do we need just k-release-dev or also the -asia and -eu sister-buckets?

Moving CI related builds into community owned bucket(s) means we can move all release-blocking/merge-blocking jobs into community owned infra, and will empower more people to help out.

ACK

I think there's a way to literally have the bucket moved between projects - would that be a better path?

Discussed offline. There does not appear to be such a way, unfortunately. So migrating piecemeal or stop-the-world-and-sync are our options.

Correct. For kubernetes-release, we might go a long way by changing dl.k8s.io
and seeing how much is left.

Unfortunately, GCP's normal billing breakdown stops at a project+SKU, but we can enable finer-grained logs.

The "stops at a project+SKU" is why I'm proposing something (k8s-release) instead of k8s-artifacts-prod. The dividing line in my head is k8s-artifacts-prod is for final/static release artifacts for every single kubernetes subproject; k8s-release is for all the intermediary stuff (per-pr, postsubmits, nightlies, etc) related to kubernetes

Where would the current 'kubernetes-release' bucket go? Are we OK conflating
the bill for dev, pull, and release together? Or should we make a "one bucket
per project" rule for these?

Regardless, we should put an LB VIP in front of whatever bucket(s) so we don't
have to deal with this again. See ensure-prod-storage-gclb.sh for example. We
could maybe set up ONE VIP to cover everything (except the existing VIP, which
exposes at the root, sigh).

I am fine with the overall plan to make new buckets and copy stuff over, then
flip pointers, for these. We will need to consider kubernetes-release soon,
too.

BTW - I have no idea where kubernetes-release-pull lives.

It's part of the kubernetes-jenkins project. That's going to be a fun one to migrate away from. But we don't have to do that today.

[spiffxp]

Do we need just k-release-dev or also the -asia and -eu sister-buckets?

So I added equivalents for these. But I have to admit I'm not sure how the existing kubernetes-release buckets are mirrored to their asia/eu siblings. I would be curious to see how much traffic these buckets are seeing compared to the non-geo (us, I guess) bucket.

I misspoke earlier, gs://kubernetes-release-pull is under kubernetes-jenkins-pull. I confirmed it has no geo-specific buckets.

[spiffxp]

To unblock the current set of release-blocking jobs I don't think mirroring to geo-specific buckets is required. I think looking at existing traffic will help us decide how much of a priority setting up mirroring is.

[MushuEE]

/lgtm

[spiffxp]

/hold cancel

[spiffxp]

Running ./infra/gcp/ensure-release-projects.sh

[spiffxp]

re: #1110 (comment)

So I added equivalents for these. But I have to admit I'm not sure how the existing kubernetes-release buckets are mirrored to their asia/eu siblings. I would be curious to see how much traffic these buckets are seeing compared to the non-geo (us, I guess) bucket.

I did some digging and discovered a few things re: syncing to regional buckets:

• kubernetes-release-dev hasn't been synced to regional buckets for over a year
• kubernetes-release-eu/-asia buckets serve two orders of magnitude less traffic than kubernetes-release (I can only see back by six weeks

Given that, I think regional syncing should not be a blocker to moving forward. If we decide it's necessary after the fact, we should look at using Storage Transfer Service (https://cloud.google.com/storage-transfer/docs/create-manage-transfer-console) to sync