Cleanup usage of kubernetes-release-pull in kubernetes presubmits

[amwat]

What should be cleaned up or changed:
We stage builds to gs://kubernetes-release-pull in almost every presubmit job.
But from what I can tell nothing is actually consuming those builds since the jobs also use extract=local .
It's a non-trivial overhead to upload the release tars in every presubmit and we should remove all the non-required usages.

Provide any links for context:
https://cs.k8s.io/?q=kubernetes-release-pull&i=nope&files=&repos=

test-infra/kubetest/extract_k8s.go

Lines 449 to 466 in c4628a3

	case local:
	url := util.K8s("kubernetes", "_output", "gcs-stage")
	files, err := ioutil.ReadDir(url)
	if err != nil {
	return err
	}
	var release string
	for _, file := range files {
	r := file.Name()
	if strings.HasPrefix(r, "v") {
	release = r
	break
	}
	}
	if len(release) == 0 {
	return fmt.Errorf("No releases found in %v", url)
	}
	return getKube(fmt.Sprintf("file://%s", url), release, extractSrc)

Random GCE provider job: https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1293275406807339008#1:build-log.txt%3A903

/cc @spiffxp @BenTheElder @MushuEE

---

EDIT(@spiffxp): I made a list of the offending jobs going off the criteria --extract=local and --stage=gs://kubernetes-release-pull/*

• if the job triggers for a single branch it's labeled as job@branch
• if the job triggers for all branches it's labeled as job
• there are no presubmits that trigger for N branches (where all > N > 1)
• there are no periodics or postsubmits that touch gs://kubernetes-release-pull
• this picks up some --provider=aws jobs (kops), it remains to be seen whether they need --stage or not

EDIT(@BenTheElder): I removed the outdated checklist and instead i'm going to provide a search: https://github.com/search?q=repo%3Akubernetes%2Ftest-infra+%22--stage%3Dgs%3A%2F%2Fkubernetes-release-pull%22&type=code

[BenTheElder]

we should test this in a canary just because this stuff is old and brittle and I can't remember why we were doing this anymore 🙃

[amwat]

update:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/92316/pull-kubernetes-e2e-gce-no-stage/1294345203125063682/#1:build-log.txt%3A355

the local path

test-infra/kubetest/extract_k8s.go

Line 450 in c4628a3

url := util.K8s("kubernetes", "_output", "gcs-stage")

comes from
https://github.com/kubernetes/release/blob/8d6bd15010efeec44018e4847860d464d2682d97/lib/releaselib.sh#L1245-L1247

which shouldn't be needed since we already have them under
bazel-bin/build/release-tars (but without the hashes)
https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/92316/pull-kubernetes-e2e-gce-no-stage/1294345203125063682/#1:build-log.txt%3A338

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[BenTheElder]

still worth doing?

[spiffxp]

/remove-lifecycle rotten
I think so. The other option is to continue as-is, meaning jobs that use this bucket need to switch to use k8s-release-pull as they migrate to k8s-infra.

[amwat]

sadly looks like those gcs links have been gced.
seems like one of the steps involved as part of --stage is copying the artifacts from the bazel output path to the make output path _output/gcs-stage and then uploading them to gcs.

and our presubmit jobs are configured to --extract=local instead of --extract=bazel while using --build=bazel
so they were relying on them being in the make output path.
https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-cloud-provider/gcp/gcp-gce.yaml#L48

testing out in the canary job: #20427

[spiffxp]

/milestone v1.21
/sig testing
/wg-k8s-infra

[amwat]

We have a succesful run at https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-no-stage/1352340847076577280

not sure why the total test duration is higher as compared to
https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1351850610516824064

but we atleast saved 154 seconds of stage time (which should be the only delta here)

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/92316/pull-kubernetes-e2e-gce-no-stage/1352340847076577280/artifacts/junit_runner.xml

as compared to

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/97894/pull-kubernetes-e2e-gce/1351850610516824064/artifacts/junit_runner.xml

and 1.84 GiB of unnecessary GCS uploads

$ gsutil du -sh gs://kubernetes-release-pull/ci/pull-kubernetes-e2e-gce/v1.18.16-rc.0.3+9f5c61d324a62b
1.84 GiB     gs://kubernetes-release-pull/ci/pull-kubernetes-e2e-gce/v1.18.16-rc.0.3+9f5c61d324a62b

[spiffxp]

/priority important-soon

[spiffxp]

/assign @amwat @spiffxp
Assigning to us for now. If we think this is eligible for /help or don't have time to do it ourselves we can writeup how to proceed

[jbpratt]

Based on #22892 (comment) (and the changes being reverted), how should we proceed with this? I started working through this and realized I was re-doing @spiffxp's changes 😄

[SD-13]

Hi, I am interested to work on this issue but I have some questions or queries.

1. Seems like the list of jobs to be fixed is outdated
2. Please help me to understand the fix we need to follow here
   I don't think the fix @amwat mentioned here here don't apply because we are not using bazel to build.
   So as discussed here config/jobs: run no-stage on k8s-infra, drop extract #24238 (comment), can we now remove extract and stage?
   Please feel free to correct me if I am wrong

cc @spiffxp @BenTheElder @ameukam

[BenTheElder]

Sorry, a couple of the people you pinged don't work on tthis anymore and I'm kinda buried.

I've lost context on this one.

[BenTheElder]

I'm not sure we ever got no-stage working? It's hard to follow at this point.

[BenTheElder]

#28176 renamed the test job, testing in kubernetes/kubernetes#126563

[BenTheElder]

It does, it will stage to a generated bucket under the rented boskos project (which the boskos janitors should clean up if they don't already), so we can carefully start dropping these I think ... very belatedly.

[BenTheElder]

beginning bulk migration in #33259, starting with a subset of optional, non-blocking, not always_run jobs

We have to drop both --extract=local and --stage at the same time. We don't need to locally extract what we just built, it's running fine and uploading to a bucket under the boskos project.

You can see sample runs in kubernetes/kubernetes#126563

Inspect these logs:
https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/126563/pull-kubernetes-e2e-gce-cos-no-stage/1820909457454927872
https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/126563/pull-kubernetes-e2e-gce-pull-through-cache/1821254221408768000

[BenTheElder]

If anyone wants to help:

• Break these up into easily reverted commits
• Make sure to drop both extract=local and stage at the same time
• Make sure the jobs you're touching are not required for merge, we'll do those last
• You must agree to follow-up to make sure you didn't break anything shortly after merging these, and definitely before doing any more. If you're not confident in / familiar with this part, I'd ask that you select a different issue, we need to get this sorted out as part of migrating to the community infra in the immediate future but we don't want to break CI especially at this point in the release cycle, and I expect to be done before we get to the safer periods in the release cycle.

NOTE: spiffxp and amwat don't work on Kubernetes anymore. I'm taking over this problem.

[BenTheElder]

#33278 does everything but the one remaining PR blocking job, for which we'll wait a bit and check some more things

[BenTheElder]

Once we have test results we can do #33280, and then I'll delete the bucket

[BenTheElder]

This is done, I just need to follow-up with eliminating that bucket.

[BenTheElder]

Done!