Add build and release for csi proxy to gcs bucket #53

jingxu97 · 2020-05-21T21:50:45Z

This PR adds building and publishing csi-proxy binary to gcs bucket k8s-artifacts-csi
The prow job for publishing the binary in test-infra is here kubernetes/test-infra#17683

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

k8s-ci-robot · 2020-05-21T21:50:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jingxu97

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jingxu97]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Add a prow job to publish csi-proxy binary csi-proxy PR to add cloudbuild.yaml kubernetes-csi/csi-proxy#53

jingxu97 · 2020-05-27T16:53:16Z

This PR is ready for review. PTAL. Thanks!

jingxu97 · 2020-05-27T16:54:48Z

cc @justaugustus

Add a prow job to publish csi-proxy binary csi-proxy PR to add cloudbuild.yaml kubernetes-csi/csi-proxy#53

jingxu97 · 2020-05-28T16:58:43Z

cc @verult

msau42 · 2020-05-28T19:02:06Z

build/cloudbuild.yaml

@@ -0,0 +1,17 @@
+# Build the module.
+steps:
+- name: gcr.io/cloud-builders/go:debian


Do we want to use the k8s build images that have specific versions of golang, like in https://github.com/kubernetes-csi/csi-release-tools/pull/87/files#diff-e5fc65747fb9417968a3295dc1ac5111R26?

I think go:debian get the latest version of go GoogleCloudPlatform/cloud-builders#132. Based on that PR's comment, seems like it also does not use specific version of go. @pohly might help on this?

Please have a look at kubernetes-csi/csi-release-tools#86 (comment) an the PRs linked to there.

In a nutshell, the approach there is to install the right version of the Go toolchain before using it, using the normal build rules for Windows binaries.

The relevant code is https://github.com/pohly/csi-release-tools/blob/bd416901d4b7cad7bc70a39f508b68e2a9fff1f6/cloudbuild.sh which sources release-tools/prow.sh and then invokes image building with run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make push-multiarch.

You could do something similar and use run_with_go "${CSI_PROW_GO_VERSION_BUILD}" make build to produce the csi-proxy.exe.

got it. Thanks!
I updated it.

msau42 · 2020-05-28T19:03:58Z

build/cloudbuild.yaml

+    set -ex
+    IFS=- read str1 rev <<< "${_GIT_TAG}"
+    make build REV=${rev}
+    cp bin/csi-proxy.exe bin/csi-proxy${_GIT_TAG}.exe


We may want to consider enhancing this later with the logic in https://github.com/kubernetes-csi/csi-release-tools/pull/87/files#diff-841c5d7c9fb7c0ba0fb2f452298ed526R147 to support "canary" images

msau42 · 2020-05-28T19:04:48Z

build/cloudbuild.yaml

+  _GIT_TAG: '12345'
+artifacts:
+  objects:
+    location: 'gs://k8s-artifacts-csi'


This is the "prod" bucket which I believe we should only push to by promoting from staging. Is there a staging bucket we can use instead?

before I confirmed with bartsmykla@ that there is no such process yet.

Correct, there's no GCS promotion process yet. I talk about it a little here: kubernetes/release#964 (comment)

I wrote gh2gcs very recently, which supports uploading GitHub releases to a GCS bucket.

See the usage here: kubernetes/release#1311

So if you publish release tarballs to GitHub, you'll be able to do something like this to upload to GCS:

$ time gh2gcs --log-level debug --org kubernetes-csi --repo csi-proxy --bucket k8s-artifacts-csi --release-dir release --tags v0.0.0,v0.0.1,v0.0.2

If you run the command w/o the --tags flag, it'll iterate over all releases and upload them.
You could add a step in the cloudbuild.yaml to do this.

Eventually (hopefully very soon), I'm going to update the tool to accept a yaml config, so you can add a config file somewhere, that looks roughly like this:

- org: foo repo: bar tags: [] gcsBucket: bucket-name releaseDir: release-dir

Let me know what you think!

@justaugustus thank you very much for providing this tool. For now, I will follow other csi repo build step first. Later we will try to see how to improve it with your tool.

I couldn't quite follow the discussion, but is the suggestion to put ci images under a subfolder called "dev/ci", and then "promote" releases to "release/"? That way it's easy for someone to easily distinguish what is a production ready image vs a dev image.

jingxu97 · 2020-06-02T22:00:10Z

@pohly @msau42 @justaugustus PR is updated. PTAL. thanks!

msau42 · 2020-06-02T22:52:55Z

cloudbuild.yaml

+  _PULL_BASE_REF: 'master'
+artifacts:
+  objects:
+    location: 'gs://disksize-bucket'


Is this location correct?

oh, sorry, that's for my local testing. fixed it.

msau42 · 2020-06-02T22:54:42Z

build/cloudbuild.yaml

+  _GIT_TAG: '12345'
+artifacts:
+  objects:
+    location: 'gs://k8s-artifacts-csi'


I couldn't quite follow the discussion, but is the suggestion to put ci images under a subfolder called "dev/ci", and then "promote" releases to "release/"? That way it's easy for someone to easily distinguish what is a production ready image vs a dev image.

jingxu97 · 2020-06-09T20:47:09Z

Updated, PTAL @msau42 @pohly

pohly

/lgtm

cloudbuild.yaml

msau42 · 2020-06-10T17:37:56Z

cloudbuild.yaml

+  _PULL_BASE_REF: 'master'
+artifacts:
+  objects:
+    location: 'gs://k8s-artifacts-csi'


Sorry for the late change, but for container repo, we decided to not use the storage-csi image repo, and instead have one "k8s-sig-storage" repo for all sig related projects. Do we want to follow a similar naming scheme here?

I tried to search a little bit and only found we have k8s-artifacts-cni and k8s-artifacts-prod were used. So overall we only have very few of such binary packages. And the name of artifacts is very close to the binary itself. So I feel I am ok with csi name.

jingxu97 · 2020-06-11T06:03:00Z

@pohly @msau42 I updated a small change. PTAL. Thanks!

jingxu97 · 2020-06-22T21:50:38Z

@pohly @msau42 I updated with a small change due to naming. Could you please add /lgtm again? Thanks!

msau42 · 2020-06-23T18:30:15Z

cloudbuild.yaml

+artifacts:
+  objects:
+    location: 'gs://k8s-artifacts-csi/dev'
+    paths: 'bin/csi-proxy-${_PULL_BASE_REF}.exe'


Ah sorry one more thought came to me. Do we want to have a "csi-proxy/bin/" directory in case in the future we have more than one binary we want to put here?

you mean the gs bucket location as gs://k8s-artifacts-csi/csi-proxy/bin? Or put it in paths: csi-proxy/bin/csi-proxy-${_PULL_BASE_REF}.exe?

For the bucket, if there are other binaries, they will have different names? For the paths here, I am not sure adding csi-proxy is useful?

either way. I think it's better organized when each binary is in separate directories, especially when we start to get many versions stored.

got it. But I thought paths is the local path where the file is stored locally, and location is the remote store where the file will be uploaded. So we should add it to location?
How about this gs://k8s-artifacts-csi/csi-proxy/dev?

This PR adds building and publishing csi-proxy binary to gcs bucket k8s-artifacts-csi

msau42 · 2020-06-30T18:49:48Z

/lgtm

k8s-ci-robot requested review from msau42 and PatrickLang May 21, 2020 21:50

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 21, 2020

jingxu97 added a commit to jingxu97/test-infra that referenced this pull request May 21, 2020

Add prow job to publishing csi-proxy binary

35acaad

Add a prow job to publish csi-proxy binary csi-proxy PR to add cloudbuild.yaml kubernetes-csi/csi-proxy#53

jingxu97 mentioned this pull request May 21, 2020

Add prow job to publishing csi-proxy binary kubernetes/test-infra#17683

Closed

jingxu97 force-pushed the May/build branch from 0b8f4a9 to 4569f10 Compare May 27, 2020 16:52

jingxu97 changed the title ~~WIP: Add build and release for csi proxy to gcs bucket~~ Add build and release for csi proxy to gcs bucket May 27, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 27, 2020

jingxu97 force-pushed the May/build branch 2 times, most recently from c22ffe8 to 6039148 Compare May 27, 2020 16:57

jingxu97 added a commit to jingxu97/test-infra that referenced this pull request May 27, 2020

Add prow job to publishing csi-proxy binary

801bd56

Add a prow job to publish csi-proxy binary csi-proxy PR to add cloudbuild.yaml kubernetes-csi/csi-proxy#53

msau42 reviewed May 28, 2020

View reviewed changes

jingxu97 force-pushed the May/build branch from 6039148 to 0a6c55e Compare June 2, 2020 21:33

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 2, 2020

msau42 reviewed Jun 2, 2020

View reviewed changes

jingxu97 force-pushed the May/build branch from 0a6c55e to 05d7821 Compare June 3, 2020 03:32

pohly approved these changes Jun 10, 2020

View reviewed changes

k8s-ci-robot assigned pohly Jun 10, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 10, 2020

msau42 reviewed Jun 10, 2020

View reviewed changes

cloudbuild.yaml Show resolved Hide resolved

msau42 reviewed Jun 10, 2020

View reviewed changes

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jun 11, 2020

jingxu97 force-pushed the May/build branch from 05d7821 to 5b07fac Compare June 11, 2020 02:29

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 11, 2020

msau42 reviewed Jun 23, 2020

View reviewed changes

Add build and release for csi proxy to gcs bucket

9dab535

This PR adds building and publishing csi-proxy binary to gcs bucket k8s-artifacts-csi

jingxu97 force-pushed the May/build branch from 5b07fac to 9dab535 Compare June 30, 2020 17:28

k8s-ci-robot assigned msau42 Jun 30, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 30, 2020

k8s-ci-robot merged commit f72dab1 into kubernetes-csi:master Jun 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add build and release for csi proxy to gcs bucket #53

Add build and release for csi proxy to gcs bucket #53

jingxu97 commented May 21, 2020 •

edited

Loading

k8s-ci-robot commented May 21, 2020

jingxu97 commented May 27, 2020

jingxu97 commented May 27, 2020

jingxu97 commented May 28, 2020

msau42 May 28, 2020

jingxu97 May 28, 2020

pohly May 29, 2020

pohly May 29, 2020

jingxu97 Jun 2, 2020

msau42 May 28, 2020

jingxu97 May 28, 2020

msau42 May 28, 2020

jingxu97 May 28, 2020

justaugustus May 28, 2020 •

edited

Loading

jingxu97 Jun 2, 2020

msau42 Jun 2, 2020

jingxu97 commented Jun 2, 2020

msau42 Jun 2, 2020

jingxu97 Jun 3, 2020

msau42 Jun 2, 2020

jingxu97 commented Jun 9, 2020

pohly left a comment

msau42 Jun 10, 2020

jingxu97 Jun 10, 2020

jingxu97 commented Jun 11, 2020

jingxu97 commented Jun 22, 2020

msau42 Jun 23, 2020

jingxu97 Jun 30, 2020

msau42 Jun 30, 2020

jingxu97 Jun 30, 2020

msau42 commented Jun 30, 2020

Add build and release for csi proxy to gcs bucket #53

Add build and release for csi proxy to gcs bucket #53

Conversation

jingxu97 commented May 21, 2020 • edited Loading

k8s-ci-robot commented May 21, 2020

jingxu97 commented May 27, 2020

jingxu97 commented May 27, 2020

jingxu97 commented May 28, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justaugustus May 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jingxu97 commented Jun 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jingxu97 commented Jun 9, 2020

pohly left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jingxu97 commented Jun 11, 2020

jingxu97 commented Jun 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Jun 30, 2020

jingxu97 commented May 21, 2020 •

edited

Loading

justaugustus May 28, 2020 •

edited

Loading