Replace gRPC code generation tool from Znly/protoc to Buf #2344

forsaken628 · 2024-06-06T07:40:23Z

What this PR does / why we need it:
Replace gRPC code generation tool from Znly/protoc to Buf , and some fix

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2141

Checklist:

Docs included if any changes are user facing

Signed-off-by: forsaken628 <forsaken628@gmail.com>

forsaken628 · 2024-06-07T00:47:48Z

medianstop doesn't have any unit tests, and it doesn't look like there's any reason it has to be implemented in python. Can I re-implement it in go later and add unit tests?
@andreyvelich

Signed-off-by: forsaken628 <forsaken628@gmail.com>

andreyvelich · 2024-06-14T14:24:36Z

medianstop doesn't have any unit tests, and it doesn't look like there's any reason it has to be implemented in python. Can I re-implement it in go later and add unit tests?

Actually, we have unit tests here: https://github.com/kubeflow/katib/blob/master/test/unit/v1beta1/earlystopping/test_medianstop_service.py

Any specific reason do you want to implement in Go, ?
We are trying to keep our optimization algorithms in Python since Data Science community are more familiar with it.

andreyvelich

Thank you for this great contribution @forsaken628!
I left a few comments.
@tenzen-y @johnugeorge Should we try to merge it before the release to unblock: #2346 ?

hack/update-codegen.sh

hack/update-proto.sh

andreyvelich · 2024-06-14T14:32:17Z

buf.gen.yaml

@@ -0,0 +1,12 @@
+version: v2
+plugins:
+- remote: buf.build/protocolbuffers/go:v1.33.0


Do we need to keep buf config for Go packages outside of /apis/manager/v1beta1 directory ?

I'm prefer global buf.gen.yaml, but there will be a lot of changes for python. Well, that's fine.

andreyvelich · 2024-06-14T14:36:19Z

test/conftest.py

+root = os.path.join(os.path.dirname(__file__), "..")
+path.extend(
+    [
+        os.path.join(root, "pkg/apis/manager/v1beta1/python"),
+        os.path.join(root, "pkg/apis/manager/health/python"),
+        os.path.join(root, "pkg/metricscollector/v1beta1/common"),
+        os.path.join(root, "pkg/metricscollector/v1beta1/tfevent-metricscollector"),
+    ]
+)


Thank you for doing this!

Signed-off-by: forsaken628 <forsaken628@gmail.com>

tenzen-y · 2024-06-14T16:23:44Z

Thank you for this great contribution @forsaken628! I left a few comments. @tenzen-y @johnugeorge Should we try to merge it before the release to unblock: #2346 ?

SGTM

Signed-off-by: forsaken628 <forsaken628@gmail.com>

tenzen-y · 2024-06-15T09:39:57Z

@forsaken628 As I mentioned here, could you open a PR to implement the custom readiness check to resolve flake E2E tests?

tenzen-y

Thank you for this great contribution!
/approve

I leave lgtm on @andreyvelich

google-oss-prow · 2024-06-15T11:18:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

forsaken628 · 2024-06-15T12:07:18Z

@forsaken628 As I mentioned here, could you open a PR to implement the custom readiness check to resolve flake E2E tests?

katib/cmd/katib-controller/v1beta1/main.go

Line 151 in e6bd3e7

    
           if err := mgr.AddReadyzCheck("readyz", hookServer.StartedChecker()); err != nil {

There's already a webhook ready check added before manger start, and I'm wondering why it's still getting the
connection refused

tenzen-y · 2024-06-15T12:33:47Z

@forsaken628 As I mentioned here, could you open a PR to implement the custom readiness check to resolve flake E2E tests?

katib/cmd/katib-controller/v1beta1/main.go

Line 151 in e6bd3e7

if err := mgr.AddReadyzCheck("readyz", hookServer.StartedChecker()); err != nil {

There's already a webhook ready check added before manger start, and I'm wondering why it's still getting the connection refused

As I described in the comment, the default checker can not consider if the certs are ready.

https://github.com/kubeflow/training-operator/blob/9a9edecf146bb0c2cccaeb3bec912ab2dfa5d8c5/cmd/training-operator.v1/main.go#L230-L236

andreyvelich · 2024-06-15T15:17:22Z

Thank you for doing this @forsaken628!
Please can you cherry-pick this PR to the release branch: release-0.17 ?

andreyvelich · 2024-06-15T15:17:46Z

/lgtm

) * Replace gRPC code generation tool from Znly/protoc to Buf Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * del build.sh Signed-off-by: forsaken628 <forsaken628@gmail.com> * cleanup Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix test Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * refine Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * rm outter yaml Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> --------- Signed-off-by: forsaken628 <forsaken628@gmail.com>

) * Replace gRPC code generation tool from Znly/protoc to Buf Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * del build.sh Signed-off-by: forsaken628 <forsaken628@gmail.com> * cleanup Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix test Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * refine Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * rm outter yaml Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> --------- Signed-off-by: forsaken628 <forsaken628@gmail.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

…branch (#2362) * Fix TestReconcileBatchJob (#2350) * update Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * update Signed-off-by: forsaken628 <forsaken628@gmail.com> * update Signed-off-by: forsaken628 <forsaken628@gmail.com> * update Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * cleanup Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * update Signed-off-by: forsaken628 <forsaken628@gmail.com> * use gomock Signed-off-by: forsaken628 <forsaken628@gmail.com> --------- Signed-off-by: forsaken628 <forsaken628@gmail.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Use cache-dependency-path in actions/setup-go for CI workflow (#2355) Signed-off-by: forsaken628 <forsaken628@gmail.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Replace already closed github.com/golang/mock with go.uber.org/mock (#2357) * replace gomock Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * revert Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> --------- Signed-off-by: forsaken628 <forsaken628@gmail.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Replace gRPC code generation tool from Znly/protoc to Buf (#2344) * Replace gRPC code generation tool from Znly/protoc to Buf Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * del build.sh Signed-off-by: forsaken628 <forsaken628@gmail.com> * cleanup Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix test Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * refine Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * rm outter yaml Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> --------- Signed-off-by: forsaken628 <forsaken628@gmail.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Upgrade the protobuf version to >=4.21.12,<5 (#2358) Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * [SDK] Fix empty list for env variables and numpy version (#2360) * [SDK] Fix empty list for env variables Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix numpy version in tests Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: forsaken628 <forsaken628@gmail.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Co-authored-by: coldWater <forsaken628@gmail.com> Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

) * Replace gRPC code generation tool from Znly/protoc to Buf Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * del build.sh Signed-off-by: forsaken628 <forsaken628@gmail.com> * cleanup Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix test Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * refine Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> * rm outter yaml Signed-off-by: forsaken628 <forsaken628@gmail.com> * fix Signed-off-by: forsaken628 <forsaken628@gmail.com> --------- Signed-off-by: forsaken628 <forsaken628@gmail.com>

forsaken628 added 2 commits June 6, 2024 14:44

Replace gRPC code generation tool from Znly/protoc to Buf

e3164fc

Signed-off-by: forsaken628 <forsaken628@gmail.com>

fix

9fc7f71

Signed-off-by: forsaken628 <forsaken628@gmail.com>

google-oss-prow bot added the size/XXL label Jun 6, 2024

google-oss-prow bot requested review from andreyvelich, anencore94 and johnugeorge June 6, 2024 07:40

forsaken628 added 6 commits June 6, 2024 15:44

del build.sh

ff684ff

Signed-off-by: forsaken628 <forsaken628@gmail.com>

cleanup

dfb8e0b

Signed-off-by: forsaken628 <forsaken628@gmail.com>

fix test

2412819

Signed-off-by: forsaken628 <forsaken628@gmail.com>

fix

c5b06e0

Signed-off-by: forsaken628 <forsaken628@gmail.com>

fix

bf9f8cd

Signed-off-by: forsaken628 <forsaken628@gmail.com>

refine

ccd7a4c

Signed-off-by: forsaken628 <forsaken628@gmail.com>

fix

619012c

Signed-off-by: forsaken628 <forsaken628@gmail.com>

andreyvelich mentioned this pull request Jun 12, 2024

Replace gRPC code generation tool from Znly/protoc to Buf #2295

Closed

1 task

andreyvelich reviewed Jun 14, 2024

View reviewed changes

Merge remote-tracking branch 'up/master' into pb

6dacf89

Signed-off-by: forsaken628 <forsaken628@gmail.com>

forsaken628 added 2 commits June 15, 2024 09:48

rm outter yaml

8d2bacb

Signed-off-by: forsaken628 <forsaken628@gmail.com>

fix

6032f5d

Signed-off-by: forsaken628 <forsaken628@gmail.com>

tenzen-y mentioned this pull request Jun 15, 2024

Python SDK for Katib and Kubeflow Pipelines can not be installed at the same time #2346

Closed

tenzen-y reviewed Jun 15, 2024

View reviewed changes

google-oss-prow bot added the approved label Jun 15, 2024

google-oss-prow bot assigned andreyvelich Jun 15, 2024

google-oss-prow bot added the lgtm label Jun 15, 2024

google-oss-prow bot merged commit 0d190b9 into kubeflow:master Jun 15, 2024
60 checks passed

forsaken628 deleted the pb branch June 16, 2024 03:41

andreyvelich mentioned this pull request Jun 18, 2024

Cherry pick of #2350 #2355 #2357 #2344 #2358 #2360 into release-0.17 branch #2362

Merged

Electronic-Waste mentioned this pull request Jul 25, 2024

[SDK] grpc-related bugs in Python SDK #2395

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace gRPC code generation tool from Znly/protoc to Buf #2344

Replace gRPC code generation tool from Znly/protoc to Buf #2344

forsaken628 commented Jun 6, 2024

forsaken628 commented Jun 7, 2024

andreyvelich commented Jun 14, 2024

andreyvelich left a comment

andreyvelich Jun 14, 2024

forsaken628 Jun 15, 2024

andreyvelich Jun 14, 2024

tenzen-y commented Jun 14, 2024

tenzen-y commented Jun 15, 2024

tenzen-y left a comment

google-oss-prow bot commented Jun 15, 2024

forsaken628 commented Jun 15, 2024

tenzen-y commented Jun 15, 2024 •

edited

Loading

andreyvelich commented Jun 15, 2024

andreyvelich commented Jun 15, 2024

Replace gRPC code generation tool from Znly/protoc to Buf #2344

Replace gRPC code generation tool from Znly/protoc to Buf #2344

Conversation

forsaken628 commented Jun 6, 2024

forsaken628 commented Jun 7, 2024

andreyvelich commented Jun 14, 2024

andreyvelich left a comment

Choose a reason for hiding this comment

andreyvelich Jun 14, 2024

Choose a reason for hiding this comment

forsaken628 Jun 15, 2024

Choose a reason for hiding this comment

andreyvelich Jun 14, 2024

Choose a reason for hiding this comment

tenzen-y commented Jun 14, 2024

tenzen-y commented Jun 15, 2024

tenzen-y left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Jun 15, 2024

forsaken628 commented Jun 15, 2024

tenzen-y commented Jun 15, 2024 • edited Loading

andreyvelich commented Jun 15, 2024

andreyvelich commented Jun 15, 2024

tenzen-y commented Jun 15, 2024 •

edited

Loading