KEP-2170: Add TrainJob and TrainingRuntime APIs #2223

andreyvelich · 2024-08-15T23:17:27Z

I added APIs for TrainJob, TrainingRuntime, and ClusterTrainingRuntime resources.

/assign @kubeflow/wg-training-leads @kannon92 @mimowo @vsoch @ahg-g @kuizhiqing @alculquicondor @zw0610 @franciscojavierarceo @shravan-achar

/hold for review.

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

ahg-g · 2024-08-15T23:20:09Z

@danielvegamyhre

andreyvelich · 2024-08-15T23:20:24Z

pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go

+	// Number of training nodes.
+	// TODO (andreyvelich): Do we want to support dynamic num of nodes in TrainJob for PyTorch elastic: `--nnodes=1:4` ?
+	NumNodes *int32 `json:"numNodes,omitempty"`


@tenzen-y What do you think about using *string here since for the elastic training user can set:

torchrun --nnodes=1:4

I prefer int here to refer to active/effective number instead of string.
IMO, the string value of nnodes if calculated from a config from like ElasticPolicy.

IIRC, we can specify the elastically policy in the JobSetSpec. So, let's use the typed int here.
After we find some advantages for string, we can introduce IntOrString like Deployment rollingUpdate.

coveralls · 2024-08-15T23:21:37Z

Pull Request Test Coverage Report for Build 10566761788

Details

0 of 622 (0.0%) changed or added relevant lines in 3 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-1.7%) to 31.801%

Changes Missing Coverage	Changed/Added Lines	%
pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go	3	0.0%
pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go	3	0.0%
pkg/apis/kubeflow.org/v2alpha1/zz_generated.deepcopy.go	616	0.0%

Files with Coverage Reduction	New Missed Lines	%
pkg/controller.v1/mpi/mpijob.go	1	91.06%

Totals
Change from base Build 10512072223:	-1.7%
Covered Lines:	3950
Relevant Lines:	12421

💛 - Coveralls

kannon92 · 2024-08-15T23:30:20Z

pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go

+}
+
+// TranJobList is a collection of training jobs.
+type TranJobList struct {


TrainJobList

Great catch!

kannon92 · 2024-08-16T00:56:34Z

pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go

+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	// Specification of the desired ClusterTrainingRuntime.
+	Spec TrainingRuntimeSpec `json:"spec,omitempty"`


Should the spec ever be empty for ClusterTrainingRUntime?

Not really, but I noticed that for all Kubernetes APIs the spec is set with omitempty: https://github.com/kubernetes/api/blob/master/apps/v1/types.go#L820.
@tenzen-y @kannon92 Any specific reason why we do this ?

IIUC, in any case, the spec field is defined as an optional field in the Kubernetes. So, the optional TrainingRuntime spec would be better.

kannon92 · 2024-08-16T00:58:16Z

pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go

+type TorchSpec struct {
+	// Number of processes per node.
+	// This value is inserted into the `--nproc-per-node` argument of the `torchrun` CLI.
+	// Supported values: `auto`, `cpu`, `gpu`, or int value.


You could probably use KubeBuilder validations for the enums here.

As we discussed offline, we will add validations in the separate PRs.
Tracking issue:

KEP-2170: Implement validations for TrainJob #2209

KEP-2170: Implement validations for TrainingRuntime and ClusterTrainingRuntime #2219

terrytangyuan · 2024-08-16T01:49:37Z

pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go

+type MLSpec struct {
+	// Configuration for the PyTorch runtime.
+	TorchSpec *TorchSpec `json:"torchSpec,omitempty"`
+
+	// Configuration for the MPI Runtime.
+	MPISpec *MPISpec `json:"mpiSpec,omitempty"`
+}


One concern I have is that frameworks configurations/spec change quite often. Have we considered using configmap for this so that we don't have a lot of responsibilities to maintain the compatibility, etc.?

Our goal is not to add all frameworks configurations here, but only parameters that require additional orchestration such as Elastic Policy, MPISpec. For example, in the future we can add SlurmSpec or FluxSpec here as we discussed with @vsoch here: #2171 (comment).

For Torch our assumption is that torchrun CLI is quite stable and probably won't change in the near future.

@terrytangyuan How do you think we can use ConfigMap for those parameters ?

tenzen-y · 2024-08-16T07:32:30Z

@andreyvelich First of all, could you generate / createregister.go, deepcopygen and so on by controller-tools?

Unless those functions, we can not use the API in the controllers/webhooks.

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich · 2024-08-16T13:45:14Z

@andreyvelich First of all, could you generate / createregister.go, deepcopygen and so on by controller-tools?

Unless those functions, we can not use the API in the controllers/webhooks.

I registered APIs with scheme and added deepcopygen via controller-gen. I think, we can add the defaulters and other parameters required for clients, listers, informers, in the following PRs.
Does it look good @tenzen-y ?

tenzen-y · 2024-08-16T13:47:39Z

@andreyvelich First of all, could you generate / createregister.go, deepcopygen and so on by controller-tools?
Unless those functions, we can not use the API in the controllers/webhooks.

I registered APIs with scheme and added deepcopygen via controller-gen. I think, we can add the defaulters and other parameters required for clients, listers, informers, in the following PRs. Does it look good @tenzen-y ?

That sounds good to me. We can create a separate issue "KEP-2170: Provide client-go library for TrainJob and TrainingRuntime".

tenzen-y · 2024-08-16T13:50:30Z

@andreyvelich First of all, could you generate / createregister.go, deepcopygen and so on by controller-tools?
Unless those functions, we can not use the API in the controllers/webhooks.

I registered APIs with scheme and added deepcopygen via controller-gen. I think, we can add the defaulters and other parameters required for clients, listers, informers, in the following PRs. Does it look good @tenzen-y ?

That sounds good to me. We can create a separate issue "KEP-2170: Provide client-go library for TrainJob and TrainingRuntime".

Created: #2224

kannon92 · 2024-08-16T20:53:32Z

pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go

+
+// PodGroupSpec represents a PodGroup configuration to enable gang-scheduling.
+type PodGroupSpec struct {
+	// Plugin for the gang-scheduling.


Are we going forward with a default?

By default, the gang-scheduling is disabled for TrainJob, since it requires plugin to be installed (coscheduling or volcano).

That makes sense.

kannon92 · 2024-08-16T20:54:31Z

pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go

+	NumNodes *int32 `json:"numNodes,omitempty"`
+
+	// JobSet configuration which will be used by TrainJob.
+	JobSetSpec *jobsetv1alpha2.JobSetSpec `json:",inline"`


Why inline here?

I think, we discussed it before, here is the example: https://github.com/kubeflow/training-operator/tree/master/docs/proposals/2170-kubeflow-training-v2#pytorch-distributed-runtime.
This will make API consistent with the JobSet. WDYT @kannon92 @tenzen-y ?

I'm asking because I don't know what that argument actually does. I usually only see if for really small objects. Never a Spec so not sure if that means literally we are putting the object "inline" or it skips protobuf or api generation? like I've seen it for type..

We want to give user functionality to set the whole JobSet spec under Training Runtimes.

type TrainingRuntimeSpec struct { MLPolicy *MLPolicy `json:"mlPolicy,omitempty"` JobSetSpec jonsetv2alpha2.JobSetSpec `json:"spec"` } type MLPolicy struct { // Number of training nodes. // Defaults to 1. NumNodes *int32 `json:"numNodes,omitempty"` MLPolicySource `json:",inline"` } type MLPolicySource struct { PyTorch ... }

Maybe, we want to dedicated field for the JobSetSpec so that we can identify the JobSetSpec.
@andreyvelich @kannon92 WDYT?

kannon92 · 2024-08-16T20:54:56Z

pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go

+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	// Specification of the desired TrainingRuntime.
+	Spec TrainingRuntimeSpec `json:"spec"`


Suggested change

Spec TrainingRuntimeSpec `json:"spec"`

Spec TrainingRuntimeSpec `json:"spec, omitempty"`

Similar to other specs?

Nice Catch!

kannon92 · 2024-08-16T20:56:46Z

pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go

+}
+
+// PodSpecOverrides represents the custom overrides that will be applied for the TrainJob's resources.
+type PodSpecOverrides struct {


@tenzen-y do we need schedulingGates here? I can't remember what we decided a few weeks ago..

Yes, we do not need to add the schedulingGate since we would expect the schedulingGates are added by the webhook based on the Pod creation event.

Indeed, Kueue adds the schedulingGates to Pods, not Job by webhook, when the Pods are created.

kuizhiqing

Great work !

kuizhiqing · 2024-08-19T12:14:40Z

pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go

+	Suspend *bool `json:"suspend,omitempty"`
+
+	// ManagedBy field indicates the controller that manages a TrainJob.
+	ManagedBy *string `json:"managedBy,omitempty"`


If the type of ManagedBy is string, I propose ControllerName inspired by SchedulerName.
Since ManagedBy reminder me of managedFields which is another struct.

This is a byproduct of https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/4368-support-managed-by-for-batch-jobs so I think we should not change this.

Yeah, that's right. This API is similar to Batch Job managedBy.
Here is the implementation from @mszadkow and @mimowo in V1 APIs: #2203

Thanks, I've missed it.

kuizhiqing · 2024-08-19T12:20:24Z

pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go

+	// Number of training nodes.
+	// TODO (andreyvelich): Do we want to support dynamic num of nodes in TrainJob for PyTorch elastic: `--nnodes=1:4` ?
+	NumNodes *int32 `json:"numNodes,omitempty"`


I prefer int here to refer to active/effective number instead of string.
IMO, the string value of nnodes if calculated from a config from like ElasticPolicy.

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

shravan-achar · 2024-08-19T21:05:11Z

pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go

+	NumProcPerNode *int32 `json:"numProcPerNode,omitempty"`
+
+	// Implementation name for the MPI to create the appropriate hostfile.
+	MPIImplementation *MPIImplementation `json:"mpiImplementation"`


Since MPIImplementation is a const, does this need to be a pointer? With a pointer, a nil is a valid value. Do we want user to pass a nil value?

Also with omitempty not specified, the zero value of this would be nil. The zero value will be included in all patch requests which seems unnecessary.

Maybe you meant to include the omitempty but accept a nil value?

Yes, I think we should remove pointer from here. @tenzen-y What do you think ?

// Implementation name for the MPI to create the appropriate hostfile. // Defaults to OpenMPI MPIImplementation *MPIImplementation `json:"mpiImplementation,omitempty"`

In most cases of optional fields, we should use the omitempty tag.
Please refer to more details here: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#optional-vs-required

shravan-achar · 2024-08-19T21:25:17Z

pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go

+// Every training runtime contains `trainer` container which represents Trainer.
+type Trainer struct {
+	// Docker image for the training container.
+	Image string `json:"image,omitempty"`


Could Image be a pointer to the string? The validation can differentiate between unspecified and empty string.

This marks the field as required in the CRD but a nil value can be passed.

@tenzen-y @shravan-achar Do we want to introduce pointer or +optional for image since user can skip the image for Trainer?
E.g. Kubernetes for container users +optional validation: https://github.com/kubernetes/api/blob/master/core/v1/types.go#L2708C6-L2709

As far as I understand, we can skip specifying this image field when the TrainingRuntime has the image.

// +optional Image *string `json:"image,omitempty"`

Regarding to the optional vs required, please refer to the above my comment.

shravan-achar · 2024-08-19T21:29:14Z

pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go

+// the `dataset-initializer` container in the `Initializer` Job.
+type DatasetConfig struct {
+	// Storage uri for the dataset provider.
+	StorageUri string `json:"storageUri"`


Could StorageUri be a pointer to the string? The validation could differentiate between unspecified and empty string.

Also omitempty is still necessary? From a PATCH perspective, if this field is unspecified, its zero value will be included in the serialization.

Let's make the storageUri optional given that Training Runtime can contain the default dataset and model.

shravan-achar · 2024-08-19T21:30:11Z

pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go

+// InputModel represents the desired pre-trained model configuration.
+type InputModel struct {
+	// Storage uri for the model provider.
+	StorageUri string `json:"storageUri"`


Same comment here as on L141

shravan-achar · 2024-08-19T21:30:56Z

pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go

+	Env []corev1.EnvVar `json:"env,omitempty"`
+
+	// Reference to the TrainJob's secrets to download dataset.
+	SecretRef corev1.SecretReference `json:"secretRef,omitempty"`


Why not take a pointer to corev1.SecretReference ?

Nice catch!

shravan-achar · 2024-08-19T21:32:05Z

pkg/apis/kubeflow.org/v2alpha1/trainjob_types.go

+// PodSpecOverrides represents the custom overrides that will be applied for the TrainJob's resources.
+type PodSpecOverrides struct {
+	// Names of the training job replicas in the training runtime template to apply the overrides.
+	TargetReplicatedJobs []string `json:"targetReplicatedJobs"`


Don't see the benefit of skipping omitempty . Probably we should include almost everywhere. Let me know your thoughts.

If we skip it from here, how we can make the targetReplicatedJobs mandatory parameter ?

Yes. It's possible to use Kubebuilder tags to make it required on the CRD. The omitempty will help with serialization. kubernetes-sigs/controller-tools#944

Suggested change

TargetReplicatedJobs []string `json:"targetReplicatedJobs"`

// +required

TargetReplicatedJobs []string `json:"targetReplicatedJobs"`

We can make this field as required like this.

google-oss-prow · 2024-08-21T13:56:06Z

@andreyvelich: GitHub didn't allow me to assign the following users: shravan-achar, kannon92.

Note that only kubeflow members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

Are there any other comments before we can merge this PR and start working on the controller implementation ?

/assign @shravan-achar @tenzen-y @kannon92 @kuizhiqing @terrytangyuan @johnugeorge
/hold cancel

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

kannon92 · 2024-08-22T16:50:23Z

LGTM on my end.

andreyvelich · 2024-08-22T16:52:23Z

We had some discussions with @tenzen-y about APIs and we proposed the following changes:

Define JobSetTemplateSpec under TrainingRuntimeSpec API. For some use-cases, Platform Engineers might want to add custom labels and annotations to JobSet for various features, such as alpha.jobset.sigs.k8s.io/exclusive-topology or alpha.jobset.sigs.k8s.io/node-selector: Document labels, annotations and taints for JobSet kubernetes/website#47383.
We don't want to define custom propagation mechanism from TrainingRuntime metadata to the JobSet, since it is not straighforward.
Move numNodes under MLSpec
We are still debating between MLSpec vs MLPolicy API name. Any thoughts @kannon92 @kubeflow/wg-training-leads @kuizhiqing @shravan-achar @vsoch ?

andreyvelich · 2024-08-22T17:33:30Z

/rerun-all

andreyvelich · 2024-08-22T18:14:50Z

pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go

+}
+
+// TorchMLSpecSource represents a PyTorch runtime configuration.
+type TorchMLSpecSource struct {


@tenzen-y I removed standalone from TorchMLSpecSource API.
I think, we can detect when numNodes=1 and numProcPerNode>1 to set the standalone parameter which sets the default values for rendezvous.

When users will need it, we can add it in the future.

That sounds good to me.
Thank you for the update.

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich · 2024-08-26T19:31:44Z

We made the final changes with @tenzen-y:

Rename MLSpec to MLPolicy since spec usually represents another Kubernetes resources that will be deployed.
Rename PodGroupSpec to PodGroupPolicy for the same reason, and make various schedulers (coscheduling, volcano, or YuniKorn) as one of API.

If we don't have any followup suggestions, we can merge it.

tenzen-y

Thank you for the updates!
/hold
/lgtm
/approve

Feel free to merge this PR.
Additionally, could you update the KEP in a separate PR?

google-oss-prow · 2024-08-26T20:02:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tenzen-y · 2024-08-26T20:02:36Z

/hold

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

tenzen-y · 2024-08-26T20:30:38Z

/lgtm

andreyvelich

Thanks everyone for the review, we should be ready to merge it.
I will create followup PR to update KEP with the correct APIs.
/hold cancel

KEP-2170: Add TrainJob and TrainingRuntime APIs

6865663

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

google-oss-prow bot added the do-not-merge/hold label Aug 15, 2024

google-oss-prow bot requested review from jinchihe and kuizhiqing August 15, 2024 23:17

google-oss-prow bot added the size/L label Aug 15, 2024

andreyvelich commented Aug 15, 2024

View reviewed changes

kannon92 reviewed Aug 15, 2024

View reviewed changes

kannon92 reviewed Aug 16, 2024

View reviewed changes

terrytangyuan reviewed Aug 16, 2024

View reviewed changes

andreyvelich added 2 commits August 16, 2024 13:42

Fix TrainJobList

ed830c8

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Register APIs with scheme

66e7049

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

google-oss-prow bot added size/XXL and removed size/L labels Aug 16, 2024

Add SchemeGroupVersion

bfa1f20

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich mentioned this pull request Aug 16, 2024

KEP-2170: Kubeflow Training V2 API #2170

Open

19 tasks

kannon92 reviewed Aug 16, 2024

View reviewed changes

kuizhiqing reviewed Aug 19, 2024

View reviewed changes

andreyvelich added 2 commits August 19, 2024 16:57

Fix TrainingRuntimeSpec omitempty

2bf13c9

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Generate manifests only for v1

72a933e

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

shravan-achar reviewed Aug 19, 2024

View reviewed changes

google-oss-prow bot removed the do-not-merge/hold label Aug 21, 2024

google-oss-prow bot assigned kuizhiqing, tenzen-y and terrytangyuan Aug 21, 2024

Use pointer for MPIImplementation

49a004c

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich force-pushed the issue-2206-add-apis branch from d70db47 to 49a004c Compare August 21, 2024 16:39

Update the JobSetTemplate API

c28a166

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich commented Aug 22, 2024

View reviewed changes

Rename PodGroupPolicy and MLPolicy APIs

06e7653

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

tenzen-y reviewed Aug 26, 2024

View reviewed changes

google-oss-prow bot added the lgtm label Aug 26, 2024

google-oss-prow bot added the approved label Aug 26, 2024

google-oss-prow bot added do-not-merge/hold and removed lgtm labels Aug 26, 2024

Update comments

7aa4094

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich force-pushed the issue-2206-add-apis branch from 338cc29 to 7aa4094 Compare August 26, 2024 20:26

google-oss-prow bot added the lgtm label Aug 26, 2024

andreyvelich commented Aug 27, 2024

View reviewed changes

google-oss-prow bot removed the do-not-merge/hold label Aug 27, 2024

google-oss-prow bot merged commit 181191e into kubeflow:master Aug 27, 2024
38 of 39 checks passed

andreyvelich deleted the issue-2206-add-apis branch August 27, 2024 16:39

andreyvelich mentioned this pull request Aug 28, 2024

KEP-2170: Update Training V2 APIs in the KEP #2240

Merged

	Spec TrainingRuntimeSpec `json:"spec"`
	Spec TrainingRuntimeSpec `json:"spec, omitempty"`

	TargetReplicatedJobs []string `json:"targetReplicatedJobs"`
	// +required
	TargetReplicatedJobs []string `json:"targetReplicatedJobs"`

KEP-2170: Add TrainJob and TrainingRuntime APIs #2223

KEP-2170: Add TrainJob and TrainingRuntime APIs #2223

Conversation

andreyvelich commented Aug 15, 2024

ahg-g commented Aug 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Aug 15, 2024 • edited Loading

Pull Request Test Coverage Report for Build 10566761788

Details

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y commented Aug 16, 2024 • edited Loading

andreyvelich commented Aug 16, 2024

tenzen-y commented Aug 16, 2024

tenzen-y commented Aug 16, 2024

Choose a reason for hiding this comment

andreyvelich Aug 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kuizhiqing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shravan-achar Aug 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

google-oss-prow bot commented Aug 21, 2024

kannon92 commented Aug 22, 2024

andreyvelich commented Aug 22, 2024

andreyvelich commented Aug 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyvelich commented Aug 26, 2024 • edited Loading

tenzen-y left a comment • edited Loading

Choose a reason for hiding this comment

google-oss-prow bot commented Aug 26, 2024

tenzen-y commented Aug 26, 2024

tenzen-y commented Aug 26, 2024

andreyvelich left a comment

Choose a reason for hiding this comment

coveralls commented Aug 15, 2024 •

edited

Loading

tenzen-y commented Aug 16, 2024 •

edited

Loading

andreyvelich Aug 19, 2024 •

edited

Loading

tenzen-y Aug 21, 2024 •

edited

Loading

shravan-achar Aug 19, 2024 •

edited

Loading

andreyvelich commented Aug 26, 2024 •

edited

Loading

tenzen-y left a comment •

edited

Loading