KEP-2170: Implement runtime framework

[tenzen-y]

What this PR does / why we need it:
Brief Design: https://docs.google.com/presentation/d/1HyEsBa7hxWpIoBXaX6uECiB48FWB85SG1kx15mO8hug/edit#slide=id.g30596bfee76_0_202

I implemented the runtime framework interfaces.
The responsibilities are the following:

• /runtime.v2/core: This contains the actual Kubeflow Job Pipeline like TrainigRuntime (not CRD), which is an internal concept.
  These pipelines build objects or create reconcile builders. We will add some pipelines in the future like SingleHostTrainingRuntime.

• /runtime.v2/framework: This contains the Kubeflow Job Pipeline Framework, which has some extension points in the following, and we will add some extension points in the future.

  • WatchExtensionPlugin
  • EnforcePodGroupPolicyPlugin
  • EnforceMLPolicyPlugin
  • CustomValidationPlugin
  • ComponentBuilderPlugin

• /runtime.v2/framework/plugins: This contains the Kubeflow Job Pipeline Framework plugins, which implement the Framework extension points. Each of these plugins is performed in Kubeflow Job Pipeline Framework extension points.

  • coscheduleing
  • jobset (Under development)
  • mpi
  • plainml
  • torch

Additionally, I did not implement all plugins. So, I will open an issue and delegate plugin implementation contributors who are interested in this project.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #
Part-of #2290

Checklist:

• Docs included if any changes are user facing

[coveralls]

Pull Request Test Coverage Report for Build 11372731768

Details

• 9 of 9 (100.0%) changed or added relevant lines in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 11330381194:	0.0%
Covered Lines:	73
Relevant Lines:	73

---

💛 - Coveralls

[andreyvelich]

Why do we need kueue dependency ?

[tenzen-y]

This dependency came from

training-operator/pkg/runtime.v2/runtime.go

Line 125 in 22da8af

PodRequests: kueuelr.TotalRequests(&spec.podSpec),

.

This allows us to set the appropriate required resources for PodGroup. If we remove this dependency, we need to just copy Kueue's "TotalRequests" function here. I believe that just coping and pasting is not the ideal way.

[terrytangyuan]

It would be great to reduce the dependency. Maybe copy and paste is okay in this case as long as we provide a reference to the original source

[tenzen-y]

That's not simple implementation. That's so complex, multiple files and lines codes.
So, I would propose keeping it here and then (after kube 1.32) switching to the kube library as I mentioned in #2280.

[andreyvelich]

Yes, we discussed with @tenzen-y offline that after k/k separates this utility function, we will remove dependency on Kueue.

[andreyvelich]

Do we need default options for Info object ?

[tenzen-y]

Suggested change

	options := defaultOptions
	for _, opt := range opts {
	opt(&options)
	}
	options := InfoOptions{}
	for _, opt := range opts {
	opt(&options)
	}

Do you recommend this?

[andreyvelich]

Maybe, I am trying to understand how are your planning to use InfoOptions in other parts ?
@tenzen-y What are the differences between Info{} and InfoOptions{} ?

[tenzen-y]

The infoOptions is object to set up the Info object.
This approach allows us to dynamically specify the parameters to the Info.

When we get rid of the InfoOptions, we need to specify all parameters everytime or need to directly pass the Info object.

[tenzen-y]

This approach is well-known to avid the following function:

// Even if all parameters is not used, all parameters should be specified.
func Foo(paramA string, paramB int, paramC int32, paramD bool, paramE int64)

// After introduced `infoOptions`.
func Foo(params ...InfoOption)

[andreyvelich]

I see, so maybe let's name it as InfoOption, not defaultOptions to make it clearer, and since we are not going to have default values for Info object.

[tenzen-y]

we are not going to have default values for Info object.

Actually, this is the default info Option. Here, this means that the default is an empty struct.
So, the below indicates to initialize options as a default values (currently default has empty fields)

options := defaultOptions

[tenzen-y]

In the future, we need to consider if we should the default parameters to infoOption. For example common labels and annotations.

[andreyvelich]

I see, that makes sense.

[andreyvelich]

How are we using this while enforcing the MLPolicy ?

[tenzen-y]

~~ if info == nil || info.MLPolicy != nil { ~~

I wanted to implement this in line 43. Maybe I failed to reabase.
Let me fix this.

[tenzen-y]

NVM above comments.

[tenzen-y]

spec:
  mlPolicy:
    numNodes: 1

We can imagine this situation.

[andreyvelich]

I see, so we override the value that we set here, right ?

training-operator/pkg/runtime.v2/runtime.go

Lines 120 to 125 in 0c376d3

	for _, spec := range options.podSpecReplicas {
	info.TotalRequests[spec.name] = TotalResourceRequest{
	Replicas: spec.replicas,
	// TODO: Need to address LimitRange and RuntimeClass.
	PodRequests: kueuelr.TotalRequests(&spec.podSpec),
	}

[tenzen-y]

Yes, this plainml codes try to update with the proper one.

[andreyvelich]

Should the indexer be equal to the Golang struct or json name ?
E.g. the trainingRuntimeSpec is named as .spec

training-operator/pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go

Line 77 in 22d5dde

Spec TrainingRuntimeSpec `json:"spec,omitempty"`

[tenzen-y]

We can specify the arbitrary key name. But the key is global within the training-ooerator.

[andreyvelich]

We should be ready to merge this.
Thank you for this great work @tenzen-y!
/lgtm
/assign @terrytangyuan @johnugeorge @kannon92

[google-oss-prow]

@andreyvelich: GitHub didn't allow me to assign the following users: kannon92.

Note that only kubeflow members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

We should be ready to merge this.
Thank you for this great work @tenzen-y!
/lgtm
/assign @terrytangyuan @johnugeorge @kannon92

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kannon92]

I don't really have context to review this at the moment.

I'll leave that to kubeflow members.

[tenzen-y]

We should be ready to merge this. Thank you for this great work @tenzen-y! /lgtm /assign @terrytangyuan @johnugeorge @kannon92

Thanks for the review!

[andreyvelich]

/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johnugeorge]

/hold cancel