Training: Reorganized Training Operator Docs

[andreyvelich]

Related: kubeflow/training-operator#1998.
I created the following sections for Training Operator docs:

• Overview
• Installation
• Getting Started
• User Guides
• Reference

A few points:

• @StefanoFioravanzo @kubeflow/wg-training-leads Any ideas on what we could add to Why Training Operator ? section ? Initially, we can just add some basic info.
• I didn't move CRDs to reference in this PR since we don't have time to discuss how we are going to generate them. What do you think we should do in this PR ?
• Do we need to have working example in GettingStarted page ? Would it be too complicated to consume ?

/hold for review

/assign @StefanoFioravanzo @kubeflow/wg-training-leads @hbelmiro @kuizhiqing @droctothorpe @franciscojavierarceo
Looking for your feedback!

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[franciscojavierarceo]

nit: my recommendation would be to populate this even with a trivial single layer NN that would actually run for this example. It helps users get started that may just be executing copy-pasta.

[andreyvelich]

Sure, let me try to add something simple.

[StefanoFioravanzo]

@andreyvelich thanks for this!

Any ideas on what we could add to Why Training Operator? section ?

Let's start with something simple and iterate in future PRs. We can start by answering questions like:

• How does training operator simplify distributed training with respect to a more traditional approach?
• How does Kubernetes help in solving these problems?
• Why is Training Operator part of the Kubeflow ecosystem?

I didn't move CRDs to reference in this PR since we don't have time to discuss how we are going to generate them. What do you think we should do in this PR ?

Makes sense. I'd keep the scope of this PR to the restructuring you already implemented. Let's iterate on content separtely. We can address each framework's user guide in dedicated PRs.

Do we need to have working example in GettingStarted page ? Would it be too complicated to consume ?

Getting Started should have an end-to-end working (yet simple) example. Generally people just want to copy paste some stuff, run it, and see results. Then you typically link some more advanced tutorials or user guides at the end

[andreyvelich]

That makes sense @StefanoFioravanzo, I added initial ideas for Why Training Operator ? and also I added the AI/ML lifecycle diagram that we can re-use in various Kubeflow components to explain which stage of lifecycle each component addresses (e.g. Spark Operator, Model Registry, Katib, Notebooks, KServe).
Please let me know what do you think @kubeflow/wg-training-leads @StefanoFioravanzo ?

[franciscojavierarceo]

I love this diagram. Based on my comment here, it may make sense to also feature serving along with model serving (though I note in my comment that there are some architectural choices to be had on which order the serving should happen). Regardless, I think it's worth stating explicitly that feature serving is a component in model serving.

[andreyvelich]

That makes sense, do we want to update the diagram once we discuss the architecture for Feature Serving ?
For this diagram, I just took diagram that we worked together with @ronaldpetty @zanetworker for CNCF WG AI WhitePaper
https://www.cncf.io/wp-content/uploads/2024/03/cloud_native_ai24_031424a-2.pdf

[franciscojavierarceo]

Oh nice, looks like that was very recent. @ronaldpetty @zanetworker let me know if you have any thoughts/opinions there. Would love for feature serving to be included in this as I think it is becoming increasingly more important.

[andreyvelich]

@franciscojavierarceo Not sure how to make it simple in this diagram. What do you think about this:

[franciscojavierarceo]

This may be overkill...but here's my attempt at it. It's technically missing the measurement required to actually repeat the process (e.g., a click for a recommendation engine).

[andreyvelich]

That makes sense, thanks @franciscojavierarceo!
A few questions:

1. Isn't the feature store also use for Model Training and Optimization ?
2. Should we split the Model Development, Iteration, and Optimization with:
   Model Experimentation and Development (Kubeflow Notebooks) --> Model Optimization and Hyperparameter Tuning (Katib) (similar to this flow: https://www.kubeflow.org/docs/started/architecture/#introducing-the-ml-workflow).
   Since I also want to use this diagram in other docs: Kubeflow Notebooks, Kubeflow Katib.

[franciscojavierarceo]

Isn't the feature store also use for Model Training and Optimization ?

In some sense yes. Feature selection is often done during Model Development, Iteration, and Optimization but that is downstream of Feature Extraction. In short, you have to pull all of the features you want first before you can select which ones are best for your model, so this diagram is more representative of how that flow actually works. Let me know if you have additional thoughts there.

Should we split the Model Development, Iteration, and Optimization with:
Model Experimentation and Development (Kubeflow Notebooks) --> Model Optimization and Hyperparameter Tuning (Katib) (similar to this flow: https://www.kubeflow.org/docs/started/architecture/#introducing-the-ml-workflow).

How about this:

[andreyvelich]

That looks good, I would also split Model Experimentation with Chose ML Algorithms + Code Model and HP Tuning + Architecture Search.
That will allow us to explain users on which stage Kubeflow Notebooks are used on which Kubeflow Katib is used

[franciscojavierarceo]

Will send you a message on slack

[zanetworker]

@franciscojavierarceo this sounds like a good discussion to have on #wg-artificial-intelligence :)

If we are looking for abstractions, I'd categorize it as Data prep, model building (training, tuning, experimentation, feature engineering,...), model serving & deployment (pull from registry, deploying inference,...), then operation & eval (monitoring,..).

I.e., there are phases in the lifecycle (building, serving, operations and iteration), components (feature stores, registries, infra,...), and personas (highlighted in the white paper, but also here: https://tag-runtime.cncf.io/wgs/cnaiwg/glossary/#data-engineers, https://tag-runtime.cncf.io/wgs/cnaiwg/glossary/#data-engineers).

Aligning, reusing the existing language across would be great and if there are language gaps we could patch in cnai as well :)

[andreyvelich]

@franciscojavierarceo I added working example, does it look good ?

[hbelmiro]

Broken link.

Suggested change

	| PaddlePaddle | [PaddleJob](/docs/components/training/user-guides/paddlepaddle/) |
	| PaddlePaddle | [PaddleJob](/docs/components/training/user-guides/paddle/) |

[andreyvelich]

Nice catch!

[andreyvelich]

@franciscojavierarceo and I made a few changes for the AI/ML lifecycle diagram, so it would be easier to use it in other Kubeflow Components doc (e.g. Katib, FEAST, Model Registry, KServe, Notebooks, Spark Operator).

[andreyvelich]

This PR should be ready unless you have any other comments.
/hold cancel

[hbelmiro]

/lgtm