add a tensorflow example and update task documentation

[kannon92]

Fixes #78

I wanted to add some documentation updates to tasks also as the above story says "to add a task" but we had no writeup in tasks. I decided to link to examples and write some words around these examples.

Added a really simple example for Tensorflow. I used https://github.com/kubeflow/training-operator/blob/master/examples/tensorflow/simple.yaml as a template.

There are some issues with this one actually.

The logs of the pods have some warnings around deprecations.

[ec2-user@ip-172-31-93-184 jobset-kevin]$ kubectl logs tensorflow-tensorflow-0-0-5l6bb
WARNING:tensorflow:From /var/tf_mnist/mnist_with_summaries.py:39: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please write your own downloading logic.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:252: wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please use urllib or similar directly.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: __init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
2023-08-10 13:42:32.288488: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA

[danielvegamyhre]

Can you add a note about what the expected output should be / how to check if the example is working?

[danielvegamyhre]

just noticed this is using kubeflow, which imo kind of defeats the purpose of using jobset, since kubeflow is presumably creating TFJobs under the hood, a CRD which encapsulates the TF distributed training primitives like the leader, workers, parameter server, which when modeled as a JobSet would each be a different ReplicatedJob, so they can have different success/failure policies applied to them and so on.

It's also handling the setting of the TF_CONFIG env var which is something the user still needs to configure when using JobSets, although automating the configuration of these kinds of env vars is part of the future plans for JobSet.

[kannon92]

So the image is mostly just a KubeFlow hosted image that builds/deploys tensorflow. It’s not using KubeFlow.

If you’d prefer to just host a tensorflow image in Jobset (like we did for PyTorch) we could go that route. I have very little experience with tensorflow so I was grabbing workable examples from the internet.

[danielvegamyhre]

Yeah, if the user wants to use this image I don't see any reason to use JobSet here, as none of its features are being used, it's just running a single Indexed Job. Is the headless service being utilized via pod-to-pod communication via pod hostnames at least? What happens when we set enableDNSHostnames: false to explicitly not create the headless service, does the example still work?

[kannon92]

I'll have to check. Where is your source for the pytorch image btw?

[danielvegamyhre]

I don't have the Dockerfile or code for it published publicly but I can share it somewhere. Perhaps I should include the training script and Dockerfile in examples/pytorch along with the JobSet yaml that is already there?

[kannon92]

Yea I think that is good idea IMO. I notice that these images/code can code rot pretty quickly.

I brought the tensorflow example being out-of-date on kubeflow ie kubeflow/training-operator#1884.

Looks like they have the source defined in the repo and some pushes to the images. Maybe we don't need to go that far but it would be nice to have a way to up-date these if they rot too much.

[ahg-g]

I think this is fine, the image has nothing to do with kubeflow itself, it is just the training script.

[ahg-g]

one minor nit

[kannon92]

@danielvegamyhre or @ahg-g any outstanding items on this PR?

[ahg-g]

/lgtm
/approve
/label tide/merge-method-squash

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, kannon92

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment