UAT test scenario to use GPU

[nishant-dash]

Context

Can we have a UAT test (or set of tests) that helps run a test workload that utilizes one or more gpus? Ideally this would be both a notebook and a pipeline job.
It may be tricky to make it generic enough to run in any environment out of the box, but just having a test there with minimal assumptions is better than having no tests for it.

This will be super helpful to have when running validation on various cloud deployments.

What needs to get done

UAT test (or set of tests) that helps run a test workload that utilizes one or more gpus
Ideally this would be both a notebook and a pipeline job.

Definition of Done

Working UATs for notebook and pipelines that test gpu utilization successfully.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6358.

This message was autogenerated

[orfeas-k]

Thank you for the proposal @nishant-dash . We will use this issue to implement the notebook itself. First, we will have to enable the driver though in order to enable running the notebook in an automated way, which means this will be worked on after #130, #131, #132.

EDIT: The design should be linked to the KF113 epic.

[orfeas-k]

The proposed notebook in #139 uses kfp SDK in order to create an experiment and run it. The pipeline:

• Schedules runs in a GPU node. That means that if there is no NVIDIA GPU, the run's pod will remain pending and cause the test to timeout and fail.
• Runs code which uses a Tensorflow framework function in order to detect if it can find a GPU in the node. If it doesn't find one, it raises resulting the test to timeout and fail.