Accommodate for tainted nodes to match kubeflow pipeline pods

[Shreyanand]

Is your feature request related to a problem? Please describe.
Environment: Elyra UI on the Open Data Hub Jupyterhub

Feature: When we try to run a Kubeflow pipeline, a notebook that uses GPU fails to run. If we look at the scheduled pod it shows an error that it couldn't find the right node because of the taints on the nodes.

Error:
0/13 nodes are available: 3 Insufficient nvidia.com/gpu, 3 node(s) had taint {node-role.kubernetes.io/infra: }, that the pod didn't tolerate, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 4 node(s) had taint {odh/notebook: true}, that the pod didn't tolerate.

Describe the solution you'd like
There should be a way to add toleration to the pods scheduled for running gpu workloads. This would make sure that the pod is able to find the right tainted node.

Describe alternatives you've considered
Another way to solve this would be to remove the taints from the nodes but that is not a recommended practice.

cc @HumairAK

[ptitzler]

Please subscribe to #2823 for updates.