[bug or feature] Job pods cannot be deleted

[gaocegege]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

Create a experiment with job, and delete it. The pods created by the job cannot be deleted. I am not sure if it is a bug or a feature of k8s job.

NAME                                                         READY   STATUS    RESTARTS   AGE
pod/katib-controller-5b8bdd684b-chvbh                        1/1     Running   0          6m21s
pod/katib-db-65966f85f-m4xrf                                 1/1     Running   0          21m
pod/katib-manager-c68bcdd8-bvtqb                             1/1     Running   0          21m
pod/katib-manager-rest-7b54c7cddd-fnfph                      1/1     Running   0          21m
pod/katib-suggestion-bayesianoptimization-785bb94b7b-zpp5j   1/1     Running   0          21m
pod/katib-suggestion-random-76c5c9c6d4-rfvd6                 1/1     Running   0          21m
pod/katib-ui-554789d55b-49rvk                                1/1     Running   0          21m
pod/random-experiment-89kswt8j-69c4b                         0/1     Error     0          13m
pod/random-experiment-89kswt8j-f9bmg                         0/1     Error     0          14m
pod/random-experiment-bzfrbbvv-gl2hh                         0/1     Error     0          13m
pod/random-experiment-bzfrbbvv-jll4j                         0/1     Error     0          14m
pod/random-experiment-m6ph9bsb-jlr9w                         0/1     Error     0          14m
pod/random-experiment-m6ph9bsb-lb6gc                         0/1     Error     0          13m
pod/random-experiment-rm95htxz-qrqm4                         0/1     Error     0          13m
pod/random-experiment-rm95htxz-rxddl                         0/1     Error     0          13m
pod/random-experiment-z277ck6x-g556k                         0/1     Error     0          14m
pod/random-experiment-z277ck6x-wfjrk                         0/1     Error     0          14m

NAME                                            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/katib-controller                        ClusterIP   10.0.0.83    <none>        443/TCP    21m
service/katib-db                                ClusterIP   10.0.0.176   <none>        3306/TCP   21m
service/katib-manager                           ClusterIP   10.0.0.204   <none>        6789/TCP   21m
service/katib-manager-rest                      ClusterIP   10.0.0.179   <none>        80/TCP     21m
service/katib-suggestion-bayesianoptimization   ClusterIP   10.0.0.19    <none>        6789/TCP   21m
service/katib-suggestion-random                 ClusterIP   10.0.0.73    <none>        6789/TCP   21m
service/katib-ui                                ClusterIP   10.0.0.200   <none>        80/TCP     21m

NAME                                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/katib-controller                        1/1     1            1           21m
deployment.apps/katib-db                                1/1     1            1           21m
deployment.apps/katib-manager                           1/1     1            1           21m
deployment.apps/katib-manager-rest                      1/1     1            1           21m
deployment.apps/katib-suggestion-bayesianoptimization   1/1     1            1           21m
deployment.apps/katib-suggestion-random                 1/1     1            1           21m
deployment.apps/katib-ui                                1/1     1            1           21m

NAME                                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/katib-controller-5b8bdd684b                        1         1         1       21m
replicaset.apps/katib-db-65966f85f                                 1         1         1       21m
replicaset.apps/katib-manager-c68bcdd8                             1         1         1       21m
replicaset.apps/katib-manager-rest-7b54c7cddd                      1         1         1       21m
replicaset.apps/katib-suggestion-bayesianoptimization-785bb94b7b   1         1         1       21m
replicaset.apps/katib-suggestion-random-76c5c9c6d4                 1         1         1       21m
replicaset.apps/katib-ui-554789d55b                                1         1         1       21m

What did you expect to happen:

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Kubeflow version:
• Minikube version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[andreyvelich]

In v1alpha all components of StudyJob are deleted when StudyJob is deleted. I think, we have missed something in the Experiment controller.