Support postgres as a katib db

[anencore94]

What this PR does / why we need it:

• Implement postgres for katib backend and support it as a built-in feature of katib.
• Add kustomization configurations for using postgres.
• Support e2e test on both mysql and postgres.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #915

Checklist:

• Docs included if any changes are user facing

[coveralls]

Coverage increased (+0.6%) to 73.433% when pulling 13cf73b on anencore94:feature/postgres-for-katib-backend-db into 42bc6a9 on kubeflow:master.

[gaocegege]

/assign @johnugeorge @andreyvelich

[tenzen-y]

Thanks for implementing this great feature! @anencore94
It looks good to me about core logic!

[johnugeorge]

Should we keep add postgres support as a separate Kustomization file similar to katib-standalone?

https://github.com/kubeflow/katib/tree/master/manifests/v1beta1/installs

It is good to add this in GitHub actions. In that case, We have to add an extra option in the setup script to take db manager backend(mysql, postgres)
https://github.com/kubeflow/katib/blob/master/.github/workflows/template-e2e-test/action.yaml#L27

What do you think?

[anencore94]

Should we keep add postgres support as a separate Kustomization file similar to katib-standalone?
It is good to add this in GitHub actions. In that case, We have to add an extra option in the setup script to take db manager backend(mysql, postgres)

Yeap, I also think that looks more safe to use both Database :)
So you mean that

1. add another folder with kustomization configurations with postgres.
2. add option to setup katib with postgres so that we could run all e2e test for both database.
   right ? @johnugeorge

[johnugeorge]

Should we keep add postgres support as a separate Kustomization file similar to katib-standalone?
It is good to add this in GitHub actions. In that case, We have to add an extra option in the setup script to take db manager backend(mysql, postgres)

Yeap, I also think that looks more safe to use both Database :) So you mean that

1. add another folder with kustomization configurations with postgres.
2. add option to setup katib with postgres so that we could run all e2e test for both database.
   right ? @johnugeorge

Yes. Correct.

[anencore94]

@johnugeorge Ok, I'll add that feature with this PR. Thanks for your comments :)

[tenzen-y]

Did you face the same error on your local machine?

No, In my local minikube environments, validatingwebhook works well : When I create invalid-expereiment.yaml, both enviroments(with mysql or with postgresql) reports expected error message as follows:

Error from server: error when creating "invalid-experiment.yaml": admission webhook "validator.experiment.katib.kubeflow.org" denied the request: unable to get Suggestion config data for algorithm invalid-algorithm: failed to find suggestion config for algorithm: invalid-algorithm in ConfigMap: katib-config

@tenzen-y

@anencore94 I see.
Can you insert kubectl logs -l "katib.kubeflow.org/component in (controller,db-manager,cert-generator)" -n kubeflow --all-containers=true to this section?

[tenzen-y]

Thanks for your contribution! @anencore94
I'm looking forward to releasing this feature!

I left a few comments.

[tenzen-y]

@kubeflow/wg-automl-leads Can you re-kick fail GH-Action Jobs?

[anencore94]

@tenzen-y Thanks for your detailed reviews :) I applied your comments.

[tenzen-y]

Thanks for updating PR! @anencore94
/lgtm

[johnugeorge]

Great work.
One question: Is it better to add readiness probe than static 30 sec sleep?

[anencore94]

Great work. One question: Is it better to add readiness probe than static 30 sec sleep?

Absolutely agree. I'll add such feature to katib-controller as a follow-up PR. @johnugeorge

[johnugeorge]

Thanks
/lgtm

[johnugeorge]

/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anencore94, johnugeorge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tenzen-y]

@anencore94
It might be better to add documentation about env vars for PostgreSQL to this section.

[anencore94]

@tenzen-y Sure, Thanks for checking :)