Set up controllers using goroutines to start the manager quickly

[tenzen-y]

What this PR does / why we need it:
The training-operator has many controllers, and each controller needs access to the kube-apiserver while launching.
So, in the heavy load cluster, the training operator can not start by timeout period.

Related to: #1841

By this PR, the training operator starts the manager not to wait for completed to register all controllers.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

• Docs included if any changes are user facing

[coveralls]

Pull Request Test Coverage Report for Build 5631716043

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 6 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-0.06%) to 40.336%

---

Files with Coverage Reduction	New Missed Lines	%
pkg/controller.v1/mpi/mpijob_controller.go	6	80.12%

Totals
Change from base Build 5625056103:	-0.06%
Covered Lines:	3861
Relevant Lines:	9572

---

💛 - Coveralls

[johnugeorge]

Great. This is a needed fix which will reduce the start up time. However, timeout issue can still happen right?

[tenzen-y]

Great. This is a needed fix which will reduce the start up time. However, timeout issue can still happen right?

Yea, the potential issue still is left. However, I believe this improvement is effective for many situations since the manager starts without waiting for registering controllers.

1. Register controllers to the manager in another thread.
2. Register probe server to manager.
3. Start the manager without waiting for register controllers to the manager. (probe servers are ready)

[johnugeorge]

Are there side effects in declaring readiness before controllers are ready?

[tenzen-y]

Are there side effects in declaring readiness before controllers are ready?

I think there aren't exist side effects even if users create XXXJobs before all controllers are registered to the manager.
Because controllers reconcile against XXXJobs after started.

[johnugeorge]

Thanks @tenzen-y
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment