Improve speed of execution for integration tests

[axsaucedo]

Fixes #1032 and #1084

This PR includes:

• Namespaced tests (which could allow for running outside KIND in main cluster if we want in future)
• Support for parallel run of tests (and also defaults to single execution)
• Extra KIND cluster config to allocate memory to kubeadm and system pods
• Refactor tests to remove replicated methods
• Added retries to tests
• Only build engine/wrapper/operator if files were modified
• Speed of execution gone down to 20 mins (ran with pytest with 6 workers in larger cluster)

It seems that there are still some failures, which are caused from the master node crashing. This results in the Seldon (and cert) pods crashing, resulting in failure of kubectl apply for new models (due to the mutating/validating webhook not being reachable).

An attempt to fix this was to add the requirements for min memory/CPU for system components in the KIND config file, but it still seems to be an issue. Having larger instances would probably reduce the failures due to this.

This is something that should be investigated further, but seems to be a bug related to KIND. Here is an example of the pods which fail.

kube-system            kube-controller-manager-kind-control-plane   0/1     CrashLoopBackOff    7          38m
cert-manager           cert-manager-cainjector-5cd66c9c45-2wldz     0/1     CrashLoopBackOff    7          37m
seldon-system          seldon-controller-manager-86c88444c8-k6s4l   0/1     CrashLoopBackOff    7          36m

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[adriangonz]

Niiice work! Getting it down to 20 mins it's awesome!

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/test integration

[axsaucedo]

/cc @cliveseldon
/cc @adriangonz

Integration tests now all pass whenever KIND doesn't break 🚀

[ukclivecox]

/approve

[seldondev]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cliveseldon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cliveseldon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment