upgrading from 1.1 to 1.2 with existing sdep leads to volume mount error

[ryandawsonuk]

Seems to stem from #1863

Problem appears to be that mutating webhook applies the volume mount to the sdep. So my sdep (iris example) contains:

          volumeMounts:
          - mountPath: /etc/podinfo
            name: podinfo

The mutating webhook doesn't modify this on the upgrade to 1.2, if it runs at all (searching logs it seems it has not). The result is an error in the operator when trying to apply both mounts:

2020-06-26T15:03:48.105Z        ERROR   controller-runtime.controller   Reconciler error        {"controller": "seldon-controller-manager", "request": "seldon/iris", "error": "Deployment.apps \"iris-default-0-iris-container\" is invalid: [spec.template.spec.containers[0].volumeMounts[0].name: Not found: \"podinfo\", spec.template.spec.containers[0].volumeMounts[1].mountPath: Invalid value: \"/etc/podinfo\": must be unique]"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88

We could fix by adjusting this check (and any other volFound checks) -

seldon-core/operator/apis/machinelearning.seldon.io/v1/seldondeployment_webhook.go

Line 146 in ce5f873

volFound := false

to check for both the old and new volume mount names.

We should not change the mount path as that is used in the wrapper and then models would have to be rewrapped.

[ryandawsonuk]

Upgrade tests don't pick up on this because it repeatedly checks http requests. But the existing pods keep running.