Seldon controller pod crashes on deploy when no internet connection available

[zjorgensenbits]

I'm running v1.0 of Seldon and I've noticed that if I disconnect internet access and then try to apply a SeldonDeployment with Kubectl, the seldon-controller-manager enters a CrashLoopBackoff state and I get the following error:

Internal error occurred: failed calling webhook "mseldondeployment.kb.io": Post https://seldon-webhook-service.seldon.svc:443/mutate-machinelearning-seldon-io-v1alpha2-seldondeployment?timeout=30s: dial tcp 10.43.115.4:443: connect: network is unreachable

If I then reconnect internet access and kill the seldon-controller-manager pod, it respawns fine and I'm able to deploy again without problems.

I'm interested to know what causes this to happen and whether it is resolvable, as I will need to be able to deploy models without an internet connection.

This could be related to issue #1361, but I'm not sure.

[ukclivecox]

Seems it could be related to #1361 - it should be a cluster local IP address so I'm not sure why its happening. It looks like the 10.43.x.x address is broken when you close off the internet. How are you "disconneting the internet"?

[zjorgensenbits]

I have a single node cluster running on a laptop with K3s. I'm just disconnecting from the wireless access point...

[adriangonz]

I'm not too familiar with K3s but it seems that it may need some extra config when there is no internet gateway set. @zjorgensenbits have you checked the following issue k3s-io/k3s#1103?

[zjorgensenbits]

No, I hadn't seen that issue -- thanks for the reference. I'll try that workaround in a bit and report back on what happens.

[ukclivecox]

Closing - please reopen if an issue