dind deployment flake: Timeout waiting for 3 nodes to report readiness

[0xmichalis]

Any dind deployment failures in ci will result in an error message similar to the following:

............................................................
[ERROR] Timeout waiting for 3 nodes to report readiness
[ERROR] PID 30612: hack/dind-cluster.sh:45: `return 1` exited with status 1.
[INFO]      Stack Trace: 
[INFO]        1: hack/dind-cluster.sh:45: `return 1`
[INFO]        2: hack/dind-cluster.sh:114: wait-for-cluster
[INFO]        3: hack/dind-cluster.sh:344: start
[INFO]   Exiting with code 1.
[INFO] Saving cluster configuration
[ERROR] Failed to deploy cluster for plugin: {multitenant}

Diagnosing the problem will require looking at the dind cluster artifacts saved by the job.

[0xmichalis]

Latest base ami failure: https://ci.openshift.redhat.com/jenkins/view/All/job/ami_build_origin_int_rhel_base/161/consoleFull#14075125555898c58db7602c31c0eab717

[0xmichalis]

https://ci.openshift.redhat.com/jenkins/view/All/job/merge_pull_request_origin/1271/consoleFull#2086332504575ef9a2e4b031e1b25ce320

[tnozicka]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/14910/test_pull_request_origin_extended_networking_minimal/8761/

Stopping dind cluster 'nettest'
Starting dind cluster 'nettest' with plugin 'redhat/openshift-ovs-multitenant' and runtime 'dockershim'
Waiting for ok
............................................................................
Done
Waiting for 3 nodes to report readiness
........................................................................................................................
[ERROR] Timeout waiting for 3 nodes to report readiness
[ERROR] PID 31630: hack/dind-cluster.sh:41: `return 1` exited with status 1.
... skipping 7 lines ...
[ERROR] Failed to deploy cluster for plugin: {multitenant}
... skipping 10712 lines ...
[ERROR] 1 plugin(s) failed one or more tests
[ERROR] PID 29201: test/extended/networking-minimal.sh:6: `NETWORKING_E2E_MINIMAL=1 "${OS_ROOT}/test/extended/networking.sh"` exited with status 1.
... skipping 7 lines ...
########## FINISHED STAGE: FAILURE: RUN EXTENDED TESTS [00h 08m 10s] ##########

[sosiouxme]

Got it twice today. https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18654/test_pull_request_origin_extended_networking_minimal/14578/ is the latest

[gabemontero]

See https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18703/test_pull_request_origin_extended_networking_minimal/14586/

Perhaps related to the networking flakes @smarterclayton mentioned on the mailing list.

[danwinship]

(The recent burst of failures was caused by the CA serial number bug that was fixed by #18713)

[danwinship]

Actually, the recent (and still continuing) burst of failures is caused by an actual docker-in-docker setup bug revealed by the new CA-serial-number-handling code (#18765), not related to #18713.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close