machine-config operator does not initialize

[rimvydukas]

Describe the bug
I did three installations with the latest OKD release and on two of these, the machine-config operatore did not initialize:
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
machine-config False True True 72m Cluster not available for 4.9.0-0.okd-2022-02-12-140851

NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master False True True 3 0 0 3 90m

Version
4.9.0-0.okd-2022-02-12-140851
vSphere UPI

How reproducible
It seems that this is happening on every installation.

I need help with the advice - is it a bug and how can I fix it? I'm quite a newbie with OKD, so sorry if I've missed some required info.

[vrutkovs]

Looks similar to #963, could you attach a log bundle from failed installation?

[vrutkovs]

This one is also mysterious to us :( We'd need to debug on-bootstrap MCS and find out why it generates the wrong rendered- MC.

If your machine-config operator has "rendered-master-... not found" in oc describe co machine-config then its certainly #963 and its worth continuing there

[rimvydukas]

Yes, I'm almost sure, that my issue is the same as in your mentioned issue. I was able to fix this problem this time:) Deleted two mentioned machine configs, required rendered-master was created and every missing and failed part was updated. This didn'twork btw on my first ever OKD install. But everything went well on another two later ones :) One thing: I also had the same issue and with 4.9.0-0.okd-2021-11-28-035710.

[vrutkovs]

Sadly this seems to be a long-standing issue, first noticed in 4.8.

However it mostly affects UPI, I don't think I saw that in IPI ever

[fortinj66]

Just happened to me with IPI vSphere on both the Current stable and todays latest nightly:

Extension:
    Master:  pool is degraded because nodes fail with "3 nodes are reporting degraded status on sync": "Node poc-c1v4-79kq7-master-2 is reporting: \"machineconfig.machineconfiguration.openshift.io \\\"rendered-master-9d3c5a0d2e10266cd337b38ccc36ab68\\\" not found\", Node poc-c1v4-79kq7-master-0 is reporting: \"machineconfig.machineconfiguration.openshift.io \\\"rendered-master-9d3c5a0d2e10266cd337b38ccc36ab68\\\" not found\", Node poc-c1v4-79kq7-master-1 is reporting: \"machineconfig.machineconfiguration.openshift.io \\\"rendered-master-9d3c5a0d2e10266cd337b38ccc36ab68\\\" not found\""
    Worker:  all 2 nodes are at latest configuration rendered-worker-c4edd36eeca3de2f23b7c7289cd1c803

[fortinj66]

oc delete mc 99-master-okd-extensions 99-okd-master-disable-mitigations resolved the issue