RHCOS 4.11 build failures across all arches

[miabbott]

On Friday March 18 around 5PM EST, all the RHCOS 4.11 builds started to fail.

The Jenkins jobs have since been reaped, so there are no logs from the original failures around that time.

The most recent failures show the following:

Input state hash: 5dbb470fdfe40cef875b7f5a83b206c3163bc780d1715a6a918c27327e8c13a5
No apparent changes since previous commit; use --force-nocache to override
commit: 776c95fb7057a6de62ecbab1a9b727dae4e2ecebe63d11691574edbd2d570e86 image: f223d9ed30f3e137e8d1d2c768405b0f0c2105dc7ff9996734918deb7586b2a7
Commit 776c95fb7057a6de62ecbab1a9b727dae4e2ecebe63d11691574edbd2d570e86 unchanged; reusing previous build's rpm-ostree metadata
ostree diff commit from: 776c95fb7057a6de62ecbab1a9b727dae4e2ecebe63d11691574edbd2d570e86
ostree diff commit to:   776c95fb7057a6de62ecbab1a9b727dae4e2ecebe63d11691574edbd2d570e86

New image input checksum: f223d9ed30f3e137e8d1d2c768405b0f0c2105dc7ff9996734918deb7586b2a7
None
New build ID: 411.85.202203181611-0-1
�[0m�[31merror: �[0mBuilding oci: Writing ostree root to blob: Exporting commit: No such metadata object 50773817e4519629fb061cb3cfe4ddae0a996c12336d087042481fbeab1a380c.dirmeta

Since it is happening on all arches, this may be a coreos-assembler issue. The most recent failures are using coreos/coreos-assembler@abcbd34

But that commit shouldn't affect building the ostree commit.

Previous known good coreos-assembler commit is coreos/coreos-assembler@3ab9a24

$ git shortlog 3ab9a24dd8e20b676db856a0e6dc300dc653a088..abcbd34b34012a3d91b6c024ebe21fd9dcce54b0
Benjamin Gilbert (3):
      cosalib/ova: stop templating SCSI and network adapters
      cosalib/ova: template VMware HW and OS versions from image.yaml
      cosalib/ova: enable UEFI in VMware OVA

Jonathan Lebon (1):
      mantle/network/ssh: use ExtendedAgent for SSHAgent

Nothing jumps out there, so perhaps a new version of ostree/rpm-ostree landed in the cosa container?

The shortlog of openshift/os from last good to current failure doesn't look suspicious.

$ git shortlog d92496d5b46bd121b78d5ed62b2152e3fd4ed3f2..fc75bde9780e7d17c36b0821082093eda7c9802f
Benjamin Gilbert (1):
      image: specify VMware OS and HW versions

OpenShift Merge Robot (1):
      Merge pull request #748 from bgilbert/vmware

[jlebon]

Are you sure this isn't just good ol' coreos/coreos-assembler#972? The latest openshift/os commit is #748 which only changed image.yaml.

[miabbott]

Good eyes, Jonathan. The error looked different enough that it didn't trigger anything in my memory. (And honestly, I forgot that coreos/coreos-assembler#972 was still an issue...)