Build status should show why it failed when the build pod runs out of ephemeral disk space

[davewalter]

Details

When the build pod errors due to a full ephemeral storage disk, the Build's Ready status condition is set to false without a useful reason or message that indicates why.

status:
  conditions:
  - lastTransitionTime: "2023-07-11T23:37:02Z"
    status: "False"
    type: Succeeded

Inspecting the build pod's status, the Ready status condition simply indicates that the pod has failed:

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-07-11T23:34:51Z"
    reason: PodFailed
    status: "False"
    type: Ready

The status.initContainerStatuses[{name: "build"}].state shows that the container has terminated with an exit code of 2 and a reason of Error:

status:
  initContainerStatuses:
  - name: build
    ...
    ready: false
    state:
      terminated:
        containerID: ...
        exitCode: 2
        finishedAt: "2023-07-11T23:37:01Z"
        reason: Error
        startedAt: "2023-07-11T23:34:56Z"

The only useful status from the build pod is the status.message field, which states "Pod ephemeral local storage usage exceeds the total limit of containers".

It would be helpful to be able to see the useful error message reflected in the Build (and Image) succeeded status conditions in the human-readable message field. It would also be nice to see the machine-readable reason field copied from the build pod's ready status condition into the succeeded conditions for both resources.

[tomkennedy513]

Hey @davewalter, what version of kpack are you using?

[davewalter]

Hi @tomkennedy513,

This testing was performed with v0.10.0. I see that v0.11.0 includes changes #1221 and #1226 to improve the status conditions on the Image resource based on the value from the corresponding Build resource. From what I can tell, it doesn't look like the build reconciler's conditionForPod helper function sets a reason/message in the Build's Succeeded condition (other than when the status is unknown).

I have taken a stab at updating the build reconciler to set suitable reasons/messages based on the status of the build pod. Please let me know if you have any questions or comments.

[pviraj59]

Resolved by #1258