OCI manifest created timestamp wrong with v2.2.0

[baal-lgln]

Description

We are using a cosign container inside our pipeline to sign newly build images.
Up until v2.1.1 (public.ecr.aws/bitnami/cosign:2.1.1) this works without problems:
cosign sign --yes --key cosign.key {{image}}@sha256:{{digest}}
That creates a new OCI entry inside our IBM Container Registry:

{
  "Id": "sha256:[...]",
  "Parent": "",
  "Comment": "",
  "Created": "2023-10-12T14:52:17.989015938Z",
[...]
  "Size": 251,
  "VirtualSize": 251,
  "RootFS": {
    "Type": "layers",
    "Layers": [
      "sha256:[...]"
    ]
  },
  "ManifestType": "application/vnd.oci.image.manifest.v1+json"
}

After updating to v2.2.0 (public.ecr.aws/bitnami/cosign:2.2.0) the Created timestamp is totally wrong and leads to this:

{
  "Id": "sha256:[...]",
  "Parent": "",
  "Comment": "",
  "Created": "0001-01-01T00:00:00Z",
 [...]
  "Size": 251,
  "VirtualSize": 251,
  "RootFS": {
    "Type": "layers",
    "Layers": [
      "sha256:[...]"
    ]
  },
  "ManifestType": "application/vnd.oci.image.manifest.v1+json"
}

As we have some rules to delete older entries inside our registry these signatures are deleted soon after creation.
Image can be verified with cosign verify in both cases. The output does not differ.
This can be reproduced with different images.

Version

public.ecr.aws/bitnami/cosign:2.2.0

[ztcjoe93]

I encountered this issue some time back and dug into it for a bit -- this is part of a commit that reverts a previous implementation to ensure that the produced artifacts are deterministic, which is pushed out in 2.2.0 as part of the milestones.

I believe container registries that uses timestamp as a housekeeping rule (i.e GitLab, GCR) will be affected since effectively the signatures are deleted after creation. No clue on any alternative workarounds, but personally I pinned the version to 2.1.1 for the time being to mitigate this.

[haydentherapper]

@jonjohnsonjr did you want to chime in on this?
To my knowledge, this is done so that container builds are reproducible.

[avasile8709]

Hello,

I have the same issue here, using GitLab pipeline.

Do you have any new information about this issue?

[uwevanheesch]

We encounter the same error in Gitlab.

[Lerentis]

kind of a bummer to write here again for the same reason :/
this effectively blocks anyone who is using housekeeping jobs to update to any release after 2.1.
imo a very big sacrifice for a gain that is rather theoretical. if there is no workaround available i would suggest to reconsider this decision @jonjohnsonjr

[dheeg]

@cpanato

We encountered the same issue back in 2022. Then #2108 was merged in order to fix the issue for GitLab users with automated registry cleanup rules.

Now #3121 was merged which reverts the fix and is itself flagged as "fix".

• I only see one issue (Artifacts are non-deterministic #3120) which was written by the same person which also created the PR. Did customers/users ask for this "fix"? To me it looks like more users are asking for the opposite.
• Was there a strategic decision made? There are clearly pros and cons for both solutions and the merged PR doesn't contain any substantial discussion, more an single opinion.

[baal-lgln]

v2.3.0 was just released (public.ecr.aws/bitnami/cosign:2.3.0-debian-12-r0) and sadly nothing has changed. This is still a huge problem for us.

[bobcallaway]

v2.3.0 was just released (public.ecr.aws/bitnami/cosign:2.3.0-debian-12-r0) and sadly nothing has changed. This is still a huge problem for us.

Did you use the new command line argument?

--honor-create-timestamp

[baal-lgln]

--honor-create-timestamp does not exist for cosign sign, seems to be the --record-creation-timestamp flag. With cosign sign --record-creation-timestamp=true it finally works 👍

    --record-creation-timestamp=false:
	set the createdAt timestamp in the signature artifact to the time it was created; by default, cosign sets this
	to the zero value

Would be great to have this kind of change added to the changelog (at least I couldn't find it).