RFC: Unset buildid for reproducible builds

[imjasonh]

If the same source is built on two different machines, ideally, they should produce the same binary, same image, same layer digests, same config diffids, etc.

This isn't the case today. If you take a recent ghcr.io/google/ko image produced by CI, and try to build the same image from the same commit, using the same Go compiler version, and using the same base image by digest (aided by #592), you still end up with different images.

$ git checkout f8d0aca1abeab6cf3d1800d15acc0592c7c976b7
$ ci=ghcr.io/google/ko@sha256:8fa68c86562684f946e92086a6a4d1f17ee116fff32e157e68f96dbf948b0783
# set base image to golang:1.17@digest in .ko.yaml, based on CI build logs
$ local=$(KO_DOCKER_REPO=gcr.io/imjasonh go run ./ build ./ --platform=all)
...
2022/02/11 11:01:22 Published gcr.io/imjasonh/ko-98b8c7facdad74510a7cae0cd368eb4e@sha256:73d4746ef0423a24b2e36705d5f648f8f6ed8ad2f70a74c43a1d799ea4135936

Note these images have different digests.

Only considering --platform=linux/amd64 for now, their manifests differ in the top layers produced by ko:

$ function manifest-platform-diff {
  diff <(crane manifest $1 --platform=linux/amd64 | jq) <(crane manifest $2 --platform=linux/amd64 | jq)
}
$ manifest-platform-diff $ci $local
7c7
<     "digest": "sha256:52fbe2d8d6f679aaf218c9696bd52cf46a0acf5734e55f477f767fcfc8171fac"
---
>     "digest": "sha256:431bf119280969354ea1b7f01d5057886cde0c5f415a9d6a702fb415b1f9e795"
52,53c52,53
<       "size": 14447415,
<       "digest": "sha256:103dce014357d1447a0ab5bdb2e0820ebe5b3825dc39ec094bec0833641c6f3c"
---
>       "size": 14451848,
>       "digest": "sha256:e73e6295963605d46fa599595e807f93d78035bbab37d5055c284d5fa597fe1b"

Configs diff only in the last diffID:

$ function config-diff {
  diff <(crane config $1 --platform=linux/amd64 | jq) <(crane config $2 --platform=linux/amd64 | jq)
}
$ config-diff $ci $local
91c91
<       "sha256:26b11d7436027c1a8b68d2f263694cb5d004cbaf82cf12fabd3e8bc57b83b675"
---
>       "sha256:a744dfc522c644d427686a3111398e19ce8ba62eb4d1091f7e4cf9773ab2dacc"

The Go binaries in each image:

$ function extract-file {
  crane export $1 - | tar -Oxf - $2      
}
$ extract-file $ci /ko-app/ko > cibin
$ extract-file $local /ko-app/ko > localbin

These are different sizes, and have different buildids embedded:

$ go tool buildid cibin
wfsXdWzess1HPZwqxpza/w1W8Gpkeb188ZJ6UexW7/fAQZlgOa-X8oaRDhGERu/OOojyWok9JTAbBpACeWS
$ go tool buildid localbin
fURyWN8_9H1KmClFNJrE/HWiELKLTvC1MFSvyxVmG/dsAR_RpmjFrxsRtRLtDJ/n5VsQTn6NFmq0lhuM8r4

We use buildid to avoid rebuilding/compressing/digesting binaries when KOCACHE is set, but I think maybe if KOCACHE isn't set we should consider also passing -ldflags=-buildid= as recommended here

Or, perhaps making it an opt-in choice like ko build ./ --reproducible?

[jonjohnsonjr]

Is the only diff the buildid?

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Keep fresh with the 'lifecycle/frozen' label.

[dprotaso]

I believe this was fixed in go1.14

golang/go#33772

[dprotaso]

It seems like it wasn't fixed.

Compiling two binaries on two different machines with -trimpath resulted in different checksums. Dropping the buildid gave me identical binaries.