Do not upload snap if its contents didn't change

[abitrolly]

A different approach than #36 to avoid refresh of snap store on each commit to repository.

1. Build reproducible snap on every commit
2. Check if the built shap package is already in the store

[abitrolly]

Building reproducible snap

Check what prevents repeatability. Create v2.snap and v3.snap.

cd snapcrafting/yakshaveinc
snapcraft_fedora.sh
cp yakshaveinc_eternal_amd64.snap v2.snap
snapcraft_fedora.sh
cp yakshaveinc_eternal_amd64.snap v3.snap

Use https://diffoscope.org/ to inspect differences.

time podman run --rm -t -w $(pwd) -v $(pwd):$(pwd):Z,ro \
      registry.salsa.debian.org/reproducible-builds/diffoscope v2.snap v3.snap --text-color=always

[abitrolly]

--- v2.snap
+++ v3.snap
├── unsquashfs -s {}
│ @@ -1,9 +1,9 @@
│  Found a valid SQUASHFS 4:0 superblock
│ -Creation or last append time Sat Aug  1 18:36:27 2020
│ +Creation or last append time Sat Aug  1 20:03:23 2020
│  Filesystem size 576 bytes (0.56 Kbytes / 0.00 Mbytes)
│  Compression xz
│  Block size 131072
│  Filesystem is exportable via NFS
│  Inodes are compressed
│  Data is compressed
│  Uids/Gids (Id table) are compressed
├── unsquashfs -d  -lls {}
│ @@ -1,6 +1,6 @@
│  Parallel unsquashfs: Using 4 processors
│  1 inodes (1 blocks) to write
│
│  drwxr-xr-x root/root                27 2020-05-26 13:34
│  drwxr-xr-x root/root                32 2020-05-26 13:34 /meta
│ --rw-r--r-- root/root               275 2020-08-01 18:36 /meta/snap.yaml
│ +-rw-r--r-- root/root               275 2020-08-01 20:03 /meta/snap.yaml
podman run --rm -t -w $(pwd) -v $(pwd):$(pwd):Z,ro  v2.snap v3.snap   219.04s user 42.56s system 46% cpu 9:18.56 total

Need a way to set timestamps in squashfs.

Using SOURCE_DATE_EPOCH=2461551217(year 2048) may help - https://reproducible-builds.org/docs/system-images/

[abitrolly]

snapcraft 4.1.1 doesn't understand SOURCE_DATE_EPOCH. This doesn't alter the timestamp to year 2048,

podman run -e SOURCE_DATE_EPOCH=2461551217 --rm -it -v /home/anatoli/a/linux/snapcrafting/yakshaveinc:/src:Z \
        -w /src yakshaveinc/snapcraft:core18 snapcraft

Repo search also reveals that snapcraft doesn't understand anything about this env var - https://github.com/snapcore/snapcraft/search?q=SOURCE_DATE_EPOCH&unscoped_q=SOURCE_DATE_EPOCH

[abitrolly]

Filled a bug to track support for SOURCE_DATE_EPOCH on snapcraft https://bugs.launchpad.net/snapcraft/+bug/1890046

In the meanwhile the solution is to patch existing .snap archive to overwrite timestamps. Tools that may do this.

• https://github.com/plougher/squashfs-tools - need to be compiled from C, might be already packaged
• https://github.com/AgentD/squashfs-tools-ng - fork of the above
• https://salsa.debian.org/reproducible-builds/strip-nondeterminism
• https://ide.kaitai.io/ - if I decide to parse snap SquashFS myself (see features)

[lamby]

Unfortunately, strip-nondeterminism is not the place to fix this issue. From a very very quick glance at the above, you should be able to achieve what you need via mainline & packaged mksquashfs (it would not "need to be compiled [or recompiled] from C").

[abitrolly]

I already started playing with https://ide.kaitai.io/ and so far I like it very much. )

[lamby]

Just my 0.02 BTC but manually hacking the contents of a SquashFS image does not feel like it is solving the problem at the right abstraction level.

[abitrolly]

On the other hand archive format that has only one right tool to modify, it doesn't look any better than proprietary tool or proprietart standard despite the presence of the source code. I don't think it is wrong to crush png's, remove extra layers from docker images or adjusting timestamps in an archive. Isn't it the Unix philosophy one tool - one function?

[abitrolly]

squashfs-tools plougher/squashfs-tools#90 (comment) are not going to help with modifying Squashfs to strip non-determinism.