Binary schema data is not reproducible.

[pmeiyu]

Hello,

I am building rime-data on Guix. Guix is a functional package manager. It requires the binary output of any package to be reproducible. But it seems that the binary schema data files produced by rime_deployer are not reproducible, thus the binary output of rime-data is not reproducible.

Developers of librime, can you confirm that the binary schema files produced by rime_deployer are not reproducible? Is this by design?

[nameoverflow]

What does deterministic exactly mean?

[pmeiyu]

@nameoverflow Deterministic means Reproducible Builds.

A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.

[lotem]

@nameoverflow
I think they meant that the data files are not the same across rebuild, not the rime_deployer binary.

[pmeiyu]

@nameoverflow
I think they meant that the data files are not the same across rebuild, not the rime_deployer binary.

Yes. The binary schema data files produced by rime_deployer are not reproducible. Sorry I didn't make myself clear.

[lotem]

Given the same set of input files, the build output are the same on the same machine.
Here's how I tested it:

make -f Makefile.xcode thirdparty
make -f Makefile.xcode
cd xbuild/bin
LD_LIBRARY_PATH=../lib/Release Release/rime_deployer --build
mv build build.1
LD_LIBRARY_PATH=../lib/Release Release/rime_deployer --build
mv build build.2
for x in build.1/*; do diff $x ${x/build.1/build.2} || (echo "diff found"; break); done

If the issue is in reproducibly fetching the set of data files with rime/plum, then librime is not the responsible part.

We've tried to make binary data files work for different CPU architectures. But it turns out data files built on different machines are compatible but not strictly identical.
#121 (comment)

[pmeiyu]

We've tried to make binary data files work for different CPU architectures.

That's not necessary.

Could it be caused by YAML schema file's filesystem timestamps? Does the binary schema file contain timestamp information?

[pmeiyu]

I tested on my computer. Touch all the YAML files and rebuild. The result is different. So this is indeed caused by YAML files' timestamp.

[lotem]

I'm aware of the issue with timestamps, yet decide to leave it there a little longer.
Timestamps provide a quick way to detect user modifications to any YAML config files, this is important to ensure good performance as there is a quick check on every startup.

But the situation may change in the future. I plan to make the data deployment process fully separate from the IME, run on user demand.
Then, in offline deployment, we can afford less time critical checksum comparison with input files.

[eagleoflqj]

Fixed at #720