Cannot build enclave reproducibly

[tiziano88]

I cannot seem to reproducibly build the enclave code on my machine. I am using the following steps:

1. docker run -it --rm -v bazel-cache:/root/.cache/bazel -v "${PWD}":/opt/my-project -w /opt/my-project gcr.io/asylo-framework/asylo
   • [execute the following steps from within the running Docker container]
2. bazel build --config=sgx-sim //asylo/examples/hello_world
3. cp bazel-bin/asylo/examples/hello_world/hello_enclave_unsigned.so enclave_$(date --iso-8601=seconds).so
4. bazel clean
5. bazel build --config=sgx-sim //asylo/examples/hello_world
6. cp bazel-bin/asylo/examples/hello_world/hello_enclave_unsigned.so enclave_$(date --iso-8601=seconds).so
7. sha1sum enclave_*

Note that the resulting hashes differ, e.g. on my machine I get the following:

b88b0156ebc354ef15cf706fdf4428e998b68356  enclave_2019-10-01T18:51:50+00:00.so
e48909687b68f586e1c3b4facf66cad2b4e51f1a  enclave_2019-10-01T18:55:54+00:00.so

ref project-oak/oak#241
cc @anghelcovici @deeglaze @KeithMoyer @annasapek

[tiziano88]

FWIW I analyzed the difference between the two resulting binaries with diffoscope, and it seems most (all?) of the differences are around functions such as:

• FromkLinuxSysconfConstant
• TokLinuxSysconfConstant
• FromkLinuxBaseSignalNumber
• FromkLinuxIpV6OptionName
• TokLinuxErrorNumber

From a quick look at the repository, these seems to be defined in https://github.com/google/asylo/blob/5ab253e2db786a9cccbc397aac2338531d4aa119/asylo/platform/system_call/type_conversions/manual_types_functions.cc .

[KeithMoyer]

@tiziano88, thanks for the good bug report with simple reproducer! I've root caused this and we'll be working on a fix (it should be fairly simple).

We have some code generation for translating between constant values when marshaling across the enclave boundary, and the python dictionary used at one stage of this is itself non-deterministic in ordering. This results in the order these functions are generated to be different, which breaks reproducability. Sorting the dictionary keys (and thus the outputted function order) should fix the issue you're seeing.

[tiziano88]

Thanks for the fix @carmenyh , I have tried pulling in the latest version of Asylo including your fix, but I am still seeing inconsistent results. The difference seem still related to the ordering of symbols such as:

• TokLinuxRusageTarget
• FromkLinuxTcpOptionName

I suggest reopening this issue until the root cause is identified.

[KeithMoyer]

@tiziano88, this was automatically closed via commit description annotation prematurely. Reopening