-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure enclave code can be reproducibly built #241
Comments
This is the result of building the same artifact twice on the same machine, and diffing the output using |
Possibly relevant: google/asylo@9c557c2 cc @deeglaze |
Linkstamping is indeed a cause for different bits for identity build inputs, which is why we disabled it for reliable trust-building on attestable bits. You will have to use the same compiler and compiler flags to get the same bits, so please consistently use the distributed Asylo toolchain. If you have different toolchains straddling internal (e.g., corp build system) and external (github.com/google/asylo/tree/master/asylo/distrib/toolchain), then you're going to have a tough time getting the same bits out. |
Thanks @deeglaze ! We are indeed using the Asylo toolchain, in fact running everything in the Asylo Docker image, but getting different binaries even when rebuilding twice in a row on the same machine and with the same Docker image. |
FWIW I am now specifying the |
Disable stamping in order to attempt to get reproducibility. Note that after this change, builds are still not reproducible, so there must be some other source of discrepancy. Ref. project-oak#241
That is not good indeed! I will take this as an issue with Asylo and see what we can do about it. Do you Considering how Bazel caches its build artifacts, I'm not confident that we'd have a simple way to create a regression test for this. build, sha256, clean, build, sha256, compare, perhaps? A few experiments I just ran with some of our test enclaves haven't discovered discrepancies, so it might just be specific to y'all's code using |
Disable stamping in order to attempt to get reproducibility. Note that after this change, builds are still not reproducible, so there must be some other source of discrepancy. Ref. project-oak#241
Indeed that's exactly what I'm doing in tiziano88@b621c57 to check for this behaviour. Is that enough information for you to reproduce the issue? Is there anything I can try to narrow down the issue? e.g. should I specify |
Disable stamping in order to attempt to get reproducibility. Note that after this change, builds are still not reproducible, so there must be some other source of discrepancy. Ref. project-oak#241
Disable stamping in order to attempt to get reproducibility. Note that after this change, builds are still not reproducible, so there must be some other source of discrepancy. Ref. project-oak#241
stamp = 0 is for linking the final binary together, so there shouldn't be a need add linkstamping to dependency libraries. A way I'd probably go about narrowing down what could be a problematic dependency to try to build different libraries into a do-nothing enclave and see which libraries could be introducing the non-determinism. |
Thanks, I'll try that, though that worries me as I guess it means that any dependency may reintroduce non-determinism at any point in the future? |
If your dependencies don't have bitwise reproducible builds due to their methods of compilation, then you'll be hard-pressed to get reproducible builds when you depend on them. For now, instead of comparing checksums of the entire ELF files, may I suggest that you focus your tests on the measured bits of the ELF file as exposed in the sgx_generate_sigstruct rule? You'll get full machine paths in debug information, so you're best off comparing your -c opt builds' sigstruct material for cross-machine reproducibility. |
Whatever you find is the problem, please let us know and we'll start compiling a list of debug steps for Asylo users to try when building enclaves for attestable bits. |
Disable stamping in order to attempt to get reproducibility. Note that after this change, builds are still not reproducible, so there must be some other source of discrepancy. Ref. project-oak#241
Disable stamping in order to attempt to get reproducibility. Note that after this change, builds are still not reproducible, so there must be some other source of discrepancy. Ref. #241
Status update: so far I tried removing dependencies on external source (mainly |
I have now stripped it down to essentially an empty EnclaveApplication, and still not getting reproducibility :/ |
I am still comparing binary hashes though. @deeglaze is this not expected to be reproducible at all? Will try with sigstruct now anyways. |
@deeglaze to use the
|
- fix a couple missing dependencies - add rules to build sigstruct, used for SGX remote attestation Ref project-oak#241
- fix a couple missing dependencies - add rules to build sigstruct, used for SGX remote attestation Ref project-oak#241
- fix a couple missing dependencies - add rules to build sigstruct, used for SGX remote attestation Ref #241
The issue with the new version of Asylo was solved by bumping version of absl too, and it seems the root cause of the reproducibility issue is google/asylo#41. I will try again once that is fixed and update this issue. |
Add an additional pipeline running on Google Cloud Build, which builds the enclave binary and pushes it to Google Cloud Storage. The intention is for this to be used to build reference binary artifacts to be deployed on Borg in Google if necessary, once we get full reproducible builds (#241). Example run: https://pantheon.corp.google.com/cloud-build/builds/700d7aea-6799-4822-afa1-ec3758e88faf?project=oak-ci
Add an additional pipeline running on Google Cloud Build, which builds the enclave binary and pushes it to Google Cloud Storage. The intention is for this to be used to build reference binary artifacts to be deployed on Borg in Google if necessary, once we get full reproducible builds (#241). Always use Bazel cache for builds and for tests. Example run: https://pantheon.corp.google.com/cloud-build/builds/700d7aea-6799-4822-afa1-ec3758e88faf?project=oak-ci
Add an additional pipeline running on Google Cloud Build, which builds the enclave binary and pushes it to Google Cloud Storage. The intention is for this to be used to build reference binary artifacts to be deployed on Borg in Google if necessary, once we get full reproducible builds (#241). Always use Bazel cache for builds and for tests. Example run: https://pantheon.corp.google.com/cloud-build/builds/700d7aea-6799-4822-afa1-ec3758e88faf?project=oak-ci
Add an additional pipeline running on Google Cloud Build, which builds the enclave binary and pushes it to Google Cloud Storage. The intention is for this to be used to build reference binary artifacts to be deployed on Borg in Google if necessary, once we get full reproducible builds (#241). Always use Bazel cache for builds and for tests. Example run: https://pantheon.corp.google.com/cloud-build/builds/700d7aea-6799-4822-afa1-ec3758e88faf?project=oak-ci
Add an additional pipeline running on Google Cloud Build, which builds the enclave binary and pushes it to Google Cloud Storage. The intention is for this to be used to build reference binary artifacts to be deployed on Borg in Google if necessary, once we get full reproducible builds (#241). Always use Bazel cache for builds and for tests. Example run: https://pantheon.corp.google.com/cloud-build/builds/700d7aea-6799-4822-afa1-ec3758e88faf?project=oak-ci
This finally allows building enclave code reproducibly (project-oak#241). For the record, I got the following output from `scripts/check_enclave_reproducibility`: ``` 5d3b12032df2c95728f2ecf0f646abdc67c46f2f ./diff/oak_enclave_unsigned_e66a750eaafcaec35b50ba1df0298f342a8aa8f1_2019-10-14T20:05:25+00:00.so 5d3b12032df2c95728f2ecf0f646abdc67c46f2f ./diff/oak_enclave_unsigned_e66a750eaafcaec35b50ba1df0298f342a8aa8f1_2019-10-14T20:10:46+00:00.so ```
This finally allows building enclave code reproducibly (#241). For the record, I got the following output from `scripts/check_enclave_reproducibility`: ``` 5d3b12032df2c95728f2ecf0f646abdc67c46f2f ./diff/oak_enclave_unsigned_e66a750eaafcaec35b50ba1df0298f342a8aa8f1_2019-10-14T20:05:25+00:00.so 5d3b12032df2c95728f2ecf0f646abdc67c46f2f ./diff/oak_enclave_unsigned_e66a750eaafcaec35b50ba1df0298f342a8aa8f1_2019-10-14T20:10:46+00:00.so ```
We should have a way of testing that building the enclave code (trusted application) on the same machine at different times, and also on different machines, results in the same (bitwise) artifact.
The text was updated successfully, but these errors were encountered: