Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Bootstrappable FCMP++ #9440

Draft
wants to merge 129 commits into
base: master
Choose a base branch
from
Draft

Conversation

tobtoht
Copy link
Contributor

@tobtoht tobtoht commented Aug 17, 2024

This is a WIP draft PR for reproducible release builds with Rust FFI.

Based on #9436 (FCMP++ integration) and #8929 (Bootstrappable builds).

Tools

  • Binutils 2.41
  • LLVM 17.0.6
  • Rust 1.82.0

Sub PRs

To-do (incomplete):

  • Fix non-determinism on aarch64
  • Vendor Rust deps in release source archive
  • Test all builds
Target Builds Reproducible Bootstrappable Tested
x86_64-linux-gnu 🮱 🮱 🮱
arm-linux-gnueabihf 🮱 🮱 🮱
aarch64-linux-gnu 🮱 🮱 🮱
riscv64-linux-gnu 🮱 🮱 🮱
i686-linux-gnu 🮱 🮱 🮱
i686-w64-mingw32 🮱 🮱 🮱
x86_64-w64-mingw32 🮱 🮱 🮱
x86_64-unknown-freebsd 🮱 🮱 🮱 [1]
x86_64-apple-darwin 🮱 🮱 🮱 [2]
aarch64-apple-darwin 🮱 🮱 🮱 [2]
arm-linux-androideabi 🮱 🮱 ❌ [3]
aarch64-linux-android 🮱 🮱 ❌ [3]

[1] = links against precompiled freebsd system libraries
[2] = links against precompiled libraries from the darwin SDK
[3] = built with a precompiled toolchain from the Android NDK (there is currently no way around this)

Guix

Changes to the build environment

Includes packages that are available in the guix container that we use to build Monero.

package old new
llvm n/a 17.0.6
rust n/a 1.82.0

Changes to transitive build dependencies

Includes (transitive) build dependencies that are not included in the build environment.

package old new
gdb n/a 12.1
libomp 11.1.0 17.0.6
procps n/a 4.0.3

@tobtoht tobtoht mentioned this pull request Aug 17, 2024
@kayabaNerve
Copy link

Rust 1.69 is being used within Monero for its LLVM/ld compatibility. There's no need to go newer and that may be simpler.

@tobtoht
Copy link
Contributor Author

tobtoht commented Aug 17, 2024

In this case, newer is easier. Guix only supports the latest version of Rust (that it has available).

This PR currently uses binutils 2.38, llvm 17.0.6. Will see if I run into problems with windows targets.

@MrCyjaneK
Copy link

MrCyjaneK commented Aug 21, 2024

Hey - I've seen that NDK task is unchecked, you can grab latest NDK patch from here: https://github.com/MrCyjaneK/monero_c/blob/master/patches/monero/0013-android-ndk-version-bump.patch and https://github.com/MrCyjaneK/monero_c/blob/master/patches/monero/0013-rename-arm-armv7a.patch it cointains some hacks - but hopefully will be useful.

- If locked output migration step completes, then program exits
while migration step to grow the tree is in progress, make sure
the migration picks back up where it left off growing the tree.
- Make sure db cursor gets set in all cases when renaming
block infn table.
- Removing the sign bit from key images enables an optimization for
fcmp's.
- If an fcmp includes a key image with sign bit cleared,while the
same key image with sign bit set exists in the chain already via a
ring signature, then the fcmp would be a double spend attempt and
the daemon must be able to detect and reject it.
- In order for the daemon to detect such double spends, upon
booting the daemon, we clear the sign bit from all key images
already in the db. We also make sure that all key images held in
memory by the pool have sign bit cleared as well.
- Key images with sign bit cleared are a new type:
`crypto::key_image_y`. The sign bit can be cleared via
`crypto::key_image_to_y`.
- The `_y` denotes that the encoded point is now the point's y
coordinate.
- In order to maintain backwards compatibility with current RPC
consumers, the daemon keeps track of which key images have sign
bit cleared and not, so that upon serving
`spent_key_image_info::id_hash`, the daemon can re-construct the
original key image and serve it to clients.
Speeds up inverting many elems at once 95%+
- Moved functions around in unit_tests/curve_trees.{h,cpp} to ease
using the in-memory Global tree across tests
- Introduced PathV1 struct, which is a path in the tree containing
whole chunks at each layer
- Implemented functions to get_path_at_leaf_idx and get_tree_root
on in-memory Global tree
- Cleanly separate logic to set the hash_offset that we use when
calling hash_trim and hash_grow from the logic used to determine
which old child values we need from the tree
- The core logic error was not properly setting the range of
children needed from the tree when
need_last_chunk_remaining_children is true. The fix makes sure
to use the correct range, and to set hash_offset appropriately
for eveery case.
- In the case that get_next_layer_reduction doesn't actually
need to do any hashing, only tell the caller to trim to boundary,
the function now short-circuits and doesn't continue with hashing
- batch_start is the simplest function to use to resize db, since
resizing requires no active txns.
- batch_stop makes sure no active txns.
- need to decrement txns before calling migrate() so that do_resize
does not deadlock in wait_no_active_txns
@tobtoht tobtoht force-pushed the guix-rust branch 9 times, most recently from d0fe0ba to e89ffb1 Compare December 14, 2024 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants