[Foundry] Investigate HTS "embedding" in Foundry

[acuarica]

HTS emulation as currently implemented posses two main issues

1. Fully re-implement HTS behavior. HTS contains many validations and customizations, which can be difficult to get right in Solidity.
2. Maintenance. Even if we manage to fully replicate HTS behavior (and that’s a big IF) in EVM bytecode, we have now to essentially maintain two implementation of HTS: The one in services and the one implemented here. This can be error prone and cumbersome to maintain.

An alternative to HTS "emulation" could be HTS "embedding”. The idea is to somehow, embed hedera-services behavior into Foundry or Hardhat. Given this approach would be tailored to each dev environment, we will probably end up writing one “adapter” for each dev env we want to provide forking support for.

Possible solution for Foundry

Foundry allows users to call an arbitrary executable through their ffi cheatcode. We could, in principle, build a custom hedera-services executable that only contains HTS System Contract related behavior. This can be an application that takes the calldata in the command line and returns the appropriate value.

There are some issues that we need to solve in order to implement this approach

Tasks

Preview Give feedback

1. 1. How to fetch remote state into the HTS process.
2. 2. Ensure HTS state is isolated from the rest of the (Foundry's) EVM state.
3. 3. How to maintain HTS state between calls. Foundry does not have a hook to kill a process when the tests finish, meaning we cannot have a running service. Which implies that each call to HTS would invoke a new instance of the executable. Keep in mind that tests don't share state, meaning each call to HTS could use different states.
4. 4. Create a native executable embedding the JVM and HTS from hedera-services. This should be as lightweight as possible, given it would be invoked on each HTS call.
5. 5. Ensure that HTS behavior related classes do not depend on Besu EVM. Besu shouldn't be needed because HTS does not run EVM bytecode, only Java code.

PoC

Tasks

Preview Give feedback

1. Investigate how the Mirror Node simulates eth_call requests. This might give us an idea how to embed and extract hedera-services for our purposes.

Related Issues & Discussions

• Implement non standard json rpc methods for dev mode hedera-local-node#252

Related Work

Just for reference, Arbitrum also uses “precompiles” or System Contracts to provide additional functionality https://docs.arbitrum.io/build-decentralized-apps/precompiles/reference. And it also suffers from the same problem as us regarding forking https://docs.arbitrum.io/for-devs/troubleshooting-building#why-does-my-transaction-revert-with-invalidfeopcode-when-using-foundry

• Why does my transaction revert with InvalidFEOpcode when using Foundry?

  Foundry and other similar development tools that enable chain forking, do not support Arbitrum precompiles. If your transaction is calling a precompile, it is likely that it will revert with InvalidFEOpcode.

  To rule out that possibility, it is recommended to send the transaction with a different tool.

Foundry

Arbitrum users have similar issues regarding forking, for example foundry-rs/foundry#5085.

It seems zskSync opts for custom Foundry support foundry-rs/foundry#7624. We may want to implement a solution similar to https://github.com/matter-labs/foundry-zksync.

There is an ongoing effort in Foundry to support precompiles (in the context of EVM-compatible networks with additional functionality)

• meta(compatibility): handle EVM Semantic differences across chains foundry-rs/foundry#748
• meta(extensions): tracking issue for extensions / plugins foundry-rs/foundry#8266
• Feature: Custom Precompiles for Anvil  foundry-rs/foundry#1505
• Support injecting custom precompiles to anvil foundry-rs/foundry#7498
• meta(precompiles): tracking issue for non-Ethereum precompiles foundry-rs/foundry#7262

However, it does not look like allows external code to injected yet foundry-rs/foundry#7589. Only available when using Anvil as a library.

Hardhat

Similar discussions to support EVM-compatible networks with additional semantics also happen in Hardhat NomicFoundation/hardhat#1902.

[arianejasuwienas]

Analysis (wip)

Storage dependencies are marked in orange in the attached diagrams.

---

1. How to fetch remote state into the HTS process.

1. Besu configurations: Besu-related configurations and presets can likely be left mocked somehow since no actual emulation involving it will be executed.
2. Smart Contract Services memory: Smart Contract Services memory should not require additional presets for this process.
3. Token-related storages: These need to be preloaded. Two potential entry points for initialization include:
   • During memory warming: Preloading could be handled during the memory warming phase.
   • NativeQueryHandlers replacement: Replace the NativeQueryHandlers implementations with custom alternatives to provide different results during the first run.

---

2. Ensure HTS state is isolated from the rest of the EVM state.

The EVM-related storage slots should not be necessary for running this script since no interaction with actual smart contracts occurs at any stage. Looks like the Proccessor is not using EVM emulation then.

---

3. How to maintain HTS state between calls. Foundry does not have a hook to kill a process when the tests finish, meaning we cannot have a running service. Which implies that each call to HTS would invoke a new instance of the executable. Keep in mind that tests don't share state, meaning each call to HTS could use different states.

• The updater executes on every transaction, and during commits, the network state is dumped into persistent storage (database or files, depending on the setup).
• This is why the consensus node can be restarted during tests without negative impacts, as long as the overall architecture remains intact.
• If the database approach currently requires an extra process, we should consider replacing it with a file-based structure to streamline operations.

---

5. Ensure that HTS behavior related classes do not depend on Besu EVM. Besu shouldn't be needed because HTS does not run EVM bytecode, only Java code.

Since we are extending Besu classes (in the code crucial for us) we can not drop it entirely. For precompile-specific operations (e.g., those directed to 0x167), Besu dependency may still be irrelevant, as long as we make sure no other calls will be routed to the service described in point 4.

[acuarica]

2. Ensure HTS state is isolated from the rest of the (Foundry's) EVM state.

(I amended the description to clarify what I meant)

For already existing tokens, I'm guessing that would be the case (HTS state is self-contained in the Java realm). However, the case for createFungibleToken and family methods would be different. These methods, besides changing HTS state, also should change Foundry's EVM state: at least by signaling that there is code in the resulting token address (we can deploy HIP719 if needed). The main takeaway I guess is that we need to communicate back the token address to Foundry's EVM.

[arianejasuwienas]

Conclusions from Tuesday's meeting (November 21, 2024) with @lukelee-sl and @acuarica:

• Even though the state is stored in the VMT and flushed periodically, it remains separate from the logic of different services. This separation should make it feasible to feed the VMT with data independently.
• Extracting only the app, HTS, and contracts services (along with their dependencies, like the libs jar) appears possible! However, we need to consider whether this is something we truly want to pursue.
• The Java application start time might be significant, so it’s worth investigating if this could pose any issues for us and if there is really anything we can do about it.
• Running the actual HTS code for local tests—might be beneficial in situations where we aren’t forking at all. And it actually would be good point for PoC since it wouldn’t require us to preload the data from the remote data source.
• We can check how the eth_call requests simulation is being approached in the mirror node project.

[arianejasuwienas]

The MirrorNode team implemented a solution to process calls and estimate gas in a way that avoids changing the state of the hashgraph. They used 'temporary' states within the scope of individual transactions to simulate the call behaviour without altering the actual state of the network. It could be useful for us if we choose to keep the node running throughout the entire testing process instead of shutting it down. However, this approach doesn't fully address our primary challenge—feeding the local state with data from the forked network. Unlike our situation, the MirrorNode operates on a state that is already fully synced with the network it pulls data from.

https://github.com/hashgraph/hedera-mirror-node/blob/0253168285275abdd89c4b6b60c45953a6249b2e/hedera-mirror-web3/src/main/java/com/hedera/mirror/web3/evm/contracts/execution/MirrorEvmTxProcessorImpl.java#L54

[acuarica]

It could be useful for us if we choose to keep the node running throughout the entire testing process instead of shutting it down.

But this scenario would require some sort of evm_revert/evm_snapshot, right?

However, this approach doesn't fully address our primary challenge—feeding the local state with data from the forked network.

As you mentioned, if we could replace the 'temporary' states with the state needed from the remote network, that would be a win, right?

[acuarica]

Just for future reference, in case we actually need it, we can evaluate building native images of services

• https://www.graalvm.org/latest/reference-manual/native-image/
• https://teavm.org/

[arianejasuwienas]

Investigation Progress:

Submitting a transaction through the local node:

• The Json RPC Relay is not a bottleneck in this process, even though it performs some validation checks on its own.
• The issue lies with the consensus node, which requires the transaction to be signed. While this isn't inherently a problem, it poses a challenge for us because we can't access the signature at the level of Solidity code execution via Foundry (and for SC to SC calls it does not even exist).
• Additionally, gas settings are required to run state-changing calls, which adds complexity.
• I can trigger deployment actions and create a tailored Hedera Node image specifically for this Proof of Concept, so I'll be able to test the whole infrastructure without the validation of the signature. I'm preparing the codebase for these tests.

Reverting transactions / rolling back to a specific point in time on our node:

• This also presents challenges. Certain stored information (in key-value format) already reflects the accumulated state over time. As a result, removing the most recent transaction is not sufficient since balances and other data are updated dynamically during execution.

[arianejasuwienas]

Current progress update

Added functionality to:

1. Read the state of the running local node into the smart contract (SC) at the Foundry level using FFI.
2. Modify the state of the Hashgraph by adding new transactions to the Hedera local node as the operator via the JavaScript SDK.
3. An additional JavaScript script determines which method was called, decodes the input bytes, makes the SDK request and prepares the response in the format of the eth call.
4. As a Proof of Concept (PoC), the current implementation supports creating new tokens and performing fungible token transfers between accounts (transfers from account A to account B). These transfers are signed by the operator, which required temporarily disabling approval checks on the HTS side.
5. Dynamic Account Creation
   When the token recipient is not an active account on the Hedera side, the script creates the account dynamically before transferring the token.

Current Challenges:

1. Gas Cost Handling:
   • Gas costs are fully covered by the operator on the HTS side, meaning normal transaction signer costs are bypassed.
   • An alternative approach (removing signature checks) was previously explored but would require significant changes to HTS due to the absence of a from field in transactions.
2. Account Balance Discrepancies:
   • Accounts created via the SDK must have a predefined balance on the Hedera side, which differs from the Foundry side. This leads to balance mismatches at the outset.
3. Transaction Reverting and Snapshots:
   • Properly handling transaction reverts and creating snapshots of the Hedera local node state will require:
   • Adding new methods to the gRPC API.
   • Potentially adopting a completely new approach to state management.
4. Local Node Lifecycle Management:
   • The current solution relies on the Hedera local node running continuously in the background, which is impractical.
   • Potential solutions:
   • Option 1: Start the local node before tests and stop it afterward. This is easier but not fail-safe (e.g., tests being interrupted unexpectedly).
   • Option 2: Start a new local node instance for each operation and terminate it once the result is returned. While more robust, this approach would require:
     • Extracting only HTS functionality from the Hedera architecture.
     • Optimizing startup times, which may not be feasible given the Java-based implementation.
5. Remote State Fetching:
   • To support forking, remote state fetching is a crucial feature for the solution's usability.