Move account data to persistent storage

[sambley]

Problem

The cost of RAM is much higher that adds up to the cost of operating a full node (16GB of RAM is the same cost as 500GB of high speed NVMe SSDs). Look into ways to reduce the RAM usage by moving some of the data onto SSDs and have them loaded / stored on demand.

Summary of Changes

Implements #2769

To help reduce RAM usage of the nodes, persist storage of accounts across NVMe SSDs and load / store them on a need basis from SSDs.

Store account information across two files: Index and Data
Index: Contains offset into data
Data: Contains the length followed by the account data

The accounts are split across NVMe SSDs using the pubkey as the key.

TODOs:

• The account data is stored currently across 2 directories and need to be changed to use NVMe
• Look into performance bottleneck due to using the storage and look to see if the account could be partitioned across multiple directories to parallelize load and store operations.
• Error handling and remove files across runs

Snapshot and version numbering is not planned for this release.

Fixes # #2499

[aeyakovenko]

Rad!!! Can @sakridge and @garious take a look as well.

[aeyakovenko]

I think the thread count in the replay stage or the process transactions stage would need to be equal to or higher than the ‘q’ setting on the NVEs.

[aeyakovenko]

@sambley did you get a chance to test it on our 2 nve machine?

[garious]

@sambley, the text you have under "Problem" in the PR description doesn't describe a problem. It's a summary of a solution. What problem are you solving precisely? At what point does the RAM usage of accounts affect a metric? Or what metric will improve if we merge this PR?

[aeyakovenko]

@sambley, the text you have under "Problem" in the PR description doesn't describe a problem. It's a summary of a solution. What problem are you solving precisely? At what point does the RAM usage of accounts affect a metric? Or what metric will improve if we merge this PR?

@garious the problem is that the cost of ram is higher than ssds. 16gb of ram is the same cost as 500gb of high speed NVE. so about 30x improvement in cost per full node. Multiply that by 16k fullnodes for an ethereum sized network.

[garious]

@aeyakovenko, our cost per fullnode goal is 5k USD. What's the current cost? With a 30x improvement for this particular component, what does that drop the cost to?

[aeyakovenko]

@garious cost per allocated byte would be roughly 0.000121875 U.S. dollars / byte for 15,000 nodes at $130 per 16gb. But you can't really build systems with more than 128gb per system cheaply. Motherboards that support more ram are either more expensive, or are not as flexible for GPUs or other components. It is about $15k for a 1TB of ddr per system, and that board doesn't support GPUs. So the price per byte is likely to be higher since the maximum the entire account space can take up is going to be the smallest node that is in the supermajority (desired finality size).

[aeyakovenko]

@garious at 128gb per system, we end up with a maximum of account number of about 1b, and only if we can optimize the Account instance allocation to fit entirely into 128 bytes. It might be doable, but is going to be hard.

[sambley]

@anatoly, I tried it out on the 2 nve m/c today and am seeing the average tps to be twice as slow when number of accounts is closer to 100000 and seems to degrade quite drastically for larger number of accounts. Still looking into it to see what is causing the degradation.

[aeyakovenko]

@sambley, that’s good data! Can you play around with the qd and IO settings?

@sakridge is journaling disabled?

[sambley]

@anatoly, yes I will play around with the settings

[aeyakovenko]

@sambley, my GitHub handle is @aeyakovenko. I wish I was @anatoly :)

[garious]

cc #1884

[aeyakovenko]

@sambley, my guess is performance is better while the file system is in the Linux ram cache.

[garious]

@sambley, does this PR assume SSDs are available? Or is there some way to get the original behavior when there are no SSDs available (like on a developer machine)?

[aeyakovenko]

@garious we can figure out how to factor out the load/store to persistent storage. In theory, the linux kernel caches the file system in ram, so if the on disk accounts fit in ram it shouldn't be noticable for developers.

[sakridge]

@sambley, that’s good data! Can you play around with the qd and IO settings?

@sakridge is journaling disabled?

@aeyakovenko I haven't formatted the drives, @sambley should have access to the machine now though.

*edit saw that he already ran the experiment so it's a question for @sambley

[sambley]

@aeyakovenko, formatted the drives for ext4, so journaling should be enabled. I have rewritten the implementation to use memory mapped I/O and that seems to perform on par for atleast 100000 accounts as expected. Will try out for larger number of accounts and see how it behaves.

[aeyakovenko]

@sambley Awesome! What TPS are you seeing? Journaling might be significantly worse for writes in some cases. We would need to profile with both. I think there might be a bunch of parameters to tune there too.

[sambley]

@aeyakovenko, its hitting only a mean TPS of 30K, would experiment tuning the different parameters to see which one provides better results.

[aeyakovenko]

@sambley the spec for those has 500,000 random writes per sec with qd32

How many reads and writes are we doing per tx? Can you profile a mmap file on those devices as well?

[aeyakovenko]

@sambley, another thing to try would be to append any new accounts, like one file per bank thread, and store the file+index offset in the tree. Appending is well optimized by all the drivers and the hardware.

[sambley]

@garious, updated patch with your other review comments.
Just to clarify, you would like to fallback to older accounts interface if the paths is set to None and only use the persistent store changes if the paths is set

[garious]

This is an amazing contribution. Thanks so much!

[garious]

@sakridge, still waiting for changes?

[sakridge]

@garious it doesn't do the fallback to the hashmap-only implementation. It also takes over the Bank::id with another value, I'm not sure if that's safe because I don't know what exactly we are using that for before.

[sakridge]

@sambley I was trying to rebase this change and I found it doesn't pass this test which seems to cause it to fail one of the integration tests: #2961

[sambley]

@sambley I was trying to rebase this change and I found it doesn't pass this test which seems to cause it to fail one of the integration tests: #2961

I am working on adding the fallback mechanism to hashmap implementation and will fix the test failure as well

[sakridge]

@sambley sweet, thanks!

[sakridge]

@sambley we also decided the fallback to hashmap-only implementation is not necessary today, so prioritize that last.

[sambley]

@sambley we also decided the fallback to hashmap-only implementation is not necessary today, so prioritize that last.

@sakridge, fixed the test failures. Let me know if we want to merge this and work on the fallback mechanism or any pending issues in separate PR.

[garious]

This looks good to me. I'm okay with this being merged if @sakridge approves. The quantity of unsafe calls definitely makes me uncomfortable about not having a fallback, so I hope we can get a follow-up PR shortly after. That PR doesn't necessarily have to add a fallback though. Finding a way to eliminate all (maybe most) of the unsafe calls might be sufficient. What's important to me is that if some component in Solana causes a seemingly-random crash, that there's some build variation where the compiler can guarantee that's not possible. That means all Rust, only safe uses of unsafe (need to manually reason through each instance), and no C/C++ libs under the hood. We're not there yet, but this PR moves us in the opposite direction because of those unsafe, unaligned memory accesses.

[sakridge]

Seems good to me.