Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature DB File warmer #7050

Merged
merged 4 commits into from
May 20, 2024
Merged

Feature DB File warmer #7050

merged 4 commits into from
May 20, 2024

Conversation

asdacap
Copy link
Contributor

@asdacap asdacap commented May 20, 2024

  • As the state is being read, the database will be cached in the OS cache, therefore the block processing improve over time. It takes about 30 minutes of constant block processing (assuming catching up) or about 2 weeks in real time for this effect to continue making consistent testing hard to achieve. Plus if I just want it to run as fast as it could to test for memory issue for example, it take some time.
  • This PR introduce a flag that will read the files from DB on start so that the OS cache is filled up early on.
  • On a system with 192GB of RAM (max RAM possible on a consumer level hardware at the moment) this can cache the whole state db resulting in a close to 0 iops at ssd level.
  • On a system with lower available memory than the size of the db, it will attempt to priority lower level and newer file first, skipping other file entirely. I did not test the effect on such sysstem, but I don't have much hope due to the random nature of the workload.
  • Graph is (before, after, before, after). sudo sysctl -w vm.drop_caches=3 was run between runs.
    Screenshot from 2024-05-20 13-42-48

Types of changes

What types of changes does your code introduce?

  • New feature (a non-breaking change that adds functionality)
  • Optimization

Testing

Requires testing

  • Yes
  • No

If yes, did you write tests?

  • Yes
  • No

Notes on testing

  • Tested with forward syncing node from an old backup.

@asdacap asdacap requested a review from rubo as a code owner May 20, 2024 05:51
Copy link
Member

@LukaszRozmej LukaszRozmej left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. @benaadams something for you to check out too!

src/Nethermind/Nethermind.Db.Test/DbOnTheRocksTests.cs Outdated Show resolved Hide resolved
src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs Outdated Show resolved Hide resolved
Comment on lines +243 to +257
long totalSize = 0;
fileMetadatas = fileMetadatas.TakeWhile(metadata =>
{
availableMemory -= (long)metadata.metadata.FileSize;
bool take = availableMemory > 0;
if (take)
{
totalSize += (long)metadata.metadata.FileSize;
}
return take;
})
// We reverse them again so that lower level goes last so that it is the freshest.
// Not all of the available memory is actually available so we are probably over reading things.
.Reverse()
.ToList();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: as fileMetadatas are a List to begin with, instead of TakeWhile, you could just reduce its Count with: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.collectionsmarshal.setcount?view=net-8.0#system-runtime-interopservices-collectionsmarshal-setcount-1(system-collections-generic-list((-0))-system-int32)

and just reverse, without making another list.

@benaadams
Copy link
Member

benaadams commented May 20, 2024

Interesting. @benaadams something for you to check out too!

I can confirm that rebooting tanks performance 😉

Then needs a while to brew

@benaadams
Copy link
Member

benaadams commented May 20, 2024

Interesting. @benaadams something for you to check out too!

I can confirm that rebooting tanks performance 😉

Then needs a while to brew

Can confirm this is straight back into good perf after a reboot (this is on top of all the other PRs)

image

@asdacap asdacap merged commit 1df6ed6 into master May 20, 2024
67 checks passed
@asdacap asdacap deleted the feature/db-file-warmer branch May 20, 2024 13:33
@benaadams
Copy link
Member

Perhaps default this to true if you have enough memory for the given statedb directory?

@asdacap
Copy link
Contributor Author

asdacap commented May 21, 2024

Not sure on that. I'm still erring towards false. Its not known if the user is on VPS for example, or there are other software running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants