Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

reduce ship memory usage #577

Closed
wants to merge 14 commits into from
Closed

reduce ship memory usage #577

wants to merge 14 commits into from

Conversation

huangminghuang
Copy link
Contributor

@huangminghuang huangminghuang commented Dec 15, 2022 •
edited
Loading

This PR addresses the issue #203 to reduce memory consumption for state history plugin.

@huangminghuang huangminghuang marked this pull request as ready for review December 16, 2022 16:53
@heifner heifner added the OCI Work exclusive to OCI team label Dec 17, 2022
heifner
heifner requested changes Dec 19, 2022
View reviewed changes
Copy link
Member

@heifner heifner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't get to reviewing the test code. I did test with EOS and WAX snapshots.

libraries/state_history/create_deltas.cpp Outdated Show resolved Hide resolved
libraries/state_history/create_deltas.cpp Outdated Show resolved Hide resolved
libraries/state_history/create_deltas.cpp Outdated Show resolved Hide resolved
libraries/state_history/create_deltas.cpp Outdated Show resolved Hide resolved
libraries/state_history/create_deltas.cpp Outdated Show resolved Hide resolved
plugins/state_history_plugin/session.hpp Outdated Show resolved Hide resolved
plugins/state_history_plugin/session.hpp Outdated Show resolved Hide resolved
plugins/state_history_plugin/session.hpp Outdated Show resolved Hide resolved
plugins/state_history_plugin/session.hpp Outdated Show resolved Hide resolved
plugins/state_history_plugin/session.hpp Outdated Show resolved Hide resolved
@spoonincode spoonincode linked an issue Dec 20, 2022 that may be closed by this pull request
near double memory usage when starting from a snapshot with state history plugin enabled #203
Closed
@spoonincode
Copy link
Member

Can we document some before/after numbers?

@huangminghuang
Copy link
Contributor Author

Can we document some before/after numbers?

Based on my testing with mainnet staring with latest snapshot. The memory used by nodeos stayed at 25.4G during store_chain_state stage in this branch; while it shot up to 40.6G with the main branch. Using ship_streamer --fetch-delta to stream the data from nodeos right after that, the nodeos memory barely went up to 26.0G in this branch, while it shot up more than 40+ G in the main branch before it went down because the RAM been reallocated to ship_streamer.

This was referenced Jan 9, 2023
Merged
Merged
@heifner heifner mentioned this pull request Jan 20, 2023
Merged
@heifner
Copy link
Member

heifner commented Jan 20, 2023

Conflicts resolved in #628 which is probably the best place to start/continue review of these changes.

@huangminghuang
Copy link
Contributor Author

Superseded by PR #628

@huangminghuang huangminghuang deleted the huangminghuang/ship-memory branch March 9, 2023 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@heifner heifner heifner approved these changes

@spoonincode spoonincode Awaiting requested review from spoonincode

Assignees
No one assigned
Labels
OCI Work exclusive to OCI team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

near double memory usage when starting from a snapshot with state history plugin enabled
3 participants
@huangminghuang @spoonincode @heifner