Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid copying back memory in FileChecksummer when data is valid #179

Merged
merged 1 commit into from
May 31, 2023

Conversation

animetosho
Copy link
Contributor

The current FileChecksummer almost always performs a memory copy once a read has occurred. This is because it fills the entire buffer, and every Jump call can, at most, consume half that buffer, leading to keep being greater than 0 in all cases except reaching the end-of-file.

This seemingly unnecessary copying around of memory seems rather wasteful.
A solution may be to turn the buffer into some sort of circular buffer, however implementing such is rather complex/fiddly.

With the goal of simplicity, I've implemented a much easier approach that yields most of the benefit - instead of filling the buffer completely, only fill it half way, most of the time. Since the buffer can only be consumed til the half-way point, this is safe.
Assuming all data is valid, this makes keep == 0 stay true throughout the process, avoiding the memory copy.

This leaves cases where data isn't valid, causing Step or Jump (with smaller distances) to be called. In such cases, it just reverts to existing behavior of filling the buffer completely, and moving the data to the beginning of the buffer (in Jumps case). If the data becomes valid again, it'll revert to filling the buffer halfway.

The assumption is that most data will be valid in typical usage scenarios, leading to a speedup most of the time.

Also the memmove in Step can be replaced with memcpy as the regions are guaranteed to be non-overlapping.

This may be less efficient than a circular buffer, but is much easier to implement
@BlackIkeEagle BlackIkeEagle merged commit ff990c0 into Parchive:master May 31, 2023
@animetosho animetosho deleted the avoid_checksum_copy branch May 31, 2023 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants