Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstd split sequence decode/execute #100

Closed
wants to merge 1 commit into from

Conversation

klauspost
Copy link
Owner

@klauspost klauspost commented Apr 23, 2019

The basic idea is that we don't need the stream history to decode the sequences.

So we do history in two parts:

  1. Decode as much as we can without any history.
  2. Receive decoder history.
  3. Decode sequences.
  4. Send decoder history to next block. Next block goto 2)
  5. Wait for stream history
  6. Execute decoded sequences.
  7. Send stream history to next block. Next block goto 6)

This should allow more concurrency for stream decoding.

We probably should keep the combined code for buffer decoding.

So this would require each decoder to have a channel on which it can receive the decoder history, if it is interested.

Estimated speed boost would be in the area of 1.5x

This should allow more concurrency for stream decoding.

We probably should keep the combined code for buffer decoding.
@klauspost
Copy link
Owner Author

It does look like it is feasible to split sequence decoding and execution.

This is the execution time distribution on silencia corpus:

image

This could well be worth the synchronization cost.

@klauspost
Copy link
Owner Author

klauspost commented Apr 23, 2019

Having execution always run on the same goroutine would also be a bonus. Not sure if that is feasible designwise.

...Maybe if we sent decoded sequences instead of receiving history that could work.

@klauspost klauspost marked this pull request as draft December 17, 2020 11:44
@klauspost klauspost mentioned this pull request Jan 25, 2022
@klauspost
Copy link
Owner Author

Tracked in #477

@klauspost klauspost closed this Jan 25, 2022
@klauspost klauspost deleted the split-sequence-decoding branch March 11, 2023 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant