Insert in tx #51
Closed
Insert in tx #51
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
don't wait for a blob to be written to disk before sending it downstream don't wait for the disk store to be walked before starting everything up
update quic fix tests
swap size to bytes
increase idle timeout to avoid errors downstream add option to delete blobs from DB if storage doesn't have it (for future local tracking)
fix store init try fixing unreasonable db bottleneck
reduce touch time to every 6 hours
add cache for blobs not found
|
client side fix at lbryio/lbry-sdk#3308 if we can make that faster it should also help not having incomplete data on shutdown. |
anbsky
reviewed
May 21, 2021
anbsky
reviewed
May 21, 2021
anbsky
reviewed
May 21, 2021
|
I'm in favor of this implementation. Aside from a few nitpicks, I don't have any qualms with it. |
lyoshenka
force-pushed
the
insert_in_tx
branch
2 times, most recently
from
0470b73
to
b404458
Compare
The problem is that inserting an sd blob with ~5k blobs takes longer than 30 seconds. So the client times out and retries the request. At that point, reflector is not done inserting so it replies with a smaller number of blobs than it should. The client uploads that many blobs and marks the stream as reflected. The remaining blobs never get uploaded. Doing the insert inside a transaction should be faster than doing 10k (2 per blob) inserts independently.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
my version of @shyba's #50. see which you like better (or maybe you'll have an even better idea?)
wrap blob insertion in tx. fixes lbryio/lbry-sdk#3296
The problem is that inserting an sd blob with ~5k
blobs takes longer than 30 seconds. So the client
times out and retries the request. At that point,
reflector is not done inserting so it replies with
a smaller number of blobs than it should. The client
uploads that many blobs and marks the stream as
reflected. The remaining blobs never get uploaded.
Victor says doing the insert inside a transaction should be
faster than doing 10k (2 per blob) inserts
independently.
He's also doing a client-side fix for this.