message: Sequencer incrementally tracks pending partial sequences #345
Conversation
If there are a large number of tracked message producers, Sequencer.HasPending() can become a hot loop since it's very frequently run. It's slow because it walks every journal producer to see if it has a larger offset than the last checkpoint offset. Instead, maintain a set of *partialSeq which begin a new transactional sequence, and clear each *partialSeq upon dequeue of a sequence or rollback. Or, clear _all_ pending sequences upon a Checkpoint.
There was a problem hiding this comment.
LGTM
I have to admit that I find the idea of using a pointer as a map key deeply confusing. I don't really have a better idea, though, and my testing has shown that it works, even after re-assigning the value behind the pointer. I'm assuming that you didn't want to use the JournalProducer as the key because that would involve copying the data? Either way, I think this looks good to merge, and we can talk later if you think there's room for improvement. This might just be one of those Go things
|
Thanks. It works because the allocated Given that, the |
If there are a large number of tracked message producers, Sequencer.HasPending() can become a hot loop since it's very frequently run. It's slow because it walks every journal producer to see if it has a larger offset than the last checkpoint offset.
Instead, maintain a set of *partialSeq which begin a new transactional sequence, and clear each *partialSeq upon dequeue of a sequence or rollback. Or, clear all pending sequences upon a Checkpoint.
This change is