-
Notifications
You must be signed in to change notification settings - Fork 366
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement safe storage handles #7934
Conversation
25ad40f
to
fc310cc
Compare
Quickly drafting to check whether there are some remnants from the first four iterations that have made it into the final thing and that could just be removed/simplified now, which is very possible because it's all a big blur at this point 🫠 |
Nope, I mostly just ran into differently shaped walls. |
f417f9d
to
2fc33cc
Compare
🙏 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I learned a lot reviewing this PR and I really appreciate nice documentation! I believe it's clear to me how this addresses use cases like streaming and why the protection is in place, but folks with more experience with the code base I'm sure can provide more valuable feedback.
|
||
// NOTE: None of these fields should ever be publicly exposed, either directly or through a method, | ||
// as it is always possible to go back to an actual `RwLock` via `RwLockWriteGuard::rwlock`. | ||
// Doing so would defeat the deadlock protection that the `StorageEngine` offers. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
perhaps it's worth documenting an example use case of doing the wrong thing without the protection that StorageEngine provides?
This implements
ChunkStoreHandle
&QueryCacheHandle
which, among other things, allow for streaming dataframes across FFI and network barriers.Those handles on their own are extremely problematic though: letting them loose all across the codebase effectively wraps the entire codebase in a semantically-unsafe{} block where every row interacting with these handles can lead to very nasty race conditions and even deadlocks.
That's why this PR also introduces the
StorageEngine
type, which makes using these handles actually safe in practice:Balancing these safety guarantees with the flexibility we need (today... and tomorrow!) for all our corner use cases has proven to be a subtle art...
ChunkStoreHandle
andQueryCacheHandle
#7486Checklist
main
build: rerun.io/viewernightly
build: rerun.io/viewerCHANGELOG.md
and the migration guideTo run all checks from
main
, comment on the PR with@rerun-bot full-check
.