re_datastore: store component rows larger than a threshold behind an indirection #487
Closed
2 tasks
Labels
⛃ re_datastore
affects the datastore itself
Let's say you have an arbitrarily large component (e.g. an
Image
, whose contents we have no control over).As the contents of the
Image
gets larger and larger, the amount of rows we can fit in a single bucket gets smaller and smaller.In the worst case, you get all of the overhead of bucketing, for none of the benefits.
Not only that, but it might also be the case that the contents of the
Image
are actually identical e.g. every other frame.So not only we're wasting bucketing overhead, we're wasting space too.
In both situations, hashing the contents of the image if it gets beyond a certain threshold, and storing that hash instead of the actual contents, fixes the issue.
Of course, we need to store an extra map to resolve the hashes into actual data at query time.
TODO:
The text was updated successfully, but these errors were encountered: