You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, whenever a new change is stored in a dataset, we add all incoming and outgoing references belonging to that change-entity to incoming and outgoing reference indexes.
We do this with a timestamp attached to the reference, so that point in time queries can follow the references that were active at the time.
However, often the references do not really change when an entity is updated. In these cases we don't really need to store a new index-item everytime. we just need to track changes to the reference state (in practise: the deleted state of a reference changes).
So, to optimize reference indexes, we could - as part of datahubs garbage collection process - compact reference indexes. e.g. remove identical(timestamp disregarded) successive refences from incoming and outgoing indexes.
We could also extend the logic in dataset.StoreEntities to skip adding new ref-index entries if they are not changed
The text was updated successfully, but these errors were encountered:
Currently, whenever a new change is stored in a dataset, we add all incoming and outgoing references belonging to that change-entity to incoming and outgoing reference indexes.
We do this with a timestamp attached to the reference, so that point in time queries can follow the references that were active at the time.
However, often the references do not really change when an entity is updated. In these cases we don't really need to store a new index-item everytime. we just need to track changes to the reference state (in practise: the deleted state of a reference changes).
So, to optimize reference indexes, we could - as part of datahubs garbage collection process - compact reference indexes. e.g. remove identical(timestamp disregarded) successive refences from incoming and outgoing indexes.
We could also extend the logic in dataset.StoreEntities to skip adding new ref-index entries if they are not changed
The text was updated successfully, but these errors were encountered: