fs-index: Track API
#85
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Benchmark for c2371c5Click to view benchmark
|
kirillt
reviewed
Aug 15, 2024
kirillt
reviewed
Aug 15, 2024
kirillt
reviewed
Aug 15, 2024
tareknaser
force-pushed
the
fs_index_track_API
branch
from
fe60070
to
91645e2
Compare
Benchmark for 95ad4a0Click to view benchmark
|
Benchmark for f653836Click to view benchmark
|
kirillt
reviewed
Sep 1, 2024
fs-index/README.md
Outdated
| @@ -10,6 +10,7 @@ The most important struct in this crate is `ResourceIndex` which comes with: | |||
|
|
|||
| - **Reactive API** | |||
| - `update_all`: Method to update the index by rescanning files and returning changes (additions/deletions). | |||
| - `update_one`: Method to update the index by rescanning a single file (addition/deletion/modification). | |||
There was a problem hiding this comment.
Should be mentioned in other section. "Reactive" means that we have a stream of updates and we react on these updates, even if we manually poll the stream. But this "cherry-pick" is completely manual approach, not reactive.
There was a problem hiding this comment.
Good point. What about the term "Selective API"?
kirillt
reviewed
Sep 1, 2024
kirillt
reviewed
Sep 1, 2024
kirillt
reviewed
Sep 1, 2024
kirillt
reviewed
Sep 1, 2024
| @@ -666,3 +666,245 @@ fn test_update_all_files_with_same_hash() { | |||
| assert_eq!(index.collisions().len(), 1, "{:?}", index); | |||
| }); | |||
| } | |||
|
|
|||
| /// Simple test for tracking a single resource addition. | |||
There was a problem hiding this comment.
In the arklib PR we also tried to implement an automated test scenario, where we'd test one specific invariant that "any sequence of update_one calls must be equivalent to update_all on the full data". My idea how to approach it is something like:
- Start with empty directory.
- Generate an update, randomly one of 3 kinds (modification and deletion could be generated only when we have some files).
- Call
update_oneon the updated file. - In isolation, build new index from the current state of the directory.
- Compare outputs of step 3 and step 4. Must be equal.
This is not exactly a unit test, because it's slow and tests multiple corner cases in one scenario. But I think it must be useful to run it periodically. By the way, I think that Ishan and Pushkar do in fs-storage with property-based testing should reflect the same idea.
Do you think it's feasible to implement in this PR?
There was a problem hiding this comment.
Sounds like a plan.
We could create multiple tests to verify this as well.
To do this properly and cleanly, we’ll likely need to set up some helper functions.
Given the scope, I think this should be handled in its own PR. This current PR is blocking #36, so once it's merged, I can add the additional tests along with the index watch API.
What do you think?
Benchmark for ac3828dClick to view benchmark
|
Signed-off-by: Tarek <tareknaser360@gmail.com>
Signed-off-by: Tarek <tareknaser360@gmail.com>
Signed-off-by: Tarek <tareknaser360@gmail.com>
Signed-off-by: Tarek <tareknaser360@gmail.com>
Signed-off-by: Tarek <tareknaser360@gmail.com>
Signed-off-by: Tarek <tareknaser360@gmail.com>
Signed-off-by: Tarek <tareknaser360@gmail.com>
tareknaser
force-pushed
the
fs_index_track_API
branch
from
7b07c9d
to
3ac8300
Compare
Benchmark for 52515f0Click to view benchmark
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a new method to the
ResourceIndexin thefs-indexcrate. The added method,update_one, allows the index to be updated with the latest information from the file system for a single resource. It handles updates for resources that have been added, removed, or modified.Fixes #84
Changes
ResourceIndex::update_onetest_track_removaltest_track_modificationtest_track_removal_with_collisiontest_track_additiontest_track_modification_with_collisiontest_track_addition_with_collisionindex_update_one