Add watch_index method and ark-cli watch command
#36
Conversation
Benchmark for 341c426Click to view benchmark
|
|
It's a good PR, and it seems to be pretty straightforward to complete it, but I'm afraid that merging it before the other index refactorings could make porting ARK-Builders/arklib#72 too difficult. Because we'd need to add one more function to the index, and at the same time we need to check diffs while porting ARK-Builders/arklib#72. |
Benchmark for 0332bd7Click to view benchmark
|
Signed-off-by: Tarek <tareknaser360@gmail.com>
Signed-off-by: Tarek <tareknaser360@gmail.com>
Signed-off-by: Tarek <tareknaser360@gmail.com>
|
Updated the watch API to call |
Benchmark for dc67cc3Click to view benchmark
|
Signed-off-by: Tarek <tareknaser360@gmail.com>
Signed-off-by: Tarek <tareknaser360@gmail.com>
|
There appear to be some unexpected events coming from the
This situation requires further investigation. Additionally, we need to test this scenario alongside other |
|
@tareknaser does only simultaneous deletion cause problems? Does simultaneous addition work fine? |
Benchmark for b8ed0bdClick to view benchmark
|
Yes and yes |
Signed-off-by: Tarek <tareknaser360@gmail.com>
|
README should be updated to explicitly state in which folder this command should be run: If |
Benchmark for 4fe7076Click to view benchmark
|
Signed-off-by: Tarek <tareknaser360@gmail.com>
Signed-off-by: Tarek <tareknaser360@gmail.com>
I added a note on how to run the example and mentioned that more can be done with |
Benchmark for 1bbb897Click to view benchmark
|
fs-index/src/watch.rs
Outdated
|
|
||
| let relative_path = file.strip_prefix(&root_path)?; | ||
| log::info!("Relative path: {:?}", relative_path); | ||
| index.update_one(relative_path)?; |
There was a problem hiding this comment.
The result should be used to provide user with actual updates.
There was a problem hiding this comment.
So, we have 2 approaches to choose from:
- Make
update_onereturn sameIndexUpdatetype asupdate_all(simple). Thenwatch_indexwould return same type to the user. We could batch updates made in some interval to pack events together, but that's optional (if you find this idea useful, we can create a follow-up task). - Alternatively, we could specialize
update_one, so Track API and Watch API would become more powerful comparing to Reactive API. The extra power I mean is more finely-grained events, similar to whatnotify-rsprovides: not only add/remove, but also rename/modify. This is more difficult though, and we would need to unify results fromupdate_allandupdate_onebefore returning fromwatch_index. I suggest creating a follow-up issue for future consideration of this approach.
There was a problem hiding this comment.
I chose the first approach to get things running faster and keep it simpler for now. I also plan to add tests soon.
Next, I want to add integration tests for this functionality. We could either add integration tests for fs-index directly or implement CI shell scripts to test ark-cli watch <PATH> for an end-to-end approach—possibly both?
Do you think CI shell scripts for ark-cli watch would be sufficient, or should we also include programmatic tests? For example, running the watcher in a separate thread and doing many create/delete operations to verify the results. Now that I think about it, writing these tests programmatically could be complex. What’s your take?
There was a problem hiding this comment.
I suggest creating a follow-up issue for future consideration of this approach.
tracked in #89
There was a problem hiding this comment.
Do you think CI shell scripts for ark-cli watch would be sufficient, or should we also include programmatic tests? For example, running the watcher in a separate thread and doing many create/delete operations to verify the results. Now that I think about it, writing these tests programmatically could be complex. What’s your take?
Agree, I think we can achieve proper result by simple shell script. I imagine it like this:
- Run
ark-cli watchin background, direct its output to dedicated log file. - Randomly modify folder content and write the performed actions into another log file.
- Then compare the two log files.
|
I spent some time today looking into different ways to use Order is very important in our case because we need events to happen in the right order (for example, we don’t want to see "
While looking through the examples, I also noticed that we might want to use the Debouncer. The file system can sometimes send multiple events for what is really just one change, which could cause problems. For example, it might trigger I'm now testing this in a smaller example and looking at how to set up the event stream properly. |
- Refactored `watch_index` to return an async stream of `WatchEvent` using `async_stream` and `tokio` - Integrated notify debouncer to collapse multiple rapid events into a single event - Updated the `watch_index` example to reflect the new implementation - Marked `ResourceId` as `Sync` and `Send` to allow usage in async contexts - Modified `ark-cli watch` to align with these changes Signed-off-by: Tarek <tareknaser360@gmail.com>
…Update<Id>` - Updated `update_one()` to return `IndexUpdate<Id>` with detailed changes - Modified `WatchEvent::UpdatedOne` to include all file changes - Added getters for `Timestamped<Item>` for easier access to data - Updated `index_watch` example and `ark-cli watch` to adapt to these changes Signed-off-by: Tarek <tareknaser360@gmail.com>
Benchmark for e4987f5Click to view benchmark
|
Signed-off-by: Tarek <tareknaser360@gmail.com>
Benchmark for a95e06aClick to view benchmark
|
| impl<Item> Timestamped<Item> { | ||
| pub fn item(&self) -> &Item { | ||
| &self.item | ||
| } | ||
|
|
||
| /// Return the last modified time | ||
| pub fn last_modified(&self) -> SystemTime { | ||
| self.last_modified | ||
| } | ||
| } |
There was a problem hiding this comment.
Just curious, why do we need this getters?
| @@ -515,6 +531,7 @@ impl<Id: ResourceId> ResourceIndex<Id> { | |||
| self.id_to_paths.remove(&id.item); | |||
| } | |||
|
|
|||
| result.removed.insert(id.item); | |||
There was a problem hiding this comment.
So, if I'm not mistaken, this line should be moved under the condition above:
if self.id_to_paths[&id.item].is_empty() {
self.id_to_paths.remove(&id.item);
// emit the removed event
}
I.e. only removal of the last path with same id should be considered real removal.
And we could log duplicates removal in else branch of same condition.
| result | ||
| .added | ||
| .insert(id, HashSet::from([timpestamped_path])); |
There was a problem hiding this comment.
We could wrap this into similar condition, checking that the amount of paths mapped to the id is zero. And log that duplicate was introduced in the corresponding else branch.
This should allow us relax this requirement to update_one:
/// - In case of a addition, the resource was not already in the index
|
Nitpick, but we can define aliases Then we could immediately return from This should be more readable. By the way, it seems that we could simplify From API point of view, we don't need a collection of paths attached to the addition event, only one path (representative). We might need separate events to track duplicates. Something like |
Description
This pull request adds a new method to
fs-indexcrate,watch_index, to monitor file system changes and automatically update the index.Additionally, it adds a new command to
ark-clito make this functionality accessible to users.This change is the first step of addressing issue #21.
Testing
An example of the new method's usage is in the
fs-indexcrate atfs-index/examples/index_watch.rs.To run the example, run the following command:
This command monitors the index at the
test-assets/directory and automatically updates it upon any file system changes.