Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize dir store to have multi-layer structure #64

Merged
merged 4 commits into from
Sep 22, 2024

Conversation

AbdelrahmanElawady
Copy link
Contributor

Description

Optimize dir store to have multi-layer structure where the first layer contains directories with the first byte (like: 1f) then under each directory we have all blobs that have keys prefixed with this byte (like: 1f677468c9b4e09db7630b52b4a0cc799c3f86e26e7147fc61e30fdd94f81dde, 1f1b11c3509a473b77a0ffdc8d7e6a744da92293ecb3d8c38a436dd4dbdb27b5).

Example tree here.

Copy link
Member

@muhamadazmy muhamadazmy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once small comment, then feel free to merge if works fine

rfs/src/store/dir.rs Outdated Show resolved Hide resolved

let path = match fs::try_exists(&dir_path).await {
Ok(true) => dir_path.join(file_name),
Ok(false) => self.root.join(file_name),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of adding this handling, why not create a once run migration script (maybe on the store new/initialization).
So we don't need to do unnecesary dir checking on each get

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually there is absolutely no need to check if dir exist or not on on a get. Trying to read the file directly is good enough. we will still get NotFound if the file or any of the directories on the path don't exist

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ much better


let path = match fs::try_exists(&dir_path).await {
Ok(true) => dir_path.join(file_name),
Ok(false) => self.root.join(file_name),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually there is absolutely no need to check if dir exist or not on on a get. Trying to read the file directly is good enough. we will still get NotFound if the file or any of the directories on the path don't exist

@rawdaGastan rawdaGastan merged commit a6e52b0 into master Sep 22, 2024
2 checks passed
@rawdaGastan rawdaGastan deleted the development_optimize_dir_store branch September 22, 2024 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants