Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

radisk file hashing #1005

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

radisk file hashing #1005

wants to merge 2 commits into from

Conversation

sirpy
Copy link
Collaborator

@sirpy sirpy commented Aug 31, 2020

because of SEA file paths that include pubkey can be very long.
radisk uses filenames as lexical index based on paths which doesnt work with long paths (>255 bytes on linux).

this solution solves it by:

  • save file as sha256(filename)

  • get file as sha256(filename)

  • keep in-memory index of used filenames ie index[hash] = true

  • keep on-disk list of original filenames ([dbname].idx)

  • whenever we get a put request we check the in-memory index if we already have this filename if we do proceed as usual if not we append the filename to the on-disk index

  • on initialization step (ie store.list) we read the .idx file instead of iterating over directory content for filenames, so rad has all the index information it needs

@sirpy sirpy requested a review from amark August 31, 2020 08:23
@amark
Copy link
Owner

amark commented Sep 1, 2020

w00h00! 👏

%C (something like that) is the "folder" file, maybe we can use/merge this with the idx file? So for instance, having the .idx file in the same folder could lead to collision (why I save tmps outside)

@sirpy
Copy link
Collaborator Author

sirpy commented Sep 1, 2020

@amark
well if you can explain the structure of %C, i didnt see it was used anywhere, so maybe we can use it as idx.
but %C is saved inside the directory, so why would saving the .idx there would lead to collision? it might break if you have multiple instances running with same "opt.file" but that would break radisk anyways no?

@sirpy
Copy link
Collaborator Author

sirpy commented Nov 25, 2020

@amark this is important

@nsreed
Copy link

nsreed commented May 1, 2021

Well, I tried writing my own fix for the fun of it, but Gun internals are still over my head. I humbly bump this PR, as a solution to #1070

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants