-
Notifications
You must be signed in to change notification settings - Fork 262
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add symlinks to data files instead of copying #583
feat: Add symlinks to data files instead of copying #583
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you verify this also fixes #563? Thanks!
@gaurav274 This doesn't throw an error while trying to load a missing file. But it throws the dataset missing file exception when we try to run a SELECT on a missing file.
I can maybe add a similar exception when you load a missing file. |
Okay, so it does say 0 files loaded. We can discuss, what the behavior should be if some files are missing while loading and selecting. Btw, in this case if no file was loaded, why does it say missing file? Shouldn't it just execute with empty output? |
It says missing file during select because the original dataset file doesn't exist. That is the check that I have implemented as of now. I think if files are missing while loading we should raise an immediate error there. |
Can we merge this PR, or is there anything remaining? |
This PR adds a major change (with little code).
Current Behavior
We copy the dataset file that the user loads into the
eva_datasets
folder.Change
This PR modifies this behavior to add a symlink from
eva_datasets
to the file on the user's filesystem. In case the user later deletes the file, an appropriate error is thrown.Testing