-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiple features on file search and download #243
Comments
This is currently my procedure to retrieve the data for i, d in enumerate(data):
irods_path = d['COLL_NAME'] + "/" + d['DATA_NAME']
local_path = "./" + irods_path
# Create the folder structure
os.makedirs(os.path.dirname(local_path), exist_ok=True)
print("Downloading: " + str(i) + " of " + str(len(data)) + " " + local_path, end="\r")
if not os.path.exists(local_path):
download(session, irods_path=irods_path, local_path=local_path) |
Hi @jjkoehorst Thanks for the feedback!
|
Conflicting checksums for unlock/home/wur.fdp/stu_hiv-composition-gut/obs_o_samd00665758/sam_samd00665758/metagenomic_amplicon_illumina/asy_drr519722/NGTAX_Silva138.1-picrust_100_DRR519722/provenance.ttl.68bb1fbf513a3703bfcdc03e692a79fc sha2:DS9/8OmoTZoqYgNvinWluzwgio06KRB/xmycWV1zqVk= That I think you calculate the sha2? and irods is calculating the md5? I don't "own" / manage the irods instance so I cannot easily change the checksum settings on irods... |
MD5 will be taken into account: #248 For more verbosity on uploads, downloads and synchronisations you can use the
You can then either execute all operations with |
ad 3. I am a bit hesitant to introduce that to our data transfers. It is convenient if you just want to create the direct folder. However, to make it generic we would have to check if someone wants to add a full collection/folder subtree and that should not happen automatically but done consciously by the programmer. |
True, @chStaiger makes sense and its just one line of code extra. Another topic regarding downloads... We often need to download a few thousand files. In a Jupyter notebook you often start all the steps and then it needs to do checksum compute again right? Is it possible to do a size check only? |
@jjkoehorst There are no size checks available. What you can do with the new PR #254 is skip the file/data object if it exists. Perhaps for now that suffices for you? |
That could lead to thousands of warnings which migth also not be very convenient... Could add a "ignore_warning" or "silent" mode ? |
In the next release we just opted for the warnings: overwrite==False, ignore_err=False -> FileExistError Mainly because our current main user group are less experienced users and not informing them in any way they might get the wrong impression of the state of the data. |
I understand and agree :) My suggestion was not to change it but to add something to be able to silence it. |
I will release the next version without that feature and we will discuss internally whether we want to do a quickfix for that in a bug-fix version or directly work on a more thorough solution #255 |
WIth that, can we close this issue, or did I overlook something that needs to go into another ticket to pick it up in the development? |
You can suppress it using the warnings library, see for example: https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings |
@qubixes good point! I will remove the issue. |
I think the feature requests have been addressed, so I'm closing this issue. If there is a new feature request that is related to the ones in this issue, just open a new issue (one per feature ideally). |
The application works great and we try to incorporate it in Jupyter notebooks however there are some small things that we would like to see added if possible.
The search function when searching for a file we have to add a
%collection%/%.txt
/
so ibridges knows I am searching for a file now right? Is it possible to have a collection / data variable instead of only path?Skip if exists, currently it throws an error if a file already exists ... (we know please ignore and skip? or a file size check?) ignore error is a bit overkill as this is not really an error?
The folder does not exists where you want to download the file to... can ibridges please create this? (param mkdir=true?)
The text was updated successfully, but these errors were encountered: