TemplateFlow permissions error in 1.3.0 #1500
Comments
|
I'm still getting the error with a singularity image built off the new docker image:
pinging @yarikoptic just in case he may spot where we are doing it wrong. |
|
I guess the filesystem layer of a Singularity image can't be written on top of. |
|
So move templateflow out of /opt ? |
|
I think it means we can't use pre-downloaded templates in Singularity. |
|
Iirc you could specify overlay and then you would be able to |
|
A quick workaround while we figure this out that worked for me:
|
|
Or image could just come with them, ie get them while generating it or that is too heavy? |
|
We are doing that for the most standard templates at this moment. |
|
TBH it's not really clear why it's trying to do anything with |
|
while I am fetching the monster of the fmriprep image to check (is "docker pull poldracklab/fmriprep" enough or should I give some tag?) -- from the log it seems that some are there and some (those failing like |
|
yeap - file is not there: $> docker run -it --rm --entrypoint bash poldracklab/fmriprep
root@6d325eccc7eb:/tmp# ls -l /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz
lrwxrwxrwx 1 root root 182 Feb 8 01:10 /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz -> .git/annex/objects/X2/fq/URL-s255798--https&c%%files.osf.io%v1%resourc-9d3f2d9def2df5e2c10c61e8ae34204f/URL-s255798--https&c%%files.osf.io%v1%resourc-9d3f2d9def2df5e2c10c61e8ae34204f
root@6d325eccc7eb:/tmp# ls -lL /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz
ls: cannot access '/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz': No such file or directoryas some others root@6d325eccc7eb:/tmp# cd /opt/templateflow/tpl-OASIS30ANTs/
root@6d325eccc7eb:/opt/templateflow/tpl-OASIS30ANTs# git annex list
here
|origin
||web
|||bittorrent
||||
X_X_ tpl-OASIS30ANTs_res-01_T1w.nii.gz
__X_ tpl-OASIS30ANTs_res-01_desc-4_dseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_desc-6_dseg.nii.gz
X_X_ tpl-OASIS30ANTs_res-01_desc-BrainCerebellumExtraction_mask.nii.gz
__X_ tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz
X_X_ tpl-OASIS30ANTs_res-01_desc-brain_T1w.nii.gz
X_X_ tpl-OASIS30ANTs_res-01_desc-brain_mask.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-BS_probseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-CBM_probseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-CGM_probseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-CSF_probseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-SCGM_probseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-WM_probseg.nii.gz
X_X_ tpl-OASIS30ANTs_res-01_label-brain_probseg.nii.gz |
|
Yep, the fastest solution would be to pull everything (check http://singularity.lbl.gov/archive/docs/v2-2/create-image#read-only-vs-read-write). |
|
Okay, I'm going to close this via bca40d1 Singularity users willing to use other templates not pre-loaded in the image will need to bind a writable folder in the host to |
|
Just checked the new singularity image does not attempt to download anything. Will release a patched version soon. |
|
Okay, just to return to this, @yarikoptic, assuming the files are present, should datalad be calling git-annex when the files exist? I can't find the source code for templateflow, but I assume that it's making some call to datalad. |
|
who knows @effigies ;-) probably not since for |
|
but what lead to reopening -- you got permission denied again @effigies ? isn't templateflow a part of fmriprep? (I saw it there IIRC) |
|
TemplateFlow is kind of its own thing, and I'm trying to figure out how it works. Oscar's fix did not resolve issues for users, so I'm trying to figure out why we're getting the error seen. There's currently a full traceback here: https://gist.github.com/Gilles86/75c3b848f1973d947f0838a1020f7f76#file-fmriprep-1-3-0-post1 |
|
The problem seems to be the lockfile, which apparently gets held even for read-only operations. That may be what's being described in your post, @yarikoptic. Do you understand what he means by |
|
Oh, I see |
|
Yes, please try! If works - great. Regardless, we should workaround on the level of datalad. For find here we definitely not need merging annex branch info. |
|
I was able to reproduce the bug with a new singularity image built from |
…g updated remote information
git-annex pretty much for any command which could potentially benefit
from updated availability information present in updated remote git-annex
branches will try to merge those first. But if that fails, the whole call
fails, even if what we are asking does not require updated remote information.
Use cases where we ran into it already:
- datalad ls -L of a dataset owned by someone else
- datalad get files whenever files load is already there but partition
is not writeable (singularity image, see nipreps/fmriprep#1500 (comment)
So with this change we set -c annex.merge-annex-branches=false for those
git annex invocations where I think we do not really need any git-annex
branch being merged since we care only about local stuff.
This commit comes without a test since I have failed to simulate the
failing scenario without sudo chown someonelse -R testrepo . We could
probably do that on Travis, but I thought first to check if there
could be better ideas
|
fix on datalad side is WiP: datalad/datalad#3164 |
…g updated remote information
git-annex pretty much for any command which could potentially benefit
from updated availability information present in updated remote git-annex
branches will try to merge those first. But if that fails, the whole call
fails, even if what we are asking does not require updated remote information.
Use cases where we ran into it already:
- datalad ls -L of a dataset owned by someone else
- datalad get files whenever files load is already there but partition
is not writeable (singularity image, see nipreps/fmriprep#1500 (comment)
So with this change we set -c annex.merge-annex-branches=false for those
git annex invocations where I think we do not really need any git-annex
branch being merged since we care only about local stuff.
This commit comes without a test since I have failed to simulate the
failing scenario without sudo chown someonelse -R testrepo . We could
probably do that on Travis, but I thought first to check if there
could be better ideas
…g updated remote information
pretty much for any command which could potentially benefit
from updated availability information present in updated remote git-annex
branches git-annex will try to merge those first. But if that fails, the whole call
fails, even if what we are asking does not require updated remote information.
Use cases where we ran into it already:
- datalad ls -L of a dataset owned by someone else
- datalad get files whenever files load is already there but partition
is not writeable (singularity image, see nipreps/fmriprep#1500 (comment)
So with this change we set -c annex.merge-annex-branches=false for those
git annex invocations where I think we do not really need any git-annex
branch being merged since we care only about local stuff.
This commit comes without a test since I have failed to simulate the
failing scenario without sudo chown someonelse -R testrepo . We could
probably do that on Travis, but I thought first to check if there
could be better ideas
Relevant traceback:
My suspicion is that the solution would be to add
umask 000beforedataladcommands in the Dockerfile.Full log.
cc @Gilles86 @oesteban
The text was updated successfully, but these errors were encountered: