[ENH] Major update to pointing to files within, outside of, or remote to a current BIDS dataset

[sappelhoff]

This PR implements a suggestion by @effigies to solve several long standing issues with how we refer to files within, outside of, and remote to a current BIDS dataset. (see: https://github.com/bids-standard/bids-specification/projects/6)

These fields are affected:

• IntendedFor
• SpatialReference
• AssociatedEmptyRoom
• B0FieldIdentifier
• B0FieldSource
• BasedOn
• RawSources
• Sources

Specifically, @effigies suggests (and I agree) to use a URI scheme like bids:<dataset-name>:/absolute/path/within/dataset to refer to data.

<dataset-name>s are declared as name-to-location mappings in a new DatasetLinks object in dataset_description.json.

The whole proposal is fleshed out in the changes of this PR in the "Common Principles" section.

I also tried to make all other adjustments (examples, metadata field descriptions, etc.) so that you can have an impression of what will be touched by these changes. I may have missed something, please help me find everything.

This is my interpretation of @effigies proposal, so please correct me where I went wrong, Chris. And all other reviewers: Please help us to make this a good revision to consistently refer to files in BIDS!

This is intended to close several issues:

closes #790
closes #471
closes #718
#757 (or at least, pave the way for it to be closed)

[VisLab]

Line 388: Throughout the BIDS specification there are metadata...
not is metadata

[VisLab]

Line 392 Suggest rewording of

"failed to consistently describe relative to what these pointers should be specified,
and the exact syntax to be used.

To:
"failed to consistently describe what these pointers should be relative to.

Line 394: suggest rewording of:

"Yet even with perfectly well and consistently defined "relative paths",

to:

"Yet even with consistently defined "relative paths",

[VisLab]

Having gone through the entire proposal, I find it well-thought out. However, when I started reading, I became confused on line 407:

bids:<dataset-name>:/absolute/path/within/dataset

because I didn't understand what <dataset-name> could be if I wanted to specify a file within the dataset. It became clear later that this was the "local" bids::/.

Could we put a brief sentence here explaining that for local paths <dataset-name> will be empty and other cases will be explained in the examples?

[sappelhoff]

@robertoostenveld I believe I have addressed all your open points (thanks for the review) - could you please verify and either raise new points or approve of the changes?

If you can't do it now, an estimated date would also be helpful :-)

[effigies]

Here are some fiddly comments. I will write up an additional post with my more general perspective.

[effigies]

Suggested change

	that `derivatives/` *folder* is not considered a BIDS dataset (see [Storage of Derivated Datasets](#storage-of-derived-datasets)),
	that `derivatives/` *folder* is not considered a BIDS dataset (see [Storage of Derived Datasets](#storage-of-derived-datasets)),

[effigies]

Generally we don't treat <bids>/derivatives/ itself as a derivative dataset but a container for derivative datasets.

Suggested change

	the BIDS URIs within the nested derivatives dataset MUST be resolved with respect to `/derivatives/dataset_description.json`,
	the BIDS URIs within the nested derivatives dataset MUST be resolved with respect to `/derivatives/<derivative>/dataset_description.json`,

[effigies]

The contents of sourcedata/, code/ and unspecified derivatives are not specified, so it does not make sense to me to provide an interpretation for BIDS URIs found within them. We can't stop people from using them, and a tool from doing its best to find an appropriate BIDS root to resolve relative to, but this feels out-of-scope.

[effigies]

What does this mean for the validator? Should the validator reject relative paths, have a relative-path rejecting mode that something like OpenNeuro can use, or something else?

[effigies]

RawSources are in a source dataset. We probably want something like:

Suggested change

	"RawSources": ["bids::/sub-01/func/sub-01_task-rest_bold.nii.gz"]
	"RawSources": ["bids:src:/sub-01/func/sub-01_task-rest_bold.nii.gz"]

[effigies]

This is a unique string, not a URI. It's just an identifier so that you can do things like:

fieldmap = layout.get(B0FieldIdentifier="myidentifier", extension=".nii.gz")

[effigies]

Again, identifier, not a URI.

applies_to = layout.get(B0FieldSource="myidentifier", extension=".nii.gz")

[effigies]

Apologies for not getting back to this sooner, as I see significant discussion has happened. I would like to push back on a particular concept here, but I recognize that this may have been settled to others' satisfaction. If that's the case, please use a thumbs-down (:-1:) on my post to indicate that you don't want to reconsider this. Please use a thumbs-up (:+1:) to indicate that you agree with my proposal, and a confused (😕) to indicate that you would like to re-discuss. I will leave it to @sappelhoff to decide whether this can be summarily dismissed, accepted, or re-litigated.

---

In the context of BIDS URIs, a dataset with a given <dataset-name> does not have to be a BIDS dataset, or a neuroimaging dataset. It may just as well be a (sub-)directory containing one or more arbitrary files.

I would prefer bids:<dataset-name> to always refer to a BIDS dataset, but the path within the dataset could be in a non-standard location. For example, supposing I have non-standard derivatives, I could say bids::/derivatives/special_deriv/path/to/file, rather than bids:special_deriv:/path/to/file.

In practice, I don't see an obvious way to enforce this distinction for subdirectories short of requiring that the target be validated, but I think we should not encourage this.

We present a few use cases, where I would recommend alternative practices:

Refer to a file outside of a dataset but on the same host

[...]
To link for example to an MRI file that is stored in a
local directory that is NOT nested in the raw BIDS directory:

bids:canonical:/single_subj_T1.nii

{
    "DatasetLinks": {
        "canonical": "file://../../common/matlab/spm12/canonical"
    }
}

I would instead encourage users to use a regular URI such as https://github.com/spm/spm12/raw/r7771/canonical/single_subj_T1.nii. Alternately, to ensure access, one could copy matlab/spm12/canonical/single_subj_T1.nii into sourcedata/ and refer to it as bids::/sourcedata/spm12/canonical/single_subj_T1.nii

Refer to a remote file

[...]
bids:my-location:/Conte69_Atlas/Conte69.R.midthickness.32k_fs_LR.surf.gii

{
    "DatasetLinks": {
        "my-location": "https://github.com/mgxd/brainplot/raw/master/brainplot"
    }
}

I would treat this the same, actually. I would either use a remote URI (https://github.com/mgxd/brainplot/raw/master/brainplot/Conte69_Atlas/Conte69.R.midthickness.32k_fs_LR.surf.gii) or encourage people to copy it into sourcedata/.

---

To put a fine point on it, I see BIDS URIs as solving two problems:

1. We have a variety of inconsistent ways of referencing files within and between BIDS datasets.
2. Related datasets (typically a source dataset <-> derivatives relation) might be organized in multiple ways, as described in Source vs. raw vs. derived data.

Dataset links seemed like an elegant solution as it allowed us to unify both within and between, while simultaneously making wholesale reorganization tractable via editing a single dictionary at the root of the dataset.

Using a URI schema allows us to further unify with external resources because now we can say "Any reference to a file is a URI", which is easily validated. The indirection is less clearly needed for unrelated datasets or external resources, and so I would rather not introduce it. Additionally, to me bids: says "Here is a file in a BIDS dataset", not "This is a reference coming from a BIDS dataset". So if we are going to support non-BIDS datasets with this mechanism, I would recommend a different schema than bids:.

As a final, minor point, I also worry that diluting the meaning of bids:<dataset> to potentially mean non-BIDS datasets will complicate adding in authorities that could respond by returning specific files, such as bids://openneuro.org/ds000001:/sub-01/anat/sub-01_T1w.nii.gz. I don't know that it would actually be a problem, however.

[sappelhoff]

thanks a lot for the thorough look, Chris. I'll deal with this once I am back from vacation. All others who want to help: Please read the post and "vote" as requested :-)

[satra]

I would prefer bids:<dataset-name> to always refer to a BIDS dataset,

i think the proposal really comes down to this statement from @effigies above. but, the two examples right after point to non-bids-dataset references (spm, conte69). i concur with the above sentiment that bids: only allows for a BIDS dataset. perhaps one should not overload this to point to other things, and simply use persistent identifiers for those things to prevent link rot. yes, individuals can always copy those resources into a bids dataset and use the canonical form, but then i would want the provenance (checksum, source repo/location, etc.,.) to be tracked.

i thought the intent of this PR was to refer to files in bids datasets rather than general resources, which then gets into provenance, parameters of operations, etc.,. if the latter is to be considered, how would this relate to space related links and potential extensions being proposed in bep 14 and the work in templateflow? perhaps a way to consider is to focus on bids datasets to start with and see which use cases are being left out?

[VisLab]

Having read through all of the comments and thinking about it, I basically agree with comments of @effigies. Is it correct that this proposal is:

1. bids:<dataset-name> always refers to a BIDS dataset.
2. Anything that doesn't start with bids: has to be directly downloadable from a standard URI.

I do have a question though..... what is a BIDS dataset? I realized that I am not completely clear on what the minimum requirements are for a directory to be considered "BIDS". Is there a strictly enforced rule about what sub-directories must/are allowed to be under the top level directory? What is the minimum requirement for files in the root?

[effigies]

To commenters/reviewers on this PR, I have opened #918 as an alternative that addresses my concerns. In the significant scale-back of scope, it's possible that I took out too much clarifying text that was added previously. I would appreciate your review over there.

Pinging especially @nicholst @robertoostenveld @satra @VisLab, who previously reviewed.

[nicholst]

This a quite extensive PR... is it possible to get rendered? (Sorry if I should know the URL magic!)

[effigies]

@nicholst I would look at #918, which is intended to replace this one and is much more compact. It's rendered here: https://bids-specification--918.org.readthedocs.build/en/918/

For reference, this one is rendered here: https://bids-specification--820.org.readthedocs.build/en/820/

You can find these links as the readthedocs.org check (bottom line of this screenshot):

[nicholst]

Thanks @effigies! For the directions to the rendered version, and to #918. Indeed, the only issues I caught on this PR (text that desribes the evolution of treatment of URIs, which doesn't fit in an 'archival' version of the standard) have been addressed in #918.

[sappelhoff]

closing this, as #918 has completely taken over and all discussion seems resolved.