[ENH] Add "multipart DWI" acquisitions and refactor DWI specifications

[oesteban]

Addresses the use case of DWI sequences that need to be split into several runs because, e.g., the scanner would exceed temperature specifications due to fast gradient-switching.

This PR also refactors the text of DWIs to better accommodate the addition and make DWIs more consistent with the rest of the file.

The approach to solve this is similar to that of #622 for fieldmaps.

References: nipreps/dmriprep#43.

[mattcieslak]

I really like the MultipartID idea! This is an elegant solution and will save some headaches for pipeline software. It will cause some headaches for data curators, though.

[effigies]

Does it make sense to use the split entity?

[oesteban]

Does it make sense to use the split entity?

I honestly don't know whether that would be backward compatible, would it?
If so, I'd be fine with that (without thinking a lot about it) -- although split would then almost completely void run of semantics for DWI: if you do the exercise of imagining split elevated to the definitions in common principles after "Run" then their definitions (run vs. split) will be almost the same (differing just in particularities of f/dMRI), don't you think?

[tsalo]

If so, I'd be fine with that (without thinking a lot about it) -- although split would then almost completely void run of semantics for DWI: if you do the exercise of imaging split elevated to the definitions in common principles after "Run" then their definitions will be almost the same (differing just in particularities of f/dMRI), don't you think?

I'd say that the difference also comes in in how you analyze the data. In the case of separate DWI runs, would one just concatenate them in the fourth dimension and preprocess like that? In my (extremely limited) experience, most of the time when you have multiple DWI runs you're going to change the phase encoding direction and you'd undistort them differentially. In the case of having to break the protocol in half for the sake of the scanner, you'd just concatenate and analyze as one run, right?

EDIT: To clarify, if so then in that sense split and run would be different. split is more like "multiple parts to the same original parameters" while run is separate samples. I hope that makes sense...

[oesteban]

I'd say that the difference also comes in in how you analyze the data. In the case of separate DWI runs, would one just concatenate them in the fourth dimension and preprocess like that? In my (extremely limited) experience, most of the time when you have multiple DWI runs you're going to change the phase encoding direction and you'd undistort them differentially. In the case of having to break the protocol in half for the sake of the scanner, you'd just concatenate and analyze as one run, right?

Not sure this is how runs should be interpreted. I think BIDS should provide the researcher with the resources to encode their intent with the acquisition protocol (the what). I'm not convinced it should describe the processing (the how) in this particular instance.

EDIT: To clarify, if so then in that sense split and run would be different. split is more like "multiple parts to the same original parameters" while run is separate samples. I hope that makes sense...

I was alluding at this before: is it worth increasing the complexity to gain such little granularity of the semantics?

[mattcieslak]

The two levels of groups are 1 the images share the same B0 field (were run with the same shim setting) and 2 they have the same phase encoding direction/TRT. In HCP protocols the scans are supposed to share the same B0 field but can have opposite phase encoding directions. Because TOPUP/eddy correct everything at once, everything under the same level 1 group goes in to the algorithm. For simpler motion correction methods you might only want to look at images that are in the same level 2 group. If we add a B0GroupID, the user can specify which scans are in the same level 1 group, and you can split them into level 2 groups based in PhaseEncodingDirection/TRT. The B0GroupID tag would also be super useful for fieldmaps in general

[francopestilli]

@tsalo @oesteban nice discussion here. In general, concatenating the two DWI scans is an accepted and frequent approach. After motion compensation and other artifacts removal, the directions originally measured in the BVECS can be assumed to be slightly different in the two scans (each scan get's preprocessed slightly differently so the reference frame in the BVECS can be assumed to be slightly different). Note that this approach assumes that the BVECS are also changed (not just the DWI images) as part of artifacts removal to account for the changes in the image.

Another common approach is to keep the data separated for test-retest reliability and cross-validation for the final model solutions.

Now, I am not sure this comment is helpful. Because it does really address the issue of whether split or run should be preferred. Does split assumes they MUT be the united (they are part of a single data set that is split)? If the answer is yes perhaps run would cover more options.

Finally, there the chance for a scary future where there will be a BVEC file per voxel. This is in theory already required every time we apply a non-linear transform the DWI data. But currently, we avoid doing that, because all solutions sounds like a nightmare.

[oesteban]

The two levels of groups are 1 the images share the same B0 field (were run with the same shim setting) and 2 they have the same phase encoding direction/TRT. In HCP protocols the scans are supposed to share the same B0 field but can have opposite phase encoding directions. Because TOPUP/eddy correct everything at once, everything under the same level 1 group goes in to the algorithm. For simpler motion correction methods you might only want to look at images that are in the same level 2 group. If we add a B0GroupID, the user can specify which scans are in the same level 1 group, and you can split them into level 2 groups based in PhaseEncodingDirection/TRT. The B0GroupID tag would also be super useful for fieldmaps in general

Please check for B0FieldIdentifier in #622. I think both PRs should take different routes.

[oesteban]

@tsalo @oesteban nice discussion here. In general, concatenating the two DWI scans is an accepted and frequent approach. After motion compensation and other artifacts removal, the directions originally measured in the BVECS can be assumed to be slightly different in the two scans (each scan get's preprocessed slightly differently so the reference frame in the BVECS can be assumed to be slightly different). Note that this approach assumes that the BVECS are also changed (not just the DWI images) as part of artifacts removal to account for the changes in the image.

Another common approach is to keep the data separated for test-retest reliability and cross-validation for the final model solutions.

Now, I am not sure this comment is helpful. Because it does really address the issue of whether split or run should be preferred. Does split assumes they MUT be the united (they are part of a single data set that is split)? If the answer is yes perhaps run would cover more options.

I think this is the key of the proposals here and in #622, and basically touches upon @satra's comment in #239 (comment) regarding whether BIDS should encode intent and/or automation.

This proposal is thought to allow encoding intent but not automation. IMHO, your comment touches upon the automation concept - prescribing not just what is the motivation for acquiring these particular elements of the data, but also giving instructions on how they should be processed and analyzed.

I'm confident that the resistance against allowing BIDS to encode intent will be pretty low. In a way, not doing so would go against the very principles of BIDS.

But I'm not so sure about automation, I think that requires a higher level discussion by the community as a whole.

At this moment, I guess the main question for the fate of this PR is whether we want to unfold the concept of split from the concept of run. To me, they are two faces of the same coin, and I don't mind keeping run (which could be somewhat biased towards the fMRI community) as long as the specs state explicitly that bvals/bvecs can change across runs. I have added a sentence to state that in this PR, and it is kind of implicit in the specs right now. This is basically the conversation @tsalo wanted to bring up, right?

[effigies]

The point of split- in ephys data is to say "For technical reasons, these files that would otherwise be one file have had to be split." In that case, it's mostly due to a maximum file size, but I don't see a sequence having to be split over multiple acquisitions for temperature reasons as fundamentally different. The idea is that "These data are meant to be analyzed as a unit."

If I understand this correctly, a set of files with the same MultipartID are considered the same? Or is it that they will have a sequence of MultipartIDs that impose an ordering? Relatedly, does the order matter, or is that up to software to determine an order? This might be all in my head, but I would interpret split-1 ... split-4 as unambiguously declaring an intended order of concatenation while run-1 ... run-4 declares an order of acquisition.

On balance, the question between run and split for me would be: Can the data be meaningfully analyzed in isolation? If so, run- makes sense. If they must be analyzed together, and could just as well be concatenated, then making it clear within the filename with split- makes sense to me.

[oesteban]

On balance, the question between run and split for me would be: Can the data be meaningfully analyzed in isolation? If so, run- makes sense. If they must be analyzed together, and could just as well be concatenated, then making it clear within the filename with split- makes sense to me.

So here there is a prescription of how you should process the data (just flagging it). That said, the problem with these kinds of DWI datasets is that, in theory, you could just stitch the splits together. In practice, they are typically preprocessed separately (mostly head motion correction) and then merged together.

I think here we want to preserve the order of acquisition. That is currently doable with run-. The addition of MultipartID just would tag runs to signal the notion that they can potentially be directly merged - but perhaps more sophisticated processing would be better before merging (i.e., some hierarchical head-motion correction/between-run-alignment).

I also proposed the new metadata key because I assume there is less friction in adding metadata affecting only DWI. If we were to accept split- for DWI then I don't see much of a point in having run- for them. Keeping the protocol's provenance (i.e., a less opinionated track of the ordering in which each run was acquired) seems a simpler change.

That all said, sorry for my ignorance -- I didn't know split- already existed for ephys (meaning, I looked at it as a bigger change before learning the keyword and semantics are already somewhere in BIDS).

Will have a look into your comments later, thanks for taking the time 👍

[francopestilli]

On balance, the question between run and split for me would be: Can the data be meaningfully analyzed in isolation? If so, run- makes sense. If they must be analyzed together, and could just as well be concatenated, then making it clear within the filename with split- makes sense to me.

So here there is a prescription of how you should process the data (just flagging it). That said, the problem with these kinds of DWI datasets is that, in theory, you could just stitch the splits together. In practice, they are typically preprocessed separately (mostly head motion correction) and then merged together.

I think here we want to preserve the order of acquisition. That is currently doable with run-. The addition of MultipartID just would tag runs to signal the notion that they can potentially be directly merged - but perhaps more sophisticated processing would be better before merging (i.e., some hierarchical head-motion correction/between-run-alignment).

I also proposed the new metadata key because I assume there is less friction in adding metadata affecting only DWI. If we were to accept split- for DWI then I don't see much of a point in having run- for them. Keeping the protocol's provenance (i.e., a less opinionated track of the ordering in which each run was acquired) seems a simpler change.

That all said, sorry for my ignorance -- I didn't know split- already existed for ephys (meaning, I looked at it as a bigger change before learning the keyword and semantics are already somewhere in BIDS).

Will have a look into your comments later, thanks for taking the time 👍

@oesteban I am supportive of using -run instead of adding -split there is sufficient overlap in the usage here that we do not need to add a new term. My 2 cents.

[effigies]

That said, the problem with these kinds of DWI datasets is that, in theory, you could just stitch the splits together. In practice, they are typically preprocessed separately (mostly head motion correction) and then merged together.

It reads like you're interpreting split- as requiring concatenation before any processing, which I don't mean to imply. I can take a series apart and process each volume individually if I want; that's an algorithmic decision. The point of split is to clearly mark to tools and users that a set of files cannot be conceptually separated.

I am supportive of using -run instead of adding -split there is sufficient overlap in the usage here that we do not need to add a new term.

MultipartID would be a new term. split- already exists within BIDS.

If the contention is that adding split- makes run- redundant, then:

1. Moving to split- or MultipartID will require all DWI datasets to be updated.
2. Neither is actually necessary, since you can always interpret run- as split- right now.

If they are not the same concept, then using split- involves renaming files in some datasets, rather than adding metadata in a way that respects the inheritance principle. This seems simpler to me and allows people to instantly distinguish multi-part sequences from multiple runs.

I think PyBIDS at least is robust enough to make the selection of related files trivial in either case. Not sure about BIDS-MATLAB (@Remi-Gau may be able to comment).

---

Following a maintainers meeting, I would say there is a strong hesitation to introducing an entirely new mechanism for a problem that already has a solution in BIDS. I think to add the metadata field, I'd like to see a use case that can't be satisfied by the entity.

[oesteban]

The point of split is to clearly mark to tools and users that a set of files cannot be conceptually separated.

Then I fully understood it, and that's why my impression is that run still better represents the case of DWI data. Perhaps we are misrepresenting the problem.

Typically, when you have a GE scanner but you want to run an HCP-like sequence, you first take the gradient scheme of the HCP sequence and try to find a good split of it. However, if you break the continuity of the acquisition, then the sequence of gradients is not "compensated" anymore. Also, because you have to insert breaks, researchers typically acquire in a more resilient manner and perhaps place the b=0 volumes at different places. You may optimize the ordering of each split, with several criteria: minimize temperature increase, or permitting that a failed run (e.g., you run out of time, or the participant hit the panic button) does not turn the multipart acquisition unusable. I think the only detail that does not match well with the current run- definition is having constant gradients (bvecs/bvals) across runs - which is something that seems to be implicitly allowed by the specs examples at the moments (and, I haven't checked, but the validator probably allows).

introducing an entirely new mechanism

introducing literally a piece of metadata of string type (i.e., a tag, a label, or a variable name if you will). The mechanism (sidecar JSON) is already there.

for a problem that already has a solution in BIDS.

It hasn't:

dwi/
   sub-1_dir-ap_run-1.nii.gz
   sub-1_dir-ap_run-2.nii.gz
   sub-1_dir-pa_run-1.nii.gz
   sub-1_dir-pa_run-2.nii.gz

Currently, there's no way to say that these 4 are meant to be just one file (regardless it makes sense or not).

With the already existing split- (in ephys), this could go like:

dwi/
   sub-1_split-1_dir-ap_run-1.nii.gz
   sub-1_split-2_dir-ap_run-2.nii.gz
   sub-1_split-3_dir-pa_run-1.nii.gz
   sub-1_split-4_dir-pa_run-2.nii.gz

If the majority sees this clearer, easier, and already existing (IMHO all the adaptations you mentioned above that are necessary means it really doesn't exist for DWI), then I'm fine with it.

But I believe this doesn't really resolve the issue. Now say we have two multipart DWIs with 4 files each:

dwi/
   sub-1_split-1_dir-ap_run-1.nii.gz
   sub-1_split-1_dir-ap_run-3.nii.gz
   sub-1_split-2_dir-ap_run-2.nii.gz
   sub-1_split-2_dir-ap_run-4.nii.gz
   sub-1_split-3_dir-pa_run-1.nii.gz
   sub-1_split-3_dir-pa_run-3.nii.gz
   sub-1_split-4_dir-pa_run-2.nii.gz
   sub-1_split-4_dir-pa_run-4.nii.gz

you better have your acq- label still available (and use it as a sort of MultipartID) because otherwise, you can't untangle this mess.

To me, this seems easier:

dwi/                          # MultipartID
   sub-1_dir-ap_run-1.nii.gz  # dwi1
   sub-1_dir-ap_run-2.nii.gz  # dwi1
   sub-1_dir-ap_run-3.nii.gz  # dwi2
   sub-1_dir-ap_run-4.nii.gz  # dwi2
   sub-1_dir-pa_run-1.nii.gz  # dwi1
   sub-1_dir-pa_run-2.nii.gz  # dwi1
   sub-1_dir-pa_run-3.nii.gz  # dwi2
   sub-1_dir-pa_run-4.nii.gz  # dwi2

When I see this, I know that the designer of the protocol wants me to merge together runs 1 and 2 of both pe directions (which means more massaging than just head-motion correction), and runs 3 and 4 (both dir).

I realize that my first example could also show how split- runs short. Say that you want to combine by-2:

dwi/
   sub-1_split-1_dir-ap_run-1.nii.gz
   sub-1_split-1_dir-ap_run-2.nii.gz
   sub-1_split-2_dir-pa_run-1.nii.gz
   sub-1_split-2_dir-pa_run-2.nii.gz

Again, you need an acq-label to say if you should combine same or opposed dir-. On the contrary:

dwi/
   sub-1_dir-ap_run-1.nii.gz  # dwi1
   sub-1_dir-ap_run-2.nii.gz  # dwi2
   sub-1_dir-pa_run-1.nii.gz  # dwi1
   sub-1_dir-pa_run-2.nii.gz  # dwi2

seems pretty clear (regardless of whether this combination makes sense or not, nobody forces you to follow what the metadata says - e.g., skipping slice-timing correction when you have the times).

[effigies]

Okay, so it looks like it isn't the case that you're grouping things that are just the same sequence, split because the scanner needs time to cool, but are doing arbitrary grouping. I definitely agree that you can't do arbitrary grouping with split-, which is an index that imposes a sequence on files with otherwise constant names.

You should update the examples to match what you've said here. It may be extremely obvious to DWI practitioners, but the existing examples look to me like a candidate for split-.

[edickie]

Nice discussion. Sound good to me.

[arokem]

Looks good to me! Thanks.

Just a couple of small questions/typo corrections

[effigies]

LGTM. Small comments.

[effigies]

I'm assuming so, but just to confirm, does this match current practice? This seems to be a new explicit requirement, although I see that there's never been _sbref.bval.

[oesteban]

If you see a _sbref.bval file then it should be filled with zeros. There could be a use case for nonzero (but very low b-vals << 1000) but why don't use _dwi which is more appropriate?

This does match current practice, and as you were pointing, it was implicit in the spec. I just made it explicit, if that makes sense.

[mattcieslak]

There's one large case, which is siemens scanners can't make a b=0 only series without b > 0 scan included. So the sbref will include a few b=0 images followed by a random b=1000. +1 for optional gradient files for sbrefs

[oesteban]

I think the BIDS curator has two options in that case:

• Store a _dwi file, although there are not enough orientations to fit a diffusion model this is indeed a diffusion-weighted MRI.
• During conversion, drop the b > 0 volume to store a _sbref file.

Allowing _sbref.bval adds quite a bit of complexity to PyBIDS and software and irregularity (w.r.t. sbrefs under func/), while IMHO misrepresenting the nature of the file.

Would you reconsider your +1 to having optional gradient files for sbrefs?

[mattcieslak]

Dropping the b > 0 images would be a processing step, so the sbrefs would no longer be "raw" like the other images. This is actually the same issue that the reverse phase encoding direction images have - the fmaps/*_epi images require .bval and .bvec files to be interpreted correctly for distortion correction.

These are major BIDS limitations that are prevent dmri workflows from using real BIDS and require users put the extra gradient information files into .bidsignore. I think there is another PR open for this, though. I'd be happy if there was even a sidecar field that has the bvals in it, since the gradient directions are irrelevant fir sbrefs and epi fieldmaps.

[oesteban]

Dropping the b > 0 images would be a processing step, so the sbrefs would no longer be "raw" like the other images.

Dropping the b > 0 would be the same as people are actually doing with nonsteady states in functional MRI (at some places), for instance. Or if you intendedly leave out of conversion a whole bunch of acquisitions that are otherwise already in the DICOM structure. I'm not saying anyone should do this, but it is an option - this is outside of BIDS purview.

This is actually the same issue that the reverse phase encoding direction images have - the fmaps/*_epi images require .bval and .bvec files to be interpreted correctly for distortion correction.

I don't think this is the same issue. If you have diffusion weights (i.e., nonzero b-values) then you are indeed acquiring DWI - and these files should not be under fmap/ regardless of whether you can fit a diffusion model or not. I think BIDS has sufficient specifications to write these files under dwi/ as _dwi.nii files with corresponding gradient info.

The problem of estimating a fieldmap using these DWIs is a completely different issue, which is actually addressed in #622.

Finally, accepting sbref.bval files would probably not be backward compatible (and a big change I would leave for another PR to keep this one with a focused scope).

These are major BIDS limitations that are prevent dmri workflows from using real BIDS and require users put the extra gradient information files into .bidsignore.

Can you point at some publicly shared dataset?

[nicholst]

I'm weak on diffusion but I read for clarity and general BIDS consistency... LGTM!

[mattcieslak]

Looks great, very excited to use these new features! The only concern is I would really appreciate some clarity on the bvec orientation requirements

[mattcieslak]

How will the pipelines know that the sbref is intended to be used as the anatomical reference for the multiband sequence? Does this go in the sidecar?

[oesteban]

This particular is out of the scope of the refactor - let's take this one to the finish line and then see if extending the IntendedFor to sbrefs could do.

[oesteban]

What's the protocol for approved PRs? I don't think it would hurt to get more reviews and I'm happy to push for a bit longer, but I'm unclear as to when and who should hit merge on this one :)

[effigies]

Five working days from last commit, so this can be merged on Thursday if there are no further changes needed.

[francopestilli]

@oesteban @arokem Some of us collect B0's files in a separate scan. This in theory can be saved as fmap/epi but I wonder whether we should envision a tag to keep track of the fact that these files really pertain DWI. Does it make sense? I cc Dan here, working on a related project for brainlife.io @dlevitas

[oesteban]

@oesteban @arokem Some of us collect B0's files in a separate scan. This in theory can be saved as fmap/epi but I wonder whether we should envision a tag to keep track of the fact that these files really pertain DWI. Does it make sense? I cc Dan here, working on a related project for brainlife.io @dlevitas

Is the goal of these b0s to inform fieldmap estimation? If so, the fmap/epi should be fine. However, I feel that by the fact that these files really pertain DWI you actually mean that there are other purposes and with current BIDS the intent for these files to also inform fieldmap estimation cannot be stated if they are stored under dwi.

I think that, with the clarifications to the bvecs/bvals table in this PR, you could store the b0s as _dwi and have both bvecs/bvals files be all zeros. If those b0s are matched with some dwi, you could add an acq-b0 to disentangle the files in a very semantic way.

Then, with #622, you'd be able to indicate that these b0s are meant to be applied in fieldmap estimation.

Finally, allowing a dwi/b0 suffix could also be interesting - but I think that's is beyond the scope of this PR. WDYT?

[francopestilli]

depending on the scanner and sequence some of the BVALs might have very small values in locations where 0 is expected (Siemens sometimes adds 5, 10, 15 as an example). Similarly, values for non-zero BVALs can be not-rounded off say 990 or 995 instead of 1,000 or 2020, 2010 or 1995 instead of 2000. I think this is fine for 80% of the situations but it might require changes in the future.

[oesteban]

I think @mattcieslak initiated the right place for this conversation with #352. For a cleaner diff that assumes this PR has been already merged, please use oesteban/bids-specification@enh/dwi-combinations...mattcieslak:dwi_tsv

[oesteban]

Thank you all.

Merge and split discussions over:

• [ENH] Use TSV for DWI gradients, and allow both voxel- and world- coordinates conventions #352 - gradient information storage, all b = 0 dwi/epi/sbrefs, etc;
• [ENH] Allow encoding the fieldmapping intent of the protocol #622 - fieldmap information intent

?

[effigies]

@sappelhoff If you have no objections, this is eligible to be merged.

[sappelhoff]

No objections from my side 👍