✨ Seamless ingress of fmriprep outputs

[sgiavasis]

Related problem

It's not easy (or possible) to directly ingress already-computed outputs into C-PAC from other pipelines (or specifically fmriprep) in order to perform further processing.

Proposed feature

The ability to ingress fmriprep output data directly into the C-PAC resource pool, so that users can then proceed to perform further processing. Example: pulling in the confounds.tsv and also the BOLD time series that has already been preprocessed and warped to template; then, C-PAC can proceed with denoising and anything else, including calculating connectomes and other derivatives.

Note: C-PAC v1.8+ has code/infrastructure to allow ingress of already-computed data into the resource pool, but needs to be expanded to make the process seamless.

Acceptance criteria

• New field in the data configuration YAML which holds the path to an fmriprep output directory
• Each possible layout of fmriprep outputs able to be pulled into C-PAC flexibly without issues (i.e. surface on or off)
• Clearly-defined user flow: if the user has preprocessing selected in the pipeline config, and already-preprocessed data is pulled in, C-PAC needs to handle this safely

Additional context

There is also the question and possibility of forking. It's possible we can have the user provide an fmriprep output directory and also select a modified version of the fmriprep-options preconfig which includes denoising and derivatives, and have C-PAC write out the end result of both for comparison easily.

Reference

Branch: https://github.com/FCP-INDI/C-PAC/tree/feature/fmriprep_ingress

[sgiavasis]

Unit/smaller testing targets:

• 1. Passthrough test where we see that the fmriprep output files are ingressed into CPAC and written directly back out.
• 2. Build tests which ensure multiple BOLD scan files go in and come back out with the right labels.
• 3. For those not listed in the outputs.tsv, make decisions regarding re-naming or inclusion/exclusion.
  • Regressors/Confounds TSV?
  • Other naming differences?
• 4. Build tests which ensure space-* and res-* BIDS tags are converted from specific --> generic --> specific.

Integration:

• 1. Taking the whole output dataset in, will C-PAC smoothly proceed with denoising/nuisance regression and derivative calculation?
• 2. Are there guardrails for the intersection between fmriprep-preprocessed native-space data and a CPAC pipeline config requesting native-space processing? (Until point no.3 below is complete- post v1.8.6).
• 3. Taking only the T1w-related outputs in, and anatomical preprocessing and registration turned Off in C-PAC, will C-PAC seamlessly proceed with BOLD preprocessing, coregistration, and final BOLD time-series warp to template?

[sgiavasis]

@e-kenneally resolves issue with the func datasource connection.

Next:

• Update the internal guardrails to accommodate a data dictionary that works with fmriprep output directory files.
• Try using fmriprep's confounds.tsv directly with AFNI 3dTproject to see if it works out of the box.
• Ascertain BIDS standard for confounds/regressors and change CPAC's naming of regressors to confounds if necessary.

[e-kenneally]

Current issue: quadruple-nested workdir folders - there is a _scan-{scan_name} folder for EVERY scan inside every _scan-{scan-name} folder in the working directory.

This is what I know about the problem so far:

• I see the same forking issue (super nested workdir folders) if I create a datasource node for every func file, or just for each key
• Forking doesn’t happen if I only pass one scan name for every resource in the inputnode.iterables field , but then only one version / scan of each file ends up in the func output
• Everything runs smoothly and the scan iterables are handled when scan iterables handled outside of loop, but then not tied to the correct ingress node, just the last one to be created in the loop.
  - Also, this is the case where they aren’t correctly matched up with the files.

[sgiavasis]

This was resolved in v1.8.6.