Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

↔️ Fmriprep Ingress #1952

Merged
merged 79 commits into from
Oct 25, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
e521615
create fmriprep ingress infrastructure
e-kenneally Mar 14, 2023
8a47cd2
add generalizability to output ingress
e-kenneally Mar 15, 2023
f16b3f3
Modify ingress_output_dir to be generalizable
e-kenneally Mar 31, 2023
cc532ef
parse outdir BIDS tags
e-kenneally Apr 7, 2023
b91c7d0
Add dseg to outputs.tsv
e-kenneally Apr 7, 2023
99123fe
add 'scan' field to create_general_datasource
e-kenneally Apr 7, 2023
1edd2ec
current status of fmriprep ingress
e-kenneally Apr 18, 2023
87aaf50
:hammer: fix scan iterables workflow
e-kenneally Apr 25, 2023
d092e9d
Merge pull request #1936 from FCP-INDI/fmriprep_ingress
e-kenneally Apr 27, 2023
beaaff6
Handle resolution in filenames
e-kenneally Apr 27, 2023
79111b0
remove old ingress function
e-kenneally Apr 27, 2023
140858e
make regressors parsing generalizable
e-kenneally May 22, 2023
353162a
Add ingress_regressors to pipeline blocks
e-kenneally May 22, 2023
3a0c123
Modularize outdir_ingress workflow
e-kenneally May 22, 2023
6de4488
Add functionality to select ingressed regressors
e-kenneally May 22, 2023
faa0ac4
Create nuisance regressor ingress functionality
e-kenneally May 22, 2023
4381e65
Add desc-confounds_timeseries to outputs.tsv
e-kenneally May 22, 2023
10663b7
Add ability to select regressors from ingressed file
e-kenneally May 22, 2023
33090f5
:bulb: Update comments based on default preconfig
ChildMindInstituteCNL May 22, 2023
d328e87
make regressors file parsing flexible
e-kenneally May 25, 2023
4a94f39
Add input options for derivatives to allow outdir ingress postproc
e-kenneally May 25, 2023
3777b34
Add fmriprep-ingress pipeline config
e-kenneally May 25, 2023
1aaaa21
allow for no TR value
e-kenneally May 25, 2023
894623b
add input options for generalizability
e-kenneally May 25, 2023
9f31356
Merge branch 'feature/fmriprep_ingress' of https://github.com/FCP-IND…
e-kenneally May 25, 2023
3ccb0ae
:bulb: Update comments based on default preconfig
ChildMindInstituteCNL May 25, 2023
98c26c9
Merge branch 'develop' into feature/fmriprep_ingress
nx10 Jun 1, 2023
bebcd55
:bulb: Update comments based on default preconfig
ChildMindInstituteCNL Jun 1, 2023
2fbb66a
Add guardrail to handle bad data
e-kenneally Jun 6, 2023
a90d4b3
Make default value 'off' for ingress_regressors
e-kenneally Jun 6, 2023
36e3c54
:wrench: Add guardrail for bad data
e-kenneally Jun 8, 2023
c048059
Update default vals for fmriprep-ingress preconfig
e-kenneally Jun 8, 2023
3dc5cbb
:bulb: Update comments based on default preconfig
ChildMindInstituteCNL Jun 8, 2023
229e1c5
:hammer: fix typo
e-kenneally Jun 14, 2023
b7f1580
:hammer: Fix typo
e-kenneally Jun 14, 2023
d772dad
:pencil2: Rename regressors to desc-confounds_timeseries
e-kenneally Jun 15, 2023
8887284
Merge branch 'feature/fmriprep_ingress' of https://github.com/FCP-IND…
e-kenneally Jun 15, 2023
bb65ad9
Modularize fs ingress for use alongside outdir ingress
e-kenneally Jun 15, 2023
d348589
Minor changes
e-kenneally Jun 27, 2023
85c4acf
Fix typo
e-kenneally Jun 27, 2023
ce7c136
Merge remote-tracking branch 'origin/develop' into feature/fmriprep_i…
e-kenneally Jul 10, 2023
bbf4f83
:bulb: Update comments based on default preconfig
ChildMindInstituteCNL Jul 10, 2023
7fa4339
Merge branch 'develop' into feature/fmriprep_ingress
e-kenneally Jul 10, 2023
25fe362
fix formatting error
e-kenneally Jul 10, 2023
f8f7d4c
Merge branch 'feature/fmriprep_ingress' of https://github.com/FCP-IND…
e-kenneally Jul 10, 2023
065e1e0
Make new nodeblock version for ingress_regressors
e-kenneally Jul 11, 2023
32ab65a
Fix data-dependent template error
e-kenneally Jul 13, 2023
5dac2b4
Merge remote-tracking branch 'origin/develop' into feature/fmriprep_i…
e-kenneally Jul 25, 2023
b93585d
:arrows_counterclockwise: Remove loop to fix iterables forking issue
e-kenneally Jul 27, 2023
4728982
⬆️ Upgrade Ubuntu (18.04 LTS Bionic Beaver → 22.04 LTS Jammy Jellyfis…
shnizzedy Jul 28, 2023
e7f5018
Workaround to match scan names with files
e-kenneally Aug 7, 2023
76221e2
fix config error for multiple scans
e-kenneally Aug 7, 2023
c2136f1
:bug: Fix template renaming issue
e-kenneally Aug 30, 2023
d6b2cb2
:bug: Fix template renaming issue
e-kenneally Aug 30, 2023
0042713
Merge remote-tracking branch 'origin/devlop' into feature/fmriprep_in…
e-kenneally Sep 1, 2023
e8de7d3
Pass correct mask for f/alff
e-kenneally Sep 14, 2023
3974cc1
Handle multiple templates in outdir
e-kenneally Sep 14, 2023
97af910
:bulb: Update comments based on default preconfig
ChildMindInstituteCNL Sep 14, 2023
27a4505
:wrench: Fix docstring
e-kenneally Sep 22, 2023
567b5d8
:bug: Fix template bug
e-kenneally Sep 22, 2023
fc873eb
Merge branch 'feature/fmriprep_ingress' of https://github.com/FCP-IND…
e-kenneally Sep 22, 2023
1a1c073
Merge 'origin/develop' -> feature/fmriprep_ingress
e-kenneally Sep 22, 2023
48d36e7
fix tuple error
e-kenneally Sep 22, 2023
0ef2bfe
:bug: Fix crash when no derivatives_dir provided
e-kenneally Sep 25, 2023
68ee445
Merge remote-tracking branch 'origin/develop' into feature/fmriprep_i…
e-kenneally Sep 25, 2023
28c9b42
Merge origin/develop into feature/fmriprep_ingress
e-kenneally Oct 9, 2023
2389209
:sound: Added fmriprep-ingress to changelog
e-kenneally Oct 13, 2023
90b4162
Merge branch 'develop' into feature/fmriprep_ingress
e-kenneally Oct 13, 2023
5f955c0
Make default value off for 'create_regressors'
e-kenneally Oct 25, 2023
8cd9d9d
:bulb: Update comments based on default preconfig
ChildMindInstituteCNL Oct 25, 2023
bd0ab4f
Add exception to prevent mutually exclusive config options
e-kenneally Oct 25, 2023
2ac64a5
Merge branch 'feature/fmriprep_ingress' of https://github.com/FCP-IND…
e-kenneally Oct 25, 2023
676705c
:pencil2: Fix typo!
e-kenneally Oct 25, 2023
7b11d1d
Update ccs-options preconfig
e-kenneally Oct 25, 2023
c9a968b
:bulb: Update comments based on default preconfig
ChildMindInstituteCNL Oct 25, 2023
3920b84
Update abcd-options preconfig
e-kenneally Oct 25, 2023
2b8c98d
Update fmriprep preconfig
e-kenneally Oct 25, 2023
4e5a764
:bulb: Update comments based on default preconfig
ChildMindInstituteCNL Oct 25, 2023
af72ff8
Merge branch 'develop' into feature/fmriprep_ingress
sgiavasis Oct 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- Some automatic handling of user-provided BIDSy atlas names.
- `sig_imports` static method decorator for `Function` nodes, to accommodate type hinting in signatures of `Function` node functions.
- Ability to ingress an FmriPrep output directory and run derivatives with C-PAC
- FmripPrep-ingress preconfig that runs derivatives
- `switch_is_off`, `switch_is_on` and `switch_is_on_off` methods to `Configuration` class

## Fixed
Expand Down
15 changes: 9 additions & 6 deletions CPAC/alff/alff.py
Original file line number Diff line number Diff line change
Expand Up @@ -287,8 +287,12 @@ def alff_falff(wf, cfg, strat_pool, pipe_num, opt=None):
[
"space-template_res-derivative_desc-denoisedNofilt_bold",
"space-template_res-derivative_desc-preproc_bold",
"space-template_desc-preproc_bold"
],
[
"space-template_res-derivative_desc-bold_mask",
"space-template_desc-bold_mask"
],
"space-template_res-derivative_desc-bold_mask",
)
],
outputs=["space-template_alff", "space-template_falff"],
Expand All @@ -303,14 +307,13 @@ def alff_falff_space_template(wf, cfg, strat_pool, pipe_num, opt=None):
cfg.amplitude_low_frequency_fluctuation['lowpass_cutoff']
alff.get_node('hp_input').iterables = ('hp', alff.inputs.hp_input.hp)
alff.get_node('lp_input').iterables = ('lp', alff.inputs.lp_input.lp)

node, out = strat_pool.get_data(["space-template_res-derivative_desc-denoisedNofilt_bold",
"space-template_res-derivative_desc-preproc_bold"])
"space-template_res-derivative_desc-preproc_bold",
"space-template_desc-preproc_bold"])
wf.connect(node, out, alff, 'inputspec.rest_res')

node, out = strat_pool.get_data("space-template_res-derivative_desc-bold_mask")
node, out = strat_pool.get_data(["space-template_res-derivative_desc-bold_mask",
"space-template_desc-bold_mask"])
wf.connect(node, out, alff, 'inputspec.rest_mask')

outputs = {
'space-template_alff': (alff, 'outputspec.alff_img'),
'space-template_falff': (alff, 'outputspec.falff_img')
Expand Down
23 changes: 16 additions & 7 deletions CPAC/nuisance/bandpass.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,18 +111,28 @@ def bandpass_voxels(realigned_file, regressor_file, bandpass_freqs,

else:
with open(regressor_file, 'r') as f:
header = [f.readline() for x in range(0,3)]

regressor = np.loadtxt(regressor_file)
header = []

# header wouldn't be longer than 5, right? I don't want to
# loop over the whole file
for i in range(5):
line = f.readline()
if line.startswith('#') or isinstance(line[0], str):
header.append(line)

# usecols=[list]
regressor = np.loadtxt(regressor_file, skiprows=len(header))
Yc = regressor - np.tile(regressor.mean(0), (regressor.shape[0], 1))
Y_bp = np.zeros_like(Yc)
for j in range(regressor.shape[1]):

# Modify to allow just 1 regressor column
shape = regressor.shape[0] if len(regressor.shape) < 1 else regressor.shape[1]
for j in range(shape):
Y_bp[:, j] = ideal_bandpass(Yc[:, j], sample_period,
bandpass_freqs)

regressor_bandpassed_file = os.path.join(os.getcwd(),
'regressor_bandpassed_demeaned_filtered.1D')

with open(regressor_bandpassed_file, "w") as ofd:
# write out the header information
for line in header:
Expand All @@ -131,7 +141,6 @@ def bandpass_voxels(realigned_file, regressor_file, bandpass_freqs,
nuisance_regressors = np.array(Y_bp)
np.savetxt(ofd, nuisance_regressors, fmt='%.18f',
delimiter='\t')

return bandpassed_file, regressor_bandpassed_file


Expand Down Expand Up @@ -163,4 +172,4 @@ def afni_1dBandpass(in_file, highpass, lowpass, tr=1):
tr, highpass, lowpass, in_file, out_file)
os.system(cmd)

return out_file
return out_file
229 changes: 165 additions & 64 deletions CPAC/nuisance/nuisance.py

Large diffs are not rendered by default.

23 changes: 18 additions & 5 deletions CPAC/pipeline/cpac_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,8 @@
erode_mask_boldCSF,
erode_mask_boldGM,
erode_mask_boldWM,
nuisance_regression_template
nuisance_regression_template,
ingress_regressors
)

from CPAC.surface.surf_preproc import surface_postproc
Expand Down Expand Up @@ -867,7 +868,8 @@ def build_anat_preproc_stack(rpool, cfg, pipeline_blocks=None):
pipeline_blocks = []

# T1w Anatomical Preprocessing
if not rpool.check_rpool('desc-reorient_T1w'):
if not rpool.check_rpool('desc-reorient_T1w') and \
not rpool.check_rpool('desc-preproc_T1w'):
anat_init_blocks = [
anatomical_init
]
Expand Down Expand Up @@ -1303,9 +1305,19 @@ def build_workflow(subject_id, sub_dict, cfg, pipeline_name=None,
erode_mask_boldCSF,
erode_mask_boldGM,
erode_mask_boldWM]
nuisance += nuisance_masks + choose_nuisance_blocks(cfg, generate_only)
nuisance += nuisance_masks + choose_nuisance_blocks(cfg, rpool, \
generate_only)

pipeline_blocks += nuisance

pipeline_blocks.append(ingress_regressors)

# Check for mutually exclusive options
if cfg.nuisance_corrections['2-nuisance_regression']['ingress_regressors'] and \
cfg.nuisance_corrections['2-nuisance_regression']['create_regressors']:
err_msg = "[!] Ingress_regressors and create_regressors can't both run! " \
" Try turning one option off.\n "
raise Exception(err_msg)

apply_func_warp = {}
_r_w_f_r = cfg.registration_workflows['functional_registration']
Expand Down Expand Up @@ -1392,8 +1404,9 @@ def build_workflow(subject_id, sub_dict, cfg, pipeline_name=None,
'nuisance_corrections', '2-nuisance_regression', 'space'
] and not generate_only
if nuisance_template:
pipeline_blocks += [(nuisance_regression_template,
("desc-preproc_bold", "desc-stc_bold"))]
pipeline_blocks += [nuisance_regression_template]
# pipeline_blocks += [(nuisance_regression_template,
# ("desc-preproc_bold", "desc-stc_bold"))]

# PostFreeSurfer and fMRISurface
if not rpool.check_rpool('space-fsLR_den-32k_bold.dtseries'):
Expand Down
Loading