♻️ Split ResourcePool into three classes

[shnizzedy]

(This PR is FCP-INDI#2131 with a different target branch)

Description

Separates https://github.com/FCP-INDI/C-PAC/blob/5ce5d379f8de00ca73032d7beafa28c29f792668/CPAC/pipeline/engine.py#L69-L1413 into three classes: Resource, ResourcePool, and StratPool.

Technical details

• With this refactor, ResourcePools have configs and workflows https://github.com/FCP-INDI/C-PAC/blob/836d100324ff1e91b4cd9cf6eaac17e891471698/CPAC/pipeline/engine/resource.py#L486-L500 https://github.com/FCP-INDI/C-PAC/blob/8d8094131ca709b3a09e7a201c03deb5e6a8c0b9/CPAC/pipeline/engine/resource.py#L525-L528 https://github.com/FCP-INDI/C-PAC/blob/8d8094131ca709b3a09e7a201c03deb5e6a8c0b9/CPAC/pipeline/engine/resource.py#L584-L587 as attributes, and StratPools have configs as attributes https://github.com/FCP-INDI/C-PAC/blob/8d8094131ca709b3a09e7a201c03deb5e6a8c0b9/CPAC/pipeline/engine/resource.py#L3024-L3034 https://github.com/FCP-INDI/C-PAC/blob/8d8094131ca709b3a09e7a201c03deb5e6a8c0b9/CPAC/pipeline/engine/resource.py#L3041 so they workflows and configs don't need to be passed around as much as they have historically. Since this PR is already massive and we're planning substantial engine changes, the only variable-passing efficiencies implemented here are the ones in internal methods.
• This is probably missing a few things, and not all of the mappings are 1:1, but here's roughly where functions/methods/classes moved from/to in this refactor (hopefully this flowchart'll help with merge conflicts)

flowchart LR
subgraph "(before)"
subgraph "cpac.pipeline (before)"
subgraph "cpac_pipeline"
list_blocks.before(list_blocks)
end
subgraph "engine (before)"
initiate_rpool
NodeData
run_node_blocks
set_iterables
strip_template
subgraph "NodeBlock (before)"
NodeBlockFunction.before(NodeBlockFunction)
nodeblock.before(nodeblock)
NodeBlock.__init__.before(__init__)
NodeBlock.get_name(get_name)
check_output.before(check_output)
check_null
NodeBlock.connect_block(connect_block)
grab_tiered_dct
end
subgraph "ResourcePool (before)"
ResourcePool.__init__.before(__init__)
ResourcePool.append_name(append_name)
back_propogate_template_name.before(back_propogate_template_name)
check_rpool.before(check_rpool)
copy_resource
copy_rpool
derivative_xfm
filter_name
filtered_movement
flatten_prov
gather_pipes
generate_prov_list
get.before(get)
get_cpac_provenance
get_data.before(get_data)
get_entire_rpool
get_json
get_json_info.before(get_json_info)
get_name
get_pipe_idxs
get_pipe_number
get_pool_info
set_pool_info
get_raw_label.before(get_raw_label)
get_resource_from_prov
get_resource_strats_from_prov
get_resources
get_strat_info
get_strats
node_data
post_process
regressor_dct.before(regressor_dct)
set_data.before(set_data)
update_resource
end
wrap_block
ingress_raw_anat_data
ingress_freesurfer
ingress_raw_func_data
ingress_output_dir
json_outdir_ingress
func_outdir_ingress
ingress_pipeconfig_paths
end
end
end
subgraph "(after)"
subgraph cpac.pipeline.engine
subgraph nodeblock
nodeblock.after(nodeblock)
NodeBlockFunction
subgraph NodeBlock
NodeBlock.__init__(__init__)
check_output
list_blocks
end
end
subgraph resource
ResourceData
_check_null
resource.set_iterables(set_iterables)
resource.strip_template(strip_template)
subgraph Resource
Resource.get_json(get_json)
data
json
keys
cpac_provenance
end
subgraph _Pool
_Pool.__init__(__init__)
_Pool.check_rpool(check_rpool)
_Pool.get(get)
_Pool.get_resource_from_prov(get_resource_from_prov)
_Pool.keys(keys)
set_data
subgraph ResourcePool
ResourcePool.__init__.after(__init__)
back_propogate_template_name.after(back_propogate_template_name)
ResourcePool.connect_block(connect_block)
ResourcePool.derivative_xfm(derivative_xfm)
ResourcePool.gather_pipes(gather_pipes)
ResourcePool.get(get)
ResourcePool.get_data(get_data)
ResourcePool.get_json(get_json)
get_json_info.after(get_json_info)
_get_pipe_number
get_raw_label.after(get_raw_label)
ResourcePool.get_resource_strats_from_prov(get_resource_strats_from_prov)
ResourcePool.get_strats(get_strats)
ResourcePool.ingress_raw_anat_data(ingress_raw_anat_data)
ResourcePool.ingress_freesurfer(ingress_freesurfer)
ResourcePool.ingress_raw_func_data(ingress_raw_func_data)
ResourcePool.ingress_output_dir(ingress_output_dir)
ResourcePool.json_outdir_ingress(json_outdir_ingress)
ResourcePool.func_outdir_ingress(func_outdir_ingress)
ResourcePool.post_process(post_process)
ResourcePool.ingress_pipeconfig_paths(ingress_pipeconfig_paths)
end
subgraph StratPool
StratPool.__init__(__init__)
StratPool.append_name(append_name)
StratPool.copy_resource(copy_resource)
StratPool.filter_name(filter_name)
StratPool.filtered_movement(filtered_movement)
StratPool.get(get)
StratPool.get_cpac_provenance(get_cpac_provenance)
StratPool.get_data(get_data)
StratPool.get_json(get_json)
regressor_dct
end
end
end
end
end
ResourcePool.__init__.before --> _Pool.__init__
_Pool.__init__ -.-> ResourcePool.__init__.after
_Pool.__init__ -.-> StratPool.__init__
back_propogate_template_name.before --> back_propogate_template_name.after
ResourcePool.append_name --> StratPool.append_name
check_rpool.before --> _Pool.check_rpool
copy_resource --> StratPool.copy_resource
derivative_xfm --> ResourcePool.derivative_xfm
filter_name --> StratPool.filter_name
filtered_movement --> StratPool.filtered_movement
gather_pipes --> ResourcePool.gather_pipes(gather_pipes)
get_cpac_provenance --> StratPool.get_cpac_provenance
get_data.before --> ResourcePool.get_data
get_data.before --> StratPool.get_data
get_json --> Resource.get_json
get_json --> ResourcePool.get_json
get_json --> StratPool.get_json
get_json_info.before --> get_json_info.after
get_pipe_number --> _get_pipe_number
get_resources --> _Pool.keys
get_raw_label.before --> get_raw_label.after
get_resource_strats_from_prov --> ResourcePool.get_resource_strats_from_prov
get_resource_from_prov --> _Pool.get_resource_from_prov
get_strats --> ResourcePool.get_strats
node_data --> DELETED
post_process --> ResourcePool.post_process
regressor_dct.before --> regressor_dct
set_data.before --> set_data
copy_rpool --> DELETED
get.before --> _Pool.get
_Pool.get -.-> ResourcePool.get
_Pool.get -.-> StratPool.get
get_name --> DELETED("🗑️")
flatten_prov --> DELETED
generate_prov_list --> DELETED
get_pipe_idxs --> DELETED
get_pool_info --> DELETED
set_pool_info --> DELETED
get_entire_rpool --> DELETED
get_strat_info --> DELETED
update_resource --> DELETED
NodeBlock.__init__.before --> NodeBlock.__init__
NodeBlock.get_name --> DELETED
check_null --> _check_null
grab_tiered_dct --> DELETED
NodeBlock.connect_block --> ResourcePool.connect_block
wrap_block --> DELETED
ingress_raw_anat_data --> ResourcePool.ingress_raw_anat_data
ingress_freesurfer --> ResourcePool.ingress_freesurfer(ingress_freesurfer)
ingress_raw_func_data --> ResourcePool.ingress_raw_func_data(ingress_raw_func_data)
ingress_output_dir --> ResourcePool.ingress_output_dir(ingress_output_dir)
json_outdir_ingress --> ResourcePool.json_outdir_ingress(json_outdir_ingress)
func_outdir_ingress --> ResourcePool.func_outdir_ingress(func_outdir_ingress)
set_iterables --> resource.set_iterables
strip_template --> resource.strip_template
initiate_rpool --> ResourcePool.__init__.after
run_node_blocks --> DELETED
NodeData --> ResourceData
NodeBlockFunction.before --> NodeBlockFunction
nodeblock.before --> nodeblock.after
check_output.before --> check_output
list_blocks.before --> list_blocks

Loading

Tests

I converted these tests to pytest tests and turned them back on: https://github.com/FCP-INDI/C-PAC/blob/41f15ce50de05dca71a92dc96bf86df0c59bc83c/CPAC/pipeline/test/test_engine.py#L45-L101

Screenshots

Checklist

• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the develop branch of the repository.
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I updated tests for the changes I made.
• I updated the changelog.
• I added or updated documentation (📝 Document "♻️ Split ResourcePool into three classes" FCP-INDI/fcp-indi.github.io#325).
• I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[sgiavasis]

Great. Thanks for the rebase. Merging this in first as it is more congruent with the codebase as-is.