🔬 Automate heatmap generation

[shnizzedy]

Related problem

Automatic heatmaps / leaderboards generation is specific to frmiprep-options

Proposed feature

Simpler C-PAC-only automatic heatmaps that generate every time

Acceptance criteria

• ♻️ Heatmap filematching is way too complicated and should be simplified CPAC_regtest_pack#2
• I think 'heatmap' is too vague and I don't understand how 'leaderboard' describes these heatmaps (but maybe that's just me). I imagine these will be called *_desc-«something»_quality.png ― the question is what do we call them?
• In its current form the table compares intermediates between two runs. Design questions here:
  • Do we want to gather up these intermediates and output them (configurable on/off)? Maybe all together in a big pickle 🥒?
  • ✅ Which intermediates do we want? How flexible can we be with this? #1760
  • 👔 What do we want to include in the heatmap every run? #1759
  • Do we want a util that compiles a multi-subject heatmap in FCP-INDI/CPAC_regtest_pack or FCP-INDI/cpac? Related to the 🥒 question above, how do we want to store any necessary data that isn't otherwise part of the output directory?

Alternatives

no response

Additional context

no response

[shnizzedy]

sgiavasis/CPAC_regtest_pack#1 may resolve this, or we might require more. Rather than a symmetric matrix (like above), this code creates a subject-sessions × features matrix like

The available features are defined in configs/features.py, and simply defining a key / naming convention in that file should make that key available through the entire workflow.

[sgiavasis]

I pulled in the changes: sgiavasis/CPAC_regtest_pack#1

Next is to plan if/how to integrate into the main C-PAC code base, or if we want to maintain a separate dev kit.

[shnizzedy]

MVP

This collection of pull requests won't address everything in this issue, but they will afford a working collection of automations that

1. Run a small set of end-to-end workflows on downsampled (5mm isometric) data (integration tests) on our HPC when a PR is opened or updated
2. Calculate correlation statistics against those same workflows run via the previous C-PAC release on the same data (regression tests)
3. Push those correlation statistics to GitHub
4. Build visualizations (some interactive) and push those visualizations in a comment to the relevant PR or commit when these tests are run

These PRs should be merged in this order:

1. 👽️ Remove deprecated xlink XML prefix C-PAC_regression_dashboard#2
2. 👔 Move creation of correlations.json to cpac-correlations CLI CPAC_regtest_pack#12
3. 👔 Build interactive SVG, build a PNG from that interactive SVG, include the PNG in automated comment C-PAC_regression_dashboard#3
4. 👔 Restructure automated integration and regression testing flow childmindresearch/slurm_testing#2
5. 🔖 Set canonincal dependencies + initial version and CHANGELOG C-PAC_regression_dashboard#4
6. 🔖 Set canonincal dependencies + initial version and CHANGELOG childmindresearch/slurm_testing#3
7. 👷 Automate lite integration and regression tests #2162

Future improvements

• full (non-downsampled) tests
• tests for preconfigs other than default
• expanded test subject set
• fix / avoid crashes
• Move non-SLURM dependency management to GitHub Actions (build a check suite for integration runs, trigger regression tests once that check suite completes)
• Abbreviate paths in correlations table
• Purge dead code