Alternative minification encoding

[Lestropie]

In discussing the utility of the NeuroDocker minification process for another project, a prospective alternative utilisation came to mind.

Currently, what happens for a given external dependency (eg. FSL) is:

1. A container is built that contains the full installation of that software.
2. A set of exemplar MRtrix3 scripts is executed to determine which features of that external software may be utilised.
3. The features of that software not utilised are deleted from the container.
4. The remaining features are tarballed and uploaded online.
5. When building the MRtrix3 image, that tarball is downloaded.

What I dislike about this strategy is that we are very explicitly providing parallel downloads of other peoples' softwares.

What we could do instead is:

4. Generate and store in the main repository a text file that encodes which files need to be retained / removed from the installation of that software.
5. When building the MRtrix3 image, retain / remove the relevant files.

This means that if anyone is logging download counts of their software at their end, image build triggers will still count toward that total.

(Note this isn't the only change I want to make w.r.t. containers, but it might be a contribution in the right direction)

[Lestropie]

In looking into #2921, I'd like to promote this to the 3.1.0 milestone. In retrospect I really don't like bundling substantial portions of other people's software, and then having downloads of such that the creators of those softwares can't track, even if it is technically more reliable if it's fully under our control in that way.

@daljit46 is this something you could take on?

We'd need:

• For each external dependency:
  • After generating the minified container, run something like find -print to get a list of all preserved files and directories
  • Add text files containing those lists to the main repository
  • Modify container recipes to install / build the full external dependency, then delete all files / directories that don't appear in the corresponding preservation list
• Soft deprecation of https://github.com/MRtrix3/containers repository:
  • Move content to an eg. containers/ sub-directory, to live alongside the file lists
  • Update README instructions to instead instruct on committing updates to external software versions / file lists rather than uploading tarballs to OSF

[daljit46]

@daljit46 is this something you could take on?

Sure :)