Copy build log and artifacts to a permanent location after failures

[gkaf89]

The files can be build in some selected build path (--buildpath), and the logs of successful compilation are then concentrated to some other location for permanent storage (--logfile-format). Logs of failed builds remain in the build path location so that they can be inspected.

However, this setup is problematic when building software in HPC jobs. Quite often in HPC systems the build path is set to some fast storage local to the node, like NVME raid mounted on /tmp or /dev/shm (as suggested in the documentation: https://docs.easybuild.io/configuration/#buildpath). The node storage is often wiped out after the end of a job, so the log files and the artifacts are no longer available after the termination of the job.

This commit adds an option to accumulate errors in some more permanent location, so they can be easily inspected after a failed build.

[gkaf89]

I am not sure what is the best way to select the build directory so that I can move it to a more permanent location. That is at the moment I am recreating the location of the build path and then copy the directory to the destination path:

source_build_path = os.path.join(buildpath, name, version, toolchain)
dest_build_path = os.path.join(err_log_path, name, version, toolchain)
copy_dir(source_build_path, dest_build_path)

• Is there some variable holding the build path, or even the relative build path (i.e. os.path.join(name, version, toolchain))?
• Should we extract this functionality to a module?

[boegel]

@gkaf89 The builddir variable that is set in each easyblock instance hold the path to the build directory for that particular easyconfig.
You can determine the relative path via the build_path() function that is available from easybuild.tools.config, that should report the top directory that corresponds to the buildpath EasyBuild configuration option (see also https://docs.easybuild.io/configuration/#buildpath).

For, for example, for example-1.2.3-GCC-12.3.0.eb, the builddir path would be something like /tmp/myuser/easybuild/build/example/1.2.3/GCC-12.3.0/, with buildpath set to /tmp/myuser/easybuild/build.
Not that the actual build directory in which the compilation is being done would be one level deeper, corresponding to the unpacked source tarball, so something like /tmp/myuser/easybuild/build/example/1.2.3/GCC-12.3.0/example-1.2.3/.

So, I think you could create a subdirectory in the permanent storage location that uses the name of the easyconfig file (to keep it simple), and copy the contents of builddir in there.
You do somehow want to make sure that the target path is unique though, because you could have multiple builds ongoing on different nodes that would all copy to the same permanent location in the end...

[akesandgren]

@gkaf89 The builddir variable that is set in each easyblock instance hold the path to the build directory for that particular easyconfig. You can determine the relative path via the build_path() function that is available from easybuild.tools.config, that should report the top directory that corresponds to the buildpath EasyBuild configuration option (see also https://docs.easybuild.io/configuration/#buildpath).

For, for example, for example-1.2.3-GCC-12.3.0.eb, the builddir path would be something like /tmp/myuser/easybuild/build/example/1.2.3/GCC-12.3.0/, with buildpath set to /tmp/myuser/easybuild/build. Not that the actual build directory in which the compilation is being done would be one level deeper, corresponding to the unpacked source tarball, so something like /tmp/myuser/easybuild/build/example/1.2.3/GCC-12.3.0/example-1.2.3/.

So, I think you could create a subdirectory in the permanent storage location that uses the name of the easyconfig file (to keep it simple), and copy the contents of builddir in there. You do somehow want to make sure that the target path is unique though, because you could have multiple builds ongoing on different nodes that would all copy to the same permanent location in the end...

Yeah, the thing to copy should be builddir into a path with the diff of buildpath and builddir based in permanent-storage-location. Just make sure to remove old remnants of that first :-)

[boegel]

@gkaf89 If you need any help with this, do let us know!

[gkaf89]

@boegel The commit is ready. I won't have enough time to familiarize myself with the test framework for the EasyBlockTest class to prepare a test before the next release.

The commit can be tested by modifying the configuration options of some easyconfig that uses the system toolchain to cause a failure. For instance I added the option

configopts = '--some-invalid-option'

in zlib-1.3.1.eb. The result is that the temporary log file in the build directory and the extracted source code are copied in a permanent location.

[boegel]

@gkaf89 There's a problem with the tests, looks like test_toy_build was broken by the changes being made here?
See for example https://github.com/easybuilders/easybuild-framework/actions/runs/10805964835/job/29973948603

[gkaf89]

The failure is caused because the target location for permanent storage is the same as the source location. The steps I am following to resolve the issue:

• add a source/destination check to avoid a hard failure, and
• detect how the source and the destination path end with the same value in the test.

[gkaf89]

@boegel Some edge cases where uncovered by the tests. The latest commit resolves the issue.

I leave it up you if you prefer to move it to version 5. I am not familiar enough with the tests to test the PR extensively.

[boegel]

@gkaf89 As briefly discussed during the conf call today, it would be good if you could add a test (or enhance an existing one, like test_toy_broken) to verify that the added functionality works as intended (and keeps working).

Do let us know if you need any help with that!