Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CentOS build action sporadically running OOM #324

Closed
mattelser opened this issue Jun 8, 2023 · 0 comments · Fixed by #332
Closed

CentOS build action sporadically running OOM #324

mattelser opened this issue Jun 8, 2023 · 0 comments · Fixed by #332
Assignees
Labels

Comments

@mattelser
Copy link
Contributor

The github build action for CentOS is inconsistently running out of memory during some sort of packman installation stage.
Here's an example output from this failed run

Package successfully installed to /var/tmp/packman/chk/nv-usd/20.08.nv.1.2.2568.0bec9a14-linux64_py37-centos_debug-main
Package 'rtx_plugins' at version '104.2+release.104.2.271-890c40c7-release.890c40c7.tc.linux-x86_64' is missing from local storage.
Downloading from https://d4i3qtqj3r0z5.cloudfront.net/rtx_plugins%40104.2%2Brelease.104.2.271-890c40c7-release.890c40c7.tc.linux-x86_64.7z (1.29 GiB)
No continuous progress report because this is not a proper terminal. Be patient ...
100% (speed 168 MiB/s)
Total of 7.88 seconds
No continuous progress report because this is not a proper terminal. Patience is a virtue ...
Traceback (most recent call last):
  File "/github/workspace/extern/nvidia/tools/repoman/build.py", line 35, in <module>
    run_command()
  File "/github/workspace/extern/nvidia/tools/repoman/build.py", line 31, in run_command
    packmanapi.pull(os.path.join(deps_folder, deps_target), platform=options.platform_target)
  File "/var/tmp/packman/packman-common/6.45.1/packman/utils.py", line 551, in wrapper_message_block
    return func(*args, **kwargs)
  File "/var/tmp/packman/packman-common/6.45.1/packman/main.py", line 1120, in pull
    project_path, platform, postscript, remotes, include_tags, exclude_tags, tokens
  File "/var/tmp/packman/packman-common/6.45.1/packman/main.py", line 1177, in pull_with_variable_file
    args=args,
  File "/var/tmp/packman/packman-common/6.45.1/packman/main.py", line 1338, in pull_dependencies
    packager.install_package(target_filename, install_path)
  File "/var/tmp/packman/packman-common/6.45.1/packman/packager.py", line 320, in install_package
    archive7z.extract_archive_to_folder(package_path, output_folder)
  File "/var/tmp/packman/packman-common/6.45.1/packman/archive7z.py", line 254, in extract_archive_to_folder
    archive.raise_if_insufficient_space_for_decompression(output_folder, uncompressed_size)
  File "/var/tmp/packman/packman-common/6.45.1/packman/archive.py", line 104, in raise_if_insufficient_space_for_decompression
    utils.raise_if_less_space(output_path, uncompressed_archive_size * 2)
  File "/var/tmp/packman/packman-common/6.45.1/packman/utils.py", line 507, in raise_if_less_space
    f"Not enough free space available at '{abs_path}' - requested {get_pretty_size(num_bytes)} "
packman.errors.PackmanError: Not enough free space available at '/var/tmp/packman/chk/rtx_plugins/ver-n6hjmb2x' - requested 8.42 GiB  but only 8.42 GiB bytes available!
CMake Error at extern/CMakeLists.txt:46 (message):
  Gathering Nvidia libraries failed


-- Configuring incomplete, errors occurred!
See also "/github/workspace/build/CMakeFiles/CMakeOutput.log".

Notably 8.4GiB is larger than the default runner size, so either we have a non-default runner or we're operating in some sort of swap/overhead provided when the runner actually succeeds. If we're not on one of the larger runner sizes, can we sign up for the larger runner size beta?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant