Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opencv4: cuda: fix libstdc++ errors downstream, plus upkeep #265070

Merged
merged 5 commits into from
Nov 13, 2023

Conversation

SomeoneSerge
Copy link
Contributor

@SomeoneSerge SomeoneSerge commented Nov 2, 2023

Description of changes

Closes #258327

Adds a test for and fixes the following libstdc++ mismatch error:

python3.11-pytlsd> ==================================== ERRORS ====================================
python3.11-pytlsd> _______________________ ERROR collecting tests/tests.py ________________________
python3.11-pytlsd> ImportError while importing test module '/build/source/tests/tests.py'.
python3.11-pytlsd> Hint: make sure your test modules/packages have valid Python names.
python3.11-pytlsd> Traceback:
python3.11-pytlsd> /nix/store/ffll6glz3gwx342z0ch8wx30p5cnqz1z-python3-3.11.5/lib/python3.11/importlib/__init__.py:126: in import_module
python3.11-pytlsd>     return _bootstrap._gcd_import(name[level:], package, level)
python3.11-pytlsd> tests/tests.py:7: in <module>
python3.11-pytlsd>     from skimage.transform import pyramid_reduce
python3.11-pytlsd> /nix/store/zr2pijvixax0say6n6d1s0mg3f970xv2-python3.11-scikit-image-0.21.0/lib/python3.11/site-packages/skimage/transform/__init__.py:38: in <module>
python3.11-pytlsd>     from .radon_transform import (radon, iradon, iradon_sart,
python3.11-pytlsd> /nix/store/zr2pijvixax0say6n6d1s0mg3f970xv2-python3.11-scikit-image-0.21.0/lib/python3.11/site-packages/skimage/transform/radon_transform.py:3: in <module>
python3.11-pytlsd>     from scipy.interpolate import interp1d
python3.11-pytlsd> /nix/store/nab40hl5f2hk4i1x5rlkimmn20dyqh40-python3.11-scipy-1.11.3/lib/python3.11/site-packages/scipy/interpolate/__init__.py:167: in <module>
python3.11-pytlsd>     from ._interpolate import *
python3.11-pytlsd> /nix/store/nab40hl5f2hk4i1x5rlkimmn20dyqh40-python3.11-scipy-1.11.3/lib/python3.11/site-packages/scipy/interpolate/_interpolate.py:12: in <module>
python3.11-pytlsd>     from . import _fitpack_py
python3.11-pytlsd> /nix/store/nab40hl5f2hk4i1x5rlkimmn20dyqh40-python3.11-scipy-1.11.3/lib/python3.11/site-packages/scipy/interpolate/_fitpack_py.py:10: in <module>
python3.11-pytlsd>     from ._bsplines import BSpline
python3.11-pytlsd> /nix/store/nab40hl5f2hk4i1x5rlkimmn20dyqh40-python3.11-scipy-1.11.3/lib/python3.11/site-packages/scipy/interpolate/_bsplines.py:9: in <module>
python3.11-pytlsd>     from scipy.optimize import minimize_scalar
python3.11-pytlsd> /nix/store/nab40hl5f2hk4i1x5rlkimmn20dyqh40-python3.11-scipy-1.11.3/lib/python3.11/site-packages/scipy/optimize/__init__.py:422: in <module>
python3.11-pytlsd>     from ._linprog import linprog, linprog_verbose_callback
python3.11-pytlsd> /nix/store/nab40hl5f2hk4i1x5rlkimmn20dyqh40-python3.11-scipy-1.11.3/lib/python3.11/site-packages/scipy/optimize/_linprog.py:21: in <module>
python3.11-pytlsd>     from ._linprog_highs import _linprog_highs
python3.11-pytlsd> /nix/store/nab40hl5f2hk4i1x5rlkimmn20dyqh40-python3.11-scipy-1.11.3/lib/python3.11/site-packages/scipy/optimize/_linprog_highs.py:20: in <module>
python3.11-pytlsd>     from ._highs._highs_wrapper import _highs_wrapper
python3.11-pytlsd> E   ImportError: /nix/store/n0gf2wl3g6229q275r4cxs2yhnam0xvg-gcc-11.4.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/nab40hl5f2hk4i1x5rlkimmn20dyqh40-python3.11-scipy-1.11.3/lib/python3.11/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-311-x86_64-linux-gnu.so)
python3.11-pytlsd> =========================== short test summary info ============================
python3.11-pytlsd> ERROR tests/tests.py
python3.11-pytlsd> !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
python3.11-pytlsd> =============================== 1 error in 0.28s ===============================
python3.11-pytlsd> /nix/store/bbxdw4rgwwl3gnajri82yidr1nlsfskf-stdenv-linux/setup: line 1596: pop_var_context: head of shell_variables not a function context
error: builder for '/nix/store/jfpbrdgvfr2qbac02lml42p1kvh4p1nf-python3.11-pytlsd-0.0.5.drv' failed with exit code 2;

...observed when importing cv2 (built with cuda support and the older nvcc-compatible gcc) before importing a native python extension from nixpkgs (which uses the newer libstdc++) like so: https://github.com/iago-suarez/pytlsd/blob/81e0fee35f1fdaf546ec73c71ad3ed488c8ee83b/tests/tests.py#L3-L7

The test passes for the opencv3 attribute because it's just built with the newer gcc(12). I decided not to touch it further, but my expectation is that opencv3 with CUDA will start failing once we bump gcc.

The tests can be run as follows:

❯ nix build -f . --arg config '{ cudaSupport = true; cudaCapabilities = [ "8.6" ]; enableForwardCompat = false; allowUnfree = true; }' opencv4.tests.no-libstdcxx-errors
❯ nix build -f . --arg config '{ cudaSupport = true; cudaCapabilities = [ "8.6" ]; enableForwardCompat = false; allowUnfree = true; }' opencv3.tests.no-libstdcxx-errors

You can check out the first commit and verify that opencv4.no-libstdcxx-errors fails until it's updated in the next commit

In the longer run we should kill backendStdenv and instead implement #226165

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.11 Release Notes (or backporting 23.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@NixOS/cuda-maintainers

@SomeoneSerge SomeoneSerge added 6.topic: python 6.topic: cuda Parallel computing platform and API labels Nov 2, 2023
@@ -37,7 +37,7 @@
, blas
, enableContrib ? true

, enableCuda ? config.cudaSupport && stdenv.hostPlatform.isx86_64
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I dropped this without thinking. Why do we ignore config on non-x86_64 in the first place?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c4adedd

config.cudaSupport can be true and cudatoolkit doesn't work with i686,
which can happen inside pkgsi686Linux wrapping.

OK, so this wasn't about aarch64 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, why would anyone set config.cudaSupport = true on an i686 device?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should move this to a separate commit though

@SomeoneSerge SomeoneSerge changed the title Fix/opencv cuda opencv4: cuda: fix libstdc++ errors Nov 2, 2023
@SomeoneSerge SomeoneSerge changed the title opencv4: cuda: fix libstdc++ errors opencv4: cuda: fix libstdc++ errors downstream Nov 2, 2023
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild 10.rebuild-linux: 0 This PR does not cause any packages to rebuild labels Nov 2, 2023
@SomeoneSerge SomeoneSerge changed the title opencv4: cuda: fix libstdc++ errors downstream opencv4: cuda: fix libstdc++ errors downstream, plus upkeep Nov 6, 2023
Copy link
Member

@samuela samuela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

diff LGTM!

@SomeoneSerge SomeoneSerge added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Nov 7, 2023
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/1207

@mweinelt mweinelt merged commit 7f5b82d into NixOS:master Nov 13, 2023
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: cuda Parallel computing platform and API 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild 10.rebuild-linux: 0 This PR does not cause any packages to rebuild 12.approvals: 1 This PR was reviewed and approved by one reputable person
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

opencv4 and scipy cause glibc version conflicts with cudaSupport
4 participants