Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build/pkgs/gap: Update to 4.13.1, require >= 4.13 #38169

Closed
wants to merge 15 commits into from

Conversation

mkoeppe
Copy link
Contributor

@mkoeppe mkoeppe commented Jun 7, 2024

📝 Checklist

  • The title is concise and informative.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation and checked the documentation preview.

⌛ Dependencies

Copy link

github-actions bot commented Jun 7, 2024

Documentation preview for this PR (built with commit ebee948; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

The automatically run incremental builds do not work well because of the soname change.

Building from scratch: https://github.com/mkoeppe/sage/actions/runs/9425219803, https://github.com/mkoeppe/sage/actions/runs/9425219809

@mkoeppe mkoeppe self-assigned this Jun 8, 2024
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

The Linux tests are OK, but on the macOS platforms, we see
(e.g. https://github.com/mkoeppe/sage/actions/runs/9425219809/job/25969750545#step:10:3687)

  [sagemath_doc_html-none]   [spkg-install]     from sage.libs.gap.libgap import libgap
  [sagemath_doc_html-none]   [spkg-install] ImportError: dlopen(/Users/runner/work/sage/sage/src/sage/libs/gap/libgap.cpython-39-darwin.so, 0x0002): Library not loaded: 'libgap.9.dylib'
  [sagemath_doc_html-none]   [spkg-install]   Referenced from: '/Users/runner/work/sage/sage/src/sage/libs/gap/libgap.cpython-39-darwin.so'
  [sagemath_doc_html-none]   [spkg-install]   Reason: tried: 'libgap.9.dylib' (no such file), '/usr/local/lib/libgap.9.dylib' (no such file), '/usr/lib/libgap.9.dylib' (no such file), '/Users/runner/work/sage/sage/src/doc/libgap.9.dylib' (no such file), '/usr/local/lib/libgap.9.dylib' (no such file), '/usr/lib/libgap.9.dylib' (no such file)

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

I also see this in a local build on macOS.
@culler Could this be an rpath problem?

@culler
Copy link
Contributor

culler commented Jun 8, 2024

@mkoeppe Yes, it looks like a missing or incorrect rpath. Since you have a local build, you could do this:

  • Download and install macher
  • run macher info sage/src/sage/libs/gap/libgap.cpython-39-darwin.so
    That will show you which libraries libgap needs, and where it is looking for them. The load paths that it tries are obtained by substituting each of its rpaths for the @rpath variable in the load command.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

Thanks! I get

$ macher info src/sage/libs/gap/libgap.cpython-312-darwin.so
Slice: 0
Filetype: MH_BUNDLE
Architecture: x86_64
Load Commands:
    LC_SEGMENT_64 __TEXT [0 : 110592]
    LC_SEGMENT_64 __DATA_CONST [110592 : 114688]
    LC_SEGMENT_64 __DATA [114688 : 118784]
    LC_SEGMENT_64 __LINKEDIT [118784 : 165680]
    LC_DYLD_INFO_ONLY
    LC_SYMTAB: offset is 122544, 1039 symbols, strings in [140160 : 165680]
    LC_DYSYMTAB
    LC_UUID: 402330AF-4268-39AD-BC64-FD0A50F99343
    LC_BUILD_VERSION: min = 14.0.0, sdk = 14.4.0
    LC_SOURCE_VERSION: 0.0.0.0.0
    LC_LOAD_DYLIB: libgap.9.dylib
    LC_LOAD_DYLIB: /usr/local/opt/gmp/lib/libgmp.10.dylib
    LC_LOAD_DYLIB: /usr/lib/libSystem.B.dylib
    LC_RPATH: /Users/mkoeppe/s/sage/sage-rebasing/worktree-pristine/local/lib
    LC_FUNCTION_STARTS
    LC_DATA_IN_CODE

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

Looks to me like the libgap.9.dylib has been installed without properly setting the install_name

$ otool -L local/lib/libgap.9.dylib
local/lib/libgap.9.dylib:
	libgap.9.dylib (compatibility version 10.0.0, current version 10.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.120.2)
	/usr/local/opt/gmp/lib/libgmp.10.dylib (compatibility version 16.0.0, current version 16.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.12)
	/usr/local/opt/readline/lib/libreadline.8.dylib (compatibility version 8.2.0, current version 8.2.0)

@culler
Copy link
Contributor

culler commented Jun 8, 2024

The error seems to be that
LC_LOAD_DYLIB: libgap.9.dylib
should be
LC_LOAD_DYLIB: @rpath/libgap.9.dylib

So the problem is not that the rpath is missing, the load command itself is wrong. (This is in the python extension module, libgap.so, not in libgap.dylib). Adding an -rpath option won't help.

You could repair the load path with macher or install_name_tool in a post-build script. But it would be better if you could get the linker to use the correct load command for libgap when building the extension module.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

Thanks! I agree

@culler
Copy link
Contributor

culler commented Jun 8, 2024

I am not sure what you mean by the install name. I think you might mean the LC_ID_DYLIB load command. Running macher info on libgap.dylib in SageMath 10.3, I see:
LC_ID_DYLIB: /private/var/tmp/sage-10.3-current/local/lib/libgap.8.dylib
As far as I know, that is set by the linker to whatever filename is being written by the linker. And also as far as I know it is not used for anything and would not cause a library not to load if it differs from the actual path or does not exist. You could check, nonetheless, and see what you get on your local build.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

Yes, same thing, as per man ld:

   Options when creating a dynamic library (dylib)
     -install_name name
             Sets an internal "install path" (LC_ID_DYLIB) in a dynamic library. Any clients linked against the library will record that path as the way dyld should
             locate this library.  If this option is not specified, then the -o path will be used.  This option is also called -dylib_install_name for compatibility.

@culler
Copy link
Contributor

culler commented Jun 8, 2024

I think the rpath may be added when the linker sees a -L command and when it actually links with a library in that directory. Can you find the link command used for libgap.cpython-312-darwin.so.

Also I don't understand the path src/sage/libs/gap/libgap.cpython-312-darwin.so -- why isn't that library in the venv?
That is probably just one of those mysteries of the sage build process, though.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

I don't understand the path src/sage/libs/gap/libgap.cpython-312-darwin.so -- why isn't that library in the venv?

We are using an editable build

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

@culler
Copy link
Contributor

culler commented Jun 8, 2024

Hmmm. The ld man page does suggest that if there were an install_name in libgap.so then it would have added the @rpath to the load command.

You can use macher set_id to add an LC_ID_DYLIB load command. You could try that on libgap.so and rebuild the extension module to see if it fixes the problem.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

Yes, it's a fix that I've had to make several times in past, basically for every package that tries to build a shared library without using libtool...

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 8, 2024

It works now. Thanks a lot for the discussion @culler

@mkoeppe mkoeppe added this to the sage-10.4 milestone Jun 8, 2024
@antonio-rojas
Copy link
Contributor

Works for me with system GAP, modulo the already known segfaults in sage/libs/gap/element.pyx with GCC 14 (which I assume are independent of the GAP version)

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 9, 2024

Thank you!

vbraun pushed a commit to vbraun/sage that referenced this pull request Jun 10, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

- Based on sagemath#37884 by @tornaria
- Fixes sagemath#37616

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->

- Depends on sagemath#38144 (merged here for testing)
    
URL: sagemath#38169
Reported by: Matthias Köppe
Reviewer(s):
In gap 4.13 there are some improvements, e.g. converting fp groups to
permutation groups, computing abelianization of fp groups, which lead to
different generators.

This commit fixes doctests so they pass using gap 4.13.
@kiwifb
Copy link
Member

kiwifb commented Aug 5, 2024

Needs rebasing against 10.5b1. sage/algebras/fusion_rings/fusion_double.py had some small changes that prevent merging this PR cleanly. Guilty party #38155.

@kiwifb
Copy link
Member

kiwifb commented Aug 6, 2024

I still have a couple of doctests failure with gap 4.13.1 after applying this PR (once corrected for #38155). Or at least I would think they are related to gap 4.13.1

sage -t --long --random-seed=321973423627906963011264308926181188136 /usr/lib/python3.12/site-packages/sage/groups/cubic_braid.py
**********************************************************************
File "/usr/lib/python3.12/site-packages/sage/groups/cubic_braid.py", line 285, in sage.groups.cubic_braid.CubicBraidElement._richcmp_
Failed example:
    sorted(CBG3)    # indirect doctest
Expected:
    [(c0*c1^-1)^2, c0*c1^-1*c0, c0^-1*c1*c0^-1, c0^-1*c1^-1*c0,
     c1*c0^-1*c1, c0^-1*c1^-1*c0^-1, c0^-1*c1^-1, c1^-1*c0*c1^-1,
     c0*c1^-1*c0^-1, c0^-1*c1, c0^-1*c1*c0, c0*c1^-1, c1*c0^-1,
     c1^-1*c0^-1, c1^-1*c0, c1*c0, c0^-1, c0*c1*c0^-1, c0*c1*c0,
     c0, c0*c1, c1^-1, c1, 1]
Got:
    [(c0^-1*c1)^2,
     c0*c1^-1*c0,
     c0^-1*c1*c0^-1,
     c0^-1*c1^-1*c0,
     c1*c0^-1*c1,
     c0^-1*c1^-1*c0^-1,
     c0^-1*c1^-1,
     c0*c1*c0^-1*c1,
     c0*c1^-1*c0^-1,
     c0^-1*c1,
     c0^-1*c1*c0,
     c0*c1^-1,
     c1*c0^-1,
     c1^-1*c0^-1,
     c1^-1*c0,
     c1*c0,
     <[ [ 1, -1 ] ]|c0^-1>,
     c0*c1*c0^-1,
     c0*c1*c0,
     <[ [ 1, 1 ] ]|c0>,
     c0*c1,
     c1^-1,
     c1,
     1]
**********************************************************************
1 item had failures:

and

File "/usr/lib/python3.12/site-packages/sage/combinat/matrices/latin.py", line 2447, in sage.combinat.matrices.latin.p3_group_bitrade_generators
Failed example:
    p3_group_bitrade_generators(3)
Expected:
    ((2,6,7)(3,8,9),
     (1,2,3)(4,7,8)(5,6,9),
     (1,9,2)(3,7,4)(5,8,6),
     Permutation Group with generators [(2,6,7)(3,8,9), (1,2,3)(4,7,8)(5,6,9)])
Got:
    ((4,5,6)(8,10,12)(11,15,14),
     (1,2,3)(7,8,11)(9,12,15)(10,14,13),
     (1,3,2)(4,6,5)(7,14,8)(9,11,12)(10,13,15),
     Permutation Group with generators [(4,5,6)(8,10,12)(11,15,14), (1,2,3)(7,8,11)(9,12,15)(10,14,13)])
**********************************************************************
1 item had failures:

I guess it could be down to the presence or absence or wrong version of particular gap packages, it would be nice to get to the bottom of it.

@kiwifb
Copy link
Member

kiwifb commented Aug 6, 2024

I notice that #37884 was marking those two failures that I observe as random, but these changes have not been included in this PR.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 9, 2024

Merged the latest version of #37884

Copy link
Member

@kiwifb kiwifb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 10, 2024

Thank you!

@vbraun
Copy link
Member

vbraun commented Aug 24, 2024

I'm getting the same failures in src/sage/categories/simplicial_sets.py as the CI

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 24, 2024

@tornaria @kiwifb Do you have a fix for these somewhere already?

@enriqueartal
Copy link
Contributor

I took a look at the failures in src/sage/categories/simplicial_sets.py. It seems they are all related to distinct orderings in sets of generators, as I think it happens for other tests. Is it not enough to change the output of the tests?

@kiwifb
Copy link
Member

kiwifb commented Sep 1, 2024

OK, so I looked at the CI.

It looks like something is not installing the right gap or rebuilding correctly.

@orlitzky
Copy link
Contributor

I took a look at the failures in src/sage/categories/simplicial_sets.py. It seems they are all related to distinct orderings in sets of generators, as I think it happens for other tests. Is it not enough to change the output of the tests?

The root of this issue is that

sage: H.<e> = FreeGroup()
sage: G = H/[e^2]
sage: [g for g in G]

used to return [1, e], but now it returns [1, <[ [ 1, 1 ] ]|e>].

@enriqueartal
Copy link
Contributor

I took a look at the failures in src/sage/categories/simplicial_sets.py. It seems they are all related to distinct orderings in sets of generators, as I think it happens for other tests. Is it not enough to change the output of the tests?

The root of this issue is that

sage: H.<e> = FreeGroup()
sage: G = H/[e^2]
sage: [g for g in G]

used to return [1, e], but now it returns [1, <[ [ 1, 1 ] ]|e>].

This works in a build not using this branch but develop with system gap in Fedora 40. On the other side I have tests with time outs. For sage/groups/perm_gps/permgroup.py after a while ctrl+c gives this crash. By the way, libgap.so.9 is used.

@orlitzky
Copy link
Contributor

orlitzky commented Oct 9, 2024

The simplicial_sets.py errors persist even when only gapdoc, transgrp, smallgrp, and primgrp are installed :(

@orlitzky
Copy link
Contributor

orlitzky commented Oct 9, 2024

But it PASSES if I additionally install

  • autpgrp
  • alnuth
  • crisp
  • ctbllib
  • factint
  • fga
  • irredsol
  • laguna
  • polenta
  • polycyclic
  • resclasses
  • sophus
  • tomlib

and their dependencies. Time to binary search through the list to find out which one does the magic.

@orlitzky
Copy link
Contributor

orlitzky commented Oct 9, 2024

Ok, it's the polenta package that makes the simplicial sets tests pass. Here are what I see as the remaining work items:

  1. Rebase
  2. Add a feature to src/sage/features/gap.py for polenta
  3. Mark these two simplicial sets tests as # needs gap_package_polenta
  4. Backport Fixes to hash functions for lists gap-system/gap#5796 to the GAP spkg so that the optional GRAPE tests pass

@orlitzky orlitzky mentioned this pull request Oct 12, 2024
vbraun pushed a commit to vbraun/sage that referenced this pull request Oct 20, 2024
    
Follow-up to:

* sagemath#37884
* sagemath#38169

With the four additional work items I mentioned in a comment on the
latter:

1. Everything has been rebased
2. There's a new feature to detect the polenta GAP package
3. The three failing simplicial sets tests have been marked `# needs
gap_package_polenta`
4. I backported gap-system/gap#5796 to the GAP
spkg so that the optional GRAPE tests will pass (untested).

Let's see what the CI has to say...
    
URL: sagemath#38804
Reported by: Michael Orlitzky
Reviewer(s): Dima Pasechnik, Enrique Manuel Artal Bartolo
vbraun pushed a commit to vbraun/sage that referenced this pull request Oct 23, 2024
    
Follow-up to:

* sagemath#37884
* sagemath#38169

With the four additional work items I mentioned in a comment on the
latter:

1. Everything has been rebased
2. There's a new feature to detect the polenta GAP package
3. The three failing simplicial sets tests have been marked `# needs
gap_package_polenta`
4. I backported gap-system/gap#5796 to the GAP
spkg so that the optional GRAPE tests will pass (untested).

Let's see what the CI has to say...
    
URL: sagemath#38804
Reported by: Michael Orlitzky
Reviewer(s): Dima Pasechnik, Enrique Manuel Artal Bartolo
vbraun pushed a commit to vbraun/sage that referenced this pull request Oct 26, 2024
    
Follow-up to:

* sagemath#37884
* sagemath#38169

With the four additional work items I mentioned in a comment on the
latter:

1. Everything has been rebased
2. There's a new feature to detect the polenta GAP package
3. The three failing simplicial sets tests have been marked `# needs
gap_package_polenta`
4. I backported gap-system/gap#5796 to the GAP
spkg so that the optional GRAPE tests will pass (untested).

Let's see what the CI has to say...
    
URL: sagemath#38804
Reported by: Michael Orlitzky
Reviewer(s): Dima Pasechnik, Enrique Manuel Artal Bartolo
@orlitzky
Copy link
Contributor

Done as part of #38804

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Gap 4.13 released, some doctest failures
9 participants