Doctest hide option: Better detection of hidden packages #36741

soehms · 2023-11-20T07:56:32Z

This PR implements the suggestion made in #36696 (comment). This means that it introduces a counter in the feature class to detect the number of events a feature has been asked to be present while it was hidden. This allows to remove the call of the is_present method in doctest/control.py.

In order to test it I ran sage -tp --all --hide=all. Ideally this should give All tests passed. But I got two failing files. One of those was src/sage/features/databases.py because database_cremona_mini_ellcurve was detected to be optional even thought it is a standard package. This is a leftover of #35820 which I fix here.

Similarily I got 37 failures in src/sage/groups/abelian_gps/abelian_group.py since gap_package_polycyclic was classified optional even though it is a Gap standard package since Gap 4.10 (as well as gap_package_atlasrep). But in this case a corresponding fix, i.e.:

 def all_features():
-    return [GapPackage("atlasrep", spkg="gap_packages"),
+    return [GapPackage("atlasrep", spkg="gap_packages", type='standard'),
             GapPackage("design", spkg="gap_packages"),
             GapPackage("grape", spkg="gap_packages"),
             GapPackage("guava", spkg="gap_packages"),
             GapPackage("hap", spkg="gap_packages"),
-            GapPackage("polycyclic", spkg="gap_packages"),
+            GapPackage("polycyclic", spkg="gap_packages", type='standard'),
             GapPackage("qpa", spkg="gap_packages"),
             GapPackage("quagroup", spkg="gap_packages")]

is not the correct way (it gives UserWarning: Feature gap_package_polycyclic is declared standard, but it is provided by gap_packages, which is declared optional in SAGE_ROOT/build/pkgs). I would say, the correct fix would be to remove both Gap packages from the all_features list and erase the corresponding optional tags in permgroup_named.py, distance_regular.pyx, abelian_aut.py, abelian_group.py and abelian_group_gap.py. If you agree I will open another PR for this.

BTW: there seem to be more packages with ambiguous type (detected with current Docker image):

Digest: sha256:790197bb223bd7e20b0a2e055aa7b4c5846dc86d94b2425cd233cb6160a5b944
Status: Downloaded newer image for sagemath/sagemath:develop
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 10.2.rc4, Release Date: 2023-11-17                │
│ Using Python 3.11.1. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: from sage.features.all import all_features
sage: [(f, f._spkg_type()) for f in all_features() if f.is_present() and f._spkg_type() != 'standard']
[(Feature('sage.misc.cython'), 'optional'),
 (Feature('database_cremona_mini_ellcurve': Cremona's database of elliptic curves),
  'optional'),
 (Feature('gap_package_atlasrep'), 'optional'),
 (Feature('gap_package_polycyclic'), 'optional'),
 (Feature('sage.libs.ecl'), 'optional'),
 (Feature('sage.libs.gap'), 'optional'),
 (Feature('sage.libs.singular'), 'optional'),
 (Feature('sage.numerical.mip'), 'optional')]
sage:
sage: [(f, f._spkg_type()) for f in all_features() if not f.is_present() and f._spkg_type() == 'standard']
[(Feature('sagemath_doc_html'), 'standard'), (Feature('sage.sat'), 'standard')]

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

mkoeppe · 2024-01-09T22:09:24Z

I would say, the correct fix would be to remove both Gap packages from the all_features list and erase the corresponding optional tags in permgroup_named.py, distance_regular.pyx, abelian_aut.py, abelian_group.py and abelian_group_gap.py. If you agree I will open another PR for this.

I agree

soehms · 2024-02-04T21:42:00Z

I would say, the correct fix would be to remove both Gap packages from the all_features list and erase the corresponding optional tags in permgroup_named.py, distance_regular.pyx, abelian_aut.py, abelian_group.py and abelian_group_gap.py. If you agree I will open another PR for this.

I agree

See #37151.

tornaria · 2024-02-20T11:13:45Z

Would something like this work?

 def all_features():
-    return [GapPackage("atlasrep", spkg="gap_packages"),
+    return [GapPackage("atlasrep", spkg="gap", type='standard'),
             GapPackage("design", spkg="gap_packages"),
             GapPackage("grape", spkg="gap_packages"),
             GapPackage("guava", spkg="gap_packages"),
             GapPackage("hap", spkg="gap_packages"),
-            GapPackage("polycyclic", spkg="gap_packages"),
+            GapPackage("polycyclic", spkg="gap", type='standard'),
             GapPackage("qpa", spkg="gap_packages"),
             GapPackage("quagroup", spkg="gap_packages")]

src/sage/features/join_feature.py

src/sage/features/databases.py

mkoeppe · 2024-03-10T20:07:08Z

src/sage/features/__init__.py

@@ -802,7 +808,7 @@ class StaticFile(FileFeature):
        To install no_such_file...you can try to run...sage -i some_spkg...
        Further installation instructions might be available at http://rand.om.
    """
-    def __init__(self, name, filename, *, search_path=None, **kwds):
+    def __init__(self, name, filename, search_path=None, type='optional', **kwds):


the * there was there intentionally. It marks the following parameters as keyword-only (cannot be passed as positional)

Thanks! I will do this later on.

This id done, now! I took the occasion to also improve some doctest tags concerning the Gap package Grape which I came across.

src/sage/features/__init__.py

Co-authored-by: Matthias Köppe <mkoeppe@math.ucdavis.edu>

github-actions · 2024-03-11T22:06:43Z

Documentation preview for this PR (built with commit 03670bd; changes) is ready! 🎉

mkoeppe

LGTM, thanks!

soehms · 2024-03-18T18:57:17Z

LGTM, thanks!

Thanks, as well!

tornaria · 2024-03-29T14:11:57Z

@soehms: this

+        # For multiprocessing of doctests, the data self._num_hidings should be
+        # shared among subprocesses. Thus we use the Value class from the
+        # multiprocessing module (cf. self._seen of class AvailableSoftware)
+        from multiprocessing import Value
+        self._num_hidings = Value('i', 0)

breaks on musl libc. It seems there is a limit on how many RLocks one can create (?). This results in unability to do any testing, see:

https://github.com/void-linux/void-packages/actions/runs/8454977933/job/23161359657?pr=49571#step:7:42494

Running it locally gives the exact same failure. And I can also reproduce with:

$ python
Python 3.12.2 (main, Feb 15 2024, 06:47:30) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Value
>>> l = [Value('i', 0) for _ in range(200)]
>>> l = [Value('i', 0) for _ in range(200)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.12/multiprocessing/context.py", line 135, in Value
    return Value(typecode_or_type, *args, lock=lock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/sharedctypes.py", line 79, in Value
    lock = ctx.RLock()
           ^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/context.py", line 73, in RLock
    return RLock(ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 194, in __init__
    SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 24] No file descriptors available

It seems multiprocessing.Value can receive a lock so maybe it's possible to use the same RLock for all the features? (or even a Lock since the code doesn't seem to recurse)

BTW, the documentation of multiprocessing explicitly warns that self._num_hidings.value += 1 or is not safe (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Value)

tornaria · 2024-03-29T15:02:56Z

Actually this:

$ python
Python 3.12.2 (main, Feb 15 2024, 06:47:30) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from threading import Lock
>>> from multiprocessing import Value
>>> l = [Value('i', 0, lock=Lock()) for _ in range(1000000)]
>>>

works ok (same with RLock), so maybe this can just use one Lock per feature to protect the short section where _num_hidings is incremented?

As a matter of fact, is a lock really necessary? It seems to me that nothing too bad will happen if two threads try to increment at the same time since the value is only used for reporting and not to take any decision (also it seems this is not protected atm so not using a lock should be equivalent to what is done now).

In summary, perhaps

         # For multiprocessing of doctests, the data self._num_hidings should be
         # shared among subprocesses. Thus we use the Value class from the
         # multiprocessing module (cf. self._seen of class AvailableSoftware)
         from multiprocessing import Value
-        self._num_hidings = Value('i', 0)
+        self._num_hidings = Value('i', 0, lock=False)

is the simplest solution, with lock=threading.Lock() (or RLock) an alternative but in that case one should also acquire the lock to protect the sections that read-then-modify the value.

I don't understand why the lock used by default is worse than threading.Lock()... It's also much slower to create, and it also fails on glibc (it just has a higher limit):

$ time python -c 'from multiprocessing import Value; [Value("i", 0) for _ in range(100_000)]'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.12/multiprocessing/context.py", line 135, in Value
    return Value(typecode_or_type, *args, lock=lock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/sharedctypes.py", line 79, in Value
    lock = ctx.RLock()
           ^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/context.py", line 73, in RLock
    return RLock(ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 194, in __init__
    SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 12] Cannot allocate memory

real	0m43.571s
user	0m37.865s
sys	0m5.153s

mkoeppe · 2024-03-29T20:16:44Z

Also on macOS I see new multiprocessing-related issues that may be coming from this PR

mkoeppe · 2024-03-29T20:27:05Z

perhaps

         # For multiprocessing of doctests, the data self._num_hidings should be
         # shared among subprocesses. Thus we use the Value class from the
         # multiprocessing module (cf. self._seen of class AvailableSoftware)
         from multiprocessing import Value
-        self._num_hidings = Value('i', 0)
+        self._num_hidings = Value('i', 0, lock=False)

is the simplest solution,

Can confirm that this fixes the problem on macOS (repro: just ./sage -fixdoctests --help).

Overall, I would suggest to remove the direct and unconditional use of multiprocessing from sage.features.
Perhaps sage.doctest can put such Value attributes into features that are to be hidden.

soehms · 2024-03-30T11:56:57Z

Sorry for causing trouble again, especially that I missed the warning in the documentation!

As a matter of fact, is a lock really necessary?

I agree that this is not necessary. In addition, there even is no need to increase the value, since I finally refrained from printing the exact number of hidings.

Unfortunately, I cannot work on this before Tueseday. If you don't want to wait, feel free to start on it.

tornaria · 2024-03-30T15:44:37Z

As a matter of fact, is a lock really necessary?

I agree that this is not necessary. In addition, there even is no need to increase the value, since I finally refrained from printing the exact number of hidings.

So, shall we get rid of the _num_hidings attribute altogether? (in fact, the self._seen is used as a shared cache, so the feature is_present()` method will be called only once (unless there's a very precise timing between two threads).

I looked a little bit into it and here's an idea:

In features, implement a simple hide() / unhide() / is_hidden() interface.
The only state is _hidden, not shared (for parallel doctesting this will be set before fork so it's ok)
Implement AvailableSoftware.hidden() similar to AvailableSoftware.seen() so the logic stays in that class and internals don't leak to sage.doctest.control (if necessary use _seen[idx] to record hidden state and/or number of hidings, or add another shared array)

Unfortunately, I cannot work on this before Tueseday. If you don't want to wait, feel free to start on it.

I could do something like I described above, if you think it's ok, but I'd still wait for you to review it.

Meanwhile, the lock=False workaround seems good enough.

soehms · 2024-03-30T18:39:09Z

I could do something like I described above, if you think it's ok, but I'd still wait for you to review it.

Sounds good! Any solution you find that preserves the current behavior and improves the existing implementation is good for me. Many thanks!

This affects musl libc and macos, see: sagemath#36741 (comment) We are working on a better fix to avoid using multiprocessing in sage.features, but this will do meanwhile. There is agreement that disabling the lock is completely harmless, since the counter is not really used for anything; disabling the lock fixes the issues that we had on 10.4.beta0.

This affects musl libc and macos, see: sagemath#36741 (comment) We are working on a better fix to avoid using multiprocessing in sage.features (cf sagemath#36741 (comment)), but this will do meanwhile. There is agreement that disabling the lock is completely harmless, since the counter is not really used for anything; disabling the lock fixes the issues that we had on 10.4.beta0. ### 📝 Checklist - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. URL: sagemath#37702 Reported by: Gonzalo Tornaría Reviewer(s): Sebastian Oehms

antonio-rojas · 2024-05-03T16:48:14Z

This breaks tests in doctest/control.py if meataxe is installed

sage -t --long --random-seed=192739335773734784141473037713226530622 /usr/lib/python3.12/site-packages/sage/doctest/control.py
**********************************************************************
File "/usr/lib/python3.12/site-packages/sage/doctest/control.py", line 1453, in sage.doctest.control.DocTestController.run
Failed example:
    DC.run()
Expected:
    Running doctests with ID ...
    Using --optional=sage...
    Features to be detected: ...
    Doctesting 1 file.
    sage -t ....py
        [4 tests, ... s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: ... seconds
        cpu time: ... seconds
        cumulative wall time: ... seconds
    Features detected...
    0
Got:
    Running doctests with ID 2024-05-03-18-34-37-b15fc869.
    Running with SAGE_LOCAL='/usr' and SAGE_VENV='/usr'
    Using --optional=sage
    Features to be detected: 4ti2,SAGE_SRC,benzene,bliss,buckygen,conway_polynomials,csdp,cvxopt,cvxopt,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_cubic_hecke,database_ellcurves,database_graphs,database_jones_numfield,database_knotinfo,dvipng,ecm,fpylll,fricas,gap_package_atlasrep,gap_package_design,gap_package_grape,gap_package_guava,gap_package_hap,gap_package_polycyclic,gap_package_qpa,gap_package_quagroup,gfan,graphviz,imagemagick,ipython,jmol,jupymake,kenzo,latte_int,lrcalc_python,lrslib,matroid_database,mcqd,meataxe,mpmath,msolve,nauty,networkx,numpy,palp,pandoc,pdf2svg,pdftocairo,pexpect,phitigra,pillow,plantri,polytopes_db,polytopes_db_4d,pplpy,primecountpy,ptyprocess,pynormaliz,pyparsing,python_igraph,requests,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.libs.braiding,sage.libs.ecl,sage.libs.flint,sage.libs.gap,sage.libs.linbox,sage.libs.m4ri,sage.libs.ntl,sage.libs.pari,sage.libs.singular,sage.misc.cython,sage.modular,sage.modules,sage.numerical.mip,sage.plot,sage.rings.complex_double,sage.rings.finite_rings,sage.rings.function_field,sage.rings.number_field,sage.rings.padics,sage.rings.polynomial.pbori,sage.rings.real_double,sage.rings.real_mpfr,sage.sat,sage.schemes,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,scipy,singular,sphinx,symengine_py,sympy,tdlib,threejs
    Doctesting 1 file.
    sage -t --random-seed=0 /tmp/tmpoo6ocsi3/tmp_rn4vfilc.py
        [5 tests, 0.11 s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: 0.1 seconds
        cpu time: 0.1 seconds
        cumulative wall time: 0.1 seconds
    Features detected for doctesting: meataxe,sage.libs.singular,sage.modules,sage.rings.finite_rings,sage.rings.number_field,sage.rings.real_mpfr,sage.symbolic,sympy
    Features that have been hidden: meataxe
    0
**********************************************************************
File "/usr/lib/python3.12/site-packages/sage/doctest/control.py", line 1471, in sage.doctest.control.DocTestController.run
Failed example:
    DC.run()
Expected:
    Running doctests with ID ...
    Using --optional=sage
    Features to be detected: ...
    Doctesting 1 file.
    sage -t ....py
        [4 tests, ... s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: ... seconds
        cpu time: ... seconds
        cumulative wall time: ... seconds
    Features detected...
    0
Got:
    Running doctests with ID 2024-05-03-18-34-38-be855998.
    Running with SAGE_LOCAL='/usr' and SAGE_VENV='/usr'
    Using --optional=sage
    Features to be detected: 4ti2,SAGE_SRC,bliss,conway_polynomials,csdp,cvxopt,cvxopt,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_cubic_hecke,database_ellcurves,database_graphs,database_jones_numfield,database_knotinfo,dvipng,ecm,fpylll,fricas,gap_package_atlasrep,gap_package_design,gap_package_grape,gap_package_guava,gap_package_hap,gap_package_polycyclic,gap_package_qpa,gap_package_quagroup,gfan,graphviz,imagemagick,ipython,jmol,jupymake,kenzo,latte_int,lrcalc_python,lrslib,matroid_database,mcqd,meataxe,mpmath,msolve,nauty,networkx,numpy,palp,pandoc,pdf2svg,pdftocairo,pexpect,phitigra,pillow,plantri,polytopes_db,polytopes_db_4d,pplpy,primecountpy,ptyprocess,pynormaliz,pyparsing,python_igraph,requests,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.libs.braiding,sage.libs.ecl,sage.libs.flint,sage.libs.gap,sage.libs.linbox,sage.libs.m4ri,sage.libs.ntl,sage.libs.pari,sage.libs.singular,sage.misc.cython,sage.modular,sage.modules,sage.numerical.mip,sage.plot,sage.rings.complex_double,sage.rings.finite_rings,sage.rings.function_field,sage.rings.number_field,sage.rings.padics,sage.rings.polynomial.pbori,sage.rings.real_double,sage.rings.real_mpfr,sage.sat,sage.schemes,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,scipy,singular,sphinx,symengine_py,sympy,tdlib,threejs
    Doctesting 1 file.
    sage -t --random-seed=0 /tmp/tmpoo6ocsi3/tmp_rn4vfilc.py
        [5 tests, 0.12 s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: 0.1 seconds
        cpu time: 0.1 seconds
        cumulative wall time: 0.1 seconds
    Features detected for doctesting: meataxe,sage.libs.singular,sage.modules,sage.rings.finite_rings,sage.rings.number_field,sage.rings.real_mpfr,sage.symbolic,sympy
    Features that have been hidden: meataxe
    0
**********************************************************************
File "/usr/lib/python3.12/site-packages/sage/doctest/control.py", line 1489, in sage.doctest.control.DocTestController.run
Failed example:
    DC.run()                              # optional - meataxe
Expected:
    Running doctests with ID ...
    Using --optional=sage
    Features to be detected: ...
    Doctesting 1 file.
    sage -t ....py
        [4 tests, ... s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: ... seconds
        cpu time: ... seconds
        cumulative wall time: ... seconds
    Features detected...
    Features that have been hidden: ...meataxe...
    0
Got:
    Running doctests with ID 2024-05-03-18-34-38-c89eac33.
    Running with SAGE_LOCAL='/usr' and SAGE_VENV='/usr'
    Using --optional=sage
    Features to be detected: 4ti2,SAGE_SRC,bliss,conway_polynomials,csdp,cvxopt,cvxopt,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_cubic_hecke,database_ellcurves,database_graphs,database_jones_numfield,database_knotinfo,dvipng,ecm,fpylll,fricas,gap_package_atlasrep,gap_package_design,gap_package_grape,gap_package_guava,gap_package_hap,gap_package_polycyclic,gap_package_qpa,gap_package_quagroup,gfan,graphviz,imagemagick,ipython,jmol,jupymake,kenzo,latte_int,lrcalc_python,lrslib,matroid_database,mcqd,meataxe,mpmath,msolve,nauty,networkx,numpy,palp,pandoc,pdf2svg,pdftocairo,pexpect,phitigra,pillow,plantri,polytopes_db,polytopes_db_4d,pplpy,primecountpy,ptyprocess,pynormaliz,pyparsing,python_igraph,requests,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.libs.braiding,sage.libs.ecl,sage.libs.flint,sage.libs.gap,sage.libs.linbox,sage.libs.m4ri,sage.libs.ntl,sage.libs.pari,sage.libs.singular,sage.misc.cython,sage.modular,sage.modules,sage.numerical.mip,sage.plot,sage.rings.complex_double,sage.rings.finite_rings,sage.rings.function_field,sage.rings.number_field,sage.rings.padics,sage.rings.polynomial.pbori,sage.rings.real_double,sage.rings.real_mpfr,sage.sat,sage.schemes,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,scipy,singular,sphinx,symengine_py,sympy,tdlib,threejs
    Doctesting 1 file.
    sage -t --random-seed=0 /tmp/tmpoo6ocsi3/tmp_rn4vfilc.py
        [5 tests, 0.12 s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: 0.1 seconds
        cpu time: 0.1 seconds
        cumulative wall time: 0.1 seconds
    Features detected for doctesting: meataxe,sage.libs.singular,sage.modules,sage.rings.finite_rings,sage.rings.number_field,sage.rings.real_mpfr,sage.symbolic,sympy
    Features that have been hidden: meataxe
    0
**********************************************************************
1 item had failures:
   3 of  23 in sage.doctest.control.DocTestController.run
    [230 tests, 3 failures, 1.71 s]
----------------------------------------------------------------------
sage -t --long --random-seed=192739335773734784141473037713226530622 /usr/lib/python3.12/site-packages/sage/doctest/control.py  # 3 doctests failed
----------------------------------------------------------------------
Total time for all tests: 1.8 seconds
    cpu time: 0.2 seconds
    cumulative wall time: 1.7 seconds
Features detected for doctesting: meataxe,sage.libs.singular,sage.modules,sage.rings.finite_rings,sage.rings.number_field,sage.rings.real_mpfr,sage.symbolic,sympy

mkoeppe · 2024-05-03T17:55:40Z

This breaks tests in doctest/control.py if meataxe is installed

See

Doctests failures in sage/doctest/control.py when sagemath-meataxe is installed #37905

soehms · 2024-05-09T16:42:00Z

This breaks tests in doctest/control.py if meataxe is installed

I suspect that in your test you had differences in control.py with the development branch, similar to those in PR #37857 or the patch for the Gentoo distribution. If so, this should be fixed by the suggestion I made at #37857 (comment) or by the fix I've implemented in #37737 (see #37737 (comment) and commit f881e2b). If not, please let me know how I can reproduce it.

antonio-rojas · 2024-05-09T21:33:22Z

I am using an unmodified doctest/control.py. #37737 fixes the doctests here.

From sagemath#36741 (comment) > Overall, I would suggest to remove the direct and unconditional use of multiprocessing from sage.features. > Perhaps sage.doctest can put such Value attributes into features that are to be hidden. This PR implements the suggestion made in sagemath#36741 (comment) > I looked a little bit into it and here's an idea: > > * In features, implement a simple hide() / unhide() / is_hidden() interface. > * The only state is _hidden, not shared (for parallel doctesting this will be set before fork so it's ok) > * Implement AvailableSoftware.hidden() similar to AvailableSoftware.seen() so the logic stays in that class and internals don't leak to sage.doctest.control (if necessary use _seen[idx] to record hidden state and/or number of hidings, or add another shared array) Fixes sagemath#37905 ### 📝 Checklist  - [ ] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies    URL: sagemath#37737 Reported by: Sebastian Oehms Reviewer(s): Matthias Köppe, Sebastian Oehms

fix_doctest_hide_option initial

8d0147f

soehms added this to the sage-10.3 milestone Nov 20, 2023

soehms added the c: doctest framework label Nov 20, 2023

soehms mentioned this pull request Nov 20, 2023

Fix SSL timeout in doctest / internet feature #36696

Merged

2 tasks

soehms requested review from tornaria and mkoeppe November 20, 2023 18:01

soehms added the s: needs review label Nov 20, 2023

soehms and others added 5 commits December 3, 2023 22:31

Merge branch 'sagemath:develop' into fix_doctest_hide_option

fd91a03

Merge branch 'sagemath:develop' into fix_doctest_hide_option

9ada8e3

Merge branch 'develop' into fix_doctest_hide_option

7f5a591

Merge branch 'develop' into fix_doctest_hide_option

b61c3f5

Merge branch 'sagemath:develop' into fix_doctest_hide_option

267dc15

Merge branch 'sagemath:develop' into fix_doctest_hide_option

0f4ebc7

soehms mentioned this pull request Jan 23, 2024

Let standard Gap packages be standard Sage packages #37151

Open

5 tasks

soehms and others added 2 commits January 23, 2024 19:20

Merge branch 'develop' into fix_doctest_hide_option

7728853

Merge branch 'sagemath:develop' into fix_doctest_hide_option

8560fc7

Merge branch 'sagemath:develop' into fix_doctest_hide_option

0718be2

soehms added 2 commits February 25, 2024 21:56

Merge branch 'sagemath:develop' into fix_doctest_hide_option

bca471b

Merge branch 'sagemath:develop' into fix_doctest_hide_option

7083396

mkoeppe reviewed Mar 10, 2024

View reviewed changes

src/sage/features/join_feature.py Outdated Show resolved Hide resolved

mkoeppe reviewed Mar 10, 2024

View reviewed changes

src/sage/features/databases.py Outdated Show resolved Hide resolved

mkoeppe reviewed Mar 10, 2024

View reviewed changes

src/sage/features/__init__.py Outdated Show resolved Hide resolved

soehms and others added 4 commits March 11, 2024 08:01

Fix docstring in join_feature.py

42adc04

Co-authored-by: Matthias Köppe <mkoeppe@math.ucdavis.edu>

Fix typo in databases.py

dd7e714

Co-authored-by: Matthias Köppe <mkoeppe@math.ucdavis.edu>

Fix docstring in features/__init__.py

eb908b4

Co-authored-by: Matthias Köppe <mkoeppe@math.ucdavis.edu>

36741: Insert lost argument

1835ed4

36741: pycodestyle fixes

03670bd

mkoeppe approved these changes Mar 17, 2024

View reviewed changes

mkoeppe added s: positive review and removed s: needs review labels Mar 17, 2024

mkoeppe removed this from the sage-10.3 milestone Mar 20, 2024

vbraun merged commit 821d8da into sagemath:develop Mar 25, 2024
16 checks passed

mkoeppe added this to the sage-10.4 milestone Mar 25, 2024

github-actions bot removed the s: positive review label Mar 26, 2024

soehms deleted the fix_doctest_hide_option branch March 26, 2024 20:47

tornaria mentioned this pull request Mar 31, 2024

Workaround for multiprocessing shared memory limits #37702

Merged

3 tasks

soehms mentioned this pull request Apr 3, 2024

Follow up #36741: improve code structure #37737

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Doctest hide option: Better detection of hidden packages #36741

Doctest hide option: Better detection of hidden packages #36741

soehms commented Nov 20, 2023 •

edited

Loading

mkoeppe commented Jan 9, 2024

soehms commented Feb 4, 2024

tornaria commented Feb 20, 2024

mkoeppe Mar 10, 2024

soehms Mar 11, 2024

soehms Mar 11, 2024

github-actions bot commented Mar 11, 2024

mkoeppe left a comment

soehms commented Mar 18, 2024

tornaria commented Mar 29, 2024

tornaria commented Mar 29, 2024

mkoeppe commented Mar 29, 2024

mkoeppe commented Mar 29, 2024

soehms commented Mar 30, 2024

tornaria commented Mar 30, 2024

soehms commented Mar 30, 2024

antonio-rojas commented May 3, 2024

mkoeppe commented May 3, 2024

soehms commented May 9, 2024

antonio-rojas commented May 9, 2024

Doctest hide option: Better detection of hidden packages #36741

Doctest hide option: Better detection of hidden packages #36741

Conversation

soehms commented Nov 20, 2023 • edited Loading

📝 Checklist

⌛ Dependencies

mkoeppe commented Jan 9, 2024

soehms commented Feb 4, 2024

tornaria commented Feb 20, 2024

mkoeppe Mar 10, 2024

Choose a reason for hiding this comment

soehms Mar 11, 2024

Choose a reason for hiding this comment

soehms Mar 11, 2024

Choose a reason for hiding this comment

github-actions bot commented Mar 11, 2024

mkoeppe left a comment

Choose a reason for hiding this comment

soehms commented Mar 18, 2024

tornaria commented Mar 29, 2024

tornaria commented Mar 29, 2024

mkoeppe commented Mar 29, 2024

mkoeppe commented Mar 29, 2024

soehms commented Mar 30, 2024

tornaria commented Mar 30, 2024

soehms commented Mar 30, 2024

antonio-rojas commented May 3, 2024

mkoeppe commented May 3, 2024

soehms commented May 9, 2024

antonio-rojas commented May 9, 2024

soehms commented Nov 20, 2023 •

edited

Loading