Include the `fq_default` type #97

GiacomoPope · 2023-10-19T16:12:31Z

I did this work a few weeks ago, then dropped it because I got married and now I'm trying to implement other work for a paper so my brain is VERY outside of python-flint world but thought I'd show at least roughly what's happening for this type and what we'll need.

Main thing is that we have these "union structs" for the fq_default type which aren't implemented in cython, so we need to do nesting instead.

oscarbenjamin · 2023-10-19T21:57:29Z

Note also previous discussion about this in gh-91.

oscarbenjamin · 2023-10-19T22:14:14Z

The way that you have defined the fq_default* structs in Cython here seems reasonable to me. I'm actually unsure if it is necessary to define anything about these structs at all if only fq_default_* functions are being used. Maybe these structs can just be treated as opaque at the Cython level. If there is any need to convert them then maybe functions like fq_default_get_fmpz_mod_poly are sufficient.

I think it is only necessary to define the struct members if we want to use direct member access but if the fq_default* functions are used instead then it is probably fine (possibly safer?) to just leave the struct definitions empty.

GiacomoPope · 2023-10-20T11:31:55Z

After I've wrapped up a few other projects, I'll come back to this then and at least get the fq_default doing some standard stuff. Then we can maybe start looking at adding the generic types in

oscarbenjamin · 2023-10-20T16:00:10Z

It's always worth remembering that it is most important just to get the basic features in. Trying to add all the possible functions that finite fields can support can take a lot longer than just making it possible to construct the objects in the first place and do some basic arithmetic.

Also once a type is added it becomes a lot easier for other people in future to add specific features if they want them.

GiacomoPope · 2024-01-17T21:41:00Z

The topic of this work came up at a Sage days I am currently attending and with some of my research code finished I now have more time to start working more on this again. Latest commit was simply merging master back into this branch to be up to date with the work I've missed.

Co-Authored-By: Luca De Feo <1315842+defeo@users.noreply.github.com>

GiacomoPope · 2024-08-06T08:05:04Z

@oscarbenjamin I'm being stupid with the local build...

If I run:

source bin/activate
./bin/build_inplace.sh

I get an error from the current setup:

Traceback (most recent call last):
  File "/Users/Jack/Documents/GitHub/python-flint/setup.py", line 143, in <module>
    ext_modules=cythonize(ext_modules, compiler_directives=compiler_directives),
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Jack/Library/Python/3.12/lib/python/site-packages/Cython/Build/Dependencies.py", line 1010, in cythonize
    module_list, module_metadata = create_extension_list(
                                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Jack/Library/Python/3.12/lib/python/site-packages/Cython/Build/Dependencies.py", line 845, in create_extension_list
    for file in nonempty(sorted(extended_iglob(filepattern)), "'%s' doesn't match any files" % filepattern):
  File "/Users/Jack/Library/Python/3.12/lib/python/site-packages/Cython/Build/Dependencies.py", line 117, in nonempty
    raise ValueError(error_msg)
ValueError: 'src/flint/types/fmpz_mpoly_q.pyx' doesn't match any files

At the moment, I do testing by running

pip install .

and then working directly with this :)

Is this now "best practice" for development? The meson stuff has changed the internals since i last worked on python-flint

oscarbenjamin · 2024-08-06T14:55:31Z

Yes, it has changed because of the meson stuff which I need to properly document somewhere.

The setup.py is no longer needed but is just there for compatibility. The build_inplace script uses setup.py so should not be used any more. In meson there is no concept of an in-place build because all builds are always out-of-tree in a build directory.

There are several ways of doing this now:

First way:

Use something like pip as a build frontend which will invoke the meson-python build backend (see PEP 517). That means that e.g. pip install . will work.

You can also use this to create a meson-python editable install like:

pip install meson meson-python cython
pip install --no-build-isolation --editable .

This editable install will be built in the ./build directory. Every time you import flint from Python it will trigger an incremental rebuild.

Second way:

Don't bother with pip and the whole PEP 517 business. Run meson commands directly to build things instead. First get a clean setup:

pip uninstall python-flint
rm -r build build-install

Now think of these commands as ./configure, make and then make install:

meson setup build --pkg-config-path=$(pwd)/.local/lib/pkgconfig
meson compile -C build
meson install -C build --destdir ../build-install

Now the build-install directory looks like this:

$ tree build-install/
build-install/
└── usr
    └── local
        └── lib
            └── python3.11
                └── site-packages
                    └── flint
                        ├── flint_base
                        │   ├── flint_base.cpython-311-x86_64-linux-gnu.so
                        │   ├── flint_context.cpython-311-x86_64-linux-gnu.so
                        │   ├── __init__.py
                        │   └── __pycache__
                        │       └── __init__.cpython-311.pyc
...

Add that directory to PYTHONPATH and run the tests:

export PYTHONPATH=$(pwd)/build-install/usr/local/lib/python3.11/site-packages
export LD_LIBRARY_PATH=$(pwd)/.local/lib
python -m flint.test

Third way:

Running the meson commands directly is tedious so spin (pip install spin) acts as a frontend to make this nicer. Make sure you have a clean setup (no editable install, etc) first:

pip uninstall python-flint
rm -r build build-install

Now spin can build and run the tests for you. If you have flint installed system wide (rather than in a local directory) then it should be as simple as running spin test which will build everything, install it into build-install, add to PYTHONPATH and run all the tests. This builds in parallel and uses incremental rebuilds so it's a lot faster.

I don't have Flint installed system wide so I need to configure the build first like this:

spin build -- --pkg-config-path=$(pwd)/.local/lib/pkgconfig -Dadd_flint_rpath=true

That configuration is persistent so now I can just run e.g. spin test to rebuild and run tests.

Fourth way:

It is also possible with spin to use an editable install and spin install will set that up for you by running pip. Note that spin is just a convenience frontend. It prints out all of the commands that it executes in case you want to run them yourself with different arguments or something.

Note that these different ways are not compatible with each other so you have to choose one. You probably need a deep clean when switching from one way to another because they use the build directory in incompatible ways which is why I recommend:

pip uninstall python-flint
rm -r build build-install

I'm not sure what should be the recommended way right now. Probably using spin without an editable install should be recommended although the editable install is quite convenient. I have spent a long time learning how all the pieces work and so I find all of these things easy to use now but I'm not sure what others would prefer. I think @Jake-Moss prefers to use spin without an editable install.

With meson all building is out of tree so if you have an old checkout where you have been doing in-place builds then it is probably best to clean out all the old .c etc files with git clean. The meson build will not generate those files in the source tree any more.

GiacomoPope · 2024-08-06T15:03:14Z

AMAZING! Thanks so much for such a detailed answer :)

I'll work at finally getting this type done this evening

GiacomoPope · 2024-08-06T16:26:45Z

@oscarbenjamin this is still very much unfinished, but I wanted to point to the arithmetic boilerplate code in the flint_scalar class -- is this something valuable? I thought it might be worth having another PR / issue where we discuss some generic functions which we think should be included for all these base classes as a guide for people implementing new types?

This links again to what you said in the other PR with pow_trunc -- having expectations with boilerplates in the base class could help?

oscarbenjamin · 2024-08-06T17:12:19Z

I wanted to point to the arithmetic boilerplate code in the flint_scalar class -- is this something valuable?

Yes, I think so.

I think that for some very basic types like nmod, fmpz, etc it is important for the subclass to implement most/all of its own boiler-plate for maximum speed when e.g. adding nmod + nmod or fmpz + int. For most other types like polynomials, matrices etc it matters much less because each operation is generally more expensive. Cython's subclass v-tables are a fast dispatch mechanism in my experience so the overhead of splitting the implementation of a method between super and subclass is quite small for cdef classes.

In general though I think it would be better if most types could inherit something that manages most of the __add__ etc boiler-plate like checking compatible types, coercing int etc. As you say it would help with defining what the expected methods should be as well.

I thought it might be worth having another PR / issue where we discuss some generic functions which we think should be included for all these base classes as a guide for people implementing new types?

I think that makes sense.

Jake-Moss · 2024-08-07T01:48:55Z

I think @Jake-Moss prefers to use spin without an editable install.

Yep that's correct! IIRC the editable install doesn't support prompting a compilation from anything other than importing. This didn't jell with me, I prefer my in-editor write, compile, test workflow with an explicit (or automatic) compilation step.

However, not using an editable install means all my python-flint related tools need to be aware of spin to properly setup PYTHON_PATH, so I configure my emacs setup with the Python executable name as spin with the necessary options. This means I can have an explicit compilation step (with my choice of arguments) if I desire it via spin build, and any invocation of spin prompts an incremental build. So things like launching pytest or a python repl first call spin, building if necessary, then running as normal.

IMO it's pretty close to optimal, only thing left for me to tinker with is to write some spin layout for direnv and drop the emacs specific configuration, but prefixing things with spin works well enough for now.

oscarbenjamin · 2024-08-07T10:14:32Z

IMO it's pretty close to optimal

Okay, so maybe I'll just write some docs that explain how to use spin like that. I can give some context for how to set things up the other ways but it is good to have a guide that clearly describes one way of doing things.

Another thing I'm not sure about:

Do either of you build against a system install of Flint or use a local build?

I always use a local build in $(pwd)/.local because I want to have local control over which version of Flint is being used. The bin/build_dependencies_unix.sh script sets up the local build of GMP, MPFR and Flint to make this work. Do you use that script?

Using a local build complicates things a bit because you need to fiddle with environment variables. Hence the bin/activate script:

python-flint/bin/activate

Lines 1 to 3 in 069d24d

    
           export C_INCLUDE_PATH=$(pwd)/.local/include 
        
           export LIBRARY_PATH=$(pwd)/.local/lib 
        
           export LD_LIBRARY_PATH=$(pwd)/.local/lib

With spin/meson there is a better way than using environment variables but it just complicates exactly the instructions that are needed for setting up the development build. Firstly meson uses pkgconfig to locate dependencies like Flint and so rather than setting LIBRARY_PATH or C_INCLUDE_PATH you set PKG_CONFIG_PATH which you can do with the --pkg-config-path command line argument. That needs to be done in the call to meson setup or spin build. With spin this means that you need to have a first explicit call to spin build before you can then use e.g. spin test because arguments like --pkg-config-path cannot be passed to spin test (its arguments are all passed to pytest rather than to meson).

Setting --pkg-config-path means that the build completes but it is also necessary to be able to find the libflint shared library at runtime. That is what LD_LIBRARY_PATH is for on Linux (or DYLD_LIBRARY_PATH on macos or PATH on Windows). At least on Linux or macos there is a better way than using environment variables which is to add an rpath entry. The meson-python editable install does this automatically. When using spin it happens automatically on macos but not on Linux. I've added a meson configuration option called add_flint_rpath for this. Possibly it makes sense to have this option on by default somehow but this is needed as well.

This is all why for me the first command I need to run is either of:

meson setup build --pkg-config-path=$(pwd)/.local/lib/pkgconfig -Dadd_flint_rpath=true
spin build -- --pkg-config-path=$(pwd)/.local/lib/pkgconfig -Dadd_flint_rpath=true

It might be handy to put this into a convenient script like bin/setup_spin.sh or something so then the instructions to set up a dev checkout are:

git clone git@github.com:flintlib/python-flint.git
cd python-flint
pip install -r requirements-dev.txt
bin/build_dependencies_unix.sh
bin/setup_spin.sh

And then spin test, spin docs, spin sdist etc should work. I think one thing missing right now is spin doctest.

I'm not sure though if I can assume though that:

Everyone wants to use a local build of Flint when working on python-flint.
The path $(pwd)/.local/lib/pkgconfig is always correct. It might be lib64 or something on some systems.
It is always fine to add the rpath entry. I think at least if this is just advice for a dev checkout then probably it is fine. There are apparently reasons why meson does not do this by default (Relative rpaths are used for a pkgconfig dependency on Linux mesonbuild/meson#13046).

GiacomoPope · 2024-08-07T11:53:56Z

@oscarbenjamin small issue with doc tests -- the default type picked between 3.0 and 3.1 seems to be different, which means I can't get things to pass for both...

One option would be simply to not have the context type included on the __repr__

oscarbenjamin · 2024-08-07T12:37:59Z

the default type picked between 3.0 and 3.1 seems to be different, which means I can't get things to pass for both...

Could just mark the doctest to be skipped.

One option would be simply to not have the context type included on the __repr__

It might be nicer if it displayed as something other than just an integer. The multivariate polynomials use an enum for monomial orderings. Perhaps something similar could be used here:

In [1]: import flint

In [2]: ctx = flint.fmpz_mpoly_ctx(3, flint.Ordering.lex, ["x", "y", "z"])

In [3]: ctx
Out[3]: fmpz_mpoly_ctx(3, '<Ordering.lex: 0>', ('x', 'y', 'z'))

I might prefer that to just show as this and for this to be valid though:

fmpz_mpoly_ctx(3, 'lex', ('x', 'y', 'z'))

Using integers makes sense in C but in Python strings are nicer. Enums are also good but it is nice if you can shortcut them with something like 'lex' that is still meaningful.

Jake-Moss · 2024-08-07T14:00:01Z

Do either of you build against a system install of Flint or use a local build?

I've been using something pretty similar with a local build, just in a different location. I've controlled everything via a .envrc where I set PKG_CONFIG_PATH, LIBRARY_PATH, LD_LIBRARY_PATH, and C_INCLUDE_PATH appropriately. Very recently I've switched to using a Nix flake to manage everything where I can override various options without building from source manually. Having pkg-config with nix is enough that I don't need to configure anything else.

The path $(pwd)/.local/lib/pkgconfig is always correct. It might be lib64 or something on some systems.

My certainly uses (or rather used) lib64.

It is always fine to add the rpath entry. I think at least if this is just advice for a dev checkout then probably it is fine. There are apparently reasons why meson does not do this by default (Relative rpaths are used for a pkgconfig dependency on Linux mesonbuild/meson#13046).

After having read that issue I think it's probably fine for a quick automated install script to add the rpath entry if they have not installed flint via a package manager. It's always possible for contributors to do things themselves if anything goes wrong.

GiacomoPope · 2024-08-07T14:00:16Z

Amazing. I'll work on this this evening if I have time. For coverage, how best should I check this? coverage run ?

Jake-Moss · 2024-08-07T14:04:53Z

It might be nicer if it displayed as something other than just an integer. The multivariate polynomials use an enum for monomial orderings. Perhaps something similar could be used here:
In [1]: import flint

In [2]: ctx = flint.fmpz_mpoly_ctx(3, flint.Ordering.lex, ["x", "y", "z"])

In [3]: ctx
Out[3]: fmpz_mpoly_ctx(3, '<Ordering.lex: 0>', ('x', 'y', 'z'))
I might prefer that to just show as this and for this to be valid though:
fmpz_mpoly_ctx(3, 'lex', ('x', 'y', 'z'))
Using integers makes sense in C but in Python strings are nicer. Enums are also good but it is nice if you can shortcut them with something like 'lex' that is still meaningful.

I should be able to fix that by using a Python based enum.Enum class instead of the cpdef enum as we can then specify strings as the constants. Not sure why I opted for the cython-based class over the Python one in the first place. I've been slacking on #164, I can make the change after that.

oscarbenjamin · 2024-08-07T14:44:41Z

For coverage, how best should I check this? coverage run ?

I'm not sure. This is one thing that does not currently work with meson because Cython gets confused about the out of tree build. I showed (in cython/cython#6186 (comment)) how to patch Cython and then add something to python-flint's coverage config that would make it work I think.

Jake have you managed to measure coverage with the meson build somehow?

GiacomoPope · 2024-08-09T13:10:03Z

Maybe just declare it as returning a pointer to the struct?

I tried this but I couldnt get cython right to find out how to return the pointer as a C-type -- i get errors about it trying to convert the type to a Python object however i refactor it

GiacomoPope · 2024-08-09T13:11:45Z

The above is also a nice idea -- i need a break from this but if you want, please add this change (either as a diff or a commit) -- im a little cython confused atm :D

oscarbenjamin · 2024-08-09T13:24:17Z

i need a break from this

Fair enough maybe to make things easier we can just postpone adding is_primitive or comment it out for now.

oscarbenjamin · 2024-08-09T13:25:55Z

src/flint/flint_base/flint_base.pyx

+        other = self._any_as_self(other)
+        if other is NotImplemented:
+            return NotImplemented
+        return self._sub_(other, swap=True)


Rather than having swap=True I would just have two methods _sub_ and _rsub_.

The idea is that __sub__ handles all conditionality so that _sub_ and _rsub_ do not need to check anything. Also keyword arguments can add overhead that I would avoid at this level.

OK I've done this for the base class and added rsub, rdiv.

GiacomoPope · 2024-08-09T13:33:36Z

postpone adding is_primitive

Maybe we can circle back to this when we deal with fq_embed which will need more complex use of the fq_ctx etc and we'll need to build a bunch of helpers to get correct contexts

oscarbenjamin · 2024-08-09T13:33:59Z

src/flint/types/fq_default.pyx

+    def _add_(self, other):
+        """
+        Assumes that __add__() has ensured other is of type self
+        """
+        cdef fq_default res
+        res = self.ctx.new_ctype_fq_default()
+        fq_default_add(res.val, self.val, (<fq_default>other).val, self.ctx.val)
+        return res


I would add types for the parameters here and make this cdef I think:

cdef fq_default _add_(fq_default self, fq_default other): cdef fq_default res = self.ctx.new_ctype_fq_default() fq_default_add(res.val, self.val, other.val, self.ctx.val) return res

I'm not sure if making this cdef makes much difference though.

I had to use cpdef instead of cdef -- is this OK? I suppose this is needed to return the python object

I'm not sure what is the best way to make this work. If we want to use this widely though (for other types) then it would be good to benchmark, look at the generated C code, try alternatives etc.

I am concerned about the overhead of these methods. See e.g.: sympy/sympy#26932. I got 10% speedup on a ~~macro~~ micro-benchmark that used fmpz just by inlining the type checking in __add__:

diff --git a/src/flint/types/fmpz.pyx b/src/flint/types/fmpz.pyx index 0b16375..0a025b8 100644 --- a/src/flint/types/fmpz.pyx +++ b/src/flint/types/fmpz.pyx @@ -1,3 +1,5 @@ +from cpython.object cimport PyTypeObject, PyObject_TypeCheck + from flint.flint_base.flint_base cimport flint_scalar from flint.utils.typecheck cimport typecheck from flint.utils.conversion cimport chars_from_str @@ -185,6 +187,10 @@ cdef class fmpz(flint_scalar): return -self def __add__(s, t): + if PyObject_TypeCheck(t, <PyTypeObject*>fmpz): + u = fmpz.__new__(fmpz) + fmpz_add((<fmpz>u).val, (<fmpz>s).val, (<fmpz>t).val) + return u cdef fmpz_struct tval[1] cdef int ttype = FMPZ_UNKNOWN u = NotImplemented

It still came out notably slower than when using gmpy2 though so there is definitely some overhead in the Cython wrapper that could be reduced. In some cases gmpy2 is 2x faster for the macro benchmark.

Maybe we should just prioritise speed for all these scalar types and accept we have to hand write all the add methods etc. seems silly to lose a bunch of performance just for a few smaller functions.

We need to measure the effects. I think it is fine to have a framework in the superclass but then inline common cases. Note that e.g. for SymPy the case that really matters is the common case like fmpz*fmpz and to a much lesser extent fmpz*int. We could have an fmpz.__add__ method that optimises that common case as hard as possible but then just falls back on something slower that handles coercions etc:

class fmpz(scalar): def __add__(self, other): if PyTypeCheck(...): # Fast optimised case else: # Fall back on slower generic routine return super().__add__(self, other)

It makes sense to start out having a nice generic routine in the superclass that handles everything correctly and then after doing timing measurements micro-optimise the cases that matter.

OK. We can work this way then. I think int * type and type * type are the ones worth doing first as I think it'll be common for people to do things like 5 * type or type + 1 in their code and it would be nice for this to be fast without having to do one = type(one); type + one

Exactly and almost always the ints would be small so the fast path could use fmpz_mul_si and friends rather than general int -> fmpz coercion.

oscarbenjamin · 2024-08-09T13:36:35Z

I'll give this a full read later but I think it's probably good to go.

oscarbenjamin

Broadly this is good but I found a bunch of small things.

oscarbenjamin · 2024-08-09T15:56:03Z

src/flint/types/fq_default.pyx

+        - 2: `fq_default_ctx.FQ_NMOD`: Using `fq_nmod_t`,
+        - 3: `fq_default_ctx.FQ`: Using `fq_t`.
+        """
+        return fq_default_ctx_type(self.val)


This should convert to the enum type like fq_default_type(fq_default_ctx_type(self.val)).

Also the docstring here is missing types like NMOD and FMPZ_MOD.

Probably best to choose a single docstring that lists and explains the types (perhaps the fq_default_type docstring) and then have other places refer to it in a way that the html docs would make a clickable link like:

See :class:`~.fq_default_type` for possible types.

oscarbenjamin · 2024-08-09T16:01:04Z

src/flint/types/fq_default.pyx

+            4 : "NMOD",
+            5 : "FMPZ_MOD",
+        }
+        return FQ_TYPES[self.fq_type]


I think it should be possible to get the string from the enum if self.fq_type returns the enum like:

return self.fq_type._name_

Then we don't need to duplicate the names and numbers here.

oscarbenjamin · 2024-08-09T16:03:18Z

src/flint/types/fq_default.pyx

+    @staticmethod
+    def _parse_input_fq_type(fq_type):
+        # Allow the type to be denoted by strings or integers
+        FQ_TYPES = {
+            "FQ_ZECH" : 1,
+            "FQ_NMOD" : 2,


This dictionary is already provided by the enum type:

In [35]: fq_default_type._member_map_ Out[35]: {'DEFAULT': <fq_default_type.DEFAULT: 0>, 'FQ_ZECH': <fq_default_type.FQ_ZECH: 1>, 'FQ_NMOD': <fq_default_type.FQ_NMOD: 2>, 'FQ': <fq_default_type.FQ: 3>, 'NMOD': <fq_default_type.NMOD: 4>, 'FMPZ_MOD': <fq_default_type.FMPZ_MOD: 5>}

I guess this function can just be:

try: return fq_default_type._member_map_[fq_type] except KeyError: pass try: return fq_default_type._value2member_map_[fq_type] except KeyError: pass raise ValueError(f"fq_type should be one of {fq_default_type._member_names_}")

I'm getting an error that fq_default_type doesn't have a member map... I'll try and figure out why

It isn't in the docs actually. This is though:

In [75]: fq_default_type.__members__ Out[75]: mappingproxy({'DEFAULT': <fq_default_type.DEFAULT: 0>, 'FQ_ZECH': <fq_default_type.FQ_ZECH: 1>, 'FQ_NMOD': <fq_default_type.FQ_NMOD: 2>, 'FQ': <fq_default_type.FQ: 3>, 'NMOD': <fq_default_type.NMOD: 4>, 'FMPZ_MOD': <fq_default_type.FMPZ_MOD: 5>})

I guess for _value2member_map_ we need to call it:

In [76]: fq_default_type(2) Out[76]: <fq_default_type.FQ_NMOD: 2>

Hm but something weird happens:

In [79]: fq_default_type(50) Out[79]: <fq_default_type.FQ_NMOD|48: 50>

I have it available in python:

In [1]: from flint import fq_default_ctx, fq_default_type In [2]: fq_default_type Out[2]: <flag 'fq_default_type'> In [3]: fq_default_type._member_map_ Out[3]: {'DEFAULT': <fq_default_type.DEFAULT: 0>, 'FQ_ZECH': <fq_default_type.FQ_ZECH: 1>, 'FQ_NMOD': <fq_default_type.FQ_NMOD: 2>, 'FQ': <fq_default_type.FQ: 3>, 'NMOD': <fq_default_type.NMOD: 4>, 'FMPZ_MOD': <fq_default_type.FMPZ_MOD: 5>} In [4]: type(fq_default_type) Out[4]: enum.EnumType

But for compilation of the cython I get:

Error compiling Cython file: ------------------------------------------------------------ ... return fq_type @staticmethod def _parse_input_var(var): try: return fq_default_type._member_map_[fq_type] ^ ------------------------------------------------------------ /Users/Jack/Documents/GitHub/python-flint/src/flint/types/fq_default.pyx:65:34: _member_map_ not a known value of fq_default_type

I guess that the C-level enum is different in the cpdef enum.

Maybe using the enum type for validation isn't quite a good fit and we just need to have a dict somewhere to map values from one to the other.

Either way we need there to be a single place where the list of values is stored rather than several different dicts spread around.

oscarbenjamin · 2024-08-09T16:17:22Z

src/flint/types/fq_default.pyx

+    Finite fields can be initialized in one of two possible ways. The
+    first is by providing characteristic and degree:
+
+        >>> fq_default_ctx(5, 2, 'y', fq_type=1)
+        fq_default_ctx(5, 2, 'y', x^2 + 4*x + 2, 'FQ_ZECH')


Probably better to use fq_type='FQ_ZECH' or fq_type=fq_default_type.FQ_ZECH here rather than fq_type=1. We shouldn't encourage writing code that uses magic values like that.

src/flint/types/fq_default.pyx

oscarbenjamin · 2024-08-09T16:38:02Z

src/flint/types/fq_default.pyx

+        if typecheck(obj, int):
+            # For small integers we can convert directly
+            if obj < 0 and obj.bit_length() < 31:
+                fq_default_set_si(fq_ele, <slong>obj, self.val)
+            elif obj.bit_length() < 32:
+                fq_default_set_ui(fq_ele, <ulong>obj, self.val)
+            # For larger integers we first convert to fmpz


The signed to unsigned cast here is unsafe:

In [65]: F = fq_default_ctx(2) In [66]: F(-2**31+1) --------------------------------------------------------------------------- OverflowError Traceback (most recent call last) Cell In[66], line 1 ----> 1 F(-2**31+1) File fq_default.pyx:467, in flint.types.fq_default.fq_default_ctx.__call__() File fq_default.pyx:492, in flint.types.fq_default.fq_default.__init__() File fq_default.pyx:388, in flint.types.fq_default.fq_default_ctx.set_any_as_fq_default() File fq_default.pyx:359, in flint.types.fq_default.fq_default_ctx.set_any_scalar_as_fq_default() OverflowError: can't convert negative value to ulong

I would just use a single set_si case here for both positive and negative values. Can probably catch OverflowError like:

slong cdef i try: i = ...

oscarbenjamin · 2024-08-09T16:41:04Z

src/flint/types/fq_default.pyx

+                    and self.prime() == other.prime()
+                    and self.modulus() == other.modulus())
+        else:
+            raise TypeError(f"Cannot compare fq_default_ctx with {type(other)}")


__eq__ should return NotImplemented rather than raising TypeError:

In [67]: F == None --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[67], line 1 ----> 1 F == None File fq_default.pyx:451, in flint.types.fq_default.fq_default_ctx.__eq__() TypeError: Cannot compare fq_default_ctx with <class 'NoneType'>

Generally considered a faux-pas for == to raise an exception rather than turning into False.

oscarbenjamin · 2024-08-09T16:49:06Z

src/flint/types/fq_default.pyx

+        return f"fq_default({self.to_list(), self.ctx.__repr__()})"
+
+    def __hash__(self):
+        return hash((self.to_polynomial(), hash(self.ctx)))


I think we probably need coverage measurement to make sure that cases like this are checked:

In [69]: hash(F(4)) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[69], line 1 ----> 1 hash(F(4)) File fq_default.pyx:556, in flint.types.fq_default.fq_default.__hash__() AttributeError: 'flint.types.fq_default.fq_default' object has no attribute 'to_polynomial'

oscarbenjamin · 2024-08-09T16:52:30Z

src/flint/types/fq_default.pyx

+        # For nmod we can convert by taking it as an int
+        # Ignores the modulus of nmod
+        if typecheck(obj, nmod):
+            fq_default_set_ui(fq_ele, <ulong>(<nmod>obj).val, self.val)
+            return 0
+
+        # For fmpz_mod we can also convert directly
+        # Assumes that the modulus of the ring matches the context
+        if typecheck(obj, fmpz_mod):
+            fq_default_set_fmpz(fq_ele, (<fmpz_mod>obj).val, self.val)
+            return 0


If this function is being used by __add__ etc then the modulus needs to be checked:

In [70]: F = fq_default_ctx(3) In [71]: F(2) + nmod(2, 7) Out[71]: 1

oh i thought i handled this... i must have forgotten to do it. -- oh, i did it for equality but not coercion for the addition

GiacomoPope · 2024-08-09T17:44:09Z

I think I have covered many of the comments above, hopefully all. but happy to re-review if needed.

oscarbenjamin · 2024-08-09T17:49:34Z

src/flint/types/fq_default.pxd

+"""
+Enum for the fq_default context types:
+
+- 1. `fq_default_ctx.FQ_ZECH`: Use `fq_zech_t`,
+- 2. `fq_default_ctx.FQ_NMOD`: Use `fq_nmod_t`,
+- 3. `fq_default_ctx.FQ`: Use `fq_t`.
+- 4. `fq_default_ctx.NMOD`: Use `nmod` for degree = 1,
+- 5. `fq_default_ctx.FMPZ_MOD`: Use `fmpz_mod` for degree = 1.
+
+These can be manually selected, or type: `fq_default_ctx.DEFAULT` can be used
+for the implementation to be automatically decided by Flint (default),
+"""
+cpdef enum fq_default_type:
+    DEFAULT  = 0


Does the docstring not need to inside the enum statement?

oscarbenjamin · 2024-08-09T17:50:12Z

One thing I realised is that we need to add this to the docs.

oscarbenjamin · 2024-08-09T17:50:47Z

We need a doc/source/fq_default.rst.

oscarbenjamin · 2024-08-12T15:02:45Z

Let's get this one in now. It would be good to improve the test coverage but

I suppose that can be done later. I think all the substantial issues were resolved.

Thanks!

GiacomoPope · 2024-08-12T16:15:09Z

Maybe we can find a hacky way to make pytest run the doctests and have coverage test this? If we get a good coverage test done im happy to circle back and improve this

oscarbenjamin · 2024-08-12T16:36:26Z

Maybe we can find a hacky way to make pytest run the doctests and have coverage test this?

We can still use setuptools for it but I guess the setup.py just needs fixing:

$ PYTHONPATH=src bin/coverage.sh

I think this is enough to fix it:

diff --git a/setup.py b/setup.py
index d260834..82ce32d 100644
--- a/setup.py
+++ b/setup.py
@@ -103,7 +103,6 @@ ext_files = [
     ("flint.types.fmpz_mod_mat", ["src/flint/types/fmpz_mod_mat.pyx"]),
 
     ("flint.types.fmpq_mpoly", ["src/flint/types/fmpq_mpoly.pyx"]),
-    ("flint.types.fmpz_mpoly_q", ["src/flint/types/fmpz_mpoly_q.pyx"]),
 
     ("flint.types.arf", ["src/flint/types/arf.pyx"]),
     ("flint.types.arb", ["src/flint/types/arb.pyx"]),

oscarbenjamin · 2024-08-12T16:38:55Z

I did have a look on Friday at getting Cython coverage to work with meson. The problem is that Cython's coverage plugin works by using heuristics based on setuptools. It basically needs a rewrite. Probably the rewritten version should be maintained as part of something like spin rather than Cython.

GiacomoPope and others added 2 commits October 19, 2023 17:10

Add a VERY rough start on adding fq

1bd96ee

Merge branch 'flintlib:master' into add_fq

2488e5f

Merge branch 'master' into add_fq

20aa30b

GiacomoPope and others added 3 commits August 5, 2024 22:17

Merge branch 'master' into add_fq

277794f

Merge branch 'master' into add_fq

942152a

further work on fq_default with luca de feo

3d637f6

Co-Authored-By: Luca De Feo <1315842+defeo@users.noreply.github.com>

add line to meson build

b728899

more boilerplate code

c8fe699

GiacomoPope added 2 commits August 7, 2024 10:15

add fq_default to the doctests

1307116

start on type conversions and comparisons

6bf96a7

untested arithemtic

7e68a0c

begin adding tests

7347924

oscarbenjamin reviewed Aug 9, 2024

View reviewed changes

remove is_primitive and modify base template

172c9d8

oscarbenjamin reviewed Aug 9, 2024

View reviewed changes

modify _arith_ to use cpdef

ce8cd4d

oscarbenjamin reviewed Aug 9, 2024

View reviewed changes

GiacomoPope added 6 commits August 9, 2024 18:21

fix some but not all reviewer comments -- need to stop for the day

ee4ae24

change docstrings for the type stuff

38b6ce1

ensure the modulus matches for scalar coercion

52f8847

use try / except for slong conversion

66c54f4

move docstring to class

163f7cc

refactor out where the dict is for types

e32a6dd

oscarbenjamin reviewed Aug 9, 2024

View reviewed changes

GiacomoPope added 3 commits August 9, 2024 18:52

move docstring

876fd2a

add docs

e79b397

check modulus for nmod_poly and fmpz_mod_poly

f789f22

oscarbenjamin merged commit 09af886 into flintlib:master Aug 12, 2024
31 checks passed

GiacomoPope deleted the add_fq branch August 12, 2024 16:41

oscarbenjamin mentioned this pull request Aug 26, 2024

Implement Finite Field Types and Conversion to SageMath #50

Open

Include the fq_default type #97

Include the fq_default type #97

Conversation

GiacomoPope commented Oct 19, 2023

oscarbenjamin commented Oct 19, 2023

oscarbenjamin commented Oct 19, 2023

GiacomoPope commented Oct 20, 2023

oscarbenjamin commented Oct 20, 2023

GiacomoPope commented Jan 17, 2024

GiacomoPope commented Aug 6, 2024 • edited Loading

oscarbenjamin commented Aug 6, 2024

GiacomoPope commented Aug 6, 2024

GiacomoPope commented Aug 6, 2024

oscarbenjamin commented Aug 6, 2024

Jake-Moss commented Aug 7, 2024

oscarbenjamin commented Aug 7, 2024

GiacomoPope commented Aug 7, 2024 • edited Loading

oscarbenjamin commented Aug 7, 2024

Jake-Moss commented Aug 7, 2024

GiacomoPope commented Aug 7, 2024

Jake-Moss commented Aug 7, 2024

oscarbenjamin commented Aug 7, 2024

GiacomoPope commented Aug 9, 2024

GiacomoPope commented Aug 9, 2024

oscarbenjamin commented Aug 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GiacomoPope commented Aug 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GiacomoPope Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

oscarbenjamin Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GiacomoPope Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oscarbenjamin commented Aug 9, 2024

oscarbenjamin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GiacomoPope Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

GiacomoPope commented Aug 9, 2024

Choose a reason for hiding this comment

oscarbenjamin commented Aug 9, 2024

oscarbenjamin commented Aug 9, 2024

oscarbenjamin commented Aug 12, 2024

GiacomoPope commented Aug 12, 2024

oscarbenjamin commented Aug 12, 2024

oscarbenjamin commented Aug 12, 2024

Include the `fq_default` type #97

Include the `fq_default` type #97

GiacomoPope commented Aug 6, 2024 •

edited

Loading

GiacomoPope commented Aug 7, 2024 •

edited

Loading

GiacomoPope Aug 9, 2024 •

edited

Loading

oscarbenjamin Aug 9, 2024 •

edited

Loading

GiacomoPope Aug 9, 2024 •

edited

Loading

GiacomoPope Aug 9, 2024 •

edited

Loading