Update to upstream ctypesgen version

[nilason]

Following discussion on #1647 I swapped content of python/grass/ctypes/ctypesgen with latest ctypesgen code from ctypesgen/ctypesgen@0681f8e.

This for more easy overview for initial estimation on degree of diversion and may serve as a starting point in a eventual transition.

I made no effort to make it actually work. Don't try this at home :)

This is now a working PR with latest version of ctypesgen. This needs to be tested.

Addresses trac ticket #3900.

[nilason]

It turned out it was almost as straightforward as swapping ctypesgen version code base.

It now actually works, as tested on Mac. It now seems that updating to latest version of ctypesgen is actually feasible. I keep it in draft mode, but I modified the title of the PR to reflect this.

Please do test!

I also would appreciate suggestions to fix CIs Flake8 failure. Preamble code is now included in every generated py file (with an indentical String class among others in each). With the current (old) version preamble code is collected in a (one) common file.

[neteler]

Great work! I have checked out this PR and compiled it successfully on Fedora 33. The GUI works.
What to look for specifically? Perhaps you have test recommendations...

[neteler]

OK, I managed to trigger one.. switching to 3D NVIZ mode in the GUI, leads to

Starting 3D view mode...                                                        
Exception in thread
Thread-13
:
Traceback (most recent call last):
  File "/usr/lib64/python3.9/threading.py", line 954, in
_bootstrap_inner

self.run()
  File "/home/mneteler/software/grass_master/dist.x86_64-pc-
linux-gnu/gui/wxpython/nviz/mapwindow.py", line 68, in run

self._display = wxnviz.Nviz(self.log, self.progressbar)
  File "/home/mneteler/software/grass_master/dist.x86_64-pc-
linux-gnu/gui/wxpython/nviz/wxnviz.py", line 124, in
__init__

G_set_error_routine(errfunc)
ctypes
.
ArgumentError
:
argument 1: <class 'TypeError'>: expected CFunctionType
instance instead of CFunctionType
Exception ignored in:
<function Nviz.__del__ at 0x7f22ba163f70>
Traceback (most recent call last):
  File "/home/mneteler/software/grass_master/dist.x86_64-pc-
linux-gnu/gui/wxpython/nviz/wxnviz.py", line 143, in __del__

del self.data
AttributeError
:
data
Traceback (most recent call last):
  File "/home/mneteler/software/grass_master/dist.x86_64-pc-
linux-gnu/gui/wxpython/mapdisp/toolbars.py", line 239, in
OnSelectTool

self.parent.AddNviz()
  File "/home/mneteler/software/grass_master/dist.x86_64-pc-
linux-gnu/gui/wxpython/mapdisp/frame.py", line 489, in
AddNviz

self.MapWindow3D = GLWindow(
  File "/home/mneteler/software/grass_master/dist.x86_64-pc-
linux-gnu/gui/wxpython/nviz/mapwindow.py", line 195, in
__init__

self.decoration["arrow"]["size"] = self._getDecorationSize()
  File "/home/mneteler/software/grass_master/dist.x86_64-pc-
linux-gnu/gui/wxpython/nviz/mapwindow.py", line 1253, in
_getDecorationSize

size = self._display.GetLongDim() / 8.0
AttributeError
:
'NoneType' object has no attribute 'GetLongDim'

[nilason]

OK, I managed to trigger one.. switching to 3D NVIZ mode in the GUI, leads to

Same for me, will look into this.

[nilason]

So, now I believe I have actually solved remaining major issues, including nviz 3D mode. Running the tests locally now shows no regression from master, and remaining ones are likely unrelated to this PR.

I included a directory with patches already applied to ctypesgen source, to keep the record. Description of patches needed, which in some way or another need to be addressed upstreams (for this to make any sense):

POINTER patch

https://trac.osgeo.org/grass/ticket/2748
https://trac.osgeo.org/grass/ticket/3641

Every generated GRASS library bridge part (gis.py, raster.py etc.) is a standalone product.
E.g. parts of libgis (include/grass/gis.h) used in libraster (etc/python/grass/lib/raster.py) are also defined in raster.py. This way there is a definition of struct_Cell_head in both gis.py and raster.py -- a situation ctypes doesn't approve of. This patch seems to fix related errors in GRASS. Manually remove e.g. the gis.py parts in raster.py and make a "import" also did the work, but that would be more difficult to implement as part of ctypesgen file generation.

Loader and preamble patch

Replaces sed introduced with:
"Move ctypesgen boilerplate to common module"
59eeff4

Mac patches

#385
#981

[neteler]

Excellent @nilason: switching to 3D NVIZ mode in the GUI now works! I justed tested that for now (Fedora 33) but will keep the patch locally active to explore it further.

[nilason]

Excellent @nilason: switching to 3D NVIZ mode in the GUI now works! I justed tested that for now (Fedora 33) but will keep the patch locally active to explore it further.

Great! Thanks for further testing. I'm rather confident this will work in general use, but there may always be surprises.

[neteler]

Probably time to merge in order to receive testing on Windows and Mac?

[nilason]

Probably time to merge in order to receive testing on Windows and Mac?

That would be the ultimate testing arena!

[petrasovaa]

I will test myself, but also there are some failing tests that seem related (in Ubuntu 18/20 tests).

[wenzeslaus]

Please, rebase or merge with the master branch to get a better working test result reporting (stderr in CI instead of stdout, downloadable report on failure).

[nilason]

Locally running python3 -m grass.gunittest.main --grassdata $HOME/grassdata --location nc_spm_full_v2alpha2 --location-type nc --min-success 100 gives 3 (and a half) error:

python/grass/pygrass/vector/testsuite/test_pygrass_vector_doctests.py
        ...
        File "etc/python/grass/pygrass/vector/table.py", line 789, in grass.pygrass.vector.table.Link.connection
        Failed example:
            cur.execute("SELECT cat, name, value from %s" %
                        link.table_name)              # doctest: +ELLIPSIS
        Expected:
            <sqlite3.Cursor object at ...>
        Got:
            <pysqlite2.dbapi2.Cursor object at 0x7f8391b903b0>
        ....


scripts/r.import/testsuite/test_parallel.sh
scripts/r.import/testsuite/test_r_import.py
        ....
        ERROR: Unable to open datasource <data/data2.asc>
        ERROR: Unable to open datasource <data/data2.asc>
        ERROR: Unable to read GDAL dataset <data/data2.asc>
        ....


temporal/t.rast.export/testsuite/test_rast_export.py
        ....
        ERROR: Value <AAIGrid> out of range for parameter <format>
        	Legal range: VRT,GTiff,COG,NITF,HFA,ELAS,DTED,PNG,JPEG,MEM,GIF,XPM,BMP,PCIDSK,PCRaster,ILWIS,SGI,SRTMHGT,Leveller,Terragen,ISIS3,ISIS2,PDS4,VICAR,ERS,FIT,GRIB,RMF,WMS,RST,INGR,GSAG,GSBG,GS7BG,R,WEBP,Rasterlite,MBTiles,CALS,WMTS,MRF,PNM,PAux,MFF,MFF2,BT,LAN,IDA,LCP,GTX,NTv2,CTable2,KRO,ROI_PAC,RRASTER,BYN,ARG,USGSDEM,NWT_GRD,ADRG,BLX,PostGISRaster,SAGA,XYZ,HF2,JPEGLS,ZMap,SIGDEM,GPKG,NGW,ENVI,EHdr,ISCE
        ERROR: Unable to export raster map <a_0>
        ....

Two are GDAL related, the other apparently using a different python sqlite driver than expected. Non are seemingly ctypesgen related.

[petrasovaa]

Two are GDAL related, the other apparently using a different python sqlite driver than expected. Non are seemingly ctypesgen related.

Before the rebasing there were errors in the Ubuntu 18/20 CI tests, I think it was related to 3D rasters, they seemed to be related to ctypesgen. I would check once the CI is working again.

[nilason]

Two are GDAL related, the other apparently using a different python sqlite driver than expected. Non are seemingly ctypesgen related.

Before the rebasing there were errors in the Ubuntu 18/20 CI tests, I think it was related to 3D rasters, they seemed to be related to ctypesgen. I would check once the CI is working again.

I did at some point see similar errors, but I can't remember when, how...
I think we should wait for all CIs giving green light before merging, but it would be great if someone could spare the time compiling and running the tests locally. If there are indeed failures, it will be a lot easier to track those issues down.

[nilason]

At last, I found which old patch for Windows that was needed for ctypesgen to work! This now runs on all platforms without regression and independent of OSGeo4W changes. This may now be merged.

[nilason]

If there are no objections, I will go ahead and merge this in a few days.

[marisn]

If there are no objections, I will go ahead and merge this in a few days.

Yes, please.

[nilason]

I take it no one objects merging this, so I'll throw it in.