Incompatibility with python3.10 #458

sssangha · 2022-03-15T08:20:23Z

In the rangecoreg step, topsApp crashes with the following error message:

....
  File "/u/trappist-r0/ssangha/conda_installation/stable_mach14_2022/envs/ARIA-tools/lib/python3.10/site-packages/isce/components/iscesys/C
omponent/Component.py", line 179, in __call__
    return getattr(self, self.__class__.__name__.lower())(*args)
  File "/u/trappist-r0/ssangha/conda_installation/stable_mach14_2022/envs/ARIA-tools/lib/python3.10/site-packages/isce/components/mroipac/a
mpcor/Ampcor.py", line 335, in ampcor
    self.setState()
  File "/u/trappist-r0/ssangha/conda_installation/stable_mach14_2022/envs/ARIA-tools/lib/python3.10/site-packages/isce/components/mroipac/a
mpcor/Ampcor.py", line 484, in setState
    ampcor.setImageDataType1_Py(str(self.imageDataType1))
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

As documented in other cases (https://stackoverflow.com/questions/70705404/systemerror-py-ssize-t-clean-macro-must-be-defined-for-formats), this error results from isce not being compatible with python3.10.

However, I've confirmed this error is circumvented with using an earlier version (python 3.9.10). So until this compatibility issue is resolved, <python3.10 dependencies should be enforced.

The text was updated successfully, but these errors were encountered:

See isce-framework#458 for detailed description of issue addressed here in the Readme

sssangha · 2022-03-15T08:24:30Z

While I suggested a change to the readme in PR #459 , we need to also impose a hard constraint for now in the conda distribution, with which I experienced these issues

rtburns-jpl · 2022-03-24T20:58:28Z

I pushed a commit to #424 to try to fix this, can you try again on that branch and see if it works now?

yunjunz · 2022-05-17T03:50:11Z

I got the similar SystemError: PY_SSIZE_T_CLEAN as below while running dem.py using conda installed isce2 in python 3.10.

dem.py -a stitch -b 37 40 -119 -114 -r -s 1 -c
...
API open (WR): demLat_N37_N40_Lon_W119_W114.dem.wgs84
Traceback (most recent call last):
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/applications/dem.py", line 171, in <module>
    sys.exit(main())
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/applications/dem.py", line 141, in main
    demImg = ds.correct()
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/components/contrib/demUtils/DemStitcher.py", line 862, in correct
    return cg(image,conversionType) if image else cg(self._image,conversionType)
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/components/contrib/demUtils/Correct_geoid_i2_srtm.py", line 182, in __call__
    self.correct_geoid_i2_srtm()
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/components/contrib/demUtils/Correct_geoid_i2_srtm.py", line 196, in correct_geoid_i2_srtm
    self.setState()
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/components/contrib/demUtils/Correct_geoid_i2_srtm.py", line 239, in setState
    correct_geoid_i2_srtm.setGeoidFilename_Py(self.geoidFilename)
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

I tried to add #define PY_SSIZE_T_CLEAN to correct_geoid_i2_srtmmodule.cpp#L30, then the following Segmentation fault error came out instead:

API open (R): ./demLat_N37_N40_Lon_W119_W114.dem
API close:  ./demLat_N37_N40_Lon_W119_W114.dem
Writing geotrans to VRT for ./demLat_N37_N40_Lon_W119_W114.dem
GDAL open (R): ./demLat_N37_N40_Lon_W119_W114.dem.vrt
API open (WR): demLat_N37_N40_Lon_W119_W114.dem.wgs84
Segmentation fault

EJFielding · 2022-07-19T14:58:13Z

I hit this problem with a Python 3.10 installed by Anaconda also. Is there a way to define the required PY_SSIZE_T_CLEAN globally in all of ISCE2 to make it compatible with Python 3.10? Or does it require other changes?

rtburns-jpl · 2022-07-19T19:02:37Z

If you're using cmake, you can try export CFLAGS=-DPY_SSIZE_T_CLEAN and export CXXFLAGS=-DPY_SSIZE_T_CLEAN before running cmake configure, that should set those macros globally for all C/C++ code. Is that sufficient to make it compatible?

EJFielding · 2022-07-20T02:54:55Z

I ran into this problem in the NISAR On-demand system where the environment setup notebook installs ISCE2 (and ISCE3) with Anaconda and a yml file that lists isce2 and all the dependencies. Anaconda defaults to Python 3.10 now.

rtburns-jpl · 2022-07-20T17:55:29Z

@yunjunz I think I figured out why you're getting a segfault here. Python3.10 changed the size type, so any time we use PyArg_ParseTuple with a 's#' argument we need to change the result var from an int to a Py_ssize_t. Can you check if defining PY_SSIZE_T_CLEAN along with this patch this fixes your dem.py workflow?

--- a/contrib/demUtils/correct_geoid_i2_srtm/bindings/correct_geoid_i2_srtmmodule.cpp
+++ b/contrib/demUtils/correct_geoid_i2_srtm/bindings/correct_geoid_i2_srtmmodule.cpp
@@ -172,12 +172,13 @@ PyObject * setConversionType_C(PyObject* self, PyObject* args)
 PyObject * setGeoidFilename_C(PyObject* self, PyObject* args)
 {
     char * varChar;
-    int  var;
+    Py_ssize_t var;
     if(!PyArg_ParseTuple(args, "s#", &varChar ,&var))
     {
         return NULL;
     }
-    setGeoidFilename_f(varChar,&var);
+    int ivar = var;
+    setGeoidFilename_f(varChar, &ivar);
     return Py_BuildValue("i", 0);
 }

I'll try to fix the other occurrences of this in isce2 to hopefully get us working with python 3.10.

yunjunz · 2022-07-21T17:37:32Z

Thank you for fixing @rtburns-jpl. I was trying to test it, but there is a problem while downloading SRTM from the USGS website, so I could not confirm if the fix works or not.

yuankailiu · 2022-07-21T20:08:37Z

Thank you for fixing @rtburns-jpl. I was trying to test it, but there is a problem while downloading SRTM from the USGS website, so I could not confirm if the fix works or not.

Yeh, @yunjunz, right now, it shows an URL error for me

curl -n  -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O http://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N4
1W125.SRTMGL1.hgt.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   299    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 503
2022-07-21 12:48:57,859 - isce.contrib.demUtils.DemStitcher - WARNING - There was a problem in retrieving the file  http://e4ftl01.cr.
usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N41W125.SRTMGL1.hgt.zip. Exception

rtburns-jpl · 2022-07-22T19:18:01Z

That's unfortunate. I thought this was a temporary outage but maybe their download url format has changed? Should we open a separate issue for that?

EJFielding · 2022-07-22T19:32:58Z

If they changed the SRTM download URLs, then I would recommend a new issue. I was going to open one earlier this week but then it worked for me again at least once. I did not find any announcements about changes but they tend to change it about every two years.

AhmetMA · 2022-07-24T04:19:32Z

If they changed the SRTM download URLs, then I would recommend a new issue. I was going to open one earlier this week but then it worked for me again at least once. I did not find any announcements about changes but they tend to change it about every two years.

A recently updated thread regarding the connectivity issues:
https://forum.earthdata.nasa.gov/viewtopic.php?t=2262#p11891

sisi-ali · 2022-08-04T12:23:39Z

I have had this error for about a month now.
Has anyone encountered this error before and solved it

EJFielding · 2022-08-04T21:38:53Z

I have had this error for about a month now. Has anyone encountered this error before and solved it

Hi sisi-ali,

Yes, we are all getting similar errors. It seems the Land Processes DAAC servers are overloaded for a long time. It works occasionally.

sisi-ali · 2022-08-05T05:38:58Z

thank for reply me
But I can't download even though I'm always trying.
You have no solution for this problem. I really need this problem to be solved so that I can move forward with my work

yunjunz · 2022-08-10T21:56:12Z

@yunjunz I think I figured out why you're getting a segfault here. Python3.10 changed the size type, so any time we use PyArg_ParseTuple with a 's#' argument we need to change the result var from an int to a Py_ssize_t. Can you check if defining PY_SSIZE_T_CLEAN along with this patch this fixes your dem.py workflow?
--- a/contrib/demUtils/correct_geoid_i2_srtm/bindings/correct_geoid_i2_srtmmodule.cpp
+++ b/contrib/demUtils/correct_geoid_i2_srtm/bindings/correct_geoid_i2_srtmmodule.cpp
@@ -172,12 +172,13 @@ PyObject * setConversionType_C(PyObject* self, PyObject* args)
 PyObject * setGeoidFilename_C(PyObject* self, PyObject* args)
 {
     char * varChar;
-    int  var;
+    Py_ssize_t var;
     if(!PyArg_ParseTuple(args, "s#", &varChar ,&var))
     {
         return NULL;
     }
-    setGeoidFilename_f(varChar,&var);
+    int ivar = var;
+    setGeoidFilename_f(varChar, &ivar);
     return Py_BuildValue("i", 0);
 }
I'll try to fix the other occurrences of this in isce2 to hopefully get us working with python 3.10.

@rtburns-jpl with #560 being merged, I confirmed dem.py works fine under python 3.10 with the changes you laid out. Thank you and cheers!

EJFielding · 2022-08-10T22:23:31Z

@sisi-ali Please get the new version of ISCE2. We have finally found the necessary change to the DEM downloading code in #560.

sisi-ali · 2022-08-21T08:18:47Z

Hi everyone
I want to calculate the closure_phase_bias.py using the MintPy software that I per-processed on ISCE,
but I cant calculate with (https://github.com/insarlab/MintPy/blob/main/mintpy/closure_phase_bias.py)
I would be grateful if you could guide me

EJFielding · 2022-08-22T00:05:39Z

@sisi-ali It would be better to ask on the MintPy discussion site https://github.com/insarlab/MintPy/discussions or on their Google Group.

sisi-ali · 2022-10-11T08:54:35Z

hi dear Eric Fielding I want to calculate the closure_phase_bias.py using the MintPy software that I per-processed on ISCE, but I cant calculate with ( https://github.com/insarlab/MintPy/blob/main/mintpy/closure_phase_bias.py) I would be grateful if you could guide me ‫‪Eric Fielding‬‏ ***@***.***‬‏> در تاریخ پنجشنبه ۱۱ اوت ۲۰۲۲ ساعت ۲:۵۳ نوشت:‬

…

@sisi-ali <https://github.com/sisi-ali> Please get the new version of ISCE2. We have finally found the necessary change to the DEM downloading code in #560 <#560>. — Reply to this email directly, view it on GitHub <#458 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZ3AKUSOWIULYCD26XTE3T3VYQTW7ANCNFSM5QX4THTA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

krive050 · 2022-10-21T20:16:25Z

Hello,

I running ISCE 2.6.1. The topsApp.py crashes at the DEM steps when it is trying to convert to DEM.

I am getting the same error "SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats"

Is anyone able to help me with this? Thanks!

EJFielding · 2022-10-21T21:25:46Z

@krive050 The easy workaround is to change your Python to version 3.9 instead of version 3.10.

rtburns-jpl · 2022-11-01T20:48:11Z

@krive050 Please check if #534 resolves your issue. I will merge that PR once I have verification that any workflow is fixed by it.

JolinnaYoung · 2023-12-25T08:36:34Z

@yunjunz Hi Dr.Yunjun! I met the same question when using dem.py with Python's version = 3.10. I've tried to add #define PY_SSIZE_T_CLEAN to [correct_geoid_i2_srtmmodule.cpp#L30] but nothing changed at all. Could you please give me hints on solving this problem? Many thanks!

EJFielding · 2023-12-25T19:42:21Z

@yunjunz Hi Dr.Yunjun! I met the same question when using dem.py with Python's version = 3.10. I've tried to add #define PY_SSIZE_T_CLEAN to [correct_geoid_i2_srtmmodule.cpp#L30] but nothing changed at all. Could you please give me hints on solving this problem? Many thanks!

Hi Jolinna, you should upgrade to ISCE2 version 2.6.3. It has been updated to work with Python 3.10.

JolinnaYoung · 2023-12-26T02:14:58Z

@yunjunz Hi Dr.Yunjun! I met the same question when using dem.py with Python's version = 3.10. I've tried to add #define PY_SSIZE_T_CLEAN to [correct_geoid_i2_srtmmodule.cpp#L30] but nothing changed at all. Could you please give me hints on solving this problem? Many thanks!

Hi Jolinna, you should upgrade to ISCE2 version 2.6.3. It has been updated to work with Python 3.10.

Hi Eric, thanks for your kind reply! I'll try it!

sssangha added a commit to sssangha/isce2 that referenced this issue Mar 15, 2022

Edited to reflect python3.10 incompatibility

6aaf69d

See isce-framework#458 for detailed description of issue addressed here in the Readme

sssangha added a commit to sssangha/isce2 that referenced this issue Mar 15, 2022

Edited to reflect python3.10 incompatibility

207e36e

See isce-framework#458 for detailed description of issue addressed here in the Readme

sssangha mentioned this issue Mar 15, 2022

Edited to reflect python3.10 incompatibility #459

Closed

jaha2600 mentioned this issue Mar 15, 2022

db file doesnt exist - run_08_timeseries_misreg #460

Closed

EJFielding mentioned this issue Jul 20, 2022

DEM.py issues with Python 3.10 #506

Closed

rtburns-jpl linked a pull request Jul 21, 2022 that will close this issue

Fix cpython bindings for python 3.10+ Py_ssize_t #534

Merged

yunjunz mentioned this issue Jul 31, 2022

Tutorial for closure_phase_bias.py insarlab/MintPy-tutorial#40

Merged

jhkennedy mentioned this issue Aug 29, 2022

Pin python <3.10 because of an ISCE2 incompatibility ASFHyP3/hyp3-autorift#158

Merged

rtburns-jpl added the bug Something isn't working label Nov 1, 2022

rtburns-jpl closed this as completed in #534 Jan 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incompatibility with python3.10 #458

Incompatibility with python3.10 #458

sssangha commented Mar 15, 2022

sssangha commented Mar 15, 2022

rtburns-jpl commented Mar 24, 2022

yunjunz commented May 17, 2022

EJFielding commented Jul 19, 2022

rtburns-jpl commented Jul 19, 2022

EJFielding commented Jul 20, 2022

rtburns-jpl commented Jul 20, 2022

yunjunz commented Jul 21, 2022

yuankailiu commented Jul 21, 2022

rtburns-jpl commented Jul 22, 2022

EJFielding commented Jul 22, 2022

AhmetMA commented Jul 24, 2022

sisi-ali commented Aug 4, 2022

EJFielding commented Aug 4, 2022

sisi-ali commented Aug 5, 2022

yunjunz commented Aug 10, 2022

EJFielding commented Aug 10, 2022

sisi-ali commented Aug 21, 2022

EJFielding commented Aug 22, 2022

sisi-ali commented Oct 11, 2022 via email

krive050 commented Oct 21, 2022

EJFielding commented Oct 21, 2022

rtburns-jpl commented Nov 1, 2022

JolinnaYoung commented Dec 25, 2023

EJFielding commented Dec 25, 2023

JolinnaYoung commented Dec 26, 2023

Incompatibility with python3.10 #458

Incompatibility with python3.10 #458

Comments

sssangha commented Mar 15, 2022

sssangha commented Mar 15, 2022

rtburns-jpl commented Mar 24, 2022

yunjunz commented May 17, 2022

EJFielding commented Jul 19, 2022

rtburns-jpl commented Jul 19, 2022

EJFielding commented Jul 20, 2022

rtburns-jpl commented Jul 20, 2022

yunjunz commented Jul 21, 2022

yuankailiu commented Jul 21, 2022

rtburns-jpl commented Jul 22, 2022

EJFielding commented Jul 22, 2022

AhmetMA commented Jul 24, 2022

sisi-ali commented Aug 4, 2022

EJFielding commented Aug 4, 2022

sisi-ali commented Aug 5, 2022

yunjunz commented Aug 10, 2022

EJFielding commented Aug 10, 2022

sisi-ali commented Aug 21, 2022

EJFielding commented Aug 22, 2022

sisi-ali commented Oct 11, 2022 via email

krive050 commented Oct 21, 2022

EJFielding commented Oct 21, 2022

rtburns-jpl commented Nov 1, 2022

JolinnaYoung commented Dec 25, 2023

EJFielding commented Dec 25, 2023

JolinnaYoung commented Dec 26, 2023