python plugin fails to build on aarch64 (oracle linux) #1164

unxed · 2022-01-01T12:45:10Z

[ 99%] Generating _pyfar.cpp
Traceback (most recent call last):
  File "/home/opc/far2l/python/src/pythongen.py", line 43, in <module>
    ffi.cdef(data, packed=True)
  File "/usr/lib64/python3.6/site-packages/cffi/api.py", line 107, in cdef
    self._cdef(csource, override=override, packed=packed)
  File "/usr/lib64/python3.6/site-packages/cffi/api.py", line 121, in _cdef
    self._parser.parse(csource, override=override, **options)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 315, in parse
    self._internal_parse(csource)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 338, in _internal_parse
    self._parse_decl(decl)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 437, in _parse_decl
    self._get_struct_union_enum_type('enum', node)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 706, in _get_struct_union_enum_typ
e
    tp = self._build_enum_type(explicit_name, type.values)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 845, in _build_enum_type
    nextenumvalue = self._parse_constant(enum.value)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 832, in _parse_constant
    "simple numeric constant" % exprnode.coord.line)
cffi.error.FFIError: :308: unsupported expression: expected a simple numeric constant
make[2]: *** [python/CMakeFiles/python.dir/build.make:74: python/_pyfar.cpp] Error 1
make[1]: *** [CMakeFiles/Makefile2:1203: python/CMakeFiles/python.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

The text was updated successfully, but these errors were encountered:

elfmz · 2022-01-01T16:09:51Z

ping @m32

m32 · 2022-01-01T21:52:03Z

Cant reproduce this error, fresh build:
[ 97%] Built target NetRocks-WebDAV
[ 97%] Generating _pyfar.cpp
generating /devel/far2l/build/python/_pyfar.cpp
Scanning dependencies of target python
[ 98%] Building CXX object python/CMakeFiles/python.dir/src/python.cpp.o
[ 98%] Building CXX object python/CMakeFiles/python.dir/_pyfar.cpp.o
[ 98%] Linking CXX shared module ../install/Plugins/python/plug/python.far-plug-wide
[ 98%] Built target python

It must be around line 308, something like :
enum ... {
KEY_PRNTSCRN = 1<<0,
...
}

unxed · 2022-01-02T00:39:44Z

Line 308 of what file? May I gather some other debug info somehow?

elfmz · 2022-01-02T00:57:22Z

First of all, try to build on completely fresh pulled repo (if it wasn't so)

m32 · 2022-01-02T03:03:09Z

File "/home/opc/far2l/python/src/pythongen.py", line 43, in
pythongen line 43: it parses farwin.h, farcolor.hpp, farkeys.hpp and plugin.hpp identical to installffi from pluginmanager.py, when pythongen fails then pluginmanager will fail too, adding print(fqname) before open will show which file is parsed

File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 832, in _parse_constant
"simple numeric constant" % exprnode.coord.line)
cffi.error.FFIError: :308: unsupported expression: expected a simple numeric constant
308 is line number in parsed file

I have also build on fresh checkout - no errors

unxed · 2022-01-02T07:51:54Z

it is
/home/opc/far2l/python/configs/plug/far2l/farkeys.hpp
looks like "unsupported" is about expressions in enum values definitions.

temporary commented out such expressions, now it fails with

/home/opc/far2l/python/configs/plug/far2l/plugin.hpp
Traceback (most recent call last):
  File "/home/opc/far2l/python/src/pythongen.py", line 44, in <module>
    ffi.cdef(data, packed=True)
  File "/usr/lib64/python3.6/site-packages/cffi/api.py", line 107, in cdef
    self._cdef(csource, override=override, packed=packed)
  File "/usr/lib64/python3.6/site-packages/cffi/api.py", line 121, in _cdef
    self._parser.parse(csource, override=override, **options)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 315, in parse
    self._internal_parse(csource)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 338, in _internal_parse
    self._parse_decl(decl)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 437, in _parse_decl
    self._get_struct_union_enum_type('enum', node)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 706, in _get_struct_union_enum_typ
e
    tp = self._build_enum_type(explicit_name, type.values)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 845, in _build_enum_type
    nextenumvalue = self._parse_constant(enum.value)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 790, in _parse_constant
    return int(s, 16)
ValueError: invalid literal for int() with base 16: '0x000000ffUL'
make[2]: *** [python/CMakeFiles/python.dir/build.make:74: python/_pyfar.cpp] Error 1
make[1]: *** [CMakeFiles/Makefile2:1203: python/CMakeFiles/python.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

unxed · 2022-01-02T08:00:18Z

$ cat /etc/oracle-release
Oracle Linux Server release 8.4
$ uname -m
aarch64

m32 · 2022-01-02T10:20:17Z

0x000000ffUL is invalid in python, should be 0x000000ff

elfmz · 2022-01-02T10:33:09Z

May remove that suffixes if it only reason of, dunno why they're there - they're useless in definition of enum's fields. However its then unclear why works for intel builds.

unxed · 2022-01-02T11:05:05Z

May remove that suffixes if it only reason of, dunno why they're there - they're useless in definition of enum's fields. However its then unclear why works for intel builds.

Worked! Thanks

unxed · 2022-01-02T11:09:12Z

So it looks like UL suffixes may safely be removed. But I don't know what to do with expressions in ENUMs

elfmz · 2022-01-02T11:10:53Z

can you try to replace the with ul suffixes? (lowercased) may be it will work (and will be safer change)

unxed · 2022-01-02T11:15:44Z

ValueError: invalid literal for int() with base 16: '0x000000fful'

elfmz · 2022-01-02T11:16:53Z

but why it works for intel? may be oracle linux has some older FFI of whatelse?

elfmz · 2022-01-02T11:18:56Z

while suffixes may be removed, removing complex expressions from farkeys.hpp is harder.. BTW regarding suffixes it seems it really was a bug there and fixed - https://foss.heptapod.net/pypy/cffi/-/issues/407

unxed · 2022-01-02T11:19:29Z

how do i check cffi version used by far2l?

m32 · 2022-01-02T12:32:25Z

run this command in python/bin folder:
pip freeze |grep cffi
on ubuntu cffi=1.15.0

unxed · 2022-01-02T13:01:24Z

$ ./pip freeze |grep cffi
You are using pip version 9.0.3, however version 21.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
cffi==1.11.5

m32 · 2022-01-02T13:54:57Z

KEY_CTRLSHIFTA = (KEY_CTRL | KEY_SHIFT) + 'A' - this is quite a complicated expression and cffi works fine.

https://pypi.org/project/cffi has 1.15.0 for cffi on aarch64 for python >= 3.6 and one of them should be downloaded when far2l is built

Which version of Python is on your machine?

unxed · 2022-01-02T14:29:53Z

$ python --version
Python 3.6.8
cd far2l/build/install/Plugins/python/plug/python/bin
./python --version
Python 3.6.8
./python3 --version
Python 3.6.8

unxed · 2022-01-02T14:33:56Z

...build/install/Plugins/python/plug/python/bin$ ./pip show cffi                                   ↑
Name: cffi
Version: 1.11.5
Summary: Foreign Function Interface for Python calling C code.
Home-page: http://cffi.readthedocs.org
Author: Armin Rigo, Maciej Fijalkowski
Author-email: python-cffi@googlegroups.com
License: MIT
Location: /usr/lib64/python3.6/site-packages
Requires: pycparser
You are using pip version 9.0.3, however version 21.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

unxed · 2022-01-02T14:39:44Z

running

cd far2l/python/configs/plug/python/bin
./python -m pip install --upgrade pip
./python -m pip install --upgrade cffi

solved an issue with expressions in ENUMs for me

elfmz · 2022-01-02T15:01:55Z

What about UL suffixes? still making problems?

unxed · 2022-01-02T15:20:34Z

What about UL suffixes? still making problems?

nope :)

elfmz · 2022-01-02T21:43:50Z

so issue resolved? however due to looked at that code, i've made old planned change - refactored python plugin build, and now it doesn't create intermediate stuff in sources tree, but instead putting all that stuff into build directory. Also i've made preprocessing of far2l's header files also during build, instead of having 'hard-commited' copies on git.
It all looks working, but it would be nice if @m32 would look at this change just in case...

unxed · 2022-01-02T21:57:55Z

tried a fresh git checkout... still errors if trying to build without cffi update by running

./python -m pip install --upgrade pip
./python -m pip install --upgrade cffi

in far2l/build/python/python/bin folder

maybe we should keep this issue open for anyone who will encounter the same errors also?

elfmz · 2022-01-02T22:01:25Z

but i didnt fix it, for now it too complicated to eliminate that complex expressions, especially if this fixable just by updating python package

unxed · 2022-01-02T22:30:26Z

may python packages be updated somehow from build script?

m32 · 2022-01-03T10:34:26Z

so issue resolved? however due to looked at that code, i've made old planned change - refactored python plugin build, and now it doesn't create intermediate stuff in sources tree, but instead putting all that stuff into build directory. Also i've made preprocessing of far2l's header files also during build, instead of having 'hard-commited' copies on git. It all looks working, but it would be nice if @m32 would look at this change just in case...

there is still a file renaming in pluginmanager.py
for fname in ("farwin.h", "farcolor.h", "farkeys.h", "farplug-wide.h"):

i did the same, with pycpp (preprocessor) i automatically change the headers to one far2l.hh file. I added one _FAR_PYTHON_GEN variable to the preprocessor definition, but I'm still testing it.
one of the changes is in farkeys.hppL KEY_BRACKET = '[' not liked by cffi

INT_PTR - is hard coded for 64-bit systems, after this change fartty works on raspberry

may python packages be updated somehow from build script?
yes, in another add_custom_command(TARGET python POST_BUILD COMMAND ...python -m pip install --upgrade ....)

m32 · 2022-01-03T23:59:41Z

I did it by hand so i did it automagically :)

#1164 - automagic generation of far2lcffi.py file

elfmz · 2022-01-04T00:17:52Z

but you broken 32 bits again:( or not?

m32 · 2022-01-04T00:20:25Z

32/64 - I haven't touched that yet, in pythongen.py there is if True: for this task

elfmz · 2022-01-04T00:25:22Z

you may check if target bitness is 64 at prebuild.sh stage by
echo $'#ifdef __LP64__\nTARGET64\n#endif' | $4 -E - | grep TARGET64
(and for that need to return passing "${CMAKE_C_COMPILER}" as $4 again)

elfmz · 2022-01-04T00:27:18Z

-E invokes compiler in preprocessor mode, i.e. it doesnt compile, just processes macroses. See how it works:

$ echo $'#ifdef __LP64__\nTARGET64\n#endif' | gcc -E - | grep TARGET64
TARGET64
$ echo $'#ifdef __LP64__\nTARGET64\n#endif' | gcc -m32 -E - | grep TARGET64
$

m32 · 2022-01-04T00:27:48Z

however, I prefer in python:
if ctypes.sizeof(ctypes.cint) == ctypes.sizeof(ctypes.void_p):
# 32 bit platform
pass
else:
# 64 bit platform
cpp.define("LP64")

this works on pi and my laptop

elfmz · 2022-01-04T00:28:23Z

build system not always == target, sometimes its a cross-compilations :)

elfmz · 2022-01-04T00:29:26Z

..albeit im not sure that python plugin build fully OK for cross-compilation in other aspects..

m32 · 2022-01-04T00:29:39Z

then we need python as cross compiler ? would be hard todo

m32 · 2022-01-04T00:31:33Z

I also compile python4android at home, only problems and it's not stable

m32 · 2022-01-04T00:39:23Z

How to find PATH_MAX and NAME_MAX in python - i have hardcoded defaults from sys/limits.h ?
Or maybe all the parameters of the platform should be calculated in the shell?

elfmz · 2022-01-04T00:49:10Z

in similar way, later i will add creating some incpy/target.h with that things obtained from gcc during configuration, but its tomorrow, its too late today already

m32 · 2022-01-04T01:15:47Z

done in python #1173

elfmz · 2022-01-04T09:37:29Z

still, i prefer to make it in cross-compile-friendly, so see 90b09cc

m32 · 2022-01-04T09:40:02Z

It's OK for me

m32 · 2022-01-04T11:12:00Z

Looks finished to me.
@unxed

unxed · 2022-01-04T22:45:47Z

still
cffi.error.FFIError: :307: unsupported expression: expected a simple numeric constant
on fresh build folder if pip & cffi are not upgraded by hands

if it is not considered a problem, may close

m32 · 2022-01-04T23:35:54Z

Maybe your python package cache is corrupted?
rm -rf ~/.cache/pip

unxed · 2022-01-04T23:43:51Z

tried rm -rf ~/.cache/pip (and removed build/python after it), still errors

elfmz · 2022-01-04T23:53:29Z

check now plz

unxed · 2022-01-05T09:17:38Z

Cache entry deserialization failed, entry ignored
Collecting pip
  Using cached https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03
ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-21.3.1
Requirement already satisfied: cffi in /usr/lib64/python3.6/site-packages (1.11.5)
Collecting debugpy
  Using cached debugpy-1.5.1-py2.py3-none-any.whl (4.1 MB)
Collecting pcpp
  Using cached pcpp-1.30-py2.py3-none-any.whl (91 kB)
Requirement already satisfied: pycparser in /usr/lib/python3.6/site-packages (from cffi) (2.14)
Installing collected packages: pcpp, debugpy
Successfully installed debugpy-1.5.1 pcpp-1.30
WARNING: yacc table file version is out of date
Traceback (most recent call last):
  File "/home/opc/far2l/python/src/pythongen.py", line 71, in <module>
    from far2lcffi import ffi, cffi
  File "/home/opc/far2l/build/python/incpy/far2lcffi.py", line 3362, in <module>
  File "/usr/lib64/python3.6/site-packages/cffi/api.py", line 107, in cdef
    self._cdef(csource, override=override, packed=packed)
  File "/usr/lib64/python3.6/site-packages/cffi/api.py", line 121, in _cdef
    self._parser.parse(csource, override=override, **options)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 315, in parse
    self._internal_parse(csource)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 338, in _internal_parse
    self._parse_decl(decl)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 437, in _parse_decl
    self._get_struct_union_enum_type('enum', node)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 706, in _get_struct_union_enum_ty
pe
    tp = self._build_enum_type(explicit_name, type.values)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 845, in _build_enum_type
    nextenumvalue = self._parse_constant(enum.value)
  File "/usr/lib64/python3.6/site-packages/cffi/cparser.py", line 790, in _parse_constant
    return int(s, 16)
ValueError: invalid literal for int() with base 16: '0x000000ffUL'
make[2]: *** [python/CMakeFiles/python.dir/build.make:80: python/incpy/far2lcffi.py] Error 1
make[1]: *** [CMakeFiles/Makefile2:1247: python/CMakeFiles/python.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

unxed · 2022-01-05T09:18:33Z

adding

 	"$DST/python/bin/python" -m pip install --upgrade cffi

just after

 	"$DST/python/bin/python" -m pip install --upgrade pip

fixes build

elfmz · 2022-01-05T09:26:51Z

now?

unxed · 2022-01-05T09:54:51Z

great! closing.

m32 · 2022-01-05T09:55:00Z

File "/usr/lib64/python3.6/site-packages/cffi/api.py", line 107, in cdef

It looks like you are not using virtualenv - python3-cffi is in system installed packages, which is outdated
sudo apt uninstall python3-cffi

unxed · 2022-01-05T11:13:35Z

Unfortunately I am not the only user of that aarch64 machine, so I am afraid to uninstall packages there

m32 · 2022-01-05T11:15:54Z

Another option is virtualenv with no system packages:
$PYTHON -m venv "$DST/python"

unxed mentioned this issue Jan 2, 2022

far2l ecosystem #647

Open

elfmz added a commit that referenced this issue Jan 4, 2022

Merge pull request #1172 from m32/master

eab8777

#1164 - automagic generation of far2lcffi.py file

elfmz added a commit that referenced this issue Jan 4, 2022

python: do pip upgrade on prebuild step (touch #1164)

28ad920

unxed closed this as completed Jan 5, 2022

python plugin fails to build on aarch64 (oracle linux) #1164

python plugin fails to build on aarch64 (oracle linux) #1164

Comments

unxed commented Jan 1, 2022

elfmz commented Jan 1, 2022

m32 commented Jan 1, 2022 • edited Loading

unxed commented Jan 2, 2022

elfmz commented Jan 2, 2022

m32 commented Jan 2, 2022

unxed commented Jan 2, 2022

unxed commented Jan 2, 2022 • edited Loading

m32 commented Jan 2, 2022

elfmz commented Jan 2, 2022

unxed commented Jan 2, 2022 • edited Loading

unxed commented Jan 2, 2022

elfmz commented Jan 2, 2022

unxed commented Jan 2, 2022

elfmz commented Jan 2, 2022

elfmz commented Jan 2, 2022

unxed commented Jan 2, 2022

m32 commented Jan 2, 2022

unxed commented Jan 2, 2022

m32 commented Jan 2, 2022

unxed commented Jan 2, 2022 • edited Loading

unxed commented Jan 2, 2022

unxed commented Jan 2, 2022

elfmz commented Jan 2, 2022

unxed commented Jan 2, 2022

elfmz commented Jan 2, 2022

unxed commented Jan 2, 2022 • edited Loading

elfmz commented Jan 2, 2022

unxed commented Jan 2, 2022

m32 commented Jan 3, 2022

m32 commented Jan 3, 2022

elfmz commented Jan 4, 2022

m32 commented Jan 4, 2022

elfmz commented Jan 4, 2022

elfmz commented Jan 4, 2022

m32 commented Jan 4, 2022 • edited Loading

elfmz commented Jan 4, 2022

elfmz commented Jan 4, 2022

m32 commented Jan 4, 2022

m32 commented Jan 4, 2022

m32 commented Jan 4, 2022

elfmz commented Jan 4, 2022

m32 commented Jan 4, 2022

elfmz commented Jan 4, 2022

m32 commented Jan 4, 2022

m32 commented Jan 4, 2022

unxed commented Jan 4, 2022

m32 commented Jan 4, 2022

unxed commented Jan 4, 2022

elfmz commented Jan 4, 2022

unxed commented Jan 5, 2022

unxed commented Jan 5, 2022 • edited Loading

elfmz commented Jan 5, 2022

unxed commented Jan 5, 2022

m32 commented Jan 5, 2022

unxed commented Jan 5, 2022

m32 commented Jan 5, 2022

m32 commented Jan 1, 2022 •

edited

Loading

unxed commented Jan 2, 2022 •

edited

Loading

unxed commented Jan 2, 2022 •

edited

Loading

unxed commented Jan 2, 2022 •

edited

Loading

unxed commented Jan 2, 2022 •

edited

Loading

m32 commented Jan 4, 2022 •

edited

Loading

unxed commented Jan 5, 2022 •

edited

Loading