tests.test_contextvars.ContextVarsTests.test_break_ctxvars on 32-bit Linux PowerPC

[glaubitz]

Running the testsuite on 32-bit Linux PowerPC fails immediately with a segmentation fault:

(sid_powerpc-dchroot)glaubitz@perotto:~/greenlet/build/lib.linux-ppc-cpython-312/greenlet$ PYTHONPATH=/home/glaubitz/greenlet/build/lib.linux-ppc-cpython-312 python3.12 -m
 unittest discover -v
test_break_ctxvars (tests.test_contextvars.ContextVarsTests.test_break_ctxvars) ... Segmentation fault (core dumped)
(sid_powerpc-dchroot)glaubitz@perotto:~/greenlet/build/lib.linux-ppc-cpython-312/greenlet$

Backtrace in GDB shows:

(sid_powerpc-dchroot)glaubitz@perotto:~/greenlet/build/lib.linux-ppc-cpython-312/greenlet$ gdb /usr/bin/python3.12 core.1669529
GNU gdb (Debian 15.1-1) 15.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/python3.12...
Reading symbols from /usr/lib/debug/.build-id/a6/4f9ab5e6f2702d682cd21b2f9f8c0273be0ac6.debug...
[New LWP 1669529]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc-linux-gnu/libthread_db.so.1".
Core was generated by `python3.12 -m unittest discover -v'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0fa9855c in green_init (self=..., args=<error reading variable: Cannot access memory at address 0x0>,
    kwargs=<error reading variable: Cannot access memory at address 0x0>) at src/greenlet/greenlet.cpp:253
253         if (!PyArg_ParseTupleAndKeywords(
(gdb) bt
#0  0x0fa9855c in green_init (self=..., args=<error reading variable: Cannot access memory at address 0x0>, kwargs=<error reading variable: Cannot access memory at address 0x0>) at src/greenlet/greenlet.cpp:253
#1  0x10161ebc in type_call (type=<optimized out>, type@entry=0xfac027c <PyGreenlet_Type>, args=args@entry=(<built-in method run of _contextvars.Context object at remote 0xf740ed68>,), kwds=kwds@entry=0x0)
    at ../Objects/typeobject.c:1677
#2  0x100da1b4 in _PyObject_MakeTpCall (tstate=0x107c1f08 <_PyRuntime+235672>, callable=callable@entry=<type at remote 0xfac027c>, args=args@entry=0xf78cd664, nargs=1, keywords=<optimized out>, keywords@entry=0x0)
    at ../Objects/call.c:240
#3  0x100db0a0 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0xf78cd664, callable=<type at remote 0xfac027c>, tstate=<optimized out>) at ../Include/internal/pycore_call.h:90
#4  0x101bc314 in _PyEval_EvalFrameDefault (tstate=0x107c1f08 <_PyRuntime+235672>, frame=0xf78cd628, throwflag=<optimized out>) at Python/bytecodes.c:2715
#5  0x100dc1c8 in _PyFunction_Vectorcall (kwnames=0x0, nargsf=3, stack=0xffd0730c, func=<function at remote 0xf74a4e38>) at ../Objects/call.c:419
#6  _PyObject_FastCallDictTstate (kwargs=0x0, nargsf=3, args=0xffd0730c, callable=<function at remote 0xf74a4e38>, tstate=0x107c1f08 <_PyRuntime+235672>) at ../Objects/call.c:133
#7  _PyObject_Call_Prepend (tstate=tstate@entry=0x107c1f08 <_PyRuntime+235672>, callable=callable@entry=<function at remote 0xf74a4e38>,
    obj=obj@entry=<_RefCountChecker(testcase=<ContextVarsTests(_testMethodName='test_break_ctxvars', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], collectedDurations=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper(mode='w') at remote 0xf77da7a8>, _original_stderr=<_io.TextIOWrapper(mode='w') at remote 0xf77da820>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<...>) at remote 0xf7451c90>, showAll=True, dots=False, descriptions=True, _newline=False, durations=None, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<TestCaseMetaClass(__module__='tests.test_contextvars', _new_ctx_run=<function at remote 0xf74a65c8>, _increment=<function at remote 0xf74a6618>, _test_context=<function at remote 0xf74a6668>, test_context_propagated_by_context_run=<function at remote 0xf74a6988>, test_context_propaga...(truncated), args=args@entry=((), {}), kwargs=0x0) at ../Objects/call.c:508
#8  0x1016fb8c in slot_tp_call (
    self=self@entry=<_RefCountChecker(testcase=<ContextVarsTests(_testMethodName='test_break_ctxvars', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], collectedDurations=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper(mode='w') at remote 0xf77da7a8>, _original_stderr=<_io.TextIOWrapper(mode='w') at remote 0xf77da820>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<...>) at remote 0xf7451c90>, showAll=True, dots=False, descriptions=True, _newline=False, durations=None, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<TestCaseMetaClass(__module__='tests.test_contextvars', _new_ctx_run=<function at remote 0xf74a65c8>, _increment=<function at remote 0xf74a6618>, _test_context=<function at remote 0xf74a6668>, test_context_propagated_by_context_run=<function at remote 0xf74a6988>, test_context_propaga...(truncated), args=args@entry=((), {}), kwds=kwds@entry=0x0) at ../Objects/typeobject.c:8779

This issue was first seen in Debian with version 2.0.1: https://buildd.debian.org/status/logs.php?pkg=python-greenlet&arch=powerpc

Related bug report in Gentoo is: https://bugs.gentoo.org/923062

[jamadden]

Commit a97d81f might fix this. It reverts a minor change to be more like greenlet 1.x.

[glaubitz]

Thanks for the quick reply. Unfortunately, that change makes no difference and the bug still persists.

I wonder whether whether @he32 has run the testsuite on 32-bit NetBSD PowerPC.

@he32 Can you comment?

[jamadden]

@glaubitz Could you please try db5c781 ? Hopefully, that will at least move the crash to a different location if not fix it entirely. (If it does crash again in a different location, the traceback would be helpful again.)

[glaubitz]

No difference, unfortunately.

GDB yields the following including disassembly:

#0  0x0fa98a6c in greenlet::refs::_BorrowedGreenlet<_greenlet, &greenlet::refs::GreenletChecker>::operator-> (this=<synthetic pointer>)
    at src/greenlet/greenlet_internal.hpp:79
79          return reinterpret_cast<PyGreenlet*>(this->p)->pimpl;
(gdb) x/5i $pc
=> 0xfa98a6c <green_getrun(greenlet::refs::BorrowedGreenlet, void*)+16>:        lwz     r3,16(r9)
   0xfa98a70 <green_getrun(greenlet::refs::BorrowedGreenlet, void*)+20>:        lwz     r9,0(r3)
   0xfa98a74 <green_getrun(greenlet::refs::BorrowedGreenlet, void*)+24>:        stmw    r30,8(r1)
   0xfa98a78 <green_getrun(greenlet::refs::BorrowedGreenlet, void*)+28>:        lwz     r9,40(r9)
   0xfa98a7c <green_getrun(greenlet::refs::BorrowedGreenlet, void*)+32>:        mflr    r30
(gdb)

Btw, you can apply for an account with the GCC Compile Farm here and try it yourself on ssh gcc203.fsffrance.org. I have admin rights on this machine and could help set up the environment.

[jamadden]

Ah. That's a completely different backtrace ( at src/greenlet/greenlet_internal.hpp:79, which is called from green_getrun). I was afraid of this.

For some reason, on this platform, plain-vanilla C++ objects have a layout that's different than on any other platform. This breaks some assumptions. It's fixable, but tedious, and it makes the code harder to read and reason about from a static analysis.

I wonder why the layout is different there. Any chance you're using unusual compiler arguments?

[glaubitz]

No special compiler arguments are being used, it's built the same way as for all the other targets.

[jamadden]

Ok, that brings us to the arguments that greenlet itself supplies.

In #94 from 2015, a workaround that added -Os for ppc linux was committed.

Perhaps that's no longer necessary. Or perhaps it should just be -O0.

[glaubitz]

Ah, that could be it. @karcherm is currently looking into it and his suspicion was it could be "by value" vs. "by reference".

[karcherm]

It seems gcc on powerpc always passes structures by reference, even if their only member is a simple pointer:

#include <stdio.h>

struct X {
        int *ptr;
};

void printptr1(struct X arg)
{
        printf("%p\n", arg.ptr);
}

void printptr2(int *ptr)
{
        printf("%p\n", ptr);
}

typedef void pp(int *);
pp* p[2] = {(pp*)printptr1, printptr2};

int main(void)
{
        int j = 0x11234;
        int* iptr = &j;
        p[0](iptr);
        p[1](iptr);
}

prints the follwing output (no matter whether I use -O0 or -Os:

$ rm testme.o ; make testme.o CFLAGS="-Os -g"; make testme ; ./testme
cc -Os -g   -c -o testme.o testme.c
cc   testme.o   -o testme
0x11234
0xfff70808

[jamadden]

I can confirm, I just found it in Power Architecture 32-bit Application Binary Interface Supplement 1.0 - Linux® & Embedded, section "3.2.3.1. Parameter Passing Register Selection Algorithm", where it says registers can contain:

A struct or union that shall be treated as a pointer to the data object, ...

This is backed up by "3.2.3.2. Parameter Passing Examples". Given the setup code:

typedef struct {
  int    a;
  double dd;
} sparm;
sparm   s, t;
int     c, d, e;
long double ld;
double  ff, gg, hh;
x = func(c, ff, d, ld, s, gg, t, e, hh);

the example states that in the call to x, register r5 winds up containing a "ptr to s." There are no exceptions for single member structs. The rule that the address of a struct is the address of its first member only allows for that optimization but does not require it.

That's unique across Win/Linux/Mac x86 and x86_64, Mac/Linux aarch64, S390x, and ppc64.

[glaubitz]

If the struct just contains one pointer as a member, why not pass the pointer itself?

[jamadden]

Type safety and automatic reference counting. It caught a number of memory leaks when I added it.

[karcherm]

If the struct just contains one pointer as a member, why not pass the pointer itself?

The functions we are talking about are called by more generic code, and they get in fact called using plain pointers. Greenlet attaches semantics to the pointers it receives from the generic code by using C++ smart pointers. There are different structs for pointers that you need to free when you are done, and pointers that are managed by some other code entity, and you don't need to worry about freeing - but you also must not store copies of such pointers that live beyong the current invocation of a function, because they might get freed by the owning code any time after the call is finished. This is called a "borrowed pointer".

The code in greenlet is non-conformant to the C/C++ standard by assuming that a caller that expects to call a function that receives a plain pointer (maybe even a simple void * ) can call functions that have a "struct wrapper type" instead. The struct wrapper makes memory handling easier. Using this non-conformant way instead of adding an extra layer of indirection by writing wrapper functions that convert raw pointers to wrapped pointers and pass them to the actual worker functions simplifies the source code.

[jamadden]

Well said @karcherm , that's all true. I knew it was technically non-conformant, but across 10+ (very) different platforms, I'd never seen an ABI where it actually made a difference. Now I have, and must rework accordingly.

[jamadden]

I think I updated all the slot functions in the issue422 branch. If someone has the opportunity to test that, I'd appreciate it!

If there are still errors, I may have to take @glaubitz up on accessing the GCC compiler farm.

[glaubitz]

I think I updated all the slot functions in the issue422 branch. If someone has the opportunity to test that, I'd appreciate it!

I can confirm that the changes from the branch issue422 fix the testsuite for me on 32-bit PowerPC.

There is just one unrelated failure left which is probably a result of how I ran the testsuite.

test_version (tests.test_version.VersionTests.test_version) ... Traceback (most recent call last):
  File "/home/glaubitz/greenlet/setup.py", line 212, in <module>
    version=get_greenlet_version(),
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/glaubitz/greenlet/setup.py", line 201, in get_greenlet_version
    with open('src/greenlet/__init__.py') as f: # pylint:disable=unspecified-encoding
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'src/greenlet/__init__.py'
FAIL

Thanks a lot for fixing this!

[jamadden]

That's great, thanks! And yes, the greenlet test suite is sensitive to being run from the directory it is checked out in because of that specific test. But its failure can be ignored.