Bug | Segmentation Fault

[omerts]

Describe the issue:

When running tests, or trying to run on a camera, I am getting a segmentation fault:

Running suite(s): states_good
0%: Checks: 3, Failures: 0, Errors: 3
../tests/check/generic/states.c:114:E:general:test_state_changes_up_and_down_seq:0: (after this point) Received signal 11 (Segmentation fault)
../tests/check/generic/states.c:146:E:general:test_state_changes_up_seq:0: (after this point) Received signal 11 (Segmentation fault)
../tests/check/generic/states.c:178:E:general:test_state_changes_down_seq:0: (after this point) Received signal 11 (Segmentation fault)
Check suite states ran in 16.217s (tests failed: 3)

Reproduce the code example:

Just run: `ninja -C builddir test`

Error message:

ninja: Entering directory `builddir'
[1/2] Running all tests.
1/1 generic_states        FAIL            16.23s   exit status 3
>>> UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MALLOC_PERTURB_=156 CK_DEFAULT_TIMEOUT=20 GST_PLUGIN_SYSTEM_PATH_1_0='' GST_PLUGIN_LOADING_WHITELIST=gstreamer:gst-plugins-base:gst-plugin-pylon@/home/amram/development/gst-plugin-pylon/builddir GSETTINGS_BACKEND=memory GST_STATE_IGNORE_ELEMENTS='
' GST_PLUGIN_PATH_1_0=/home/amram/development/gst-plugin-pylon/builddir:/usr/lib/x86_64-linux-gnu/gstreamer-1.0:/usr/lib/x86_64-linux-gnu/gstreamer-1.0 GST_REGISTRY=/home/amram/development/gst-plugin-pylon/builddir/tests/check/generic_states.registry GST_PLUGIN_SCANNER_1_0=/usr/lib/x86_64-linux-gnu/gstreamer1.0/gstreamer-1.0/gst-plugin-scanner ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 /home/amram/development/gst-plugin-pylon/builddir/tests/check/generic_states
――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
Running suite(s): states_good
0%: Checks: 3, Failures: 0, Errors: 3
../tests/check/generic/states.c:114:E:general:test_state_changes_up_and_down_seq:0: (after this point) Received signal 11 (Segmentation fault)
../tests/check/generic/states.c:146:E:general:test_state_changes_up_seq:0: (after this point) Received signal 11 (Segmentation fault)
../tests/check/generic/states.c:178:E:general:test_state_changes_down_seq:0: (after this point) Received signal 11 (Segmentation fault)
Check suite states ran in 16.217s (tests failed: 3)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――


Summary of Failures:

1/1 generic_states FAIL            16.23s   exit status 3

Ok:                 0   
Expected Fail:      0   
Fail:               1   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   

Full log written to /home/amram/development/gst-plugin-pylon/builddir/meson-logs/testlog.txt
FAILED: meson-internal__test 
/usr/local/bin/meson test --no-rebuild --print-errorlogs
ninja: build stopped: subcommand failed.

Is your camera operational in Basler pylon viewer on your platform

Yes

Hardware setup & camera model(s) used

This is happening both on my personal Ubuntu 22.04 computer, and on our Jetson Xavier NX Jetpack 5.1.2.

CPU: X86_64 & ARM64
Operating system: Ubuntu 20.04 & 22.04
RAM: 32Gb & 8Gb
Cable length: 25cm
Deepstream: 6.4
Cuda: 12.3 & 11.4

Runtime information:

python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
platform: linux/x86_64/5.17.15-051715-generic
pypylon: 3.0.1 / 7.4.0.38864

[omerts]

Fixed it by using sudo while installing and running, not sure why it is needed, but it fixes it

[thiesmoeller]

Can you try to recreate the segfault while running in gdb?

Even if it is fixed for you now, it is always good to learn about possible failure paths.

[omerts]

@thiesmoeller Of course:

(gdb) set args pylonsrc ! "video/x-raw(memory:NVMM), width=1920, height=1080"  !  nvvidconv  ! "video/x-raw(memory:NVMM), width=1280, height=720" !  fakesink
(gdb) run
Starting program: /usr/bin/gst-launch-1.0 pylonsrc ! "video/x-raw(memory:NVMM), width=1920, height=1080"  !  nvvidconv  ! "video/x-raw(memory:NVMM), width=1280, height=720" !  fakesink
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0xfffff3f5f1e0 (LWP 9599)]
[New Thread 0xfffff375e1e0 (LWP 9600)]
[Thread 0xfffff375e1e0 (LWP 9600) exited]
[Thread 0xfffff3f5f1e0 (LWP 9599) exited]
[New Thread 0xfffff375e1e0 (LWP 9601)]
[New Thread 0xfffff3f5f1e0 (LWP 9602)]
[New Thread 0xfffff25f91e0 (LWP 9603)]
[New Thread 0xfffff1df81e0 (LWP 9604)]
[Thread 0xfffff1df81e0 (LWP 9604) exited]
[Thread 0xfffff25f91e0 (LWP 9603) exited]
[New Thread 0xfffff1df81e0 (LWP 9605)]
[Thread 0xfffff1df81e0 (LWP 9605) exited]
[New Thread 0xfffff1df81e0 (LWP 9606)]
[New Thread 0xfffff25f91e0 (LWP 9607)]
[New Thread 0xfffff15f71e0 (LWP 9608)]
[New Thread 0xfffff0df61e0 (LWP 9609)]
[Thread 0xfffff15f71e0 (LWP 9608) exited]
[Thread 0xfffff25f91e0 (LWP 9607) exited]
[Thread 0xfffff1df81e0 (LWP 9606) exited]
[Thread 0xfffff0df61e0 (LWP 9609) exited]
[New Thread 0xfffff0df61e0 (LWP 9610)]
[New Thread 0xfffff15f71e0 (LWP 9611)]
[New Thread 0xfffff25f91e0 (LWP 9612)]
[New Thread 0xfffff1df81e0 (LWP 9613)]
[Thread 0xfffff25f91e0 (LWP 9612) exited]
[Thread 0xfffff15f71e0 (LWP 9611) exited]
[Thread 0xfffff0df61e0 (LWP 9610) exited]
[Thread 0xfffff1df81e0 (LWP 9613) exited]
[New Thread 0xfffff1df81e0 (LWP 9614)]
[Thread 0xfffff1df81e0 (LWP 9614) exited]
[New Thread 0xfffff1df81e0 (LWP 9615)]
[New Thread 0xfffff25f91e0 (LWP 9616)]
[New Thread 0xfffff15f71e0 (LWP 9617)]
[New Thread 0xfffff0df61e0 (LWP 9618)]
[Thread 0xfffff25f91e0 (LWP 9616) exited]
[Thread 0xfffff1df81e0 (LWP 9615) exited]
[Thread 0xfffff15f71e0 (LWP 9617) exited]
[Thread 0xfffff0df61e0 (LWP 9618) exited]
[New Thread 0xfffff0df61e0 (LWP 9619)]
[New Thread 0xfffff15f71e0 (LWP 9620)]
[New Thread 0xfffff25f91e0 (LWP 9621)]
[New Thread 0xfffff1df81e0 (LWP 9622)]
[Thread 0xfffff25f91e0 (LWP 9621) exited]
[Thread 0xfffff15f71e0 (LWP 9620) exited]
[Thread 0xfffff0df61e0 (LWP 9619) exited]
[Thread 0xfffff1df81e0 (LWP 9622) exited]
[New Thread 0xfffff1df81e0 (LWP 9623)]
Setting pipeline to PAUSED ...
--Type <RET> for more, q to quit, c to continue without paging--

Thread 1 "gst-launch-1.0" received signal SIGSEGV, Segmentation fault.
0x0000fffff7366734 in gst_pylon_new (
    gstpylonsrc=gstpylonsrc@entry=0xaaaaaad21050 [GstElement|pylonsrc0], 
    device_user_name=<optimized out>, device_serial_number=<optimized out>, 
    device_index=<optimized out>, enable_correction=<optimized out>, 
    err=err@entry=0xffffffffe2b0) at ../ext/pylon/gstpylon.cpp:340
340	  self->nvsurface_layout = PROP_NVSURFACE_LAYOUT_DEFAULT;
(gdb) bt
#0  0x0000fffff7366734 in gst_pylon_new(_GstElement*, char const*, char const*, int, int, _GError**)
    (gstpylonsrc=gstpylonsrc@entry=0xaaaaaad21050 [GstElement|pylonsrc0], device_user_name=<optimized out>, device_serial_number=<optimized out>, device_index=<optimized out>, enable_correction=<optimized out>, err=err@entry=0xffffffffe2b0) at ../ext/pylon/gstpylon.cpp:340
#1  0x0000fffff736ee44 in gst_pylon_src_start(GstBaseSrc*)
    (src=<optimized out>) at ../ext/pylon/gstpylonsrc.cpp:780
#2  0x0000fffff72c9330 in gst_base_src_start
    (basesrc=basesrc@entry=0xaaaaaad21050 [GstBaseSrc|pylonsrc0])
    at gstbasesrc.c:3432
#3  0x0000fffff72c9878 in gst_base_src_activate_push
    (pad=0xaaaaab2001e0 [GstPad|src], active=<optimized out>, parent=0xaaaaaad21050 [GstObject|pylonsrc0]) at gstbasesrc.c:3806
#4  gst_base_src_activate_mode
    (pad=0xaaaaab2001e0 [GstPad|src], parent=0xaaaaaad21050 [GstObject|pylonsrc0], mode=GST_PAD_MODE_PUSH, active=<optimized out>) at gstbasesrc.c:3883
#5  0x0000fffff7ecfd54 in activate_mode_internal
    (pad=0xaaaaab2001e0 [GstPad|src], parent=0xaaaaaad21050 [GstObject|pylonsrc0], mode=GST_PAD_MODE_PUSH, active=1) at gstpad.c:1217
#6  0x0000fffff7ed042c in gst_pad_set_active
    (pad=pad@entry=0xaaaaab2001e0 [GstPad|src], active=1) at gstpad.c:1100
#7  0x0000fffff7ea9490 in activate_pads
--Type <RET> for more, q to quit, c to continue without paging--
    (vpad=<optimized out>, ret=0xffffffffe500, active=0xffffffffe56c)
    at gstelement.c:3053
#8  0x0000fffff7ebdb24 in gst_iterator_fold
    (it=it@entry=0xaaaaaac688a0, func=func@entry=0xfffff7ea9468 <activate_pads>, ret=ret@entry=0xffffffffe500, user_data=user_data@entry=0xffffffffe56c)
    at gstiterator.c:617
#9  0x0000fffff7ea9d10 in iterator_activate_fold_with_resync
    (iter=iter@entry=0xaaaaaac688a0, user_data=user_data@entry=0xffffffffe56c, func=<optimized out>) at gstelement.c:3077
#10 0x0000fffff7eabf00 in gst_element_pads_activate
    (element=element@entry=0xaaaaaad21050 [GstElement|pylonsrc0], active=<optimized out>) at gstelement.c:3113
#11 0x0000fffff7eac36c in gst_element_change_state_func
    (element=0xaaaaaad21050 [GstElement|pylonsrc0], transition=GST_STATE_CHANGE_READY_TO_PAUSED) at gstelement.c:3173
#12 0x0000fffff72c75f4 in gst_base_src_change_state
    (element=0xaaaaaad21050 [GstElement|pylonsrc0], transition=GST_STATE_CHANGE_READY_TO_PAUSED) at gstbasesrc.c:3920
#13 0x0000fffff7eae9bc in gst_element_change_state
    (element=element@entry=0xaaaaaad21050 [GstElement|pylonsrc0], transition=transition@entry=GST_STATE_CHANGE_READY_TO_PAUSED) at gstelement.c:2965
#14 0x0000fffff7eaf110 in gst_element_set_state_func
    (element=0xaaaaaad21050 [GstElement|pylonsrc0], state=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--
    at gstelement.c:2919
#15 0x0000fffff7e88154 in gst_bin_element_set_state
    (next=GST_STATE_PAUSED, current=GST_STATE_READY, start_time=0 [0:00:00.000000000], base_time=0 [0:00:00.000000000], element=0xaaaaaad21050 [GstElement|pylonsrc0], bin=0xaaaaab1120d0 [GstBin|pipeline0]) at gstbin.c:2605
#16 gst_bin_change_state_func
    (element=0xaaaaab1120d0 [GstElement|pipeline0], transition=GST_STATE_CHANGE_READY_TO_PAUSED) at gstbin.c:2947
#17 0x0000fffff7eae9bc in gst_element_change_state
    (element=element@entry=0xaaaaab1120d0 [GstElement|pipeline0], transition=GST_STATE_CHANGE_READY_TO_PAUSED) at gstelement.c:2965
#18 0x0000fffff7eaf484 in gst_element_continue_state
    (element=element@entry=0xaaaaab1120d0 [GstElement|pipeline0], ret=ret@entry=GST_STATE_CHANGE_SUCCESS) at gstelement.c:2673
#19 0x0000fffff7eae9fc in gst_element_change_state
    (element=element@entry=0xaaaaab1120d0 [GstElement|pipeline0], transition=transition@entry=GST_STATE_CHANGE_NULL_TO_READY) at gstelement.c:3004
#20 0x0000fffff7eaf110 in gst_element_set_state_func
    (element=0xaaaaab1120d0 [GstElement|pipeline0], state=<optimized out>)
    at gstelement.c:2919
#21 0x0000aaaaaaaa37f4 in main (argc=<optimized out>, argv=<optimized out>)
    at gst-launch.c:1132

[Gabben00]

Hi! I had this problem aswell and it wasnt solved with the sudo command. However i had a friend help me diagnose it. So the problem lies in gstpylon.cpp:340 as we can see from above. The problem is that when you have nvidia enabled you try to assign stuff to self but if there is an error in the giant catch block before the plugin deletes self and therefore self is null when you try to assign. So my solution was to check the errors above and see if i violated any of them. It turns out since my camera wasnt connected it threw a fault but then the segfault happens before it can report that fault.

[thiesmoeller]

great catch!

can you provide a pull request?

[Gabben00]

Sadly i have neither the time nor the knowledge of how to do this. I'm but a humble novice conveying what little information i have.