Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mono-native 6.12.0.161 build error for Fedora 38 host #180

Closed
zboszor opened this issue May 5, 2023 · 9 comments · Fixed by #183
Closed

mono-native 6.12.0.161 build error for Fedora 38 host #180

zboszor opened this issue May 5, 2023 · 9 comments · Fixed by #183

Comments

@zboszor
Copy link
Collaborator

zboszor commented May 5, 2023

Not sure what causes this. Kernel 6.2.x? GLIBC? GCC 13?

... [C parts of Mono build up to this point]
CSC     [build-linux] gensources.exe
Makefile:20: warning: overriding recipe for target 'test-local'
../../build/library.make:208: warning: ignoring old recipe for target 'test-local'
Makefile:20: warning: overriding recipe for target 'test-local'
../../build/library.make:208: warning: ignoring old recipe for target 'test-local'

=================================================================
        Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
        Native stacktrace:
=================================================================
        0x4a3dd2 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x4a4179 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x455301 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x41ba1f - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x7fe2e73bcb70 - /lib64/libc.so.6 : 
        0x49fee8 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x45a67d - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x41d1a6 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x6143ef - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x616e8f - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x572d81 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x424c59 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x42af1c - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : mono_main
        0x41b1ff - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x7fe2e73a6b4a - /lib64/libc.so.6 : 
        0x7fe2e73a6c0b - /lib64/libc.so.6 : __libc_start_main
        0x41b7d5 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : _start

=================================================================
        Telemetry Dumper:
=================================================================
No threads attached to runtime.

=================================================================
        External Debugger Dump:
=================================================================
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

=================================================================
        Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x49fee8):0x49fed8  83 c2 01 48 83 fa 01 0f 87 a9 00 00 00 83 e8 05  ...H............
0x49fee8  c6 03 e8 89 43 01 4d 85 f6 74 4d 4c 89 6b 06 41  ....C.M..tML.k.A
0x49fef8  bc 0e 00 00 00 c6 43 05 08 83 fd 0e 75 4a 4d 85  ......C.....uJM.
0x49ff08  ff 74 03 41 89 2f 89 ee 48 89 df e8 58 c1 ff ff  .t.A./..H...X...

The same crash occurs a couple of times for different DLLs until the build finally stops.

FWIW, I looked at the Fedora SRPM / spec for possible fixes.

  • adding Fedora patches didn't work
  • using -O1 instead of -O2 didn't work

I am out of ideas at the moment.

@ajlennon Any idea?
@ajlennon
Copy link
Member

ajlennon commented May 5, 2023

How very odd. No I don't really know I am afraid. I only run Ubuntu myself.

Have you had a go at restricting parallel builds?

It's not related to memory usage is it?

@zboszor
Copy link
Collaborator Author

zboszor commented May 6, 2023

How very odd. No I don't really know I am afraid. I only run Ubuntu myself.

Have you had a go at restricting parallel builds?

No, because the mono recipe runs make directly, restricting parallel builds to -j1 out of the box.

The recipe should use oe_runmake to do parallel build. But using it doesn't influence anything regarding the observed crashes, only the speed to reproduce them.

It's not related to memory usage is it?

It doesn't seem so.

@ajlennon
Copy link
Member

ajlennon commented May 6, 2023

Most odd. Has anybody upstream had anything to say about this maybe?

@ajlennon
Copy link
Member

ajlennon commented May 6, 2023

This may not be a good idea but potentially we could try to replicate this by making a docker CI build container using Fedora rather than Ubuntu?

@zboszor
Copy link
Collaborator Author

zboszor commented May 8, 2023 •
edited
Loading

This is indeed strange.

The crash dump (i.e. the bytes as shown in the Memory around native instruction pointer message is identical across different DLL builds but it's sporadic in the sense that sometimes different DLLs show the issue.

And it WAS working previously. So it's an upgrade that causes this.

(after a lot of trial and error today)

TADA: 6.2.12 kernel allows building mono-native, 6.2.13 and 6.2.14 started crashing the Mono build.

What's left is to

  • test the build with kernel 6.3.1 and
  • test our Mono based stuff on an image with 6.2.14 and 6.3.1.

@zboszor
Copy link
Collaborator Author

zboszor commented May 8, 2023

Sadly, 6.3.1 also crashes the build.

@zboszor
Copy link
Collaborator Author

zboszor commented May 8, 2023

Reported the issue in the Fedora bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2196250

@ajlennon
Copy link
Member

ajlennon commented May 8, 2023

Very very odd indeed eh

@zboszor zboszor mentioned this issue May 9, 2023
Closed
@zboszor
Copy link
Collaborator Author

zboszor commented May 9, 2023

FWIW, the cross-compiled Mono seems to work with kernel 6.2.14.

  • mono cert-sync.exe works
  • our in-house C application with the embedded Mono runtime works

@zboszor zboszor mentioned this issue May 9, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix mmap segfault on recent kernels zboszor/meta-mono
2 participants
@ajlennon @zboszor