dolphin-emu build seems to output a bad binary when using mold 1.1

[kzdixon]

As stated in the title, building dolphin-emu appears to spit out a corrupted binary when using mold.

$ mold -v
mold 1.1 (89612b709638b90c8a044e2f96f47d28054ba789; compatible with GNU ld)

$ gcc --version
gcc (GCC) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 7 5800X 8-Core Processor
    CPU family:          25
    Model:               33
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            0
    Frequency boost:     enabled
    CPU max MHz:         4887.3042
    CPU min MHz:         2200.0000
    BogoMIPS:            7600.20
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht
                         syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid
                          aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c r
                         drand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit w
                         dt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba
                         ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushop
                         t clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero
                          irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid d
                         ecodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmu
                         lqdq rdpid overflow_recov succor smca fsrm
Virtualization features:
  Virtualization:        AMD-V
Caches (sum of all):
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    4 MiB (8 instances)
  L3:                    32 MiB (1 instance)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-15
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Full AMD retpoline, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling
  Srbds:                 Not affected
  Tsx async abort:       Not affected

---

#########################################################################
# ARCHITECTURE, COMPILE FLAGS
#########################################################################
#
CARCH="x86_64"
CHOST="x86_64-pc-linux-gnu"

#-- Compiler and Linker Flags
CPPFLAGS="$CFLAGS"
CFLAGS="-march=native -B/usr/lib/mold -Ofast -pipe -fno-plt \
	-fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -pthread -Wno-error -w"
CXXFLAGS="$CFLAGS -Wp,-D_GLIBCXX_ASSERTIONS"
LDFLAGS="-Wl,-O4,--sort-common,--as-needed,-z,relro,-z,now,-lpthread"
RUSTFLAGS="-C opt-level=2 -C target-cpu=native -C link-arg=-fuse-ld=/usr/lib/mold"
#-- Make Flags: change this for DistCC/SMP systems
MAKEFLAGS="-j$(nproc) --quiet"
#-- Debugging flags
DEBUG_CFLAGS="-g -fvar-tracking-assignments"
DEBUG_CXXFLAGS="-g -fvar-tracking-assignments"
#DEBUG_RUSTFLAGS="-C debuginfo=2"
DEBUG_MAKEFLAGS="-j1 --debug=bvij"
CC_LD=mold
CXX_LD=mold
CMAKE_C_COMPILER="command gcc $CFLAGS"
CMAKE_CXX_COMPILER="command g++ $CXXFLAGS"
CMAKE_LINKER="command ld $LDFLAGS"

(Clean/purged build from dolphin-emu-git AUR package.)

$ readelf -p .comment /usr/bin/dolphin-emu
String dump of section '.comment':
  [     0]  GCC: (GNU) 11.2.0
  [    12]  mold 1.1 (89612b709638b90c8a044e2f96f47d28054ba789; compatible with GNU ld)

$ paru -Ss dolphin-emu-git
aur/dolphin-emu-git 5.0.r15933.g900a0b0eee-1 [+111 ~2.53] [Installed: 5.0.r16081.g5b3b6a7f1d-1]
    A Gamecube / Wii emulator - git version

$ dolphin-emu --version
Illegal instruction (core dumped)

$ dolphin-emu
Illegal instruction (core dumped)

Does not work. Instant core dump.

---

Removed -B/usr/lib/mold from CFLAGS

(Clean/purged build from dolphin-emu-git AUR package.)

$ readelf -p .comment /usr/bin/dolphin-emu
String dump of section '.comment':
  [     0]  GCC: (GNU) 11.2.0

$ paru -Ss dolphin-emu-git
aur/dolphin-emu-git 5.0.r15933.g900a0b0eee-1 [+111 ~2.53] [Installed: 5.0.r16081.g5b3b6a7f1d-1]
    A Gamecube / Wii emulator - git version

$ dolphin-emu --version
Dolphin 5.0-16081

$ dolphin-emu

Works and runs successfully.

---

Disabling LTO via -fno-lto doesn't seem to help either as the result is the same.

[rui314]

Thank you for your report. If it fails with an illegal instruction error, it is very likely that mold corrupted the binary. However, unfortunately, I couldn't reproduce the issue by checking out dolphin-emu from https://github.com/dolphin-emu/dolphin and built on my machine. If you check it out from the git repository and build it yourself, does it still fail?

Here is what I did:

git clone git@github.com:dolphin-emu/dolphin.git
cd dolphin
git submodule update --init
mkdir build
cd build
cmake -GNinja ..
mold -run ninja
./Binaries/dolphin-emu --version

[kzdixon]

Building with your instructions seems to work fine.
Adding in all of my flags as env vars for another build on the repo seems to give me the Illegal instruction (core dumped) so one of those isn't playing nicely when mold is in use. Will try to narrow it down.

[rui314]

I exported all your variables as below, but it still succeeded.

export CARCH="x86_64"
export CHOST="x86_64-pc-linux-gnu"

#-- Compiler and Linker Flags
export CPPFLAGS="$CFLAGS"
export CFLAGS="-march=native -B/usr/lib/mold -Ofast -pipe -fno-plt \
        -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -pthread -Wno-error -w"
export CXXFLAGS="$CFLAGS -Wp,-D_GLIBCXX_ASSERTIONS"
export LDFLAGS="-Wl,-O4,--sort-common,--as-needed,-z,relro,-z,now,-lpthread"
export RUSTFLAGS="-C opt-level=2 -C target-cpu=native -C link-arg=-fuse-ld=/usr/lib/mold"
#-- Make Flags: change this for DistCC/SMP systems
export MAKEFLAGS="-j$(nproc) --quiet"
#-- Debugging flags
export DEBUG_CFLAGS="-g -fvar-tracking-assignments"
export DEBUG_CXXFLAGS="-g -fvar-tracking-assignments"
#DEBUG_RUSTFLAGS="-C debuginfo=2"
export DEBUG_MAKEFLAGS="-j1 --debug=bvij"
export CC_LD=mold
export CXX_LD=mold
export CMAKE_C_COMPILER="command gcc $CFLAGS"
export CMAKE_CXX_COMPILER="command g++ $CXXFLAGS"
export CMAKE_LINKER="command ld $LDFLAGS"

cmake -GNinja ..
ninja
./Binaries/dolphin-emu --version

What am I missing?

[kzdixon]

Honestly not sure... Using the default arch makepkg flags also results in the same issue when mold is used to wrap around ninja apparently.

For Reference these are the defaults for makepkg.conf:

#########################################################################
# ARCHITECTURE, COMPILE FLAGS
#########################################################################
#
CARCH="x86_64"
CHOST="x86_64-pc-linux-gnu"

#-- Compiler and Linker Flags
#CPPFLAGS=""
CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions \
        -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security \
        -fstack-clash-protection -fcf-protection"
CXXFLAGS="$CFLAGS -Wp,-D_GLIBCXX_ASSERTIONS"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now"
#RUSTFLAGS="-C opt-level=2"
#-- Make Flags: change this for DistCC/SMP systems
MAKEFLAGS="-j8"
#-- Debugging flags
DEBUG_CFLAGS="-g -fvar-tracking-assignments"
DEBUG_CXXFLAGS="-g -fvar-tracking-assignments"
#DEBUG_RUSTFLAGS="-C debuginfo=2"

Could be CPU-family specific unless you're also on Ryzen.

Appears to be isolated to CFLAGS/CXXFLAGS of course. The below flags are what I trimmed down to from the default arch CFLAGS/CXXFLAGS to get it to successfully build with mold.

$ echo $CFLAGS
-fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection

$ echo $CXXFLAGS
-fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS

Which means that getting rid of -fno-plt was the culprit for my system apparently. Rerunning with all the arch makepkg.conf defaults sans -fno-plt also works just fine.

$ gcc --help=optimizers | grep fno-plt
  -fplt                       Use PLT for PIC calls (-fno-plt: load the address from GOT at call site).

Rerunning the AUR installation of dolphin-emu-git with -fno-plt removed from my actual makepkg.conf also seems to work just fine; Thus it appears there is a weird interaction on Ryzen (assuming our CPUs are not the same) when mold is used alongside -fno-plt for dolphin-emu!

Haven't seen other builds with similar behavior quite yet but I only just recently swapped over to mold and these FLAGS.

And just to sanity check, here's the readelf output:

$ readelf -p .comment /usr/bin/dolphin-emu

String dump of section '.comment':
  [     0]  GCC: (GNU) 11.2.0
  [    12]  mold 1.1 (89612b709638b90c8a044e2f96f47d28054ba789; compatible with GNU ld)

[rui314]

Thanks! I added -no-plt and indeed it causes a segmentation fault. It looks like we are handling R_X86_64_GOTPCRELX in an incorrect way. Let me prepare a fix.

[rui314]

To verify the fix, please rebuild mold after git pull and try to build dolphin-emu again.

[kzdixon]

$ paru -Ss mold-git
aur/mold-git v1.0.1_11_g4ccbd24c-1 [+4 ~0.37] [Installed: v1.1_14_g9f348308-1]
    A Modern Linker

Clean install of dolphin-emu-git:

$ dolphin-emu --version
Dolphin 5.0-16081

$ readelf -p .comment /usr/bin/dolphin-emu

String dump of section '.comment':
  [     0]  GCC: (GNU) 11.2.0
  [    12]  mold 1.1 (9f3483088206dd6f939d7b2c090570cf1f83d728; compatible with GNU ld)

All is well!