Crash on ARM?

[Alkarex]

Hello,
Testing on Raspberry Pi 4, with Raspberry Pi OS, 32-bit (arm32v7) seems to show that simply having the auth_openidc module enabled prevents Apache from starting.

Setup

Starting from a fresh Raspberry Pi OS Lite based on Debian 11.

sudo apt install apache2 libapache2-mod-auth-openidc
sudo service apache2 restart

Result

Apache produces no logs whatsoever, but there is a bit in syslog:

$ sudo cat /var/log/syslog
Jun 13 16:33:37 raspberrypi systemd[1]: Starting The Apache HTTP Server...
Jun 13 16:33:37 raspberrypi apachectl[13267]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Jun 13 16:33:37 raspberrypi systemd[1]: Started The Apache HTTP Server.
Jun 13 16:33:37 raspberrypi systemd[1]: apache2.service: Main process exited, code=killed, status=7/BUS
Jun 13 16:33:37 raspberrypi systemd[1]: apache2.service: Failed with result 'signal'.
Jun 13 16:33:37 raspberrypi systemd[1]: apache2.service: Scheduled restart job, restart counter is at 5.
Jun 13 16:33:37 raspberrypi systemd[1]: Stopped The Apache HTTP Server.
Jun 13 16:33:37 raspberrypi systemd[1]: apache2.service: Start request repeated too quickly.
Jun 13 16:33:37 raspberrypi systemd[1]: apache2.service: Failed with result 'signal'.
Jun 13 16:33:37 raspberrypi systemd[1]: Failed to start The Apache HTTP Server.

Additional info

The same error happens natively or in Docker, and also with Debian 12.
The same test works fine on x86-64 platform.

Downstream related issues:

• Fix OpenID Connect crash on ARM FreshRSS/FreshRSS#5463
• [BUG] docker status Restarting (135) FreshRSS/FreshRSS#5460

[zandbelt]

thanks for reporting: unfortunately we don't maintain the packaging for Raspberry PI and we cannot reproduce locally so a server debug log file would help us to try and address it

[andrewmcguinness]

I have also found this on a Raspberry Pi Zero 2 W

Details below from my debian bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055453

Apache doesn't start up with the module enabled on a raspberry pi zero 2 W

If I disable the module it runs OK

Looks like a memory alignment issue

systemctl status reports:
BEGIN

x apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled)
Active: failed (Result: signal) since Mon 2023-11-06 15:55:19 GMT; 22s ago
Duration: 15ms
Docs: https://httpd.apache.org/docs/2.4/
Process: 2167 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
Main PID: 2171 (code=killed, signal=BUS)
CPU: 251ms

Nov 06 15:55:19 raspberrypi systemd[1]: apache2.service: Main process exited, code=killed, status=7/BUS
Nov 06 15:55:19 raspberrypi systemd[1]: apache2.service: Failed with result 'signal'.

END

dmesg has

[10932.990013] Alignment trap: apache2 (2111) PC=0x76bdc824 Instr=0xed837b00 Address=0x751be419 FSR 0x801
[10932.990044] Alignment trap: not handling instruction ed837b00 at [<76bdc824>]
[10932.990050] 8<--- cut here ---
[10932.990074] Unhandled fault: alignment exception (0x801) at 0x751be419
[10932.990091] [751be419] *pgd=04221835, *pte=1553a75f, *ppte=1553ac7f

The same error happens every time.

[zandbelt]

I suddenly remember something similar here OpenIDC/liboauth2#21 and I guess I should adopt that fix here as well; can you run with LogLevel auth_openidc:debug and see where about the crash is happing?

[andrewmcguinness]

I only get this:

[Mon Nov 06 17:28:39.092321 2023] [auth_openidc:info] [pid 3176:tid 1995275648] mod_auth_openidc-2.4.12.3 - init - cjose 0.6.2.1, OpenSSL 3.0.8 7 Feb 2023, EC=yes, GCM=yes, Memcache=yes, Redis=yes, JQ=no

[andrewmcguinness]

adding OIDCCacheType file did not make a difference

[zandbelt]

oww, then I still have no clue about how to fix this...

[Alkarex]

Would it help if I provide an SSH access to a blank Raspberry Pi 4 or 5?

[zandbelt]

unfortunately not

[hmoffatt]

Can you run httpd in gdb and see what crashes?

[andrewmcguinness]

OK, so I did manage to set up my chroot environment to build the module from source. However, the tests were failing

src/cache/common.c:105 [3] [0] oidc_cache_mutex_post_config: apr_global_mutex_create failed to create mutex (4) on file /tmp/mod_auth_openidc_shm_mutex.523743.3e828680: Permission denied (13)
cfg->cache->post_config failed!
FAIL test/test (exit status: 255)

I assume this is probably some weirdness relating to the chroot (/tmp is mounted into it from the x86_64 host, I don't know....)

So I hacked the Makefile.am to set -DOIDC_LOCK=APR_LOCK_DEFAULT

That worked.

Just to check, I tried that out on my actual Pi Zero 2.

That worked too.

Some confounding factors:

1. My build environment is Debian bookworm, not Raspbian bookworm
2. While trying to solve the mutex issue, I switched from the stable version 2.4.12.3 to the latest version 2.4.14.2
3. The OIDC_LOCK change

but in summary, it looks like this is more of a distro/packaging problem than a source problem.

I will give a compile of 2.4.12.3 a try too, but not immediately; now I have working apache I want to get back to my own project.

[zandbelt]

that explains the mutex behaviour at lock time but not the segmentation fault at startup

[sanzinger]

The crash is happening exactly here because of unaligned access:

#0  oidc_cache_shm_post_config (s=0x804028) at src/cache/shm.c:110
110				i++, OIDC_CACHE_SHM_ADD_OFFSET(t, cfg->cache_shm_entry_size_max)) {
(gdb) info locals
cfg = 0x83eb10
context = 0x83cc40
rv = 0
__FUNCTION__ = "oidc_cache_shm_post_config"
i = 1
t = 0x67478219

In order to resolve the issue on ARM, you can change the OIDC_DEFAULT_CACHE_SHM_ENTRY_SIZE_MAX property as follows:

diff --git a/src/config.c b/src/config.c
index f1f7a66..8cedfcd 100644
--- a/src/config.c
+++ b/src/config.c
@@ -116,7 +116,7 @@
 /* default max cache size for shm */
 #define OIDC_DEFAULT_CACHE_SHM_SIZE 10000
 /* default max cache entry size for shm: # value + # key + # overhead */
-#define OIDC_DEFAULT_CACHE_SHM_ENTRY_SIZE_MAX 16384 + 512 + 17
+#define OIDC_DEFAULT_CACHE_SHM_ENTRY_SIZE_MAX 16384 + 512 + 32
 /* for issued-at timestamp (iat) checking */
 #define OIDC_DEFAULT_IDTOKEN_IAT_SLACK 600
 /* for file-based caching: clean interval in seconds */

to have the offsets 8 byte aligned.

[zandbelt]

@sanzinger thanks for confirming that it is a memory alignment issue; can you verify c97f306 ?

[sanzinger]

Tested it and it does not work. Still same issue. Only setting OIDC_DEFAULT_CACHE_SHM_ENTRY_SIZE_MAZ to an aligned value works (as suggested above).

The compiler hint to the struct does not solve the problem. The point is that pointer is derived from oidc_cfg.cache_shm_entry_size_max in macro OIDC_CACHE_SHM_ADD_OFFSET which is unaligned (because of the "overhead" of 17), the compiler cannot do anything about it.

So the simplest way is to make sure that oidc_cfg.cache_shm_entry_size_max is always a multiple of 8.

[andrewmcguinness]

So when I changed the locking type to get around the (probably unrelated) build error in my chroot, I avoided using all this shm code and that's why it worked. Makes sense.

[zandbelt]

Tested it and it does not work. Still same issue. Only setting OIDC_DEFAULT_CACHE_SHM_ENTRY_SIZE_MAZ to an aligned value works (as suggested above).

The compiler hint to the struct does not solve the problem. The point is that pointer is derived from oidc_cfg.cache_shm_entry_size_max in macro OIDC_CACHE_SHM_ADD_OFFSET which is unaligned (because of the "overhead" of 17), the compiler cannot do anything about it.

So the simplest way is to make sure that oidc_cfg.cache_shm_entry_size_max is always a multiple of 8.

see 4a4e198

[sanzinger]

Thank you, this version works now. 👍

[zandbelt]

this is now in 2.4.15 https://github.com/OpenIDC/mod_auth_openidc/releases/tag/v2.4.15