The overwritten tail from the dbg logs is c0c0c0c0c0c0c0c0 != c0c0c0c0c0c0c00a... which seems to indicate that the compacted message buffer from mc_compact_cb is causing the LF to overflow.

From frame 2 of the backtrace we have the fragment of issue
#2 0x0000563098f42bc2 in qm_debug_frag (f=f@entry=0x7f75c2e64068, qm=<optimized out>) at mem/q_malloc.c:122

Is possible to print the contents of this fragment? I was unsuccessful, but maybe this will shed some light into the unexpected buffer.

Any other ideas for debugging the issue?

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

Hey team! This one was tricky to identify, but is related to miscalculated buffer length due to header sanitization. If you apply the first commit from the PR that I just submitted (a5b6d2e) and send a request with some non-compacted header that will be sanitized when parsed then you will see the new LM_BUG report an overflow.

Example:

# set header with extra leading whitespace
Min-SE:  3600

Sep 22 00:40:21 [118] CRITICAL:compression:mc_compact_cb: 
>>> buffer overflow: calculated=1357, actual=1358

It seems you have hit a programming bug.
Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues

In the PR that I submitted, I removed the shortcutting on header assembly. Let me know if you have a better solution!

Hi @john08burke , thanks for your work here !! Are you talking about #2632 , right ?

Hey @bogdan-iancu, yes that's the one! Let me know if you need more info to reproduce. You should be able to trigger the overflow log if you cherry-pick a5b6d2e. Ideally Q_MALLOC+DBG should catch it too, but in my test env I could never get it to dump!

I also realized that I detailed the bug in my commit (ba89136), but didn't really put the info here! The cause of the overflow seems to be with the shortcutting of non-compact SIP headers when re-assembling after compaction, especially in case when "sanitization" was performed. The calculation for memory allocation is always done with the hdr_field.body field which removes leading white spaces (and probably more cleanups, but I didn't check). However, when re-assembling the compact headers the module uses the hdr_field.name field which is the raw header. Normally this is OK, but in case of sanitization this leads to buffer overflow.

For example, something like:

Min-SE:  3600

where there is an extra space will lead to allocation len being 1 byte less than what will be written to buffer!

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

keepalive, I can provide an env to replicate this if that would help!

Hey @bogdan-iancu! Any chance we can try to get this one reviewed and closed out? I've been running with the submitted patch for ~2months and haven't seen the crash since.

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

Marking as closed due to lack of progress for more than 30 days. If this issue is still relevant, please re-open it with additional details.