Offline messages are not delivered to client when SASL2 is used

[dedmurash]

Lost messages
Updated the server to version 24.07. Messages sent to users who are not online started to get lost. The server connects to the base on Maria. The web interface shows a message when the user appears online, it goes away, but the client does not display it. I used both mobile and computer clients, it is not their fault

[licaon-kter]

Client have activated MAM per account too?

[dedmurash]

Client have activated MAM per account too?

The server is configured so that archiving does not work for anyone for security reasons and for this it is worth: default: never

[dedmurash]

I understand that if messages are saved in the database, the client will download them from there. But the conditions are such that there will be no archiving. The message is on the server and waits for the subscriber to appear, when the client appears on the network, the message disappears from the database but does not reach the client. Somewhere the error may be related to the new type of authentication. On the stable version 23.10 everything works and there are no problems

[licaon-kter]

So close the issue, as messages are not lost but disappear as configured?

Or maybe explain differently?!

[dedmurash]

So close the issue, as messages are not lost but disappear as configured?

Or maybe explain differently?!

As a result, the interlocutor does not receive anything if he was not online at that moment. and after the connection is established, he also does not receive anything. But the server probably thinks that everything is in order and deletes messages from the database

[dedmurash]

I have tested it myself, two accounts cannot send messages to each other when they are temporarily offline. In different modes and with and without encryption. The message is in the spool table and waits for the receiver to appear online. After it appears, the message disappears, and the receiver does not receive anything. I remind you that there is no such thing in server version 23.10

[licaon-kter]

Which clients?

Edit your config .yml so it has no sensitive data and put it on https://gist.github.com

[dedmurash]

Which clients?

Convarsation and Gajim checked on both the behavior is the same

[dedmurash]

I did some research and realized that the problem comes from Conversation. It uses the new sasl protocol scram-sha1-plus. Gajim and aTalk don't have such problems.

[licaon-kter]

Explain how SASL impacts message retrieval.

[dedmurash]

I can only suggest these are new connection options or it is bind2 that is to blame. Since later versions of the server do not support them yet. At some point the transaction does not reach. I do not even know how to solve my problem. I even installed aTalk and it received a message. So that you understand what the conditions were. Subscriber A is online, Subscriber B is offline. The server archiving by default is never. Subscriber A sends a message to Subscriber B. Subscriber B connects to the server after a while. If the server has archiving - always, then he receives the message, if not, then Subscriber B will not have a message. Thank you very much for trying to understand the problem and I know that you are also a member of conversation support. Maybe it would be better to open this issue on the codeberg website and Daniel Gultsch will be able to find the problem

[licaon-kter]

When you disable MAM, messages get stored by https://docs.ejabberd.im/admin/configuration/modules/#mod_offline instead

Offline messages get delivered to the first client that comes online, afaik

[dedmurash]

yes, everything is correct and I am also trying to make it work on version 24.07. and it worked on version 23.10. but as soon as I updated, my subscribers started complaining that messages were not reaching them. I started checking on my own and was really convinced that there was a problem. I work through the MySQL database and monitored the message flow. The message is in the spool table in the database, and when a client appears, it disappears into thin air. and it is not in the database or on the client. but with the gajim and aTalk clients there are no such problems, all messages arrived clearly and there are no losses

[licaon-kter]

What's the point of disabling MAM if you want the messages to be stored anyway?

from mod_offline link: "A user is considered offline if no session presence priority > 0 are currently open."

My guess is, the client binds a session and when it comes back it resumes it, so the user was never offline 🤷 hence messages get dropped?

see https://xmpp.org/extensions/xep-0386.html and https://xmpp.org/extensions/xep-0388.html and etc

[dedmurash]

I want to achieve maximum security with this setting. and let the messages be stored only on the client and it will be on their conscience. if the server is captured, there will be nothing to take from it. and secondly, I don't have to periodically get rid of memory overflow.
It turns out that with the advent of sasl2, a problem arises with the ban on storing the archive on the server. it looks like there is work for Daniel Gultsch.

[licaon-kter]

I want to achieve maximum security with this setting

enable OMEMO, then you don't care

let the messages be stored only on the client

but they are on the server anyway, if the server is captured the "enemy" will just keep it online and read them from the database

but if OMEMO is used, it's useless for the "enemy" as it's all encrypted

I don't have to periodically get rid of memory overflow.

setup cron to clean up MAM after some days

It turns out that with the advent of sasl2, a problem arises with the ban on storing the archive on the server.

I guess you could say this is a bug, eg. rename issue to "Offline messages are not delivered to client when SASL2 is used" or so.

it looks like there is work for Daniel Gultsch.

it looks like your clients are not up to date (you have Gajim 1.9.5 yes? atalk does not support sasl2?)

do read: https://notes.valdikss.org.ru/jabber.ru-mitm/ and https://monal-im.org/post/00004-sasl/ to understand why clients/servers pushed for SASL/Bind2

[dedmurash]

Thank you very much for the useful articles. Everything became clear to me.

enable OMEMO, then you don't care

yes of course omemo is good encryption I read a lot about it, but the last audit was in 2016, that's a long time ago. I really want to believe that it is still reliable

it looks like your clients are not up to date (you have Gajim 1.9.5 yes? atalk does not support sasl2?)

That's right. That's why I came to the conclusion that the problem is in the new protocols that both ejabberd and conversation have.

perhaps you are right, I renamed the problem on your recommendation

[licaon-kter]

yes of course omemo is good encryption I read a lot about it, but the last audit was in 2016, that's a long time ago. I really want to believe that it is still reliable

nothing changed, afaik

also it's implemented around in multiple apps and frameworks and languages

[iNPUTmice]

I believe that offline message queue deployments are valid deployments and I’m aware that the Bind 2 XEP says the server MUST clear offline messages. However those things are at odds with each other and I believe the XEP should probably lower that to a SHOULD and/or ejabberd should either introduce a config setting to ignore that task or make this depend on mod_mam being loaded or even better the user settings.

I’m also aware that @mwild1 is working on using SASL2 user agents to track which clients have received which messages with the goal to offer similar security guarantees for MAM as one would get from only using offline messages. However until this is working and ported to ejabberd we probably need a server side setting.

In the mean time @dedmurash can probably try to disable bind 2 on the server? Not sure if that’s possible.

[dedmurash]

Thank you all very much for the detailed explanation. I especially respect you @iNPUTmice for your work and the team that works with you. And I changed the settings of my server, now it will store messages and changed the privacy policy until better times.

[dedmurash]

Here's what I noticed. The server is in a hurry with the deletion actions, I think that a slight delay in sending a message after connecting to the user would solve the problem, even if it’s 1 second or 2, you need to check. And the welcome message for new accounts still doesn’t go through.

[Socalix]

Just in case my prior logs help, I went through the same pain and debugging and got to the same conclusion about BIND 2 spec deleting the offline messages prematurely. I've reported it on conversations with detailed logs several months ago -- https://codeberg.org/iNPUTmice/Conversations/issues/296

I ended up reverting my server back to v23.10 until hopefully this issue is resolved, which was a pain because all clients complained about lowering authentication level and required manual confirmation.

[dedmurash]

In version 24.10 the problem is not solved, I checked again and everything is still in the same place.

[dedmurash]

I apologize. I checked when the client had strict parameters for saving messages.

[licaon-kter]

Wasn't this fixed in Conversations 2.17+ ? Or is it about other clients too?

[dedmurash]

I think the Conversations client can't fix the server problem. I use this client and tested it on version 2.17.2. Unfortunately, messages keep disappearing, an example is that when configuring the greeting module during registration, it sends a message to a new user. But even this message doesn't get through. I don't know how else to explain it, if you are a server administrator, experiment with archiving and you will understand that the server deletes the message from the spool table ahead of time. And also if archiving is enabled, messages are loaded only from the archive table, and the spool no longer participates in the work.

[badlop]

The message is in the spool table in the database, and when a client appears, it disappears into thin air. and it is not in the database or on the client. but with the gajim and aTalk clients there are no such problems, all messages arrived clearly and there are no losses

the Bind 2 XEP says the server MUST clear offline messages.

got to the same conclusion about BIND 2 spec deleting the offline messages prematurely.

Right, the exact text in the XEP says:

Upon processing the bind request, the server MUST perform several operations, including:
- Clear the offline messages for this user, if any, without sending them (as they will be provided by MAM).

And that was directly added to mod_offline.erl in commit efffc31

c2s_handle_bind2_inline(#{jid := #jid{luser = LUser, lserver = LServer}} = State, _Els) ->
    delete_all_msgs(LUser, LServer),
    State.

I believe the XEP should probably lower that to a SHOULD and/or ejabberd should either introduce a config setting to ignore that task or make this depend on mod_mam being loaded or even better the user settings.

Seems reasonable: the XEP says to remove offline because that same content is assumed to be in MAM. If the later is not true (MAM is disabled), then that rule doesn't apply (let's keep offline so it can be sent)

I've implemented that, it would be great to get some feedback:

• preliminary patch: badlop@fc5010d
• installers: https://github.com/badlop/ejabberd/actions/runs/11794063466
• container: https://github.com/badlop/ejabberd/pkgs/container/ejabberd/304157374?tag=dependabot sha256:c9c53a6858e32bc6595db5e717c0ecfe9fe90e92e28d935c8794f98b419e34b1