RabbitMQ terminates a connection due to an exception: {writer,message_not_understood}

[GitJadhav]

Describe the bug

The following rabbitmq error is causing the NodeJS to crash and has led to the crash of our platform, where rabbitmq is saying I am going to abort this channel and nodejs is not designed to handle this exception.

2023-10-25 05:59:13.610193-07:00 [error] <0.1010.0> Error on AMQP connection <0.1010.0> (10.135.45.39:50716 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn', state: running), channel 1:
2023-10-25 05:59:13.610193-07:00 [error] <0.1010.0>  {writer,message_not_understood,{#Ref<0.4068250614.1356857350.73626>,ok}}
2023-10-25 05:59:13.610367-07:00 [warning] <0.1010.0> Non-AMQP exit reason '{writer,message_not_understood,
2023-10-25 05:59:13.610367-07:00 [warning] <0.1010.0>                               {#Ref<0.4068250614.1356857350.73626>,ok}}'
2023-10-25 05:59:13.639616-07:00 [error] <0.1010.0> Error on AMQP connection <0.1010.0> (10.135.45.39:50716 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn', state: running), channel 1:error while terminating:
2023-10-25 05:59:13.639616-07:00 [error] <0.1010.0> shutdown

The reason to pull a plug as it is “writer,message_not_understood” is an internal message and there could be several reasons for this including frame is corrupted.
Technically - whenever rabbitmq aborts a channel (i.e calls shutdown on channel ) and that message is invoked by our node socket library, node will crash leading to our platform crash.

Reproduction steps

1.Create a 3 Node RabbitMQ Cluster ( 3.11.7 )
2.Running with RabbitMQ defaults ( memory 40% , buffering - 128MB )
3.Our Platform crashed due to the following RabbitMQ Error
...
2023-10-25 05:59:13.610193-07:00 [error] <0.1010.0> {writer,message_not_understood,{#Ref<0.4068250614.1356857350.73626>,ok}}
2023-10-25 05:59:13.610367-07:00 [warning] <0.1010.0> Non-AMQP exit reason '{writer,message_not_understood,
2023-10-25 05:59:13.610367-07:00 [warning] <0.1010.0> {#Ref<0.4068250614.1356857350.73626>,ok}}'

Expected behavior

RabbitMQ Error must clarify the root cause of the symptom

Additional context

This rabbitmq message "writer,message_not_understood" is invoked by our node socket library, node will crash leading to our product platform crash .
Trying to understand this symptom and get some feedback from the community - if somebody has witnessed this same error.

[lukebakken]

Hello, and thanks for using RabbitMQ.

As I'm sure you found via a search, this is not an error that has been reported, ever...

https://www.google.com/search?q=%22message_not_understood%22+rabbitmq

...so it must be unique to your environment.

If you can provide actual reproduction steps, that would be very helpful. The steps you provided are insufficient. It might be helpful to have your full log files instead of a few lines. If you can provide them, ATTACH them to this discussion in compressed form.

Your code should handle channel closures, obviously.

[michaelklishin]

@GitJadhav no, this error cannot be caused by a "corrupt frame" because the (Erlang) process that runs into it is the process that takes a frame data structure and serializes it, then writes it to the connection socket.

Somehow in your case it receives a different message which it does not know how to format because it is not a frame ({#Ref<0.4068250614.1356857350.73626>,ok}, which looks like an info message from the socket to me). I don't recall seeing something like this in years, so we'd need more details on how this can be reproduced.

In general, client connections can fail for all kinds of reasons, your applications must be ready to handle that. That's why connection recovery sequence is a feature some clients support and other client libraries usually document (some authors think it should not be a client library feature because it won't always work perfectly well for every app). This is particularly true for consumers.

You cannot claim that "due to this exception our system crashes", if your system cannot handle a connection or channel closure, it will eventually crash for a broad range of other reasons, be it a network connectivity loss on the host that hosts the app, or a genuine channel exception, it's only a matter of time.

[michaelklishin]

Alternatively, a frame writer could log a warning and keep going instead of immediately terminating like it does today. I'm sure some members of the core team would not agree with this solution.

That won't change the fact that your applications must handle connection failure.

[GitJadhav]

Please find rabbitMQ logs with the {writer, message_not_understand} exception
rabbit-writer-message_not_understood-log.txt

[lukebakken]

Your logs show many busy_dist_port messages, which suggests that you are using mirroring in your cluster with large message bodies. Is that true? If so, you should compress your messages. Please ensure that you are only mirroring to a quorum of nodes and not all of them.

Your logs also have many client unexpectedly closed TCP connection lines, which means your application is not closing the AMQP TCP connection gracefully. This indicates a bug in your application or in the client library you are using.

There is nothing in the logs that would help us reproduce the message_not_understood error, however.

[GitJadhav]

@lukebakken @michaelklishin I really appreciate your time in reviewing this situation and sharing some insights
With context to your suggestion
Alternatively, a frame writer could log a warning and keep going instead of immediately terminating like it does today. I'm sure some members of the core team would not agree with this solution.

We are deliberately shutting down on our side to as we see the 541 - INTERNAL ERROR as a red flag
Can you please recommend some logging mechanism to capture the packet data to determine this error signature on RabbitMQ Server
(message_not_understood {#Ref<0.4068250614.1356857350.73626>,ok}
Something like tcpdump - as mentioned in this link
https://www.rabbitmq.com/amqp-wireshark.html

[michaelklishin]

This does not provide any additional information. We still do not know what process has sent the ref(), ok that the socket writer does not know how to handle.

We can make it ignore such messages since they don't seem to be relevant to what the writer process does.

[lukebakken]

I don't think this case should be ignored.

@GitJadhav - has this issue happened more than once? If so, how many times, vs the uptime of your environment?

I'm assuming you still do not have reliable steps to reproduce it.

[michaelklishin]

@lukebakken by "ignored" I mean ignored by the writer process. It cannot do much about them so at least it should not fail.

Ideally it should log some context (maybe we can figure out what the sending process is) at debug level.

[lukebakken]

OK we can do that. There is definitely a bug to be fixed. I will open an issue.

[lukebakken]

#9991

[gomoripeti]

Just want to add that we've seen this strange {Ref, ok} stray message as well before, so the reporter is not alone. (Unfortunately it was reported to happen about once a month, so completely non-reproducible)

Only info that I have:

• RabbitMQ 3.11.10 Erlang 25.3 single node cluster
• Shovel in on-confirm mode (src: local exchange; dest: exchange on a remote cluster)
• The shovel's dest writer received this stray message, which stopped/crashed the shovel and restarted in the middle of shoveling a burst of a large amount of messages (~10K).

In RabbitMQ I only found one place in ra that sends a message in such format, but I have no idea how it can reach a writer process. Maybe a late response from OTP code-base.

[lukebakken]

If it's not too much trouble, could you follow up every time you see this issue? It would be great to get a feel for the affected RabbitMQ and Erlang versions, as well as any other relevant details (like you have provided).

In RabbitMQ I only found one place in ra that sends a message in such format

Could you please link to the line(s) of code? Thank you!

[gomoripeti]

• unfortunately the customer who saw this 3 months ago switched to a different configuration and hasn't reported the issue since. (they saw this happening 1 time per month over 3 months. About 6 months ago there was an upgrade from RabbitMQ 3.9.11/Erlang 24.1.7 to RabbitMQ 3.11.10/Erlang 25.3)
• this is the code in ra that sends a {Ref, ok} https://github.com/rabbitmq/ra/blob/main/src/ra_server_proc.erl#L349
• only classic queues (so no quorum or stream queues, no mqtt plugin)
• my gut feeling is that this stray message comes from Erlang/OTP, like the inet driver or similar and that it might be fixed in a newer version than 25.3. But I couldn't pin point any candidate in the Erlang codebase nor found any github issue that looked relevant in the erlang repo.
• When I mentioned shovels I did not mean to imply that the issue is related to the shovel plugin, quite the opposite that in our case it happened with the dst side of a shovel ie the writer of an amqp client while the reported saw it on the server side of an amqp connection. So this seems like a lower level issue than shovels. The traffic pattern of a burst of messages might be more relevant but I don't know how exactly.

[GitJadhav]

@lukebakken Sharing some insights on the failure episodes - We have seen this failure 25-30 times since 10/7 to 11/22
This is resulting in following error on the client side - we are restarting our platform as part of error exception handling .
Oct 13 14:39:53 ppollpghn00093 node[3377274]: 'Connection to RabbitMQ has been lost: Connection closed: 541 (INTERNAL-ERROR) with message "INTERNAL_ERROR"'

Our RabbitMQ Specs

• RabbitMQ 3.11.7 , Erlang 25.2.1
• 3 Node Cluster
• Shovel in on-confirm mode
  {rabbitmq_shovel,
  [{defaults,
  [{prefetch_count,1000},
  {ack_mode,on_confirm},
  {publish_fields,[]},
  {publish_properties,[]},
  {reconnect_delay,5}]}]},

PATTERN ANALYSIS
1.We have been processing similar payloads ( 2MB ) in earlier RabbitMQ version and never seen this RabbitMQ failure {writer, message_not_undertood}

• RabbitMQ 3.8.9 , Erlang 23.3.2
• 3 Node Cluster
• Shovel Feature - Disabled
  2.From RabbitMQ writer source code notes - Based on this reference
  Writers perform internal buffering. When the amount of data
  %% buffered exceeds a threshold, a socket flush is performed.
  %% See FLUSH_THRESHOLD for details.
  %% When a socket write fails, writer will exit.

As we were seeing busy_dist_port errors -- I have requested to enable tuning on the buffering side ( set MAX values ) to confirm if this will provide enough internal buffering resources to handle big payloads ( >2MB)
Path - /etc/rabbitmq/rabbitmq.conf
max_message_size = 536870911 ( default - 128MB , Max - 512MB )
Path - etc/rabbitmq/rabbitmq-env.conf
RABBITMQ_DISTRIBUTION_BUFFER_SIZE = 2097151 ( default - 192MB , Max - 2GB )

3.Trigger Source
We are seeing the RabbitMQ Server failing with this stray message {Ref,Ok} , with increased load volume of jobs
We have each job processing big payload ( 1.5M to 2MB data )

4. Is this a side-effect from Shovel plugin feature causing some contention and leading to this stray message {Ref,Ok} state ??Referring to @gomoripeti mention on he has come across this stray message {Ref,Ok}
   I noticed he has shovel plugin enabled
   The only difference that I see on our Environment is we recently enabled Shovel plugin feature.

5.We do see this erlang crash log captured in RabbitMQ log folder - Will that give us any leads ??
Is this even normal to see an erlang crash ??
I can share it in my next response
erl_crash.dump

[lukebakken]

Is this even normal to see an erlang crash ??

This is the kind of crash we would like to avoid by finding the cause of it. If you can compress and attach the erl_crash.dump file it might provide clues.

• Did you see these errors before enabling the Shovel plugin while using a 3.11.x version of RabbitMQ?
• Do the large messages cross the Shovel connection?
• Is there any chance you can compress the large messages to see if a smaller size eliminates the problem?
• Do you use quorum queues?
• Can you reproduce this in an environment using the latest version of RabbitMQ and Erlang? 3.11.7 is neither the latest 3.11.x version nor the latest version of RabbitMQ.

[lukebakken]

@gomoripeti @GitJadhav

I have opened the following issue:

#9991

Please read the questions I have asked above and supply additional information in issue #9991. Thank you!

[michaelklishin]

#9994

[GitJadhav]

@lukebakken
@michaelklishin
Follow Up
#9799 was my original issue ticket - It got converted to #9803 discussion ticket - FYI
I am seeing this ticket #9991 and now #9994
Will we have access and visibility to all this tickets ??
I filed this ticket (#9799) as an issue describing the rabbitMQ failure { Ref , Ok } message_not_understood error symptom and it was converted to an discussion and we have a lot of context captured in this ticket
Is there a way we can link this ticket to the new ticket that you opened or Can we make this current ticket active for reviewing at your end ??

Responding to your queries
1.We upgraded RabbitMQ to RabbitMQ 3.11.7 , Erlang 25.2.1 and enabled "shovel" feature on this version
2.Do the large messages cross the Shovel connection? I believe so from shovel functionality description below **
3.Is there any chance you can compress the large messages to see if a smaller size eliminates the problem? NO
Instead we are allocating the MAX buffer and testing to see if that mitigates this failure
4.Do you use quorum queues? We use classic queues
5.Can you reproduce this in an environment using the latest version of RabbitMQ and Erlang? TBD / WIP
We are in the process of replicating with current staged version - if we can replicate the issue and get an STR
We will upgrade the RabbitMQ stack and confirm the behavior
6.We have requested the customer to share the erlang_crash logs -- it is a big file - will have it compressed and attach in

QUESTION
**From google research
Shovel is a core RabbitMQ plugin that unidirectionally moves messages from a source to a destination. It will move messages from a source (queue) in one cluster to a destination (e.g. exchange, topic, etc) in another cluster. It is basically a transfer worker that runs as part of the RabbitMQ cluster.
1.Can you clarify the PROS and CONS of using "shovel" plugin ??
2.Is this a recommended feature to be enabled in PROD Env ??

[michaelklishin]

@GitJadhav #9991 already links to this discussion

[michaelklishin]

@GitJadhav I don't see how your questions about Shovels are relevant to this discussion.

Our team strongly prefers issues and discussions to be focussed. Perhaps open a new discussion with your questions about Shovels.

If you don't need to move messages from A to B then you don't have to use Shovel.

[GitJadhav]

@michaelklishin @lukebakken
Sharing the extended log trace -- Does this provide any hint / clue ??

RABBITMQ LOGS

2023-10-16 02:45:52.161854-07:00 [error] <0.31244.98> Error on AMQP connection <0.31244.98> (10.135.45.50:57870 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn', state: running), channel 1:
2023-10-16 02:45:52.161854-07:00 [error] <0.31244.98>  {writer,message_not_understood,{#Ref<0.3036637736.3039035399.137025>,ok}}
2023-10-16 02:45:52.163218-07:00 [warning] <0.31244.98> Non-AMQP exit reason '{writer,message_not_understood,
2023-10-16 02:45:52.163218-07:00 [warning] <0.31244.98>                               {#Ref<0.3036637736.3039035399.137025>,ok}}'
2023-10-16 02:45:52.196877-07:00 [error] <0.31244.98> Error on AMQP connection <0.31244.98> (10.135.45.50:57870 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn', state: running), channel 1:error while terminating:
2023-10-16 02:45:52.196877-07:00 [error] <0.31244.98> shutdown
2023-10-16 02:45:52.345402-07:00 [info] <0.31244.98> closing AMQP connection <0.31244.98> (10.135.45.50:57870 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn')
2023-10-16 02:45:55.377101-07:00 [warning] <0.4414.102> closing AMQP connection <0.4414.102> (10.135.45.50:57918 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn'):
2023-10-16 02:45:55.377101-07:00 [warning] <0.4414.102> client unexpectedly closed TCP connection
2023-10-16 02:45:55.377264-07:00 [warning] <0.5068.102> closing AMQP connection <0.5068.102> (10.135.45.50:57916 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn'):
2023-10-16 02:45:55.377264-07:00 [warning] <0.5068.102> client unexpectedly closed TCP connection
2023-10-16 02:45:55.386297-07:00 [warning] <0.2884.100> closing AMQP connection <0.2884.100> (10.135.45.50:57902 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn'):
2023-10-16 02:45:55.386297-07:00 [warning] <0.2884.100> client unexpectedly closed TCP connection
2023-10-16 02:45:55.386417-07:00 [warning] <0.15320.102> closing AMQP connection <0.15320.102> (10.135.45.50:57886 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn'):
2023-10-16 02:45:55.386417-07:00 [warning] <0.15320.102> client unexpectedly closed TCP connection
2023-10-16 02:45:55.393413-07:00 [warning] <0.979.101> closing AMQP connection <0.979.101> (10.135.45.50:53366 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn'):
2023-10-16 02:45:55.393413-07:00 [warning] <0.979.101> client unexpectedly closed TCP connection
2023-10-16 02:45:55.393552-07:00 [warning] <0.32526.100> closing AMQP connection <0.32526.100> (10.135.45.50:53354 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn'):
2023-10-16 02:45:55.393552-07:00 [warning] <0.32526.100> client unexpectedly closed TCP connection
2023-10-16 02:45:55.394565-07:00 [error] <0.16166.102>     supervisor: {<0.16166.102>,rabbit_channel_sup_sup}
2023-10-16 02:45:55.394565-07:00 [error] <0.16166.102>     errorContext: shutdown_error
2023-10-16 02:45:55.394565-07:00 [error] <0.16166.102>     reason: noproc
2023-10-16 02:45:55.394565-07:00 [error] <0.16166.102>     offender: [{nb_children,1},
2023-10-16 02:45:55.394565-07:00 [error] <0.16166.102>                {id,channel_sup},
2023-10-16 02:45:55.394565-07:00 [error] <0.16166.102>                {mfargs,{rabbit_channel_sup,start_link,[]}},
2023-10-16 02:45:55.394565-07:00 [error] <0.16166.102>                {restart_type,temporary},
2023-10-16 02:45:55.394565-07:00 [error] <0.16166.102>                {shutdown,infinity},
2023-10-16 02:45:55.394565-07:00 [error] <0.16166.102>                {child_type,supervisor}]
2023-10-16 02:45:55.403503-07:00 [warning] <0.16370.102> closing AMQP connection <0.16370.102> (10.135.45.50:53452 -> 10.61.253.65:5671, vhost: '/iap', user: 'pronghorn'):
2023-10-16 02:45:55.403503-07:00 [warning] <0.16370.102> client unexpectedly closed TCP connection

As we are able to replicate this issue only in PROD ENV , and we were still seeing errors in logs
I have advised the customer to set their buffering parameters to MAX and monitor their PROD ENV
Got the following Buffering parameters TUNING added into their PROD - RabbitMQ Configuration
Path : /etc/rabbitmq/rabbitmq.conf
max_message_size = 536870911 ( default - 128MB , MAX - 512MB )
Path : /etc/rabbitmq/rabbitmq-env.conf ( distribution busy buffer limit )
RABBITMQ_DISTRIBUTION_BUFFER_SIZE=2097151 ( viz 2GB MAX , default - 128MB )

RESULTS
After this tuning , we are not seeing <busy_dist_port> errors anymore-FYI
Monitoring for the stray message { Ref , OK} message_not_understood failure

[lukebakken]

Please see how I used <details>, <summary> and ``` to put those logs behind a dropdown so they don't take up screen space.

There is nothing materially different in those logs than what has already been provided.

[GitJadhav]

@michaelklishin @lukebakken

Referring to this Update
Closes #9991.

The goal at this point is to avoid connection termination. Those who would like to investigate these messages further now have a place to add extra logging or tracing.

What Rabbit and Erlang version has this extra debug logging or tracing capabilities ?

Current Rabbit/Erlang version
RabbitMQ 3.11.7 , Erlang 25.2.1

As this RabbitMQ version 3.11.7 is outdated
What would you recommend if we have to upgrade RabbitMQ /Erlang to confirm if we are able to replicate the issue in latest version ?
NOTE- We are using NODE - NodeJs v18.15.0 , and we are using Classic_Queue mirroring

[GitJadhav]

@michaelklishin @lukebakken
The following fix was merged into 3.11.27
(see rabbit_writer: ignore unknown messages (backport #9994) (backport #9996) by mergify[bot] ·
Pull Request #10005 · rabbitmq/rabbitmq-server ) with the goal to ‘no longer kill the connection’ or terminate the connection .
I also see that “community support for 3.11” ends dec 31 2023.
Any ETA on RabbitMQ version availability with the above fix - Will we have release 3.11.27 before Dec 31 ??

[michaelklishin]

Nothing prevents us from shipping a 3.11 release in 2024 if we choose to.

…

On Tue, 5 Dec 2023 at 13:08, GitJadhav ***@***.***> wrote: @michaelklishin <https://github.com/michaelklishin> @lukebakken <https://github.com/lukebakken> The following fix was merged into 3.11.27 (see rabbit_writer: ignore unknown messages (backport #9994) (backport #9996) by mergify[bot] · Pull Request #10005 · rabbitmq/rabbitmq-server <#10005> ) with the goal to ‘no longer kill the connection’ or terminate the connection . I also see that “community support for 3.11” ends dec 31 2023. Any ETA on when we can use the RabbitMQ version with the above fix - Will we have release 3.11.27 before Dec 31 ?? — Reply to this email directly, view it on GitHub <#9803 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAIQWOXET6KUHILNT7FT3YH5PKTAVCNFSM6AAAAAA6R7ZFI6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TONRXGU2DI> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

-- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

[GitJadhav]

@michaelklishin Sounds good - Thank you
We were planning our upgrade MOP and timelines and hence wanted to confirm on the 3.11.27 availability .

[GitJadhav]

@michaelklishin
With reference to this update
https://github.com/rabbitmq/rabbitmq-server/milestone/185?closed=1
What is the process to expedite our request for release of patch 3.11.27
Is this "unknown messaging" fix merged in 3.12.x ??
As we are deciding on the best go forward path with RabbitMQ Upgrade considering support timelines .

[michaelklishin]

When a new release ships, you will find it under Releases in this GitHub repository and then on https://rabbitmq.com/changelog.html.

Every issue or PR worth mentioning has a milestone set to it here on GitHub.

[michaelklishin]

There is currently one outstanding PR for 3.11.27. It may ship next week but we do not offer any ETA promises.

[michaelklishin]

https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.11.27

[GitJadhav]

@michaelklishin
Thank you for sharing 3.11.27 updates
I do see 3.11.27 release notes and rpm files
https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.11.27
I am aware it has not yet been released in change log
https://rabbitmq.com/changelog.html

I had a follow up question - As we are planning to upgrade our dependency stack to Rabbit 3.12.11 , with the consideration of 3.11.x EOL timelines
@gomoripeti
Will this fix be ported in 3.12.11 ??
Core Broker
Bug Fixes
Avoids a rare exception that could stop TCP socket writes on a client connection.

GitHub issues: #9991, #9803

[michaelklishin]

Answered in #10005 (comment) and #10005 (comment)

[GitJadhav]

@michaelklishin @gomoripeti

The latest comments from @gomoripeti referring to another customer observation link

FYI..
We just did some internal smoke test with RabbitMQ 3.11.27 , Erlang 25.2.1
We just upgraded our RabbitMQ version from 3.11.7 --> 3.11.27 , and left Erlang/OTP at 25.2.1

@gomoripeti is saying stray message {Ref,Ok} is coming from Erlang/OTP ,like the inet driver or similar and that it might be fixed in a newer version than 25.3.
*this is the code in ra that sends a {Ref, ok} https://github.com/rabbitmq/ra/blob/main/src/ra_server_proc.erl#L349
But I couldn't pin point any candidate in the Erlang codebase nor found any github issue that looked relevant in the erlang repo.

With that mention - I have the following clarification - Appreciate if you can provide us an path forward
1.Do you recommend us to test with RabbitMQ 3.11.27 , Erlang/OTP 25.3.x
2.Will this stray message { Ref , OK } issue even occur with RabbitMQ 3.12.10 , Erlang/OTP 25.3.x

[lukebakken]

With that mention - I have the following clarification - Appreciate if you can provide us an path forward
1.Do you recommend us to test with RabbitMQ 3.11.27 , Erlang/OTP 25.3.x
2.Will this stray message { Ref , OK } issue even occur with RabbitMQ 3.12.10 , Erlang/OTP 25.3.x

Why do you think anyone has the ability to answer those questions, considering nobody has an idea of what may be the cause?

Wait for the next RabbitMQ release which will ignore and log these stray messages.

[michaelklishin]

3.11 goes out of general support in less than three weeks. We recommend using 3.12 on Erlang 26.

Not every change has a public issue, not every issue makes into release notes, and not every change is intentional. All of these are true for Erlang as well as other large projects.

[michaelklishin]

Well, if this system is currently on 3.11, then adopting 3.11.27 ASAP makes sense. 3.12.11 won't ship this week but 3.11.27 is already out.

[GitJadhav]

@michaelklishin
Thank you for all the prompt support and assist with the deliverables .Appreciate it
We have completed Upgrade and Sanity tests
RabbitMQ 3.11.27 , Erlang/OTP 25.2.1
RabbitMQ 3.11.27 , Erlang/OTP 25.3.2.7
NOTE -- Confirming we do see enhanced logging - FYI
We will certify RabbitMQ 3.12.11 ( once it is shipped out) with Erlang 26

[michaelklishin]

@GitJadhav thank you for reporting back. If this improves the stability of your system, that's great to hear.

The logging is arguably not that useful in its current form.

[GitJadhav]

@michaelklishin @lukebakken
This rabbitmq fix to ignore stray message ( writer , message_not_understood ) did help resolve our platform failure symptom .
Just wanted to share this update -- and thank your team for the contribution .
I hope this will help others in this community if they are facing a similar problem with regards to resolution steps.

[lukebakken]

Thank you for letting us know!