RabbitMQ 3.13.0 nodes with peer discovery enabled fails to start in an IPv6-only environment

[mnasiadka]

Describe the bug

In OpenStack Kolla-Ansible we support ipv6-only OpenStack deployments, including RabbitMQ.
The following config worked up until RMQ 3.13:

https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/kolla_configs/rabbitmq/rabbitmq.conf

https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/kolla_configs/rabbitmq/index.html

Logs and crash dump here: https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/kolla/rabbitmq/index.html

Reproduction steps

1. Use config from description to start RMQ 3.13

...

Expected behavior

Runs correctly

Additional context

No response

[dumbbell]

Hi! Thank you for the report.

How do you specify the path to the inetrc file?

The peer discovery subsystem starts a temporary intermediate node to query some properties of remote nodes. I suspect that this temporary node doesn't use this inetrc file.

[mnasiadka]

Here is the container log: https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/container_logs/rabbitmq.txt

We just put erl_inetrc under /etc/rabbitmq/erl_inetrc with this content: https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/kolla_configs/rabbitmq/erl_inetrc

Is there something more we should be doing?

[dumbbell]

No, this is a limitation in the way that temporary node is started probably. I will try to reproduce locally.

[lukebakken]

I suspect that this temporary node doesn't use this inetrc file

I'm sure you're right @dumbbell

[michaelklishin]

This may be worth delaying 3.13.1 over.

[dumbbell]

I'm starting to look at this issue today, bu I can't find where this /etc/rabbitmq/erl_inetrc file is configured. RabbitMQ itself doesn't look at a default location, so it must come from something outside of RabbitMQ.

Could you please share the output of the following two commands?

• rabbitmqctl eval 'os:getenv().'
• rabbitmqctl eval 'application:get_all_env(kernel).'

[artificial-intelligence]

we set this via the environment, see the jinja template here: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/rabbitmq/templates/rabbitmq-env.conf.j2 (line 10)

I don't currently have shell access to the CI env where this fails, but as you can see from the git history this part of the code didn't change since 2019, so here are the commands you wanted the output from, from a similar environment, but with a slightly older rmq release.
you can see the erl_inetrc is loaded there as well, I just edited out some stuff like hostnames and IPs:

rabbitmqctl eval 'os:getenv().'
["KOLLA_DISTRO_PYTHON_VERSION=3.10",
 "RABBITMQ_CLUSTER_COOKIE=REDACTED",
 "RABBITMQ_LOG_DIR=/var/log/kolla/rabbitmq","ROOTDIR=/usr/lib/erlang",
 "SHLVL=0","SYS_PREFIX=","RABBITMQ_NODENAME=rabbit@REDACTED",
 "RABBITMQ_PID_FILE=/var/lib/rabbitmq/mnesia/rabbitmq.pid",
 "RABBITMQ_HOME=/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.11.23",
 "RABBITMQ_LOG_BASE=/var/log/kolla/rabbitmq","RABBITMQ_DIST_PORT=25672",
 "OLDPWD=/","PROGNAME=erl","PWD=/var/lib/rabbitmq",
 "PATH=/usr/lib/erlang/erts-13.2.2.3/bin:/usr/lib/erlang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
 "PLUGINS_DIR=/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.11.23/plugins",
 "LANG=en_US.UTF-8","MNESIA_BASE=/var/lib/rabbitmq/mnesia",
 "KOLLA_SERVICE_NAME=rabbitmq","HOME=/var/lib/rabbitmq",
 "KOLLA_BASE_DISTRO=ubuntu","KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
 "HOSTNAME=REDACTED","KOLLA_BASE_ARCH=x86_64","ERL_EPMD_ADDRESS=REDACTED",
 "ERL_INETRC=/etc/rabbitmq/erl_inetrc","ERL_MAX_ETS_TABLES=50000",
 "ERL_MAX_PORTS=65536",
 "ERL_LIBS=/usr/lib/rabbitmq/lib/rabbitmq_server-3.11.23/plugins",
 "ERL_EPMD_PORT=4369","DEBIAN_FRONTEND=noninteractive",
 "ENABLED_PLUGINS_FILE=/etc/rabbitmq/enabled_plugins",
 "ERL_CRASH_DUMP=/var/log/kolla/rabbitmq/erl_crash.dump","EMU=beam",
 "BINDIR=/usr/lib/erlang/erts-13.2.2.3/bin",
 "CONFIG_FILE=/etc/rabbitmq/rabbitmq",
 "ADVANCED_CONFIG_FILE=/etc/rabbitmq/advanced.config"]

and the other command (your command missed a ' at the end):

 rabbitmqctl eval 'application:get_all_env(kernel).'
[{shell_docs_ansi,auto},
 {net_ticktime,60},
 {shutdown_func,{rabbit_prelaunch,shutdown_func}},
 {inet_default_connect_options,[{nodelay,true}]},
 {connect_all,true},
 {logger_level,notice},
 {inet_dist_listen_min,25672},
 {inetrc,'/etc/rabbitmq/erl_inetrc'},
 {logger_sasl_compatible,false},
 {logger,
     [{handler,default,logger_std_h,
          #{config => #{type => standard_io},
            formatter =>
                {logger_formatter,
                    #{legacy_header => true,single_line => false}}}}]},
 {inet_dist_listen_max,25672},
 {inet_dist_use_interface,{REDACTED}},
 {net_tickintensity,4},
 {prevent_overlapping_partitions,false}]

HTH in the debugging process.

[dumbbell]

Thank you very much! Yes, it definitely helps.

I fixed the command in my comment, thanks.

[dumbbell]

As a side question, why do you set both $ERL_INETRC and -kernel inetrc …?

This won’t change the debuggint or fix anything, I’m just curious.

[dumbbell]

@mnasiadka, @artificial-intelligence: I prepared a patch in pull request #10759. I’m not sure yet how to create a testcase for this. Could you please give it a try in your environment?

[mnasiadka]

@dumbbell I'm afraid we have no experience in building a package with that PR content - is there an option you would make an rpm or deb package available for testing?
Sorry for the delayed response.

[dumbbell]

@mnasiadka: Sure. I took a shortcut and replaced the rabbit_peer_discovery.beam module inside the RabbitMQ 3.13.0 Debian package :-) This is the only module modified in this patch, so I expect it to work. Here it is:

https://people.freebsd.org/~dumbbell/rabbitmq/rabbitmq-server_3.13.0-1_all.deb

[dormanze]

I used the new ported code and it didn't solve my problem. The error message is as follows:

2024-03-26 13:47:31.437298+08:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> BOOT FAILED
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> ===========
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> Exception during startup:
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> error:{case_clause,{ok,'/usr/local/rabbitmq_server-3.13.0/etc/rabbitmq/erl_inetrc'}}
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>     rabbit_peer_discovery:maybe_add_inetrc_arguments/1, line 442
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>     rabbit_peer_discovery:query_node_props/1, line 392
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>     rabbit_peer_discovery:sync_desired_cluster/3, line 188
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>     rabbit_db:init/0, line 65
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>     rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 51
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>     rabbit_boot_steps:run_step/2, line 58
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>     rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 22
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>     rabbit_boot_steps:run_boot_steps/1, line 23
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>   crasher:
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     initial call: application_master:init/4
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     pid: <0.247.0>
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     registered_name: []
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     exception exit: {{case_clause,{ok,'/usr/local/rabbitmq_server-3.13.0/etc/rabbitmq/erl_inetrc'}},
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>                      {rabbit,start,[normal,[]]}}
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>       in function  application_master:init/4 (application_master.erl, line 142)
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     ancestors: [<0.246.0>]
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     message_queue_len: 1
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     messages: [{'EXIT',<0.248.0>,normal}]
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     links: [<0.246.0>,<0.44.0>]
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     dictionary: []
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     trap_exit: true
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     status: running
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     heap_size: 2586
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     stack_size: 28
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>     reductions: 176
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>   neighbours:
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>
2024-03-26 13:47:32.446210+08:00 [notice] <0.44.0> Application rabbit exited with reason: {{case_clause,{ok,'/usr/local/rabbitmq_server-3.13.0/etc/rabbitmq/erl_inetrc'}},{rabbit,start,[normal,[]]}}

my erl_inetrc:

{inet6, true}.
{file, hosts, "/etc/hosts"}.
{file, resolv, "/etc/resolv.conf"}.

For other configurations, see: #10840

[dumbbell]

@dormanze: Thank you for testing!

The patch was not ready anyway and should not have been merged. I reverted it and will work on an improved patch.

[dormanze]

@dumbbell Thank you for your reply. Please let me know if you need me to cooperate with uploading any configuration.

[dormanze]

Hi @dumbbell , any progress on this issue? We desperately need to adapt to 3.13. If there is a fix, please let me know in time to cooperate with the verification. Thank you very much.

[michaelklishin]

@dormanze it's incredibly rude to nudge open source maintainers to hurry up to deliver a fix in a tool you very likely do not pay for, and never have. We have a PR ready but it has shown an improvement only in some environments.

If you are desperately in need of a fix, you are welcome to contribute it, pay for commercial support, or patiently wait for the maintainers to do it.

[dumbbell]

@dormanze: The bug fix is probably ok, but I'm still working on the testsuite to have more confidence. It is challenging to test the part of peer discovery that triggered the regression @mnasiadka and you hit. The current test I have for IPv6 looks good, but having the equivalent for TLS inter-node communication is slightly harder to get right.

Your specific error is interesting. You used single quotes to quote the inetrc filename. Erlang uses this quoting for a type that is not a string. We could tell you to use double quotes and be done with it. But I think RabbitMQ should handle the use of single quotes from users likely used to Bourne shell quoting and do what’s needed internally to use the parameter as a filename. I’m considering this in the testsuite.

[dormanze]

I verified the addition of the double quotes and my problem was indeed solved. Thank you very much.@dumbbell

[dumbbell]

I’ve merged the patch to the main branch and will leave it there for a week or so. If no regressions are spotted, I will backport it to 3.13.x. Sorry for the time it took to merge, there were unrelated failures in CI that blocked it.

If you have a chance, please test the main branch :-)

[dumbbell]

The patch was integrated into the 3.13.x release branch and will be part of RabbitMQ 3.13.2.