AMQP 1.0 connection authentication stopped working against the HTTP authN backend starting with 3.11.10

[lukebakken]

Discussed in #9031

^{Originally posted by motmot80 August 8, 2023}

Describe the bug

Since RabbitMQ 3.11.10 it isnt possible to authenticate amqp-v1.0 clients with the http-auth-plugin.

• RMQ 3.11.9 with Erlang 25.3.2.3: working
• RMQ 3.10.25​ with Erlang 25.3.2.3: working
• RMQ 3.11.10 with Erlang 25.3.2.3: not working => auth_failure,"Refused"

It seems that the lastest RMQ versions may have broken the plugin combination

• rabbitmq_auth_backend_http
• rabbitmq_amqp1_0

Reproduction steps

1. Install RMQ > 3.11.10
2. Install REST-Mock as http auth plugin counter part replying with success
3. Activate and configure rabbitmq_auth_backend_http
4. Activate rabbitmq_amqp1_0
5. Connect AMQPv1.0 client (QPID-JMS, amqpnetlite, ...) with username & password for http-auth

Minimal java client:

org.apache.qpid.jms.JmsConnectionFactory jmsConnectionFactory = new org.apache.qpid.jms.JmsConnectionFactory("amqps://host:5671");
jmsConnectionFactory.setUsername("BOB");
jmsConnectionFactory.setPassword("PASSWORD");
jmsConnectionFactory.createConnection().createSession();

Expected behavior

Same behavior as RMQ 3.11.9 or 3.10.25 (< 3.11.10):

Authentication is working

Additional context

Maybe the Oauth 2 refactorings have broken the interface between the amqpv1 and the http-auth plugin.

Thanks in advance and best regards
Thomas

RabbitMQ stack trace

2023-08-09 09:32:15.505849-07:00 [info] <0.822.0> accepting AMQP connection <0.822.0> (127.0.0.1:39236 -> 127.0.0.1:5672)
2023-08-09 09:32:15.513829-07:00 [debug] <0.822.0> User 'admin' failed authentication by backend rabbit_auth_backend_internal
2023-08-09 09:32:15.513994-07:00 [debug] <0.822.0> auth_backend_http: GET http://localhost:8000/auth/user?username=admin&password=foobar
2023-08-09 09:32:15.514031-07:00 [debug] <0.822.0> auth_backend_http: request timeout: 15000, connection timeout: 15000
2023-08-09 09:32:15.516584-07:00 [debug] <0.822.0> auth_backend_http: response code is 200, body: "allow administrator"
2023-08-09 09:32:15.516672-07:00 [debug] <0.822.0> User 'admin' authenticated successfully by backend rabbit_auth_backend_http
2023-08-09 09:32:15.516725-07:00 [info] <0.822.0> AMQP 1.0 connection <0.822.0>: user 'admin' authenticated
2023-08-09 09:32:15.525831-07:00 [debug] <0.822.0> AMQP 1.0 connection.open frame: hostname = localhost, extracted vhost = /, idle_timeout = 60000
2023-08-09 09:32:15.531141-07:00 [warning] <0.830.0> AMQP 0-9-1 client call timeout was 70000 ms, is updated to a safe effective value of 130000 ms
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> User 'admin' authentication failed with exit:{unknown_auth_props,<<"admin">>,
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>                                               [{rabbit_auth_backend_http,
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>                                                 #Fun<rabbit_auth_backend_http.0.99357657>}]}:
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> [{rabbit_auth_backend_internal,user_login_authentication,2,
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>                                [{file,"rabbit_auth_backend_internal.erl"},
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>                                 {line,99}]},
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>  {rabbit_access_control,try_authenticate,3,
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>                         [{file,"rabbit_access_control.erl"},{line,86}]},
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>  {rabbit_access_control,'-check_user_login/2-fun-0-',4,
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>                         [{file,"rabbit_access_control.erl"},{line,51}]},
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>  {lists,foldl,3,[{file,"lists.erl"},{line,1350}]},
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>  {rabbit_access_control,check_user_login,2,
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>                         [{file,"rabbit_access_control.erl"},{line,36}]},
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>  {rabbit_direct,connect,5,[{file,"rabbit_direct.erl"},{line,103}]},
2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0>  {erpc,execute_call,4,[{file,"erpc.erl"},{line,589}]}]
2023-08-09 09:32:15.537889-07:00 [warning] <0.830.0> Closing session for connection <0.822.0>:
2023-08-09 09:32:15.537889-07:00 [warning] <0.830.0> {auth_failure,"Refused"}
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>   crasher:
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     initial call: rabbit_amqp1_0_session_process:init/1
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     pid: <0.830.0>
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     registered_name: []
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     exception exit: {auth_failure,"Refused"}
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>       in function  gen_server2:init_it/6 (gen_server2.erl, line 600)
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     ancestors: [<0.828.0>,<0.824.0>,<0.821.0>,<0.820.0>,<0.737.0>,<0.736.0>,
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>                   <0.735.0>,<0.733.0>,<0.732.0>,rabbit_sup,<0.239.0>]
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     message_queue_len: 0
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     messages: []
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     links: [<0.828.0>]
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     dictionary: [{gen_server_call_timeout,130000}]
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     trap_exit: true
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     status: running
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     heap_size: 1598
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     stack_size: 28
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>     reductions: 7518
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>   neighbours:
2023-08-09 09:32:15.538062-07:00 [error] <0.830.0>
2023-08-09 09:32:15.538336-07:00 [error] <0.822.0> Error on AMQP 1.0 connection <0.822.0> (running), channel 0:
2023-08-09 09:32:15.538336-07:00 [error] <0.822.0> <<"Reader error: {error,\n               {{auth_failure,\"Refused\"},\n                {child,undefined,channel,\n                 {rabbit_amqp1_0_session_process,start_link,\n                  [{0,<0.822.0>,<0.829.0>,\n                    {user,<<\"admin\">>,\n                     [administrator],\n                     [{rabbit_auth_backend_http,\n                       #Fun<rabbit_auth_backend_http.0.99357657>}]},\n                    <<\"/\">>,1048568,\n                    {amqp_adapter_info,\n                     {0,0,0,0,0,65535,32512,1},\n                     5672,\n                     {0,0,0,0,0,65535,32512,1},\n                     39236,<<\"127.0.0.1:39236 -> 127.0.0.1:5672\">>,\n                     {'AMQP',\"1.0\"},\n                     [{ssl,false},\n                      {authz_backends,\n                       [{rabbit_auth_backend_http,\n                         #Fun<rabbit_auth_backend_http.0.99357657>}]}]},\n                    <0.825.0>}]},\n                 transient,true,300000,worker,\n                 [rabbit_amqp1_0_session_process]}}}\n[{rabbit_amqp1_0_reader,send_to_new_1_0_session,3,\n                        [{file,\"rabbit_amqp1_0_reader.erl\"},{line,736}]},\n {rabbit_amqp1_0_reader,handle_1_0_frame,4,\n                        [{file,\"rabbit_amqp1_0_reader.erl\"},{line,317}]},\n {rabbit_amqp1_0_reader,recvloop,2,\n                        [{file,\"rabbit_amqp1_0_reader.erl\"},{line,120}]},\n {rabbit_reader,run,1,[{file,\"rabbit_reader.erl\"},{line,453}]},\n {rabbit_reader,start_connection,5,[{file,\"rabbit_reader.erl\"},{line,352}]},\n {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]">>

[lukebakken]

Hm, even with RabbitMQ 3.11.9 I can see that the password is not passed to the auth backend for an AMQP 1.0 connection:

INFO <QueryDict: {'username': ['admin']}>
INFO "GET /auth/user?username=admin HTTP/1.1" 200 4

This is what I see when I run rabbitmqctl authenticate_user admin foobar:

INFO <QueryDict: {'username': ['admin'], 'password': ['foobar']}>
INFO "GET /auth/user?username=admin&password=foobar HTTP/1.1" 200 19

...and this is what I see with a regular AMQP 0.9.1 connection via this test program:

INFO <QueryDict: {'username': ['admin'], 'password': ['foobar']}>
INFO "GET /auth/user?username=admin&password=foobar HTTP/1.1" 200 19
INFO <QueryDict: {'username': ['admin'], 'vhost': ['/'], 'ip': ['::ffff:127.0.0.1'], 'tags': ['administrator']}>
INFO "GET /auth/vhost?username=admin&vhost=%2F&ip=%3A%3Affff%3A127.0.0.1&tags=administrator HTTP/1.1" 200 5

Continuing to investigate...

cc @michaelklishin

[lukebakken]

@motmot80 well, testing with the RabbitMQ 3.10.x branch still fails. It appears that the AMQP 1.0 plugin hasn't sent the supplied password to the HTTP auth backend for quite a while now:

INFO <QueryDict: {'username': ['admin']}>
INFO "GET /auth/user?username=admin HTTP/1.1" 200 4

In your environment, does your HTTP auth server bother to look at the password supplied or is it just the username?

I'm moving on to just fixing this issue rather than trying to understand when it broke, or if it ever worked 😹

[lukebakken]

It looks like PR #6931 is what broke the combination of the AMQP 1.0 plugin and multiple auth backends. cc @MarcialRosales @michaelklishin

[michaelklishin]

Addressed in #9045.

@motmot80 would you be able to test #9045 with an OCI (Docker) image or do you need a specific package (e.g. Debian or Windows)?

[michaelklishin]

@MarcialRosales @lukebakken is this not applicable to main and v3.12.x?

[MarcialRosales]

@michaelklishin yes, it should be applied to main and v3.12.x. Let me know if there are any conflicts. If there is any conflict around the seleniun tests, just do not merge those changes.
In any case, I am putting together a PR with main as base with the fix plus some other fixes around the test scripts. I am tagging it with backport to v3.12.x. I will make sure there are no conflicts.

[lukebakken]

Re-opening since #9045 is not the correct fix.

#9052 should do it.

[motmot80]

@motmot80 would you be able to test #9045 with an OCI (Docker) image or do you need a specific package (e.g. Debian or Windows)?

@michaelklishin We retested https://github.com/rabbitmq/rabbitmq-server-binaries-dev/releases/tag/v3.11.21-alpha.16 fix in one of our dev reference environments (RHEL 8, Erlang 25.3.2.3).

Works like a charm!

Thanks for the quick support.

Best regards
Thomas

[motmot80]

@lukebakken After upgraded to 3.12.13-debian-12-r2 the problem occurs again.

[lukebakken]

Hi @motmot80 - could you please do the following:

• Open a new issue
• Test with RabbitMQ 3.13.2 in your environment

Thanks

[MarcialRosales]

Hi @motmot80 , I am not able to reproduce this issue in 3.12.13 (docker image). Could you please provide rabbitmq logs with debug level enabled and also the token you are using?
Thanks