Report more TLS-related values via HTTP API

[pickettphil]

Using /api/overview or /api/nodes, it would be useful to see all ssl_options reported. In previous versions, we saw something like the following reported:

{"node":"rabbit@localhost","description":"RabbitMQ Management","path":"/","port":"15672","ssl":"true","ssl_opts":{"cacertfile":"/etc/rabbitmq/ca.pem","certfile":"/etc/rabbitmq/cert.pem","keyfile":"/etc/rabbitmq/key.pem"}}

Our configuration also specified verify, fail_if_no_peer_cert, versions ,.... As a result, we don't know what Rabbit does if any of these were overridden from command line.

I tested with both 3.5.7 and 3.6.1 and don't see any ssl_options reported at all.

[essen]

@pickettphil I can't reproduce so far. I see the options as expected on current stable when starting a single node with management enabled.

Can I see the relevant part of the configuration file? Is this with only management started or is there other nodes? Is the correct configuration file picked up at startup?

[michaelklishin]

@essen are trying with the tip of stable or 3.6.1?

On 30 mar 2016, at 12:21, Loïc Hoguin notifications@github.com wrote:

@pickettphil I can't reproduce so far. I see the options as expected on current stable when starting a single node with management enabled.

Can I see the relevant part of the configuration file? Is this with only management started or is there other nodes? Is the correct configuration file picked up at startup?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

[essen]

@michaelklishin Stable after doing "make up".

[michaelklishin]

Please try with 3.6.1 to compare ;)

On 30 mar 2016, at 13:18, Loïc Hoguin notifications@github.com wrote:

@michaelklishin Stable after doing "make up".

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[essen]

No problem with the 3.6.1 tag (should I try the release directly?)

[michaelklishin]

There is no harm in trying 3.6.1 from a release tarball but I expect there to be no difference. We should wait for more details from @pickettphil.

[pickettphil]

I have the following rabbitmq.config:

[
{rabbit, [
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"/opt/pivotal/ssl/testca/cacert.pem"},
{certfile,"/opt/pivotal/ssl/server/cert.pem"},
{keyfile,"/opt/pivotal/ssl/server/key.pem"},
{verify,verify_peer},
{fail_if_no_peer_cert,false}]}
]}
].

and am running RabbitMQ 3.6.1 on Erlang 18.2 on CentOS 7. I see the
following:

http://localhost:15672/api/overview

{"management_version":"3.6.1","rates_mode":"basic","exchange_types":[{"name":"direct","description":"AMQP
direct exchange, as per the AMQP
specification","enabled":true},{"name":"headers","description":"AMQP
headers exchange, as per the AMQP
specification","enabled":true},{"name":"fanout","description":"AMQP fanout
exchange, as per the AMQP
specification","enabled":true},{"name":"topic","description":"AMQP topic
exchange, as per the AMQP
specification","enabled":true}],"rabbitmq_version":"3.6.1","cluster_name":"rabbit@rmqremote","erlang_version":"18.2","erlang_full_version":"Erlang/OTP
18 [erts-7.2] [source] [64-bit] [async-threads:64] [hipe]
[kernel-poll:true]","message_stats":{},"queue_totals":{"messages":0,"messages_details":{"rate":0.0},"messages_ready":0,"messages_ready_details":{"rate":0.0},"messages_unacknowledged":0,"messages_unacknowledged_details":{"rate":0.0}},"object_totals":{"consumers":0,"queues":0,"exchanges":8,"connections":0,"channels":0},"statistics_db_event_queue":0,"node":"rabbit@rmqremote
","statistics_db_node":"rabbit@rmqremote
","listeners":[{"node":"rabbit@rmqremote
","protocol":"amqp","ip_address":"::","port":5672},{"node":"rabbit@rmqremote
","protocol":"amqp/ssl","ip_address":"::","port":5671},{"node":"rabbit@rmqremote
","protocol":"clustering","ip_address":"::","port":25672}],"contexts":[{"node":"rabbit@rmqremote","description":"RabbitMQ
Management","path":"/","port":"15672"}]}

http://localhost:15672/api/nodes

[{"cluster_links":[],"disk_free":14689619968,"disk_free_details":{"rate":0.0},"fd_used":49,"fd_used_details":{"rate":0.0},"io_read_avg_time":0,"io_read_avg_time_details":{"rate":0.0},"io_read_bytes":1,"io_read_bytes_details":{"rate":0.0},"io_read_count":1,"io_read_count_details":{"rate":0.0},"io_sync_avg_time":0,"io_sync_avg_time_details":{"rate":0.0},"io_write_avg_time":0,"io_write_avg_time_details":{"rate":0.0},"mem_used":50617064,"mem_used_details":{"rate":196.8},"mnesia_ram_tx_count":13,"mnesia_ram_tx_count_details":{"rate":0.0},"proc_used":202,"proc_used_details":{"rate":0.0},"sockets_used":0,"sockets_used_details":{"rate":0.0},"partitions":[],"os_pid":"12087","fd_total":1024,"sockets_total":829,"mem_limit":410145587,"mem_alarm":false,"disk_free_limit":50000000,"disk_free_alarm":false,"proc_total":1048576,"rates_mode":"basic","uptime":277807,"run_queue":0,"processors":1,"exchange_types":[{"name":"direct","description":"AMQP
direct exchange, as per the AMQP
specification","enabled":true},{"name":"headers","description":"AMQP
headers exchange, as per the AMQP
specification","enabled":true},{"name":"fanout","description":"AMQP fanout
exchange, as per the AMQP
specification","enabled":true},{"name":"topic","description":"AMQP topic
exchange, as per the AMQP
specification","enabled":true}],"auth_mechanisms":[{"name":"PLAIN","description":"SASL
PLAIN authentication
mechanism","enabled":true},{"name":"AMQPLAIN","description":"QPid AMQPLAIN
mechanism","enabled":true},{"name":"RABBIT-CR-DEMO","description":"RabbitMQ
Demo challenge-response authentication
mechanism","enabled":false}],"applications":[{"name":"amqp_client","description":"RabbitMQ
AMQP Client","version":"3.6.1"},{"name":"asn1","description":"The Erlang
ASN1 compiler version
4.0.1","version":"4.0.1"},{"name":"compiler","description":"ERTS CXC 138
10","version":"6.0.2"},{"name":"crypto","description":"CRYPTO","version":"3.6.2"},{"name":"inets","description":"INETS
CXC 138 49","version":"6.1"},{"name":"kernel","description":"ERTS CXC 138
10","version":"4.1.1"},{"name":"mnesia","description":"MNESIA CXC 138
12","version":"4.13.2"},{"name":"mochiweb","description":"MochiMedia Web
Server","version":"2.13.0"},{"name":"os_mon","description":"CPO CXC 138
46","version":"2.4"},{"name":"public_key","description":"Public key
infrastructure","version":"1.1"},{"name":"rabbit","description":"RabbitMQ","version":"3.6.1"},{"name":"rabbit_common","description":"","version":"3.6.1"},{"name":"rabbitmq_management","description":"RabbitMQ
Management
Console","version":"3.6.1"},{"name":"rabbitmq_management_agent","description":"RabbitMQ
Management
Agent","version":"3.6.1"},{"name":"rabbitmq_web_dispatch","description":"RabbitMQ
Web Dispatcher","version":"3.6.1"},{"name":"ranch","description":"Socket
acceptor pool for TCP
protocols.","version":"1.2.1"},{"name":"sasl","description":"SASL CXC 138
11","version":"2.6.1"},{"name":"ssl","description":"Erlang/OTP SSL
application","version":"7.2"},{"name":"stdlib","description":"ERTS CXC 138
10","version":"2.7"},{"name":"syntax_tools","description":"Syntax
tools","version":"1.7"},{"name":"webmachine","description":"webmachine","version":"1.10.3"},{"name":"xmerl","description":"XML
parser","version":"1.3.9"}],"contexts":[{"description":"RabbitMQ
Management","path":"/","port":"15672"}],"log_file":"/var/log/rabbitmq/rabbit@rmqremote.log
","sasl_log_file":"/var/log/rabbitmq/rabbit@rmqremote-sasl.log
","db_dir":"/var/lib/rabbitmq/mnesia/rabbit@rmqremote
","config_files":["/etc/rabbitmq/rabbitmq.config"],"net_ticktime":60,"enabled_plugins":["rabbitmq_management"],"name":"rabbit@rmqremote
","type":"disc","running":true}]

Phil

On Wed, Mar 30, 2016 at 3:21 AM, Loïc Hoguin notifications@github.com
wrote:

@pickettphil https://github.com/pickettphil I can't reproduce so far. I
see the options as expected on current stable when starting a single node
with management enabled.

Can I see the relevant part of the configuration file? Is this with only
management started or is there other nodes? Is the correct configuration
file picked up at startup?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#163 (comment)

[pickettphil]

Oh, I see why I am seeing something different. The management plugin was configured to use SSL and it is those "ssl_opts" that are shown. This makes sense since SSL options for rabbit use the "ssl_options" key while SSL options for the management plugin use the "ssl_opts" key which is what is shown in the results above. I'd be more interested in seeing the server ssl options than the management plugin's ssl options, but reporting both may be best.

[pickettphil]

That's it. The management plugin's ssl options are reported just fine. The server's ssl options are not reported at all.

With config:

[
{rabbit, [
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"/opt/pivotal/ssl/testca/cacert.pem"},
{certfile,"/opt/pivotal/ssl/server/cert.pem"},
{keyfile,"/opt/pivotal/ssl/server/key.pem"},
{verify,verify_peer},
{fail_if_no_peer_cert,false}]}
]},
{rabbitmq_management, [
%% {listener, [{port, 15672}]}
{listener, [{port, 15672},
{ssl, true},
{ssl_opts, [{cacertfile, "/opt/pivotal/ssl/testca/cacert.pem"},
{certfile, "/opt/pivotal/ssl/server/cert.pem"},
{keyfile, "/opt/pivotal/ssl/server/key.pem"},
{verify, verify_none},
{fail_if_no_peer_cert,false}]}
]}
]}
].

At URL https://localhost:15672/api/overview, I see the following:

{"management_version":"3.6.1","rates_mode":"basic","exchange_types":[{"name":"topic","description":"AMQP topic exchange, as per the AMQP specification","enabled":true},{"name":"direct","description":"AMQP direct exchange, as per the AMQP specification","enabled":true},{"name":"fanout","description":"AMQP fanout exchange, as per the AMQP specification","enabled":true},{"name":"headers","description":"AMQP headers exchange, as per the AMQP specification","enabled":true}],"rabbitmq_version":"3.6.1","cluster_name":"rabbit@rmqremote","erlang_version":"18.2","erlang_full_version":"Erlang/OTP 18 [erts-7.2] [source] [64-bit] [async-threads:64] [hipe] [kernel-poll:true]","message_stats":{},"queue_totals":{"messages":0,"messages_details":{"rate":0.0},"messages_ready":0,"messages_ready_details":{"rate":0.0},"messages_unacknowledged":0,"messages_unacknowledged_details":{"rate":0.0}},"object_totals":{"consumers":0,"queues":0,"exchanges":8,"connections":0,"channels":0},"statistics_db_event_queue":0,"node":"rabbit@rmqremote","statistics_db_node":"rabbit@rmqremote","listeners":[{"node":"rabbit@rmqremote","protocol":"amqp","ip_address":"::","port":5672},{"node":"rabbit@rmqremote","protocol":"amqp/ssl","ip_address":"::","port":5671},{"node":"rabbit@rmqremote","protocol":"clustering","ip_address":"::","port":25672}],"contexts":[{"node":"rabbit@rmqremote","description":"RabbitMQ Management","path":"/","port":"15672","ssl":"true","ssl_opts":{"cacertfile":"/opt/pivotal/ssl/testca/cacert.pem","certfile":"/opt/pivotal/ssl/server/cert.pem","keyfile":"/opt/pivotal/ssl/server/key.pem","verify":"verify_none","fail_if_no_peer_cert":"false"}}]}

[essen]

Understood, will take a look. Thanks!

[essen]

Looks like we don't currently save listener SSL options. So first we need to record those when starting listeners and then add them to management. One question though: do we want just the SSL options or do we also care about some TCP options (like backlog for example)?

[ppickett-pivotal]

Maybe some history would help. The customer had added a base64 encoded RabbitMQ config via the RMQ Tile. They had assumed that the tuple "{verify, verify_peer}" was set because this was in their encoded config. As it turned out, this supplied config does not override the defaults of the tile. They had attempted to use the management API to view all ssl_options, but this and several other options were not displayed. I suggested that they use rabbitmqctl eval 'application:get_env(rabbit, ssl_options).' as a workaround.

That being said, I think just the SSL options are ok, but I guess the more information that we could show, the better. I think there is a fair amount of leeway for you to decide what is best.

[essen]

Thanks, will experiment to see what makes sense, perhaps having both TCP and SSL options while filtering defaults out would be good enough.

[ppickett-pivotal]

BTW - It was a default of the tile that was not being overridden which may or may not be a RabbitMQ default (to make things even more confusing).

[essen]

I have opened two PRs with a proposal for changes. This requires adding a field to a Mnesia table though; I'm not sure what's the procedure/policy regarding upgrades yet.

Output looks like this:

    "listeners": [
        {
            "node": "rabbit@localhost",
            "protocol": "amqp",
            "ip_address": "::",
            "port": 5672,
            "socket_opts": {
                "backlog": 128,
                "nodelay": true,
                "linger": [
                    true,
                    0
                ],
                "exit_on_close": false
            }
        },
        {
            "node": "rabbit@localhost",
            "protocol": "amqp/ssl",
            "ip_address": "::",
            "port": 5671,
            "socket_opts": {
                "backlog": 128,
                "nodelay": true,
                "linger": [
                    true,
                    0
                ],
                "exit_on_close": false,
                "versions": [
                    "tlsv1.2",
                    "tlsv1.1",
                    "tlsv1"
                ],
                "cacertfile": "/home/essen/rabbitmq/umbrella-master/deps/rabbitmq_test/certs/testca/cacert.pem",
                "certfile": "/home/essen/rabbitmq/umbrella-master/deps/rabbitmq_test/certs/server/cert.pem",
                "keyfile": "/home/essen/rabbitmq/umbrella-master/deps/rabbitmq_test/certs/server/key.pem"
            }
        },
        {
            "node": "rabbit@localhost",
            "protocol": "clustering",
            "ip_address": "::",
            "port": 25672,
            "socket_opts": []
        }
    ],

[michaelklishin]

There are schema upgrade functions but we cannot ship changes that alter schema in stable releases.

[essen]

Thought so. No problem then, will switch to master. :-) Thanks.

[essen]

I have pushed the same for master. I will however do a few more tests to make sure I didn't miss anything.

[essen]

The PRs add socket options to the /api/overview. What they don't do is change /api/nodes or /api/connections.

• For /api/nodes there's actually no listener information at all currently. Should I add it?
• For /api/connections there was already SSL information.

Also of note is that the PRs add more than just SSL information; they add all the socket options that were configured. One could therefore see the backlog value for example.

[michaelklishin]

@essen lets add listener information to GET /api/nodes, yes.

Another question: we should check if STOMP, MQTT, and their WebSockets counterparts need updating for this.

[michaelklishin]

Currently the broker doesn't start for me even after doing full clean and wiping RabbitMQ database directory:

Offline change; changes will take effect at broker restart.
MAKE="make" ERL_LIBS="/home/antares/git/umbrella/deps/rabbit/apps" RABBITMQ_NODENAME="rabbit" RABBITMQ_NODE_IP_ADDRESS="" RABBITMQ_NODE_PORT="" RABBITMQ_PID_FILE="/tmp/rabbitmq-test-instances/rabbit/rabbit.pid" RABBITMQ_LOG_BASE="/tmp/rabbitmq-test-instances/rabbit/log" RABBITMQ_MNESIA_BASE="/tmp/rabbitmq-test-instances/rabbit/mnesia" RABBITMQ_PLUGINS_DIR="/home/antares/git/umbrella/deps/rabbit/plugins" RABBITMQ_PLUGINS_EXPAND_DIR="/tmp/rabbitmq-test-instances/rabbit/plugins" RABBITMQ_SERVER_START_ARGS="" RABBITMQ_ENABLED_PLUGINS_FILE="/tmp/rabbitmq-test-instances/rabbit/enabled_plugins" \
  RABBITMQ_ALLOW_INPUT=true \
  /home/antares/git/umbrella/deps/rabbit/scripts/rabbitmq-server
Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]

Eshell V7.0  (abort with ^G)
(rabbit@ubuntu)1> 
  ##  ##
  ##  ##      RabbitMQ 0.0.0. Copyright (C) 2007-2016 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
  ######  ##
  ##########  Logs: /tmp/rabbitmq-test-instances/rabbit/log/rabbit.log

              Starting broker...

BOOT FAILED
===========

Error description:
    init:start_em/1
    init:start_it/1
    rabbit:start_it/1 line 427
    rabbit:broker_start/0 line 306
    rabbit:start_apps/1 line 472
    app_utils:manage_applications/6 line 117
    lists:foldl/3 line 1262
    rabbit:'-handle_app_error/1-fun-0-'/3 line 489
throw:{could_not_start,rabbit,
          {{schema_integrity_check_failed,
               [{table_attributes_mismatch,rabbit_listener,
                    [node,protocol,host,ip_address,port,opts],
                    [node,protocol,host,ip_address,port]}]},
           {rabbit,start,[normal,[]]}}}
Log file(s) (may contain more information):
   /tmp/rabbitmq-test-instances/rabbit/log/rabbit.log

{"init terminating in do_boot",{could_not_start,rabbit,{{schema_integrity_check_failed,[{table_attributes_mismatch,rabbit_listener,[node,protocol,host,ip_address,port,opts],[node,protocol,host,ip_address,port]}]},{rabbit,start,[normal,[]]}}}}

Could be something on my end, too.

[michaelklishin]

I managed to get past the above but now on the first HTTP API request after startup I'm getting

2016-04-10 04:19:01.226 [error] <0.430.0> CRASH REPORT Process <0.430.0> with 0 neighbours crashed with reason: bad function [{'_',[],[{[<<"api">>,<<"overview">>],[],rabbit_mgmt_wm_overview,[]},{[<<"api">>,<<"cluster-name">>],[],rabbit_mgmt_wm_cluster_name,[]},{[<<"api">>,<<"nodes">>],[],rabbit_mgmt_wm_nodes,[]},{[<<"api">>,<<"nodes">>,node],[],rabbit_mgmt_wm_node,[]},{[<<"api">>,<<"extensions">>],[],rabbit_mgmt_wm_extensions,[]},{[<<"api">>,<<"all-configuration">>],[],rabbit_mgmt_wm_definitions,[]},{[<<"api">>,<<"definitions">>],[],rabbit_mgmt_wm_definitions,[]},{[<<"api">>,<<"definitions">>,vhost],[],rabbit_mgmt_wm_definitions,...},...]}] in rabbit_web_dispatch_sup:'-loopfun/1-fun-0-'/2 line 78

and port 15672 becomes unresponsive.

[essen]

Didn't touch that so that's weird. I'll do a clean fetch and see if I can reproduce.

[essen]

I think you still have web dispatch on stable, because there's no such thing as a loop fun anymore on master, see https://github.com/rabbitmq/rabbitmq-web-dispatch/blob/master/src/rabbit_web_dispatch_sup.erl#L78 (and no mention of "loop" at all even).

[michaelklishin]

@essen you are right, I will give it another shot in a bit.