SSL Letsencrypt

[jgustavo99]

• OS: Ubuntu 18.04
• EMQ: 3.1

Hello!

How can I add the ssl letsencrypt certificate in EMQX (Websocket and MQTT)?

I've already tried using the tutorial: https://medium.com/@emqtt/using-lets-encrypt-certificates-in-emq-b11e0e57efa6

But I did not succeed since the tutorial is from version 2.3,

[zhengyupan]

Hi @jgustavo99 In principle the TLS configuration of 2.3 and 3.1 is very similar. Could you describe at which step do you have an issue exactly? If you can give us the output on emqx, it could be helpful.

[chimit]

The same problem in v3.1-rc.1. Did exactly what is described in this article, but nothing works - HTTPS dashboard, WSS, TLS.

2019-04-09 10:50:10.193 [error] crasher:
    initial call: emqx_connection:init/1
    pid: <0.1741.0>
    registered_name: []
    exception error: no match of right hand side value {error,
                                                        {ssl_error,
                                                         {options,
                                                          {certfile,
                                                           "/etc/letsencrypt/live/broker.abc.com/fullchain.pem",
                                                           {error,eacces}}}}}
      in function  emqx_connection:init/1 (src/emqx_connection.erl, line 146)
    ancestors: [<0.1618.0>,<0.1617.0>,esockd_sup,<0.1351.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1618.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 5777
  neighbours:
2019-04-09 10:50:10.195 [error] supervisor: 'esockd_connection_sup - <0.1618.0>'
    errorContext: connection_crashed
    reason: {{badmatch,
                 {error,
                     {ssl_error,
                         {options,
                             {certfile,
                                 "/etc/letsencrypt/live/broker.abc.com/fullchain.pem",
                                 {error,eacces}}}}}},
             [{emqx_connection,init,1,
                  [{file,"src/emqx_connection.erl"},{line,146}]},
              {proc_lib,init_p_do_apply,3,
                  [{file,"proc_lib.erl"},{line,249}]}]}
    offender: [{pid,<0.1741.0>},
               {name,connection},
               {mfargs,
                   {emqx_connection,start_link,
                       [[{deflate_options,[]},
                         {max_conn_rate,500},
                         {active_n,100},
                         {zone,external}]]}}]
2019-04-09 10:50:10.278 [error] crasher:
    initial call: emqx_connection:init/1
    pid: <0.1745.0>
    registered_name: []
    exception error: no match of right hand side value {error,
                                                        {ssl_error,
                                                         {options,
                                                          {certfile,
                                                           "/etc/letsencrypt/live/broker.abc.com/fullchain.pem",
                                                           {error,eacces}}}}}
      in function  emqx_connection:init/1 (src/emqx_connection.erl, line 146)
    ancestors: [<0.1618.0>,<0.1617.0>,esockd_sup,<0.1351.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1618.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 5777
  neighbours:
2019-04-09 10:50:10.280 [error] supervisor: 'esockd_connection_sup - <0.1618.0>'
    errorContext: connection_crashed
    reason: {{badmatch,
                 {error,
                     {ssl_error,
                         {options,
                             {certfile,
                                 "/etc/letsencrypt/live/broker.abc.com/fullchain.pem",
                                 {error,eacces}}}}}},
             [{emqx_connection,init,1,
                  [{file,"src/emqx_connection.erl"},{line,146}]},
              {proc_lib,init_p_do_apply,3,
                  [{file,"proc_lib.erl"},{line,249}]}]}
    offender: [{pid,<0.1745.0>},
               {name,connection},
               {mfargs,
                   {emqx_connection,start_link,
                       [[{deflate_options,[]},
                         {max_conn_rate,500},
                         {active_n,100},
                         {zone,external}]]}}]

[fluffis]

Copying the certificate + key to /etc/emqx/certs/ and chown to emqx:emqx solves the error, but of cause creates an issue with the certbot reissue cycle and possible an issue with the privkey.

[chimit]

@turtleDeng any progress on this?

[jgustavo99]

Copying the certificate + key to /etc/emqx/certs/ and chown to emqx:emqx solves the error, but of cause creates an issue with the certbot reissue cycle and possible an issue with the privkey.

But then what would be the best way to solve the problem? No problems with certificate renewal, etc.

[i906]

For renewal, I followed this guide: https://certbot.eff.org/docs/using.html#renewing-certificates and created this post deploy hook.

#!/bin/sh

set -e

for domain in $RENEWED_DOMAINS; do
        case $domain in
        example.com)
                daemon_cert_root=/etc/emqx/certs

                # Make sure the certificate and private key files are
                # never world readable, even just for an instant while
                # we're copying them into daemon_cert_root.
                umask 077

                cp "$RENEWED_LINEAGE/cert.pem" "$daemon_cert_root/cert.pem"
                cp "$RENEWED_LINEAGE/privkey.pem" "$daemon_cert_root/key.pem"
                cp "$RENEWED_LINEAGE/chain.pem" "$daemon_cert_root/cacert.pem"

                # Apply the proper file ownership and permissions for
                # the daemon to read its certificate and key.
                chown emqx:emqx "$daemon_cert_root/cert.pem" \
                        "$daemon_cert_root/key.pem" \
                        "$daemon_cert_root/cacert.pem"

                chmod 400 "$daemon_cert_root/cert.pem" \
                        "$daemon_cert_root/key.pem" \
                        "$daemon_cert_root/cacert.pem"

                service emqx restart >/dev/null
                ;;
        esac
done

[nightwolfz]

I managed to make it work in a slightly different way.

Use the script from @i906 but comment out the line that copies cacert.pem, then in your emqx.conf set it as following:

#### From letsencrypt
listener.wss.external.keyfile = /etc/emqx/certs/key.pem
listener.wss.external.certfile = /etc/emqx/certs/cert.pem

#### Has to be the one from EMXQ
listener.wss.external.cacertfile = /etc/emqx/certs/cacert.pem

[chimit]

Still waiting for a simple built-in solution.

[citkane]

From the Certbot docs:
https://certbot.eff.org/docs/using.html#where-are-my-certificates

It is OK to:

chmod 0755 /etc/letsencrypt/{live,archive}
chgrp emqx /etc/letsencrypt/live/<domain>/privkey.pem
chgrp emqx /etc/letsencrypt/archive/<domain>/privkey1.pem
chmod 0640 /etc/letsencrypt/live/<domain>/privkey.pem
chmod 0640 /etc/letsencrypt/archive/<domain>/privkey1.pem

If you will never downgrade to an older version of Certbot.

I can confirm that this works on a fresh cert install.
The docs claim that these permissions will be retained across renewals, but I have not tested this yet.

My question is:
Does EMQX cache the cert files and will EMQX need to be restarted in a Certbot post deploy hook?

[a9965]

Hi.
I've been stuck in this problem for two days and I can't quite understand what the procedure is.

I direct the certificate path to live or archive?

Why do we give emqx permissions in the archive folder?

Thank you.

[zmstone]

sorry for the super late reply.
Erlang doc gave some hint: http://erlang.org/doc/man/ssl.html#clear_pem_cache-0
i.e. Erlang will cache PEMs and check changes on disk. I am not entirely sure about the check interval though.

As for SLL certificate files configuration, I believe this is the answer:
#3720 (comment)

[teenkevo]

For renewal, I followed this guide: https://certbot.eff.org/docs/using.html#renewing-certificates and created this post deploy hook.

#!/bin/sh

set -e

for domain in $RENEWED_DOMAINS; do
        case $domain in
        example.com)
                daemon_cert_root=/etc/emqx/certs

                # Make sure the certificate and private key files are
                # never world readable, even just for an instant while
                # we're copying them into daemon_cert_root.
                umask 077

                cp "$RENEWED_LINEAGE/cert.pem" "$daemon_cert_root/cert.pem"
                cp "$RENEWED_LINEAGE/privkey.pem" "$daemon_cert_root/key.pem"
                cp "$RENEWED_LINEAGE/chain.pem" "$daemon_cert_root/cacert.pem"

                # Apply the proper file ownership and permissions for
                # the daemon to read its certificate and key.
                chown emqx:emqx "$daemon_cert_root/cert.pem" \
                        "$daemon_cert_root/key.pem" \
                        "$daemon_cert_root/cacert.pem"

                chmod 400 "$daemon_cert_root/cert.pem" \
                        "$daemon_cert_root/key.pem" \
                        "$daemon_cert_root/cacert.pem"

                service emqx restart >/dev/null
                ;;
        esac
done

When should I run this script. And is it possible to create some logic to run it automatically?

[zmstone]

There is no need to restart emqx for certs renewal, emqx relads the files everything 2 minutes or so.
Unless you’d like to force all clients to reconnect immediately.

When should I run this script

anytime after the new certs are issued.

And is it possible to create some logic to run it automatically?

Do you mean inside emqx?
I’m afraid not, because certs renewal is not quite a standardised procedure

[teenkevo]

There is no need to restart emqx for certs renewal, emqx relads the files everything 2 minutes or so. Unless you’d like to force all clients to reconnect immediately.

When should I run this script

anytime after the new certs are issued.

And is it possible to create some logic to run it automatically?

Do you mean inside emqx? I’m afraid not, because certs renewal is not quite a standardised procedure

Thank you so much, you don’t how helpful this is.

[JiapengLi]

emqx ctl pem_cache clean all will force clear pem cache.

share my script in case someone may need.

#!/bin/bash

RENEWED_LINEAGE=/etc/letsencrypt/live/YOUR_DOMAIN
daemon_cert_root=/etc/emqx/certs

# Make sure the certificate and private key files are
# never world readable, even just for an instant while
# we're copying them into daemon_cert_root.
umask 077

cp "$RENEWED_LINEAGE/cert.pem" "$daemon_cert_root/cert.pem"
cp "$RENEWED_LINEAGE/privkey.pem" "$daemon_cert_root/key.pem"
cp "$RENEWED_LINEAGE/chain.pem" "$daemon_cert_root/cacert.pem"

# Apply the proper file ownership and permissions for
# the daemon to read its certificate and key.
chown emqx:emqx "$daemon_cert_root/cert.pem" \
        "$daemon_cert_root/key.pem" \
        "$daemon_cert_root/cacert.pem"

chmod 400 "$daemon_cert_root/cert.pem" \
        "$daemon_cert_root/key.pem" \
        "$daemon_cert_root/cacert.pem"
        
emqx ctl pem_cache clean all