Debian - check_nrpe 3.0.1-3 Could not complete SSL handshake (rc=-1 SSL-error=5)

[Worst45]

On stable Debian 8.7, I installed:

• the 'testing' version of nagios-nrpe-plugins on my Nagios host (v3.0.1-3 amd64)
• the testing version of nagios-nrpe-server on another host (v3.0.1-3 amd64)

When I try to connect to the NRPE server:

• if I use '-n' option, everything is OK
• If SSL is activated, I get an error

Without SSL (-n option)

Client:

# /usr/lib/nagios/plugins/check_nrpe -H 192.168.x.y -n
NRPE v3.0.1

# tail /var/log/syslog
Apr 19 17:19:29 xxx check_nrpe: Remote 192.168.x.y accepted a Version 3 Packet

Server:

# tail /var/log/syslog
Apr 19 17:19:29 xxx nrpe[9899]: Connection from 192.168.x.z port 29371
Apr 19 17:19:29 xxx nrpe[9899]: Host address is in allowed_hosts
Apr 19 17:19:29 xxx nrpe[9899]: Host 192.168.x.z is asking for command '_NRPE_CHECK' to be run...
Apr 19 17:19:29 xxx nrpe[9899]: Response to 192.168.x.z: NRPE v3.0.1
Apr 19 17:19:29 xxx nrpe[9899]: Return Code: 0, Output: NRPE v3.0.1
Apr 19 17:19:29 xxx nrpe[9899]: Connection from 192.168.x.z closed.

With SSL

Client:

# /usr/lib/nagios/plugins/check_nrpe -H 192.168.x.y
CHECK_NRPE: Error - Could not connect to 192.168.x.y: Connection reset by peer

# tail /var/log/syslog
Apr 19 17:22:32 xxx check_nrpe: Error: Could not complete SSL handshake with 192.168.x.y: rc=-1 SSL-error=5

Server:

# tail /var/log/syslog
Apr 19 17:22:32 xxx nrpe[9902]: Connection from 192.168.x.z port 29627
Apr 19 17:22:32 xxx nrpe[9902]: Host address is in allowed_hosts
Apr 19 17:22:32 xxx nrpe[9902]: Error: Request packet version was invalid!
Apr 19 17:22:32 xxx nrpe[9902]: Could not read request from client 192.168.x.z, bailing out...
Apr 19 17:22:32 xxx nrpe[9902]: Connection from 192.168.x.z closed.

# cat /etc/nagios/nrpe.d/myconfig.cfg
allowed_hosts=127.0.0.1,192.168.x.y
nrpe_user=amavis
ssl_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
ssl_privatekey_file=/etc/ssl/private/ssl-cert-snakeoil.key
debug=1
ssl_logging=0xff
...
command[check_xxx]=....

What I tried:

• On the server, I tested the default snakeoil certificate, but also new generated ones:

ssh-keygen -t rsa -b 4096 -N "" -f /etc/nagios/nrpe.key
chmod 400 /etc/nagios/nrpe.key

• I tried to reduce the ssl_logging level (I read on another issue that it had lead to a segfault). Same error
• I tried to comment all the SSL stuff (ssl_* variables). Same error

More details

Nagios host:

/usr/lib/nagios/plugins/check_nrpe -V

NRPE Plugin for Nagios
Copyright (c) 1999-2008 Ethan Galstad (nagios@nagios.org)
Version: 3.0.1
Last Modified: 09-08-2016
License: GPL v2 with exemptions (-l for more info)
SSL/TLS Available: OpenSSL 0.9.6 or higher required

Server host:

/usr/sbin/nrpe -h

NRPE - Nagios Remote Plugin Executor
Copyright (c) 1999-2008 Ethan Galstad (nagios@nagios.org)
Version: 3.0.1
Last Modified: 09-08-2016
License: GPL v2 with exemptions (-l for more info)
SSL/TLS Available, OpenSSL 0.9.6 or higher required
TCP Wrappers Available

Usage: nrpe [-n] -c <config_file> [-4|-6] <mode>

...

I am on the same network, with no firewall on the path

[jfrickson]

If check_http -n -H 192.168.x.y works, then SSL is not enabled on the other end. Check the startup file and see if the command has the -n in it.

[jfrickson]

I installed 3.0.1 for testing issue #112. While in there, I discovered that /etc/default/nagios-nrpe-server has NRPE_OPTS="-n". Change that to NRPE_OPTS= and it should work fine.

[Widmo]

I have the same issue, option "-n" does not work.

[hedenface]

@Widmo Did you check the /etc/default/nagios-nrpe-server file to see see if that option were in there? And are you able to be a bit more descriptive? The original issue here was that -n was the only flag working, not that it wasn't working.

[Widmo]

Yes, i check it.

I had Debian 8, NRPE configured correctly with icinga2 server (doesn't matter)

It works:

/usr/lib/nagios/plugins/check_nrpe -H debian8.domain.com
NRPE v2.15

after upgrade debian do 9

/usr/lib/nagios/plugins/check_nrpe -H debian9.domain.com
CHECK_NRPE: Error - Could not connect to x.x.x.x: Connection reset by peer

Debian 9 logs:

Jun 19 15:16:18 unifi nrpe[25633]: Connection from x.x.x.x port 60593
Jun 19 15:16:18 unifi nrpe[25633]: Host address is in allowed_hosts
Jun 19 15:16:18 unifi nrpe[25633]: Error: Request packet version was invalid!
Jun 19 15:16:18 unifi nrpe[25633]: Could not read request from client x.x.x.x, bailing out...
Jun 19 15:16:18 unifi nrpe[25633]: Connection from x.x.x.x closed.

It look like SSL problem.

The same query, without SSL:

/usr/lib/nagios/plugins/check_nrpe -H debian9.domain.com -n
NRPE v3.0.1

[static-max]

Same problem like @Widmo, tough with nrpe-server on Ubuntu 17.04 Zesty

[viyullas]

Same problem here. Nagios host is a Debian8 and i have check_nrpe versions 2.* and 3.*
Both give the same problem when checking a server with Debian 9 and nrpe server 3.0.1

[Widmo]

@jfrickson can we open this issue? Thank You :)

[imtek]

Same issue as @viyullas :(

[hedenface]

@jfrickson Is no longer available to re-open issues. I'll take a peek when I get a few.

[hedenface]

Likely related to the OpenSSL changes. Can I see the versions of those with problems of your ssl libs used for compiling?

[imtek]

OpenSSL 1.1.0f 25 May 2017 on Debian 9 (Stretch)

[hedenface]

Before I can get around to this, maybe all of you could check out the following document and see if this can help with a fix for the time being:

https://support.nagios.com/kb/article.php?id=519

FYI: this issue is actually related to SSL and not NRPE, if I understand the problem correctly. Thanks!

[imtek]

When i explicitly mention "ssl_client_certs=0" in my nrpe.cfg it does not work (i never had it mentioned before by the way).

But i did not use SSL before at all. So i guess "NRPE" now expects you to use a certificate or there is still a issue with NRPE in regards to SSL?

[viyullas]

I didn't compile the nrpe server, used the one form debian 9 repository.

[imtek]

@viyullas same here.

[hedenface]

Compiling or installing a distributed package won't make any difference with this issue. It has to do with OpenSSL dropping support for the method we use to initiate anonymous SSL connections.

So, the options right now (as I see them) are:

1. Use the certificate generation method documented in the link I provided.
2. Use a different OS that still contains the necessary portion of the OpenSSL (e.g.: an older distro) (you probably shouldn't do this).
3. Manually compile an older version of OpenSSL on your system (anything pre-1.1.0 I believe) (please don't do this).
4. Don't use any security (please don't do this).
5. Use a different agent, such as NCPA.

We're actually reviewing our options pretty intensely here, and an actual fix should be incorporated into the codebase shortly.

[viyullas]

Are you planning to update nrpe to stop using the deprecated method?
Bacause those could be used as a workaround but we will update to Debian 9 in a near future and all the agents will be failing.

[imtek]

Forcing people to use ssl certificates on NRPE communication is a bit harsh, was this changed announced? Or are we mistaken?

Like @viyullas said i think a lot of people use NRPE without ssl certificates and this will force a lot of people to reconfigure their whole monitoring environment.

[hedenface]

@imtek - I don't know how many times to say it but this was a change from OpenSSL, NOT NRPE - The ability to use certificates has been present in NRPE for a while now.

@viyullas - I am furiously working on trying to get another anonymous method (AECDH) working. As soon as I have a fix I'll push it to the maint branch and issue a statement of some kind to this post.

[imtek]

@hedenface no problem thank you for explaining, maybe i did not understand it fully. But if you are not using any certificates why would this impact NRPE/Nagios?

[hedenface]

Because something has changed in the newer versions of OpenSSL that are causing the anonymous method we use to stop working. Unfortunately this is the most technical answer I have at this time :)

I am working on this as a priority, as it is not only affecting the few of you in this thread, but actually quite a lot more people.

I will keep the thread updated with information I find along with any other relevant info.

[viyullas]

Thanks a lot @hedenface

[hedenface]

@skelgaard Yes, but you're still stuck with a truly insecure method of transport unless you use public/private keys.

[skelgaard]

well i transport no password or other important info over the nrpe, so if it is secure isn't really a concern... and they all run over private network, so not a problem either.
I should however proberly setup keys yes... but will that solve the problem ? also if some servers run on the old debian still.

[hedenface]

I'd like to say that setting up public/private keys would solve the problem as it sits without the need for any recompiling, as long as use_adh is set to 0 everywhere.

[andysh-uk]

I'm suffering with this issue also on a Debian 8 system upgraded to Debian 9, however even after compiling the latest "maint" branch, I'm still seeing it.

I checkout out the git repo, checked out the "maint" branch and ran:

./configure
make all

Have I missed a step?

root@mickey:~# /root/nrpe/src/check_nrpe -H 127.0.0.1 -n
NRPE v3.1.1

root@mickey:~# /root/nrpe/src/check_nrpe -H 127.0.0.1
CHECK_NRPE: Error - Could not connect to 127.0.0.1: Connection reset by peer

root@mickey:~# tail /var/log/syslog
Jun 26 08:00:12 mickey nrpe[27590]: Error: Request packet version was invalid!
Jun 26 08:00:12 mickey nrpe[27590]: Could not read request from client 127.0.0.1, bailing out...
Jun 26 08:00:12 mickey check_nrpe: Error: (nerrs = 0)(!log_opts) Could not complete SSL handshake with 127.0.0.1: rc=-1 SSL-error=5

Unfortunately my hosting provider run the Nagios system which interacts with this agent on my server for health-checks, so I can't modify or confirm the Nagios configuration.

[hedenface]

@pandy06269 The maint branch is considered unstable at the moment, as this is where all the changes are going for 3.2.0. I swear it worked at some point yesterday :)

[hedenface]

@pandy06269 It should be fixed (again) via 1715dcd

[sebastic]

NRPE 3.1.1 built with OpenSSL 1.1.0 is now available in stretch-backports for Debian stable users.

[theseal]

I'm not sure if this is the correct place for this post, might be better
as a BTS. But since we got both the upstream developer and the Debian
maintainer aware of this thread I give it a shot.

When trying to apply 99b0de9 on 3.0.1 as bundled in Stretch I found the
that the package doesn't depend on openssl 1.1.x:

Build-Depends: debhelper (>= 9),
               dh-autoreconf,
               dh-systemd,
               libssl1.0-dev,
               libwrap0-dev,
               openssl

So the upstream patch won't help in this case.

After some further digging I think I found the issue. It is
probably
https://anonscm.debian.org/git/pkg-nagios/pkg-nrpe.git/commit/?id=a4b095ea
in the Debian package which break TLS(DH?) in nrpe 3.0.1. My take on the
commit message is that it is supposed to fix build issues, not to disable DH.

By disabling need_dh in the configure script with
11_reproducible_dh.h.patch:

[...]
--- a/configure.ac
+++ b/configure.ac
@@ -307,7 +307,7 @@ AC_ARG_ENABLE([ssl],
 dnl Optional SSL library and include paths
 if test x$check_for_ssl = xyes; then
        # need_dh should only be set for NRPE
-       need_dh=yes
+       need_dh=no
        AC_NAGIOS_GET_SSL
 fi

We skip the following part of the configure script (love the indentation…):

        if test x$found_ssl = xyes -a x$need_dh = xyes; then

                # Find the openssl program

                if test x$need_dh = xyes; then
                        # Extract the first word of "openssl", so it can be a program name with args.
set dummy openssl; ac_word=$2
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
$as_echo_n "checking for $ac_word... " >&6; }
if ${ac_cv_path_sslbin+:} false; then :
  $as_echo_n "(cached) " >&6
else
  case $sslbin in
  [\\/]* | ?:[\\/]*)
  ac_cv_path_sslbin="$sslbin" # Let the user override the test with a path.
  ;;
  *)
  as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
for as_dir in $ssl_dir/sbin$PATH_SEPARATOR$ssl_dir/bin$PATH_SEPARATOR$PATH
do
  IFS=$as_save_IFS
  test -z "$as_dir" && as_dir=.
    for ac_exec_ext in '' $ac_executable_extensions; do
  if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
    ac_cv_path_sslbin="$as_dir/$ac_word$ac_exec_ext"
    $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
    break 2
  fi
done
  done
IFS=$as_save_IFS

  test -z "$ac_cv_path_sslbin" && ac_cv_path_sslbin="value-if-not-found"
  ;;
esac
fi
sslbin=$ac_cv_path_sslbin
if test -n "$sslbin"; then
  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $sslbin" >&5
$as_echo "$sslbin" >&6; }
else
  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
$as_echo "no" >&6; }
fi


                        $as_echo "#define USE_SSL_DH 1" >>confdefs.h

                        # Generate DH parameters
                        if test -f "$sslbin"; then
                                echo ""
                                echo "*** Generating DH Parameters for SSL/TLS ***"
                                # awk to strip off meta data at bottom of dhparam output
                                $sslbin dhparam -C 2048 | awk '/^-----/ {exit} {print}' > include/dh.h
                        fi
                fi
        fi

Which includes the definition of USE_SSL_DH:

[...]
                        $as_echo "#define USE_SSL_DH 1" >>confdefs.h
[...]

And that is causing the code to disable DH in nrpe.c:

[...]
#ifndef USE_SSL_DH
        ssl_opts = SSL_OP_ALL;
        sslprm.allowDH = 0;
#endif
[...]
#ifdef USE_SSL_DH
                dh = get_dh2048();
                SSL_CTX_set_tmp_dh(ctx, dh);
                DH_free(dh);
#endif

I'm no C programmer and not very good at openssl but I guess that it is
probably best to modify the patch so it won't remove the definition of
USE_SSL_DH, and after that it might be possible to release it to
stable-updates. That at least works with tests in our environment and
creates backwards compatibility with older check_nrpe.

[hedenface]

@theseal See c339ba2 which was committed for #143.

And yes, the indentation in the configure script is wonderful. The entire block around the need_dh is as follows:

need_dh=yes
AC_ARG_WITH([need_dh],
        AS_HELP_STRING([--with-need-dh],[set to 'no' to not include Diffie-Hellman SSL logic]),
        [need_dh=$withval],
        [nrpe_group=need_dh])

dnl Optional SSL library and include paths
if test x$check_for_ssl = xyes; then
        # need_dh should only be set for NRPE
        #need_dh=yes
        AC_NAGIOS_GET_SSL
fi

[sebastic]

@theseal wrote:

When trying to apply 99b0de9 on 3.0.1 as bundled in Stretch I found the that the package doesn't depend on openssl 1.1.x:

NRPE 3.0.1 fails to build with OpenSSL 1.1.0 in stretch, the support for OpenSSL 1.1.0 was added in NRPE 3.1.x, which is now available in stretch-backports.

Both the NRPE 3.0.x & 3.1.x packages have the issue that setting need_dh to no, either with the reproducible build patch or the --with-need-dh configure option, also disables the USE_SSL_DH definition. This is a bug in the package that was not spotted before, thanks for sharing your findings!

I've updated the Debian package to address this issue in the following revision:

• 3.2.0-2 (unstable)
• 3.1.1-1~bpo9+2 (stretch-backports)
• 3.0.1-3+deb9u1 (stretch)

The updates for unstable & stretch-backports have been uploaded and will be available on the mirrors in a few hours.

@theseal, can you build the proposed update for stretch and confirm that also fixes the issue for you?
The sources are available in the stretch branch of the package git repository: https://anonscm.debian.org/git/pkg-nagios/pkg-nrpe.git/log/?h=stretch

[imtek]

After getting the new nrpe package from backports the issues still remain for me (running without a certificate) like a lot of people (will move to certificate validation soon). The allowed host is still running Debian Jessie (due to these issues on Stretch).

Jul 6 15:41:11 hostname nrpe[14160]: CONN_CHECK_PEER: is this a blessed machine: x.x.x.x port 10971
Jul 6 15:41:11 hostname nrpe[14160]: Connection from x.x.x.x port 10971
Jul 6 15:41:11 hostname nrpe[14160]: is_an_allowed_host (AF_INET): is host >x.x.x.x< an allowed host >x.x.x.x<
Jul 6 15:41:11 hostname nrpe[14160]: is_an_allowed_host (AF_INET): host is in allowed host list!
Jul 6 15:41:11 hostname nrpe[14160]: Host address is in allowed_hosts
Jul 6 15:41:11 hostname nrpe[14160]: Error: Request packet version was invalid!
Jul 6 15:41:11 hostname nrpe[14160]: Could not read request from client x.x.x.x, bailing out...
Jul 6 15:41:11 hostname nrpe[14160]: Connection from x.x.x.x closed.

Am i missing something?

[sebastic]

nagios-nrpe (3.1.1-1~bpo9+1) in stretch-backports doesn't have the USE_SSL_DH fix yet. In a few hours the -2 upload will be available which should fix the issue.

SSL support (without certificates configured) hasn't worked with the NRPE 3.x Debian packages, which is why SSL support was disabled by default (with the -n option) as documented in /usr/share/doc/nagios-nrpe-server/NEWS.Debian.gz which is shown on upgrade when apt-listchanges is installed.

[imtek]

@sebastic hopefully the patch(es) will help with that so that we can upgrade all our jessie machines to stretch, we never enabled SSL on our setup SSL support (without certificates configured) has never been configured on our side (just working without any certificates right now).

[theseal]

@sebastic Just built the proposed update for stretch and it worked like it should, great work! 👍

[imtek]

Upgraded to 3.1.1-1~bpo9+2, no changes for me am i missing something?

[sebastic]

@imtek, if you enable ssl_logging you'll probably see the following in /var/log/syslog:

nrpe[4634]: Error: Could not complete SSL handshake with <n.n.n.n>: no shared cipher
nrpe[4634]: Error: This could be because you have not specified certificate or ca-certificate files

This is related to the @SECLEVEL=0 changes from NRPE 3.2.0, you need to modify the /etc/nagios/nrpe.cfg to include (this is the new default in NRPE 3.2.0):

ssl_cipher_list=ALL:!MD5:@STRENGTH:@SECLEVEL=0

After restarting the nagios-nrpe-server service you should be able to use check_nrpe <= 2.15 with nrpe >= 3.1.1.

nrpe[4686]: Connection from <n.n.n.n> port 34692
nrpe[4686]: Remote <n.n.n.n> - SSL Version: TLSv1.2
nrpe[4686]: Remote <n.n.n.n> - TLSv1.2, Cipher is ADH-AES256-GCM-SHA384
nrpe[4686]: SSL Not asking for client certification

nagios-nrpe (3.1.1-1~bpo9+3) has been uploaded to stretch-backports which sets the ssl_cipher_list, this change is only required for NRPE 3.1.x built with OpenSSL 1.1.0.

[imtek]

Set both options and also upgraded NRPE to 3.1.1-1~bpo9+2, still getting same errors (logfiles identical tho those i pasted yesterday) :(

[hedenface]

@imtek I don't have a solution for you, but I did just commit 25c9dee which will allow for better debugging when seeing those particular messages. (Both the nonssl and ssl error messages were exactly the same)

[imtek]

@hedenface thanks, hopefully that will get into a Debian package release some day. For now i cannot upgrade any of my Debian machines because of this, moving to ssl certificates is my only option i think (or i hope at least).

Ugh ...

[sebastic]

You can upgrade your Debian machines to stretch, but you'll need to disable SSL until the proposed-update is available. I've done this for all my private systems and at $DAYJOB.

I've finished testing the proposed-update to confirm that SSL support (without certificates configured) between NRPE 2.x (2.15 on Debian jessie) and NRPE 3.x (3.0.1 & 3.1.1 on Debian stretch and 3.2.0 on Debian unstable) works as expected again.

Debian Bug #867567 has been filed to request permission from the Debian stable Release Manager to upload the proposed-update.

[imtek]

@sebastic how can i disable SSL when i never configured SSL? Thanks!

Because -n is already set in de /etc/default file for NRPE.

[sebastic]

The check_nrpe command also needs to use the -n option if the NRPE daemon it connects to has SSL disabled via -n too.

[imtek]

@sebastic you are a champ, a little search and replace on my nagios box and i am up and running again on Stretch!

[imtek]

I still have a issue on CentOS 7 when applying -n on monitoring probes side, how can i disable ssl on nrpe (of the checked host) on CentOS ? on Debian i can change it in /etc/default/nagios-nrpe-server but in CentOS there is no such file?

[sebastic]

Edit /etc/sysconfig/nrpe, the file is/was sourced by the init script and configed as EnvironmentFile in the systemd service.

[sgsax]

In debugging this problem I discovered a change in /etc/default/nagios-nrpe-server (for debian/ubuntu) between v2.15 and 3.0.1. In v2.15 the variable is DAEMON_OPTS and in 3.0.1 it is NRPE_OPTS. I couldn't figure out why I was getting the SSL handshake error, even though I was specifying -n on the client side and had -n set in defaults per my previous host configuration. Once I found the new var name and changed it, the daemon ran with my preferred defaults.

[sebastic]

/etc/default/nagios-nrpe-server is a conffile managed by dpkg, after distribution upgrades you need to merge changes from the new configuration files into your own if you had modified the files (otherwise dpkg will install the new version). The following lists all the conffile that you need to inspect after an upgrade:

find /etc -name "*.dpkg-*"
find /etc -name "*.ucf-*"

Files with the .dpkg-new/.ucf-new & .dpkg-dist/.ucf-dist extension are for the new version in the upgraded package, files with the .dpkg-old/.ucf-old & .dpkg-remove/.ucf-remove are the old configuration files that can generally be removed.

RedHat/Fedora/CentOS work similiarly with upgrades where the files use the .rpmnew extension. It's the responsibility of the system administrator to merge the changes from the new configuration files and to preserve customizations.

[sebastic]

nagios-nrpe (3.0.1-3+deb9u1) with the fix for this issue has been released as part of the Debian 9.1 stable update.

The updated package also enables SSL support by default again too.

[torfsen]

@sebastic Seems to work fine over here, thanks!