Upgrading to 2024.2.2 Issue

[geeksailor]

Devs,
Just attempted to run an upgrade from 2023.45.1 to 2024.2.2 and I ran into the following Perl issue with Net-SSLeay:

duration: 2690.974
name: cpan install Digest::CRC
result: true
start_time: '10:26:18.870233
cmd_|-remnux-perl-packages-exiftool_|-cpan install Image::ExifTool_|-run:
id: remnux-perl-packages-exiftool
run_num: 824
sls: remnux.perl-packages.exiftool
changes: {}
comment: 'One or more requisite failed: remnux.perl-packages.net-ssleay.remnux-perl-packages-net-ssleay-install'
duration: 0.007
name: cpan install Image::ExifTool
result: false
--
\ 13:53:58 GMT\nRunning install for module 'Net::SSLeay'\nChecksum for /root/.cpan/sources/authors/id/C/CH/CHRISN/Net-SSLeay-1.94.tar.gz
\ ok\n'YAML' not installed, will not store persistent state\nConfiguring C/CH/CHRISN/Net-SSLeay-1.94.tar.gz
\ with Makefile.PL\nDo you want to run external tests?\nThese tests will
\ fail if you do not have network connectivity. [n] n\n*** If there are
\ build errors, test failures or run-time malfunctions,\n try to use the
\ same compiler and options to compile your OpenSSL,\n Perl, and Net::SSLeay.\n
Warning: No success on command[/usr/bin/perl Makefile.PL INSTALLDIRS=site]\n
\ CHRISN/Net-SSLeay-1.94.tar.gz\n /usr/bin/perl Makefile.PL INSTALLDIRS=site
\ -- NOT OK"
comment: Command "cpan install Net::SSLeay" run
duration: 3446.299
name: cpan install Net::SSLeay
result: false

Full Saltstack log:
saltstack.log

grep'ed down results:
errfiltered.log

I was able to do clean install on a new Ubuntu 20.04.6 (Mate) VM last night using the addon mode with no issues. Not really sure why upgrading is giving me issues with this.

when I run just cpan it runs the following:

Loading internal logger. Log::Log4perl recommended for better logging
Sorry, we have to rerun the configuration dialog for CPAN.pm due to
some missing parameters. Configuration will be written to
<</home/{user}/.cpan/CPAN/MyConfig.pm>>

CPAN.pm requires configuration, but most of it can be done automatically.
If you answer 'no' below, you will enter an interactive dialog for each
configuration option instead.

Would you like to configure as much as possible automatically? [yes]

Did something possible get reset with this package that I need to reconfigure before trying to run the upgrade again? I am not confirming any changes at this point till I can do some more research and make sure that going through these choice menus won't possibly screw up the installation process further.

EDIT: /home/{user}/.cpan/CPAN/MyConfig.pm only has 1; in it which may be a config written by the net-ssleay.sls in the salt.

Thanks and cheers!

[geeksailor]

Another thing, the following is in the CPAN MyConfig.pm for root:

$CPAN::Config = {
'applypatch' => q[],
'auto_commit' => q[0],
'build_cache' => q[100],
'build_dir' => q[/root/.cpan/build],
'build_dir_reuse' => q[0],
'build_requires_install_policy' => q[yes],
'bzip2' => q[/usr/bin/bzip2],
'cache_metadata' => q[1],
'check_sigs' => q[0],
'cleanup_after_install' => q[0],
'colorize_output' => q[0],
'commandnumber_in_prompt' => q[1],
'connect_to_internet_ok' => q[1],
'cpan_home' => q[/root/.cpan],
'ftp_passive' => q[1],
'ftp_proxy' => q[],
'getcwd' => q[cwd],
'gpg' => q[/usr/bin/gpg],
'gzip' => q[/usr/bin/gzip],
'halt_on_failure' => q[0],
'histfile' => q[/root/.cpan/histfile],
'histsize' => q[100],
'http_proxy' => q[],
'inactivity_timeout' => q[0],
'index_expire' => q[1],
'inhibit_startup_message' => q[0],
'keep_source_where' => q[/root/.cpan/sources],
'load_module_verbosity' => q[none],
'make' => q[/usr/bin/make],
'make_arg' => q[],
'make_install_arg' => q[],
'make_install_make_command' => q[/usr/bin/make],
'makepl_arg' => q[INSTALLDIRS=site],
'mbuild_arg' => q[],
'mbuild_install_arg' => q[],
'mbuild_install_build_command' => q[./Build],
'mbuildpl_arg' => q[--installdirs site],
'no_proxy' => q[],
'pager' => q[/usr/bin/less],
'patch' => q[/usr/bin/patch],
'perl5lib_verbosity' => q[none],
'prefer_external_tar' => q[1],
'prefer_installer' => q[MB],
'prefs_dir' => q[/root/.cpan/prefs],
'prerequisites_policy' => q[follow],
'recommends_policy' => q[1],
'scan_cache' => q[atstart],
'shell' => q[/bin/bash],
'show_unparsable_versions' => q[0],
'show_upload_date' => q[0],
'show_zero_versions' => q[0],
'suggests_policy' => q[0],
'tar' => q[/usr/bin/tar],
'tar_verbosity' => q[none],
'term_is_latin' => q[1],
'term_ornaments' => q[1],
'test_report' => q[0],
'trust_test_report_history' => q[0],
'unzip' => q[/usr/bin/unzip],
'urllist' => [q[http://www.cpan.org/]],
'use_prompt_default' => q[0],
'use_sqlite' => q[0],
'version_timeout' => q[15],
'wget' => q[/usr/bin/wget],
'yaml_load_code' => q[0],
'yaml_module' => q[YAML],
};
1;
END

Could something possibly overwrote the user config causing it to require reconfiguration?

[digitalsleuth]

It looks like the actual cause of the error is on line 66026:

# [ERROR   ] stderr: /usr/local/bin/openssl: error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory
Use of uninitialized value $output in scalar chomp at Makefile.PL line 425.
Use of uninitialized value $output in pattern match (m//) at Makefile.PL line 428.
Use of uninitialized value $output in pattern match (m//) at Makefile.PL line 431.
Use of uninitialized value $output in concatenation (.) or string at Makefile.PL line 440.
*** OpenSSL version test failed
    (`' has been returned)
    Either you have bogus OpenSSL or a new version has changed the version
    number format. Please inform the authors!
# [ERROR   ] retcode: 1

Could you let me know what version of openssl is installed (if any) currently in that VM?

[geeksailor]

@digitalsleuth Apt says 1.1.1f. When I try to anything with openssl like help or version it tells me:

openssl: error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory.

Guess my libssl.so.3 somehow disappeared. I'll try doing a re-install of openssl and see if that fixes it.

[digitalsleuth]

I tried the same process (brand new Ubuntu Mate 20.04.6, fully updated). Default OpenSSL for me is 1.1.1f as well, with no REMnux installed. Attempted the exiftool install with no issues.

Before re-installing, you could try running sudo ldconfig to see if that resolves the openssl issue. If it does, I can account for that in the state file.

[geeksailor]

Oops, sorry buddy I didn't wait and already am trying a reinstall now.

I also think I might have installed the latest OpenSSL (3.x) from their site or git to be more current and had the APT repo package installed as well, but I'm not sure on that since I can't get a version.

Quick question though, If I do reinstall OpenSSL but using the git to install 3.x again, do you know if REMnux has any dependencies that require 1.1.1 to operate? Just curious since 1.1.1 hit its last release back in 2018 and its LTS/EOL was back in September of 2023.

Cheers!

[geeksailor]

UPDATE 1: I tried uninstalled the APT package and trying the command again and this error still persists. So I'm not sure if there is another package somewhere that the link in /usr/local/bin is referencing that the system is looking to and it doesn't have this required file? Hopefully a clean install will fix it.

UPDATE 2: I tried running the command you suggested to see if it fixes it since OpenSSL seems to still be here somewhere while running the make and it didn't seem to do anything.

[digitalsleuth]

Did you try the "sudo ldconfig" command?

[geeksailor]

UPDATE 3: Reinstall didn't fix it but did get me the latest version at least. A quick Google-fu and found a suggestion to try sudo ldconfig /usr/local/lib64/. So I ran the command and it seems to have fixed it. and seems to be working with no libssl.so issues.

Trying another REMnux upgrade now that is fixed! Will let you know how it goes.

[geeksailor]

Oops. Don't think I did try with sudo.

[digitalsleuth]

If you're already trying again, simply try the sudo ldconfig command without /usr/local/lib64/ to see if that resolves the issue first.

[geeksailor]

UPDATE 4: REMnux upgrade failed again and I'm back to having the libssl.so issue again. I think this was because a part of the Salt install is to check for if OpenSSL is installed in the APT packages and if not it installs it. And the Git build and install method does not register in the APT packages. When I run a openssl version it show it as:

OpenSSL 3.3.0-dev (Library: OpenSSL 3.3.0-dev )

I think the APT install overrides part the Git build/install and I end up having a mishmash of both. So I guess both are here but the 3.3.0 is the dominating version.

Next thing I'm gonna try is to see if I can find a way to register the Git build/install with APT packages so REMnux see's OpenSSL is installed and doesn't need to run an APT installation. If that works than I want to try to remove 1.1.1 and try running the remnux upgrade again. If I don't end up finding a solution though and making this work, I'll just remove 3.3.0 and stick with trying an upgrade with the 1.1.1 version. But I want to see if I can keep the 3.3.0 and make REMnux work with it. Just as an experiment. Time to do some Google-fu.

Also, I checked my history. I did actually run sudo ldconfig before, I just forgot. I did try running it again with no /usr/local/lib64/ and it again seemed to do nothing and the libssl.so issue persisted. So I tried running it with the path again and the error went away. So I guess that needed.

[digitalsleuth]

I'm running some testing now, I'll let you know how it goes, hopefully it resolves this issue.

[geeksailor]

Sounds good. Also appears the latest stable version available for OpenSSL is 3.2.0 but I guess when I did a git clone for the build I made, it pulled down 3.3.0-dev. So I might pull 3.2.0 to build and install so I at least have a stable version. I like current but I'm not looking to use the dev version of it if I don't have to.

[digitalsleuth]

So, I have tried about 10 different ways to intentionally make this break, but I've been unable to duplicate the error. Are you using the ISO downloaded from the Ubuntu page, or is it one you had previously built but are installing REMnux into? If it's a previous build, there may have been an error induced by an upgrade to the SSL package. Even after the successful install, the openssl version is still 1.1.1f.

I'll manually install openssl 3+ to see the best way to account for that in the future, but in order to get everything to work during the install, I would recommend temporarily removing openssl 3, then installing it after completion (provided you want/need version 3+).

[geeksailor]

I downloaded and used the ISO from Ubuntu Mate's site to build the system I am using.
For about a week I had an issue with my box locking up on the login screen after boot before even being able to put in a password. Not sure what was the cause but I removed 3.0 from my system, tried a few tweaks and I guess something worked because it's back working again. Long story short, I'm back to just having 1.1.1f on my system for now till I can make sure it wasn't having both that might have been a part of what messed it up.
I haven't tried running an update yet since then but I'll do that this morning to see if it is working again now that I only have 1.1.1f installed.

[geeksailor]

If you take a peek over at the salt-state repo and the new issue I just opened (REMnux/salt-states#274) I guess uninstalling OpenSSL v3 fixed my issue with that. That problem doesn't seem to be giving me any more issue with the update/upgrade process. And whatever issue I was having with Radare during the upgrade/upgrade process, I seemed to have figured out a work around to make it work. So I'll close this issue out since my original issue seems to be fixed.

Thanks Digital for taking some time to see if you can figure it out!
Cheers!